We do use kedro with snowflake. We did write a custom dataset to handle all the data engineering part like change data capture. At the moment we are looking for an observability solution. Does someone have expirience with a monitoring/observability solution? The use case is that we can monitor the data processed over time and the number of successfull data tests. The solutions, we have in mind are • hosting kedro-viz/ml flow and adjusting the capabilites • Introduce opentelemetry, a backend and graphana Best regards

OpenLineage is probably a better fit than OpenTelemetry; it is very much inspired by OpenTelemetry, but it's built for the Data world (as opposed to the Services world): openlineage.io/blog/openlineage-takes-inspiration-from-opentelemetry github.com/kedro-org/kedro/discussions/5728 is timely and may be of interest to you—@Ravi Kumar Pilla and @Sajid Alam leading this work, but I feel like design partners for these things never hurt. 🙂 (They could better say whether at a stage where looking for that soon.)

Thank you @Deepyaman Datta Hi @Ralf Kowatsch, It would help us to get your thoughts on github.com/kedro-org/kedro/discussions/5728 as we shape data lineage on Kedro. Let me know if you would like to get on a call to discuss more on this. cc: @Sajid Alam Thank you