Hi, is there someone using kedro with snowpark/snowflake ? What is the best way to implement generic and custom data tests? I have the following options so far • great expecation ◦ (+) Is already integrated in kedro ◦ (+) Is integrated with snowfalke ◦ (+) covers generic and custom checks • pandera ◦ (+) Is already integrated in kedro ◦ (-) Is more for pandas than snowpark ◦ (+) covers generic and custom checks • data contracts ◦ (-) Is not integrated in kedro ◦ (+) DB oriented ◦ (+) covers generic and custom checks • custom implementation ◦ (+) less dependencies ◦ (-) More maintenance What are. your thoughts?

Pandera supports more than pandas, including backends for Spark, Polars, and Ibis. Narwhals-backed backends will also be available in the next release (very soon 🤞). I haven't tried it, but found Snowpark Checkpoints Validators work via Pandera schemas: https://docs.snowflake.com/en/developer-guide/snowpark/python/checkpoints-validators Could be worth looking into (I think they execute lazily, but haven't tested myself). May also be of interest: https://github.com/kedro-org/kedro/discussions/5602 cc @Sajid Alam In the end, between GE and Pandera, you need to decide whether you want a data validation framework (put it front and center) or lightweight library (you're able to validate data frames, but you don't want a lot of machinery to enable it). (Disclosure: I am a Pandera maintainer, having implemented most of the Ibis and Narwhals-backed backends)

Thanks for the fast response and the mentioned points. I totally see the advantages of pandera as a light weigth validator. My data test's are against the backend which is snowflake. In a former architectual decision we decided on snowpark and not ibis. Therefore, if I would choose Pandera I would have the following options • Use modin ◦ I spoced modin and neglected it because i run into to many issues. In specific transformations I moved to pandas locally • Use Ibis ◦ I would have a mixture of different dataframes ( snowpark,ibis,pandas) which I try to avoid since they all have similar purpose and introducing all of them increases the extrinsic load for the dev. • Write snowpark backend for Pandera ◦ The additional value would be minimal since there is already Ibis

Did you look into the Snowpark Checkpoints Validators stuff? I think that would be the best (or only) option if you want to go with Pandera with PySpark. If somebody were to support Snowpark in Narwhals (see https://github.com/narwhals-dev/narwhals/issues/1419), then that might provide an easy path forward to do so in Pandera, but I don't think that's trivial based on the issue discussion.

Hey, I'm not really a fan of any of the available options. I'm now trying to implement a light data test framework myself. BUT. I cant really think of a nice solution. My requirements are • The data test has to happen before the end of the transaction. ◦ The reason is, that I want to be able to apply test to the whole dataset and abort the transaction in case of an error. • The data test has to happen after all the transformation eg. 1. I built a registry and register all methods with a decorator a. Pretty ugly because the scope is super big and i'ts implicit 2. I reference the path to the test in the yaml and add the possibility to add data_test to the data_set a. Pretty ugly because I lose all the casting and I'm wandering arround in the dir looking for a module which I load during runtime Is there an option that I overlooked?