Hey team. Looking into some advice or insights on how to think about unit testing complex nodes in kedro (or rather nodes taking in complex data with a lot of edge cases). In these cases I usually follow the approach of integrating a lot functionality into a single node, composed of several smaller private functions. My question: How to best test the node's actual output (standard stuff like

column a shouldn't have any nulls

,

column b should never be lower than 10

))? • I feel like it would be impossible to create dummy data to account for all edge cases in the test function itself • Reading from the production input table, on the other hand, defeats the purpose of unit testing. • Does it make sense to generate synthetic or sample data from the input tables to the node and store it somewhere to be read at testing time?

Someone will reply to you shortly. In the meantime, this might help:

Hi Pedro, I think it would be beneficial to separate them into two different cases. Unit testing and data validations are quite different in terms of tooling.

Hi! Check out

pandera

. That library allows you to define all conditions you want to enforce for your dataset, and validate any dataset, in your example I guess the output of that function, against them. Pandera can also generate synthetic data automatically, based on schema definition. This is literally what it’s designed for: > column a shouldn't have any nulls, column b should never be lower than 10))

Thanks guys! • @Nok Lam Chan i would say it's the complexity of the inputs (several columns, billions of rows which hide several edge cases, several input tables in some cases), rather than the complexity of the node affecting the way tests can be thought of here. The problem applies to the smaller private functions in the node, because inputs/outputs are also complex. I'm struggling to declare sample inputs and outputs in the test that are not ridiculously extensive and complex • @Yury Fedotov good point, I have been using pandera for a while too, but that's intended for runtime checks I believe - i.e., if a given contract is not met, you can either stop execution or handle it gracefully. But here I am interested in testing this at CI/CD time

column a shouldn't have any nulls

,

column b should never be lower than 10

))?

this sounds like property-based testing to me! https://hypothesis.readthedocs.io/en/latest/quickstart.html

@Nishant Rathi here

thank you Pedro!