🦜 Hi Team! 🦜 QQ about running K pipelines in Jenkins CI. We have pipelines with fabricated data that use the same nodes as pipelines with real data, and it would already be a great integration test to run all our fabricated pipelines after unit tests in our CI. Are these case studies / examples for how to do this, eg. how to handle the pipeline output. Also, do we need to remove the fabricated pipeline output from the catalog to keep it a MemoryDataSet for CI to access if we don't want to write to disk every time CI runs? Thanks! 🙏

Code API!

Could you elaborate a little bit? 😅

In my opinion, Kedro itself does a pretty good job of integration testing pipelines using behavior-driven tests. It's a bit more involved, but it does things like creating a new directory for the project and cleaning up that directory once tests are done. You can take a look at it in the Kedro repo, if you're interested. At a high level, if you have data you want to pass without persisting (i.e. using the code API as @datajoely suggests, and you create a

MemoryDataSet(data=my_input_data)

), or you can create a temporary folder that you use for input/output data and clean it up after your tests.

sorry I meant to come back to this