https://kedro.org/ logo
#questions
Title
# questions
b

Balazs Konig

01/19/2023, 10:55 AM
🦜 Hi Team! 🦜 QQ about running K pipelines in Jenkins CI. We have pipelines with fabricated data that use the same nodes as pipelines with real data, and it would already be a great integration test to run all our fabricated pipelines after unit tests in our CI. Are these case studies / examples for how to do this, eg. how to handle the pipeline output. Also, do we need to remove the fabricated pipeline output from the catalog to keep it a MemoryDataSet for CI to access if we don't want to write to disk every time CI runs? Thanks! 🙏
d

datajoely

01/19/2023, 10:56 AM
Code API!
b

Balazs Konig

01/19/2023, 10:57 AM
Could you elaborate a little bit? 😅
d

Deepyaman Datta

01/19/2023, 12:09 PM
In my opinion, Kedro itself does a pretty good job of integration testing pipelines using behavior-driven tests. It's a bit more involved, but it does things like creating a new directory for the project and cleaning up that directory once tests are done. You can take a look at it in the Kedro repo, if you're interested. At a high level, if you have data you want to pass without persisting (i.e. using the code API as @datajoely suggests, and you create a
MemoryDataSet(data=my_input_data)
), or you can create a temporary folder that you use for input/output data and clean it up after your tests.
d

datajoely

01/19/2023, 12:09 PM
sorry I meant to come back to this
some examples here
2 Views