Vassilis Kalofolias
02/07/2023, 3:15 PMSequentialRunner
instead of using the session
of jupyter.
For example I would like to run the same pipeline in a loop with different partitions of a PartitionedDataSet
and I find it weird to call %reload_ext kedro.ipython
in a loop.
Is this discouraged practice? What is the benefit of having a session in jupyter if you develop interactively? (related but not answering my question: https://kedro-org.slack.com/archives/C03RKP2LW64/p1668423931294329)
Thanks a lot!datajoely
02/07/2023, 3:17 PMkedro run
once you’ve moved your jupyter prototypes into your codebaseVassilis Kalofolias
02/07/2023, 3:28 PMNok Lam Chan
02/07/2023, 3:40 PMsession.run()
or it’s saved on disk (use catalog.load(“xxx”)` to retrieve it.
After the code is developed, you can wrap the cell as a function and put it back as a node into the pipeline.catalog.load()
a dataset with type PartitionedDataSet, it should return an iterable and you can simply run a for loop on it, the same as what you would do in a node.Vassilis Kalofolias
02/07/2023, 4:00 PMSequentialRunner.run()
outside the session
, right?
I mean this scenario:
pipeline_1 -> new_node -> pipeline_2
Nok Lam Chan
02/09/2023, 5:41 AM