Vassilis Kalofolias02/07/2023, 3:15 PM
instead of using the
of jupyter. For example I would like to run the same pipeline in a loop with different partitions of a
and I find it weird to call
in a loop. Is this discouraged practice? What is the benefit of having a session in jupyter if you develop interactively? (related but not answering my question: https://kedro-org.slack.com/archives/C03RKP2LW64/p1668423931294329) Thanks a lot!
datajoely02/07/2023, 3:17 PM
once you’ve moved your jupyter prototypes into your codebase
Vassilis Kalofolias02/07/2023, 3:28 PM
Nok Lam Chan02/07/2023, 3:40 PM
or it’s saved on disk (use catalog.load(“xxx”)` to retrieve it. After the code is developed, you can wrap the cell as a function and put it back as a node into the pipeline.
a dataset with type PartitionedDataSet, it should return an iterable and you can simply run a for loop on it, the same as what you would do in a node.
Vassilis Kalofolias02/07/2023, 4:00 PM
, right? I mean this scenario:
pipeline_1 -> new_node -> pipeline_2
Nok Lam Chan02/09/2023, 5:41 AM