Hello channel :wave: I have two Kedro pipelines, ...
# questions
Hello channel 👋 I have two Kedro pipelines, say TrainA and TrainB. The pipelines are quite similar in that • most node pairs are identical, i.e. they use the same function and the same input/output configuration • some nodes use the same function and their input/output configuration differs only by the parameters that are read in. Right now, the two pipelines live in two different Python modules, namely tr`train_a.py` and
. My idea is to separate the process from the product, i.e. creating a general pipeline for training that can be used to train both A and B given their respective configuration parameters—they can possibly live in their own
so to avoid override and/or conflict. The two questions related are: 1. does this harmonise with the Kedro principles? 2. is this possible in Kedro? Thank you. 🙏
I think this is a perfect use-case for our namespaced pipelines construct (also known as modular pipelines) as you can
🤩 1
đź‘Ť 1
if you provide namespaces you also get the nice big boxes on Kedro-Viz (see demo.kedro.org)
Thank you! I will try straight away and I will follow up in this thread. 🤩
@datajoely I gave it a try, created a namespaced modular pipeline, but there is one thing I don't get. Suppose I have my general
pipeline instead of
. However, I still have two different parameter configuration YAML files that live respectively inside
. How can I call the training pipeline twice, each time with one specific configuration? I hope it makes sense.
so the idea of this pattern is that the code is static, but it possible to override the catalog or parameter information sort of like a sub-class
yes, this rephrasing is correct—the code is made in such a way that when you call e.g.
inside one node, both configuration YAML file will be passing the required parameter.
@datajoely I was following up on this thread, as I have not solved it yet. Did you have a chance to think about it? 🙂
can you not use the same parameter key twice?
I am afraid I don't fully understand, can you elaborate on that?