https://kedro.org/ logo
#questions
Title
# questions
p

Paul Mora

01/10/2024, 8:28 AM
Hey team, hope you are all doing well! We are currently working with dynamic kedro pipelines through the usage of OmegaConfigLoader. We are basically following this article as the general guideline. In case other people have adopted that already: Our question is how would one trigger only one pipeline using the CLI. E.g. one has like 10 pipelines in the variants but would like to trigger only one of those ten through a simple command in CLI instead of removing the other 9 variants and run the general command. Does that question make sense? CC: @Kasper Janehag
āœ… 1
m

Merel

01/10/2024, 2:37 PM
Maybe @marrrcin can help here? šŸ™‚
šŸ’™ 1
m

marrrcin

01/11/2024, 8:54 AM
@Paul Mora Iā€™m the author of this article, happy to help! šŸ™‚ Take a look at the Running the dynamic pipelines in Kedro paragraph in the article: > ā€¢ Run everything for one of the namespaces, e.g. `kedro run --namespace price_predictor`: this will run the
data_processing
pipeline once and all variants of the
data_science
pipeline: base, candidate1, candidate2 etc. > ā€¢ Run only the single variant:
kedro run --namespace price_predictor.candidate1
will only run the
data_science
pipeline in the
candidate1
variant, skipping the rest. This requires the
data_processing
pipeline to be run first in order to consume input data. > ā€¢ Run only the single variant and also run the
data_processing
pipeline:
kedro run --namespace price_predictor --tags candidate1
> So depending on your configuration, you either run with
--namespace
and/or
--tags
.
šŸ’™ 1
p

Paul Mora

01/11/2024, 9:11 AM
Thanks a lot guys šŸ™‚ very clear
Hey @marrrcin I have a quick follow-up question if that is okay šŸ™‚ In case you would for example run 10 of those pipelines with the dynamic config loader. And we then assume that we get 10 different output files - how would we best place a consolidation function that e.g. concatenates the 10 output files? Of course we could run two separate pipelines one after another in which the first creates the files and the second consolidates, but would it be also possible to do it within one step?
p

Paul Mora

01/12/2024, 11:09 AM
Glorious - thanks @marrrcin you kedro god!
šŸ˜Ž 2
šŸ™‚ 3