https://kedro.org/ logo
#questions
Title
# questions
z

Zubin Roy

03/19/2024, 11:12 AM
Hi All. I have a question about modular pipelines. Is there a way to setup modular pipelines (where you have multiple modular pipelines created) to run either a specific modular pipeline or multiple (but not all) through the
kedro run --pipeline
command. The reason for this is we have a codebase where we may want to run all our modular pipelines or may want to run one or multiple depending on our requirements. I came up with a solution using parameters (shown below) but this feels slightly hacky to me and was wondering if there was a better way to do this either through the kedro run commands or something else. Thanks!
# loads context parameters
config_loader = ConfigLoader("conf", env="s3")
params = config_loader.get("parameters*")["odds_portal_run_arguments"]
# functionality built like this to allow for a single or multiple modular pipelines to be run
modular_pipelines = []
if params["run_rugby_pipeline"]:
modular_pipelines.append(extract_data_pipeline)
modular_pipelines.append(clean_and_merge_pipeline)
if params["run_sa_premier_league_pipeline"]:
modular_pipelines.append(south_africa_prem_pipeline)
# kedro pipeline cannot be treated like a list object and so has to be added through this method like a single object for multiple
# pipelines to be run at the same time
for pipe_no in range(len(modular_pipelines)):
if pipe_no == 0:
all_pipes = modular_pipelines[0]
else:
all_pipes += modular_pipelines[pipe_no]
return all_pipes
m

marrrcin

03/19/2024, 11:15 AM
Use either namespaces or tags to run specific pipelines / parts of those pipelines:
kedro run --pipeline=your_pipeline --namespace=one_of_the_modular_pipelines
or
kedro run --pipeline=your_pipeline --tags some_tag
or both. Keep in mind that you have to construct the pipelines "the right way" - which is - assigning the namespaces and tags as per your requirements. Seems like you're loading the
ConfigLoader
on your own and I sense some anti-pattern here and you're trying to "hack" Kedro.
this 1
z

Zubin Roy

03/19/2024, 11:27 AM
Thanks @marrrcin. Can you chain tags? e.g. run multiple tags
m

marrrcin

03/19/2024, 11:30 AM
Yes
z

Zubin Roy

03/19/2024, 11:33 AM
Cool. Thank you!