Hi All. I have a question about modular pipelines. Is there a way to setup modular pipelines (where you have multiple modular pipelines created) to run either a specific modular pipeline or multiple (but not all) through the

kedro run --pipeline

command. The reason for this is we have a codebase where we may want to run all our modular pipelines or may want to run one or multiple depending on our requirements. I came up with a solution using parameters (shown below) but this feels slightly hacky to me and was wondering if there was a better way to do this either through the kedro run commands or something else. Thanks!

# loads context parameters

config_loader = ConfigLoader("conf", env="s3")

params = config_loader.get("parameters*")["odds_portal_run_arguments"]

# functionality built like this to allow for a single or multiple modular pipelines to be run

modular_pipelines = []

if params["run_rugby_pipeline"]:

modular_pipelines.append(extract_data_pipeline)

modular_pipelines.append(clean_and_merge_pipeline)

if params["run_sa_premier_league_pipeline"]:

modular_pipelines.append(south_africa_prem_pipeline)

# kedro pipeline cannot be treated like a list object and so has to be added through this method like a single object for multiple

# pipelines to be run at the same time

for pipe_no in range(len(modular_pipelines)):

if pipe_no == 0:

all_pipes = modular_pipelines[0]

else:

all_pipes += modular_pipelines[pipe_no]

return all_pipes

Use either namespaces or tags to run specific pipelines / parts of those pipelines:

kedro run --pipeline=your_pipeline --namespace=one_of_the_modular_pipelines

or

kedro run --pipeline=your_pipeline --tags some_tag

or both. Keep in mind that you have to construct the pipelines "the right way" - which is - assigning the namespaces and tags as per your requirements. Seems like you're loading the

ConfigLoader

on your own and I sense some anti-pattern here and you're trying to "hack" Kedro.

Thanks @marrrcin. Can you chain tags? e.g. run multiple tags

Yes

Cool. Thank you!