Yong Bang Xiang
09/12/2023, 10:25 AMkedro run --pipeline=data_science
, but how could i run two or three instead of one (nor all)? in particular, i'm looking to do something like kedro run --pipeline=data_science+evaluation
where i run these two selected pipelines onlymarrrcin
09/12/2023, 10:30 AMregister_pipelines
. You can use https://docs.python.org/3/library/itertools.html#itertools.combinations for that if you have a lot of pipelines.datajoely
09/12/2023, 10:51 AMkedro run --pipeline=data_science & kedro run --pipeline=data_science+evaluation
run in parallelkedro run --pipeline=data_science && kedro run --pipeline=data_science+evaluation
run in sequenceNok Lam Chan
09/12/2023, 10:52 AMpipeline_data_science
+ pipeline_eavluation
+
or -
quite easilyJuan Luis
09/12/2023, 1:55 PMdef register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
A mapping from pipeline names to ``Pipeline`` objects.
"""
pipelines = find_pipelines()
pipelines["__default__"] = sum(pipelines.values())
pipelines["except-train"] = ???
return pipelines
Nok Lam Chan
09/12/2023, 2:00 PMingest
, process
, train
,eval
You can have
pipelines["all"] = ingest + process + train + eval
pipelines["all_except_eval"] = pipelines["all"] - eval
Juan Luis
09/12/2023, 2:26 PMfrom .pipelines.model_training import create_pipeline as create_model_training_pipeline
...
pipelines["all"] = sum(pipelines.values())
pipelines["all_except_eval"] = pipelines["all"] - create_model_training_pipeline()
is there a better way of doing it? not sure what eval
is in your example @Nok Lam ChanNok Lam Chan
09/12/2023, 2:28 PMPipeline
object - your snippets should work. But more likely you would structure your pipeline in modular pipeline fashion, so you have pipelines/eval/
, find_pipelines
should take care of it and you can just get it from pipelines["eval"]
Juan Luis
09/12/2023, 2:48 PMpipelines = find_pipelines()
, when pipelines["eval"]
will be my object, gotcha ๐๐ผYong Bang Xiang
09/15/2023, 8:00 AM