Łukasz Janiec
02/24/2025, 9:03 AMSequentialRunner
to run all of my pipelines - Copilot suggested something like this, but I cannot get it to work:
import os
from kedro.runner import SequentialRunner
from kedro.framework.session import KedroSession
def main():
# Get the parent directory of the current directory
current_directory = os.path.dirname(os.path.abspath(__file__))
project_path = os.path.dirname(current_directory)
# Initialize the Kedro session
with KedroSession.create(package_name="data_science_pipelines", project_path=project_path) as session:
context = session.load_context()
# Initialize the SequentialRunner
runner = SequentialRunner()
# Get all pipeline names
pipelines = context.pipelines
# Start all pipelines
for pipeline_name, pipeline in pipelines.items():
print(f"Starting pipeline: {pipeline_name}")
runner.run(pipeline, context.catalog)
print(f"Finished pipeline: {pipeline_name}")
if __name__ == "__main__":
main()
Hall
02/24/2025, 9:03 AMŁukasz Janiec
02/24/2025, 9:04 AMpipelines
fieldLorenzo Castellino
02/24/2025, 9:12 AMproject_path = Path.cwd()
bootstrap_project(project_path)
with KedroSession.create(project_path=project_path) as session:
pipelines = find_pipelines()
Mind you I am not sure this is the best/correct way to do it, maybe the real Kedro experts can help us both in this regardŁukasz Janiec
02/24/2025, 9:45 AMŁukasz Janiec
02/24/2025, 9:53 AMfind_pipelines
?
EDIT: Got it, from kedro.framework.project import find_pipelines
Nok Lam Chan
02/25/2025, 2:21 PMŁukasz Janiec
03/11/2025, 1:53 PMNok Lam Chan
03/11/2025, 1:58 PMpipelines["everything"] = pipelines["train"] + pipelines["test"] + pipelines["detection"]
when you need to run everything, do kedro run -p everthing
, and when you only need a subset of it, do kedro run -p detection