I was wondering if it's possible to create a pipeline with nodes having different requirements? E.g. node_1 uses numpy==2.0 and node_2 uses numpy==1.0. With Kubeflow pipelines each node can be supplied with a custom docker image with different dependencies. I was reading about micro-packaging in Kedro but it would seem that after importing a packaged pipeline all requirements would be merged (from docs - Kedro will also parse any requirements packaged with the micro-package and add them to project level

<http://requirements.in|requirements.in>

). Is there an automatic way of combining two different pipelines with different and separate requirements when converting a Kedro pipeline into a workflow orchestration pipeline (i.e. Kubeflow Pipeline, Prefect, Airflow, etc) ?

hi @Sergey S, indeed the closest we have is micro-packaging. I'm not aware of any solution to combine different pipelines, most plugins are built around projects and not micro-pipelines as far as I understand. cc @datajoely @marrrcin

This problem is also covered in the MLOps synthesis https://github.com/kedro-org/kedro/issues/3094 We have some internal tooling for this sort of problem, but it’s not yet open source. We have observed people maintain multiple concurrent Kedro projects as a solution here, but it’s far from ideal.

Thanks @Juan Luis I have read in the docs that Kedro does allow per pipeline requirements. So I could have two pipelines in the project, one having numpy==2.0 and the other with numpy==1.0. However running locally with that set up will not work, since locally Kedro uses a single python environment. But it seems in theory possible doing that when converting Kedro pipelines to workflow orchestration steps (e.g. with Kubeflow) since each pipeline and each docker image would be supplied with pipeline specific requirements.txt

indeed:

• A modular pipeline might have external dependencies specified in a local

requirements.txt

file.

• Pipeline specific dependencies are scooped up during the micro-packaging process.

https://docs.kedro.org/en/stable/nodes_and_pipelines/modular_pipelines.html#providing-modular-pipeline-specific-dependencies we have a pending task to revisit our micro-pipeline flow https://github.com/kedro-org/kedro/milestone/21 we'd love to hear your thoughts when you have more experience with it @Sergey S