Hi everyone! I am new to Kedro and I am wondering how we can run pipelines conditionally in a scalable way. For example let's say i have 4 pipelines a,b,c,d. First we run a, then upon its values we either run b or c->d. Parallel question can we finish the run based on the output of a pipeline?

Hey, welcome!

For now I am learning, but the example I am working on requires me to: 1 - Load the data 2 - Classify the data 3- Depending on classification apply post-processing a or b. I know i could create a single pipeline for the two post processing but it doesnt sound scalable, if after the classification i want to apply 10 nodes for example

you can trigger the Kedro pipelines conditionally though! pseudocode:

kedro run -p load_and_classify

if GET(<s3://results>)["post_processing"] == "a":
    kedro run -p post_processing_a
else:
    kedro run -p post_processing_b

Interesting! So for example I could change the run.py script?

What is the run.py script?

kedro run

I think the idea is that, you don't need to modify Kedro to do conditional. Instead, you apply the conditional logics outside of Kedro. In fact this is very reasonable as you scale out, it's likely you will split out your pipeline into multipart, expose them through an API or services. It could be as simple as a script like this:

if result == "a":
   trigger_kedro_pipeline_a
if result == "b":
   trigger_kedro_pipeline_b

Within each Kedro pipeline, it is deterministic.

what if I want to vizualize it though with the kedro viz? and visualize all the output data from the nodes that were triggered

You can visualise part of the pipeline, but you won't have a single view that stitch up the entire run, as you triggers multiple kedro run.

very clear response, thank you so much guys, cant wait to work on real projects with it

So there is two main way of running Kedro: 1. CLI, i.e.

kedro run

2. Python API, which you usually create a

session

first and then do a

session.run

(it's what

kedro run

do behind the scene anyway)

Sorry to ask that much but seems very interesting, would you have an example script? you can write some dummy variables names like pipeline_a, i will understand

let me create an example, probably useful for others later too.

you're very nice, thank you so much

Sorry got distracted for a bit. You can find it here now, see the README instruction. https://github.com/noklam/kedro-example/tree/master/conditional-kedro-runs

Thank you so much very helpful!

There is an alternative approach that allows you to do everything in kedro, with a custom hook. But it only works in specific cases. We have a very specific use-case where we have to skip some nodes. In the end, we created a dedicated

kedro_env

with a specific global arg that is used in a before pipeline run hook that replaces the nodes to skip with a dummy node. It’s not really conditional based on node output like your example, but just wanted to share what’s possible

could you elaborate on the method/use-case?

@Noah Sarfati If datasets are not in the result, you need to load it with

catalog.load(<dataset_name>)

. This is because during kedro run we try to optimise memory and throw away dataset as soon as they are not needed.

In another entry I asked about a turn around for an issue: I'm facing troubles because the Kedro pipeline is running in a Google App Engine, and

kedro run

tries to read pyproject.toml, but in App Engine the filesystem is in read-only mode. So, I was perusing the documentation and it looks like if creating the session with the argument save_on_close=False could solve it. Make it sense?

Are you using the session at all? You can check your settings.py. By default if you are not using viz it shouldn't write to anything.

I've just tried in local with from kedro.framework.session import KedroSession from kedro.framework.startup import bootstrap_project from pathlib import Path bootstrap_project(Path.cwd()) with KedroSession.create(save_on_close=False) as session: session.run() Following this link: https://docs.kedro.org/en/stable/kedro_project_setup/session.html Thanks for the hint. I'm not using viz, because I'm just running a pipeline in Google App Engine. Even if the arisen error ("ERROR Failed to read the file: plugin.py:111 /workspace/pyproject.toml. [Errno 30] Read-only file system: '/workspace/pyproject.toml' ") doesn't stop the execution, I want to fix it, and overall to understand it well.

@Oscar Villa what version of

kedro

and

kedro-telemetry

packages you have? (if

kedro-telemetry

is not installed in GAE, please say so too)

Hi, @Juan Luis. Thanks for taking the time. I deleted the App Engine service completely and now I don't have a way to be sure of the version I used that time. I'm going to try again and be back with the packages versions and detailed errors.

thanks @Oscar Villa , I suppose you don’t have a full traceback? also, Kedro-telemetry is not installed?

I was editing/updating the message while App Engine loads...

okay, I suspect that

kedro-telemetry

is trying to modify your

pyproject.toml

and then you observe that

Read-only file system

error cc @Elena Khaustova

Yes, it was the issue: if I remove telemetry from the requirements.txt, the error disappears. Thank you a lot, @Juan Luis for your support. I can go ahead with Kedro as our DS framework in a clean fashion 🚀.

opened https://github.com/kedro-org/kedro-plugins/issues/781 to track this, gracias @Oscar Villa!