Hello everyone 😄 I am trying to integrate some conditional flow in my kedro pipeline. My first node checks if a certain file is present or not. If that file is present then certain nodes are supposed to run and if it isn't, then another set is supposed to run. One thing I could do was use tags but that would require running the check before calling the pipeline. But is it possible to use a value which is checked at runtime and not use the tags on nodes? Is it possible to use hooks to implement this? Any help is appreciated!

Hi, Kedro doesn't support dynamic pipelines but you can implement your custom logic using hooks which allow you to inject custom logic at different points in the pipeline run. You can create a custom hook that checks the file's existence and modifies the pipeline before it is run. https://docs.kedro.org/en/stable/hooks/introduction.html

Hello, sorry I missed this. Thank you for your help 🙂 How exactly can I modify the pipeline? I thought of 3 things - • Check if file exists with before_pipeline_run hook, change the value of a flag accordingly. In the pipeline definition, check the value of this flag and return the required pipeline. I wasn't sure at which point the hook runs - will it be before or after the flag is checked in the pipeline definition. • Change the default pipeline in pipeline_registry based on the hook - not sure if this is possible • Use the before_node_run hook - check if the file exists before each of the required nodes. Problem with this is that I don't want the pipeline to stop execution based on the condition, instead move on to the next node. Not sure if any of these can actually solve the problem. It would be great if you could give me some direction