Hello, sorry I missed this. Thank you for your help š How exactly can I modify the pipeline? I thought of 3 things -
⢠Check if file exists with before_pipeline_run hook, change the value of a flag accordingly. In the pipeline definition, check the value of this flag and return the required pipeline. I wasn't sure at which point the hook runs - will it be before or after the flag is checked in the pipeline definition.
⢠Change the default pipeline in pipeline_registry based on the hook - not sure if this is possible
⢠Use the before_node_run hook - check if the file exists before each of the required nodes. Problem with this is that I don't want the pipeline to stop execution based on the condition, instead move on to the next node.
Not sure if any of these can actually solve the problem. It would be great if you could give me some direction