I have a pipeline with two different data processi...
# questions
m
I have a pipeline with two different data processing pipelines, I would like to control which branch is run using parameters.yml. Both pipelines are part of the default pipeline. Is there a way I can use the
before_pipeline_run
hook to prevent the pipeline from running? The logic is working in the hook to know which pipeline I don't want to run but I can't figure out a way to prevent the pipeline from running.
n
Not directly answering your question. Alternative is having them as two separate pipeline registry and do
kedro run --pipeline a
and
kedro run --pipeline b
instead of injecting dynamic behavior
m
Thanks for the suggestion! I'm sure that's the best way to do it. I had also tried skipping individual nodes in
before_node_run
, which I was able to get to not run nodes but was still loading the data for unused nodes.
m
What about tags? Do you really have to use
parameters.yml
?