Hi Team, can I add an optional node to pipeline, I...
# questions
Hi Team, can I add an optional node to pipeline, It should be executed only if specific parameter is set?
You can do a
if the parameter value is null for example, but the pipeline needs to be static.
Will that create problem while running?, like if a parameter is set to
, I want to run the node and get a dataframe for example and if its
, I want to move to next node without any output
I am confused about the catalog implementation for this node
If you plan to output something from this node, then it will show some challenges
E.g. you cannot return
for the data catalog
Why do you need such behaviour? Maybe you should implement this logic within as single node, rather than splitting them
This is one of our teams questions regarding kedro, I am not exactly aware of why they need this
When it comes to Kedro itself, pipelines need to be static DAGs. You can work-around some of the issues (like here, with the pseudo-conditional node, but it’s not natively supported right now).
Got it, thanks
I think @marrrcin is right here. To supplement, a static DAG is preferred due to simplicity and reproducibility. If you think about pipeline visualisation, when you have multiple switches within nodes, it is very difficult to reason the execution logic (it is almost like a hidden graph within your node). Especially when you have a large computation graph. So even when you can do this, it’s still best to avoid this. We do recognize in some cases it’s unavoidable. As of the moment, the approach suggested by @marrrcin is quite common