Hello kedro Team!
Can we have use the same output for multiple nodes ? I have asynchronous kedro pipelines updating specific columns of the same dataset, at different frequencies. If not, any other approaches to consider ?
01/22/2023, 4:21 PM
So Kedro enforces an acyclic execution graph - so you won’t be able to create ‘cycles’ by definition. Conditional logic is a bit difficult in Kedro due to the way we want to encourage reproducibility.
Perhaps the way to do it is to manage the different frequencies into separate registered pipelines so you can execute via
kedro run --pipeline name_1
and then have a join step which will pick up the different slices irrespective of how they were produced.