Hi kedroids In my pipeline I have this logic for 2 nodes nod

Question

Hi kedroids In my pipeline I have this logic for 2 nodes node 1 reads a table and executes a data process only to new items that are not in the table node 2 Executes transformations to those new items and append them to the same table I m getting this error ```CircularDependencyError Circular dependencies exist among these items node1 node2 ``` Yes the output of node 2 is an input for node 1 My goal is to not process all the items every time I run the pipeline but only the new items not in that table How can I do this Thanks slightly smiling face

datajoely · Answer

so you re in a funny place where Kedro s design decisions will fight this we ve built things to guarantee reproducibility and this sort of out of Kedro DAG operation is hard to do

datajoely · Answer

If you really want to do this potentially kedro hooks allow you to set things up <https docs kedro org en stable hooks introduction html|before pipeline run>

Nok Lam Chan · Answer

Any chance you can use IncrementaDataSet for this

Jose Nuñez · Answer

~I don t suggest you doing this but gonna say it anyways it may work as a workaround for now ~ ~instead of doing ~ ~All DATA gt ~ ~ ~ ~ gt ~ ~ Current Data gt ~

Jose Nuñez · Answer

Try this instead ONLY as a workaround catalog yaml actual data filepath 01 raw actual data csv actual data updated filepath 02 intermediate actual data updated csv nodes py Node1 1 Reads lt all data gt and compares it to lt actual data gt 2 calculate lt new records gt 3 do the transformations on lt new records gt 4 Do some magic to avoid circular dependency 5 return the new lt actual data updated gt The trick on step 4 is to manually save <http pd to|pd to> csv the lt new data updated gt into the same path as the lt actual data gt filepath 01 raw actual data csv and without stating this on the pipeline py so that kedro doesn t see the dependency problem You can do this as a workaround but this is a very very bad practice if you are on a hurry go ahead and if not is best if you plan your pipeline better

Sebastian Cardona Lozano · Answer

Thanks all for your ideas slightly smiling face