Is there any workaround to enable a node to have the same dataset as input and output?

Can you elaborate why you want to do that? This is not possible by design to prevent circular dependencies.

Yep sure

I see, then your pipeline should starts with the altered JSON, it doesn’t have to be the same dataset from what I understand here. What’s the problem to keep them as separate dataset? If you don’t need to trigger the steps of editing json you can simply use tag or separate it out from the default pipeline and only run it when necessary

Hi Elior, This is a common need for me in some pipelines. Kedro only cares that you do not have the same dataset name in a circular dependency, but there is nothing preventing you from having dataset_a: filepath: same_path.json dataset_b: filepath: same_path.json Kedro will treat those as two separate datasets. But you should be aware that this can lead to unexpected behavior

god that's so straightforward I don't know how I couldn't think of this myself facepalming

i had a similar lightbulb moment at some point when i first encountered this need hahaha. to avoid repeating yourself in yaml config you can use yaml anchoring

Just to add to the discussion. Although the proposed workarounds are solid. Be aware that when you overwrite an existing dataset, your pipelines are no longer idempotent (running it twice produces the same result). You could achieve a similar result by just versioning your dataset somehow (input

v

, output

v+1

) where the version number v is used in the filepath and supplied at runtime (e.g as an env var). This way, your pipelines remain idempotent

that is a very very good point. in our usecase these datasets are append-only tables which helps preserve idempotency