Hi everyone! I’d like to read existing partitions and write new partitions to the same PartitionedDataSet in a single Pipeline. However, with a single DataCatalog entry this creates a CircularDependencyError. What is the proper way to handle such situations in Kedro? I can create identical Catalog entries with different names, however it doesn’t feel like the correct solution for this problem.
01/09/2023, 2:29 PM
I think you've more-or-less got the "solution"; honestly, this is a circular operation if you're writing to the same