Hi Team, also is there a way to drop certain data frames from pipelines which arent required anymore without defining them in catalogs, my assumption is that if you dont define in catalogs it continues to stay in Memorydataset, right?

Are they not required because an experiment has been abandoned?

Lets say for example I have a spark dataframe and I am converting it to pandas using transcoding for further nodes, the spark dataframe will be no longer needed, if I save it using catalog it will eat up my disk space and if I dont save it, it will continue to stay in my MemoryDataSet covering memory

oh in that sense delete the catalog entry and the python GC will deal with it

How are you transcoding but at the same time it's a

MemoryDataSet

?

Yes I believe this is managed by Kedro, essentially the

Runner

class has its own mini GC mechanism to track if a dataset is still needed.