Hi Team, also is there a way to drop certain data ...
# questions
Hi Team, also is there a way to drop certain data frames from pipelines which arent required anymore without defining them in catalogs, my assumption is that if you dont define in catalogs it continues to stay in Memorydataset, right?
Are they not required because an experiment has been abandoned?
Lets say for example I have a spark dataframe and I am converting it to pandas using transcoding for further nodes, the spark dataframe will be no longer needed, if I save it using catalog it will eat up my disk space and if I dont save it, it will continue to stay in my MemoryDataSet covering memory
oh in that sense delete the catalog entry and the python GC will deal with it
if it’s spark it’s likely not even your memory, it’s the clusters memory and it’s lazily evaluated so it should have no impact there!
How are you transcoding but at the same time it's a
And `MemoryDataSet`'s are automatically cleaned up, and even released as soon as they're no longer needed by the pipeline (i.e. usually sooner than end of pipeline).
Yes I believe this is managed by Kedro, essentially the
class has its own mini GC mechanism to track if a dataset is still needed.