Hi everyone!, I'm rather new exploring Kedro, and I would appreciate the help on a (probably silly) problem I'm facing. It is an error I'm getting when working with KedroSession. I am trying to run the following code, which works fine until

session.run()

, without the part below it.

with KedroSession.create(env="databricks") as session:

session.run()

context = session.load_context()
    catalog = context.catalog
    my_dataset =catalog.load("x_train@pandas")
    my_dataset.head()

Yet when I try to access the elements inside the catalog through

catalog.load("x_train@pandas")

I get the error:

DataSetError: Data for MemoryDataSet has not been saved yet

I am not sure what is the best way to access the in memory elements that were part of the run, and I presume there must be a way, so any help here would be very helpful! Thanks in advance for your help community :)

I am not sure what is the best way to access the in memory elements that were part of the run, and I presume there must be a way, so any help here would be very helpful!

There is no way. `MemoryDataset`s are not persisted anywhere; if you want to access them later, use a physical dataset.

Thanks @Deepyaman Datta for the clarification.