Emilio Gagliardi
09/14/2023, 8:10 PMmarrrcin
09/15/2023, 8:36 AMkedro run
on a single machine you could pass around any python object. The quirk is that by default, the MemoryDataSet
uses copy/deep copy mode and assing for DataFrames, whereas you should probably explicitly use MemoryDataSet(copy_mode="assign")
instead.
You can do this via catalog:
catalog.yml
my_llama_index:
type: MemoryDataSet
copy_mode: assign
Iñigo Hidalgo
09/15/2023, 12:11 PMtype
?
I had a hiccup with that default copy mode for memorydatasets with my Ibis dataset.Emilio Gagliardi
09/15/2023, 8:45 PMmarrrcin
09/18/2023, 7:04 AM"{dataset_name}":
type: MemoryDataSet
copy_mode: assign
it overrides the “default” dataset, because it’s effectively a “catch-all” datasetIñigo Hidalgo
09/18/2023, 2:11 PM