Marc Gris
09/21/2023, 10:13 AMmarrrcin
09/21/2023, 10:15 AMload
will return a function that will then load the dataset based on the params you pass to it:
class LazyDataSet(AbstractDataSet):
# constructors and other stuff
def _load():
def lazy_loader(path):
return PickleDataSet(path).load()
return lazy_loader
And then you do this in 2 nodes:
1. node(inputs="from_sql_query", func=<extract the path you need>, outputs="path_you_need")
2. node(inputs=["path_you_need", "lazy_dataset"], lambda path, lazy: lazy(path))
Marc Gris
09/21/2023, 10:30 AMmarrrcin
09/21/2023, 1:18 PM