Lukas Innig
11/15/2023, 10:31 AMcatalog.list()
in a kedro jupter lab instance, it doesn’t return all registered datasets. anything that’s using a dataset factory seems to be missing. Is there a way to infer those somehow?Ankita Katiyar
11/15/2023, 10:40 AMcatalog.exists(dataset_name)
for dataset in pipeline["__default__"].data_sets():
catalog.exists(dataset)
And then catalog.list() should list themLukas Innig
11/15/2023, 10:42 AM_exists
method for this to work. That’s quite useful!Ankita Katiyar
11/15/2023, 10:47 AMLukas Innig
11/15/2023, 10:48 AM2023-11-15 10:42:25,539 - kedro.io.core - WARNING - 'exists()' not implemented for 'DataRobotProjectDataset'. Assuming output does not exist.
But it’s fine - I’m actually rather happy to implement a custom exists
methodAnkita Katiyar
11/15/2023, 10:52 AMLukas Innig
11/15/2023, 10:53 AMNok Lam Chan
11/15/2023, 11:34 AMcatalog.list(pipeline=<name>)
is not possible because catalog is not aware of a Pipeline object. This is coordianted by the session/runner instead😶Lukas Innig
11/15/2023, 11:37 AMNok Lam Chan
11/15/2023, 11:49 AM