When I say

catalog.list()

in a kedro jupter lab instance, it doesn’t return all registered datasets. anything that’s using a dataset factory seems to be missing. Is there a way to infer those somehow?

The factory datasets are registered after they are first used during a session run. CHecking for their existence will also register them to the catalog ->

catalog.exists(dataset_name)

for dataset in pipeline["__default__"].data_sets():
  catalog.exists(dataset)

And then catalog.list() should list them

that’s brilliant! Thank you

It should work without the _exists method too since internally it just sees if the catalog entry exists, doesn’t call the datasets’s exists method

I am getting a

2023-11-15 10:42:25,539 - kedro.io.core - WARNING - 'exists()' not implemented for 'DataRobotProjectDataset'. Assuming output does not exist.

But it’s fine - I’m actually rather happy to implement a custom

exists

method

Ah, my bad. It does need it!

ah you’re right. It does appear in the list

@Lukas Innig What do you expected to see instead? factory is created during a pipeline run, so the result gonna depends on which pipeline you are running.

I guess I was expecting to see all the datasets that I specified in catalog.yml. to be honest I haven’t thought too deeply about it

I created this ticket to document the workaround. https://github.com/kedro-org/kedro/issues/3312 I can’t think of a nice solution just now but I will be keep this in mind.