Hi, I have a dataset factory specified, and when I do

catalog.list()

I don't see the entires that match the factory (so in the end I don't know that they exist at all). However,

catalog.load("<name>")

works. Is this the intended way?

which version of Kedro are you running?

0.19.5

Hey Kacper, yeah, the factory datasets are not eagerly registered to the catalog but at the first instance they are loaded or checked for existence.

@Ankita Katiyar do you think it makes sense to provide a way of triggering this?

I see, now that i loaded them i can see them when I do

catalog.list()

The workaround is -

for dataset in pipeline["__default__"].datasets():
  catalog.exists(dataset)

Can confirm that doing this:

for dataset in pipelines["__default__"].datasets():
    catalog.exists(dataset)

makes the entries show up in the

catalog.list()

(might throw a warning if the dataset does not exist just yet)

Great! Thanks for the help 🙌

would you mind adding a comment to the issue so we have your usecase recorded? 😄

yeah sure!

It depends though, if you have a huge catalog, loading all datasets eagerly might not be what you want. It could be very time and memory consuming.

@Merel Should we put this back into inbox? The priority of this isn't high originally, but we have seen multiple requests

In that case, let's just up the priority. No need to re-discuss in grooming 🙂