Hi Team 🙂 (CC @Antony Milne @Max S @Petar Pejovic as this may interest you!) I have a question regarding the possibility of using dataset factories without a pipeline that define each single datatset in the context of a Vizro dashboard. Basically, I have defined dataset factories for use only in my Vizro Dashboards to load files that I do not know the name apriori using a pattern. For now I use a partitioned dataset that gives me all the available files with their loader function but for a significant time delay as they are potentially millions of files. Is it possible? One way could simply be to try and create a dummy pipeline that uses the dataset I need and ask the catalog for the load function but I wanted to know if there's a nicer way! Just to make it clear to anyone outside of the team, Vizro is a (really nice 🙂) Python dashboarding framework built on top of plotly Dash and Pydantic, that has an integration with Kedro data catalog and inbuilt data caching. Thanks a lot :)

Correct me if I'm wrong this will be way nicer in Kedro 1.0.0 @Elena Khaustova?

the possibility of using dataset factories without a pipeline that define each single datatset

You can use dataset factories without having a pipeline at all. If I understood the purpose correctly, you can just define factories in

catalog.yaml

and then just do

df = catalog.load(dataset_name)

, the rest of the steps are not needed. The resolution happens under the hood. This method allows you to see the resolved configuration without adding a dataset to the catalog:

ds_config = config_resolver.resolve_pattern(dataset_name)

Indeed, it works by just calling

load

! I am not sure why I really thought you needed to resolve yourself specific dataset created from a factory before you could actually load it. Thanks a lot @Elena Khaustova for explaining this :)