How do I access the data catalog from within a pip...
# questions
How do I access the data catalog from within a pipeline? A use case for EDA is to run a function on all (or multiple) data sets. A rough sketch might look like this:
Copy code
target_datasets = [...]
for dataset in catalog.datasets:
    if dataset in target_datasets:
            node(dataset.describe(), inputs=dataset, outputs=f'{dataset}_describe')
You can use hooks for this:, specifically
could help here.
@Merel Thank you. I am unfamiliar with how to set up hooks but I will review the documentation.
Hopefully the docs are clear, but feel free to ask more questions 🙂
@Merel Thanks. I get a feeling that I should have read other parts of the documentation before understanding what is going on in the documentation for hooks. I see this:
Copy code
def after_catalog_created(
    catalog: DataCatalog,
    conf_catalog: Dict[str, Any],
    conf_creds: Dict[str, Any],
    save_version: str,
    load_versions: Dict[str, str],
) -> None:
I am not sure where
is supposed to be imported from, or how it is intended that I implement functionality into
. For example, if I wanted to call
on each data set in the catalog and then include those tabular results back into the catalog I am unsure how it is intended for me to do that. Do I just monkey patch
or is there a method available for instances of
? It just isn't concrete for me yet 😅
Reading the docs is always a good idea! But maybe you can explain a bit more what you're trying to achieve? Would you want to do a describe on every dataset every time you run the pipeline or just once? And would those results be saved as new catalog entries or do you want to append them to the existing?
(FYI for more learning our youtube channel might be helpful
❤️ 1
@Merel I am watching this video. I'll see where I get 🙂 If it doesn't make sense then I'll answer your question.
👍 1

👍 1
@Merel I got a proof of concept hook working. Thank you for your help!
@Merel I do have another question. Here is a prototype hook
Copy code
class PrintCatalog:

    def after_catalog_created(self, catalog: DataCatalog, conf_catalog: Dict[str, Any]) -> None:
        for thing in dir(catalog.datasets):
                print(getattr(catalog.datasets, thing).load().describe())
            except Exception as e:
Naturally the catch-all Exception is not great, but not permanent. Just trying to take a look at what I am accessing. How would I add new entries to the data catalog which in turn would get saved while the hook is executing? Like if I wanted to write the results of
to a the datasets folder in my Kedro project.
Ah, the
method on
Copy code
Help on function add in module

add(self, dataset_name: 'str', dataset: 'AbstractDataset', replace: 'bool' = False) -> 'None'
    Adds a new ``AbstractDataset`` object to the ``DataCatalog``.
        dataset_name: A unique data set name which has not been
            registered yet.
        dataset: A data set object to be associated with the given data
            set name.
        replace: Specifies whether to replace an existing dataset
            with the same name is allowed.
        DatasetAlreadyExistsError: When a data set with the same name
            has already been registered.
        >>> from kedro_datasets.pandas import CSVDataset
        >>> io = DataCatalog(datasets={
        >>>                   'cars': CSVDataset(filepath="cars.csv")
        >>>                  })
        >>> io.add("boats", CSVDataset(filepath="boats.csv"))
👍 1