https://kedro.org/ logo
Powered by
#questions
Title
# questions
h

Hervé Lauwerier

11/10/2022, 4:51 PM
Hello Team, how to I use a generated catalog with create_pipeline. I have my catalog generated with a function and I would like my pipeline to be aware of it. But when I run it i got all my datasets
Copy code
not found in the DataCatalog
Kedro viz does display all my nodes properly
Copy code
def create_pipeline(**kwargs) -> Pipeline:
    conf_path = str(settings.CONF_SOURCE)
    conf_loader = TemplatedConfigLoader(conf_source=conf_path, env="local")
    parameters = conf_loader.get("parameters*", "parameters*/**", "**/parameters*")
    db_connector = LocalDbConnector(parameters)
    tables = generate_table_list(parameters=parameters, db_connector=db_connector)
    io = generate_catalog_from_table_list(tables=tables, db_connector=db_connector)
    return pipeline(
        generate_nodes(tables=tables)
    )
io contains here all my datasets.
Copy code
def generate_catalog_from_table_list(tables, db_connector):
    io = DataCatalog()
    for table in tables:
        # Add sql catalog entry
        io.add(
            f"sql_{table.get_raw_table_name()}",
            SQLTableDataSet(
                credentials=dict(con=db_connector.connection_string()+";DATABASE="+table.database.get_local_database_name()),
                table_name=f"{table.name}"
            ), replace = True
        )
How do I use io inside my pipeline?
I used @hook_impl def after_catalog_created(self, catalog):
a

Allen Ma

11/11/2022, 7:20 AM
HI @Hervé Lauwerier I don’t think that’s supported https://github.com/kedro-org/kedro/blob/main/kedro/framework/session/session.py#L391 https://github.com/kedro-org/kedro/blob/main/kedro/runner/runner.py#L73 Of course, I’m not sure if I’m right. What does kedro team say
3 Views
Powered by