Hello Team, how to I use a generated catalog with ...
# questionsh
Hervé Lauwerier
11/10/2022, 4:51 PMHello Team, how to I use a generated catalog with create_pipeline. I have my catalog generated with a function and I would like my pipeline to be aware of it. But when I run it i got all my datasets
Copy code
not found in the DataCatalogHervé Lauwerier
11/10/2022, 4:51 PMKedro viz does display all my nodes properly
Hervé Lauwerier
11/10/2022, 4:52 PM Copy code
def create_pipeline(**kwargs) -> Pipeline:
conf_path = str(settings.CONF_SOURCE)
conf_loader = TemplatedConfigLoader(conf_source=conf_path, env="local")
parameters = conf_loader.get("parameters*", "parameters*/**", "**/parameters*")
db_connector = LocalDbConnector(parameters)
tables = generate_table_list(parameters=parameters, db_connector=db_connector)
io = generate_catalog_from_table_list(tables=tables, db_connector=db_connector)
return pipeline(
generate_nodes(tables=tables)
)Hervé Lauwerier
11/10/2022, 4:53 PMio contains here all my datasets.
Copy code
def generate_catalog_from_table_list(tables, db_connector):
io = DataCatalog()
for table in tables:
# Add sql catalog entry
io.add(
f"sql_{table.get_raw_table_name()}",
SQLTableDataSet(
credentials=dict(con=db_connector.connection_string()+";DATABASE="+table.database.get_local_database_name()),
table_name=f"{table.name}"
), replace = True
)Hervé Lauwerier
11/10/2022, 6:50 PMHow do I use io inside my pipeline?
Hervé Lauwerier
11/10/2022, 9:25 PMI used @hook_impl
def after_catalog_created(self, catalog):
a
Allen Ma
11/11/2022, 7:20 AMHI @Hervé Lauwerier
I don’t think that’s supported
https://github.com/kedro-org/kedro/blob/main/kedro/framework/session/session.py#L391
https://github.com/kedro-org/kedro/blob/main/kedro/runner/runner.py#L73
Of course, I’m not sure if I’m right. What does kedro team say
4 Views