Fazil B. Topal
08/02/2023, 1:14 PMkedro run
or do i have to explicitly use this python object and load the data on my own?
• Is it possible to define some sections in the yaml file and other parts in python? I know i can do something in the hooks but I wanted to check if there is way where this catalog variable would be accessible by the user?
Thanks in advance! 🙂Merel
08/03/2023, 9:23 AMFazil B. Topal
08/03/2023, 9:26 AMcatalog
variable to kedro. That's what i was trying to find out.Merel
08/03/2023, 9:28 AMcatalog.py
what’s meant is that you can load the catalog itself directly in python. It doesn’t mean that your catalog can be python. The example in the docs:
from <http://kedro.io|kedro.io> import DataCatalog
from kedro_datasets.pandas import (
CSVDataSet,
SQLTableDataSet,
SQLQueryDataSet,
ParquetDataSet,
)
io = DataCatalog(
{
"bikes": CSVDataSet(filepath="../data/01_raw/bikes.csv"),
"cars": CSVDataSet(filepath="../data/01_raw/cars.csv", load_args=dict(sep=",")),
"cars_table": SQLTableDataSet(
table_name="cars", credentials=dict(con="sqlite:///kedro.db")
),
"scooters_query": SQLQueryDataSet(
sql="select * from cars where gear=4",
credentials=dict(con="sqlite:///kedro.db"),
),
"ranked": ParquetDataSet(filepath="ranked.parquet"),
}
)
shows how to use the catalog within python codeFazil B. Topal
08/03/2023, 9:31 AMnode(example_func, inputs='scooters_query')
will it work directly or what do I need to do it make it work? Using context hook to inject my catalog to KedroContext?Merel
08/03/2023, 9:38 AMkedro run
. You could create a custom DataCatalog
class and then register it in settings.py
https://docs.kedro.org/en/latest/kedro_project_setup/settings.html#application-settings. But that would require overwriting the DataCatalog
which isn’t exactly what you want I think?Fazil B. Topal
08/03/2023, 9:44 AMDataCatalog
but i simpled wanted to see if I have the io
variable in your code snippet above, what would be the way to hook this up to kedro run ...
Merel
08/03/2023, 9:58 AMJo Stichbury
08/03/2023, 10:19 AMFazil B. Topal
08/03/2023, 10:23 AMNok Lam Chan
08/03/2023, 12:27 PMFazil B. Topal
08/03/2023, 3:30 PMNok Lam Chan
08/03/2023, 7:58 PM