https://kedro.org/ logo
#questions
Title
# questions
l

Lucia Gonzalez

11/13/2023, 2:48 PM
Hello! How are you? I have a question regarding kedro's parameters loading. i want to be able to load parameters configurations from a database. I mean, someone changes my model's parameters by changing values on a database, and I need to update my parameters with that information. What i've been thinking is to use a kedro Hook after_context_created to extract the parameters from the db, and dump them in the parameters.yaml file. Do you think this is a good approach?
n

Nok Lam Chan

11/13/2023, 3:10 PM
Do you update the parameters or fetch the whole configuration from db? You can have your custom ConfigLoader that read from database instead of a file. Alternatively you can override certain config with hooks as you suggested. Config loader has a dictionary interface, so you can override the parameters as in you are assigning value to a dictionary
l

Lucia Gonzalez

11/13/2023, 6:05 PM
Only the parameters from db, i have to fetch them, process them, and I want the kedro run to run with the updated parameters. I will look the ConfigLoader as you suggest
Where should i override the config loader?
I'm sorry, just now im reading the config loaders docs, and I just don't understand in which part of the project (file/folder/dir?) should I write the code to read the parameters from the db. Also, I was hoping to benefit from kedro's experiment tracking funcionality, and I guess that by using configLoaders that shouldn't be possible? Also, I need to refresh parameters from the database before each kedro run...
n

Nok Lam Chan

11/14/2023, 5:35 AM
Possible ways: 1. You can use hook to load external config - See https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig, I am not sure thought if kedro-viz experiment tracking can track this. Cc @Rashida Kanchwala @Nero Okwa to confirm this 2. Another possible way to do this is to use the “resolver”, you can implement a custom resolver that read config from database. https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-resolvers-in-the-omegaconfigloader
l

Lucia Gonzalez

11/14/2023, 11:30 AM
Thank you very very much for your answers. I will be working on this today, and I'll keep you posted if i reach a solution 🙂 thanks again
😄 1
r

Rashida Kanchwala

11/14/2023, 11:42 AM
Kedro-viz experiment tracking only supports two types of tracking datasets i.e. MetricsDataset and JSONDataset. What ever metrics/data you want to track you will have to define them as an output dataset in your conf. Pls refer to docs here - https://docs.kedro.org/en/stable/experiment_tracking/index.html#set-up-experiment-tracking-datasets
l

Lucia Gonzalez

11/14/2023, 1:57 PM
Thanks! What i meant with kedro-viz and experiment tracking is that I've read that the parameters defined in parameters.yml are tracked alongside the run. Would that work if i override the parameters.yml file with a configLoader?
Another question that i have about the config loaders, is at what moment of the context creation are they executed? Can I use datacatalog objects in this moment? Or are they not defined yet? I ask this in order to know if i can connect my oracledb from which I need to fetch the parameters values as a DataCatalog instance and access it from the configloader, or if I should connect externally using a python conector such as sqlalchemy?
thank you in advance for all the help 😄
3 Views