Hello! How are you? I have a question regarding kedro's parameters loading. i want to be able to load parameters configurations from a database. I mean, someone changes my model's parameters by changing values on a database, and I need to update my parameters with that information. What i've been thinking is to use a kedro Hook after_context_created to extract the parameters from the db, and dump them in the parameters.yaml file. Do you think this is a good approach?
Nok Lam Chan
11/13/2023, 3:10 PM
Do you update the parameters or fetch the whole configuration from db?
You can have your custom ConfigLoader that read from database instead of a file. Alternatively you can override certain config with hooks as you suggested. Config loader has a dictionary interface, so you can override the parameters as in you are assigning value to a dictionary
11/13/2023, 6:05 PM
Only the parameters from db, i have to fetch them, process them, and I want the kedro run to run with the updated parameters. I will look the ConfigLoader as you suggest
Where should i override the config loader?
I'm sorry, just now im reading the config loaders docs, and I just don't understand in which part of the project (file/folder/dir?) should I write the code to read the parameters from the db. Also, I was hoping to benefit from kedro's experiment tracking funcionality, and I guess that by using configLoaders that shouldn't be possible? Also, I need to refresh parameters from the database before each kedro run...
Thanks! What i meant with kedro-viz and experiment tracking is that I've read that the parameters defined in parameters.yml are tracked alongside the run. Would that work if i override the parameters.yml file with a configLoader?
Another question that i have about the config loaders, is at what moment of the context creation are they executed? Can I use datacatalog objects in this moment? Or are they not defined yet? I ask this in order to know if i can connect my oracledb from which I need to fetch the parameters values as a DataCatalog instance and access it from the configloader, or if I should connect externally using a python conector such as sqlalchemy?