Hello, good afternoon, I was adding a catalog item...
# questionsj
Javier Hernandez
01/12/2023, 1:51 PMHello, good afternoon, I was adding a catalog item from a database that has timeseries data and I wanted to have a dynamic param so when I run the code It retrieves the data regarding a certain day.
I ran into this post:
https://github.com/kedro-org/kedro/issues/1089
But I am failing to understand what is the reasoning behind. I would like to understand what would be the approach in kedro. Loading the whole table from sql and then run the transformations in code. But that would make quite inefficient when running in production.
d
datajoely
01/12/2023, 1:52 PMhey this is possible, but it’s not encouraged since you lose reproducibility
datajoely
01/12/2023, 1:52 PMbut you can use templated config to have the data param in
globals.ymldatajoely
01/12/2023, 1:53 PMyou can then use a hook to inject a CLI argument into that if you really want
j
Javier Hernandez
01/12/2023, 1:57 PMbut my initial thought on this is that dynamic parameters is obtained with code with another node. Could I use the after_node_run to set this dynamic parameter in the catalog?
d
datajoely
01/12/2023, 1:57 PMso when you get into that territory you are breaking some of Kedro’s key design decisions
datajoely
01/12/2023, 1:58 PMbecause we’re striving for reproducibility we don’t like conditional logic an the nodes shouldn’t be aware of IO
datajoely
01/12/2023, 1:58 PMit’s a tricky one because there are valid reasons users want to do this, but it’s an area Kedro has been designed to not enable
j
Javier Hernandez
01/12/2023, 2:00 PMso the kedro approach is to have a static catalog entries
d
datajoely
01/12/2023, 2:00 PMit was designed that way initially
datajoely
01/12/2023, 2:01 PMnow there are ways to do what you’re looking for, but it involves hooks like you say
j
Javier Hernandez
01/12/2023, 2:01 PMthanks for the answer
Javier Hernandez
01/12/2023, 2:02 PMIt's now more clear to me
6 Views