https://kedro.org/ logo
#questions
Title
# questions
j

Jordan Barlow

09/11/2023, 12:59 PM
Does anyone have an example of how to access params passed at the CLI in a hook? I know this isn't a very Kedronic pattern, but it seems a necessary workaround for my use case
d

datajoely

09/11/2023, 1:02 PM
I’d argue it is Kedriffic! The params should be in the
extra_params
attr of the
before_pipeline_run
hook
K 1
j

Jordan Barlow

09/11/2023, 1:25 PM
I'll add that one to my vocabulary! Do you have any thoughts about if this will work if I'm trying to dynamically generate catalog entries? I think what I really need is to be able to override globals from the CLI, something like this:
Copy code
${<http://globals.site|globals.site>}_${globals.service}_usage:
  type: pandas.SQLQueryDataSet
  filepath: data/01_raw/${globals.service}.sql
  load_args:
    index_col: "my_index_col"
    params:
      start_date: ${globals.start_date}
      end_date: ${globals.end_date}
  credentials: ${globals.site.db_creds}
  layer: Raw
But as I understand, this isn't currently possible with catalog templating. Dynamically generating the catalog at runtime via hooks seems like the next best option.
d

datajoely

09/11/2023, 1:29 PM
You can do it with a omega conf resolver
j

Jordan Barlow

09/11/2023, 1:53 PM
Thanks, I'll look into this!
@datajoely So am I right in saying that this can now (as of
0.18.13
) be done right from
settings.py
? Something like:
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "runtime_param": lambda x: self.runtime_params[x],
    }
}
Though I don't understand the OmegaConf implementation well enough to know the correct syntax here, surely
self
is not right
d

datajoely

09/11/2023, 2:28 PM
hey let me look into this
They create a new config loader class which is how they can use self
j

Jordan Barlow

09/11/2023, 2:45 PM
Oh yeah I saw this, but I wasn't sure where
ExtendedOmegaConfigLoader
was implemented (within
settings.py
?) or what the contents of
register_custom_resolver
was supposed to be!
d

datajoely

09/11/2023, 3:19 PM
it can be anywhere you just need to import it into setting.py and use it under
CONFIG_LOADER_CLASS
j

Jordan Barlow

09/11/2023, 3:35 PM
Cheers, that makes sense! It would be good to know what's going on in that static method 🤔
d

datajoely

09/11/2023, 4:01 PM
@Fazil B. Topal are you able to share more of your snippet?
🥳 1
f

Fazil B. Topal

09/11/2023, 4:11 PM
that is the whole snippet so far
❤️ 1
j

Jordan Barlow

09/11/2023, 4:27 PM
@Fazil B. Topal Thank you very much for this!
🥳 1
2 Views