https://kedro.org/ logo
#questions
Title
# questions
i

Iñigo Hidalgo

03/13/2024, 11:30 AM
I am using dataset factories and variable interpolation to set up some dynamic pipelines while avoiding repeating config as much as possible.
Copy code
_path_placeholder_config:
  root: <abfs://container>
  signal_path: "{geography}/{technology}/{signal}/{granularity}"
  model_path: ${.signal_path}/{model_name}

"{geography}__{technology}__{signal}__{granularity}__predictions":
  type: xxx
  path: ${_path_placeholder_config.root}/predictions/${_path_placeholder_config.signal_path}/predictions.parquet
  credentials: blob_storage
  write_mode: append
  versioned: true


_model_object_config:
  root: ${_path_placeholder_config.root}/models/${_path_placeholder_config.model_path}
  versioned: true

"{geography}__{technology}__{signal}__{granularity}__{model_name}__model_obj":
  type: axpo.kedro.datasets.python_object_dataset.PythonObjectBlob
  credentials: blob_storage
  filepath: ${_model_object_config.root}/fitted_model.pickle
  versioned: ${_model_object_config.versioned}

"{geography}__{technology}__{signal}__{granularity}__{model_name}__model_obj#autogluon":
  type: axpo.kedro.autogluon.autogluon_dataset.AutoGluonModelDataset
  credentials: blob_storage
  path: ${_model_object_config.root}/fitted_model
  versioned: ${_model_object_config.versioned}
I am not 100% clear on the order of OmegaConf interpolation vs the combination of base/, dev/ environment config. If I have the above in my base/, would you expect it to be possible for me to override the
_path_placeholder_config
catalog entry from local/ and have that change affect the dataset factories? I will test this myself eventually, but I would like to have a clearer understanding of the order of operations before I fully commit to building out this sort of structure.
n

Nok Lam Chan

03/13/2024, 1:19 PM
https://noklam.github.io/blog/posts/kedro_config_loader/2023-11-16-kedro-config-loader-dive-deep.html I never have time to finish writing it but the diagram should still helps
i

Iñigo Hidalgo

03/13/2024, 6:01 PM
Thx Nok, that answers my question exactly
you cannot interpolate a value from another environment
does this only refer to the direct interpolation i referred to above? or omegaconf custom resolvers also cannot resolve across environments?
n

Nok Lam Chan

03/13/2024, 6:04 PM
globals
is the only thing that works cross environment IIRC. custom resolver wouldn't work directly because everything is resolved within the same env, then a dict merge happened between the two environment.
👍 1
i

Iñigo Hidalgo

03/13/2024, 6:19 PM
not that i will attempt to do this, but if i wanted to modify this behavior, where would i find this behavior? the omegaconfigloader? do you think this question is related to this issue? https://github.com/kedro-org/kedro/issues/3086
👀 1
n

Nok Lam Chan

03/13/2024, 6:24 PM
Maybe, but I am not sure. Currently the config loader and the DataCatalog has clear responsibility. So It is the DataCatalog responsibility to convert that funny template string to a "dataset pattern".
In other words, ConfigLoader is not aware of "dataset pattern", it's merely a dictionary of string
r

Rennan Haro

03/14/2024, 6:32 PM
deleted :)