I am using dataset factories and variable interpolation to set up some dynamic pipelines while avoiding repeating config as much as possible.

_path_placeholder_config:
  root: <abfs://container>
  signal_path: "{geography}/{technology}/{signal}/{granularity}"
  model_path: ${.signal_path}/{model_name}

"{geography}__{technology}__{signal}__{granularity}__predictions":
  type: xxx
  path: ${_path_placeholder_config.root}/predictions/${_path_placeholder_config.signal_path}/predictions.parquet
  credentials: blob_storage
  write_mode: append
  versioned: true


_model_object_config:
  root: ${_path_placeholder_config.root}/models/${_path_placeholder_config.model_path}
  versioned: true

"{geography}__{technology}__{signal}__{granularity}__{model_name}__model_obj":
  type: axpo.kedro.datasets.python_object_dataset.PythonObjectBlob
  credentials: blob_storage
  filepath: ${_model_object_config.root}/fitted_model.pickle
  versioned: ${_model_object_config.versioned}

"{geography}__{technology}__{signal}__{granularity}__{model_name}__model_obj#autogluon":
  type: axpo.kedro.autogluon.autogluon_dataset.AutoGluonModelDataset
  credentials: blob_storage
  path: ${_model_object_config.root}/fitted_model
  versioned: ${_model_object_config.versioned}

I am not 100% clear on the order of OmegaConf interpolation vs the combination of base/, dev/ environment config. If I have the above in my base/, would you expect it to be possible for me to override the

_path_placeholder_config

catalog entry from local/ and have that change affect the dataset factories? I will test this myself eventually, but I would like to have a clearer understanding of the order of operations before I fully commit to building out this sort of structure.

https://noklam.github.io/blog/posts/kedro_config_loader/2023-11-16-kedro-config-loader-dive-deep.html I never have time to finish writing it but the diagram should still helps

Thx Nok, that answers my question exactly

you cannot interpolate a value from another environment

does this only refer to the direct interpolation i referred to above? or omegaconf custom resolvers also cannot resolve across environments?

globals

is the only thing that works cross environment IIRC. custom resolver wouldn't work directly because everything is resolved within the same env, then a dict merge happened between the two environment.

not that i will attempt to do this, but if i wanted to modify this behavior, where would i find this behavior? the omegaconfigloader? do you think this question is related to this issue? https://github.com/kedro-org/kedro/issues/3086

Maybe, but I am not sure. Currently the config loader and the DataCatalog has clear responsibility. So It is the DataCatalog responsibility to convert that funny template string to a "dataset pattern".