Hi everyone, I'm trying to use Jinja2 syntax on Kedro 0.18.4 to dynamically define the variable

storage_type

, this is my how

globals

YAML looks like:

storage_mode: "local"

storage:
  local: "data/"
  gcp: "<gs://my-bucket/data/>"

data:
  {% if storage_mode == 'local' %}
  storage_type: ${storage.local}
  {% elif storage_mode == 'gcp' %}
  storage_type: ${storage.gcp}
  {% endif %}
  player_tags: ${storage_type}/01_player_tags
  raw_battlelogs: ${storage_type}/02_raw_battlelogs
  raw_metadata: ${storage_type}/03_raw_metadata
  enriched_data: ${storage_type}/04_enriched_data
  curated_data: ${storage_type}/05_curated_data
  viz_data: ${storage_type}/06_viz_data
  feature_store: ${storage_type}/07_feature_store
  model_registry: ${storage_type}/08_model_registry

I'm not familiar with this type of syntax, and I'm getting a

ScannerError

Not 100% confident, but I don't imagine you can use Jinja in your `globals.yaml`; when loading from globals patterns (https://github.com/kedro-org/kedro/blob/main/kedro/config/templated_config.py#L139),

ac_template=false

, which basically means you can't use Jinja (from what I remember).

Mmm nope, it seems that it's not working also tried the next:

storage:
  local: "data/"
  gcp: "<gs://my-bucket/data/>"

storage_type: "{{ storage.local }}"

data:
  player_tags: "{{ storage_type }}/01_player_tags"
  raw_battlelogs: "{{ storage_type }}/02_raw_battlelogs"
  raw_metadata: "{{ storage_type }}/03_raw_metadata"
  enriched_data: "{{ storage_type }}/04_enriched_data"
  curated_data: "{{ storage_type }}/05_curated_data"
  viz_data: "{{ storage_type }}/06_viz_data"
  feature_store: "{{ storage_type }}/07_feature_store"
  model_registry: "{{ storage_type }}/08_model_registry"

Mmm nope, it seems that it's not working also

Yeah, I didn't expect it to work, which is why I asked. AFAIK you can't template in globals. 0.18.5 (should be an easy upgrade from 0.18.4) adds the

OmegaConfigLoader

, which could potentially have a bit more flexibility in this space, as well as being the way forward from a Kedro standpoint. But I'd defer to somebody else working more closely on this functionality if you want to ake that route; I haven't used it much.

Don't worry and thanks for the suggestion @Deepyaman Datta. I'll take a look

Instead of injecting conditions immediately, I would do separate them into

env

instead. What I would do is to have two environment

base

and

gcp

base/parameters.yml

storage_type: "{{ storage_type }}"

data:
  player_tags: "{{ storage_type }}/01_player_tags"
  raw_battlelogs: "{{ storage_type }}/02_raw_battlelogs"
  raw_metadata: "{{ storage_type }}/03_raw_metadata"
  enriched_data: "{{ storage_type }}/04_enriched_data"
  curated_data: "{{ storage_type }}/05_curated_data"
  viz_data: "{{ storage_type }}/06_viz_data"
  feature_store: "{{ storage_type }}/07_feature_store"
  model_registry: "{{ storage_type }}/08_model_registry"

base/globals.yml

storage_path: "data/"

gcp/globals.yml
storage_path: "<gs://my-bucket/data/>"

Thanks @Nok Lam Chan, yes that approach makes more sense. I'll give a try!

Awesome!