Rob
04/22/2023, 6:09 PMstorage_type
, this is my how globals
YAML looks like:
storage_mode: "local"
storage:
local: "data/"
gcp: "<gs://my-bucket/data/>"
data:
{% if storage_mode == 'local' %}
storage_type: ${storage.local}
{% elif storage_mode == 'gcp' %}
storage_type: ${storage.gcp}
{% endif %}
player_tags: ${storage_type}/01_player_tags
raw_battlelogs: ${storage_type}/02_raw_battlelogs
raw_metadata: ${storage_type}/03_raw_metadata
enriched_data: ${storage_type}/04_enriched_data
curated_data: ${storage_type}/05_curated_data
viz_data: ${storage_type}/06_viz_data
feature_store: ${storage_type}/07_feature_store
model_registry: ${storage_type}/08_model_registry
I'm not familiar with this type of syntax, and I'm getting a ScannerError
Deepyaman Datta
04/22/2023, 6:15 PMac_template=false
, which basically means you can't use Jinja (from what I remember).storage_type: ${storage.local}
, does it work fine?Rob
04/22/2023, 7:35 PMstorage:
local: "data/"
gcp: "<gs://my-bucket/data/>"
storage_type: "{{ storage.local }}"
data:
player_tags: "{{ storage_type }}/01_player_tags"
raw_battlelogs: "{{ storage_type }}/02_raw_battlelogs"
raw_metadata: "{{ storage_type }}/03_raw_metadata"
enriched_data: "{{ storage_type }}/04_enriched_data"
curated_data: "{{ storage_type }}/05_curated_data"
viz_data: "{{ storage_type }}/06_viz_data"
feature_store: "{{ storage_type }}/07_feature_store"
model_registry: "{{ storage_type }}/08_model_registry"
Deepyaman Datta
04/23/2023, 3:31 PMMmm nope, it seems that it's not working alsoYeah, I didn't expect it to work, which is why I asked. AFAIK you can't template in globals. 0.18.5 (should be an easy upgrade from 0.18.4) adds the
OmegaConfigLoader
, which could potentially have a bit more flexibility in this space, as well as being the way forward from a Kedro standpoint. But I'd defer to somebody else working more closely on this functionality if you want to ake that route; I haven't used it much.Rob
04/23/2023, 3:34 PMNok Lam Chan
04/24/2023, 12:51 PMenv
instead.
What I would do is to have two environment base
and gcp
base/parameters.yml
storage_type: "{{ storage_type }}"
data:
player_tags: "{{ storage_type }}/01_player_tags"
raw_battlelogs: "{{ storage_type }}/02_raw_battlelogs"
raw_metadata: "{{ storage_type }}/03_raw_metadata"
enriched_data: "{{ storage_type }}/04_enriched_data"
curated_data: "{{ storage_type }}/05_curated_data"
viz_data: "{{ storage_type }}/06_viz_data"
feature_store: "{{ storage_type }}/07_feature_store"
model_registry: "{{ storage_type }}/08_model_registry"
base/globals.yml
storage_path: "data/"
gcp/globals.yml
storage_path: "<gs://my-bucket/data/>"
Rob
04/24/2023, 2:13 PMNok Lam Chan
04/25/2023, 4:52 PM