Iñigo Hidalgo
03/30/2023, 9:58 AMdatajoely
03/30/2023, 10:01 AMIñigo Hidalgo
03/30/2023, 10:01 AMdatajoely
03/30/2023, 10:02 AMIñigo Hidalgo
03/30/2023, 10:04 AMJuan Luis
03/30/2023, 10:04 AMdatajoely
03/30/2023, 10:06 AMIñigo Hidalgo
03/30/2023, 10:10 AMdatajoely
03/30/2023, 10:22 AMIñigo Hidalgo
03/30/2023, 10:23 AMNok Lam Chan
03/30/2023, 12:37 PMIñigo Hidalgo
03/30/2023, 12:50 PMdatajoely
03/30/2023, 12:51 PMIñigo Hidalgo
03/30/2023, 12:55 PMdatajoely
03/30/2023, 12:56 PMIñigo Hidalgo
03/30/2023, 1:00 PMsales_intermediate:
filename: ...
layer: ...
save_kwargs: ...
`validations.yml`:
sales_intermediate:
expected_schema:
col1: ...
col2: ...
datajoely
03/30/2023, 1:01 PMsales_intermediate:
filename: ...
layer: ...
save_kwargs: ...
validations : ${../validations.yml}
Iñigo Hidalgo
03/30/2023, 1:02 PMNok Lam Chan
03/30/2023, 1:04 PMIñigo Hidalgo
03/31/2023, 9:21 AMself._check_duplicates(seen_file_to_keys)
, which might be inherited from anyconf so could be removed to facilitate the behavior I asked about above. This triggered a further thought though: I haven't been following recent developments, but do you distinguish the merging done within environments and across environments (e.g. when merging local & base)? I ask because at the moment (on kedro 0.17.1) I have some datasets defined in the base catalog, and when I want to save those datasets locally instead of blob, I overwrite them in the local environment with a different dataset definition. At the moment, since anyconf replaces the entire dictionary with the one from local, it overwrites correctly, but I was wondering what the expected behavior would be if you use OmegaConf's subkey merging.
conf/
base/
catalog/
datasets.yml
validations.yml # subkeys should be merged with those in the same environment
local/
catalog/
datasets.yml # top-level keys (dataset names) and their dictionary here should overwrite those in base (including those from validations?)
Nok Lam Chan
03/31/2023, 2:50 PM