Mate Scharnitzky
06/22/2023, 2:05 PMkedro==0.18.3
that pins pytest~=6.2
which is conflict with pandera
, a new dependency we want to introduce. kedro==0.18.5
has already pytest~=7.2
, so we’re not far from resolving this conflict.
On the other hand, in order to upgrade to a higher Kedro version, we would need to change our custom JinjaTemplatedConfigLoader
that inherits from AbstractConfigLoader
, as both 0.18.4
and 0.18.5
introduced changes to configuration management, the latter OmegaConf
specifically. Also, 0.18.6
fixes some regressions in 0.18.5
.
Question: based on the above context, what Kedro would you suggest for us to upgrade to?
• It seems at least we need to go to 0.18.6
but maybe we can aim all the way to 0.18.10
?
• Also, do you have a migration guide about how to migrate from a custom config loader to OmegaConf bearing in mind that we need to use multi-runner
as well?
Thank you!
@Kasper Janehag, @Jaskaran Singh SidanaJuan Luis
06/22/2023, 2:10 PMJinjaTemplatedConfigLoader
works with 0.18.3 but not with 0.18.6 or any other later 0.18.x
, we'd love to know more. it's definitely not the intention to introduce breaking changes before major releasesMate Scharnitzky
06/22/2023, 2:14 PM0.18.10
and I’m happy to report back the issues we identify.Nok Lam Chan
06/22/2023, 2:16 PMMate Scharnitzky
06/23/2023, 3:03 PM0.18.10
. After resolving a few dependency conflicts, all of our kedro pipelines’ integration tests fail with the below error:
│ /home/circleci/.pyenv/versions/3.9.17/lib/python3.9/collections/__init__.py: │
│ 1058 in __getitem__ │
│ │
│ 1055 │ │ │ return self.data[key] │
│ 1056 │ │ if hasattr(self.__class__, "__missing__"): │
│ 1057 │ │ │ return self.__class__.__missing__(self, key) │
│ ❱ 1058 │ │ raise KeyError(key) │
│ 1059 │ │
│ 1060 │ def __setitem__(self, key, item): │
│ 1061 │ │ self.data[key] = item │
╰───────────────���──────────────────────────────────────────────────────────────╯
KeyError: 'logging'
Juan Luis
06/23/2023, 3:04 PMMate Scharnitzky
06/23/2023, 3:05 PMNok Lam Chan
06/23/2023, 3:06 PMlogging
wasn’t available? Check if you logging files are actually there maybe?Juan Luis
06/23/2023, 3:06 PMNok Lam Chan
06/23/2023, 3:07 PMdev
for 0.19, I would be very surprise if this leaked into the release😅Juan Luis
06/23/2023, 3:08 PMMate Scharnitzky
06/23/2023, 3:08 PMNok Lam Chan
06/23/2023, 3:08 PMAbstractConfigLoader
Mate Scharnitzky
06/23/2023, 3:10 PMkedro registry list
would fail as wellJuan Luis
06/23/2023, 3:12 PM$ tree conf/
? to see what config files you haveMate Scharnitzky
06/23/2023, 3:13 PMNok Lam Chan
06/23/2023, 3:15 PMMate Scharnitzky
06/23/2023, 3:16 PMNok Lam Chan
06/23/2023, 3:20 PMMate Scharnitzky
06/23/2023, 3:28 PMJuan Luis
06/23/2023, 4:25 PMI think for the time being though we should not get too stuck on "if users inherit from `AbstractConfigLoader`", because from the interviews it became clear people aren't using the😅 https://github.com/kedro-org/kedro/pull/1870#issuecomment-1262028818.AbstractConfigLoader