https://kedro.org/ logo
#questions
Title
# questions
m

Mate Scharnitzky

06/22/2023, 2:05 PM
Hi Team, Currently we’re using
kedro==0.18.3
that pins
pytest~=6.2
which is conflict with
pandera
, a new dependency we want to introduce.
kedro==0.18.5
has already
pytest~=7.2
, so we’re not far from resolving this conflict. On the other hand, in order to upgrade to a higher Kedro version, we would need to change our custom
JinjaTemplatedConfigLoader
that inherits from
AbstractConfigLoader
, as both
0.18.4
and
0.18.5
introduced changes to configuration management, the latter
OmegaConf
specifically. Also,
0.18.6
fixes some regressions in
0.18.5
. Question: based on the above context, what Kedro would you suggest for us to upgrade to? • It seems at least we need to go to
0.18.6
but maybe we can aim all the way to
0.18.10
? • Also, do you have a migration guide about how to migrate from a custom config loader to OmegaConf bearing in mind that we need to use
multi-runner
as well? Thank you! @Kasper Janehag, @Jaskaran Singh Sidana
j

Juan Luis

06/22/2023, 2:10 PM
hey @Mate Scharnitzky! if your
JinjaTemplatedConfigLoader
works with 0.18.3 but not with 0.18.6 or any other later
0.18.x
, we'd love to know more. it's definitely not the intention to introduce breaking changes before major releases
the migration guide for OmegaConf is planned https://github.com/kedro-org/kedro/issues/2699
m

Mate Scharnitzky

06/22/2023, 2:14 PM
It definitely caused some issues for us, but since we relaxed/cleaned a lot of our dependencies. Let me try to move the pin to
0.18.10
and I’m happy to report back the issues we identify.
n

Nok Lam Chan

06/22/2023, 2:16 PM
I don’t expect any breaking changes for any 0.18.x release other than the dependencies issue.
m

Mate Scharnitzky

06/23/2023, 3:03 PM
Hi @Nok Lam Chan, @Juan Luis As we discussed, we created a PR to upgrade Kedro to
0.18.10
. After resolving a few dependency conflicts, all of our kedro pipelines’ integration tests fail with the below error:
Copy code
│ /home/circleci/.pyenv/versions/3.9.17/lib/python3.9/collections/__init__.py: │
│ 1058 in __getitem__                                                          │
│                                                                              │
│   1055 │   │   │   return self.data[key]                                     │
│   1056 │   │   if hasattr(self.__class__, "__missing__"):                    │
│   1057 │   │   │   return self.__class__.__missing__(self, key)              │
│ ❱ 1058 │   │   raise KeyError(key)                                           │
│   1059 │                                                                     │
│   1060 │   def __setitem__(self, key, item):                                 │
│   1061 │   │   self.data[key] = item                                         │
╰───────────────���──────────────────────────────────────────────────────────────╯
KeyError: 'logging'
Log from CI
j

Juan Luis

06/23/2023, 3:04 PM
m

Mate Scharnitzky

06/23/2023, 3:05 PM
Exactly 🙂 , full circle
I will build the app to get a more refined view of the error as this is not informative.
n

Nok Lam Chan

06/23/2023, 3:06 PM
I guess
logging
wasn’t available? Check if you logging files are actually there maybe?
What does you integration test do exactly?
j

Juan Luis

06/23/2023, 3:06 PM
but it was working before right? it's a change that we've done midway
n

Nok Lam Chan

06/23/2023, 3:07 PM
Is it running with some kedro starters?
We did remove logging but it’s done in
dev
for 0.19, I would be very surprise if this leaked into the release😅
j

Juan Luis

06/23/2023, 3:08 PM
we need to get to the bottom of this I think. I'm opening an issue
m

Mate Scharnitzky

06/23/2023, 3:08 PM
It runs the kedro pipelines for our propensity use case which includes: • the data mocker pipeline • the feature engineering pipeline • and finally the propensity pipeline
n

Nok Lam Chan

06/23/2023, 3:08 PM
I guess the error shows up very early before any of these pipelines get run?
Let me check what’s changed in
AbstractConfigLoader
m

Mate Scharnitzky

06/23/2023, 3:10 PM
yes, the mocker doesn’t even complete, I assume if I typed
kedro registry list
would fail as well
j

Juan Luis

06/23/2023, 3:12 PM
can you do
Copy code
$ tree conf/
? to see what config files you have
m

Mate Scharnitzky

06/23/2023, 3:13 PM
I’m building it locally now to be able to test it
👍🏼 1
n

Nok Lam Chan

06/23/2023, 3:15 PM
so 0.18.3 is still working and 0.18.10 breaks it. Do you know since version it starts to break?
m

Mate Scharnitzky

06/23/2023, 3:16 PM
correct
Not now, but earlier I tested 0.18.4 as well, and it failed already
👍🏼 1
m

Mate Scharnitzky

06/23/2023, 3:28 PM
I managed to replicate the error locally
j

Juan Luis

06/23/2023, 4:25 PM
we found the root cause, updating soon
👍🏼 1
I think for the time being though we should not get too stuck on "if users inherit from `AbstractConfigLoader`", because from the interviews it became clear people aren't using the
AbstractConfigLoader
.
😅 https://github.com/kedro-org/kedro/pull/1870#issuecomment-1262028818
4 Views