https://kedro.org/ logo
#questions
Title
# questions
i

Iñigo Hidalgo

02/21/2024, 3:41 PM
If I have parameters like so
Copy code
dataset_config:
   primary_key: [date, timeseries_id, unit_id]
Using OmegaConf resolvers could I somehow select
dataset_config.primary_key[1:]
in another parameter?
I'm looking at the built-in resolvers but it's a bit unclear if they serve the purpose I need
d

datajoely

02/21/2024, 3:42 PM
I think you need to register something like this
lambda x,y : x[y:]
i

Iñigo Hidalgo

02/21/2024, 3:45 PM
and then I would do
Copy code
CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "index_list": lambda x, y: x[y:]
}}
${index_list:dataset_config.primary_key, 1}
?
d

datajoely

02/21/2024, 3:46 PM
Yes exactly
🤞
i

Iñigo Hidalgo

02/21/2024, 3:47 PM
testing now, ill report back in a couple mins 🙂
🤞 1
got sidetracked as i just discovered relative interpolation 🤯 https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#id26
this is a huge gamechanger for me btw, idk if it's in the documentation specifically, but it makes some config sososo much easier!!!
❤️ 2
it seems i'm missing a small thing on the resolver btw, taking this example
${index_list:dataset_config.primary_key, 1}
this is returning
ataset_config.primary_key
So it's parsing my interpolation key as a string, This works though:
${index_list:${dataset_config.primary_key}, 1}
d

datajoely

02/21/2024, 4:04 PM
so there is no requirement for it to be a Lambda I guess, if you provide a
def
can you put a breakpoint in it?
i

Iñigo Hidalgo

02/21/2024, 4:05 PM
yeah, it was just a question of omegaconf not interpreting just the pure string as a thing to be interpolated, but my final line works exactly as it should!! so my question is fully solved. I have to say Omegaconf is a huge upgrade, and it seems I'm only scratching the barest surfacrt
❤️ 1
i have to say relative imports are actually a bigger find for me than the custom resolver
K 1
but it's only really super useful within the scope of dynamic pipelines @marrrcin
👍 1
d

datajoely

02/21/2024, 4:09 PM
it’s honestly such a huge huge moment for the team @Merel deserves a lot of credit for the thoughtful design and @Ankita Katiyar for doing most of the implementation (sorry if I forgot anyone) 🚀. The opus of config research written by @Hamza a lonnnnng time ago underpinned most of this thinking too. I have Issue 891 etched into my brain lol. The config system has been the one of the biggest blockers for us doing some of the more ambitious features we have planned since we knew we were going to replace it and it was embedded so deep down it touched EVERYTHING. Now we’ve shipped it OmegConf - not only do you have new super powers like the above, but it unlocks so much we’re now in a position to cook up 👨‍🍳
❤️ 7
i

Iñigo Hidalgo

02/21/2024, 4:13 PM
I wanna say huge kudos to everybody involved in all the development on kedro over the past couple of years. The difference in UX from 0.17.1 and current kedro is incredible. The basics are the same but it's become such a pleasure to use and develop. Whereas before it felt like sacrifices were made in order to fit the "kedro" way, now it feels like we have the necessary tools to extend the kedro way to fit many different needs.
🥹 3
❤️ 8
m

Merel

02/21/2024, 4:42 PM
Thank you for your kind words @Iñigo Hidalgo, your feedback is incredibly valuable and it makes my day reading a comment like this 🥰
🎉 1
n

Nok Lam Chan

02/21/2024, 5:01 PM
Don't forget to leave a star on https://github.com/omry/omegaconf too :D
🌟 2
j

Juan Luis

02/21/2024, 5:25 PM
it's so exciting to read this thread, especially knowing that it's been a looooong journey for you @Iñigo Hidalgo to migrate these pipelines from ancient Kedro versions!! thanks a bunch for your heartwarming feedback K
j

Janick Spirig

02/22/2024, 6:21 PM
Hi @Iñigo Hidalgo Thanks for this information! I am working on something similar at the moment. Just one question, where exactly in the code do you call:
${index_list:${dataset_config.primary_key}, 1}
? Do you then have
primary_key: [timeseries_id, unit_id]
available in a dict?
i

Iñigo Hidalgo

02/22/2024, 6:58 PM
Hi Janick It would be a further parameter, so
Copy code
dataset_config:
  primary_key: [date, timeseries_id, unit_id]

...

somewhere:
  else:
    single_timeseries_id: ${index_list:${dataset_config.primary_key}, 1}
I could then access
params:somewhere.else.single_timeseries_id
which would be passed to the node as
[timeseries_id, unit_id]
migrate these pipelines from ancient Kedro versions
@Juan Luis i have to admit, i cheated there. in the end, the cost-benefit just wasn't in favor of actually migrating that older code to newer kedro. instead i was able to get buy-in to start reimplementing it in a much more modular way.
👍🏼 1