https://kedro.org/ logo
#questions
Title
# questions
m

Muhammed Afnas

02/23/2024, 9:50 AM
Hi folks, I have a query regarding accessing the kedro catalog of a current kedro session inside pipeline.py file inside a pipeline. or if there is a way to access the MemoryDataset added via hook in the pipeline.py file.
d

datajoely

02/23/2024, 10:00 AM
so there are dataset hooks which you can utilise as they pass through the pipeline
m

marrrcin

02/23/2024, 10:18 AM
Yeah, but first maybe you should determine why do you need to do that @Muhammed Afnas? Every time I see similar question there’s a hint that someone tries to do something against Kedro way of doing things.
Smells a lot like dynamic pipelines every time 🙈 THERE IS NO BUILT IN WAY TO ACCESS DATASETS / PARAMETERS DURING PIPELINE GENERATION. Maybe we should add auto responder for that @datajoely 😄
d

datajoely

02/23/2024, 10:26 AM
We need a page on the docs
or a video explaining our thoughts on the topic
perhaps I should make that
m

Muhammed Afnas

02/23/2024, 10:27 AM
Thank you datajoely! hey @marrrcin, thanks for your response. My use case is to use the same kedro pipeline for multiuser activity via gunicorn. But I was facing the issue of local kedro file getting replaced with new user if a user is running a process already. I was trying to make the pipeline dynamic by adding pid to filenames.
d

datajoely

02/23/2024, 10:27 AM
you could do an omegaconf resolver to add that I think
m

marrrcin

02/23/2024, 10:27 AM
Yeah, runtime param
m

Muhammed Afnas

02/23/2024, 10:47 AM
just found that omegaconfigloader is not available in kedro 0.18. also in this case, can i pass the runtime parameters directly by passing to session.run()?
n

Nok Lam Chan

02/23/2024, 11:12 AM
@Muhammed Afnas OmegaConfigLoader is available in 0.18, it should be non breaking to upgrade kedro version within 0.18.x series
this 1
Can you try bump it to 0.18.14?
2 Views