Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Hi folks,

I have a query regarding accessing the kedro catalog of a current kedro session inside pipeline.py file inside a pipeline. or if there is a way to access the MemoryDataset added via hook in the pipeline.py file.

so there are dataset hooks which you can utilise as they pass through the pipeline

Yeah, but first maybe you should determine why do you need to do that <@U04QZJM3B28>? Every time I see similar question there’s a hint that someone tries to do something against Kedro way of doing things.

Smells a lot like dynamic pipelines every time :see_no_evil:  THERE IS NO BUILT IN WAY TO ACCESS DATASETS / PARAMETERS DURING PIPELINE GENERATION.

Maybe we should add auto responder for that <@U03R8FW4HUZ> :smile:

or a video explaining our thoughts on the topic

Thank you datajoely!
hey <@U045L91RV9D>, thanks for your response.
My use case is to use the same kedro pipeline for multiuser activity via gunicorn. But I was facing the issue of local kedro file getting replaced with new user if a user is running a process already.
I was trying to make the pipeline dynamic by adding pid to filenames.

you could do an omegaconf resolver to add that I think

<https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-override-configuration-with-runtime-parameters-with-the-omegaconfigloader|https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-override-configuration-with-[…]rameters-with-the-omegaconfigloader>

just found that omegaconfigloader is not available in kedro 0.18.
also in this case, can i pass the runtime parameters directly by passing to session.run()?

<@U04QZJM3B28> OmegaConfigLoader is available in 0.18, it should be non breaking to upgrade kedro version within 0.18.x series