Hi, everyone! I'm trying to parametrize a folder name via

--params

at runtime, but Kedro isn’t picking it up. In my

parameters.yml

I have:

data_ingestion:
  queries:
    queries_folder: "${runtime_params:folder}"

Then, in the pipeline creation:

conf_path = str(settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path)
params = conf_loader["parameters"]

queries_folder = params["data_ingestion"]["queries"]["queries_folder"]

query_files = [f for f in os.listdir(queries_folder) if f.endswith(".sql")]

When I run:

kedro run -p data_ingestion_s3 --params=folder=custom_folder

I get an error saying

"folder " not found, and no default value provided.

Has anyone used runtime parameters inside parameter files like this? Do you know if this is expected, or should I be loading params differently? I would appreciate any guidance you could give me! Thanks 🙂 Note: I am using kedro version 1.0.0

Hi @Lívia Pimentel, I tried reproducing this but could not find any issue. I wonder why the error which says "folder " not found - with a trailing space.

@Ravi Kumar Pilla I can reproduce: https://github.com/deepyaman/new-kedro-project

I wonder why the error which says "folder " not found - with a trailing space.

I feel like this is just a mistake. @Lívia Pimentel Is this in line with where you're manually instantiating the config loader? I tried both in

pipeline_registry.py

and in a

create_pipeline

function, and I got the same error as you both times. Haven't had a chance to investigate more.

Hello, @Ravi Kumar Pilla and @Deepyaman Datta. Thank you for your responses. Yes, that aligns with where I was instantiating the config loader. Actually, I also tried a simpler example and encountered the same error: In parameters.yml:

model_options:
  random_state: "${runtime_params:random}"

When running the same pipeline with kedro run -p <pipeline_name> --params random=3, I got:

random not found, and no default value provided.

I'm wondering if this might be related to how runtime_params is being loaded or passed into the configuration context. Do I need to adjust the config loader settings, or is there a specific place where runtime_params must be defined for this to work?

Hi @Lívia Pimentel, Apologies for the delay in response. A bit of context - When you do

kedro run --params

, behind the scene, a KedroSession is created and this creates a session store which stores the runtime_params. Then a

session.run

is triggered which creates a

KedroContext

. KedroContext needs a ConfigLoader instance while instantiation and we inject runtime_params at this stage while instantiating ConfigLoader. This is how it works automatically. --- Issue - Coming to the issue above, when you try to create a

conf_loader

i.e., manually -

conf_loader = OmegaConfigLoader(conf_source=conf_path)

, you are not passing the runtime_params. So there is no way conf_loader would know what the runtime param is. You can read more about what OmegaConf is used for here from docstring - https://github.com/kedro-org/kedro/blob/main/kedro/config/omegaconf_config.py#L43 --- Solution - To summarize, OmegaConf is used to resolve config files that are present on disk. If you want the resolution to take into account the runtime_params, you need to pass them manually

conf_loader = OmegaConfigLoader(conf_source=conf_path, runtime_params={"folder": "test"})

or let kedro handle it via

KedroSession

and

KedroContext

. If you have further questions, please let us know. Thank you

Thanks for the details, @Ravi Kumar Pilla. @Lívia Pimentel I think this seems similar to https://github.com/kedro-org/kedro/issues/2169#issuecomment-1722914953; would

kedro-boot

help you? I got interested last night and tried it out—long overdue. I've updated the repo: https://github.com/deepyaman/new-kedro-project/tree/9f14fc99ffe86e3e79d251d1f13de0da082f7b6b You can run:

kedro run -p data_processing --itertime-params=folder=custom_folder,companies_filename=companies

As you can see, injecting parameters into dataset definitions works, but I still don't know/haven't figured out how to load parameters from the session. Some questions: • For @Lívia Pimentel: What are you trying to accomplish with the SQL queries folder? I wonder if this is something that could be represented as a dataset. • @Takieddine Kadiri @Yolan Honoré-Rougé 🙏 (or whoever): Do you know if there's any reason

kedro-boot

doesn't respect the resolver for params? Is this a limitation related to the lifecycle stage @Ravi Kumar Pilla mentioned above, or is it just not implemented? • More generally: I can't recall/figure out if there's anyway way to get a copy of the session object. I think no... but just want to confirm. That would probably make a lot of things easier. ◦ Sub-question: Is

kedro-boot

different in any way here, that you can easily refetch the active session. • If this all works using

kedro-boot

, the repo I have had did a fair bit of hacking to include that as the session class for run; wonder if it would be possible to make

KedroSessionClass

configurable, too. 🙂 I think it should be.