Hi, I’m on kedro 0.18.3 trying to override a templated variable in the data catalog with runtime configuration. So

catalog.yml

has

filepath: "${configurable_filepath}"

and I’d like to do

kedro run --params configurable_filepath:/path/to/file

. A similar question was asked previously https://linen-discord.kedro.org/t/2203662/Hi-all-I-have-a-beginner-question-on-Kedro-0-18-2-I-have-a-T with writing a custom TemplatedConfigLoader as solution: https://github.com/noklam/kedro_gallery/blob/master/template_config_loader_demo/src/template_config_loader_demo/settings.py Is this the recommended approach or is there a way of achieving what I want without writing a custom TemplatedConfigLoader accessing private variables? Is there no other way to add all runtime parameters to the global dict? I’d really want to avoid that if possible in case a future kedro update changes things.

In general we don’t expect filepaths to be dynamically it makes reproducibility very difficult, is why it’s not natively supported

The example would be running the same job/pipeline with multiple different inputs, i.e. filepaths

Well we have a solution for that called Modular Pipelines https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html

Say you need to run a pipeline for a varying number of inputs, where the inputs may come from a query in a database. That doesn’t seem compatible, or at least convenient, with a static catalog.

ah gotcha - that goes against the principle of reproducibility

I don’t see that.. If I really wanted to reproduce it I could keep track of all the inputs I’ve had over time and reproduce it. It’s just not convenient to represent the input as something static

so you can absolutely achieve this, probably the best way to do so is with a pipeline hook which mutates the catalog

But… I’m not here to argue about kedro’s design choices ofc 🙂

but it’s a piece where we’re opinionated on because it makes things difficult to maintain / debug in the past

Yes, ok. I’m just thinking in reality data source are often changing. At least in my world 🙂 But then I know there is no “easy” way.

hooks are super nice to work with

ok, thanks