Hey team Is there a way to have pipeline specific parameters.yml, and spark.yml? I'd ideally like to colocate pipeline config in the same folder as the pipelines for easier collaboration and I would like pipeline runs to only load pipeline specific configurations I guess what I am looking for is "composable pipeline projects"

Hey Ahmed, just to clarify, what you want to do is to put the

parameters.yml

file in the same directory as your pipeline files, like

nodes.py

and

pipeline.py

?

Yeah and catalog.yml I guess I am trying to understand why are they top level in conf/ and what is the design decision behind that Also, what would that look like in large pipeline projects?

Kedro’s design philosophy is that configuration should be global, reproducible, and environment-aware, and that pipelines should be modular in code, not in config loading. The config refers to the Kedro project as a whole, this is why it's at top level. On top of that, the config is loaded before the pipelines as well. And this allows you to share things like parameters, datasets, credentials, etc. between multiple pipelines without need for code duplication. You can check the docs for some options of how you can organize your parameters in a way that works for your project: https://docs.kedro.org/en/stable/configure/configuration_basics But to load it from the pipeline directory itself, you'd need to change the way the ConfigLoader works.