Hello! I'd appreciate a bit of help with an issue that I'm facing. I have a kedro pipeline that is writing artifacts to a s3 bucket. In my project, I've got various envs; staging, dev, various experiments, etc. I'd like to separate the artifacts based on the env, but I only have one s3 bucket, so I was thinking to just do it with directories in my bucket, like so:

"example_data_{x}_{y}":
  type: pandas.ParquetDataset
  filepath: "<s3://path/to/{ENV}/rest/of/path/example_data_{x}_{y}.pqt>"
  credentials: my_s3_creds

Currently

ENV

is an environment variable. I have figured out how to make the artifact's name dynamic, but how can I make the artifact's s3 filepath dynamic as well? Many thanks in advance! I'm using kedro

0.19.4

Can it be done in a static way, like literally have this configuration in all the environments you mentions with a different default? That’s one of the point of having a catalog file that describes all the data in a specific environment

Maybe? Maybe I never quite understood why the catalog was designed to be static, and if I understood that then I'd have my solution? As it is, I have a single code repository, and my deployment system reads my code repository and adds its own env vars, which affect what data sources (mostly SQL) that I read from and write to when the code executes. Should I have more than one catalog.yml maybe? In that case, how do I tell kedro to read one and not the other (based on an env var)?

Kedro run --env? I think this is exactly why it should be static, dynamic config are hard to understand and deployment team are not the one who build the pipeline. They should have control of things that they need to control but not for configuration like hyperparameter of your model. https://docs.kedro.org/en/0.19.6/configuration/configuration_basics.html#configuration-environments Have you perhaps seen this already?