This is sort of a follow up on previous question that was solved. Say I have a project that ingests a large dataset, but I only process a part of it -- it is a big time series, I want to process a specific month. I want to pass a CLI argument to do that, which I currently can by overriding a parameter. However, I'd also like the output to be written to different places depending on the argument (that is, I want the e.g. filename to be prefixed with the CLI argument).
Turns out that CLI parameters do not override global parameters, which is what is used when templating the catalog.
So I'm stuck on how to do this without having to rewrite the global parameters file before running kedro.
Nok Lam Chan
03/17/2023, 8:15 PM
While templating catalog may work, it sounds like PartitionedDataSet can be a good fit here
03/18/2023, 12:43 PM
Thanks @Nok Lam Chan! I think it will work! I never thought about using