Rob
02/13/2023, 4:52 PM{category}_output_1.parquet
, {category}_output_2.parquet
and so on...
Any alternative suggestion is welcome 🙂datajoely
02/13/2023, 4:56 PMRob
02/13/2023, 5:00 PMdatajoely
02/13/2023, 5:01 PMafter_pipeline_run
hookRob
02/13/2023, 5:02 PMIan Whalen
02/13/2023, 5:06 PMRob
02/13/2023, 5:11 PMFlorianGD
02/13/2023, 5:19 PMTemplatedConfigLoader
, where you update the globals_dict
with the runtime_params
. This way, the parameters that you pass can be fed into the catalog definition. Then, in your catalog.yml
, you can define your entry this way:
output1:
type: pandas.ParquetDataSet
path: ${category:NOT_DEFINED}_output_1.parquet
Then when you run your pipeline; kedro run --pipeline my_pipeline --params category:category1
, the path will be updated according to the passed parameterNOT_DEFINED
, but according to your use case, not having a default and failing when you do not provide the parameter could be a safer/better choiceRob
02/13/2023, 5:30 PMSebastian Pehle
02/14/2023, 7:13 PM