How to combine multiple similar datasets into one in Kedro dynamically?
I use Kedro to run pipelines for multiple models that each generate a CSV file with the same schema. However, I need to combine each of the generated CSV files into one and do some post processing which will be my final output.
This is what my catalog looks like:
catalog.yml
model1.output:
type: pandas.CSVDataSet
filepath: ${data_path}/output/model1/output.csv
model2.output:
type: pandas.CSVDataSet
filepath: ${data_path}/output/model2/output.csv
model3.output:
type:...