Sergey S
11/13/2023, 9:18 PMcatalog.yaml
? I am asking, because there was this handy kedro plug-in kedro-wings, unfortunately the last commit is from 3 years ago and is probably outdated for modern Kedro.
The feature itself was very handy, the plug-in would infer the dataset types from the input and output names (and so using the appropriate dataset class to load and save), e.g.:
node(
split_data,
inputs=['01_raw/iris.csv', 'params:example_test_data_ratio'],
outputs=dict(
train_x="02_intermediate/example_train_x.csv"
train_y="02_intermediate/example_train_y.csv"
)
Kedro-wings would automatically populate catalog.yaml
. Concrete use case - training runs with 10 plots and 3 txt log files. Currently, I would have to manually define 13x datasets in catalog.yaml
and manually maintain it if I add or remove node saving data to disk.
Is something like this possible today or a different recommended way of solving this?marrrcin
11/13/2023, 9:25 PMSergey S
11/13/2023, 9:28 PM