Hey folks! Just started using kedro. Is there any

kedro

command to import datasets from a path into my data directory in the project? Thanks!

it’s possible to do via hooks, but we believe this pattern is a blocker to reproducibility

what about something that copies the files to

data/

and creates some basic config files? (I think this is more in line with what @Juan Marin had in mind, correct me if I'm wrong Juan)

Yeah, my idea was a command which you give a path for a certain file (eg reviews.csv in my downloads folder) and it will add it to your catalog.yml with its corresponding type and a specified filepath (eg: data/01_raw) where the data will also be added

well you can use the

PartitionedDataSet

or

IncrementalDataSet

to scoop up new files which get added https://kedro.readthedocs.io/en/stable/data/kedro_io.html#incremental-loads-with-incrementaldataset

mentioned this thread in https://github.com/kedro-org/kedro/issues/3919#issuecomment-2151576857