https://kedro.org/ logo
#questions
Title
# questions
j

Juan Marin

01/19/2023, 12:32 PM
Hey folks! Just started using kedro. Is there any
kedro
command to import datasets from a path into my data directory in the project? Thanks!
๐Ÿ‘€ 1
d

datajoely

01/19/2023, 12:57 PM
itโ€™s possible to do via hooks, but we believe this pattern is a blocker to reproducibility
j

Juan Luis

01/19/2023, 1:50 PM
what about something that copies the files to
data/
and creates some basic config files? (I think this is more in line with what @Juan Marin had in mind, correct me if I'm wrong Juan)
j

Juan Marin

01/19/2023, 2:10 PM
Yeah, my idea was a command which you give a path for a certain file (eg reviews.csv in my downloads folder) and it will add it to your catalog.yml with its corresponding type and a specified filepath (eg: data/01_raw) where the data will also be added
d

datajoely

01/19/2023, 2:14 PM
well you can use the
PartitionedDataSet
or
IncrementalDataSet
to scoop up new files which get added https://kedro.readthedocs.io/en/stable/data/kedro_io.html#incremental-loads-with-incrementaldataset
๐Ÿ‘ 1
2 Views