Rachid Cherqaoui
07/17/2023, 9:04 PMPartitionedDataSet
function from <http://kedro.io|kedro.io>
to load a data but I've just seen that this function doesn't take the delimiter into account, how can I solve this? (knowing that I'm working on csv files on my local, here is the code used : data_set = PartitionedDataSet(
path = "data/01_raw/Tableaux",
dataset= CSVDataSet,
filename_suffix= ".csv",
load_args= {"delimiter": ";", "header": 0,"encoding": "utf-8"}
Deepyaman Datta
07/17/2023, 9:17 PMload_args
should be on the underlying dataset, not on the PartitionedDataset
. See https://github.com/kedro-org/kedro/blob/main/kedro/io/partitioned_dataset.py#L166-L167. Pass a dict, if you're doing it in Python and not YAML.Rachid Cherqaoui
07/17/2023, 9:20 PMDeepyaman Datta
07/17/2023, 9:30 PM>>> pds = PartitionedDataset(path="kedro/", dataset={"type": "pandas.CSVDataSet", "load_args": {"delimiter": ";"}}, filename_suffix=".csv")
>>> pds.load()
{'blah': <bound method AbstractVersionedDataSet.load of <kedro_datasets.pandas.csv_dataset.CSVDataSet object at 0x149d97850>>, 'foo': <bound method AbstractVersionedDataSet.load of <kedro_datasets.pandas.csv_dataset.CSVDataSet object at 0x149d97940>>}
>>> pds.load()["blah"]()
dog eat dog.1
0 1 2 3
1 4 5 6
>>> pds.load()["foo"]()
cats eat mice
0 5 6 7
1 8 9 0
>>>
(kedro) deepyaman@Deepyamans-MacBook-Air kedro % cat kedro/*.csv
dog;eat;dog
1;2;3
4;5;6
cats;eat;mice
5;6;7
8;9;0
Rachid Cherqaoui
07/17/2023, 9:30 PM