Hello
I have a problem when using CSVDataSet and d...
# questionsd
Dawid Bugajny
10/13/2023, 8:44 AMHello
I have a problem when using CSVDataSet and downloading data from Azure (abfs protocol) - after 8h the operation is canceled due to timeout. This does not happen every time, but I would like to avoid such a situation. Had anyone had this kind of problem? Maybe there is a solution other than a try-catch implementation?
d
datajoely
10/13/2023, 8:57 AMIf this is the case I would think about subclassing the dataset and building in a retry
m
marrrcin
10/13/2023, 9:00 AMDo you have a CSV so large that it takes >8h to read it?
d
datajoely
10/13/2023, 9:00 AMoh I misread that!
datajoely
10/13/2023, 9:00 AMI thought it said 8th!
datajoely
10/13/2023, 9:00 AMgood point!
d
Dawid Bugajny
10/13/2023, 9:07 AMDataset has only 37.6 MB, so it isn't so large.
d
datajoely
10/13/2023, 9:34 AMyeah this should take milliseconds to complete
datajoely
10/13/2023, 9:35 AMSomething I like to do to prototype these sort of things:
ā¢ open a notebook
ā¢ import
from kedro.datasets.pandas import CSVDataSetimportlibš 1
m
marrrcin
10/13/2023, 9:42 AMI would verify the networking and configuration (of the execution environment, not Kedro) first; then plain python with fsspec then Kedro
š 1
d
Dawid Bugajny
10/13/2023, 9:45 AMOkey, thanks š