Hello I have a problem when using CSVDataSet and downloading

Question

Hello I have a problem when using CSVDataSet and downloading data from Azure abfs protocol after 8h the operation is canceled due to timeout This does not happen every time but I would like to avoid such a situation Had anyone had this kind of problem Maybe there is a solution other than a try catch implementation

datajoely · Answer

If this is the case I would think about subclassing the dataset and building in a retry

marrrcin · Answer

Do you have a CSV so large that it takes gt 8h to read it

datajoely · Answer

oh I misread that

datajoely · Answer

I thought it said 8th

datajoely · Answer

good point

Dawid Bugajny · Answer

Dataset has only 37 6 MB so it isn t so large

datajoely · Answer

yeah this should take milliseconds to complete

datajoely · Answer

Something I like to do to prototype these sort of things open a notebook import `from kedro datasets pandas import CSVDataSet` Construct the dataset there to test the right configuration All Kedro does behind the scenes is pass your YAML config to `importlib` so this is a really nice way to tinker with the config quickly

marrrcin · Answer

I would verify the networking and configuration of the execution environment not Kedro first then plain python with fsspec then Kedro

Dawid Bugajny · Answer

Okey thanks slightly smiling face