Hello fellow Kedroids kedro robot face I m having a very str

Question

Hello fellow Kedroids kedro robot face I m having a very strange issue when saving a file to parquet I m getting this error `DataSetError Failed while saving data to data set ParquetDataSet filepath= Users jose darnott PycharmProjects planta litio data 01 raw data sql parquet load args= engine pyarrow protocol=file save args= engine pyarrow Duplicate column names found timestamp lims BFIL CO3 lims BFIL Ca ` It s basically showing all the columns inside the dataframe here I m showing only 3 of them My catalog entry looks like this ```data sql type pandas ParquetDataSet filepath data 01 raw data sql parquet load args engine pyarrow save args engine pyarrow layer III``` I m using kedro==0 18 8 pandas==2 0 1 pyarrow==12 0 0 The problem is quite similiar to this issue from 2022 but in my case removing the load and save args as the OP mentions won t solve my problem This is quite puzzling since I just did a clipboard inside the node before returning my output open it on a jupyter notebook and I see no problems with the dataframe I can even save it to parquet without any issues So that makes me thing the problem comes from kedro Anyways as a workaround I m saving the dataframe as csv and it s working just fine But I d like to find a way to make the parquet work again since this is a huge file Thanks in advance party parrot

Nok Lam Chan · Answer

Can you post the full traceback or try to scroll up the error a bit This is raised by Kedro but there should be an error throw by either pandas or pyarrow I suspect

Nok Lam Chan · Answer

And do you have `kedro datasets` installed

Jose Nuñez · Answer

yes kedro datasets==1 2 0

Jose Nuñez · Answer

``` 06 12 23 11 38 31 INFO Kedro project planta litio session py 360 06 12 23 11 38 34 INFO Running node get local credentials get sql credentials None gt sql credentials node py 329 INFO Saving data to sql credentials SharedMemoryDataSet data catalog py 382 06 12 23 11 38 35 INFO Loading data from diccionario raw ExcelDataSet data catalog py 343 INFO Loading data from sql credentials SharedMemoryDataSet data catalog py 343 INFO Loading data from params sql data date init MemoryDataSet data catalog py 343 INFO Loading data from params sql data date end MemoryDataSet data catalog py 343 INFO Running node download data sql azure node py 329 download data azure diccionario raw sql credentials params sql data date init params sql data date end gt data sql 06 12 23 11 38 36 WARNING Users jose darnott PycharmProjects planta litio src planta litio pipelines pre processing nodes py 344 UserWarning pandas warnings py 109 only supports SQLAlchemy connectable engine connection or database string URI or sqlite3 DBAPI2 connection Other DBAPI2 objects are not tested Please consider using SQLAlchemy df = pd read sql query data db connection 06 12 23 11 38 41 INFO Saving data to data sql ParquetDataSet data catalog py 382 RemoteTraceback Traceback most recent call last File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro io core py line 214 in save self save data File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro datasets pandas parquet dataset py line 188 in save data to parquet bytes buffer self save args File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages pandas core frame py line 2889 in to parquet return to parquet File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages pandas io parquet py line 411 in to parquet impl write File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages pandas io parquet py line 159 in write table = self api Table from pandas df from pandas kwargs File pyarrow table pxi line 3681 in pyarrow lib Table from pandas File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages pyarrow pandas compat py line 570 in dataframe to arrays convert fields = get columns to convert df schema preserve index File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages pyarrow pandas compat py line 352 in get columns to convert raise ValueError ValueError Duplicate column names found timestamp The above exception was the direct cause of the following exception Traceback most recent call last File Users jose darnott opt miniconda3 envs planta litio lib python3 8 concurrent futures process py line 239 in process worker r = call item fn call item args call item kwargs File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro runner parallel runner py line 122 in run node synchronization return run node node catalog hook manager is async session id File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro runner runner py line 319 in run node node = run node sequential node catalog hook manager session id File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro runner runner py line 435 in run node sequential catalog save name data File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro io data catalog py line 384 in save dataset save data File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro io core py line 613 in save super save data File Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro io core py line 221 in save raise DataSetError message from exc kedro io core DataSetError Failed while saving data to data set ParquetDataSet filepath= Users jose darnott PycharmProjects planta litio data 01 raw data sql parquet load args= engine pyarrow protocol=file save args= engine pyarrow Duplicate column names found timestamp The above exception was the direct cause of the following exception ```

Jose Nuñez · Answer

``` Traceback most recent call last Users jose darnott opt miniconda3 envs planta litio bin kedro 8 in lt module gt Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro framework cli cli py 211 in main 208 209 init plugins 210 cli collection = KedroCLI project path=Path cwd 211 cli collection 212 Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages click core py 1 130 in call Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro framework cli cli py 139 in main 136 137 138 try 139 super main 140 args=args 141 prog name=prog name 142 complete var=complete var Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages click core py 1 055 in main Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages click core py 1 657 in invoke Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages click core py 1 404 in invoke Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages click core py 7 60 in invoke Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro framework cli project py 472 in run 469 with KedroSession create 470 env=env conf source=conf source extra params=params 471 as session 472 session run 473 tags=tag 474 runner=runner is async=is async 475 node names=node names Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro framework session session py 426 in run 423 424 425 try 426 run result = runner run 427 filtered pipeline catalog hook manager session id 428 429 self run called = True Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro runner ru nner py 91 in run 88 89 Asynchronous mode is enabled for loading and saving data 90 91 self run pipeline catalog hook manager session id 92 93 Pipeline execution completed successfully 94 Users jose darnott opt miniconda3 envs planta litio lib python3 8 site packages kedro runner pa rallel runner py 334 in run 331 break pragma no cover 332 done futures = wait futures return when=FIRST COMPLETED 333 for future in done 334 node = future result 335 done nodes add node 336 337 Decrement load counts and release any datasets we Users jose darnott opt miniconda3 envs planta litio lib python3 8 concurrent futures base py 4 37 in result 434 if self state in CANCELLED CANCELLED AND NOTIFIED 435 raise CancelledError 436 elif self state == FINISHED 437 return self get result 438 439 self condition wait timeout 440 Users jose darnott opt miniconda3 envs planta litio lib python3 8 concurrent futures base py 3 89 in get result 386 def get result self 387 if self exception 388 try 389 raise self exception 390 finally 391 Break a reference cycle with the exception in self exception 392 self = None DataSetError Failed while saving data to data set ParquetDataSet filepath= Users jose darnott PycharmProjects planta litio data 01 raw data sql parquet load args= engine pyarrow protocol=file save args= engine pyarrow Duplicate column names found timestamp ```

Nok Lam Chan · Answer

Inteesting can you double check if it works with `<http df to|df to> parquet` ``` bytes buffer = BytesIO data to parquet bytes buffer self save args ``` This is literally what Kedro s does behind the scene we are using the same logic whatever pandas is doing Kedro handles the path but nothing more

Nok Lam Chan · Answer

This is weird Please try to do this 1 `<http df to|df to> parquet` see if it works 2 If 1 works then try ``` from io import BytesIO bytes buffer = BytesIO data to parquet bytes buffer ```

Nok Lam Chan · Answer

If 1 works but 2 fail then I would suggest to create a minimal reproducible example and open an issue on either `pandas` or `pyarrow`

Jose Nuñez · Answer

I just added the <http df to|df to> parquet just above the return of the node and it fails in same fashion as above so you right this is probably not a direct problem with kedro

Nok Lam Chan · Answer

Maybe try to pickle it and load it from a notebook There is good chance that there are just duplicate columns

Nok Lam Chan · Answer

Or if you are Pycharm VSCode user it maybe easier to attach a debugger and step through the breakpoints <https docs kedro org en stable development set up pycharm html debugging>