Hello I have a catalog using transcoding pandas spark see re

Question

Hello I have a catalog using transcoding pandas spark see reply in the thread to see how it looks I then have a node that saves output in pandas version e g `df pandas` and the node right after that loads it in spark e g `df spark` When running the whole pipeline it throws the error below when it tries to load `df spark` Interestingly though if I restart the pipeline from the node that failed the error is no longer produced and the pipeline runs fine Any ideas what could be the issue pyarrow version is 6 0 1 Thanks ``` Traceback most recent call last TRUNCATED NotADirectoryError blahblah user giulio 05 model input master parquet 2023 10 09T18 45 28 662Z master parquet The above exception was the direct cause of the following exception TRUNCATED DataSetError Failed while loading data from data set ParquetDataSet filepath=blahblah user giulio 05 model input master parquet load args= engine pyarrow protocol=s3 save args= engine pyarrow index False version=Version load=None save= 2023 10 09T18 45 28 662Z dptx fingerprint user giulio myc screen 1k 05 model input master parquet 2023 10 09T18 45 28 662Z master parquet```

Giulio Morina · Answer

``` spark parquet ds spark parquet type spark SparkDataSet file format parquet versioned True layer model input load args header True save args mode overwrite header True pandas parquet ds pandas parquet layer model input versioned True type pandas ParquetDataSet save args index False Ensure no extra col with index is created when loading with Spark engine pyarrow df spark << spark parquet filepath df pandas << pandas parquet filepath ```