any clue what could have been the issue i could be missing s

Question

any clue what could have been the issue i could be missing some minor thing here < Rob> is it similar to your issue could u help

Rob · Answer

Hey < meharji arumilli> as I understood as Spark has a lazy execution if prev your processed data haven t performed an action you should set it as a `MemoryDataSet` So in you catalog place something like this ```your pipeline type MemoryDataSet copy mode assign``` Notice the copy mode and for more reference check this doc

Rob · Answer

Note If you haven t performed an action an you set the `copy mode= deecopy ` default setting I guess probably will fail since your data is still part of the Spark s Logical Plan

meharji arumilli · Answer

afaik ```your pipeline type MemoryDataSet copy mode assign``` should be used when Sometimes you might want to use Spark objects that aren t `DataFrame` as inputs and outputs in your pipeline as per the docs <https kedro readthedocs io en 0 17 3 11 tools integration 01 pyspark html use kedro s built in spark datasets to load and save raw data>

meharji arumilli · Answer

here my object `preprocessed data ` is a sparkDataFrame and here im trying to save to local storage

meharji arumilli · Answer

```preprocessed data type spark SparkDataSet filepath data $ project 05 model input df preprocessed parquet file format parquet```

meharji arumilli · Answer

should i still place this in catalog ```preprocessed data type MemoryDataSet copy mode assign```

Rob · Answer

Try doing that if you haven t performed an Spark Action this works on dataframes that have only been exposed to Spark Transformations Otherwise be sure to execute the action over your dataframe But you can try both

meharji arumilli · Answer

```preprocessed data type MemoryDataSet copy mode assign preprocessed data type spark SparkDataSet filepath data $ project 05 model input df preprocessed parquet file format parquet```

meharji arumilli · Answer

```but this already has an issue as we use `preprocessed data` twice in the catalog```

Rob · Answer

``preprocessed data`` is the name of your output I think Place it as ``` lt Pipeline Name gt type MemoryDataSet copy mode assign preprocessed data type spark SparkDataSet filepath data $ project 05 model input df preprocessed parquet file format parquet```

Rob · Answer

Output sorry

meharji arumilli · Answer

``` feature engineering type MemoryDataSet copy mode assign preprocessed data type spark SparkDataSet filepath data $ project 05 model input df preprocessed parquet file format parquet```

meharji arumilli · Answer

i have added now as u suggested and the error code is different now

meharji arumilli · Answer

`raise Py4JJavaError ` `py4j protocol Py4JJavaError An error occurred while calling o742 save ` ` java lang ClassNotFoundException cloud PathOutputCommitProtocol` `at URLClassLoader findClass URLClassLoader java 387 ` `at java lang ClassLoader loadClass ClassLoader java 418 ` `at java lang ClassLoader loadClass ClassLoader java 351 `

Rob · Answer

Mmm yes now that a Java issue

Rob · Answer

Be sure to have the Java JDK 8 and Spark correctly set up

meharji arumilli · Answer

`$ java version` `openjdk version 1 8 0 292 ` `OpenJDK Runtime Environment AdoptOpenJDK build 1 8 0 292 b10 ` `OpenJDK 64 Bit Server VM AdoptOpenJDK build 25 292 b10 mixed mode `

Rob · Answer

I did this using Windows but what worked for me was setting the <https github com robguilarr spark winutils 3 3 1|hadoop dll> file and the winutils exe according to your spark version + Plus setting your system variables If you are using MacOS or Linux I d recommend you to ask the community

meharji arumilli · Answer

hmm i did installed hadoop 3 3 4 on mac but probably need to re do this to make sure things are ok

meharji arumilli · Answer

but could the java error be related to this

Rob · Answer

It s better to ask being honest

meharji arumilli · Answer

ok thanks BTW do u happen to save and load model pipeline files in pyspark + kedro framework would be helpful if u can give a hint

Rob · Answer

I haven t worked with models yet And you re welcome

meharji arumilli · Answer

slightly smiling face

meharji arumilli · Answer

thanks