Hi everyone, I am facing the following issue when ...
# questionsa
Andreas_Kokolantonakis
05/09/2023, 2:51 PMHi everyone, I am facing the following issue when I am trying to read a CSV with spark. With pandas, it works fine but with spark, it seems I need some extra configurations. Could you please point me in the right direction? thank you in advance!
j
Juan Luis
05/09/2023, 2:57 PMwelcome @Andreas_Kokolantonakis! I think the file was deleted and cannot be seen
a
Andreas_Kokolantonakis
05/09/2023, 2:57 PMyep sorry
Andreas_Kokolantonakis
05/09/2023, 2:57 PMj
Javier del Villar
05/09/2023, 3:03 PMyou should add s3a:// to the filepath
Javier del Villar
05/09/2023, 3:05 PMsorry! Hi! hehehe
Javier del Villar
05/09/2023, 3:06 PM@Andreas_Kokolantonakis you see examples here: https://docs.kedro.org/en/stable/kedro_datasets.spark.SparkDataSet.html#kedro_datasets.spark.SparkDataSet
☝🏼 1
a
Andreas_Kokolantonakis
05/09/2023, 5:06 PMtried all the recommended approaches, still getting the same error
Andreas_Kokolantonakis
05/09/2023, 5:06 PMI used context.py to load a new session with s3 configurations. no luck, is anyone able to help me further please?! thanks
j
Juan Luis
05/09/2023, 5:06 PMcould you paste the traceback that corresponds to the
s3a://a
Andreas_Kokolantonakis
05/09/2023, 5:08 PM@Juan Luis
Andreas_Kokolantonakis
05/09/2023, 5:09 PMI used the pyspark starter, and added the s3AFile system as a configuration in the spark.yml file
Andreas_Kokolantonakis
05/09/2023, 5:09 PMbut still no luck
j
Javier del Villar
05/09/2023, 5:16 PMNow you have a different error now, you are missing the AWS connector in your spark's classpath
j
Juan Luis
05/09/2023, 5:16 PMthe error is slightly different now indeed:
Class org.apache.hadoop.fs.s3a.S3AFileSystem not founda
Andreas_Kokolantonakis
05/09/2023, 5:18 PMyep, but I am adding this config in spark.yml. How I can make sure that kedro will use ProjectContext(KedroContext) to initialise the spark session?
j
Javier del Villar
05/09/2023, 5:18 PMsearching the channel for that error I got:
https://kedro-org.slack.com/archives/C03RKP2LW64/p1681471022814169
a
Andreas_Kokolantonakis
05/09/2023, 5:19 PMwill try to follow this!
Andreas_Kokolantonakis
05/10/2023, 7:23 AMhi everyone, how I can make sure that kedro will use spark.yml and context.py when initializing a spark session? does it work out of the box or I need somehow to point at it? thanks!
j
Juan Luis
05/10/2023, 7:56 AMhi everyone, how I can make sure that kedro will use spark.yml and context.py when initializing a spark session? does it work out of the box or I need somehow to point at it? thanks!
n
Nok Lam Chan
05/10/2023, 9:52 AM@Andreas_Kokolantonakis What version of kedro are you using? If you are using 0.18.x you should be able to do this with https://docs.kedro.org/en/stable/integrations/pyspark_integration.html#initialise-a-sparksession-using-a-hook
j
Juan Luis
05/10/2023, 10:25 AMI think it's the other Andreas @Andreas_Kokolantonakis 😄
a
Andreas_Kokolantonakis
05/10/2023, 11:43 AM@Haris Michailidis
n
Nok Lam Chan
05/10/2023, 12:09 PM😅sorry I need to work on my skills at tagging people…
🙈 1
3 Views