Hi all! I have a kedro project that is being intia...
# questionsa
Anirudh Dahiya
02/01/2023, 1:14 PMHi all! I have a kedro project that is being intiated with a pyspark session. Till date, I never had any issues when running pipelines or opening a jupyter notebook from my project's directory. However today I am facing this error -
Copy code
Exception: Java gateway process exited before sending its port numberd
datajoely
02/01/2023, 1:43 PMCan you kill the kernel and see if persists?
a
Anirudh Dahiya
02/01/2023, 1:45 PMIt does. Not sure, what is causing the error.
d
datajoely
02/01/2023, 1:46 PMand are you running against a remote spark cluster or all locally
a
Anirudh Dahiya
02/01/2023, 1:46 PMremote spark cluster
d
datajoely
02/01/2023, 1:46 PMand you’re configuring your spark stuff with
spark.yamla
Anirudh Dahiya
02/01/2023, 1:47 PMspark.yml in kedro
d
datajoely
02/01/2023, 1:48 PMOkay messing around with those variables is really the only course of action
datajoely
02/01/2023, 1:48 PMall Kedro is doing is passing those arguments to the session builder stuff
Copy code
SparkSession.builder.appName('myapp').getOrCreatedatajoely
02/01/2023, 1:49 PMa
Anirudh Dahiya
02/01/2023, 1:51 PMAlright thank you - will look into it, would it be helpful if I paste the contents of my spark.yml here?
d
datajoely
02/01/2023, 1:51 PMMaybe if someone else sees it
datajoely
02/01/2023, 1:51 PMbut I’m not sure
a
Anirudh Dahiya
02/01/2023, 1:51 PMAlright, let me fiddle around with it
Anirudh Dahiya
02/01/2023, 1:51 PMThank you for your help
d
datajoely
02/01/2023, 1:52 PMgood luck! sorry can’t do more here
a
Anirudh Dahiya
02/01/2023, 1:52 PMThat's alright, you've given me something to work with
o
Olivia Lihn
02/01/2023, 2:02 PMhi @Anirudh Dahiya if you want share the contents of spark.yml and i can take a look at those
a
Anirudh Dahiya
02/01/2023, 2:03 PMHi @Olivia Lihn, thank you. Please find below the configuration of spark.yml
Anirudh Dahiya
02/01/2023, 2:03 PM Copy code
# You can define spark specific configuration here.
spark.sql.execution.arrow.pyspark.enabled: true
spark.ui.port: 4050
spark.driver.bindAddress: 127.0.0.1
spark.driver.memory: 180g
spark.driver.maxResultSize: 70g
spark.driver.memoryOverhead: 40g
spark.network.timeout: 1000s
spark.hadoop.fs.s3a.connection.maximum: 1000
spark.debug.maxToStringFields: 10000
spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.broadcastTimeout: 600
spark.executor.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.driver.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
spark.local.dir: /data1/temp
spark.sql.autoBroadcastJoinThreshold: 50000000
spark.speculation: true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: 2
spark.jars.packages: org.apache.hadoop:hadoop-aws:2.9.2,com.databricks:spark-redshift_2.11:2.0.1,org.apache.avro:avro:1.8.1,org.apache.spark:spark-avro_2.11:2.4.4
spark.jars: /packages/RedshiftJDBC42-no-awssdk-1.2.55.1083.jar
# <https://kedro.readthedocs.io/en/stable/tools_integration/pyspark.html#tips-for-maximising-concurrency-using-threadrunner>
spark.scheduler.mode: FAIRo
Olivia Lihn
02/01/2023, 2:04 PMare you sure the cluster is configured correctly? If this same spark.yml configuration file worked running pipelines from your cli and on jupyter notebook, then it might be because there's a difference between you remote cluster configuration and this file
Olivia Lihn
02/01/2023, 2:05 PMwhat i would do is, if you can, check connecting or creating a spark session in your cluster and verify the config above works (as a first step)
a
Anirudh Dahiya
02/01/2023, 2:14 PMAlright let me do that
Anirudh Dahiya
02/01/2023, 2:15 PMThank you - is there a way to check the configuration of the remote cluster?
o
Olivia Lihn
02/01/2023, 2:15 PMtry connecting to the cluster and sending a very simple spark job (something like "hello world")
11 Views