#questions
Title
# questions
a
Anirudh Dahiya
02/01/2023, 1:14 PMHi all! I have a kedro project that is being intiated with a pyspark session. Till date, I never had any issues when running pipelines or opening a jupyter notebook from my project's directory. However today I am facing this error -
Copy code
Exception: Java gateway process exited before sending its port numberd
datajoely
02/01/2023, 1:43 PMCan you kill the kernel and see if persists?
a
Anirudh Dahiya
02/01/2023, 1:45 PMIt does. Not sure, what is causing the error.
d
datajoely
02/01/2023, 1:46 PMand are you running against a remote spark cluster or all locally
a
Anirudh Dahiya
02/01/2023, 1:46 PMremote spark cluster
d
datajoely
02/01/2023, 1:46 PMand you’re configuring your spark stuff with
spark.yamla
Anirudh Dahiya
02/01/2023, 1:47 PMspark.yml in kedro
d
datajoely
02/01/2023, 1:48 PMOkay messing around with those variables is really the only course of action
all Kedro is doing is passing those arguments to the session builder stuff
Copy code
SparkSession.builder.appName('myapp').getOrCreatea
Anirudh Dahiya
02/01/2023, 1:51 PMAlright thank you - will look into it, would it be helpful if I paste the contents of my spark.yml here?
d
datajoely
02/01/2023, 1:51 PMMaybe if someone else sees it
but I’m not sure
a
Anirudh Dahiya
02/01/2023, 1:51 PMAlright, let me fiddle around with it
Thank you for your help
d
datajoely
02/01/2023, 1:52 PMgood luck! sorry can’t do more here
a
Anirudh Dahiya
02/01/2023, 1:52 PMThat's alright, you've given me something to work with
o
Olivia Lihn
02/01/2023, 2:02 PMhi @Anirudh Dahiya if you want share the contents of spark.yml and i can take a look at those
a
Anirudh Dahiya
02/01/2023, 2:03 PMHi @Olivia Lihn, thank you. Please find below the configuration of spark.yml
Copy code
# You can define spark specific configuration here.
spark.sql.execution.arrow.pyspark.enabled: true
spark.ui.port: 4050
spark.driver.bindAddress: 127.0.0.1
spark.driver.memory: 180g
spark.driver.maxResultSize: 70g
spark.driver.memoryOverhead: 40g
spark.network.timeout: 1000s
spark.hadoop.fs.s3a.connection.maximum: 1000
spark.debug.maxToStringFields: 10000
spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.broadcastTimeout: 600
spark.executor.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.driver.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
spark.local.dir: /data1/temp
spark.sql.autoBroadcastJoinThreshold: 50000000
spark.speculation: true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: 2
spark.jars.packages: org.apache.hadoop:hadoop-aws:2.9.2,com.databricks:spark-redshift_2.11:2.0.1,org.apache.avro:avro:1.8.1,org.apache.spark:spark-avro_2.11:2.4.4
spark.jars: /packages/RedshiftJDBC42-no-awssdk-1.2.55.1083.jar
# <https://kedro.readthedocs.io/en/stable/tools_integration/pyspark.html#tips-for-maximising-concurrency-using-threadrunner>
spark.scheduler.mode: FAIRo
Olivia Lihn
02/01/2023, 2:04 PMare you sure the cluster is configured correctly? If this same spark.yml configuration file worked running pipelines from your cli and on jupyter notebook, then it might be because there's a difference between you remote cluster configuration and this file
what i would do is, if you can, check connecting or creating a spark session in your cluster and verify the config above works (as a first step)
a
Anirudh Dahiya
02/01/2023, 2:14 PMAlright let me do that
Thank you - is there a way to check the configuration of the remote cluster?
o
Olivia Lihn
02/01/2023, 2:15 PMtry connecting to the cluster and sending a very simple spark job (something like "hello world")
5 Views