Anirudh Dahiya
02/01/2023, 1:14 PMException: Java gateway process exited before sending its port number
Has anyone faced this error before?datajoely
02/01/2023, 1:43 PMAnirudh Dahiya
02/01/2023, 1:45 PMdatajoely
02/01/2023, 1:46 PMAnirudh Dahiya
02/01/2023, 1:46 PMdatajoely
02/01/2023, 1:46 PMspark.yaml
in Kedro or environment varialbes?Anirudh Dahiya
02/01/2023, 1:47 PMdatajoely
02/01/2023, 1:48 PMSparkSession.builder.appName('myapp').getOrCreate
Anirudh Dahiya
02/01/2023, 1:51 PMdatajoely
02/01/2023, 1:51 PMAnirudh Dahiya
02/01/2023, 1:51 PMdatajoely
02/01/2023, 1:52 PMAnirudh Dahiya
02/01/2023, 1:52 PMOlivia Lihn
02/01/2023, 2:02 PMAnirudh Dahiya
02/01/2023, 2:03 PM# You can define spark specific configuration here.
spark.sql.execution.arrow.pyspark.enabled: true
spark.ui.port: 4050
spark.driver.bindAddress: 127.0.0.1
spark.driver.memory: 180g
spark.driver.maxResultSize: 70g
spark.driver.memoryOverhead: 40g
spark.network.timeout: 1000s
spark.hadoop.fs.s3a.connection.maximum: 1000
spark.debug.maxToStringFields: 10000
spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.broadcastTimeout: 600
spark.executor.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.driver.extraJavaOptions: -Dcom.amazonaws.services.s3.enableV4=true
spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
spark.local.dir: /data1/temp
spark.sql.autoBroadcastJoinThreshold: 50000000
spark.speculation: true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: 2
spark.jars.packages: org.apache.hadoop:hadoop-aws:2.9.2,com.databricks:spark-redshift_2.11:2.0.1,org.apache.avro:avro:1.8.1,org.apache.spark:spark-avro_2.11:2.4.4
spark.jars: /packages/RedshiftJDBC42-no-awssdk-1.2.55.1083.jar
# <https://kedro.readthedocs.io/en/stable/tools_integration/pyspark.html#tips-for-maximising-concurrency-using-threadrunner>
spark.scheduler.mode: FAIR
Olivia Lihn
02/01/2023, 2:04 PMAnirudh Dahiya
02/01/2023, 2:14 PMOlivia Lihn
02/01/2023, 2:15 PM