https://kedro.org/ logo
#questions
Title
# questions
m

Maurits

12/07/2022, 5:29 PM
Hi all, I'm facing a
java.lang.OutOfMemoryError: Java heap space
error storing a JSON-file of 2.5M rows on AWS S3 via a Kedro pipeline. ECS Compute has 104 GB memory already. Any recommendation how to configure this? Repartition experience? Spark config? Or work around it?
d

datajoely

12/07/2022, 6:09 PM
This is a spark configuration error not actually a Kedro one. So yeah edit
spark.yml
to do in Kedro or provide env vars to do independently
3 Views