hey team, question regarding kedro and spark. is there a way to enable spark log4j to output into the log file / terminal output?
I have this line in my spark hook:
_spark_session.sparkContext.setLogLevel("WARN")
doing some reading, it seems like I might need a log4j.properties file and specify the file in spark.yml.
is this the right path forward?