Hi! Is there a way to access the SparkSession (

spark

) from within kedro nodes when running pipelines on Azure Databricks? Our client wants to use some shared utility functions they have which passes in the spark session to do custom conversions but it doesn’t seem to be accessible from via the node

Node < .... > failed with error: name 'spark' is not defined

You need to do getOrCreateSession to retrieve the singleton but it should work in a node

thank you! that worked 🙂