https://kedro.org/ logo
#questions
Title
# questions
n

Nanshan Li

11/28/2023, 4:23 PM
Hi! Is there a way to access the SparkSession (
spark
) from within kedro nodes when running pipelines on Azure Databricks? Our client wants to use some shared utility functions they have which passes in the spark session to do custom conversions but it doesn’t seem to be accessible from via the node
Copy code
Node < .... > failed with error: name 'spark' is not defined
d

datajoely

11/28/2023, 8:36 PM
You need to do getOrCreateSession to retrieve the singleton but it should work in a node
👍 1
n

Nanshan Li

11/28/2023, 9:19 PM
thank you! that worked 🙂
2 Views