Flavien05/03/2023, 10:26 AM
project on Databricks (and have good hope to convince my team to go for
). The documentation is very well written, thanks for that. Scrolling through the messages in Slack, I did not find a way to directly use the object
provided directly in the Databricks notebooks. Is there any way to do so?
Juan Luis05/03/2023, 10:31 AM
directly, let me check
Flavien05/03/2023, 10:39 AM
which is getting the session through
. I am not familiar at all with the inner workings of Spark and Databricks. As Databricks provided a dedicated session (properly) setup, I would have liked it to be used when running:
as per the documentation.
with KedroSession.create() as session: session.run(pipeline_name="mon_petit_pipeline")
Juan Luis05/03/2023, 12:12 PM
has a static
method that, as you spotted, gets or creates an existing one. if I understand correctly how Databricks works, this should get the appropriate
, even if it's created by Databricks. • when passing a
around, you can access its
property, which yields the session that created it. I'm mindful these answers are somewhat generic (I promised I generated them with my own brain and not ChatGPT 😬) but, if none of them are quite what you were looking for, would you mind sharing a bit more detail?
Flavien05/03/2023, 12:26 PM
in my team — mentioning
every week for a year now 😅.
Juan Luis05/03/2023, 12:53 PM