Yetunde
09/14/2022, 8:43 AMAshish Verma
09/14/2022, 9:05 AMmetadata = bootstrap_project(Path.cwd())
with KedroSession.create(metadata.package_name) as session:
session.run()
Yetunde
09/14/2022, 9:34 AMTynan
09/14/2022, 9:43 AMMerel
09/14/2022, 9:44 AMAshish Verma
09/14/2022, 9:53 AMTynan
09/14/2022, 10:00 AMRogier Vlijm
09/15/2022, 7:58 AMAntony Milne
09/15/2022, 9:11 AMKedroContext
? Does is instantiate spark or just use the in-built spark
object in databricks?Rogier Vlijm
09/15/2022, 9:12 AMAntony Milne
09/15/2022, 9:19 AMspark
instance:
def _get_databricks_object(name: str):
"""Gets object called `name` from the user namespace."""
return IPython.get_ipython().user_ns.get(name) # pragma: no cover
_get_databricks_object("spark")
spark
object rather than calling getOrCreate
Rogier Vlijm
09/15/2022, 9:23 AMObject 'SparkDataSet' cannot be loaded from 'kedro.extras.datasets.spark'. Please see the documentation on how to install relevant dependencies for kedro.extras.datasets.spark.SparkDataSet
pip install kedro[spark.SparkDataSet]
:pip install hdfs
now)