Hi
I have a weird error.
I have a databricks work...
# questionse
Erwin
09/01/2023, 4:37 PMHi
I have a weird error.
I have a databricks workflows, which clone the repo and uses a notebook.
So first step 1:
1. identify where the code was cloned
Erwin
09/01/2023, 4:37 PM2. intall reqs
Erwin
09/01/2023, 4:38 PM3. load catalog to show some data
Erwin
09/01/2023, 4:38 PMbut fails:
Copy code
MissingConfigException: No files of YAML or JSON format found in /Workspace/Repos/.internal/02476ba86f_commits/880770e941404812856252f77bc24948806a60c2/conf/base or /Workspace/Repos/.internal/02476ba86f_commits/880770e941404812856252f77bc24948806a60c2/conf/databricks_dev matching the glob pattern(s): ['spark*/']Erwin
09/01/2023, 4:39 PMwhich is weird cause the spark file exists:
Erwin
09/01/2023, 4:39 PMthis way of executing worked untill kedro 0.18.12.
a
Ankita Katiyar
09/01/2023, 4:43 PMTry changing the
config_pattern['spark*']Ankita Katiyar
09/01/2023, 4:44 PMI think it might be looking for the spark config in a folder starting with
sparke
Erwin
09/01/2023, 4:58 PMI chaged it
but still same error:
Copy code
File /Workspace/Repos/.internal/02476ba86f_commits/8707aed67bbe02e5476a3b8cea916cb72591f1b4/src/ml_minsur_de_lingo/hooks.py:26, in SparkHooks.after_context_created(self, context)
21 """Initialises a SparkSession using the config
22 defined in project's conf folder.
23 """
25 # Load the spark configuration in spark.yaml using the config loader
---> 26 parameters = context.config_loader.get("spark")
28 spark_conf = SparkConf().setAll(parameters.items())
30 # Initialise the spark session
File /usr/lib/python3.10/_collections_abc.py:819, in Mapping.get(self, key, default)
817 'D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.'
818 try:
--> 819 return self[key]
820 except KeyError:
821 return default
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-56c57ddd-cda0-4310-8d72-02a8d8b856d7/lib/python3.10/site-packages/kedro/config/omegaconf_config.py:208, in OmegaConfigLoader.__getitem__(self, key)
205 config.update(env_config)
207 if not processed_files and key != "globals":
--> 208 raise MissingConfigException(
209 f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
210 f" the glob pattern(s): {[*self.config_patterns[key]]}"
211 )
212 return config
MissingConfigException: No files of YAML or JSON format found in /Workspace/Repos/.internal/02476ba86f_commits/8707aed67bbe02e5476a3b8cea916cb72591f1b4/conf/base or /Workspace/Repos/.internal/02476ba86f_commits/8707aed67bbe02e5476a3b8cea916cb72591f1b4/conf/databricks_dev matching the glob pattern(s): ['spark*']Erwin
09/01/2023, 4:59 PMinside
conf/databricks_deva
Ankita Katiyar
09/01/2023, 5:04 PMOoh try doing
parameters = config_loader["spark"]SparkHook👍 1
e
Erwin
09/01/2023, 11:25 PMdid not work. this is really weird, in my local computer works fine even using the flag
--env databricks_devErwin
09/03/2023, 6:16 PM@Ankita Katiyar I was able to reproduce the error
1. with this path
/Users/Erwin_Paillacan/Projects/Workspace/Repos/.internal/02476ba86f_commits/e3754f960754e93d5dadcd17a74901cf7fb65fda/some_project/conf/base/Users/Erwin_Paillacan/Projects/Workspace/Repos/internal/02476ba86f_commits/e3754f960754e93d5dadcd17a74901cf7fb65fda/some_project/conf/baseinternal.internal Copy code
No files of YAML or JSON format found in /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/base or /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/databricks_dev matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']👍 1
Erwin
09/03/2023, 6:20 PMin kedro 0.18.12 and OmegaConfigLoader the dot was not a problem
a
Ankita Katiyar
09/03/2023, 6:31 PMAhh, I see, this was a bug we fixed recently - https://github.com/kedro-org/kedro/pull/2977
Ankita Katiyar
09/03/2023, 6:34 PMBut I suppose it shouldn’t be excluding hidden folders outside of
conf/👍 1
e
Erwin
09/03/2023, 6:35 PM