Hi I have a weird error I have a databricks workflows which

Question

Hi I have a weird error I have a databricks workflows which clone the repo and uses a notebook So first step 1 1 identify where the code was cloned

Erwin · Answer

2 intall reqs

Erwin · Answer

3 load catalog to show some data

Erwin · Answer

but fails ```MissingConfigException No files of YAML or JSON format found in Workspace Repos internal 02476ba86f commits 880770e941404812856252f77bc24948806a60c2 conf base or Workspace Repos internal 02476ba86f commits 880770e941404812856252f77bc24948806a60c2 conf databricks dev matching the glob pattern s spark ```

Erwin · Answer

which is weird cause the spark file exists

Erwin · Answer

this way of executing worked untill kedro 0 18 12

Ankita Katiyar · Answer

Try changing the `config pattern` to ` spark `

Ankita Katiyar · Answer

I think it might be looking for the spark config in a folder starting with `spark`

Erwin · Answer

I chaged it but still same error ```File Workspace Repos internal 02476ba86f commits 8707aed67bbe02e5476a3b8cea916cb72591f1b4 src ml minsur de lingo hooks py 26 in SparkHooks after context created self context 21 Initialises a SparkSession using the config 22 defined in project s conf folder 23 25 Load the spark configuration in spark yaml using the config loader gt 26 parameters = context config loader get spark 28 spark conf = SparkConf setAll parameters items 30 Initialise the spark session File usr lib python3 10 collections abc py 819 in Mapping get self key default 817 D get k d gt D k if k in D else d d defaults to None 818 try gt 819 return self key 820 except KeyError 821 return default File local disk0 ephemeral nfs envs pythonEnv 56c57ddd cda0 4310 8d72 02a8d8b856d7 lib python3 10 site packages kedro config omegaconf config py 208 in OmegaConfigLoader getitem self key 205 config update env config 207 if not processed files and key = globals gt 208 raise MissingConfigException 209 f No files of YAML or JSON format found in base path or env path matching 210 f the glob pattern s self config patterns key 211 212 return config MissingConfigException No files of YAML or JSON format found in Workspace Repos internal 02476ba86f commits 8707aed67bbe02e5476a3b8cea916cb72591f1b4 conf base or Workspace Repos internal 02476ba86f commits 8707aed67bbe02e5476a3b8cea916cb72591f1b4 conf databricks dev matching the glob pattern s spark ```

Erwin · Answer

inside `conf databricks dev` there is no spark yml but the one from base is supposed to be loaded

Ankita Katiyar · Answer

Ooh try doing `parameters = config loader spark ` in your `SparkHook`

Erwin · Answer

did not work this is really weird in my local computer works fine even using the flag ` env databricks dev` I will try to debug more

Erwin · Answer

< Ankita Katiyar> I was able to reproduce the error 1 with this path ` Users Erwin Paillacan Projects Workspace Repos internal 02476ba86f commits e3754f960754e93d5dadcd17a74901cf7fb65fda some project conf base` fails 2 with this one works fine ` Users Erwin Paillacan Projects Workspace Repos internal 02476ba86f commits e3754f960754e93d5dadcd17a74901cf7fb65fda some project conf base` the only difference in both path is `internal` and ` internal` when a dot is in the path the config loader gets confused and fails ```No files of YAML or JSON format found in Workspace Repos internal 02476ba86f commits 20adcb96c8bec99ac2fad8b78b25158e7d968fa4 conf base or Workspace Repos internal 02476ba86f commits 20adcb96c8bec99ac2fad8b78b25158e7d968fa4 conf databricks dev matching the glob pattern s catalog catalog catalog ```

Erwin · Answer

in kedro 0 18 12 and OmegaConfigLoader the dot was not a problem

Ankita Katiyar · Answer

Ahh I see this was a bug we fixed recently <https github com kedro org kedro pull 2977>

Ankita Katiyar · Answer

But I suppose it shouldn t be excluding hidden folders outside of `conf ` Thanks for opening an issue for it

Erwin · Answer