Context: I am trying to run a job in Databricks based off the .whl file packaged by Kedro: Kedro Version: 0.18.3 Error:

ValueError: Given configuration path either does not exist or is not a valid directory: /databricks/driver/conf/base

Q1: Is the issue on due to the .whl file itself or the way I am configuring the job ? Q2: Would this be due to a permissions issue on the environment I am using ?

I think configuration isn't packaged using

kedro package

, so you would need to add the config for your job in the expected place. You can also modify the place where it looks for configuration by changing

CONF_SOURCE

in

settings.py

.

@Deepyaman Datta - so this would involve uncommenting

CONF_SOURCE

in

setting.py

and having it be like the following ?

CONF_SOURCE = "conf/base"

@shawn that depends if u have access to put the configuration to that particular location

the following lines of code work resolved the issue:

include_package_data = True,

packages = ['.'].