hi < Jens Peder Meldgaard> + < Juan Luis> i m trying to foll Kedro #questions

hi <@U0769K3GSD9> + <@U046V0QC124>! i'm trying to ...

Juan Luis

02/14/2025, 6:05 PM

hi @Jens Peder Meldgaard + @Juan Luis! i'm trying to follow along but i'm running into an issue. when i run

kedro databricks init

i see that

conf/base/databricks.yml

is created but

iris/databricks.yml

(the asset bundle definition) is not created. so then when i call

kedro databricks deploy

i get this error

Copy code

FileNotFoundError: 
No `databricks.yml` file found. Maybe you forgot to initialize the Databricks bundle?

You can initialize the Databricks bundle by running:

kedro databricks init `````` i can confirm i'm connected via the databricks CLI. any thoughts here? Thread in Slack conversation

Juan Luis

02/14/2025, 6:06 PM

@Nik Linnane could you show the output of the

kedro databricks init

command? cc @Jens Peder Meldgaard

Nik Linnane

02/14/2025, 6:08 PM

Copy code

kedro databricks init
2025-02-14 13:06:43,963 - kedro.framework.project - INFO - Using 'conf/logging.yml' as logging configuration. You can change this by setting the KEDRO_LOGGING_CONFIG environment variable accordingly.

Please select your cloud provider:
1. Azure
2. AWS
3. GCP
 [1]: 2
2025-02-14 13:06:56,964 - Command - ERROR - Creating databricks configuration: Usage: databricks [OPTIONS] COMMAND [ARGS]...
2025-02-14 13:06:56,964 - Command - ERROR - Creating databricks configuration: Try 'databricks -h' for help.
2025-02-14 13:06:56,964 - Command - WARNING - Creating databricks configuration (['databricks', 'bundle', 'init', '/var/folders/9k/g9p7q1wn1772b8bwrjt7wv3r0000gn/T/tmp3fd7uxxu', '--config-file', '/var/folders/9k/g9p7q1wn1772b8bwrjt7wv3r0000gn/T/tmp4vv7us71', '--output-dir', '/Users/Nik_Linnane/Desktop/databricks_test/iris']): Usage: databricks [OPTIONS] COMMAND [ARGS]...
Try 'databricks -h' for help.
2025-02-14 13:06:56,964 - iris - INFO - Creating databricks configuration: Wrote databricks.yml
2025-02-14 13:06:56,966 - iris - WARNING - Creating bundle override configuration: conf/base/databricks.yml already exists.
2025-02-14 13:06:56,966 - iris - INFO - Substituting DBFS paths: Checking conf/local/catalog.yml
2025-02-14 13:06:56,966 - iris - WARNING - Substituting DBFS paths: conf/local/catalog.yml does not exist.
2025-02-14 13:06:56,966 - iris - INFO - Substituting DBFS paths: Checking conf/base/catalog.yml

Juan Luis

02/14/2025, 6:10 PM

some underlying error with

databricks bundle init

Jens Peder Meldgaard

02/14/2025, 6:16 PM

Well, as it says in the error message there already exist a databricks.yml file. So removing that would probably resolve the issue

Nik Linnane

02/14/2025, 6:17 PM

sorry, forgot i had already ran it. i deleted

conf/base/databricks.yml

and reran

kedro databricks init

Copy code

kedro databricks init
2025-02-14 13:16:32,918 - kedro.framework.project - INFO - Using 'conf/logging.yml' as logging configuration. You can change this by setting the KEDRO_LOGGING_CONFIG environment variable accordingly.

Please select your cloud provider:
1. Azure
2. AWS
3. GCP
 [1]: 2
2025-02-14 13:16:37,020 - Command - ERROR - Creating databricks configuration: Usage: databricks [OPTIONS] COMMAND [ARGS]...
2025-02-14 13:16:37,021 - Command - ERROR - Creating databricks configuration: Try 'databricks -h' for help.
2025-02-14 13:16:37,021 - Command - WARNING - Creating databricks configuration (['databricks', 'bundle', 'init', '/var/folders/9k/g9p7q1wn1772b8bwrjt7wv3r0000gn/T/tmpe47utvp4', '--config-file', '/var/folders/9k/g9p7q1wn1772b8bwrjt7wv3r0000gn/T/tmp94vmamwc', '--output-dir', '/Users/Nik_Linnane/Desktop/databricks_test/iris']): Usage: databricks [OPTIONS] COMMAND [ARGS]...
Try 'databricks -h' for help.
2025-02-14 13:16:37,021 - iris - INFO - Creating databricks configuration: Wrote databricks.yml
2025-02-14 13:16:37,044 - iris - INFO - Creating bundle override configuration: Wrote conf/base/databricks.yml
2025-02-14 13:16:37,044 - iris - INFO - Substituting DBFS paths: Checking conf/local/catalog.yml
2025-02-14 13:16:37,045 - iris - WARNING - Substituting DBFS paths: conf/local/catalog.yml does not exist.
2025-02-14 13:16:37,045 - iris - INFO - Substituting DBFS paths: Checking conf/base/catalog.yml

Nik Linnane

02/14/2025, 6:18 PM

still no

iris/databricks.yml

Jens Peder Meldgaard

02/14/2025, 6:24 PM

Sooo… I don’t quite know what went wrong. But of the top of my head. Did you authenticate with the databricks client beforehand? ‘databricks auth login’

Nik Linnane

02/14/2025, 6:25 PM

i authenticated with a personal access token using

databricks configure --token

- would that make a difference?

Jens Peder Meldgaard

02/14/2025, 7:22 PM

Not really. Did you use the default profile?

Nik Linnane

02/14/2025, 8:04 PM

yes

Jens Peder Meldgaard

02/15/2025, 7:49 AM

@Nik Linnane Hi again, Could I get you to share which version of the Databricks CLI that you are using? I can't quite seem to reproduce the error that you are experiencing.. 😞

Jens Peder Meldgaard

02/15/2025, 8:05 AM

Scratch that... I found out what happened. So it seems that you are using the python package

databricks-cli

. Unfortunately, the

kedro-databricks

plugin is heavily reliant on the

databricks bundle

command group, which was introduced in their new CLI implementation. You can find more information on how to update the Databricks CLI here: https://docs.databricks.com/en/dev-tools/cli/install.html

Jens Peder Meldgaard

02/15/2025, 8:06 AM

I hope that resolves your issues. If not, please let me know 🙂

Nik Linnane

02/18/2025, 3:54 PM

ahh okay i will try that! appreciate you looking into this 🙂

Nik Linnane

02/20/2025, 3:41 PM

hi @Jens Peder Meldgaard! so that worked (now using Databricks CLI v0.241.2), but im now encountering an issue where after running

kedro databricks deploy

a job is never created. i made a new issue here

Jens Peder Meldgaard

02/20/2025, 3:46 PM

Hey! Happy to hear that. I’m currently working on version 0.8.0, here the deployment issue should have been resolved.

thankyou 1

2 Views

Open in Slack

Previous Next