Hi Team,
We are using databricks setup for running...
# questionsa
Ankit Kansal
11/27/2023, 2:42 PMHi Team,
We are using databricks setup for running kedro pipelines and using databricks.ManagedTable** dataset using
Workflow
1. We develop the code in notebooks - run it test it.
2. We have a little tweak of the ManagedTable class to accept file path as only external tables are allowed
3. Deploy the code through azure pipelines --
facing challenges hereAnkit Kansal
11/27/2023, 5:54 PM@Juan Cruz
Ankit Kansal
11/27/2023, 5:55 PMI know the issue, we just need to update the databricks.ManagedTable** dataset and make it accept filepath variable.
Now i have done the above but it is working in one workspace while not working in another
In another it is giving import error
Ankit Kansal
11/27/2023, 10:50 PMerror screenshot
j
Juan Luis
11/28/2023, 8:08 AMhi @Ankit Kansal! what happens if you do
Copy code
from claimscostutilities.kedro_datasets.custom_managed_table_dataset import ManagedTableDatasetsa
Ankit Kansal
11/28/2023, 8:32 AMHi @Juan Luis thanks for checking, the above import works without giving error
Ankit Kansal
11/28/2023, 8:33 AMimage.png
j
Juan Luis
11/28/2023, 8:36 AMgot it. how are you loading Kedro in the first case?
%load_ext kedro.ipythona
Ankit Kansal
11/28/2023, 9:54 AMThis is my requirements.txt
Ankit Kansal
11/28/2023, 9:55 AMimage.png
👀 1
Ankit Kansal
11/28/2023, 10:06 AM1. The same code works when you run the databricks git workspace.
2. The code fails with above exception when run in databricks workspace folder.
j
Juan Luis
11/28/2023, 10:22 AMthanks @Ankit Kansal, that's helpful. this tells me that the databricks notebook probably has some missing dependency somehow.
In your https://kedro-org.slack.com/archives/C03RKP2LW64/p1701165302683509?thread_ts=1701096140.614109&cid=C03RKP2LW64 screenshot, when you do
session.run(pipeline_name="...")