Hey team. Any experience deploying kedro pipelines...
# questionsp
Pedro Sousa Silva
04/10/2024, 3:49 PMHey team. Any experience deploying kedro pipelines as tasks in Databricks jobs? Basically having Databricks Workflows manage the dependencies between kedro pipelines
n
Nok Lam Chan
04/10/2024, 3:51 PMd
datajoely
04/10/2024, 3:52 PMWe have a few different Databricks approaches on our docs
datajoely
04/10/2024, 3:52 PMLet us know if you have any questions!
p
Pedro Sousa Silva
04/10/2024, 3:56 PMSo I had read that page already and my understanding is that it applies only in a case where you want to run your entire pipeline
Pedro Sousa Silva
04/10/2024, 3:57 PMI was trying to build the wheel for the project on the go on the first task through something like:
Copy code
import os
import subprocess
import sys
username = 'username'
project_root_str = f'/Workspace/Users/{username}/repo-name/kedro-project-name'
subprocess.check_call(
[
sys.executable,
'-m',
'pip',
'install',
project_root_str
]
)
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path
project_root = Path(project_root_str)
bootstrap_project(project_root)
with KedroSession.create(project_path=project_root, env='prod') as session:
session.run(pipeline_name='p01_setup')n
Nok Lam Chan
04/10/2024, 4:23 PMIt seems a bit weird to have pip install in the script, isn't databricks handle the package in its way?
4 Views