Anyone tried deploying their kedro pipeline in Pre...
# questionse
Eluard Camota
02/08/2024, 10:58 AMAnyone tried deploying their kedro pipeline in Prefect Cloud. Thanks..
P 1
a
Ahdra Merali
02/08/2024, 1:29 PMNot personally, but we've got docs on Prefect deployment - have you run into any issues?
e
Eluard Camota
02/09/2024, 5:30 AMyup, it was working when deploying locally but when I used their cloud platform, the flow cannot be retrieved.
K 1
j
Juan Luis
02/09/2024, 9:41 AMcould you share a screenshot or traceback of how the error looks like?
e
Eluard Camota
02/09/2024, 12:05 PM Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 425, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 245, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 162, in get_directory
copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func)
File "/usr/local/lib/python3.10/shutil.py", line 557, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/E:\\Repositories\\Back-tester\\historical-data-ingestion/E:\\Repositories\\Back-tester\\historical-data-ingestion'j
Juan Luis
02/09/2024, 12:06 PMyour absolute Windows paths seem to be leaking into the configuration somehow @Eluard Camota 🤔
e
Eluard Camota
02/09/2024, 1:31 PMYup, using this function won't work with the prefect cloud.
Copy code
def prefect_deploy(
pipeline_name, env, deployment_name, work_pool_name, work_queue_name, version
):
"""Register a Kedro pipeline as a Prefect flow."""
# Pipeline name to execute
pipeline_name = pipeline_name or "__default__"
# Use standard deployment configuration for local execution. If you require a different
# infrastructure, check the API docs for Deployments at: <https://docs.prefect.io/latest/api-ref/prefect/deployments/>
deployment = Deployment.build_from_flow(
flow=my_flow,
name=deployment_name,
path=str(Path.cwd()),
version=version,
parameters={
"pipeline_name": pipeline_name,
"env": env,
},
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
work_pool_name=work_pool_name,
work_queue_name=work_queue_name,
# schedule={"cron": "0 0 * * 6"}
)
deployment.apply()