https://kedro.org/ logo
#questions
Title
# questions
e

Eluard Camota

02/08/2024, 10:58 AM
Anyone tried deploying their kedro pipeline in Prefect Cloud. Thanks..
P 1
a

Ahdra Merali

02/08/2024, 1:29 PM
Not personally, but we've got docs on Prefect deployment - have you run into any issues?
e

Eluard Camota

02/09/2024, 5:30 AM
yup, it was working when deploying locally but when I used their cloud platform, the flow cannot be retrieved.
K 1
j

Juan Luis

02/09/2024, 9:41 AM
could you share a screenshot or traceback of how the error looks like?
e

Eluard Camota

02/09/2024, 12:05 PM
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 425, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 245, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=from_path, local_path=".")
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 162, in get_directory
    copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func)
  File "/usr/local/lib/python3.10/shutil.py", line 557, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/E:\\Repositories\\Back-tester\\historical-data-ingestion/E:\\Repositories\\Back-tester\\historical-data-ingestion'
j

Juan Luis

02/09/2024, 12:06 PM
your absolute Windows paths seem to be leaking into the configuration somehow @Eluard Camota 🤔
e

Eluard Camota

02/09/2024, 1:31 PM
Yup, using this function won't work with the prefect cloud.
Copy code
def prefect_deploy(
    pipeline_name, env, deployment_name, work_pool_name, work_queue_name, version
):
    """Register a Kedro pipeline as a Prefect flow."""

    # Pipeline name to execute
    pipeline_name = pipeline_name or "__default__"

    # Use standard deployment configuration for local execution. If you require a different
    # infrastructure, check the API docs for Deployments at: <https://docs.prefect.io/latest/api-ref/prefect/deployments/>
    deployment = Deployment.build_from_flow(
        flow=my_flow,
        name=deployment_name,
        path=str(Path.cwd()),
        version=version,
        parameters={
            "pipeline_name": pipeline_name,
            "env": env,
        },
        infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
        work_pool_name=work_pool_name,
        work_queue_name=work_queue_name,
        # schedule={"cron": "0 0 * * 6"}
    )

    deployment.apply()
I think it's because of the path, I might need to store the flow code into a docker image and use that to deploy to prefect cloud so it can access it. Still figuring it.