Hello Everyone, Anyone ever tried using prefect on...
# questions
e
Hello Everyone, Anyone ever tried using prefect on their pipelines?
Copy code
Encountered exception during execution:
Traceback (most recent call last):
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\kedro\framework\startup.py", line 147, in _validate_source_path
    source_path.relative_to(project_path)
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 816, in relative_to
    raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: 'C:\\Users\\KodeCraft-3\\AppData\\Local\\Temp\\tmpcc05q7ogprefect\\src' is not in the subpath of 'C:\\Users\\KODECR~1\\AppData\\Local\\Temp\\tmpcc05q7ogprefect' OR one path is relative and the other is absolute.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\prefect\engine.py", line 829, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\prefect\_internal\concurrency\calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\prefect\_internal\concurrency\calls.py", line 315, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\KODECR~1\AppData\Local\Temp\tmpcc05q7ogprefect\register_prefect_flow.py", line 61, in my_flow
    metadata = bootstrap_project(project_path)
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\kedro\framework\startup.py", line 174, in bootstrap_project
    _add_src_to_path(metadata.source_dir, project_path)
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\kedro\framework\startup.py", line 158, in _add_src_to_path
    _validate_source_path(source_dir, project_path)
  File "C:\Users\KodeCraft-3\AppData\Local\Programs\Python\Python310\lib\site-packages\kedro\framework\startup.py", line 149, in _validate_source_path
    raise ValueError(
ValueError: Source path 'C:\Users\KodeCraft-3\AppData\Local\Temp\tmpcc05q7ogprefect\src' has to be relative to your project root 'C:\Users\KODECR~1\AppData\Local\Temp\tmpcc05q7ogprefect'.
07:19:56 PM
prefect.flow_runs
ERROR
Finished in state Failed("Flow run encountered an exception. ValueError: Source path 'C:\\Users\\KodeCraft-3\\AppData\\Local\\Temp\\tmpcc05q7ogprefect\\src' has to be relative to your project root 'C:\\Users\\KODECR~1\\AppData\\Local\\Temp\\tmpcc05q7ogprefect'.")
Im encountering this error, I just followed the documentation
j
hello @Eluard Camota, I think this might be a bug in Kedro related to the use of temporary directories https://github.com/kedro-org/kedro/issues/2346 can you give more details on how did you build your project and where are you running it from?
e
I'm running it on windows system, after i created the kedro project using
kedro new
, I then created a pipeline named historical_cubic_spline_iv using the command
kedro create pipeline
Attached is the file structure of my project. According to the kedro docs on using prefect, I shall create a
register_prefect_flow.py
on my root directory which I did.
j
can you paste here the absolute path to
register_prefect_flow.py
@Eluard Camota? should be something like
C:\\...\\register_prefect_flow.py
e
D:\Repositories\data-playground\register_prefect_flow.py
j
okay, I think Prefect might be generating these temporary directories then. I'm curious, are you acquaintance of @Kyle Niosco? they reported a similar issue a month ago, with the path starting from
C:\Users\KodeCraft-2
(or, everybody is naming their Windows users
KodeCraft
? 😅)
e
Hahaha yup, I think he fixed it already might try to fix it with his solution. Thank you 😁
j
ah that's great to know 😄 how did you fix it?? would love to share it as a workaround!
e
He just replaced the
project_path
instead of
Path.cwd()
to the absolute path of the root directory, it seems like
Path.cwd()
doesn't read the root directory of the project file.
Now we face a new challenge 😄, its how we run both the kedro project and prefect in the same docker image. haha
j
keep us posted 🙂
e
sure, will do. Thanks
Update: I was able to generate the docker image of my kedro project using this docs -> https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker, but since my prefect is still self-hosted you should run the docker based from the command here https://docs.prefect.io/2.13.8/guides/docker/#building-a-docker-image.
j
Perfect 👏🏼