https://kedro.org/ logo
#questions
Title
# questions
t

tom kurian

09/20/2023, 4:35 PM
Is there any docs that explains how kedro can be integrated with ray framework, I have a DS pipeline that creates a KG and I would like to do a POC on ray framework.
j

Juan Luis

09/20/2023, 4:37 PM
hi Tom, unfortunately we don't have that yet - apparently some people have done but they didn't share the code https://github.com/kedro-org/kedro/issues/479 I recently reopened this issue so we can tackle it soon
t

tom kurian

09/20/2023, 4:38 PM
aah, let me try it myself and of it is a successful one, I will share for sure 🙂
d

Deepyaman Datta

09/20/2023, 4:48 PM
It can be done very similarly to Dask deployment. This will have a lot more changes than necessary, but you can get inspiration from
ray_executor.py
(combined with the Dask deployment guide).
n

Nok Lam Chan

09/20/2023, 6:14 PM
If I remember correctly, using Ray’s function decorator you can just add it to a Kedro node/function.
d

Deepyaman Datta

09/20/2023, 6:17 PM
This will have a lot more changes than necessary, but you can get inspiration from
ray_executor.py
Sorry, just noticed I forgot to include a link. 😂 https://github.com/kedro-org/kedro/pull/995/files#diff-dbf04ed0f48883285bfb145a14ce9d7a1d33f6808ba3a375d239f4ba8949a742
10 Views