Is there any docs that explains how kedro can be integrated with ray framework, I have a DS pipeline that creates a KG and I would like to do a POC on ray framework.

hi Tom, unfortunately we don't have that yet - apparently some people have done but they didn't share the code https://github.com/kedro-org/kedro/issues/479 I recently reopened this issue so we can tackle it soon

aah, let me try it myself and of it is a successful one, I will share for sure 🙂

It can be done very similarly to Dask deployment. This will have a lot more changes than necessary, but you can get inspiration from

ray_executor.py

(combined with the Dask deployment guide).

If I remember correctly, using Ray’s function decorator you can just add it to a Kedro node/function.

This will have a lot more changes than necessary, but you can get inspiration from

ray_executor.py

Sorry, just noticed I forgot to include a link. 😂 https://github.com/kedro-org/kedro/pull/995/files#diff-dbf04ed0f48883285bfb145a14ce9d7a1d33f6808ba3a375d239f4ba8949a742