Hi everyone, I am trying to run Kedro with Ray, co...
# questions
h
Hi everyone, I am trying to run Kedro with Ray, considering the potential of ray distributed and Kedro pipelines an integration would help a lot. Has someone tried something similar, Need some guidance on how to proceed!
d
Yeah, people have. At a high level, the deployment pattern can be pretty similar to how it is for Dask (which I know you've experimented with). https://github.com/kedro-org/kedro/pull/995/files#diff-dbf04ed0f48883285bfb145a14ce9d7a1d33f6808ba3a375d239f4ba8949a742 is a publicly-available project with Ray execution, but it also includes significant changes to the Kedro framework code (since that was the purpose of the hack project). There are people who have proven it out within QuantumBlack, too, but I can't find them on this channel.
j
for future reference, I see you also chimed in https://github.com/kedro-org/kedro/issues/479 👍🏼