04/01/2023, 9:02 PM
Hi everyone, I am trying to run Kedro with Ray, considering the potential of ray distributed and Kedro pipelines an integration would help a lot. Has someone tried something similar, Need some guidance on how to proceed!
04/02/2023, 5:22 AM
Yeah, people have. At a high level, the deployment pattern can be pretty similar to how it is for Dask (which I know you've experimented with).
is a publicly-available project with Ray execution, but it also includes significant changes to the Kedro framework code (since that was the purpose of the hack project). There are people who have proven it out within QuantumBlack, too, but I can't find them on this channel.
04/02/2023, 6:19 PM
for future reference, I see you also chimed in