Hi Kedro Team! Hope you are doing well! 🎉 We are looking at best practices on how to deploy Kedro pipelines on Cloud (AWS or Azure). ☁️ We are currently thinking of creating a deployment procedure to deploy Kedro pipelines on Amazon Lambda functions for the following benefits: • Collaboration: Kedro pipelines can process intermediary data hosted on a shared object store (e.g., S3). • Compute: Computational heavy Kedro pipelines can be run in the Cloud to alleviate local compute resources. • Access private resources: Kedro pipelines can access privately deployed APIs in the same AWS account. Ask: • Are the above benefits the usual benefits to run Kedro in the Cloud? Any benefit to be challenged or added that were not mentioned? • Are there any alternative Cloud resources we could look into to achieve similar benefits? E.g., running Kedro on AWS Step Functions as mentioned in your doc OR running Kedro on EC2? We would love to hear your thoughts to confirm some of our discussions within our team! Please feel free to ping us for a call if you prefer to communicate synchronously. 🙂 @Cyril Verluise @Roberto P. Palomares

Thanks @Ze Wen Wu ! In general, we are curious to know what is usually recommended to deploy kedro pipelines on the cloud. Cheers all!

I was trying to remember a past discussion on this topic, but it was https://kedro-org.slack.com/archives/C03RKP2LW64/p1689789306348409, so I'm sure you all already are aware of it... As (somewhat) mentioned previously, Kedro is not opinionated as to your deployment target, and one of the benefits of Kedro is that you can map to various deployment targets. That said, if I were to suggest something, it would be to deploy to Sagemaker Pipelines on AWS, Azure ML Pipelines on Azure, Vertex AI Pipelines on Google--but that's just an opinion and may differ from what others say. ;)

Thanks!