Hi Kedro Team! Hope you are doing well! š
We are looking at best practices on how to deploy Kedro pipelines on Cloud (AWS or Azure). āļø
We are currently thinking of creating a deployment procedure to deploy Kedro pipelines on Amazon Lambda functions for the following benefits:
ā¢
Collaboration: Kedro pipelines can process intermediary data hosted on a shared object store (e.g., S3).
ā¢
Compute: Computational heavy Kedro pipelines can be run in the Cloud to alleviate local compute resources.
ā¢
Access private resources: Kedro pipelines can access privately deployed APIs in the same AWS account.
Ask:
⢠Are the above benefits the usual benefits to run Kedro in the Cloud? Any benefit to be challenged or added that were not mentioned?
⢠Are there any alternative Cloud resources we could look into to achieve similar benefits? E.g., running Kedro on
AWS Step Functions as mentioned in your doc OR running Kedro on EC2?
We would love to hear your thoughts to confirm some of our discussions within our team! Please feel free to ping us for a call if you prefer to communicate synchronously. š
@Cyril Verluise @Roberto P. Palomares