Hi all, Has anyone used Kedro for building pipeli...
# questions
Hi all, Has anyone used Kedro for building pipelines in a repo which also houses non-pipeline code, like lambda functions? I am bringing in Kedro, but also need some way of porting over our existing lambda functions to live in the same repo as our pipelines. Splitting them out into a separate repo is not really feasible due to the common code used by each and the extra work/dependency management that would introduce. We use aws sam as a framework of sorts within each of our lambda functions, so could just put them in
next to
and have some 3rd directory with shared code
, but thought there may be a different way to go about this! Thanks!
Hi @Baden Ashford, interesting question! Is this more about the organisation of code or you need to use Kedro with Lambda?
Hi @Nok Lam Chan, probably the former. But can Kedro be used with Lambda?
It can be run with lambda, but we would recommend Step Function over Lambda if serverless is a requirement. See https://docs.kedro.org/en/stable/deployment/aws_step_functions.html
👍 1
Can you elaborate more what’s the challenge here? In general, if the code need to be separate out, then they deserve to be a standalone module. Your current approach looks fine to me. Having 3 separate directory • A kedro project (with pipelines) • A standalone common utilies • A Lambda specific directory
I will move it out from the Kedro project if none of the Lambda/common need to access the parameters etc
The challenge is just organising the code so we are not working against the kedro framework whilst being able to have the repo house both pipelines and lambdas. From the sounds of it there are no obvious footguns and it will just take some thought and discipline from us to keep things in order! Thanks for your help @Nok Lam Chan
Awesome, I think you have a good approach already : )
🙌 1
also please play back your learnings, it would be great to document this for future kedroid s
👌 1