Anyone here ever use kedro to run glue jobs? I know someone was talking about a glue runner plugin at one point.

It never materialised if it did exist!

@Nishant Kumar Might be able to help advise on how he's done it.

Cool. Yeah I'm just curious if anyone's experimented with it beyond initial exploration

Hello Andrew, yes, we did run Glue jobs with Kedro. Ib very short, we did: •

kedro package

to create a wheel file for the pipeline and push on S3. Docs here • use

%additional_python_modules

magic to get wheel from S3 and installed. • FInally, just import your project as mentioned in the kedro documentation.

from kedro_spaceflights.__main__ import main

main(
    ["--pipeline", "__default__"]
)  # or simply main() if you don't want to provide any arguments

that did it for us. 🙂

Thabks @Nishant Kumar . How "manual" is that deployment process for you? Could you see automating it fairly easily via CI/CD?

yes absolutely, these 2 steps i.e.: 1. kedro package 2. moving the wheel to S3 are completely automated.

@Nishant Kumar do you buy any chance have any examples of your glue script?

I'll create an example and share the URL of the git issue by Monday

Also quick question: was this for glue python job or glue spark job? Basically we're trying to run something like the kedro pyspark example in glue.

yes, it was a glue python job.