https://kedro.org/ logo
#questions
Title
# questions
a

Andrew Stewart

11/11/2022, 4:47 PM
Anyone here ever use kedro to run glue jobs? I know someone was talking about a glue runner plugin at one point.
d

datajoely

11/11/2022, 4:48 PM
It never materialised if it did exist!
y

Yetunde

11/11/2022, 4:52 PM
@Nishant Kumar Might be able to help advise on how he's done it.
a

Andrew Stewart

11/11/2022, 4:53 PM
Cool. Yeah I'm just curious if anyone's experimented with it beyond initial exploration
n

Nishant Kumar

11/14/2022, 11:10 AM
Hello Andrew, yes, we did run Glue jobs with Kedro. Ib very short, we did: •
kedro package
to create a wheel file for the pipeline and push on S3. Docs here • use
%additional_python_modules
magic to get wheel from S3 and installed. • FInally, just import your project as mentioned in the kedro documentation.
Copy code
from kedro_spaceflights.__main__ import main

main(
    ["--pipeline", "__default__"]
)  # or simply main() if you don't want to provide any arguments
that did it for us. šŸ™‚
šŸ‘ 1
I need to update the git issue on kedro. This is a good nudge. I'll do this week šŸ™‚
Let me know Andrew, if you will have any questions
a

Andrew Stewart

11/14/2022, 2:50 PM
Thabks @Nishant Kumar . How "manual" is that deployment process for you? Could you see automating it fairly easily via CI/CD?
n

Nishant Kumar

11/14/2022, 6:03 PM
yes absolutely, these 2 steps i.e.: 1. kedro package 2. moving the wheel to S3 are completely automated.
šŸ‘ 1
a

Andrew Stewart

11/18/2022, 4:30 AM
@Nishant Kumar do you buy any chance have any examples of your glue script?
n

Nishant Kumar

11/18/2022, 9:44 AM
I'll create an example and share the URL of the git issue by Monday
šŸŽ‰ 1
K 1
ā¤ļø 1
a

Andrew Stewart

11/18/2022, 7:50 PM
Also quick question: was this for glue python job or glue spark job? Basically we're trying to run something like the kedro pyspark example in glue.
n

Nishant Kumar

11/24/2022, 1:55 PM
yes, it was a glue python job.
3 Views