New blog post How to build a custom Kedro runner kedro In th Kedro #announcements

New blog post: How to build a custom Kedro runner ...

Juan Luis

06/22/2023, 9:38 AM

New blog post: How to build a custom Kedro runner K In this post our colleague @Nok Lam Chan explains how to write an efficient custom Kedro pipeline runner that continues executing a set of nodes after encountering non-catastrophic failure points. Enjoy! https://kedro.org/blog/build-a-custom-kedro-runner (Source code at https://github.com/kedro-org/kedro/blob/feat/softfail-runner/kedro/runner/softfail_runner.py)

kedroid 11

📣 4

👍 1

K 7

❤️ 2

Jose Nuñez

06/22/2023, 1:44 PM

Great post @Nok Lam Chan keep on going!! 🦜K QQ: Is this SoftFailRunner going to be merged into the main branch eventually or was this just a show case on how to create a new runner?

thankyou 1

Juan Luis

06/22/2023, 1:47 PM

@Jose Nuñez we're discussing just that 😄 https://github.com/kedro-org/kedro/issues/503#issuecomment-1602354520

Nok Lam Chan

06/22/2023, 1:54 PM

Stay tune! We are going to open a ticket shortly, please upvote it if you think this is useful and we could plan accordingly. Currently we didn’t plan to merge it into

main

in the coming weeks as we have some other priorities. Short term plan is have it as example and links with doc, and a follow-up ticket to merge that into

main

Nok Lam Chan

06/22/2023, 1:56 PM

It works pretty fine if you are mostly using

kedro run

only, there are some caveats if you need to consume the output. i.e.

Copy code

result = session.run()
do_something_about(result) # This isn't available in the current implementation.

We will need to fix this before we merge it into the collections of runners.

Nok Lam Chan

06/22/2023, 3:22 PM

The issue is here now ! Add documentation for the soft-fail runner if/when it is released in a supported version.

22 Views

Open in Slack

Previous Next