Hello everyone! We'd love to make a small announcement for our

argo-kedro

plugin. The plugin bundles some of the utilities we've been using to run large Kedro pipelines on Argo. This is still very early stage, and we're happy to hear any feedback you may have! Core functionalities are: • Workflow construction:

argo-kedro

constructs an Argo Workflow manifest from your Kedro pipeline for execution on your cluster. This ensures that the Kedro pipeline definition remains the single source of truth. • Defining compute resources:

argo-kedro

exposes a custom

Node

type that can be used to control the compute resources available to the node. • Node fusing: To maximize parallelisation,

argo-kedro

executes each Kedro node in a dedicated Argo task. The plugin exposes a

FusedPipeline

object that can be used to co-locate nodes for execution on a single Argo task. https://pypi.org/project/argo-kedro/ https://github.com/everycure-org/kedro-argo

love this!

@Nok Lam Chan very happy to hear any feedback, first time on this 🙂