Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Hi friends. We're wondering if we should upgrade our kedro -&gt; argo manual jinja templating kung-fu by leveraging <https://hera-workflows.readthedocs.io/>
Has anyone thought about this as well? It could be a nice way to give kedro a stable go-to codebase to deploy kedro to k8s clusters

it wouldn't solve the container build/push/registry bits, that sadly is still separate AFAIK but it does feel like a very clean way to map kedro nodes to DAG steps in argo and submit the workflows

Do you have an idea on how to implement this? Been thinking about this too for a while, but I don’t want to map kedro nodes 1:1 to tasks in a DAG as that would be wasteful (pod startup overhead)

<@U04HQRFPM0C> <https://github.com/kedro-org/kedro-plugins/blob/main/kedro-airflow/RELEASE.md#release-0100>

I am not sure if this is implemented in a more generic sense, but in `kedro-airflow` there is now a feature to group pipeline in a higher level (namespace). It will also try to handle the issue with MemoryDataset.

If this is something useful, feel free to raise an issue to ask a feature request/discussion