Would kedro users be opposed defining nodes with decorators? I have written a simple implementation but as I've only recently started using kedro I wonder if I'm missing anything: The syntax would be:

from kedro.pipeline import Pipeline, node, pipeline

@node(inputs=1, outputs="first_sum")
def step1(number):
   return number + 1

@node(inputs="first_sum", outputs="second_sum")
def step2(number):
   return number + 1 

@node(inputs="second_sum", outputs="final_result")
def step3(number):
   return number + 2

pipeline = pipeline(
   [
       step1,
       step2,
       step3,
   ]
)

the node name could be inferred from the function name

Someone will reply to you shortly. In the meantime, this might help:

Related: https://github.com/kedro-org/kedro/issues/2471

the functions could even be decorated in the project nodes.py and then pipeline definition would just become:

from kedro.pipeline import Pipeline, node, pipeline
from .nodes import step1,step2,step3

pipeline = pipeline(
   [
       step1,
       step2,
       step3,
   ]
)

Personally I don't strongly against it, but to me it's mostly a syntax sugar. It will make simple things simpler but the complex cases more difficult. That's the main tradeoff

It's pure syntactic sugar yes

What complex cases would this not cover though?

You get into funny situations where different decorators would conflict, for example combining this with Pandera might be painful We also try to have only one way of doing things, whilst this rule is broken in some places, it can cause headaches

say you want to reuse the function -> now you can't because it's a node with pipeline specific details

☝️

I don't think it's an issue with reusability of functions, I replied to this concern in the github thread: https://github.com/kedro-org/kedro/issues/2471#issuecomment-2598338855

yeah which is why I think it's in my opinion not worth the trouble

I agree with this:

It will make simple things simpler but the complex cases more difficult. That's the main tradeoff

I personally like to move all of my functions outside of my kedro project into their own well tests python package which is available on a private PyPI like repo. This is also means my flow logic isn't coupled to my business logic and say I needed to swap Kedro for something else it wont be a pain

Fair enough! Just wanted to see what the community thought of it, thanks for the insights @Nok Lam Chan @datajoely

so in summary I think our current pattern enables high cohesion but low coupling between Kedro's framework and your business logic

The approach you have taken it's slightly different, so you keep the function but have a separate thin node wrapper for what you call "step"

@node(inputs=["a", "b"], outputs="sum")
def pipeline_step(a, b): return reusable_fn(a, b)

Is this simpler than

node(reuable_fn, "a", "b", outputs="sum"

? It's a few more keystroke, though maybe slightly clearer since the arguments are highlight at the top

Curious, where do you store your private pypi repos? At my old work we had artifactory but I'm not sure what folks used out there

So my new place we use the one GCP provides

^don't be discourage if you find this works for you, fundamentally there is nothing wrong IMO. We aim to serve the broad audience so we try to keep this simple

Is this simpler than

node(reuable_fn, "a", "b", outputs="sum"

? It's a few more keystroke, though maybe slightly clearer since the arguments are highlight at the top

@Nok Lam Chan yeah I'm not really sure which one is simpler at that point hence agreeing with your point that it complicates the "advanced" use cases

Bit late to the party, but I would suggest not binding input and output names to the

node

. Instead, I think you can get something halfway like:

@node
def step1(number):
    return number + 1

@node
def step2(number):
    return number + 1

@node
def step3(number):
    return number + 2

@pipeline
def my_pipe(my_input):
    first_sum = step1(something)
    second_sum = step2(first_sum)
    final_result = step3(second_sum)
    return final_result

my_pipe(1)

This is definitely not 100% what it should look like, but I think the benefit is that you are still constructing the DAG--even though it's function calls--at pipeline definition time.

I implemented something like this at work a while ago, but never really used it. I think a better solution to the issue I was trying to solve would be a kedro vscode plugin that can better show all the pipelines, nodes, catalog entries

The vs code extension now exits! https://marketplace.visualstudio.com/items?itemName=kedro.Kedro

Came across this library today which is relevant to what we were discussing:

Venusian is a library which allows you to defer the action of decorators. Instead of taking actions when a function, method, or class decorator is executed at import time, you can defer the action until a separate "scan" phase.

https://docs.pylonsproject.org/projects/venusian/en/latest/#using-venusian

Maybe late to the discussion, but we parameterize our various nodes, and I don't see how this would work well with decorators. Our pipeline looks a bit like this : 1. Split the data (1 Global, easy to do with a decorator) 2. Train a model (There is 1 per scenario, and its input is the output of the split_data node) 3. Tests with custom test cases that changes as real world data comes in (There is 1 per scenario, and it's input is its associated "train" node's output) 4. Summary/Reporting (1 global, and its input is all of the previous Tests from each scenarios)