Hi Everyone, I was experimenting with

@singledispatchmethod

from the

functools

library to refactor my code and create “per-model type” implementations of

fit()

,

predict()

…). Unfortunately, this results in a

ValueError: Invalid Node definition: first argument must be a function, not 'singledispatchmethod'.

And indeed in kedro/pipeline/node.py:72

if not callable(func):
            raise ValueError(
                _node_error_message(
                    f"first argument must be a function, not '{type(func).__name__}'."
                )
            )

Is this “rejection” of functools.singledispatchmethod a “un-intended collateral” of the test (in which case I could make a pull request to handle it) or are there some things “down the line” that would justify not allow the use of functools & co ? 🙂 Thx

hi @Marc Gris, I tried the example from the docs and methods decorated with

@singledispatchmethod

are indeed callable:

In [3]: class Negator:
   ...:     @singledispatchmethod
   ...:     def neg(self, arg):
   ...:         raise NotImplementedError("Cannot negate a")
   ...: 
   ...:     @neg.register
   ...:     def _(self, arg: int):
   ...:         return -arg
   ...: 
   ...:     @neg.register
   ...:     def _(self, arg: bool):
   ...:         return not arg
   ...: 

In [4]: n = Negator()

In [5]: n.neg(10)
Out[5]: -10

In [6]: n.neg
Out[6]: <function __main__.Negator.neg(self, arg)>

In [7]: callable(n.neg)
Out[7]: True

can you share a snippet of your code to see what's going wrong?

Hi @Juan Luis Thx for your swift reply & check. Actually… You’re totally right and I messed up 😅 I used

@singledispatchmethod

to decorate a function instead of a method… With

@singledispatch

everyone does work perfectly. Mea Culpa 🙏 Since I cannot delete feel free to delete my message, please feel free to do so and declutter the thread 🙂 Thx again M.

@Marc Gris I’d love to learn more about this pattern and how it works for multiple model types! How is it working for you?

no worries @Marc Gris! a little bit of rubber ducking in the morning is always good rubberduck

Hi @datajoely 🙂 I’m afraid I can’t be of much help since I’m still experimenting / playing around. But basically, and as you may already know

@singledispatch

allows to do pattern matching and have different implementations of the decorated function for different input types (for the 1st arg). The idea for me was to avoid a messy / gigantic function body with many conditional branches like:

def train(model,**kwargs): 
    if isinstance(model, SuperModel):
         ...
    elif isinstance(model, MegaModel):
        ... 
    elif ... 
    elif ... 
    [...]
    else:
         raise NotImplementedError

So far, I was avoiding this by using dictionaries mapping from model name or types to specific functions, with things like:

TRAINERS = {SuperModel: train_super_model, 
            MegaModel: train_mega_model} 

def train(model, **kwargs):
    try:
        _train = TRAINERS[type(model)]
    except KeyError:
         raise NotImplementedError
    return _train(model, **kwargs)

I could have stuck to the above, but I’m assuming that

@singledispatch

exists “for many other good reasons which I’m not perceiving yet” and am therefore testing it out with things like:

@singledispatch
def train(model, **kwargs):
    raise NotImplementedError 

@train.register(SuperModel)
def _(model: SuperModel, **kwargs):
    # some beautiful code here ;-)

Since all of this is still completely experimental, please allow to return the question: How do you handle different models from different libraries with different inputs & methods in a single pipeline ? 🙂 Thx in advance (and for your youtube tutorial as well 😉 ) M.

Personally I use the method you've been using until now of having a mapping of functions to types, it's clear and simple enough and I like making things explicit. Admittedly I'd never heard of singledispatch before today.