I have two data preprocessing pipelines that diffe...
# questions
m
I have two data preprocessing pipelines that differ only by *one node function*—the signature is identical, the name and the body are slightly different. Can I create a modular pipeline where I am able to provide the function name that the node must call as its input? An example would be:
Copy code
preprocessing_a = pipeline(pipe=preprocess, inputs={"params:function": "func_a"})
preprocessing_b = pipeline(pipe=preprocess, inputs={"params:function": "func_b"})
d
I would do this with
importlib.import_module
and essentially call the function within your function
or if you know what you’re importing you can do something like this
Copy code
def some_func(df, target:str, path):
   writer = getattr(df,f'to_{target}')
   writer(path)

some_func(df, "csv", "/path/to/csv")
some_func(df, "parquet", "/path/to/pq"