https://kedro.org/ logo
Join Slack
Powered by
How can I pass a callable to `converters` for `rea...
# questions
s
How can I pass a callable to 
converters
for 
read_excel
E.g.:
Copy code
dataset:
  type: pandas.ExcelDataset
  filepath: file.xlsx
  load_args:
    converters:
      col: str
    dtype_backend: pyarrow
I have a column that contains mixed types. If I do not specify the converter, then pyarrow accept the df at all. But, specify dataset in catalog as above results in passing the string 
"str"
to read_excel instead of the func 
str
. Is there any way around this?
In regular code, this works:
Copy code
pd.read_excel(
    "file.xlsx",
    converters={"Col": str},
    dtype_backend="pyarrow",
)
This raises an exception [i.e. i need the converter]:
Copy code
pd.read_excel(
    "file.xlsx",
    dtype_backend="pyarrow",
)
And this is equivalent to what kedro does; also raises exception:
Copy code
pd.read_excel(
    "file.xlsx",
    converters={"Col": "str"},
    dtype_backend="pyarrow",
)
d
Omegaconf resolver!
s
Do you mean there's already a resolver for this use-case? Or just add:
Copy code
"custom_resolvers": {
        "builtins": lambda name: getattr(builtins, name),
    },
d
the examples on the docs IIRC is a polars example for this very problem
s
l
I would love to have a more formal dependency injection system in Kedro though, we've been using it for ages and I can't imagine a life without it anymore
Hi btw @Simon Bull 🙂
👋 1
n
I would love to have a more formal dependency injection system in Kedro though
What is missing?
l
We use a dependency injection system in the params, as follows:
Copy code
# params
dataset_search_experiment:

  # Experiments to run, each experiment should at least define a `task_fn` that is
  # a valid LangFuse task function. Additional entries are passed as kwargs to the task function.
  # <https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk#usage-with-langfuse-datasets>
  experiments:
    - name: exp-gpt-5-mini
      model: openai:gpt-5-mini-2025-08-07
      prompt_version: 1
      task_fn:
        _object: pipelines.dataset_search.pipeline.generate_dataset_search_task
Copy code
# nodes.py

# essentially inject object scans inputs for _object keyword and instantiates the object/fn
@inject_object()
def run_experiments(experiments):
   # thanks to decorator, the task_fn is now properly loaded as function
   experiments[0].task_fn()
Happy to show you if relevant
n
This somewhat remind me of the hydra config loader.
Copy code
task_fn: ${di:pipelines.dataset_search.pipeline.generate_dataset_search_task}
This is essentially the same as having a resolver like this?
Copy code
def di(dependency):
   return importlib.import_module(dependency)
l
Its alike indeed, though very often we DI nested objects, e.g., a hyperparameter tuner with a specific model nested inside. This is a little trickier with the setup above.
n
Do you have an example of it?
l
e.g.,
Copy code
model_tuning_args:
        tuner:
          _object: matrix.pipelines.modelling.tuning.GaussianSearch
          estimator:
            _object: lightgbm.LGBMClassifier
            n_jobs: 16
            random_state: ${globals:random_state}
            device: cuda
            # objective: binary Confirm with chunyu if this is needed as we are doing multiclass.
            boosting_type: gbdt
            force_row_wise: true
            # Built-in early stopping (works without fit_params)
            early_stopping_round: 50
          n_calls: 20
          splitter:
            _object: sklearn.model_selection.StratifiedKFold
            n_splits: 2
            random_state: ${globals:random_state}
            shuffle: True
        features: # Features use regex, source_0, source_1, .., target_0, target_1
          - source_\d+
          - target_\d+
        target_col_name: y
👍🏼 1
👀 1
@Nok Lam Chan We very much drive Kedro nodes through their interfaces, and then implement classes to get custom behaviour in, this then tracks behaviour through the parameters which I think works really well, e.g., with example above we respect the splitter interface from sklearn, but also implement our own custom splitting methods
n
Do I understand correctly this is equivalent to something like 
GaussianSearch(estimator, n_calls, splitter)
?
l
true, but each of them in turn are objects, if the 
_object
kw is found, the object with the path is instantiated, with construction args as the other kvs in the yaml dict
n
gotcha. I see as resolver doesn't have a natural way to support these nested constructor cleanly. I think this is slightly magical, but reasonable approach to deal with these case. Essentially the current Kedro will enforce you to do something like:
Copy code
def run_experiments(experiments, model_tuning_args): # This is still "pure" config
      # This is equivalent to the _object instantiate bit
      experiments = resolve_objects(experiments)  
      model_tuning_args = resolve_objects(model_tuning_args)
      ...
I guess this is your own implementation, just curious how different is it compare to hydra? (it use 
_target
instead) https://hydra.cc/docs/advanced/instantiate_objects/overview/
As I remember there are teams that taking similar approach with hydra + hook
Copy code
@hook_impl
    def before_node_run(xxx):
        # Iterate config and auto instantiate
Though I think the decorator approach is better in a way that at least this is more transparent.
l
I see, did not know hydra had this
Cheers for digging into!
4 Views
Open in Slack
PreviousNext
Powered by