Hi everyone! I'm trying to create a hook to overwrite some parameters if scoring pipeline runs, but it does not seem to be working (the parameters dont get written - if not present - not overwritten - if present-). The code im using is the following:

def before_pipeline_run(self, run_params, catalog: DataCatalog) -> None:
        """Change feature inclusion parameters for
        scoring pipeline
        """
        if run_params["pipeline_name"] == "scoring":
            # retrieve feature_list from catalog
            feature_list_df = catalog.load("modeling.feature_selection_report")
            feature_list = list(feature_list_df[feature_list_df.selected == True].feature.unique())

            # get list of feature engineering pipelines
            params = catalog.load("parameters")
            feateng_pipes = [fteng_name for fteng_name in params.keys() if fteng_name.endswith("_fteng")]

            # overwrite parameters
            for pipeline in feateng_pipes:
                catalog.add_all(
                    {f"params:{pipeline}.feature_inclusion_params.feature_list": feature_list,
                    f"params:{pipeline}.feature_inclusion_params.enable_regex": True},
                    replace=True
                )

I also tried using

run_params["params"]

without any luck, and tried returning the catalog but no luck. The hook runs (tested with print statements), so my guess is i'm missing something. Thanks!

catalog.add

/

catalog.add_all

only adds

AbstractDataSet

entries to the catalog, without saving them, my guess is that you should have sth like this:

catalog.add_all(
                    {f"params:{pipeline}.feature_inclusion_params.feature_list": MemoryDataSet(feature_list),
                    f"params:{pipeline}.feature_inclusion_params.enable_regex": MemoryDataSet(True)},
                    replace=True
                )

Thanks Marrrcin! i'll give it a try

You should also consider calling

catalog.add_feed_dict

instead <-- this is what Kedro actually does to add parameters to the catalog

yeap, i tried that first and didnt work either, but i think with the MemoryDataSet might work

Where do you print the params?

just after (inside the hooks imp) and then inside the node (just to make sure)

OK, but do you use

parameters

or your specific keys e.g.

f"params:{pipeline}.feature_inclusion_params.feature_list"

?

i use my specific keys

Thank you @marrrcin awesome help here!

🙂 ❤️