Hello, I am trying to override a bool parameter using the CLI (running from bash):

kedro run --params round_occupancy:False

However the

False

is read as a string. Is there a way to pass a boolean instead? Note that the original param is correctly read from the Yaml file as a bool.

false

I think

The Yaml is read correctly. The problem is when I pass it to the CLI (from bash).

kedro run --params round_occupancy:False

-> python reads str

'False'

kedro run --params "round_occupancy: false"

-> python reads str

'false'

ah that’s annoying

Hmm that's weird, there seems to be a problem with argument

--config

(or I am doing something wrong):

$ poetry run kedro run --config config.yaml 

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/bin/kedro:8 in         │
│ <module>                                                                                         │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/cli.py:211 in main                                                     │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1130 in __call__                                                             │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/cli.py:139 in main                                                     │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1055 in main                                                                 │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1655 in invoke                                                               │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:920 in make_context                                                          │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1378 in parse_args                                                           │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:2360 in handle_parse_result                                                  │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:2322 in process_value                                                        │
│                                                                                                  │
│ /home/<username>/<--------         path_to_kedro_project    ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/utils.py:377 in _config_file_callback                                  │
│                                                                                                  │
│   374 │   section = ctx.info_name                                                                │
│   375 │                                                                                          │
│   376 │   if value:                                                                              │
│ ❱ 377 │   │   config = anyconfig.load(value)[section]                                            │
│   378 │   │   ctx.default_map.update(config)                                                     │
│   379 │                                                                                          │
│   380 │   return value                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'run'

hmm

I tried with: v1:

params:\n\tround_occupancy: False

v2:

parameters:
  round_occupancy: False

v3:

params:
  round_occupancy: False

and all failed?

Yes..

aahh let me try

😅 fantastic

OK now next level: What is the easiest way to override parameters programmatically? (without CLI)? I am trying to run a pipeline in jupyter, with different parameters for each run.

hooks

I work in VS code so editing a file is not difficult. But I was hoping to make it all in code (without editing a file) in a self-contained notebook.

I guess you could use the python API, but you do lose some features, this is a typical scaffold for writing a Kedro pipeline in Pytest

from <http://kedro.io|kedro.io> import DataCatalog
from kedro.runner import SequentialRunner

from my_pipe import create_pipeline


def test_pipeline():
    pipe = create_pipeline()
    runner = SequentialRunner()
    catalog = DataCatalog()
    runner.run(pipe, catalog)

So theoretically I should change directly the dictionary context.params?

you don’t get some things like hooks

yeah this is what I do to edit datasets:

catalog_edited = catalog.shallow_copy()
catalog_edited.add_all(..., replace=True)

But I have to check if it works with parameters.

I have a feeling there may be a issue with that approach - but in my opinion it SHOULD be the way for users to do this, Would you mind checking?

yeah, this use-case is really important for me if I want my data scientists to use the lib 🙂 I'll let you know how it goes (or ask more questions :P)

👍

So the easiest way for Jupyter is to reload kedro with a dict of extra_params, bypassing the magic: This fails to get

False

as a bool:

%reload_kedro --params co2_base:0.000415,round_occupancy:False

So we can do this instead:

from kedro.ipython import reload_kedro
reload_kedro(extra_params={"co2_base":0.000415, "round_occupancy": False})

This is a bit hacky (even though this is a public function so the magic has no benefit). I am now searching for a way to do it without a session. The problem is that the catalog has all levels of parameters, for example:

> catalog.list()

[
    'dataset1',
    'dataset2',
    'dataset3',
    'parameters',
    'params:param1',
    'params:group1',
    'params:group1.param1',
    'params:group1.param2',
    'params:group2',
    'params:group2.param1',
    'params:group2.param2',
    'params:group2.param3'
]

So the tricky thing is to not overwrite params individually, especially for the nested ones. This nesting is done through

kedro.framework.context.context.KedroContext._get_feed_dict()

that automatically nests parameters to create a ready input for

add_feed_dict()

. Unfortunately this function is not public... I think

add_feed_dict()

should have an option like

recursive: bool=False

to have this behavior. This would allow for reproducing the behaviour of session in notebooks easily so that we reuse pipelines for prototyping more easily:

catalog with datasets
2. catalog.add_feed_dict(nested_params_dict, recursive=True)

def test_pipeline():
    pipe = create_pipeline()
    runner = SequentialRunner()
    catalog = DataCatalog()
    params_dict = {...}
    catalog.add_feed_dict(params_dict, recursive=True)
    runner.run(pipe, catalog)

The alternative would be possible if

run

would allow for

extra_params

so the annoying answer

I am completely ok with that! I just want to make sure that the general kedro moto "don't set up dynamic runs" will not lead to making such things more difficult in the future.

well I think things will get better

I am very happy to hear that :)) It is a great tool for data engineering right now, and onboarding also prototyping people would only make it more useful :))

Hello @Vassilis Kalofolias - the

False

issue from the CLI is now fixed in 0.18.5 if you’d like to upgrade