Vassilis Kalofolias
02/16/2023, 11:06 AMkedro run --params round_occupancy:False
However the False
is read as a string. Is there a way to pass a boolean instead? Note that the original param is correctly read from the Yaml file as a bool.datajoely
02/16/2023, 11:06 AMfalse
I thinkIn [3]: import yaml
In [4]: yaml.safe_load('test: false')
Out[4]: {'test': False}
Vassilis Kalofolias
02/16/2023, 11:22 AMkedro run --params round_occupancy:False
-> python reads str 'False'
kedro run --params "round_occupancy: false"
-> python reads str 'false'
datajoely
02/16/2023, 11:22 AMkedro run --config config.yaml
which you can do in an ugly one liner this way
echo 'params:\n\tround_occupancy: False' > config.yaml && kedro run --config config.yaml
Vassilis Kalofolias
02/16/2023, 2:01 PM--config
(or I am doing something wrong):
$ poetry run kedro run --config config.yaml
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/bin/kedro:8 in │
│ <module> │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/cli.py:211 in main │
│ │
│ 208 │ """ │
│ 209 │ _init_plugins() │
│ 210 │ cli_collection = KedroCLI(project_path=Path.cwd()) │
│ ❱ 211 │ cli_collection() │
│ 212 │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1130 in __call__ │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/cli.py:139 in main │
│ │
│ 136 │ │ ) │
│ 137 │ │ │
│ 138 │ │ try: │
│ ❱ 139 │ │ │ super().main( │
│ 140 │ │ │ │ args=args, │
│ 141 │ │ │ │ prog_name=prog_name, │
│ 142 │ │ │ │ complete_var=complete_var, │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1055 in main │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1655 in invoke │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:920 in make_context │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:1378 in parse_args │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:2360 in handle_parse_result │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/click/core.py:2322 in process_value │
│ │
│ /home/<username>/<-------- path_to_kedro_project ------>/.venv/lib/python3.9/site-pac │
│ kages/kedro/framework/cli/utils.py:377 in _config_file_callback │
│ │
│ 374 │ section = ctx.info_name │
│ 375 │ │
│ 376 │ if value: │
│ ❱ 377 │ │ config = anyconfig.load(value)[section] │
│ 378 │ │ ctx.default_map.update(config) │
│ 379 │ │
│ 380 │ return value │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'run'
datajoely
02/16/2023, 2:02 PMconfig.yml
?Vassilis Kalofolias
02/16/2023, 2:05 PMparams:\n\tround_occupancy: False
v2:
parameters:
round_occupancy: False
v3:
params:
round_occupancy: False
datajoely
02/16/2023, 2:10 PMVassilis Kalofolias
02/16/2023, 2:10 PMdatajoely
02/16/2023, 2:11 PMrun
Vassilis Kalofolias
02/16/2023, 2:11 PMdatajoely
02/16/2023, 2:16 PMVassilis Kalofolias
02/16/2023, 2:16 PMdatajoely
02/16/2023, 2:16 PMbefore_pipeline_run
hook has access to the catalog
and you can just mutate them thereVassilis Kalofolias
02/16/2023, 2:20 PMdatajoely
02/16/2023, 2:21 PMfrom <http://kedro.io|kedro.io> import DataCatalog
from kedro.runner import SequentialRunner
from my_pipe import create_pipeline
def test_pipeline():
pipe = create_pipeline()
runner = SequentialRunner()
catalog = DataCatalog()
runner.run(pipe, catalog)
Vassilis Kalofolias
02/16/2023, 2:22 PMdatajoely
02/16/2023, 2:22 PMcatalog.add()
but I’m not 100% sureVassilis Kalofolias
02/16/2023, 2:24 PMcatalog_edited = catalog.shallow_copy()
catalog_edited.add_all(..., replace=True)
But I have to check if it works with parameters.datajoely
02/16/2023, 2:25 PMVassilis Kalofolias
02/16/2023, 2:26 PMdatajoely
02/16/2023, 2:26 PMVassilis Kalofolias
02/16/2023, 4:35 PMFalse
as a bool:
%reload_kedro --params co2_base:0.000415,round_occupancy:False
So we can do this instead:
from kedro.ipython import reload_kedro
reload_kedro(extra_params={"co2_base":0.000415, "round_occupancy": False})
This is a bit hacky (even though this is a public function so the magic has no benefit).
I am now searching for a way to do it without a session. The problem is that the catalog has all levels of parameters, for example:
> catalog.list()
[
'dataset1',
'dataset2',
'dataset3',
'parameters',
'params:param1',
'params:group1',
'params:group1.param1',
'params:group1.param2',
'params:group2',
'params:group2.param1',
'params:group2.param2',
'params:group2.param3'
]
So the tricky thing is to not overwrite params individually, especially for the nested ones. This nesting is done through kedro.framework.context.context.KedroContext._get_feed_dict()
that automatically nests parameters to create a ready input for add_feed_dict()
.
Unfortunately this function is not public... I think add_feed_dict()
should have an option like recursive: bool=False
to have this behavior. This would allow for reproducing the behaviour of session in notebooks easily so that we reuse pipelines for prototyping more easily:
catalog with datasets
2. catalog.add_feed_dict(nested_params_dict, recursive=True)
def test_pipeline():
pipe = create_pipeline()
runner = SequentialRunner()
catalog = DataCatalog()
params_dict = {...}
catalog.add_feed_dict(params_dict, recursive=True)
runner.run(pipe, catalog)
The alternative would be possible if run
would allow for extra_params
datajoely
02/16/2023, 4:37 PMVassilis Kalofolias
02/16/2023, 4:39 PMdatajoely
02/16/2023, 4:40 PMVassilis Kalofolias
02/16/2023, 4:40 PMdatajoely
02/21/2023, 10:45 AMFalse
issue from the CLI is now fixed in 0.18.5 if you’d like to upgrade