Hello I am trying to override a bool parameter using the CLI

Question

Hello I am trying to override a bool parameter using the CLI running from bash `kedro run params round occupancy False` However the `False` is read as a string Is there a way to pass a boolean instead Note that the original param is correctly read from the Yaml file as a bool

datajoely · Answer

`false` I think

datajoely · Answer

```In 3 import yaml In 4 yaml safe load test false Out 4 test False ```

Vassilis Kalofolias · Answer

The Yaml is read correctly The problem is when I pass it to the CLI from bash `kedro run params round occupancy False` gt python reads str ` False ` `kedro run params round occupancy false ` gt python reads str ` false `

datajoely · Answer

ah that s annoying

datajoely · Answer

okay it s then getting coerced by click badly

datajoely · Answer

so there is a way to get round this sort of

datajoely · Answer

`kedro run config config yaml` which you can do in an ugly one liner this way `echo params n tround occupancy False gt config yaml amp amp kedro run config config yaml`

Vassilis Kalofolias · Answer

Hmm that s weird there seems to be a problem with argument ` config` or I am doing something wrong ```$ poetry run kedro run config config yaml Traceback most recent call last home lt username gt lt path to kedro project gt venv bin kedro 8 in lt module gt home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages kedro framework cli cli py 211 in main 208 209 init plugins 210 cli collection = KedroCLI project path=Path cwd 211 cli collection 212 home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 1130 in call home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages kedro framework cli cli py 139 in main 136 137 138 try 139 super main 140 args=args 141 prog name=prog name 142 complete var=complete var home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 1055 in main home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 1655 in invoke home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 920 in make context home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 1378 in parse args home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 2360 in handle parse result home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages click core py 2322 in process value home lt username gt lt path to kedro project gt venv lib python3 9 site pac kages kedro framework cli utils py 377 in config file callback 374 section = ctx info name 375 376 if value 377 config = anyconfig load value section 378 ctx default map update config 379 380 return value KeyError run ```

datajoely · Answer

hmm

datajoely · Answer

can you show me the contents of your `config yml`

Vassilis Kalofolias · Answer

I tried with v1 ```params n tround occupancy False``` v2 ```parameters round occupancy False``` v3 ```params round occupancy False```

datajoely · Answer

and all failed

Vassilis Kalofolias · Answer

Yes

datajoely · Answer

datajoely · Answer

it looks like the top level key needs to be `run`

datajoely · Answer

sorry I wasn t aware

Vassilis Kalofolias · Answer

aahh let me try

Vassilis Kalofolias · Answer

muscle skin tone 3 it worked Thanks a lot

datajoely · Answer

sweat smile fantastic

Vassilis Kalofolias · Answer

OK now next level What is the easiest way to override parameters programmatically without CLI I am trying to run a pipeline in jupyter with different parameters for each run

Vassilis Kalofolias · Answer

sorry for insisting on these dynamic use cases sweat smile

datajoely · Answer

hooks

datajoely · Answer

so the `before pipeline run` hook has access to the `catalog` and you can just mutate them there

datajoely · Answer

but jupyter may make that a bit difficult

datajoely · Answer

jupyterlab may be easier as you could just open a seperate window edit the file and then reload kedro in another

Vassilis Kalofolias · Answer

I work in VS code so editing a file is not difficult But I was hoping to make it all in code without editing a file in a self contained notebook

datajoely · Answer

I guess you could use the python API but you do lose some features this is a typical scaffold for writing a Kedro pipeline in Pytest ``` from <http kedro io|kedro io> import DataCatalog from kedro runner import SequentialRunner from my pipe import create pipeline def test pipeline pipe = create pipeline runner = SequentialRunner catalog = DataCatalog runner run pipe catalog ```

datajoely · Answer

you don t need any YAML for that

Vassilis Kalofolias · Answer

So theoretically I should change directly the dictionary context params

datajoely · Answer

you don t get some things like hooks

datajoely · Answer

I think you can do `catalog add ` but I m not 100 sure

Vassilis Kalofolias · Answer

yeah this is what I do to edit datasets ```catalog edited = catalog shallow copy catalog edited add all replace=True ``` But I have to check if it works with parameters

datajoely · Answer

I have a feeling there may be a issue with that approach but in my opinion it SHOULD be the way for users to do this Would you mind checking

Vassilis Kalofolias · Answer

yeah this use case is really important for me if I want my data scientists to use the lib slightly smiling face I ll let you know how it goes or ask more questions P

datajoely · Answer

+1

Vassilis Kalofolias · Answer

So the easiest way for Jupyter is to reload kedro with a dict of extra params bypassing the magic This fails to get `False` as a bool ``` reload kedro params co2 base 0 000415 round occupancy False``` So we can do this instead ```from kedro ipython import reload kedro reload kedro extra params= co2 base 0 000415 round occupancy False ``` This is a bit hacky even though this is a public function so the magic has no benefit I am now searching for a way to do it without a session The problem is that the catalog has all levels of parameters for example ``` gt catalog list dataset1 dataset2 dataset3 parameters params param1 params group1 params group1 param1 params group1 param2 params group2 params group2 param1 params group2 param2 params group2 param3 ``` So the tricky thing is to not overwrite params individually especially for the nested ones This nesting is done through `kedro framework context context KedroContext get feed dict ` that automatically nests parameters to create a ready input for `add feed dict ` Unfortunately this function is not public I think `add feed dict ` should have an option like `recursive bool=False` to have this behavior This would allow for reproducing the behaviour of session in notebooks easily so that we reuse pipelines for prototyping more easily ```catalog with datasets 2 catalog add feed dict nested params dict recursive=True def test pipeline pipe = create pipeline runner = SequentialRunner catalog = DataCatalog params dict = catalog add feed dict params dict recursive=True runner run pipe catalog ``` The alternative would be possible if `run` would allow for `extra params`

Vassilis Kalofolias · Answer

What is in your opinion the most kedro acceptable style My goal is to make it easy for a junior data scientist to use notebooks with a ready mature kedro project

datajoely · Answer

so the annoying answer

datajoely · Answer

is this will get better in the future

datajoely · Answer

but today you need to hack about

datajoely · Answer

both our Jupyter workflow and our Parameter consistency pieces are under development

Vassilis Kalofolias · Answer

I am completely ok with that I just want to make sure that the general kedro moto don t set up dynamic runs will not lead to making such things more difficult in the future

datajoely · Answer

well I think things will get better

Vassilis Kalofolias · Answer

I am very happy to hear that It is a great tool for data engineering right now and onboarding also prototyping people would only make it more useful

Vassilis Kalofolias · Answer

Anyway thanks a lot for the quick feedback You are doing a great work

Vassilis Kalofolias · Answer

shall I right about the above findings in some discussion about future dev somewhere

datajoely · Answer

Hello < Vassilis Kalofolias> the `False` issue from the CLI is now fixed in 0 18 5 if you d like to upgrade