https://kedro.org/ logo
#questions
Title
# questions
f

Felipe Vianna

07/06/2023, 1:29 PM
Hello folks. Is there any way I can use globals params from env other than base? problem: I have some configurations that are parsed from globals to other params files by reference
my_param: ${my_param_in_globals.yml}
for some reason we can parse params like that just from base globals.yml. and when i try to get from live/globals.yml it’s not taken.
j

Juan Luis

07/06/2023, 1:34 PM
hello Felipe! are you using the default
ConfigLoader
or something else? also, what version of Kedro are you using?
f

Felipe Vianna

07/06/2023, 1:36 PM
Hello Juan! I’m using
Copy code
from kedro.config import TemplatedConfigLoader
import os

CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {"globals_pattern": "*globals.yml",
                      "globals_dict": os.environ,
                      }
👍🏼 1
kedro, version 0.18.5
👍🏼 1
Any idea why --env live is not imposing the live/globals.yml values?
j

Juan Luis

07/06/2023, 1:51 PM
maybe it's because of the
globals_pattern
🤔 what about
Copy code
"globals_pattern": "**/globals.yml",
@Felipe Vianna?
👎🏼 1
hmm that was a shot in the dark and possibly not a good one. pinging my colleagues @Merel @Ahdra Merali
f

Felipe Vianna

07/06/2023, 1:57 PM
Let me try it
that was not it.
m

Merel

07/06/2023, 3:20 PM
I’m not quite sure I understand the setup. What config files do you have? And what’s the part that isn’t working?
Do you mean that when you have your globals file somewhere else than
base
the values aren’t passed on
kedro run
?
f

Felipe Vianna

07/06/2023, 3:57 PM
I have some values in live/globals.yml that I want to be taken when I do
kedro run --env live
but it always take the values in base
one thing that might help to debug. Is there a way to fetch what env is running? I’m assuming the --env live might not be working correctly, so it would be nice if I could log the env in some of the py file
m

Merel

07/06/2023, 4:14 PM
You could use the
before_pipeline_run
hook which has
run_params
and contains
env
f

Felipe Vianna

07/06/2023, 4:20 PM
I think I need something much simpler than hooks.. I really just need to be able to parse the ENV in the
kedro run --env ENV
m

Merel

07/06/2023, 4:20 PM
A quick hook I made that does the trick:
Copy code
class PipelineHooks:
    @property
    def _logger(self):
        return logging.getLogger(self.__class__.__name__)

    @hook_impl
    def before_pipeline_run(
            self, run_params: dict[str, Any], pipeline: Pipeline, catalog: DataCatalog
    ) -> None:
        self._logger.warning(f"ENV: {run_params['env']}")
f

Felipe Vianna

07/06/2023, 4:22 PM
That seems to be what I need =] thank you!
👍 1
another question. How can I do variable substitution passing the param through the CLI? eg.:
KEDRO_ENV=test kedro run --pipeline live_pipe --params=run_end_date:"2023-07-02 07:30:00"
the expected result is that the
run_end_date
would be taken from the CLI params instead of the globals.yml
m

Merel

07/07/2023, 8:16 AM
At the moment that doesn’t work out of the box. There’s a work around described on this ticket: https://github.com/kedro-org/kedro/issues/2640
f

Felipe Vianna

07/07/2023, 8:51 AM
I did that in settings, and ran the command above but got
Copy code
Failed to format pattern '${run_end_date}': no config value found, no default provided
j

Juan Luis

07/07/2023, 11:08 AM
hi @Felipe Vianna, did you manage to fix your original issue of
globals
not being picked up? happy to have another look at this and try to craft a reproducer
f

Felipe Vianna

07/07/2023, 11:23 AM
I did a work a workaround by running
KEDRO_ENV=live kedro run ….
and then in the py file I used the config loader passing the env I could get with os.getenv(). that forced the usage of the right globals.
j

Juan Luis

07/07/2023, 11:29 AM
I'm happy that you found a workaround although I must say that definitely feels less more complicated than it should be. I'll keep you posted if I find a better way, even if it's just for users finding a similar problem in the future. thanks @Felipe Vianna!
f

Felipe Vianna

07/07/2023, 11:38 AM
Thank you juan!