https://kedro.org/ logo
#questions
Title
# questions
a

Artur Dobrogowski

03/21/2024, 12:12 PM
What can cause
conf_source
in omegaconfigloader to duplicate project name? I've encountered the error before but don't remember the cause. self.context.config_loader.conf_source = '/home/adobrogo/projects/kedro/kedro-vertexai/spaceflights/spaceflights/conf' Answer: When migrating to 0.19 check your
KedroSession.create
arguments
n

Nok Lam Chan

03/21/2024, 12:13 PM
which version of kedro are you on? and how
CONF_SOURCE
is defined in
settings.py
a

Artur Dobrogowski

03/21/2024, 12:13 PM
its not defined, version 0.19.3
n

Nok Lam Chan

03/21/2024, 12:15 PM
where is the project root? i.e. where is your
pyproject.toml
?
a

Artur Dobrogowski

03/21/2024, 12:15 PM
it's spaceflights-pandas template - it's in root dir of spaceflights
n

Nok Lam Chan

03/21/2024, 12:16 PM
there are two
spaceflights
in the path you provided, is it the outer or the inner one?
a

Artur Dobrogowski

03/21/2024, 12:17 PM
this is the path it looks for, when I run debug mode I've read the conf_source param of
context.config_loader
and that's its value that is incorrect. I only have 1 spaceflights in the path
n

Nok Lam Chan

03/21/2024, 12:18 PM
Oh interesting, does
kedro run
or any kedro project command works as expected? is it just this attribute looks suspicious
image.png
a

Artur Dobrogowski

03/21/2024, 12:19 PM
yes it does...
the problem arises when I try to manually load/patch config loader
n

Nok Lam Chan

03/21/2024, 12:19 PM
Looks fine for me, actually , what is
self.context.config_loader.config_loader
? Is this your own code or coming from kedro
a

Artur Dobrogowski

03/21/2024, 12:20 PM
Copy code
cl: AbstractConfigLoader = self.context.config_loader
        try:
            if self.CONFIG_KEY not in cl.config_patterns.keys():
                cl.config_patterns.update(
                    {
                        self.CONFIG_KEY: [
                            self.CONFIG_FILE_PATTERN,
                            f"{self.CONFIG_FILE_PATTERN}/**",
                        ]
                    }
                )
            vertex_conf = self._ensure_obj_is_dict(
                self.context.config_loader.get(self.CONFIG_KEY)
            )
I remember the cause before was when someone mixed accidently templated config loader with omegaconf. I think I've swapped all instances of TemplatedConfig class to OmegaConf, but maybe not
^ this code throws error
nevermind, I'll keep looking for that
n

Nok Lam Chan

03/21/2024, 12:22 PM
Is this a subclass of ConfigLoader? this isn't coming from Kedro so it's hard for me to determine what's going on
it's likely an incorrect implementation
a

Artur Dobrogowski

03/21/2024, 12:22 PM
yes I've probably missed something to update
trying to bump kedro-vertex to kedro 0.19
šŸ‘€ 1
n

Nok Lam Chan

03/21/2024, 12:23 PM
maybe @marrrcin know something about this?
a

Artur Dobrogowski

03/21/2024, 12:24 PM
I doubt it he remembers šŸ˜„, probably would need to dig in as well
m

marrrcin

03/21/2024, 12:47 PM
Yeah, so this code was to handle cases during Kedro's transition from ConfigLoader/TemplatedConfigLoader into OmegaConf. Where does it throw?
a

Artur Dobrogowski

03/21/2024, 12:48 PM
Copy code
[03/21/24 13:47:29] WARNING  Failed to confirm consent. No data was sent to Heap. Exception: [Errno 2] No such file or directory:                                             plugin.py:306
                             '/home/adobrogo/projects/kedro/kedro-vertexai/spaceflights/spaceflights/.telemetry'                                                                           
ā•­ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ Traceback (most recent call last) ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•®
ā”‚ /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro_vertexai/context_hel ā”‚
ā”‚ per.py:66 in config                                                                              ā”‚
ā”‚                                                                                                  ā”‚
ā”‚   63 ā”‚   ā”‚   ā”‚   ā”‚   ā”‚   }                                                                       ā”‚
ā”‚   64 ā”‚   ā”‚   ā”‚   ā”‚   )                                                                           ā”‚
ā”‚   65 ā”‚   ā”‚   ā”‚   vertex_conf = self._ensure_obj_is_dict(                                         ā”‚
ā”‚ ā± 66 ā”‚   ā”‚   ā”‚   ā”‚   self.context.config_loader.get(self.CONFIG_KEY)                             ā”‚
ā”‚   67 ā”‚   ā”‚   ā”‚   )                                                                               ā”‚
ā”‚   68 ā”‚   ā”‚   except MissingConfigException:                                                      ā”‚
ā”‚   69 ā”‚   ā”‚   ā”‚   if not isinstance(cl, OmegaConfigLoader):                                       ā”‚
ā”‚                                                                                                  ā”‚
ā”‚ /home/adobrogo/.pyenv/versions/3.10.4/lib/python3.10/_collections_abc.py:819 in get              ā”‚
ā”‚                                                                                                  ā”‚
ā”‚    816 ā”‚   def get(self, key, default=None):                                                     ā”‚
ā”‚    817 ā”‚   ā”‚   'D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.'                    ā”‚
ā”‚    818 ā”‚   ā”‚   try:                                                                              ā”‚
ā”‚ ā±  819 ā”‚   ā”‚   ā”‚   return self[key]                                                              ā”‚
ā”‚    820 ā”‚   ā”‚   except KeyError:                                                                  ā”‚
ā”‚    821 ā”‚   ā”‚   ā”‚   return default                                                                ā”‚
ā”‚    822                                                                                           ā”‚
ā”‚                                                                                                  ā”‚
ā”‚ /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro/config/omegaconf_con ā”‚
ā”‚ fig.py:199 in __getitem__                                                                        ā”‚
ā”‚                                                                                                  ā”‚
ā”‚   196 ā”‚   ā”‚   else:                                                                              ā”‚
ā”‚   197 ā”‚   ā”‚   ā”‚   base_path = str(Path(<http://self._fs.ls|self._fs.ls>("", detail=False)[-1]) / self.base_env)       ā”‚
ā”‚   198 ā”‚   ā”‚   try:                                                                               ā”‚
ā”‚ ā± 199 ā”‚   ā”‚   ā”‚   base_config = self.load_and_merge_dir_config(  # type: ignore[no-untyped-cal   ā”‚
ā”‚   200 ā”‚   ā”‚   ā”‚   ā”‚   base_path, patterns, key, processed_files, read_environment_variables      ā”‚
ā”‚   201 ā”‚   ā”‚   ā”‚   )                                                                              ā”‚
ā”‚   202 ā”‚   ā”‚   except UnsupportedInterpolationType as exc:                                        ā”‚
ā”‚                                                                                                  ā”‚
ā”‚ /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro/config/omegaconf_con ā”‚
ā”‚ fig.py:286 in load_and_merge_dir_config                                                          ā”‚
ā”‚                                                                                                  ā”‚
ā”‚   283 ā”‚   ā”‚   # noqa: too-many-locals                                                            ā”‚
ā”‚   284 ā”‚   ā”‚                                                                                      ā”‚
ā”‚   285 ā”‚   ā”‚   if not self._fs.isdir(Path(conf_path).as_posix()):                                 ā”‚
ā”‚ ā± 286 ā”‚   ā”‚   ā”‚   raise MissingConfigException(                                                  ā”‚
ā”‚   287 ā”‚   ā”‚   ā”‚   ā”‚   f"Given configuration path either does not exist "                         ā”‚
ā”‚   288 ā”‚   ā”‚   ā”‚   ā”‚   f"or is not a valid directory: {conf_path}"                                ā”‚
ā”‚   289 ā”‚   ā”‚   ā”‚   )                                                                              ā”‚
ā•°ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•Æ
MissingConfigException: Given configuration path either does not exist or is not a valid directory: /home/adobrogo/projects/kedro/kedro-vertexai/spaceflights/spaceflights/conf/base
I noticed that it got it wrong even at the telemetry warning before
I think it was working before... I'm very puzzled. kedro run/kedro ipython get it right
this happens only when I try to
kedro vertexai run-once
probably some context-loading helper class does something in way that was changed but I can't track it down yet
a

Artur Dobrogowski

03/21/2024, 12:52 PM
yea I'm not sure how to debug it though
maybe I can link it instead packaging and installing and then try to use debugger
m

marrrcin

03/21/2024, 12:53 PM
Yup, that's the most convenient way - just remember to use poetry in both places
a

Artur Dobrogowski

03/21/2024, 12:54 PM
I hate configuring debugger in vs code šŸ˜„
šŸ¤£ 1
m

marrrcin

03/21/2024, 12:57 PM
pycharm ftw
n

Nok Lam Chan

03/21/2024, 1:02 PM
> I hate configuring debugger in vs code šŸ˜„ (edited) Same here, but recently I learnt it's less annoying than it used to be. At least for the case of Kedro I always have to look up the path of the executable before. Now you can choose
Python module
and add args, then it's done. This is equivalent to
python -m kedro <args>
(i.e. args = ["run"] 99% of the time)
a

Artur Dobrogowski

03/22/2024, 3:07 PM
So FYI @Nok Lam Chan the issue was change in API for
KedroSession.create
šŸ‘šŸ¼ 1
šŸŽ–ļø 1