What can cause `conf_source` in omegaconfigloader ...
# questionsa
Artur Dobrogowski
03/21/2024, 12:12 PMWhat can cause
conf_sourceKedroSession.createn
Nok Lam Chan
03/21/2024, 12:13 PMwhich version of kedro are you on? and how
CONF_SOURCEsettings.pya
Artur Dobrogowski
03/21/2024, 12:13 PMits not defined, version 0.19.3
n
Nok Lam Chan
03/21/2024, 12:15 PMwhere is the project root? i.e. where is your
pyproject.tomla
Artur Dobrogowski
03/21/2024, 12:15 PMit's spaceflights-pandas template - it's in root dir of spaceflights
n
Nok Lam Chan
03/21/2024, 12:16 PMthere are two
spaceflightsa
Artur Dobrogowski
03/21/2024, 12:17 PMthis is the path it looks for, when I run debug mode I've read the conf_source param of
context.config_loadern
Nok Lam Chan
03/21/2024, 12:18 PMOh interesting, does
kedro runNok Lam Chan
03/21/2024, 12:19 PMimage.png
a
Artur Dobrogowski
03/21/2024, 12:19 PMyes it does...
Artur Dobrogowski
03/21/2024, 12:19 PMthe problem arises when I try to manually load/patch config loader
n
Nok Lam Chan
03/21/2024, 12:19 PMLooks fine for me, actually , what is
self.context.config_loader.config_loadera
Artur Dobrogowski
03/21/2024, 12:20 PM Copy code
cl: AbstractConfigLoader = self.context.config_loader
try:
if self.CONFIG_KEY not in cl.config_patterns.keys():
cl.config_patterns.update(
{
self.CONFIG_KEY: [
self.CONFIG_FILE_PATTERN,
f"{self.CONFIG_FILE_PATTERN}/**",
]
}
)
vertex_conf = self._ensure_obj_is_dict(
self.context.config_loader.get(self.CONFIG_KEY)
)Artur Dobrogowski
03/21/2024, 12:21 PMI remember the cause before was when someone mixed accidently templated config loader with omegaconf. I think I've swapped all instances of TemplatedConfig class to OmegaConf, but maybe not
Artur Dobrogowski
03/21/2024, 12:21 PM^ this code throws error
Artur Dobrogowski
03/21/2024, 12:22 PMnevermind, I'll keep looking for that
n
Nok Lam Chan
03/21/2024, 12:22 PMIs this a subclass of ConfigLoader? this isn't coming from Kedro so it's hard for me to determine what's going on
Nok Lam Chan
03/21/2024, 12:22 PMit's likely an incorrect implementation
a
Artur Dobrogowski
03/21/2024, 12:22 PMyes I've probably missed something to update
Artur Dobrogowski
03/21/2024, 12:22 PMtrying to bump kedro-vertex to kedro 0.19
š 1
n
Nok Lam Chan
03/21/2024, 12:23 PMmaybe @marrrcin know something about this?
a
Artur Dobrogowski
03/21/2024, 12:24 PMI doubt it he remembers š, probably would need to dig in as well
m
marrrcin
03/21/2024, 12:47 PMYeah, so this code was to handle cases during Kedro's transition from ConfigLoader/TemplatedConfigLoader into OmegaConf. Where does it throw?
a
Artur Dobrogowski
03/21/2024, 12:48 PM Copy code
[03/21/24 13:47:29] WARNING Failed to confirm consent. No data was sent to Heap. Exception: [Errno 2] No such file or directory: plugin.py:306
'/home/adobrogo/projects/kedro/kedro-vertexai/spaceflights/spaceflights/.telemetry'
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Traceback (most recent call last) āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā®
ā /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro_vertexai/context_hel ā
ā per.py:66 in config ā
ā ā
ā 63 ā ā ā ā ā } ā
ā 64 ā ā ā ā ) ā
ā 65 ā ā ā vertex_conf = self._ensure_obj_is_dict( ā
ā ā± 66 ā ā ā ā self.context.config_loader.get(self.CONFIG_KEY) ā
ā 67 ā ā ā ) ā
ā 68 ā ā except MissingConfigException: ā
ā 69 ā ā ā if not isinstance(cl, OmegaConfigLoader): ā
ā ā
ā /home/adobrogo/.pyenv/versions/3.10.4/lib/python3.10/_collections_abc.py:819 in get ā
ā ā
ā 816 ā def get(self, key, default=None): ā
ā 817 ā ā 'D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.' ā
ā 818 ā ā try: ā
ā ā± 819 ā ā ā return self[key] ā
ā 820 ā ā except KeyError: ā
ā 821 ā ā ā return default ā
ā 822 ā
ā ā
ā /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro/config/omegaconf_con ā
ā fig.py:199 in __getitem__ ā
ā ā
ā 196 ā ā else: ā
ā 197 ā ā ā base_path = str(Path(<http://self._fs.ls|self._fs.ls>("", detail=False)[-1]) / self.base_env) ā
ā 198 ā ā try: ā
ā ā± 199 ā ā ā base_config = self.load_and_merge_dir_config( # type: ignore[no-untyped-cal ā
ā 200 ā ā ā ā base_path, patterns, key, processed_files, read_environment_variables ā
ā 201 ā ā ā ) ā
ā 202 ā ā except UnsupportedInterpolationType as exc: ā
ā ā
ā /home/adobrogo/projects/kedro/virtualenv/lib/python3.10/site-packages/kedro/config/omegaconf_con ā
ā fig.py:286 in load_and_merge_dir_config ā
ā ā
ā 283 ā ā # noqa: too-many-locals ā
ā 284 ā ā ā
ā 285 ā ā if not self._fs.isdir(Path(conf_path).as_posix()): ā
ā ā± 286 ā ā ā raise MissingConfigException( ā
ā 287 ā ā ā ā f"Given configuration path either does not exist " ā
ā 288 ā ā ā ā f"or is not a valid directory: {conf_path}" ā
ā 289 ā ā ā ) ā
ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
MissingConfigException: Given configuration path either does not exist or is not a valid directory: /home/adobrogo/projects/kedro/kedro-vertexai/spaceflights/spaceflights/conf/baseArtur Dobrogowski
03/21/2024, 12:48 PMI noticed that it got it wrong even at the telemetry warning before
Artur Dobrogowski
03/21/2024, 12:49 PMI think it was working before... I'm very puzzled. kedro run/kedro ipython get it right
Artur Dobrogowski
03/21/2024, 12:49 PMthis happens only when I try to
kedro vertexai run-onceArtur Dobrogowski
03/21/2024, 12:50 PMprobably some context-loading helper class does something in way that was changed but I can't track it down yet
m
marrrcin
03/21/2024, 12:51 PMa
Artur Dobrogowski
03/21/2024, 12:52 PMyea I'm not sure how to debug it though
Artur Dobrogowski
03/21/2024, 12:52 PMmaybe I can link it instead packaging and installing and then try to use debugger
m
marrrcin
03/21/2024, 12:53 PMYup, that's the most convenient way - just remember to use poetry in both places
a
Artur Dobrogowski
03/21/2024, 12:54 PMI hate configuring debugger in vs code š
š¤£ 1
m
marrrcin
03/21/2024, 12:57 PMpycharm ftw
n
Nok Lam Chan
03/21/2024, 1:02 PM> I hate configuring debugger in vs code š (edited)
Same here, but recently I learnt it's less annoying than it used to be. At least for the case of Kedro I always have to look up the path of the executable before. Now you can choose
Python modulepython -m kedro <args>a
Artur Dobrogowski
03/22/2024, 3:07 PMSo FYI @Nok Lam Chan the issue was change in API for
KedroSession.createšš¼ 1
šļø 1