Hi everyone. I'm trying to deploy my kedro pipelin...
# plugins-integrations
s
Hi everyone. I'm trying to deploy my kedro pipeline in Vertex AI in GCP using the kedro-vertexai plugin, but I have problems with the installation. When I run
kedro info
in the CLI, I got this error at the beginning and I can't use any command of this plugin:
Copy code
WARNING  Failed to load kedro_vertexai.cli commands from EntryPoint(name='vertexai', value='kedro_vertexai.cli:commands', group='kedro.project_commands'). Full exception:    utils.py:338
                             invalid syntax (generator.py, line 191)                                                                                                                                          
[01/12/23 23:05:27] WARNING  Failed to load kedro_vertexai.cli commands from EntryPoint(name='vertexai', value='kedro_vertexai.cli:commands', group='kedro.project_commands'). Full exception:    utils.py:338
                             invalid syntax (generator.py, line 191)                                                                                                                                          

 _            _
| | _____  __| |_ __ ___
| |/ / _ \/ _` | '__/ _ \
|   <  __/ (_| | | | (_) |
|_|\_\___|\__,_|_|  \___/
v0.18.4

Kedro is a Python framework for
creating reproducible, maintainable
and modular data science code.

Installed plugins:
kedro_docker: 0.3.1 (entry points:project)
kedro_telemetry: 0.2.3 (entry points:cli_hooks,hooks)
kedro_vertexai: 0.6.0 (entry points:hooks,project)
kedro_viz: 5.1.1 (entry points:global,line_magic)
I'm working on a Vertex AI workbench wich has Anaconda and Jupyter. I'd appreciate your help to solve this issue. Thanks!! PD: kedro-docker and kedro-viz work perfect. #questions #plugins-integrations
e
could you try with the newer kedro-vertexai plugin version? the most recent one is
0.8.1
-> https://pypi.org/project/kedro-vertexai/
☝️ 1
1
s
Hi @em-pe. Thanks for your advice. Effectively I tried with version 0.8.1 and Python 3.9 (I was working with the default version 3.7) and it worked and created the pipeline in Vertex AI. Now I have another issue, because when the pipeline starts to run in Vertex it rised the following error:
Copy code
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/kedro:8 in <module>                                           │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/cli/cli.py:211 in     │
│ main                                                                         │
│                                                                              │
│   208 │   """                                                                │
│   209 │   _init_plugins()                                                    │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                 │
│ ❱ 211 │   cli_collection()                                                   │
│   212                                                                        │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/click/core.py:1130 in __call__        │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/cli/cli.py:139 in     │
│ main                                                                         │
│                                                                              │
│   136 │   │   )                                                              │
│   137 │   │                                                                  │
│   138 │   │   try:                                                           │
│ ❱ 139 │   │   │   super().main(                                              │
│   140 │   │   │   │   args=args,                                             │
│   141 │   │   │   │   prog_name=prog_name,                                   │
│   142 │   │   │   │   complete_var=complete_var,                             │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/click/core.py:1055 in main            │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/click/core.py:1657 in invoke          │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/click/core.py:1404 in invoke          │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/click/core.py:760 in invoke           │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/cli/project.py:381 in │
│ run                                                                          │
│                                                                              │
│   378 │   tag = _get_values_as_tuple(tag) if tag else tag                    │
│   379 │   node_names = _get_values_as_tuple(node_names) if node_names else n │
│   380 │                                                                      │
│ ❱ 381 │   with KedroSession.create(env=env, extra_params=params) as session: │
│   382 │   │   session.run(                                                   │
│   383 │   │   │   tags=tag,                                                  │
│   384 │   │   │   runner=runner(is_async=is_async),                          │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/session/session.py:18 │
│ 0 in create                                                                  │
│                                                                              │
│   177 │   │   session._store.update(session_data)                            │
│   178 │   │                                                                  │
│   179 │   │   # we need a ConfigLoader registered in order to be able to set │
│ ❱ 180 │   │   session._setup_logging()                                       │
│   181 │   │   return session                                                 │
│   182 │                                                                      │
│   183 │   def _get_logging_config(self) -> Dict[str, Any]:                   │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/session/session.py:19 │
│ 5 in _setup_logging                                                          │
│                                                                              │
│   192 │   def _setup_logging(self) -> None:                                  │
│   193 │   │   """Register logging specified in logging directory."""         │
│   194 │   │   try:                                                           │
│ ❱ 195 │   │   │   logging_config = self._get_logging_config()                │
│   196 │   │   except MissingConfigException:                                 │
│   197 │   │   │   self._logger.debug(                                        │
│   198 │   │   │   │   "No project logging configuration loaded; "            │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/framework/session/session.py:18 │
│ 4 in _get_logging_config                                                     │
│                                                                              │
│   181 │   │   return session                                                 │
│   182 │                                                                      │
│   183 │   def _get_logging_config(self) -> Dict[str, Any]:                   │
│ ❱ 184 │   │   logging_config = self._get_config_loader()["logging"]          │
│   185 │   │   # turn relative paths in logging config into absolute path     │
│   186 │   │   # before initialising loggers                                  │
│   187 │   │   logging_config = _convert_paths_to_absolute_posix(             │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/config/config.py:111 in         │
│ __getitem__                                                                  │
│                                                                              │
│   108 │   │   )                                                              │
│   109 │                                                                      │
│   110 │   def __getitem__(self, key):                                        │
│ ❱ 111 │   │   return self.get(*self.config_patterns[key])                    │
│   112 │                                                                      │
│   113 │   def __repr__(self):  # pragma: no cover                            │
│   114 │   │   return (                                                       │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/config/config.py:125 in get     │
│                                                                              │
│   122 │   │   return _remove_duplicates(self._build_conf_paths())            │
│   123 │                                                                      │
│   124 │   def get(self, *patterns: str) -> Dict[str, Any]:  # type: ignore   │
│ ❱ 125 │   │   return _get_config_from_patterns(                              │
│   126 │   │   │   conf_paths=self.conf_paths, patterns=list(patterns)        │
│   127 │   │   )                                                              │
│   128                                                                        │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/kedro/config/common.py:69 in          │
│ _get_config_from_patterns                                                    │
│                                                                              │
│    66 │                                                                      │
│    67 │   for conf_path in conf_paths:                                       │
│    68 │   │   if not Path(conf_path).is_dir():                               │
│ ❱  69 │   │   │   raise ValueError(                                          │
│    70 │   │   │   │   f"Given configuration path either does not exist "     │
│    71 │   │   │   │   f"or is not a valid directory: {conf_path}"            │
│    72 │   │   │   )                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: Given configuration path either does not exist or is not a valid 
directory: /home/kedro_docker/conf/local
This error also raises when I run the container with
docker run
in the CLI in the local machine. Nevertheless, when I run it with
kedro docker run
It runs perfectly. I'm following the quick start guide https://kedro-vertexai.readthedocs.io/en/0.8.1/source/03_getting_started/01_quickstart.html Thanks again! #questions #plugins-integrations
m
Seems like the conf/local is in .dockerignore, either run in a different Kedro environment or add those files to the docker image
s
Hi @marrrcin/@em-pe. Thanks for your time. You were right, I had to add the
conf/local
and
log
folders to the Docker image although kedro-vertex didn´t do. On the other hand, maybe do you have an example or a more detailed explanation of the vertexai.yml to configure the resources for specific nodes or to send notifications?
e
It’s actually done on purpose - by convention
conf/local
should contain only the configuration that you are using on your local machine with
kedro run
. The basic config should land under
conf/base
which is the default when you don’t specify environment or in
conf/<your-env>
if you want to use the custom one. When it comes to the documentation I believe all we have at the moment are the comments that land in the generated default config file - see https://github.com/getindata/kedro-vertexai/blob/9c07518af6476cdcb87dba2304027b52d733096e/kedro_vertexai/config.py#L7
👍 1