https://kedro.org/ logo
Join Slack
Powered by
hey all, looking to deploy kedro pipelines in AWS ...
# questions
m
hey all, looking to deploy kedro pipelines in AWS SageMaker. The kedro-sagemaker plugin looks useful, however, does not seem to be actively maintained, and does not work with kedro 19. Any other suggestions? Cheers!
j
Hi Mark, Thank you for your question. I will look into it.
m
thanks Jitendra!
d
Hi @Mark Einhorn, We’re currently revising Kedro deployment methods and prioritising our next steps in that direction. I believe for SageMaker, you should still be able to use the plugin to transform Kedro pipelines, even with Kedro 0.19—try unpinning the dependency in plugins 
pyproject.toml
. We’d greatly appreciate it if you could spend 15-20 minutes with us for an interview where we can discuss your experience deploying Kedro projects. Your insights would be incredibly valuable in shaping our next priorities for the Kedro deployment milestone.
👍 2
m
thanks Dimitry! Will give it a go!
👍 1
g
Hi @Dmitry Sorokin, I have the same issue. When you say
Copy code
try unpinning the dependency in plugins pyproject.toml
do you mean building the plugin locally with Poetry and then using it? Would you recommend just publishing with a different repo name?
If anyone is interested, I have: 1. unpinned the upper bound from 
pyproject.toml
and changed the package name 2. rebuilt poetry lock with 
--no-update
3. built and published the package 4. successfully rebuilt my Kedro project with the following dependencies (I have the mlflow plugin and build failed using a cached PyYAML-5.4.1)
Copy code
kedro
kedro-viz
kedro-mlflow
gab-kedro-sagemaker # <------ here put your the changed packaged name
snowflake-sqlalchemy
PyYAML >= 6
I will tell you if it works
👍 1
m
very cool @Gabriele Cacchioni, keen to hear how it goes!
d
yes, please tell us how it works, @Gabriele Cacchioni! If you will have issues with Omega Config Loader during execution, you can change it to previous Config Loader, because the plugin is most likely incompatible with new Omega Config Loader. For doing that you can go to the 
settings.py
file inside of 
src/your_project_name/
folder and change from 
from kedro.config import OmegaConfigLoader
to 
from kedro.config import ConfigLoader
👍 1
g
Ok it breaks here
Copy code
❯ kedro sagemaker init -h
[...] REDACTED OLD LOG

[10/16/24 12:42:50] WARNING  Failed to load kedro_sagemaker.cli     utils.py:366
                             commands from                                      
                             EntryPoint(name='sagemaker',                       
                             value='kedro_sagemaker.cli:commands',              
                             group='kedro.project_commands'). Full              
                             exception: cannot import name                      
                             'AbstractDataSet' from 'kedro.io'                  
                             (/Users/gabriele/opt/anaconda3/envs/ke             
                             dro-environment-2/lib/python3.10/site-             
                             packages/kedro/io/__init__.py)
Basically it fails to load the entry point 
EntryPoint(name='sagemaker', value='kedro_sagemaker.cli:commands', group='kedro.project_commands')
The stack is: • 
main()
(
__main__.py
, line 47, in the generated kedro project) • 
run = _find_run_command(package_name)
(
__main__.py
, line 42) • 
plugins = load_entry_points("project")
(
__main__.py
, line 18) • 
loaded_entry_point = _safe_load_entry_point(entry_point)
(kedro.framework.cli.utils.py line 391) The subsequent call to 
entry_point.load()
fails for the 
sagemaker
entry point. Any suggestions?
In the same environment, I can do:
Copy code
from kedro.io import AbstractDataset
with no issues 🤔 That was a lie and this is the problem:
Copy code
from kedro.io import AbstractDataSet

Traceback (most recent call last):
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-46e9178f7ea0>", line 1, in <module>
    from kedro.io import AbstractDataSet
ImportError: cannot import name 'AbstractDataSet' from 'kedro.io' (/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro/io/__init__.py)
Ok here is the error reproduced
Copy code
import _frozen_importlib as _bootstrap
_bootstrap._gcd_import('kedro_sagemaker.cli', None, 0)
[10/16/24 12:57:59] INFO     Using 'conf/logging.yml' as logging __init__.py:270
                             configuration. You can change this                 
                             by setting the KEDRO_LOGGING_CONFIG                
                             environment variable accordingly.                  
Traceback (most recent call last):
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-e53a65254232>", line 1, in <module>
    _bootstrap._gcd_import('kedro_sagemaker.cli', None, 0)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro_sagemaker/cli.py", line 12, in <module>
    from kedro_sagemaker.cli_functions import (
  File "/Users/gabriele/Applications/PyCharm <http://Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py|Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py>", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro_sagemaker/cli_functions.py", line 12, in <module>
    from kedro_sagemaker.generator import KedroSageMakerGenerator
  File "/Users/gabriele/Applications/PyCharm <http://Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py|Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py>", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro_sagemaker/generator.py", line 42, in <module>
    from kedro_sagemaker.datasets import SageMakerModelDataset
  File "/Users/gabriele/Applications/PyCharm <http://Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py|Professional.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py>", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro_sagemaker/datasets.py", line 12, in <module>
    from <http://kedro.io|kedro.io> import AbstractDataSet, DataSetError
ImportError: cannot import name 'AbstractDataSet' from '<http://kedro.io|kedro.io>' (/Users/gabriele/opt/anaconda3/envs/kedro-environment-2/lib/python3.10/site-packages/kedro/io/__init__.py)
LOL it’s a renaming thing
Copy code
## Breaking changes to the API

## Upcoming deprecations for Kedro 0.19.0
* Renamed abstract dataset classes, in accordance with the [Kedro lexicon](<https://github.com/kedro-org/kedro/wiki/Kedro-documentation-style-guide#kedro-lexicon>). Dataset classes ending with "DataSet" are deprecated and will be removed in 0.19.0. Note that all of the below classes are also importable from `<http://kedro.io|kedro.io>`; only the module where they are defined is listed as the location.
🫠 1
facepalming
m
so it's working now?
g
I will try to rename the imports and add it to the fork
🤞 1
d
yea, sorry for that, Datasets renaming was one of the parts of 0.19 breaking changes
m
think everyone of our developers has been caught out by it before 😂
g
Could have been worse 😄
Copy code
============================== 73 failed, 109 passed, 9 xfailed, 8 xpassed, 79 warnings in 616.57s (0:10:16) ==============================
Gonna go YOLO mode and see if this somehow works
d
@Gabriele Cacchioni - you can try to use different approach also - you can create a container for your kedro project using kedro-docker plugin - it works well. Then use kedro-sagemaker plugin only to convert your kedro project to sagemaker pipeline syntax. I mean - you can try to use current plugin itself with new(current) kedro version installed. The only problem with that approach should be with OmegaConfLoader, and I wrote above how to handle it. Then you can receive pipeline and move it manually to Sagemaker, as well as a Docker container
m
Hey @Gabriele Cacchioni just wanted to follow up on this and see where you got to?
g
I tried to fix the plugin but I kept getting errors, so I decided not to use it. I am packaging the pipeline on my own
👍 1
Open in Slack
PreviousNext
Powered by