Hello Team Does anyone have experience in deploying kedro pi

Question

Hello Team Does anyone have experience in deploying kedro pipelines to aws stepfunctions please let me know i need some urgent help with that

Deepyaman Datta · Answer

I don t have experience with it but just checking are you following <https kedro readthedocs io en stable deployment aws step functions html> and running into specific issues with it Providing some more context around your issues are will probably help people understand who can help best

Suryansh Soni · Answer

< Deepyaman Datta> i have deployed my kedro pipeline on aws stepfuncitons but when i test it i get a os error

Suryansh Soni · Answer

` ` ` error OSError ` ` cause ` ` errorMessage Errno 38 Function not implemented ` ` errorType OSError ` ` requestId 47cd9191 2a44 4811 908b 3c5f3874e396 ` ` stackTrace ` ` File home app lambda handler py line 6 in handler n configure project radar ml n ` ` File home app kedro framework project init py line 243 in configure project n settings configure settings module n ` ` File home app dynaconf base py line 182 in configure n self wrapped = Settings settings module=settings module kwargs n ` ` File home app dynaconf base py line 238 in init n self validators validate n ` ` File home app dynaconf validator py line 441 in validate n validator validate n ` ` File home app kedro framework project init py line 46 in validate n super validate settings args kwargs n ` ` File home app dynaconf validator py line 207 in validate n self validate items n ` ` File home app dynaconf validator py line 236 in validate items n self default settings self n ` ` File home app kedro framework project init py line 37 in validator func n return getattr importlib import module module class name n ` ` File usr local lib python3 8 importlib init py line 127 in import module n return bootstrap gcd import name level package level n ` ` File lt frozen importlib bootstrap gt line 1014 in gcd import n ` ` File lt frozen importlib bootstrap gt line 991 in find and load n ` ` File lt frozen importlib bootstrap gt line 961 in find and load unlocked n ` ` File lt frozen importlib bootstrap gt line 219 in call with frames removed n ` ` File lt frozen importlib bootstrap gt line 1014 in gcd import n ` ` File lt frozen importlib bootstrap gt line 991 in find and load n ` ` File lt frozen importlib bootstrap gt line 975 in find and load unlocked n ` ` File lt frozen importlib bootstrap gt line 671 in load unlocked n ` ` File lt frozen importlib bootstrap external gt line 843 in exec module n ` ` File lt frozen importlib bootstrap gt line 219 in call with frames removed n ` ` File home app kedro framework session init py line 4 in lt module gt n from session import KedroSession n ` ` File home app kedro framework session session py line 26 in lt module gt n from kedro framework session store import BaseSessionStore n ` ` File home app kedro framework session store py line 46 in lt module gt n class ShelveStore BaseSessionStore n ` ` File home app kedro framework session store py line 49 in ShelveStore n lock = Lock n ` ` File usr local lib python3 8 multiprocessing context py line 68 in Lock n return Lock ctx=self get context n ` ` File usr local lib python3 8 multiprocessing synchronize py line 162 in init n SemLock init self SEMAPHORE 1 1 ctx=ctx n ` ` File usr local lib python3 8 multiprocessing synchronize py line 57 in init n sl = self semlock = multiprocessing SemLock n ` ` ` ` ` ` `

Nok Lam Chan · Answer

I think it s probably these environment don t support disable the multiprocessing context

Nok Lam Chan · Answer

At the moment `ShelveStore` create a `Lock` when `store py` get imported

Nok Lam Chan · Answer

We will probably move the `ShelveStore` to a separate module to prevent this from happening but it will take some more times to the next release < Suryansh Soni> Would you be able to test it out If I push the changes to a temporary branch

Suryansh Soni · Answer

yes i should be < Nok Lam Chan>

Suryansh Soni · Answer

are available for a quick chat

Lim H. · Answer

Sorry for butting in in step 2 2 in the guide it shows that you need to patch the Lock Might want to patch the SemLock as well

Lim H. · Answer

When I wrote this guide I found patching Lock was all you needed and I got the pipeline to run but looking at your exception maybe something has changed and patching SemLock is also neccessary

Suryansh Soni · Answer

< Lim H > then how can i add that change

Suryansh Soni · Answer

with patch Sem Lock

Lim H. · Answer

```with patch multiprocessing Lock patch multiprocessing SemLock from kedro framework session import KedroSession with KedroSession create env= aws as session session run node names= node to run ``` I haven t tested this but something like this

Lim H. · Answer

based off this line in the stacktrace ```self semlock = multiprocessing SemLock n ```

Lim H. · Answer

But generally I regret writing this guide I no longer think running Kedro on StepFunction is a good idea If you can try AWS Batch

Suryansh Soni · Answer

< Lim H > my team wants to deploy kedro on serverless framework that is why we are approaching this problem that way

Lim H. · Answer

+1 on lambda you need to keep patching unimplemented python runtime features so the general approach is follow the stack trace and patch away

Suryansh Soni · Answer

alright thank you

Suryansh Soni · Answer

i will implement this and let you know what happends

Nok Lam Chan · Answer

This is amazing < Lim H > thanks a lot

Suryansh Soni · Answer

still getting this issue after implementing the fix

Suryansh Soni · Answer

errorMessage Errno 38 Function not implemented errorType OSError requestId 9adc74c9 bf3d 4bcb a551 97fd172a809b stackTrace File home app lambda handler py line 6 in handler n configure project radar ml n File home app kedro framework project init py line 243 in configure project n settings configure settings module n File home app dynaconf base py line 182 in configure n self wrapped = Settings settings module=settings module kwargs n File home app dynaconf base py line 238 in init n self validators validate n File home app dynaconf validator py line 441 in validate n validator validate n File home app kedro framework project init py line 46 in validate n super validate settings args kwargs n File home app dynaconf validator py line 207 in validate n self validate items n File home app dynaconf validator py line 236 in validate items n self default settings self n File home app kedro framework project init py line 37 in validator func n return getattr importlib import module module class name n File usr local lib python3 9 importlib init py line 127 in import module n return bootstrap gcd import name level package level n File lt frozen importlib bootstrap gt line 1030 in gcd import n File lt frozen importlib bootstrap gt line 1007 in find and load n File lt frozen importlib bootstrap gt line 972 in find and load unlocked n File lt frozen importlib bootstrap gt line 228 in call with frames removed n File lt frozen importlib bootstrap gt line 1030 in gcd import n File lt frozen importlib bootstrap gt line 1007 in find and load n File lt frozen importlib bootstrap gt line 986 in find and load unlocked n File lt frozen importlib bootstrap gt line 680 in load unlocked n File lt frozen importlib bootstrap external gt line 850 in exec module n File lt frozen importlib bootstrap gt line 228 in call with frames removed n File home app kedro framework session init py line 4 in lt module gt n from session import KedroSession n File home app kedro framework session session py line 26 in lt module gt n from kedro framework session store import BaseSessionStore n File home app kedro framework session store py line 46 in lt module gt n class ShelveStore BaseSessionStore n File home app kedro framework session store py line 49 in ShelveStore n lock = Lock n File usr local lib python3 9 multiprocessing context py line 68 in Lock n return Lock ctx=self get context n File usr local lib python3 9 multiprocessing synchronize py line 162 in init n SemLock init self SEMAPHORE 1 1 ctx=ctx n File usr local lib python3 9 multiprocessing synchronize py line 57 in init n sl = self semlock = multiprocessing SemLock n

Deepyaman Datta · Answer

I d guess the patch isn t getting applied Maybe you need to patch ` multiprocessing SemLock`

Suryansh Soni · Answer

can you show me a code snippet on how to do that

Deepyaman Datta · Answer

`patch multiprocessing SemLock ` instead of `patch multiprocessing SemLock ` I m still not 100 sure this will work since patching isn t that straightforward and you need to patch it the way it gets imported but it looks right to me based on how it s being called in `multiprocessing synchronize py` I also didn t know you can do multiple patches in a `with` statement TIL

Suryansh Soni · Answer

< Deepyaman Datta> still causing the same issue

Suryansh Soni · Answer

does work

Suryansh Soni · Answer

there is some major disconnect

Nok Lam Chan · Answer

Does the patching solution work at the end If so would you mind sharing what works < Suryansh Soni>

Suryansh Soni · Answer

Hello < Nok Lam Chan> that patch solution didnt work still got the same issue looks like with patch multiprocessing SemLock is trying to override the method that doesnt exist

Suryansh Soni · Answer

There should be a different way to handle this instead of with patch multiprocessing