Xinghong Fang
02/27/2023, 3:14 AM_multiprocessing.SemLock is not implemented
issue when launching the pipeline. A quick google search bring me to this issue https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda Looks like AWS Lambda's python runtime is missing /dev/shm
, which seems to be needed by the KedroSession
Has anyone successfully ran a kedro pipeline on AWS Lambda? Thanks in advance!Nok Lam Chan
02/27/2023, 7:49 AMXinghong Fang
02/27/2023, 8:35 AM0.18.4
, I can only get around the issue if I patch the multiprocessing.Lock
function (not sure if it is safe to do)from pathlib import Path
from unittest.mock import patch
def handle_event(event, context):
print("================ TEST PATCH ===============")
with patch("multiprocessing.Lock"):
from kedro.framework.cli.cli import KedroCLI
cli_collection = KedroCLI(project_path=Path.cwd())
cli_collection(args=["run"])
we are using the default sequential runner, but I still feel skeptical about patching out the lock.Nok Lam Chan
02/27/2023, 8:45 AMkedro
shouldn’t have any dependency on multiprocessing
unless ShelveStore
is specified in settings.py
Xinghong Fang
02/27/2023, 9:29 AMfrom pathlib import Path
def handle_event(event, context):
from kedro.framework.cli.cli import KedroCLI
cli_collection = KedroCLI(project_path=Path.cwd())
cli_collection(args=["run"])
This code now runs without error. Thanks!Nok Lam Chan
02/27/2023, 9:30 AM