Hi gang, Is there any reasonable way to make async...
# questions
a
Hi gang, Is there any reasonable way to make asyncio and kedro work together? I have some external IO client that provides only async interface and I can't make it work from sync context, since there is already a running and managed asyncio loop somewhere in kedro. Do coroutines as kedro nodes make any sense?
h
Someone will reply to you shortly. In the meantime, this might help:
d
I don't think there is an async loop in Kedro • Unless you run
--async
in your runner CLI args • Kedro Viz is a FastAPI app which uses asyncio But the normal kedro workflow is synchronous, even when using ParallelRunner
d
Do coroutines as kedro nodes make any sense?
Not going to be very helpful here, but I randomly had somebody bring this up a few months back, that why couldn't Kedro just go all-in on
asyncio
and coroutines for execution. But I don't know that anybody has actually tried it. If you do, let us know? šŸ™‚
m
šŸ‘ 1
a
yea datajoely - we use --async with runner
yes @Deepyaman Datta I've also wondered that... It would make scheduling nodes executions and parallelism of data loading easy and out of the box
multiprocessing is really unreliable with speed ups and suitable only for some specific tasks that are parallel & indepdendent by nature, while threading has GIL problem
I am not super familiar with asyncio and am not aware if they managed to get around GIL via syncing with the await contracts - concurrency topic in python is very convoluted because of GIL