https://kedro.org/ logo
#questions
Title
# questions
f

FlorianGD

12/01/2023, 10:28 AM
Hi, I have a pipeline that uses a lot of memory. I profiled it with
memory_profiler
, and there is a huge memory usage, before the first node runs. This is critical for me because I am running it on a kubernetes pods, and if I use too much memory, the pod gets killed. Attached is the profile of a
dummy
pipeline that I created that prints a parameter and does nothing else. We can see that it uses nearly 200MiB of memory. I am not sure what I should look for to improve this memory usage. Is it the catalog? If so, how can I reduce it? Is it something else from kedro? Is it outside of kedro?
👀 1
For what it's worth, I am using
kedro==0.18.4
and
kedro-datasets==1.8.0
j

Juan Luis

12/01/2023, 10:51 AM
thanks a lot @FlorianGD, this is really helpful. could you open an issue with this same message and screenshot? and if you can share the "dummy" project, much better. we've been keeping an eye on the long startup times but never really looked at the memory (btw I'm supposed to be a maintainer of memory-profiler but Fabian has moved on and I've been neglecting the issue tracker... glad it still works)
😮 3
f

FlorianGD

12/01/2023, 10:53 AM
OK, will do!