Hi, I have a pipeline that uses a lot of memory. I...
# questions
Hi, I have a pipeline that uses a lot of memory. I profiled it with
, and there is a huge memory usage, before the first node runs. This is critical for me because I am running it on a kubernetes pods, and if I use too much memory, the pod gets killed. Attached is the profile of a
pipeline that I created that prints a parameter and does nothing else. We can see that it uses nearly 200MiB of memory. I am not sure what I should look for to improve this memory usage. Is it the catalog? If so, how can I reduce it? Is it something else from kedro? Is it outside of kedro?
👀 1
For what it's worth, I am using
thanks a lot @FlorianGD, this is really helpful. could you open an issue with this same message and screenshot? and if you can share the "dummy" project, much better. we've been keeping an eye on the long startup times but never really looked at the memory (btw I'm supposed to be a maintainer of memory-profiler but Fabian has moved on and I've been neglecting the issue tracker... glad it still works)
😮 3
OK, will do!