Hi Kedro team, I’m wondering if it is possible to ...
# questions
Hi Kedro team, I’m wondering if it is possible to run the kedro commands in combination with something like scalene or https://pypi.org/project/memory-profiler/ since both kedro and those packages are run with
python -m <module>
I guess this depends on what you want to do but you can use
to track memory profiling in Kedro, using Hooks: https://docs.kedro.org/en/stable/hooks/examples.html#add-memory-consumption-tracking
@Markus Sagen can you try
python -m scalene -m kedro run
for the record, I just tested
python -m scalene -m kedro run
and it gave a "program did not run long enough to profile", and
mprof run kedro run
gave a "process no longer exists". it would be good for us to document how to integrate Kedro with these profiling tools. @Markus Sagen would you like to open an issue in our repository?
Absolutely! Would be great to see such tools be supported with Kedro 👍
I also got the same results when running it locally initially, so great to see that it occurs on other machines as well
👍🏼 1