Is there a guide on developing new integrations in the

kedro run

level? What I'm looking for is to develop an integration with memray but since memray is executed by

memray run my_script.py

I really have no clue on how to go about activating it when using

kedro run

Hi @Elior Cohen this is a good question! I don’t actually know the right answer to the CLI question (I’ve asked for help on this one) but perhaps you could use the python context manager to do this, possibly in a

after_context_created

hook https://bloomberg.github.io/memray/api.html

But how will I control this dynamically? I'm aiming for developing an extension where if you installed memray you would be able to execute

kedro run -memray ....

So I’m waiting on one of the engineers to right CLI way to do this - but my suggestion write the memray outputs to the

*.bin

file during the lifecycle of a run and then you could view the file like they do on the 3rd example here https://bloomberg.github.io/memray/examples/README.html

@Elior Cohen according to memray’s docs, you could do either way:

memray run [options] file.py [args]
memray run [options] -m module [args]

As each Kedro project is also a Python package with an entrypoint for the runs, you could effectively do:

memray run -m <your_kedro_project_package>

If you are doing that in a plugin, I would recommend that you create a new CLI command (e.g.

kedro memray

) which will pass on all arguments to the underlying Kedro run. You could read more about running Kedro project as a package here: https://kedro.readthedocs.io/en/stable/tutorial/package_a_project.html

mmm

Check `memray`’s function that is called on their entrypoint and uses it directly, and not as executable: https://github.com/bloomberg/memray/blob/main/setup.py#L324-L328

ill look at that

So the function is here: https://github.com/bloomberg/memray/blob/main/src/memray/commands/__init__.py#L119

I've done this before with a similar profiler. You should probably look at using the programmatic API (https://bloomberg.github.io/memray/api.html), not invoking as a script, and then you can set up in

before_pipeline_run

or something, and do any reporting in

after_pipeline_run

hook or something.

@Deepyaman Datta the hooks seems the correct way to go about it, I'll check that out

@Elior Cohen if you do get something working I’d love to share it with the community via docs or a plugin - so let us know how you get on and shout if you need to discuss anything further 🙂

Will do

so you can actually provide your own run command which can registerd with a plugin https://kedro.readthedocs.io/en/stable/development/commands_reference.html#customise-or-override-project-specific-kedro-commands so if you wanted to create a

kedro-memray

plugin with it’s own run command and

before|after

hooks that would work neatly?

thx, I'll check it out

This is a plugin a wrote a while back if you want to see me do some of the same things

cool I'll check it out

I think (in kinda an ugly way) I add a

--simple

option to the run command here https://github.com/datajoely/kedro-rich/blob/d92651a7579c048e265d8b07fc4e45ab4ed66b87/kedro_rich/rich_cli.py#L64