Hi all! Is there a parameter/option to have kedro do some cleanup after node executions in a pipeline? I have a rather largish pipeline made up of relatively small nodes. Each individual node when executed independently runs just fine, but the entire pipeline itself runs out of memory. Trying not to have to scale up the server to run the pipeline unless absolutely necessary

Hi @Daniel Montero, it might be a good idea to use hooks here, Kedro hooks allow you to execute custom logic at different stages of the pipeline execution, including after each node execution. You can check it out here, https://docs.kedro.org/en/stable/hooks/introduction.html.