Hi team, are there any best practices for optimizing spark code within Kedro pipelines? I have a large pipeline that executes at the last node due to lazy eval. I would like to look at execution plans, etc. Any suggestions? I suppose this would apply to Polars/Ibis/other similar frameworks.

You can analyze execution plan in Spark UI, once you run the job.

This is up to the execution engine - i.e. Polars / Spark is gonna to have completely different execution plan. Ibis is different in that catagory