Hello Team, How to deploy kedro project with pyspark to Google Cloud Dataproc (a spark cluster) is that even possible? and how to provoke it to run only one pipeline of the project not all pipelines Normally we pass conf.zip file and main file: main.py file This Dataproc runs on top of GCE Google Cloud Engine (not Kubernetes) so we cannot use docker

you can define your run arguments in YAML https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#configure-kedro-run-arguments and include this in the default run command

You can take a look at https://github.com/getindata/kedro-starters/tree/develop/getindata_kedro_starters/pyspark-iris-running-on-gcp-dataproc-serverless 😉

oooooh

They are not on the most recent versions of Kedro though 😞