https://kedro.org/ logo
#questions
Title
# questions
a

Abdullah Alsqoor

01/21/2024, 11:07 AM
Hello Team, How to deploy kedro project with pyspark to Google Cloud Dataproc (a spark cluster) is that even possible? and how to provoke it to run only one pipeline of the project not all pipelines Normally we pass conf.zip file and main file: main.py file This Dataproc runs on top of GCE Google Cloud Engine (not Kubernetes) so we cannot use docker
d

datajoely

01/21/2024, 12:35 PM
you can define your run arguments in YAML https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#configure-kedro-run-arguments and include this in the default run command
are you using
kedro package
?
d

datajoely

01/22/2024, 10:11 AM
oooooh
@marrrcin we need to promote those a little better, maybe even mention in the docs 🤔
🤔 1
m

marrrcin

01/22/2024, 10:13 AM
They are not on the most recent versions of Kedro though 😞
We’ve tested them on 18
2 Views