Hello Team, How to deploy kedro project with pysp...
# questions
Hello Team, How to deploy kedro project with pyspark to Google Cloud Dataproc (a spark cluster) is that even possible? and how to provoke it to run only one pipeline of the project not all pipelines Normally we pass conf.zip file and main file: main.py file This Dataproc runs on top of GCE Google Cloud Engine (not Kubernetes) so we cannot use docker
you can define your run arguments in YAML https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#configure-kedro-run-arguments and include this in the default run command
are you using
kedro package
@marrrcin we need to promote those a little better, maybe even mention in the docs 🤔
🤔 1
They are not on the most recent versions of Kedro though 😞
We’ve tested them on 18