Hadeel Mustafa
05/25/2023, 4:25 PMredshift-spark
in kedro before? appreciate the help if someone can show me an example on how can this be done, specifically the driver used for redshift.
Thanks in advance!spark.yml
(under conf) add the driver as the following line
spark.jars.packages: com.amazon.redshift:redshift-jdbc42:2.1.0.14
then use this setup to load redshift dataset
type: spark.SparkJDBCDataSet
url: jdbc:<redshift://HOST/DATABASE_NAME>
credentials: db_credentials
load_args:
properties:
driver: com.amazon.redshift.jdbc42.Driver
save_args:
properties:
driver: com.amazon.redshift.jdbc42.Driver