https://kedro.org/ logo
Powered by
#questions
Title
# questions
h

Hadeel Mustafa

05/25/2023, 4:25 PM
hey everyone! Has anyone used 
redshift-spark
in kedro before? appreciate the help if someone can show me an example on how can this be done, specifically the driver used for redshift. Thanks in advance!
Fixed - in 
spark.yml
(under conf) add the driver as the following line 
spark.jars.packages: com.amazon.redshift:redshift-jdbc42:2.1.0.14
then use this setup to load redshift dataset
Copy code
type: spark.SparkJDBCDataSet
  url: jdbc:<redshift://HOST/DATABASE_NAME>
  credentials: db_credentials
  load_args:
    properties:
      driver: com.amazon.redshift.jdbc42.Driver
  save_args:
    properties:
      driver: com.amazon.redshift.jdbc42.Driver
Powered by