hey everyone! Has anyone used `redshift-spark` in ...
# questions
h
hey everyone! Has anyone used
redshift-spark
in kedro before? appreciate the help if someone can show me an example on how can this be done, specifically the driver used for redshift. Thanks in advance!
Fixed - in
spark.yml
(under conf) add the driver as the following line
spark.jars.packages: com.amazon.redshift:redshift-jdbc42:2.1.0.14
then use this setup to load redshift dataset
Copy code
type: spark.SparkJDBCDataSet
  url: jdbc:<redshift://HOST/DATABASE_NAME>
  credentials: db_credentials
  load_args:
    properties:
      driver: com.amazon.redshift.jdbc42.Driver
  save_args:
    properties:
      driver: com.amazon.redshift.jdbc42.Driver