Redshift Data Source for Apache Spark

A library to load data into Spark SQL DataFrames from Amazon Redshift, and write them back to Redshift tables. Amazon S3 is used to efficiently transfer data in and out of Redshift, and JDBC is used to automatically trigger the appropriate COPY andUNLOAD commands on Redshift.

This library is more suited to ETL than interactive queries, since large amounts of data could be extracted to S3 for each query execution. If you plan to perform many queries against the same Redshift tables then we recommend saving the extracted data in a format such as Parquet.

• Installation
  • Snapshot builds
    
• Usage:
  • Data sources API: Scala, Python, SQL, R
  • Hadoop InputFormat
    
• Configuration
  • Authenticating to S3 and Redshift
  • Encryption
  • Parameters
    
• Additional configuration options
  • Configuring the maximum size of string columns
  • Setting a custom column type
  • Configuring column encoding
  • Setting descriptions on columns
    
• Transactional Guarantees
• Common problems and solutions
• S3 bucket and Redshift cluster are in different AWS regions
• Migration Guide
  

Installation

This library requires Apache Spark 2.0+ and Amazon Redshift 1.0.963+.

For version that works with Spark 1.x, please check for the 1.x branch.

You may use this library in your applications with the following dependency information:

Scala 2.10

groupId: com.databricksartifactId: spark-redshift_2.10version: 3.0.0-preview1

Scala 2.11

groupId: com.databricksartifactId: spark-redshift_2.11version: 3.0.0-preview1

You will also need to provide a JDBC driver that is compatible with Redshift. Amazon recommend that you use their driver, which is distributed as a JAR that is hosted on Amazon's website. This library has also been successfully tested using the Postgres JDBC driver.

Note on Hadoop versions: This library depends on spark-avro, which should automatically be downloaded because it is declared as a dependency. However, you may need to provide the corresponding avro-mapred dependency which matches your Hadoop distribution. In most deployments, however, this dependency will be automatically provided by your cluster's Spark assemblies and no additional action will be required.

Note on Amazon SDK dependency: This library declares a provided dependency on components of the AWS Java SDK. In most cases, these libraries will be provided by your deployment environment. However, if you get ClassNotFoundExceptions for Amazon SDK classes then you will need to add explicit dependencies on com.amazonaws.aws-java-sdk-core andcom.amazonaws.aws-java-sdk-s3 as part of your build / runtime configuration. See the comments inproject/SparkRedshiftBuild.scala for more details.

Snapshot builds

Master snapshot builds of this library are built using jitpack.io. In order to use these snapshots in your build, you'll need to add the JitPack repository to your build file.

• In Maven:

  <repositories>   <repository>     <id>jitpack.io</id>     <url>https://jitpack.io</url>   </repository></repositories>

  then

  <dependency>  <groupId>com.github.databricks</groupId>  <artifactId>spark-redshift_2.10</artifactId>  <!-- For Scala 2.11, use spark-redshift_2.11 instead -->  <version>master-SNAPSHOT</version></dependency>

• In SBT:

  resolvers += "jitpack" at "https://jitpack.io"

  then

  libraryDependencies += "com.github.databricks" %% "spark-redshift" % "master-SNAPSHOT"