Changes

New Features:

• Add experimental support for using CSV as an intermediate data format when writing back to Redshift (#73 / #288). This can significantly speed up large writes and also allows saving of tables whose column names are unsupported by Avro's strict schema validation rules (#84).

Performance enhancements:

• The read path is now based on Spark 2.0's new FileFormat-based data source, allowing it to benefit from performance improvements in FileScanRDD, such as automatic coalescing of partitions based on input size (#289).

Usability enhancements:

• Attempt to automatically detect when the Redshift cluster and S3 bucket are in different regions in order to provide more informative error messages (#285).
• Document this library's support for encrypted load / unload (#189).
• Document this library's security-related configurations, including an extensive discussion of the different communication channels and data sources and how each may be authenticated and encrypted (#291). Forwarding of Spark credentials to Redshift now requires explicit opt-in.

Bug fixes:

• Fix a NumberFormatException which occurred when reading the special floating-point values NaN and Infinity from Redshift (#261 / #269).
• Pass AWSCredentialProviders instead of AWSCredentials instances in order to avoid expiration of temporary AWS credentials between different steps of the read or write operation (#200 / #284).
• IAM instance profiles authentication no longer requires temporary STS keys (or regular AWS keys) to be explicitly acquired / supplied by user code (#173 / #274, #276 / #277).