release v2.7.0

Upgrade-time behavior changes

In v2.7.0, following the refactoring of sink-into-table, the system will automatically migrate fragments of all legacy tables to a unified pattern.

• Migration process: Upon upgrading and starting the cluster, a background task will automatically initiate the migration for all tables. You can see meta logs like:
  • "migrated table fragments table_id={table_id}" indicate a successful migration.
  • "failed to migrate legacy table fragments" indicate an error on migration.
• Impact: Migration errors triggering these logs will not affect existing jobs or system availability. The system will continuously retry the migration.
  • If the migration fails but does not impact your usage, please report the issue via Slack or GitHub issue and wait for bug fix in the next version.
  • To disable automatic migration, set enable_legacy_table_migration = false in the [meta] section of your toml file (requires meta node restart).
• Limitations & Workaround: You cannot create a new sink-into-table or drop an existing one on a table that has not completed migration. If the background migration is stuck or taking too long and you urgently need to perform these DDL operations, you can force a migration by running:

  ALTER TABLE {table_name} ADD COLUMN {col_name} int;
  ALTER TABLE {table_name} DROP COLUMN {col_name};
  

  After this, you can use it as a normal table.

SQL features

• SQL commands:
  • Supports vector index Top-N lookups in streaming queries, enabling MVs and sinks to enrich each row with nearest-neighbor results. https://github.com/risingwavelabs/risingwave/pull/23288
  • Supports ALTER FRAGMENT ... SET PARALLELISM command to alter the parallelism of streaming fragments. https://github.com/risingwavelabs/risingwave/pull/23523
  • Reduces reserved keywords in EXPLAIN options (e.g., BACKFILL, TRACE) by parsing them as identifiers.https://github.com/risingwavelabs/risingwave/pull/23208
  • Adds schema name to name column in the output of SHOW [TABLE|VIEW|SOURCE|SINK|...] commands. https://github.com/risingwavelabs/risingwave/pull/23119
  • When DROP/CANCEL takes longer than slow_ddl_notification_secs (30s), you can run RECOVER for immediate effect. https://github.com/risingwavelabs/risingwave/pull/23613
  • Supports ALTER TABLE ... ADD COLUMN command on tables with incoming sinks.https://github.com/risingwavelabs/risingwave/pull/23766
  • Supports querying vector index in append-only streaming. https://github.com/risingwavelabs/risingwave/pull/23288
• SQL functions & operators:
  • Supports parsing STRUCT< > as empty struct type. https://github.com/risingwavelabs/risingwave/pull/23099
  • Supports rw_cluster_id()  function to return the cluster's unique identifier as a VARCHAR. https://github.com/risingwavelabs/risingwave/pull/23260
• System catalog:
  • Adds system catalog rw_recovery_info to display the database-level recovery state. https://github.com/risingwavelabs/risingwave/pull/23728
  • Adds column parallelism_policy in system catalog rw_fragments. https://github.com/risingwavelabs/risingwave/pull/23524

Connectors

• Breaking change: When decoding Protobuf messages, RisingWave returns NULL for missing optional fields and default values for all other fields. https://github.com/risingwavelabs/risingwave/pull/23120
• Supports using CREATE SOURCE directly for UPSERT-format sources requiring a primary key, without creating a table first. https://github.com/risingwavelabs/risingwave/pull/22856
• File sink now uses UUIDs instead of executor_id in output file names. https://github.com/risingwavelabs/risingwave/pull/23981
• Supports on-demand manual refresh for Iceberg batch tables with position deletes and equity deletes. https://github.com/risingwavelabs/risingwave/pull/23527
• Technical preview: Supports periodic automatic refresh for batch tables with persistent state tracking and improved observability. https://github.com/risingwavelabs/risingwave/pull/23737
• Technical preview: Supports files-with-delete Iceberg compaction and introduces additional compaction parameters. https://github.com/risingwavelabs/risingwave/pull/23660, https://github.com/risingwavelabs/risingwave/pull/23809
• Supports Azure Data Lake Storage Gen2 for Iceberg sink when using REST catalog. https://github.com/risingwavelabs/risingwave/pull/23350
• Supports configuring S3 and Glue with independent credentials for Iceberg connector when using Glue catalog. https://github.com/risingwavelabs/risingwave/pull/23775
• Support assuming IAM roles for S3 and Glue when enable_config_load is enabled for Iceberg connector when using Glue catalog. https://github.com/risingwavelabs/risingwave/pull/23788
• Supports key-pair authentication using RSA private keys for Snowflake sink. https://github.com/risingwavelabs/risingwave/pull/23525
• Supports JDBC sink connector for SQL Server. https://github.com/risingwavelabs/risingwave/pull/23416
• Changes the default value of commit_checkpoint_interval from 10 to 60 for Iceberg sink. https://github.com/risingwavelabs/risingwave/pull/23552
• Supports is_exactly_once for Iceberg sink by default. https://github.com/risingwavelabs/risingwave/pull/23553
• Supports vended credentials for Iceberg connector when using REST catalog. https://github.com/risingwavelabs/risingwave/pull/23567
• Supports configuring postgres.is.aws.rds to specify if the upstream PostgreSQL is hosted on AWS RDS. https://github.com/risingwavelabs/risingwave/pull/23420
• Supports using Base64-encoded JSON strings for BigQuery credentials. https://github.com/risingwavelabs/risingwave/pull/23100
• Supports TLS connections to Redis sink using rediss:// protocol. https://github.com/risingwavelabs/risingwave/pull/23223

Installation and deployment

• Counts the CPU cores of all nodes (instead of only compute nodes) in the cluster against the cpu_core_limit specified in the license key. https://github.com/risingwavelabs/risingwave/pull/23390
• Changes cpu_core_limit in the license key to rwu_limit to limit both CPU cores and total cluster memory. https://github.com/risingwavelabs/risingwave/pull/23403
• Breaking change: Requires the correct aud claim for authentication. https://github.com/risingwavelabs/risingwave/pull/23155
• Supports LDAP authentication to verify users against an external LDAP directory server. https://github.com/risingwavelabs/risingwave/pull/23150

Cluster configuration changes

• Adds session variable enable_index_selection to control whether to enable index selection for both streaming and batch queries. https://github.com/risingwavelabs/risingwave/pull/23092
• Technical preview: Adds session variable enable_locality_backfill to control locality backfilling, which extends index-based optimizations to the entire backfill pipeline. https://github.com/risingwavelabs/risingwave/pull/23275
• Adds configuration option RW_SSL_ENFORCE to control SSL enforcement for security. https://github.com/risingwavelabs/risingwave/pull/23306

Assets

• Run this version from Docker: docker run -it --pull=always -p 4566:4566 -p 5691:5691 risingwavelabs/risingwave:v2.7.0-standalone single_node
• Prebuilt all-in-one library for Linux
• Source code (zip)
• Source code (tar.gz)
• risectl - a CLI tool for managing and accessing RisingWave clusters