2.3.9 Release Note

Bug fix

• [Bug][Avro] Fix avro miss convert short type to int (#8459)
• [Bug][CI] Fix all-connectors and update-modules run simultaneously (#8343)
• [Bug] [connector-file] When the data source field is less than the target (Hive) field，it will throw null pointer exception#8150 (#8200)
• [Bug][connectors-v2] fix mongodb bson convert exception (#8044)
• [Bug] [connectors-v2] The Hadoop Source/Sink fails with Unable to find valid Kerberos Ticket. (#7809)
• [Bug][Connector-v2] MongoDB CDC Set SeatunnelRow's tableId (#7935)
• [Bug][Core] Local mode cancel running JobStatusRunner (#7943)
• [bugfix][core] Fix the problem of incorrect association between metrics and nodes (#7799)
• [BUG] Fixed avro format support for storing null (#8424)
• [BugFix][Excel] Fix read formulas/number cell value of excel (#8316)
• [Bug][Kafka] kafka reads repeatedly (#8465)
• [Fix][API] Fix column length can not be long (#8039)
• [Fix][CI] Fix ci dependency check error (#8262)
• [Fix][CI] Fix CI error caused by set classloader-cache-mode to true (#7940)
• [Fix][CI] Fix schedule test be canceled (#8024)
• [Fix][CI] time-consuming module optimization (#8052)
• [Fix][Connecotr-V2] Fix clickhouse sink does not support composite primary key (#8021)
• [Fix][Connector-starrocks] Fix drop column bug for starrocks (#8216)
• [Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)
• [fix][connector-v2][connector-assert] Optimize Assert Sink verification method (#8356)
• [Fix][Connector-V2][connector-file-base-hadoop] Fixed HdfsFile source load the krb5_path configuration (#7870)
• [Fix][Connector-V2] Fix AbstractSingleSplitReader lock useless when do checkpoint (#7764)
• [Fix][Connector-V2] Fix cdc use default value when value is null (#7950)
• [Fix][Connector-V2] Fixed clickhouse connectors cannot stop under multiple parallelism (#7921)
• [Fix][Connector-V2] Fix file binary format sync convert directory to file (#7942)
• [Fix][Connector-V2] Fix hana type loss of precision (#7912)
• [Fix][Connector-V2] Fix hive krb5 path not work (#8228)
• [Fix][Connector-V2] Fix kafka format_error_handle_way not work (#7838)
• [Fix][connector-v2]Fix Paimon table connector Error log information. (#7873)
• [Fix][Connector-V2] Fix When reading Excel data, string and date type conversion errors (#7796)
• [Fix][Connector-V2][FTP] Fix FTP connector connection_mode is not effective (#7865)
• [Fix][Connector-V2]Oceanbase vector database is added as the source server (#7832)
• [fix][docker] update Dockerfile (#8421)
• [Fix][Docker] Workflow "push-docker" failed run with error "No space left on device" (#8339)
• [Fix][Doris] Fix catalog not closed (#8415)
• [fix] fix get job metrics get NPE (#8093)
• [Fix][Flink] Fix NPE when get null row from upstream transform (#8469)
• [Fix] [Flink/Spark] Fix transform stream choose wrong when not configure source_table_name (#7907)
• [Fix][Kafka] Fix in kafka streaming mode can not read incremental data (#7871)
• [fix][log]Fix log returns interface issue and adds test code (#8387)
• [fix] move docker build plugin into dist module (#8338)
• [Fix] Set all snappy dependency use one version (#8423)
• fix transforms-v2 nlpmodel typo (#8430)
• [Fix][Zeta] Fix apply resource again for another pipeline when restore one pipeline (#7965)
• [Fix][Zeta] Fix job client print error msg even job finished (#7981)
• [Fix][Zeta] Fix worker node metrics acquisition (#7862)
• [Fix][Zeta] Introduce SeaTunnel CompletableFuture to prevent ForkJoinPool thread shortage (#8445)
• [Fix][Zeta] Remove duplicate LIFECYCLE_WRITER_CLOSE events (#7984)
• [Fix][Zeta] Set AsyncLogger.ThreadNameStrategy=UNCACHED to avoid thread name cache (#8215)
• [Hotfix][CDC] Fix occasional database connection leak when read snapshot split (#7918)
• [Hotfix][Config] Fix configuration key sort disorder (#7893)
• [hotfix] [connector-cdc-oracle ] support read partition table (#8265)
• [hotfix][connector-elasticsearch-sink] Convert index to lowercase (#8429)
• [hotfix][connector-v2][hive]Fix hive3 e2e dependencies (#8045)
• [Hotfix][Connector-V2][kafka] fix kafka sink config exactly-once exception (#7857)
• [Hotfix][Core][Flink] SeaTunnel flink engine support application mode on yarn (#7762)
• [Hotfix][Docs]Correct errors in the FAQ doc (#8048)
• [Hotfix][E2E] Fix OracleCDCIT compile error (#8023)
• [HotFix]Fix Dead Link In ObsFile.md (#8251)
• [hotfix] fix exceptions caused by operator priority in connector-clickhouse when using sharding_key (#8162)
• [Hotfix] Fix redis sink NPE (#8171)
• [hotfix][gbase] Fix gbase driver link (#8110)
• [Hotfix][Transform][Sql] function concat_ws handle array type， avoid write [Ljava.lang.String. (#8369)
• [Hotfix][Transforms-v2] DynamicCompile Plugin compatibility fix (#8057)
• [Hotfix][Zeta] Fix the dependency conflict between the guava in hadoop-aws and hive-exec (#7986)
• [Hotfix][Zeta][Log] Fix zeta parallelStream log trace for job (#8456)

Docs

• [Chore] delete chinese desc in code (#8306)
• [Core][Flink] Resolve streaming source high CPU usage (#8354)
• [doc][connector][hive] Add hive chinese doc (#8169)
• [Doc][Format] Add protobuf doc (#7989)
• [Doc][hive] Rename result_table_name to plugin_output in hive doc (#8203)
• [Docs] [Elasticsearch] fix the doc about index name replacement (#8392)
• [Doc][Transform-v2] Update doc for Array function (#8419)
• [Fix][Doc] Correct hive-jdbc config useKerberos to use_kerberos (#7896)
• [Fix][Doc] fix backup-count config (#8147)
• [Fix][Doc] fix doc tip style and add document format specification (#8378)
• [Fix][Doc] Fixed unreachable links (#8131)
• [Fix][Doc] Fixing errors in the documentation and adding information (#8353)
• [Fix][Doc] Fix jetty doc error (#7883)
• [Fix][Doc] Fix key description error in redis zh doc (#7993)
• [Fix][Doc] Fix klaviyo doc 404 link (#7969)
• [Fix][Doc] Fix outdated plugin path in oracle documentation (#8403)
• [Fix][Doc] Fix rest-api missing content (#8384)
• [Fix][Doc] Fix wrong link in about/index.html (#8486)
• [Fix][Doc] Fix zh doc build error (#7882)
• [Fix][Doc] Modify error name SEATNUNNEL to SEATUNNEL (#8257)
• [Fix][Doc] Modify the example of enable-dynamic-port in the documentation. (#8331)
• [Fix][docs] fix wrong config in setup.md (#8483)
• [Fix][Doc] Update Elasticsearch Sink and Oracle CDC Source Sample Configuration (#8425)
• [Improve][doc] Add doc name check (#8140)
• [Improve][Doc] Add enable_header_write parameter in File doc (#8397)
• [Improve][Doc] Add file_filter_pattern example to doc (#7922)
• [Improve][Doc] Add some connector icons (#7944)
• [Improve][Doc] Add the way of configure jvm options in client (#8091)
• [Improve][Doc] Correcting typo errors in logging (#7874)
• [Improve][Doc] Correction to docker doc for get container ip (#7801)
• [Improve][Doc] Fix chinese doc miss grafana dashborad config (#7955)
• [Improve][Doc] Improve the connector-v2 develop doc (#8190)
• [Improve][Doc] Move seatunnel engine deployment doc (#8239)
• [Improve][Docs] Add driver reference to doris/starrocks docs (#8414)
• [Improve][Docs] Edit the description about startup.mode for Opengauss-CDC (#7815)
• [Improve][Doc] Update connector v2 contribute guide (#8375)
• [Improve][Doc] Updated telemetry doc (#7827)

Feature

• [Feature][API] Support timestamp with timezone offset (#8367)
• [Feature][CDC] Add 'schema-changes.enabled' options (#8252)
• [Feature][CDC] Add 'schema-changes.enabled' options (#8285)
• [Feature]Check Chinese comments in the code (#8319)
• [Feature][ClickhouseFile] Support add publicKey to identity (#8351)
• [Feature][Clickhouse] Support sink savemode (#8086)
• [Feature][connector-elasticsearch] elasticsearch support nested type (#8462)
• [Feature] [connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885)
• [Feature] [Connector-Redis] Redis connector support delete data (#7994)
• [Feature][Connectors] LocalFile Support reading gz (#8025)
• [Feature][Connector-V2] Add prometheus source and sink (#7265)
• [Feature][Connector-V2[Doris]Support sink ddl (#8250)
• [Feature][Connector-V2]Ftp file source support multiple table (#7795)
• [Feature][Connector-V2]Jdbc chunk split add snapshotSplitColumn config #7794 (#7840)
• [Feature][Connector-V2] Jdbc DB2 support upsert SQL (#7879)
• [Feature][Connector-V2] Piamon Sink supports changelog-procuder is lookup and full-compaction mode (#7834)
• [Feature][Connector-V2]Sftp file source support multiple table (#7824)
• [Feature][Connector-V2] Starrocks implements multi table sink (#8467)
• [Feature][Connector-V2] StarRocks-sink support schema evolution (#8082)
• [Feature][Connector-V2]Support Doris Fe Node HA (#8311)
• [Feature][Connector-V2] Support MaxCompute save mode (#8277)
• [Feature][connector-v2]Support opengauss jdbc connnector using opengauss driver. (#7622)
• [Feature][Connector-v2] Support S3 filesystem of paimon connector (#8036)
• [Feature][Connector-v2] Support schema evolution for Oracle connector (#7908)
• [Feature][Connector-v2] Support schema evolution for paimon sink (#8211)
• [Feature][Connector-V2] Support the jdbc connector for highgo db (#8282)
• [Feature][Connector-V2] Support use EasyExcel as read excel engine (#8064)
• [Feature][Connector-V2] Support write cdc changelog event into hudi sink (#7845)
• [Feature][Core] Rename result_table_name/source_table_name to plugin_input/plugin_output (#8072)
• [Feature][Core] Support cdc task ddl restore for zeta (#7463)
• [Feature][Core] Support read arrow data (#8137)
• [Feature][Doc]Paimon Souce changelog update (#8187)
• [Feature][Doris] Support multi-table source read (#7895)
• [Feature][E2E] Add E2E test case time analysis (#8028)
• [Feature][E2E] Add hive3 e2e test case (#8003)
• [Feature][E2E][Kerberos] Support for Kerberos in e2e (#8108)
• [Feature][Elasticsearch] Support sink ddl (#8412)
• [Feature][Engine UI] partial pages of the engine (#7602)
• [Feature][File] Support config null format for text file read (#8109)
• [Feature][Iceberg] Support custom delete sql for sink savemode (#8094)
• [Feature][Jdbc] Add Jdbc default dialect for all jdbc series database without dialect (#8132)
• [Feature][JDBC source] pg support char types (#8420)
• [Feature][Jdbc] Support sink ddl for postgresql (#8276)
• [Feature][K8S deploy] add helm chart (#8291)
• [Feature][Kafka] Add debezium_record_table_filter and fix error (#8391)
• [Feature][Mongodb-CDC] Support multi-table read (#8029)
• [Feature][MySQL-CDC] Support database/table wildcards scan read (#8323)
• [Feature][Redis] Flush data when the time reaches checkpoint.interval (#8198)
• [Feature][Redis] Flush data when the time reaches checkpoint.interval and update test case (#8308)
• [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)
• [Feature] [rest-api] Support Rest Api to upload file and submit task (#8442)
• [Feature][RestAPI] Support submit job with seatunnel style hocon format config (#8000)
• [Feature][Rest api] Use jetty to replace Hazelcast native rest api (#7647)
• [Feature][Transform-SQL]Support sql transform to generate UUID (#7881)
• [Feature][Transform] Sql transform support array and split function (#8016)
• [Feature][Transform] Support merge shading-tables to one table (#8360)
• [Feature][Transform] Support rename table/column (#8170)
• [Feature][transforms-v2] Support append only stream from cdc source (#7763)
• [Feature][transform] transform support explode (#7928)
• [Feature][Transform-v2] Add metadata transform (#7899)
• [Feature][Transform-V2] Spark support transform with multi-table (#8340)
• [Feature][Transform-V2] Support transform with multi-table (#7628)
• [Feature][Zeta] Add COS support for checkpoint storage (#7931)
• [Feature][Zeta]Add jar path precheck when job submit on master (#7976)
• [Feature][Zeta][Core] Support rest api get logs (#7818)
• [Feature] [Zeta] Optimize CoordinatorService ThreadPool Configuration to Prevent Potential OOM (#8241)
• [Feature][Zeta] Submit job scheduling support pending (#7693)
• [Feature][Zeta] Support delete logs regularly (#7787)
• [Imporve][ClickhouseFile] Directly connect to each shard node to obtain the corresponding path (#8449)
• [Imporve][ClickhouseFile] Improve clickhousefile generation parameter configuration (#8293)
• [Improve] Allow nullable for SeaTunnelRowType convert to Column (#8098)
• [Improve][api] Improve inheritance relationship of AlterTableColumnEvent (#7825)
• [Improve][API] Move AlterTableNameEvent parent (#7869)
• [improve][API] support metrics interface in rest v2 (#8312)
• [Improve][Api] Supports simultaneous config of conditions and optional options (#7788)
• [Improve][API] Unified tables_configs and table_list (#8100)
• [Improve][CI]add code specification check (#8209)
• [Improve][CI] Fix module runs CI repeatedly (#8284)
• [Improve][CI] Improve oracle schema evolution e2e case (#8224)
• [Improve][CI] Optimizing CI time cost (#7997)
• [Improve][CI]skip ui module, improve module dependent (#8225)
• [Improve][ClickhouseFile] Added attach sql log for better debugging (#8315)
• [Improve][ClickhouseFile] ClickhouseFile Connector's rsync transmission supports specifying users (#8236)
• [Improve][ClickhouseFile] Clickhouse supports option configuration when connecting to shard nodes (#8297)
• [Improve][ClickhouseFile Connector] Unified specifying clickhouse file generation path (#8302)
• [Improve][ClickhouseFile] Improve rsync log output (#8332)
• [Improve][connector-e2e][Hive]Optimize script and method names (#8249)
• [Improve][Connector-file-base] Improved file allocation algorithm for subtasks. (#8453)
• [Improve][Connector][Hive] skip temporary hidden directories (#8402)
• [Improve][Connector-V2] Add doris/starrocks create table with comment (#7847)
• [Improve][Connector-V2] Add pre-check for table enable cdc (#8152)
• [Improve][Connector-V2] Add pre-check starrocks version before exeucte alter table field name (#8237)
• [Improve][Connector-V2] Add some debug log when create dir in (S)FTP (#8286)
• [Improve][Connector-V2] Add table info when multi table sink error (#8163)
• [Improve][Connector-V2] Change File Read/WriteStrategy setSeaTunnelRowTypeInfo to setCatalogTable (#7829)
• [Improve][Connector-V2] Fix SqlServer cdc memory leak (#8083)
• [Improve][Connector-V2][FTP] delete duplicated code (#7915)
• [Improve][Connector-V2] Improve jdbc merge table from path and query when type is decimal (#7917)
• [Improve][Connector-V2] Improve schema evolution on column insert after for mysql-jdbc (#8017)
• [improve][connector-v2][kafka-connector] Add e2e code for Kafka Kerberos authentication (#8159)
• [Improve][Connector-V2] Optimized DefaultSaveModeHandlerconstruct method (#8179)
• [Improve][Connector-V2] Redis support custom key and value (#7888)
• [Improve][Connector-V2] Reduce the create times of iceberg sink writer (#8155)
• [Improve][Connector-V2] Reduce the request times of iceberg load table (#8149)
• [Improve][Connector-V2] RocketMQ Sink add message tag config (#7996)
• [Improve][Connector-V2] Save data savemode operation when schema savemode created new table (#8178)
• [Improve][Connector-v2] Support checkpoint in batch mode for paimon sink (#8333)
• [Improve][Connector-V2] Support read comment when jdbc dialect without catalog (#8196)
• [Improve][Connector-V2] The interface supports jdbc respects the target database field type (#8031)
• [Improve][Connector-v2] Use checkpointId as the commit's identifier instead of the hash for streaming write of paimon sink (#7835)
• [Improve][Connector-v2] Use regex to match filedName placeholders in jdbc sink (#8222)
• [Improve][Core] Add protobuf transform test case (#7914)
• [Improve][Core] Add test case to check deprecated result_table_name and source_table_name (#8153)
• [Improve][Core] Refactor common options of column/row (#7911)
• [Improve][dist]add shade check rule (#8136)
• [Improve][dist]add UT class name check (#8182)
• [Improve][E2E] improve doris initialize (#8446)
• [Improve][E2E] Improve Druid E2E Case (#8077)
• [Improve][E2E] improve kafka e2e (#8295)
• [Improve][E2E] improve oracle e2e (#8292)
• [Improve][E2E] modify the method of obtaining JobId (#7880)
• [Improve][Elasticsearch] Truncate the exception message body for request errors (#8263)
• [Improve][Example] Add log config for job mdc (#7876)
• [Improve][Example] Improve zeta local/cluster example (#7877)
• [Improve][Examples]Enable Flink Example WebUI (#8355)
• [Improve][Excel] Support read blank string & auto type-cast (#8111)
• [Improve][Fake] Improve memory usage when split size is large (#7821)
• [Improve][Formats] Support not primary-key table for debezium format (#7836)
• [Improve][Format] Support complex data type parse of debezium_json (#8330)
• [Improve][Iceberg] Filter catalog table primaryKey is empty (#8413)
• [Improve][Iceberg] Support table comment for catalog (#7936)
• [Improve] Improve AlterTableSchemaEventHandler (#8050)
• [improve] improve hazelcast joiner, lite node can't be election as master (#8261)
• [Improve][Jdbc] Improve ddl write validate (#8158)
• [Improve][Jdbc] Optimize index name conflicts when create table for postgresql (#7875)
• [Improve][Jdbc] Refactor ddl change (#8134)
• [Improve][Jdbc] Support nvarchar in dm (#8270)
• [Improve][Jdbc] Support postgresql inet type. (#7820)
• [Improve][Jdbc] Support save mode for the sink of jdbc-dm (#7814)
• [Improve][Kafka] Support custom topic for debezium compatible format (#8145)
• [Improve][POM] Bump the version of maven-shade-plugin to 3.4.1 (#8067)
• [Improve][PostgreSQL CDC]-PostgresSourceOptions description error (#7813)
• [Improve] Remove unused code (#8167)
• [Improve][Rest api]add verifyHtmlResponse test case (#7919)
• [Improve][rest api] improve rest-api doc (#7804)
• [Improve][RestApi] Unify the code logic of REST API V1 and V2 (#7920)
• [Improve][RestApi] Update context-path to / as default path (#7800)
• [Improve][Sls] Add sls sink connector、e2e、doc (#7830)
• [Improve][Test] Enable some spark test case on jdk 11 (#8238)
• [Improve][Transform] Auto mark column length with jsonpath transform (#7636)
• [Improve][Transform] gz support excel (#8181)
• [Improve][Transform] Optimize transform structure (#8071)
• [Improve][Transform] Rename sql transform table name from 'fake' to 'dual' (#8298)
• [Improve][transform-v2] Support dynamic types for array function (#8139)
• [Improve] update clickhouse connector, use factory to create source/sink (#7946)
• [Improve] Update snapshot version to 2.3.9 (#7841)
• [Improve] Update snapshot version to 2.3.9 (#7849)
• [Improve][Zeta] Fixed issues related to log deletion during ci (#8076)
• [Improve][Zeta] Jetty server enable dynamic port (#7903)
• [Improve][Zeta] Make sure the local mode CompletableFuture behavior same as server mode (#8476)
• [Improve][Zeta] Make the default value of classloader-cache-mode is true (#7938)
• [Improve][Zeta] Remove default jvm memory option (#8068)
• [Improve][Zeta] Split classloader in config parse phase (#8193)
• [Improvment][Build] Provide option to disable source jar creation. (#7680)
• [Zeta][Log] Fix trace parallel stream (#8206)