Release 5.5.2-SNAPSHOT

API Changes

New Features

Kernel: Add firebird SQL parser module and database type #33773

Enhancements

Kernel: Add arguments not null check when creating RouteUnit - #33382
Kernel: Add index columns not empty judgement for IndexColumnTokenGenerator - #33384
Kernel: Add binding to owner table - #33533
DistSQL: Check inline expression when create sharding table rule with inline sharding algorithm - #33735
SQL Parser: Support parsing Doris BITXOR - #33258
SQL Parser: Support parsing Doris INSTR - #33289
SQL Parser: Support parsing Doris STRRIGHT - #33393
SQL Parser: Support parsing MySQL by adding non-reserved keywords in BaseRule.g4 file according to MySQL 8.4 doc - #33846
Storage: Support setting hive_conf_list, hive_var_list and sess_var_list for jdbcURL when connecting to HiveServer2 - #33749
Storage: Support connecting to HiveServer2 through database connection pools other than HikariCP - #33762
Storage: Partial support for connecting to embedded ClickHouse chDB - #33786
SQL Federation: Upgrade calcite version to 1.38.0 and update all license info in LICENSE file - #33279
Transaction: Bump the minimum Seata Client version for Seata AT integration to 2.2.0 - #33872
JDBC: Add show database name for JDBC when execute SHOW COMPUTE NODES - #33437
Kernel: Add binding to owner table - #33533
SQL Parser: Support parsing Doris EXTRACT_URL_PARAMETER - #33571
JDBC: Support ZonedDateTime on ResultSet - #33660
Proxy: Add query parameters and check for MySQL kill processId - #33274
Proxy: Support table not exist exception for PostgreSQL proxy - #33885
Proxy Native: Change the Base Docker Image of ShardingSphere Proxy Native - #33263
Proxy Native: Support connecting to HiveServer2 with ZooKeeper Service Discovery enabled in GraalVM Native Image - #33768
Proxy Native: Support local transactions of ClickHouse under GraalVM Native Image - #33801
Sharding: Support GroupConcat function for aggregating multiple shards in MySQL, OpenGauss, Doris - #33808
Proxy Native: Support Seata AT integration under Proxy Native in GraalVM Native Image - #33889
Agent: Simplify the use of Agent's Docker Image - #33356
Metadata: Add load-table-metadata-batch-size props to concurrent load table metadata - #34009
SQL Binder: Add sql bind logic for create table statement - #34074
SQL Binder: Support create index statement sql bind - #34112

Bug Fixes

Mode: Fixes JDBCRepository improper handling of H2-database in memory mode - #33281
Mode: Fixes duplicate column names added when index changed in DDL - #33982
SQL Binder: Fixes table does not exist exception when use HintManager#setDatabaseName to transparent - #33370
SQL Parser: Fixes LiteralExpressionSegment cast exception in SQL parser - #33332
SQL Parser: Fixes PostgreSQL and openGauss time extract function parse week and quarter error - #33564
SQL Binder: Use Multimap and CaseInsensitiveString to replace CaseInsensitiveMap for supporting MySQL multi table join with same table alias - #33303
SQL Binder: Fixes the combine statement cannot find the outer table when bind - #33357
SQL Binder: Fixes SQL performance issues caused by repeated subquery fetches - #33361
SQL Parser: Fixes MySQL parse zone unreserved keyword error - #33720
SQL Parser: Fixes MySQL range parse error when use table owner - #33874
Storage: Fixes cannot connect to HiveServer2 using remote Hive Metastore Server - #33837
Proxy: Fixes BatchUpdateException when execute INSERT INTO ON DUPLICATE KEY UPDATE in proxy adapter - #33796
Sharding: Remove ShardingRouteAlgorithmException check logic temporarily to support different actual table name configuration - #33367
Sharding: Fixes SQL COUNT with GROUP BY to prevent incorrect row returns - #33380
Sharding: Fixes avg, sum, min, max function return empty data when no query result return - #33449
Encrypt: Fixes merge exception without encrypt rule in database - #33708
SQL Binder: Fixes the expression segment cannot find the outer table when binding - #34015
Proxy: Fixes "ALL PRIVILEGES ON DB.*" is not recognized during SELECT privilege verification for MySQL - #34037
Encrypt: Use sql bind info in EncryptInsertPredicateColumnTokenGenerator to avoid wrong column table mapping - #34110

Change Logs

MILESTONE

Release 5.5.1

API Changes

Authority: Mark privilege provider ALL_PERMITTED as deprecated and will be removed in future
DistSQL: Remove optional param usageCount from show storage units
Readwrite-splitting: Change dataSources to dataSourceGroups for YAML

New Features

Kernel: Add new doris, hive and presto sql parser module and database type

Enhancements

Kernel: Support sql hint extract when sql contains dbeaver hint comment
Kernel: Add extract combine left select in extractFromSelectStatementWithoutProjection method
Metadata: Collect table type for PostgreSQL
Metadata: Add assisted node to create and delete databases
Mode: Change shadow、sharding's algorithms node path to shadow_algorithms and sharding_algorithms node path
DistSQL: Check privilege when registering or altering storage unit
DistSQL: Check duplicate actual data nodes when creating or altering sharding table rule
DistSQL: Add like support for show storage units
DistSQL: Rollback if import database configuration failed
DistSQL: add table type to result set of show logical tables
Proxy: Make the results of show tables in order
Proxy: Trigger metadata collection when creating and deleting database and table
Proxy: Optimize Agent to support collecting metrics data from multiple data sources when used with Driver
SQL Parser: Support PostgreSQL COLLATION and CONSTRAINT keywords
SQL Parser: Support MySQL change replication with no server ids
SQL Parser: Support MySQL row and column aliases with ON DUPLICATE KEY UPDATE
SQL Parser: Support MySQL CALL SQL with mysql prefix
SQL Parser: Support MySQL create loadable function returns int/dec
SQL Parser: Support parsing MySQL CREATE TABLESPACE/TABLE in Mysql-8
SQL Parser: Support parsing MySQL CREATE INDEX with ENGINE_ATTRIBUTE
SQL Parser: Support parsing MySQL LOAD DATA with @ variable
SQL Parser: Support parsing MySQL START REPLICA statement
SQL Parser: Support parsing MySQL json table function
SQL Parser: Enhance mysql json function visit statement result
SQL Parser: Enhance Oracle update statement as alias parse and add test case
SQL Parser: Add MySQL column not null SQL parsing
SQL Parser: Add MySQL charset sql parsing
SQL Parser: Support Oracle column length unit type SQL parsing
SQL Parser: Support PostgreSQL, openGauss function table and update from parse
SQL Parser: Support cte with postgres and openGauss
SQL Parser: Optimize the timezone support of PostgreSQLDateValueParser
SQL Parser: Support PostgreSQL limit and offset as null
SQL Federation: Support SQL federation bit_count function for MySQL
SQL Federation: Improve atan and anan2 sql function for MySQL
SQL Federation: Support federated query mysql bin function
SQL Federation: Support for federated query NOT operator
Transaction: Support for switching transaction types
Use same transaction type in one transaction in jdbc adapter
Proxy Native: Add more graalvm reachability metadata for caffeine cache
Sharding: Revise all local index for sharding table and add object uniqueness level spi to control index token generator
Sharding: Add NullsOrderType.LOW and NullsOrderType.HIGH to handle NULL order by in sharding feature
Add inline sharding algorithms match actual data nodes check
Encrypt: Add unsupported check for combine statement with encrypt columns
Encrypt: Support select distinct(column) encrypt rewrite and refactor SubstitutableColumnNameToken build logic
Encrypt: Support like concat nested concat statement rewrite with encrypt feature
Pipeline: Add SHARDING_TOTAL_COUNT impl of JobExecutorServiceHandler to improve CPU core requirement
Pipeline: Support page query for inventory dumper and data consistency streaming query
Pipeline: Use case-insensitive identifiers to enhance the table metadata loader
Pipeline: Support primary key columns ordering for standard pipeline table metadata loader

Bug Fixes

Kernel: Replace ThreadLocal with TransmittableThreadLocal in HintManager to support cross Thread usage
Kernel: Use ConnectionPropertiesParser spi to parse jdbcUrl to solve oracle url parse error
Kernel: Fixes column count assert exception caused by postgresql system table judge
Metadata: Keep in-used storage node when unregister storage unit
Metadata: Fixes the generated key column name case insensitivity error
SQL Parser: Support PostgreSQL do nothing conflict action
SQL Parser: Fix oracle nvl function and interval hour parse error
DistSQL: Fixes load single table error after creating logical data source
DistSQL: Fixes set default single table storage unit to random failed
DistSQL: Fixes set default single table storage unit to logical data source failed
DistSQL: Fixes NPE when import metadata
Fixes the problem of missing storage unit when registering storage unit
SQL Binder: Add TableAvailable interface for CloseStatementContext, MoveStatementContext, FetchStatementContext and fix SQL rewrite test case
SQL Federation: Fixes federated query LocalDateTime conversion
SQL Federation: Fixes push down SQL execute error when sql contains chinese character with SQL federation
SQL Federation: Fixes federation query binary type data query
SQL Federation: Fixes null result in federated query for a single projected column
SQL Federation: Fixes SQL federation unknown type exception caused by calcite wrong result type with bigint
Transaction: Fixes savepoint manager not cleaned up in distributed transactions
Transaction: Fixes PostgreSQL rollback only
Transaction: Fixes transaction context not cleaned up when xa transaction is committed
Transaction: Fixes setSavepoint method invocation not cleaning up
Transaction: Fixes the issue that cursor is not rewritten
Proxy: Support binary type bool value in PostgreSQL
Proxy: Support array type prepared param in PostgreSQL
Proxy: Support binary protocol value for text array in PostgreSQL
Proxy: Fixes duplicate results when querying information_schema.SCHEMATA
Proxy: Fixes incorrect results for querying information_schema.SCHEMATA
Proxy: Fixes NPE when execute show table status
Proxy: Fixes no database selected exception occurs when show tables from database
Proxy: Fixes the error that the process does not exit after proxy startup fail
Proxy: Fixes the error that the persisted system metadata was not cleared after the database was deleted in the PostgreSQL scenario
Proxy: Fixes no database selected exception in the query SQL
JDBC: Fixes the NPE when it does not contain a specified logic database in Driver
Sharding: Fixes Sharding column not tracked through aliases
Sharding: Fixes alter view exception when config sharding rule and binding table rule
Sharding: Fixes is need accumulate logic
Encrypt: Fixes show create table wrong result with encrypt when data type contains float
Encrypt: Add insert select rewrite for encrypt
Encrypt: Fixes the issue where updating a non-encrypted table and using a subquery on an encrypted table
Pipeline: Fixes default data source lost when create migration job
Pipeline: InventoryTaskSplitter compatible with BigInteger primary key
Resolve the issue where "zip file closed" in versions prior to SpringBoot 2.3

Change Log

MILESTONE

Release 5.5.0

API Changes

Kernel: Remove the sqlCommentParseEnabled configuration in SQL Parser Rule to reduce code complexity
SQL Parser: Consider removing sqlCommentParseEnabled config in SQLParser rule
Federation: Add allQueryUseSQLFederation config for sql federation
Proxy: Remove the outdated schemaName configuration of the Proxy

New Features

DistSQL: New syntax for query plugin (SPI) implementation
DistSQL: New syntax for managing SQL_TRANSLATOR rule

Enhancements

Authority: Add isSuper option for user
SQL Parser: Add EOF to throw exception when parse distsql rollback migration statement
SQL Parser: Support more plsql statement parse and add plsql parse assert logic
SQL Parser: Support parse index hint
SQL Parser: Support mysql intersect combine operation sql parse
SQL Parser: Support parse chinese white space for oracle
SQL Parser: Fix mysql TimeStampDiff function parse
SQL Parser: Fix sqlServer unqualified shorthand parsing
SQL Parser: Support sqlServer SEARCH unreserved word parsing
SQL Parser: Add sql server MEMBER unreserved word
DistSQL: Use JSON format to output props in RQL
DistSQL: Optimize REFRESH DATABASE METADATA logic
DistSQL: SHOW COMPUTE NODES supports JDBC nodes
DistSQL: Optimize syntax of REFRESH DATABASE METADATA
DistSQL: Optimize the output of SHOW TABLE METADATA
SQL Binder: Add ParameterMarkerSegmentBinder logic for Oracle MergeStatementBinder
SQL Federation: Support mysql cross join statement for sql federation
Transaction: Add advice message in XATransactionCheckPrivilegeFailedException
Proxy: Add MySQL precompiled parameter verification to avoid turning on rewriteBatchedStatements=true when BenchmarkSQL connects to Proxy, causing an ArrayIndexOutOfBoundsException exception when the Proxy parameterCount exceeds 65535.
Proxy: Fix mysql client multi statements option in protocol
Sharding: Validate duplicate sharding actual data nodes
Sharding: Support null condition value routing
Pipeline: Improve CDC stability and performance
Pipeline: MemoryPipelineChannel supports configurable zero queue size for less memory consumption
Pipeline: Show dedicated error when mode type is not Cluster
Pipeline: Ignore error message in status DistSQL result on job cancelling

Bug Fixes

MetaData: Fixes database system schema is not built when create database
Metadata: Fixes NPE of system schema builder rule
Governance: Fixes thread blocking problem when create logic database for Etcd register center
Governance：Fixes register storage units and create feature rules failure when use Standalone mode
SQL Parser: Fixes PostgreSQL NPE when parse columnRef
SQL Parser: Fixes npe cause by parse MySQL select window statement
SQL Federation: Fixes Object 'DUAL' not found exception when execute select 1 from dual with sql federation
Transaction: Fixes xa auto commit in executeQuery
Proxy: Restore original databaseName in connectionSession after unicast
Proxy: Fixes show tables can be executed without use database
Proxy: Fixes the incorrect current database after unicast routing
Proxy: Fixes the problem that show tables can be executed before use database
Proxy: Fixes multi statements with specified database name
Sharding: Fixes routing error when joining tables in uppercase
Sharding: Fixes drop sharding table exception when table name is uppercase
Sharding: Fixes generated key with upper case column name
Readwrite-splitting: Fix check exception when using shadow data source
Pipeline: Fixes commit/rollback migration job doesn't drop related consistency check job when check job is not completed
Pipeline: Fixes show consistency check status stop_time display

Change Log

MILESTONE

Release 5.4.1

New Features

Metadata: Standalone mode adapts to metadata new structure
Governance: Governance supports register instance level data source
Proxy: Supports dbcp and c3p0 connection pools

Enhancements

Mode: Improve Standalone mode JDBC type impl reset data on initialization
JDBC: Move jdbc core META-INF/services/java.sql.Driver from test to main
Encrypt: Add duplicate name check for derived columns and logical columns
Encrypt: Deny DDL for cipher columns in Proxy
Encrypt: Add the default type for derived columns to varchar(4000)
Pipeline: Isolate ShardingSphereDataSource Standalone repository in pipeline
Pipeline: Disable system-schema-metadata-enabled in pipeline
Pipeline: Add algorithm columns in SHOW MIGRATION CHECK ALGORITHMS DistSQL result
Pipeline: Add type_alias column in SHOW MIGRATION CHECK ALGORITHMS DistSQL result

Bug Fixes

Single Table: Fixes not switched active version when CREATE/DROP table modifies the Single rule configuration
JDBC: Fixes JDBC memory leak with MySQL in the 5.4.0 version
Pipeline: Fixes get inventory position not correctly on breakpoint resuming when table names are similar
Pipeline: Fixes CDC importer not start on breakpoint resuming when first inventory task is finished

Change Log

MILESTONE

Release 5.4.0

API Changes

Metadata: Change sharding broadcast tables to global broadcast tables
JDBC: Remove exclamation mark (!) for global rules
DistSQL: Simplify keywords ASSISTED_QUERY, LIKE_QUERY in encrypt DistSQL
DistSQL: Optimize SQL_PARSER rule syntax
Encrypt: Adjust encryption yaml API to distinguish between encrypt, like, and assisted query configurations
Encrypt: Remove plain column and queryWithCipherColumn configuration in encrypt feature
Readwrite-splitting: Refactor read/write splitting api
Proxy: Remove property proxy-instance-type configuration
Proxy: Remove property proxy-backend-executor-suitable
Proxy: Remove property proxy-mysql-default-version
Scaling: Refactor commit rollback streaming to drop streaming
Sharding: Merge ShardingCacheRule into ShardingRule (Experimental)

New Features

DistSQL: New syntax to manage SQL_FEDERATION rule
Proxy: Support Unix Domain Socket

Enhancements

Scaling: CDC supports pure incremental mode
Scaling: CDC supports exporting data by transaction
Scaling: CDC supports MySQL and PostgreSQL
Scaling: CDC supports single table
Scaling: CDC supports all openGauss data types
Scaling: CDC supports replication reconnection
Scaling: Remove DataConsistencyCalculateAlgorithmChooser
Scaling: Improve performance of integer unique key table inventory data splitting
Scaling: Adjust process configuration default value to reduce resource consumption
Scaling: Auto refresh table metadata for migration
Scaling: Compatible with openGauss existing replication slot reuse when database not existing
Scaling: Show data consistency check status result should be empty when it's not completed
Scaling: Enable concurrent CRC32 match on source and target
Scaling: Pipeline job compatible with sharding rule audit strategy
Metadata: Refactor metadata persistence structure
Metadata: Optimize the process of loading single table metadata
Metadata: Support MySQL/PostgreSQL/openGauss system tables empty query
DistSQL: Add support for transactionalReadQueryStrategy for read/write splitting rule
DistSQL: Enhanced algorithm properties check
Transaction: Add privilege check
Transaction: Remove the TransactionTypeHolder and only create the current transaction manager
Parser: Support MySQL LOAD DATA and LOAD XML statement with single table or broadcast table
Parser: Improve the parsing support of high-priority SQL statements in the test results of the MySQL test program
Parser: Oracle dialect parser now supports Chinese comma
Encrypt: Support query of encrypt column in projection subquery when use encrypt feature
Kernel: Adds table existence metadata check for INSERT, DELETE, UPDATE and SELECT statements
JDBC: Implement batch execution for ShardingSphereStatement
Proxy: Frontend supports SSL/TLS
Proxy: Support Flush message for PostgreSQL/openGauss Proxy
Proxy: Support data type bit, bool for PostgreSQL Proxy

Bug Fixes

Scaling: Fix pipeline job failure status persistence and usage
Scaling: Fix CDC DELETE event Record.beforeList is null
Scaling: Fix openGauss mppdb decoding plugin single quote issue
Scaling: Fix execute engine not closed after job stopping
Scaling: Fix stop job before task starting
Metadata: Fix case sensitive issue when loading schema meta data with H2 database
Metadata: Fix "object not found" exception when config PostgreSQL/openGauss schema name as database name
DistSQL: Fix wrong result of check_table_metadata_enabled when execute SHOW DIST VARIABLE
Encrypt: Fix SQL rewrite exception when use PostgreSQL/openGauss encrypt like feature
Sharding: Support null sharding condition pass to sharding algorithm to allow user control null value route
Parser: Support BETWEEN AND expression parsing in MySQL Projection
Mask: Fix wrong mask result when config same value of from-x and to-y with KEEP_FROM_X_TO_Y
Infra: Fix ClassNotFoundException may occur when missing pgjdbc
Proxy: Fix MySQL packet out of order when client sending pipelining requests

Change Log

MILESTONE

5.3.2

API Changes

Proxy: Add property system-log-level, support dynamic change of log level by DistSQL.
DistSQL: Remove Hint-related DistSQL, users can use SQL Hint instead

New Features

Scaling: Support any type of column unique key table

Enhancements

Scaling: Use stream query for inventory dump and data consistency check
Scaling: Compatible with VARBINARY column type in MySQL binlog parsing
Scaling: Refactor AbstractSimplePipelineJob.execute to blocking
Scaling: Improve table records count calculation
Scaling: Support proxy sharding rule absent for migration job
Scaling: Add useServerPrepStmts=false for pipeline job on MySQL
Scaling: Improve datetime/time parsing for MySQL binlog
Scaling: Add global status for prepare stage
Scaling: Add netTimeoutForStreamingResults for pipeline job on MySQL
Authority: Support specifying password authentication method
Authority: Add md5 authentication support for openGauss protocol
Agent: Add more metrics for JDBC
Kernel: ShardingSphere Driver configuration supports Apollo
Kernel: Adjust SKIP_ENCRYPT_REWRITE SQL Hint to SKIP_SQL_REWRITE to support more scenarios
Kernel: Support openGauss EXPLAIN PERFORMANCE
Encrypt: Like supports concat function

Bug Fixes

Scaling: Fix unicode char and special char decoding for PostgreSQL incremental task
Scaling: Fix Migration not support PostgreSQL json type
DistSQL: CREATE SHARDING TABLE RULE supports NONE strategy
Kernel: Fix use system database error when data sources are empty
Kernel: Fix set worker-id does not take effect with Standalone mode
Kernel: Clear storage node information when delete readwrite-splitting and database discovery rules
Kernel: Fix the abnormal problem of Column index out of range in single table complex query
Kernel: Fix PostgreSQL like lower case failed.
Kernel: Fixed the exception of built-in metabase data collection when the front and back database types were inconsistent
Kernel: Fix the problem of routing error reporting under certain table names
Kernel: Fix MySQL create procedure parse error
Kernel: Fix union extract table name NPE
Kernel: Fix upper case table constraint not rewrite error
Kernel: Fix failed to parse PostgreSQL / openGauss SQL contains money type
Kernel: Fix PostgreSQL / openGauss positional parameter rule
Kernel: Fix PostgreSQL / openGauss failed to parse const with type cast
Kernel: Fix Chinese characters encode exception when execute select with sql federation engine
Kernel: Fix IndexOutOfBoundsException when execute set variable statement int jdbc adapter
Kernel: Fix index does not exist exception when execute drop index statement
Proxy: Properly handle number sign in MySQL Proxy binary protocol
Proxy: Fix PostgreSQL Proxy failed to handle bytea data type
Proxy: Fix PostgreSQL Proxy failed to parse time value with microseconds
Proxy: Fix PostgreSQL protocol codec for date type in binary format
Proxy: Fix possible CCE PostgreSQLInt2BinaryProtocolValue
Proxy: Fix possible error when client pass quoted charset to PostgreSQL/openGauss Proxy

Change Log

MILESTONE

5.3.1

New Features

Kernel: Add new data masking, dynamic data masking features, and built-in data masking algorithms
Scaling: Basic support of CDC feature
DistSQL: Add masking rule related DistSQL

Enhancements

Kernel: Cluster mode avoids secondary refresh of metadata
Kernel: SHOW COMPUTE NODES supports displaying the version number of each instance
Kernel: System database add cluster information table
Kernel: Standalone mode persistent metadata supports MySQL
Kernel: SQL HINT performance improvement
Kernel: Restore routing to the specified database with Hint
Encrypt: Supports underscore wildcards for Encrypt CharDigestLikeEncryptAlgorithm
Kernel: Support SQL federation SELECT NULLS LAST/FIRST statement
Kernel: Refactor encrypt integration test logic and add more test cases
Kernel: Add salt props for MD5MaskAlgorithm, MD5EncryptAlgorithm
Kernel: Refactor ShardingConditionEngine to support SPI configuration
DistSQL: Add algorithm type check for CREATE SHARDING TABLE RULE

Bug Fixes

Fix the problem of ZooKeeper cluster error reporting when ShardingSphere connects to Kubernetes
Kernel: Fix use Consul in cluster mode start up failure
DB Discovery: Close heartbeat job when drop discovery rule
Kernel: Fix wrong decide result when execute same sharding condition subquery with SQL federation
Kernel: Fix priority problem of UNION, INTERSECT, EXCEPT set operation in SQL Federation for PostgreSQL and openGauss dialect
Kernel: Fix create view index out of range exception when view contains set operator
Kernel: Add XA resource exceeds length check
Kernel: Fix transaction support for spring requires_new
Encrypt: Fix AESEncryptAlgorithm decrypt exception when config char type with PostgreSQL and openGauss
Encrypt: Fix abnormal expansion result for shorthand when encrypt subquery contains an alias
Kernel: Fix unsigned flag of column metadata was not serialized
Kernel: Fix PostgreSQL / openGauss select fetch parsing issue to support federation execution engine
Proxy: Fix packet sequence ID may be incorrect if error occurred in MySQL Proxy
Proxy: Fix error occur in Proxy when using PostgreSQL composite type
Proxy: Set proper column definition flag for MySQL COM_STMT_PREPARE
Proxy: When querying PG metadata through Proxy and the result set is empty, the labels are lost

Change Log

MILESTONE

5.3.0

API Changes

DistSQL: Refactor syntax API, please refer to the user manual
Proxy: Change the configuration style of global rule, remove the exclamation mark
Proxy: Allow zero-configuration startup, enable the default account root/root when there is no Authority configuration
Proxy: Remove the default logback.xml and use API initialization
JDBC: Remove the Spring configuration and use Driver + YAML configuration instead

Enhancements

DistSQL: New syntax REFRESH DATABASE METADATA, refresh logic database metadata
Kernel: Support DistSQL REFRESH DATABASE METADATA to load configuration from the governance center and rebuild MetaDataContext
Support postgresql/openGauss setting transaction isolation level
Scaling: Increase inventory task progress update frequency
Scaling: DATA_MATCH consistency check support breakpoint resume
Scaling: Support drop consistency check job via DistSQL
Scaling: Rename column from sharding_total_count to job_item_count in job list DistSQL response
Scaling: Add sharding column in incremental task SQL to avoid broadcast routing
Scaling: Sharding column could be updated when generating SQL
Scaling: Improve column value reader for DATA_MATCH consistency check
DistSQL: Encrypt DistSQL syntax optimization, support like query algorithm
DistSQL: Add properties value check when REGISTER STORAGE UNIT
DistSQL: Remove useless algorithms at the same time when DROP RULE
DistSQL: EXPORT DATABASE CONFIGURATION supports broadcast tables
DistSQL: REGISTER STORAGE UNIT supports heterogeneous data sources
Encrypt: Support Encrypt LIKE feature
Automatically start distributed transactions when executing DML statements across multiple shards
Kernel: Support client \d for PostgreSQL and openGauss
Kernel: Support select group by, order by statement when column contains null values
Kernel: Support parse RETURNING clause of PostgreSQL/openGauss Insert
Kernel: SQL HINT performance improvement
Kernel: Support mysql case when then statement parse
Kernel: Supporting data source level heterogeneous database gateway
(Experimental) Sharding: Add sharding cache plugin
Proxy: Support more PostgreSQL datetime formats
Proxy: Support MySQL COM_RESET_CONNECTION
Scaling: Improve MySQLBinlogEventType.valueOf to support unknown event type
Kernel: Support case when for federation

Bug Fix

Scaling: Fix barrier node created at job deletion
Scaling: Fix part of columns value might be ignored in DATA_MATCH consistency check
Scaling: Fix jdbc url parameters are not updated in consistency check
Scaling: Fix tables sharding algorithm type INLINE is case-sensitive
Scaling: Fix incremental task on MySQL require mysql system database permission
Proxy: Fix the NPE when executing select SQL without storage node
Proxy: Support DATABASE_PERMITTED permission verification in unicast scenarios
Kernel: Fix the wrong value of worker-id in show compute nodes
Kernel: Fix route error when the number of readable data sources and weight configurations of the Weight algorithm are not equal
Kernel: Fix multiple groups of readwrite-splitting refer to the same load balancer name, and the load balancer fails problem
Kernel: Fix can not disable and enable compute node problem
JDBC: Fix data source is closed in ShardingSphereDriver cluster mode when startup problem
Kernel: Fix wrong rewrite result when part of logical table name of the binding table is consistent with the actual table name, and some are inconsistent
Kernel: Fix startup exception when use SpringBoot without configuring rules
Encrypt: Fix null pointer exception when Encrypt value is null
Kernel: Fix oracle parsing does not support varchar2 specified type
Kernel: Fix serial flag judgment error within the transaction
Kernel: Fix cursor fetch error caused by wasNull change
Kernel: Fix alter transaction rule error when refresh metadata
Encrypt: Fix EncryptRule cast to TransparentRule exception that occurs when the call procedure statement is executed in the Encrypt scenario
Encrypt: Fix exception which caused by ExpressionProjection in shorthand projection
Proxy: Fix PostgreSQL Proxy int2 negative value decoding incorrect
Proxy: PostgreSQL/openGauss support describe insert returning clause
Proxy: Fix gsql 3.0 may be stuck when connecting Proxy
Proxy: Fix parameters are missed when checking SQL in Proxy backend
Proxy: Enable MySQL Proxy to encode large packets
Kernel: Fix oracle parse comment without whitespace error
DistSQL: Fix show create table for encrypt table

Refactor

Scaling: Reverse table name and column name when generating SQL if it's SQL keyword
Scaling: Improve increamental task failure handling
Kernel: Governance center node adjustment, unified hump to underscore

Change Log

MILESTONE

5.2.1

New Feature

Add ShardingSphere default system database to support global metadata management
Support asynchronous data consistency check
Added support for Consul governance center
Added support for Nacos governance center
Added support for the view function in the governance center

Enhancement

SQL Federation engine adds ADVANCED executor and adapts to openGauss database
Support ShardingSphere Proxy startup after read-write splitting read database is disabled
SQL HINT supports force sharding route
Show processlist supports showing Proxy connections (sleep, active)
Optimized ShardingSphere-JDBC data source configuration error message
ShardingSphere-JDBC supports SpringBoot 3.x version
Support load MySQL, PostgreSQL, openGauss and SQLServer view metadata
Update Snakeyaml to 1.33 and open YAML 3MB limit
Reuse cached connections as possible when unicast sharding
Support Parsing ALTER ROLE in Oracle
Add support of ALTER RESOURCE COST for Oracle
Support parsing Drop Materialized View in Oracle
Support parsing DROP LIBRARY in Oracle
Support parsing DROP JAVA in Oracle
Support parsing DROP PLUGGABLE DATABASE in Oracle
Support parsing DROP INDEX TYPE in Oracle
Support Parsing ALTER PACKAGE in openGauss
Support openGauss select offset, count statement parse and remove useless syntax in PostgreSQL grammar
Add max_size to openGauss syntax
Optimize alter view/drop view parse logic and fix alter view refresher bug
Add sql parser error detail to ParseCancellationException
Add support for parse OptOnConflict for postgres
Enhance support for ALTER TABLE and ALTER VIEW in PostgreSQL
Add a missing keyword in PostgreSQL Declare Statement
Add json function support to mysql parser
ShardingSphere-Proxy automatically adapts to cgroup memory limits in Docker environment
Show migration status DistSQL respond new added error_message column
Show migration status respond new added processed_records_count column
Support MySQL 8 caching_sha2_password authentication in incremental dump
Improve drop pipeline process configuration
Support unique key table migration
Support migrate table to new name table
Improve thread pool usage in pipeline task and runner
Support cancelable data consistency check
DistSQL: When creating or altering readwrite-splitting rule, check duplicate write or read resources
DistSQL: Add validity check for ALTER SHARDING BINDING TABLE RULES
Standalone mode H2 support persistent metadata
Fix openGauss cursor execution in xa transaction
Added transaction related exceptions

Bug Fix

Generate proper placeholder for PostgreSQL when rewriting
Fix opengauss update set parse exception
Fix parse exception when execute insert statement with negative value
Fix wrong connectDescriptorUrlPattern in OracleDataSourceMetaData
Fix insert SQL garbled after sharding rewrote in special rules
Fix exception when execute select * from information_schema.tables
Fix exception when execute alter view rename
Fix PostgreSQL check data source permission when using rolsuper
DistSQL: fix NPE for REFRESH TABLE METADATA when there is no resource
Fix Unmodified table metadata when modify rules
Fix database discovery
The MySQL.NORMAL_REPLICATION algorithm cannot find primary node"
Fixed using etcd to build a cluster event not aware
Fix NPE occurred when transaction management is not created
Fix routing exception that occurs when the sharding value of the InlineShardingAlgorithm algorithm exceeds Integer

API Changes

SQL HINT syntax format adjust to SQL-style format
DistSQL: Remove syntax COUNT DATABASE RULES
ShardingSphere mode remove overwrite configuration
Agent: Optimize configuration of agent.yaml

Change Log

MILESTONE

5.2.0

New Feature

Support SQL audit for sharding feature
Support MySQL show processlist and kill process id feature
Scaling: Add dedicated DistSQL for data migration
Scaling: Basic support migrate data to heterogeneous database
DistSQL: New syntax CREATE MIGRATION PROCESS CONFIGURATION
DistSQL: New syntax ALTER MIGRATION PROCESS CONFIGURATION
DistSQL: New syntax SHOW MIGRATION PROCESS CONFIGURATION
DistSQL: New syntax ADD MIGRATION SOURCE RESOURCE
DistSQL: New syntax SHOW SQL_TRANSLATOR RULE
DistSQL: New syntax CREATE SHARDING AUDITOR
DistSQL: New syntax ALTER SHARDING AUDITOR
DistSQL: New syntax SHOW SHARDING AUDIT ALGORITHMS

Enhancement

Support column visible feature for MySQL, Oracle, SQLServer and H2
Support cartesian product configuration for read write splitting
Support spring namespace and spring boot usage for sql translator
Support JSR-310 Year and Month in IntervalShardingAlgorithm
Support broadcast table update/delete limit statement
Support create index on table(column) statement rewrite when config encrypt
Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
Support encrypt column rewrite when execute column is null in predicate
Support encrypt show create table return logic columns
Support create table with index statement rewrite when config encrypt
Support PostgreSQL create operator statement parse
Support PostgreSQL create materialized view statement parse
Support PostgreSQL nested comments parse
Support PostgreSQL alter subscription statement parse
Support PostgreSQL create group statement parse
Support PostgreSQL alter statictics statement parse
Support PostgreSQL create foreign table statement parse
Support PostgreSQL alter server statement parse
Support PostgreSQL create foreign data wrapper statement parse
Support PostgreSQL create event trigger statement parse
Support PostgreSQL security label statement parse
Support PostgreSQL reindex statement parse
Support PostgreSQL reassign owned statement and refresh materialized view statement parse
Support PostgreSQL prepare transaction statement parse
Support PostgreSQL create collation statement parse
Support PostgreSQL lock statement parse
Support PostgreSQL alter rule statement parse
Support PostgreSQL notify statement parse
Support PostgreSQL unlisten statement parse
Support Oracle alter function and alter hierarchy statement parse
Support Oracle alter pluggable database statement parse
Support Oracle alter materialized view log statement parse
Support Oracle alter diskgroup statement parse
Support Oracle alter operator statement parse
Support oracle alter cluster statement parse
Support oracle alter audit policy statement parse
Support Oracle alter index type statement parse
Support Oracle lock table statement parse
Support Oracle alter java statement parse
Support Oracle inline constraint statement parse
Support openGauss geometric operator statement parse
Optimize MySQL visible/invisible parse of create/alter table statements
Support scope of variable prefixed with @@ in MySQL SET statement parse
Support MySQL create procedure with create view parse
Support column segments parse in create index on table statement
Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
Support encrypt column rewrite when execute column is null in predicate
Support encrypt show create table return logic columns
Support create table with index statement rewrite when config encrypt
Support parsing ALTER LOCKDOWN PROFILE in Oracle
Support parsing ALTER MATERIALIZED VIEW in Oracle
Support parsing ALTER MATERIALIZED ZONEMAP in Oracle
Support parsing ALTER LIBRARY in Oracle
Support parsing ALTER INMEMORY JOIN GROUP in Oracle
Support parsing DROP OPERATOR in Oracle
Support parsing DROP RESTORE POINT in Oracle
Support parsing CREATE RESTORE POINT in Oracle
Support parsing DROP INMEMORY JOIN GROUP in Oracle
Support parsing create_bit_xor_table in MySQL
Support parsing MySQL DO statement
Support parsing DropServer in openGauss
Support parsing CREATE AGGREGATE In openGauss
Support parsing ALTER ROUTINE in PostgreSQL
Add PostgreSQL Create Cast Statement
Add PostgreSQL Create Aggregate Statement
Support fetch/move/close cursor statement in PostgreSQL
Support Parsing ALTER PUBLICATION in PostgreSQL
Add PostgreSQL Create Access Method Statement
Support Parsing ALTER POLICY in PostgreSQL
Support parsing ALTER OPERATOR in PostgreSQL
Add PostgreSQL Copy Statement
Add PostgreSQL Comment Statement
Support listen statement in PostgreSQL
Support DECLARE cursor statement
Add default serverConfig in helm charts
Assemble openGauss JDBC Driver into Proxy distribution
ShardingSphere-Proxy listen on specified IP addresses
Support COM_STMT_SEND_LONG_DATA in MySQL Proxy
SELECT VERSION() support alias in MySQL Proxy
Fix openGauss Proxy could not be connected if no resource defined
Support using JRE defined in JAVA_HOME in ShardingSphere-Proxy's startup script
Avoid client blocked when OOM occurred in ShardingSphere-Proxy
Support using c3p0 in ShardingSphere-JDBC
Support SET NAMES with value quoted by double-quote
Connection.prepareStatement with columns arguments is available in ShardingSphere-JDBC
Scaling: Improve MySQL connect and reconnect
Scaling: Fix MySQL json column may cause leak at incremental task
Scaling: Add permission check for PostgreSQL data sources
Scaling: Incremental migration support for MySQL MGR mode
Scaling: Improve job progress persistence
Scaling: Start job DistSQL execute and return synchronously
Scaling: Inventory migration support table has primary key and unique key
Scaling: Close unerlying ElasticJob when stopping job
Scaling: Improve logical replication slot name generation for PostgreSQL and openGauss
Scaling: Make query DistSQL could be executed when no database selected
DistSQL: Add worker_id to the result set of SHOW INSTANCE LIST & SHOW INSTANCE INFO
DistSQL: Improve the result of EXPORT DATABASE CONFIG
DistSQL: Support more databases for FORMAT SQL
DistSQL: Optimize the execution logic of CREATE TRAFFIC RULE
DistSQL: Support assistEncryptor for Encrypt RDL
DistSQL: Add sharding algorithm type check when CREATE SHARDING TABLE RULE
Support database discovery to configure multiple groups of high availability under the same logic database
Support ShardingSphere-Proxy to start up under empty logic library
Support for isolating EventBus events by instance
Support the database to detect changes in the master node and restart the detection heartbeat task
Support ShardingSphere-Proxy to generate new worker-id when re-registering in cluster mode
Thrown exception when inserting expression value in shadow column on executing insert
Support distributed transactions across multiple logical databases
Support executing truncate in XA & PostgreSQL
Support alter local transaction rule with DistSQL
Support global transaction manager
Delete support for branch transaction on proxy

Bug Fix

Fix single table metadata refresh error caused by filtering DataSourceContainedRule
Fix parsing exception caused by the null value of MySQL blob type
Fix PostgreSQL/openGauss reset statement parse error
Fix wrong parameter rewrite when use sharding and encrypt
Fix the failed conversion of Month related classes on IntervalShardingAlgorithm
Fix NullPointerException when execute select union statement contains subquery
Fix wrong encrypt rewrite result due to incorrect order of metadata
Fix MySQL trim function parse error
Fix MySQL insert values with _binary parse error
Fix MySQL syntax error cannot be thrown to client
Avoid EventLoop blocked because of closing JDBC resources
Correct server status flags returned by MySQL Proxy
Fix a possible connection leak issue if Proxy client disconnected in transaction
Fixed a possible consistency issue with the statement being executed when the Proxy client is disconnected
Avoid pooled connection polluted by executing SET statements
Make SHOW TABLES FROM work in ShardingSphere-Proxy
Fix PostgreSQL DDL could not be executed by Extended Query
Fix SHOW VARIABLES could not be executed in PostgreSQL Proxy without resource
Fix FileNotFoundException when use ShardingSphere Driver with SpringBoot fatjar
Scaling: Fix the problem that the table contains both primary key and unique index at inventory migration
Scaling: Improve incremental migration, support the latest position in the middle of batch insert event
Scaling: Fix the error caused by null field value in openGauss incremental migration
DistSQL: Fix incorrect strategy name in result of SHOW SHARDING TABLE RULES
DistSQL: Fix current rule config is modified in advance when ALTER SHARDING TABLE RULE
DistSQL: Fix connection leak when ALTER RESOURCE
DistSQL: Fix CREATE TRAFFIC RULE failed when load balance algorithm is null
Fix that the monitoring heartbeat task was not stopped when the database was discovered and the logical library was deleted
Fix cluster mode ShardingSphere-JDBC load all logic database
Fix worker-id generated by SnowflakeKeyGenerateAlgorithm in cluster mode may exceed the maximum value
Fix DistSQL adding shadow algorithm exception without shadow data source
Fix cross-database data source confusion caused by same data source name in multiple logical databases
Fix RUL DistSQL execution failure in transaction
Fix begin for PostgreSQL & openGauss
Agent: Fixed the error of null value in contextManager when collecting metric data

API Changes

Remove SQL passthrough to data source feature
Add new assistedQueryEncryptorName and remove QueryAssistedEncryptAlgorithm interface
Refactor readwrite-splitting api to improve user experience
Remove check-duplicate-table-enabled configuration
Remove useless config item show-process-list-enabled configuration
Scaling: Change keyword for part of data migration DistSQL
Scaling: Redesign part of data migration DistSQL
DistSQL: Unify parameter type specification
DistSQL: Split SHOW INSTANCE MODE to SHOW MODE INFO and SHOW INSTANCE INFO
DistSQL: Change DROP SCALING jobId to CLEAN MIGRATION jobId
DistSQL: Remove COUNT INSTANCE RULES
Add database found that high availability supports all the slave libraries to go offline, and the main library undertakes the read traffic configuration
SnowflakeKeyGenerateAlgorithm supports configuring worker-id in standalone mode
Replace sourceDataSourceName with productionDataSourceName in Shadow API Configuration
Authority: Remove deprecated native authority provider

Refactor

ShardingSphere metadata refactoring for splitting actual metadata and logical metadata
Use ConnectionContext, QueryContext to remove ThreadLocal in FetchOrderByValueQueuesHolder, TrafficContextHolder, SQLStatementDatabaseHolder and TransactionHolder
Modify the default value of the ShardingSphere-Proxy version in the helm chart
Docker container will exit if ShardingSphere-Proxy failed to startup
Helm Charts in ShardingSphere repository are transferred to sub-project shardingsphere-on-cloud
Scaling: Plenty of refactor for better code reuse
DistSQL: Add a new category named RUL
Refactor the schedule module and split it into cluster schedule and standalone schedule
Remove memory mode, keep standalone mode and cluster mode
Refactoring metadata table loading logic and persistence logic
Refactoring distributed locks to retain the most concise interface design
Refactor : Unify The Creation for Proxy Containers in IT from ENV Modules
Refactor : Unify The Configuration for container created by testcontainer

Change Log

MILESTONE

5.1.2

New Feature

Kernel: Alpha version to support SQL dialect translate for MySQL and PostgreSQL
Kernel: Support custom schema for PostgreSQL and openGauss
Kernel: Support create/alter/drop view statement for PostgreSQL and openGauss
Kernel: Support openGauss cursor statement
Kernel: Support use customize system database
Kernel: Support get openGauss and MySQL create SQL
Kernel: Support get postgres create SQL
Proxy: Official support for quickly deploying a ShardingSphere-Proxy cluster with a ZooKeeper cluster in Kubernetes using Helm
JDBC: Support ShardingSphere JDBC Driver
Scaling: Support PostgreSQL auto create table
Scaling: Support scaling for table with customized schema in PostgreSQL and openGauss
Scaling: Support scaling for table with text primary key and without integer primary key
Mode: Registry center supports PG/openGauss three-level structure
Mode: Registry center supports database-level distributed lock

Enhancement

Kernel: Support copy statement for PostgreSQL and openGauss
Kernel: Support alter/drop index statement for PostgreSQL
Kernel: Support update force index statement for MySQL
Kernel: Support create/alter/drop schema for openGauss
Kernel: Optimize RoundRobinReplicaLoadBalanceAlgorithm and RoundRobinTrafficLoadBalanceAlgorithm logic
Kernel: Optimize metadata loading logic when frontendDatabaseType and backendDatabaseType are different
Kernel: Refactor meta data load logic
Kernel: Optimize show processlist statement
Kernel: Improve performance about large tables loaded
Kernel: Support execute comment statement
Kernel: Support view in sharding rule
Kernel: Support parsing CREATE ROLLBACK SEGMENT in Oracle
Kernel: Support Parsing DROP TYPE in openGauss
Kernel: Support Parsing ALTER TYPE in openGauss
Kernel: Support parsing DROP DISKGROUP in Oracle
Kernel: Support parsing CREATE DISKGROUP in Oracle
Kernel: Support parsing DROP FLASHBACK ARCHIVE in Oracle
Kernel: Support Parsing CHECKPOINT in openGauss
Kernel: Support parsing CREATE FLASHBACK ARCHIVE in Oracle
Kernel: Add PostgreSQL Close Statement
Kernel: Support Parsing DROP CAST in openGauss
Kernel: Support parsing CREATE CAST in openGauss
Kernel: Support parsing CREATE CONTROL FILE in Oracle
Kernel: Support Parsing DROP DIRECTORY in openGauss
Kernel: Support parsing ALTER DIRECTORY in openGauss
Kernel: Support parsing CREATE DIRECTORY in openGauss
Kernel: Add PostgreSQL Checkpoint Statement
Kernel: Support parsing DROP SYNONYM in openGauss
Kernel: Support parsing CREATE SYNONYM in openGauss
Kernel: Support parsing ALTER SYNONYM in openGauss
Kernel: Add PostgreSQL CALL Statement
Kernel: Support parsing CREATE PFILE in Oracle
Kernel: Support parsing CREATE SPFILE in Oracle
Kernel: Support parsing ALTER SEQUENCE in Oracle
Kernel: Support parsing CREATE CONTEXT in Oracle
Kernel: Support Parsing ALTER PACKAGE in oracle
Kernel: Support parsing CREATE SEQUENCE in Oracle
Kernel: Support parsing ALTER ATTRIBUTE DIMENSION in Oracle
Kernel: Support parsing ALTER ANALYTIC VIEW in Oracle
Kernel: Use ShardingSphere SPI in SQLVisitorFacade
Kernel: Use ShardingSphere SPI in DatabaseTypedSQLParserFacade
Kernel: Support parsing ALTER OUTLINE in Oracle
Kernel: Support parsing DROP OUTLINE in Oracle
Kernel: Support parsing drop edition in oracle
Kernel: Support WITH Common Table Expression of SQLServer
Kernel: Exclude parenthesis from SubquerySegment's start and stop index in withClause
Kernel: Refactor JoinTableSegment
Kernel: Support parsing DROP SYNONYM in Oracle
Kernel: Support parsing CREATE DIRECTORY in Oracle
Kernel: Support parsing CREATE SYNONYM in Oracle
Kernel: Support for XmlNamespaces Clause of SQLServer SELECT Statement
Kernel: Support parsing Alter Database Dictionary in Oracle
Kernel: Support FOR Clause of SQLServer SELECT Statement
Kernel: Support Parsing ALTER DATABASE LINK in Oracle
Kernel: Support CREATE EDITION Parsing in Oracle
Kernel: Support parsing ALTER TRIGGER in Oracle
Kernel: Add SQLServer REVERT Statement
Kernel: Support Parsing DROP TEXT SEARCH in PostgreSQL
Kernel: Add drop server for PostgreSQL
Kernel: Support Parsing ALTER VIEW in Oracle
Kernel: Add drop access method for PostgreSQL
Kernel: Support Parsing DROP ROUTINE in PostgreSQL
Kernel: Proofread SQLServer DROP USER Statement
Kernel: Support parsing DROP TRIGGER in Oracle
Kernel: Support parsing Drop subscription in PostgreSQL
Kernel: Add drop operator class for PostgreSQL
Kernel: Support parsing DROP PUBLICATION in PostgreSQL
Kernel: Support Parsing DROP VIEW in Oracle
Kernel: Support Parsing DROP TRIGGER in PostgreSQL
Kernel: Support Parsing DROP DIRECTORY in Oracle
Kernel: Support Parsing DROP STATISTICS for PostgreSQL
Kernel: Add drop type SQL parser for PostgreSQL
Kernel: Support Parsing DROP RULE in PostgreSQL
Kernel: Proofread SQLServer ALTER LOGIN Statement
Kernel: Support parsing PostgreSQL DROP FOREIGN DATA WRAPPER
Kernel: Small changes to PostgreSQL DROP EVENT TRIGGER statement
Proxy: ShardingSphere-Proxy MySQL supports receiving MySQL packet more than 16 MB
Proxy: Supports netty parameter ChannelOption.SO_BACKLOG configurable in ShardingSphere-Proxy
Proxy: Optimize so-reuseaddr in netty to solve the problem of port occupied
Proxy: Docker image of ShardingSphere-Proxy supports aarch64 platform
Proxy: Make server version configurable in ShardingSphere-Proxy MySQL
Proxy: Supports more character sets in ShardingSphere-Proxy PostgreSQL/openGauss
Proxy: Make default port configurable in ShardingSphere-Proxy
Scaling: Compatible with HA ports for openGauss:3.0 when thread_pool enabled
Scaling: Optimize ZooKeeper event handling in PipelineJobExecutor to avoid blocking ZooKeeper events
Scaling: Make table name case-insensitive in whole process
Scaling: Improve replication slot cleanup for PostgreSQL and openGauss
Scaling: Improve lock protection for job preparation
Scaling: Support PostgreSQL insert on conflict do update
Scaling: Do not cache data source in GlobalDataSourceRegistry to avoid possible shared resource close issue
Scaling: Reuse data source pool as more as possible to reduce working database connections
DistSQL: REFRESH TABLE METADATA supports specifying PostgreSQL's schema
DistSQL: ALTER SHARDING TABLE RULE add validation of binding table
Mode: ShardingSphere-JDBC supports configuring database connection name
Distributed Transaction: DistSQL is prohibited from executing within a transaction
Distributed Transaction: autocommit = 0, DDL part of DML will automatically open the transaction

Bug Fix

Kernel: Fix parsing error about show statement for PostgreSQL and openGauss
Kernel: Fix parsing error about time extract function for for PostgreSQL and openGauss
Kernel: Fix parsing error about select mod function for for PostgreSQL and openGauss
Kernel: Fix PSQLException when execute join statement with multi schema in readwrite scenario
Kernel: Fix wrong route result when execute create schema statement in encrypt scenario
Kernel: Fix npe when execute drop schema if exist statement
Kernel: Fix wrong route result when execute SELECT LAST_INSERT_ID() AS id; statement
Kernel: Fix npe when execute use database when database doesn't contains datasource
Kernel: Fix create function with set var
Proxy: Fix NPE caused by column's case unmatched in PostgreSQLComDescribeExecutor
Proxy: Complete command tags for schema DDL in ShardingSphere-Proxy PostgreSQL / openGauss
Scaling: Fix MySQL unsigned type null value cause error during increment task
Scaling: Fix resource leak caused by error occurred when creating DataSource in ShardingSphere-Scaling
Scaling: Fix ShardingSphereDataSource creation ignoring other rules
Scaling: Fix on preparation job could not be stopped
Scaling: Fix data source property url and jdbcUrl compatibility
Scaling: Fix openGauss logical replication slot creation, avoid possible incremental data loss
Scaling: Update local job status before persisting job status to registry center, make sure it won't be overwritten later
Scaling: Handling null value in TestDecodingPlugin for PostgreSQL
DistSQL: Fix SET VARIABLE modification not taking effect in stand-alone and memory mode
DistSQL: Fix the inconsistency between SHOW INSTANCE LIST display data and actual data
DistSQL: Fix capitalization sensitivity in sharding DistSQL
Mode: Fix the new version metadata lost data after the Scaling changes the table sharding rules
Distributed Transaction: Fix getIndexInfo with catalog

API Changes

DistSQL: Change EXPORT SCHEMA CONFIG to EXPORT DATABASE CONFIG
DistSQL: Change IMPORT SCHEMA CONFIG to IMPORT DATABASE CONFIG
DistSQL: Change SHOW SCHEMA RESOURCES to SHOW DATABASE RESOURCES
DistSQL: Change COUNT SCHEMA RULES to COUNT DATABASE RULES
Mode: Adjust db-discovery algorithm configuration
Authority: Authority provider ALL_PRIVILEGES_PERMITTED updated to ALL_PERMITTED
Authority: Authority provider SCHEMA_PRIVILEGES_PERMITTED updated to DATABASE_PERMITTED

Refactor

Scaling: Refactor JobConfiguration, prepare for different types of jobs reuse and extension
Mode: Optimize compute node structure of the registry center
Mode: Use uuid instead of ip@port as instance id

Change Log

MILESTONE

5.1.1

New Feature

Kernel: support alter materialized view for PostgreSQL
Kernel: support declare for PostgreSQL
Kernel: support discard for PostgreSQL
Kernel: Add mode to parser to support $$ in PostgreSQL
Kernel: Support MySQL create tablespace statement parse
Scaling: Implement stop source writing and restore source writing
Scaling: Support partial tables scale-out
DistSQL: New DistSQL syntax: SHOW UNUSED RESOURCES
Mode: Added persistent XA Recovery Id to Governance Center
Mode: Database discovery adds delayed master-slave delay function
Distributed Transaction: Add savepoint support for ShardingSphere proxy
Distributed Transaction: Support auto rollback when report exception in transaction block for PostgreSQL and openGauss
Distributed Transaction: Make it is easy to use with Narayana
Distributed Transaction: Add savepoint support for ShardingSphere-JDBC

Enhancement

Kernel: Refactor kernel to improve performance
Proxy: Reduce Docker image size of ShardingSphere-Proxy
Proxy: ShardingSphere-Proxy supports set names statements
Proxy: ShardingSphere-Proxy MySQL supports multi statements
Scaling: Only one proxy node could do data consistency check in proxy cluster
Scaling: Replace scaling input and output config fields type from int to Integer
Scaling: Update MySQL checksum SQL
Scaling: Improve scaling job progress deletion in reset and progress check before starting job
Scaling: Improve FinishCheckJob data consistency check when target tables already have the same data as source tables
Scaling: Break scaling job ASAP when there is unsupported table since primary key
Scaling: Reuse ClusterPersistRepository of proxy in PipelineAPIFactory
Scaling: Update jobId generation algorithm, and make it support idempotency
DistSQL: Support configuration data type and length when CREATE/ALTER ENCRYPT RULE
DistSQL: Unify the display results of SHOW ALL VARIABLES and SHOW VARIABLE
DistSQL: Remove the effect of binding order when DROP BINDING TABLE RULES
DistSQL: Add column mode_type in the result of SHOW INSTANCE LIST
DistSQL: Add validation to the mode when ENABLE/DISABLE INSTANCE
DistSQL: Check if the rule is in used when DROP READWRITE_SPLITTING RULE
DistSQL: Check duplicate resource names when CREATE READWRITE_SPLITTING RULE
DistSQL: Add column delay_time to the result of SHOW READWRITE_SPLITTING READ RESOURCES
DistSQL: Support IF EXISTS when DROP RULE
DistSQL: Optimize the prompt information of connection failure when ADD/ALTER RESOURCE
Mode: Add schema-level global distributed locks
Mode: Add schema version number to support batch execution of DistSQL
Mode: Persistent metadata optimization in cluster mode
Mode: The database discovery add the schemaName identifier when create a JOB

Refactor

Kernel: Refactor test case for encrypt
Kernel: Refactor metadata to support PostgreSQL database and schema
Scaling: Remove HikariCP dependency in pipeline modules
Mode: Refactor governance center storage node structure
Mode: Refactor governance center meta data structure
Mode: Adjust the database discovery MGR module to MySQL module

Bug Fix

Kernel: Fix function with no parameter
Kernel: Fix InsertValueContext.getValue cast exception
Kernel: Fix aggregate distinct column error
Kernel: Fix NPE when rewrite parameter with schema
Kernel: Fix NPE caused by GeneratedKeysResultSet not return columnName in read-write splitting
Kernel: Fix show tables statement loses part of the single table
Kernel: Fix ModShardingAlgorithm wrong route result when exist same suffix table
Kernel: Fix sql parse error when contains key in assignment clause and optimize index parse
Kernel: Fix NumberFormatException when sharding algorithm config number props
Kernel: Fix wrong metadata when config single dataSource for read-write splitting
Kernel: Fix statement close exception when use BatchPreparedStatementExecutor
Kernel: Fix rewrite lowercase logic when sql contains shorthand projection
Kernel: Fix NullPointerException when start up proxy with memory mode
Proxy: Fix literals may be replaced by mistake in PostgreSQL/openGauss protocol
Proxy: Fix ShardingSphere-Proxy PostgreSQL with multi-schema cannot be connected by PostgreSQL JDBC Driver 42.3.x
Proxy: Fix timestamp nanos inaccurate in ShardingSphere-Proxy MySQL
Proxy: Complete ShardingSphere-Proxy PostgreSQL codec for numeric in binary format
Proxy: Potential performance issue and risk of OOM in ShardingSphere-JDBC
Proxy: Fix Operation not allowed after ResultSet closed occasionally happens in ShardingSphere-Proxy MySQL
Proxy: Fix NPE causes by ShardingSphere-JDBC executeBatch without addBatch
Scaling: Fix failed or stopped job could not be started by DistSQL except restarting proxy
DistSQL: Fix parsing exception for inline expression when CREATE SHARDING TABLE RULE
DistSQL: Fix parsing exception when password is keyword password in ADD RESOURCE statement
Mode: Fixed loss of compute nodes due to ZooKeeper session timeout
Mode: Fixed the case of the table name in the governance center
Mode: DistSQL enable disable instance refresh in-memory compute node status
Mode: Fixed database discovery unable to create Rule through DistSQL

Change Log

MILESTONE

5.1.0

New feature

Support SQL hint
New DistSQL syntax: SHOW AUTHORITY RULE
New DistSQL syntax: SHOW TRANSACTION RULE
New DistSQL syntax: ALTER TRANSACTION RULE
New DistSQL syntax: SHOW SQL_PARSER RULE
New DistSQL syntax: ALTER SQL_PARSER RULE
New DistSQL syntax: ALTER DEFAULT SHARDING STRATEGY
New DistSQL syntax: DROP DEFAULT SHARDING STRATEGY
New DistSQL syntax: CREATE DEFAULT SINGLE TABLE RULE
New DistSQL syntax: SHOW SINGLE TABLES
New DistSQL syntax: SHOW SINGLE TABLE RULES
New DistSQL syntax: SHOW SHARDING TABLE NODES
New DistSQL syntax: CREATE/ALTER/DROP SHARDING KEY GENERATOR
New DistSQL syntax: SHOW SHARDING KEY GENERATORS
New DistSQL syntax: REFRESH TABLE METADATA
New DistSQL syntax: PARSE SQL, Output the abstract syntax tree obtained by parsing SQL
New DistSQL syntax: SHOW UNUSED SHARDING ALGORITHMS
New DistSQL syntax: SHOW UNUSED SHARDING KEY GENERATORS
New DistSQL syntax: CREATE/DROP SHARDING SCALING RULE
New DistSQL syntax: ENABLE/DISABLE SHARDING SCALING RULE
New DistSQL syntax: SHOW SHARDING SCALING RULES
New DistSQL syntax: SHOW INSTANCE MODE
New DistSQL syntax: COUNT SCHEMA RULES
Scaling: Add rateLimiter configuration and QPS TPS implementation
Scaling: Add DATA_MATCH data consistency check
Scaling: Add batchSize configuration to avoid possible OOME
Scaling: Add streamChannel configuration and MEMORY implementation
Scaling: Support MySQL BINARY data type
Scaling: Support MySQL YEAR data type
Scaling: Support PostgreSQL BIT data type
Scaling: Support PostgreSQL MONEY data type
Database discovery adds support for JDBC Spring Boot
Database discovery adds support for JDBC Spring Namespace
Database discovery adds support for openGauss
Shadow DB adds support for logical data source transfer
Add data type validator for column matching shadow algorithm
Add support for xa start/end/prepare/commit/recover in encrypt case with only one data source

API Change

Redesign the database discovery related DistSQL syntax
In DistSQL, the keyword GENERATED_KEY is adjusted to KEY_GENERATE_STRATEGY
Native authority provider is marked as deprecated and will be removed in a future version
Scaling: Move scaling configuration from server.yaml to config-sharding.yaml
Scaling: Rename clusterAutoSwitchAlgorithm SPI to completionDetector and refactor method parameter
Scaling: Data consistency check API method rename and return type change
Database discovery module API refactoring
Readwrite-splitting supports static and dynamic configuration
Shadow DB remove the enable configuration
Shadow algorithm type modified

Enhancement

Improve load multi single table performance
Remove automatically added order by primary key clause
Optimize binding table route logic without sharding column in join condition
Support update sharding key when the sharding routing result keep the same
Optimize rewrite engine performance
Support select union/union all ... statements by federation engine
Support insert on duplicate key update sharding column when route context keep same
Use union all to merge sql route units for simple select to improve performance
Supports autocommit in ShardingSphere-Proxy
ShardingSphere openGauss Proxy supports sha256 authentication method
Remove property java.net.preferIPv4Stack=true from Proxy startup script
Remove the verification of null rules for JDBC
Optimize performance of executing openGauss batch bind
Disable Netty resource leak detector by default
Supports describe prepared statement in PostgreSQL / openGauss Proxy
Optimize performance of executing PostgreSQL batched inserts
Add instance_id to the result of SHOW INSTANCE LIST
Support to use instance_id to perform operations when enable/disable a proxy instance
Support auto creative algorithm when CREATE SHARDING TABLE RULE, reducing the steps of creating rule
Support specifying an existing KeyGenerator when CREATE SHARDING TABLE RULE
DROP DATABASE supports IF EXISTS option
DATANODES in SHARDING TABLE RULE supports enumerated inline expressions
CREATE/ALTER SHARDING TABLE RULE supports complex sharding algorithm
SHOW SHARDING TABLE NODES supports non-inline scenarios (range, time, etc.)
When there is only one read data source in the readwrite-splitting rule, it is not allowed to be disabled
Scaling: Add basic support of chunked streaming data consistency check
Shadow algorithm decision logic optimization to improve performance

Refactor

Refactor federation engine scan table logic
Avoid duplicated TCL SQL parsing when executing prepared statement in Proxy
Scaling: Add pipeline modules to redesign scaling
Scaling: Refactor several job configuration structure
Scaling: Precalculate tasks splitting and persist in job configuration
Scaling: Add basic support of pipeline-core code reuse for encryption job
Scaling: Add basic support of scaling job and encryption job combined running
Scaling: Add input and output configuration, including workerThread and rateLimiter
Scaling: Move blockQueueSize into streamChannel
Scaling: Change jobId type from integer to text
Optimize JDBC to load only the specified schema
Optimize meta data structure of the registry center
Rename Note shadow algorithm to HINT shadow algorithm

Bug Fix

Support parsing function
Fix alter table drop constrain
Fix optimize table route
Support Route resource group
Support parsing binlog
Support postgreSql/openGauss '&' and '|' operator
Support parsing openGauss insert on duplicate key
Support parse postgreSql/openGauss union
Support query which table has column contains keyword
Fix missing parameter in function
Fix sub query table with no alias
Fix utc timestamp function
Fix alter encrypt column
Support alter column with position encrypt column
Fix delete with schema for postgresql
Fix wrong route result caused by oracle parser ambiguity
Fix projection count error when use sharding and encrypt
Fix npe when using shadow and readwrite_splitting
Fix wrong metadata when actual table is case insensitive
Fix encrypt rewrite exception when execute multiple table join query
Fix parsing chinese
Fix encrypt exists sub query
Fix full route caused by the MySQL BINARY keyword in the sharding condition
Fix getResultSet method empty result exception when using JDBCMemoryQueryResult processing statement
Fix incorrect shard table validation logic when creating store function/procedure
Fix null charset exception occurs when connecting Proxy with some PostgreSQL client
Fix executing commit in prepared statement cause transaction status incorrect in MySQL Proxy
Fix client connected to Proxy may stuck if error occurred in PostgreSQL with non English locale
Fix file not found when path of configurations contains blank character
Fix transaction status may be incorrect cause by early flush
Fix the unsigned datatype problem when query with PrepareStatement
Fix protocol violation in implementations of prepared statement in MySQL Proxy
Fix caching too many connections in openGauss batch bind
Fix the problem of missing data in SHOW READWRITE_SPLITTING RULES when db-discovery and readwrite-splitting are used together
Fix the problem of missing data in SHOW READWRITE_SPLITTING READ RESOURCES when db-discovery and readwrite-splitting are used together
Fix the NPE when the CREATE SHARDING TABLE RULE statement does not specify the sub-database and sub-table strategy
Fix NPE when PREVIEW SQL by schema.table
Fix DISABLE statement could disable readwrite-splitting write data source in some cases
Fix DIABLE INSTANCE could disable the current instance in some cases
Fix the issue that user may query the unauthorized logic schema when the provider is SCHEMA_PRIVILEGES_PERMITTED
Fix NPE when authority provider is not configured
Scaling: Fix DB connection leak on XA initialization which triggered by data consistency check
Scaling: Fix PostgreSQL replication stream exception on multiple data sources
Scaling: Fix migrating updated record exception on PostgreSQL incremental phase
Scaling: Fix MySQL 5.5 check BINLOG_ROW_IMAGE option failure
Scaling: Fix PostgreSQL xml data type consistency check
Fix database discovery failed to modify cron configuration
Fix single read data source use weight loadbalance algorithm error
Fix create redundant data source without memory mode
Fix column value matching shadow algorithm data type conversion exception

Change Log

MILESTONE

5.0.0

New feature

Support parsing SQL comment
New DistSQL syntax: shadow rule management
New DistSQL syntax: scaling job management
New DistSQL syntax: disable proxy instance
New DistSQL syntax: disable readwrite-splitting read data source
New DistSQL syntax: DROP SHARDING ALGORITHM
New DistSQL syntax: ALTER RESOURCE
New DistSQL syntax: CREATE SHARDING ALGORITHM
New DistSQL syntax: CREATE DEFAULT SHARDING [TABLE | DATABASE] STRATEGY
New DistSQL syntax: SHOW ALL VARIABLE
New DistSQL syntax：SHOW VARIABLE variableName;
Support 3 modes, including Memory, Standalone and Cluster mode
Proxy supports for openGauss
Scaling: Add basic support for openGauss
Scaling: Add incremental task completion detect algorithm SPI interface
Scaling: Add data consistency check algorithm SPI interface
Scaling: Basic support of create table on target automatically for MySQL and openGauss
Scaling: Support auto switch cluster configuration when job finished
Scaling: Add more DistSQL support such as data consistency check, etc

API Change

Add schema name configuration for ShardingSphere-JDBC
Add default sharding column configuration
Change the default authority provider from NATIVE to ALL_PRIVILEGES_PERMITTED
SCTL syntax adjustment, merged with DistSQL RAL syntax
SHOW RESOURCES DistSQL is adjusted to SHOW SCHEMA RESOURCES
Remove shadow logical field, support shadow algorithm

Enhancement

Support parsing MySQL union/union all statement
Support PostgreSQL ABORT statement
Refactor and improve metadata loading process
Support PostgreSQL CREATE INDEX statement to generate index automatically when no index is specified
Support SQL statement execution with logical schema
Support binding tables configuration with different sharding columns
Optimize kernel performance
Proxy supports queries to part of information_schema tables to optimize client connection experience
DistSQL supports using quotation marks to use keywords as parameter names
The password in the ADD RESOURCE statement supports special characters
ADD RESOURCE supports custom JDBC parameters and connection pool properties
DROP RESOURCE supports optional parameter ignore single tables, used to ignore single table rule restrictions
Support the use of DistSQL to create sharding table rule based on readwrite-splitting rules
SHOW DATABASES statement supports like syntax
CREATE SHARDING TABLE RULE supports the use of inline expressions to specify resources
CREATE SHARDING TABLE RULE supports configuration using dataNodes
CREATE SHARDING TABLE RULE supports reuse of existing algorithms
SET VARIABLE, support to modify proxy configuration
PostgreSQL's protocol enhancements (Such as supports Portal, unspecified type)
Using Netty executor to optimize Proxy performance in specified scenarios
Make memory strictly fetch size configurable in Proxy
Scaling: Improve support for PostgreSQL
Scaling: Support concurrent data consistency check of source and target side

Refactor

Refactor the SingleTable feature to support Encrypt multiple data sources
Adjust the persistent data structure of the registry center state node
Remove the SQL rewrite of DML for Shadow
Support the SQL routing of DDL for Shadow
Scaling: Refactor default implementation of incremental task completion detect algorithm
Scaling: Refactor default implementation of data consistency check algorithm
Scaling: Remove HTTP API and binary distribution

Bug Fix

Fix sharding interval algorithm
Fix SHOW INDEX FROM TABLE FROM TABLE statement rewrite exception
Fix Encrypt multi tables join query rewrite exception
Fix subquery index out of range exception
Fix wrong result of Oracle paging query
Fix the rewrite exception when KeyGenerateStrategy is not configured in the Sharding feature
Fix federation executor engine exception caused by Oracle dialect case
Fix Sharding and Encrypt integration usage rewrite exception
Fix Oracle metadata loading exception
Fix the issue that SHOW RESOURCES statement cannot display custom attributes
Fix the issue that SQL execution exception is not thrown
Fix Etcd can not send node added event
Fix PostgreSQL rows contains null value may be missing in query result
Fix PostgreSQL metadata columns are out-of-order
Fix client character set may be incorrect in Proxy

Change Log

MILESTONE

5.0.0-beta

New feature

New DistSQL to load and show all the ShardingSphere configuration rules
Support join SQL from different database instances
Support multiple backend database types for new feature of database gateway
Support creating and updating the authentication online
Add new automated agent module

API Change

QueryReplica configuration item is replaced by read-write-splitting
Authentication configuration item of ShardingProxy is replaced by AUTHORITY
Optimize the datasource configuration for ShardingJDBC with dataSourceClassName
New API for automated sharding table configuration to provide standard and automated sharding tables
Remove configuration item acceptor-size from ShardingProxy
Create a built-in sharding algorithm SPI which allows users to directly configure the class name as in 4.x

Enhancement

Improve metadata loading process distinctly
Greatly enhance the SQL parsing for Oracle, SQLServer and PostgreSQL
Support loading privileges from MySQL/PostgreSQL/SQLServer/Oracle
Support DDL statement for encryption feature
Support rewrite owner table name of projection if using sharding and encrypt together
When using SELECT * from encrypt SQL, add quote char for rewritten column to avoid conflict with keyword
Support PostgreSQL JSON/JSONB/pattern matching operator parse
Support MySQL/PostgreSQL CREATE/ALTER/DROP TABLESPACE statement
Support PostgreSQL PREPARE, EXECUTE, DEALLOCATE statement
Support PostgreSQL EXPLAIN statement
Support PostgreSQL START/END TRANSACTION statement
Support PostgreSQL ALTER/DROP INDEX statement
Support CREATE TABLESPACE of PostgreSQL
Support MySQL CREATE LOADABLE FUNCTION statement
Support MySQL/PostgreSQL ALTER TABLE RENAME statement
Support PostgreSQL Close command

Refactor

New schema structure in registry center
Remove Nacos and Apollo config center support
ShardingScaling leverages elasticJob as its job distribution
Refactor the metadata content and its update process

Bug Fix

Bug fix for cannot use SELECT * wildcard when readwrite-splitting only
Fix instance error when the custom sharding algorithm does not match the configuration type.
Fix NoSuchTableException when execute drop table if exists statement
Fix wrong table rewrite in UPDATE ... SET ... statement
Fix wrong table rewrite in CREATE/ALTER TABLE statement foreign key reference table
Fix projection owner check exception when exist subquery temporary table
Fix Oracle/SQL92 SELECT ... WHERE ... LIKE statement class cast exception
Fix MySQL SELECT EXISTS ... FROM ... statement parse error
Fix wrong result of SHOW INDEX statement
Fix SELECT ... GROUP BY ... statement rewrite and merge result error
Fix CREATE TABLE statement rewrite error for encrypt
Fix exception occur in PostgreSQL Proxy when reading text format parameter values
Enhance the support of array object for PostgreSQL Proxy
Fix the bug of Datetype casting for ShardingProxy
PostgreSQL Proxy supports using numeric type
Fix PostgreSQL Proxy transaction command complete packet's tag incorrect
Fix PostgreSQL Proxy may return packet which is not expected by client

Change Log

MILESTONE

5.0.0-alpha

Build & Dependencies

Upgrade the minimum supported version of JDK to Java8
Update Zookeeper to version 3.6.x and curator to version 5.1.0
Update Google Guava to version 29.0-jre

New Feature

The pluggable architecture is available and support function extension through the SPI
Independent SQL parsing engine is available to get SQL parsed AST for the different database dialects
New RDL(Rule Definition Language) feature for ShardingSphere Proxy supports creating sharding rules and sharding tables
ShardingSphere-Scaling supports resuming data migration from break-point
ShardingSphere-Scaling supports using ShardingSphere JDBC to migrate data to new cluster
ShardingSphere shadow database is available

API Changes

New sharding/encryption/shadow/queryReplica API
New sharding algorithm and strategy API
New API for ShardingSphere Scaling to create task
Remove DefaultDataSourceName setting item
The separator of the parameter configuration item is changed from the dot ‘.’ to minus sign ‘-’
Change parameter allow.range.query.with.inline.sharding from global parameter to sharding algorithm parameter

Refactor

Refactor the architecture of SQL parsing module base on the database dialects
Use SPI mechanism to reconstruct online metadata change processing
Rename Orchestration module to Governance module
Rename MasterSlave module to QueryReplica module
Refactor the metadata structure in the governance registration center
Refactor GovernmentShardingSphereDataSource
ShardingSphere UI merges configuration center and registration center display layout

Enhancement

The enhancement for MySQL and PostgreSQL SQL syntax definition and parsing process
The enhancement for sub-queries in different database dialects
Support MySQL view operations for non-sharding tables
Support MySQL stored function and procedure operations for non-sharding tables
Support SQLServer Top syntax
Optimize the metadata loading to improve the startup speed
Optimize batch insert performance
Supports the use of Oracle RAC JDBC URL
XA transaction manager adds support for Oracle database
ShardingSphere Proxy supports the use of p6sy driver
Add help information to the ShardingSphere Proxy startup script

Bug Fixes

Fix alias rewriting error when processing OrderBy condition
Fix SQL rewriting error when MYSQL Insert statement contains expression
Fix parameter calculation error in update on duplicate SQL
Fix generatedKeys gets wrong when batch inserting
Fix the abnormal issue of multi-table verification in DML statement update operation
Fix the NPE problem caused by executing SQL when the table does not exist
Fix the exception when using the Show table command for an unconfigured table
Fix metadata loading error when Oracle database has multiple qualified users
Fix the issue that replica node cannot be enabled online
Fix the problem that ShardingSphere-JDBC does not support PostgreSQL array type
Fix the problem that ShardingSphere-Proxy does not respond when querying long blob data

Change Logs

MILESTONE

4.1.1

Enhancement

Add Sharding-Scaling & Sharding-UI dockerfile
update MySQL & PostgreSQL version for proxy

Bug Fixes

Fix parser syntax rule of SUBSTRING and CONVERT
Fix parser ColumnSegment ClassCastException
Fix TableMetaData load error when use Sharding-JDBC with oracle
Fix getSchema NPE when use Sharding-JDBC with oracle
Fix Sharding-JDBC parse sql NPE in PostgreSQL
Fix Sharding-Proxy receive error response for PostgreSQL JDBC client
Fix Sharding-Proxy response number of update is 0 for PostgreSQL JDBC client
Fix Sharding-Proxy receive null for PostgreSQL column meta data
Fix Sharding-Scaling NPE in MySQL incremental task

Change Logs

MILESTONE

4.1.0

New Features

Support scaling for ShardingSphere (alpha version)
Move etcd registry center from experiment repo to apache repo
Upgrade to Java 8

Enhancement

Optimization for Sharding Parser with ANTLR Visitor improving the parsing performance of long SQL by 100%~1000%
Use multiple threads to load metadata for different data sources
Support allow.range.query.with.inline.sharding option
The docker of ShardingSphere-Proxy supports loading external lib packages
Support integration with Spring using @ShardingSphereTransactionType
Enhance ShardingDataSource to compatible with Seata in micro-service distribution transaction

Refactor

Remove leaf key generator

Bug Fixes

Fix an exception caused by using a CHAR/VARCHAR type column as an order by item
Refine DataTypeName syntax rules of all database dialects
Fix an exception caused by executing BEGIN using prepared statement of MySQL C API
Fix the problem that ALTER TABLE fails to execute when the field type of the table contains Integer/Double/BigDecimal
Fix the problem of the stop index dislocation of segment with alias
Fix the problem that overwriting SQL SELECT * FROM tbl WHERE encrypt_col = ? AND (sharding_key = ? OR sharding_key = ?) when using sharding + encrypt would throw StringIndexOutOfBoundsException
Fix the problem of incorrect decoding after AES encoding when using ShardingSphere-Proxy in Spring Boot
Fix a long-time problem of adding schema dynamically in ShardingSphere-UI

Change Logs

MILESTONE

4.0.1

Bug Fixes

Using guava cache to fix parsing deadlock.
Oracle insert SQL could not work in encrypt mode.
Proxy for PostgreSQL decode parameters error in all types except String.
COM_STM_EXECUTE of proxy for MySQL could not support sysbench.
None sharding strategy could not config in spring-boot.
WasNull field was wrong in GroupByStreamMergeResult.
Metadata.getColumns could not work in JDBC.
IN operator contains space and \n \t \r could not supported by parser.

Enhancement

Optimize antlr performance using two-stage parsing strategy.
Add class filter constructor to restrict the illegal class from YAML.

Change Logs

MILESTONE

4.0.0

API Changes

Change package and maven groupId form io.shardingsphere to org.apache.shardingsphere.
Adjust ShardingSphere-JDBC configuration API.
Adjust persist structure for registry center.

New Features

SQL92 Syntax available.
ShardingSphere-Proxy for PostgreSQL protocol available.
SQL 100% compatible if route to single data node.
Less-than(<), greater-than(>) and Less-than-equal(<=), greater-than-equal(>=) for sharding key operator available.
DISTINCT SQL syntax available.
Broadcast table available.
LEAF key generator available.
XA Transaction available, Atomikos, Narayana and Bitronix integrated.
BASE Transaction available, Seata integrated.
Data encrypt available.
Skywalking plugin available.
ShardingSphere-UI available, an orchestration management platform.

Enhancement

MariaDB supported.
Improve the compatibility of SQL parsing.
SELECT FOR UPDATE route to primary data source only.
Hint in ShardingSphere-Proxy available.
Make configuration of orchestration consistent between ShardingSphere-JDBC and ShardingSphere-Proxy.
Renew modified data sources only, not renew all the data sources.
Vibrate configurable for Snowflake key generator.

Bug Fixes

Improve the compatibility of JDBC Driver URL.
Delete statement with alias available.
Check and disable updating sharding column.
Fix wrong type of TINYINT and SMALLINT as INTEGER.

Change Logs

MILESTONE #3
MILESTONE #4
MILESTONE #5
MILESTONE #6
MILESTONE #7
MILESTONE #8
MILESTONE #9

4.0.0.RC3

New Features

ShardingSphere-UI, an orchestration management platform for ShardingSphere comes online.
Not only SQLs from MySQL, PostgreSQL, SQLServer, Oracle, but any SQL92 Syntax can be parsed correctly and used in ShardingSphere.

Enhancement

Support using less-than character(<) and greater-than character(>) for sharding data.
When primary and replica dataSources exist, support executing SELECT FOR UPDATE on primary data source.
Support hint in ShardingSphere-Proxy.
Finish parsing DAL syntax for MySQL.
Make configuration of orchestration compatible between ShardingSphere-JDBC and ShardingSphere-Proxy.

Bug Fixes

Through Bug fix, the feature of encryption becomes much stable and applicable.
Support delete statement with alias.
Check and disable updating sharding column.
Fix wrong type of TINYINT and SMALLINT as INTEGER.

Refactor

Rename optimized module to preprocessor module.
Decouple rewrite core module and sharding/encrypt features.

Change Logs

MILESTONE

4.0.0.RC2

API Changes

Optimize and delete API and configuration item of sharding logic index.
Update the API of encryption to support the encrypted and plain data coexistence.

New Features

Integration of Seata for distributed transaction.
User can do data encryption by using ShardingProxy.
User can use Leaf-segment generator to get distributed ID.
Support Skywalking plugin for application performance management.

Enhancement

Renew modified dataSources, not all the datasources to improve performance for configuration orchestration.
Improve the compatibility of SQL parsing.

Refactor

Remove DatabaseType enum, use dynamic SPI to load data source type.
The parse engine upgrade from the 2nd generation to 3rd.
The Refactoring of SQL rewriting module.

Change Logs

MILESTONE

4.0.0.RC1

Merge all change logs of version 3.1.0.M1, 3.1.0, 3.1.0.1 and 4.0.0.M1. First apache release.

API Changes

Adjust persist structure for orchestration's registry center.
Adjust ShardingSphere-JDBC configuration API.
Change package and maven groupId form io.shardingsphere to org.apache.shardingsphere.
Adjust spring-boot-starter.

New Features

XA Transaction available.
Data encrypt available.
Use PostgreSQL protocol access ShardingSphere-Proxy available.
DISTINCT SQL syntax available.
Broadcast table.
All SQL 100% compatible if route to single data node (MySQL Only).

Change Logs

MILESTONE #3
MILESTONE #4
MILESTONE #5
MILESTONE #6

3.0.0

Milestones

ShardingSphere-Proxy launch. Support the use of ShardingSphere in the form of database to support for MySQL CLI and GUI client

New Features

Core

ISSUE #290 Support batch INSERT
ISSUE #501 Support OR
ISSUE #980 Support DCL
ISSUE #1111 Support MySQL DAL

ShardingSphere-Proxy

ISSUE #902 Support XA transaction
ISSUE #916 Support authorization
ISSUE #936 Support registry center
ISSUE #1046 Support multiple logic databases

Enhancements

Core

ISSUE #373 Support order by ?
ISSUE #610 Route unicast for DQL without table
ISSUE #701 Caching parsed results to improve performance
ISSUE #773 Support sharding and autoincrement key of INSERT without column names
ISSUE #935 Use YAML instead of JSON to store configurations in registry center
ISSUE #1004 Properties can configure for sharding and replica query independent
ISSUE #1205 Execute engine enhancement

ShardingSphere-JDBC

ISSUE #652 Support Spring Boot Starter 2.X
ISSUE #702 Support $->{..} for inline expression
ISSUE #719 Support inject key generator objects by spring namespace
ISSUE #720 Support inject sharding algorithm objects by spring namespace

Sharding-Opentracing

ISSUE #1172 Opentracing enhancement

API changes

ISSUE #1153 Adjust the maven artifactId for orchestration module
ISSUE #1203 Adjust Spring namespace xsd for sharding and replica query
ISSUE #1289 Adjust hint API
ISSUE #1302 Refine package structure
ISSUE #1305 Deprecated and remove sharding-jdbc-transaction-parent module
ISSUE #1382 Remove type configuration in orchestration module

Bug Fixes

Core

ISSUE #569 Failed to parse SQL for Oracle when ROWNUM is not at end
ISSUE #628 Support data type jsonb for PostgreSQL
ISSUE #646 When aliases in SELECT ITEMS correspond to the real column names of GROUP BY or ORDER BY, there is no need to generate derived columns
ISSUE #806 NOT IN parse exception
ISSUE #827 Endless loop for bad SQL like SELECT * FROM table WHERE id IN ()
ISSUE #919 Inline expression with groovy may cause memory leak
ISSUE #993 Fail to parsing PostgreSQL due to the quotation
ISSUE #1015 Support SQL like SELECT id, COUNT(*) FROM table GROUP BY 1,2
ISSUE #1120 Derived columns of GROUP BY / ORDER BY appear in query result
ISSUE #1186 Dead lock may occur on MEMORY_STRICTLY mode when get connection on concurrency environment
ISSUE #1265 RoundRobinReplicaLoadBalanceAlgorithm throw an ArrayIndexOutOfBoundsException when AtomicInteger overflow

ShardingSphere-JDBC

ISSUE #372 Reuse PreparedStatement cause cache of route result do not clean
ISSUE #629 Support transaction isolation on JDBC
ISSUE #735 Unexpected replica datasource routing result when using Round-robin load-balance algorithm in Mybatis
ISSUE #1011 Can't resolve placeholder in Spring Boot YAML configuration

2.0.3

New Features

Core

ISSUE #600 Support TCL

Bug Fixes

Core

ISSUE #540 Support SQL that alias is the keyword
ISSUE #577 Support new line for YAML configuration

ShardingSphere-JDBC

ISSUE #522 Replica database does not need to execute the DDL for replica query

2.0.2

Enhancements

Core

ISSUE #475 Support CREATE INDEX
ISSUE #525 Support DROP INDEX

Bug Fixes

Core

ISSUE #521 ShardingProperties is invalid in YAML configuration
ISSUE #529 Table name capitalization cannot be queried
ISSUE #541 IS NOT NULL parse error
ISSUE #557 When GROUP BY and ORDER BY aliases are inconsistent, stream merging should be used
ISSUE #559 Support parsing numbers beginning with minus and decimal (e.g. -.12)
ISSUE #567 Add escape char for derived columns or alias when using MySQL

ShardingSphere-JDBC

ISSUE #520 Exception is no longer DuplicateKeyException when the unique key conflict

2.0.1

Enhancements

Core

ISSUE #490 Using rownum greater than or equal to or less than or equal to the result of paging is incorrect in Oracle
ISSUE #496 Logical table names in sharding configuration can be case sensitive
ISSUE #497 Close registry center gracefully

Bug Fixes

ShardingSphere-JDBC

ISSUE #489 Uses RuntimeBeanReference to prevent the creation of InnerBean on spring namespace
ISSUE #491 Can't close connection by ResultSet.getStatement().getConnection().close()

2.0.0

Milestones

API adjust. Brand new groupId and artifactId for Maven, package name and spring namespace name. Simplify and enhance API configuration, inline expression fully configuration support
Support spring-boot-starter of ShardingSphere-JDBC
Dynamic configuration. ZooKeeper and etcd can be used as registry to dynamically modify data sources and sharding configurations
Database orchestration. Fusing database access procedures to access databases and disable access to replica databases
ConfigMap support. Predefined metadata can be obtained in the sharding and replica query strategy
Tracking system support. You can view the invocation chain of ShardingSphere-JDBC through sky-walking and other Opentracing based APM systems

Enhancements

Core

ISSUE #386 Support SQL that does not contain table names, such as SELECT 1

ShardingSphere-JDBC

ISSUE #407 Support Hyphen properties for sharding-jdbc-spring-boot-starter
ISSUE #424 Providing SQL overall execution events

Bug Fixes

Core

ISSUE #387 Prevent errors from keywords process when '`' exists in function + column name
ISSUE #419 When SQL is rewritten, it does not determine whether alias is a keyword without the escape character, which results in SQL exception
ISSUE #464 SQL if the varchar type is not closed due to the absence of matching single quotes, and the next varchar in SQL is the wrong SQL of Chinese characters, it will lead to higher use of CPU

ShardingSphere-JDBC

ISSUE #394 Can't only close statement
ISSUE #398 Use Hint routing to shield case sensitivity
ISSUE #404 Sharding-jdbc's spring-boot-starter does not support HikariDataSource
ISSUE #436 Replica query, when the RoundRobin algorithm is configured from the database and MyBatis is used, it can only be routed to the same replica database
ISSUE #452 Sharding of DDL statements to more than one table causes a connection leak
ISSUE #472 Before Connection executes createStatement, it calls getMetaData first and then setAutoCommit can not take effective connection to the database that was created later

1.5.4.1

Bug Fixes

ISSUE #382 The query cannot be completed without sharding rule

1.5.4

Bug Fixes

ISSUE #356 In the Where condition of SQL, the REGEXP operator is compatible with non sharding columns
ISSUE #362 Replica query using PreparedStatement does not invoke the setParameter method to cause errors
ISSUE #370 Error in calling getGeneratedKeys using native self increment primary key
ISSUE #375 Data can not be obtained after paging second pages route to a single node
ISSUE #379 When Mybatis is used to call Connection.getMetaData (), the connection is not close correct

1.5.3

Enhancements

ISSUE #98 Replica query load balancing strategy support configuration
ISSUE #196 Replica query and sharding configuration independence

Bug Fixes

ISSUE #349 Incorrect function of ResultSet.wasNull causes null numeric type in DB to zero
ISSUE #351 Tables that are included in the default data source but not in TableRule configuration are not properly executed
ISSUE #353 In the Where condition of SQL, it is compatible with non sharding columns !=, !> and !< operator
ISSUE #354 In the Where condition of SQL, NOT operators are compatible with non-sharding columns

1.5.2

Milestones

The test engine of quality assurance, each SQL can run 60 test cases of different dimensions

Enhancements

ISSUE #335 Support the GROUP BY + custom function SQL
ISSUE #341 Support ORDER BY xxx NULLS FIRST | LAST statement of Oracle

Bug Fixes

ISSUE #334 Parsing ORDER BY with functions will resolve the following ASC and DESC to the name attribute of OrderItem
ISSUE #335 JOIN parsing is incorrect using the full name association of the table
ISSUE #346 Parsing table name error of DDL statement DROP TABLE IF EXISTS USER

1.5.1

New Features

ISSUE #314 Support DDL type SQL

Changes

ISSUE #327 Close sql.show configuration by default

Bug Fixes

ISSUE #308 Invalid return of database native GeneratedKey
ISSUE #309 ORDER BY and GROUP BY in sub-queries are not included in the analytic context
ISSUE #313 Support <> operator
ISSUE #317 Parameter of LIMIT can not be type of Long
ISSUE #320 SQL rewriting error of GROUP BY + LIMIT
ISSUE #323 Parsing ORDER BY + Aggregation Expression error

1.5.0

Milestones

The new SQL parsing module removes the dependence on Druid. We only need to parse the sharding context, and adopt a "semi understanding" concept for SQL to further improve performance and compatibility, and reduce code complexity
The new SQL rewrite module adds an optimized rewrite module
The new SQL merge module is rebuilt into 3 merging engines: streaming, memory and decorator

New Features

Support Oracle, SQLServer and PostgreSQL
Non functional sub-query support

Enhancements

ISSUE #256 Configurable display sharding execute to SQL log
ISSUE #291 Processing SQL use stream mode that contains only GroupBy

Changes

Simplify the distributed self increasing sequence. Each table is supported by simplifying a multiple self increasing sequence to a single table supporting only a single distributed self increasing sequence, and no longer supporting workerID settings through environment variables
Remove support for OR

Bug Fixes

ISSUE #239 LIMIT routes to multiple query result sets. If there is only one result set that is not empty, the result of paging is incorrect
ISSUE #263 Sharding and logical table configuration can be case insensitive
ISSUE #292 When the memory mode handles GROUP BY statement, if there is paging information, it needs to be rewritten
ISSUE #295 LIMIT 0 does not filter the result set according to paging restrictions

1.4.2

Enhancements

ISSUE #219 Thread performance optimization
ISSUE #215 Aggregated result set of stream sort StreamingOrderByReducerResultSet performance optimization
ISSUE #161 When the result sets are merged, heap sort can be used to improve performance

Bug Fixes

ISSUE #212 More meaningful hints for missing data source rules
ISSUE #214 table_name.column_name in (?,?) in WHERE can't parser expression
ISSUE #180 Batch execution of Update return inaccuracy
ISSUE #225 The last character of automatic generation of Id is not 0

1.4.1

Enhancements

ISSUE #191 Generating KeyGenerator of workerId based on IP of host
ISSUE #192 Get workerId's KeyGenerator based on HOSTNAME's digital suffix
ISSUE #210 Routing to single database and single table to remove supplementary SQL statement fragments

Bug Fixes

ISSUE #194 Some component exceptions in the close method of Connection, Statement, ResultSet and other interfaces cause the close method of another component to be not invoked
ISSUE #199 Sharding and reuse PreparedStatement object cause route error
ISSUE #201 Event transmission missing before batch operation execution
ISSUE #203 Merge events sent by the batch operation
ISSUE #209 Parallel execution of multiple limit queries leads to IndexOutOfBoundsException

1.4.0

Enhancements

Automatic generation key implementation, including

ISSUE #162 Implementation of distributed primary key algorithm
ISSUE #163 Acquisition of a self increasing sequence of JDBC interfaces
ISSUE #171 Sharding-jdbc-core coordination automatic generation sequence transformation
ISSUE #172 The configuration of YAML and Spring supports the self increasing sequence

Bug Fixes

ISSUE #176 The wasNull flag of AbstractMemoryResultSet is not reset in time

1.3.3

Enhancements

ISSUE #59 PreparedStatement can call the correct underlying set method according to the parameter type when setting parameters

Bug Fixes

ISSUE #149 When INSERT IGNORE INTO, if the data is duplicated, the value returned to -1 when ignored, and it should be returned to 0
ISSUE #118 In the same thread, DQL is executed first, then DML is executed, and DML operation is executed from the replica database
ISSUE #122 In cases where connections are not available (such as network interruption), transactions should be interrupted rather than retry
ISSUE #152 PreparedStatement's cache causes an array out of bound
ISSUE #150 With the latest SQLServer jdbc driver compatibility problem, Product Name should be changed from SQLServer to Microsoft SQL Server
ISSUE #166 Druid data source stat filter multi-thread error reporting should be added to database connection level synchronization

1.3.2

Enhancements

ISSUE #79 Optimizes limit for only one target table, does not modify limit offset

Bug Fixes

ISSUE #36 ShardingPreparedStatement cannot set parameters repeatedly
ISSUE #114 When ShardingPreparedStatement performs batch tasks, it repeatedly analyzes SQL and leads to OOM
ISSUE #33 According to the MySQL document, queries similar to limit 100 and -1 format are not supported
ISSUE #124 The return value of com.dangdang.ddframe.rdb.sharding.jdbc.adapter.AbstractStatementAdapter.getUpdateCount does not conform to the JDBC specification
ISSUE #141 Multi-thread executor parameter setting failure

1.3.1

Enhancements

ISSUE #91 Open support for Statement.getGeneratedKeys can return the original database self increase primary key
ISSUE #92 Query DQL statement event sending

Bug Fixes

ISSUE #89 Use replica query with sharding hint leads to conflict
ISSUE #95 Write operations in the same thread read from the primary database changed to the same thread and within the same connection

1.3.0

New Features

ISSUE #85 New feature for replica query

Enhancements

ISSUE #82 TableRule can import the dataSourceName attribute to specify the data source corresponding to the TableRule
ISSUE #88 Release restrictions on other databases, support standard SQL, do not support personalized paging statements

Bug Fixes

ISSUE #81 Associative table query uses OR query condition to resolve the result exceptions

1.2.1

Refactor

ISSUE #60 API adjust, remove ShardingDataSource, use factory instead
ISSUE #76 ShardingRule and TableRule change to Builder pattern
ISSUE #77 ShardingRule and TableRule change to Builder pattern

Enhancements

ISSUE #61 Add the logical table name to the ShardingValue class
ISSUE #66 Statement on the JDBC tier supports get/set MaxFieldSize, MaxRows and QueryTimeout
ISSUE #72 Batch inserts supporting select union all
ISSUE #78 Simplifying sharding only configuration, without configuring logical table and real table correspondence
ISSUE #80 Simplifying the configuration that does not sharding, specifying the default data source, do not need configure TableRule

Bug Fixes

ISSUE #63 No table name or table alias is added to the ORDER BY and GROUP BY derivation columns
ISSUE #65 Performance enhancement for parsing condition context
ISSUE #67 The soft transaction log cannot be deleted when routed to multiple tables
ISSUE #71 Routing single sharding key by OFFSET of LIMIT error
ISSUE #75 MemoryTransactionLogStorage retry times update concurrency problem

1.2.0

New Features

ISSUE #53 The relationship between the real table and the logical table is not configured, and the real table is dynamically calculated by the sharding algorithm
ISSUE #58 Soft transaction: the initial version of the best effort type

Refactor

ISSUE #49 Adjust the property configuration
ISSUE #51 Refactor of Hint interface

Bug Fixes

ISSUE #43 The yaml file contains Chinese, and the operating system mode is not UTF-8 encoding, resulting in yaml can not be parsed
ISSUE #48 Yaml file is not closed after reading
ISSUE #57 At the analytic level, we can identify subqueries to ensure that the behavior of supplementary columns can be accurately positioned

1.1.0

New Features

ISSUE #40 Support YAML configuration
ISSUE #41 Support Spring namespace configuration
ISSUE #42 Support inline expression configuration

Bug Fixes

ISSUE #25 The problem of repeated results will appear under the OR expression

1.0.1

Enhancements

ISSUE #39 Support the use of Hint method to register the key value to SQL routing

Bug Fixes

ISSUE #11 The count function returns incorrectly without aliases
ISSUE #13 The Insert statement does not write column names or write column names but column names do not contain sharding fields, occur broadcast route
ISSUE #16 For now a new connection pool is executed every time SQL is executed. Instead, each ShardingDataSource object should be changed to share a pool
ISSUE #18 When query Count, getObject() throws Exception: Unsupported data type: Object
ISSUE #19 In SUM and AVG functions, merger is not executed if aliases are not added, and null pointer exception fired if aliases are added
ISSUE #38 The compatibility between JPA and ShardingSphere-JDBC. JPA automatically add the column aliases of SELECT, resulting in ORDER BY obtaining ResultSet data only by aliases rather than column names

1.0.0

Initial version

Files

RELEASE-NOTES.md

Latest commit

History

RELEASE-NOTES.md

File metadata and controls

Release 5.5.2-SNAPSHOT

API Changes

New Features

Enhancements

Bug Fixes

Change Logs

Release 5.5.1

API Changes

New Features

Enhancements

Bug Fixes

Change Log

Release 5.5.0

API Changes

New Features

Enhancements

Bug Fixes

Change Log

Release 5.4.1

New Features

Enhancements

Bug Fixes

Change Log

Release 5.4.0

API Changes

New Features

Enhancements

Bug Fixes

Change Log

5.3.2

API Changes

New Features

Enhancements

Bug Fixes

Change Log

5.3.1

New Features

Enhancements

Bug Fixes

Change Log

5.3.0

API Changes

Enhancements

Bug Fix

Refactor

Change Log

5.2.1

New Feature

Enhancement

Bug Fix

API Changes

Change Log

5.2.0

New Feature

Enhancement

Bug Fix

API Changes

Refactor

Change Log

5.1.2

New Feature

Enhancement

Bug Fix

API Changes

Refactor

Change Log

5.1.1

New Feature

Enhancement

Refactor

Bug Fix

Change Log

5.1.0

New feature