forked from facebook/mysql-5.6
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Bug#26414532: MYSQLRPLSYNC ERRORS OUT BECAUSE SLAVE IS USING
--SUPER-READ-ONLY OPTION Problem: ======== After upgrading MySQL server from 5.7.17 to 5.7.18 'mysqlrplsync' utility tool fails with following error. ERROR: Query failed. 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement When this occurs, the Slave_SQL_Running on the slave server stops running and needs to be manually started again. The slave server does have the super-read-only option set to true. When running MySQL 5.7.17 this wasn't an issue. Analysis: ======== When replication information repository is set to 'TABLE' few replication specific administration commands listed below, will try to update the repository tables. 1) START SLAVE 2) STOP SLAVE 3) CHANGE MASTER 4) RESET SLAVE 5) RESET MASTER 6) SET GLOBAL GTID_PURGED 7) FLUSH BINARY LOGS 8) 'gtid_executed' TABLE compression thread. First three commands were fixed as part of Bug#22097534: SUPER_READ_ONLY ABORTS STOP SLAVE IF RELAY_LOG_INFO_REPOSITORY=TABLE, DBG CRASH The reset of the commands were fixed as part of Bug#22857926: ASSERTION `! IS_SET()' AT SQL_ERROR.CC:38 IN READ_ONLY MODE FOR MANY RPL CMDS. During the Bug#22857926 fix, the previous Bug#22097534 fix specific changes were reverted a new approach was taken. The new approach used a flag named 'ignore_global_read_lock' whose intention is to ignore the read only option and allow commits to replication specific system tables. If this flag is set then the additional check for 'read only' option is not done. But the above fix works fine only when 'AUTOCOMMIT=ON'. In the case of AUTOCOMMIT=OFF the 'ignore_global_read_lock' flag is not set. Hence commands fail to ignore read_only restriction. STOP SLAVE command behavior explained in detail: STOP SLAVE command is an IMPLICIT COMMIT command, i.e parser will append an explicit BEGIN and COMMIT for this command. BEGIN STOP SLAVE; COMMIT; STOP SLAVE command will try to stop both the applier thread and receiver thread and update their positions in 'slave_relay_log_info' and 'slave_master_info' tables. Case: AUTOCOMMIT=ON. 1) BEGIN 2) Update 'slave_relay_log_info' table and commit as a real transaction. System_table_access::close_table will invoke ha_commit_trans with 'ignore_global_read_lock=true'. Since autocommit is 'ON' each statement will do a complete commit and clean up the transaction context. 3) Update 'slave_master_info' also does the same. 4) COMMIT. COMMIT will invoke 'trans_commit_implicit'. It will check if there any active transactions which needs to be committed implicitly. Since step 2-3 are real commit transactions this final commit has nothing to do. It will exit without proceeding further. Case: AUTOCOMMIT=OFF. 1) BEGIN 2) Update 'slave_relay_log_info' table. This transaction is considered as a multi statement transaction mode, it will not commit on its own. It will wait for final COMMIT execution. 3) Update 'slave_master_info' also does the same. 4) COMMIT. COMMIT will invoke 'trans_commit_implicit'. It will check if there any active transactions which needs to be committed . Since there are active transaction sessions from step 2-3 'ha_commit_trans' will be called as shown below. 'trans_commit_implicit' will invoke ha_commit_trans (thd,all=true,ignore_global_read_lock=false) This will not pass the read_only check and the command will fail. This issue happens for commands that update info tables other than 'gtid_executed' table. In case of 'gtid_executed' table irrespective of autocommit variable all the transactions are considered as 'real' transactions and they will do complete commit. Fix: === Introduced a new variable, to flag the sql command under execution to ignore the read_only/super_read_only. If the variable is set then the command is allowed to execute even though read_only/super_read_only is enabled otherwise command is blocked.
- Loading branch information
Sujatha Sivakumar
committed
Aug 24, 2017
1 parent
1c582ac
commit dee592d
Showing
16 changed files
with
437 additions
and
147 deletions.
There are no files selected for viewing
134 changes: 134 additions & 0 deletions
134
mysql-test/extra/rpl_tests/rpl_ignore_super_read_only.test
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| # ==== Purpose ==== | ||
| # | ||
| # Verify that we permit replication operations with super_read_only=ON. | ||
| # | ||
| # ==== Implementation ==== | ||
| # | ||
| # 1) SET GLOBAL super_read_only to 1 on slave | ||
| # 2) Execute 'STOP SLAVE' command | ||
| # 3) Execute 'CHANGE REPLICATION FILTER' command | ||
| # 4) Execute 'CHANGE MASTER TO' command | ||
| # 5) Execute 'RESET SLAVE FOR CHANNEL <channel_name>' command | ||
| # 7) Execute 'RESET SLAVE ALL FOR CHANNEL <channel_name>' command | ||
| # 8) Execute 'RESET SLAVE' command | ||
| # 9) Execute 'START SLAVE' command | ||
| # 10) Execute 'RESET MASTER' command (with GTID_MODE=ON will assert) | ||
| # 11) Execute 'FLUSH BINARY LOGS' command (with GTID_MODE=ON will assert) | ||
| # 12) Execute 'SET GLOBAL gtid_purged='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1'' | ||
| # command | ||
| # 14) Check that replication works fine | ||
| # 15) Restore GLOBAL super_read_only to 0 | ||
| # | ||
| # ==== References ==== | ||
| # | ||
| # Bug#22097534: SUPER_READ_ONLY ABORTS STOP SLAVE IF | ||
| # RELAY_LOG_INFO_REPOSITORY=TABLE, DBG CRASH | ||
| # Bug#22857926: ASSERTION `! IS_SET()' AT SQL_ERROR.CC:38 IN READ_ONLY MODE FOR | ||
| # MANY RPL CMDS. | ||
| # Bug#26414532: MYSQLRPLSYNC ERRORS OUT BECAUSE SLAVE IS USING --SUPER-READ-ONLY | ||
| # OPTION | ||
| # | ||
|
|
||
| if (!$autocommit_opt) | ||
| { | ||
| --die !!!ERROR IN TEST: you must set $autocommit_opt | ||
| } | ||
|
|
||
| --source include/rpl_connection_master.inc | ||
| CREATE TABLE t1(a INT); | ||
| INSERT INTO t1 VALUES(1); | ||
| DROP TABLE t1; | ||
| --source include/sync_slave_sql_with_master.inc | ||
|
|
||
| SET @saved_value_super= @@GLOBAL.super_read_only; | ||
| SET @saved_value= @@GLOBAL.read_only; | ||
| SET @saved_autocommit= @@GLOBAL.autocommit; | ||
| SET GLOBAL super_read_only= 1; | ||
| --eval SET AUTOCOMMIT= $autocommit_opt | ||
| SHOW VARIABLES like '%autocommit%'; | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case1: STOP SLAVE command | ||
| --echo #################################################################### | ||
| --source include/stop_slave.inc | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case2: CHANGE REPLICATION FILTER command | ||
| --echo #################################################################### | ||
| CHANGE REPLICATION FILTER REPLICATE_DO_DB=(test); | ||
| CHANGE REPLICATION FILTER REPLICATE_DO_DB=(); | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case3: CHANGE MASTER command | ||
| --echo #################################################################### | ||
| CHANGE MASTER TO MASTER_CONNECT_RETRY=20; | ||
| CHANGE MASTER TO MASTER_HOST='dummy' FOR CHANNEL 'aaa'; | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case4: RESET SLAVE FOR CHANNEL/RESET SLAVE ALL/RESET SLAVE | ||
| --echo # commands | ||
| --echo #################################################################### | ||
| RESET SLAVE FOR CHANNEL 'aaa'; | ||
| RESET SLAVE ALL FOR CHANNEL 'aaa'; | ||
| RESET SLAVE; | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case5: START SLAVE command | ||
| --echo #################################################################### | ||
| --source include/start_slave.inc | ||
| --source include/rpl_connection_master.inc | ||
| --source include/sync_slave_sql_with_master.inc | ||
|
|
||
| # Check that mysql.gtid_executed table is empty. | ||
| --let $assert_text= mysql.gtid_executed table must have zero records | ||
| --let $assert_cond= [SELECT count(*) FROM mysql.gtid_executed] = 0 | ||
| --source include/assert.inc | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case6: FLUSH BINARY LOGS command | ||
| --echo #################################################################### | ||
| # FLUSH BINARY LOGS asserts when it is trying to update gtid_executed table | ||
| # during binary log rotation. | ||
| FLUSH BINARY LOGS; | ||
|
|
||
| # Check that an entry is updated in gtid_executed table without causing any | ||
| # assert. | ||
| --disable_query_log | ||
| if ( `SELECT @@GLOBAL.GTID_MODE = "ON"` ) | ||
| { | ||
| --let $table=mysql.gtid_executed | ||
| --let $count=1 | ||
| --source include/wait_until_rows_count.inc | ||
| } | ||
| --enable_query_log | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case7: RESET MASTER command | ||
| --echo #################################################################### | ||
| RESET MASTER; | ||
|
|
||
| --echo #################################################################### | ||
| --echo # Test Case8: SET GLOBAL GTID_PURGED command | ||
| --echo #################################################################### | ||
| --let $master_uuid=`SELECT @@SERVER_UUID` | ||
| --replace_result $master_uuid MASTER_UUID | ||
| --eval SET GLOBAL gtid_purged= '$master_uuid:1' | ||
|
|
||
|
|
||
| --echo "Clean up" | ||
| # Reset slave to clean state | ||
| SET AUTOCOMMIT= @saved_autocommit; | ||
| SET GLOBAL super_read_only= @saved_value_super; | ||
| SET GLOBAL read_only= @saved_value; | ||
|
|
||
| # Start fresh slave | ||
| --source include/stop_slave.inc | ||
| --replace_result $MASTER_MYPORT MASTER_PORT | ||
| --replace_column 2 #### | ||
| --eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=$MASTER_MYPORT, MASTER_USER='root'; | ||
| --source include/start_slave.inc | ||
|
|
||
| --source include/rpl_connection_master.inc | ||
| --source include/sync_slave_sql_with_master.inc | ||
|
|
||
| --source include/rpl_reset.inc |
28 changes: 28 additions & 0 deletions
28
mysql-test/suite/rpl/r/rpl_gtid_compress_thd_ignore_super_read_only.result
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| include/master-slave.inc | ||
| Warnings: | ||
| Note #### Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| [connection master] | ||
| [connection slave] | ||
| call mtr.add_suppression("You need to use --log-bin*"); | ||
| SET @saved_value_super= @@GLOBAL.super_read_only; | ||
| SET @saved_value= @@GLOBAL.read_only; | ||
| SET GLOBAL super_read_only= 1; | ||
| [connection master] | ||
| CREATE TABLE t1(a INT) ENGINE=INNODB; | ||
| INSERT INTO t1 VALUES(1); | ||
| INSERT INTO t1 VALUES(2); | ||
| INSERT INTO t1 VALUES(3); | ||
| INSERT INTO t1 VALUES(4); | ||
| DROP TABLE t1; | ||
| include/sync_slave_sql_with_master.inc | ||
| # | ||
| # Verify that gtid_executed table has MASTER_UUID:1:6 | ||
| # | ||
| SELECT * FROM mysql.gtid_executed; | ||
| source_uuid interval_start interval_end | ||
| MASTER_UUID 1 6 | ||
| "Clean up" | ||
| SET GLOBAL super_read_only= @saved_value_super; | ||
| SET GLOBAL read_only= @saved_value; | ||
| include/rpl_end.inc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
mysql-test/suite/rpl/t/rpl_gtid_compress_thd_ignore_super_read_only-slave.opt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| --skip-log-bin --gtid_executed_compression_period=5 |
64 changes: 64 additions & 0 deletions
64
mysql-test/suite/rpl/t/rpl_gtid_compress_thd_ignore_super_read_only.test
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # ==== Purpose ==== | ||
| # | ||
| # Verify that we permit replication operations with super_read_only=ON. | ||
| # | ||
| # ==== Implementation ==== | ||
| # | ||
| # 1) Have gtid_mode on and binlog less slave. | ||
| # 2) set super_read_only=on on slave server. | ||
| # 3) Set gtid_executed_compression_period=5 so that compression thread wakes | ||
| # up after 5 gtid transactions are executed. | ||
| # 4) Execute 6 transaction on master so that it triggers compression thread | ||
| # on slave to update mysql.gtid_executed table. | ||
| # 5) Verify that Slave server is up and running. | ||
| # 6) Check that replication works fine | ||
| # 7) Restore GLOBAL super_read_only to 0 | ||
| # | ||
| # ==== References ==== | ||
| # | ||
| # Bug#22097534: SUPER_READ_ONLY ABORTS STOP SLAVE IF | ||
| # RELAY_LOG_INFO_REPOSITORY=TABLE, DBG CRASH | ||
| # Bug#22857926: ASSERTION `! IS_SET()' AT SQL_ERROR.CC:38 IN READ_ONLY MODE | ||
| # FOR MANY RPL CMDS. | ||
| # Bug#26414532: MYSQLRPLSYNC ERRORS OUT BECAUSE SLAVE IS USING | ||
| # --SUPER-READ-ONLY OPTION | ||
| # | ||
|
|
||
| # Test in this file only makes sense in standard replication, | ||
| # so it is skipped in group replication. | ||
| --source include/not_group_replication_plugin.inc | ||
| # Gtids are mandatory for the test. | ||
| --source include/have_gtid.inc | ||
| # Test in this file is binlog format agnostic, thus no need | ||
| # to rerun it for every format. | ||
| --source include/have_binlog_format_row.inc | ||
| --source include/master-slave.inc | ||
|
|
||
| --source include/rpl_connection_slave.inc | ||
| call mtr.add_suppression("You need to use --log-bin*"); | ||
| SET @saved_value_super= @@GLOBAL.super_read_only; | ||
| SET @saved_value= @@GLOBAL.read_only; | ||
| SET GLOBAL super_read_only= 1; | ||
|
|
||
| --source include/rpl_connection_master.inc | ||
| --let $master_uuid= `SELECT @@GLOBAL.SERVER_UUID` | ||
| CREATE TABLE t1(a INT) ENGINE=INNODB; | ||
| INSERT INTO t1 VALUES(1); | ||
| INSERT INTO t1 VALUES(2); | ||
| INSERT INTO t1 VALUES(3); | ||
| INSERT INTO t1 VALUES(4); | ||
| DROP TABLE t1; | ||
| --source include/sync_slave_sql_with_master.inc | ||
|
|
||
| --echo # | ||
| --echo # Verify that gtid_executed table has MASTER_UUID:1:6 | ||
| --echo # | ||
| --replace_result $master_uuid MASTER_UUID | ||
| SELECT * FROM mysql.gtid_executed; | ||
|
|
||
| --echo "Clean up" | ||
| # Reset slave to clean state | ||
| SET GLOBAL super_read_only= @saved_value_super; | ||
| SET GLOBAL read_only= @saved_value; | ||
|
|
||
| --source include/rpl_end.inc |
Oops, something went wrong.