rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true #9216

zhongxiaoyi738 · 2025-02-28T07:33:19Z

zhongxiaoyi738
Feb 28, 2025

[rocketmq@broker-01-68d4859b4d-b7gq7 bin]$ more /opt/conf/broker-01.conf
brokerClusterName = DefaultCluster
brokerName=broker-01
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n1
sendMessageThreadPoolNums=16

[rocketmq@broker-02-84cc5b59fb-d2n2r bin]$ more /opt/conf/broker-02.conf
brokerClusterName = DefaultCluster
brokerName=broker-02
listenPort=31866
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n2
sendMessageThreadPoolNums=16

[rocketmq@broker-03-584bb8879d-5b9tq bin]$ more /opt/conf/broker-03.conf
brokerClusterName = DefaultCluster
brokerName=broker-03
listenPort=31869
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n3
sendMessageThreadPoolNums=16

broker-02
#Topic #Group #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 0 133 133 0 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 1 121 121 0 2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 2 135 134 1 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 3 121 121 0 2025-01-19 14:43:35

broker-03
#Topic #Group #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 0 133 133 0 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 1 121 121 0 2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 2 135 135 0 2025-02-27 16:39:54
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 3 121 121 0 2025-01-19 14:43:35

francisoliverlee · 2025-03-03T02:47:11Z

francisoliverlee
Mar 3, 2025
Collaborator

any error logs?
DLeger Model not suggest, try Controller Mode

6 replies

zhongxiaoyi738 Mar 10, 2025
Author

brokerClusterName = DefaultCluster
brokerName=broker-01
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
dLegerSelfId=n1
sendMessageThreadPoolNums=16

1、brokerName、dLegerGroup不一致
2、dLegerPeers 与 dLegerSelfId 不一致
请问这2个不一致会有影响么？
感觉BrokerController.handleSlaveSynchronize这个定时任务没有启动哦，jstack打印出来BrokerController只有一个线程

francisoliverlee Mar 11, 2025
Collaborator

1、brokerName、dLegerGroup不一致
不影响

2、dLegerPeers 与 dLegerSelfId 不一致
dLegerSelfId是dLegerPeers的一部分。具体看这里

check peer state change log in codes

zhongxiaoyi738 Mar 11, 2025
Author

hi，我改了一下rocketMQ的源码（加了很多调试日志），新增加的日志如下：（masterAddrBak=null）

2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize role=SLAVE
2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize ready slaveSyncFuture
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----handleSlaveSynchronize run syncAll
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncConsumerOffset masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncDelayOffset masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncSubscriptionGroupConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

zhongxiaoyi738 Mar 11, 2025
Author

dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
dLegerSelfId=n1

我们这样配置，对么？
如果用域名，应该如何配置

francisoliverlee Mar 12, 2025
Collaborator

满足这个逻辑即可

zhongxiaoyi738 · 2025-03-11T08:26:47Z

zhongxiaoyi738
Mar 11, 2025
Author

补充，从节点的日志摘录（刚刚启动时有报错，但是后续恢复正常了）

[rocketmq@broker-02-7dcffbb679-jmxlb rocketmqlogs]$ more broker_default.log
2025-03-11 16:10:46 INFO main - instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@458c1321

2025-03-11 16:11:46 INFO voteInvokeExecutor_3 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:47 INFO QuorumAckChecker-n2 - [n2][CANDIDATE] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:49 INFO StateMaintainer - [n2] [PARSE_VOTE_RESULT] cost=2600 term=24 memberNum=3 allNum=2 acceptedNum=0 notReadyTermNum=0 biggerLedgerNum=0 alreadyHasLeader=false maxTerm=24 result=WAIT_TO_REVOTE
2025-03-11 16:11:49 INFO NettyServerPublicExecutor_3 - [n2][ChangeRoleToFollower] from term: 24 leaderId: n1 and currTerm: 24
2025-03-11 16:11:49 ERROR voteInvokeExecutor_4 - Send vote request failed, info[group=RaftNode00,term=24,code=200,local=null,remote=n3,leader=n2]
org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to broker-03.rocketmq-flow.svc.cluster.local:40911 failed
at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeAsync(NettyRemotingClient.java:535) ~[rocketmq-remoting-4.9.2.jar:4.9.2]
at io.openmessaging.storage.dledger.DLedgerRpcNettyService.lambda$vote$3(DLedgerRpcNettyService.java:164) ~[dledger-0.2.2.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_322]
2025-03-11 16:11:49 INFO voteInvokeExecutor_4 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:50 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:53 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
后面一直打印的日志，跟最后一行一致

0 replies

zhongxiaoyi738 · 2025-03-11T08:35:27Z

zhongxiaoyi738
Mar 11, 2025
Author

补充问题
rocketMQ4.9.2，开启enableDLegerCommitLog=true
topic、consumerOffset等元数据，是如何同步的？

2 replies

francisoliverlee Mar 12, 2025
Collaborator

DLeger模式只涉及commitlog是raft的，元数据应该还是一样同步。不过貌似有个bug，dleger模式不能为null，不过我没有验证，你可以打日志check下

rocketmq/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java

Line 1139 in c840b20

this.slaveSynchronize.setMasterAddr(null);

zhongxiaoyi738 Mar 12, 2025
Author

请问你提到的这个bug有修复么？

我加了调试日志
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true #9216

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true #9216

zhongxiaoyi738 Feb 28, 2025

must be unique

must be unique

must be unique

Replies: 3 comments · 8 replies

francisoliverlee Mar 3, 2025 Collaborator

zhongxiaoyi738 Mar 10, 2025 Author

francisoliverlee Mar 11, 2025 Collaborator

zhongxiaoyi738 Mar 11, 2025 Author

zhongxiaoyi738 Mar 11, 2025 Author

francisoliverlee Mar 12, 2025 Collaborator

zhongxiaoyi738 Mar 11, 2025 Author

zhongxiaoyi738 Mar 11, 2025 Author

francisoliverlee Mar 12, 2025 Collaborator

zhongxiaoyi738 Mar 12, 2025 Author

zhongxiaoyi738
Feb 28, 2025

Replies: 3 comments 8 replies

francisoliverlee
Mar 3, 2025
Collaborator

zhongxiaoyi738 Mar 10, 2025
Author

francisoliverlee Mar 11, 2025
Collaborator

zhongxiaoyi738 Mar 11, 2025
Author

zhongxiaoyi738 Mar 11, 2025
Author

francisoliverlee Mar 12, 2025
Collaborator

zhongxiaoyi738
Mar 11, 2025
Author

zhongxiaoyi738
Mar 11, 2025
Author

francisoliverlee Mar 12, 2025
Collaborator

zhongxiaoyi738 Mar 12, 2025
Author