Skip to content

Commit

Permalink
Merge branch 'dev-yj-forinner'
Browse files Browse the repository at this point in the history
  • Loading branch information
yangjing committed Sep 11, 2020
2 parents 11cc443 + 8d3d535 commit ee8c7ad
Show file tree
Hide file tree
Showing 7 changed files with 70 additions and 7 deletions.
8 changes: 4 additions & 4 deletions doc/replayer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
<br>

## 一、名词解释
* **SUT**: System Under Test, 在这里特指被测业务服务。目前仅支持对外提供HTTP接口的SUT回放。
* **Inbound**: 指SUT对外提供的HTTP接口的请求和响应,即Inbound Request/Response。
* **Outbound**: 指SUT提供的http接口内调用的下游请求和响应,即Outboud Request/Response。目前支持的Outbound协议有 MYSQL、REDIS、HTTP、Binary Thrift、Compact Thrift。
* **SUT**: System Under Test, 在这里特指被测业务服务。
* **Inbound**: 指SUT对外提供的 HTTP/Thrift 接口的请求和响应,即Inbound Request/Response。目前支持对外提供 HTTP/Thrift 接口的SUT回放
* **Outbound**: 指SUT提供的 http/thrift 接口内调用的下游请求和响应,即Outboud Request/Response。目前支持的Outbound协议有 MYSQL、REDIS、HTTP、Binary Thrift、Compact Thrift。
* **流量**: 指在TCP层对SUT录制的Inbound请求/响应 和 对应的Outbound请求/响应。
* **回放**: 基于录制的一条流量,Replayer-Agent根据Inbound Request构造HTTP Request并对SUT发起请求;其中SUT的Outbound请求会发送到Mock Server进行流量匹配并返回Outbound响应;最后,Replayer-Agent将收到的SUT HTTP Response与Inbound Response做对比,给出回放结果。
* **回放**: 基于录制的一条流量,Replayer-Agent根据Inbound Request构造 HTTP/Thrift Request并对SUT发起请求;其中SUT的Outbound请求会发送到Mock Server进行流量匹配并返回Outbound响应;最后,Replayer-Agent将收到的SUT HTTP/Thrift Response与Inbound Response做对比,给出回放结果。
* **Replayer-Agent**: 包括回放过程的Web Server和Mock Server。其中Web Server默认监听8998端口,主要负责流量搜索,回放请求的构造和发起,以及回放结果的对比和展示。
* **Mock Server**: 默认监听**3515**端口,主要负责接收SUT的下游请求,然后与回放流量的Outbound请求匹配,将最匹配的Outbound响应返回SUT。
* **噪音**: 即在Inbound Response对比和Outbound Request匹配过程里,出现的不影响回放结果和匹配度的diff字段,如时间戳。Replayer-Agent回放结果页支持上报噪音,方便再次回放时精确回放结果。
Expand Down
9 changes: 9 additions & 0 deletions doc/replayer/guide/troubleshoot.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,15 @@ sharingan已经在滴滴内部服务了一批go业务模块,现将各业务模

* **解决**:参考 [覆盖率统计回放-2. 配置并启动SUT](../replayer-codecov.md#2-配置并启动SUT) 修改编译命令,设置-coverpkg为指定目录

<br>

##### 7. 本地回放时,首页搜索不到流量
* **现象**:按 [本地流量回放](../replayer-local.md) 修改好相关配置后,启动replayer-agent,但首页搜索不到流量。

* **原因**:本地流量文件,单条过大,超过100K

* **解决**:请按 [回放配置文件-flow](../replayer-conf.md#6-flow) 修改line_max_size即可。

<br>
<br>

Expand Down
8 changes: 8 additions & 0 deletions doc/replayer/replayer-conf.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,3 +91,11 @@ default字段非空时,支持不同业务部门的模块读取自不同的es
> 注意:
>
> 使用Biz配置的前提是, default字段非空。所以,如果想使用部门字段地址,请确认default字段非空。
##### 6. [flow]

标识流量的一些基本信息。

* un_gzip = 0 # 回放的流量是否做了gzip解压;默认0未解压;1解压(mock服务会删除header Content-Encoding: gzip)
* line_max_size = 512 # 本地回放时,读取本地的每条流量最大512K, 不设置默认100K;如需更大,可以按需修改。

4 changes: 4 additions & 0 deletions replayer-agent/conf/app.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ level = "info"
[outbound]
server_addr = "127.0.0.1:3515" # mock server addr

[flow]
un_gzip = 0 # 回放的流量是否做了gzip解压;默认0未解压;1解压(mock服务会删除header Content-Encoding: gzip)
line_max_size = 512 # 本地回放时,每条流量最大512K, 不设置默认100K

[http_api]
# dsl_get = "http://{{your_domain}}/dsl?project=%s"
# dsl_push = "http://{{your_domain}}/dsl"
Expand Down
30 changes: 30 additions & 0 deletions replayer-agent/logic/outbound/connection.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ import (
"bytes"
"context"
"errors"
"fmt"
"net"
"regexp"
"strconv"
"time"

"github.com/didi/sharingan/replayer-agent/common/handlers/conf"
"github.com/didi/sharingan/replayer-agent/common/handlers/tlog"
"github.com/didi/sharingan/replayer-agent/model/pool"
"github.com/didi/sharingan/replayer-agent/model/recording"
Expand Down Expand Up @@ -200,6 +202,13 @@ func (cs *ConnState) match(ctx context.Context, request []byte) error {
}
response := callOutbound.MatchedResponse
response = bytes.Replace(response, []byte("Connection: keep-alive\r\n"), []byte("Connection: close\r\n"), -1)

// for unzip flow
ungzip := conf.Handler.GetInt("flow.un_gzip")
if ungzip == 1 {
response = resetContentLength(ctx, response)
}

_, err := cs.conn.Write(response)
if err != nil {
tlog.Handler.Errorf(ctx, tlog.DebugTag, "errmsg=write back response failed||err=%s", err)
Expand Down Expand Up @@ -231,6 +240,27 @@ func (cs *ConnState) match(ctx context.Context, request []byte) error {
return nil
}

// resetContentLength 对于录制时做了gzip解压的流量,重新计算Content-Length
func resetContentLength(ctx context.Context, data []byte) []byte {
var contents [][]byte

if !bytes.Contains(data, []byte("Content-Encoding: gzip\r\n")) {
return data
}

bodySplit := []byte("\r\n\r\n")
if contents = bytes.Split(data, bodySplit); len(contents) != 2 {
return data
}

// 如果流量录制时,对流量做了gzip解压,则header Content-Length值相对会偏小
newLength := fmt.Sprintf("Content-Length: %d\r\n", len(contents[1])) // 计算body长度
data = contentLengthRegex.ReplaceAll(data, []byte(newLength))
data = bytes.Replace(data, []byte("Content-Encoding: gzip\r\n"), []byte(""), -1)

return data
}

// applySimulation
func applySimulation(ctx context.Context, sim func(ctx context.Context, request []byte) []byte,
request []byte, conn net.Conn, callOutbound *replaying.CallOutbound) error {
Expand Down
7 changes: 4 additions & 3 deletions replayer-agent/logic/worker/replayer.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,11 @@ func (r *Replayer) ReplaySessionDoreplay(ctx context.Context, session *replaying

{
// add header
traceHeader := fmt.Sprintf("\r\nSharingan-Replayer-Traceid: %s", traceID)
timeHeader := fmt.Sprintf("\r\nSharingan-Replayer-Time: %d", session.CallFromInbound.OccurredAt)
traceHeader := fmt.Sprintf("Sharingan-Replayer-Traceid: %s\r\n", traceID)
timeHeader := fmt.Sprintf("Sharingan-Replayer-Time: %d\r\n", session.CallFromInbound.OccurredAt)
s := bytes.Split(request, []byte("\r\n"))
s[0] = append(s[0], []byte(traceHeader+timeHeader)...)
// add at the front for the thrift inbound
s[0] = append([]byte(traceHeader+timeHeader), s[0]...)
request = bytes.Join(s, []byte("\r\n"))
}

Expand Down
11 changes: 11 additions & 0 deletions replayer-agent/utils/helper/fileops.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"sync"
"time"

"github.com/didi/sharingan/replayer-agent/common/handlers/conf"
"github.com/didi/sharingan/replayer-agent/common/handlers/tlog"
)

Expand Down Expand Up @@ -89,6 +90,16 @@ func ReadLines(confFile string) ([]string, error) {

var contents []string
scanner := bufio.NewScanner(file)

maxCapacity := 100 * 1024
// for local replay, when flow is too big
lineSize := conf.Handler.GetInt("flow.line_max_size")
if lineSize > 0 {
maxCapacity = lineSize * 1024
}
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)

for scanner.Scan() {
contents = append(contents, scanner.Text())
}
Expand Down

0 comments on commit ee8c7ad

Please sign in to comment.