Skip to content

Commit

Permalink
update private url
Browse files Browse the repository at this point in the history
  • Loading branch information
yufeng committed Apr 22, 2020
1 parent 623a3d2 commit edcdca0
Show file tree
Hide file tree
Showing 16 changed files with 48 additions and 48 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sharingan

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/didichuxing/sharingan/blob/master/LICENSE)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/didi/sharingan/blob/master/LICENSE)

Sharingan,中文名:写轮眼,是漫画《火影忍者》中的一种瞳术,具有复制、幻术等能力,在幻术世界里,一切因素包括时间,地点,质量等,都由施术者掌控。

Expand Down Expand Up @@ -35,7 +35,7 @@ Sharingan(写轮眼)是一个基于golang的流量录制回放工具,录

```shell
# Step1: 下载sharingan项目
$ git clone https://github.com/didichuxing/sharingan.git
$ git clone https://github.com/didi/sharingan.git
$ cd sharingan

# Step2: 使用定制版golang,以go1.13为例
Expand Down Expand Up @@ -83,7 +83,7 @@ $ 页面选择要回放的流量点执行 # 内置提前录制好的3
* 修改golang源码,对外暴露Hook接口。「所有改动通过官方测试用例」
* 提供API串联不同goroutine之间的网络调用。「常见的http、mysql、redis流量都不需要特别设置」
* 提供单独的agent筛选流量、控制比例。
* 更多参考:[流量录制实现原理](https://github.com/didichuxing/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%BD%95%E5%88%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)
* 更多参考:[流量录制实现原理](https://github.com/didi/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%BD%95%E5%88%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)

### 3.4、回放方案

Expand All @@ -92,7 +92,7 @@ $ 页面选择要回放的流量点执行 # 内置提前录制好的3
* 重置时间:将程序执行时间回放到录制时刻,消除时间带来的干扰。
* 去除噪音:提供API可以将已知的噪音流量去掉,如:traceID,每次请求本来就不一样。
* 常见协议解析:会解析http、mysql、redis、thrift等协议,方便diff对比。
* 更多参考:[流量回放实现原理](https://github.com/didichuxing/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%9B%9E%E6%94%BE%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)
* 更多参考:[流量回放实现原理](https://github.com/didi/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%9B%9E%E6%94%BE%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)

## 四、效果展示

Expand All @@ -107,7 +107,7 @@ $ 页面选择要回放的流量点执行 # 内置提前录制好的3

#### 4.1.2、批量回放

批量回放的并发度默认是10,可通过增加[-parallel](https://github.com/didichuxing/sharingan/blob/master/replayer-agent/control.sh#L160)参数修改。
批量回放的并发度默认是10,可通过增加[-parallel](https://github.com/didi/sharingan/blob/master/replayer-agent/control.sh#L160)参数修改。
![main](./doc/images/replayer_p.png)

### 4.2、覆盖率报告
Expand Down
18 changes: 9 additions & 9 deletions doc/recorder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

### 1、使用定制版golang【必须】

* 目前支持go1.10 ~ go1.14,参考:[golang安装](https://github.com/didichuxing/sharingan-go/tree/recorder)
* 目前支持go1.10 ~ go1.14,参考:[golang安装](https://github.com/didi/sharingan-go/tree/recorder)

```shell
# go1.13使用示例,github域名不太稳定,失败可以换成其它方式,参考golang安装
Expand All @@ -25,11 +25,11 @@ export PATH=$GOROOT/bin:$PATH
import _ "github.com/didichuxing/sharingan"
```

* 参考:[example](https://github.com/didichuxing/sharingan/blob/master/example/main.go)
* 参考:[example](https://github.com/didi/sharingan/blob/master/example/main.go)

#### 2.2、特殊设置【非必须】

* 背景:使用goroutine对外网络调用时,需要显示的传递goroutineID,否则链路无法串联起来,详见:[链路追踪](https://github.com/didichuxing/sharingan/wiki/%E9%93%BE%E8%B7%AF%E8%BF%BD%E8%B8%AA)
* 背景:使用goroutine对外网络调用时,需要显示的传递goroutineID,否则链路无法串联起来,详见:[链路追踪](https://github.com/didi/sharingan/wiki/%E9%93%BE%E8%B7%AF%E8%BF%BD%E8%B8%AA)
* tip1:定时任务的流量不会录制,涉及到的代码不需要修改。「我们只录制对外http接口整个流程的流量」
* tip2:http请求主流程不等待结果的异步网络调用,不需要设置。「我们只录制http请求阶段确定的流量」
* tip3:常见的第三方包「http、redis、mysql、thrift等」,经测试都可以正常进行录制,不需要修改。
Expand All @@ -48,7 +48,7 @@ import "github.com/didichuxing/sharingan"
+ }(sharingan.GetCurrentGoRoutineID())
```

* 参考:[example](https://github.com/didichuxing/sharingan/blob/master/example/recorder/main.go)
* 参考:[example](https://github.com/didi/sharingan/blob/master/example/recorder/main.go)

### 3、指定tag编译【必须】

Expand All @@ -68,7 +68,7 @@ cd /path/to/your/project && ./$project // 使用上一步编译生成二进
```

* 录制成功标志:指定文件/tmp/recorder.log存在流量,一条流量占一行。
* 线下流量回放,参考:[本地回放](https://github.com/didichuxing/sharingan/blob/master/doc/replayer/replayer-local.md)
* 线下流量回放,参考:[本地回放](https://github.com/didi/sharingan/blob/master/doc/replayer/replayer-local.md)

#### 4.2、线上流量录制「流量发送给recorder-agent」

Expand All @@ -78,9 +78,9 @@ export RECORDER_TO_AGENT="127.0.0.1:9003" // 指定agent地址,确保recorde
cd /path/to/your/project && ./$project // 使用上一步编译生成二进制文件启动项目
```

* 启动recorder-agent:[recorder-agent](https://github.com/didichuxing/sharingan/blob/master/doc/recorder/recorder-agent.md)
* 启动recorder-agent:[recorder-agent](https://github.com/didi/sharingan/blob/master/doc/recorder/recorder-agent.md)
* 录制成功标志:/path/to/your/sharingan/recorder-agent/log/recorder.log存在流量,一条流量占一行。
* 线上流量回放,参考:[流量回放](https://github.com/didichuxing/sharingan/tree/master/doc/replayer)
* 线上流量回放,参考:[流量回放](https://github.com/didi/sharingan/tree/master/doc/replayer)

## 二、最佳实践【**线上流量录制,强烈推荐**

Expand Down Expand Up @@ -120,13 +120,13 @@ fi

## 三、录制原理

[录制原理详解](https://github.com/didichuxing/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%BD%95%E5%88%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)
[录制原理详解](https://github.com/didi/sharingan/wiki/%E6%B5%81%E9%87%8F%E5%BD%95%E5%88%B6%E5%AE%9E%E7%8E%B0%E5%8E%9F%E7%90%86)

## 四、常见问题

### 1、对正常服务影响

* [压测详情](https://github.com/didichuxing/sharingan/blob/master/doc/recorder/hey.md)
* [压测详情](https://github.com/didi/sharingan/blob/master/doc/recorder/hey.md)
* 建议只在一台机器上开启录制,其它机器不受任何影响。【降低影响面,参考最佳实现】
* 建议添加pprof监控,观察服务健康状况,逐步调高录制机器权重。【有问题可以下线机器,或者在机器上用正常bin重启】

Expand Down
2 changes: 1 addition & 1 deletion doc/recorder/recorder-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
### 2、编译 && 启动

```shell
git clone https://github.com/didichuxing/sharingan.git
git clone https://github.com/didi/sharingan.git
cd sharingan/recorder-agent
go build
nohup ./recorder-agent >> run.log 2>&1 &
Expand Down
4 changes: 2 additions & 2 deletions doc/replayer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@

首先,确保本地存在go环境(官方或定制版均可,且版本无要求), 并根据本地go版本选择go mod或glide来安装sharingan的依赖。
```shell script
git clone https://github.com/didichuxing/sharingan.git
git clone https://github.com/didi/sharingan.git
&& cd sharingan && go mod download # 低版本go执行 glide install
```
然后编译并启动Replayer-Agent:
Expand All @@ -51,7 +51,7 @@ cd ./replayer-agent && go build && nohup ./replayer-agent >> run.log 2>&1 &

> 需要使用定制的golang,并通过指定tag来编译 引入回放包的SUT代码。
首先,配置定制版golang环境,目前支持go1.10 ~ go1.14,参考:[golang安装](https://github.com/didichuxing/sharingan-go/tree/recorder)
首先,配置定制版golang环境,目前支持go1.10 ~ go1.14,参考:[golang安装](https://github.com/didi/sharingan-go/tree/recorder)
```shell script
curl https://raw.githubusercontent.com/didichuxing/sharingan-go/recorder/install/go1.13 | sh
&& export GOROOT=/tmp/recorder-go1.13
Expand Down
2 changes: 1 addition & 1 deletion doc/replayer/replayer-sut.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ sh sut_replayer.sh build cov //覆盖率回放
```text
curl: (7) Failed to connect to raw.githubusercontent.com port 443: Connection refused
```
可能是raw.githubusercontent.com域名不通,建议配置个代理;或者 配置hosts (151.101.108.133 raw.githubusercontent.com);再或者 自己安装go后,重新执行脚本构建。可参考: [sharingan-go安装](https://github.com/didichuxing/sharingan-go/tree/recorder)
可能是raw.githubusercontent.com域名不通,建议配置个代理;或者 配置hosts (151.101.108.133 raw.githubusercontent.com);再或者 自己安装go后,重新执行脚本构建。可参考: [sharingan-go安装](https://github.com/didi/sharingan-go/tree/recorder)

<br>

Expand Down
10 changes: 5 additions & 5 deletions doc/replayer/replayer-theory.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Golang的流量回放 主要基于 [sharingan/replayer](../../replayer) 包 及
2. 如何在录制的流量里,选择最合适的Outbound返回给SUT。

## 二、Outbound请求拦截点
看过 [流量录制拦截点选择](https://github.com/didichuxing/sharingan/wiki/%E6%8B%A6%E6%88%AA%E7%82%B9%E9%80%89%E6%8B%A9) 可知,golang的录制是在语言标准库层面做的拦截。
看过 [流量录制拦截点选择](https://github.com/didi/sharingan/wiki/%E6%8B%A6%E6%88%AA%E7%82%B9%E9%80%89%E6%8B%A9) 可知,golang的录制是在语言标准库层面做的拦截。

同理,流量回放也是在语言标准库层面做的拦截。修改系统调用 [syscall.Connect](https://github.com/golang/go/blob/release-branch.go1.10/src/syscall/syscall_unix.go#L225) 方法,将原本的socket地址sa替换为Mock Server地址。
```shell script
Expand All @@ -35,7 +35,7 @@ func Connect(fd int, sa Sockaddr) (err error) {
[sharingan/replayer](../../replayer) 包拦截了SUT的Outbound请求,将其转发给Replayer-Agent的Mock Server。
![replay-theory](https://github.com/didichuxing/sharingan/raw/master/doc/wiki/images/replay_theory.png)
![replay-theory](https://github.com/didi/sharingan/raw/master/doc/wiki/images/replay_theory.png)
如上图,回放剧本的传递过程如下:
1. 用户浏览Web Server首页(:8998),筛选一个流量,点击回放
Expand Down Expand Up @@ -66,7 +66,7 @@ Mock Server有个非常重要的工作,就是匹配Outbound请求,这直接
下面简化下匹配算法核心步骤:
![replay-match](https://github.com/didichuxing/sharingan/raw/master/doc/wiki/images/replay_match.png)
![replay-match](https://github.com/didi/sharingan/raw/master/doc/wiki/images/replay_match.png)
a) 匹配当前请求时,若上一次匹配成功,则从上一次匹配成功的请求(lastMatchedIndex)的下一个请求开始匹配,否则就从第一个请求开始匹配;
Expand All @@ -91,7 +91,7 @@ c) 判断权重最高的Outbound请求是否达标(权重是否超过阈值)
### 1. 并发原理
![replay_parallel](https://github.com/didichuxing/sharingan/raw/master/doc/wiki/images/replay_parallel.png)
![replay_parallel](https://github.com/didi/sharingan/raw/master/doc/wiki/images/replay_parallel.png)
如上图,基本思路如下:
Expand Down Expand Up @@ -134,7 +134,7 @@ func (c *conn) Write(b []byte) (int, error) {
SUT每接收Web Server的一个http请求,都会开启一个goroutine,同时,一般都会在这同一个goroutine内完成下游请求的调用,最终将http响应返回给Web Server。所以,针对这种情况,在同一个goroutine内维护sessionID是很简单的事。
但 对于通过新起一个goroutine来调用下游请求的SUT,维护sessionID需要基于录制里讲到的 [链路追踪](https://github.com/didichuxing/sharingan/wiki/%E9%93%BE%E8%B7%AF%E8%BF%BD%E8%B8%AA) 原理,基于定制版的golang实现。
但 对于通过新起一个goroutine来调用下游请求的SUT,维护sessionID需要基于录制里讲到的 [链路追踪](https://github.com/didi/sharingan/wiki/%E9%93%BE%E8%B7%AF%E8%BF%BD%E8%B8%AA) 原理,基于定制版的golang实现。
```shell script
// GetCurrentGoRoutineId get RoutineId in case delegate
Expand Down
6 changes: 3 additions & 3 deletions example/replayer/sut_replayer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ local_ip="127.0.0.1"
local_port="8998"

# set github about go
git_replayer_agent='https://github.com/didichuxing/sharingan/raw/master/replayer-agent'
git_replayer_agent='https://github.com/didi/sharingan/raw/master/replayer-agent'
git_go='https://raw.githubusercontent.com/didichuxing/sharingan-go/recorder'
install_go_md='https://github.com/didichuxing/sharingan-go/tree/recorder'
install_go_md='https://github.com/didi/sharingan-go/tree/recorder'
VERSION="go1.13"

function install_go() {
Expand Down Expand Up @@ -70,7 +70,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
6 changes: 3 additions & 3 deletions example/replayer/sut_replayer_glide.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ local_ip="127.0.0.1"
local_port="8998"

# set github about go
git_replayer_agent='https://github.com/didichuxing/sharingan/raw/master/replayer-agent'
git_replayer_agent='https://github.com/didi/sharingan/raw/master/replayer-agent'
git_go='https://raw.githubusercontent.com/didichuxing/sharingan-go/recorder'
install_go_md='https://github.com/didichuxing/sharingan-go/tree/recorder'
install_go_md='https://github.com/didi/sharingan-go/tree/recorder'
VERSION="go1.10"

function install_go() {
Expand Down Expand Up @@ -70,7 +70,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
6 changes: 3 additions & 3 deletions example/replayer/sut_replayer_gomod.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ local_ip="127.0.0.1"
local_port="8998"

# set github about go
git_replayer_agent='https://github.com/didichuxing/sharingan/raw/master/replayer-agent'
git_replayer_agent='https://github.com/didi/sharingan/raw/master/replayer-agent'
git_go='https://raw.githubusercontent.com/didichuxing/sharingan-go/recorder'
install_go_md='https://github.com/didichuxing/sharingan-go/tree/recorder'
install_go_md='https://github.com/didi/sharingan-go/tree/recorder'
VERSION="go1.13"

function install_go() {
Expand Down Expand Up @@ -70,7 +70,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
2 changes: 1 addition & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function install(){
VERSION=$1

# base set
GIT_URL="https://github.com/didichuxing/sharingan-go"
GIT_URL="https://github.com/didi/sharingan-go"

# param
uname=`uname`
Expand Down
2 changes: 1 addition & 1 deletion replayer-agent/control.sh
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
2 changes: 1 addition & 1 deletion replayer-agent/control_glide.sh
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
2 changes: 1 addition & 1 deletion replayer-agent/control_gomod.sh
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ function install_go() {
# install go
function install_go_tar() {
# get go url
releases_go='https://github.com/didichuxing/sharingan-go/releases/download/'
releases_go='https://github.com/didi/sharingan-go/releases/download/'
if [[ "$OSTYPE" =~ ^darwin.* ]]; then
goSuffix="$VERSION.recorder/$VERSION.darwin-amd64.tar.gz"
elif [[ "$OSTYPE" =~ ^linux.* ]]; then
Expand Down
2 changes: 1 addition & 1 deletion replayer-agent/public/js/auto.js
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ const TimeSleep = 0
}
this.$notify({
title: '提示',
message: msg+'</br><a href="/coverage?project='+ Global.Project +'" target="_blank"><button type="button" class="el-button el-button--text"><span>覆盖率报告</span></button></a>&nbsp;&nbsp;<a href="https://github.com/didichuxing/sharingan/blob/master/doc/replayer/replayer-codecov.md" target="_blank"><button type="button" class="el-button el-button--text"><span>覆盖率使用手册</span></button></a>',
message: msg+'</br><a href="/coverage?project='+ Global.Project +'" target="_blank"><button type="button" class="el-button el-button--text"><span>覆盖率报告</span></button></a>&nbsp;&nbsp;<a href="https://github.com/didi/sharingan/blob/master/doc/replayer/replayer-codecov.md" target="_blank"><button type="button" class="el-button el-button--text"><span>覆盖率使用手册</span></button></a>',
type: 'success',
dangerouslyUseHTMLString: true,
offset: 100,
Expand Down
Loading

0 comments on commit edcdca0

Please sign in to comment.