Skip to content

Commit

Permalink
Update perf script and doc - better default run_tests (istio#4416)
Browse files Browse the repository at this point in the history 
* Update perf script and doc - better default run_tests

run_tests now run ing->f1 ing->f2 f1->f2 f2->f1 and saves the json and
starts report ui (if fortio is installed)

Simplify some of the setup and doc

Use 443 as the http port on the VM (should not get auto cleaned up)

Only run the through istio ingress test by default

Don’t delete everything by default (when not sourcing)

Fortio 0.8.x doesn’t need percentiles passed

* Typo

* Fix vm install to rebuild after go get with the -X taking action

* Fix ingress to f2 label

* More readme updates

* Fix up readme

* Change labels
  • Loading branch information
@ldemailly
ldemailly authored Mar 21, 2018
1 parent 6511bec commit 6c191a1
Show file tree
Hide file tree
Showing 2 changed files with 110 additions and 50 deletions.
92 changes: 64 additions & 28 deletions tools/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Istio Load Testing User Guide
### Introduction
This guide provides step-by-step instructions for using the `setup_perf_cluster.sh` load testing script.
The script deploys a GKE cluster, an Istio service mesh and a GCE VM. The script then runs [Fortio](https://github.com/istio/fortio/)
on the VM, 2 pods within the cluster (non-Istio) and 2 pods within the Istio mesh. The following diagram provides
additional details of the deployment:
The script deploys a GKE cluster, an Istio service mesh and a GCE VM. The script then runs [Fortio](https://github.com/istio/fortio/#fortio)
on the VM, 2 pods within the cluster (non-Istio) and 2 pods within the Istio mesh.

It should not be too difficult to adapt the script to other cloud providers or environments and contributions for additional automated setup are welcome.

The following diagram provides additional details of the deployment:

![Deployment Diagram](perf_setup.svg)

Expand All @@ -12,7 +15,7 @@ graphing results and as a backend echo server.

### Download a Release or Clone Istio

From release (either [official](https://github.com/istio/istio/releases) or [dailies](https://github.com/istio/istio/wiki/Daily-builds)):
It's recommended you use a release (either [official](https://github.com/istio/istio/releases) or [dailies](https://github.com/istio/istio/wiki/Daily-builds)):
```
curl -L https://git.io/getLatestIstio | sh - # or download the daily TGZ
```
Expand All @@ -22,11 +25,18 @@ From source:
$ git clone https://github.com/istio/istio.git && cd istio
```

### Install fortio locally

Optional but recommended:

If not already present from building from source,
Install fortio: `go get istio.io/fortio` (so you can run `fortio report` to visualize the results)

### Prepare the Istio Deployment Manifest and Istio Client

__Option A:__ (From release) Make sure `istioctl` is in your path is the one matching the downloaded release.

For instance, in `~/tmp/istio-0.5.0/` run:
For instance, in `~/tmp/istio-0.6.0/` run:
```
export PATH=`pwd`/bin:$PATH
# check 'which istioctl' and 'istioctl version' returns the correct version
Expand All @@ -37,21 +47,13 @@ $ ln -s $GOPATH/src/istio.io/istio/tools
```
If you want to get newer version of the tools, you can `rm -rf tools/` and do the symlink above to use your updated/newer script.

__Option B:__ (From source) Build the deployment manifest and `istioctl` binary:
```
$ ./install/updateVersion.sh # This step is only needed when using Istio from source.
```
Follow the steps in the [Developer Guide](https://github.com/istio/istio/blob/master/DEV-GUIDE.md) to build the `istioctl` binary. Make sure it does `istioctl kube-inject` producing the HUB/TAG you expect.
Make the kubectl binary executable.
```
$ chmod +x ./istioctl
```

Move the binary in to your PATH.
__Option B:__ (Advanced users, not recommended, from source) Build the deployment manifest and `istioctl` binary:
```
$ mv ./istioctl /usr/local/bin/istioctl
$ ./install/updateVersion.sh # This step is only needed when using Istio from source and may or may not work/need additional hub/tags/...
```

Follow the steps in the [Developer Guide](https://github.com/istio/istio/blob/master/DEV-GUIDE.md) to build the `istioctl` binary.
Make sure the binary is first in to your PATH.
Make sure it does `istioctl kube-inject` producing the HUB/TAG you expect.

### Set Your Google Cloud Credentials (optional/one time setup)
This is not necessary if you already have working `gcloud` commands and you
Expand All @@ -67,17 +69,19 @@ If you do not have a Google Cloud account, [set one up](https://cloud.google.com
The `setup_perf_cluster.sh` script can be customized. View the script and modify the default variables if needed.
For example, to update the default gcloud zone (us-east4-b):
```
$ ZONE=us-west1-a
$ export ZONE=us-west1-a
```
If you change either the `PROJECT` or the `ZONE`, make sure to run `update_gcp_opts` before calling the other functions.

The script tries to guess your `PROJECT` but it's safer to set it explicitly. (and use a new empty project if possible)

### Source the Script
```
# Set PROJECT and ZONE first then
$ source tools/setup_perf_cluster.sh
```
__Note:__ `setup_perf_cluster.sh` can be used as a script or sourced and functions called interactively.

Inside Google, you may need to rerun setup_vm_firewall multiple times.

### Run the Functions
Expand All @@ -104,16 +108,48 @@ istio-system istio-mixer-3192291716-psskv 3/3
istio-system istio-pilot-3663920167-4ns3g 2/2 Running 0 7m
<SNIP>
```
You can now run the performance tests, either from the command line or interactively using the UIs (see next section). For command lines there are a couple of examples in the `run_tests` function:

Make sure your ingress is ready:
```
$ run_tests
$ kubectl get ing -n istio
NAME HOSTS ADDRESS PORTS AGE
istio-ingress * 35.188.254.231 80 1m
```

The first test case uses the default loadbalancer and no Istio mesh or Istio Ingress Controller. The following command tells
You can now run the performance tests, either from the command line or interactively using the UIs (see next section).
For command lines there are a couple of examples in the `run_tests` functions, it will run 4 tests
and start fortio report so you can graph the result on [http://localhost:8080/](http://localhost:8080/)

```
$ run_tests
+++ VM Ip is 35.199.55.254 - visit (http on port 443 is not a typo:) http://35.199.55.254:443/fortio/
+++ In k8s fortio external ip: http://35.199.37.178:8080/fortio/
+++ In k8s non istio ingress: http://35.227.201.148/fortio/
+++ In k8s istio ingress: http://35.188.241.231/fortio1/fortio/ and fortio2
Using istio ingress to fortio1:
### Running: curl -s http://35.199.55.254:443/fortio/?labels=ingress+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://35.188.241.231/fortio1/echo | tee ing-to-f1.json | grep ActualQPS
"ActualQPS": 439.8723210634554,
Using istio ingress to fortio2:
### Running: curl -s http://35.199.55.254:443/fortio/?labels=ingress+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://35.188.241.231/fortio2/echo | tee ing-to-f2.json | grep ActualQPS
"ActualQPS": 540.2583184971915,
Using istio f1 to f2:
### Running: curl -s http://35.188.241.231/fortio1/fortio/?labels=f1+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv2:8080/echo | tee f1-to-f2.json | grep ActualQPS
"ActualQPS": 439.5027107832303,
Using istio f2 to f1:
### Running: curl -s http://35.188.241.231/fortio2/fortio/?labels=f2+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv1:8080/echo | tee f2-to-f1.json | grep ActualQPS
"ActualQPS": 330.49386695603846,
```
And then you will see:
![Single Graph Screen Shot](https://user-images.githubusercontent.com/3664595/37693480-231ac8c0-2c7d-11e8-9b3a-4e77a06f2d37.png)
![Multi Graph Screen Shot](https://user-images.githubusercontent.com/3664595/37693481-232efdf4-2c7d-11e8-92b4-8a6e088d3357.png)


For comparison and reference you can also run `run_fortio_test1` uses the default loadbalancer and no Istio mesh or Istio Ingress Controller.

The following command tells
Fortio on the VM to run a load test against the Fortio echo server running in the Kubernetes cluster:
```
### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_FORTIO_EXT_IP:8080/echo
### Running: curl http://$VM_URL/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_FORTIO_EXT_IP:8080/echo
```
The following arguments are passed to the Fortio server running on the GCE VM:

Expand All @@ -127,15 +163,15 @@ The following arguments are passed to the Fortio server running on the GCE VM:
| load=Start | Tells Fortio to be a load generator |
| url=http://$K8S_FORTIO_EXT_IP:8080/echo | The target to load test |

The second test case uses the Fortio Ingress with no Istio mesh and the same arguments as the first test:
You can also run `run_fortio_test2` which uses the Fortio Ingress with no Istio mesh and the same arguments as the first test:
```
### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$NON_ISTIO_INGRESS/echo
### Running: curl http://$VM_URL/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$NON_ISTIO_INGRESS/echo
```

The third test case uses the Istio Ingress with the same arguments as the first test. This is the test that performs load testing
The tests from `run_tests` uses the Istio Ingress with the same arguments. This is the test that performs load testing
of the Istio service mesh:
```
### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$ISTIO_INGRESS/fortio1/echo
### Running: curl http://$VM_URL/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$ISTIO_INGRESS/fortio1/echo
```
Compare the test results to understand the load differential between the 3 test cases.

Expand All @@ -145,7 +181,7 @@ Fortio provides a [Web UI](https://github.com/istio/fortio#webgraphical-ui) that
can be used to perform load testing. You can call the `get_ips` function to obtain Fortio endpoint information for further load testing:
```
$ get_ips
+++ VM Ip is $VM_IP - visit http://$VM_IP/fortio/
+++ VM Ip is $VM_IP - visit http://$VM_URL/
+++ In k8s fortio external ip: http://$EXTERNAL_IP:8080/fortio/
+++ In k8s non istio ingress: http://$NON_ISTIO_INGRESS_IP/fortio/
+++ In k8s istio ingress: http://$ISTIO_INGRESS_IP/fortio1/fortio/ and fortio2
Expand Down
68 changes: 46 additions & 22 deletions tools/setup_perf_cluster.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
# Sets up a cluster for perf testing - GCP/GKE
# tools/setup_perf_cluster.sh
# Notes:
# * See README.md
# * Make sure istioctl in your path is the one matching your release/crd/...
# * You need to update istio-auth.yaml or run from a release directory:
# source tools/setup_perf_cluster.sh
Expand All @@ -11,7 +12,8 @@
#
# This can be used as a script or sourced and functions called interactively
#

# The script must be run/sourced from the parent of the tools/ directory
#
PROJECT=${PROJECT:-$(gcloud config list --format 'value(core.project)' 2>/dev/null)}
ZONE=${ZONE:-us-east4-b}
CLUSTER_NAME=${CLUSTER_NAME:-istio-perf}
Expand Down Expand Up @@ -56,10 +58,16 @@ function update_gcp_opts() {
}

function Execute() {
echo "### Running:" "$@"
echo "### Running:" "$@" 1>&2
"$@"
}

function ExecuteEval() {
echo "### Running:" "$@" 1>&2
eval "$@"
}


function create_cluster() {
Execute gcloud container clusters create $CLUSTER_NAME $GCP_OPTS --machine-type=$MACHINE_TYPE --num-nodes=$NUM_NODES --no-enable-legacy-authorization
}
Expand All @@ -85,36 +93,35 @@ function delete_vm() {
}

function run_on_vm() {
echo "*** Remote run: \"$1\""
echo "*** Remote run: \"$1\"" 1>&2
Execute gcloud compute ssh $VM_NAME $GCP_OPTS --command "$1"
}

function setup_vm() {
Execute gcloud compute instances add-tags $VM_NAME $GCP_OPTS --tags http-server,allow-8080
Execute gcloud compute instances add-tags $VM_NAME $GCP_OPTS --tags https-server
run_on_vm '(sudo add-apt-repository ppa:gophers/archive > /dev/null && sudo apt-get update > /dev/null && sudo apt-get upgrade --no-install-recommends -y && sudo apt-get install --no-install-recommends -y golang-1.9-go make && mv .bashrc .bashrc.orig && (echo "export PATH=/usr/lib/go-1.9/bin:\$PATH:~/go/bin"; cat .bashrc.orig) > ~/.bashrc ) < /dev/null'
}

function setup_vm_firewall() {
Execute gcloud compute --project=$PROJECT firewall-rules create default-allow-http --network=default --action=ALLOW --rules=tcp:80 --source-ranges=0.0.0.0/0 --target-tags=http-server || true
Execute gcloud compute --project $PROJECT firewall-rules create allow-8080 --direction=INGRESS --action=ALLOW --rules=tcp:8080 --target-tags=port8080 || true
Execute gcloud compute --project=$PROJECT firewall-rules create default-allow-https --network=default --action=ALLOW --rules=tcp:443 --source-ranges=0.0.0.0/0 --target-tags=https-server || true
}

function delete_vm_firewall() {
Execute gcloud compute --project=$PROJECT firewall-rules delete default-allow-http -q
Execute gcloud compute --project $PROJECT firewall-rules delete allow-8080 -q
Execute gcloud compute --project=$PROJECT firewall-rules delete default-allow-https -q
}

function update_fortio_on_vm() {
run_on_vm 'go get istio.io/fortio && cd go/src/istio.io/fortio && git fetch && git checkout latest_release && make submodule-sync && go install -ldflags "-X istio.io/fortio/version.tag=$(git describe --tag --match v\*) -X istio.io/fortio/version.buildInfo=$(git rev-parse HEAD)" . && sudo setcap 'cap_net_bind_service=+ep' `which fortio` && fortio version'
run_on_vm 'go get istio.io/fortio && cd go/src/istio.io/fortio && git fetch && git checkout latest_release && make submodule-sync && go build -o ~/go/bin/fortio -ldflags "-X istio.io/fortio/version.tag=$(git describe --tag --match v\*) -X istio.io/fortio/version.buildInfo=$(git rev-parse HEAD)" . && sudo setcap 'cap_net_bind_service=+ep' `which fortio` && fortio version'
}

function run_fortio_on_vm() {
run_on_vm 'pkill fortio; nohup fortio server -http-port 80 > ~/fortio.log 2>&1 &'
run_on_vm 'pkill fortio; nohup fortio server -http-port 443 > ~/fortio.log 2>&1 &'
}

function get_vm_ip() {
VM_IP=$(gcloud compute instances describe $VM_NAME $GCP_OPTS |grep natIP|awk -F": " '{print $2}')
echo "+++ VM Ip is $VM_IP - visit http://$VM_IP/fortio/"
VM_URL="http://$VM_IP:443/fortio/"
echo "+++ VM Ip is $VM_IP - visit (http on port 443 is not a typo:) $VM_URL"
}

# assumes run from istio/ (or release) directory
Expand Down Expand Up @@ -214,16 +221,29 @@ function get_istio_ingress_ip() {

function run_fortio_test1() {
echo "Using default loadbalancer, no istio:"
Execute curl "http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$FORTIO_K8S_IP:8080/echo"
Execute curl "$VM_URL?json=on&save=on&qps=-1&t=30s&c=48&load=Start&url=http://$FORTIO_K8S_IP:8080/echo"
}
function run_fortio_test2() {
echo "Using default ingress, no istio:"
Execute curl "http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_INGRESS_IP/echo"
Execute curl "$VM_URL?json=on&save=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_INGRESS_IP/echo"
}
function run_fortio_test3() {
echo "Using istio ingress:"
Execute curl "http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$ISTIO_INGRESS_IP/fortio1/echo"
function run_fortio_test_istio_ingress1() {
echo "Using istio ingress to fortio1:"
ExecuteEval curl -s "$VM_URL?labels=ingress+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://$ISTIO_INGRESS_IP/fortio1/echo" \| tee ing-to-f1.json \| grep ActualQPS
}
function run_fortio_test_istio_ingress2() {
echo "Using istio ingress to fortio2:"
ExecuteEval curl -s "$VM_URL?labels=ingress+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://$ISTIO_INGRESS_IP/fortio2/echo" \| tee ing-to-f2.json \| grep ActualQPS
}
function run_fortio_test_istio_1_2() {
echo "Using istio f1 to f2:"
ExecuteEval curl -s "http://$ISTIO_INGRESS_IP/fortio1/fortio/?labels=f1+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv2:8080/echo" \| tee f1-to-f2.json \| grep ActualQPS
}
function run_fortio_test_istio_2_1() {
echo "Using istio f2 to f1:"
ExecuteEval curl -s "http://$ISTIO_INGRESS_IP/fortio2/fortio/?labels=f2+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv1:8080/echo" \| tee f2-to-f1.json \| grep ActualQPS
}


# Run canonical perf tests.
# The following parameters can be supplied:
Expand Down Expand Up @@ -294,7 +314,6 @@ function run_canonical_perf_test() {
;;
esac

PERCENTILES="50%2C+75%2C+90%2C+99%2C+99.9"
GRANULARITY="0.001"

LABELS="${LABEL}+${DRIVER}+${TARGET}+Q${QPS}+T${DURATION}+C${CLIENTS}"
Expand All @@ -308,7 +327,7 @@ function run_canonical_perf_test() {

echo "Running '${LABELS}' and storing results in ${OUT_FILE}"

URL="${DRIVER_URL}/?labels=${LABELS}&url=${TARGET_URL}&qps=${QPS}&t=${DURATION}&c=${CLIENTS}&p=${PERCENTILES}&r=${GRANULARITY}&json=on&save=on&load=Start"
URL="${DRIVER_URL}/?labels=${LABELS}&url=${TARGET_URL}&qps=${QPS}&t=${DURATION}&c=${CLIENTS}&r=${GRANULARITY}&json=on&save=on&load=Start"
#echo "URL: ${URL}"

curl -s "${URL}" -o "${OUT_FILE}"
Expand Down Expand Up @@ -366,9 +385,14 @@ function get_ips() {
function run_tests() {
update_gcp_opts
get_ips
run_fortio_test1
run_fortio_test2
run_fortio_test3
# run_fortio_test1
# run_fortio_test2
run_fortio_test_istio_ingress1
run_fortio_test_istio_ingress2
run_fortio_test_istio_1_2
run_fortio_test_istio_2_1
echo "Graph the results:"
fortio report &
}


Expand Down Expand Up @@ -397,5 +421,5 @@ if [[ $SOURCED == 0 ]]; then
#setup_vm_firewall
#get_ips
#install_istio_svc
delete_all
#delete_all
fi

0 comments on commit 6c191a1

Please sign in to comment.