This guide provides step-by-step instructions for using the setup_perf_cluster.sh load testing script.
The script deploys a GKE cluster, an Istio service mesh and a GCE VM. The script then runs Fortio
on the VM, 2 pods within the cluster (non-Istio) and 2 pods within the Istio mesh.
It should not be too difficult to adapt the script to other cloud providers or environments and contributions for additional automated setup are welcome.
The following diagram provides additional details of the deployment and the main 4 istio-ified scenarios (which are ran twice each, once at max qps and once at fixed 400qps):
The deployment provides a basis for Istio performance characterization. Fortio is used to perform load testing, graphing results and as a backend echo server.
It's recommended you use a release (either official or dailies):
curl -L https://git.io/getLatestIstio | sh - # or download the daily TGZ
From source:
$ git clone https://github.com/istio/istio.git && cd istio
Optional but recommended:
If not already present from building from source,
Install fortio: go get fortio.org/fortio (so you can run fortio report to visualize the results)
Option A: (From release) Make sure istioctl is in your path is the one matching the downloaded release.
For instance, in ~/tmp/istio-0.6.0/ run:
export PATH=`pwd`/bin:$PATH
# check 'which istioctl' and 'istioctl version' returns the correct version
For versions before 0.5.0 (the tools/ directory is now part of the release)
$ ln -s $GOPATH/src/istio.io/istio/tools
If you want to get newer version of the tools, you can rm -rf tools/ and do the symlink above to use your updated/newer script.
Option B: (Advanced users, not recommended, from source) Build the deployment manifest and istioctl binary:
$ ./install/updateVersion.sh # This step is only needed when using Istio from source and may or may not work/need additional hub/tags/...
Follow the steps in the Developer Guide to build the istioctl binary.
Make sure the binary is first in to your PATH.
Make sure it does istioctl kube-inject producing the HUB/TAG you expect.
This is not necessary if you already have working gcloud commands and you
did gcloud auth login at least once.
$ gcloud auth login
# Or
$ export GOOGLE_APPLICATION_CREDENTIALS=/my/gce/creds.json
If you do not have a Google Cloud account, set one up.
The setup_perf_cluster.sh script can be customized. View the script and modify the default variables if needed.
For example, to update the default gcloud zone (us-east4-b):
$ export ZONE=us-west1-a
If you change either the PROJECT or the ZONE, make sure to run update_gcp_opts before calling the other functions.
The script tries to guess your PROJECT but it's safer to set it explicitly. (and use a new empty project if possible)
# Set PROJECT and ZONE first then
$ source tools/setup_perf_cluster.sh
Note: setup_perf_cluster.sh can be used as a script or sourced and functions called interactively.
Inside Google, you may need to rerun setup_vm_firewall multiple times.
The output of source tools/setup_perf_cluster.sh provides a list of available functions or
you can view the functions from within the setup_perf_cluster.sh script. The most common workflow is:
$ setup_all
Obtaining latest ubuntu xenial image name... (takes a few seconds)...
<SNIP>
### Running: kubectl apply -n istio -f tools/cache_buster.yaml
The deployment is now complete. You can verify the deployment using standard kubectl commands:
$ kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
fortio fortio1-1966733334-xj5f6 1/1 Running 0 8m
fortio fortio2-3044850348-v5f74 1/1 Running 0 8m
istio-system istio-ca-1363003450-gvtmn 1/1 Running 0 7m
istio-system istio-ingress-1732553340-gv41r 1/1 Running 0 7m
istio-system istio-mixer-3192291716-psskv 3/3 Running 0 8m
istio-system istio-pilot-3663920167-4ns3g 2/2 Running 0 7m
<SNIP>
Make sure your ingress is ready:
$ kubectl get ing -n istio
NAME HOSTS ADDRESS PORTS AGE
istio-ingress * 35.188.254.231 80 1m
You can now run the performance tests, either from the command line or interactively using the UIs (see next section).
For command lines there are a couple of examples in the run_tests functions, it will run 4 tests
and start fortio report so you can graph the result on http://localhost:8080/
$ run_tests
+++ VM Ip is 35.199.55.254 - visit (http on port 443 is not a typo:) http://35.199.55.254:443/fortio/
+++ In k8s fortio external ip: http://35.199.37.178:8080/fortio/
+++ In k8s non istio ingress: http://35.227.201.148/fortio/
+++ In k8s istio ingress: http://35.188.241.231/fortio1/fortio/ and fortio2
Using istio ingress to fortio1:
### Running: curl -s http://35.199.55.254:443/fortio/?labels=ingress+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://35.188.241.231/fortio1/echo | tee ing-to-f1.json | grep ActualQPS
"ActualQPS": 439.8723210634554,
Using istio ingress to fortio2:
### Running: curl -s http://35.199.55.254:443/fortio/?labels=ingress+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://35.188.241.231/fortio2/echo | tee ing-to-f2.json | grep ActualQPS
"ActualQPS": 540.2583184971915,
Using istio f1 to f2:
### Running: curl -s http://35.188.241.231/fortio1/fortio/?labels=f1+to+f2\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv2:8080/echo | tee f1-to-f2.json | grep ActualQPS
"ActualQPS": 439.5027107832303,
Using istio f2 to f1:
### Running: curl -s http://35.188.241.231/fortio2/fortio/?labels=f2+to+f1\&json=on\&save=on\&qps=-1\&t=30s\&c=48\&load=Start\&url=http://echosrv1:8080/echo | tee f2-to-f1.json | grep ActualQPS
"ActualQPS": 330.49386695603846,
And then you will see:
For comparison and reference you can also run run_fortio_test1 uses the default loadbalancer and no Istio mesh or Istio Ingress Controller.
The following command tells Fortio on the VM to run a load test against the Fortio echo server running in the Kubernetes cluster:
### Running: curl http://$VM_URL/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_FORTIO_EXT_IP:8080/echo
The following arguments are passed to the Fortio server running on the GCE VM:
| Argument | Description |
|---|---|
| json=on | Sets output in json format |
| qps=-1 | Requested queries per second to "max" |
| t=30s | Requested duration to run load test |
| c=48 | Number of connections/goroutine/threads |
| qps=-1 | Requested queries per second to "max" |
| load=Start | Tells Fortio to be a load generator |
| url=http://$K8S_FORTIO_EXT_IP:8080/echo | The target to load test |
You can also run run_fortio_test2 which uses the Fortio Ingress with no Istio mesh and the same arguments as the first test:
### Running: curl http://$VM_URL/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$NON_ISTIO_INGRESS/echo
The tests from run_tests uses the Istio Ingress with the same arguments. This is the test that performs load testing
of the Istio service mesh:
### Running: curl http://$VM_URL/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$ISTIO_INGRESS/fortio1/echo
Compare the test results to understand the load differential between the 3 test cases.
Fortio provides a Web UI that
can be used to perform load testing. You can call the get_ips function to obtain Fortio endpoint information for further load testing:
$ get_ips
+++ VM Ip is $VM_IP - visit http://$VM_URL/
+++ In k8s fortio external ip: http://$EXTERNAL_IP:8080/fortio/
+++ In k8s non istio ingress: http://$NON_ISTIO_INGRESS_IP/fortio/
+++ In k8s istio ingress: http://$ISTIO_INGRESS_IP/fortio1/fortio/ and fortio2
Then visit http://$ISTIO_INGRESS_IP/fortio1/fortio/ or http://$ISTIO_INGRESS_IP/fortio2/fortio/ to generate a load to one of the Fortio echo servers:
echosrv1.istio.svc.cluster.local:8080 or echosrv2.istio.svc.cluster.local:8080.
Fortio provides additional load testing capabilities not covered by this document. For more information, refer to the Fortio documentation
There is a set of canonical tests in run_canonical_perf_tests.sh script that runs tests by changing parameters in
various dimensions:
- Number of clients
- QPS
- Cached v.s. non-cached
If you have a change that you think might affect performance, then you can run these tests to check the affects.
To establish a baseline, simply deploy a perf cluster using the instructions above. Then run
run_canonical_perf_tests.sh to establish the baseline. You will see output that looks like this:
> run_canonical_perf_tests.sh
+++ In k8s istio ingress: http://<ip>/fortio1/fortio/ and fortio2
Running 'canonical+fortio2+echo1+Q100+T1s+C16' and storing results in /tmp/istio_perf.cpxCcs/canonical_fortio2_echo1_Q100_T1s_C16.json
+++ In k8s istio ingress: http://<ip>/fortio1/fortio/ and fortio2
Running 'canonical+fortio2+echo1+Q400+T1s+C16' and storing results in /tmp/istio_perf.cpxCcs/canonical_fortio2_echo1_Q400_T1s_C16.json
...
You can check the Fortio UI of the respective drivers to see the results. Also, you can checkout the raw json files that gets stored in the temporary folder that is in the output above:
ls /tmp/istio_perf.cpxCcs/
canonical_fortio2_echo1_Q1000_T1s_C16.json canonical_fortio2_echo1_Q100_T1s_C20.json canonical_fortio2_echo1_Q1200_T1s_C24.json canonical_fortio2_echo1_Q400_T1s_C16.json
canonical_fortio2_echo1_Q1000_T1s_C20.json canonical_fortio2_echo1_Q100_T1s_C24.json canonical_fortio2_echo1_Q1600_T1s_C16.json canonical_fortio2_echo1_Q400_T1s_C20.json
canonical_fortio2_echo1_Q1000_T1s_C24.json canonical_fortio2_echo1_Q1200_T1s_C16.json canonical_fortio2_echo1_Q1600_T1s_C20.json canonical_fortio2_echo1_Q400_T1s_C24.json
canonical_fortio2_echo1_Q100_T1s_C16.json canonical_fortio2_echo1_Q1200_T1s_C20.json canonical_fortio2_echo1_Q1600_T1s_C24.json out.csv
You can run fortio report -data-dir /tmp/istio_perf.cpxCcs/ to see all the results and graph them/compare them by visiting http://localhost:8080
Alternatively, notice the out.csv file in the folder. This file contains all the data in the individual json files, and can be
imported into a spreadsheet:
> cat /tmp/istio_perf.cpxCcs/out.csv
Label,Driver,Target,qps,duration,clients,min,max,avg,p50,p75,p90,p99,p99.9
canonical,fortio2,echo1,1200,1s,16,0.00243703,0.059164527,0.0134183966225,0.0108966942149,0.01594375,0.02405,0.048646875,0.0575867009348
canonical,fortio2,echo1,1200,1s,24,0.003420898,0.086621239,0.0248239801951,0.0203296703297,0.0303731343284,0.0494375,0.080344304428,0.085993545542
...
To test the affects of your change, simply update your cluster with your binaries by following the Developer Guide and rerun the tests again. To ensure you're tracking the results of your changes correctly, you can explicitly specify a label:
# Notice the "mylabel" parameter below:
#
> run_canonical_perf_tests.sh mylabel
+++ In k8s istio ingress: http://<ip>/fortio1/fortio/ and fortio2
Running 'mylabel+fortio2+echo1+Q400+T1s+C16' and storing results in /tmp/istio_perf.0XuSIH/mylabel_fortio2_echo1_Q400_T1s_C16.json
+++ In k8s istio ingress: http://<ip>/fortio1/fortio/ and fortio2
...
After the run, you can find the new results both in Fortio UI, and also in the temporary folder:
> ls /tmp/istio_perf.0XuSIH/
mylabel_fortio2_echo1_Q1000_T1s_C16.json mylabel_fortio2_echo1_Q100_T1s_C20.json mylabel_fortio2_echo1_Q1200_T1s_C24.json mylabel_fortio2_echo1_Q400_T1s_C16.json
mylabel_fortio2_echo1_Q1000_T1s_C20.json mylabel_fortio2_echo1_Q100_T1s_C24.json mylabel_fortio2_echo1_Q1600_T1s_C16.json mylabel_fortio2_echo1_Q400_T1s_C20.json
mylabel_fortio2_echo1_Q1000_T1s_C24.json mylabel_fortio2_echo1_Q1200_T1s_C16.json mylabel_fortio2_echo1_Q1600_T1s_C20.json mylabel_fortio2_echo1_Q400_T1s_C24.json
mylabel_fortio2_echo1_Q100_T1s_C16.json mylabel_fortio2_echo1_Q1200_T1s_C20.json mylabel_fortio2_echo1_Q1600_T1s_C24.json out.csv
Use the delete_all function to remove everything done by the setup_all function. The following delete functions are used by
delete_all and may be called individually:
$ delete_istio
$ delete_cluster
$ delete_vm
$ delete_vm_firewall