HTTP/2 Tracing With eBPF Uprobes Demo

Use eBPF uprobes to trace HTTP/2 headers, without any changes to the application code.

What is this demo?

This demo provides the gRPC client and server, and the uprobe tracer for Observing HTTP/2 Traffic is Hard, but eBPF Can Help blog post. Follow the instructions provided below to trace HTTP/2 message headers with eBPF uprobes.

Prerequisites

• This demo only works on Linux. The code was tested on Ubuntu 20.04.3 LTS with 5.4 kernel.
• Install BCC.
• All of the code is written in Golang.
• Various Go packages might need to be installed, follow the directives to install them when any of the go build commands failed.

Trace HTTP/2 headers with wireshark

First launch wireshark, and then launch gRPC client & server. When launching wireshark, you'll need to set the port for HTTP/2 protocol, follow the menu Edit > Preferences > Protocols > HTTP2, set HTTP2 TCP port to 50051, as shown in the picture below:

# Launch wireshark with display filter on the gRPC server's listening port 50051.
sudo wireshark -Y "tcp.port == 50051 && http2" -i any -k

# Build and launch gRPC server.
go build -o /tmp/grpc_server server/main.go && /tmp/grpc_server

# Build and launch gRPC client.
go build -o /tmp/grpc_client client/main.go && /tmp/grpc_client --count 10

Click the packet in the capture window containing the text 200 OK:

The decoded HTTP/2 headers can be seen as below:

Now launch wireshark after launching gRPC client & server, such that wireshark only capture the traffic in the middle of the connection. This time, wireshark cannot decode the headers as shown in the screenshot below.

go build -o /tmp/grpc_server server/main.go && /tmp/grpc_server
go build -o /tmp/grpc_client client/main.go && /tmp/grpc_client --count 20
sudo wireshark -Y "tcp.port == 50051 && http2" -i any -k

Trace HTTP/2 headers with uprobe tracer

Run the commands below to see the uprobe tracer tracing the HTTP/2 messages from the gRPC server to client:

go build -o /tmp/grpc_server server/main.go && /tmp/grpc_server

# Build and launch the uprobe tracer.
# If failed, see instructions in the Development section below to change gobpf version.
go build -o /tmp/uprobe_trace ./uprobe_trace && \
 sudo -E /tmp/uprobe_trace --binary=/tmp/grpc_server

go build -o /tmp/grpc_client client/main.go && /tmp/grpc_client --count 10

The output of the uprobe tracer looks like:

Development

• If you want to modify the gRPC protobuf, you'll need to install: protocol buffer compiler and go protobuf plugin.
• If you have modified the gRPC protobuf file, you'll need to run the following command to update the generated go source files:

  protoc --go_out=. --go-grpc_out=. --go-grpc_opt=require_unimplemented_servers=false proto/greet.proto
  

• Change gobpf version in go.mod if go build failed when building uprobe tracer with error like below:

  github.com/iovisor/gobpf/bcc ../../../pkg/mod/github.com/iovisor/[email protected]/bcc/module.go:261:33:
  not enough arguments in call to _C2func_bpf_attach_uprobe
        have (_Ctype_int, uint32, *_Ctype_char, *_Ctype_char, _Ctype_ulong, _Ctype_int)
        want (_Ctype_int, uint32, *_Ctype_char, *_Ctype_char, _Ctype_ulong, _Ctype_int, _Ctype_uint)
  

  In go.mod, uncomment any of the gobpf versions listed to replace the exiting one. Until the build succeeds. Gobpf is a wrapper of BCC's libbpf APIs, but gobpf is not released through the same channel as BCC, so the build failed when the versions are not compatible.

Bugs & Features

Feel free to file a bug or an issue for a feature request. You can also join our Slack community.