Integration tests do not clean up properly

[ccressent]

Integration tests do not appear to clean up properly after being run, leading to various issues.

Current Behavior

• After certain integration tests run, about 130MB of etcd state is left in a /tmp/sensu* directory. This can slowly fill up /tmp and lead to strange system behaviour (Slack crash, test failures, ...).

• After certain integration tests run, a subsequent test may be unable to open a new gRPC connection, leading to test timeouts and/or failure.

Expected Behavior

• Tests clean up after themselves, leaving nothing behind and returning to the same state as before they were run.
• Back to back integration tests work instead of timing out.

Steps to Reproduce (for bugs)

The keepalived package can be used to reproduce both issues:

$ cd backend/keepalived
$ GOCACHE=off go test -tags=integration -v .
$ ls /tmp/sensu*
$ du -hs /tmp/sensu*

In backend/keepalived/integration_test.go, duplicate the TestKeepaliveMonitor() function so that 2 identical integration tests run back to back.

$ GOCACHE=off go test -tags=integration -v .

After the first test run, the second, identical test will be stuck with repeating error messages:

WARNING: 2019/01/16 10:32:35 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:43657: connect: connection refused"; Reconnecting to {127.0.0.1:43657 0  <nil>}

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
• Installation method: git
• Operating System: Arch Linux, kernel 4.19