Skip to content

Commit

Permalink
add more feature ideas to the readme
Browse files Browse the repository at this point in the history
  • Loading branch information
raphi011 committed Nov 18, 2024
1 parent 7302663 commit 6c7ba14
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,20 @@ This is where Handoff comes in. You can run Handoff alongside your system as a s

On top of that there is a baked in web ui where you can look up the test results and manually trigger new runs.

Other valuable (future) features:
Other valuable (future/planned) features:

* Matching and linking test runs to logs / traces / metrics generated by the
systems under test (SUT) for easier debugging.
* Running smoke tests on new deployments for automatic rollbacks if necessary (argocd, flux).
* Identifying flaky tests.
* Fighting flaky tests via
* auto detection
* triggering multiple scheduled runs of the same test suite to provoke a test failure which can be debugged thereafter
* Maintaining historic test run data for deeper insights on the
stability of the platform and making it available via a `/metrics` endpoint.
* Triggering of alerts on test failures (e.g. via pagerduty).
* Automated notifications on test runs (e.g. via slack messages).
* Github Integration: adding test run results to relevant PRs after they were merged and deployed.
*

## Example

Expand Down Expand Up @@ -117,17 +120,21 @@ Once handoff is green in the dashboard you should be able to open up the ui here
- [ ] authentication providers (LDAP, Oauth2, ...)
- [ ] (Feature) Test Suite Metadata (desription in markdown, links to e.g. github, documentation, handbook, ...) shown in UI
- [ ] (Feature) Automatic test run retries/backoff on failures
- [ ] (Feature) Add a limit to scheduled runs (stop after X runs or X mins)
- [ ] (Feature) CLI program to run tests and wait for results, export results to json (via the server's http api)
- [ ] (Feature) Dashboard UI that shows handoff statistics, running tests, resource usage (cpu, memory, active go routines...) etc
- [ ] (Feature) Opt-in test timeouts through t.Context and / or providing wrapped handoff functions ( e.g. http clients) to be used in tests that implement test timeouts
- [ ] (Feature) Add test-suite labels
- [ ] (Feature) If test was run within the context of a PR maybe we can figure out who the author was (via commit emails) and send an email if a test failed.
- [ ] (Feature) Allow running of handoff as headless/cli mode (without http server) that returns a code != 0 if a test has failed (e.g. in github actions CI)
- [ ] (Feature) Add an option to the helm chart to support remote debugging through dlv
- [ ] (Feature) Image for [helm chart](https://helm.sh/docs/topics/chart_tests/) tests for automated helm release rollbacks
- [ ] (Feature) Test suite namespaces for grouping
- [ ] (Feature) Asynchronous plugin hooks with callbacks for slow operations (e.g. http calls)
- [ ] (Plugin) Pagerduty - triger alerts/incidents on failed e2e tests
- [ ] (Plugin) Slack - send messages to slack channels when tests pass / fail
- [ ] (Plugin) Github - pr status checks
- [ ] (Plugin) Jira - add test run results to a PR.
- [ ] (Plugin) Prometheus / Loki / Tempo / ELK stack - find and fetch logs/traces/metrics that are created by tests (e.g. for easier debugging) - e.g. via correlation ids
- [ ] (Technical) Idempotency in post requests via a key to avoid duplicate test runs
- [x] (Technical) Server configuration through either ENV vars or cli flags
Expand Down

0 comments on commit 6c7ba14

Please sign in to comment.