Clarify test suites

[DanielOaks]

So this issue is basically to have a discussion around our test suite(s), and how they work, and our intentions for future work. Especially given Oragono's position as basically 'my reference implementation' for my own IRC protocol documentation, and my want for it to inter-operate in a large way with most other servers out there.

Integrated tests

This basically means our unit tests. We don't have any integration-test stuff inside the server right now, and... with how much we change things, I don't think integration tests are a realistic thing to have as a part of the code anyway.

We'll extend these as we can, but they don't prevent issues like us sending numerics incorrectly or things along those lines.

Our irctest fork

This piece of software works as a reasonable sanity-check. Especially for new IRCv3 features that we've written new tests for. However:

• It's a bit difficult to actually run.
• It doesn't really test our inter-operation that well, or things like our numeric parameters, though it could probably be extended to do this a bit better.
• Can restart the server with different configuration options to test things in a more complete way.
• There's a fair bit of internal magic that I'm not entirely happy with.

Plainly put, it works fine. It's not amazing, but I'm happy with how we've taken it so far.

ircts

This is a new general piece of IRC testing software that I want to develop. Basically, it's like irctest, but written from the ground up with what I feel like is a slightly cleaner goal in mind.

Downsides:

• It's, um, not written yet.
• We'll need to write new tests for it from scratch.
• It's intentionally designed to run on a live, running server, rather than to restart the server with different configuration files like irctest does. Because of this, it takes more work for us to do tests using it.

Upsides:

• Focus on compatibility between different servers.
• Focus on testing reply parameters, something that we've explicitly run into a number of times.
• Ease of use (download binary, run (or clone from source, go run).

---

Basically, irctest is our existing, working suite of tests that does serve us well. ircts is a new piece of software that I want to work on, useful both for Oragono and for me testing things for documentation purposes.

I think that's where my head is at the moment in terms of test suites, their primary purposes / tradeoffs, and all. I'd like to eventually make ircts the primary test suite for Ora, but that isn't gonna happen for a fair while because, well, that bit of software needs to be written.

[slingamn]

A couple arguments in favor of irctest:

• Restarting the server every time helps with test reliability (we're automatically immune to nondeterministic test ordering effects)
• Restarting the server lets us test with multiple versions of the config, although we aren't doing this yet: make it easier to modify the config for tests DanielOaks/irctest#25
• There is a need to test Oragono-specific features, or just to write tests that are sufficiently specific that they don't pass on other ircds