bbt README

• Overview
  • What does the behavior description look like?
  • Partial parsing
  • Step arguments
  • One more example
• Why should I use bbt?
  • Write once
  • Write a real documentation, not just a ".feature" Gherkin file
  • Be proficient in no time
  • No-fuss no-wait run
  • Ready to publish output
• Status of the project
• Installation
• Help and comments
• Further reading

Overview

bbt is a simple tool to black box check the behavior of an executable through Command Line Interface (CLI).
Hence the name: bbt stands for Black Box Tester.

bbt targets both specification of the behavior and end-to-end test automation for the very common case of apps taking some input and producing some output.
It enable developers to write and execute comprehensive test scenarios in just a few minutes.

The outstanding feature of btt is that it directly uses your behavior documentation in plain english.
There is no script nor other file to write.

What does the behavior description look like?

The behavior is described in almost natural English, in Markdown, using the BDD / Gherkin usual pattern Given / When / Then.
Here is a minimal example:

### Scenario: I want to know gcc version

- When I run `gcc --version`
- Then the output contains `14.2.0`

resulting in:

Scenario: I want to know gcc version

• When I run gcc --version
• Then the output contains 14.2.0

bbt being about documentation and simplicity, Markdown¹ is a perfect fit.

Let's consider a slightly more complete example:

(Markdown source here)

We have:

1. An "Overview" header and a first text
   All this is ignored, because:

   • bbt processes only Gherkin headers # Features, # Background, and # Scenario or # Example.
   • bbt considers all text lines as comment, except Step lines.

   bbt staying out of the way, you're free to use markdown almost without constraints to draft nice documentations.

2. A "Feature" and a "Scenario" header (followed by the feature/scenario name)
   bbt is now awake, and waiting for step lines.
   Note that header's level is ignored (#### Scenario, is equal to # Scenario for bbt), you're free to structure the file as you want.

3. Steps
   Steps are lines starting with - Given, - When, - Then, - And, - But, that contain the things to check or do.

Partial parsing

A distinctive feature of bbt is that it seems to directly understand those almost normal English sentences like:

- When I run `sut --quiet input.txt`
- Then there is no output

This is achieved thanks to a partial parser. It means that there is no rigid grammar, because bbt takes into account only some keywords to recognize the skeleton of the sentence.

So when you write:

• Then I should get version 15.0.0 (Fix #2398 and #2402)

bbt actually reads:

• then get version 15.0.0

And this is what gives the ability to write steps in almost natural language.

Step arguments

Step's argument may be strings or multiline text.

As per MDG, bbt uses :

• multiline text (expected output, file content, etc) : fenced code blocks, that is a text between two "```" lines
• strings (file name, command to run, etc.) : code span, that is a string between backticks

It's not only to nicely highlight inputs in the doc, but also because otherwise the analysis of the steps would be too complex.

One more example

This example shows how simple it is to run a gcc sanity test, that compiles and runs the ubiquitous Hello Word.

Installation

is available thanks to the Alire package manager.

1. To install Alire on your platform, refer to Alire

2. Then to install bbt:

   alr get bbt  
   cd bbt  
   alr build  

3. Move the bbt exec somewhere in your PATH

Or, to get the latest version:

git clone https://github.com/LionelDraghi/bbt  
cd bbt  
alr build 

(if you don't want to install alr, just run gprbuild instead of alr build)

First use

The easiest way to start is illustrated by Simon in it's ada_caser project.
He just created a scenario file called tests.md, and put in the README a reference to that file and the command line to run the tests.
Thats'it, he didn't even need to create a "tests" directory.

Why should I use bbt?

Write once

Specification is the only source of truth. This is bbt most interesting feature, there is nothing else: no intermediate representation, no glue code, no scripting language, no duplication of the original source at all.

With two main consequences:

1. writing a test is a matter of minutes,
2. there is no more place for a discrepancy between documentation and tests.

Alternative tools exist, some are mentioned in my quick overview of some comparable tools.
But as far as I know, bbt is the only one to provide such a direct "run the doc" approach.

Write a real documentation, not just a ".feature" Gherkin file

bbt effectiveness does not come at the cost of limiting documentation readability or expressiveness:

• First, the vast majority of the file is just plain markdown : use it, structure it the way you like, give as much context as you want, and use all Markdown cool extensions (for example graphics with Mermaid);
• Second, even the part that is interpreted by bbt, the steps, is written in readable English thanks to the partial parsing.

Nice consequence, bbt scenarios may be written by non coders people.

Be proficient in no time

bbt Steps uses a limited English subset, with a vocabulary dedicated to test with no-surprise keywords like run, output, contains, etc.

Although simple, you don't have to learn this subset by heart, you may :

• ask for a template scenario by running bbt -ct (or --create_template), or
• ask for the complete grammar with bbt -lg (or --list_grammar).

No-fuss no-wait run

To run a scenario : bbt my_scenario.md
Or to run all the md files in the tests tree bbt -r tests

bbt has no dependencies on external lib or tools (diff, for example) and can be run as is on major native platforms.

Ready to publish output

bbt output is in Markdown format. You can adjust the detail level with the usual "-q" and "-v" options.

The output cross-references the executed scenario files: if a test fails, just click on the link and you are in the scenario.
You can push it on GitHub without further processing.

To see what it looks like, consider bbt own tests.

Test results are generated when running bbt, by just using the -o option (--output).

Status of the project

bbt is in an early stage, meaning that interface and behavior are subject to changes.
Feel free to make features suggestions in bbt discussions.

The code has grown fast in 2024, and is far from being clean.
Nevertheless, bbt is working, and has as a serious test base.

A very conclusive test on the effectiveness of bbt as being conducted on the day 4 of Advent of Code 2024's challenges.
Tests where easy and fast to setup, allowing to stay most of the time focus on coding.

In real life, the acc project has largely migrated to BBT, resulting in a drastically reduced number of files and a significant gain in maintenability and readability of the tests.
Other people are using it too.

btt compile on Linux, Windows and Mac OS, and the test suite is run on the three platforms.
On Windows, some tests fail because of expected output containing Unix style path, and one relative to escaping command line written for Linux. bbt syntax is not yet able to specify per platform expected results, so this is not going to be fixed quickly.
On MacOS, it may be usefull to set the environment variable GNAT_FILE_NAME_CASE_SENSITIVE to 1, cf. discussion here to avoid small glitches on file names.

That said, I'm not aware of any Windows or MacOS specific bug.

Help and comments

Comments are welcome here

Further reading

• User Guide: concepts, command, features...
• Developer Guide: design overview, issues, fixme...
• References: syntax, grammar, and more details on non obvious behavior
• Project status: changelog, tests, TDL...
• Command line help

Footnotes

1. More precisely, this is a subset of the existing Markdown with Gherkin (MDG) format. ↩