tests: minor fixes, add README.md

tests/README.md

@@ -0,0 +1,149 @@
+# Testing Arduino ESP8266 Core
+
+## Testing on host
+
+Some features of this project can be tested by compiling and running the code on the PC, rather than running it on the ESP8266. Tests and testing infrastructure for such features is located in `tests/host` directory of the project.
+
+Some hardware features, such as Flash memory and HardwareSerial, can be emulated on the PC. Others, such as network, WiFi, and other hardware (SPI, I2C, timers, etc) are not yet emulated. This limits the amount of features which can be tested on the host.
+
+### Adding a test case
+
+Tests are written in C++ using [Catch framework](https://github.com/catchorg/Catch2).
+
+See .cpp files under tests/host/core/ for a few examples how to write test cases.
+
+When adding new test files, update `TEST_CPP_FILES` variable in tests/host/Makefile to compile them.
+
+If you want to add emulation of a certain feature, add it into tests/host/common/ directory.
+
+
+### Running test cases
+
+To run test cases, go to tests/host/ directory and run `make`. This will compile and run the tests. 
+
+If all tests pass, you will see "All tests passed" message and the exit code will be 0.
+
+Additionally, test coverage info will be generated using `gcov` tool. You can use some tool to analyze coverage information, for example `lcov`:
+
+    lcov -c -d . -d ../../cores/esp8266 -o test.info
+    genhtml -o html test.info
+
+This will generate an HTML report in `html` directory. Open html/index.html in your browser to see the report.
+
+**Note to macOS users:** you will need to install GCC using Homebrew or MacPorts. Before running `make`, set `CC`, `CXX`, and `GCOV` variables to point to GCC tools you have installed. For example, when installing gcc-5 using Homebrew:
+
+    export CC=gcc-5
+    export CXX=g++-5
+    export GCOV=gcov-5
+
+When running `lcov` (which you also need to install), specify `gcov` binary using `--gcov-tool $(which $GCOV)` (assuming you have already set `GCOV` environment variable).
+
+## Testing on device
+
+Most features and libraries of this project can not be tested on host. Therefore testing on an ESP8266 device is required. Such tests and the test infrastructure are located in tests/device directory of this project.
+
+### Test cases
+
+Tests are written in the form of Arduino sketches, and placed into tests/device/test_xxx directories. These tests are compiled using Arduino IDE, so test file name should match the name of the directory it is located in (e.g. test_foobar/test_foobar.ino). Tests use a very simple BSTest library, which handles test registration and provides `TEST_CASE`, `CHECK`, `REQUIRE`, and `FAIL` macros, similar to [Catch](https://github.com/catchorg/Catch2).
+
+*Note: we should migrate to Catch framework with a custom runner.*
+
+Here is a simple test case written with BSTest:
+
+```c++
+#include <BSTest.h>
+#include <test_config.h>
+
+BS_ENV_DECLARE();
+
+void setup()
+{
+    Serial.begin(115200);
+    BS_RUN(Serial);
+}
+
+
+TEST_CASE("this test runs successfully", "[bs]")
+{
+    CHECK(1 + 1 == 2);
+    REQUIRE(2 * 2 == 4);
+}
+```
+
+BSTest is a header-only library, so necessary static data is injected into the sketch using `BS_ENV_DECLARE();` macro.
+
+`BS_RUN(Serial)` passes control to the test runner, which uses `Serial` stream to communicate with the host. If you need to do any preparation before starting tests, for example connect to an AP, do this before calling `BS_RUN`.
+
+`TEST_CASE` macro defines a test case. First argument is human-readable test name, second contains optional set of tags (identifiers with square brackets). Currently only one tag has special meaning: `[.]` can be used to mark the test case as ignored. Such tests will not be skipped by the test runner (see below).
+
+### Test execution
+
+Once `BS_RUN` is called, BSTest library starts by printing the *menu*, i.e. the list of tests defined in the sketch. For example:
+
+```
+>>>>>bs_test_menu_begin
+>>>>>bs_test_item id=1 name="this test runs successfully" desc="[bs]"
+>>>>>bs_test_menu_end
+```
+
+Then it waits for the test index to be sent by the host, followed by newline.
+
+Once the line number is received, the test is executed, and feedback is printed:
+```
+>>>>>bs_test_start file="arduino-esp8266/tests/device/test_tests/test_tests.ino" line=13 name="this test runs successfully" desc="[bs]"
+>>>>>bs_test_end line=0 result=1 checks=2 failed_checks=0
+```
+
+Or, in case the test fails:
+```
+>>>>>bs_test_start file="arduino-esp8266/tests/device/test_tests/test_tests.ino" line=19 name="another test which fails" desc="[bs][fail]"
+>>>>>bs_test_check_failure line=22
+>>>>>bs_test_check_failure line=24
+>>>>>bs_test_end line=0 result=0 checks=4 failed_checks=2
+```
+
+BSTest library also contains a Python script which can "talk" to the ESP8266 board and run the tests, tests/device/libraries/BSTest/runner.py. Normally it is not necessary to use this script directly, as the top level Makefile in tests/device/ directory can call it automatically (see below).
+
+### Test configuration
+
+Some tests need to connect to WiFi AP or to the PC running the tests. This configuration should be set in tests/device/libraries/test_config/test_config.h. This file is not tracked in Git. Create it by copying from tests/device/libraries/test_config/test_config.h.template, and modifying settings in that file.
+
+### Building and running the tests
+
+Makefile in tests/device/ directory handles compiling, uploading, and executing test cases.
+
+Here are some of the supported targets:
+
+- `virtualenv`: prepares Python virtual environment inside tests/device/libaries/BSTest/virtualenv/. This has to be run once on each computer where tests are to be run. This target will use `pip` to install several Python libraries required by the test runner (see tests/device/libaries/BSTest/requirements.txt).
+
+- `test_xxx/test_xxx.ino`: compiles, uploads, and runs the tests defined in `test_xxx/test_xxx.ino` sketch. Some extra options are available, these can be passed as additional arguments to `make`:
+    - `NO_BUILD=1`: don't compile the test.
+    - `NO_UPLOAD=1`: don't upload the test.
+    - `NO_RUN=1`: don't run the test.
+    - `V=1`: enable verbose output from compilation, upload, and test runner.
+
+    For example, `make test_newlib/test_newlib.ino V=1` will compile, upload, and run all tests defined in `test_newlib/test_newlib.ino`.
+
+    For each test sketch, test results are stored in `tests/device/.build/test_xxx.ino/test_result.xml`. This file is an xUnit XML file, and can be read by a variety of tools, such as Jenkins.
+
+- `test_report`: Generate HTML test report from xUnit XML files produced by test runs.
+
+- `all` (or just `make` without a target): Run tests from all the .ino files, and generate HTML test report.
+
+### Host-side helpers
+
+Some tests running on the device need a matching part running on the host. For example, HTTP client test might need a web server running on the host to connect to. TCP server test might need to be connected to by TCP client running on the host. To support such use cases, for each test file, an optional Python test file can be provided. This Python file defines setup and teardown functions which have to be run before and after the test is run on the device. `setup` and `teardown` decorators bind setup/teardown functions to the test with specified name:
+
+```python
+from mock_decorators import setup, teardown
+
+@setup('WiFiClient test')
+def setup_wificlient_test(e):
+    # create a TCP server
+
+@teardown('WiFiClient test')
+def teardown_wificlient_test(e):
+    # delete TCP server
+```
+
+

tests/device/Makefile

@@ -10,9 +10,11 @@ UPLOAD_BAUD ?= 921600
 UPLOAD_BOARD ?= nodemcu
 BS_DIR ?= libraries/BSTest
 DEBUG_LEVEL ?= DebugLevel=None____
-FQBN ?= esp8266com:esp8266:generic:CpuFrequency=80,FlashFreq=40,FlashMode=DIO,UploadSpeed=115200,FlashSize=4M1M,ResetMethod=none,Debug=Serial,$(DEBUG_LEVEL)
-BUILD_TOOL = $(ARDUINO_IDE_PATH)/arduino-builder
-TEST_CONFIG = libraries/test_config/test_config.h
+FQBN ?= esp8266com:esp8266:generic:CpuFrequency=80,FlashFreq=40,FlashMode=dio,UploadSpeed=115200,FlashSize=4M1M,LwIPVariant=v2mss536,ResetMethod=none,Debug=Serial,$(DEBUG_LEVEL)
+BUILD_TOOL := $(ARDUINO_IDE_PATH)/arduino-builder
+TEST_CONFIG := libraries/test_config/test_config.h
+TEST_REPORT_XML := test_report.xml
+TEST_REPORT_HTML := test_report.html
 
 ifeq ("$(UPLOAD_PORT)","")
 $(error "Failed to detect upload port, please export UPLOAD_PORT manually")
@@ -57,6 +59,7 @@ ifneq ("$(NO_BUILD)","1")
 		$@
 endif
 ifneq ("$(NO_UPLOAD)","1")
+	@echo Uploading binary
 	$(SILENT)$(ESPTOOL) $(UPLOAD_VERBOSE_FLAG)  \
 		-cp $(UPLOAD_PORT) \
 		-cb $(UPLOAD_BAUD) \

tests/device/libraries/test_config/test_config.h.template

@@ -1,9 +1,12 @@
 #pragma once
 
+// Tests which use STA mode will connect to this AP:
 #define STA_SSID "test_wifi"
 #define STA_PASS "test_wifi_pass"
 
+// Tests which use AP mode will set up a SoftAP with these parameters:
 #define AP_SSID "test_wifi_ap"
 #define AP_PASS "test_wifi_ap_pass"
 
+// IP address of the PC running the tests (needed for HTTP client and HTTP server tests)
 #define SERVER_IP "192.168.10.1"

tests/host/Makefile

@@ -4,8 +4,8 @@ CORE_PATH := ../../cores/esp8266
 
 # I wasn't able to build with clang when -coverage flag is enabled, forcing GCC on OS X
 ifeq ($(shell uname -s),Darwin)
-CC  := gcc
-CXX := g++
+CC  ?= gcc
+CXX ?= g++
 endif
 GCOV ?= gcov