Testing

Add tests here for MRtrix3 commands to ensure that the output is consistent with expectation. If you intend to help with development of MRtrix3, it is recommended you run the enclosed tests regularly to ensure your changes do not introduce regressions.

Recommended command to fully test everything is working:

./configure && ./build && ./run_tests 

Look in testing.log for details of any failures encountered.

To run the tests

Simply run the ./run_tests script from within the MRtrix3 toplevel folder:

./run_tests

This will fetch the testing data as a submodule from a separate dedicated repo, build the testing executables, then run the tests. All activities are logged to the testing.log file - take a look in there for details of any failures.

Note that the ./run_tests script will not build the executables to be tested. This is to allow developers to build and test individual commands without needing to rebuild all other commands. If you want to test all commands, make sure you run ./build first.

Running tests for a single command

You can run a test for a single command simply by adding the name of the command as the argument:

./run_tests mrconvert

This means you don't need to build everything to test one particular command. For example, this is a construct I commonly use:

./build release/bin/mrconvert && ./run_tests mrconvert

Adding tests

Add a script to the tests/ folder. Each line of these scripts constitutes a single test, and will be run as a single unit. Use && and || bash constructs if needed to create compound commands. Each of these lines should return a zero exit code if successful. You can test the output of your commands against your expected output using the testing_diff_image command (note other commands are available to check various types of output - look in testing/cmd for the full list).

Note that this script will be invoked directly in the context set up by the run_tests script, so does not need to be executable, or to set up any redirection, or to uses a hash-bang line to specify the interpreter. Just add commands to be run, and if any of them produce a non-zero exit code, this will be caught by the run_tests script. All commands will also be logged.

The testing will consider each line of the test scripts to correspond to an individual test, and report as such. If any of your tests need multiple commands, simply list them all on the same line, separated by semicolons or other constructs such as && or ||. Look within existing test scripts for examples.

Temporary files

If your tests need to create temporary files, make sure they are prefixed with 'tmp' and are not placed in subfolders - the run_tests script will make sure these are deleted prior to running the next set of tests.

Adding test data

If needed, you can add test data to the test_data repo, preferably within its own subfolder if you don't anticipate these data will be suitable for testing other commands. Please keep the size of these data small to ensure this repository doesn't grow too large. In general, you really don't need large images to verify correct operation.

These data are included as a submodule within the main MRtrix3 repository, as a means of ensuring that the version of the data used for the tests is recorded within the main repo, while keeping its history separate, and hence on a separate repository. This separation helps to keep the size of the main repository small, and avoid large downloads for users. The downside is that making changes to the test data is more complicated. For here a full introduction into git submodules.

To add data, you will need to:

• add your data to the test_data repo;

• update the main MRtrix3 repo to reflect the new version of the data needed for the tests.

Note: if you need to modify exising data in a way that would break existing tests, you will need to be very careful with how you do this - see the relevant section below for details.

Adding the data to the test_data repo repository

This can be done using a number of approaches:

1. Using the GitHub website

Probably the simplest approach is simply to upload your new data via the GitHub interface. This will commit directly to the master branch of the test_data repo.

2. From a separate standalone repo:

Simply clone the test_data repo, add your data, commit, and push:

$ git clone [email protected]:MRtrix3/test_data.git
$ cd test_data
$ git add your_data.mif
$ git commit 
$ git push

You can delete the standalone repo once you have pushed your changes.

3. Directly within the MRtrix3 installation

Adding data can be done by pushing commits on the master branch of the submodule within the MRtrix3 folder. Note that you will need to have run ./run_tests at least once to initialise the submodule. For this to work, you will need to override the default URL for the test_data submodule in your local installation, since the default URL is the public HTTPS version, which is read-only. This only needs to be done once, using the following instructions:

$ git config submodule.testing/data.url [email protected]:MRtrix3/test_data.git
$ rm -rf testing/data .git/modules/testing/
$ ./run_tests

After this, you should be able to add data by committing directly to the master branch. Note that by default, the submodule will be in detached HEAD state, since it is locked to the specific version specified in the main MRtrix3 folder. Remember to checkout master before committing:

$ cd testing/data
$ git checkout master
$ git add your_data.mif
$ git commit 
$ git push

Updating the main MRtrix3 repo to make use of the latest commits

Once your new data have been committed to the test_data repo, all you need to do is tell the main MRtrix3 repository to record the new SHA1 for the latest version of the test data.

The first thing to do is ensure your main MRtrix3 repo has already initialised the testing/data submodule. The simplest way to do this is to just run ./run_tests (you can abort as soon as the 'fetching test data... OK' line has completed). Alternatively, you can run the relevant command directly yourself:

$ git submodule update --init

At this point, you can update the submodule with the latest version on the test_data repo:

$ git submodule update --remote

Note: this will record the SHA1 of the current master branch on the test_data repo. See the next section if you need to record any other branch or SHA1.

At this point, it's good practice to check your git status, and make sure everything makes sense. For example, assuming I'd just added some test data, and modified the tests for mrconvert:

$ git status
On branch master
Your branch is up-to-date with 'github/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   testing/data (new commits)
        modified:   testing/tests/mrconvert

no changes added to commit (use "git add" and/or "git commit -a")

If everything looks in order, you can add these to the staging area:

$ git add testing/data 
$ git add testing/tests/mrconvert

then commit and push:

$ git commit
$ git push

At this point, you can check whether things are working correctly by running the ./run_tests scripts, and checking that the expected checksum shows up afterwards when running:

$ git submodule
 b6d0e018d0bad96edec423b81ef1b006a94932fb testing/data (b6d0e01)

It should show the expected version, without a leading + sign.

What to do when modifying existing test data

Generally, adding data to the test_data repo is straightforward, and has no consequences for existing tests. However, there may be cases where changes in the code modify the output of some commands, and the test data therefore needs to be updated to match. If these modifications are pushed to the master branch of the test_data repo, everything will be fine initially, since the master branch will still pull in the specific SHA1 commit that was recorded at the time. However, problems will occur if other developers also push data to the test_data repo after that point, since these changes will most likely be pushed to its master branch, on top of the earlier modifications.

For example, imagine the following situation:

• Jack fixes a critical bug in dwi2fod that leads to big change in the expected output, and commits these changes to branch dwi2fod_bug_fix.

• Jack then modifies the data used in the dwi2fod tests in the test_data repo, pushes these to its master branch, and records the new updated SHA1 into the dwi2fod_bug_fix branch.

• later on, Gill adds a new command to her own branch featureX

• Gill adds tests for hew new command, along with new test data. Following the instructions, she pushes this additional data onto the master branch of the test_data repo.

• Gill's tests now fail, because her version of the test data now also includes Jack's modifications, but her featureX branch does not include Jack's corresponding modifications to the code in dwi2fod_bug_fix.

So this is clearly not the best outcome...

Solution: push modifications to separate branch

To avoid these issues, the best thing to do is to push any changes that modify existing data to a separate branch on the test_data repo, and merge these to master when the corresponding branch on the main MRtrix3 repo has also been merged. This allows other developers to keep pushing additional data to master without anything breaking.

Technically, there is no reason to merge the branch on the test_data repo to master since run_tests will pull in the relevant commit via its SHA1 checksum. However, merging to master ensures that the branch can be deleted without risk of losing the relevant commit.

To do this, developers need to push their changes to a different branch than master on the test_data repo. This can be done using for example:

$ git clone [email protected]:MRtrix3/test_data.git
$ cd test_data
$ git checkout -b dwi2fod_bug_fix
# make changes to test data...
$ git commit -a
$ git push origin dwi2fod_bug_fix

At that point, the modifications should be on the dwi2fod_bug_fix branch only. So far so good.

The next difficulty is updating the main MRtrix3 repo with that particular branch. As mentioned above, git submodule update --remote will pull in the current master branch, so we need to do something else. The solution is to manually checkout the relevant branch. First, make sure the submodule has been initialised (either run ./run_tests or git submodule update --init as per instructions above). Then:

$ cd testing/data
$ git fetch
$ git checkout dwi2fod_bug_fix

At which point you can go back to the toplevel folder of the main repo, and commit these changes as before:

$ cd ../..
$ git add testing/data
$ git commit 
$ git push