Disclaimer: The views expressed here are my own; they do not reflect the views of my current and past employers.
We will be working on (gradually) migrating the majority of the code base to the Stryker.
So I’ll update my post to use the latest Stryker framework. The following will be the updated post with all the code example migrated to the Stryker framework:
- Code example: github link
Last November (2015) I attended the EuroStar Software Testing Conference, and was introduced to a interesting idea called mutation testing. Ask yourself: “How do I ensure my (automated) unit test suite is good enough?”. Did you miss any important test? Is your test always passing so it didn’t catch anything? Is there anything un-testable in your code such that your test suite can never catch it?
Introduction to Mutation Testing
Mutation testing tries to “test your tests” by deliberately inject faults (called “mutants”) into your program under test. If we re-run the tests on the crippled program, our test suite should catch it (i.e. some test should fail.) If you missed some test, the error might slip through and the test will pass. Borrowing terms from Genetics, if the test fails, we say the mutant is “killed” by the test suite; on the opposite, if the tests passes, we say the mutant survived.
The goal of mutation testing is to kill all mutants by enhancing the test suite. Although 100% kill is usually impossible for even small programs, any progress on increasing the number can still benefit your test a lot.
The concept of mutation testing has been around for quite a while, but it didn’t get very popular because of the following reasons: first, it is slow. The number of possible mutations are just too much, re-compiling (e.g. C++) and re-run the test will take too long. Various methods has been proposed to lower the number of tests we need to run without sacrifice the chance of finding problems. The second is the “equivalent mutation” problem, which we’ll discuss in more detail in the examples.
This framework supports mutants like changing the math operators, changing logic operators or even removing conditionals and block statements. You can find a full list of mutations with examples here.
Setting up the environment
You can follow the quickstart guide to install everything you need. The guide has a nice interactive menu so you can choose your favorite build system, test runner, test framework and reporting format. In this blog post I’ll demonstrate with my favorite combination: Vanilla NPM + Mocha + Mocha + clear-text.
npm installed. (I recommended using nvm).
There are a few Node packages you need to install,
Here are the list of packaged that I’ve installed:
A simple test suite
I created a simple program in
src/calculator.js, which has two functions:
The first function is called
substractPositive, it substract
num1 is a positive number. If
num1 is not positive, it will return
0 instead. It doesn’t make much sense, but it’s just for demonstrative purpose.
The second is a simple
add function that adds two numbers. It has a unnecessary
if...else... statement, which is also used to demonstrate the power of mutation testing.
The two functions are tested using
This is a test file running using
mocha, The first verifies
substractPositive(1, -1) returns
2. The second tests
2. If you run
mocha in your commandline, you’ll see both the test passes. If you run
mocha in the commandline you’ll see the output:
So this test suite looks OK, it exercises both function and verifies its output, but is it good enough? Let’s verify this by some mutation testing.
Running the mutation testing
To run the mutation testing, we first need to create a config file for the stryker mutator. Create a file called
stryker.conf.js in the project’s root directory and paste the following input into it:
You can see we choose
mocha as the
testFramework, and we tell the Stryker framework to run the test files in
test/**/*.js, and the source files are in
Then we need to tell
npm how to run Stryker in the
package.json file. Simply add the following code in your
This will add a
npm run stryker command to
npm run and execute the
stryker -c stryker.conf.js script.
Finding Missing Tests
Now if we run
npm run stryker, you’ll see the following test result:
As you can see, it tested 22 kinds of mutations, but 14 (22-8=14) of them survived!
Let’s look at a survived mutant:
It tells us that by replacing the
>= in line 2 of our
calculator.js file, the test will pass.
If you look the line, the problem is pretty clear
We didn’t test the boundary values, if
num1 == 0, then the program should go to the
else branch and returns
0. By changing the
>=, the program will go into the
return num1 - num2 branch instead and returns
0 - num2!
This is one of the problem mutation testing can solve, it tells you which test case you missed. The solution is very simple, we can add a test like this:
If you run mutation testing again, the problem with the
substractPositive function should go away.
Equivalent Mutation and Dead Code
Sometimes a mutation will not change the behavior of the program, so no matter what test you write, you can never make it fail. For example, a mutation may disable caching in your program, the program will run slower but the behavior will be exactly the same, so you’ll have a mutation you can never kill. This kind of mutation is called “equivalent mutation”.
Equivalent mutation will make you overestimate your mutation survival rate. And they time to debug, but may not reveal useful information about your test suite. However, some equivalent mutations do reveal issues about your program under test.
Let look at the mutation results again:
If you look at the code, you’ll find that all the branches of the
if...else... statement returns the same thing. So no matter how you mutate the
if...else... conditions, or even remove one branch that was not reached, the function will always return the correct result.
Because we found that the
if...else... is useless, we can simplify the function to only three lines:
If you run the mutation test again, you can see all the mutations being killed.
This is one of the side benefit of equivalent mutations, although your test suite is fine, it tells you that your program code has dead code or untestable code.