class: center ##Mutation testing for JavaScript ### Shing Lyu ??? top, middle, bottom left, center, right --- ###Existing Frameworks for JS * [Mutator](http://ortask.com/mutator/) (Ask the sales rep!) * [AjaxMutator](http://knishiura-lab.github.io/AjaxMutator/) (Java+PHP??) * [Mutandis](https://github.com/saltlab/mutandis) (Academic, from UBC) * [grunt-mutation-testing](https://github.com/jimivdw/grunt-mutation-testing) (we'll use this) --- ### Mutation Testing Revisited * Test your test suite * missing tests? always passing tests? etc. * by injecting deliberate faults (mutant) into the test target * Drawbacks: * sloooooow * equivalent mutations --- ### Mutation Testing * Inject mutant * Run test ||| |--|-| | failed | => mutant dies| => .green[good test!]| | passed | => mutant survived| => .red[problem!]| --- ### Setting up the test ```javascript // ===== calculator.js ===== function substractPositive(num1, num2){ if (num1 > 0){ return num1 - num2; } else { return 0 } } module.exports.substractPositive = substractPositive ``` ```javascript //===== test_calculator.js ===== ar assert = require('assert'); var cal = require('../src/calculator.js') describe('Calculator', function(){ it('substractPositive', function(){ assert.equal('2', cal.substractPositive(1, -1)); }); }) ``` --- ### Test Result ```bash npm install -g mocha ``` ```bash % mocha Calculator ✓ substractPositive 1 passing (5ms) ``` --- ### Building a grunt job for mutation testing ```javascript // ===== Gruntfile.js ===== module.exports = function(grunt) { grunt.initConfig({ mutationTest: { options: { testFramework: 'mocha' }, target: { options:{ code: ['src/*.js'], specs: 'test/test_*.js', mutate: 'src/*.js' } } } }); grunt.loadNpmTasks('grunt-mutation-testing'); grunt.registerTask('default', ['mutationTest']); }; ``` --- ### Setup and Run ```bash npm install -g grunt-cli npm install . # grunt, grunt-mutation-testing, karma grunt ``` --- ### Mutation result ```bash % grunt Running "mutationTest:target" (mutationTest) task (17:34:23.955) INFO [mutation-testing]: Mutating file: /tmp/mutation-testing1151016-11404-alnz38/src/calculator.js (17:34:23.957) INFO [mutation-testing]: Mutating line 1, 1/22 (5%) (17:34:23.959) INFO [mutation-testing]: Mutating line 10, 2/22 (9%) (17:34:24.677) INFO [mutation-testing]: Mutating line 23, 3/22 (14%) (17:34:24.679) INFO [mutation-testing]: Mutating line 24, 4/22 (18%) (17:34:24.680) INFO [mutation-testing]: Mutating line 2, 5/22 (23%) (17:34:24.682) INFO [mutation-testing]: Mutating line 2, 6/22 (27%) /src/calculator.js:2:11 Replaced > with >= -> SURVIVED (17:34:24.684) INFO [mutation-testing]: Mutating line 2, 7/22 (32%) (17:34:24.686) INFO [mutation-testing]: Mutating line 2, 8/22 (36%) (17:34:24.688) INFO [mutation-testing]: Mutating line 3, 9/22 (41%) (17:34:24.689) INFO [mutation-testing]: Mutating line 3, 10/22 (45%) (17:34:24.690) INFO [mutation-testing]: Mutating line 6, 11/22 (50%) /src/calculator.js:6:5 Removed return 0 -> SURVIVED ... 12 of 22 unignored mutations are tested (54%). Done, without errors. ``` --- ### Types of mutations  ??? background-image: url(img/mutants.png) --- ### Problem 1: Missing test case ```bash /src/calculator.js:2:11 Replaced > with >= -> SURVIVED /src/calculator.js:6:5 Removed return 0 -> SURVIVED /src/calculator.js:6:12 Replaced 0 with 1 -> SURVIVED ``` ```javascript 1 function substractPositive(num1, num2){ 2 if (num1 > 0){ 3 return num1 - num2; 4 } 5 else { 6 return 0 7 } 8 } ``` --- ### Fixing Problem 1 * We didn't test if `num1 <= 0`, * We didn't varify `return 0` ```javascript it('substractPositive', function(){ assert.equal('0', cal.substractPositive(0, -1)); }); ``` --- ### Equivalent mutation * Mutation which doesn't change behavior, so not test can catch it * "false alarm" * May reveal real problem, but not with the test suite * dead code * internal mechanism (e.g. cache, state machine) --- ### Problem 2: Dead Code ```javascript 10 function add(num1, num2){ 11 if (num1 == num2){ 12 return num1 + num2; 13 } 14 else if (num1 > num2){ 15 return num1 + num2; 16 } 17 else { 18 return num1 + num2; 19 } 20 } ``` ```javascript /src/calculator.js:11:11 Replaced == with != -> SURVIVED /src/calculator.js:14:16 Replaced > with >= -> SURVIVED /src/calculator.js:14:16 Replaced > with <= -> SURVIVED /src/calculator.js:15:5 Removed return num1 + num2; -> SURVIVED /src/calculator.js:15:16 Replaced + with - -> SURVIVED /src/calculator.js:18:5 Removed return num1 + num2; -> SURVIVED /src/calculator.js:18:16 Replaced + with - -> SURVIVED ``` --- ### Fixing Problem 2 * Remote the dead code ```javascript function add(num1, num2){ return num1 + num2; } ``` --- class: center, middle #Thank you