Python Mutant Testing (PyMuTester)
What is PyMuTester?
PyMuTester is tool to facilitate Mutant Testing (a.k.a Mutant Analysis or Program Mutation) on software systems written in Python. Its main purpose is to assist you in improving your existing unit tests to cover missing checks and "loopholes" in your testing. It works by making small changes (technically known as mutants) to your Python application's source code and re-run your unit tests over these mutated application source code. Since the mutants usually go against the specifications, your unit tests should fail in such tests. If the unit tests still pass, then that is an indication that your unit tests might have missed out some checks!
Currently, PyMuTester leverages on nose testing framework to execute the re-runs. The nose testing framework also supports test cases written using doctests and unittest.
Disclaimer: Although PyMuTester will modify your source code to create mutants, these modifications are all performed in memory. The source files on the filesystem are never modified!
Why Mutant Testing?
In any non-trival software development projects, testing is always a crucial part of the software development lifecycle (SDLC). Agile development methodology stresses rapid delivery and responsiveness to requirement changes. For these to happen, sound testing techniques are crucial to ensure that existing, delivered software components do not deviate from their specifications after changes have been made in other parts of the software system.
Most experienced software developers and/or testers will agree that one of the major hurdles in testing is to determine if the testing is comprehensive and complete: Have we done enough testing on the software system within the limited resources and time? Did we miss testing any parts of the code? Mutant testing answers these questions by testing your unit test cases with various mutated forms of the original source code. In fact, one can say that mutant testing actually perform tests on your testing; the improvements on the software system are just a side-effect.
Download
PyMuTester is licensed under Apache License version 2.0. You may download PyMuTester from Python Package Index (PyPI). If you have installed setuptools on your system, you can also run easy_install pymutester to install PyMuTester.
Installation and Use
To install PyMuTester, simply extract the contents from the archive and run python setup.py install. This will install the mutester package into your python distribution.
- Perform your unit tests as usual using the nosetest commands.
- When all your unit tests pass in nosetest, run your tests again through the mutant-nosetest command. The command line arguments to the mutant-nosetest command should contain paths pointing to your test cases and exclude plugins (those parameters starting with --with-) parameters. The command line arguments must have one or more --mutant-path parameters which specify the folder of the source code to be mutated. Only Python source files found in these folders will be mutated. If you want to exclude any source code folder from mutation, use one or more --mutant-exclude parameters.
For example,
$> nosetest --with-coverage --with-xunit tests/mock_tests .................. ---------------------------------------------------------------------- Ran 18 tests in 0.044s
OK
$> mutant-nosetest --mutant-path /project/myapp tests/mock_tests
..................
----------------------------------------------------------------------
Ran 18 tests in 0.064s
OK
*************************
Starting mutation test...
*************************
Mutating RawConfigParser.has_option (/project/myapp/ConfigParser.py:373)...
*** IFNOT-1... Mutant killed
*** IFNOT-2... Mutant killed
Mutating ConfigParser.items (/project/myapp/ConfigParser.py:578)...
*** IFNOT-1... Mutant not reached
--- /project/myapp/ConfigParser.py (original)
+++ /project/myapp/ConfigParser.py (mutant-IFNOT-1)
@@ -591,7 +591,8 @@
try:
d.update(self._sections[section])
except KeyError:
- if section != DEFAULTSECT:
+ if (not (section != DEFAULTSECT)):
raise NoSectionError(section)
# Update with the entry specific variables
if vars:
*** IFNOT-2... Mutant killed
Mutant Test Results
Total: 58
Alive: 0 (0.0%) Killed: 55 (94.8%) Unreachable: 3 (5.2%)For the mutant testing to proceed, all the given unit tests must succeed. As the mutant testing progresses iteratively on the source code, the test result will be printed to the screen. The mutant is killed if at least one of the unit test cases has raised an assertion, or the mutated application crashes in the execution. If the mutated code is not executed, the mutant will be marked as unreachable. If the mutated code executes successfully without being caught by the test cases, it is considered alive.
PyMuTester will output the mutated source code in unified diff format if the mutant is alive or unreachable at the end of each mutation tests. You should compare the mutated source code with your application source, and verify if the mutant warrants any further attention/action from you.
Available Mutators in PyMuTester
If-Condition Negation
This mutator mutates the test conditional clause in the if-statement, by negating the final results of the test with a not keyword.
Loop Skipping
This mutator mutates the bodies of for- and while-loops. A continue statement is inserted on every other loop iteration, and prevents the rest of the loop body from executing.