Historically it has been common to use different test coverage metrics to estimate how well a given set of tests cover the tested code. In the automotive industry this has been an important source of documentation of the fact that production code is thoroughly tested. The problem with these kinds of metrics is that they can't prove that testing is good or sufficient, they can only prove that tests are not sufficient.
A relatively new approach that has gained some interest is mutation testing. In mutation testing a tool is used to run the tests on "mutations" of the production code. If the tests fail a large portion of the mutations then they are likely good at detecting unwanted changes in the production code and if the tests pass for a large portion of the mutations then the tests are likely not so good at detecting unwanted changes and that would be an indication that tests need to be improved.
This approach has the benefit of actually evaluating the tests ability to detect unwanted changes rather than just monitoring how much of the production code is executed during test.
At Zenseact we are so far only using coverage metrics. Using mutation testing as a complement to coverage metrics could improve confidence in our test suites and the quality of our products.
Mutation on has been widely explored in Java (including a little in automotive). However, for C++ or wider aspects of automotive development (e.g., c++, CI / CD, etc.) there is not so much empirical evaluations.
Zenseacts goal is to try mutation testing on a limited set of tests in our product code. If we succeed, we will get new insights to the quality of the tests in question and a documented way of adopting mutation testing for the rest of our code base. We would also get data which we can use to support any decisions on whether a wider adoption of mutation testing would be worth pursuing at Zenseact.
In this master thesis project, you will:
- Selecting a couple of candidate tools for evaluation.
- Setting up mutation testing to work with a small open-source project.
- Comparing two mutation testing tools, for speed, ease of use, effectiveness etc.
- Setting up mutation testing to work with part of Zenseacts code base.
- Evaluating the challenges of using mutation testing at large scale within Zenseact.
- Evaluating what human factors can influence the adoption and success of those mutation tools in the tool chain.
- Potentially creating some useful mutators.
- Collecting results of coverage measurements and mutation testing.
- Analyzing the result from mutation testing.
- Formulating a strategy for both efficient and effective use of mutation testing in continuous integration.
- Evaluating the potential usefulness of mutation testing for Zenseact.
We are looking for 1 or 2 students, preferably with good knowledge of
- programming (specifically you need to be comfortable working with C++)
Please send in individual applications with CV, motivational letter and grade transcripts.
Planned start: January 2022, with some flexibility.
Final application date: 15 of November 2021, but we will screen candidates continuously, so please submit your application as soon as possible.
Duration: 30 ECTS
For questions regarding the project, please contact: firstname.lastname@example.org