facebook tracking

Master Thesis - Evaluation of mutation based testing

Automotive software development is a rapidly growing business. Autonomous drive is the engineering challenge of the decade and verification of AD software is a key challenge for this new business.
Scroll to content

Background 

Historically it has been common to use different test coverage metrics to estimate how well a given set of tests cover the tested code. In the automotive industry this has been an important source of documentation of the fact that production code is thoroughly tested. The problem with these kinds of metrics is that they can't prove that testing is good or sufficient, they can only prove that tests are not sufficient.

A relatively new approach that has gained some interest is mutation testing. In mutation testing a tool is used to run the tests on "mutations" of the production code. If the tests fail a large portion of the mutations then they are likely good at detecting unwanted changes in the production code and if the tests pass for a large portion of the mutations then the tests are likely not so good at detecting unwanted changes and that would be an indication that tests need to be improved.
This approach has the benefit of actually evaluating the tests ability to detect unwanted changes rather than just monitoring how much of the production code is executed during test.

At Zenseact we are so far only using coverage metrics. Using mutation testing as a complement to coverage metrics could improve confidence in our test suites and the quality of our products.

Mutation on has been widely explored in Java (including a little in automotive). However, for C++ or wider aspects of automotive development (e.g., c++, CI / CD, etc.) there is not so much empirical evaluations.

Zenseacts goal is to try mutation testing on a limited set of tests in our product code. If we succeed, we will get new insights to the quality of the tests in question and a documented way of adopting mutation testing for the rest of our code base. We would also get data which we can use to support any decisions on whether a wider adoption of mutation testing would be worth pursuing at Zenseact.

Project Description 

In this master thesis project, you will: 

  • Selecting a couple of candidate tools for evaluation.
  • Setting up mutation testing to work with a small open-source project.
  • Comparing two mutation testing tools, for speed, ease of use, effectiveness etc.
  • Setting up mutation testing to work with part of Zenseacts code base.
  • Evaluating the challenges of using mutation testing at large scale within Zenseact.
  • Evaluating what human factors can influence the adoption and success of those mutation tools in the tool chain.
  • Potentially creating some useful mutators.
  • Collecting results of coverage measurements and mutation testing.
  • Analyzing the result from mutation testing.
  • Formulating a strategy for both efficient and effective use of mutation testing in continuous integration.
  • Evaluating the potential usefulness of mutation testing for Zenseact.


Qualifications 

We are looking for 1 or 2 students, preferably with good knowledge of

  • programming (specifically you need to be comfortable working with C++)
  • testing

Further information

Please send in individual applications with CV, motivational letter and grade transcripts. 

Planned start: January 2022, with some flexibility.

Final application date: 15 of November 2021, but we will screen candidates continuously, so please submit your application as soon as possible.

Duration: 30 ECTS 

For questions regarding the project, please contact: kim.viggedal@zenseact.com

Additional information

  • Remote status

    Flexible remote

Or, know someone who would be a perfect fit? Let them know!

Gothenburg, Sweden

Lindholmspiren 2
417 56 Gothenburg, Sweden Directions View page

Teamtailor

Career site by Teamtailor