Skip to main content

"What's a good code coverage to have?"

February 10, 2017

“What is a good speed to drive?”. The answer depends on a lot of things like: where do you drive (in a residential area, on a highway), are there speed limits, what are the weather conditions, etc. But more importantly, since when is driving at a particular speed a goal? Getting to a certain location at a certain time is the actual goal. Measuring speed only helps you to track progress on achieving this goal. “What’s a good percentage of code coverage to have?” is like the speed question. It too depends on the context and isn’t a goal. Then what does code coverage actually tell you and how can we use it to achieve the real goal: creating quality software?

Running (unit) tests on your code is an important part of assessing the quality of your product. By compiling code, we verify that the source code is syntactically correct (it leads to something that we can run). By testing it, we verify that it is semantically correct (does it behave the way it should). Code coverage is a measure used to describe the degree to which code is exercised by your tests. For this, we use tools and preferably make them part of our Continuous Integration/Delivery pipeline. Tools like SonarQube can help to generate insights based on these measurements. What’s important to realize is that this measure doesn’t tell you anything about the quality of the tests being run and therefore that code that’s covered by tests automatically has a high quality.

One way to define quality software is to look at it from an external and internal point of view. From an external point of view, customers experiencing bugs in production should be a rare occurrence. From internal point of view, developers should feel comfortable adding or changing functionality without being afraid of introducing bugs that slip into production. You could say that you test enough if both are the case.

Keeping track of code coverage can help you to get a partial answer about whether or not you test enough. It does this by showing the ratio between tested and untested code. It’s interesting to know what code is covered, but it’s far more interesting to know what code is not covered by your tests. In code that is not covered by tests, it’s easier to introduce a bug that gets into production than in code that is tested (well).

So does that mean you should strive for 100% code coverage? There are always parts of code that are hard to test (e.g. I/O, multi-threaded and network code) and the benefits are not always worth the costs. If you strive for a code coverage of 100% it sounds like you make code coverage a goal and not using it as an aid. If your way of working includes writing unit tests for every new feature or change (preferably before writing production code by practicing Test Driven Development), you will automatically end up with a code coverage of about 80%-90% for those features or changes.

Beware however, to set bars (e.g. we need 80% code coverage), especially if you do it for somebody else. I’ve seen developers writing bogus tests to increase code coverage just because a certain percentage was forced upon them. The tests they created exercised the code, but didn’t actually verify if the behavior was correct. Martin Fowler calls this anti-pattern Assertion free testing. As a result, it the code coverage was high but could't be used for anything useful. A nice quote of Brian Marick about this:

I expect a high level of coverage. Sometimes managers require one. There's a subtle difference.

Measuring code coverage can help increase software quality by identifying untested parts of your application. Beware to make having a high percentage of coverage a goal on its own and don’t force it on others. If you do, chances are it will lead to counter-productive behavior. If you start with a low coverage, it’s far more effective to make writing unit tests for all new functionality part of your way of working. Combine this with identifying high-risk areas that are currently uncovered and write test for those areas. This way, you’ll soon reap the benefits and achieve the real goal: increase the quality of your software in an effective way.

What did you think about this post?