SEARCH FINANCIAL SERVICES INFRASTRUCTURE SECURITY SCIENCE INTERVIEWS

 

     

Research on code coverage

By Jaime Jorge,  CEO and Co-founder of Codacy

January 10, 2017

Best practices dictate that you should track code coverage. Code coverage helps you assess the proportion of code that was tested and (more importantly) not tested.

Why is it important to track coverage? Our users report that coverage is important because it is the confidence level of code being correct and with less bugs. Weíve written previously about code coverage, this is a follow up based on industry data.

In this article we share some evidence of how different companies do code coverage, based on a survey of 680 companies about their code quality practices. Weíll answer three questions:

  1. Who tracks code coverage? What happens when you donít?
  2. What thresholds are used for code coverage?
  3. What workflow rules are implemented with code coverage?

Without further adoÖ

Who tracks code coverage? What happens when you donít?

Letís first look at some of the numbers from our research.

Code coverage breakdown

Half of the companies track code coverage. Weíll see in a second how that impacts code quality and other parts of development.

This means that half of the companies cannot trust their code to be without bugs.

The bigger the company the more likely it is to track code coverage, as shown in the chart below.

Coverage tracking by company size

You can see that the larger the company the more likely it is to be tracking code coverage.

Not surprisingly, the older the company the more likely it is to track code coverage too.

Coverage tracking by company age

So the first major question we need to ask is: what is the impact of not tracking coverage?

Here is a quick summary of the impact:

  • 68% of people who do not monitor code coverage feel they should have more time to maintain code.
  • 59% of those who do monitor code coverage feel they should have more time to maintain code.
  • People who monitor code coverage perceive their code quality to be 7,2% better

Conclusion: although subtle, there are benefits to tracking code coverage. 
People who track code coverage feel that the quality of their code is higher, and donít feel they need to spend more time reviewing code. This the likely reason why more mature companies, which tend to be larger and older, implement code coverage systematically.

Code coverage thresholds?

Code coverage at any given time is really just a number. Itís the evolution of that number that matters. Code coverage that goes down is an indication that new code is not being tested therefore the confidence level is going down. If itís going up, normally itís a good sign.

Minimum code coverage threshold to accept a Pull Request? Most companies donít

Only 34% of companies require a minimum code coverage threshold to accept a pull request.

What is surprising is the difference between those 34% and 65%.

Here is the real difference between those two groups:

  • Companies that require a minimum code coverage threshold perceive their code quality to be 6,4% better.
  • Companies that donít enforce code coverage thresholds say their biggest problem is technical debt.
  • On the other hand, quality comes at a cost: companies that enforce code coverage thresholds say their biggest problem is speed of delivery.

Another important question is: what is the ideal threshold?

Code coverage popularity

The vast majority of companies that have a code coverage target have set it at 80%. This means the companies that do track code coverage actually set a very high bar with each pull request.

Conclusion: having code coverage thresholds increased perceived code quality and decreased time spent on tackling technical debt. Companies that required this threshold set the bar, on average, at 80%.

Code coverage workflow rules?

Our final consideration is about code coverage workflow rules
Do companies require code code coverage to increase whenever new code is introduced? Or do they enforce other rules, for example requiring code coverage not to go down?

First question is: how many companies have code coverage workflow rules?

Terms of Use | Copyright © 2002 - 2016 CONSTITUENTWORKS SM  CORPORATION. All rights reserved. | Privacy Statement