C# Code Coverage with Azure DevOps

I have a couple of times wanted to implement code coverage stats in build pipelines. But every time I couldn’t make it work and abandoned trying it more. Recently it came again. I wanted code coverage to new build pipeline, tried it, and failed (again). Even if I had done it just earlier in Microsoft Learn. I was frustrated (and bit shamed); why I can’t do this, it can’t be difficult!

I made a decision: I will finally learn it. A good way to learn it is to write a blog post about it. Then I will find this recipe later when I will need it again. There are many blog posts about code coverage in Azure DevOps but hopefully, this helps someone else also. At least in other blog posts, there wasn’t a case where there are many unit test projects; everyone seems to have a simple case with only one unit test project. Here is an instruction to get code coverage for solutions with multiple unit test projects.

I have used .Net Framework because that was the target framework I was recently fighting against. But I also tested that this works also for .Net Core without any changes to the codes and pipelines shown here. The test framework is MSTest.

TL;DR

• Add coverlet.msbuild NuGet to all unit test projects.
• Use following YAML in dotnet test in build pipeline with relative path in CoverletOutput parameter. This generates code coverage report files.

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'
    arguments: '/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=./MyCoverage/'
    publishTestResults: true

• Add the following YAML after dotnet test to publish code coverage report to build.

- task: PublishCodeCoverageResults@1
  displayName: 'Publish Code Coverage Results'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/**/MyCoverage/coverage.cobertura.xml'
    failIfCoverageEmpty: true

• Enjoy code coverage reports in builds.

Simple Unit Test

Here is our simple static class with two methods to start with.

public static class MyMathOperations
{
  public static int Sum(int a, int b)
  {
    return a + b;
  }

  public static int Difference(int a, int b)
  {
    return a - b;
  }
}

At the beginning we have only one unit test.

[TestClass]
public class MyMathOperationsTests
{
  [TestMethod]
  public void TestSum()
  {
    Assert.AreEqual(10, MyMathOperations.Sum(4, 6));
  }
}

What should be code coverage? This should be a simple one. We have two one-line methods and only one is tested. So code coverage should be 50%.

How to Get Code Coverage Locally?

I will use coverlet to get code coverage in Cobertura format. When adding a new .Net Core MSTest project in Visual Studio, the unit test project will have coverlet.collector NuGet installed at the very beginning. But in my case, I couldn’t make this work. Instead, I installed coverlet.msbuild NuGet (v2.8.0).

Here is a dotnet test command to run tests and get code coverage in Cobertura format to MyCoverage folder.

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=./MyCoverage/

...

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1.0577 Seconds

Calculating coverage result...
  Generating report '.\MyCoverage\coverage.cobertura.xml'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+
| MyMath | 50%  | 100%   | 50%    |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 50%  | 100%   | 50%    |
+---------+------+--------+--------+
| Average | 50%  | 100%   | 50%    |
+---------+------+--------+--------+

We got our “official” code coverage and it is 50% what was our expected code coverage! As we can see it outputs code coverage ascii table and tells report has been generated to MyCoverage\coverage.cobertura.xml. Here is how the coverage.cobertura.xml file looks like.

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0.5" branch-rate="1" version="1.9" timestamp="1585509793" lines-covered="3" lines-valid="6" branches-covered="0" branches-valid="0">
  <sources>
    <source>D:\</source>
  </sources>
  <packages>
    <package name="MyMath" line-rate="0.5" branch-rate="1" complexity="2">
      <classes>
        <class name="MyMath.MyMathOperations" filename="ohjelmointi\CodeCoverage\MyMath\MyMathOperations.cs" line-rate="0.5" branch-rate="1" complexity="2">
          <methods>
            <method name="Sum" signature="(System.Int32,System.Int32)" line-rate="1" branch-rate="1">
              <lines>
                <line number="6" hits="1" branch="False" />
                <line number="7" hits="1" branch="False" />
                <line number="8" hits="1" branch="False" />
              </lines>
            </method>
            <method name="Difference" signature="(System.Int32,System.Int32)" line-rate="0" branch-rate="1">
              <lines>
                <line number="11" hits="0" branch="False" />
                <line number="12" hits="0" branch="False" />
                <line number="13" hits="0" branch="False" />
              </lines>
            </method>
          </methods>
          <lines>
            <line number="6" hits="1" branch="False" />
            <line number="7" hits="1" branch="False" />
            <line number="8" hits="1" branch="False" />
            <line number="11" hits="0" branch="False" />
            <line number="12" hits="0" branch="False" />
            <line number="13" hits="0" branch="False" />
          </lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

We don’t have to understand this (at least in real-world coverage reports won’t be this simple) but we can see there is a code coverage report for the Sum method in lines 11-17, and then there is a code coverage for the Difference method in lines 18-24. Next we will generate the same report in Azure DevOps build pipeline.

Azure DevOps Build Pipeline

At first, we create a really simple build pipeline that only runs tests.

trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'

Let’s now add code coverage but putting the same arguments to the dotnet test command as we put earlier when we generated code coverage locally. Only change is in CoverletOutput where we put $(Build.SourcesDirectory)/MyCoverage/ instead of ./MyCoverage.

trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'
    arguments: '/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/MyCoverage/'
    publishTestResults: true

Now when we look at the build logs we can see that there is the same code coverage ascii table as we got locally earlier.

But when we look at the Code Coverage tab, there isn’t any code coverage.

Seems that code coverage data is generated as we see it in build logs. But we want it to appear in the Code Coverage tab also. There is a Publish Code Coverage Results task that does this for us.

Because our code coverage report is in Cobertura format, we will put codeCoverageTool: 'Cobertura'. The next one is the most important: summaryFileLocation. This is where a code coverage report should be found. If we look at previous build logs we can find that it created this to d:\a\1\s\MyCoverage\coverage.cobertura.xml. And here d:\a\1\s is $(Build.SourcesDirectory) that we defined in dotnet test‘s arguments to output code coverage report. Here is this new step to publish code coverage results in build pipeline.

- task: PublishCodeCoverageResults@1
  displayName: 'Publish Code Coverage Results'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/MyCoverage/coverage.cobertura.xml'
    failIfCoverageEmpty: true

Now after the build, there is a report in the Code Coverage tab!

That is a fine report but there are more detail reports for each class. Just click the class name at the end of the page (MyMath.MyMathOperations in the above image) and we can see line by line view.

Code Coverage from Many Unit Test Projects

Often we have many unit test projects so the previous example is too simple in real life. Let’s create another project and unit test project for it to explore this situation.

public static class MyBooleanOperations
{
  public static bool Xor(bool a, bool b)
  {
    if (a == b)
      return false;
    else
      return true;
  }
}

[TestClass]
public class BooleanOperationsTests
{
  [TestMethod]
  public void Xor_BothAreTrue_ReturnsFalse()
  {
    Assert.IsFalse(MyBooleanOperations.Xor(true, true));
  }
}

Let’s commit this and run build in Azure DevOps which triggers our build. Build logs show us that both unit test projects are run and code coverage reports are also generated.

Test run for d:\a\1\s\MyBooleanTests\bin\Debug\net472\MyBooleanTests.dll
...
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 2.9576 Seconds

Calculating coverage result...
  Generating report 'd:\a\1\s\MyCoverage\coverage.cobertura.xml'

+-----------+------+--------+--------+
| Module    | Line | Branch | Method |
+-----------+------+--------+--------+
| MyBoolean | 80%  | 50%    | 100%   |
+-----------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 80%  | 50%    | 100%   |
+---------+------+--------+--------+
| Average | 80%  | 50%    | 100%   |
+---------+------+--------+--------+

Test run for d:\a\1\s\MyMathTests\bin\Debug\net472\MyMathTests.dll
...
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1.6457 Seconds

Calculating coverage result...
  Generating report 'd:\a\1\s\MyCoverage\coverage.cobertura.xml'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+
| MyMath | 50%  | 100%   | 50%    |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 50%  | 100%   | 50%    |
+---------+------+--------+--------+
| Average | 50%  | 100%   | 50%    |
+---------+------+--------+--------+

But when we go to the code coverage tab it seems that only MyMathTests is reported.

At first, I thought this is useless for multiple unit test projects. But then I realized one argument in dotnet test command: /p:CoverletOutput=$(Build.SourcesDirectory)/MyCoverage/. And when looking more to our build logs we can find this twice: Generating report 'd:\a\1\s\MyCoverage\coverage.cobertura.xml'. Each unit test project’s code coverage report is being saved to the same place. And thus the later report has overwritten the first report!

One solution for this is to give different paths for each code coverage report. This can be done by putting the relative path to CoverletOutput argument in dotnet test (line 13) and using a wild card in PublishCodeCoverageResults‘ summaryFileLocation (line 20).

trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'
    arguments: '/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=./MyCoverage/'
    publishTestResults: true

- task: PublishCodeCoverageResults@1
  displayName: 'Publish Code Coverage Results'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/**/MyCoverage/coverage.cobertura.xml'
    failIfCoverageEmpty: true

Now after build, we have code coverage report from both unit test projects.

One Small Shortage with Coverage Percent

Unfortunately, I am not 100% happy with this. There is one small shortage: code coverage percent in build’s summary page isn’t always valid. In our case it shows 80% code coverage:

But when we go to code coverage tab it says code coverage to be 63.6%.

There is a significant difference and it is not because of code coverage calculation (line, branch, etc.). I know both are line coverages. What is the reason for the difference? Which is right? The later, code coverage tab, has the right code coverage percent. The reason is that both tabs take this number from different files!

Summary tab takes it from alphanumerically first unit test project’s coverage.cobertura.xml file. In our case, it is from MyBooleanTests project. Fortunately, code coverage tab’s percent is from the merged code coverage report and includes all unit test projects.

If there are many unit test projects, the summary tab’s code coverage percent is unfortunately not valid. See valid value from the code coverage tab.

Conclusion

In this blog post, I walked through how to get code coverage report when there are many unit test projects. Just remember that the build’s summary tab’s coverage percent isn’t a valid one. Otherwise, it works fine. (Read TL;DR at the beginning for a summary of how to do all this.)

Sources

• Coverlet
• Perform code coverage testing from Microsoft Learn