Reaching 100% Code Coverage in Rust

As of commit aa3536d0, Trane can boast a code coverage score of 100%. I had difficulty finding a comprehensive guide when I was getting started, so this post is meant to explain how I got there, the tools I used, the problems I ran into, and the lessons I learned. Hopefully this will save time to someone else who is trying to add test coverage to a Rust project.

The Tools

I used grcov to generate the coverage information. This requires that the tests are run in the nightly version of Rust with special flags that create the coverage information. I used the grcov GitHub Action to generate the report. This is a fork of the original repository, which appears to be abandoned. The fork is needed to support excluding lines from the coverage report (more on that later).

For comparing the coverage reports I used Coveralls. This is a free service for open-source repositories that provides a web interface for comparing the reports between commits. Here's the example report of the commit mentioned above. I used the official Coveralls GitHub Action to upload the report to the service.

Here are the configuration files for reference:

The file to set up all the GitHub actions: coverage.yaml
The file to configure grcov: grcov.yml

Configuring grcov

I ran into multiple issues with grcov which required some configuration to work around them.

The initial report included coverage information for all the dependencies. Figuring out the fix for this issue took me a while, but I fixed by setting the ignore option in the configuration file to exclude any directory with .cargo in its path.
The initial reports included branch coverage information. I decided to stick initially to line coverage, as I could not get it working with the fix to the previous issue. This required that I set the branch option to false in the configuration file. I might revisit this decision later and try to get branch coverage working.
Some lines were marked as not covered, even though they don't contain actual code. They are ignored by setting the excl-line in the configuration file.
- Some files have the first line marked as not covered. All my files have module-level comments, so I fixed the issue by excluding all lines that start with //! from the coverage report.
- #[derive... lines are marked as not covered. This is because the code generated by those macros is not covered. Given that I trust the compiler to generate the correct code, I decided to ignore those lines.

Other issues

I ran into an issue in which fields in a struct or some variants of an enum would appear as not covered in the Coveralls report. This is related to the issue with #[derive...] lines. In particular, if a type implements the Clone trait, but the struct or some variant of an enum is not cloned anywhere in the code, some fields will appear as not covered. The fix is to remove the Clone trait if you are not using it or add tests that clone the type if it's an enum and not all variants are cloned in the code.

Some other issues with partial hits using grcov and Rust have been reported. See this issue for more details.

There's also the problem that grcov does not appear to be maintained at the moment, apart from automated commits updating its dependencies. I do not know whether the software has been abandoned or is mostly complete at this point. However, issues have not been addressed in a while, which would indicate that the latter option is more likely.

Getting to 100% Code Coverage

In this section, I will go over some of the steps I took to get to 100% code coverage. The first step was configuring grcov to exclude any arbitrary line or section from the coverage report. This is done in two steps.

The first step was to modify the configuration file to exclude any line with the grcov-excl-line string from the final results. This allows me to exclude lines that are hard to test or not very relevant from the coverage report. For example:

The modules that use SQLite call a method to prepare a statement. This is a fallible operation, but this particular line will never fail because the statement does not change. If a bad change to the statement were introduced, the tests would fail anyway. Here's an example from the practice_stats module, which stores the results of the user's practice sessions:

#![allow(unused)]
fn main() {
// Retrieve the exercise trials from the database.
let connection = self.pool.get()?;
let mut stmt = connection
    .prepare_cached(
        "SELECT score, timestamp from practice_stats WHERE unit_uid = (
          SELECT unit_uid FROM uids WHERE unit_id = ?1)
          ORDER BY timestamp DESC LIMIT ?2;",
    )
    .with_context(|| "cannot prepare statement")?; //grcov-excl-line
}

Adding an entry to the search index is also a fallible operation, but there are already tests that verify that search work, so there's little value in having a test that triggers this error. This is the code from the course_library module, which opens the courses stored in the file system for use with Trane:

#![allow(unused)]
fn main() {
  // Add the lesson manifest to the search index.
  Self::add_to_index_writer(
      index_writer,
      lesson_manifest.id,
      &lesson_manifest.name,
      &lesson_manifest.description,
      &lesson_manifest.metadata,
  )?; // grcov-excl-line
}

The second step was to configure grcov to exclude entire sections of code from the report. For example, the Trane struct that is the main entry point for the library reimplements all the interfaces so that users do not have to access the fields of the struct to call those methods. Thus, those lines appeared as uncovered in the report but are actually covered by the all the other tests. I fixed this by configuring the excl-start and excl-stop options to exclude sections of the code from the final report. Here's an example of some of the code that was excluded:

#![allow(unused)]
fn main() {
impl ReviewList for Trane {
    fn add_to_review_list(&mut self, unit_id: &Ustr) -> Result<()> {
        self.review_list.write().add_to_review_list(unit_id)
    }

    fn remove_from_review_list(&mut self, unit_id: &Ustr) -> Result<()> {
        self.review_list.write().remove_from_review_list(unit_id)
    }

    fn all_review_list_entries(&self) -> Result<Vec<Ustr>> {
        self.review_list.read().all_review_list_entries()
    }
}
}

Once those two steps were configured, the coverage was around 90%. The remaining 10% consisted of edge cases that had not been tested yet. For example, there was no test to ensure that trying to open a Trane library with an invalid path would fail. Below is a list of example commits that introduce additional tests to cover those edge cases:

There are about a dozen more commits that gradually took the coverage to 100%. Finally, the commit linked at the top of the post was the first one to reach 100% code coverage. This commit adds a test to verify that the score cache does not return a score for courses that have all of its lessons added to the blacklist.

Conclusion

The effort to get to 100% code coverage was not very time-consuming and uncovered some minor bugs that had gone unnoticed. However, the core of the logic was already covered by existing tests and none of the bugs that were uncovered were critical. Still, I think the effort has given me more confidence to make changes to the code base without worrying about introducing new bugs or modifying existing behavior.

The Trane Book