Theory: Covering the code with tests

As your project grows, it gets difficult to determine which code we have already tested and which haven't. At the same time, this need arises regularly. It usually happens when not all team members write tests not in a responsible way. In this case, the quality of the project may suffer.

But there is a solution. We can measure the code testability. In this lesson, we'll get acquainted with a metric that helps calculate the number of tests and the testing quality.

How test coverage works

Developers refer to the code coverage metric as code test coverage. We analyze it by test frameworks that consider the relationship of the lines involved in the tests to all lines of the source code. For example, if there's a conditional construction in the code not covered by tests, then all lines of code in this construction aren't covered.

In Pytest, it is simple to measure coverage. It is enough to install one dependency and run the tests with the right flag:

poetry add pytest-cov

poetry run pytest --cov

# There is a sample output
tests/test_example.py ..      [ 66%]
tests/test_hexlet_pytest.py . [100%]

Name                        Stmts   Miss  Cover
-----------------------------------------------
hexlet_pytest/__init__.py       1      0   100%
hexlet_pytest/example.py        4      1    75%
-----------------------------------------------
TOTAL                           5      1    80%

After all the tests, Pytest outputs a summary table for each file. It shows the percentage of code test coverage. The example above shows that we covered 100% of the code in the file _init_.py, but only 75% is in the example.py file. The total code coverage is 80%.

Note that coverage is highly dependent on which tests we have performed. If some tests fail with an error, then Pytest will show much less coverage — the tests won't get to the whole code.

Therefore, coverage is measured when all the tests are green. This statistic helps you find places where there aren't enough tests. We can add them as we move along. This statistic will begin to skyrocket if there are no tests in the project. But as we get closer to 90 percent, we'll have to fight for every line of code.

What should the coverage be

Coverage doesn't guarantee that the code will work correctly in all situations.

We cannot track logical errors in the code by coverage alone. It requires tests for the same functionality but with a different data set. As a rule, these are tests for borderline cases. There is good practice in development: before fixing bugs, you need to write tests that reproduce them, and only then you can fix them.

What coverage is considered acceptable? 100% coverage looks beautiful, but it is too difficult to achieve it. Moreover, it's pointless for most projects — the effort spent won't pay off. Most developers agree that 80% is a good coverage. We can stop there.