80

I've just started using Coverage.py module and so decided to make a simple test to check how it works.

Sample.py

def sum(num1, num2):
    return num1 + num2


def sum_only_positive(num1, num2):
    if num1 > 0 and num2 > 0:
        return num1 + num2
    else:
        return None

test.py 

from sample import sum, sum_only_positive

def test_sum():
    assert sum(5, 5) == 10

def test_sum_positive_ok():
    assert sum_only_positive(2, 2) == 4

def test_sum_positive_fail():
    assert sum_only_positive(-1, 2) is None

As you see, all my code is covered with tests and py.test says all of them pass. I expect Coverage.py to show 100% coverage. Well, no.

Coverage.py results

Well, Coverage.py may not see test.py file, so I copied test functions to sample.py file and ran Coverage again:
enter image description here

Then I added this block of code:

if __name__ == "__main__":
    print(sum(2, 4))
    print(sum_only_positive(2, 4))
    print(sum_only_positive(-1, 3))

and removed all test functions. After that, Coverage.py shows 100%:

enter image description here

Why is it so? Shouldn't Coverage.py show code test coverage, not just execution coverage? I've read an official F.A.Q. for Coverage.py, but can't find the solution.
Since many SO users are familiar with code testing and code coverage, I hope you can tell me, where am I mistaken.

I have just one thought here: Coverage.py may simply watch which lines of code aren't executed so I should write tests for those lines. But there're lines which are executed already but aren't covered with tests so Coverage.py will fail here.

Improve this question
4
  • 1
    How do you invoke coverage / pytest? Commented Apr 9, 2016 at 13:21
  • @Rogalski pytest: python -m py.test test.py and coverage: python -m coverage run sample.py (on Windows) Commented Apr 9, 2016 at 13:25
  • it doesn't show 100%, it shows the same lines as not covered...it doesn't work for me still. i copied pasted your code and make sure i have py.test and coverage.py with pip. i am seeing same in command line and in intellij, please LMK. Commented Apr 5, 2019 at 17:15
  • Please see demo-06-test-coverage and demo-07-coverage-reports-combined as an example how to configure "coverage" in a project using the src-layout and pyproject.toml. Commented Dec 11, 2023 at 13:17

5 Answers 5

Reset to default
46

Coverage looks for a .coverage file to read and generate that report for you. Py.test on its own does not create one. You need py.test plugin for coverage:

pip install pytest-cov

If you already have it, then you can run both at once like this:

py.test test.py --cov=sample.py

Which means run test module test.py and record/display coverage report on sample.py.

If you need to have multiple test runs and accumulate their recorded coverage and then display a final report, you can run it like this:

py.test test.py --cov=sample.py --cov-report=
py.test test.py --cov=sample2.py --cov-report=
py.test test.py --cov=sample3.py --cov-report=

Which means run test module test.py and record (only) coverage on sample.py - don't display a report.

Now you can run coverage command separately for a complete report:

coverage report -m

The command above simply displays a formatted coverage report based on the accumulated .coverage data file from previous test runs. -m means show lines missed i.e. lines not covered by tests:

Name        Stmts   Miss  Cover   Missing
-----------------------------------------
sample.py       6      0   100%

Coverage supports more switches like --include and --omit to include/exclude files using path patterns. For more info check out their docs: https://coverage.readthedocs.io/en/6.0.2/source.html?highlight=reporting#reporting

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Doesn't work if the file to examine is a submodule in a package that is imported in the test. Whatever I specify at --cov=<> - just the file name, relative or absolute path - I get "ERROR: Failed to generate report: No data to report."
"asterisk/mydir/asterisk" works for omit. I would write * for asterisk but it turns into bold.
Use --branch to also add branch coverage.
You could also do something like coverage run --source=<module-name> -m pytest <test-location> instead of py.test test.py --cov=sample.py if the additional dependency isn't worth it.
ERROR: usage: py.test [options] [file_or_dir] [file_or_dir] [...] py.test: error: unrecognized arguments: --cov=test.py
|
29

It's a little hard to parse through your experiments, and you haven't included the command lines you used with each experiment. But: if you run the tests with:

python -m py.test test.py

then you can run them under coverage.py with:

coverage run -m py.test test.py
Improve this answer

5 Comments

By the way, I've included commands used to run tests and coverage just under my post in second comment: pytest: python -m py.test test.py and coverage: python -m coverage run sample.py (on Windows). I see your second command differs from mine, will check this, thank you!
Maybe the problem here is a basic misunderstanding: you aren't supposed to run the tests, and then coverage. You are supposed to use coverage to run your tests (the way I recommend), or enable coverage during the running of tests (for example with a test runner's coverage plugin).
Doesn't work for me :( coverage -m -> No such option -m. coverage run py.test test.py -> Unknown command py.test (py.test is installed)
@NedBatchelder This method includes test.py in the generated report. Is that considered a best practice when using a tool like coverage? By that I mean is the intent to always measure both the coverage on the module to be tested and the test itself? If not how would I exclude the test.py coverage from the report?
I see no reason to exclude tests from measurement, and only advantages. You have code in your tests, you want to know that it is being executed. Coverage.py can tell you that.
27

The below command worked for me:

coverage run --source=sample -m pytest test.py

coverage report -m
Improve this answer

2 Comments

A heads up for anyone else stumbling upon this question. As of 2021 & Coverage v5.3.1, this is the recommended way to do it.
I am getting No data to report. ;/
6

putting __init__.py in every test folder resolves the original question and shows correct coverage.

Improve this answer

Comments

1

For module, you can install coverage and pytest. Then, to run a coverage for the tests and corresponding module, you can do the following.

This will run pytests in the directory tests and use pkg_name as the source:

coverage run --source=pkg_name -m pytest -x tests

then, when you generate the report using coverage report, you will get a list of coverages for each of the script in your package (pkg_name). Something like this:

Name Stmts Miss Cover
pkg_name\main.py 10 10 0%
pkg_name\__init__.py 3 0 100%
pkg_name\module\M1.py 37 29 22%
pkg_name\module_init_.py 0 0 100%
Total ### ### X%
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.