View Issue Details

IDProjectCategoryView StatusLast Update
0001674XdebugCode Coveragepublic2019-08-05 18:58
Reporterjoerieger@quickenloans.comAssigned Toderick 
PrioritynormalSeveritymajorReproducibilityalways
Status confirmedResolutionopen 
PlatformLinuxOSDebianOS Version
Product Version2.7.2 
Target Version3.0.0devFixed in Version 
Summary0001674: Inconsistent Path & Branch Coverage Reported
DescriptionDepending on what, or possibly how many, files are analyzed for path & branch coverage, XDebug reports significantly different results for path & branch coverage.

I've set up a GitHub repo here with a Docker container that consistently reproduces the output.
https://github.com/jmrieger/xdebug-issue

I've tried this on the following setups:
php-fpm from Docker, 7.3.3, and PHPUnit 8 master
php-fpm from Homebrew, 7.3.6, and PHPUnit 8 master
php-fpm from Homebrew, 7.1.29, with PHPUnit 7.5.7-6
Steps To Reproduce1. Gather code coverage on PHPUnit\Framework\Assert, and analyze the results. When calling xdebug_stop_code_coverage, pass 1 to clear results.
1a. In my testing, clearing results had no bearing on the results.
2. Properly, 193 functions are found in that class.
3. Run PHPUnit::main(), passing a flag to only test one file: tests/unit/Util/JsonTest.php
4. Re-run the same coverage gathering steps. Only 2 functions are returned.
5. Re-running step 4 also returns only 2 functions.
Additional InformationCould also fall into the category of "Usage Problems (Wrong Results)", but I'll leave that up to you to reassign.
Tagsbranches, coverage, Docker, functions, path
Operating System
PHP Version7.3.2-7.3.4

Activities

derick

2019-07-19 11:48

administrator   ~0005074

This test case is way too big. Can you please give me something where:
- I don't need Docker
- There is a simple test, without all of phpunit, where two code coverage runs show this problem?

You need to show what your expected result is (not just "two functions in file", but the real contents, and what you expected. Small test cases are best, and ones that don't require PHPUnit are highly recommended.

joerieger@quickenloans.com

2019-07-19 12:12

reporter   ~0005075

I will do my best to reproduce a more minimal, self-contained example. Apparent scope + size were one of the reasons I went with the Docker + PHPUnit route, but I understand the debugging difficulty.

joerieger@quickenloans.com

2019-07-19 17:50

reporter   ~0005081

Hi derick - I updated the linked repository to include a more self-contained example that doesn't run on Docker and doesn't use PHPUnit. It looks like any run through a file, regardless of if the functions are static or not, will produce a different count of the functions within the file.

Direct link to the 3 files needed for this example:
https://github.com/jmrieger/xdebug-issue/blob/master/driver.php
https://github.com/jmrieger/xdebug-issue/blob/master/SampleClass.php
https://github.com/jmrieger/xdebug-issue/blob/master/SampleStaticClass.php

It's not as "small" as it could be, but I did want to demonstrate that the unexpected behavior isn't limited to whether a function is static or not.

joerieger@quickenloans.com

2019-07-19 17:56

reporter   ~0005082

Additional clarity on my statement above: The first pass through when processing code coverage returns a result I expect - it identifies the actual number of functions present in the specified file, as well as coverage information for each of the functions, regardless of if the function was called.

The second pass through each file, the number of functions present in the specified file only includes functions which were called during the code coverage run. I expect the behavior to be the actual number of functions in the file, with only the branch and path information varying.

derick

2019-08-05 18:34

administrator   ~0005106

Hi,

I've had a moderately good look, and with your smaller example I can reproduce it. I think I know that the problem is that the xdebug_code_coverage_stop() call also removes all discovered information regarding branches, paths, and possible executable/non-executable lines. This can be fixed, but it is hard to do that without refactoring much of this code. I therefore don't think I want to do the refactoring for Xdebug 2.8, but rather leave it to 3.0. It will also likely drastically improve performance.

With as caveat being me not knowing how PHP_CodeCoverage exactly works, I do believe Sebastian has some workarounds for the merging of runs. In theory, you should be able to find out for each "run" which lines, paths, and branches *have* been hit; just not which ones that haven't been. You should be able to extrapolate that from earlier runs (and you'll know which lines won't have been hit too). If it's code that hasn't been seen before, these new functions will have the full list of branches, paths, and lines in the output. I know, it's not ideal, but I think it is possible to construct a full result for each non-first "run" with some logic.

Happy to chat about this on IRC or something like that about it — please email if you'd like to do that.

For now, I'll have to re-target this for Xdebug 3.0.

cheers,
Derick

joerieger@quickenloans.com

2019-08-05 18:58

reporter   ~0005108

Re-targeting to 3.0 is more than acceptable, given the scale of changes you mentioned.

I did note that much of Sebastian's code revolves around merging runs, though how I ran into this issue was related to that implementation - if you unintentionally forget the coverage data (or even intentionally discard it, say for indirectly covered code), you lose the visibility into earlier coverage data.

I'll check my calendar and see if there's a time we can meet on IRC. Any additional support that I can provide to help with the improvements I'm more than glad to provide.

Thanks,
Joe

Issue History

Date Modified Username Field Change
2019-06-12 19:38 joerieger@quickenloans.com New Issue
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: branches
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: coverage
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: Docker
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: functions
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: path
2019-06-28 11:19 derick Target Version => 2.8.0dev
2019-07-19 11:48 derick Assigned To => derick
2019-07-19 11:48 derick Status new => feedback
2019-07-19 11:48 derick Note Added: 0005074
2019-07-19 12:12 joerieger@quickenloans.com Note Added: 0005075
2019-07-19 12:12 joerieger@quickenloans.com Status feedback => assigned
2019-07-19 17:50 joerieger@quickenloans.com Note Added: 0005081
2019-07-19 17:56 joerieger@quickenloans.com Note Added: 0005082
2019-08-05 18:34 derick Status assigned => confirmed
2019-08-05 18:34 derick Note Added: 0005106
2019-08-05 18:35 derick Target Version 2.8.0dev => 3.0.0dev
2019-08-05 18:58 joerieger@quickenloans.com Note Added: 0005108