View Issue Details

IDProjectCategoryView StatusLast Update
0001674XdebugCode Coveragepublic2021-08-26 17:02
Reporterjoerieger@quickenloans.com Assigned Toderick  
PrioritynormalSeveritymajorReproducibilityalways
Status confirmedResolutionopen 
PlatformLinuxOSDebian 
Product Version2.7.2 
Target Version3.2dev 
Summary0001674: Inconsistent Path & Branch Coverage Reported
DescriptionDepending on what, or possibly how many, files are analyzed for path & branch coverage, XDebug reports significantly different results for path & branch coverage.

I've set up a GitHub repo here with a Docker container that consistently reproduces the output.
https://github.com/jmrieger/xdebug-issue

I've tried this on the following setups:
php-fpm from Docker, 7.3.3, and PHPUnit 8 master
php-fpm from Homebrew, 7.3.6, and PHPUnit 8 master
php-fpm from Homebrew, 7.1.29, with PHPUnit 7.5.7-6
Steps To Reproduce1. Gather code coverage on PHPUnit\Framework\Assert, and analyze the results. When calling xdebug_stop_code_coverage, pass 1 to clear results.
1a. In my testing, clearing results had no bearing on the results.
2. Properly, 193 functions are found in that class.
3. Run PHPUnit::main(), passing a flag to only test one file: tests/unit/Util/JsonTest.php
4. Re-run the same coverage gathering steps. Only 2 functions are returned.
5. Re-running step 4 also returns only 2 functions.
Additional InformationCould also fall into the category of "Usage Problems (Wrong Results)", but I'll leave that up to you to reassign.
Tagsbranches, coverage, Docker, functions, path
Operating System
PHP Version7.3.2-7.3.4

Activities

derick

2019-07-19 11:48

administrator   ~0005074

This test case is way too big. Can you please give me something where:
- I don't need Docker
- There is a simple test, without all of phpunit, where two code coverage runs show this problem?

You need to show what your expected result is (not just "two functions in file", but the real contents, and what you expected. Small test cases are best, and ones that don't require PHPUnit are highly recommended.

joerieger@quickenloans.com

2019-07-19 12:12

reporter   ~0005075

I will do my best to reproduce a more minimal, self-contained example. Apparent scope + size were one of the reasons I went with the Docker + PHPUnit route, but I understand the debugging difficulty.

joerieger@quickenloans.com

2019-07-19 17:50

reporter   ~0005081

Hi derick - I updated the linked repository to include a more self-contained example that doesn't run on Docker and doesn't use PHPUnit. It looks like any run through a file, regardless of if the functions are static or not, will produce a different count of the functions within the file.

Direct link to the 3 files needed for this example:
https://github.com/jmrieger/xdebug-issue/blob/master/driver.php
https://github.com/jmrieger/xdebug-issue/blob/master/SampleClass.php
https://github.com/jmrieger/xdebug-issue/blob/master/SampleStaticClass.php

It's not as "small" as it could be, but I did want to demonstrate that the unexpected behavior isn't limited to whether a function is static or not.

joerieger@quickenloans.com

2019-07-19 17:56

reporter   ~0005082

Additional clarity on my statement above: The first pass through when processing code coverage returns a result I expect - it identifies the actual number of functions present in the specified file, as well as coverage information for each of the functions, regardless of if the function was called.

The second pass through each file, the number of functions present in the specified file only includes functions which were called during the code coverage run. I expect the behavior to be the actual number of functions in the file, with only the branch and path information varying.

derick

2019-08-05 18:34

administrator   ~0005106

Hi,

I've had a moderately good look, and with your smaller example I can reproduce it. I think I know that the problem is that the xdebug_code_coverage_stop() call also removes all discovered information regarding branches, paths, and possible executable/non-executable lines. This can be fixed, but it is hard to do that without refactoring much of this code. I therefore don't think I want to do the refactoring for Xdebug 2.8, but rather leave it to 3.0. It will also likely drastically improve performance.

With as caveat being me not knowing how PHP_CodeCoverage exactly works, I do believe Sebastian has some workarounds for the merging of runs. In theory, you should be able to find out for each "run" which lines, paths, and branches *have* been hit; just not which ones that haven't been. You should be able to extrapolate that from earlier runs (and you'll know which lines won't have been hit too). If it's code that hasn't been seen before, these new functions will have the full list of branches, paths, and lines in the output. I know, it's not ideal, but I think it is possible to construct a full result for each non-first "run" with some logic.

Happy to chat about this on IRC or something like that about it — please email if you'd like to do that.

For now, I'll have to re-target this for Xdebug 3.0.

cheers,
Derick

joerieger@quickenloans.com

2019-08-05 18:58

reporter   ~0005108

Re-targeting to 3.0 is more than acceptable, given the scale of changes you mentioned.

I did note that much of Sebastian's code revolves around merging runs, though how I ran into this issue was related to that implementation - if you unintentionally forget the coverage data (or even intentionally discard it, say for indirectly covered code), you lose the visibility into earlier coverage data.

I'll check my calendar and see if there's a time we can meet on IRC. Any additional support that I can provide to help with the improvements I'm more than glad to provide.

Thanks,
Joe

derick

2021-08-26 17:02

administrator   ~0005993

Hi!

I have now finally sat down to have a good look at this, as well as some
other issues. In order to fix this, I had to split the analysing part
(scanning opcodes) and collecting data during execution, into two separate
data structures. This work is part of the `improve-code-coverage <https://github.com/derickr/xdebug/tree/improve-code-coverage>`_ branch.

The good news is that this fixes the problem, and several related parts.

The bad news is that this makes code coverage twice as slow.

This is likely because there are now more data structures, and hence memory
allocations and frees are involved, but I've not been able to pinpoint the
real cause, nor have I come up with some plan to address this new slow down.

I therefore don't want to merge this branch yet, and I'm moving the target
for this Xdebug 3.2. But, please try out the new branch, as it's possible
that there are still some pending issues.

cheers,
Derick

Issue History

Date Modified Username Field Change
2019-06-12 19:38 joerieger@quickenloans.com New Issue
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: branches
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: coverage
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: Docker
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: functions
2019-06-12 19:38 joerieger@quickenloans.com Tag Attached: path
2019-06-28 11:19 derick Target Version => 2.8.0dev
2019-07-19 11:48 derick Assigned To => derick
2019-07-19 11:48 derick Status new => feedback
2019-07-19 11:48 derick Note Added: 0005074
2019-07-19 12:12 joerieger@quickenloans.com Note Added: 0005075
2019-07-19 12:12 joerieger@quickenloans.com Status feedback => assigned
2019-07-19 17:50 joerieger@quickenloans.com Note Added: 0005081
2019-07-19 17:56 joerieger@quickenloans.com Note Added: 0005082
2019-08-05 18:34 derick Status assigned => confirmed
2019-08-05 18:34 derick Note Added: 0005106
2019-08-05 18:35 derick Target Version 2.8.0dev => 3.0dev
2019-08-05 18:58 joerieger@quickenloans.com Note Added: 0005108
2020-11-18 15:32 derick Target Version 3.0dev => 3.1dev
2021-08-26 17:02 derick Target Version 3.1dev => 3.2dev
2021-08-26 17:02 derick Note Added: 0005993