View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0001674||Xdebug||Code Coverage||public||2019-06-12 19:38||2021-08-26 17:02|
|Summary||0001674: Inconsistent Path & Branch Coverage Reported|
|Description||Depending on what, or possibly how many, files are analyzed for path & branch coverage, XDebug reports significantly different results for path & branch coverage.|
I've set up a GitHub repo here with a Docker container that consistently reproduces the output.
I've tried this on the following setups:
php-fpm from Docker, 7.3.3, and PHPUnit 8 master
php-fpm from Homebrew, 7.3.6, and PHPUnit 8 master
php-fpm from Homebrew, 7.1.29, with PHPUnit 7.5.7-6
|Steps To Reproduce||1. Gather code coverage on PHPUnit\Framework\Assert, and analyze the results. When calling xdebug_stop_code_coverage, pass 1 to clear results. |
1a. In my testing, clearing results had no bearing on the results.
2. Properly, 193 functions are found in that class.
3. Run PHPUnit::main(), passing a flag to only test one file: tests/unit/Util/JsonTest.php
4. Re-run the same coverage gathering steps. Only 2 functions are returned.
5. Re-running step 4 also returns only 2 functions.
|Additional Information||Could also fall into the category of "Usage Problems (Wrong Results)", but I'll leave that up to you to reassign.|
|Tags||branches, coverage, Docker, functions, path|
This test case is way too big. Can you please give me something where:
- I don't need Docker
- There is a simple test, without all of phpunit, where two code coverage runs show this problem?
You need to show what your expected result is (not just "two functions in file", but the real contents, and what you expected. Small test cases are best, and ones that don't require PHPUnit are highly recommended.
||I will do my best to reproduce a more minimal, self-contained example. Apparent scope + size were one of the reasons I went with the Docker + PHPUnit route, but I understand the debugging difficulty.|
Hi derick - I updated the linked repository to include a more self-contained example that doesn't run on Docker and doesn't use PHPUnit. It looks like any run through a file, regardless of if the functions are static or not, will produce a different count of the functions within the file.
Direct link to the 3 files needed for this example:
It's not as "small" as it could be, but I did want to demonstrate that the unexpected behavior isn't limited to whether a function is static or not.
Additional clarity on my statement above: The first pass through when processing code coverage returns a result I expect - it identifies the actual number of functions present in the specified file, as well as coverage information for each of the functions, regardless of if the function was called.
The second pass through each file, the number of functions present in the specified file only includes functions which were called during the code coverage run. I expect the behavior to be the actual number of functions in the file, with only the branch and path information varying.
I've had a moderately good look, and with your smaller example I can reproduce it. I think I know that the problem is that the xdebug_code_coverage_stop() call also removes all discovered information regarding branches, paths, and possible executable/non-executable lines. This can be fixed, but it is hard to do that without refactoring much of this code. I therefore don't think I want to do the refactoring for Xdebug 2.8, but rather leave it to 3.0. It will also likely drastically improve performance.
With as caveat being me not knowing how PHP_CodeCoverage exactly works, I do believe Sebastian has some workarounds for the merging of runs. In theory, you should be able to find out for each "run" which lines, paths, and branches *have* been hit; just not which ones that haven't been. You should be able to extrapolate that from earlier runs (and you'll know which lines won't have been hit too). If it's code that hasn't been seen before, these new functions will have the full list of branches, paths, and lines in the output. I know, it's not ideal, but I think it is possible to construct a full result for each non-first "run" with some logic.
Happy to chat about this on IRC or something like that about it — please email if you'd like to do that.
For now, I'll have to re-target this for Xdebug 3.0.
Re-targeting to 3.0 is more than acceptable, given the scale of changes you mentioned.
I did note that much of Sebastian's code revolves around merging runs, though how I ran into this issue was related to that implementation - if you unintentionally forget the coverage data (or even intentionally discard it, say for indirectly covered code), you lose the visibility into earlier coverage data.
I'll check my calendar and see if there's a time we can meet on IRC. Any additional support that I can provide to help with the improvements I'm more than glad to provide.
I have now finally sat down to have a good look at this, as well as some
other issues. In order to fix this, I had to split the analysing part
(scanning opcodes) and collecting data during execution, into two separate
data structures. This work is part of the `improve-code-coverage <https://github.com/derickr/xdebug/tree/improve-code-coverage>`_ branch.
The good news is that this fixes the problem, and several related parts.
The bad news is that this makes code coverage twice as slow.
This is likely because there are now more data structures, and hence memory
allocations and frees are involved, but I've not been able to pinpoint the
real cause, nor have I come up with some plan to address this new slow down.
I therefore don't want to merge this branch yet, and I'm moving the target
for this Xdebug 3.2. But, please try out the new branch, as it's possible
that there are still some pending issues.
|2019-06-12 19:firstname.lastname@example.org||New Issue|
|2019-06-12 19:email@example.com||Tag Attached: branches|
|2019-06-12 19:firstname.lastname@example.org||Tag Attached: coverage|
|2019-06-12 19:email@example.com||Tag Attached: Docker|
|2019-06-12 19:firstname.lastname@example.org||Tag Attached: functions|
|2019-06-12 19:email@example.com||Tag Attached: path|
|2019-06-28 11:19||derick||Target Version||=> 2.8.0dev|
|2019-07-19 11:48||derick||Assigned To||=> derick|
|2019-07-19 11:48||derick||Status||new => feedback|
|2019-07-19 11:48||derick||Note Added: 0005074|
|2019-07-19 12:firstname.lastname@example.org||Note Added: 0005075|
|2019-07-19 12:email@example.com||Status||feedback => assigned|
|2019-07-19 17:firstname.lastname@example.org||Note Added: 0005081|
|2019-07-19 17:email@example.com||Note Added: 0005082|
|2019-08-05 18:34||derick||Status||assigned => confirmed|
|2019-08-05 18:34||derick||Note Added: 0005106|
|2019-08-05 18:35||derick||Target Version||2.8.0dev => 3.0dev|
|2019-08-05 18:firstname.lastname@example.org||Note Added: 0005108|
|2020-11-18 15:32||derick||Target Version||3.0dev => 3.1dev|
|2021-08-26 17:02||derick||Target Version||3.1dev => 3.2dev|
|2021-08-26 17:02||derick||Note Added: 0005993|