Coverage information does not seem to be right

[ronisbr]

Hi guys!

After the new version of Coverage, the coverage information does not seem to be right. Take a look, for example, at the function compose_rotation here:

https://coveralls.io/builds/19787299/source?filename=src/compose_rotations.jl#L53

It says that it is not tested at all. However, take a look at the test here:

https://github.com/JuliaSpace/ReferenceFrameRotations.jl/blob/6fd8162b51958ed809fe36c268eb11bd0a506a43/test/test_compose_rotations_quaternions.jl#L49

The last version of Coverage was saying that ReferenceFrameRotations.jl has a coverage of 100% and now it is 74%.

Notice that the same thing happened with Codecov.

[ronisbr]

By the way, I am also wondering why it is marking this line as it was not executed:

https://codecov.io/gh/JuliaSpace/ReferenceFrameRotations.jl/src/master/src/DCM.jl#L345

whereas the surrounding lines were executed.

[ronisbr]

Using --inline=no seems to fix the problem. However, the tests take ages to complete. Is there any other workaround?

[ronisbr]

And I could not use --inline=no in Travis. It is working only on local analysis. The command I used was:

julia --check-bounds=yes --color=yes --inline=no -e "VERSION >= v\"0.7.0-DEV.5183\" && using Pkg; Pkg.test(\"ReferenceFrameRotations\", coverage=true)"

[ararslan]

Sounds like JuliaLang/julia#28192

[ronisbr]

@ararslan yes, maybe. However, I started to see the problem today. The last coverage result, which was 2 days ago, was working perfectly.

[ararslan]

cc @fingolfin

[ararslan]

Yeah looking around at various PRs with coverage integration, I think we're going to need to revert the latest changes to this package unless we can come up with a quick fix. Something is very wrong.

[fingolfin]

The old coverage was "accidentally correct". The problem is that due to a bug in Coverage.jl, it reported lines that do not contain code, and lines that are not covered, the same way. You can see that in e.g. https://coveralls.io/builds/18398285/source?filename=src/compose_rotations.jl only one line is marked green, but no lines red; so that means it thinks that exactly that one line contains code. Which is of course wrong.

When I fixed this, the other lines of code started to be marked as "is code". But Julia still reports that there is no coverage for them, so now they get marked as "uncovered code".

That's unfortunate, but other than either enabling --inline=no, the only other fix I can think of is to improve how Julia reports code coverage. Specifically, it could try to attribute coverage to inlined functions (I think for C/C++, GCC somehow manages that quite well).

I do not see any way JuliaCoverage.jl can help with that, other than reverting the fix and returning to broken coverage reporting. Which will result in great coverage scores for everybody, with a lot of uncovered code not reported as such. I.e., pick what kind of error you prefer, first or second order :/.

[fingolfin]

@ararslan I am not sure anything is wrong in Coverage.jl, other than it not hiding the effects of JuliaLang/julia#28192 anymore.

That said, one could of course try to mixup things further, and e.g. not report macros as "code"; then the example @ronisbr reports for src/compose_rotations.jl would be "fixed". But not https://codecov.io/gh/JuliaSpace/ReferenceFrameRotations.jl/src/master/src/DCM.jl#L345

[ronisbr]

I understood, thanks @fingolfin . Indeed, running with --inline=no fixed the problem. However, I was not able to run this no Travis. When I tried to enable, it made no difference.

[fingolfin]

As a matter of fact, I wonder if any project using Coverage.jl with Julia 0.7 or 1.0 had a coverage score other than 100% prior to my fixes; I don't see how, at least. While that looks great, it is also completely useless...

[ronisbr]

I see. The local runs with inline=no showed 98% of coverage. I just need then to make this work in Travis somehow... ideas?

[fingolfin]

What did you try? I am looking at https://github.com/JuliaSpace/ReferenceFrameRotations.jl/blob/master/.travis.yml now. I assume you edited the four calls to Julia there to insert --inline=no? And that then did not help (tested how? via a PR to your own project?)

[KristofferC]

Currently, Pkg would have to be tweaked to run the tests with --inline=no.