Skip to content

Conversation

@jmid
Copy link
Collaborator

@jmid jmid commented Feb 24, 2023

I've played with computing coverage to have an idea of how much ground we cover.
For now, the approach takes a copy of each tested Stdlib module and instruments that with bisect_ppx.

I do not consider this to be particularly pretty... 😅
For example, this will not report any coverage on extern-defined functions
Nevertheless, I consider it good enough to obtain some basic numbers.

$ make coverage-summary 
bisect-ppx-report summary --per-file
 28.84 %   109/378    src/array/array.ml
100.00 %     6/6      src/atomic/atomic.ml
 17.28 %    42/243    src/buffer/buffer.ml
  4.29 %    34/792    src/bytes/bytescp.ml
 24.75 %    25/101    src/domain/domain.ml
 27.78 %   115/414    src/ephemeron/ephemeron.ml
 36.94 %   140/379    src/floatarray/float.ml
 28.57 %   104/364    src/hashtbl/hashtbl.ml
 12.50 %     8/64     src/io/in_channel.ml
  0.00 %     0/8      src/io/out_channel.ml
 81.48 %    22/27     src/lazy/lazy.ml
 43.40 %    23/53     src/queue/queue.ml
 41.03 %    16/39     src/semaphore/semaphore.ml
 65.38 %    17/26     src/stack/stack.ml
 43.48 %    10/23     src/sys/sys.ml
 68.57 %   168/245    src/weak/weak.ml
 26.53 %   839/3162   Project coverage

These were obtained by running the below command for the instrumented modules and computing a summary and report at the end:

dune runtest -f src/array --instrument-with bisect_ppx
....
dune runtest -f src/weak --instrument-with bisect_ppx

I'm also including the HTML in a discard-able commit (don't know if it is viewable through the GitHub UI).

Note:

  • For Bigarray no coverage is reported! (4/5 tested functions defined as extern it seems)
  • For Bytes the simple approach needed adjusting to avoid conflicts with the Bytes required by qcheck-core
  • For Domain, instrumenting a copy crashes on my machine:
    $ dune runtest -f src/domain --instrument-with bisect_ppx
    File "src/domain/dune", line 15, characters 7-23:
    15 |  (name domain_spawntree)
                ^^^^^^^^^^^^^^^^
    Command got signal SEGV.
    File "src/domain/dune", line 6, characters 7-23:
    6 |  (name domain_joingraph)
               ^^^^^^^^^^^^^^^^
    Command got signal SEGV.
    
    Some coverage is reported though - probably up to the crash
  • For Dynlink - no immediate dynlink.ml or dynlink/dynlink.ml to copy
  • For Out_channel coverage is for some reason reported as 0%. I'm unsure why.
  • For Thread and Threadomain - no immediate thread.ml or threads/thread.ml to copy

I'm not sure we would want this to be merged.
It gives us some numbers nevertheless that we can compare to, e.g., with the multiple-ts of #112

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants