Skip to content

Conversation

@mmiller-max
Copy link
Member

The changes here mean that if a custom TestSet has any of the fields properties, time_taken, start_time or hostname then they are used in the report. This could break/be strange if these fields are in the test set but used for other things and not what we use them for. Is it better to only populate these values in a report when ReportingTestSets are used or to keep the flexibility for custom TestSets?

Closes #61

@mmiller-max mmiller-max force-pushed the mm/fix-custom-testsets branch 2 times, most recently from a81841a to 6a6357c Compare January 4, 2021 08:51
@oxinabox
Copy link
Member

oxinabox commented Jan 6, 2021

I wonder if rather than checking e.g. hasproperty(ts, :time_taken)....
we should just instead do:

time_taken(ts::ReportingTestSet) = ts.time_taken
time_taken(ts::AbstractTestSet) = nothing  # or some dummy value like -1ms

and then it is easy enough to overload that to add support for them

@mmiller-max
Copy link
Member Author

Yeah that's better, probably worth doing for the description and results too, although #3 mentions that these could be in base

@oxinabox
Copy link
Member

oxinabox commented Jan 7, 2021

Adding things in Base takes ages, so probably not worth it.

@codecov-io
Copy link

codecov-io commented Jan 26, 2021

Codecov Report

Merging #63 (410234a) into master (c4ecb80) will decrease coverage by 1.36%.
The diff coverage is 85.71%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #63      +/-   ##
==========================================
- Coverage   92.00%   90.63%   -1.37%     
==========================================
  Files           5        6       +1     
  Lines         375      395      +20     
==========================================
+ Hits          345      358      +13     
- Misses         30       37       +7     
Impacted Files Coverage Δ
src/TestReports.jl 100.00% <ø> (ø)
src/v1_compat.jl 0.00% <0.00%> (ø)
src/testsets.jl 95.04% <80.00%> (-4.96%) ⬇️
src/recordproperty.jl 100.00% <100.00%> (ø)
src/to_xml.jl 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4ecb80...410234a. Read the comment docs.

@mmiller-max
Copy link
Member Author

OK I've updated it following your suggestions. I've left results and properties for now as it was more involved (we set both of these as we flatten TestSets, for example. I'll add some notes to #3.

If you're happy with this, I'll add some more docs and tests to make sure its all being properly tested

@mmiller-max mmiller-max force-pushed the mm/fix-custom-testsets branch 2 times, most recently from 9651770 to fb869fa Compare January 31, 2021 03:54
@mmiller-max mmiller-max force-pushed the mm/fix-custom-testsets branch from fb869fa to 410234a Compare January 31, 2021 21:39
@mmiller-max mmiller-max changed the title RFC: Allow and test for custom TestSets in pkg tests Allow and test for custom TestSets in pkg tests Jan 31, 2021
@mmiller-max mmiller-max merged commit cd0feec into master Jan 31, 2021
@mmiller-max mmiller-max deleted the mm/fix-custom-testsets branch January 31, 2021 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CustomTestSets may be broken

4 participants