[Perf] Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	fb5f07f9c580bdcd5d0726d1391c2a52a01030f8
Compare	ae4dc9cef942258098164beae987589c9d569215
Diff	Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
K_Max_Elements - Duration of single invocation	419.83 ns	442.70 ns	1.05	0.00	False
HeapSort - Duration of single invocation	52.78 μs	55.83 μs	1.06	0.03	False
HeapSort - Duration of single invocation	1.70 μs	1.87 μs	1.10	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>*'

Related Issues

Regressions

• [Perf] Windows/x86: 24 Regressions on 10/2/2022 4:42:49 PM
• [Perf] Alpine/x64: 21 Regressions on 10/2/2022 4:42:49 PM
• [Perf] Linux/x64: 37 Regressions on 10/2/2022 4:42:49 PM

Improvements

• [Perf] Linux/x64: 20 Improvements on 10/2/2022 4:42:49 PM

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.K_Max_Elements(Size: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 442.70104704328384 > 440.7245315214986.
IsChangePoint: Marked as a change because one of 10/2/2022 4:08:26 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -43.82269926441918 (T) = (0 -443.35472183754894) / Math.Sqrt((1.4099182202912361 / (18)) + (7.7759195658518 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.05728774395496787 = (419.3321301343226 - 443.35472183754894) / 419.3321301343226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 55.826369119802784 > 55.34462434146614.
IsChangePoint: Marked as a change because one of 9/3/2022 11:44:47 AM, 9/23/2022 1:16:35 AM, 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -29.270366449344046 (T) = (0 -55822.14208501988) / Math.Sqrt((35281.66684117917 / (14)) + (315056.12359184 / (35))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (35) - 2, .025) and -0.05964015689879864 = (52680.28181226355 - 55822.14208501988) / 52680.28181226355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.866617259837963 > 1.7848844771642938.
IsChangePoint: Marked as a change because one of 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -19.536857311570802 (T) = (0 -1858.4396812229638) / Math.Sqrt((645.542544125046 / (16)) + (408.03389280808665 / (35))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (35) - 2, .025) and -0.08202864547093051 = (1717.5512764859534 - 1858.4396812229638) / 1717.5512764859534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[DrewScoggins]

Seems related to #76155

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	fb5f07f9c580bdcd5d0726d1391c2a52a01030f8
Compare	ae4dc9cef942258098164beae987589c9d569215
Diff	Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
K_Max_Elements - Duration of single invocation	419.83 ns	442.70 ns	1.05	0.00	False
HeapSort - Duration of single invocation	52.78 μs	55.83 μs	1.06	0.03	False
HeapSort - Duration of single invocation	1.70 μs	1.87 μs	1.10	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>*'

Related Issues

Regressions

• [Perf] Windows/x86: 24 Regressions on 10/2/2022 4:42:49 PM
• [Perf] Alpine/x64: 21 Regressions on 10/2/2022 4:42:49 PM
• [Perf] Linux/x64: 37 Regressions on 10/2/2022 4:42:49 PM

Improvements

• [Perf] Linux/x64: 20 Improvements on 10/2/2022 4:42:49 PM

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.K_Max_Elements(Size: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 442.70104704328384 > 440.7245315214986.
IsChangePoint: Marked as a change because one of 10/2/2022 4:08:26 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -43.82269926441918 (T) = (0 -443.35472183754894) / Math.Sqrt((1.4099182202912361 / (18)) + (7.7759195658518 / (35))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (35) - 2, .025) and -0.05728774395496787 = (419.3321301343226 - 443.35472183754894) / 419.3321301343226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 55.826369119802784 > 55.34462434146614.
IsChangePoint: Marked as a change because one of 9/3/2022 11:44:47 AM, 9/23/2022 1:16:35 AM, 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -29.270366449344046 (T) = (0 -55822.14208501988) / Math.Sqrt((35281.66684117917 / (14)) + (315056.12359184 / (35))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (35) - 2, .025) and -0.05964015689879864 = (52680.28181226355 - 55822.14208501988) / 52680.28181226355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.866617259837963 > 1.7848844771642938.
IsChangePoint: Marked as a change because one of 10/2/2022 5:38:34 AM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -19.536857311570802 (T) = (0 -1858.4396812229638) / Math.Sqrt((645.542544125046 / (16)) + (408.03389280808665 / (35))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (35) - 2, .025) and -0.08202864547093051 = (1717.5512764859534 - 1858.4396812229638) / 1717.5512764859534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	DrewScoggins
Assignees:	-
Labels:	os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, area-CodeGen-coreclr, untriaged
Milestone:	-

[JulieLeeMSFT]

Seems related to #76155

@SingleAccretion could you take a look?

[SingleAccretion]

could you take a look?

Yes, of course.

[JulieLeeMSFT]

Hi @SingleAccretion, 8/14 is RC1 code snap and 0 bug date. Do you plan to look into these 5 issues assigned to you or do you need help from our team?