Skip to content

Regressions from recent change in Enumerable/String  #64992

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 215c328e039b3a6b6bfdc6514253307105f25816
Compare ca6716a658137b55d29b3ce100d5d14eb0edd67e
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterfaceInterfaceMethodLongHierarchy - Duration of single invocation 343.02 μs 434.44 μs 1.27 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodLongHierarchy


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 434.43776620370363 > 360.16713994565214.
IsChangePoint: Marked as a change because one of 12/3/2021 12:15:47 AM, 12/26/2021 4:30:45 AM, 1/28/2022 6:30:08 PM, 1/31/2022 1:37:35 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -7.992472074571784 (T) = (0 -426668.4222380961) / Math.Sqrt((965734216.7506899 / (22)) + (492133022.74529207 / (52))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (52) - 2, .025) and -0.15852951931967613 = (368284.4632984827 - 426668.4222380961) / 368284.4632984827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 215c328e039b3a6b6bfdc6514253307105f25816
Compare ca6716a658137b55d29b3ce100d5d14eb0edd67e
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 222.48 ns 239.27 ns 1.08 0.07 False
WhereAny_LastElementMatches - Duration of single invocation 258.96 ns 285.18 ns 1.10 0.09 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 239.27208808510542 > 233.7122841434742.
IsChangePoint: Marked as a change because one of 1/31/2022 1:37:35 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -12.826789876065815 (T) = (0 -234.1997613536838) / Math.Sqrt((0.8492934570542772 / (22)) + (42.165541206139075 / (52))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (52) - 2, .025) and -0.05316225669485286 = (222.3776629525964 - 234.1997613536838) / 222.3776629525964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 285.1834081591006 > 271.80694088092423.
IsChangePoint: Marked as a change because one of 12/2/2021 8:37:26 PM, 12/7/2021 12:20:12 AM, 12/8/2021 6:13:42 AM, 12/17/2021 7:05:08 PM, 2/7/2022 2:49:00 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -30.337380283490962 (T) = (0 -285.0143803057061) / Math.Sqrt((39.02563193166371 / (71)) + (0.07871479182178301 / (3))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (71) + (3) - 2, .025) and -0.08787425924788754 = (261.9920251654392 - 285.0143803057061) / 261.9920251654392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 215c328e039b3a6b6bfdc6514253307105f25816
Compare ca6716a658137b55d29b3ce100d5d14eb0edd67e
Diff Diff

Regressions in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableHashSet - Duration of single invocation 15.93 μs 20.77 μs 1.30 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<Int32>.ImmutableHashSet(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.76785312640576 > 16.658499762854014.
IsChangePoint: Marked as a change because one of 1/31/2022 1:37:35 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -104.9294748860685 (T) = (0 -21051.73394744585) / Math.Sqrt((19881.098816749425 / (22)) + (76796.96306669693 / (52))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (52) - 2, .025) and -0.32133820424881193 = (15932.131440499654 - 21051.73394744585) / 15932.131440499654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 215c328e039b3a6b6bfdc6514253307105f25816
Compare ca6716a658137b55d29b3ce100d5d14eb0edd67e
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 1.44 ms 1.69 ms 1.18 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.690661756756757 > 1.502581166502463.
IsChangePoint: Marked as a change because one of 1/31/2022 1:37:35 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -55.5956689561246 (T) = (0 -1735731.1983655638) / Math.Sqrt((350600548.6628888 / (22)) + (656938221.8443089 / (52))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (52) - 2, .025) and -0.20656855668343535 = (1438568.2344786676 - 1735731.1983655638) / 1438568.2344786676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 215c328e039b3a6b6bfdc6514253307105f25816
Compare ca6716a658137b55d29b3ce100d5d14eb0edd67e
Diff Diff

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Append_Char_Capacity - Duration of single invocation 180.04 ns 192.01 ns 1.07 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 192.0087062413456 > 189.21173907550073.
IsChangePoint: Marked as a change because one of 1/31/2022 1:37:35 PM, 2/8/2022 5:50:06 AM falls between 1/30/2022 5:09:41 PM and 2/8/2022 5:50:06 AM.
IsRegressionStdDev: Marked as regression because -44.315123284896586 (T) = (0 -191.7010662098259) / Math.Sqrt((0.39241636051399265 / (22)) + (2.6694428660464187 / (52))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (52) - 2, .025) and -0.06473447551369219 = (180.0458899551813 - 191.7010662098259) / 180.0458899551813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimeuntriagedNew issue has not been triaged by the area owner

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions