Skip to content

[Perf] Linux/x64: Regressions from peephole optimization of removing mov instruction #82776

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayGet - Duration of single invocation 3.13 μs 4.46 μs 1.42 0.02 False
BitArrayCopyToByteArray - Duration of single invocation 96.60 ns 127.13 ns 1.32 0.14 False
BitArrayGet - Duration of single invocation 23.90 ns 34.88 ns 1.46 0.04 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.461169894969991 > 3.289212771233317.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -14.192708936531552 (T) = (0 -4716.691601345517) / Math.Sqrt((9.822328743776907 / (32)) + (174040.5699213365 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.504894189657222 = (3134.234708168993 - 4716.691601345517) / 3134.234708168993 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToByteArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 127.12562998486031 > 101.52213180950997.
IsChangePoint: Marked as a change because one of 1/9/2023 8:29:54 AM, 2/24/2023 12:21:41 AM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -8.126516233947049 (T) = (0 -123.38147901596396) / Math.Sqrt((121.98309277943181 / (26)) + (42.6614140605677 / (20))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (20) - 2, .025) and -0.20782692933852506 = (102.15162124554939 - 123.38147901596396) / 102.15162124554939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.87986771120701 > 24.999256065982223.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -15.458065785909893 (T) = (0 -36.27278652177423) / Math.Sqrt((0.12361635534773109 / (32)) + (8.78341090100157 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.5119226217582704 = (23.991165949743692 - 36.27278652177423) / 23.991165949743692 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Indexer - Duration of single invocation 6.59 ns 7.75 ns 1.18 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.Indexer


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.754013429754235 > 6.91930064855726.
IsChangePoint: Marked as a change because one of 1/9/2023 8:29:54 AM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -18.06856534409327 (T) = (0 -7.849148221895618) / Math.Sqrt((0.08106357917856769 / (32)) + (0.01803876575879867 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.16592026403053478 = (6.732148384454233 - 7.849148221895618) / 6.732148384454233 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ctor_seeded - Duration of single invocation 288.88 ns 334.54 ns 1.16 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.ctor_seeded


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 334.54439670556474 > 303.22118724924195.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 2/9/2023 4:10:34 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -28.13552155284546 (T) = (0 -335.3118175594031) / Math.Sqrt((120.06256187109301 / (32)) + (0.3903661036166245 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.19493348939057553 = (280.6112813278122 - 335.3118175594031) / 280.6112813278122 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EncodeUtf16 - Duration of single invocation 94.93 ns 105.09 ns 1.11 0.01 False
EncodeUtf16 - Duration of single invocation 39.12 ns 42.63 ns 1.09 0.07 False
EncodeUtf16 - Duration of single invocation 44.68 ns 52.52 ns 1.18 0.03 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

Histogram

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,&lorem ipsum=dolor sit amet,512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 105.08947862018871 > 99.79948470820757.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 1/13/2023 4:18:46 AM, 1/19/2023 8:10:32 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -56.346263592222016 (T) = (0 -105.41317440800262) / Math.Sqrt((0.6596366346693843 / (32)) + (0.2051774884768605 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.11158636240394233 = (94.83129514114734 - 105.41317440800262) / 94.83129514114734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: JavaScript,&Hello+<World>!,16)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.634198435479036 > 41.106720723386296.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -9.668193430807563 (T) = (0 -43.61043461158647) / Math.Sqrt((1.2671574185008387 / (32)) + (2.4781612593950197 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.11505061278282146 = (39.11072207094556 - 43.61043461158647) / 39.11072207094556 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,&lorem ipsum=dolor sit amet,16)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.5210271986192 > 45.67678367654555.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -29.87002404008044 (T) = (0 -54.94319482671539) / Math.Sqrt((0.5996188002245728 / (32)) + (1.6887168745314194 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.25462849484147976 = (43.792401537681776 - 54.94319482671539) / 43.792401537681776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in Benchstone.BenchI.EightQueens

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 1.52 μs 1.67 μs 1.10 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.EightQueens*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.EightQueens.Test


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.670665690263447 > 1.5945792605011049.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -71.42958256579422 (T) = (0 -1661.3700495909973) / Math.Sqrt((80.9257841259283 / (32)) + (17.92842994860785 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.09160360383729522 = (1521.9536137026407 - 1661.3700495909973) / 1521.9536137026407 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in SciMark2.kernel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
benchFFT - Duration of single invocation 582.08 ms 618.81 ms 1.06 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'

Payloads

Baseline
Compare

Histogram

SciMark2.kernel.benchFFT


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 618.8149386428572 > 613.46088937.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -23.223727106585116 (T) = (0 -619396407.0052328) / Math.Sqrt((3549708936860.585 / (32)) + (33141561698981.934 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.06272655567516185 = (582837046.5549564 - 619396407.0052328) / 582837046.5549564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.29 ms 2.44 ms 1.07 0.11 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes\s+\w+", Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4422080998613045 > 2.318360970081688.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -6.462520023890406 (T) = (0 -2415307.357646216) / Math.Sqrt((5513564343.765095 / (32)) + (6282623070.951262 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.07144385544859774 = (2254254.710000612 - 2415307.357646216) / 2254254.710000612 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 2.06 μs 2.43 μs 1.18 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4325265260126057 > 2.188240009532451.
IsChangePoint: Marked as a change because one of 12/10/2022 11:25:25 PM, 12/12/2022 5:25:39 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -19.194996594965662 (T) = (0 -2420.3929016343386) / Math.Sqrt((7597.081606857718 / (32)) + (433.1677897629344 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.14931029161277198 = (2105.9525171726405 - 2420.3929016343386) / 2105.9525171726405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentDictionary - Duration of single invocation 2.45 μs 3.02 μs 1.23 0.23 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<Int32, Int32>.ConcurrentDictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0160619943713036 > 2.583393129227729.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/13/2023 6:54:14 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -13.203644400238474 (T) = (0 -3072.6138806619056) / Math.Sqrt((32902.739846669465 / (32)) + (23682.775345265425 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.2888391207980084 = (2384.0166170308676 - 3072.6138806619056) / 2384.0166170308676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 9.54 μs 10.85 μs 1.14 0.36 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<Int32>.Dictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.84762242552295 > 10.440739742627738.
IsChangePoint: Marked as a change because one of 2/13/2023 6:54:14 PM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -8.796371030115006 (T) = (0 -11319.443344266809) / Math.Sqrt((185974.49802628352 / (32)) + (147649.02918055665 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.11035901660176996 = (10194.3994465049 - 11319.443344266809) / 10194.3994465049 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 6387a9eb56098a889021190747d31f07246dd9f2
Compare cabaab5e90e19256ae404970091350bb6f0f7139
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 288.33 ms 325.90 ms 1.13 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 325.899507 > 302.9142510925.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/24/2023 8:06:06 PM, 2/28/2023 7:53:18 AM falls between 2/19/2023 7:20:18 PM and 2/28/2023 7:53:18 AM.
IsRegressionStdDev: Marked as regression because -55.410304359968514 (T) = (0 -325987617.64383835) / Math.Sqrt((13451161913235.465 / (32)) + (47558701461.590576 / (14))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (14) - 2, .025) and -0.12441372838461481 = (289917856.22554374 - 325987617.64383835) / 289917856.22554374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions