[Perf] Windows/arm64: 4 Regressions on 5/19/2023 1:23:34 PM #18109
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | Windows 10.0.25094 |
| Queue | AmpereWindows |
| Baseline | f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5 |
| Compare | 1ec5ff876baf614275dcdc1d6ba435e94d78e4b1 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.Json.Tests.Perf_Get
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| GetUInt16 - Duration of single invocation | 1.08 μs | 1.48 μs | 1.37 | 0.64 | False | |||||
| GetByte - Duration of single invocation | 896.71 ns | 1.49 μs | 1.66 | 0.64 | False | |||||
| GetSByte - Duration of single invocation | 1.32 μs | 1.55 μs | 1.18 | 0.65 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'Payloads
Histogram
System.Text.Json.Tests.Perf_Get.GetUInt16
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4847233551149166 > 1.060045006244306.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/25/2023 4:37:39 AM falls between 5/16/2023 7:34:13 AM and 5/25/2023 4:37:39 AM.
IsRegressionStdDev: Marked as regression because -19.918582225797298 (T) = (0 -1640.013784925023) / Math.Sqrt((4817.321369434108 / (14)) + (7683.155868154366 / (12))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (12) - 2, .025) and -0.6156516596371746 = (1015.0788229272882 - 1640.013784925023) / 1015.0788229272882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Get.GetByte
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/17/2023 10:06:24 AM, 5/25/2023 4:37:39 AM falls between 5/16/2023 7:34:13 AM and 5/25/2023 4:37:39 AM.
IsRegressionStdDev: Marked as regression because -12.540102231114659 (T) = (0 -1652.5272212217737) / Math.Sqrt((5721.05155790613 / (5)) + (12936.867050969637 / (7))) is less than -2.2281388519818126 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (7) - 2, .025) and -0.7097072858292335 = (966.5556407922035 - 1652.5272212217737) / 966.5556407922035 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Tests.Perf_Get.GetSByte
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/17/2023 10:06:24 AM, 5/25/2023 4:37:39 AM falls between 5/16/2023 7:34:13 AM and 5/25/2023 4:37:39 AM.
IsRegressionStdDev: Marked as regression because -9.15745972401329 (T) = (0 -1650.9953323934608) / Math.Sqrt((6032.962428688225 / (5)) + (8471.284920080012 / (6))) is less than -2.2621571627982036 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (6) - 2, .025) and -0.39631008231694764 = (1182.3987761041622 - 1650.9953323934608) / 1182.3987761041622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | Windows 10.0.25094 |
| Queue | AmpereWindows |
| Baseline | f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5 |
| Compare | 1ec5ff876baf614275dcdc1d6ba435e94d78e4b1 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Tests.Perf_UInt64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryParseHex - Duration of single invocation | 4.90 ns | 6.53 ns | 1.33 | 0.88 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'Payloads
Histogram
System.Tests.Perf_UInt64.TryParseHex(value: "0")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.528472707638346 > 4.451811446843922.
IsChangePoint: Marked as a change because one of 3/17/2023 5:40:49 AM, 4/25/2023 1:02:54 PM, 5/19/2023 4:22:12 AM, 5/25/2023 4:37:39 AM falls between 5/16/2023 7:34:13 AM and 5/25/2023 4:37:39 AM.
IsRegressionStdDev: Marked as regression because -8.586233400164787 (T) = (0 -6.870974893326252) / Math.Sqrt((1.0117893305984484 / (14)) + (0.5039163092740299 / (12))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (12) - 2, .025) and -0.7313443450420021 = (3.9685778932437406 - 6.870974893326252) / 3.9685778932437406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository