Skip to content

Conversation

@radical
Copy link
Member

@radical radical commented Sep 2, 2022

Issue: #2575

@radical radical requested a review from adamsitnik September 2, 2022 01:54
@radical
Copy link
Member Author

radical commented Sep 2, 2022

CI is not very usable right now, so I can't build this on dotnet/runtime.

@radical
Copy link
Member Author

radical commented Sep 2, 2022

@adamsitnik I'm curious about this from the original issue #2575:

// Benchmark: Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]
// *** Execute ***
// Launch: 1 / 1
// Execute: /home/helixbot/.jsvu/v8 --expose_wasm test-main.js -- --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName "System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits)" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 67  in /datadisks/disk1/work/A6C10953/w/B67009A6/e/performance/artifacts/bin/for-running/MicroBenchmarks/95a3fee8-6d92-4c8b-8059-369b73cd756f/bin/net7.0/browser-wasm/AppBundle
Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
Incoming arguments: --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits) --job PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0 --benchmarkId 67
Application arguments: --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits) --job PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0 --benchmarkId 67
console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
console.info: Initializing.....
// BeforeAnythingElse

// Benchmark Process Environment Information:
// Runtime=.NET Core (Mono) 8.0.0-ci, Wasm NativeAOT
// GC=Non-concurrent Workstation
// HardwareIntrinsics=
// Job: Job-WFMUPX(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)

The run starts, and it throws:

// BeforeActualRun

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.ArgumentException: Destination is too short. (Parameter 'destination')
   at System.Numerics.BigIntegerCalculator.PowCore(Span`1 , Int32 , ReadOnlySpan`1 , ReadOnlySpan`1 , Span`1 , Span`1 )
   at System.Numerics.BigIntegerCalculator.Pow(ReadOnlySpan`1 , ReadOnlySpan`1 , ReadOnlySpan`1 , Span`1 )
   at System.Numerics.BigInteger.ModPow(BigInteger , BigInteger , BigInteger )
   at System.Numerics.Tests.Perf_BigInteger.ModPow(BigIntegers arguments)
   at BenchmarkDotNet.Autogenerated.Runnable_67.WorkloadActionUnroll(Int64 invokeCount)
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode mode, IterationStage stage, Int32 index, Int64 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineActualStage.RunSpecific(Int64 invokeCount, IterationMode iterationMode, Int32 iterationCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineActualStage.Run(Int64 invokeCount, IterationMode iterationMode, Boolean runAuto, Int32 unrollFactor, Boolean forceSpecific)
   at BenchmarkDotNet.Engines.EngineActualStage.RunWorkload(Int64 invokeCount, Int32 unrollFactor, Boolean forceSpecific)
   at BenchmarkDotNet.Engines.Engine.Run()
   at BenchmarkDotNet.Autogenerated.Runnable_67.Run(IHost host, String benchmarkName)
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object , Span`1 , BindingFlags )
   --- End of inner exception stack trace ---
   at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo )
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
// AfterAll
{"name":"ExitStatus","message":"Program terminated with exit(-1)","status":-1}
No Workload Results were obtained from the run.
// Benchmark Process 4941 has exited with code 255.

// ** Remained 60 (46.9%) benchmark(s) to run. Estimated finish 2022-08-24 19:33 (0h 14m from now) **
// ***** BenchmarkRunner: Finish  *****

// * Export *
  datadisks/disk1/work/A6C10953/w/B67009A6/e/artifacts/BenchmarkDotNet.Artifacts/results/System.Numerics.Tests.Perf_BigInteger-report-github.md
  datadisks/disk1/work/A6C10953/w/B67009A6/e/artifacts/BenchmarkDotNet.Artifacts/results/System.Numerics.Tests.Perf_BigInteger-report-full.json

// * Detailed results *
Perf_BigInteger.ToByteArray: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [numberString=-2147483648]
Runtime = .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT; GC = Non-concurrent Workstation
Mean = 406.000 us, StdErr = 0.000 us (0.00%), N = 1, StdDev = 0.000 us
Min = 406.000 us, Q1 = 406.000 us, Median = 406.000 us, Q3 = 406.000 us, Max = 406.000 us
IQR = 0.000 us, LowerFence = 406.000 us, UpperFence = 406.000 us
ConfidenceInterval = [NaN us; NaN us] (CI 99.9%), Margin = NaN us (NaN% of Mean)
Skewness = NaN, Kurtosis = NaN, MValue = 2
-------------------- Histogram --------------------
[406.000 us ; 406.000 us) | @
---------------------------------------------------

Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]
Runtime = .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT; GC = Non-concurrent Workstation
There are not any results runs

// * Summary *

BenchmarkDotNet=v0.13.1.1847-nightly, OS=ubuntu 18.04
AMD EPYC 7452, 1 CPU, 2 logical cores and 1 physical core
.NET SDK=7.0.100-rc.2.22423.10
  [Host]     : .NET 7.0.0 (7.0.22.42212), X64 RyuJIT AVX2
  Job-BJXMHI : .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=Wasm  Toolchain=Wasm
InvocationCount=1  IterationCount=1  IterationTime=250.0000 ms
MaxIterationCount=20  MinIterationCount=15  RunStrategy=ColdStart
UnrollFactor=1  WarmupCount=0

This shows NA for various columns for ModPow. Does it mean that the benchmark run will get recorded as having data from this run, and have a value of NA, or 0?

|      Method | numberString |         arguments |     Mean | Error |   Median |      Min |      Max | Allocated |
|------------ |------------- |------------------ |---------:|------:|---------:|---------:|---------:|----------:|
| ToByteArray |  -2147483648 |                 ? | 406.0 us |    NA | 406.0 us | 406.0 us | 406.0 us |     104 B |
|      ModPow |            ? | 1024,1024,64 bits |       NA |    NA |       NA |       NA |       NA |         - |
Benchmarks with issues:
  Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]

The reason I'm looking at this is because of dotnet/perf-autofiling-issues#7979 which says the value improved to 0! And this failure was around the same time.

@adamsitnik
Copy link
Member

The reason I'm looking at this is because of dotnet/perf-autofiling-issues#7979 which says the value improved to 0! And this failure was around the same time.

@radical I've re-run the benchmark locally and the exported JSON file contains no statistics, but a single measurement (the jitting time). From BDN perspective I think it's a valid output.

@DrewScoggins how the Reporting System handles such cases?

{
 "DisplayInfo":"Perf_BigInteger.ModPow: Job-QRRJIW(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [arguments=1024,1024,64 bits]",
 "Namespace":"System.Numerics.Tests",
 "Type":"Perf_BigInteger",
 "Method":"ModPow",
 "MethodTitle":"ModPow",
 "Parameters":"arguments=1024,1024,64 bits",
 "FullName":"System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits)",
 "HardwareIntrinsics":"AVX2,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256",
 "Statistics":null,
 "Memory":{
    "Gen0Collections":0,
    "Gen1Collections":0,
    "Gen2Collections":0,
    "TotalOperations":0,
    "BytesAllocatedPerOperation":0
 },
 "Measurements":[
    {
       "IterationMode":"Overhead",
       "IterationStage":"Jitting",
       "LaunchIndex":1,
       "IterationIndex":1,
       "Operations":1,
       "Nanoseconds":364600
    }
 ]
}

@radical
Copy link
Member Author

radical commented Sep 6, 2022

The issue doesn't seem to have been fixed by dotnet/runtime#74585 . Still getting the same failure in performance-ci on windows - here, and here.

@DrewScoggins
Copy link
Member

I took a look at the code and the way that we handle this is here,

foreach (var report in summary.Reports)
{
// Skip individual reports with errors
if (report.HasAnyErrors())
{
continue;
}
var test = new Test();
test.Name = FullNameProvider.GetBenchmarkName(report.BenchmarkCase);
test.Categories = report.BenchmarkCase.Descriptor.Categories;
if (hasCriticalErrors)
{
test.AdditionalData["criticalErrors"] = "true";
}
var results = from result in report.AllMeasurements
where result.IterationMode == Engines.IterationMode.Workload && result.IterationStage == Engines.IterationStage.Result
orderby result.LaunchIndex, result.IterationIndex
select new { result.Nanoseconds, result.Operations};
var overheadResults = from result in report.AllMeasurements
where result.IsOverhead() && result.IterationStage != Engines.IterationStage.Jitting
orderby result.LaunchIndex, result.IterationIndex
select new { result.Nanoseconds, result.Operations };
test.Counters.Add(new Counter
{
Name = "Duration of single invocation",
TopCounter = true,
DefaultCounter = true,
HigherIsBetter = false,
MetricName = "ns",
Results = (from result in results
select result.Nanoseconds / result.Operations).ToList()
});
test.Counters.Add(new Counter
{
Name = "Overhead invocation",
TopCounter = false,
DefaultCounter = false,
HigherIsBetter = false,
MetricName = "ns",
Results = (from result in overheadResults
select result.Nanoseconds / result.Operations).ToList()
});
test.Counters.Add(new Counter
{
Name = "Duration",
TopCounter = false,
DefaultCounter = false,
HigherIsBetter = false,
MetricName = "ms",
Results = (from result in results
select result.Nanoseconds).ToList()
});
test.Counters.Add(new Counter
{
Name = "Operations",
TopCounter = false,
DefaultCounter = false,
HigherIsBetter = true,
MetricName = "Count",
Results = (from result in results
select (double)result.Operations).ToList()
});
foreach (var metric in report.Metrics.Keys)
{
var m = report.Metrics[metric];
test.Counters.Add(new Counter
{
Name = m.Descriptor.DisplayName,
TopCounter = false,
DefaultCounter = false,
HigherIsBetter = m.Descriptor.TheGreaterTheBetter,
MetricName = m.Descriptor.Unit,
Results = new[] { m.Value }
});
}
if (disassemblyDiagnoser != null && disassemblyDiagnoser.Results.TryGetValue(report.BenchmarkCase, out var disassemblyResult))
{
string disassembly = DiffableDisassemblyExporter.BuildDisassemblyString(disassemblyResult, disassemblyDiagnoser.Config);
test.AdditionalData["disasm"] = disassembly;
}
reporter.AddTest(test);

The gist is that we inspect the BDN object model and pull out results from the Reports property in the Summary object.

var results = from result in report.AllMeasurements
                              where result.IterationMode == Engines.IterationMode.Workload && result.IterationStage == Engines.IterationStage.Result
                              orderby result.LaunchIndex, result.IterationIndex
                              select new { result.Nanoseconds, result.Operations};

@adamsitnik
Copy link
Member

@DrewScoggins thanks! I've created new issue #2600

@DrewScoggins
Copy link
Member

The issue doesn't seem to have been fixed by dotnet/runtime#74585 . Still getting the same failure in performance-ci on windows - here, and here.

That is because we have not had an update of the runtime into the installer branch in 16 days, and the fix for this went in 12 days ago. I assume it is so broken because most of the focus is on the release branches these days, but I am not fully sure.

@LoopedBard3
Copy link
Member

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@LoopedBard3 LoopedBard3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming generally clear run

Copy link
Member

@adamsitnik adamsitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @radical !

@adamsitnik adamsitnik merged commit 10d532f into dotnet:main Oct 5, 2022
@radical radical deleted the enable-big-int branch October 5, 2022 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants