Stop gracefully when benchmark fails with an exception that can not be caught by a catch block #1661

adamsitnik · 2021-03-05T09:21:20Z

The original issue was reported in dotnet/performance#1701:

// Benchmark Process Environment Information:
// Runtime=.NET 6.0.0 (6.0.21.11801), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-AQTFSE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

OverheadJitting  1: 1 op, 537000.00 ns, 537.0000 us/op
WorkloadJitting  1: 1 op, 6767100.00 ns, 6.7671 ms/op

OverheadJitting  2: 16 op, 731100.00 ns, 45.6938 us/op
WorkloadJitting  2: 16 op, 2527400.00 ns, 157.9625 us/op

WorkloadPilot    1: 16 op, 7800.00 ns, 487.5000 ns/op
// Benchmark Process 3336 has exited with code -1073741819
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)

The benchmark process exited with -1073741819 which translates to 0xc0000005 which is Access Violation.

BenchmarkDotNet should:

detect failures for exceptions that can not be caught (most probably this logic has to be modified)
stop gracefully in such cases

Whoever is willing to work on that should most probably start by creating a benchmark that throws an exception that can not be caught by a catch block (AccessViolation would be the best), then create a test for it similar to this one and fix BenchmarkRunnerClean.Execute

The text was updated successfully, but these errors were encountered:

danmoseley · 2021-03-05T18:57:50Z

I wonder whether Watson grabbed a dump.

kevinsalimi · 2021-03-06T04:36:49Z

I wanna take this if still available.

adamsitnik · 2021-03-06T10:35:02Z

@kevinsalimi I've assigned you. Please let me know if you need some help.

kevinsalimi · 2021-03-24T13:50:43Z

Hello @adamsitnik. Thank you for the assignment.
I spent ample time on the issue and learned a lot during the code exploration yet I have some questions.

The AccessViolation exception is this, am I right?

After creating a benchmark, I couldn't simulate any exceptions that cannot be caught by a catch block. I tried to threw variant exceptions and scenarios but I got nothing. In addition, I wrote code below in the benchmark:

unsafe
{
    IntPtr ptr = new IntPtr(100);
    int value = Marshal.ReadInt32(ptr);
}

I caught the AccessViolationException and the Benchmark stop gracefully like below.

// Benchmark Process Environment Information:
// Runtime=.NET Framework 4.8 (4.8.4300.0), X64 RyuJIT
// GC=Concurrent Workstation
// Job: DefaultJob

OverheadJitting  1: 1 op, 510500.00 ns, 510.5000 us/op

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.InteropServices.Marshal.ReadInt32(IntPtr ptr, Int32 ofs)
   at BenchmarkDotNet.Samples.IntroArguments.Benchmark() in C:\Users\Lenovo\Documents\OpenSourceProjects\BenchmarkDotNet\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\IntroArguments.cs:line 20
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount)
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
// AfterAll
// Benchmark Process 8632 has exited with code -1
Parse error in the following line:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
.
.
.
// ***** BenchmarkRunner: Finish  *****
.
.
.

Any ideas would be appreciated.

kevinsalimi · 2021-04-19T20:13:17Z

@adamsitnik

timcassell · 2021-04-21T06:13:02Z

I'm not sure how to generate an uncatchable AccessViolationException, but you could throw an uncatchable StackOverflowException:

void Kill()
{
    Kill();
}

twallace27603 · 2021-06-10T00:54:30Z

You can also generate an uncatchable StackOverflowException by throwing it, although I love the elegant recursion. I found a great summary of "uncatchable errors on StackOverflow (the irony): https://stackoverflow.com/questions/7392783/list-of-exceptions-that-cant-be-caught-in-net. Basically there are conditions under which AccessViolationExceptions are sort of uncatchable, but it seems that the StackOverflowException is an easier candidate for testing the behavior.

timcassell · 2021-06-10T07:02:56Z

I believe if you manually throw a StackOverflowException it can be caught. It will only be un-catchable when the runtime throws it. At least that's what I observed when testing that in Unity a while ago. I haven't tested that behavior on the latest .Net runtime.

twallace27603 · 2021-06-10T11:56:45Z

@timcassell you are 100% correct. I did something embarrassingly stupid for my first comment on this project. Your approach is the best test.

danmoseley · 2021-09-10T03:04:22Z

another example, presumably same fix:

[2021/09/08 09:11:10][INFO] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[2021/09/08 09:11:10][INFO]  ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name does not resolve
[2021/09/08 09:11:10][INFO]    at System.Net.Dns.GetHostEntryOrAddressesCore(IPAddress address, Boolean justAddresses)
[2021/09/08 09:11:10][INFO]    at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
[2021/09/08 09:11:10][INFO]    at System.Net.Tests.DnsTests.GetHostEntry() in /root/git/performance/src/benchmarks/micro/libraries/System.Net.Primitives/DnsTests.cs:line 14
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_2149.WorkloadActionNoUnroll(Int64 invokeCount) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 1582163
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_2149.Run(IHost host, String benchmarkName) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 1581840
[2021/09/08 09:11:10][INFO]    --- End of inner exception stack trace ---
[2021/09/08 09:11:10][INFO]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
[2021/09/08 09:11:10][INFO]    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2021/09/08 09:11:10][INFO]    at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 3583
[2021/09/08 09:11:10][INFO] // AfterAll
[2021/09/08 09:11:10][INFO] // Benchmark Process 23065 has exited with code 255.
[2021/09/08 09:11:10][INFO] Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
[2021/09/08 09:11:10][INFO]    at System.Linq.ThrowHelper.ThrowNoMatchException()
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[2021/09/08 09:11:10][INFO]    at MicroBenchmarks.Program.Main(String[] args) in /root/git/performance/src/benchmarks/micro/Program.cs:line 42
[2021/09/08 09:11:10][INFO] $ popd
[2021/09/08 09:11:10][ERROR] Process exited with status 134

cc @adamsitnik this was the issue I hit this morning

adamsitnik added bug up-for-grabs labels Mar 5, 2021

adamsitnik added this to the v0.13.x milestone Mar 5, 2021

adamsitnik mentioned this issue Mar 5, 2021

Crash in BDN with System.InvalidOperationException dotnet/performance#1701

Closed

adamsitnik added the good first issue label Mar 5, 2021

adamsitnik assigned kevinsalimi Mar 6, 2021

dotnet / BenchmarkDotNet Public

Stop gracefully when benchmark fails with an exception that can not be caught by a catch block #1661

Stop gracefully when benchmark fails with an exception that can not be caught by a catch block #1661

adamsitnik commented Mar 5, 2021

danmoseley commented Mar 5, 2021

kevinsalimi commented Mar 6, 2021 •

edited

adamsitnik commented Mar 6, 2021

kevinsalimi commented Mar 24, 2021 •

edited

kevinsalimi commented Apr 19, 2021

timcassell commented Apr 21, 2021

twallace27603 commented Jun 10, 2021 •

edited

timcassell commented Jun 10, 2021

twallace27603 commented Jun 10, 2021

danmoseley commented Sep 10, 2021

dotnet / BenchmarkDotNet Public

Stop gracefully when benchmark fails with an exception that can not be caught by a catch block #1661

Stop gracefully when benchmark fails with an exception that can not be caught by a catch block #1661

Comments

adamsitnik commented Mar 5, 2021

danmoseley commented Mar 5, 2021

kevinsalimi commented Mar 6, 2021 • edited

adamsitnik commented Mar 6, 2021

kevinsalimi commented Mar 24, 2021 • edited

kevinsalimi commented Apr 19, 2021

timcassell commented Apr 21, 2021

twallace27603 commented Jun 10, 2021 • edited

timcassell commented Jun 10, 2021

twallace27603 commented Jun 10, 2021

danmoseley commented Sep 10, 2021

kevinsalimi commented Mar 6, 2021 •

edited

kevinsalimi commented Mar 24, 2021 •

edited

twallace27603 commented Jun 10, 2021 •

edited