Case of the Invoked Hang (.NET)

A .NET application was hanging completely when certain background tasks were occurring. Using ProcDump (http://live.sysinternals.com/Procdump.exe ) with option Procdump –ma <process name> i captured a 3 dump files about 10 seconds apart, to identify where hang was occurring.

Opening dmp file in WinDbg with SOS extension:

.loadby sos clr

The UI thread

0:000> kv
# ChildEBP RetAddr  Args to Child             
00 006df114 754d1286 00000001 00a3cc10 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
01 006df2a8 6913b957 00000001 00a3cc10 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x136 (FPO: [SEH])
02 006df2f8 6913b62c 00000000 ffffffff 00000001 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c (FPO: [Non-Fpo])
03 006df384 6913b71d 00000001 00a3cc10 00000000 clr!Thread::DoAppropriateWaitWorker+0x237 (FPO: [5,25,0])
04 006df3f0 6913b8fd 00000001 00a3cc10 00000000 clr!Thread::DoAppropriateWait+0x64 (FPO: [Non-Fpo])
05 006df43c 6913b0d2 ffffffff 00000001 00000000 clr!CLREventBase::WaitEx+0x128 (FPO: [Non-Fpo])
06 006df450 6910134d ffffffff 00000001 00000000 clr!CLREventBase::Wait+0x1a (FPO: [3,0,0])
07 006df4b4 69241bb8 00010000 006df4d8 352bbf87 clr!CLREventWaitWithTry+0x42 (FPO: [Non-Fpo])
08 006df4ec 69125dde 352bbe77 027f9cd8 00000000 clr!ThreadStore::WaitForOtherThreads+0x74 (FPO: [Non-Fpo])
09 006df51c 691247b7 352bbe43 00aecab8 00aa77d0 clr!RunMain+0x26d (FPO: [Non-Fpo])
0a 006df788 69124dbb 00000000 352bbcf3 00540000 clr!Assembly::ExecuteMainMethod+0x144 (FPO: [1,149,0])
0b 006dfc90 69124e61 352bb783 00000000 00000000 clr!SystemDomain::ExecuteMainMethod+0x651 (FPO: [0,315,0])
0c 006dfce8 69124662 352bb643 00000000 00000000 clr!ExecuteEXE+0x4c (FPO: [Non-Fpo])
0d 006dfd28 690e4e91 352bb60f 00000000 00000000 clr!_CorExeMainInternal+0xdc (FPO: [Non-Fpo])
0e 006dfd64 6a10cd87 334078bd 6a18dc60 6a10cd16 clr!_CorExeMain+0x4d (FPO: [Non-Fpo])
0f 006dfda0 6a18dd05 6a18dc60 639b32df 006dfdc4 mscoreei!_CorExeMain+0x10a (FPO: [0,10,4])
10 006dfdb0 74d33744 7ec9f000 74d33720 12f4615d mscoree!_CorExeMain_Exported+0xa5 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
11 006dfdc4 779ca064 7ec9f000 ad1ace76 00000000 kernel32!BaseThreadInitThunk+0x24
12 006dfe0c 779ca02f ffffffff 779ed7de 00000000 ntdll!__RtlUserThreadStart+0x2f (FPO: [Non-Fpo])
13 006dfe1c 00000000 6a18dc60 7ec9f000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

I took a quick look at the .NET stack traces

!eestack

….

 

Thread  21
Current frame: ntdll!NtWaitForSingleObject+0xc
ChildEBP RetAddr  Caller, Callee
0099e3a0 754c8d09 KERNELBASE!WaitForSingleObjectEx+0x99, calling ntdll!NtWaitForSingleObject
0099e414 68fc1bb7 clr!CLREventBase::Reset+0x145
0099e444 68fc1bfe clr!CLREventBase::Reset+0x18d, calling clr!CLREventBase::Reset+0x114
0099e494 68fc1b81 clr!CLREventBase::WaitEx+0x152, calling clr!CLREventBase::Reset+0x163
0099e4b8 756fcca6 combase!WindowsCreateString+0x6886, calling msvcrt!memcmp
0099e4cc 6913b0d2 clr!CLREventBase::Wait+0x1a, calling clr!CLREventBase::WaitEx
0099e4e4 69244395 clr!Thread::WaitSuspendEventsHelper+0x8a, calling clr!CLREventBase::Wait
0099e524 690533c5 clr!SafeReleasePreemp+0x234
0099e550 69244468 clr!Thread::WaitSuspendEvents+0x14, calling clr!Thread::WaitSuspendEventsHelper
0099e564 692427cc clr!Thread::RareEnablePreemptiveGC+0x8e, calling clr!Thread::WaitSuspendEvents
0099e57c 69137d3b clr!Thread::RareDisablePreemptiveGC+0x102, calling clr!Thread::RareEnablePreemptiveGC
0099e5cc 6913b694 clr!Thread::DoAppropriateWaitWorker+0x3cb, calling clr!GCCoopHackNoThread::~GCCoopHackNoThread
0099e648 6913b71d clr!Thread::DoAppropriateWait+0x64, calling clr!Thread::DoAppropriateWaitWorker
0099e69c 690bb01e clr!AcquireSafeHandle+0x33, calling clr!_EH_epilog3
0099e6b4 690bb130 clr!WaitHandleNative::CorWaitOneNative+0x163, calling clr!Thread::DoAppropriateWait
0099e72c 690bb06d clr!WaitHandleNative::CorWaitOneNative+0x4c, calling clr!LazyMachStateCaptureState
0099e750 68fb278a clr!HelperMethodFrame::Push+0x10, calling clr!GetThread
0099e758 69068141 clr!AppDomainNative::IsFinalizingForUnload+0x3a, calling clr!_EH_epilog3
0099e7bc 680cc7c1 (MethodDesc 67e4df10 +0x21 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)), calling clr!WaitHandleNative::CorWaitOneNative
0099e7d0 680cc788 (MethodDesc 67e4dedc +0x28 System.Threading.WaitHandle.WaitOne(Int32, Boolean)), calling (MethodDesc 67e4df10 +0 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
0099e7ec 0085bb9d (MethodDesc 04feadb8 +0x1a5 System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle))
0099e7f4 68fb21a6 clr!JIT_MonExitWorker+0xa, calling clr!GetThread
0099e830 0949b3e3 (MethodDesc 04feb6f4 +0x2cb System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control, System.Delegate, System.Object[], Boolean)), calling 0949a078
0099e8ac 0085b5fb (MethodDesc 04feb648 +0x4b System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])), calling 0949a038
0099e8e8 0085b598 (MethodDesc 04feb63c +0x8 System.Windows.Forms.Control.Invoke(System.Delegate)), calling (MethodDesc 04feb648 +0 System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[]))
0099e8f0 0085b3af (MethodDesc 059f56c8 +0x1c7 Main.Form1.bulkWorker_DoWork(System.Object, System.ComponentModel.DoWorkEventArgs)), calling 0085ae0c

 

This is a common cause of .NET application hang – .Invoke called from background thread. This is easily fixed by replaced .Invoke with .BeginInvoke

 

What if there is no source code, vendor is slow to fix, etc? You can patch it yourself, at your own risk…

Use ILDasm ( https://chentiangemalc.wordpress.com/2014/06/30/upgrade-exe-net-framework-with-ildasm-ilasm-and-hex-editor/ ) to dump the contents to IL format

Do a search replace from

callvirt   instance object [System.Windows.Forms]System.Windows.Forms.Control::Invoke

to

callvirt   instance class [mscorlib]System.IAsyncResult [System.Windows.Forms]System.Windows.Forms.Control::BeginInvoke

We then recompile:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\ilasm.exe Program.il /RESOURCE=Program.res

  And the hangs are gone…

What is the difference between Invoke & BeginInvoke:

  • Invoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.
  • BeginInvoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.

Some of the problems with using invoke:

  • Causes thread to wait on another thread, increases risk of deadlock in your application. Using ThreadPools may help, but under high load in “threadpool starvation” the deadlock may be triggered.
  • Invoke forces thread switch immediately that has potential to decrease application stability
  • Each Invoke requires a separate thread switch, potentially impacting application performance

Some advantage of BeginInvoke:

  • Multiple BeginInvoke can be executed in a single thread switch ‘
  • Schedules execution of delegate on other thread – thread switch occurs “when OS is ready”

However a word of caution

Invoke is required when synchronous execution is expected. If code relying on changes made by BeginInvoke immediately follows in a separate code block, you may have another issue.

i.e.

SomeControl.BeginInvoke(new Action(delegate {

  SomeControl.Text=”some value”

}));

// trying to read SomeControl.Text here is not safe, may not get expected result – move code reyling on BeginInvoke’s result into BeginInvoke or use Invoke

About chentiangemalc

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in .NET, Debugging, IL, WinDbg and tagged . Bookmark the permalink.

2 Responses to Case of the Invoked Hang (.NET)

  1. David Raasch says:

    Ummm… don’t these two lines say the exact same words??:

    Invoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.
    BeginInvoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.

    • Delegate.Invoke: Executes synchronously, on the same thread.
      Delegate.BeginInvoke: Executes asynchronously, on a threadpool thread.
      Control.Invoke: Executes on the UI thread, but calling thread waits for completion before continuing.
      Control.BeginInvoke: Executes on the UI thread, and calling thread doesn’t wait for completion.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s