Debugging / Viewing Windows Update Log on Windows 10 Insider Builds

Having confirmed with WinDbg team currently insider builds don’t get public symbols unless it is a major release (They are looking into a solution….)

How are we going to debug anything on bleeding edge systems “Fast Track Updates” in the meantime?

To workaround this problem I did the following

1) Downloaded the latest Windows 10 symbol package publicly available

https://msdn.microsoft.com/en-us/windows/hardware/gg463028.aspx

I installed them into a folder C:\win10symbols

2) Download ChkMatch tool from here http://www.debuginfo.com/tools/chkmatch.html

Note it is 32-bit only tool, so when specifying path to 64-bit files in C:\windows\system32 you must use path C:\windows\sysnative

3) Delete any files in <debugger directory>\sym folder (i.e. C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\sym )

Here is the WinDbg output:

Before ChkMatch

0:000> .reload
Reloading current modules
…..
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll –

************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  PDB not found : c:\win10symbols\symbols\dll\ntdll.pdb

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:000> !peb
PEB at 00007ff6759af000
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn’t have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing “.symopt- 100”. Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: ntdll!_PEB                                    ***
***                                                                   ***
*************************************************************************
error 3 InitTypeRead( nt!_PEB at 00007ff6759af000)…

I then ran the following command:

C:\>chkmatch -m C:\windows\sysnative\ntdll.dll c:\win10symbols\ntdll.pdb\B9A5AE37F693422C89FBF6471E0FC5F51\ntdll.pdb
ChkMatch – version 1.0
Copyright (C) 2004 Oleg Starodumov
http://www.debuginfo.com/

Executable: C:\windows\sysnative\ntdll.dll
Debug info file: c:\win10symbols\ntdll.pdb\B9A5AE37F693422C89FBF6471E0FC5F51\ntdll.pdb

Executable:
TimeDateStamp: 55cbff9a
Debug info: 2 ( CodeView )
TimeStamp: 55cbff9a  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 34  RVA: 0011ebe8  FileOffset: 0011cbe8
CodeView format: RSDS
Signature: {7edb4a59-efff-47ec-b12f-7697e9226177}  Age: 1
PdbFile: ntdll.pdb
Debug info: 13 ( Unknown )
TimeStamp: 55cbff9a  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 1084  RVA: 0011ec20  FileOffset: 0011cc20

Debug information file:
Format: PDB 7.00
Signature: {b9a5ae37-f693-422c-89fb-f6471e0fc5f5}  Age: 2

Writing to the debug information file…
Result: Success.

0:000> .reload
Reloading current modules
…..
No .natvis files found at C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers.
0:000> !peb
PEB at 00007ff6759af000
    InheritedAddressSpace:    No
    ReadImageFileExecOptions: No
    BeingDebugged:            Yes
    ImageBaseAddress:         00007ff676040000
    Ldr                       00007fffdf8b71e0
    Ldr.Initialized:          Yes

 

etc…

 

Note: This solution only works if the PDB age signature matches the current DLL, more info about this here: http://www.debuginfo.com/articles/debuginfomatch.html

Windows Update Log using PowerShell Get-WindowsUpdateLog just returning GUIDs?

1601/01/01 11:00:00.0000000 980   8592                  Unknown( 14): GUID=464bbb57-d1b3-22d8-23ef-338952442d14 (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 10): GUID=524890e0-a6db-a018-bdf5-146836fbf25d (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 11): GUID=6388909c-9384-8454-6d45-b6165f613f9a (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 11): GUID=524890e0-a6db-a018-bdf5-146836fbf25d (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 23): GUID=ccdda195-7c91-4637-05e0-79cdf332fe13 (No Format Information found).

Using Fiddler I quickly identified the PDB files it wanted:

image

 

Copying the URL column into Excel I did a quick & dirty processing. Filtered on items ending with .PDB and copied the result into a new sheet, then added formula to extract just the filename

=TRIM(RIGHT(SUBSTITUTE(A1,"/",REPT(" ",LEN(A1))),LEN(A1)))

 

Then pivot table on that column we find the PDB files this process relies on:

storewuauth.pdb
wuapi.pdb
wuauclt.pdb
wuaueng.pdb
wuautoappupdate.pdb
wuuhext.pdb

 

This resulted in running the following commands, all of which reported success.

chkmatch -m c:\Windows\WinSxS\amd64_microsoft-windows-s..e-windowsupdateauth_31bf3856ad364e35_10.0.10525.0_none_8b70d1ffa911692a\storewuauth.dll c:\win10symbols\storewuauth.pdb\5A9142A09775486CB3053E2D6D6A7B6F1\storewuauth.pdb chkmatch -m c:\windows\sysnative\wuapi.dll C:\win10symbols\wuapi.pdb\03E47413D978470EB9438AD21D5B208E1\wuapi.pdb chkmatch -m c:\windows\sysnative\wuauclt.exe c:\win10symbols\wuauclt.pdb\E0D0C18A15D24973B0AE8799EFFFC1001\wuauclt.pdb chkmatch -m c:\windows\sysnative\wuaueng.dll c:\win10symbols\wuaueng.pdb\EBADD49A0078487A8BD8DBFE7D56D88B1\wuaueng.pdb chkmatch -m c:\windows\sysnative\wuautoappupdate.dll c:\win10symbols\wuautoappupdate.pdb\2E4D201B897D4FACB97B5198FC9B2CBD1\wuautoappupdate.pdb chkmatch -m c:\windows\sysnative\wuuhext.dll c:\win10symbols\wuuhext.pdb\C94A76FA1A6546D38EBCAAE1B7C0A39C1\wuuhext.pdb

We then run PowerShell cmd

Get-WindowsUpdateLog –SymbolServer C:\win10symbols

However unlike WinDbg, this is not working, as the paths don’t match

image

So I made a copy of each folder, the copy with the same name that tracerpt.exe was looking for

image

WOW! I CAN NOW READ THE WINDOWS UPDATE LOG!!!!!!!!!!!!!!!!!!!!!!!!!!!! WOO-HOO

That was easy.

2015/09/01 10:40:01.8948051 996   9112  SLS             Retrieving SLS response from server using ETAG Ww/uzEDtiHGuCQKdqqiVoucsE3BysNngQpDk+NfngSU=_1440″…”
2015/09/01 10:40:01.8949751 996   9112  SLS             Making request with URL
HTTPS://sls.update.microsoft.com/SLS/{7971F918-A847-4430-9279-4A52D1EFE18D}/x64/10.0.10525.0/0?CH=423&L=en-US&P=&PT=0x30&WUA=10.0.10525.0
2015/09/01 10:40:22.9594201 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:40:22.9594266 996   9112  Misc            WinHttp: SendRequestToServerForFileInformation failed with 0x80072ee2; retrying with default proxy.
2015/09/01 10:40:43.9686947 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:40:43.9687321 996   9112  SLS             Failed: hr = 0x80072EE2
2015/09/01 10:40:43.9690170 996   9112  SLS             GetResponse failed with hresult 0x80072ee2…
2015/09/01 10:40:43.9690306 996   9112  Agent           Failed to retrieve SLS response data for service 7971f918-a847-4430-9279-4a52d1efe18d, error = 0x80072ee2
2015/09/01 10:40:43.9884897 996   9112  SLS             Retrieving SLS response from server using ETAG Ww/uzEDtiHGuCQKdqqiVoucsE3BysNngQpDk+NfngSU=_1440″…”
2015/09/01 10:40:43.9887935 996   9112  SLS             Making request with URL
HTTPS://sls.update.microsoft.com/SLS/{7971F918-A847-4430-9279-4A52D1EFE18D}/x64/10.0.10525.0/0?CH=423&L=en-US&P=&PT=0x30&WUA=10.0.10525.0
2015/09/01 10:41:05.0047561 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:41:05.0047606 996   9112  Misc            WinHttp: SendRequestToServerForFileInformation failed with 0x80072ee2; retrying with default proxy.
2015/09/01 10:41:26.0152702 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:41:26.0152821 996   9112  SLS             Failed: hr = 0x80072EE2
2015/09/01 10:41:26.0157665 996   9112  SLS             GetResponse failed with hresult 0x80072ee2…
2015/09/01 10:41:26.0157936 996   9112  Misc            EP: error: 0x80072EE2 : – failed to get SLS data

Posted in Fiddler, ProcMon, WinDbg, Windows 10 | Tagged | Leave a comment

Case of the Object Is Not Set To An Instance of an Object–.NET Patching

Continuing our series on patching .NET code without source ( https://chentiangemalc.wordpress.com/2015/07/31/case-of-the-black-background-window-net-patching/ )

A .NET application which worked fine on Windows 7, started throwing an exception when opening an image in Windows 10’s default editor. However the images still opened OK. Changing the default image viewer to a 3rd party program also fixed the issue.

Object reference not set to an instance of an object

image

This exception means a method is trying to be called on a null object.

To investigate I asked an engineer to take a dmp file while the error message was displayed with ProcDump http://live.sysinternals.com/procdump.exe

ProcDump -ma <process name>

Because it was a 32-bit .NET application I loaded the dmp in WinDbg x86

0:000> .load C:\windows\microsoft.net\Framework\v2.0.50727\SOS.dll
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.8662 f:0
doesn’t match desired version 2.0.50727.8000 f:0
CLRDLL: Unable to find ” on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL c:\localsymbols\mscordacwks_x86_x86_2.0.50727.8000.dll\526720FC5b0000\mscordacwks_x86_x86_2.0.50727.8000.dll
CLR DLL status: Loaded DLL c:\localsymbols\mscordacwks_x86_x86_2.0.50727.8000.dll\526720FC5b0000\mscordacwks_x86_x86_2.0.50727.8000.dll
0:000> !pe
Exception object: 030ca204
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    00EAE610 0A9E39F9 wds!StreamlineUI.MainForm.DisplayImage(System.String)+0x91

StackTraceString: <none>
HResult: 80004003
0:000> !clrstack
OS Thread Id: 0xfe4 (0)
ESP       EIP    
00ead69c 74b5bfbc [NDirectMethodFrameStandalone: 00ead69c] System.Windows.Forms.SafeNativeMethods.MessageBox(System.Runtime.InteropServices.HandleRef, System.String, System.String, Int32)
00ead6b8 65a61dc8 System.Windows.Forms.MessageBox.ShowCore(System.Windows.Forms.IWin32Window, System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean)
00ead758 65a619de System.Windows.Forms.MessageBox.Show(System.String)
00ead760 0a9e3a68 StreamlineUI.MainForm.DisplayImage(System.String)
00eae640 0a19b912 StreamlineUI.MainForm.doDataBind()
00eae668 0a19a542 StreamlineUI.MainForm.getNextForm(ACTION, DIRECTION, System.String)
00eae690 0a199fae StreamlineUI.MainForm.CheckIn(DIRECTION, System.String)
00eae6a8 0a199cfe StreamlineUI.MainForm.btnNext_Click(System.Object, System.EventArgs)
00eae6e0 65435170 System.Windows.Forms.Control.OnClick(System.EventArgs)
00eae6f8 6543055a System.Windows.Forms.Button.OnClick(System.EventArgs)
00eae708 65ccac9a System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
00eae714 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eae71c 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eae730 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eae930 011207bc [NDirectMethodFrameStandalone: 00eae930] System.Windows.Forms.UnsafeNativeMethods.SendMessage(System.Runtime.InteropServices.HandleRef, Int32, IntPtr, IntPtr)
00eae94c 65474c34 System.Windows.Forms.Control.SendMessage(Int32, IntPtr, IntPtr)
00eae968 6547453b System.Windows.Forms.Control.ReflectMessageInternal(IntPtr, System.Windows.Forms.Message ByRef)
00eae978 6546497e System.Windows.Forms.Control.WmCommand(System.Windows.Forms.Message ByRef)
00eae988 65469d9c System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
00eae98c 65464fe5 [InlinedCallFrame: 00eae98c]
00eae9ec 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eae9f4 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eaea08 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eaedd8 011207bc [InlinedCallFrame: 00eaedd8] System.Windows.Forms.UnsafeNativeMethods.CallWindowProc(IntPtr, IntPtr, Int32, IntPtr, IntPtr)
00eaedd4 6546a033 System.Windows.Forms.NativeWindow.DefWndProc(System.Windows.Forms.Message ByRef)
00eaee18 65469f8c System.Windows.Forms.Control.DefWndProc(System.Windows.Forms.Message ByRef)
00eaee1c 659994e9 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
00eaeea8 65ccc3a6 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
00eaeeac 654638ac [InlinedCallFrame: 00eaeeac]
00eaef44 654637f0 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
00eaef50 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eaef58 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eaef6c 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eaf134 011207bc [NDirectMethodFrameStandalone: 00eaf134] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
00eaf144 65479f8e System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
00eaf1e0 65479bf7 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
00eaf234 65479a41 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
00eaf264 65436911 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
00eaf278 050100ab StreamlineUI.Startup.Main()
00eaf4c8 67101b5c [GCFrame: 00eaf4c8]

CLRStack however doesn’t give us a really good indication of where our exception occurred, for this we will get the more verbose stack output:

0:000> !dumpstack

long long stack trace, then finally …

00eae610 0a9e39f8 (MethodDesc 0x10d610c +0x90 StreamlineUI.MainForm.DisplayImage(System.String)) ====> Exception Code c0000005 cxr@eae1ac exr@eae15c
00eae510 75587890 shell32!CShellExecute::Release+0x30, calling shell32!operator delete
00eae518 76e02c1b KERNELBASE!LocalFree+0x27, calling ntdll!RtlFreeHeap
00eae524 76e02c34 KERNELBASE!LocalFree+0x40, calling KERNELBASE!_SEH_epilog4
00eae55c 76e02c34 KERNELBASE!LocalFree+0x40, calling KERNELBASE!_SEH_epilog4
00eae560 010cb550 010cb550, calling KERNELBASE!GetLastError
00eae57c 6683419e (MethodDesc 0x6664a350 +0x2e System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr)), calling 66782c9c
00eae59c 6683419e (MethodDesc 0x6664a350 +0x2e System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr)), calling 66782c9c
00eae5ac 663db45d (MethodDesc 0x65e68bd0 +0x291 System.Diagnostics.Process.StartWithShellExecuteEx(System.Diagnostics.ProcessStartInfo)), calling (MethodDesc 0x6664a350 +0 System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr))
00eae5e0 663da5cd (MethodDesc 0x65e68b8c +0x39 System.Diagnostics.Process.Start()), calling (MethodDesc 0x65e68bd0 +0 System.Diagnostics.Process.StartWithShellExecuteEx(System.Diagnostics.ProcessStartInfo))
00eae5f0 663da29a (MethodDesc 0x65e68c0c +0x32 System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo)), calling 65f49dc0
00eae600 663da30c (MethodDesc 0x65e68bf4 +0x20 System.Diagnostics.Process.Start(System.String)), calling (MethodDesc 0x65e68c0c +0 System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo))
00eae608 0a9e39f2 (MethodDesc 0x10d610c +0x8a StreamlineUI.MainForm.DisplayImage(System.String)), calling mscorwks!JIT_WriteBarrierEAX
00eae638 0a19b912 (MethodDesc 0x10d6118 +0x262 StreamlineUI.MainForm.doDataBind()), calling 0570a8b8

even more long stack trace continues…

From this we find Process.Start is the last code called before the MessageBox showing the error. Now let’s decompile the function that threw the exception:

0:000> !name2ee wds!StreamlineUI.MainForm.DisplayImage
Module: 010d2c5c (WDS.exe)
Token: 0x060000d3
MethodDesc: 010d610c
Name: StreamlineUI.MainForm.DisplayImage(System.String)
JITTED Code Address: 0a9e3968
0:000> !dumpil 010d610c
ilAddr = 00ceef7c
.try
{
  IL_0000: ldarg.1
  IL_0001: call System.IO.File::Exists
  IL_0006: brtrue.s IL_003d
  IL_0008: ldstr “Unable to locate image file.”
  IL_000d: call System.Environment::get_NewLine
  IL_0012: call System.Environment::get_NewLine
  IL_0017: ldarg.1
  IL_0018: call System.String::Concat
  IL_001d: ldstr “Info”
  IL_0022: call System.Windows.Forms.MessageBox::Show
  IL_0027: pop
  IL_0028: ldarg.0
  IL_0029: ldc.i4.1
  IL_002a: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_002f: ldarg.0
  IL_0030: ldc.i4.6
  IL_0031: call StreamlineUI.MainForm::set_APPMODE
  IL_0036: ldc.i4.0
  IL_0037: stloc.1
  IL_0038: leave IL_00c2
  IL_003d: ldarg.0
  IL_003e: call StreamlineUI.MainForm::KillImageProcess
  IL_0043: pop
  IL_0044: ldarg.0
  IL_0045: ldarg.1
  IL_0046: call System.Diagnostics.Process::Start
  IL_004b: stfld StreamlineUI.MainForm::imageViewProcess
  IL_0050: ldarg.0
  IL_0051: ldfld StreamlineUI.MainForm::imageViewProcess

We can infer the call to WaitForInputIdle this is the culprit. It is calling a method on the return value from Process Start

  IL_0056: callvirt System.Diagnostics.Process::WaitForInputIdle
  

  IL_005b: pop
  IL_005c: ldarg.0
  IL_005d: call StreamlineUI.MainForm::get_APPMODE
  IL_0062: ldc.i4.2
  IL_0063: bne.un.s IL_0093
  IL_0065: ldarg.0
  IL_0066: ldfld StreamlineUI.MainForm::grdTransactions
  IL_006b: callvirt Infragistics.Win.UltraWinGrid.UltraGridBase::get_Rows
  IL_0070: callvirt Infragistics.Shared.DisposableObjectCollectionBas::get_Count
  IL_0075: brtrue.s IL_0093
  IL_0077: ldarg.0
  IL_0078: ldfld StreamlineUI.MainForm::grdTransactions
  IL_007d: callvirt Infragistics.Win.UltraWinGrid.UltraGridBase::get_DisplayLayout
  IL_0082: callvirt Infragistics.Win.UltraWinGrid.UltraGridLayout::get_Bands
  IL_0087: ldc.i4.0
  IL_0088: callvirt Infragistics.Win.UltraWinGrid.BandsCollection::get_Item
  IL_008d: callvirt Infragistics.Win.UltraWinGrid.UltraGridBand::AddNew
  IL_0092: pop
  IL_0093: ldarg.0
  IL_0094: call StreamlineUI.MainForm::EnterEditMode
  IL_0099: ldarg.0
  IL_009a: ldc.i4.0
  IL_009b: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_00a0: leave.s IL_00ba
} // end .try
.catch
{
  IL_00a2: stloc.0
  IL_00a3: ldloc.0
  IL_00a4: callvirt System.Exception::get_Message
  IL_00a9: call System.Windows.Forms.MessageBox::Show <- displays the error message
  IL_00ae: pop
  IL_00af: ldarg.0
  IL_00b0: ldc.i4.1
  IL_00b1: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_00b6: ldc.i4.0
  IL_00b7: stloc.1
  IL_00b8: leave.s IL_00c2
} // end .catch
IL_00ba: ldarg.0
IL_00bb: call System.Windows.Forms.Form::Activate
IL_00c0: ldc.i4.1
IL_00c1: ret
IL_00c2: ldloc.1
IL_00c3: ret

If you are uncomfortable with IL we can save the module then open with .NET Reflector:

0:000> lmvm wds
Browse full module list
start    end        module name
00ce0000 00d26000   wds      C (no symbols)          
    Loaded symbol image file: wds.exe
    Image path: C:\Users\user\WDSv5.4\UAT\wds.exe
    Image name: wds.exe
    Browse all global symbols  functions  data
    Has CLR image header, track-debug-data flag not set
    Timestamp:        Thu May 08 14:10:04 2014 (536B039C)
    CheckSum:         00000000
    ImageSize:        00046000
    File version:     5.0.0.4
    Product version:  5.0.0.4
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04b0
    CompanyName:      IT Dept
    ProductName:      Work Distribution System
    InternalName:     WDS.exe
    OriginalFilename: WDS.exe
    ProductVersion:   5.0.0.4
    FileVersion:      5.0.0.4
    FileDescription:  WDS Application
    LegalCopyright:   2005, IT Dept
0:000> !savemodule 00ce0000 c:\support\wds.exe
3 sections in file
section 0 – VA=2000, VASize=3f064, FileAddr=1000, FileSize=40000
section 1 – VA=42000, VASize=730, FileAddr=41000, FileSize=1000
section 2 – VA=44000, VASize=c, FileAddr=42000, FileSize=1000

The section of code at fault decompiled in C#

We expect this.imageViewProcess must be NULL when this.imageViewProcess.WaitForInputIdle() is called

private bool DisplayImage(string imagePath) { try { if (!File.Exists(imagePath)) { MessageBox.Show("Unable to locate image file." + Environment.NewLine + Environment.NewLine + imagePath, "Info"); this.IMAGEMISSING = true; this.APPMODE = 6; return false; } this.KillImageProcess(); this.imageViewProcess = Process.Start(imagePath); this.imageViewProcess.WaitForInputIdle(); if ((this.APPMODE == 2) && (this.grdTransactions.get_Rows().get_Count() == 0)) { this.grdTransactions.get_DisplayLayout().get_Bands().get_Item(0).AddNew(); } this.EnterEditMode(); this.IMAGEMISSING = false; } catch (Exception exception) { MessageBox.Show(exception.Message); this.IMAGEMISSING = true; return false; } base.Activate(); return true; }

Let’s test the theory in PowerShell, you can see when attempting to Process Start an image then call WaitForInputIdle we get an InvokeMethodOnNull exception, this is exactly the same exception thrown by our .NET application.

PS C:\Users\Malcolm> $p = [System.Diagnostics.Process]::Start("notepad.exe") $p.WaitForInputIdle() True PS C:\Users\Malcolm> $p = [System.Diagnostics.Process]::Start("C:\Users\Malcolm\Pictures\DSC01230.JPG") $p.WaitForInputIdle() You cannot call a method on a null-valued expression. At line:3 char:1 + $p.WaitForInputIdle() + ~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (:) [], RuntimeException + FullyQualifiedErrorId : InvokeMethodOnNull PS C:\Users\Malcolm>

Why Process.Start returning NULL?  Checking the .NET documentation  at https://msdn.microsoft.com/en-US/library/53ezey2s(v=vs.80).aspx

Return Value

A new Process component that is associated with the process resource, or a null reference (Nothing in Visual Basic), if no process resource is started (for example, if an existing process is reused).

Use this overload to start a process resource by specifying its file name. The overload associates the resource with a new Process component. If the process is already running, no additional process resource is started. Instead, the existing process resource is reused and no new Process component is created. In such a case, instead of returning a new Process component, Start returns a null reference (Nothing in Visual Basic) to the calling procedure.

Following the source code 

http://referencesource.microsoft.com/#System/services/monitoring/system/diagnosticts/Process.cs

we can see in how the “false” return results in null

public static Process Start(ProcessStartInfo startInfo) { Process process = new Process(); if (startInfo == null) throw new ArgumentNullException("startInfo"); process.StartInfo = startInfo; if (process.Start()) { return process; } return null; // IF process.Start returns FALSE, then we will get NULL back, like in our crashing program }

Following process.Start leads us to StartWithShellExecuteEx where we can see if shellExecuteInfo.hProcess is null, FALSE gets returned.

private bool StartWithShellExecuteEx(ProcessStartInfo startInfo) { <code…> if (shellExecuteInfo.hProcess != (IntPtr)0) { SafeProcessHandle handle = new SafeProcessHandle(shellExecuteInfo.hProcess); SetProcessHandle(handle); return true; }

return false;

 

How do we fix it if the vendor can’t/won’t/source code not available?

PATCH it…

To fix this bug we just need a simple NULL check

To do this we will use ILDASM from Windows SDK to dump the contents into a folder, open the IL file, and search for our DisplayImage method.

Our patched code will be

ldarg.0

ldfld      class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess
ldnull
cgt.un
stloc.0
ldloc.0
brfalse.s  IL_005c

Refer to http://www.ecma-international.org/publications/standards/Ecma-335.htm Partition III for an explanation of the commands.

In context:

.method private hidebysig instance bool DisplayImage(string imagePath) cil managed { // Code size 196 (0xc4) .maxstack 4 .locals init (class [mscorlib]System.Exception V_0, bool V_1) .try { IL_0000: ldarg.1 IL_0001: call bool [mscorlib]System.IO.File::Exists(string) IL_0006: brtrue.s IL_003d IL_0008: ldstr "Unable to locate image file." IL_000d: call string [mscorlib]System.Environment::get_NewLine() IL_0012: call string [mscorlib]System.Environment::get_NewLine() IL_0017: ldarg.1 IL_0018: call string [mscorlib]System.String::Concat(string, string, string, string) IL_001d: ldstr "Info" IL_0022: call valuetype [System.Windows.Forms]System.Windows.Forms.DialogResult [System.Windows.Forms]System.Windows.Forms.MessageBox::Show(string, string) IL_0027: pop IL_0028: ldarg.0 IL_0029: ldc.i4.1 IL_002a: stfld bool StreamlineUI.MainForm::IMAGEMISSING IL_002f: ldarg.0 IL_0030: ldc.i4.6 IL_0031: call instance void StreamlineUI.MainForm::set_APPMODE(valuetype [StreamlineClsLib]StreamlineClsLib.Form/ACTION) IL_0036: ldc.i4.0 IL_0037: stloc.1 IL_0038: leave IL_00c2 IL_003d: ldarg.0 IL_003e: call instance bool StreamlineUI.MainForm::KillImageProcess() IL_0043: pop IL_0044: ldarg.0 IL_0045: ldarg.1 IL_0046: call class [System]System.Diagnostics.Process [System]System.Diagnostics.Process::Start(string) IL_004b: stfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess ldarg.0 ldfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess ldnull cgt.un stloc.0 ldloc.0 brfalse.s IL_005c IL_0050: ldarg.0 IL_0051: ldfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess IL_0056: callvirt instance bool [System]System.Diagnostics.Process::WaitForInputIdle() IL_005b: pop IL_005c: ldarg.0 IL_005d: call instance valuetype [StreamlineClsLib]StreamlineClsLib.Form/ACTION StreamlineUI.MainForm::get_APPMODE() IL_0062: ldc.i4.2

We then recompile

c:\windows\microsoft.net\Framework\v2.0.50727\ilasm app.il /RESOURCE=app.res

Check the compiled fix with .NET reflector :

This looks good (!= null decompiles as > null) due to cgt.un instruction which is : Push 1 (of type int32) if value1 > value2, unsigned or unordered, else push 0.

Testing the application, yet another bug patched!

Posted in .NET, C#, Hacking, IL, MSIL, Patching, Reverse Engineering | Tagged | Leave a comment

Case of the .NET Memory Leak via Gfx Driver on Windows 8

A .NET 4 application used to index scanned documents started to fail on Windows 8

clip_image002

Typically the recommendation for Not enough storage is available to process this command might involve:

  • reduce the number of running programs;
  • remove unwanted files from the disk the paging file is on and restart the system;
  • check the paging file disk for an I/O error; or
  • install additional memory in your system

However in this case these options didn’t fix the issue.

Taking a dmp file with ProcDump –ma option (http://live.sysinternals.com/Procdump.exe) we find the following:

Load SOS extensions for debugging .NET process

0:000> .loadby sos clr
0:000> .cordll -ve -u -l

Dumping last CLR exception…

0:000> !pe
Exception object: 07559bb0
Exception type:   System.IO.FileLoadException
Message:          Could not load file or assembly ‘System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’ or one of its dependencies. Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008)
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 UNKNOWN!DocumentAutomation.Common.ServiceAdaptor.ESBV4ServicesAdaptor..ctor(System.String, System.String, System.String[,], System.String, System.TimeSpan)+0x2
    005DE908 04CE30E5 DocumentAutomation_Indexing!DocumentAutomation.Indexing.ViewModels.IndexingViewModel.OnValidateCase()+0x265
StackTraceString: <none>
HResult: 80070008
0:000> lmvm DocumentAutomation*
Browse full module list
start    end        module name
00420000 0045c000   DocumentAutomation_Indexing C (no symbols)          
    Loaded symbol image file: DocumentAutomation.Indexing.exe
    Image path: C:\Program Files (x86)\TIS\eFlow 4.5\Bin\DocumentAutomation.Indexing.exe
    Image name: DocumentAutomation.Indexing.exe
    Browse all global symbols  functions  data
    Has CLR image header, track-debug-data flag not set
    Timestamp:        Wed Mar 25 19:18:50 2015 (55126F6A)
    CheckSum:         00000000
    ImageSize:        0003C000
    File version:     1.0.0.0
    Product version:  1.0.0.0
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04b0
    ProductName:      DocumentAutomation.Indexing
    InternalName:     DocumentAutomation.Indexing.exe
    OriginalFilename: DocumentAutomation.Indexing.exe
    ProductVersion:   1.0.0.0
    FileVersion:      1.0.0.0
    FileDescription:  DocumentAutomation.Indexing
LegalCopyright:   Copyright ©  2012

Extracting the EXE in WinDbg, we can then open in .NET reflector to decompile

0:000> !SaveModule 00420000 C:\support\DocumentAutomation.Indexing.exe
3 sections in file
section 0 – VA=2000, VASize=350f4, FileAddr=200, FileSize=35200
section 1 – VA=38000, VASize=5d0, FileAddr=35400, FileSize=600
section 2 – VA=3a000, VASize=c, FileAddr=35a00, FileSize=200

Here we can find the code that threw the error. The issue is initialization of ESBV4ServicesAdaptor is failing when it tries to load System.ServiceModel.dll. You may need to take a few dmp files to confirm if it is always failing in exact same place.

Private Sub OnValidateCase()
    Try
        Dim base2 As CaseValidatorBase
        Me.AutoPopulateDocumentFields
        If Me._globalSettings.PlanValidationConnector(Me.CurrentCase.get_Plan).Equals(“Internal”, StringComparison.OrdinalIgnoreCase) Then
            base2 = New InternalCaseValidator(Me.CurrentCase, Me.eFlow.Collection, Me._globalSettings)
        Else
Dim adaptor As New ESBV4ServicesAdaptor(Me._globalSettings.GetSetting(“AaspireUserID”), Encryption.Decrypt(Me._globalSettings.GetSetting(“AaspirePassword”)), Me._globalSettings.ImagaasPlanIdsPlancodes, Me._globalSettings.GetSetting(“V4RemoteEndPoint”), New TimeSpan(0, Integer.Parse(Me._globalSettings.GetSetting(“ServiceAdaptorSendTimeoutMins”)), 0))
            base2 = New CaseValidator(Me.CurrentCase, adaptor, Me.eFlow.Collection, Me._globalSettings.GetSetting(“AaspireUserID”), Encryption.Decrypt(Me._globalSettings.GetSetting(“AaspirePassword”)))
        End If

        base2.Validate
        Me.AutoPopulateDocumentFields
        Me.SyncCaseFields
        Me.SelectedPageImage.RefreshDocumentField
        Me.Status = IndexingStatus.Validated
        If (base2.get_ValidationMsgs.Count > 0) Then
            MessageBox.Show(base2.GetMassages, “Validation Message”, MessageBoxButton.OK)
        End If
    Catch exception As InvalidCaseException
        Dim nullable As Integer?
        Dim e As InvalidCaseException = exception
        Dim messageBoxText As String = If(e.ValidationMessage.get_PageIndex.HasValue, String.Format(“Page {0}: {1}”, If(nullable = e.ValidationMessage.get_PageIndex.HasValue, New Integer?((nullable.GetValueOrDefault + 1)), Nothing), e.Message), e.Message)
        MessageBox.Show(messageBoxText, e.ValidationMessage.get_Field, MessageBoxButton.OK)
        If e.ValidationMessage.get_PageIndex.HasValue Then
            Dim classf As <>c__DisplayClass5f
            Me.SelectedPageImage = Enumerable.FirstOrDefault(Of PageImage)(Me.PageImages, New Func(Of PageImage, Boolean)(classf, DirectCast(Me.<OnValidateCase>b__5e, IntPtr)))
        End If
    Catch exception2 As Exception
MessageBox.Show(exception2.Message, “Fatal Error”, MessageBoxButton.OK, MessageBoxImage.Hand)
    End Try
End Sub

Because this is failing to initialize an object, a common issue may be out of process address space, and this app is running as 32-bit program:


Looking at memory usage…
0:000> !address -summary
Mapping file section regions…
Mapping module regions…
Mapping PEB regions…
Mapping TEB and stack regions…
Mapping heap regions…
Mapping page heap regions…
Mapping other regions…
Mapping stack trace database regions…
Mapping activation context regions…
— Usage Summary —————- RgnCount ———– Total Size ——– %ofBusy %ofTotal
Heap                                    344          5ad52000 (   1.419 GB)  78.89%   70.97%
Free                                    320           cdd5000 ( 205.832 MB)           10.05%
Image                                  1037           c462000 ( 196.383 MB)  10.66%    9.59%
<unknown>                               757           b053000 ( 176.324 MB)   9.57%    8.61%
Stack                                    55            fc0000 (  15.750 MB)   0.85%    0.77%
Other                                     9             41000 ( 260.000 kB)   0.01%    0.01%
TEB                                      18             12000 (  72.000 kB)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kB)   0.00%    0.00%
— Type Summary (for busy) —— RgnCount ———– Total Size ——– %ofBusy %ofTotal
MEM_PRIVATE                             705          60ec8000 (   1.514 GB)  84.19%   75.72%
MEM_IMAGE                              1467           d1cd000 ( 209.801 MB)  11.39%   10.24%
MEM_MAPPED                               49           5186000 (  81.523 MB)   4.43%    3.98%
— State Summary —————- RgnCount ———– Total Size ——– %ofBusy %ofTotal
MEM_COMMIT                             1686          6e10d000 (   1.720 GB)  95.60%   85.99%
MEM_FREE                                320           cdd5000 ( 205.832 MB)           10.05%
MEM_RESERVE                             535           510e000 (  81.055 MB)   4.40%    3.96%
— Protect Summary (for commit) – RgnCount ———– Total Size ——– %ofBusy %ofTotal
PAGE_READWRITE                          635          5d9f3000 (   1.463 GB)  81.32%   73.14%
PAGE_EXECUTE_READ                       118           8ac2000 ( 138.758 MB)   7.53%    6.78%
PAGE_READONLY                           496           623a000 (  98.227 MB)   5.33%    4.80%
PAGE_WRITECOPY                          252           1681000 (  22.504 MB)   1.22%    1.10%
PAGE_EXECUTE_READWRITE                   97            1f5000 (   1.957 MB)   0.11%    0.10%
PAGE_EXECUTE_WRITECOPY                   50            14c000 (   1.297 MB)   0.07%    0.06%
PAGE_READWRITE|PAGE_GUARD                36             5a000 ( 360.000 kB)   0.02%    0.02%
PAGE_NOACCESS                             2              2000 (   8.000 kB)   0.00%    0.00%
— Largest Region by Usage ———– Base Address ——– Region Size ———-
Heap                                         e570000            fcf000 (  15.809 MB)
Free                                        5e820000            d80000 (  13.500 MB)
Image                                       57388000            f1a000 (  15.102 MB)
<unknown>                                    2120000           1cae000 (  28.680 MB)
Stack                                        2000000             fd000 (1012.000 kB)
Other                                       7ed90000             23000 ( 140.000 kB)
TEB                                         7ebf7000              1000 (   4.000 kB)
PEB                                         7edbe000              1000 (   4.000 kB)
0:000> !heap -stat
_HEAP 006f0000
     Segments            00000004
         Reserved  bytes 007fc000
         Committed bytes 005c5000
     VirtAllocBlocks     0000005d
         VirtAlloc bytes e255b09c
_HEAP 00b80000
     Segments            00000009
         Reserved  bytes 04e47000
         Committed bytes 041cb000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 09170000
     Segments            00000002
         Reserved  bytes 0010e000
         Committed bytes 00018000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 00990000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 0000f000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 04780000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00008000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 04750000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00007000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 02110000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00003000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 01ff0000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00001000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
0:000> !heap -s
************************************************************************************************************************
                                              NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key                   : 0x271d989d
Termination on corruption : DISABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                    (k)     (k)    (k)     (k) length      blocks cont. heap
—————————————————————————–
006f0000 00000002    8284   5924   8176   1328   202     4   93      1   LFH
    External fragmentation  22 % (202 free blocks)
00990000 00001002     168     64     60      9     7     1    0      0   LFH
00b80000 00001002   80264  67380  80156    294    30     9    0      0   LFH
02110000 00001002      60     12     60      3     6     1    0      0     
01ff0000 00041002      60      4     60      2     1     1    0      0     
04780000 00001002     168     36     60      2     3     1    0      0   LFH
04750000 00041002     168     32     60      1     5     1    0      0   LFH
09170000 00001002    1188    100   1080     50     5     2    0      0   LFH
—————————————————————————–
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x07ca8d10
generation 1 starts at 0x07c51000
generation 2 starts at 0x02121000
ephemeral segment allocation context: none
         segment             begin         allocated  size
02120000  02121000  0311ffe4  0xffefe4(16773092)
06ab0000  06ab1000  0757e71c  0xacd71c(11327260)
07c50000  07c51000  0819fca8  0x54eca8(5565608)
Large object heap starts at 0x03121000
         segment             begin         allocated  size
03120000  03121000  03dadaa8  0xc8caa8(13159080)
Total Size:              Size: 0x2ca7e50 (46825040) bytes.
——————————
GC Heap Size:    Size: 0x2ca7e50 (46825040) bytes.

0:000> !dumpheap -stat
    Statistics:

          MT    Count    TotalSize Class Name
etc…

    60f3b340     9234       258552 System.Reflection.Emit.ScopeTree
    60f6b8fc     4647       260232 System.Reflection.RuntimePropertyInfo
    60f642d0     8139       260448 System.EventHandler
    60f70b28     9622       269416 System.Collections.ArrayList+ArrayListEnumeratorSimple
    57501c04     3616       274816 System.Windows.Data.BindingExpression
    06256688     4414       300152 TiS.Core.eFlowAPI.TisROIMiscellaneousParams
    06256160     4414       300152 TiS.Core.eFlowAPI.TisROIPostProcessingParams
    06256b64     4622       314296 TiS.Core.eFlowAPI.TisOCRGroupParamsLinksContainer
    60f7a740     3312       318672 System.Runtime.CompilerServices.ConditionalWeakTable`2+Entry[[System.Object, mscorlib],[System.Object, mscorlib]][]
    574fbd18     1294       320912 System.Windows.Controls.Border
    60f4356c     9234       332424 System.Reflection.Emit.DynamicResolver
    60f763d4    30341       364092 System.WeakReference
    60f75a78     8363       367972 System.Signature
    60f3b1d0    11905       380960 System.Reflection.Emit.SignatureHelper
    60f71238     2920       424800 System.Collections.Hashtable+bucket[]
    009846b0     4196       429232 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[System.Attribute[], mscorlib]][]
    60f6c290     9901       475248 System.RuntimeMethodInfoStub
    062551fc     4414       476712 TiS.Core.eFlowAPI.TisROIParams
    0500b718    15820       506240 TiS.Core.eFlowAPI.NamedObjectListEvent
    60f65af8     9234       517104 System.Reflection.Emit.DynamicMethod
    60f73acc    13901       641256 System.Int32[]
    60f6bed4    10749       644940 System.Reflection.RuntimeMethodInfo
    58eccce8     9114       846936 System.Windows.EffectiveValueEntry[]
    60f433b8     9234       886464 System.Reflection.Emit.DynamicILGenerator
    60f35738   107092      3501976 System.Object[]
    60f75670    51775      5672668 System.Byte[]
    60f721b4    57971      6750108 System.String
    00702bd0      223      8557414      Free
    Total 783513 objects
    Fragmented blocks larger than 0.5 MB:
        Addr     Size      Followed by
    08039088    1.4MB         08194298 System.String

 

From the memory analysis we find heap is large 1.4GB ram, but the managed objects are very small in size, so looks like we have a native memory leak. This can be further researched with

However we already had several dmp files of the process as memory grew, so we took a look at these.

A raw analysis of the dmp file showed 73% of the dmp file was filled with 0xFF – we suspect this was 0xFF 0xFF 0xFF representing white background on the scanned documents.

From !eestack we saw this stack pattern always present in each dmp file, allocating to heap

 

Current frame: ntdll!NtRemoveIoCompletion+0xc
ChildEBP RetAddr  Caller, Callee
05e6e988 748f2dfa KERNELBASE!GetQueuedCompletionStatus+0x2a, calling ntdll!NtRemoveIoCompletion
05e6e9c0 673f94e1 clr!ThreadpoolMgr::CompletionPortThreadStart+0x22f, calling kernel32!GetQueuedCompletionStatusStub
05e6ea54 6734814d clr!Thread::intermediateThreadProc+0x4d
05e6ed20 76fb81b5 ntdll!bsearch+0x6f
05e6ed44 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6ed64 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6ed7c 71db107b mscoreei!ShimLog::Log+0x1a4, calling mscoreei!__security_check_cookie
05e6ed9c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6edb8 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6edf8 76fdb1b7 ntdll!LdrpApplyLookupReference+0x1e, calling ntdll!RtlIsCriticalSectionLockedByThread
05e6ee04 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6ee08 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee38 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee3c 76fc53eb ntdll!LdrpFindOrMapDll+0xfd6, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6ee48 76fc544d ntdll!LdrpFindOrMapDll+0xf06, calling ntdll!LdrpFindOrMapDll+0xf40
05e6ee58 76fc54ac ntdll!LdrpFindOrMapDll+0x107e, calling ntdll!__security_check_cookie
05e6eeec 76fb8c78 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x2d8, calling ntdll!memset
05e6efb4 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6efd4 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6f00c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6f028 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6f05c 76fb9493 ntdll!LdrpApplyFileNameRedirection+0x96, calling ntdll!RtlDosApplyFileIsolationRedirection_Ustr
05e6f08c 76fb9424 ntdll!LdrpApplyFileNameRedirection+0xac, calling ntdll!__security_check_cookie
05e6f0ac 76fc5c5b ntdll!LdrpFindOrMapDependency+0x209, calling ntdll!LdrpFindOrMapDll
05e6f0c8 76fc5c79 ntdll!LdrpFindOrMapDependency+0x227, calling ntdll!LdrpFindOrMapDependency+0x2d9
05e6f0d8 76fc5cba ntdll!LdrpFindOrMapDependency+0x268, calling ntdll!__security_check_cookie
05e6f15c 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6f164 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f194 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f198 76fcff38 ntdll!LdrpFindLoadedDll+0x2ea, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6f1ac 76fee967 ntdll!LdrpFindLoadedDll+0x342, calling ntdll!__security_check_cookie
05e6f1bc 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f304 76fc4d3e ntdll!LdrpGetProcedureAddress+0x3d, calling ntdll!RtlImageNtHeaderEx
05e6f350 76f8e385 ntdll!LdrpResolveNonStaticDependency+0x22f, calling ntdll!LdrpDereferenceNode
05e6f384 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f398 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f3a8 76f8e47c ntdll!LdrGetProcedureAddressForCaller+0x312, calling ntdll!LdrpReleaseLoaderLock
05e6f3c4 748bf769 KERNELBASE!_KernelBaseBaseDllInitialize+0x313, calling KERNELBASE!RegKrnInitialize
05e6f3d4 748bf77a KERNELBASE!_KernelBaseBaseDllInitialize+0x324, calling KERNELBASE!__security_check_cookie
05e6f45c 76fd04c3 ntdll!RtlpLowFragHeapAllocFromContext+0x34d, calling ntdll!memset
05e6f4d8 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f4f0 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f518 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f528 7479b2e4 msvcrt!_VEC_memzero+0x36, calling msvcrt!fastzero_I
05e6f550 6315ba71 igdumdim32!OpenAdapter+0x15aa71, calling ntdll!RtlLeaveCriticalSection
05e6f55c 631550ef igdumdim32!OpenAdapter+0x1540ef, calling igdumdim32!OpenAdapter+0x15aa5e
05e6f564 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f590 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f594 63151741 igdumdim32!OpenAdapter+0x150741, calling igdumdim32!OpenAdapter+0x15403e
05e6f5a0 681b30a1 msctfui!_CRT_INIT+0x1ea, calling msctfui!__security_check_cookie
05e6f5ac 681a32c6 msctfui!DllMain+0x10, calling kernel32!GetCurrentThreadId
05e6f5bc 681b31f7 msctfui!__DllMainCRTStartup+0x107, calling msctfui!DllMain
05e6f5c4 681b366e msctfui!_SEH_epilog4_GS+0xa, calling msctfui!__security_check_cookie
05e6f5c8 681b3379 msctfui!__DllMainCRTStartup+0x289, calling msctfui!_SEH_epilog4_GS
05e6f608 631518a5 igdumdim32!OpenAdapter+0x1508a5, calling igdumdim32!OpenAdapter+0x155225
05e6f624 76fb96c4 ntdll!RtlDeactivateActivationContextUnsafeFast+0x9c, calling ntdll!__security_check_cookie
05e6f640 76fb9758 ntdll!LdrpCallInitRoutine+0x43, calling ntdll!LdrxCallInitRoutine
05e6f650 76fb9776 ntdll!LdrpCallInitRoutine+0x61, calling ntdll!_SEH_epilog4
05e6f668 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f67c 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f68c 76fd5b55 ntdll!LdrpInitializeThread+0x1af, calling ntdll!LdrpReleaseLoaderLock
05e6f698 76fd5b43 ntdll!LdrpInitializeThread+0x139, calling ntdll!_SEH_epilog4
05e6f6ec 76fd5ab4 ntdll!LdrpInitializeThread+0xeb, calling ntdll!RtlActivateActivationContextUnsafeFast
05e6f6f0 76fd5b1d ntdll!LdrpInitializeThread+0x1a4, calling ntdll!RtlDeactivateActivationContextUnsafeFast
05e6f72c 76fcaa56 ntdll!_LdrpInitialize+0x80, calling ntdll!_SEH_epilog4
05e6f774 76fca9ca ntdll!LdrInitializeThunk+0x1a, calling ntdll!NtContinue
05e6f9e0 6734813b clr!Thread::intermediateThreadProc+0x3b, calling clr!_alloca_probe_16
05e6f9f4 74cc7c04 kernel32!BaseThreadInitThunk+0x24
05e6fa08 76fcad1f ntdll!__RtlUserThreadStart+0x2f
05e6fa50 76fcacea ntdll!_RtlUserThreadStart+0x1b, calling ntdll!__RtlUserThreadStart

This seemed to be coming from user mode graphics driver. Interestingly on working machines a similar stack pattern was present, minus the user mode graphics driver.

0:000> lmvm igdumdim32
Browse full module list
start    end        module name
62b70000 64249000   igdumdim32   (deferred)            
    Image path: C:\Windows\System32\igdumdim32.dll
    Image name: igdumdim32.dll
    Browse all global symbols  functions  data
    Timestamp:        Wed Mar 04 04:29:18 2015 (54F5EF6E)
    CheckSum:         016F3784
    ImageSize:        016D9000
    File version:     10.18.14.4156
    Product version:  10.18.14.4156
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.8 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Intel Corporation
    ProductName:      Intel HD Graphics Drivers for Windows 8(R)
    InternalName:     igdumdim32.dll
    OriginalFilename: igdumdim32.dll
    ProductVersion:   10.18.14.4156
    FileVersion:      10.18.14.4156
    FileDescription:  User Mode Driver for Intel(R) Graphics Technology
    LegalCopyright:   Copyright (c) 1998-2013 Intel Corporation.

 

So I wrote a simple console C# program to test something, to show the RenderCapability ( https://msdn.microsoft.com/en-us/library/system.windows.media.rendercapability(v=vs.110).aspx ) of the device. Added reference PresentationCore

using System; using System.Windows.Media; namespace RenderCapabilityConsoleApplication { class Program { static void Main(string[] args) { Console.WriteLine("RenderCapability:"); Console.WriteLine("Tier: {0}", RenderCapability.Tier >> 16); switch (RenderCapability.Tier) { case 0: Console.WriteLine("No graphics hardware acceleration is available for the application on the device.All graphics features use software acceleration.The DirectX version level is less than version 9.0."); break; case 0x00010000: Console.WriteLine("Most of the graphics features of WPF will use hardware acceleration if the necessary system resources are available and have not been exhausted. This corresponds to a DirectX version that is greater than or equal to 9.0."); break; case 0x00020000: Console.WriteLine("Most of the graphics features of WPF will use hardware acceleration provided the necessary system resources have not been exhausted.This corresponds to a DirectX version that is greater than or equal to 9.0."); break; } Console.Write("Press <Enter>"); Console.ReadLine(); } } }

What we found was this:

  • The broken machine reported capability 0 (No hardware acceleration)
  • The working machines (Windows XP+Windows 7) reported render capability 2 (Most of the features of WPF will use hardware acceleration)

Now a machine can be forced down to capability 0 by remote controlling it. However in this case the machine was not remote controlled.

So we RDP’d to the working Win7/Windows XP machines – they were now breaking too with out of memory error, only when under RDP session:

clip_image002[8]

Running DxDiag on the affected machine, we found no video acceleration capability was available on the affected machine.

Checking device drivers we found a 3rd party video adapter installed which was used by the IT department for remote control ( CA ITCM Remote Control ) 

Unfortunately DxDiag didn’t seem to capture this display adapter in its report, we found it via Device Manager.

Removing this device driver set the machines render capability back to 0, and memory leak was gone, application now worked fine…

Posted in .NET, C#, Debugging, Windows 8 | Tagged | Leave a comment

Extract Module Info From a DMP File with PowerShell

Modify $cdb to point to CDB.exe from Windows SDK Debugging Tools.

At end of script contains usage example…

Script can be downloaded here http://1drv.ms/1MEeIqD

Set-StrictMode -Version 2.0 # path to CDB from Windows SDK Debugging Tools $cdb = "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" function Get-ModulesFromDmp { Param( [string]$FilePath ) if (-not ([System.Management.Automation.PsTypeName]'moduleinfo').Type) { Add-Type @" public struct moduleinfo { public string imagepath; public string imagename; public string timestamp; public string checksum; public string imagesize; public System.Version fileversion; public System.Version productversion; public string fileflags; public string fileos; public string filetype; public string filedate; public string translations; public string companyname; public string productname; public string internalname; public string originalfilename; public string productversiontext; public string fileversiontext; public string privatebuild; public string filedescription; public string legalcopyright; public string notes; } "@ } $moduledata = &$cdb -z "$FilePath" -c "lmv;q" $modules = @() $started = $false $unloadedmodules = $false $info = $null $moduledata.Split("`r`n") | ForEach-Object { if ($_.StartsWith("Unloaded modules")) { $unloadedmodules = $true } if ($started) { if (-Not $_.StartsWith(" ")) { if ($info -eq $null) { $info = New-Object moduleinfo } else { $modules+=$info $info = New-Object moduleinfo } } if ($_.Contains(":")) { $name=$_.Split(":")[0].Trim() $value=$_.Substring($_.IndexOf(":")+1).Trim() switch ($name) { "Image path" { $info.ImagePath = $value } "Image name" { $info.imagename = $value } "Timestamp" { $info.timestamp = $value } "Checksum" { $info.checksum = $value } "ImageSize" { $info.imagesize = $value } "File version" { $info.fileversion = New-Object System.Version($value) } "Product version" { $info.productversion = New-Object System.Version($value) } "File flags" { $info.fileflags = $value } "File OS" { $info.fileos = $value } "File type" { $info.filetype = $value } "File date" { $info.filedate = $value } "Translations" { $info.translations = $value } "CompanyName" { $info.companyname = $value } "ProductName" { $info.productname = $value } "InternalName" { $info.internalname = $value } "OriginalFilename" { $info.originalfilename = $value } "ProductVersion" { $info.productversiontext = $value } "FileVersion" { $info.fileversiontext = $value } "FileDescription" { $info.filedescription = $value } "LegalCopyright" { $info.legalcopyright = $value } "PrivateBuild" { $info.privatebuild = $value } } } else { if ($unloadedmodules) { $info.imagename = $_.Substring(21) $info.filedescription="unloaded module" } else { if ($_.StartsWith(" ")) { $info.notes += $_.Trim() } } } } else { if ($_.StartsWith("start")) { $started=$true } } } return $modules } # usage example $result = Get-ModulesFromDmp "C:\dumps\DocumentAutomation.Indexing.exe_150819_123051.dmp" # make a csv #$result | Export-Csv C:\support\out4.csv -NoTypeInformation # get a grid view $result | Out-GridView

Posted in PowerShell, WinDbg | Tagged | Leave a comment

Using PowerShell to Decode HResults

Here is some code that breaks a HResul (the 8 digit hex error codes or long negative decimal number error codes) into its subcomponents, this can make it easier to identify the source of an error…

You can read more about HRESULT here https://msdn.microsoft.com/en-us/library/cc231198.aspx

This requires PowerShell v3.0 or higher for the bitwise operations.

Script can be downloaded here: http://1drv.ms/1NtDBSL

#based on winerror.h from Windows 10 SDK function Get-HresultFailed($hr) { return $hr -lt 0 } function Get-HresultCode($hr) { return $hr -band 0xFFFF } function Get-ErrorMessage($code) { $ex = New-Object System.ComponentModel.Win32Exception($code) return $ex.Message } function Get-HresultFacility($hr) { $facilityCode = (($hr) -shr 16) -band 0x1fff switch ($facilityCode) { 0 { return "FACILITY_NULL" } 1 { return "FACILITY_RPC" } 2 { return "FACILITY_DISPATCH" } 3 { return "FACILITY_STORAGE" } 4 { return "FACILITY_ITF" } 7 { return "FACILITY_WIN32" } 8 { return "FACILITY_WINDOWS" } 9 { return "FACILITY_SSPI" } 9 { return "FACILITY_SECURITY" } 10 { return "FACILITY_CONTROL" } 11 { return "FACILITY_CERT" } 12 { return "FACILITY_INTERNET" } 13 { return "FACILITY_MEDIASERVER" } 14 { return "FACILITY_MSMQ" } 15 { return "FACILITY_SETUPAPI" } 16 { return "FACILITY_SCARD" } 17 { return "FACILITY_COMPLUS" } 18 { return "FACILITY_AAF" } 19 { return "FACILITY_URT" } 20 { return "FACILITY_ACS" } 21 { return "FACILITY_DPLAY" } 22 { return "FACILITY_UMI" } 23 { return "FACILITY_SXS" } 24 { return "FACILITY_WINDOWS_CE" } 25 { return "FACILITY_HTTP" } 26 { return "FACILITY_USERMODE_COMMONLOG" } 27 { return "FACILITY_WER" } 31 { return "FACILITY_USERMODE_FILTER_MANAGER" } 32 { return "FACILITY_BACKGROUNDCOPY" } 33 { return "FACILITY_WIA" } 33 { return "FACILITY_CONFIGURATION" } 34 { return "FACILITY_STATE_MANAGEMENT" } 35 { return "FACILITY_METADIRECTORY" } 36 { return "FACILITY_WINDOWSUPDATE" } 37 { return "FACILITY_DIRECTORYSERVICE" } 38 { return "FACILITY_GRAPHICS" } 39 { return "FACILITY_SHELL" } 39 { return "FACILITY_NAP" } 40 { return "FACILITY_TPM_SERVICES" } 41 { return "FACILITY_TPM_SOFTWARE" } 42 { return "FACILITY_UI" } 43 { return "FACILITY_XAML" } 44 { return "FACILITY_ACTION_QUEUE" } 48 { return "FACILITY_WINDOWS_SETUP" } 48 { return "FACILITY_PLA" } 49 { return "FACILITY_FVE" } 50 { return "FACILITY_FWP" } 51 { return "FACILITY_WINRM" } 52 { return "FACILITY_NDIS" } 53 { return "FACILITY_USERMODE_HYPERVISOR" } 54 { return "FACILITY_CMI" } 55 { return "FACILITY_USERMODE_VIRTUALIZATION" } 56 { return "FACILITY_USERMODE_VOLMGR" } 57 { return "FACILITY_BCD" } 58 { return "FACILITY_USERMODE_VHD" } 60 { return "FACILITY_SDIAG" } 61 { return "FACILITY_WINPE" } 61 { return "FACILITY_WEBSERVICES" } 62 { return "FACILITY_WPN" } 63 { return "FACILITY_WINDOWS_STORE" } 64 { return "FACILITY_INPUT" } 66 { return "FACILITY_EAP" } 80 { return "FACILITY_WINDOWS_DEFENDER" } 81 { return "FACILITY_OPC" } 82 { return "FACILITY_XPS" } 83 { return "FACILITY_RAS" } 84 { return "FACILITY_POWERSHELL" } 84 { return "FACILITY_MBN" } 85 { return "FACILITY_EAS" } 98 { return "FACILITY_P2P_INT" } 99 { return "FACILITY_P2P" } 100 { return "FACILITY_DAF" } 101 { return "FACILITY_BLUETOOTH_ATT" } 102 { return "FACILITY_AUDIO" } 103 { return "FACILITY_STATEREPOSITORY" } 109 { return "FACILITY_VISUALCPP" } 112 { return "FACILITY_SCRIPT" } 113 { return "FACILITY_PARSE" } 120 { return "FACILITY_BLB" } 121 { return "FACILITY_BLB_CLI" } 122 { return "FACILITY_WSBAPP" } 128 { return "FACILITY_BLBUI" } 129 { return "FACILITY_USN" } 130 { return "FACILITY_USERMODE_VOLSNAP" } 131 { return "FACILITY_TIERING" } 133 { return "FACILITY_WSB_ONLINE" } 134 { return "FACILITY_ONLINE_ID" } 153 { return "FACILITY_DLS" } 160 { return "FACILITY_SOS" } 176 { return "FACILITY_DEBUGGERS" } 208 { return "FACILITY_DELIVERY_OPTIMIZATION" } 231 { return "FACILITY_USERMODE_SPACES" } 232 { return "FACILITY_USER_MODE_SECURITY_CORE" } 234 { return "FACILITY_USERMODE_LICENSING" } 256 { return "FACILITY_SPP" } 256 { return "FACILITY_RESTORE" } 256 { return "FACILITY_DMSERVER" } 257 { return "FACILITY_DEPLOYMENT_SERVICES_SERVER" } 258 { return "FACILITY_DEPLOYMENT_SERVICES_IMAGING" } 259 { return "FACILITY_DEPLOYMENT_SERVICES_MANAGEMENT" } 260 { return "FACILITY_DEPLOYMENT_SERVICES_UTIL" } 261 { return "FACILITY_DEPLOYMENT_SERVICES_BINLSVC" } 263 { return "FACILITY_DEPLOYMENT_SERVICES_PXE" } 264 { return "FACILITY_DEPLOYMENT_SERVICES_TFTP" } 272 { return "FACILITY_DEPLOYMENT_SERVICES_TRANSPORT_MANAGEMENT" } 278 { return "FACILITY_DEPLOYMENT_SERVICES_DRIVER_PROVISIONING" } 289 { return "FACILITY_DEPLOYMENT_SERVICES_MULTICAST_SERVER" } 290 { return "FACILITY_DEPLOYMENT_SERVICES_MULTICAST_CLIENT" } 293 { return "FACILITY_DEPLOYMENT_SERVICES_CONTENT_PROVIDER" } 305 { return "FACILITY_LINGUISTIC_SERVICES" } 885 { return "FACILITY_WEB" } 886 { return "FACILITY_WEB_SOCKET" } 1094 { return "FACILITY_AUDIOSTREAMING" } 1536 { return "FACILITY_ACCELERATOR" } 1793 { return "FACILITY_MOBILE" } 1967 { return "FACILITY_SQLITE" } 1989 { return "FACILITY_UTC" } 1996 { return "FACILITY_WMAAECMA" } 2049 { return "FACILITY_WEP" } 2050 { return "FACILITY_SYNCENGINE" } 2168 { return "FACILITY_DIRECTMUSIC" } 2169 { return "FACILITY_DIRECT3D10" } 2170 { return "FACILITY_DXGI" } 2171 { return "FACILITY_DXGI_DDI" } 2172 { return "FACILITY_DIRECT3D11" } 2173 { return "FACILITY_DIRECT3D11_DEBUG" } 2174 { return "FACILITY_DIRECT3D12" } 2175 { return "FACILITY_DIRECT3D12_DEBUG" } 2184 { return "FACILITY_LEAP" } 2185 { return "FACILITY_AUDCLNT" } 2200 { return "FACILITY_WINCODEC_DWRITE_DWM" } 2201 { return "FACILITY_DIRECT2D" } 2304 { return "FACILITY_DEFRAG" } 2305 { return "FACILITY_USERMODE_SDBUS" } 2306 { return "FACILITY_JSCRIPT" } 2339 { return "FACILITY_XBOX" } 2561 { return "FACILITY_PIDGENX" } } return "FACILITY_UNKNOWN_" + $facilityCode } $hresult = 0x800700C1 Get-HResultFailed($hresult) Get-HResultFacility($hresult) Get-HResultCode($hresult) # will return error text for FACILITY_WIN32 Get-ErrorMessage($hresult)

Posted in PowerShell | Tagged | Leave a comment

Case of the Black Background Window (.NET Patching)

On Windows XP  a Windows form had a white background, something like this:

clip_image002[5]

However on Windows 8 it had a black background like this, making text on the form unreadable:

clip_image004[4]

 

This is a common AppCompat bug…where the developer instead of selecting “White” to make a Windows form background, selected Active Caption Text.

clip_image004[6]

This is a change from the default Control

clip_image002[7]

If the source code is not available/developer can’t patch it, there are a few ways we can fix this. We can change Active Title Bar colour back to White on a Windows 7 machine, save it as a .themepack and apply it to Windows 8 or later.

clip_image001

clip_image003

 

Patching .NET EXE directly:

1) Open EXE in ILDasm and click File –> Dump then select the default settings. ILDasm is found in the Windows SDK, it will be in a folder similar to C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6 Tools\

2) Open the .IL file in a text editor and search replace

SystemColors::get_ActiveCaptionText()

to

SystemColors::get_Control()

 

Typically you will only want to replace those that just precede a set_BackColor

call       valuetype [System.Drawing]System.Drawing.Color [System.Drawing]System.Drawing.SystemColors::get_ActiveCaptionText()
callvirt   instance void [System.Windows.Forms]System.Windows.Forms.Control::set_BackColor(valuetype [System.Drawing]System.Drawing.Color)

 

call       valuetype [System.Drawing]System.Drawing.Color [System.Drawing]System.Drawing.SystemColors::get_Control()
callvirt   instance void [System.Windows.Forms]System.Windows.Forms.Control::set_BackColor(valuetype [System.Drawing]System.Drawing.Color)

 

3) We then recompile the EXE, using ilasm which lives in the .NET framework directories, specifying the IL file and the .RES file output from ILDASM

 

c:\windows\microsoft.net\Framework\v2.0.50727\ilasm WorkflowApplication.il /RESOURCE=WorkflowApplication.res

4) You program should now be fixed!

For more patching examples refer to https://chentiangemalc.wordpress.com/category/patching/

Posted in .NET, AppCompat, IL, Patching | Leave a comment

Case of the Invoked Hang (.NET)

A .NET application was hanging completely when certain background tasks were occurring. Using ProcDump (http://live.sysinternals.com/Procdump.exe ) with option Procdump –ma <process name> i captured a 3 dump files about 10 seconds apart, to identify where hang was occurring.

Opening dmp file in WinDbg with SOS extension:

.loadby sos clr

The UI thread

0:000> kv
# ChildEBP RetAddr  Args to Child             
00 006df114 754d1286 00000001 00a3cc10 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
01 006df2a8 6913b957 00000001 00a3cc10 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x136 (FPO: [SEH])
02 006df2f8 6913b62c 00000000 ffffffff 00000001 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c (FPO: [Non-Fpo])
03 006df384 6913b71d 00000001 00a3cc10 00000000 clr!Thread::DoAppropriateWaitWorker+0x237 (FPO: [5,25,0])
04 006df3f0 6913b8fd 00000001 00a3cc10 00000000 clr!Thread::DoAppropriateWait+0x64 (FPO: [Non-Fpo])
05 006df43c 6913b0d2 ffffffff 00000001 00000000 clr!CLREventBase::WaitEx+0x128 (FPO: [Non-Fpo])
06 006df450 6910134d ffffffff 00000001 00000000 clr!CLREventBase::Wait+0x1a (FPO: [3,0,0])
07 006df4b4 69241bb8 00010000 006df4d8 352bbf87 clr!CLREventWaitWithTry+0x42 (FPO: [Non-Fpo])
08 006df4ec 69125dde 352bbe77 027f9cd8 00000000 clr!ThreadStore::WaitForOtherThreads+0x74 (FPO: [Non-Fpo])
09 006df51c 691247b7 352bbe43 00aecab8 00aa77d0 clr!RunMain+0x26d (FPO: [Non-Fpo])
0a 006df788 69124dbb 00000000 352bbcf3 00540000 clr!Assembly::ExecuteMainMethod+0x144 (FPO: [1,149,0])
0b 006dfc90 69124e61 352bb783 00000000 00000000 clr!SystemDomain::ExecuteMainMethod+0x651 (FPO: [0,315,0])
0c 006dfce8 69124662 352bb643 00000000 00000000 clr!ExecuteEXE+0x4c (FPO: [Non-Fpo])
0d 006dfd28 690e4e91 352bb60f 00000000 00000000 clr!_CorExeMainInternal+0xdc (FPO: [Non-Fpo])
0e 006dfd64 6a10cd87 334078bd 6a18dc60 6a10cd16 clr!_CorExeMain+0x4d (FPO: [Non-Fpo])
0f 006dfda0 6a18dd05 6a18dc60 639b32df 006dfdc4 mscoreei!_CorExeMain+0x10a (FPO: [0,10,4])
10 006dfdb0 74d33744 7ec9f000 74d33720 12f4615d mscoree!_CorExeMain_Exported+0xa5 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
11 006dfdc4 779ca064 7ec9f000 ad1ace76 00000000 kernel32!BaseThreadInitThunk+0x24
12 006dfe0c 779ca02f ffffffff 779ed7de 00000000 ntdll!__RtlUserThreadStart+0x2f (FPO: [Non-Fpo])
13 006dfe1c 00000000 6a18dc60 7ec9f000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

I took a quick look at the .NET stack traces

!eestack

….

 

Thread  21
Current frame: ntdll!NtWaitForSingleObject+0xc
ChildEBP RetAddr  Caller, Callee
0099e3a0 754c8d09 KERNELBASE!WaitForSingleObjectEx+0x99, calling ntdll!NtWaitForSingleObject
0099e414 68fc1bb7 clr!CLREventBase::Reset+0x145
0099e444 68fc1bfe clr!CLREventBase::Reset+0x18d, calling clr!CLREventBase::Reset+0x114
0099e494 68fc1b81 clr!CLREventBase::WaitEx+0x152, calling clr!CLREventBase::Reset+0x163
0099e4b8 756fcca6 combase!WindowsCreateString+0x6886, calling msvcrt!memcmp
0099e4cc 6913b0d2 clr!CLREventBase::Wait+0x1a, calling clr!CLREventBase::WaitEx
0099e4e4 69244395 clr!Thread::WaitSuspendEventsHelper+0x8a, calling clr!CLREventBase::Wait
0099e524 690533c5 clr!SafeReleasePreemp+0x234
0099e550 69244468 clr!Thread::WaitSuspendEvents+0x14, calling clr!Thread::WaitSuspendEventsHelper
0099e564 692427cc clr!Thread::RareEnablePreemptiveGC+0x8e, calling clr!Thread::WaitSuspendEvents
0099e57c 69137d3b clr!Thread::RareDisablePreemptiveGC+0x102, calling clr!Thread::RareEnablePreemptiveGC
0099e5cc 6913b694 clr!Thread::DoAppropriateWaitWorker+0x3cb, calling clr!GCCoopHackNoThread::~GCCoopHackNoThread
0099e648 6913b71d clr!Thread::DoAppropriateWait+0x64, calling clr!Thread::DoAppropriateWaitWorker
0099e69c 690bb01e clr!AcquireSafeHandle+0x33, calling clr!_EH_epilog3
0099e6b4 690bb130 clr!WaitHandleNative::CorWaitOneNative+0x163, calling clr!Thread::DoAppropriateWait
0099e72c 690bb06d clr!WaitHandleNative::CorWaitOneNative+0x4c, calling clr!LazyMachStateCaptureState
0099e750 68fb278a clr!HelperMethodFrame::Push+0x10, calling clr!GetThread
0099e758 69068141 clr!AppDomainNative::IsFinalizingForUnload+0x3a, calling clr!_EH_epilog3
0099e7bc 680cc7c1 (MethodDesc 67e4df10 +0x21 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)), calling clr!WaitHandleNative::CorWaitOneNative
0099e7d0 680cc788 (MethodDesc 67e4dedc +0x28 System.Threading.WaitHandle.WaitOne(Int32, Boolean)), calling (MethodDesc 67e4df10 +0 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean))
0099e7ec 0085bb9d (MethodDesc 04feadb8 +0x1a5 System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle))
0099e7f4 68fb21a6 clr!JIT_MonExitWorker+0xa, calling clr!GetThread
0099e830 0949b3e3 (MethodDesc 04feb6f4 +0x2cb System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control, System.Delegate, System.Object[], Boolean)), calling 0949a078
0099e8ac 0085b5fb (MethodDesc 04feb648 +0x4b System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])), calling 0949a038
0099e8e8 0085b598 (MethodDesc 04feb63c +0x8 System.Windows.Forms.Control.Invoke(System.Delegate)), calling (MethodDesc 04feb648 +0 System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[]))
0099e8f0 0085b3af (MethodDesc 059f56c8 +0x1c7 Main.Form1.bulkWorker_DoWork(System.Object, System.ComponentModel.DoWorkEventArgs)), calling 0085ae0c

 

This is a common cause of .NET application hang – .Invoke called from background thread. This is easily fixed by replaced .Invoke with .BeginInvoke

 

What if there is no source code, vendor is slow to fix, etc? You can patch it yourself, at your own risk…

Use ILDasm ( https://chentiangemalc.wordpress.com/2014/06/30/upgrade-exe-net-framework-with-ildasm-ilasm-and-hex-editor/ ) to dump the contents to IL format

Do a search replace from

callvirt   instance object [System.Windows.Forms]System.Windows.Forms.Control::Invoke

to

callvirt   instance class [mscorlib]System.IAsyncResult [System.Windows.Forms]System.Windows.Forms.Control::BeginInvoke

We then recompile:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\ilasm.exe Program.il /RESOURCE=Program.res

  And the hangs are gone…

What is the difference between Invoke & BeginInvoke:

  • Invoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.
  • BeginInvoke: Executes the specified delegate on the thread that owns the control’s underlying window handle.

Some of the problems with using invoke:

  • Causes thread to wait on another thread, increases risk of deadlock in your application. Using ThreadPools may help, but under high load in “threadpool starvation” the deadlock may be triggered.
  • Invoke forces thread switch immediately that has potential to decrease application stability
  • Each Invoke requires a separate thread switch, potentially impacting application performance

Some advantage of BeginInvoke:

  • Multiple BeginInvoke can be executed in a single thread switch ‘
  • Schedules execution of delegate on other thread – thread switch occurs “when OS is ready”

However a word of caution

Invoke is required when synchronous execution is expected. If code relying on changes made by BeginInvoke immediately follows in a separate code block, you may have another issue.

i.e.

SomeControl.BeginInvoke(new Action(delegate {

  SomeControl.Text=”some value”

}));

// trying to read SomeControl.Text here is not safe, may not get expected result – move code reyling on BeginInvoke’s result into BeginInvoke or use Invoke

Posted in .NET, Debugging, WinDbg, IL | Tagged | Leave a comment