Case of the Unwanted Chrome Search Modification

A friend’s machine had the Google Chrome home page set to


But there is no homepage set in settings.


Checking the Chrome Shortcut in Start Menu we can see it’s been modified, so we remove the piesearch parameter. This setting needs Administrative privilege to change because Chrome was installed to “All Users”


That fixed the homepage, but now searching in the omnibox resulted in searches from NOT google which had been set before:


When settings were opened it was advised this was set by the administrator, and the default could not be changed.

I did a ProcMon trace and used filter Details Contains coldsearch to identify registry keys with the value coldsearch but got no hits.

So I changed ProcMon filter to

  • Process Name is Chrome.exe
  • Result is Success

Then using ProcMon –> Tools | Summary could quickly identify the folders Chrome accessed files from:


We could see app settings loading from C:\Users\<Username>\AppData\Local\Google\Chrome

However renaming this folder didn’t remove the search setting.

To find where the setting might be stored I used SearchMyFiles )

And limited the search to those folders found with ProcMon to speed it up:


This machine was not supposed to have any Group Policy settings, so I removed the Registry.pol file, which represents Local Machine Group Policy.

Using the tool here we could view the .pol file (Requires .NET 3.5, Free registration required to download)


With the Registry.Pol file removed, Chrome was back to Normal.

Finally we cleaned up the Mozilla Firefox directory settings by backing up, then deleting the files under C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles

Microsoft Edge had not been affected, and Internet Explorer settings modification had been blocked by Windows Defender.

Posted in Group Policy, Internet Explorer, ProcMon, Sys, SysInternals | Leave a comment

Case of the Domain Join Failure Part III

Continuing the series on domain join failures from

A Windows 10 device had been joined to domain; removed from domain. The computer account in AD had been deleted and recreated. However attempting to manually join domain failed.

The most useful error message was achieved by using the legacy Control Panel’s System Properties  Network ID… button to join the computer to the domain. This provided more detailed error then the new Windows 10 Rename your PC or join a domain in System Settings.


In attempting to join domain domain we got an error referencing the old NetBIOS domain name MYDOMAIN. This was no longer accessible as WINS servers had been decommissioned. Note: During the domain join process we made no reference to the old NetBIOS name

The domain name “MYDOMAIN” might be a NetBIOS domain name.  If this is the case, verify that the domain name is properly registered with WINS.

If you are certain that the name is not a NetBIOS domain name, then the following information can help you troubleshoot your DNS configuration.

The following error occurred when DNS was queried for the service location (SRV) resource record used to locate an Active Directory Domain Controller (AD DC) for domain “MYDOMAIN”:

The error was: “DNS name does not exist.”
(error code 0x0000232B RCODE_NAME_ERROR)

The query was for the SRV record for _ldap._tcp.dc._msdcs.MYDOMAIN

Common causes of this error include the following:

– The DNS SRV records required to locate a AD DC for the domain are not registered in DNS. These records are registered with a DNS server automatically when a AD DC is added to a domain. They are updated by the AD DC at set intervals. This computer is configured to use DNS servers with the following IP addresses:

– One or more of the following zones do not include delegation to its child zone:

. (the root zone)

Unfortunately attempting a packet capture using netsh trace start capture=yes report=yes failed to capture any network traffic, so I installed WireShark and by filtering on dns found the following SRV records were requested:

All were successful except those containing MYDOMAIN

Why was Windows attempting to use the NetBIOS name? I have no idea at this point, However we did join the domain successfully using Windows 10 in-built offline domain join command djoin

From an elevated cmd prompt I ran the following:

runas /\chentiangemalc /netonly cmd.exe

And provided the password as required. This allowed us to launch a cmd line with domain credentials for network purposes, without having to be joined to the domain.

To provision the machine onto the domain:


(Using /REUSE because the computer account had been precreated)

Finally the command to actually join the machine to the domainL


After a reboot the djoin command had been successful in joining the machine to the domain…when all the Windows in-built GUI options had failed. Yet another reason to be familiar with the CLI…

Posted in Active Directory, Windows 10 | Tagged | Leave a comment

Case of the Network Name That Couldn’t Be Found

Continuing the series on .NET patching from

An application when moved from XP to Windows 8.1 started reporting error The network name cannot be found.


A dump file was taken at this point with procdump –ma option ( )

Because this is 32-bit .NET 2.0 process, we load in x86 WinDbg and load .NET 2.0 SOS extension:


0:000> .load C:\Windows\Microsoft.NET\Framework\v2.0.50727\SOS.dll
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.5485 f:0
doesn’t match desired version 2.0.50727.8009 f:0
CLRDLL: Unable to find ” on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL g:\symbols\mscordacwks_x86_x86_2.0.50727.8009.dll\53A11D015b0000\mscordacwks_x86_x86_2.0.50727.8009.dll
CLR DLL status: Loaded DLL g:\symbols\mscordacwks_x86_x86_2.0.50727.8009.dll\53A11D015b0000\mscordacwks_x86_x86_2.0.50727.8009.dll
0:000> !pe
Exception object: 02651d00
Exception type: System.ComponentModel.Win32Exception
Message: The network name cannot be found
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    0038EB74 6143C043 System_ni!System.Diagnostics.Process.StartWithShellExecuteEx(System.Diagnostics.ProcessStartInfo)+0x267
    0038EBA4 6143B1DD System_ni!System.Diagnostics.Process.Start()+0x39
    0038EBB4 6143AEAA System_ni!System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo)+0x32
    0038EBC4 6143AF1C System_ni!System.Diagnostics.Process.Start(System.String)+0x20
    0038EBCC 05853E67 wds!StreamlineUI.MainForm.DisplayImage(System.String)+0x7f

StackTraceString: <none>
HResult: 80004005

From dumping stack objects, it seems like the file path has too many double back slashes

0:000> !dso
OS Thread Id: 0x1e3c (0)
ESP/REG  Object   Name
0038ea70 02291198 System.String   
0038ea74 02651fbc System.String    The network name cannot be found
0038ea88 025add10 System.Windows.Forms.ControlBindingsCollection
0038eadc 02651fbc System.String    The network name cannot be found
0038eb20 025add10 System.Windows.Forms.ControlBindingsCollection
0038eb24 02651fbc System.String    The network name cannot be found
0038eb28 025fceb0 System.String    \\server01\\Images\\WDS\\FaxImages\\2015\09\17\104510_Fax_FILE01_0309.tif
0038eb48 02291198 System.String   
0038eb4c 02293be0 StreamlineUI.MainForm
0038eb8c 025add10 System.Windows.Forms.ControlBindingsCollection
0038eb98 025fd174 System.Diagnostics.ProcessStartInfo
0038ebb4 025fceb0 System.String    \\server01\\Images\\WDS\\FaxImages\\2015\09\17\104510_Fax_FILE01_0309.tif
0038ebb8 025fceb0 System.String    \\server01\\Images\\WDS\\FaxImages\\2015\09\17\104510_Fax_FILE01_0309.tif
0038ebcc 025fceb0 System.String    \\server01\\Images\\WDS\\FaxImages\\2015\09\17\104510_Fax_FILE01_0309.tif
0038ebd0 02293be0 StreamlineUI.MainForm

Further investigation identified the fact that these filenames came from a database, and had already been stored in this format. In Windows XP this faulty file paths didn’t cause a problem, the images still opened OK in Internet Explorer which had been configured to be default viewer for TIF files.

Without write access to the DB, and software developer unable to make a patch I worked in a simple patch. This time instead of developing the patch in MSIL I created a C# Class Library (DLL) with the following code:


using System; using System.Collections.Generic; using System.Text; using System.Text.RegularExpressions; namespace DotNetPatches { public static class FilePaths { public static string Fix(string path) { try { return Regex.Replace(path, @"(?!^)\\\\", @"\"); } catch { return path; } } } }

Now we decompile our broken EXE with ILDASM and open the IL file for editing. At beginning of .IL file we add the reference to our DLL

.assembly extern DotNetPatches { .ver 1:0:0:0 }

Using .NET reflector we had identified get_IMAGEPATH as the method responsible for retrieving our faulty file path, so we patch it to fix the value it returns by inserting a call to our Fix function.

.method public hidebysig specialname instance string get_IMAGEPATH() cil managed { // Code size 12 (0xc) .maxstack 8 IL_0000: ldarg.0 IL_0001: ldfld class [System.Windows.Forms]System.Windows.Forms.TextBox StreamlineUI.MainForm::txtImageURI IL_0006: callvirt instance string [System.Windows.Forms]System.Windows.Forms.Control::get_Text() call string [DotNetPatches]DotNetPatches.FilePaths::Fix(string) IL_000B: ret } // end of method MainForm::get_IMAGEPATH

After recompiling with ilasm this patched application now works fine in Windows 8.1

Posted in .NET, AppCompat, Patching | Tagged | Leave a comment

Case of the Win10 Windows Performance Recorder Trace That Wouldn’t Stop

Disk space kept rapidly disappearing on my C: drive…

Using WinDirStat ( it was evident majority was taken up by 40GB+ ETL files in my TEMP folder.

Running wpr –status showed no traces running


Searching registry with RegScanner ( ) for the filename of the filename WPR_initiated_WprApp_WPR Event Collector.etl we found the culprits under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\AutoLogger under keys

  • WPR_initiated_WprApp_WPR Event Collector
  • WPR_initiated_WprApp_WPR System Collector

Checking the GUID subkeys and looking at provider name we could see it was monitoring the following providers:

  • Microsoft-Antimalware-Engine
  • Microsoft-Windows-DNS-Client
  • Microsoft-Windows-URLMon
  • Microsoft-Windows-Shell-Core
  • Microsoft-Windows-NCSI
  • Microsoft-Windows-Kernel-Power
  • Microsoft-Windows-WinINet
  • Microsoft-JScript
  • Microsoft-IEFRAME
  • Microsoft-Windows-BootUX
  • Microsoft-Antimalware-Service
  • IE7
  • Microsoft-Windows-Win32k
  • Microsoft-Antimalware-RTP
  • Microsoft-Windows-WLAN-AutoConfig
  • Microsoft-IE
  • Microsoft-Windows-PDC
  • Microsoft-Windows-Kernel-EventTracing
  • Microsoft-Antimalware-AMFilter
  • Microsoft-Windows-ProcessStateManager
  • Microsoft-Windows-DotNETRuntime
  • Microsoft-Antimalware-Protection
  • Microsoft-Windows-BrokerInfrastructure

Best of all max file size was set to 0 (unlimited):


Not sure how this trace got started, but removing the keys stopped it taking up all my disk space…

Posted in Debugging, Windows 10, Windows Performance Recorder | Tagged | 1 Comment

Debugging / Viewing Windows Update Log on Windows 10 Insider Builds

Having confirmed with WinDbg team currently insider builds don’t get public symbols unless it is a major release (They are looking into a solution….)

How are we going to debug anything on bleeding edge systems “Fast Track Updates” in the meantime?

To workaround this problem I did the following

1) Downloaded the latest Windows 10 symbol package publicly available

I installed them into a folder C:\win10symbols

2) Download ChkMatch tool from here

Note it is 32-bit only tool, so when specifying path to 64-bit files in C:\windows\system32 you must use path C:\windows\sysnative

3) Delete any files in <debugger directory>\sym folder (i.e. C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\sym )

Here is the WinDbg output:

Before ChkMatch

0:000> .reload
Reloading current modules
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll –

************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  PDB not found : c:\win10symbols\symbols\dll\ntdll.pdb

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:000> !peb
PEB at 00007ff6759af000
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn’t have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing “.symopt- 100”. Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: ntdll!_PEB                                    ***
***                                                                   ***
error 3 InitTypeRead( nt!_PEB at 00007ff6759af000)…

I then ran the following command:

C:\>chkmatch -m C:\windows\sysnative\ntdll.dll c:\win10symbols\ntdll.pdb\B9A5AE37F693422C89FBF6471E0FC5F51\ntdll.pdb
ChkMatch – version 1.0
Copyright (C) 2004 Oleg Starodumov

Executable: C:\windows\sysnative\ntdll.dll
Debug info file: c:\win10symbols\ntdll.pdb\B9A5AE37F693422C89FBF6471E0FC5F51\ntdll.pdb

TimeDateStamp: 55cbff9a
Debug info: 2 ( CodeView )
TimeStamp: 55cbff9a  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 34  RVA: 0011ebe8  FileOffset: 0011cbe8
CodeView format: RSDS
Signature: {7edb4a59-efff-47ec-b12f-7697e9226177}  Age: 1
PdbFile: ntdll.pdb
Debug info: 13 ( Unknown )
TimeStamp: 55cbff9a  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 1084  RVA: 0011ec20  FileOffset: 0011cc20

Debug information file:
Format: PDB 7.00
Signature: {b9a5ae37-f693-422c-89fb-f6471e0fc5f5}  Age: 2

Writing to the debug information file…
Result: Success.

0:000> .reload
Reloading current modules
No .natvis files found at C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers.
0:000> !peb
PEB at 00007ff6759af000
    InheritedAddressSpace:    No
    ReadImageFileExecOptions: No
    BeingDebugged:            Yes
    ImageBaseAddress:         00007ff676040000
    Ldr                       00007fffdf8b71e0
    Ldr.Initialized:          Yes




Note: This solution only works if the PDB age signature matches the current DLL, more info about this here:

Windows Update Log using PowerShell Get-WindowsUpdateLog just returning GUIDs?

1601/01/01 11:00:00.0000000 980   8592                  Unknown( 14): GUID=464bbb57-d1b3-22d8-23ef-338952442d14 (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 10): GUID=524890e0-a6db-a018-bdf5-146836fbf25d (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 11): GUID=6388909c-9384-8454-6d45-b6165f613f9a (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 11): GUID=524890e0-a6db-a018-bdf5-146836fbf25d (No Format Information found).
1601/01/01 11:00:00.0000000 980   8592                  Unknown( 23): GUID=ccdda195-7c91-4637-05e0-79cdf332fe13 (No Format Information found).

Using Fiddler I quickly identified the PDB files it wanted:



Copying the URL column into Excel I did a quick & dirty processing. Filtered on items ending with .PDB and copied the result into a new sheet, then added formula to extract just the filename



Then pivot table on that column we find the PDB files this process relies on:



This resulted in running the following commands, all of which reported success.

chkmatch -m c:\Windows\WinSxS\amd64_microsoft-windows-s..e-windowsupdateauth_31bf3856ad364e35_10.0.10525.0_none_8b70d1ffa911692a\storewuauth.dll c:\win10symbols\storewuauth.pdb\5A9142A09775486CB3053E2D6D6A7B6F1\storewuauth.pdb chkmatch -m c:\windows\sysnative\wuapi.dll C:\win10symbols\wuapi.pdb\03E47413D978470EB9438AD21D5B208E1\wuapi.pdb chkmatch -m c:\windows\sysnative\wuauclt.exe c:\win10symbols\wuauclt.pdb\E0D0C18A15D24973B0AE8799EFFFC1001\wuauclt.pdb chkmatch -m c:\windows\sysnative\wuaueng.dll c:\win10symbols\wuaueng.pdb\EBADD49A0078487A8BD8DBFE7D56D88B1\wuaueng.pdb chkmatch -m c:\windows\sysnative\wuautoappupdate.dll c:\win10symbols\wuautoappupdate.pdb\2E4D201B897D4FACB97B5198FC9B2CBD1\wuautoappupdate.pdb chkmatch -m c:\windows\sysnative\wuuhext.dll c:\win10symbols\wuuhext.pdb\C94A76FA1A6546D38EBCAAE1B7C0A39C1\wuuhext.pdb

We then run PowerShell cmd

Get-WindowsUpdateLog –SymbolServer C:\win10symbols

However unlike WinDbg, this is not working, as the paths don’t match


So I made a copy of each folder, the copy with the same name that tracerpt.exe was looking for


WOW! I CAN NOW READ THE WINDOWS UPDATE LOG!!!!!!!!!!!!!!!!!!!!!!!!!!!! WOO-HOO

That was easy.

2015/09/01 10:40:01.8948051 996   9112  SLS             Retrieving SLS response from server using ETAG Ww/uzEDtiHGuCQKdqqiVoucsE3BysNngQpDk+NfngSU=_1440″…”
2015/09/01 10:40:01.8949751 996   9112  SLS             Making request with URL
2015/09/01 10:40:22.9594201 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:40:22.9594266 996   9112  Misc            WinHttp: SendRequestToServerForFileInformation failed with 0x80072ee2; retrying with default proxy.
2015/09/01 10:40:43.9686947 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:40:43.9687321 996   9112  SLS             Failed: hr = 0x80072EE2
2015/09/01 10:40:43.9690170 996   9112  SLS             GetResponse failed with hresult 0x80072ee2…
2015/09/01 10:40:43.9690306 996   9112  Agent           Failed to retrieve SLS response data for service 7971f918-a847-4430-9279-4a52d1efe18d, error = 0x80072ee2
2015/09/01 10:40:43.9884897 996   9112  SLS             Retrieving SLS response from server using ETAG Ww/uzEDtiHGuCQKdqqiVoucsE3BysNngQpDk+NfngSU=_1440″…”
2015/09/01 10:40:43.9887935 996   9112  SLS             Making request with URL
2015/09/01 10:41:05.0047561 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:41:05.0047606 996   9112  Misc            WinHttp: SendRequestToServerForFileInformation failed with 0x80072ee2; retrying with default proxy.
2015/09/01 10:41:26.0152702 996   9112  Misc            Send request failed, hr:0x80072ee2
2015/09/01 10:41:26.0152821 996   9112  SLS             Failed: hr = 0x80072EE2
2015/09/01 10:41:26.0157665 996   9112  SLS             GetResponse failed with hresult 0x80072ee2…
2015/09/01 10:41:26.0157936 996   9112  Misc            EP: error: 0x80072EE2 : – failed to get SLS data

Posted in Fiddler, ProcMon, WinDbg, Windows 10 | Tagged | 1 Comment

Case of the Object Is Not Set To An Instance of an Object–.NET Patching

Continuing our series on patching .NET code without source ( )

A .NET application which worked fine on Windows 7, started throwing an exception when opening an image in Windows 10’s default editor. However the images still opened OK. Changing the default image viewer to a 3rd party program also fixed the issue.

Object reference not set to an instance of an object


This exception means a method is trying to be called on a null object.

To investigate I asked an engineer to take a dmp file while the error message was displayed with ProcDump

ProcDump -ma <process name>

Because it was a 32-bit .NET application I loaded the dmp in WinDbg x86

0:000> .load C:\windows\\Framework\v2.0.50727\SOS.dll
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.8662 f:0
doesn’t match desired version 2.0.50727.8000 f:0
CLRDLL: Unable to find ” on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL c:\localsymbols\mscordacwks_x86_x86_2.0.50727.8000.dll\526720FC5b0000\mscordacwks_x86_x86_2.0.50727.8000.dll
CLR DLL status: Loaded DLL c:\localsymbols\mscordacwks_x86_x86_2.0.50727.8000.dll\526720FC5b0000\mscordacwks_x86_x86_2.0.50727.8000.dll
0:000> !pe
Exception object: 030ca204
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    00EAE610 0A9E39F9 wds!StreamlineUI.MainForm.DisplayImage(System.String)+0x91

StackTraceString: <none>
HResult: 80004003
0:000> !clrstack
OS Thread Id: 0xfe4 (0)
ESP       EIP    
00ead69c 74b5bfbc [NDirectMethodFrameStandalone: 00ead69c] System.Windows.Forms.SafeNativeMethods.MessageBox(System.Runtime.InteropServices.HandleRef, System.String, System.String, Int32)
00ead6b8 65a61dc8 System.Windows.Forms.MessageBox.ShowCore(System.Windows.Forms.IWin32Window, System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean)
00ead758 65a619de System.Windows.Forms.MessageBox.Show(System.String)
00ead760 0a9e3a68 StreamlineUI.MainForm.DisplayImage(System.String)
00eae640 0a19b912 StreamlineUI.MainForm.doDataBind()
00eae668 0a19a542 StreamlineUI.MainForm.getNextForm(ACTION, DIRECTION, System.String)
00eae690 0a199fae StreamlineUI.MainForm.CheckIn(DIRECTION, System.String)
00eae6a8 0a199cfe StreamlineUI.MainForm.btnNext_Click(System.Object, System.EventArgs)
00eae6e0 65435170 System.Windows.Forms.Control.OnClick(System.EventArgs)
00eae6f8 6543055a System.Windows.Forms.Button.OnClick(System.EventArgs)
00eae708 65ccac9a System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
00eae714 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eae71c 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eae730 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eae930 011207bc [NDirectMethodFrameStandalone: 00eae930] System.Windows.Forms.UnsafeNativeMethods.SendMessage(System.Runtime.InteropServices.HandleRef, Int32, IntPtr, IntPtr)
00eae94c 65474c34 System.Windows.Forms.Control.SendMessage(Int32, IntPtr, IntPtr)
00eae968 6547453b System.Windows.Forms.Control.ReflectMessageInternal(IntPtr, System.Windows.Forms.Message ByRef)
00eae978 6546497e System.Windows.Forms.Control.WmCommand(System.Windows.Forms.Message ByRef)
00eae988 65469d9c System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
00eae98c 65464fe5 [InlinedCallFrame: 00eae98c]
00eae9ec 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eae9f4 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eaea08 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eaedd8 011207bc [InlinedCallFrame: 00eaedd8] System.Windows.Forms.UnsafeNativeMethods.CallWindowProc(IntPtr, IntPtr, Int32, IntPtr, IntPtr)
00eaedd4 6546a033 System.Windows.Forms.NativeWindow.DefWndProc(System.Windows.Forms.Message ByRef)
00eaee18 65469f8c System.Windows.Forms.Control.DefWndProc(System.Windows.Forms.Message ByRef)
00eaee1c 659994e9 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
00eaeea8 65ccc3a6 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
00eaeeac 654638ac [InlinedCallFrame: 00eaeeac]
00eaef44 654637f0 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
00eaef50 65469a00 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
00eaef58 65469981 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
00eaef6c 6546985a System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
00eaf134 011207bc [NDirectMethodFrameStandalone: 00eaf134] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
00eaf144 65479f8e System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
00eaf1e0 65479bf7 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
00eaf234 65479a41 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
00eaf264 65436911 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
00eaf278 050100ab StreamlineUI.Startup.Main()
00eaf4c8 67101b5c [GCFrame: 00eaf4c8]

CLRStack however doesn’t give us a really good indication of where our exception occurred, for this we will get the more verbose stack output:

0:000> !dumpstack

long long stack trace, then finally …

00eae610 0a9e39f8 (MethodDesc 0x10d610c +0x90 StreamlineUI.MainForm.DisplayImage(System.String)) ====> Exception Code c0000005 cxr@eae1ac exr@eae15c
00eae510 75587890 shell32!CShellExecute::Release+0x30, calling shell32!operator delete
00eae518 76e02c1b KERNELBASE!LocalFree+0x27, calling ntdll!RtlFreeHeap
00eae524 76e02c34 KERNELBASE!LocalFree+0x40, calling KERNELBASE!_SEH_epilog4
00eae55c 76e02c34 KERNELBASE!LocalFree+0x40, calling KERNELBASE!_SEH_epilog4
00eae560 010cb550 010cb550, calling KERNELBASE!GetLastError
00eae57c 6683419e (MethodDesc 0x6664a350 +0x2e System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr)), calling 66782c9c
00eae59c 6683419e (MethodDesc 0x6664a350 +0x2e System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr)), calling 66782c9c
00eae5ac 663db45d (MethodDesc 0x65e68bd0 +0x291 System.Diagnostics.Process.StartWithShellExecuteEx(System.Diagnostics.ProcessStartInfo)), calling (MethodDesc 0x6664a350 +0 System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr))
00eae5e0 663da5cd (MethodDesc 0x65e68b8c +0x39 System.Diagnostics.Process.Start()), calling (MethodDesc 0x65e68bd0 +0 System.Diagnostics.Process.StartWithShellExecuteEx(System.Diagnostics.ProcessStartInfo))
00eae5f0 663da29a (MethodDesc 0x65e68c0c +0x32 System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo)), calling 65f49dc0
00eae600 663da30c (MethodDesc 0x65e68bf4 +0x20 System.Diagnostics.Process.Start(System.String)), calling (MethodDesc 0x65e68c0c +0 System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo))
00eae608 0a9e39f2 (MethodDesc 0x10d610c +0x8a StreamlineUI.MainForm.DisplayImage(System.String)), calling mscorwks!JIT_WriteBarrierEAX
00eae638 0a19b912 (MethodDesc 0x10d6118 +0x262 StreamlineUI.MainForm.doDataBind()), calling 0570a8b8

even more long stack trace continues…

From this we find Process.Start is the last code called before the MessageBox showing the error. Now let’s decompile the function that threw the exception:

0:000> !name2ee wds!StreamlineUI.MainForm.DisplayImage
Module: 010d2c5c (WDS.exe)
Token: 0x060000d3
MethodDesc: 010d610c
Name: StreamlineUI.MainForm.DisplayImage(System.String)
JITTED Code Address: 0a9e3968
0:000> !dumpil 010d610c
ilAddr = 00ceef7c
  IL_0000: ldarg.1
  IL_0001: call System.IO.File::Exists
  IL_0006: brtrue.s IL_003d
  IL_0008: ldstr “Unable to locate image file.”
  IL_000d: call System.Environment::get_NewLine
  IL_0012: call System.Environment::get_NewLine
  IL_0017: ldarg.1
  IL_0018: call System.String::Concat
  IL_001d: ldstr “Info”
  IL_0022: call System.Windows.Forms.MessageBox::Show
  IL_0027: pop
  IL_0028: ldarg.0
  IL_0029: ldc.i4.1
  IL_002a: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_002f: ldarg.0
  IL_0030: ldc.i4.6
  IL_0031: call StreamlineUI.MainForm::set_APPMODE
  IL_0036: ldc.i4.0
  IL_0037: stloc.1
  IL_0038: leave IL_00c2
  IL_003d: ldarg.0
  IL_003e: call StreamlineUI.MainForm::KillImageProcess
  IL_0043: pop
  IL_0044: ldarg.0
  IL_0045: ldarg.1
  IL_0046: call System.Diagnostics.Process::Start
  IL_004b: stfld StreamlineUI.MainForm::imageViewProcess
  IL_0050: ldarg.0
  IL_0051: ldfld StreamlineUI.MainForm::imageViewProcess

We can infer the call to WaitForInputIdle this is the culprit. It is calling a method on the return value from Process Start

  IL_0056: callvirt System.Diagnostics.Process::WaitForInputIdle

  IL_005b: pop
  IL_005c: ldarg.0
  IL_005d: call StreamlineUI.MainForm::get_APPMODE
  IL_0062: ldc.i4.2
  IL_0063: bne.un.s IL_0093
  IL_0065: ldarg.0
  IL_0066: ldfld StreamlineUI.MainForm::grdTransactions
  IL_006b: callvirt Infragistics.Win.UltraWinGrid.UltraGridBase::get_Rows
  IL_0070: callvirt Infragistics.Shared.DisposableObjectCollectionBas::get_Count
  IL_0075: brtrue.s IL_0093
  IL_0077: ldarg.0
  IL_0078: ldfld StreamlineUI.MainForm::grdTransactions
  IL_007d: callvirt Infragistics.Win.UltraWinGrid.UltraGridBase::get_DisplayLayout
  IL_0082: callvirt Infragistics.Win.UltraWinGrid.UltraGridLayout::get_Bands
  IL_0087: ldc.i4.0
  IL_0088: callvirt Infragistics.Win.UltraWinGrid.BandsCollection::get_Item
  IL_008d: callvirt Infragistics.Win.UltraWinGrid.UltraGridBand::AddNew
  IL_0092: pop
  IL_0093: ldarg.0
  IL_0094: call StreamlineUI.MainForm::EnterEditMode
  IL_0099: ldarg.0
  IL_009a: ldc.i4.0
  IL_009b: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_00a0: leave.s IL_00ba
} // end .try
  IL_00a2: stloc.0
  IL_00a3: ldloc.0
  IL_00a4: callvirt System.Exception::get_Message
  IL_00a9: call System.Windows.Forms.MessageBox::Show <- displays the error message
  IL_00ae: pop
  IL_00af: ldarg.0
  IL_00b0: ldc.i4.1
  IL_00b1: stfld StreamlineUI.MainForm::IMAGEMISSING
  IL_00b6: ldc.i4.0
  IL_00b7: stloc.1
  IL_00b8: leave.s IL_00c2
} // end .catch
IL_00ba: ldarg.0
IL_00bb: call System.Windows.Forms.Form::Activate
IL_00c0: ldc.i4.1
IL_00c1: ret
IL_00c2: ldloc.1
IL_00c3: ret

If you are uncomfortable with IL we can save the module then open with .NET Reflector:

0:000> lmvm wds
Browse full module list
start    end        module name
00ce0000 00d26000   wds      C (no symbols)          
    Loaded symbol image file: wds.exe
    Image path: C:\Users\user\WDSv5.4\UAT\wds.exe
    Image name: wds.exe
    Browse all global symbols  functions  data
    Has CLR image header, track-debug-data flag not set
    Timestamp:        Thu May 08 14:10:04 2014 (536B039C)
    CheckSum:         00000000
    ImageSize:        00046000
    File version:
    Product version:
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04b0
    CompanyName:      IT Dept
    ProductName:      Work Distribution System
    InternalName:     WDS.exe
    OriginalFilename: WDS.exe
    FileDescription:  WDS Application
    LegalCopyright:   2005, IT Dept
0:000> !savemodule 00ce0000 c:\support\wds.exe
3 sections in file
section 0 – VA=2000, VASize=3f064, FileAddr=1000, FileSize=40000
section 1 – VA=42000, VASize=730, FileAddr=41000, FileSize=1000
section 2 – VA=44000, VASize=c, FileAddr=42000, FileSize=1000

The section of code at fault decompiled in C#

We expect this.imageViewProcess must be NULL when this.imageViewProcess.WaitForInputIdle() is called

private bool DisplayImage(string imagePath) { try { if (!File.Exists(imagePath)) { MessageBox.Show("Unable to locate image file." + Environment.NewLine + Environment.NewLine + imagePath, "Info"); this.IMAGEMISSING = true; this.APPMODE = 6; return false; } this.KillImageProcess(); this.imageViewProcess = Process.Start(imagePath); this.imageViewProcess.WaitForInputIdle(); if ((this.APPMODE == 2) && (this.grdTransactions.get_Rows().get_Count() == 0)) { this.grdTransactions.get_DisplayLayout().get_Bands().get_Item(0).AddNew(); } this.EnterEditMode(); this.IMAGEMISSING = false; } catch (Exception exception) { MessageBox.Show(exception.Message); this.IMAGEMISSING = true; return false; } base.Activate(); return true; }

Let’s test the theory in PowerShell, you can see when attempting to Process Start an image then call WaitForInputIdle we get an InvokeMethodOnNull exception, this is exactly the same exception thrown by our .NET application.

PS C:\Users\Malcolm> $p = [System.Diagnostics.Process]::Start("notepad.exe") $p.WaitForInputIdle() True PS C:\Users\Malcolm> $p = [System.Diagnostics.Process]::Start("C:\Users\Malcolm\Pictures\DSC01230.JPG") $p.WaitForInputIdle() You cannot call a method on a null-valued expression. At line:3 char:1 + $p.WaitForInputIdle() + ~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (:) [], RuntimeException + FullyQualifiedErrorId : InvokeMethodOnNull PS C:\Users\Malcolm>

Why Process.Start returning NULL?  Checking the .NET documentation  at

Return Value

A new Process component that is associated with the process resource, or a null reference (Nothing in Visual Basic), if no process resource is started (for example, if an existing process is reused).

Use this overload to start a process resource by specifying its file name. The overload associates the resource with a new Process component. If the process is already running, no additional process resource is started. Instead, the existing process resource is reused and no new Process component is created. In such a case, instead of returning a new Process component, Start returns a null reference (Nothing in Visual Basic) to the calling procedure.

Following the source code

we can see in how the “false” return results in null

public static Process Start(ProcessStartInfo startInfo) { Process process = new Process(); if (startInfo == null) throw new ArgumentNullException("startInfo"); process.StartInfo = startInfo; if (process.Start()) { return process; } return null; // IF process.Start returns FALSE, then we will get NULL back, like in our crashing program }

Following process.Start leads us to StartWithShellExecuteEx where we can see if shellExecuteInfo.hProcess is null, FALSE gets returned.

private bool StartWithShellExecuteEx(ProcessStartInfo startInfo) { <code…> if (shellExecuteInfo.hProcess != (IntPtr)0) { SafeProcessHandle handle = new SafeProcessHandle(shellExecuteInfo.hProcess); SetProcessHandle(handle); return true; }

return false;


How do we fix it if the vendor can’t/won’t/source code not available?


To fix this bug we just need a simple NULL check

To do this we will use ILDASM from Windows SDK to dump the contents into a folder, open the IL file, and search for our DisplayImage method.

Our patched code will be


ldfld      class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess
brfalse.s  IL_005c

Refer to Partition III for an explanation of the commands.

In context:

.method private hidebysig instance bool DisplayImage(string imagePath) cil managed { // Code size 196 (0xc4) .maxstack 4 .locals init (class [mscorlib]System.Exception V_0, bool V_1) .try { IL_0000: ldarg.1 IL_0001: call bool [mscorlib]System.IO.File::Exists(string) IL_0006: brtrue.s IL_003d IL_0008: ldstr "Unable to locate image file." IL_000d: call string [mscorlib]System.Environment::get_NewLine() IL_0012: call string [mscorlib]System.Environment::get_NewLine() IL_0017: ldarg.1 IL_0018: call string [mscorlib]System.String::Concat(string, string, string, string) IL_001d: ldstr "Info" IL_0022: call valuetype [System.Windows.Forms]System.Windows.Forms.DialogResult [System.Windows.Forms]System.Windows.Forms.MessageBox::Show(string, string) IL_0027: pop IL_0028: ldarg.0 IL_0029: ldc.i4.1 IL_002a: stfld bool StreamlineUI.MainForm::IMAGEMISSING IL_002f: ldarg.0 IL_0030: ldc.i4.6 IL_0031: call instance void StreamlineUI.MainForm::set_APPMODE(valuetype [StreamlineClsLib]StreamlineClsLib.Form/ACTION) IL_0036: ldc.i4.0 IL_0037: stloc.1 IL_0038: leave IL_00c2 IL_003d: ldarg.0 IL_003e: call instance bool StreamlineUI.MainForm::KillImageProcess() IL_0043: pop IL_0044: ldarg.0 IL_0045: ldarg.1 IL_0046: call class [System]System.Diagnostics.Process [System]System.Diagnostics.Process::Start(string) IL_004b: stfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess ldarg.0 ldfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess ldnull cgt.un stloc.0 ldloc.0 brfalse.s IL_005c IL_0050: ldarg.0 IL_0051: ldfld class [System]System.Diagnostics.Process StreamlineUI.MainForm::imageViewProcess IL_0056: callvirt instance bool [System]System.Diagnostics.Process::WaitForInputIdle() IL_005b: pop IL_005c: ldarg.0 IL_005d: call instance valuetype [StreamlineClsLib]StreamlineClsLib.Form/ACTION StreamlineUI.MainForm::get_APPMODE() IL_0062: ldc.i4.2

We then recompile

c:\windows\\Framework\v2.0.50727\ilasm /RESOURCE=app.res

Check the compiled fix with .NET reflector :

This looks good (!= null decompiles as > null) due to cgt.un instruction which is : Push 1 (of type int32) if value1 > value2, unsigned or unordered, else push 0.

Testing the application, yet another bug patched!

Posted in .NET, C#, Hacking, IL, MSIL, Patching, Reverse Engineering | Tagged | Leave a comment

Case of the .NET Memory Leak via Gfx Driver on Windows 8

A .NET 4 application used to index scanned documents started to fail on Windows 8


Typically the recommendation for Not enough storage is available to process this command might involve:

  • reduce the number of running programs;
  • remove unwanted files from the disk the paging file is on and restart the system;
  • check the paging file disk for an I/O error; or
  • install additional memory in your system

However in this case these options didn’t fix the issue.

Taking a dmp file with ProcDump –ma option ( we find the following:

Load SOS extensions for debugging .NET process

0:000> .loadby sos clr
0:000> .cordll -ve -u -l

Dumping last CLR exception…

0:000> !pe
Exception object: 07559bb0
Exception type:   System.IO.FileLoadException
Message:          Could not load file or assembly ‘System.ServiceModel, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089’ or one of its dependencies. Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008)
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 UNKNOWN!DocumentAutomation.Common.ServiceAdaptor.ESBV4ServicesAdaptor..ctor(System.String, System.String, System.String[,], System.String, System.TimeSpan)+0x2
    005DE908 04CE30E5 DocumentAutomation_Indexing!DocumentAutomation.Indexing.ViewModels.IndexingViewModel.OnValidateCase()+0x265
StackTraceString: <none>
HResult: 80070008
0:000> lmvm DocumentAutomation*
Browse full module list
start    end        module name
00420000 0045c000   DocumentAutomation_Indexing C (no symbols)          
    Loaded symbol image file: DocumentAutomation.Indexing.exe
    Image path: C:\Program Files (x86)\TIS\eFlow 4.5\Bin\DocumentAutomation.Indexing.exe
    Image name: DocumentAutomation.Indexing.exe
    Browse all global symbols  functions  data
    Has CLR image header, track-debug-data flag not set
    Timestamp:        Wed Mar 25 19:18:50 2015 (55126F6A)
    CheckSum:         00000000
    ImageSize:        0003C000
    File version:
    Product version:
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04b0
    ProductName:      DocumentAutomation.Indexing
    InternalName:     DocumentAutomation.Indexing.exe
    OriginalFilename: DocumentAutomation.Indexing.exe
    FileDescription:  DocumentAutomation.Indexing
LegalCopyright:   Copyright ©  2012

Extracting the EXE in WinDbg, we can then open in .NET reflector to decompile

0:000> !SaveModule 00420000 C:\support\DocumentAutomation.Indexing.exe
3 sections in file
section 0 – VA=2000, VASize=350f4, FileAddr=200, FileSize=35200
section 1 – VA=38000, VASize=5d0, FileAddr=35400, FileSize=600
section 2 – VA=3a000, VASize=c, FileAddr=35a00, FileSize=200

Here we can find the code that threw the error. The issue is initialization of ESBV4ServicesAdaptor is failing when it tries to load System.ServiceModel.dll. You may need to take a few dmp files to confirm if it is always failing in exact same place.

Private Sub OnValidateCase()
        Dim base2 As CaseValidatorBase
        If Me._globalSettings.PlanValidationConnector(Me.CurrentCase.get_Plan).Equals(“Internal”, StringComparison.OrdinalIgnoreCase) Then
            base2 = New InternalCaseValidator(Me.CurrentCase, Me.eFlow.Collection, Me._globalSettings)
Dim adaptor As New ESBV4ServicesAdaptor(Me._globalSettings.GetSetting(“AaspireUserID”), Encryption.Decrypt(Me._globalSettings.GetSetting(“AaspirePassword”)), Me._globalSettings.ImagaasPlanIdsPlancodes, Me._globalSettings.GetSetting(“V4RemoteEndPoint”), New TimeSpan(0, Integer.Parse(Me._globalSettings.GetSetting(“ServiceAdaptorSendTimeoutMins”)), 0))
            base2 = New CaseValidator(Me.CurrentCase, adaptor, Me.eFlow.Collection, Me._globalSettings.GetSetting(“AaspireUserID”), Encryption.Decrypt(Me._globalSettings.GetSetting(“AaspirePassword”)))
        End If

        Me.Status = IndexingStatus.Validated
        If (base2.get_ValidationMsgs.Count > 0) Then
            MessageBox.Show(base2.GetMassages, “Validation Message”, MessageBoxButton.OK)
        End If
    Catch exception As InvalidCaseException
        Dim nullable As Integer?
        Dim e As InvalidCaseException = exception
        Dim messageBoxText As String = If(e.ValidationMessage.get_PageIndex.HasValue, String.Format(“Page {0}: {1}”, If(nullable = e.ValidationMessage.get_PageIndex.HasValue, New Integer?((nullable.GetValueOrDefault + 1)), Nothing), e.Message), e.Message)
        MessageBox.Show(messageBoxText, e.ValidationMessage.get_Field, MessageBoxButton.OK)
        If e.ValidationMessage.get_PageIndex.HasValue Then
            Dim classf As <>c__DisplayClass5f
            Me.SelectedPageImage = Enumerable.FirstOrDefault(Of PageImage)(Me.PageImages, New Func(Of PageImage, Boolean)(classf, DirectCast(Me.<OnValidateCase>b__5e, IntPtr)))
        End If
    Catch exception2 As Exception
MessageBox.Show(exception2.Message, “Fatal Error”, MessageBoxButton.OK, MessageBoxImage.Hand)
    End Try
End Sub

Because this is failing to initialize an object, a common issue may be out of process address space, and this app is running as 32-bit program:

Looking at memory usage…
0:000> !address -summary
Mapping file section regions…
Mapping module regions…
Mapping PEB regions…
Mapping TEB and stack regions…
Mapping heap regions…
Mapping page heap regions…
Mapping other regions…
Mapping stack trace database regions…
Mapping activation context regions…
— Usage Summary —————- RgnCount ———– Total Size ——– %ofBusy %ofTotal
Heap                                    344          5ad52000 (   1.419 GB)  78.89%   70.97%
Free                                    320           cdd5000 ( 205.832 MB)           10.05%
Image                                  1037           c462000 ( 196.383 MB)  10.66%    9.59%
<unknown>                               757           b053000 ( 176.324 MB)   9.57%    8.61%
Stack                                    55            fc0000 (  15.750 MB)   0.85%    0.77%
Other                                     9             41000 ( 260.000 kB)   0.01%    0.01%
TEB                                      18             12000 (  72.000 kB)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kB)   0.00%    0.00%
— Type Summary (for busy) —— RgnCount ———– Total Size ——– %ofBusy %ofTotal
MEM_PRIVATE                             705          60ec8000 (   1.514 GB)  84.19%   75.72%
MEM_IMAGE                              1467           d1cd000 ( 209.801 MB)  11.39%   10.24%
MEM_MAPPED                               49           5186000 (  81.523 MB)   4.43%    3.98%
— State Summary —————- RgnCount ———– Total Size ——– %ofBusy %ofTotal
MEM_COMMIT                             1686          6e10d000 (   1.720 GB)  95.60%   85.99%
MEM_FREE                                320           cdd5000 ( 205.832 MB)           10.05%
MEM_RESERVE                             535           510e000 (  81.055 MB)   4.40%    3.96%
— Protect Summary (for commit) – RgnCount ———– Total Size ——– %ofBusy %ofTotal
PAGE_READWRITE                          635          5d9f3000 (   1.463 GB)  81.32%   73.14%
PAGE_EXECUTE_READ                       118           8ac2000 ( 138.758 MB)   7.53%    6.78%
PAGE_READONLY                           496           623a000 (  98.227 MB)   5.33%    4.80%
PAGE_WRITECOPY                          252           1681000 (  22.504 MB)   1.22%    1.10%
PAGE_EXECUTE_READWRITE                   97            1f5000 (   1.957 MB)   0.11%    0.10%
PAGE_EXECUTE_WRITECOPY                   50            14c000 (   1.297 MB)   0.07%    0.06%
PAGE_READWRITE|PAGE_GUARD                36             5a000 ( 360.000 kB)   0.02%    0.02%
PAGE_NOACCESS                             2              2000 (   8.000 kB)   0.00%    0.00%
— Largest Region by Usage ———– Base Address ——– Region Size ———-
Heap                                         e570000            fcf000 (  15.809 MB)
Free                                        5e820000            d80000 (  13.500 MB)
Image                                       57388000            f1a000 (  15.102 MB)
<unknown>                                    2120000           1cae000 (  28.680 MB)
Stack                                        2000000             fd000 (1012.000 kB)
Other                                       7ed90000             23000 ( 140.000 kB)
TEB                                         7ebf7000              1000 (   4.000 kB)
PEB                                         7edbe000              1000 (   4.000 kB)
0:000> !heap -stat
_HEAP 006f0000
     Segments            00000004
         Reserved  bytes 007fc000
         Committed bytes 005c5000
     VirtAllocBlocks     0000005d
         VirtAlloc bytes e255b09c
_HEAP 00b80000
     Segments            00000009
         Reserved  bytes 04e47000
         Committed bytes 041cb000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 09170000
     Segments            00000002
         Reserved  bytes 0010e000
         Committed bytes 00018000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 00990000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 0000f000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 04780000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00008000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 04750000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00007000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 02110000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00003000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
_HEAP 01ff0000
     Segments            00000001
         Reserved  bytes 0000f000
         Committed bytes 00001000
     VirtAllocBlocks     00000000
         VirtAlloc bytes 00000000
0:000> !heap -s
                                              NT HEAP STATS BELOW
LFH Key                   : 0x271d989d
Termination on corruption : DISABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                    (k)     (k)    (k)     (k) length      blocks cont. heap
006f0000 00000002    8284   5924   8176   1328   202     4   93      1   LFH
    External fragmentation  22 % (202 free blocks)
00990000 00001002     168     64     60      9     7     1    0      0   LFH
00b80000 00001002   80264  67380  80156    294    30     9    0      0   LFH
02110000 00001002      60     12     60      3     6     1    0      0     
01ff0000 00041002      60      4     60      2     1     1    0      0     
04780000 00001002     168     36     60      2     3     1    0      0   LFH
04750000 00041002     168     32     60      1     5     1    0      0   LFH
09170000 00001002    1188    100   1080     50     5     2    0      0   LFH
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x07ca8d10
generation 1 starts at 0x07c51000
generation 2 starts at 0x02121000
ephemeral segment allocation context: none
         segment             begin         allocated  size
02120000  02121000  0311ffe4  0xffefe4(16773092)
06ab0000  06ab1000  0757e71c  0xacd71c(11327260)
07c50000  07c51000  0819fca8  0x54eca8(5565608)
Large object heap starts at 0x03121000
         segment             begin         allocated  size
03120000  03121000  03dadaa8  0xc8caa8(13159080)
Total Size:              Size: 0x2ca7e50 (46825040) bytes.
GC Heap Size:    Size: 0x2ca7e50 (46825040) bytes.

0:000> !dumpheap -stat

          MT    Count    TotalSize Class Name

    60f3b340     9234       258552 System.Reflection.Emit.ScopeTree
    60f6b8fc     4647       260232 System.Reflection.RuntimePropertyInfo
    60f642d0     8139       260448 System.EventHandler
    60f70b28     9622       269416 System.Collections.ArrayList+ArrayListEnumeratorSimple
    57501c04     3616       274816 System.Windows.Data.BindingExpression
    06256688     4414       300152 TiS.Core.eFlowAPI.TisROIMiscellaneousParams
    06256160     4414       300152 TiS.Core.eFlowAPI.TisROIPostProcessingParams
    06256b64     4622       314296 TiS.Core.eFlowAPI.TisOCRGroupParamsLinksContainer
    60f7a740     3312       318672 System.Runtime.CompilerServices.ConditionalWeakTable`2+Entry[[System.Object, mscorlib],[System.Object, mscorlib]][]
    574fbd18     1294       320912 System.Windows.Controls.Border
    60f4356c     9234       332424 System.Reflection.Emit.DynamicResolver
    60f763d4    30341       364092 System.WeakReference
    60f75a78     8363       367972 System.Signature
    60f3b1d0    11905       380960 System.Reflection.Emit.SignatureHelper
    60f71238     2920       424800 System.Collections.Hashtable+bucket[]
    009846b0     4196       429232 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[System.Attribute[], mscorlib]][]
    60f6c290     9901       475248 System.RuntimeMethodInfoStub
    062551fc     4414       476712 TiS.Core.eFlowAPI.TisROIParams
    0500b718    15820       506240 TiS.Core.eFlowAPI.NamedObjectListEvent
    60f65af8     9234       517104 System.Reflection.Emit.DynamicMethod
    60f73acc    13901       641256 System.Int32[]
    60f6bed4    10749       644940 System.Reflection.RuntimeMethodInfo
    58eccce8     9114       846936 System.Windows.EffectiveValueEntry[]
    60f433b8     9234       886464 System.Reflection.Emit.DynamicILGenerator
    60f35738   107092      3501976 System.Object[]
    60f75670    51775      5672668 System.Byte[]
    60f721b4    57971      6750108 System.String
    00702bd0      223      8557414      Free
    Total 783513 objects
    Fragmented blocks larger than 0.5 MB:
        Addr     Size      Followed by
    08039088    1.4MB         08194298 System.String


From the memory analysis we find heap is large 1.4GB ram, but the managed objects are very small in size, so looks like we have a native memory leak. This can be further researched with

However we already had several dmp files of the process as memory grew, so we took a look at these.

A raw analysis of the dmp file showed 73% of the dmp file was filled with 0xFF – we suspect this was 0xFF 0xFF 0xFF representing white background on the scanned documents.

From !eestack we saw this stack pattern always present in each dmp file, allocating to heap


Current frame: ntdll!NtRemoveIoCompletion+0xc
ChildEBP RetAddr  Caller, Callee
05e6e988 748f2dfa KERNELBASE!GetQueuedCompletionStatus+0x2a, calling ntdll!NtRemoveIoCompletion
05e6e9c0 673f94e1 clr!ThreadpoolMgr::CompletionPortThreadStart+0x22f, calling kernel32!GetQueuedCompletionStatusStub
05e6ea54 6734814d clr!Thread::intermediateThreadProc+0x4d
05e6ed20 76fb81b5 ntdll!bsearch+0x6f
05e6ed44 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6ed64 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6ed7c 71db107b mscoreei!ShimLog::Log+0x1a4, calling mscoreei!__security_check_cookie
05e6ed9c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6edb8 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6edf8 76fdb1b7 ntdll!LdrpApplyLookupReference+0x1e, calling ntdll!RtlIsCriticalSectionLockedByThread
05e6ee04 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6ee08 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee38 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee3c 76fc53eb ntdll!LdrpFindOrMapDll+0xfd6, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6ee48 76fc544d ntdll!LdrpFindOrMapDll+0xf06, calling ntdll!LdrpFindOrMapDll+0xf40
05e6ee58 76fc54ac ntdll!LdrpFindOrMapDll+0x107e, calling ntdll!__security_check_cookie
05e6eeec 76fb8c78 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x2d8, calling ntdll!memset
05e6efb4 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6efd4 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6f00c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6f028 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6f05c 76fb9493 ntdll!LdrpApplyFileNameRedirection+0x96, calling ntdll!RtlDosApplyFileIsolationRedirection_Ustr
05e6f08c 76fb9424 ntdll!LdrpApplyFileNameRedirection+0xac, calling ntdll!__security_check_cookie
05e6f0ac 76fc5c5b ntdll!LdrpFindOrMapDependency+0x209, calling ntdll!LdrpFindOrMapDll
05e6f0c8 76fc5c79 ntdll!LdrpFindOrMapDependency+0x227, calling ntdll!LdrpFindOrMapDependency+0x2d9
05e6f0d8 76fc5cba ntdll!LdrpFindOrMapDependency+0x268, calling ntdll!__security_check_cookie
05e6f15c 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6f164 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f194 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f198 76fcff38 ntdll!LdrpFindLoadedDll+0x2ea, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6f1ac 76fee967 ntdll!LdrpFindLoadedDll+0x342, calling ntdll!__security_check_cookie
05e6f1bc 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f304 76fc4d3e ntdll!LdrpGetProcedureAddress+0x3d, calling ntdll!RtlImageNtHeaderEx
05e6f350 76f8e385 ntdll!LdrpResolveNonStaticDependency+0x22f, calling ntdll!LdrpDereferenceNode
05e6f384 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f398 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f3a8 76f8e47c ntdll!LdrGetProcedureAddressForCaller+0x312, calling ntdll!LdrpReleaseLoaderLock
05e6f3c4 748bf769 KERNELBASE!_KernelBaseBaseDllInitialize+0x313, calling KERNELBASE!RegKrnInitialize
05e6f3d4 748bf77a KERNELBASE!_KernelBaseBaseDllInitialize+0x324, calling KERNELBASE!__security_check_cookie
05e6f45c 76fd04c3 ntdll!RtlpLowFragHeapAllocFromContext+0x34d, calling ntdll!memset
05e6f4d8 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f4f0 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f518 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f528 7479b2e4 msvcrt!_VEC_memzero+0x36, calling msvcrt!fastzero_I
05e6f550 6315ba71 igdumdim32!OpenAdapter+0x15aa71, calling ntdll!RtlLeaveCriticalSection
05e6f55c 631550ef igdumdim32!OpenAdapter+0x1540ef, calling igdumdim32!OpenAdapter+0x15aa5e
05e6f564 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f590 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f594 63151741 igdumdim32!OpenAdapter+0x150741, calling igdumdim32!OpenAdapter+0x15403e
05e6f5a0 681b30a1 msctfui!_CRT_INIT+0x1ea, calling msctfui!__security_check_cookie
05e6f5ac 681a32c6 msctfui!DllMain+0x10, calling kernel32!GetCurrentThreadId
05e6f5bc 681b31f7 msctfui!__DllMainCRTStartup+0x107, calling msctfui!DllMain
05e6f5c4 681b366e msctfui!_SEH_epilog4_GS+0xa, calling msctfui!__security_check_cookie
05e6f5c8 681b3379 msctfui!__DllMainCRTStartup+0x289, calling msctfui!_SEH_epilog4_GS
05e6f608 631518a5 igdumdim32!OpenAdapter+0x1508a5, calling igdumdim32!OpenAdapter+0x155225
05e6f624 76fb96c4 ntdll!RtlDeactivateActivationContextUnsafeFast+0x9c, calling ntdll!__security_check_cookie
05e6f640 76fb9758 ntdll!LdrpCallInitRoutine+0x43, calling ntdll!LdrxCallInitRoutine
05e6f650 76fb9776 ntdll!LdrpCallInitRoutine+0x61, calling ntdll!_SEH_epilog4
05e6f668 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f67c 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f68c 76fd5b55 ntdll!LdrpInitializeThread+0x1af, calling ntdll!LdrpReleaseLoaderLock
05e6f698 76fd5b43 ntdll!LdrpInitializeThread+0x139, calling ntdll!_SEH_epilog4
05e6f6ec 76fd5ab4 ntdll!LdrpInitializeThread+0xeb, calling ntdll!RtlActivateActivationContextUnsafeFast
05e6f6f0 76fd5b1d ntdll!LdrpInitializeThread+0x1a4, calling ntdll!RtlDeactivateActivationContextUnsafeFast
05e6f72c 76fcaa56 ntdll!_LdrpInitialize+0x80, calling ntdll!_SEH_epilog4
05e6f774 76fca9ca ntdll!LdrInitializeThunk+0x1a, calling ntdll!NtContinue
05e6f9e0 6734813b clr!Thread::intermediateThreadProc+0x3b, calling clr!_alloca_probe_16
05e6f9f4 74cc7c04 kernel32!BaseThreadInitThunk+0x24
05e6fa08 76fcad1f ntdll!__RtlUserThreadStart+0x2f
05e6fa50 76fcacea ntdll!_RtlUserThreadStart+0x1b, calling ntdll!__RtlUserThreadStart

This seemed to be coming from user mode graphics driver. Interestingly on working machines a similar stack pattern was present, minus the user mode graphics driver.

0:000> lmvm igdumdim32
Browse full module list
start    end        module name
62b70000 64249000   igdumdim32   (deferred)            
    Image path: C:\Windows\System32\igdumdim32.dll
    Image name: igdumdim32.dll
    Browse all global symbols  functions  data
    Timestamp:        Wed Mar 04 04:29:18 2015 (54F5EF6E)
    CheckSum:         016F3784
    ImageSize:        016D9000
    File version:
    Product version:
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.8 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Intel Corporation
    ProductName:      Intel HD Graphics Drivers for Windows 8(R)
    InternalName:     igdumdim32.dll
    OriginalFilename: igdumdim32.dll
    FileDescription:  User Mode Driver for Intel(R) Graphics Technology
    LegalCopyright:   Copyright (c) 1998-2013 Intel Corporation.


So I wrote a simple console C# program to test something, to show the RenderCapability ( ) of the device. Added reference PresentationCore

using System; using System.Windows.Media; namespace RenderCapabilityConsoleApplication { class Program { static void Main(string[] args) { Console.WriteLine("RenderCapability:"); Console.WriteLine("Tier: {0}", RenderCapability.Tier >> 16); switch (RenderCapability.Tier) { case 0: Console.WriteLine("No graphics hardware acceleration is available for the application on the device.All graphics features use software acceleration.The DirectX version level is less than version 9.0."); break; case 0x00010000: Console.WriteLine("Most of the graphics features of WPF will use hardware acceleration if the necessary system resources are available and have not been exhausted. This corresponds to a DirectX version that is greater than or equal to 9.0."); break; case 0x00020000: Console.WriteLine("Most of the graphics features of WPF will use hardware acceleration provided the necessary system resources have not been exhausted.This corresponds to a DirectX version that is greater than or equal to 9.0."); break; } Console.Write("Press <Enter>"); Console.ReadLine(); } } }

What we found was this:

  • The broken machine reported capability 0 (No hardware acceleration)
  • The working machines (Windows XP+Windows 7) reported render capability 2 (Most of the features of WPF will use hardware acceleration)

Now a machine can be forced down to capability 0 by remote controlling it. However in this case the machine was not remote controlled.

So we RDP’d to the working Win7/Windows XP machines – they were now breaking too with out of memory error, only when under RDP session:


Running DxDiag on the affected machine, we found no video acceleration capability was available on the affected machine.

Checking device drivers we found a 3rd party video adapter installed which was used by the IT department for remote control ( CA ITCM Remote Control ) 

Unfortunately DxDiag didn’t seem to capture this display adapter in its report, we found it via Device Manager.

Removing this device driver set the machines render capability back to 0, and memory leak was gone, application now worked fine…

Posted in .NET, C#, Debugging, Windows 8 | Tagged | Leave a comment