Experimental Use of 64-bit Dump of 32-bit .NET Process in WinDbg

A .NET dmp file is typically best captured as 32-bit for 32-bit process. On x64 system this could be using the 32-bit task manager (C:\windows\syswow64\taskmgr.exe), WinDbg (x86), or a tool like ProcDump (http://live.sysinternals.com/ProcDump.exe )

However what if a 32-bit .NET process has already been captured in 64-bit dmp file, and the issue is really hard to reproduce?

If you try to open with WinDbg x64 and try to load SOS you will get a failure:

…
wow64cpu!CpupSyscallStub+0x2:
00000000`76f21eb2 c3              ret
0:000> .loadby sos clr
The call to LoadLibrary(C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos) failed, Win32 error 0n193
    “%1 is not a valid Win32 application.”
Please check your debugger configuration and/or network access.

So we try to force load of x64 SOS:

0:000> .cordll -u -ve -I clr -lp c:\windows\microsoft.net\framework64\v4.0.30319
Automatically loaded SOS Extension
CLRDLL: Loaded DLL c:\windows\microsoft.net\framework64\v4.0.30319\mscordacwks.dll
CLR DLL status: Loaded DLL c:\windows\microsoft.net\framework64\v4.0.30319\mscordacwks.dll

However we can’t run any SOS commands successfully:

0:000> !threads
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> .cordll -ve -u -l
CLRDLL: LoadLibrary(C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll) failed, Win32 error 193
CLRDLL: Unable to find mscordacwks_AMD64_x86_4.6.42.00.dll by mscorwks search
CLRDLL: Unable to find ‘mscordacwks_AMD64_x86_4.6.42.00.dll’ on the path
CLRDLL: Unable to get version info for ‘c:\symbols\clr.dll\54EFEAF769a000\mscordacwks_AMD64_x86_4.6.42.00.dll’, Win32 error 0n87
Cannot Automatically load SOS
CLRDLL: ERROR: Unable to load DLL mscordacwks_AMD64_x86_4.6.42.00.dll, Win32 error 0n87
CLR DLL status: ERROR: Unable to load DLL mscordacwks_AMD64_x86_4.6.42.00.dll, Win32 error 0n87

Trying to use SosEX extension also fails:

0:000:x86> .load C:\debugging\sosex64.dll
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000:x86> !bhi
The target is 32-bit. 32-bit managed targets must be debugged using a 32-bit debugger.
0:000:x86> !wow64exts.sw
Switched to Host mode
0:000> !bhi
CLRDLL: CLR DLL load disabled
Unable to initialize .NET data interface. Version 4.6.42.0 of mscordacwks.dll is required.
Locate and load the correct version of mscordacwks.dll. See documentation for the .cordll command.

So we try to open dmp file in WinDbg (x86) :

0:000> .loadby sos clr
0:000> !threads
SOS does not support the current target architecture.
0:000> .load wow64exts
0:000> !wow64exts.sw
Switched to Guest (WoW) mode
0:000:x86> !threads
SOS does not support the current target architecture.

0:000:x86> .load c:\support\sosex32.dll
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000:x86> !bhi
Enumerating heap objects…Failed to load module 00000000012d3fbc. Error = 0x80131c49
Failed to load module 00000000012d3fbc. Error = 0x80131c49
Request for array class data for 0000000071635a78 failed. Error=0x80004001.
BuildHeapIndex failed. Error = 0x80004001.
0:000:x86> !wow64exts.sw
Switched to Host mode
0:000> !bhi
The target is 64-bit. 64-bit managed targets must be debugged using a 64-bit debugger.

What if we try removing the SOS architecture check, will it explode?

To do this I opened another instance of WinDbg (x86) and selected File –> Attach to a Process and selected our WinDbg (x86) process that had SOS extension loaded.

(225c.1f34): Break instruction exception – code 80000003 (first chance)
eax=feaab000 ebx=00000000 ecx=77771080 edx=10010044 esi=77771080 edi=77771080
eip=7773b270 esp=07c6f8b4 ebp=07c6f8e0 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
7773b270 cc              int     3
0:002> lmvm sos
Browse full module list
start    end        module name
54890000 54950000   sos        (deferred)            
    Image path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.dll
    Image name: sos.dll
    Browse all global symbols  functions  data
    Timestamp:        Fri Feb 27 14:50:08 2015 (54EFE970)
    CheckSum:         000B91DB
    ImageSize:        000C0000
    File version:     4.6.42.0
    Product version:  4.0.30319.0
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     SOS.dll
    OriginalFilename: SOS.dll
    ProductVersion:   4.6.42.0
    FileVersion:      4.6.42.0 built by: NETFXREL1
    PrivateBuild:     DDBLD629A
    FileDescription:  Microsoft NTSD extension for .NET Runtime
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail
0:002> s -a 0x54890000 0x54950000 “SOS does not support the current target architecture”
54893224  53 4f 53 20 64 6f 65 73-20 6e 6f 74 20 73 75 70  SOS does not sup

Let’s try and find a reference to this address. We’ll convert it into Little Endian (reverse it) and search for those bytes 24 32 89 54

0:001> s -b 0x54890000 0x54950000 24 32 89 54
548af180  24 32 89 54 89 35 ac c0-91 54 e8 11 79 03 00 83  $2.T.5…T..y…

Now let’s disassemble that function:

0:001> uf 548af180

sos!ArchQuery:
548af140 55              push    ebp
548af141 8bec            mov     ebp,esp
548af143 51              push    ecx
548af144 a1fcc09154      mov     eax,dword ptr [sos!g_ExtControl (5491c0fc)]
548af149 8d55fc          lea     edx,[ebp-4]
548af14c 56              push    esi
548af14d 52              push    edx
548af14e 50              push    eax
548af14f 8b08            mov     ecx,dword ptr [eax]
548af151 33f6            xor     esi,esi
548af153 ff9190000000    call    dword ptr [ecx+90h]
548af159 8b45fc          mov     eax,dword ptr [ebp-4]
548af15c 3d4c010000      cmp     eax,14Ch
548af161 750a            jne     sos!ArchQuery+0x2d (548af16d)  Branch

sos!ArchQuery+0x23:
548af163 e888feffff      call    sos!X86Machine::GetInstance (548aeff0)
548af168 8bf0            mov     esi,eax
548af16a 8b45fc          mov     eax,dword ptr [ebp-4]

sos!ArchQuery+0x2d:
548af16d 3dc4010000      cmp     eax,1C4h
548af172 7507            jne     sos!ArchQuery+0x3b (548af17b)  Branch

sos!ArchQuery+0x34:
548af174 e8f7fdffff      call    sos!ARMMachine::GetInstance (548aef70)
548af179 8bf0            mov     esi,eax

sos!ArchQuery+0x3b:
548af17b 85f6            test    esi,esi
548af17d 751d            jne     sos!ArchQuery+0x5c (548af19c)  Branch <- Jumps if valid platform

sos!ArchQuery+0x3f:
548af17f 6824328954      push    offset sos!`string’ (54893224) <- ERROR MESSAGE
548af184 8935acc09154    mov     dword ptr [sos!g_targetMachine (5491c0ac)],esi
548af18a e811790300      call    sos!ExtErr (548e6aa0)
548af18f 83c404          add     esp,4
548af192 b805400080      mov     eax,80004005h
548af197 5e              pop     esi
548af198 8be5            mov     esp,ebp
548af19a 5d              pop     ebp
548af19b c3              ret

sos!ArchQuery+0x5c:
548af19c 8935acc09154    mov     dword ptr [sos!g_targetMachine (5491c0ac)],esi
548af1a2 33c0            xor     eax,eax
548af1a4 5e              pop     esi
548af1a5 8be5            mov     esp,ebp
548af1a7 5d              pop     ebp
548af1a8 c3              ret
0:001> u 54893224
sos!`string’:
54893224 53              push    ebx
54893225 4f              dec     edi
54893226 53              push    ebx
54893227 20646f65        and     byte ptr [edi+ebp*2+65h],ah
5489322b 7320            jae     sos!`string’+0x29 (5489324d)
5489322d 6e              outs    dx,byte ptr [esi]
5489322e 6f              outs    dx,dword ptr [esi]
5489322f 7420            je      sos!`string’+0x2d (54893251)

We take the address of the conditional jmp we want to remove, and replace it with an unconditional jmp:

0:001> a 548af17d
548af17d jmp sos!ArchQuery+0x5c
jmp sos!ArchQuery+0x5c
548af17f <Ctrl+C>

We can now see in decompilation the jmp is gone:

sos!ArchQuery+0x3b:
548af17b 85f6            test    esi,esi
548af17d eb1d            jmp     sos!ArchQuery+0x5c (548af19c)  Branch

We continue with ‘g’ and go back to our WinDbg that has the bad dmp file open…

Now we’re getting somewhere:

Pre Patch

0:000:x86> !threads
SOS does not support the current target architecture.

Post Patch

At least now we can run some SOS commands. Many commands work, or at least partially work. They may not behave 100% correctly, but at least some of the information seems to be correct in my initial investigation.

0:000:x86> !threads
ThreadCount:      2
UnstartedThread:  0
BackgroundThread: 1
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                         Lock 
       ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception
   0    1 1ad4 0112ac20     a6028 Preemptive  0324F8A8:00000000 01124b98 1     STA System.Runtime.InteropServices.SEHException 0324f644
   2    2 1e04 01138b70     2b228 Preemptive  00000000:00000000 01124b98 0     MTA (Finalizer)

0:000:x86> !pe
Exception object: 0324f644
Exception type:   System.Runtime.InteropServices.SEHException
Message:          External component has thrown an exception.
InnerException:   <none>
StackTrace (generated):
c0000005 Exception in C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll.pe debugger extension.
      PC: 548c5f3d  VA: 00000000  R/W: 0  Parameter: 00000000

Stack based commands (!dumpstack \ !clrstack \!eestack ) however will thrown an exception like this in the debugger.

(225c.21e0): Access violation – code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=0635d844 ecx=00000000 edx=00000000 esi=778d919c edi=00fbb644
eip=548a87b3 esp=0635d26c ebp=0635d7f4 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
sos!DumpStackWorker+0x63:
548a87b3 8b01            mov     eax,dword ptr [ecx]  ds:002b:00000000=????????

And result in error displayed in WinDbg

0:000:x86> !DumpStack
OS Thread Id: 0x1ad4 (0)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: ntdll_77870000!NtWaitForSingleObject+0xc
c0000005 Exception in C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll.DumpStack debugger extension.
      PC: 548a87b3  VA: 00000000  R/W: 0  Parameter: 00000000