## Case of the Invalid Base Key Error

Continuing my series on .NET patching – refer to https://chentiangemalc.wordpress.com/2014/05/19/case-of-the-net-service-hang-patching-webclient-operation-timed-out/ for the previous articles more of an overview of using some of the tools described.

A “front end” application that was used to launch another application was failing to launch on some machines with the error after migrating to Windows 7 x64:

A quick decimal to hex conversion using Windows calculator got what looked like an HRESULT which we could do an internet / or Windows SDK headers search for value 80040203. However that didn’t turn up anything obvious.

I cannot stress the importance of first investigating the error message provided by the application. I see this first level diagnostic process overlooked all the time, and I never assume the first few levels of support have done this…

As the error mentioned “key” checking registry keys between working / broken machine was a good idea. Using Process Monitor (ProcMon) I used a filter I commonly use when getting a quick summary of what what data / registry settings an application uses (I have this saved as a ProcMon Filter)

• Process is <processname.exe> – Include
• Operation is RegQueryValue – Include
• Operation is RegSetValue – Include
• Operation is ReadFile – Include
• Operation is WriteFile – Include
• Path Contains :\ – Include
• Path Contains \\ – Include
• Path Contains SOFTWARE – Include
• Path Contains SOFTWARE\Microsoft – Exclude (Don’t use for Microsoft programs)
• Excluded Path containing DLLs – Exclude

Here we quickly identified the application loaded a number of .INI files and per-user registry keys.

And they were different on working / broken machine.

However copying the working registry keys and INI files across to the broken machine did not fix the issue…

There was no obvious issue identified with ProcMon accessing any of the registry keys or ini files (i.e. Name Not Found / Access Denied / Sharing Violation)

One other thing I noticed is the process was 64-bit, but the previous version had been 32-bit. As a quick test I used corflags.exe and changed process to 32-bit but it also didn’t resolve issue. (http://msdn.microsoft.com/en-us/library/ms164699(v=vs.110).aspx)

So now we had ruled out the configuration of application itself being the cause of the issue, it was time to look at system configuration differences.

First I used Process Explorer (http://live.sysinternals.com/ProcExp.exe) and used the Lower Pane view set to “DLLs”

I then selected our process on both a working and broken machine and clicked File –> Save and saved the output to a text file.

In the saved text file I manually removed the first half of the text file (Process List) and just left the DLL view. I then used BeyondCompare to compare the DLLs loaded by both processes.  There was one glaring difference. The working version had the following DLLs loaded, the broken version did not:

C:\Windows\assembly\NativeImages_v2.0.50727_64\mscorlib\74a5f0c2bc0d0e6e3c4ec4886b9be891\mscorlib.ni.dll
C:\Windows\assembly\NativeImages_v2.0.50727_64\System.Configuration\ac14913a11af4bfae0b8eb913a46a161\System.Configuration.ni.dll
C:\Windows\assembly\NativeImages_v2.0.50727_64\System.Data\ee5c9facac5c7dbf9c4b1e160f76daae\System.Data.ni.dll
C:\Windows\assembly\NativeImages_v2.0.50727_64\System.Drawing\88f8a6436dc95497fce0dae347646e53\System.Drawing.ni.dllC:\Windows\assembly\NativeImages_v2.0.50727_64\System\f3888a2c7b096d416ca0cfc5405219b4\System.ni.dll
C:\Windows\assembly\NativeImages_v2.0.50727_64\System.Runtime.Remo#\6c3851b925e2a31ddefb3d36bb9163cb\System.Runtime.Remoting.ni.dll
C:\Windows\assembly\NativeImages_v2.0.50727_64\System.Xml\52e2da35b160dbd254683f72a0f1b937\System.Xml.ni.dll

Native Images are generated by the Native Image Service, typically at application installation to improve .NET application performance (http://msdn.microsoft.com/en-us/library/ms165074(v=vs.110).aspx)

With this new found knowledge I decided to try and break my “working system” to do this I went to rename the C:\Windows\assembly\NativeImages_v2.0.50727_64 folder

However I got “Access Denied” This is because the folder was in use by another .NET program, I used Process Explorer Find Handle to find the process responsible and killed it off..

However it kept coming back, it was launched by Visual Studio. I should have just closed Visual Studio, but being the serial killer of processes I am, I went and killed the Process Tree. Living on the edge.

With that done I finally had my folder renamed successfully:

And then… sure enough my “Working system” was now broken!

However this gave us a good clue to a workaround, we could ngen all the relevant DLLs to fix the issue.

Simply running the following commands fixed the issue:

pushd C:\Windows\Microsoft.NET\Framework64\v2.0.50727
ngen Microsoft.VisualBasic.dll
ngen System.Configuration.dll
ngen System.Data.dll
ngen System.Drawing.dll
ngen System.dll
ngen System.Runtime.Remoting.dll
ngen System.Windows.Forms.dll
ngen System.Xml.dll

OK – so that’s a fix. But why this app…why was it breaking…

Using WinDbg to assess a dmp file taken at time of application dialog box via Task Manger, right click process “Create Dump File” along with the SOS debugging extension we were able to run command !pe (print exception)

Exception object: 0000000002f898d0
Exception type: System.Exception
Message: Invalid Base Key
InnerException: <none>
StackTrace (generated):
SP               IP               Function
000000000014D4F0 000007FF001B6CF4 Microsoft_VisualBasic!Microsoft.VisualBasic.ErrObject.Raise(Int32, System.Object, System.Object, System.Object, System.Object)+0x314
000000000014D560 000007FF001AE218 QFBox2!QFBox2.clsRegIni2.Init_Class(System.Object ByRef, System.Object ByRef)+0x2d8
000000000014D670 000007FF001B5E27 QFBox2!QFBox2.frmMain.popRemoteFODetails()+0x1c7

From this we could tell the error came from function clsRegIni2.Init_Class

Inspecting the function with .NET reflector we could see the code that triggered the issue:

public void Init_Class(ref object szBaseKey = null, ref object szINIFileName = null)
{
if (szBaseKey.ToString() == “”)
{
Information.Err().Raise(-2147220989, “RegIni Class”, “Invalid Base Key”, null, null);
}
}

So we new this issue was szBaseKey was getting passed as an empty string. A static analysis of the decompiled EXE found this value was getting set by an INI file. And the INI file had the correct value.

So I wanted to some live step-by-step debugging of the Process. .NET Reflector + Visual Studio Professional or higher make a great choice for this – you will get

• decompiled source code you can step through
• variable watch
• ability to edit variable contents during execution
• intellitrace event / call tracing

To do this I initially launched process via .NET Reflector plugin menu option in Visual Studio Debug an Executable

However when launched via this process – the issue no longer occurred!

To work around this issue I inserted a dummy messagebox in the beginning of the frmMain_Load method using Reflector + Reflexil. To do this I used 3 lines of MSIL:

Note: The pop is required because we are discarding the sSystem.Windows.Forms.DialogResult value returned by the MessageBox.Show method.

With that done I then used Visual Studio and .NET Reflector menu option to “Clear the cache” I then used “Debug an Executable” to debug my patched EXE (which I had to rename to same name as original filename to debug, making a backup of original)

I then launched my “patched” program from Windows Explorer, and waiting for the message box:

I then using Visual Studio Tools –> Attach to Process to connect to my process

You will get a security warning, if you want to continue click Attach again…

Hitting “Pause” I initially got view “Source Not Available”

Then our friend .NET Reflector did it’s thing:

(In my experience this feature often breaks, clearing .NET reflector cache repairs code enabled debugging)

I was then able to use the .NET Reflector Object Browser to browse objects, look at code and use F9 to set breakpoints where I wanted

Through static analysis with .NET reflector I had figured out that mszHKLMPath was the variable that was getting set blank, so I looked for code sections where this variable was modified  and used F9 to set break points there. Once I hit the breakpoint I used F10 to step through code, or when desired F11 to step into a method’s code. I then right clicked the variable and selected option “Add Watch”

a

Then while stepping through code I watched this value waiting for it to get blanked out…

In the end I traced it to the QueryValue function being called here, as soon as this function finished the watched variable became a blank.

One thing we could notice straight away as an issue was the variable was being passed by reference – which meant the function could modify it at will.

So this time I used F11 to “step-into” that section. When entering this function I also added a watch on szKeyName

After the RegOpenKeyEx function completed we saw our szKeyName was blanked out…Because the variable mszHKLMPath had been passed “by reference” it would become blank on exit of the QueryValue function…

It should be noted that all functions throughout the programs liberally used the “by reference” option, most likely this is the legacy of a semi-automated VB6->VB.NET conversion, were strings were passed by reference by default in VB6

However we can see from MSDN definition of RegOpenKeyEx the key name is an “IN” parameter (no OUT) so shouldn’t be passed by reference

An example of correct definition:

To fix this issue I used .NET Reflector + Reflexil to change the DllImport definition from using “by reference” to using the .NET default (by value)

The old way

<DllImport(“advapi32.dll”, EntryPoint:=”RegOpenKeyExA”, CharSet:=CharSet.Ansi, SetLastError:=True, ExactSpelling:=True)> _
Private Function RegOpenKeyEx(ByVal hKey As Integer, <MarshalAs(UnmanagedType.VBByRefStr)> ByRef lpSubKey As String, ByVal ulOptions As Integer, ByVal samDesired As Integer, ByRef phkResult As Integer) As Integer

The fixed way

<DllImport(“advapi32.dll”, EntryPoint:=”RegOpenKeyExA”, CharSet:=CharSet.Ansi, SetLastError:=True, ExactSpelling:=True)> _
Private Function RegOpenKeyEx(ByVal hKey As Integer, <MarshalAs(UnmanagedType.VBByRefStr)> ByVal lpSubKey As String, ByVal ulOptions As Integer, ByVal samDesired As Integer, ByRef phkResult As Integer) As Integer

But WHY???

Besides the obvious – don’t pass by reference, that which should be passed by value

To find out more I created a simple VB.NET console application to test the faulty code. I used VB.NET even though I prefer C# so any findings I could pass onto developers in VB, as that was what they were using.

Here I found something interesting:

Debug / Release Build – Run from Visual Studio

Debug Build – Run From Windows Explorer

Release Build – Run from Windows Explorer

So next in the Visual Studio Project properties I enabled Native debugging

Next I run the release version of my test code from Windows Explorer, then used Debug –> Attach in Visual Studio.

I then hit “Pause” to break into the debugger and selected the RegOpenKeyEx line and hit F9 to set a break point there. Then hit “Run” Then gave the program keyboard input it requested to continue..

Now I opened the disassembly window on our breakpoint:

Stepping through this we could see that when this call

call        000007FF0005C050

Returned the variable was cleared. Unfortunately Visual Studio Didn’t support “Step into” the Disassembly of this function call.

Not to mention I couldn’t use the Memory Window or Set Data Breakpoint .

This is because our process was 64-bit and 64-bit .NET doesn’t support Interop Debugging.

So in Solution I changed my project from target of ANY to that of x86

So I cleared the 32-bit Native Images folder C:\Windows\Assembly\NativeImages_v2.0.50727_32 this type in hope of reproducing our issue, then ran “Release” version from Windows Explorer…and it worked!

• Native Images hadn’t been generated for core .NET assemblies
• 64-bit only
• Release build only, when not started from Visual Studio

So I stopped messing around with Visual Studio and went back to a real debugger, WinDbg. I set a breakpoint on the RegOpenKeyExA function.

I also used the Tool CodeMachine Debugger Extension DLL to ensure I could extract the parameters quickly. As 64-bit code typically passes parameters via registers (rcx, rdx, r8, r9) this dumps the values.

http://www.codemachine.com/tool_cmkd.html

0:000> sxe ld mscorjit
0:000> g
ModLoad: 000007feef240000 000007feef3c4000   C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorjit.dll
ntdll!NtMapViewOfSection+0xa:
000000007779083a c3              ret
0:000> g
Breakpoint 0 hit
000007feff80dec0 4883ec38        sub     rsp,38h
0:000> g
Breakpoint 0 hit
000007feff80dec0 4883ec38        sub     rsp,38h
0:000> !stack -p
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SYSTEM32\MSCOREE.DLL –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\KERNEL32.dll –
Call Stack : 16 frames
Parameter[0] = ffffffff80000001
Parameter[1] = 00000000006a0bd4
Parameter[2] = 0000000000000000
Parameter[3] = 0000000000000001
01 00000000004debe0 000007ff00205b18 mscorwks!IEE+d2df
Parameter[0] = (unknown)
Parameter[1] = (unknown)
Parameter[2] = (unknown)
Parameter[3] = (unknown)
0:000> da 00000000006a0bd4
00000000006a0bd4  “SOFTWARE\COMPANY\QIK-QUBE\BUSINE”
00000000006a0bf4  “SS\”

OK we got the registry value I wanted, so I set a break point on write access…

0:000> ba w 1 00000000006a0bd4
0:000> g
Breakpoint 1 hit
ntdll!RtlpQueryDefaultUILanguage+0x539:
0000000077756f39 4989442408      mov     qword ptr [r12+8],rax ds:00000000006a0bd8=4d4f435c45524157
0:000> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000004de320 00000000777933fd : 0000000000610000 0000000000000001 00000000006a0bc0 00000000006a0bd0 : ntdll!RtlpQueryDefaultUILanguage+0x539
00000000004de660 0000000077809369 : 0000000000000000 0000000000000000 0000000000000000 000007feea42595b : ntdll!RtlAllocateHeap+0xd9d
00000000004de6e0 00000000777cb075 : 0000000000610000 ffffffff50000063 0000000000610000 000007feea3fd762 : ntdll!RtlLogStackBackTrace+0x24d9
00000000004de740 00000000777933fd : 0000000000610000 000000001bb90d01 00000000006a0bc0 00000000006a0bd0 : ntdll!MD5Final+0xd365
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\ole32.dll –
00000000004dea80 000007fefd978e56 : 0000000000000000 00000000006a0bd0 0000000000000000 0000000000000000 : ntdll!RtlAllocateHeap+0xd9d
00000000004deb00 000007feea58e592 : 00000000006a0bd0 000077038037fac3 0000000000000000 0000000002b817c0 : ole32!CoTaskMemFree+0x36
00000000004deb30 000007ff00206cb3 : 00000000004dec60 00000000006a0bd0 0000000000000001 0000000000000002 : mscorwks!IEE+0xd4fa
00000000004debc0 000007ff00205b89 : 00000000006a0bd4 0000000002b817b0 0000000000000000 0000000002b817c0 : 0x7ff00206cb3
00000000004dec90 000007ff001fbd9c : 0000000080000001 00000000004dee00 0000000000000000 0000000000000001 : 0x7ff00205b89
00000000004dedb0 000007feea58eb52 : 000007ff00033528 000007feea42c809 0000000000000000 000007ff000334f8 : 0x7ff001fbd9c
00000000004dee50 000007feea3fdea3 : 0000000000000000 000007fe00000017 000007feea347020 0000000000000000 : mscorwks!IEE+0xdaba
00000000004dee90 000007feea96a841 : 00000000004defc8 0000000000000000 0000000000000001 0000000000000000 : mscorwks!CompareAssemblyIdentity+0x44d3
00000000004def30 000007feea4d9923 : 0000000000000000 000007ff000334f8 0000000000000000 00000000004df430 : mscorwks!PreBindAssembly+0x7c241
00000000004df170 000007feea4fd288 : 0000000000610000 0000000000000000 0000000000000018 0000000000000000 : mscorwks!CreateApplicationContext+0xa9cb
00000000004df3d0 000007feeaa56e4d : 00000000004dfa20 0000000000000000 000000000069e188 0000000000000200 : mscorwks!StrongNameErrorInfo+0x3268
00000000004df6c0 000007feea50a40b : 0000000000000000 0000000000000000 0000000000000000 000007feea526a32 : mscorwks!GetAssemblyIdentityFromFile+0x1960d
00000000004dfc90 000007feea4ed02c : fffffffffffffffe 0000000000000000 00003e5100000000 0000000000000000 : mscorwks!StrongNameErrorInfo+0x103eb
00000000004dfce0 000007fef37e77ad : ffffffffffffffff 000000000068e310 0000000000000000 00000000004dfce8 : mscorwks!CorExeMain+0xac
00000000004dfd40 000007fef3265b21 : 0000000000000000 000007feea4ecf80 0000000000000000 0000000000000000 : mscoreei!CorExeMain+0x5d
00000000004dfd90 000000007753652d : 000007fef37e0000 0000000000000000 0000000000000000 0000000000000000 : MSCOREE!CorExeMain+0x69
0:000> da 00000000006a0bd4
00000000`006a0bd4  “”

Moral of the Story:

Pass by reference in .NET can result in very hard to troubleshoot bugs, please be kind to your software and pass by value.