Obtaining Correct Mscordacwks.dll for .NET WinDbg’ing

If you open a lot of .NET app user mini-dumps you have probably come across this issue:

user32!NtUserWaitMessage+0xa:
00000000`775b933a c3              ret
0:000> .load C:\windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll
0:000> !eestack -ee
Failed to load data access DLL, 0×80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of mscorwks.dll is
                in the version directory
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.
0:000> !sym noisy
noisy mode – symbol prompts on
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscordacwks.dll:2.0.50727.8000 f:0
doesn’t match desired version 2.0.50727.5477 f:0
SYMSRV:  c:\symbols\mscordacwks_AMD64_AMD64_2.0.50727.5477.dll\5265C8EE99d000\mscordacwks_AMD64_AMD64_2.0.50727.5477.dll not found
SYMSRV: 
http://msdl.microsoft.com/download/symbols/mscordacwks_AMD64_AMD64_2.0.50727.5477.dll/5265C8EE99d000/mscordacwks_AMD64_AMD64_2.0.50727.5477.dll not found
CLRDLL: Unable to find mscordacwks_AMD64_AMD64_2.0.50727.5477.dll by mscorwks search
CLRDLL: Unable to find ‘mscordacwks_AMD64_AMD64_2.0.50727.5477.dll’ on the path
DBGHELP: c:\symbols\mscorwks.dll\5265C8EE99d000\mscorwks.dll – OK
CLRDLL: Unable to get version info for ‘c:\symbols\mscorwks.dll\5265C8EE99d000\mscordacwks_AMD64_AMD64_2.0.50727.5477.dll’, Win32 error 0n87
Cannot Automatically load SOS
CLRDLL: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_2.0.50727.5477.dll, Win32 error 0n87
CLR DLL status: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_2.0.50727.5477.dll, Win32 error 0n87

It would be far too easy if Microsoft was kind enough to provide these via the symbol server. Of course you can copy mscordacwks (in this case C:\windows\Microsoft.NET\Framework64\v2.0.50727\mscordacwks.dll ) from the machine dump was generated on, rename it to the file mentioned i.e. mscordacwks_AMD64_AMD64_2.0.50727.5477.dll and put it in a folder that is in your PATH environment variable. In this case of analysing a . NET 4.0 dump on .NET 4.5 you will also need sos.dll from target machine…..

What if you don’t have access to the target machine?

You can do  a search on http://support.microsoft.com for mscordacwks version

image

If you want you can confirm the update has the file you want by checking the File Information section of the KB article.

image

Now we can go to http://catalog.update.microsoft.com and search for the KB number (You must use Internet Explorer, as site uses ActiveX controls) In this case I wanted the x64 version so I added it to the basket and downloaded it

image

Now we extract it. In this case the file was placed into a folder called Security Update for Microsoft .NET Framework 3.5.1 on Windows 7 and Windows Server 2008 R2 SP1 for x64-based Systems (KB2898857) containing the update file AMD64_X86_IA64-all-windows6.1-kb2898857-x64_c6960541776b7cc108be7eab97eaf3fdbb26a666.msu

 

expand –F:* AMD64_X86_IA64-all-windows6.1-kb2898857-x64_c6960541776b7cc108be7eab97eaf3fdbb26a666.msu C:\extract

Make a note of the .CAB file with KB number extracted, in this case Windows6.1-KB2898857-x64.cab

cd \extract

expand –F:* Windows6.1-KB2898857-x64.cab C:\extract

Now you can grab your mscordacwks.dll. Note the folder version numbers don’t match up with the file version number that we are looking for:

image

I check the file version and confirm it is correct one, then rename it

image 

I then copy the file into a directory that is in my PATH environment variable, and we re-run .cordll -ve -u –l

And have success – Loaded DLL.

0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscordacwks.dll:2.0.50727.8000 f:0
doesn’t match desired version 2.0.50727.5477 f:0
SYMSRV:  c:\symbols\mscordacwks_AMD64_AMD64_2.0.50727.5477.dll\5265C8EE99d000\mscordacwks_AMD64_AMD64_2.0.50727.5477.dll not found
SYMSRV: 
http://msdl.microsoft.com/download/symbols/mscordacwks_AMD64_AMD64_2.0.50727.5477.dll/5265C8EE99d000/mscordacwks_AMD64_AMD64_2.0.50727.5477.dll not found
CLRDLL: Unable to find mscordacwks_AMD64_AMD64_2.0.50727.5477.dll by mscorwks search
CLRDLL: Unable to find ‘SOS_AMD64_AMD64_2.0.50727.5477.dll’ on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL mscordacwks_AMD64_AMD64_2.0.50727.5477.dll
CLR DLL status: Loaded DLL mscordacwks_AMD64_AMD64_2.0.50727.5477.dll

Posted in .NET, WinDbg | Leave a comment

Case of the Broken Novell LDAP over SSL + Querying eDirectory via PowerShell

With a legacy Novell Netware environment I wanted to eliminate some manual tasks, to do this I planned to use PowerShell. First I needed to identify a Novell Netware LDAP server to connect to.

To identify a server name I logged onto a machine with Novell Client and right clicked the N icon in System Notification area and selected “Novell Connections…”

image

I then looked for the server with the * and authentication state eDirectory Services

image

I then tried some quick & dirty PowerShell to test LDAP connectivity:

#netware server to query – port 636 for SSL
$NetWareServer=New-Object System.DirectoryServices.Protocols.LdapDirectoryIdentifier("nw-1",636)

# top level place to start
$SearchStart="o=home"

# if username specified must be in DN format cn=chentiangemalc,ou=drouin,ou=aus,ou=global,o=home
$NetwareUser="cn=chentiangemalc,ou=drouin,ou=aus,ou=global,o=home"

# storing passwords in scripts is a bad idea
$NetwarePassword="NCC-1701-d"

[System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols")

$netcred=New-Object System.Net.NetworkCredential($NetwareUser,$NetwarePassword)
$ldap=New-Object System.DirectoryServices.Protocols.LdapConnection($NetwareServer)

# basic auth for novell
$ldap.AuthType=[System.DirectoryServices.Protocols.AuthType]::Basic

if ($NetwarePassword –eq "")
{
    # No Auth
    $ldap.Bind()
}
else
{
    $SessionOptions=$ldap.SessionOptions
    # enables SSL – required if using password in default eDirectory config
    $SessionOptions.SecureSocketLayer=$true

    #accept invalid certs – only uncomment if required
    # $SessionOptions.VerifyServerCertificate = {
    # $MyCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $args[1]
    ## Get the date of cert in case you want to do something with it

    #  $DCCertDate = $MyCert.NotAfter
    # $true
    #}
    $ldap.Bind($netcred)
}

$request=New-Object System.DirectoryServices.Protocols.SearchRequest(
    $SearchStart,
    "(objectClass=Group)",
    [System.DirectoryServices.Protocols.SearchScope]::Subtree,
    "*")

# wait for up to one minute – as this server is slow to respond

$WaitTime=New-Object System.TimeSpan(0,1,0)
$response=$ldap.SendRequest($request,$WaitTime)

"Found $($response.Entries.count) groups under $($SearchStart)"
$response.Entries

 

However this resulted in the following error:

Exception calling “Bind” with “1″ argument(s): “The LDAP server is unavailable.”
At c:\support\NetWareScript.ps1:30 char:11
+ $ldap.Bind <<<< ($netcred)
+ CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
Exception calling “SendRequest” with “1″ argument(s): “The LDAP server returned an unknown error.”
At c:\support\NetWareScript.ps1:39 char:28
+ $response=$ldap.SendRequest <<<< ($request)
+ CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException

 

So I tried running the code without SSL. I changed the value of NetWareServer and NetwarePassword variables:

# use default port (non-SSL)

$NetWareServer=New-Object System.DirectoryServices.Protocols.LdapDirectoryIdentifier("nw-1")

$NetwarePassword=""

 

And everything worked fine..so we worked out issue was related to SSL connection to the LDAP server.

In ConsoleOne I also tried to export LDIF configuration by clicking Wizards | NDS Import/Export Wizard

I selected “Export LDIF File” then clicked Next, filled out other details as necessary, but when export started we got a failure:

ldap_simple_bind failed: 81(Can’t contact LDAP server),

So to diagnose this I searched for the server in ConsoleOne to find the management page, I used the Find button and ensured Search subcontainers was ticked

image

We found our server

image

From here on the General Tab under Network Addresses we could find the ldap and ldaps IP addresses. But today I wanted the portal address…

image

From the portal web page I loaded NDS iMonitor and enabled LDAP and Secure Sockets trace then clicked Trace On

image

I then replicated the issue and clicked “Trace Off”

I then clicked “Trace History” and looked for a trace with the date/time matching my trace and loaded it and we could see a more useful error message:

12:54:10 A69AC440 LDAP: TLS handle allocation failed on connection 0xa66d25c0, setting err = -5873. Error stack:
error:140BA0C3:SSL routines:SSL_new:null ssl ctx

 

This error suggested SSL cert was not configured on the server, so I searched eDirectory again for *ldap*servername and clicked the Server object to open the properties:

image

Looking at server properties we could the Host Server property was blank:

image

The reason for this was because it had pointed to a server that no longer existed. (NW-1a) I hit the browse icon and selected the server we were using for LDAP queries as the Host Server and clicked Apply

Immediately after this change PowerShell was happy with SSL and LDIF Export was working once again in ConsoleOne.

(Note: The SSL/TLS Configuration Page in the LDAP Server Properties may also need to be checked for correct configuration)

Posted in eDirectory, LDAP, Novell, Power, PowerShell | Leave a comment

Case of the KB2919355 Installation Failure (Windows 8.1 Update)

We had already updated a Windows 8.1 x64 machine with KB2919442 and was starting to install Windows8.1-KB2919355-x64.msu.

Firstly it was hanging for ages at the “Installing” phase with no progress…

image

Well that is normal. What is going on here? Using ProcMon (http://live.sysinternals.com/ProcMon.exe) we used the target icon and dragged it over the “Download and Install Updates” Window to monitor activity.

image

However there was no file or registry activity going on.

So I cleared the current filter and used Tools | File Summary… I then looked at the most Active File by Path and clicked it, to display it in ProcMon filter. From this we could see that TiWorker.exe the “Windows Modules Installer Worker” was busy extracting update contents into the following folders:

  • C:\Windows\CbsTemp\
  • C:\Windows\SoftwareDistribution\Download\

For this update this phase can take quite a while due to large size of update file. You will not see any progress bar during the extraction phase.

Based on this info we reset our filter and set to filter on Process Name is TiWorker.exe

image

Once we started to see progress we would see files being written into C:\Windows\WinSxS\Temp\PendingRenames and other locations. (The PendingRenames files are files that need to be installed on reboot)

We also were able to notice with ProcMon that a log file was being written to at

C:\Windows\Logs\CBS\CBS.log

However after some time update failed with this helpful message:

Some updates were not installed

The following updates were not installed:

Update for Windows (KB2919355)

image

Checking the Windows System Event log, and looking at Source WindowsUpdateClient we found a slightly more helpful error message:

Installation Failure: Windows failed to install the following update with error 0×80070003: Update for Windows (KB2919355)

image

The error message 0×80070003 refers to “PATH NOT FOUND” So what was NOT FOUND?

We can find TIWORKER failures by searching our ProcMon log for LastError

In the detail view we can see the Data value has our error message – in decimal instead of HEX. We also see the package name that failed. We also see writing to CBS.log.

image

If we work backwards from that in our ProcMon log we quickly see a pile of PATH NOT FOUND errors:

image

If we checked the CBS.log (you will need to open Notepad or other log viewer as Administrator to open this file –> in some cases you may need to copy it into another folder before you can open it)

2014-04-03 13:55:16, Info CSI 00000e25 Delete of a missing registry value detected in the registry installer!!

Registry value name: “[l:0]“””

2014-04-03 13:55:17, Error CSI 00000e26 (F) STATUS_OBJECT_PATH_NOT_FOUND #8659314# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowSharingViolation), handle = {provider=NULL, handle=0, name= (“null”)}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES), oa = @0x64b7a4d9a8->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[64]“\??\ D:\Users\Default\AppData\Local\Microsoft\Windows\WinX \Group2″; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0x64b7a4d988, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_SYNCHRONOUS_IO_NONALERT|0×00004000), eab = NULL, eal = 0, disp = Invalid)

[gle=0xd000003a]

2014-04-03 13:55:17, Error CSI 00000e27@2014/4/3:02:55:17.825 (F) base\wcp\sil\merged\ntu\ntsystem.cpp(2155): Error STATUS_OBJECT_PATH_NOT_FOUND originated in function Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile expression: (null)

[gle=0x80004005]

2014-04-03 13:55:17, Info CBS Added C:\windows\Logs\CBS\CBS.log to WER report.

2014-04-03 13:55:17, Info CBS Added C:\windows\Logs\CBS\CbsPersist_20140403023755.log to WER report.

2014-04-03 13:55:17, Info CBS Added C:\windows\Logs\CBS\CbsPersist_20140403022331.log to WER report.

2014-04-03 13:55:17, Info CBS Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2014-04-03 13:55:17, Info CBS Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2014-04-03 13:55:17, Info CBS Not able to add SCM.EVM to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2014-04-03 13:55:19, Error CSI 00000e28 (F) STATUS_OBJECT_PATH_NOT_FOUND #8659313# from Windows::Rtl::SystemImplementation::CSystemIsolationLayer_IRtlSystemIsolationLayerTearoff::OpenFilesystemDirectory(flags = 0, da = (FILE_ALL_ACCESS), dn = [ml:520{260},l:128{64}]“\??\D:\Users\Default\AppData\Local\Microsoft\Windows\WinX\Group2″, sa = (FILE_SHARE_READ|FILE_SHARE_DELETE), oo = (FILE_DIRECTORY_FILE|FILE_SYNCHRONOUS_IO_NONALERT|FILE_OPEN_FOR_BACKUP_INTENT), dir = NULL, disp = Invalid)

[gle=0xd000003a]

2014-04-03 13:55:19, Error CSI 00000e29 (F) STATUS_OBJECT_PATH_NOT_FOUND #8124892# from PrimitiveInstaller::CCoordinator::FinalizeChanges(…)[gle=0xd000003a]

2014-04-03 13:55:19, Info CSI 00000402 SMI Primitive Installer [done]

2014-04-03 13:55:19, Error CSI 00000e2a (F) STATUS_OBJECT_PATH_NOT_FOUND #6494085# from CCSDirectTransaction::PerformChangeAnalysis(…)[gle=0xd000003a]

2014-04-03 13:55:19, Error CSI 00000e2b (F) STATUS_OBJECT_PATH_NOT_FOUND #6494084# from CCSDirectTransaction::PrepareForCommit(…)[gle=0xd000003a]

2014-04-03 13:55:19, Error CSI 00000e2c (F) STATUS_OBJECT_PATH_NOT_FOUND #6494083# from CCSDirectTransaction::GenerateComponentChangeList(…)[gle=0xd000003a]

2014-04-03 13:55:19, Error CSI 00000e2d (F) STATUS_OBJECT_PATH_NOT_FOUND #6494082# from Windows::COM::CPendingTransaction::ExtractInformationFromRtlTransaction(…)[gle=0xd000003a]

2014-04-03 13:55:19, Error CSI 00000e2e (F) HRESULT_FROM_WIN32(ERROR_PATH_NOT_FOUND) #1764404# from Windows::COM::CPendingTransaction::IStorePendingTransaction_Analyze(…)[gle=0x80070003]

2014-04-03 13:55:20, Error CSI 00000e2f (F) HRESULT_FROM_WIN32(ERROR_PATH_NOT_FOUND) #1626198# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_Commit(Flags = 47 (0x0000002f), pSink = NULL, disp = 0, coldpatching = FALSE)[gle=0x80070003]

2014-04-03 13:55:20, Error CSI 00000e30 (F) HRESULT_FROM_WIN32(ERROR_PATH_NOT_FOUND) #1626197# 313613458 us from

 

So why was D:\Users\Default folder missing?

This is because the User Profiles has been redirected using a registry key, instead of unattend.xml – Refer to “Relocation of the Users directory and the ProgramData directory to a drive other than the drive that contains the Windows directory” at http://support.microsoft.com/kb/949977 

To summarize this KB article:

  • Using ProfilesDirectory to redirect folders to a drive other than the system volume blocks upgrades.
  • Using ProfilesDirectory to point to a directory that is not the system volume will block SKU upgrades and upgrades to future versions of Windows.
  • Microsoft recommends that you do not change the location of the user profile directories or program data folders.
  • Note If you use the unattend settings to set up the operating systems that are listed in this article, we will provide commercially reasonable efforts to support your scenario.

So to fix the machine we ran:

robocopy C:\Users\Default\ D:\Users\Default\ *.* /s

To prevent these type of issues in the future however it is best to keep ProfilesDirectory on the system volume (i.e. C:\ ) This issue may not have occurred if the unattend settings had been used to put the profile on D:\

After creating a copy of default profile under D:\Users\Default we re-ran Windows8.1-KB2919355-x64.msu and updated installed without an issue

image

Some other issues with this Windows update error message are also covered under these KB articles:

Windows Update error “0×80070002″ or “0×80070003″

http://support.microsoft.com/kb/910336

Error message when you try to install updates by using the Windows Update or Microsoft Update Web site: “0×80070003″

http://support.microsoft.com/kb/956705/id-id

Posted in ProcMon, SysInternals, Troubleshooting, Windows 8, Windows 8.1 | 16 Comments

Case of the Office 365 “Something Went Wrong”

Intermittently in SharePoint Online the Office 365 toolbar would just not appear. In other cases it would appear but Outlook Web Access would get stuck at logon and eventually fail with message “something went wrong” 

image

This problem only occurred to users on the corporate network, and was not affecting users connecting to the sites externally.

Looking at a WireShark capture we could see traffic was going “direct” instead of via proxy.

We were also able to reproduce the issue while running Fiddler http://www.telerik.com/fiddler

With Fiddler we were able to identify easily:

image

In the details view fiddler provided this information:

image

Looking up these IP addresses we could find they were part of the Akamai content delivery network, used by Microsoft.

C:\Users\chentiangemalc>nslookup 184.87.112.47
Server:  UnKnown
Address:  10.211.55.1

Name:    a184-87-112-47.deploy.static.akamaitechnologies.com
Address:  184.87.112.47

C:\Users\chentiangemalc>

Checking proxy configuration we noticed a PAC file in use. We tested with proxy set manually to server and port number and the site worked. So we needed to check the PAC file. At first glance it appeared the PAC file should have resolved these URLs to Proxy

/* Externally hosted sites to be proxied */

if (dnsDomainIs(host, “cdn.sharepointonline.com”

|| dnsDomainIs(host, “.outlook.com”))
{
    return ProxyChoice(MyIP,host);
}

The problem here is don’t assume the comment “to be proxied” actually means they are going to be proxied. As we can see it is returning the value of a function ProxyChoice.

Using my PacDbg tool here http://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/ I was able to quickly figure out why it was going direct…

We can see within ProxyChoice function there was a statement that would return DIRECT

image

These statements had been added in to improve performance of heavy utilized site by bypassing the proxy.

Why did it work most of the time going DIRECT?

This is because the firewall had rules to let out HTTP/S traffic to documented Office 365 IP addresses http://office.microsoft.com/en-us/office365-sharepoint-online-enterprise-help/sharepoint-online-urls-and-ip-addresses-HA102772748.aspx

However the host names cdn.sharepointonline.com and r4.res.outlook.com are load balanced sites that are constantly returning different IP addresses. Whenever there was a failure it was because the IP address was not permitted “direct” due to firewall rules. 

Adding the new IPs to Firewall fixed the issue, but this risked breaking again when new IPs were introduced, so the PAC file was modified to ensure these went via proxy.

Posted in Fiddler, Internet Explorer | Leave a comment

Finding “Prevent Performance of First Run Wizard” in Windows 8.1 Group Policy

I’ve seen a number of people struggle to find the group policy to disable Internet Explorer’s first run Wizard in Windows 8.1. Many people across the internets have suggested to set the policy registry key via Group Policy preferences.

With a policy key in existence though we should be able to find the Group Policy setting. I like to use a fast & free search utility from NirSoft called SearchMyFiles http://www.nirsoft.net/utils/search_my_files.html (Note: This site might be blocked on some corporate networks due to the “hacking” tools available from this site)

I search C:\Windows\PolicyDefinitions folder for contents that includes part of the registry key I’m looking for. In this case I searched for DisableFirstRun which we can see is found in inetres.admx

image  

We then search the relevant ADMX for the registry value name we are looking for. We will use the displayName NoFirstRunCustomise to search the relevant .adml file.

image 

Next we must open the relevant language file – this will be in a sub folder based on the language you are using. In this case we are using US English so we open \en-US\inetres.adml

So we find the display name…and yes it has been changed to Prevent Running First Run wizard. Which makes a whole lot more sense “Prevent Performance of…” but also breaks all existing documentation on the internets regarding this policy…

image

And sure enough it is there…

image

Posted in Uncategorized | Leave a comment

Case of the Unknown Error App Crash – Debugging & Patching Someone Else’s .NET Race Condition

We were testing an application Supply Chain Guru purchased by a customer, however on launch it we would see a splash screen for a second, then fail with dialog message box:

image

Clicking OK on the message box would quit the application.

Some interesting things about this issue:

  • Issue only occurred when laptop was connected to corporate network
  • Did not occur when connected to a 4G network
  • If you launched the app many times in a row, in rare cases it would work
  • Application always launched when disconnected from network, or connected to a network other than the corporate LAN
  • The issue did not occur if you launched application via Debugger (cdb/WinDbg) and hit “g” then on initial crash hit .restart and hit “g”again
  • Issue does not occur on a “clean install” OS (with no other 3rd party software
  • The issue would not occur if ProcMon is running

When issues don’t occur with ProcMon running, often we have a bug that is called a “race condition” or “race hazard” This is when the program relies on a certain sequence and timing of uncontrollable events…and turn into a bug when they execute out of the sequence the programmer intended.

For application crashes it is always wise to check the Application event log at first, as you often get very detailed information immediately just from there.  Sure enough we get a stack trace of the crash:

Application: SCGuru.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
Stack:
   at LLamasoft.SupplyChainGuru.Core.cErrorLog.WriteMessage(System.String, Boolean)
   at LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary()
   at LLamasoft.SupplyChainGuru.Core.HelpManager._Lambda$__118(System.Object)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

This was provided to the vendor application support, but they were a bit lost. So we are going to need to help them out…

From this we can see two things:

1) From the System.Threading.IThreadPoolWorkItem.ExecuteWorkItem we have a worker thread – this confirms our suspicions of a race condition

2) A NullReferenceException occurs in function LLamasoft.SupplyChainGuru.Core.cErrorLog.WriteMessage function.

NullReferenceException in a .NET program occurs when attempting to access a property or call a method on an uninitialized object. Using .NET reflector let’s look at this function:

public static void WriteMessage(string strMessage, bool IncludeSystemData = true)
{

    GuruInformation.Instance.LogFactory.WriteLogMessage(SourceLevels.Verbose, strMessage, null, IncludeSystemData);
}

Just looking at this line of code, we can guess the most likely cause of a NullReferenceException is that GuruInformation.Instance.LogFactory has not been initialized, so attempt to access WriteLogMessage method results in a NullReferenceException.

Using .NET reflector we are also able to find the code that initializes LogFactory. In LLamasoft.SupplyChainGuru.WinClient.UserContext, within Initialize() function:

GuruInformation.Instance.InitializeLogging<LogFactory>(GuruModel.get_TemporaryModelFolder(), new Func<string>(cErrorLog.GetSystemData));

I had got a crash dump generated by an onsite engineer at time issue occurred using http://live.sysinternals.com/procdump.exe

We opened this in WinDbg (x86) from the Windows SDK. Because this is a .NET app we first load the sos debugging extension, however this failed:

0:015> .loadby sos clr
0:015> !eestack
Failed to load data access DLL, 0×80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

 

This issue is caused because you cannot debug .NET 4.0 apps with .NET 4.5.x installed, you will need

1) symbols configured to Microsoft symbol servers (i.e. run .symfix C:\symbols) and ensure mscorwks.dll comes down from MS Symbol Servers (You can verify with !sym noisy; .reload /f )

2) A copy of SOS DLL from a .NET 4.0 machine C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll then load it with the full file path (If there is space in the file path don’t add quotes!)

0:015> .unload sos
Unloading C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos extension DLL
0:015> .load C:\support\sos_4.0.dll

I used !EEstack to display all managed threads. We search for cErrorLog.WriteMessage

Which we find:

<blah blah blah>

Thread  11
Current frame: user32!NtUserWaitMessage+0×15
ChildEBP RetAddr  Caller, Callee

<blah blah blah>

System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean)), calling 52eab2a0
0be4cc04 5356844b (MethodDesc 52d21fec +0x23b System.Windows.Forms.MessageBox.ShowCore(System.Windows.Forms.IWin32Window, System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean)), calling 52eab2a0
0be4cc28 53568391 (MethodDesc 52d21fec +0×181 System.Windows.Forms.MessageBox.ShowCore(System.Windows.Forms.IWin32Window, System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean)), calling user32!GetActiveWindow
0be4cc98 535686d9 (MethodDesc 52d21f68 +0×19 System.Windows.Forms.MessageBox.Show(System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon)), calling (MethodDesc 52d21fec +0 System.Windows.Forms.MessageBox.ShowCore(System.Windows.Forms.IWin32Window, System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon, System.Windows.Forms.MessageBoxDefaultButton, System.Windows.Forms.MessageBoxOptions, Boolean))
0be4ccb8 08a8bb95 (MethodDesc 00dd5ef8 +0xf5 LLamasoft.SupplyChainGuru.WinClient.CustomExceptionHandler.OnDomainThreadException(System.Object, System.UnhandledExceptionEventArgs)), calling (MethodDesc 52d21f68 +0 System.Windows.Forms.MessageBox.Show(System.String, System.String, System.Windows.Forms.MessageBoxButtons, System.Windows.Forms.MessageBoxIcon))
<blah blah>
0be4dc84 08a8ba41 (MethodDesc 06b03f04 +0×31 LLamasoft.SupplyChainGuru.Core.cErrorLog.WriteMessage(System.String, Boolean)), calling 001d6852
0be4dcf0 08a8b5ab (MethodDesc 00ed05fc +0x1bb LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary()), calling (MethodDesc 06b03f04 +0

So then I set active thread to 11:

0:011> ~11 s

looked for the exceptions with the nested option:

0:011> !pe -nested
Exception object: 0345fc60
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    0BE4CD54 08A8C353 SCGuru!LLamasoft.SupplyChainGuru.WinClient.UserContext.get_LogFilePath()+0×13
    0BE4CD5C 08A8C311 UNKNOWN!LLamasoft.SupplyChainGuru.Core.cErrorLog.get_ErrorLogLocation()+0×31
    0BE4CD68 08A8BB15 SCGuru!LLamasoft.SupplyChainGuru.WinClient.CustomExceptionHandler.OnDomainThreadException(System.Object, System.UnhandledExceptionEventArgs)+0×75

StackTraceString: <none>
HResult: 80004003

Nested exception ————————————————————-
Exception object: 03459d50
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    0BE4DCDC 08A8BA3B UNKNOWN!LLamasoft.SupplyChainGuru.Core.cErrorLog.WriteMessage(System.String, Boolean)+0x2b
    0BE4DCF8 08A8B5AB UNKNOWN!LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary()+0x1bb
    0BE4ECF0 08A8B038 UNKNOWN!LLamasoft.SupplyChainGuru.Core.HelpManager._Lambda$__118(System.Object)+0×8
    0BE4ECF4 554FDA21 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)+0x2d
    0BE4ECFC 55484D85 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0×51
    0BE4F198 55484C8A mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x6a
    0BE4F1B0 55487F92 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x7e
    0BE4F1D4 554CB692 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+0x5a
    0BE4F1E8 554CAF1F mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0×147
    0BE4F234 554CADC5 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()+0x2d

StackTraceString: <none>
HResult: 80004003

Nested exception ————————————————————-
Exception object: 03459870
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    0BE4ECB8 08A8B479 UNKNOWN!LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary()+0×89

StackTraceString: <none>
HResult: 80004003

0:000>

Here we can see the original exception is occurring in LoadTooltipHelpDictionary() … Let’s look at what this function is. First I get the method descriptor of the function uses !name2ee (If we know the module name you can place it instead of the *)

0:011> !name2ee *!LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary
<blah blah blah>
——
Module:      00399b30
Assembly:    SupplyChainGuru.LowerCore.dll
Token:       06000a47
MethodDesc:  00ed05fc
Name:        LLamasoft.SupplyChainGuru.Core.HelpManager.LoadTooltipHelpDictionary()
JITTED Code Address: 08a8b3f0
———————————
<blah blah blah>

Now we can dump the IL code….

0:011> !dumpil 00ed05fc
ilAddr = 059440b8
IL_0000: ldarg.0
IL_0001: callvirt LLamasoft.SupplyChainGuru.Core.HelpManager::ClearTooltipHelpDictionary
.try
{
  IL_0006: newobj System.Data.DataTable::.ctor
  IL_000b: stloc.0
  .try
  {
    IL_000c: call LLamasoft.Data.DataSources.DataManager::get_Instance
    IL_0011: call LLamasoft.SupplyChainGuru.Core.CGuruLowerCore::get_Instance
    IL_0016: callvirt LLamasoft.SupplyChainGuru.Core.CGuruLowerCore::get_UserContext
    IL_001b: callvirt LLamasoft.SupplyChainGuru.Core.iUserContext::get_HelpFileDB
    IL_0020: callvirt LLamasoft.Data.DataSources.DataManager::CreateConnection
    IL_0025: stloc.1

<blah blah blah>
    } // end .try

} // end .try
.catch
{
  IL_008d: dup
  IL_008e: call Microsoft.VisualBasic.CompilerServices.ProjectDat::SetProjectError
  IL_0093: stloc.3
  IL_0094: ldstr “Failed to load help tooltips.”
  IL_0099: ldc.i4.1
  IL_009a: call LLamasoft.SupplyChainGuru.Core.cErrorLog::WriteMessage
  IL_009f: call Microsoft.VisualBasic.CompilerServices.ProjectDat::ClearProjectError
  IL_00a4: leave.s IL_00a6
} // end .catch
IL_00a6: ret

 

Looking at the code we can guess the attempt to create the connection to the HelpFileDB (a local MDB file, used to popular tool tips in the application) Is failing with a Null Reference Exception, resulting in hitting the “Catch” section of code, then attempting to write the error log. Which also fails due to a null reference exception…

So let’s inspect objects on the stack. NullReferenceException seems to be popular today :) However the object that interests me is the HelpManager that gets references in above code…

0:011> !DumpStackObjects
OS Thread Id: 0x20dc (11)
ESP/REG  Object   Name
0BE4CBDC 0345a4ac System.String    Fatal Error
0BE4CBE0 0345a4ac System.String    Fatal Error
0BE4CBF0 0345a4ac System.String    Fatal Error
0BE4CBF4 0345a4ac System.String    Fatal Error
0BE4CC44 0345a4ac System.String    Fatal Error
0BE4CC90 0345a038 System.String    An unexpected application error has occurred in one of the application’s
threads and has left the application in an unstable state.

The application is terminating.

Information about the problem has been written to the application error log at the following location:

0BE4CCB4 0345a4ac System.String    Fatal Error
0BE4CD98 03459fd8 System.UnhandledExceptionEventArgs
0BE4CE40 02b21338 System.AppDomain
0BE4CE44 02b524c0 System.UnhandledExceptionEventHandler
0BE4CE70 03459fd8 System.UnhandledExceptionEventArgs
0BE4D064 02b524c0 System.UnhandledExceptionEventHandler
0BE4D06C 02b21338 System.AppDomain
0BE4D074 03459fd8 System.UnhandledExceptionEventArgs
0BE4D0D0 03459fd8 System.UnhandledExceptionEventArgs
0BE4D0D8 03459d50 System.NullReferenceException
0BE4D174 03459fd8 System.UnhandledExceptionEventArgs
0BE4D1E8 02b524c0 System.UnhandledExceptionEventHandler
0BE4D2B0 02b21338 System.AppDomain
0BE4D3BC 03459d50 System.NullReferenceException
0BE4D568 03459e60 System.Runtime.Serialization.SafeSerializationManager
0BE4D9D8 03459d50 System.NullReferenceException
0BE4DA3C 03459d50 System.NullReferenceException
0BE4DA5C 03459d50 System.NullReferenceException
0BE4DB00 03459d50 System.NullReferenceException
0BE4DB34 03459d50 System.NullReferenceException
0BE4DB78 03459d50 System.NullReferenceException
0BE4DBC4 034536d8 System.String    Failed to load help tooltips.
0BE4DBC8 0345972c LLamasoft.Data.DataSources.DataManager
0BE4DCAC 034536d8 System.String    Failed to load help tooltips.
0BE4DCB4 0345972c LLamasoft.Data.DataSources.DataManager
0BE4DCE4 034536d8 System.String    Failed to load help tooltips.
0BE4DCE8 02b21228 System.String   
0BE4DCEC 03453740 System.Data.DataTable
0BE4DD08 03459870 System.NullReferenceException
0BE4DD1C 03459870 System.NullReferenceException
0BE4DD20 0345972c LLamasoft.Data.DataSources.DataManager
0BE4DD2C 02b21228 System.String   
0BE4DD30 03453740 System.Data.DataTable
0BE4E040 03459870 System.NullReferenceException
0BE4E054 03459870 System.NullReferenceException
0BE4E680 03459870 System.NullReferenceException
0BE4E77C 03453740 System.Data.DataTable
0BE4E784 02b21228 System.String   
0BE4E8F0 03453740 System.Data.DataTable
0BE4E8F4 02b21228 System.String   
0BE4E8F8 0345972c LLamasoft.Data.DataSources.DataManager
0BE4E8FC 02b21228 System.String   
0BE4EC7C 02b21228 System.String   
0BE4EC80 0345972c LLamasoft.Data.DataSources.DataManager
0BE4EC88 02b21228 System.String   
0BE4EC8C 03453740 System.Data.DataTable
0BE4EC9C 0345972c LLamasoft.Data.DataSources.DataManager
0BE4ECA0 03453740 System.Data.DataTable
0BE4ECAC 02b21228 System.String   
0BE4ECC0 03453740 System.Data.DataTable
0BE4ECC4 03448728 LLamasoft.SupplyChainGuru.Core.HelpManager

We can take the 2nd column value to inspect object:

0:011> !do 03448728
Name:        LLamasoft.SupplyChainGuru.Core.HelpManager
MethodTable: 00ed0698
EEClass:     00e69734
Size:        20(0×14) bytes
File:        C:\Program Files (x86)\Supply Chain Guru\SupplyChainGuru.LowerCore.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
00ed08e4  4000312        4 …nGuru.LowerCore]]  0 instance 0344873c _tooltipHelpDictionary
65a07728  4000313        8 System.Drawing.Image  0 instance 03452bbc _helpIcon
00ed0efc  4000314        c …Core.RoboHelp_CSH  0 instance 03453100 mRoboHelp
00ed0698  4000311       60 ….Core.HelpManager  0   static 03448728 _instance

 

Next we want to inspect the values of the _tooltipHelpDictionary using the MT & Field values from above:

0:011> !dumpvc 00ed08e4 4000312
Name:        System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[LLamasoft.SupplyChainGuru.Core.TooltipHelpResource, SupplyChainGuru.LowerCore]]
MethodTable: 00ed08e4
EEClass:     552a99ac
Size:        52(0×34) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
55572a94  4000c7d        0       System.Int32[]  0 instance 00000000 buckets
55d0e378  4000c7e        4 …non, mscorlib]][]  0 instance 00000000 entries
55572ad4  4000c7f       1c         System.Int32  1 instance        0 count
55572ad4  4000c80       20         System.Int32  1 instance        0 version
55572ad4  4000c81       24         System.Int32  1 instance        0 freeList
55572ad4  4000c82       28         System.Int32  1 instance        0 freeCount
55573134  4000c83        8 …Canon, mscorlib]]  0 instance 00000000 comparer
555657e8  4000c84        c …Canon, mscorlib]]  0 instance 00000000 keys
5556b88c  4000c85       10 …Canon, mscorlib]]  0 instance 00000000 values
5556f744  4000c86       14        System.Object  0 instance 00000000 _syncRoot
5556b8f0  4000c87       18 …SerializationInfo  0 instance 00000000 m_siInfo

 

What we confirm, here is that – this dictionary didn’t populate at all. So the exception must have occurred at connection to database…

By searching in .NET Reflector we found this HelpManager gets initialized when UserContext gets initialized.

image 

We find the thread in our dump with !eestack

0:00>!eestack
<blah blah blah>
Thread   9
Current frame: ntdll!ZwQueryFullAttributesFile+0×12
ChildEBP RetAddr  Caller, Callee
06c1e77c 75889bdf KERNELBASE!GetFileAttributesExW+0×93, calling ntdll!ZwQueryFullAttributesFile
06c1e7ac 68c0cf99 clr!WKS::GCHeap::SetCardsAfterBulkCopy+0×16
06c1e7b8 68c1152c clr!SystemNative::ArrayCopy+0×282
06c1e7c0 68c0da0b clr!SystemNative::ArrayCopy+0x28f, calling clr!Thread::CatchAtSafePoint
06c1e800 555100e2 (MethodDesc 552d29cc +0×52 DomainNeutralILStubClass.IL_STUB_PInvoke(System.String, Int32, WIN32_FILE_ATTRIBUTE_DATA ByRef))
06c1e824 555100e2 (MethodDesc 552d29cc +0×52 DomainNeutralILStubClass.IL_STUB_PInvoke(System.String, Int32, WIN32_FILE_ATTRIBUTE_DATA ByRef))
06c1e828 758774ec KERNELBASE!GetErrorMode+0×18, calling ntdll!ZwQueryInformationProcess
06c1e838 75877547 KERNELBASE!SetErrorMode+0×37, calling ntdll!ZwSetInformationProcess
06c1e858 5548770b (MethodDesc 552c3388 +0xa7 System.IO.File.FillAttributeInfo(System.String, WIN32_FILE_ATTRIBUTE_DATA ByRef, Boolean, Boolean)), calling 554294bc
06c1e874 554876ce (MethodDesc 552c3388 +0x6a System.IO.File.FillAttributeInfo(System.String, WIN32_FILE_ATTRIBUTE_DATA ByRef, Boolean, Boolean)), calling kernel32!SetErrorModeStub
06c1e894 55507c29 (MethodDesc 55385f14 +0×39 System.Security.Util.StringExpressionSet.AddExpressions(System.Collections.ArrayList, Boolean)), calling clr!JIT_WriteBarrierESI
06c1e8ec 554a9067 (MethodDesc 5533d5b4 +0×33 System.IO.DirectoryInfo.get_Exists()), calling (MethodDesc 552c3388 +0 System.IO.File.FillAttributeInfo(System.String, WIN32_FILE_ATTRIBUTE_DATA ByRef, Boolean, Boolean))
06c1e91c 08a81cfd (MethodDesc 05486dec +0x7d LLamasoft.SupplyChainGuru.WinClient.UserContext.CheckFolderSettingSaved(System.String, System.String))
06c1e934 05e633a5 (MethodDesc 05486610 +0×335 LLamasoft.SupplyChainGuru.WinClient.UserContext.Initialize()), calling (MethodDesc 05486dec +0 LLamasoft.SupplyChainGuru.WinClient.UserContext.CheckFolderSettingSaved(System.String, System.String))
06c1e954 00c9362a (MethodDesc 054865c4 +0x3a LLamasoft.SupplyChainGuru.WinClient.UserContext.get_Instance()), calling (MethodDesc 05486610 +0 LLamasoft.SupplyChainGuru.WinClient.UserContext.Initialize())
 

0:00>~9 s

0:009> !dumpstackobjects
OS Thread Id: 0x23a0 (9)
ESP/REG  Object   Name
06C1E7A4 03118adc System.Object[]    (System.String[])
06C1E7CC 031189b0 System.Object[]    (System.String[])
06C1E848 031187b0 System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E84C 031187b0 System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E87C 03118af0 System.Collections.ArrayList
06C1E880 03118af0 System.Collections.ArrayList
06C1E88C 03118b50 System.Collections.ArrayList
06C1E890 03118b38 System.Security.Util.StringExpressionSet
06C1E8A8 031187b0 System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E8E0 031187b0 System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E8E4 03118868 System.IO.DirectoryInfo
06C1E8E8 02eec9dc System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E910 031187b0 System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E914 02d121dc System.String    BackupFolder
06C1E918 02eec9dc System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E924 03118868 System.IO.DirectoryInfo
06C1E928 02d128f0 LLamasoft.SupplyChainGuru.GuruInformation
06C1E92C 02b5c95c LLamasoft.SupplyChainGuru.WinClient.UserContext <- This is the object we want
06C1E930 02d121dc System.String    BackupFolder
06C1E93C 02eec9dc System.String    \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Guru Model Backups
06C1E940 02d128f0 LLamasoft.SupplyChainGuru.GuruInformation
06C1E980 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1E9B8 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1EA1C 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1EA40 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1EB20 02b4aaec System.Windows.Forms.Control+ControlNativeWindow
06C1ED68 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1ED74 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1ED7C 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1ED84 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1ED8C 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1EE0C 02b4a648 LLamasoft.SupplyChainGuru.WinClient.FormSplash
06C1EE58 02b520a0 System.Windows.Forms.Application+ThreadContext
06C1EE88 02c16280 System.Reflection.RuntimeMethodInfo
06C1EE98 02b51fb0 System.Windows.Forms.ApplicationContext
06C1EEBC 02c16280 System.Reflection.RuntimeMethodInfo
06C1F364 02b51e88 System.Runtime.CompilerServices.RuntimeHelpers+TryCode
06C1F368 02b51ea8 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
06C1F36C 02b51ec8 System.Threading.ExecutionContext+ExecutionContextRunData
06C1F3A0 02c16048 System.String    LeftInternal
06C1F3B4 02b51d74 System.Threading.ThreadHelper
06C1F3B8 02c16048 System.String    LeftInternal
06C1F3C4 02c16048 System.String    LeftInternal
06C1F3D8 02b51d74 System.Threading.ThreadHelper
06C1F668 02b28fb8 System.Security.Principal.WindowsPrincipal
06C1F670 02b51d88 System.Threading.ThreadStart
06C1F678 02b51d44 System.Threading.Thread

Now we dump the UserContext object…

0:009> !do 02b5c95c
Name:        LLamasoft.SupplyChainGuru.WinClient.UserContext
MethodTable: 057e1888
EEClass:     05542700
Size:        136(0×88) bytes
File:        C:\Program Files (x86)\Supply Chain Guru\SCGuru.exe
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
5556fb08  400067d        4        System.String  0 instance 02d12ffc defaultRoutingProvider
5556fb08  400067f        8        System.String  0 instance 02b21228 m_strApplicationVersion
5556fb08  4000680        c        System.String  0 instance 02b21228 m_strApplicationTitle
5556fb08  4000681       10        System.String  0 instance 02b21228 m_strApplicationName
5556fb08  4000682       14        System.String  0 instance 02b21228 m_strDataPath
5556fb08  4000683       18        System.String  0 instance 02b21228 m_strImagePath
5556fb08  4000684       1c        System.String  0 instance 02b21228 m_strSplashImagePath
5556fb08  4000685       20        System.String  0 instance 02b21228 m_strMapIconPath
5556fb08  4000686       24        System.String  0 instance 02b21228 m_strBaseModelFilePath
5556fb08  4000687       28        System.String  0 instance 02b21228 m_strDefaultCostingDataPath
5556fb08  4000688       2c        System.String  0 instance 02b21228 m_strDataStorePath
5556fb08  4000689       30        System.String  0 instance 03117ac8 m_strMacroPath
5556fb08  400068a       34        System.String  0 instance 02b21228 m_strTimerLogPath
00000000  400068b       38                       0 instance 00000000 m_SqlServerDatabase
00000000  400068c       3c                       0 instance 00000000 m_sqlExpress
5556fb08  400068d       40        System.String  0 instance 02b21228 m_strHelpDBFileName <- this is what we are interested in
5556fb08  400068e       44        System.String  0 instance 02b21228 m_strHelpFileName
55576788  400068f       80       System.Boolean  1 instance        0 m_UseProxyServer
5556fb08  4000690       48        System.String  0 instance 02b21228 m_ProxyServerAddress
5556fb08  4000691       4c        System.String  0 instance 02b21228 m_ProxyServerPort
55576788  4000692       81       System.Boolean  1 instance        0 m_ProxyUseIntegratedSecurity
5556fb08  4000693       50        System.String  0 instance 02b21228 m_ProxyUserName
5556fb08  4000694       54        System.String  0 instance 02b21228 m_ProxyUserPassword
5556fb08  4000695       58        System.String  0 instance 02b21228 m_ProxyUserDomain
55576788  4000696       82       System.Boolean  1 instance        0 m_UseCustomAgentString
5556fb08  4000698       5c        System.String  0 instance 02b21228 m_BaseHelpLocalWebPage
5556fb08  4000699       60        System.String  0 instance 02b21228 m_WebURL
5556fb08  400069b       64        System.String  0 instance 00000000 _enterpriseBackupFolder
55576788  400069c       83       System.Boolean  1 instance        0 _enterpriseServerSqlIntegratedSecurity
5556fb08  400069d       68        System.String  0 instance 03117f94 m_strDefaultProjectFolder
5556fb08  400069e       6c        System.String  0 instance 02b7cbf4 m_strDefaultEnterpriseFolder
5556fb08  400069f       70        System.String  0 instance 02b82c70 m_strTempPackageFolder <- we are just using this as a test
5556fb08  40006a0       74        System.String  0 instance 02c597a0 m_strCustomMapLayers
5556fb08  40006a1       78        System.String  0 instance 02b21228 _enterpriseServerWebPassword
5556fb08  40006a2       7c        System.String  0 instance 00000000 _enterpriseDataFolder
057e1888  400069a       54 …lient.UserContext  0   static 02b5c95c m_instance

 

As a test I checked contents of m_strTempPackageFolder – contents look OK.

0:009> !do 02b82c70 

Name:        System.String
MethodTable: 5556fb08
EEClass:     552a8bb0
Size:        172(0xac) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      \\somecompany.local\filestore\MEL-Users\chentiangemalc\Documents\LLamasoft\Temp\Packages
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
55572ad4  4000103        4         System.Int32  1 instance       79 m_stringLength
55571f24  4000104        8          System.Char  1 instance       5c m_firstChar
5556fb08  4000105        8        System.String  0   shared   static Empty
    >> Domain:Value  00fd29b8:02b21228 <<

Now we check what we’re interested in m_strHelpDBFilename. It is empty. Note all the strings above with value of “02b21228” are empty

0:009> !do 02b21228
Name:        System.String
MethodTable: 5556fb08
EEClass:     552a8bb0
Size:        14(0xe) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:     
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
55572ad4  4000103        4         System.Int32  1 instance        0 m_stringLength
55571f24  4000104        8          System.Char  1 instance        0 m_firstChar
5556fb08  4000105        8        System.String  0   shared   static Empty
>> Domain:Value  00fd29b8:02b21228 <<

 

So why is this value not initialized, before use. We need to ask – where is the initialization? I used Visual Studio 2013 (you need professional or higher) with .NET Reflector option “Debug an Executable” and traced it to CGuruAppUI.Instance.Initialize. This allowed me to step-by-step inspect code, set break points, change variables, etc, and I could use F11 “Step-into” as needed. For live debugging a .NET app it is much easier for many tasks than WinDbg. However debugging live with WinDbg is more stable, and doesn’t require Visual Studio Professional. If you need to use WinDbg for live debugging .NET apps I recommend you also use the sosex extension from http://www.stevestechspot.com/SOSEXV40NowAvailable.aspx as it is much easier to set breakpoints vs sos.(among other features).

image

 

Right clicking the method call we can decompile it:

image

 

This brought us to this code, here we see QueueUserWorkItem used. Based on MSDN documentation “http://msdn.microsoft.com/en-us/library/system.threading.threadpool.queueuserworkitem(v=vs.110).aspx” The method executes “when the thread pool becomes available.” Sounds like prime cause of our “race condition”

 

public void Initialize()
        {
            if (CGuruLowerCore.Instance.ApplicationType == ApplicationType.SupplyChainGuru)
            {
                System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(this._Lambda$__118));
                if (this.mRoboHelp == null)
                {
                    this.mRoboHelp = new RoboHelp_CSH();
                }
                this.mRoboHelp.RH_AssociateOfflineHelp(@”http:\\supplychainguru.com\help.htm”, CGuruLowerCore.Instance.UserContext.BaseHelpLocalWebPage);
            }
        }

So with .NET Reflector + Reflexil ( http://reflexil.net/ ) I patched the binary, by adding a Sleep for 5 seconds after the CGuruAppUIInstance.Initialize function, to allow the work item to run. (5 seconds was over kill, but I was erring on safe side…)

Using reflixl I inserted the IL commands:

idc.i4 5000

call System.Void.System.Threading.Thread::Sleep(System.Int32)

When adding the “Call” method just keep in mind the browser for method is based on DLL not namespace. So do not browse to System.Threading but instead Mscorlib –> System.Threading

 

image

 

I was then able to save patched code. Because this was an EXE it was pretty straightforward.

Warning: .NET DLL patching is more of a pain, as you need to use the  “Strong Name Remover” feature of reflexil, which I have seen corrupt EXEs before…take a backup first, as strong name remover overwrites your existing files.

image

I closed, and reloaded the code in .NET reflector to confirm my IL code was correct.

image ware

Testing the application, the “Unknown Error” no longer occurred. Keep in mind I don’t recommend binary patching as a general practice for 3rd party software in production use. However it is very useful to prove cause of bug when vendor can’t resolve the issue. We can now tell vendor exactly where issue is…

Posted in Uncategorized | Leave a comment

.NET 4.5.1 and SqlConnection.Open() = an error occurred during the login process

Windows 7 x64 machines had just been upgraded from .NET 4.0 to .NET 4.5.1 via KB2858728 (http://support.microsoft.com/kb/2858728) which offers the following benefits:

  • Performance and reliability improvements
  • The ability to suspend and resume operations in the ASP.NET page framework
  • The ability to compact large object heaps on-demand
  • 64-bit Edit and Continue controls
  • The ability to trace and sample activities
  • Improvements that make SQL connections more resilient
  • The ability to manage return values

 

After the upgrade it was no longer possible to connect to remote SQL servers using code like this:

using (var conn = new SqlConnection(“<my connection string>”))
{
    conn.Open();
}

 

The conn.Open() started throwing the following exception:

A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 – The message received was unexpected or badly formatted.)

The call stack looked like this:

System.Data.dll!System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(System.Data.SqlClient.ServerInfo serverInfo = {unknown}, string newPassword = {unknown}, System.Security.SecureString newSecurePassword = {unknown}, bool redirectedUserInstance = {unknown}, System.Data.SqlClient.SqlConnectionString connectionOptions = {unknown}, System.Data.SqlClient.SqlCredential credential = {unknown}, System.Data.ProviderBase.TimeoutTimer timeout = {unknown}) 

System.Data.dll!System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(System.Data.ProviderBase.TimeoutTimer timeout = {unknown}, System.Data.SqlClient.SqlConnectionString connectionOptions = {unknown}, System.Data.SqlClient.SqlCredential credential = {unknown}, string newPassword = {unknown}, System.Security.SecureString newSecurePassword = {unknown}, bool redirectedUserInstance = {unknown}) 

System.Data.dll!System.Data.SqlClient.SqlInternalConnectionTds..ctor(System.Data.ProviderBase.DbConnectionPoolIdentity identity = {unknown}, System.Data.SqlClient.SqlConnectionString connectionOptions = {unknown}, System.Data.SqlClient.SqlCredential credential = {unknown}, object providerInfo = {unknown}, string newPassword = {unknown}, System.Security.SecureString newSecurePassword = {unknown}, bool redirectedUserInstance = {unknown}, System.Data.SqlClient.SqlConnectionString userConnectionOptions = {unknown}, System.Data.SqlClient.SessionData reconnectSessionData = {unknown}) 

System.Data.dll!System.Data.SqlClient.SqlConnectionFactory.CreateConnection(System.Data.Common.DbConnectionOptions options = {unknown}, System.Data.Common.DbConnectionPoolKey poolKey = {unknown}, object poolGroupProviderInfo = {unknown}, System.Data.ProviderBase.DbConnectionPool pool = {unknown}, System.Data.Common.DbConnection owningConnection = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown})  

System.Data.dll!System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(System.Data.ProviderBase.DbConnectionPool pool = {unknown}, System.Data.Common.DbConnection owningObject = {unknown}, System.Data.Common.DbConnectionOptions options = {unknown}, System.Data.Common.DbConnectionPoolKey poolKey = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown})  

System.Data.dll!System.Data.ProviderBase.DbConnectionPool.CreateObject(System.Data.Common.DbConnection owningObject = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}, System.Data.ProviderBase.DbConnectionInternal oldConnection = {unknown})     

System.Data.dll!System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(System.Data.Common.DbConnection owningObject = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}, System.Data.ProviderBase.DbConnectionInternal oldConnection = {unknown})     

System.Data.dll!System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection owningObject = {unknown}, uint waitForMultipleObjectsTimeout = {unknown}, bool allowCreate = {unknown}, bool onlyOneCheckConnection = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}, out System.Data.ProviderBase.DbConnectionInternal connection = {unknown}) 

System.Data.dll!System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection owningObject = {unknown}, System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}, out System.Data.ProviderBase.DbConnectionInternal connection = {unknown})  

System.Data.dll!System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(System.Data.Common.DbConnection owningConnection = {unknown}, System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}, System.Data.ProviderBase.DbConnectionInternal oldConnection = {unknown}, out System.Data.ProviderBase.DbConnectionInternal connection = {unknown})  

System.Data.dll!System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(System.Data.Common.DbConnection outerConnection = {unknown}, System.Data.ProviderBase.DbConnectionFactory connectionFactory = {unknown}, System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}) 

System.Data.dll!System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(System.Data.Common.DbConnection outerConnection = {unknown}, System.Data.ProviderBase.DbConnectionFactory connectionFactory = {unknown}, System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown}, System.Data.Common.DbConnectionOptions userOptions = {unknown}) 

System.Data.dll!System.Data.SqlClient.SqlConnection.TryOpenInner(System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown})     

System.Data.dll!System.Data.SqlClient.SqlConnection.TryOpen(System.Threading.Tasks.TaskCompletionSource<System.Data.ProviderBase.DbConnectionInternal> retry = {unknown})     

     System.Data.dll!System.Data.SqlClient.SqlConnection.Open() 

 

.NET 2.0 or NET 3.5 applications were not affected, only those targeting .NET 4.0. A quick workaround was to reset the Winsock catalog to a clean state:

netsh winsock reset

Note: All Winsock Layered Service Providers (LSPs) which were previously installed must be reinstalled. This command does not affect Winsock Name Space Provider entries.

However this workaround was only temporary – as 3rd party program repaired it’s LSPs the issue returned.

To find the provider at fault we listed all installed providers with command

netsh winsock show catalog

 

This provided the following output:

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [TCP/IP]]
Provider ID:                        {F9C047C7-FADC-42A0-90AE-078C7315BC69}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1013
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×66
Protocol Chain Length:              2
Protocol Chain: 1011 : 1001

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [UDP/IP]]
Provider ID:                        {B2E091CB-AB6F-4F98-B035-989156056E11}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1014
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×609
Protocol Chain Length:              2
Protocol Chain: 1011 : 1002

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [TCP/IPv6]]
Provider ID:                        {71B909D0-BF67-4AF0-8140-C4E47594A7C2}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1015
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×66
Protocol Chain Length:              2
Protocol Chain: 1011 : 1004

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [UDP/IPv6]]
Provider ID:                        {24FA2D3C-9977-4E79-9B3A-1B00A42ABB09}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1016
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×609
Protocol Chain Length:              2
Protocol Chain: 1011 : 1005

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [RSVP TCPv6 Service Provider]
Provider ID:                        {4D9EB2FF-D675-49E7-B6F7-F10B3D7373B0}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1017
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×2066
Protocol Chain Length:              2
Protocol Chain: 1011 : 1007

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [RSVP TCP Service Provider]
Provider ID:                        {C033EAC6-EB01-4382-B011-ADDEAE64D85C}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1018
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×2066
Protocol Chain Length:              2
Protocol Chain: 1011 : 1008

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [RSVP UDPv6 Service Provider]
Provider ID:                        {4A94DB27-6A7A-43D5-9CFC-E31B1084A271}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1019
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×2609
Protocol Chain Length:              2
Protocol Chain: 1011 : 1009

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [RSVP UDP Service Provider]
Provider ID:                        {7E4D08FF-5C59-494A-9C95-5BA057E88660}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1020
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×2609
Protocol Chain Length:              2
Protocol Chain: 1011 : 1010

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [TCP/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1001
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×20066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [UDP/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1002
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [RAW/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1003
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        3
Protocol:                           0
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [TCP/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1004
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×20066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [UDP/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1005
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        MSAFD Tcpip [RAW/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1006
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        3
Protocol:                           0
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        RSVP TCPv6 Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1007
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×22066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        RSVP TCP Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1008
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×22066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        RSVP UDPv6 Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1009
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×22609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider
Description:                        RSVP UDP Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1010
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×22609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Service Provider
Description:                        Sophos Web Intelligence LSP
Provider ID:                        {8BFC85A0-09DF-4E9D-8C89-DC87DA833CC8}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1011
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        0
Protocol:                           0
Service Flags:                      0×66
Protocol Chain Length:              0

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [TCP/IP]]
Provider ID:                        {39745C7F-E028-4E95-8B97-2A94500BDBE7}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1021
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×66
Protocol Chain Length:              2
Protocol Chain: 1012 : 1001

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [UDP/IP]]
Provider ID:                        {FED72BC6-20D5-432F-A8FA-AB4B7530552E}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1022
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×609
Protocol Chain Length:              2
Protocol Chain: 1012 : 1002

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [TCP/IPv6]]
Provider ID:                        {7B1C62BF-7D03-4216-A5A8-594D7287063C}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1023
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×66
Protocol Chain Length:              2
Protocol Chain: 1012 : 1004

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [UDP/IPv6]]
Provider ID:                        {D150F7F8-0E12-4B53-BF9A-FB492C36917E}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1024
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×609
Protocol Chain Length:              2
Protocol Chain: 1012 : 1005

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [RSVP TCPv6 Service Provider]
Provider ID:                        {84128345-9F2A-46DB-AB7F-8DEC9A93F660}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1025
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×2066
Protocol Chain Length:              2
Protocol Chain: 1012 : 1007

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [RSVP TCP Service Provider]
Provider ID:                        {E4DFA87C-B750-4735-A35F-A60F33EC98CC}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1026
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×2066
Protocol Chain Length:              2
Protocol Chain: 1012 : 1008

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [RSVP UDPv6 Service Provider]
Provider ID:                        {967D92B8-44E0-487B-8187-0CE634565A9D}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1027
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×2609
Protocol Chain Length:              2
Protocol Chain: 1012 : 1009

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Chain Entry (32)
Description:                        Sophos Web Intelligence LSP over [RSVP UDP Service Provider]
Provider ID:                        {060CA60F-4D41-4B7D-9DCF-EACC385F3639}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1028
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×2609
Protocol Chain Length:              2
Protocol Chain: 1012 : 1010

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [TCP/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1001
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×20066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [UDP/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1002
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [RAW/IP]
Provider ID:                        {E70F1AA0-AB8B-11CF-8CA3-00805F48A192}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1003
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        3
Protocol:                           0
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [TCP/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1004
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×20066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [UDP/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1005
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        MSAFD Tcpip [RAW/IPv6]
Provider ID:                        {F9EAB0C0-26D4-11D0-BBBF-00AA006C34E4}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1006
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        3
Protocol:                           0
Service Flags:                      0×20609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        RSVP TCPv6 Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1007
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×22066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        RSVP TCP Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1008
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×22066
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        RSVP UDPv6 Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1009
Version:                            2
Address Family:                     23
Max Address Length:                 28
Min Address Length:                 28
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×22609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Base Service Provider (32)
Description:                        RSVP UDP Service Provider
Provider ID:                        {9D60A9E0-337A-11D0-BD88-0000C082E69A}
Provider Path:                      %SystemRoot%\system32\mswsock.dll
Catalog Entry ID:                   1010
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        2
Protocol:                           17
Service Flags:                      0×22609
Protocol Chain Length:              1

Winsock Catalog Provider Entry
——————————————————
Entry Type:                         Layered Service Provider (32)
Description:                        Sophos Web Intelligence LSP
Provider ID:                        {8BFC85A0-09DF-4E9D-8C89-DC87DA833CC8}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp.dll
Catalog Entry ID:                   1012
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        0
Protocol:                           0
Service Flags:                      0×66
Protocol Chain Length:              0

Name Space Provider Entry
——————————————————
Description:                        Network Location Awareness Legacy (NLAv1) Namespace
Provider ID:                        {6642243A-3BA8-4AA6-BAA5-2E0BD71FDD83}
Name Space:                         15
Active:                             1
Version:                            0

Name Space Provider Entry
——————————————————
Description:                        E-mail Naming Shim Provider
Provider ID:                        {964ACBA2-B2BC-40EB-8C6A-A6DB40161CAE}
Name Space:                         37
Active:                             1
Version:                            0

Name Space Provider Entry
——————————————————
Description:                        PNRP Cloud Namespace Provider
Provider ID:                        {03FE89CE-766D-4976-B9C1-BB9BC42C7B4D}
Name Space:                         39
Active:                             1
Version:                            0

Name Space Provider Entry
——————————————————
Description:                        PNRP Name Namespace Provider
Provider ID:                        {03FE89CD-766D-4976-B9C1-BB9BC42C7B4D}
Name Space:                         38
Active:                             1
Version:                            0

Name Space Provider Entry
——————————————————
Description:                        Tcpip
Provider ID:                        {22059D40-7E9E-11CF-AE5A-00AA00A7112B}
Name Space:                         12
Active:                             1
Version:                            0

Name Space Provider Entry
——————————————————
Description:                        NTDS
Provider ID:                        {3B2637EE-E580-11CF-A555-00C04FD8D4AC}
Name Space:                         32
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        Network Location Awareness Legacy (NLAv1) Namespace
Provider ID:                        {6642243A-3BA8-4AA6-BAA5-2E0BD71FDD83}
Name Space:                         15
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        E-mail Naming Shim Provider
Provider ID:                        {964ACBA2-B2BC-40EB-8C6A-A6DB40161CAE}
Name Space:                         37
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        PNRP Cloud Namespace Provider
Provider ID:                        {03FE89CE-766D-4976-B9C1-BB9BC42C7B4D}
Name Space:                         39
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        PNRP Name Namespace Provider
Provider ID:                        {03FE89CD-766D-4976-B9C1-BB9BC42C7B4D}
Name Space:                         38
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        Tcpip
Provider ID:                        {22059D40-7E9E-11CF-AE5A-00AA00A7112B}
Name Space:                         12
Active:                             1
Version:                            0

Name Space Provider Entry (32)
——————————————————
Description:                        NTDS
Provider ID:                        {3B2637EE-E580-11CF-A555-00C04FD8D4AC}
Name Space:                         32
Active:                             1
Version:                            0

 

From this we can find our culprit – The only 3rd party providers installed here are all from Sophos. Uninstalling Sophos fixed the issue – however being the firewall / anti-virus software on the machine that was not a permanent solution. We could also fix the issue by uninstalling .NET 4.5.1 and re-installing .NET 4.0 and 4.5 would resolve the issue.

This is documented in Microsoft KB article “Applications crash and “AccessViolationException” exception occurs when you use System.Data.SqlClient after you install Visual Studio 2013 or .NET Framework 4.5.1” http://support.microsoft.com/kb/2915689 

Which is also listed as known issues for .NET Framework 4.5.1 http://support.microsoft.com/kb/2890857 

As per the article :

This problem occurs because some non-IFS Winsock Base Service Providers (BSPs) or Layered Service Providers (LSPs) that are installed on the system intercept and change the incoming and outgoing network traffic. Therefore, when the application connects to SQL Server, these BSPs or LSPs interfere with the calls to Winsock

1.    If the Service Flags value has the 0×20000 bit set, this indicates that the provider uses IFS handles and works correctly.
2.    If the 0×20000 bit is clear (not set), this indicates that the provider is a non-IFS BSP or LSP.

We can straight away see that the following LSP doesn’t have this bit set:

Entry Type:                         Layered Chain Entry
Description:                        Sophos Web Intelligence LSP over [MSAFD Tcpip [TCP/IP]]
Provider ID:                        {F9C047C7-FADC-42A0-90AE-078C7315BC69}
Provider Path:                      C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Catalog Entry ID:                   1013
Version:                            2
Address Family:                     2
Max Address Length:                 16
Min Address Length:                 16
Socket Type:                        1
Protocol:                           6
Service Flags:                      0×66
Protocol Chain Length:              2
Protocol Chain: 1011 : 1001

 

To resolve this issue you will need to remove the non-IFS compliant LSPs, or disable the relevant component in the application that installed it. If you need to keep the LSP you will need vendor to make it IFS compliant, or revert to older .NET framework patch level.

LSPs are a DLL that use Winsock APIs to insert themselves into the TCP/IP stack. Once in the stack they can modify inbound/outbound internet traffic and are commonly used by software to filter web traffic or implement parental controls, or by malware to block Windows Updates :)

Non-IFS LSPs are the most common type currently in use. non-IFS LSPs modify the socket handle to a non valid Windows IFS handle and therefore the LSP must implement all Winsock 2 methods. IFS LSPs, on the other hand, preserve the socket handle, which allows the LSP to implement only the functions it wants to intercept.

Further reading:

Understand the Mysteries of Writing a Winsock 2 Layered Service Provider

http://www.microsoft.com/msj/0599/layeredservice/layeredservice.aspx

The (new) Trouble with Select and LSPs

http://blogs.msdn.com/b/wndp/archive/2006/07/13/664737.aspx

The Darkside of Winsock (exploiting it)

http://hisown.com/Talks/spi.pdf

IFS vs non IFS LSPs

http://www.komodia.com/KomodiaLSPTypes.pdf

Categorizing Layered Service Providers and Applications

http://msdn.microsoft.com/en-us/library/windows/desktop/bb513664(v=vs.85).aspx

Posted in .NET, Windows 7 | Tagged | Leave a comment