Case of the Office Hang on Launch

After a file server migration all Microsoft Office products started to hang on launch for over a minute.

It was immediately observed that uninstalling Office File Validation fixed the launch, however it had been installed for over a year and used without any issue.

Several Excel hang dumps were created with ProcDump (http://live.sysinternals.com/ProcDump.exe )

So we looked at the DMP file with WinDbg from Windows SDK:

Running a simple “Display Stack Backtrace” command was enough to show us the cause:

0:000> k
ChildEBP RetAddr 
0014b85c 77a26a24 ntdll!KiFastSystemCallRet
0014b860 75d9179c ntdll!NtWaitForSingleObject+0xc
0014b8cc 7727c2f3 KERNELBASE!WaitForSingleObjectEx+0x98
0014b8e4 71fd7261 kernel32!WaitForSingleObjectExImplementation+0x75
0014b918 71fd6ca6 cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x1a3 <- responsible for hang
0014ba08 75bd7f27 cryptnet!CryptRetrieveObjectByUrlW+0xce
0014ba38 75be1fc6 crypt32!ChainRetrieveObjectByUrlW+0x3e
0014ba74 75c4f8f3 crypt32!CCertChainEngine::RetrieveAuthRootAutoUpdateObjectByUrlW+0x5f
0014bb00 75c5098a crypt32!CCertChainEngine::WireRetrieveAutoUpdateCab+0xd2
0014bb50 75bf931a crypt32!CCertChainEngine::GetTimeValidDisallowedCertAutoUpdateCtl+0xb9
0014bb80 75bc3a42 crypt32!CCertChainEngine::GetChainContext+0x3f
0014bbb8 75d66245 crypt32!CertGetCertificateChain+0x72
0014bc20 75d66094 wintrust!_WalkChain+0x1ca
0014bc5c 75d636a5 wintrust!WintrustCertificateTrust+0xba
0014bd78 75d626a2 wintrust!I_IsUnsignedPEFile+0x8c3
0014bd94 03425b1b wintrust!WinVerifyTrust+0x52
0014be38 7316ec9d mscorsec!GetPublisher+0xe4
0014be90 72f23fb3 mscorwks!PEFile::CheckSecurity+0xcb
0014beb8 72f23efc mscorwks!PEAssembly::DoLoadSignatureChecks+0x3a
0014bee0 72f24342 mscorwks!PEAssembly::PEAssembly+0x109
0014c17c 72f2443d mscorwks!PEAssembly::DoOpen+0x103
0014c210 72f1e29b mscorwks!PEAssembly::Open+0x79
0014c374 72edc856 mscorwks!AppDomain::BindAssemblySpec+0x247
0014c3dc 72eead55 mscorwks!AssemblySpec::LoadDomainAssembly+0x114
0014c400 72f10913 mscorwks!AssemblySpec::LoadAssembly+0x1d
0014c54c 7244123c mscorwks!AssemblyNative::Load+0x240
0014c574 7243fc50 mscorlib_ni+0x23123c
0014c5a8 7243fe1f mscorlib_ni+0x22fc50
0014c5cc 723b8c74 mscorlib_ni+0x22fe1f
0014c5e8 723b8c3a mscorlib_ni+0x1a8c74
0014c614 723c80e2 mscorlib_ni+0x1a8c3a
0014c624 72fdf468 mscorlib_ni+0x1b80e2
0014c73c 72fdf573 mscorwks!COMToCLRWorkerBody+0x1de
0014c798 72fdf9d9 mscorwks!COMToCLRWorkerDebuggerWrapper+0x37
0014c97c 0317a0d1 mscorwks!COMToCLRWorker+0x52f
WARNING: Frame IP not in any known module. Following frames may be wrong.
0014c9a4 100014ac 0x317a0d1
0014ca10 10001753 TDCExcel30+0x14ac <— loading this add-in triggers CRL check
0014ca54 10005109 TDCExcel30+0x1753
0014ca68 10002490 TDCExcel30+0x5109
0014ca7c 778a8ca6 TDCExcel30+0x2490

We can see the hang is being cause excel Add-in TDCExcel30.dll triggering a check for Certificate Revocation List (CRL)

due to cryptnet!CryptRetrieveObjectByUrlWithTimeout

So disabling TDCExcel30.dll also fixed the hang…but in this case, this add-in wasn’t to blame…

Running kv command in WinDbg we got the arguments for the function:

0:000> kv
ChildEBP RetAddr  Args to Child             
0014b85c 77a26a24 75d9179c 00000568 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0014b860 75d9179c 00000568 00000000 0014b8a8 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
0014b8cc 7727c2f3 00000568 00003a98 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0014b8e4 71fd7261 00000568 00003a98 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0014b918 71fd6ca6 03a3ff88 00000000 04205004 cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x1a3 (FPO: [Non-Fpo])
etc

We took the first args to child and dumped the unicode string at that location:

0:000> du 03a3ff88
03a3ff88  “
http://ctldl.windowsupdate.com/m”
03a3ffc8  “sdownload/update/v3/static/trust”
03a40008  “edr/en/disallowedcertstl.cab”

Testing the URL we found it worked on working machines, but URL didn’t work on broken machines.

When running Fiddler (http://www.fiddler2.com) we saw result 502

image

And in “TextView” we saw:

[Fiddler] The connection to ‘ctldl.windowsupdate.com’ failed. <br />Error: TimedOut (0x274c). <br />System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 125.56.204.123:80               

So first thing I checked was proxy settings. A PAC file was in use, so I switched to configure a direct proxy – and issue was gone.

This proved PAC file was an issue.

To find why PAC file caused an issue I launched my PacDbg tool (https://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/)

By default PacDbg will load current PAC File. I then put in URL and hit “Run”

image

The error in PAC file was immediately obvious, a change had been secretly made to PAC file and an equal sign was dropped from the file.

Inserting an EQUAL sign, PAC file worked correctly

image

Change made to PAC file, and Office started loading in <1 sec instead of 1 minute.

Note: The file server migration was just a red herring, it just happened the same time as somebody also changed the PAC file outside the change control process. (And when asked ‘Anybody change anything recently?’ the relevant person was silent)

About chentiangemalc

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in Debugging, Office, PacDbg, WinDbg and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s