This all started out on a Windows 7 deployment project very innocuously. A small group of users would occasionally (2-3 times a week, some people 1-2 times a day) try to browse a web page but get the standard Internet Explorer cannot display the webpage. Any amount of refreshing/attempting different addresses would fail, a restart of Internet Explorer was required. According to users this experience was “totally random” and they could not link it to any specific sites.
It turns out the “totally random” part was some misdirection from the users, which I was prepared for. (ref: https://chentiangemalc.wordpress.com/2012/04/09/rules-for-successful-decoding-of-user-terminology/)
Standard testing of IE in no-add ons mode, the issue still occurred. Surely this ruled out 3rd party product….
One challenge with this issue is it occurred so rarely with no known method to reproduce it.
In any case I had a strong desire to find what caused the issue, so I could create a re-producible test.
I set up netsh trace start capture=YES report=YES persistent=YES (ref: https://chentiangemalc.wordpress.com/2012/02/22/netsh-traceuse-it/) to allow for network tracing, even across reboots, left permanently running on users machine, performing constant circular logging of all network packets.
From this tracing I learnt an important fact – as soon as the issue occurred network requests completely died off. The URL the user was attempting to visit, did not even get logged here.
I also learnt another important fact, just prior to HTTP requests from Iexplore.exe dropping from the universe a PDF had been opened…
And so the fun began…
I managed to get a copy of this PDF and set up a test web page. Refresh PDF 3 times… BOOM! Page cannot be displayed…
OK so make PDF open out of browser?
Open 3 times…Boom! Same thing…IE is broken…
Ok, really weird.
One thing I noticed with Process Explorer (http://live.sysinternals.com/ProcExp.exe) is that Iexplore.exe was running at 50% CPU. Using the Threads tab I noticed high CPU usage was caused by repeated calls to WININET.dll!InternetSetStatusCallBackA
Hitting the “stack” button we got this info
Now on MSDN it is documented InternetSetStatusCallBack can fail for several reasons. Using WinDbg to attach to Iexplore.exe I used !gle to find the actual error:
LastErrorValue: (WinSock) 0x276d (10093) – Either the application has not called WSAStartup or WSAStartup failed. The application may be accessing a socket that the current active task does not own (That is, trying to share a socket between tasks), or WSACleanup has been called too many times.
In the debugging stack traces though there was no references to anything about Adobe. To get this I downloaded Application Verifier (http://www.microsoft.com/download/en/details.aspx?id=20028)
Application Verifier can put additional checks on an application to make it crash sooner, to assist finding potential root causes of bugs, that may manifest themselves later on. (For verifying drivers, use Windows in-built verifier.exe)
When you enable verifier on an application you will want to run it through a debugger, such as WinDbg so you can catch the exceptions. When I enabled Application Verifier on Iexplore.exe I re-did my test of refreshing the PDF. First two times went by fine…then Boom! Application Verifier crash. We had a few…
Verifier Crash #1 – APPLICATION_VERIFIER_LEAK_ALLOCATION (900)
A heap allocation was leaked. This stop is generated if the owner dll of the allocation was dynamically unloaded while owning resources. This leak occurred in Adobe Reader plug-in annots.api
The following plugin/DLLs were being unloaded at the time:
5af70000 5b554000 Annots.api
Timestamp: Tue Oct 26 08:26:19 2010 (4CC5F5FB)
5e010000 5e18a000 Multimedia.api
Timestamp: Tue Oct 26 08:28:21 2010 (4CC5F675)
5f780000 5f930000 EScript.api
Timestamp: Tue Oct 26 08:27:27 2010 (4CC5F63F)
13ab0000 13ad6000 SAPForms.api
Timestamp: Mon Jul 18 20:01:46 2011 (4E24048A)
6a0f0000 6a129000 WindowsMedia.mpp
Timestamp: Tue Oct 26 06:43:48 2010 (4CC5DDF4)
69c80000 69cc8000 QuickTime.mpp
Timestamp: Tue Oct 26 06:43:18 2010 (4CC5DDD6)
6ec10000 6ec2a000 MCIMPP.mpp
Timestamp: Tue Oct 26 06:43:18 2010 (4CC5DDD6)
6a130000 6a151000 Flash.mpp
Timestamp: Tue Oct 26 06:43:40 2010 (4CC5DDEC)
69c90000 69cc3000 sqmapi.dll
Timestamp: Sat Nov 20 23:07:40 2010 (4CE7BA0C)
So I then disabled annots.api disabled we got further but still more application verifier stop codes:
Verifier Crash #2 – APPLICATION_VERIFIER_LEAK_ALLOCATION (900)
A heap allocation was leaked. This stop is generated if the owner dll of the allocation was dynamically unloaded while owning resources.
In this case hang is occurring in Adobe Reader plugin IA32.api. Unfortunately this is required for this PDF to work, as this is the plug-in to get internet access. This also references SQLITE in the stack trace after attempting to “create collation”
00000000 00000000 ia32.api!Unknown+0x0
00799424 6ee9e9e5 <Unloaded_IA32.api>+0xe9e5
00799428 6ee9e7cc <Unloaded_IA32.api>+0xe7cc
0079942c 65ba2fb6 sqlite!sqlite3_free+0x1a99
00799430 65bb61a0 sqlite!sqlite3_bind_null+0x2db2
00799434 65bb67e8 sqlite!sqlite3_bind_null+0x33fa
00799438 65bb68cf sqlite!sqlite3_bind_null+0x34e1
0079943c 65bb6b50 sqlite!sqlite3_bind_null+0x3762
00799440 65bbcc7d sqlite!sqlite3_create_collation+0x14d
00799444 65bbd113 sqlite!sqlite3_create_collation+0x5e3
00799448 65bc841d sqlite!sqlite3_exec+0x5e9
0079944c 65bc84a8 sqlite!sqlite3_exec+0x674
00799450 65bc956b sqlite!sqlite3_get_table+0x10b6
00799454 65baa8e8 sqlite!sqlite3_set_auxdata+0x226
00799458 65bc99aa sqlite!sqlite3_get_table+0x14f5
0079945c 65bc9a62 sqlite!sqlite3_get_table+0x15ad
00799460 65bc841d sqlite!sqlite3_exec+0x5e9
00799464 65bc84a8 sqlite!sqlite3_exec+0x674
00799468 65bc956b sqlite!sqlite3_get_table+0x10b6
0079946c 65baa8e8 sqlite!sqlite3_set_auxdata+0x226
00799470 65bc99aa sqlite!sqlite3_get_table+0x14f5
00799474 65bc9a62 sqlite!sqlite3_get_table+0x15ad
00799478 3173a862 AcroRd32!AVAcroALM_Destroy+0x819a
0079947c 3178230f AcroRd32!AcroWinBrowserMain+0x15fed
00799480 317822bb AcroRd32!AcroWinBrowserMain+0x15f99
00799484 31c29247 AcroRd32!PDFLTerm+0x1f2e17
00799488 31acec7f AcroRd32!PDFLTerm+0x9884f
Loaded symbol image file: sqlite.dll
Image path: C:\Program Files\Adobe\Reader 10.0\Reader\sqlite.dll
Image name: sqlite.dll
Timestamp: Tue Oct 26 06:13:33 2010 (4CC5D6DD)
File version: 184.108.40.206
Product version: 220.127.116.11
File flags: 0 (Mask 17)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
ProductName: sqlite Dynamic Link Library
ProductVersion: 9, 0, 0, 1
FileVersion: 1, 0, 0, 1
FileDescription: sqlite Dynamic Link Library
LegalCopyright: Copyright (C) 2008
Continuing further we get another similar type of error:
Verifier Crash #3 – APPLICATION_VERIFIER_LEAK_HANDLE (901)
A HANDLE was leaked. This stop is generated if the owner dll of the handle was dynamically unloaded while owning
resources. This occurs in AdobeXMP.dll – XMP is Adobe’s Extensible Metadata Platform for embedding meta data about a file
I also noticed in tracing:
- WS2_32.dll throwing error: WSAHOST_NOT_FOUND 11001 (0x2AF9)
- mdnsNSP.dll throwing error: WSASERVICE_NOT_FOUND 10108 (0x277C) No such service is known. The service cannot be found in the specified name space.
- WS2_32.dll (winsock) throwing error: WSAEWOULDBLOCK 10035 (0x2733) A non-blocking socket operation could not be completed immediately.
- WinInet.DLL throwing error “ERROR_HTTP_HEADER_NOT_FOUND 12150
The requested header could not be located.
Ok so the question did any PDF cause the issue? The answer through testing was no, it was only certain PDFs. So it was time for PDF Stream Dumper from Sandspite (http://sandsprite.com/blogs/index.php?uid=7&pid=57) to analyse the PDF. Looking through the various objects one stood out – a binary object starting with “CWS” If you love your hex editors you may already know this off the top of your head – it’s a Flash Shockwave object.
Luckily Shockwave objects are typically easily reverse engineered and so I saved the stream as a .swf file and opened in AS3 Sorcerer (Or I could have just hit Decompile…)
It is interesting that issue could never be reproduced with the extracted Flash file alone, despite this flash object being the only real content in the PDF of any value. In addition the Flash files loaded about 10x faster when stand-alone then when embedded in PDF. However customer wanted to pursue the embedded option…
AS3 sorcerer is a must have tool, and very cheap, if you need to analyse other people’s shockwave problems.
Scanning the decompiled ActionScript code I was getting some clues of how this PDF may be impacting IE network connections. Basically the SWF file was dynamically generating graphs and reports based on data it was querying over the network.
The 145,000 lines of decompiled code gave clue as to the creator of the PDF with namespaces of “Xcelsius” which is a SAP product for generating reports. (http://www.sap.com/solutions/sapbusinessobjects/sme/xcelsius/index.epx)
I found at least 10 different web service queries among the code like this (And yes, this really is the password they used):
_local1.okText = “OK”;
_local1.passwordText = “Password”;
_local1.refreshOnLoad = true;
_local1.sSOAPAction = “TransportMonitor/GetReportBlock_TransportPerformance_YTD”;
_local1.sTargetnamespace = “TransportMonitor”;
_local1.sUrlendpoint = “http://pbprod01:8080/dswsbobje/qaawsservices/queryasaservice?&cuid=AZ.L5dQ7rWRNpDAi4VKf0YX&authType=secEnterprise&locale=en&timeout=60&ConvertAnyType=true”;
_local1.systemText = “System”;
_local1.userId = “User Identification”;
_local1.userNameText = “User Name”;
In any case we found reverting from Adobe Reader 10.0 to Adober Reader 9.3 resolved the issue.
Which the customer was fine…I could not reproduce issue at all with Adobe Reader 9.3; while in tandem we opened support case with Adobe to get fix for Adober Reader X …
So Adobe Reader went backwards across the fleet of many machines, after two weeks of successful use in a pilot group of 20 users.
However there was a problem here:
- The pilot group only consisted of the IT group
- A test plan had been written, but not executed
You might now know where this is going…Adobe Reader 9.3 then broke SAP PDF form plug-in (By the way this is my most hated plug-in of my life due to it’s flakiness, and SAP no longer recommend using it, sadly I had no choice here)
The SAP PDF form plug-in is located in C:\Program Files\Adobe Reader\<Reader version>\Reader\plugins\SAPForms.api
(.API files are just DLL files, renamed to .API to be Adobe plugins)
Basically it no longer became possible to do many operations in SAP management web portal – leave/shift changes/terminations/etc, Internet Explorer would just totally crash out.
As this was critical issue we didn’t have time to troubleshoot and an emergency roll-up to 10.0 was required…
This fixed the SAP plugin crashing. However we got a new problem, people could no longer embed PDFs in Word Documents. (Disclaimer: I have never desired, never will, or never plan to embed PDFs in Word documents myself. Not my idea!)
It was at this time Adobe support provided a hot-fix. This required updating Adobe 10.0 to 10.1.2 and installing the hot-fix here:
Hoo-ray! Page cannot displayed issue has disappeared. Great so deploy to a single production machine…Adobe reader will not launch…
A quick check of Report.wer files under %LOCALAPPDATA%\Microsoft\Windows\WER\ReportArchive showed the following info:
Sig.Name=Fault Module Name
Sig.Name=Fault Module Version
Sig.Name=Fault Module Timestamp
DynamicSig.Name=Additional Information 1
DynamicSig.Name=Additional Information 2
DynamicSig.Name=Additional Information 3
DynamicSig.Name=Additional Information 4
UI=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe
UI=Adobe Reader has stopped working
UI=Windows can check online for a solution to the problem.
UI=Check online for a solution and close the program
UI=Check online for a solution later and close the program
UI=Close the program
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.dll
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\AGM.dll
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\CoolType.dll
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\BIB.dll
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\ACE.dll
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\plug_ins\SAPForms.api
AppPath=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe
So what we can gues from this? If app is crashing on launch, as in this cause go up the loaded module backwards for any suspicious items. In this case
LoadedModule=C:\Program Files\Adobe\Reader 10.0\Reader\plug_ins\SAPForms.api
I renamed this file to .bak and Hello – Adobe Reader launched. Through more testing I confirmed this add-in, even after full re-installing SAP and the add-in did not like Adobe Reader 10.1.2 – trying many configurations such as disabling enhanced security & protected mode to no avail. Unfortunately this plug-in is a must for the SAP forms to work…
Downgrading to Adobe Reader 10.0.1 and SAP worked, but we still get “Page Cannot Be Displayed” error.
Thankfully there is a solution – install IE9. IE9 eliminated the page displayed error, and allow us to stay on the 10.0.1 Adobe Reader version that SAP plays happy with.
Now just to test the multitude of poorly designed web applications & add-ins (that cannot be changed/upgraded) with IE9 that abound at this customer.
I’m sure there will be lots more debugging entertainment provided by this customer for months to come.
Moral of the story:
- Don’t put shockwave in PDFs
- Don’t embed PDFs in Word docs
- Don’t use SAP Forms plugin
- Upgrade to IE9!