An Epic Battle of Adobe vs IE vs SAP

This all started out on a Windows 7 deployment project very innocuously. A small group of users would occasionally (2-3 times a week, some people 1-2 times a day) try to browse a web page but get the standard Internet Explorer cannot display the webpage. Any amount of refreshing/attempting different addresses would fail, a restart of Internet Explorer was required. According to users this experience was “totally random” and they could not link it to any specific sites.

image

It turns out the “totally random” part was some misdirection from the users, which I was prepared for. (ref: https://chentiangemalc.wordpress.com/2012/04/09/rules-for-successful-decoding-of-user-terminology/)

Standard testing of IE in no-add ons mode, the issue still occurred. Surely this ruled out 3rd party product….

One challenge with this issue is it occurred so rarely with no known method to reproduce it. 

In any case I had a strong desire to find what caused the issue, so I could create a re-producible test.

I set up netsh trace start capture=YES report=YES persistent=YES (ref: https://chentiangemalc.wordpress.com/2012/02/22/netsh-traceuse-it/) to allow for network tracing, even across reboots, left permanently running on users machine, performing constant circular logging of all network packets.

From this tracing I learnt an important fact – as soon as the issue occurred network requests completely died off. The URL the user was attempting to visit, did not even get logged here.

I also learnt another important fact, just prior to HTTP requests from Iexplore.exe dropping from the universe a PDF had been opened…

And so the fun began…

I managed to get a copy of this PDF and set up a test web page. Refresh PDF 3 times… BOOM! Page cannot be displayed…

Interesting.

OK so make PDF open out of browser?

Open 3 times…Boom! Same thing…IE is broken…

Ok, really weird.

One thing I noticed with Process Explorer (http://live.sysinternals.com/ProcExp.exe) is that Iexplore.exe was running at 50% CPU.  Using the Threads tab I noticed high CPU usage was caused by repeated calls to WININET.dll!InternetSetStatusCallBackAimage

Hitting the “stack” button we got this info

image

Now on MSDN it is documented InternetSetStatusCallBack can fail for several reasons. Using WinDbg to attach to Iexplore.exe I used !gle to find the actual error:

0:000> !gle
LastErrorValue: (WinSock) 0x276d (10093) – Either the application has not called WSAStartup or WSAStartup failed. The application may be accessing a socket that the current active task does not own (That is, trying to share a socket between tasks), or WSACleanup has been called too many times.

In the debugging stack traces though there was no references to anything about Adobe. To get this I downloaded Application Verifier (http://www.microsoft.com/download/en/details.aspx?id=20028)

Application Verifier can put additional checks on an application to make it crash sooner, to assist finding potential root causes of bugs, that may manifest themselves later on. (For verifying drivers, use Windows in-built verifier.exe)

When you enable verifier on an application you will want to run it through a debugger, such as WinDbg so you can catch the exceptions. When I enabled Application Verifier on Iexplore.exe I re-did my test of refreshing the PDF. First two times went by fine…then Boom! Application Verifier crash. We had a few…

Verifier Crash #1 – APPLICATION_VERIFIER_LEAK_ALLOCATION (900)

A heap allocation was leaked. This stop is generated if the owner dll of the allocation was dynamically unloaded while owning resources.  This leak occurred in Adobe Reader plug-in annots.api

The following plugin/DLLs were being unloaded at the time:

5af70000 5b554000   Annots.api
    Timestamp: Tue Oct 26 08:26:19 2010 (4CC5F5FB)
    Checksum:  00000000
    ImageSize:  005E4000
5e010000 5e18a000   Multimedia.api
    Timestamp: Tue Oct 26 08:28:21 2010 (4CC5F675)
    Checksum:  00000000
    ImageSize:  0017A000
5f780000 5f930000   EScript.api
    Timestamp: Tue Oct 26 08:27:27 2010 (4CC5F63F)
    Checksum:  00000000
    ImageSize:  001B0000
13ab0000 13ad6000   SAPForms.api
    Timestamp: Mon Jul 18 20:01:46 2011 (4E24048A)
    Checksum:  0002B5FD
    ImageSize:  00026000
6a0f0000 6a129000   WindowsMedia.mpp
    Timestamp: Tue Oct 26 06:43:48 2010 (4CC5DDF4)
    Checksum:  000385D6
    ImageSize:  00039000
69c80000 69cc8000   QuickTime.mpp
    Timestamp: Tue Oct 26 06:43:18 2010 (4CC5DDD6)
    Checksum:  000464BB
    ImageSize:  00048000
6ec10000 6ec2a000   MCIMPP.mpp
    Timestamp: Tue Oct 26 06:43:18 2010 (4CC5DDD6)
    Checksum:  00021733
    ImageSize:  0001A000
6a130000 6a151000   Flash.mpp
    Timestamp: Tue Oct 26 06:43:40 2010 (4CC5DDEC)
    Checksum:  0001E85B
    ImageSize:  00021000
69c90000 69cc3000   sqmapi.dll
    Timestamp: Sat Nov 20 23:07:40 2010 (4CE7BA0C)
    Checksum:  00035EE9
    ImageSize:  00033000

So I then disabled annots.api disabled we got further but still more application verifier stop codes:

Verifier Crash #2 – APPLICATION_VERIFIER_LEAK_ALLOCATION (900)
A heap allocation was leaked. This stop is generated if the owner dll of the allocation was dynamically unloaded while owning resources.

In this case hang is occurring in Adobe Reader plugin IA32.api. Unfortunately this is required for this PDF to work, as this is the plug-in to get internet access. This also references SQLITE in the stack trace after attempting to “create collation”

DEFAULT_BUCKET_ID:  APPLICATION_HANG_HungIn_LoaderLock

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_HungIn_LoaderLock

STACK_TEXT: 
00000000 00000000 ia32.api!Unknown+0x0

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ia32.api!Unknown  
00799424  6ee9e9e5 <Unloaded_IA32.api>+0xe9e5
00799428  6ee9e7cc <Unloaded_IA32.api>+0xe7cc
0079942c  65ba2fb6 sqlite!sqlite3_free+0x1a99
00799430  65bb61a0 sqlite!sqlite3_bind_null+0x2db2
00799434  65bb67e8 sqlite!sqlite3_bind_null+0x33fa
00799438  65bb68cf sqlite!sqlite3_bind_null+0x34e1
0079943c  65bb6b50 sqlite!sqlite3_bind_null+0x3762
00799440  65bbcc7d sqlite!sqlite3_create_collation+0x14d
00799444  65bbd113 sqlite!sqlite3_create_collation+0x5e3
00799448  65bc841d sqlite!sqlite3_exec+0x5e9
0079944c  65bc84a8 sqlite!sqlite3_exec+0x674
00799450  65bc956b sqlite!sqlite3_get_table+0x10b6
00799454  65baa8e8 sqlite!sqlite3_set_auxdata+0x226
00799458  65bc99aa sqlite!sqlite3_get_table+0x14f5
0079945c  65bc9a62 sqlite!sqlite3_get_table+0x15ad
00799460  65bc841d sqlite!sqlite3_exec+0x5e9
00799464  65bc84a8 sqlite!sqlite3_exec+0x674
00799468  65bc956b sqlite!sqlite3_get_table+0x10b6
0079946c  65baa8e8 sqlite!sqlite3_set_auxdata+0x226
00799470  65bc99aa sqlite!sqlite3_get_table+0x14f5
00799474  65bc9a62 sqlite!sqlite3_get_table+0x15ad
00799478  3173a862 AcroRd32!AVAcroALM_Destroy+0x819a
0079947c  3178230f AcroRd32!AcroWinBrowserMain+0x15fed
00799480  317822bb AcroRd32!AcroWinBrowserMain+0x15f99
00799484  31c29247 AcroRd32!PDFLTerm+0x1f2e17
00799488  31acec7f AcroRd32!PDFLTerm+0x9884f

Loaded symbol image file: sqlite.dll
    Image path: C:\Program Files\Adobe\Reader 10.0\Reader\sqlite.dll
    Image name: sqlite.dll
    Timestamp:        Tue Oct 26 06:13:33 2010 (4CC5D6DD)
    CheckSum:         0003F6DA
    ImageSize:        00040000
    File version:     1.0.0.1
    Product version:  9.0.0.1
    File flags:       0 (Mask 17)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    ProductName:      sqlite Dynamic Link Library
    InternalName:     sqlite
    OriginalFilename: sqlite.dll
    ProductVersion:   9, 0, 0, 1
    FileVersion:      1, 0, 0, 1
    FileDescription:  sqlite Dynamic Link Library
    LegalCopyright:   Copyright (C) 2008

Continuing further we get another similar type of error:

Verifier Crash #3 –  APPLICATION_VERIFIER_LEAK_HANDLE (901)
A HANDLE was leaked. This stop is generated if the owner dll of the handle was dynamically unloaded while owning
resources. This occurs in AdobeXMP.dll – XMP is Adobe’s Extensible Metadata Platform for embedding meta data about a file

I also noticed in tracing:

  • WS2_32.dll throwing error: WSAHOST_NOT_FOUND 11001 (0x2AF9)
  • mdnsNSP.dll throwing error: WSASERVICE_NOT_FOUND 10108 (0x277C) No such service is known. The service cannot be found in the specified name space.
  • WS2_32.dll (winsock) throwing error: WSAEWOULDBLOCK 10035 (0x2733) A non-blocking socket operation could not be completed immediately.
  • WinInet.DLL throwing error “ERROR_HTTP_HEADER_NOT_FOUND 12150
    The requested header could not be located.

~

Ok so the question did any PDF cause the issue? The answer through testing was no, it was only certain PDFs. So it was time for PDF Stream Dumper from Sandspite (http://sandsprite.com/blogs/index.php?uid=7&pid=57) to analyse the PDF. Looking through the various objects one stood out – a binary object starting with “CWS” If you love your hex editors you may already know this off the top of your head – it’s a Flash Shockwave object.

image

Luckily Shockwave objects are typically easily reverse engineered and so I saved the stream as a .swf file and opened in AS3 Sorcerer (Or I could have just hit Decompile…)

image 

It is interesting that issue could never be reproduced with the extracted Flash file alone, despite this flash object being the only real content in the PDF of any value. In addition the Flash files loaded about 10x faster when stand-alone then when embedded in PDF. However customer wanted to pursue the embedded option…

AS3 sorcerer is a must have tool, and very cheap, if you need to analyse other people’s shockwave problems.

Scanning the decompiled ActionScript code I was getting some clues of how this PDF may be impacting IE network connections. Basically the SWF file was dynamically generating graphs and reports based on data it was querying over the network.

The 145,000 lines of decompiled code gave clue as to the creator of the PDF with namespaces of “Xcelsius” which is a SAP product for generating reports. (http://www.sap.com/solutions/sapbusinessobjects/sme/xcelsius/index.epx)

I found at least 10 different web service queries among the code like this (And yes, this really is the password they used):

_local1.okText = “OK”;
            _local1.passwordText = “Password”;
            _local1.refreshOnLoad = true;
            _local1.sSOAPAction = “TransportMonitor/GetReportBlock_TransportPerformance_YTD”;
            _local1.sTargetnamespace = “TransportMonitor”;
            _local1.sUrlendpoint = “http://pbprod01:8080/dswsbobje/qaawsservices/queryasaservice?&cuid=AZ.L5dQ7rWRNpDAi4VKf0YX&authType=secEnterprise&locale=en&timeout=60&ConvertAnyType=true&#8221;;
            _local1.systemText = “System”;
            _local1.userId = “User Identification”;
            _local1.userNameText = “User Name”;

In any case we found reverting from Adobe Reader 10.0 to Adober Reader 9.3 resolved the issue.

Which the customer was fine…I could not reproduce issue at all with Adobe Reader 9.3; while in tandem we opened support case with Adobe to get fix for Adober Reader X …

So Adobe Reader went backwards across the fleet of many machines, after two weeks of successful use in a pilot group of 20 users.

However there was a problem here:

  • The pilot group only consisted of the IT group
  • A test plan had been written, but not executed

You might now know where this is going…Adobe Reader 9.3 then broke SAP PDF form plug-in (By the way this is my most hated plug-in of my life due to it’s flakiness, and SAP no longer recommend using it, sadly I had no choice here)

The SAP PDF form plug-in is located in C:\Program Files\Adobe Reader\<Reader version>\Reader\plugins\SAPForms.api

(.API files are just DLL files, renamed to .API to be Adobe plugins)

Basically it no longer became possible to do many operations in SAP management web portal – leave/shift changes/terminations/etc, Internet Explorer would just totally crash out.

As this was critical issue we didn’t have time to troubleshoot and an emergency roll-up to 10.0 was required…

This fixed the SAP plugin crashing. However we got a new problem, people could no longer embed PDFs in Word Documents. (Disclaimer: I have never desired, never will, or never plan to embed PDFs in Word documents myself. Not my idea!)

It was at this time Adobe support provided a hot-fix. This required updating Adobe 10.0 to 10.1.2 and installing the hot-fix here:

http://helpx.adobe.com/acrobat/kb/pdf-opens-grey-screen-browser.html

Hoo-ray! Page cannot displayed issue has disappeared. Great so deploy to a single production machine…Adobe reader will not launch…

A quick check of Report.wer files under %LOCALAPPDATA%\Microsoft\Windows\WER\ReportArchive showed the following info:

Version=1
EventType=APPCRASH
EventTime=129779844183155000
ReportType=2
Consent=1
UploadTime=129779844183780000
ReportIdentifier=e532a85f-7e07-11e1-b466-00155dd55900
IntegratorReportIdentifier=e532a85e-7e07-11e1-b466-00155dd55900
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=AcroRd32.exe
Sig[1].Name=Application Version
Sig[1].Value=10.1.2.45
Sig[2].Name=Application Timestamp
Sig[2].Value=4f02e382
Sig[3].Name=Fault Module Name
Sig[3].Value=AcroRd32.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=10.1.2.45
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=4f02f76f
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=0011292c
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.256.4
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=3081
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=0a9e
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=0a9e372d3b4ad19135b953a78882e789
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=0a9e
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=0a9e372d3b4ad19135b953a78882e789
UI[2]=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe
UI[3]=Adobe Reader has stopped working
UI[4]=Windows can check online for a solution to the problem.
UI[5]=Check online for a solution and close the program
UI[6]=Check online for a solution later and close the program
UI[7]=Close the program
LoadedModule[0]=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\USER32.dll
LoadedModule[5]=C:\Windows\system32\GDI32.dll
LoadedModule[6]=C:\Windows\system32\LPK.dll
LoadedModule[7]=C:\Windows\system32\USP10.dll
LoadedModule[8]=C:\Windows\system32\msvcrt.dll
LoadedModule[9]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[10]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[11]=C:\Windows\system32\RPCRT4.dll
LoadedModule[12]=C:\Windows\system32\SHELL32.dll
LoadedModule[13]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[14]=C:\Windows\system32\ole32.dll
LoadedModule[15]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[16]=C:\Windows\system32\Secur32.dll
LoadedModule[17]=C:\Windows\system32\SSPICLI.DLL
LoadedModule[18]=C:\Windows\system32\IMM32.DLL
LoadedModule[19]=C:\Windows\system32\MSCTF.dll
LoadedModule[20]=C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\Comctl32.dll
LoadedModule[21]=C:\Windows\system32\uxtheme.dll
LoadedModule[22]=C:\Windows\system32\dwmapi.dll
LoadedModule[23]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[24]=C:\Windows\system32\ntmarta.dll
LoadedModule[25]=C:\Windows\system32\WLDAP32.dll
LoadedModule[26]=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.dll
LoadedModule[27]=C:\Program Files\Adobe\Reader 10.0\Reader\AGM.dll
LoadedModule[28]=C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4940_none_50916076bcb9a742\MSVCP90.dll
LoadedModule[29]=C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4940_none_50916076bcb9a742\MSVCR90.dll
LoadedModule[30]=C:\Windows\system32\VERSION.dll
LoadedModule[31]=C:\Program Files\Adobe\Reader 10.0\Reader\CoolType.dll
LoadedModule[32]=C:\Program Files\Adobe\Reader 10.0\Reader\BIB.dll
LoadedModule[33]=C:\Program Files\Adobe\Reader 10.0\Reader\ACE.dll
LoadedModule[34]=C:\Windows\system32\profapi.dll
LoadedModule[35]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[36]=C:\Windows\system32\rsaenh.dll
LoadedModule[37]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[38]=C:\Program Files\Adobe\Reader 10.0\Reader\plug_ins\SAPForms.api
LoadedModule[39]=C:\Windows\system32\WSOCK32.dll
LoadedModule[40]=C:\Windows\system32\WS2_32.dll
LoadedModule[41]=C:\Windows\system32\NSI.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Adobe Reader
AppPath=C:\Program Files\Adobe\Reader 10.0\Reader\AcroRd32.exe

 

So what we can gues from this? If app is crashing on launch, as in this cause go up the loaded module backwards for any suspicious items. In this case

LoadedModule[38]=C:\Program Files\Adobe\Reader 10.0\Reader\plug_ins\SAPForms.api

I renamed this file to .bak and Hello – Adobe Reader launched. Through more testing I confirmed this add-in, even after full re-installing SAP and the add-in did not like Adobe Reader 10.1.2 – trying many configurations such as disabling enhanced security & protected mode to no avail. Unfortunately this plug-in is a must for the SAP forms to work…

Downgrading to Adobe Reader 10.0.1 and SAP worked, but we still get “Page Cannot Be Displayed” error.

Thankfully there is a solution – install IE9. IE9 eliminated the page displayed error, and allow us to stay on the 10.0.1 Adobe Reader version that SAP plays happy with.

Now just to test the multitude of poorly designed web applications & add-ins (that cannot be changed/upgraded) with IE9 that abound at this customer.

I’m sure there will be lots more debugging entertainment provided by this customer for months to come.

Moral of the story:

  • Don’t put shockwave in PDFs
  • Don’t embed PDFs in Word docs
  • Don’t use SAP Forms plugin
  • Upgrade to IE9!

About chentiangemalc

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in Adobe, Application Compatibility, Internet Explorer, Windows 7 and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s