Case of the Slow Logon – Anti-Virus vs 3rd Party Application

Across a customer we noticed a trend in increased logon times, logon times sometimes as long as 30 minutes. This particular issue was occurring in the legacy Windows XP environment, that had yet to upgrade to Windows 7. One of the first things we did was take a ProcMon Boot Log (http://live.sysinternals.com/ProcMon.exe) by clicking Options | Enable Boot Logging, then accepting any defaults.

image

 

The first thing I check in slow logon is the Process Tree via the Tools menu. Any nasty hanging items here could be a problem (i.e. scripts running, too many startup programs, &c)

So I looked through the list, while it seems to have plenty of candidates for optimizing, I compared it to the logon process of “fast logon” user. The process tree looked very similar. Having a baseline to compare to is invaluable in these scenarios. image

So then I clicked Tools -> File Summary and noticed a relatively large amount of file activity in a 3rd party application folder. This did not exist on the logs of the non-affected user. Interestingly this application was not configured to launch at logon, and did not appear in the process tree at all!

image

So I set my filter to include Path Contains QIK-QUBE2 to find out what process was having a field day with this folder, and we could see Sophos Anti-Virus was the one to blame:

image

In particular the Anti-Virus spent most of it’s time scanning about 600 .FIL files. I opened up some of the files and noticed why Anti-Virus found them so interesting they started with MZ, which means it is a Windows Portable Executable (EXE/DLL) being the initials of Mark Zbikowski

Further investigation found that these .FIL files were actually ZenWorks Snapshot Packages that been unnecessarily included in the MSI installer. We could determine that because they were composed of

  • *.FIL files
  • .AOT file
  • .AXT file
  • filedef.txt file

Further examination of the filedef.txt file found that these files themselves also included anti-virus definitions from another Anti-Virus vendor i.e. the filedef.txt file had lines like

52.fil=C:\Program Files\Common Files\Symantec Shared\VirusDefs\20011107.004\NAVEX15.SYS
53.fil=C:\Program Files\Common Files\Symantec Shared\VirusDefs\20011107.004\NAVEX15.VXD

However removing these files alone was not enough to improve the logon performance. The Anti-Virus process continued to use up excessive time at logon. One thing I commonly check first when processes are badly behaving is check what DLLs they load by filtering on the Load Image operation and look for anything suspicious. Sure enough we had it … Sophos was loading DLL from this 3rd party application, not the C:\windows\system32 folder which had a much newer version of the DLL

image

Opening the MSI in Orca and looking at the Registry table we could see why – the MSI was overwriting the Windows registration details with it’s own

image

Unfortunately uninstalling the application resulted in a damaged system as all these keys were deleted but not replaced with the originals.

So I captured a ProcMon of the installation and filtered on Operation is RegSetValue and Detail Contains DLL. I then saved the resultant list of DLL registrations modified in Registry as CSV, and opened in Excel.

I then used a PivotTable to get all the unique DLL paths, and used a formula to build the regsvr32 commands in this repair script. Some of these may not have existed in windows\system32 directory, but then they would just fail silently.

pushd %windir%\system32
regsvr32 /s asycfilt.dll
regsvr32 /s COMCAT.DLL
regsvr32 /s COMDLG32.OCX
regsvr32 /s msadodc.ocx
regsvr32 /s msadox.dll
regsvr32 /s msbind.dll
regsvr32 /s mscomct2.ocx
regsvr32 /s MSCOMCTL.OCX
regsvr32 /s MSDATGRD.OCX
regsvr32 /s msdbrptr.dll
regsvr32 /s msjro.dll
regsvr32 /s msmask32.ocx
regsvr32 /s MSSTDFMT.DLL
regsvr32 /s msvbvm60.dll
regsvr32 /s msvcrt.dll
regsvr32 /s NWSess.ocx
regsvr32 /s oleaut32.dll
regsvr32 /s olepro32.dll
regsvr32 /s RICHED32.DLL
regsvr32 /s RICHTX32.OCX
regsvr32 /s scrrun.dll
regsvr32 /s stdole2.tlb
regsvr32 /s VB6STKIT.DLL
regsvr32 /s wshom.ocx

popd

Once this script ran + the .FIL files removed the logon performance issue was totally gone…

Epilogue

The MSI was updated removing the unnecessary CSLID registry keys and the ZenWorks Snapshot package was removed. The MSI was also updated to repair any damage done by uninstalling the previous MSI. Everyone lived happily ever after.

About chentiangemalc

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in Logon, Performance, ProcMon and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s