Across a customer we noticed a trend in increased logon times, logon times sometimes as long as 30 minutes. This particular issue was occurring in the legacy Windows XP environment, that had yet to upgrade to Windows 7. One of the first things we did was take a ProcMon Boot Log (http://live.sysinternals.com/ProcMon.exe) by clicking Options | Enable Boot Logging, then accepting any defaults.
The first thing I check in slow logon is the Process Tree via the Tools menu. Any nasty hanging items here could be a problem (i.e. scripts running, too many startup programs, &c)
So I looked through the list, while it seems to have plenty of candidates for optimizing, I compared it to the logon process of “fast logon” user. The process tree looked very similar. Having a baseline to compare to is invaluable in these scenarios.
So then I clicked Tools -> File Summary and noticed a relatively large amount of file activity in a 3rd party application folder. This did not exist on the logs of the non-affected user. Interestingly this application was not configured to launch at logon, and did not appear in the process tree at all!
So I set my filter to include Path Contains QIK-QUBE2 to find out what process was having a field day with this folder, and we could see Sophos Anti-Virus was the one to blame:
In particular the Anti-Virus spent most of it’s time scanning about 600 .FIL files. I opened up some of the files and noticed why Anti-Virus found them so interesting they started with MZ, which means it is a Windows Portable Executable (EXE/DLL) being the initials of Mark Zbikowski
Further investigation found that these .FIL files were actually ZenWorks Snapshot Packages that been unnecessarily included in the MSI installer. We could determine that because they were composed of
- *.FIL files
- .AOT file
- .AXT file
- filedef.txt file
Further examination of the filedef.txt file found that these files themselves also included anti-virus definitions from another Anti-Virus vendor i.e. the filedef.txt file had lines like
52.fil=C:\Program Files\Common Files\Symantec Shared\VirusDefs\20011107.004\NAVEX15.SYS
53.fil=C:\Program Files\Common Files\Symantec Shared\VirusDefs\20011107.004\NAVEX15.VXD
However removing these files alone was not enough to improve the logon performance. The Anti-Virus process continued to use up excessive time at logon. One thing I commonly check first when processes are badly behaving is check what DLLs they load by filtering on the Load Image operation and look for anything suspicious. Sure enough we had it … Sophos was loading DLL from this 3rd party application, not the C:\windows\system32 folder which had a much newer version of the DLL
Opening the MSI in Orca and looking at the Registry table we could see why – the MSI was overwriting the Windows registration details with it’s own
Unfortunately uninstalling the application resulted in a damaged system as all these keys were deleted but not replaced with the originals.
So I captured a ProcMon of the installation and filtered on Operation is RegSetValue and Detail Contains DLL. I then saved the resultant list of DLL registrations modified in Registry as CSV, and opened in Excel.
I then used a PivotTable to get all the unique DLL paths, and used a formula to build the regsvr32 commands in this repair script. Some of these may not have existed in windows\system32 directory, but then they would just fail silently.
regsvr32 /s asycfilt.dll
regsvr32 /s COMCAT.DLL
regsvr32 /s COMDLG32.OCX
regsvr32 /s msadodc.ocx
regsvr32 /s msadox.dll
regsvr32 /s msbind.dll
regsvr32 /s mscomct2.ocx
regsvr32 /s MSCOMCTL.OCX
regsvr32 /s MSDATGRD.OCX
regsvr32 /s msdbrptr.dll
regsvr32 /s msjro.dll
regsvr32 /s msmask32.ocx
regsvr32 /s MSSTDFMT.DLL
regsvr32 /s msvbvm60.dll
regsvr32 /s msvcrt.dll
regsvr32 /s NWSess.ocx
regsvr32 /s oleaut32.dll
regsvr32 /s olepro32.dll
regsvr32 /s RICHED32.DLL
regsvr32 /s RICHTX32.OCX
regsvr32 /s scrrun.dll
regsvr32 /s stdole2.tlb
regsvr32 /s VB6STKIT.DLL
regsvr32 /s wshom.ocx
Once this script ran + the .FIL files removed the logon performance issue was totally gone…
The MSI was updated removing the unnecessary CSLID registry keys and the ZenWorks Snapshot package was removed. The MSI was also updated to repair any damage done by uninstalling the previous MSI. Everyone lived happily ever after.