Case of the XP Service Pack 3 Upgrade Fail

Over the weekend the drummer in my band “The Brushed Keys” ( https://www.youtube.com/watch?v=WU-U7SBPF5Y&list=PLQA4w1oo2uGQGxKTzvzcJaPcNSj8SL8pp ) had finally taken the plunge and attempted the Windows XP Service Pack 3 upgrade, only about 6 years late.

However it all went wrong when the machine would no longer startup, going to a permanent black screen with mouse cursor after the XP  logo. This occurred in all the Safe modes, and also Last Known Good Configuration.

image

Using a Windows PE boot disk I had on hand, with diagnostic utilities, I first used an offline event viewer Event Log Explorer to check the last Windows events, opening the .evt files in c:\windows\system32\config  ( http://www.eventlogxp.com/ ) This tool is particularly useful in XP environments .EVT files cannot be opened by event viewer if copied off a machine, they must be exported first. Windows Vista and later EVTX files do not suffer this problem.

From the application log we could see WinLogon.exe was crashing, but there was no fault information i.e. exception code, faulting module, etc. In addition Dr Watson logs were not generated. Without a firewire/serial cable on hand I didn’t have much diagnostic info, so decided to just revert the XP Service Pack 3 upgrade.

On the C: drive I could see System Restore was enabled, but in Windows XP you cannot use System Restore until you actually boot into the system, (Or you have a specific boot disk to handle offline XP system restore, I hadn’t used in 5 years or more so no idea where that was)

To do this, I booted into Windows PE. (but you could use Windows XP recovery console) and did the following:

1) Made a backup of C:\Windows\System32\Config folder

2) Browsed C:\System Volume Information\_restore<GUID>\RPxxx (I chose the oldest one)\snapshot folder (Note these are Hidden System Folder)

3) Copied the following files to Config folder, and renamed them taking of the _REGISTRY_MACHINE_ prefix, replacing the existing files.

  • _REGISTRY_MACHINE_SAM
  • _REGISTRY_MACHINE_SECURITY
  • _REGISTRY_MACHINE_SOFTWARE
  • _REGISTRY_MACHINE_SYSTEM
  • _REGISTRY_MACHINE_DEFAULT

4) This was probably not necessary but I also restored the user’s ntuser.dat (Also made a backup of it before replacing)

Copying

_REGISTRY_USER_USERCLASS_<SID> to C:\Documents and Settings\<User Profile>

You can see the SID by checking permissions on user profile within Windows PE, i.e. using icacls/etc.

5) Renamed C:\windows\$NTServicePackUninstall$\spuninst.txt to spuninst.cmd and ran the batch file. This deletes the Service Pack 3 files, and copies back the backed up files before the patch was applied

Unfortunately after restarting we got error

lssass.exe – System Error

When trying to update a password, this return status indicates that the value provided

as the current password is not correct.

image

After this error the computer immediately rebooted.

This issue was fixed by restoring the SAM and SECURITY files from my backup, back into C:\Windows\System32\Config (If you didn’t have a backup you could try your luck the backups in c:\windows\repair folder)

After this the machine boots fine, with all the users applications intact.

I made a 5 minute video of the process on my phone, it is here on YouTube

Uninstalling Windows XP SP3 on an Unbootable PC

https://www.youtube.com/watch?v=MTR50Z3Kh98

Posted in Windows XP | Tagged | Leave a comment

Case of the IE Hangs-Citrix HDX Flash Redirection

To start with I wanted to capture IE hangs properly. In modern versions of Internet before you capture hang dumps you ideally want to set the TabProcGrowth value to 0 under HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main

This will force IE to put all tabs in the one process. This can decrease stability, but ensures we can see everything IE doing in the one process memory snapshot.

Be aware on IE10 or later in 64-bit platforms this setting can cause compatibility issues http://support.microsoft.com/kb/2716529

Next I prefer to capture about 3 dmp files, about 10 seconds apart. If you have no tools installed you can use task manager.

Or you can use ProcDump ( http://live.sysinternals.com/ProcDump.exe ) to autodetect (must be only one process for this to work, or change iexplore.exe with PID )

This simple batch file will wait for IE to hang then generate 3 consecutive dmp files.

procdump.exe -h -ma -x c:\dumps iexplore.exe ping 1.1.1.1 -w 10000 -n 1 procdump.exe -h -ma -x c:\dumps iexplore.exe ping 1.1.1.1 -w 10000 -n 1 procdump.exe -h -ma -x c:\dumps iexplore.exe

Alternatively you can use AdPlus.exe from Windows Debugging Tools, which you can manually initiate on a hang situation. This has the benefit of generating a log file, which can be useful if you are getting someone else to collect the dmp files and they are having trouble doing so.

adplus -hang -pn iexplore.exe -o C:\dumps -quiet ping 1.1.1.1 -w 10000 -n 1 adplus -hang -pn iexplore.exe -o C:\dumps -quiet ping 1.1.1.1 -w 10000 -n 1 adplus -hang -pn iexplore.exe -o C:\dumps -quiet

After reproducing some hangs I had a collection of 5 sets of 3 hangs. Once we opened up our dmp file in WinDbg I started with !analyze –vhang

FAULTING_IP:
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000007 (Wake debugger)
  ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR:  HANG

PROCESS_NAME:  iexplore.exe

ERROR_CODE: (NTSTATUS) 0xcfffffff – <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xcfffffff – <Unable to get error code text>

DETOURED_IMAGE: 1

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

DERIVED_WAIT_CHAIN: 

Dl Eid Cid     WaitType
– — ——- ————————–
   0   3e18.5ae0 SendMessage           

WAIT_CHAIN_COMMAND:  ~0s;k;;

BLOCKING_THREAD:  00005ae0

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BusyHang

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BusyHang

LAST_CONTROL_TRANSFER:  from 74a874bb to 74a872b9

FAULTING_THREAD:  00000000

STACK_TEXT: 
003ee130 74a874bb 00040274 00000112 0000f120 user32!NtUserMessageCall+0x15
003ee1bc 74a86a8c 01425b10 00000000 00000112 user32!RealDefWindowProcWorker+0x73
003ee1dc 64d27744 00040274 00000112 0000f120 user32!RealDefWindowProcW+0x4a
003ee1f4 64d2a092 003ee220 03f150f8 003ee258 uxtheme!DoMsgDefault+0x2d
003ee204 64d20b0d 03f150f8 003ee220 00003fff uxtheme!OnDwpSysCommand+0x47
003ee258 64d20b96 00000000 00000000 0000f120 uxtheme!_ThemeDefWindowProc+0x13c
003ee274 74a8729a 00040274 00000112 0000f120 uxtheme!ThemeDefWindowProcW+0x18
003ee2bc 59247cda 00040274 00000112 0000f120 user32!DefWindowProcW+0x68
003ee2d8 592487b0 00040274 00000112 0000f120 ieframe!Detour_DefWindowProcW+0x18
003ee34c 59241fe3 00040274 00000112 0000f120 ieframe!CBrowserFrame::v_WndProc+0xd3e
003ee370 74a862fa 00040274 00000112 0000f120 ieframe!CImpWndProc::s_WndProc+0x68
003ee39c 74a87316 59241fa1 00040274 00000112 user32!InternalCallWinProc+0x23
003ee414 74a8965e 00000000 59241fa1 00040274 user32!UserCallWinProcCheckWow+0xd8
003ee458 74a896c5 01425b10 00000000 59241fa1 user32!SendMessageWorker+0x581
003ee47c 64d2a173 00040274 00000112 0000f120 user32!SendMessageW+0x7f
003ee4a4 64d20b0d 03f150f8 003ee4c0 00003fff uxtheme!OnDwpNcLButtonDown+0xc7
003ee4f8 64d20b96 00000000 00000000 00000009 uxtheme!_ThemeDefWindowProc+0x13c
003ee514 74a8729a 00040274 000000a1 00000009 uxtheme!ThemeDefWindowProcW+0x18
003ee55c 59247cda 00040274 000000a1 00000009 user32!DefWindowProcW+0x68
003ee578 592487b0 00040274 000000a1 00000009 ieframe!Detour_DefWindowProcW+0x18
003ee5ec 59241fe3 00040274 000000a1 00000009 ieframe!CBrowserFrame::v_WndProc+0xd3e
003ee610 74a862fa 00040274 000000a1 00000009 ieframe!CImpWndProc::s_WndProc+0x68
003ee63c 74a86d3a 59241fa1 00040274 000000a1 user32!InternalCallWinProc+0x23
003ee6b4 74a877c4 00000000 59241fa1 00040274 user32!UserCallWinProcCheckWow+0x109
003ee714 74a8788a 59241fa1 00000000 003ee798 user32!DispatchMessageWorker+0x3bc
003ee724 59241e74 003ee740 005795c8 04664f00 user32!DispatchMessageW+0xf
003ee798 59228df7 004c4788 0057f200 00000001 ieframe!CBrowserFrame::FrameMessagePump+0x38c
003ee7fc 59264501 00000000 004c4788 74e71420 ieframe!BrowserThreadProc+0x258
003ee824 59264459 004c4788 004c4824 004c4788 ieframe!BrowserNewThreadProc+0x95
003ef89c 592642f7 004c4788 75089058 75088861 ieframe!SHOpenFolderWindow+0x10f
003efac4 59264161 00501f20 00000001 00000000 ieframe!IEWinMain+0x1a7
003efb08 01333958 00501f20 00000001 00000000 ieframe!LCIEStartAsFrame+0x457
003efb50 0133131a 01330000 00000000 004c2d88 iexplore!wWinMain+0x3e9
003efbe4 74e7338a 7efde000 003efc30 7709bf32 iexplore!_initterm_e+0x1b0
003efbf0 7709bf32 7efde000 62daa4f5 00000000 kernel32!BaseThreadInitThunk+0xe
003efc30 7709bf05 013326b0 7efde000 ffffffff ntdll!__RtlUserThreadStart+0x70
003efc48 00000000 013326b0 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b

FOLLOWUP_IP:
uxtheme!ThemeDefWindowProcW+18
64d20b96 5d              pop     ebp

SYMBOL_STACK_INDEX:  6

SYMBOL_NAME:  uxtheme!ThemeDefWindowProcW+18

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: uxtheme

IMAGE_NAME:  uxtheme.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bdb3c

STACK_COMMAND:  ~0s ; kb

BUCKET_ID:  HANG_DETOURED_uxtheme!ThemeDefWindowProcW+18

FAILURE_BUCKET_ID:  APPLICATION_HANG_BusyHang_cfffffff_uxtheme.dll!ThemeDefWindowProcW

WATSON_STAGEONE_URL:  http://watson.microsoft.com/00000000.htm?Retriage=1

Followup: MachineOwner
———

0:000> lmvm uxtheme
start    end        module name
64d10000 64d90000   uxtheme    (pdb symbols)          g:\symbols\wuxtheme.pdb\20C669C0018E406295BFA56B7C93850F2\wuxtheme.pdb
    Loaded symbol image file: uxtheme.dll
    Image path: C:\Windows\System32\uxtheme.dll
    Image name: uxtheme.dll
    Timestamp:        Tue Jul 14 11:11:24 2009 (4A5BDB3C)
    CheckSum:         000479E1
    ImageSize:        00080000
    File version:     6.1.7600.16385
    Product version:  6.1.7600.16385
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     UxTheme.dll
    OriginalFilename: UxTheme.dll
    ProductVersion:   6.1.7600.16385
    FileVersion:      6.1.7600.16385 (win7_rtm.090713-1255)
    FileDescription:  Microsoft UxTheme Library
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

 

In this case !analyze –v  is showing us the UI thread is hung, which is correct. But it’s not the root cause of the hang.

Checking for deadlocks we can see a critical section with high lock count / contention count. Understanding Critical Sections is essential for Windows Hang Analysis. A good starting point is the MSDN documentation here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms682530(v=vs.85).aspx

0:000> !locks

CritSec ntdll!LdrpLoaderLock+0 at 771620c0
WaiterWoken        No
LockCount          27
RecursionCount     1
OwningThread       5da8
EntryCount         0
ContentionCount    125
*** Locked

CritSec PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+7a1d30 at 53753ea0
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       5da8
EntryCount         0
ContentionCount    0
*** Locked

CritSec +ee88e80 at 0ee88e80
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       5da8
EntryCount         0
ContentionCount    0
*** Locked

Scanned 2706 critical sections

We can view the thread status, and here you also see

  • thread #
  • ID – which is Process ID and Thread ID (In WinDbg often referred to as CID which is a shortcut for the undocumented structure CLIENT_ID – reference the free book “undocumented Windows 2000 secrets” http://undocumented.rawol.com/)
  • TEB (Thread Environment Block)

0:000> ~
.  0  Id: 3e18.5ae0 Suspend: 1 Teb: 7efdd000 Unfrozen
   1  Id: 3e18.5c7c Suspend: 1 Teb: 7efda000 Unfrozen
   2  Id: 3e18.1454 Suspend: 1 Teb: 7efd7000 Unfrozen
   3  Id: 3e18.5780 Suspend: 1 Teb: 7efac000 Unfrozen
   4  Id: 3e18.4808 Suspend: 1 Teb: 7efa9000 Unfrozen
   5  Id: 3e18.59cc Suspend: 1 Teb: 7efa6000 Unfrozen
   6  Id: 3e18.30a8 Suspend: 1 Teb: 7ef9f000 Unfrozen
   7  Id: 3e18.39d4 Suspend: 1 Teb: 7ef9c000 Unfrozen
   8  Id: 3e18.47a8 Suspend: 1 Teb: 7ef99000 Unfrozen
   9  Id: 3e18.4f94 Suspend: 1 Teb: 7ef96000 Unfrozen
  10  Id: 3e18.34ac Suspend: 1 Teb: 7ef8f000 Unfrozen
  11  Id: 3e18.2478 Suspend: 1 Teb: 7ef8c000 Unfrozen
  12  Id: 3e18.34d8 Suspend: 1 Teb: 7ef89000 Unfrozen
  13  Id: 3e18.d44 Suspend: 1 Teb: 7ef83000 Unfrozen
  14  Id: 3e18.5db0 Suspend: 1 Teb: 7ef7f000 Unfrozen
  15  Id: 3e18.4f1c Suspend: 1 Teb: 7ef79000 Unfrozen
  16  Id: 3e18.35e0 Suspend: 1 Teb: 7ef76000 Unfrozen
  17  Id: 3e18.31b4 Suspend: 1 Teb: 7ef73000 Unfrozen
  18  Id: 3e18.3d88 Suspend: 1 Teb: 7ef93000 Unfrozen
  19  Id: 3e18.4dec Suspend: 1 Teb: 7ef7c000 Unfrozen
  20  Id: 3e18.4bc4 Suspend: 1 Teb: 7ef6f000 Unfrozen
  21  Id: 3e18.2994 Suspend: 1 Teb: 7ef6c000 Unfrozen
  22  Id: 3e18.3ddc Suspend: 1 Teb: 7ef66000 Unfrozen
  23  Id: 3e18.5b68 Suspend: 1 Teb: 7ef63000 Unfrozen
  24  Id: 3e18.4bac Suspend: 1 Teb: 7ef5f000 Unfrozen
  25  Id: 3e18.38c4 Suspend: 1 Teb: 7ef5c000 Unfrozen
  26  Id: 3e18.3824 Suspend: 1 Teb: 7ef56000 Unfrozen
  27  Id: 3e18.41c8 Suspend: 1 Teb: 7ef53000 Unfrozen
  28  Id: 3e18.dac Suspend: 1 Teb: 7ef49000 Unfrozen
  29  Id: 3e18.1a9c Suspend: 1 Teb: 7ef3f000 Unfrozen
  30  Id: 3e18.22cc Suspend: 1 Teb: 7ef39000 Unfrozen
  31  Id: 3e18.3084 Suspend: 1 Teb: 7ef36000 Unfrozen
  32  Id: 3e18.4118 Suspend: 1 Teb: 7ef1c000 Unfrozen
  33  Id: 3e18.458c Suspend: 1 Teb: 7ef19000 Unfrozen
  34  Id: 3e18.4b90 Suspend: 1 Teb: 7ef4f000 Unfrozen
  35  Id: 3e18.3c18 Suspend: 1 Teb: 7ef46000 Unfrozen
  36  Id: 3e18.5da8 Suspend: 1 Teb: 7ef69000 Unfrozen
  37  Id: 3e18.4b84 Suspend: 1 Teb: 7ef4c000 Unfrozen
  38  Id: 3e18.5a40 Suspend: 1 Teb: 7ef29000 Unfrozen
  39  Id: 3e18.2bbc Suspend: 1 Teb: 7ef1f000 Unfrozen
  40  Id: 3e18.2a14 Suspend: 1 Teb: 7ef13000 Unfrozen
  41  Id: 3e18.16ac Suspend: 1 Teb: 7ef03000 Unfrozen
  42  Id: 3e18.5ddc Suspend: 1 Teb: 7ef33000 Unfrozen
  43  Id: 3e18.31b0 Suspend: 1 Teb: 7ef09000 Unfrozen
  44  Id: 3e18.23c4 Suspend: 1 Teb: 7eef6000 Unfrozen
  45  Id: 3e18.5f54 Suspend: 1 Teb: 7eef3000 Unfrozen
  46  Id: 3e18.1880 Suspend: 1 Teb: 7eeef000 Unfrozen
  47  Id: 3e18.58f0 Suspend: 1 Teb: 7eeec000 Unfrozen
  48  Id: 3e18.4f18 Suspend: 1 Teb: 7eee9000 Unfrozen
  49  Id: 3e18.321c Suspend: 1 Teb: 7eee6000 Unfrozen
  50  Id: 3e18.a64 Suspend: 1 Teb: 7eee3000 Unfrozen
  51  Id: 3e18.4828 Suspend: 1 Teb: 7ef86000 Unfrozen
  52  Id: 3e18.2f00 Suspend: 1 Teb: 7ef59000 Unfrozen
  53  Id: 3e18.2e24 Suspend: 1 Teb: 7eedf000 Unfrozen
  54  Id: 3e18.5b9c Suspend: 1 Teb: 7eedc000 Unfrozen
  55  Id: 3e18.37c4 Suspend: 1 Teb: 7ef26000 Unfrozen
  56  Id: 3e18.3de4 Suspend: 1 Teb: 7ef23000 Unfrozen
  57  Id: 3e18.2b80 Suspend: 1 Teb: 7efaf000 Unfrozen
  58  Id: 3e18.518c Suspend: 1 Teb: 7efa3000 Unfrozen
  59  Id: 3e18.6090 Suspend: 1 Teb: 7ef43000 Unfrozen
  60  Id: 3e18.42bc Suspend: 1 Teb: 7ef3c000 Unfrozen
  61  Id: 3e18.6074 Suspend: 1 Teb: 7ef2f000 Unfrozen
  62  Id: 3e18.4038 Suspend: 1 Teb: 7ef2c000 Unfrozen
  63  Id: 3e18.4c4 Suspend: 1 Teb: 7ef16000 Unfrozen
0:000> ~36s
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000cc8 edi=00000000
eip=7707f8d1 esp=0d9eb37c ebp=0d9eb3e8 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
7707f8d1 83c404          add     esp,4
0:036> kv
ChildEBP RetAddr  Args to Child             
0d9eb37c 767414ab 00000cc8 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0d9eb3e8 74e71194 00000cc8 ffffffff 00000000 KERNELBASE!WaitForSngleObjectEx+0x98 (FPO: [Non-Fpo])
0d9eb400 74e71148 00000cc8 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0d9eb414 5a7a31e2 00000cc8 ffffffff 0ee88e80 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0d9eb434 5a7a3359 0d9ee6a8 00000000 52fb0000 MMDevAPI!CDeviceEnumerator::DestroyHWndNotificationThread+0xf6 (FPO: [Non-Fpo])
0d9eb444 5a7a24c0 00000003 00050418 00000000 MMDevAPI!CDeviceEnumerator::ReleaseHWndNotification+0x29 (FPO: [0,0,4])
0d9eb458 5305bc4f 0ee88e00 011ce808 5c0eaf18 MMDevAPI!CDeviceEnumerator::UnregisterEndpointNotificationCallback+0x7e (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0d9ebcbc 5305b3fb 5c0ed6b4 011ce528 73736553 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xa9adf <- likely culprit
0d9ec510 5305ec4f 011ce528 0d9ecd88 53022718 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xa928b
0d9ec51c 53022718 00000001 5c0ede2c cccccccc PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xacadf
0d9ecd88 53022297 5c0ec64c 011bd900 011e0e20 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x705a8
0d9ed5e8 53066055 5c0ecdf0 011cef20 011e0ff0 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x70127
0d9ede54 52fb25b1 5c0ef510 0d9ef7ac 00000003 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xb3ee5
0d9ee6b4 52fbaf16 011e0e20 0d9eef58 530653c1 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x441
0d9ee6c0 530653c1 00000001 5c0efcfc 0d9ee6fc PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x8da6
0d9eef58 52fb10f7 5c0ee41c 00000000 00000003 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xb3251
0d9ef7b8 52fdb209 52fb0000 00000003 00000000 PseudoServerInproc2+0x10f7
0d9ef7fc 52fdb2c2 52fb0000 665d1fed 52fb0000 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x29099
0d9ef82c 665d1f5d 52fb0000 00000003 00000000 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x29152
0d9ef860 7709b990 52fb0000 00000003 00000000 IEShims!CShimBindings::s_DllMainHook+0x4a (FPO: [Non-Fpo])
0d9ef880 770b659f 665d1f14 52fb0000 00000003 ntdll!LdrpCallInitRoutine+0x14
0d9ef924 770b6786 00000000 00000000 0d9ef94c ntdll!LdrShutdownThread+0xe6 (FPO: [Non-Fpo])
0d9ef934 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0d9ef94c 74e7338a 07c1f260 0d9ef998 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
0d9ef958 7709bf32 07c1f260 6f7aa15d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0d9ef998 7709bf05 5924fe98 07c1f260 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0d9ef9b0 00000000 5924fe98 07c1f260 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

From this we can work out the issue is caused when audio is being shutdown, a critical section is blocked and deadlock is created. In this case ~40% of threads, including the primary user interface thread are waiting for access to that critical section 771620c0.

Critical sections can be found by looking for first argument in ntdll!RtlEnterCriticalSection and you can view them by command !critsec <value>

We can also see many additional 3rd party components, so we won’t be surprised if we see more issues once this one is resolved Smile 

 

0:036> ~*kv

#  0  Id: 3e18.5ae0 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr  Args to Child             
003ee130 74a874bb 00040274 00000112 0000f120 user32!NtUserMessageCall+0x15 (FPO: [7,0,0])
003ee1bc 74a86a8c 01425b10 00000000 00000112 user32!RealDefWindowProcWorker+0x73 (FPO: [Non-Fpo])
003ee1dc 64d27744 00040274 00000112 0000f120 user32!RealDefWindowProcW+0x4a (FPO: [Non-Fpo])
003ee1f4 64d2a092 003ee220 03f150f8 003ee258 uxtheme!DoMsgDefault+0x2d (FPO: [Non-Fpo])
003ee204 64d20b0d 03f150f8 003ee220 00003fff uxtheme!OnDwpSysCommand+0x47 (FPO: [Non-Fpo])
003ee258 64d20b96 00000000 00000000 0000f120 uxtheme!_ThemeDefWindowProc+0x13c (FPO: [Non-Fpo])
003ee274 74a8729a 00040274 00000112 0000f120 uxtheme!ThemeDefWindowProcW+0x18 (FPO: [Non-Fpo])
003ee2bc 59247cda 00040274 00000112 0000f120 user32!DefWindowProcW+0x68 (FPO: [Non-Fpo])
003ee2d8 592487b0 00040274 00000112 0000f120 ieframe!Detour_DefWindowProcW+0x18 (FPO: [Non-Fpo])
003ee34c 59241fe3 00040274 00000112 0000f120 ieframe!CBrowserFrame::v_WndProc+0xd3e (FPO: [4,19,4])
003ee370 74a862fa 00040274 00000112 0000f120 ieframe!CImpWndProc::s_WndProc+0x68 (FPO: [Non-Fpo])
003ee39c 74a87316 59241fa1 00040274 00000112 user32!InternalCallWinProc+0x23
003ee414 74a8965e 00000000 59241fa1 00040274 user32!UserCallWinProcCheckWow+0xd8 (FPO: [Non-Fpo])
003ee458 74a896c5 01425b10 00000000 59241fa1 user32!SendMessageWorker+0x581 (FPO: [Non-Fpo])
003ee47c 64d2a173 00040274 00000112 0000f120 user32!SendMessageW+0x7f (FPO: [Non-Fpo])
003ee4a4 64d20b0d 03f150f8 003ee4c0 00003fff uxtheme!OnDwpNcLButtonDown+0xc7 (FPO: [Non-Fpo])
003ee4f8 64d20b96 00000000 00000000 00000009 uxtheme!_ThemeDefWindowProc+0x13c (FPO: [Non-Fpo])
003ee514 74a8729a 00040274 000000a1 00000009 uxtheme!ThemeDefWindowProcW+0x18 (FPO: [Non-Fpo])
003ee55c 59247cda 00040274 000000a1 00000009 user32!DefWindowProcW+0x68 (FPO: [Non-Fpo])
003ee578 592487b0 00040274 000000a1 00000009 ieframe!Detour_DefWindowProcW+0x18 (FPO: [Non-Fpo])
003ee5ec 59241fe3 00040274 000000a1 00000009 ieframe!CBrowserFrame::v_WndProc+0xd3e (FPO: [4,19,4])
003ee610 74a862fa 00040274 000000a1 00000009 ieframe!CImpWndProc::s_WndProc+0x68 (FPO: [Non-Fpo])
003ee63c 74a86d3a 59241fa1 00040274 000000a1 user32!InternalCallWinProc+0x23
003ee6b4 74a877c4 00000000 59241fa1 00040274 user32!UserCallWinProcCheckWow+0x109 (FPO: [Non-Fpo])
003ee714 74a8788a 59241fa1 00000000 003ee798 user32!DispatchMessageWorker+0x3bc (FPO: [Non-Fpo])
003ee724 59241e74 003ee740 005795c8 04664f00 user32!DispatchMessageW+0xf (FPO: [Non-Fpo])
003ee798 59228df7 004c4788 0057f200 00000001 ieframe!CBrowserFrame::FrameMessagePump+0x38c (FPO: [Non-Fpo])
003ee7fc 59264501 00000000 004c4788 74e71420 ieframe!BrowserThreadProc+0x258 (FPO: [Non-Fpo])
003ee824 59264459 004c4788 004c4824 004c4788 ieframe!BrowserNewThreadProc+0x95 (FPO: [1,3,4])
003ef89c 592642f7 004c4788 75089058 75088861 ieframe!SHOpenFolderWindow+0x10f (FPO: [Non-Fpo])
003efac4 59264161 00501f20 00000001 00000000 ieframe!IEWinMain+0x1a7 (FPO: [Non-Fpo])
003efb08 01333958 00501f20 00000001 00000000 ieframe!LCIEStartAsFrame+0x457 (FPO: [Non-Fpo])
003efb50 0133131a 01330000 00000000 004c2d88 iexplore!wWinMain+0x3e9 (FPO: [4,9,4])
003efbe4 74e7338a 7efde000 003efc30 7709bf32 iexplore!_initterm_e+0x1b0 (FPO: [Non-Fpo])
003efbf0 7709bf32 7efde000 62daa4f5 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
003efc30 7709bf05 013326b0 7efde000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
003efc48 00000000 013326b0 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   1  Id: 3e18.5c7c Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr  Args to Child             
031af2c8 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
031af32c 7709b398 00000000 00000000 00000001 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
031af354 77096c20 771620c0 61feab49 00000000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
031af38c 7674190f 00000001 00000000 031af3b4 ntdll!LdrLockLoaderLock+0xe4 (FPO: [Non-Fpo])
031af3d8 5c3c7835 64cc0000 13fe8140 00000103 KERNELBASE!GetModuleFileNameW+0x75 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
031af418 5c3c5c2f cdb656cc 03381a00 033b4978 rsintcor32+0x7835
031af44c 5c3c7229 64cc0000 741c14c5 03381a00 rsintcor32+0x5c2f
031af69c 741c1749 03381a00 03381a00 03381a00 rsintcor32+0x7229
031af6c8 741c196f 00000228 741c1622 03381a00 csma_ldr32+0x1749
031af714 741c16bc 741c1962 741c2566 74e71432 csma_ldr32+0x196f
031af75c 741c272c cdb95aa2 00000000 00000000 csma_ldr32+0x16bc
031af7b0 74133433 00000000 cdb95a64 00000000 csma_ldr32!DllUnregisterServer+0x70d
031af7e8 741334c7 00000000 031af800 74e7338a msvcr90!_endthreadex+0x44 (FPO: [Non-Fpo])
031af7f4 74e7338a 03381a00 031af840 7709bf32 msvcr90!_endthreadex+0xd8 (FPO: [Non-Fpo])
031af800 7709bf32 03381a00 61fea085 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
031af840 7709bf05 7413345e 03381a00 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
031af858 00000000 7413345e 03381a00 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   2  Id: 3e18.1454 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr  Args to Child             
036afaec 7709c6c5 00000023 00526270 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
036afc80 74e7338a 00000000 036afccc 7709bf32 ntdll!TppWaiterpThread+0x33d (FPO: [Non-Fpo])
036afc8c 7709bf32 00526240 618ea409 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
036afccc 7709bf05 7709c599 00526240 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
036afce4 00000000 7709c599 00526240 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   3  Id: 3e18.5780 Suspend: 1 Teb: 7efac000 Unfrozen
ChildEBP RetAddr  Args to Child             
0464ea24 767415f7 00000002 0464ea74 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
0464eac0 74e719f8 0464ea74 0464eae8 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
0464eb08 74e74200 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
0464eb24 75093e16 00000002 0055b798 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
0464fb4c 750b2f6a 0055b748 0464fb70 5924febb iertutil!CForeignProcessToCurrentProcessMessaging::_vThreadProc+0xa5 (FPO: [Non-Fpo])
0464fb58 5924febb 00577b98 00000000 00000000 iertutil!CForeignProcessToCurrentProcessMessaging::_sThreadProc+0xe (FPO: [Non-Fpo])
0464fb70 74e7338a 0055b748 0464fbbc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0464fb7c 7709bf32 0055b748 6680a379 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0464fbbc 7709bf05 5924fe98 0055b748 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0464fbd4 00000000 5924fe98 0055b748 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   4  Id: 3e18.4808 Suspend: 1 Teb: 7efa9000 Unfrozen
ChildEBP RetAddr  Args to Child             
0486f730 767415f7 00000001 0486f780 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
0486f7cc 74e719f8 0486f780 0486f7f4 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
0486f814 74a9086a 00000001 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
0486f868 750a874c 00000708 00000000 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x14d (FPO: [Non-Fpo])
0486f8c4 750b2e2f 00000001 00000000 750b2dd4 iertutil!IsoThreadWindowsPumpInit+0x266 (FPO: [Non-Fpo])
0486f8e8 5924febb 0056cc80 00000000 00000000 iertutil!IsoManagerThreadNonzero_WindowsPump+0x5b (FPO: [Non-Fpo])
0486f900 74e7338a 0055b768 0486f94c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0486f90c 7709bf32 0055b768 6662a189 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0486f94c 7709bf05 5924fe98 0055b768 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0486f964 00000000 5924fe98 0055b768 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   5  Id: 3e18.59cc Suspend: 1 Teb: 7efa6000 Unfrozen
ChildEBP RetAddr  Args to Child             
04b2fc74 767415f7 00000003 04b2fcc4 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
04b2fd10 74e719f8 04b2fcc4 04b2fd38 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
04b2fd58 74e74200 00000003 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
04b2fd74 59263f31 00000003 04b2fd98 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
04b2fda8 5924febb 00000001 00000000 00000000 ieframe!MTAThread+0x54 (FPO: [Non-Fpo])
04b2fdc0 74e7338a 0055b7a8 04b2fe0c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
04b2fdcc 7709bf32 0055b7a8 6656a6c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
04b2fe0c 7709bf05 5924fe98 0055b7a8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
04b2fe24 00000000 5924fe98 0055b7a8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   6  Id: 3e18.30a8 Suspend: 1 Teb: 7ef9f000 Unfrozen
ChildEBP RetAddr  Args to Child             
04e6fe1c 76743bd5 00000000 04e6fe60 cb691217 ntdll!ZwDelayExecution+0x15 (FPO: [2,0,0])
04e6fe84 767444a5 0000ea60 00000000 04e6febc KERNELBASE!SleepEx+0x65 (FPO: [Non-Fpo])
04e6fe94 74b9d98d 0000ea60 00578278 74b9cd48 KERNELBASE!Sleep+0xf (FPO: [Non-Fpo])
04e6fea0 74b9cd48 00000000 74b9d864 00578278 ole32!CROIDTable::WorkerThreadLoop+0x14 (FPO: [1,0,4]) (CONV: stdcall) [d:\w7rtm\com\ole32\com\dcomrem\refcache.cxx @ 1345]
04e6febc 74b9d87a 74b9d864 0055b878 04e6fee4 ole32!CRpcThread::WorkerLoop+0x26 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\w7rtm\com\ole32\com\dcomrem\threads.cxx @ 257]
04e6fecc 5924febb 00578278 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x16 (FPO: [Non-Fpo]) (CONV: stdcall) [d:\w7rtm\com\ole32\com\dcomrem\threads.cxx @ 63]
04e6fee4 74e7338a 0055b878 04e6ff30 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
04e6fef0 7709bf32 0055b878 6602a7f5 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
04e6ff30 7709bf05 5924fe98 0055b878 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
04e6ff48 00000000 5924fe98 0055b878 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   7  Id: 3e18.39d4 Suspend: 1 Teb: 7ef9c000 Unfrozen
ChildEBP RetAddr  Args to Child             
04fefcb0 770b1ad0 000002e0 04fefd64 661aa6d5 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
04fefe10 74e7338a 00526478 04fefe5c 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
04fefe1c 7709bf32 00526478 661aa699 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
04fefe5c 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
04fefe74 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   8  Id: 3e18.47a8 Suspend: 1 Teb: 7ef99000 Unfrozen
ChildEBP RetAddr  Args to Child             
051bfa54 767414ab 00000540 00000001 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
051bfac0 74e71194 00000540 ffffffff 00000001 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
051bfad8 67e633b7 00000540 ffffffff 00000001 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
051bfb3c 5924febb 00000000 00000000 00000000 rasman!RasmanServiceMonitorThread+0xe7 (FPO: [Non-Fpo])
051bfb54 74e7338a 005b2eb0 051bfba0 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
051bfb60 7709bf32 005b2eb0 67ffa365 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
051bfba0 7709bf05 5924fe98 005b2eb0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
051bfbb8 00000000 5924fe98 005b2eb0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   9  Id: 3e18.4f94 Suspend: 1 Teb: 7ef96000 Unfrozen
ChildEBP RetAddr  Args to Child             
0541fa5c 767414ab 00000580 00000000 0541faa4 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0541fac8 74e71194 00000580 001b7740 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0541fae0 74e71148 00000580 001b7740 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0541faf4 765dc964 00000580 001b7740 00572a18 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0541fb14 765dc8be 00572a18 001b7740 00000000 wininet!AutoProxyResolver::WaitForMessage+0x6b (FPO: [Non-Fpo])
0541fb30 765dc825 00572a18 0541fb4c 00000000 wininet!AutoProxyResolver::PumpProxyMessage+0x75 (FPO: [Non-Fpo])
0541fb68 765dc73b 00572a18 765dc631 005b2f30 wininet!AutoProxyResolver::ProcessMessages+0x54 (FPO: [Non-Fpo])
0541fd1c 765dc63e 00572a18 0541fd40 5924febb wininet!AutoProxyResolver::AutoProxyThread+0x12a (FPO: [Non-Fpo])
0541fd28 5924febb 00572a18 00000000 00000000 wininet!AutoProxyResolver::AutoProxyThreadStart+0xd (FPO: [Non-Fpo])
0541fd40 74e7338a 005b2f30 0541fd8c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0541fd4c 7709bf32 005b2f30 67a5a549 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0541fd8c 7709bf05 5924fe98 005b2f30 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0541fda4 00000000 5924fe98 005b2f30 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  10  Id: 3e18.34ac Suspend: 1 Teb: 7ef8f000 Unfrozen
ChildEBP RetAddr  Args to Child             
05b8f904 767415f7 00000002 05b8f954 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
05b8f9a0 74e719f8 05b8f954 05b8f9c8 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
05b8f9e8 74a9086a 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
05b8fa3c 699e2006 000005c8 05b8fa74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x14d (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
05b8fa60 699e6b29 000004ff ffffffff 00000001 ieui+0x2006
05b8fa94 699e9344 05b8fad4 00000000 00000000 ieui!SetGadgetParent+0x6fa
05b8fab4 699e92a4 05b8fad4 00000000 00000000 ieui!GetMessageExA+0x3b
05b8fb08 749d1287 00000000 ca093af1 749d12e5 ieui!DllMain+0x407
05b8fb40 749d1328 05b8fb60 5924febb 006bdf88 msvcrt!_endthreadex+0x44 (FPO: [Non-Fpo])
05b8fb48 5924febb 006bdf88 00000000 00000000 msvcrt!_endthreadex+0xce (FPO: [Non-Fpo])
05b8fb60 74e7338a 0465fc68 05b8fbac 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
05b8fb6c 7709bf32 0465fc68 675ca369 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05b8fbac 7709bf05 5924fe98 0465fc68 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
05b8fbc4 00000000 5924fe98 0465fc68 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  11  Id: 3e18.2478 Suspend: 1 Teb: 7ef8c000 Unfrozen
ChildEBP RetAddr  Args to Child             
05dbf644 770b1ad0 000002dc 05dbf6f8 673faf61 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
05dbf7a4 74e7338a 00526478 05dbf7f0 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
05dbf7b0 7709bf32 00526478 673faf35 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05dbf7f0 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
05dbf808 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  12  Id: 3e18.34d8 Suspend: 1 Teb: 7ef89000 Unfrozen
ChildEBP RetAddr  Args to Child             
0551fb00 770b1ad0 000002e0 0551fbb4 67b5a4a5 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0551fc60 74e7338a 00526478 0551fcac 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0551fc6c 7709bf32 00526478 67b5a469 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0551fcac 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0551fcc4 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  13  Id: 3e18.d44 Suspend: 1 Teb: 7ef83000 Unfrozen
ChildEBP RetAddr  Args to Child             
0624f5d0 767415f7 00000001 0624f620 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
0624f66c 74e719f8 0624f620 0624f694 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
0624f6b4 74a9086a 00000001 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
0624f708 750a874c 00000624 00000000 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x14d (FPO: [Non-Fpo])
0624f764 750afc2c 00000001 00000000 005aeb00 iertutil!IsoThreadWindowsPumpInit+0x266 (FPO: [Non-Fpo])
0624f778 750b3418 006beae8 0466a3f0 0624f7a0 iertutil!IsoThreadWindowsPump+0x12 (FPO: [Non-Fpo])
0624f788 5924febb 005aeb00 00000000 00000000 iertutil!CIsoScope::RegisterThread+0xab (FPO: [Non-Fpo])
0624f7a0 74e7338a 0466a3f0 0624f7ec 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0624f7ac 7709bf32 0466a3f0 64c0af29 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0624f7ec 7709bf05 5924fe98 0466a3f0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0624f804 00000000 5924fe98 0466a3f0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  14  Id: 3e18.5db0 Suspend: 1 Teb: 7ef7f000 Unfrozen
ChildEBP RetAddr  Args to Child             
05edf930 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
05edf994 7709b398 00000000 00000000 0000fff8 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
05edf9bc 770b650d 771620c0 6709a291 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
05edfa54 770b6786 00000002 00000000 05edfbc0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
05edfa64 770c0289 00000000 6709a305 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
05edfbc0 74e7338a 00526478 05edfc0c 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
05edfbcc 7709bf32 00526478 6709a4c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05edfc0c 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
05edfc24 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  15  Id: 3e18.4f1c Suspend: 1 Teb: 7ef79000 Unfrozen
ChildEBP RetAddr  Args to Child             
0756fc5c 71a1635c 000006c0 0756fc90 0756fc84 ntdll!NtRemoveIoCompletion+0x15 (FPO: [5,0,0])
0756fc88 5924febb 71a164b3 00000000 00000000 mswsock!SockAsyncThread+0x83 (FPO: [Non-Fpo])
0756fca0 74e7338a 0468ffc8 0756fcec 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0756fcac 7709bf32 0468ffc8 65b2a429 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0756fcec 7709bf05 5924fe98 0468ffc8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0756fd04 00000000 5924fe98 0468ffc8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  16  Id: 3e18.35e0 Suspend: 1 Teb: 7ef76000 Unfrozen
ChildEBP RetAddr  Args to Child             
07a1d8c8 59247c31 046f8e70 046f8e8c 006bf680 user32!NtUserWaitMessage+0x15 (FPO: [0,0,0])
07a1f9ec 59261976 046f8e70 046fc0d0 750b340a ieframe!CTabWindow::_TabWindowThreadProc+0x7d1 (FPO: [1,2115,4])
07a1faa8 750b3418 006bf680 046dff10 07a1fad0 ieframe!LCIETab_ThreadProc+0x317 (FPO: [Non-Fpo])
07a1fab8 5924febb 046fc0d0 00000000 00000000 iertutil!CIsoScope::RegisterThread+0xab (FPO: [Non-Fpo])
07a1fad0 74e7338a 046dff10 07a1fb1c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
07a1fadc 7709bf32 046dff10 6545a3d9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07a1fb1c 7709bf05 5924fe98 046dff10 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
07a1fb34 00000000 5924fe98 046dff10 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  17  Id: 3e18.31b4 Suspend: 1 Teb: 7ef73000 Unfrozen
ChildEBP RetAddr  Args to Child             
07b3f654 767414ab 00000758 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
07b3f6c0 74e71194 00000758 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
07b3f6d8 74e71148 00000758 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
07b3f6ec 65dea262 00000758 ffffffff 65f84f60 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
07b3f72c 65de7b65 c9f34248 05bdd8d4 05bdd878 swi_filter!HTTPFilterIsEos+0x57872
07b3f76c 65de80d7 05bd28d8 07b3f7bc 05c067b8 swi_filter!HTTPFilterIsEos+0x55175
07b3f7cc 65de99f4 c9f34d24 05bdd8d4 05bdd878 swi_filter!HTTPFilterIsEos+0x556e7
07b3f800 65de8c46 c9f34d1c 65e42d14 0470e9c8 swi_filter!HTTPFilterIsEos+0x57004
07b3f838 65e34adb 65e42d14 0470e9c8 05c36d10 swi_filter!HTTPFilterIsEos+0x56256
07b3f864 65e42cee 05bdd87c c9f34db8 65e42d14 swi_filter!HTTPFilterIsEos+0xa20eb
07b3f89c 65e42d96 0470e9c8 07b3f8c0 5924febb swi_filter!HTTPFilterIsEos+0xb02fe
07b3f8a8 5924febb 05c36d10 00000000 00000000 swi_filter!HTTPFilterIsEos+0xb03a6
07b3f8c0 74e7338a 0470e9c8 07b3f90c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
07b3f8cc 7709bf32 0470e9c8 6557a1c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07b3f90c 7709bf05 5924fe98 0470e9c8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
07b3f924 00000000 5924fe98 0470e9c8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  18  Id: 3e18.3d88 Suspend: 1 Teb: 7ef93000 Unfrozen
ChildEBP RetAddr  Args to Child             
0568f914 71a16f1f 000006d4 00000001 0568f93c ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0568f954 71a16d40 000006d4 00000780 00000001 mswsock!SockWaitForSingleObject+0x1ba (FPO: [Non-Fpo])
0568fa40 755e6a28 00000001 0568fd68 0568fc64 mswsock!WSPSelect+0x3a6 (FPO: [Non-Fpo])
0568fac0 5c3cfaea 00000001 0568fd68 0568fc64 ws2_32!select+0x494 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0568fb0c 765e36f3 00000001 0568fd68 0568fc64 rsintcor32!RslLoadedTerm+0x7c6e
0568fe70 765eccbb 0568fe90 5924febb 0469deb0 wininet!ICAsyncThread::SelectThread+0x381 (FPO: [Non-Fpo])
0568fe78 5924febb 0469deb0 00000000 00000000 wininet!ICAsyncThread::SelectThreadWrapper+0xd (FPO: [Non-Fpo])
0568fe90 74e7338a 0470ea18 0568fedc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0568fe9c 7709bf32 0470ea18 678ca619 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0568fedc 7709bf05 5924fe98 0470ea18 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0568fef4 00000000 5924fe98 0470ea18 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  19  Id: 3e18.4dec Suspend: 1 Teb: 7ef7c000 Unfrozen
ChildEBP RetAddr  Args to Child             
07e8fd18 767414ab 00000888 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
07e8fd84 74e71194 00000888 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
07e8fd9c 74e71148 00000888 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
07e8fdb0 6c396bf0 00000888 ffffffff 74e713d0 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
07e8fdc0 6c396e4a 00000100 07e8fe40 00000001 winsta!CWaitEventCollect::WaitEvent+0x13 (FPO: [0,0,4])
07e8fdf4 74053072 00000000 00000008 07e8fe14 winsta!WinStationWaitSystemEvent+0x243 (FPO: [Non-Fpo])
07e8fe18 65c72111 00000000 00000008 07e8fe40 wtsapi32!WTSWaitSystemEvent+0x8d (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
07e8fe38 5924febb 65cb7380 00000000 00000000 AppSenseURLFilter+0x2111
07e8fe50 74e7338a 0470eac8 07e8fe9c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
07e8fe5c 7709bf32 0470eac8 650ca659 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07e8fe9c 7709bf05 5924fe98 0470eac8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
07e8feb4 00000000 5924fe98 0470eac8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  20  Id: 3e18.4bc4 Suspend: 1 Teb: 7ef6f000 Unfrozen
ChildEBP RetAddr  Args to Child             
080afa9c 74a8790d 080afb18 00000000 00000000 user32!NtUserGetMessage+0x15 (FPO: [4,0,0])
080afab8 5c3c1722 080afb18 00000000 00000000 user32!GetMessageW+0x33 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
080afaf0 65c80701 080afb18 00000000 00000000 rsintcor32+0x1722
080afb78 5924febb 65cb7380 00000000 00000000 AppSenseURLFilter!DllUnregisterServer+0x1c61
080afb90 74e7338a 0470ead8 080afbdc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
080afb9c 7709bf32 0470ead8 6aeea319 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
080afbdc 7709bf05 5924fe98 0470ead8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
080afbf4 00000000 5924fe98 0470ead8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  21  Id: 3e18.2994 Suspend: 1 Teb: 7ef6c000 Unfrozen
ChildEBP RetAddr  Args to Child             
081ff788 767415f7 00000002 081ff7d8 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
081ff824 74e719f8 081ff7d8 081ff84c 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
081ff86c 74e74200 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
081ff888 65c8d7ba 00000002 081ff8b0 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
081ff8d4 65c808c1 081ff914 65c91401 65cb73cc AppSenseURLFilter!DllUnregisterServer+0xed1a
081ff8dc 65c91401 65cb73cc f9ff7d20 65c91427 AppSenseURLFilter!DllUnregisterServer+0x1e21
081ff914 65c9148b 0470eac8 081ff938 5924febb AppSenseURLFilter!DllUnregisterServer+0x12961
081ff920 5924febb 06fe1260 00000000 00000000 AppSenseURLFilter!DllUnregisterServer+0x129eb
081ff938 74e7338a 0470eac8 081ff984 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
081ff944 7709bf32 0470eac8 6afba141 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
081ff984 7709bf05 5924fe98 0470eac8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
081ff99c 00000000 5924fe98 0470eac8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  22  Id: 3e18.3ddc Suspend: 1 Teb: 7ef66000 Unfrozen
ChildEBP RetAddr  Args to Child             
08d7fbc8 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
08d7fc2c 7709b398 00000000 00000000 00000003 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
08d7fc54 770b650d 771620c0 6a33a429 005046f0 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
08d7fcec 770b6786 00000002 00000000 08d7fe58 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
08d7fcfc 770c0289 00000000 6a33a69d 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
08d7fe58 74e7338a 005046f0 08d7fea4 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
08d7fe64 7709bf32 005046f0 6a33a661 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
08d7fea4 7709bf05 770b25c1 005046f0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
08d7febc 00000000 770b25c1 005046f0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  23  Id: 3e18.5b68 Suspend: 1 Teb: 7ef63000 Unfrozen
ChildEBP RetAddr  Args to Child             
08e7fa88 767414ab 0000094c 00000000 08e7fad0 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
08e7faf4 74e71194 0000094c 000927c0 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
08e7fb0c 74e71148 0000094c 000927c0 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
08e7fb20 63ec0770 0000094c 000927c0 63d3c527 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
08e7fb80 63eca2da 63ac0000 08e7fb98 63d3c535 mshtml!CDwnTaskExec::ThreadExec+0x401 (FPO: [0,17,4])
08e7fb8c 63d3c535 07b89160 08e7fbb0 5924febb mshtml!CExecFT::ThreadProc+0x4b (FPO: [Non-Fpo])
08e7fb98 5924febb 0056a428 00000000 00000000 mshtml!CExecFT::StaticThreadProc+0xe (FPO: [Non-Fpo])
08e7fbb0 74e7338a 07b89160 08e7fbfc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
08e7fbbc 7709bf32 07b89160 6a03a339 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
08e7fbfc 7709bf05 5924fe98 07b89160 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
08e7fc14 00000000 5924fe98 07b89160 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  24  Id: 3e18.4bac Suspend: 1 Teb: 7ef5f000 Unfrozen
ChildEBP RetAddr  Args to Child             
0913f700 770b1ad0 000009c4 0913f7b4 6bf7a0a5 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0913f860 74e7338a 07bc76d8 0913f8ac 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0913f86c 7709bf32 07bc76d8 6bf7a069 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0913f8ac 7709bf05 770b25c1 07bc76d8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0913f8c4 00000000 770b25c1 07bc76d8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  25  Id: 3e18.38c4 Suspend: 1 Teb: 7ef5c000 Unfrozen
ChildEBP RetAddr  Args to Child             
0ab4fc48 767414ab 00000a14 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0ab4fcb4 74e71194 00000a14 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0ab4fccc 74e71148 00000a14 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0ab4fce0 63edf509 00000a14 ffffffff 63d3c527 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0ab4fd10 63eca2da 63ac0000 0ab4fd28 63d3c535 mshtml!CTimerMan::ThreadExec+0x119 (FPO: [0,5,4])
0ab4fd1c 63d3c535 07bd85c0 0ab4fd40 5924febb mshtml!CExecFT::ThreadProc+0x4b (FPO: [Non-Fpo])
0ab4fd28 5924febb 07b79100 00000000 00000000 mshtml!CExecFT::StaticThreadProc+0xe (FPO: [Non-Fpo])
0ab4fd40 74e7338a 07bd85c0 0ab4fd8c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0ab4fd4c 7709bf32 07bd85c0 6850a549 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0ab4fd8c 7709bf05 5924fe98 07bd85c0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0ab4fda4 00000000 5924fe98 07bd85c0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  26  Id: 3e18.3824 Suspend: 1 Teb: 7ef56000 Unfrozen
ChildEBP RetAddr  Args to Child             
0aebf864 767415f7 00000002 0aebf8b4 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
0aebf900 74e719f8 0aebf8b4 0aebf928 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
0aebf948 74e74200 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
0aebf964 5adc1400 00000002 0aebf988 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
0aebf990 5ae917af 00000002 f8fd95a9 749d12e5 jscript9!Recycler::ThreadProc+0x9e (FPO: [Non-Fpo])
0aebf9cc 749d1287 0100abf8 c55a3bb5 749d12e5 jscript9!Recycler::StaticThreadProc+0x4c (FPO: [Non-Fpo])
0aebfa04 749d1328 0aebfa24 5924febb 00ff1040 msvcrt!_endthreadex+0x44 (FPO: [Non-Fpo])
0aebfa0c 5924febb 00ff1040 00000000 00000000 msvcrt!_endthreadex+0xce (FPO: [Non-Fpo])
0aebfa24 74e7338a 07bd8730 0aebfa70 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0aebfa30 7709bf32 07bd8730 680fa2b5 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0aebfa70 7709bf05 5924fe98 07bd8730 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0aebfa88 00000000 5924fe98 07bd8730 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  27  Id: 3e18.41c8 Suspend: 1 Teb: 7ef53000 Unfrozen
ChildEBP RetAddr  Args to Child             
0ad8f918 767414ab 00000b10 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0ad8f984 74e71194 00000b10 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0ad8f99c 74e71148 00000b10 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0ad8f9b0 5adc1947 00000b10 ffffffff 0100d2b8 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0ad8f9d0 5adc19b7 f8ce9665 749d12e5 07bd8730 jscript9!BackgroundCodeGenThread::GetNextCodeGenWorkItem+0x1a2 (FPO: [0,2,0])
0ad8fa00 5ae9183c f8ce965d 749d12e5 07bd8730 jscript9!BackgroundCodeGenThread::MainProc+0xa0 (FPO: [Non-Fpo])
0ad8fa38 749d1287 0100d2b8 c5693bc1 749d12e5 jscript9!BackgroundCodeGenThread::StaticThreadProc+0x4b (FPO: [Non-Fpo])
0ad8fa70 749d1328 0ad8fa90 5924febb 00ff1040 msvcrt!_endthreadex+0x44 (FPO: [Non-Fpo])
0ad8fa78 5924febb 00ff1040 00000000 00000000 msvcrt!_endthreadex+0xce (FPO: [Non-Fpo])
0ad8fa90 74e7338a 07bd8730 0ad8fadc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0ad8fa9c 7709bf32 07bd8730 683ca219 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0ad8fadc 7709bf05 5924fe98 07bd8730 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0ad8faf4 00000000 5924fe98 07bd8730 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  28  Id: 3e18.dac Suspend: 1 Teb: 7ef49000 Unfrozen
ChildEBP RetAddr  Args to Child             
0b4ff7f8 770b1ad0 000002dc 0b4ff8ac 69aba19d ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0b4ff958 74e7338a 00526478 0b4ff9a4 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0b4ff964 7709bf32 00526478 69aba161 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0b4ff9a4 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0b4ff9bc 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  29  Id: 3e18.1a9c Suspend: 1 Teb: 7ef3f000 Unfrozen
ChildEBP RetAddr  Args to Child             
0c42f8d8 767414ab 00000c34 00000000 0c42f920 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0c42f944 74e71194 00000c34 000927c0 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0c42f95c 74e71148 00000c34 000927c0 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0c42f970 63ec0770 00000c34 000927c0 63d3c527 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0c42f9d0 63eca2da 63ac0000 0c42f9e8 63d3c535 mshtml!CDwnTaskExec::ThreadExec+0x401 (FPO: [0,17,4])
0c42f9dc 63d3c535 07c6e6c0 0c42fa00 5924febb mshtml!CExecFT::ThreadProc+0x4b (FPO: [Non-Fpo])
0c42f9e8 5924febb 07b66d00 00000000 00000000 mshtml!CExecFT::StaticThreadProc+0xe (FPO: [Non-Fpo])
0c42fa00 74e7338a 07c6e6c0 0c42fa4c 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0c42fa0c 7709bf32 07c6e6c0 6ea6a289 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0c42fa4c 7709bf05 5924fe98 07c6e6c0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0c42fa64 00000000 5924fe98 07c6e6c0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  30  Id: 3e18.22cc Suspend: 1 Teb: 7ef39000 Unfrozen
ChildEBP RetAddr  Args to Child             
0cacf9b8 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0cacfa1c 7709b398 00000000 00000000 0000fffd ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0cacfa44 770b650d 771620c0 6e48a219 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0cacfadc 770b6786 00000002 00000000 0cacfc48 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0cacfaec 770c0289 00000000 6e48a48d 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0cacfc48 74e7338a 00526478 0cacfc94 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
0cacfc54 7709bf32 00526478 6e48a451 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0cacfc94 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0cacfcac 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  31  Id: 3e18.3084 Suspend: 1 Teb: 7ef36000 Unfrozen
ChildEBP RetAddr  Args to Child             
0ccbfccc 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0ccbfd30 7709b398 00000000 00000000 0000fffc ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0ccbfd58 770b650d 771620c0 6e2fa535 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0ccbfdf0 770b6786 00000002 00000000 0ccbff5c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0ccbfe00 770c0289 00000000 6e2fa799 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0ccbff5c 74e7338a 00526478 0ccbffa8 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
0ccbff68 7709bf32 00526478 6e2fa76d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0ccbffa8 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0ccbffc0 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  32  Id: 3e18.4118 Suspend: 1 Teb: 7ef1c000 Unfrozen
ChildEBP RetAddr  Args to Child             
0df7f680 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0df7f6e4 7709b398 00000000 00000000 0000fffe ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0df7f70c 770b650d 771620c0 6f13af61 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0df7f7a4 770b6786 00000002 00000000 0df7f910 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0df7f7b4 770c0289 00000000 6f13a1d5 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0df7f910 74e7338a 00526478 0df7f95c 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
0df7f91c 7709bf32 00526478 6f13a199 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0df7f95c 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0df7f974 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  33  Id: 3e18.458c Suspend: 1 Teb: 7ef19000 Unfrozen
ChildEBP RetAddr  Args to Child             
0de3f6e0 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0de3f744 7709b398 00000000 00000000 0000fffb ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0de3f76c 770b650d 771620c0 6f07a0c1 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0de3f804 770b6786 00000002 00000000 0de3f970 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0de3f814 770c0289 00000000 6f07a1b5 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0de3f970 74e7338a 00526478 0de3f9bc 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
0de3f97c 7709bf32 00526478 6f07a179 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0de3f9bc 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0de3f9d4 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  34  Id: 3e18.4b90 Suspend: 1 Teb: 7ef4f000 Unfrozen
ChildEBP RetAddr  Args to Child             
0d17fa8c 770b1ad0 00000d64 0d17fb40 6ff3a329 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0d17fbec 74e7338a 107ff210 0d17fc38 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0d17fbf8 7709bf32 107ff210 6ff3a4fd 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0d17fc38 7709bf05 770b25c1 107ff210 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0d17fc50 00000000 770b25c1 107ff210 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  35  Id: 3e18.3c18 Suspend: 1 Teb: 7ef46000 Unfrozen
ChildEBP RetAddr  Args to Child             
0d46fb98 7673770d 00000dc4 0d46fc60 0d46fbdc ntdll!NtRemoveIoCompletion+0x15 (FPO: [5,0,0])
0d46fbc4 5c3c187b 00000dc4 0d46fc4c 0d46fc60 KERNELBASE!GetQueuedCompletionStatus+0x29 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0d46fc10 73ff01dc 00000dc4 0d46fc4c 0d46fc60 rsintcor32+0x187b
0d46fc74 7401c20c c2fc9c49 7401c1d0 0470eba8 scardhook!gvch::IoCompletionPort::WorkerThread::Execute+0x5c (FPO: [Non-Fpo])
0d46fca4 5924febb 02c48e48 00000000 00000000 scardhook!ctxb::Thread::ThreadProc+0x3c (FPO: [Non-Fpo])
0d46fcbc 74e7338a 0470eba8 0d46fd08 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0d46fcc8 7709bf32 0470eba8 6fa2a5cd 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0d46fd08 7709bf05 5924fe98 0470eba8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0d46fd20 00000000 5924fe98 0470eba8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  36  Id: 3e18.5da8 Suspend: 1 Teb: 7ef69000 Unfrozen
ChildEBP RetAddr  Args to Child             
0d9eb37c 767414ab 00000cc8 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0d9eb3e8 74e71194 00000cc8 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
0d9eb400 74e71148 00000cc8 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
0d9eb414 5a7a31e2 00000cc8 ffffffff 0ee88e80 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
0d9eb434 5a7a3359 0d9ee6a8 00000000 52fb0000 MMDevAPI!CDeviceEnumerator::DestroyHWndNotificationThread+0xf6 (FPO: [Non-Fpo])
0d9eb444 5a7a24c0 00000003 00050418 00000000 MMDevAPI!CDeviceEnumerator::ReleaseHWndNotification+0x29 (FPO: [0,0,4])
0d9eb458 5305bc4f 0ee88e00 011ce808 5c0eaf18 MMDevAPI!CDeviceEnumerator::UnregisterEndpointNotificationCallback+0x7e (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0d9ebcbc 5305b3fb 5c0ed6b4 011ce528 73736553 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xa9adf
0d9ec510 5305ec4f 011ce528 0d9ecd88 53022718 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xa928b
0d9ec51c 53022718 00000001 5c0ede2c cccccccc PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xacadf
0d9ecd88 53022297 5c0ec64c 011bd900 011e0e20 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x705a8
0d9ed5e8 53066055 5c0ecdf0 011cef20 011e0ff0 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x70127
0d9ede54 52fb25b1 5c0ef510 0d9ef7ac 00000003 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xb3ee5
0d9ee6b4 52fbaf16 011e0e20 0d9eef58 530653c1 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x441
0d9ee6c0 530653c1 00000001 5c0efcfc 0d9ee6fc PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x8da6
0d9eef58 52fb10f7 5c0ee41c 00000000 00000003 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0xb3251
0d9ef7b8 52fdb209 52fb0000 00000003 00000000 PseudoServerInproc2+0x10f7
0d9ef7fc 52fdb2c2 52fb0000 665d1fed 52fb0000 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x29099
0d9ef82c 665d1f5d 52fb0000 00000003 00000000 PseudoServerInproc2!PseudoServer_IsURLInDynamicBlacklist+0x29152
0d9ef860 7709b990 52fb0000 00000003 00000000 IEShims!CShimBindings::s_DllMainHook+0x4a (FPO: [Non-Fpo])
0d9ef880 770b659f 665d1f14 52fb0000 00000003 ntdll!LdrpCallInitRoutine+0x14
0d9ef924 770b6786 00000000 00000000 0d9ef94c ntdll!LdrShutdownThread+0xe6 (FPO: [Non-Fpo])
0d9ef934 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0d9ef94c 74e7338a 07c1f260 0d9ef998 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
0d9ef958 7709bf32 07c1f260 6f7aa15d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0d9ef998 7709bf05 5924fe98 07c1f260 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0d9ef9b0 00000000 5924fe98 07c1f260 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  37  Id: 3e18.4b84 Suspend: 1 Teb: 7ef4c000 Unfrozen
ChildEBP RetAddr  Args to Child             
0d2af6c0 770b1ad0 00000ce0 0d2af774 6fcea0e5 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0d2af820 74e7338a 1083f7a8 0d2af86c 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0d2af82c 7709bf32 1083f7a8 6fcea0a9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0d2af86c 7709bf05 770b25c1 1083f7a8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0d2af884 00000000 770b25c1 1083f7a8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  38  Id: 3e18.5a40 Suspend: 1 Teb: 7ef29000 Unfrozen
ChildEBP RetAddr  Args to Child             
100ef95c 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
100ef9c0 7709b398 00000000 00000000 6e700000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
100ef9e8 770a3ab8 771620c0 72eaa2ed 100efb18 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
100efa28 5c3c1550 6e700000 dea258d4 100efb18 ntdll!LdrUnloadDll+0x2a (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
100efa54 76742d2c 6e700000 00000000 100efabc rsintcor32+0x1550
100efa64 7367f5c6 6e700000 100efb18 109b0f78 KERNELBASE!FreeLibrary+0x15 (FPO: [Non-Fpo])
100efabc 73695252 6e700000 100efb04 7369a081 sophos_detoured+0xf5c6
100efac8 7369a081 6e700000 00005a40 754f825e sophos_detoured+0x25252
100efb04 754f8aae 6e700000 6b637453 754f8210 sophos_detoured!Detoured+0x1401
100efb3c 754f82ed 00000000 00000000 754f82a5 crypt32!FreeDllWaitForCallback+0x161 (FPO: [Non-Fpo])
100efb58 5924febb 00000001 00000000 00000000 crypt32!ILS_WaitForThreadProc+0x44 (FPO: [Non-Fpo])
100efb70 74e7338a 07bf6ee8 100efbbc 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
100efb7c 7709bf32 07bf6ee8 72eaa379 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
100efbbc 7709bf05 5924fe98 07bf6ee8 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
100efbd4 00000000 5924fe98 07bf6ee8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  39  Id: 3e18.2bbc Suspend: 1 Teb: 7ef1f000 Unfrozen
ChildEBP RetAddr  Args to Child             
116ff7cc 767415f7 00000002 116ff81c 00000001 ntdll!ZwWaitForMultipleObjects+0x15 (FPO: [5,0,0])
116ff868 74e719f8 116ff81c 116ff890 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
116ff8b0 74e74200 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
116ff8cc 5adc1400 00000002 116ff8f0 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
116ff8f8 5ae917af 00000002 e3799551 749d12e5 jscript9!Recycler::ThreadProc+0x9e (FPO: [Non-Fpo])
116ff934 749d1287 0f1c18f8 dede38dd 749d12e5 jscript9!Recycler::StaticThreadProc+0x4c (FPO: [Non-Fpo])
116ff96c 749d1328 116ff98c 5924febb 010588b0 msvcrt!_endthreadex+0x44 (FPO: [Non-Fpo])
116ff974 5924febb 010588b0 00000000 00000000 msvcrt!_endthreadex+0xce (FPO: [Non-Fpo])
116ff98c 74e7338a 046cb820 116ff9d8 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
116ff998 7709bf32 046cb820 738ba11d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
116ff9d8 7709bf05 5924fe98 046cb820 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
116ff9f0 00000000 5924fe98 046cb820 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  40  Id: 3e18.2a14 Suspend: 1 Teb: 7ef13000 Unfrozen
ChildEBP RetAddr  Args to Child             
1183fcd8 767414ab 00001120 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
1183fd44 74e71194 00001120 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98 (FPO: [Non-Fpo])
1183fd5c 74e71148 00001120 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75 (FPO: [Non-Fpo])
1183fd70 5adc1947 00001120 ffffffff 01090160 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
1183fd94 5adc19b7 e39591a1 749d12e5 07c9b738 jscript9!BackgroundCodeGenThread::GetNextCodeGenWorkItem+0x1a2 (FPO: [0,2,0])
1183fdc4 5ae9183c e3959199 749d12e5 07c9b738 jscript9!BackgroundCodeGenThread::MainProc+0xa0 (FPO: [Non-Fpo])
1183fdfc 749d1287 01090160 de323f85 749d12e5 jscript9!BackgroundCodeGenThread::StaticThreadProc+0x4b (FPO: [Non-Fpo])
1183fe34 749d1328 1183fe54 5924febb 010588b0 msvcrt!_endthreadex+0x44 (FPO: [Non-Fpo])
1183fe3c 5924febb 010588b0 00000000 00000000 msvcrt!_endthreadex+0xce (FPO: [Non-Fpo])
1183fe54 74e7338a 07c9b738 1183fea0 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
1183fe60 7709bf32 07c9b738 7367a665 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
1183fea0 7709bf05 5924fe98 07c9b738 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
1183feb8 00000000 5924fe98 07c9b738 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  41  Id: 3e18.16ac Suspend: 1 Teb: 7ef03000 Unfrozen
ChildEBP RetAddr  Args to Child             
11e1f7c4 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
11e1f828 7709b398 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
11e1f850 770902a9 771620c0 7305a029 00000074 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
11e1f8ec 770901e2 755e0000 00000000 00000074 ntdll!LdrGetProcedureAddressEx+0x159 (FPO: [Non-Fpo])
11e1f908 76741e59 755e0000 00000000 00000074 ntdll!LdrGetProcedureAddress+0x18 (FPO: [Non-Fpo])
11e1f930 73161e95 755e0000 00000074 109a0798 KERNELBASE!GetProcAddress+0x44 (FPO: [Non-Fpo])
11e1f978 73161eda 755e0000 73193040 0c8be458 webio!__delayLoadHelper2+0xe9 (FPO: [Non-Fpo])
11e1f9e0 731799ad 10000001 ff000002 00000000 webio!_tailMerge_WS2_32_dll+0xd
11e1fa20 731b1894 10000001 ff000002 00000000 webio!WebTerminate+0x22
11e1fa40 731b183f 11e1faac 11e1fa70 770b326f winhttp!WINHTTP_DLL::_Terminate+0x64 (FPO: [Non-Fpo])
11e1fa4c 770b326f 11e1faac 04746908 108007b8 winhttp!WINHTTP_DLL::_SafeTerminateDll+0x10 (FPO: [Non-Fpo])
11e1fa70 770b2b65 11e1faac 10800818 7305a315 ntdll!TppTimerpExecuteCallback+0x10f (FPO: [Non-Fpo])
11e1fbd0 74e7338a 00526478 11e1fc1c 7709bf32 ntdll!TppWorkerThread+0x572 (FPO: [Non-Fpo])
11e1fbdc 7709bf32 00526478 7305a4d9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
11e1fc1c 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
11e1fc34 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  42  Id: 3e18.5ddc Suspend: 1 Teb: 7ef33000 Unfrozen
ChildEBP RetAddr  Args to Child             
11a0fd7c 770b1ad0 000002d8 11a0fe30 7344a619 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
11a0fedc 74e7338a 00526478 11a0ff28 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
11a0fee8 7709bf32 00526478 7344a7ed 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
11a0ff28 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
11a0ff40 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  43  Id: 3e18.31b0 Suspend: 1 Teb: 7ef09000 Unfrozen
ChildEBP RetAddr  Args to Child             
12a4fd94 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
12a4fdf8 7709b398 00000000 00000000 0b6f1448 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
12a4fe20 770b650d 771620c0 7040a67d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
12a4feb8 770b6786 00000000 00000000 12a4fee0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
12a4fec8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
12a4fee0 74e7338a 0ee18b80 12a4ff2c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
12a4feec 7709bf32 0ee18b80 7040a7e9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
12a4ff2c 7709bf05 5924fe98 0ee18b80 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
12a4ff44 00000000 5924fe98 0ee18b80 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  44  Id: 3e18.23c4 Suspend: 1 Teb: 7eef6000 Unfrozen
ChildEBP RetAddr  Args to Child             
12bafce0 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
12bafd44 7709b398 00000000 00000000 0000fff9 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
12bafd6c 770b650d 771620c0 705ea6c1 00526478 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
12bafe04 770b6786 00000002 00000000 12baff70 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
12bafe14 770c0289 00000000 705ea7b5 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
12baff70 74e7338a 00526478 12baffbc 7709bf32 ntdll!TppWorkerThread+0x856 (FPO: [Non-Fpo])
12baff7c 7709bf32 00526478 705ea779 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
12baffbc 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
12baffd4 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  45  Id: 3e18.5f54 Suspend: 1 Teb: 7eef3000 Unfrozen
ChildEBP RetAddr  Args to Child             
12d8f920 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
12d8f984 7709b398 00000000 00000000 06cc2020 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
12d8f9ac 770b650d 771620c0 703ca281 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
12d8fa44 770b6786 00000000 00000000 12d8fa6c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
12d8fa54 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
12d8fa6c 74e7338a 04746d68 12d8fab8 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
12d8fa78 7709bf32 04746d68 703ca27d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
12d8fab8 7709bf05 5924fe98 04746d68 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
12d8fad0 00000000 5924fe98 04746d68 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  46  Id: 3e18.1880 Suspend: 1 Teb: 7eeef000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e71faf0 770b1ad0 000002d8 0e71fba4 6c95a495 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0e71fc50 74e7338a 00526478 0e71fc9c 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0e71fc5c 7709bf32 00526478 6c95a459 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0e71fc9c 7709bf05 770b25c1 00526478 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0e71fcb4 00000000 770b25c1 00526478 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  47  Id: 3e18.58f0 Suspend: 1 Teb: 7eeec000 Unfrozen
ChildEBP RetAddr  Args to Child             
12fcf694 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
12fcf6f8 7709b398 00000000 00000000 06cc1708 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
12fcf720 770b650d 771620c0 7018af7d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
12fcf7b8 770b6786 00000000 00000000 12fcf7e0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
12fcf7c8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
12fcf7e0 74e7338a 0ee8cf90 12fcf82c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
12fcf7ec 7709bf32 0ee8cf90 7018a0e9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
12fcf82c 7709bf05 5924fe98 0ee8cf90 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
12fcf844 00000000 5924fe98 0ee8cf90 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  48  Id: 3e18.4f18 Suspend: 1 Teb: 7eee9000 Unfrozen
ChildEBP RetAddr  Args to Child             
131ff674 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
131ff6d8 7709b398 00000000 00000000 0b6eeb78 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
131ff700 770b650d 771620c0 71fbaf5d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
131ff798 770b6786 00000000 00000000 131ff7c0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
131ff7a8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
131ff7c0 74e7338a 04746d68 131ff80c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
131ff7cc 7709bf32 04746d68 71fba0c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
131ff80c 7709bf05 5924fe98 04746d68 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
131ff824 00000000 5924fe98 04746d68 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  49  Id: 3e18.321c Suspend: 1 Teb: 7eee6000 Unfrozen
ChildEBP RetAddr  Args to Child             
1333fe20 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
1333fe84 7709b398 00000000 00000000 0b712b30 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
1333feac 770b650d 771620c0 71d7a781 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
1333ff44 770b6786 00000000 00000000 1333ff6c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
1333ff54 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
1333ff6c 74e7338a 0ee18b80 1333ffb8 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
1333ff78 7709bf32 0ee18b80 71d7a77d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
1333ffb8 7709bf05 5924fe98 0ee18b80 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
1333ffd0 00000000 5924fe98 0ee18b80 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  50  Id: 3e18.a64 Suspend: 1 Teb: 7eee3000 Unfrozen
ChildEBP RetAddr  Args to Child             
12e9f874 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
12e9f8d8 7709b398 00000000 00000000 06cc2328 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
12e9f900 770b650d 771620c0 700da15d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
12e9f998 770b6786 00000000 00000000 12e9f9c0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
12e9f9a8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
12e9f9c0 74e7338a 07bfd338 12e9fa0c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
12e9f9cc 7709bf32 07bfd338 700da2c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
12e9fa0c 7709bf05 5924fe98 07bfd338 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
12e9fa24 00000000 5924fe98 07bfd338 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  51  Id: 3e18.4828 Suspend: 1 Teb: 7ef86000 Unfrozen
ChildEBP RetAddr  Args to Child             
1345f784 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
1345f7e8 7709b398 00000000 00000000 06cd05e8 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
1345f810 770b650d 771620c0 71a1a06d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
1345f8a8 770b6786 00000000 00000000 1345f8d0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
1345f8b8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
1345f8d0 74e7338a 0ee8cf90 1345f91c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
1345f8dc 7709bf32 0ee8cf90 71a1a1d9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
1345f91c 7709bf05 5924fe98 0ee8cf90 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
1345f934 00000000 5924fe98 0ee8cf90 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  52  Id: 3e18.2f00 Suspend: 1 Teb: 7ef59000 Unfrozen
ChildEBP RetAddr  Args to Child             
136dfbe4 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
136dfc48 7709b398 00000000 00000000 0b6ee568 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
136dfc70 770b650d 771620c0 7189a5cd 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
136dfd08 770b6786 00000000 00000000 136dfd30 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
136dfd18 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
136dfd30 74e7338a 10a211c0 136dfd7c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
136dfd3c 7709bf32 10a211c0 7189a5b9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
136dfd7c 7709bf05 5924fe98 10a211c0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
136dfd94 00000000 5924fe98 10a211c0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  53  Id: 3e18.2e24 Suspend: 1 Teb: 7eedf000 Unfrozen
ChildEBP RetAddr  Args to Child             
0f97fc94 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0f97fcf8 7709b398 00000000 00000000 0b6ee260 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0f97fd20 770b650d 771620c0 6d73a57d 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0f97fdb8 770b6786 00000000 00000000 0f97fde0 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0f97fdc8 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0f97fde0 74e7338a 10a211c0 0f97fe2c 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
0f97fdec 7709bf32 10a211c0 6d73a6e9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0f97fe2c 7709bf05 5924fe98 10a211c0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0f97fe44 00000000 5924fe98 10a211c0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  54  Id: 3e18.5b9c Suspend: 1 Teb: 7eedc000 Unfrozen
ChildEBP RetAddr  Args to Child             
139ff750 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
139ff7b4 7709b398 00000000 00000000 06ccd9c8 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
139ff7dc 770b650d 771620c0 717ba0b1 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
139ff874 770b6786 00000000 00000000 139ff89c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
139ff884 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
139ff89c 74e7338a 0465ee48 139ff8e8 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
139ff8a8 7709bf32 0465ee48 717ba02d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
139ff8e8 7709bf05 5924fe98 0465ee48 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
139ff900 00000000 5924fe98 0465ee48 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  55  Id: 3e18.37c4 Suspend: 1 Teb: 7ef26000 Unfrozen
ChildEBP RetAddr  Args to Child             
13ccf9d0 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
13ccfa34 7709b398 00000000 00000000 0b6f0218 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
13ccfa5c 770b650d 771620c0 7128a231 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
13ccfaf4 770b6786 00000000 00000000 13ccfb1c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
13ccfb04 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
13ccfb1c 74e7338a 07c1f270 13ccfb68 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
13ccfb28 7709bf32 07c1f270 7128a3ad 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
13ccfb68 7709bf05 5924fe98 07c1f270 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
13ccfb80 00000000 5924fe98 07c1f270 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  56  Id: 3e18.3de4 Suspend: 1 Teb: 7ef23000 Unfrozen
ChildEBP RetAddr  Args to Child             
0eb0fe38 74a8790d 0eb0feb0 00000000 00000000 user32!NtUserGetMessage+0x15 (FPO: [4,0,0])
0eb0fe54 5c3c1722 0eb0feb0 00000000 00000000 user32!GetMessageW+0x33 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
0eb0fe8c 74b8a44e 0eb0feb0 00000000 00000000 rsintcor32+0x1722
0eb0fecc 74b8853b 00007530 74e71151 10a227f0 ole32!CDllHost::STAWorkerLoop+0x81 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\w7rtm\com\ole32\com\objact\dllhost.cxx @ 957]
0eb0fee8 74b8a4ac 0eb0ff0c 74b9cd48 74cb7b68 ole32!CDllHost::WorkerThread+0xd0 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\w7rtm\com\ole32\com\objact\dllhost.cxx @ 825]
0eb0fef0 74b9cd48 74cb7b68 74b9d864 10a227f0 ole32!DLLHostThreadEntry+0xd (FPO: [Non-Fpo]) (CONV: stdcall) [d:\w7rtm\com\ole32\com\objact\dllhost.cxx @ 758]
0eb0ff0c 74b9d87a 74b9d864 07c1fa10 0eb0ff34 ole32!CRpcThread::WorkerLoop+0x26 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\w7rtm\com\ole32\com\dcomrem\threads.cxx @ 257]
0eb0ff1c 5924febb 10a227f0 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x16 (FPO: [Non-Fpo]) (CONV: stdcall) [d:\w7rtm\com\ole32\com\dcomrem\threads.cxx @ 63]
0eb0ff34 74e7338a 07c1fa10 0eb0ff80 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
0eb0ff40 7709bf32 07c1fa10 6c54a745 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0eb0ff80 7709bf05 5924fe98 07c1fa10 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0eb0ff98 00000000 5924fe98 07c1fa10 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  57  Id: 3e18.2b80 Suspend: 1 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr  Args to Child             
0db5f924 770b1ad0 0000019c 0db5f9d8 6f51a241 ntdll!ZwWaitForWorkViaWorkerFactory+0x12 (FPO: [2,0,0])
0db5fa84 74e7338a 005046f0 0db5fad0 7709bf32 ntdll!TppWorkerThread+0x216 (FPO: [Non-Fpo])
0db5fa90 7709bf32 005046f0 6f51a215 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0db5fad0 7709bf05 770b25c1 005046f0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0db5fae8 00000000 770b25c1 005046f0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  58  Id: 3e18.518c Suspend: 1 Teb: 7efa3000 Unfrozen
ChildEBP RetAddr  Args to Child             
1037f850 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
1037f8b4 7709b398 00000000 00000000 06ccdfd8 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
1037f8dc 770b650d 771620c0 72d3a1b1 7197d290 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
1037f974 770b6786 00000000 00000000 1037f99c ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
1037f984 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
1037f99c 74e7338a 10a2d960 1037f9e8 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
1037f9a8 7709bf32 10a2d960 72d3a12d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
1037f9e8 7709bf05 5924fe98 10a2d960 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
1037fa00 00000000 5924fe98 10a2d960 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  59  Id: 3e18.6090 Suspend: 1 Teb: 7ef43000 Unfrozen
ChildEBP RetAddr  Args to Child             
0ffbfb14 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0ffbfb78 7709b398 00000000 00000000 011ce528 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0ffbfba0 770b650d 771620c0 6d1fa4fd 5305b630 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0ffbfc38 770b6786 00000000 00000000 0ffbfc60 ntdll!LdrShutdownThread+0x50 (FPO: [Non-Fpo])
0ffbfc48 5924fec9 00000000 00000000 00000000 ntdll!RtlExitUserThread+0x2a (FPO: [Non-Fpo])
0ffbfc60 74e7338a 13e4f778 0ffbfcac 7709bf32 ieframe!Detour_DefWindowProcA+0x7a (FPO: [Non-Fpo])
0ffbfc6c 7709bf32 13e4f778 6d1fa469 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0ffbfcac 7709bf05 5924fe98 13e4f778 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0ffbfcc4 00000000 5924fe98 13e4f778 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  60  Id: 3e18.42bc Suspend: 1 Teb: 7ef3c000 Unfrozen
ChildEBP RetAddr  Args to Child             
1200f544 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
1200f5a8 7709b398 00000000 00000000 1200f610 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
1200f5d0 770902a9 771620c0 70e4aea9 74aeaae6 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
1200f66c 770901e2 74780000 1200f6a8 00000000 ntdll!LdrGetProcedureAddressEx+0x159 (FPO: [Non-Fpo])
1200f688 76741e59 74780000 1200f6a8 00000000 ntdll!LdrGetProcedureAddress+0x18 (FPO: [Non-Fpo])
1200f6b0 74aad75a 74780000 74aeaae6 00000000 KERNELBASE!GetProcAddress+0x44 (FPO: [Non-Fpo])
1200f6f8 74aad6dc 74780000 74af0004 0002003e user32!__delayLoadHelper2+0xe9 (FPO: [Non-Fpo])
1200f71c 5a7a216d 07c9e240 5a7a27e1 13e4f708 user32!_tailMerge_CFGMGR32_dll+0xd
1200f7b4 5a7a27ee 1200f7d4 5924febb 0ee88e00 MMDevAPI!CDeviceEnumerator::PnpNotificationThread+0x33b (FPO: [Non-Fpo])
1200f7bc 5924febb 0ee88e00 00000000 00000000 MMDevAPI!CDeviceEnumerator::PnpNotificationThreadWrapper+0xd (FPO: [Non-Fpo])
1200f7d4 74e7338a 13e4f708 1200f820 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
1200f7e0 7709bf32 13e4f708 70e4a0e5 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
1200f820 7709bf05 5924fe98 13e4f708 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
1200f838 00000000 5924fe98 13e4f708 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  61  Id: 3e18.6074 Suspend: 1 Teb: 7ef2f000 Unfrozen
ChildEBP RetAddr  Args to Child             
14e0fbac 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
14e0fc10 7709b398 00000000 00000000 58580000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
14e0fc38 770a3ab8 771620c0 7604a4bd 585a2c40 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
14e0fc78 5c3c1550 58580000 da4c5e24 585a2c40 ntdll!LdrUnloadDll+0x2a (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
14e0fca4 76741833 58580000 07c9e2e0 0e3c3058 rsintcor32+0x1550
14e0fcb8 74e8d562 58580000 00000000 14e0fce4 KERNELBASE!FreeLibraryAndExitThread+0x28 (FPO: [Non-Fpo])
14e0fcc8 585a2d34 58580000 00000000 58580000 kernel32!FreeLibraryAndExitThreadStub+0x10 (FPO: [Non-Fpo])
14e0fce4 5924febb 00000001 00000000 00000000 icaendpoint!RegistryMonitor::ThreadProc+0xf4 (FPO: [Non-Fpo])
14e0fcfc 74e7338a 07c9e2e0 14e0fd48 7709bf32 ieframe!Detour_DefWindowProcA+0x6c (FPO: [Non-Fpo])
14e0fd08 7709bf32 07c9e2e0 7604a58d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
14e0fd48 7709bf05 5924fe98 07c9e2e0 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
14e0fd60 00000000 5924fe98 07c9e2e0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

  62  Id: 3e18.4038 Suspend: 1 Teb: 7ef2c000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e90f4ac 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0e90f510 7709b398 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0e90f538 7709c0e9 771620c0 6c74ad09 7ef2c000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0e90f5cc 7709be8c 0e90f63c 6c74aedd 00000000 ntdll!LdrpInitializeThread+0xc6 (FPO: [Non-Fpo])
0e90f618 7709beb9 0e90f63c 77060000 00000000 ntdll!_LdrpInitialize+0x1ad (FPO: [Non-Fpo])
0e90f628 00000000 0e90f63c 77060000 00000000 ntdll!LdrInitializeThunk+0x10 (FPO: [Non-Fpo])

  63  Id: 3e18.4c4 Suspend: 1 Teb: 7ef16000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e20f998 7709b4b4 00000210 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
0e20f9fc 7709b398 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
0e20fa24 7709c0e9 771620c0 6cc4a27d 7ef16000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
0e20fab8 7709be8c 0e20fb28 6cc4a3c1 00000000 ntdll!LdrpInitializeThread+0xc6 (FPO: [Non-Fpo])
0e20fb04 7709beb9 0e20fb28 77060000 00000000 ntdll!_LdrpInitialize+0x1ad (FPO: [Non-Fpo])
0e20fb14 00000000 0e20fb28 77060000 00000000 ntdll!LdrInitializeThunk+0x10 (FPO: [Non-Fpo])

Contacting Citrix support, they confirmed this was an issue within this component.

0:036> lmvm PseudoServerInproc2
start    end        module name
52fb0000 537af000   PseudoServerInproc2 PseudoServerInproc2.dll
    Loaded symbol image file: PseudoServerInproc2.dll
    Symbol file: PseudoServerInproc2.dll
    Image path: C:\Program Files (x86)\Citrix\system32\PseudoServerInproc2.dll
    Timestamp:        Sat Apr 12 01:51:11 2014 (53480F6F)
    CheckSum:         007FD94B
    ImageSize:        007FF000
    File version:     6.2.9.100
    Product version:  6.2.9.100
    File flags:       8 (Mask 3F) Private
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Citrix Systems, Inc.
    ProductName:      Citrix ICA Host
    InternalName:     HDX Flash v2 PseudoServerInproc
    OriginalFilename: PseudoServerInproc2.dll
    ProductVersion:   6.2
    FileVersion:      6.2.9.100
    FileDescription:  HDX MediaStream For Flash v2 Server DLL
    LegalCopyright:   Copyright 1990-2010 Citrix Systems, Inc.

Citrix have a private hotfix for this particular issue, if you experience the same you should contact Citrix support to receive the fix.

As a workaround to Flash Redirection issues you can use Citrix Policy to stop certain sites from using Flash Redirection, this will increase load on Citrix server.

To check if a site is using Flash redirection or not, take a screenshot within the Citrix session, if you see a black square where the Flash content is being played, then the site is being redirected. If you see the content in the screen capture it isn’t being redirected.

In addition to improve Flash Redirection performance ensure you are using the latest Citrix Receiver Client, and supported browser/flash combination on both server and client.

Note:

One of the affected web applications was thought not to use Flash. But taking a Fiddler trace and searching for .swf we found a JavaScript file

image

Checking this JavaScript file we found it initiated Flash.

Posted in Citrix, Internet Explorer, WinDbg | Tagged | Leave a comment

Patching a PAC File To Improve Performance

Having taken the trouble to write a PAC file debugger ( http://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/ ) I have to say I’ve seen some pretty horrendous PAC files, where attempts have been made to put the entire network design in this little JavaScript file. In the more extreme cases I consider the PAC file is more like a major application to manage that requires a specialist development team.

By far the most frequent cause of hangs/lockups/slow web performance I’ve seen due to PAC file being processed is because of DNS lookups, in particular IsInNet. Check http://www.websense.com/content/support/library/web/v76/pac_file_best_practices/PAC_best_pract.aspx for some good tips on high performance PAC files.

In particular this issue seems to be worse when the websites request a hostname that can’t resolve successfully. In many cases this failure to resolve a hostname will happen in the background and not be visible to the user, but can be diagnosed with a packet capture tool like WireShark, Network Monitor, Fiddler/etc.

In some cases however because the logic of the PAC file has grown so complex, it can take significant effort to make it compact and high performing once again.

In these cases I’ve found a workaround which frequently improves the performance, having seen this simple change result in certain applications drop from minutes to seconds.

This logic does the following:

  • Checks if HOST is an IPv4 address, with shExpMatch – if it’s an IP address, the script continues on as normal (This script as is does not cater for IPv6)
  • If HOST is NOT an IPv4 address we check if we can resolve the HOST. If we can’t, we immediately return DIRECT. (No Proxy)
function FindProxyForURL(url, host) { if (!shExpMatch(host, "/^\d+\.\d+\.\d+\.\d+$/g")) { if(!isResolvable(host)) { return "DIRECT"; }}

If you are using a PAC file, I like to ensure I can always test with a direct proxy/no proxy in cases of slow performances or unexplained web issues.

This can be done without changing your browser proxy settings, by using the Custom Proxy Browser Tool I put together, also here http://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/

Posted in Fiddler, Internet Explorer | Tagged | Leave a comment

Case of the DllHost.exe Crash

A problem case had been going on for sometime about DllHost.exe crashing, aka COM Surrogate Host across many Citrix Servers. There were about 1200 crashes a week.

We set up a server setup to capture dmp files on application crash. Due to a previous case where 10,000 instances of werfault.exe had been running on a Citrix Server however werfault.exe had been disabled  from launching via this method by setting under

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Werfault.exe

REG_SZ value Debugger to NUL

image

 

Due to this being disabled we couldn’t use the Windows in-built app dumping  here: http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

To work around this issue we set Debugger under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug to a little PowerShell script with %ld as the first parameter. (Process ID)

image

The main point of this script was to do some additional check on free disk space, as in this case D:\ also handled SCCM cache and we needed to ensure plenty remained available.

The way this is handled is first check if disk has 10GB free minimum, if it doesn’t we don’t do any dmp files.

Next after dmp file is written we check if the folder has 10GB worth of dmp files, if so the oldest ones are deleted until 10GB of files remain.

Warning: Quick & Dirty Ugly scripting follows.

 

Param( [Parameter(Mandatory=$true,Position=0)] [string]$procID) [bool]$ok=$false; # force a single instance at a time # just to limit load, paranoia/etc $m = New-Object System.Threading.Mutex($true, "AutoProcDump.Debugger", [ref]$ok); if (!$ok) { "Another instance is already running." return 0 } #Get D: Drive freespace $driveData = Get-WmiObject -class win32_LogicalDisk -filter "Name = 'D:'" | select "FreeSpace" $driveDataSize = ([int]($driveData.FreeSpace/1GB)) # Check if D: is Less than 10GB if ($driveDataSize -le 10) { return } $Folder = "D:\Tools\Dumps" $psi = New-Object System.Diagnostics.ProcessStartInfo $psi.FileName="D:\Tools\ApplicationCrash\procdump.exe" $psi.UseShellExecute=$false $psi.WorkingDirectory=$Folder $psi.Arguments="-accepteula -ma $($procID)" Write-Host $Proc $psi.Arguments $p=[System.Diagnostics.Process]::Start($psi) $p.WaitForExit(60000) if (!$p.HasExited) { # running too long $p.Kill() } #put our folder we want to check here $folder2 = "D:\Tools\Dumps" #now we need to see how big that folder is $foldersize = (Get-ChildItem $folder2 | Measure-Object -property length -sum ) #and convert it to GB's $GBsize = "{0:N5}" -f ($foldersize.sum/ 1GB) #now, let's check to see if it's over 10 GBs If ($GBsize -gt 10) #if it is, we want to DO the following {do #Let's get the 1st file (sorted by lastwrite time and remove it {dir $folder2\*.dmp | sort lastwritetime | select -first 1 | remove-item -force #now let's recheck the folder size $foldersize = (Get-ChildItem $folder2 | Measure-Object -property length -sum ) $GBsize = "{0:N5}" -f ($foldersize.sum/ 1GB) #print the folder size for testing $Gbsize } #is the folder less than 10gb? Yes, we are done. No, go back and delete another file until ($GBsize -lt 10) Write-Host "Deletes Done" } else {"No deletes Needed"} return

However after all this, no dumps were collected. The reason – the issue was not occurring on the test server, even after a week.

Then almost by accident, when I was looking for some dmp files I had run

dir *.dmp /s

And found a hidden cache of hundreds upon hundreds of mini-dump files in the D:\EdgeSight\EdgeSight folder on the Citrix server. Better than nothing, I’ll take what I can get.

So I got a list of all citrix servers, and stole all the minidumps I could find.

FOR /F %i IN (server_list.txt) DO ( xcopy \\%i\d$\EdgeSight\EdgeSight\FaultReports C:\support\minidumps /s /q )

 

However these dump files had funny random looking names.

To fix this I ran an automated script against all the dmp files, based on one from Volume 1, http://www.patterndiagnostics.com/ultimate-memory-analysis-reference

.symfix C:\symbols .reload vertarget r kv 100 !analyze -v r kv 100 ub eip u eip uf eip dps esp-3000 esp+3000 dpu esp-3000 esp+3000 dpa esp-3000 esp+3000 lmv ~*k q

I then saved the above in a file C:\support\autodbg.txt and ran a single command line to process all dmp files in current folder (Cdb.exe was accessible from this DIR)

If running from a batch file change %I to %%i

FOR /f "delims=/" %i IN ('dir *.dmp /b') DO ( cdb -z "%i" -command "$$><C:\support\autodbg.txt" > "%i.txt" )

Then I used this process to rename all the dmp files, this added a prefix to our dmp files of process name _ bucket ID from the !analyze –v output:

$files=Get-ChildItem -Path c:\support\minidumps -Filter *.txt ForEach ($file in $files) { $sr = New-Object System.IO.StreamReader($file.FullName) $text=$sr.ReadToEnd() $sr.Close() #ignore invalid dmp files if (!$text.Contains("Could not open dump file")) { $proc_start=$text.IndexOf("PROCESS_NAME:")+"PROCESS_NAME:".Length $proc_end=$text.IndexOf("`n",$proc_start) $bucket_start=$text.IndexOf("BUCKET_ID:")+"BUCKET_ID:".Length $bucket_end=$text.IndexOf("`n",$bucket_start) $proc=$text.Substring($proc_start,$proc_end-$proc_start).Trim() $bucket=$text.Substring($bucket_start,$bucket_end-$bucket_start).Trim() $dmpFileName=$file.FullName.Replace(".txt",".dmp") $dstFileName=[String]::Format("{0}_{1}_{2}.dmp",$proc,$bucket,$file.BaseName) Rename-Item $dmpFilename $dstFileName } }

This converted a folder looking like

image

to

image

Unfortunately these are all mini-dumps. But we still have some important information. Looking at !analyze –v output

FAULTING_IP: ole32!CStdMarshal::CreateStub+8c 000007fe`fe0dc170 498b0424 mov rax,qword ptr [r12] EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 000007fefe0dc170 (ole32!CStdMarshal::CreateStub+0x000000000000008c) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 0000000000000000 Parameter[1]: 00000000027621c8 Attempt to read from address 00000000027621c8 DEFAULT_BUCKET_ID: INVALID_POINTER_READ PROCESS_NAME: dllhost.exe ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s. EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s. EXCEPTION_PARAMETER1: 0000000000000000 EXCEPTION_PARAMETER2: 00000000027621c8 READ_ADDRESS: 00000000027621c8 FOLLOWUP_IP: esint+2190 00000000`6bc22190 ?? ??? DETOURED_IMAGE: 1 MOD_LIST: <ANALYSIS/> LAST_CONTROL_TRANSFER: from 000007fefe0dc063 to 000007fefe0dc170 FAULTING_THREAD: 0000000000001d70 PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ STACK_TEXT: 00000000`03cdeaa0 000007fe`fe0dc063 : 00000000`00000001 00000000`002de9f0 00000000`03cdeb98 00000000`03cdeb60 : ole32!CStdMarshal::CreateStub+0x8c 00000000`03cdeb30 000007fe`fe0dbf32 : 00000000`002fb4a8 00000000`00000000 00000000`002f97b0 00000000`00000000 : ole32!CStdMarshal::ConnectSrvIPIDEntry+0x2f 00000000`03cdeb80 000007fe`fe0e21ef : 00000000`00000000 00000000`002fb4a8 00000000`03cdeca0 00000000`00326780 : ole32!CStdMarshal::MarshalServerIPID+0xb6 00000000`03cdec20 000007fe`fe0e209f : 00000000`00000001 000007fe`fe0e2018 00000000`00000002 00000000`00000001 : ole32!CStdMarshal::MarshalIPID+0x34 00000000`03cdec60 000007fe`ffa0ff85 : 00000000`00000006 00000000`03cdf140 00000000`03cded60 00000000`00000001 : ole32!CRemoteUnknown::RemQueryInterface+0x2f5 00000000`03cded30 000007fe`ffabb68e : 00000000`00000006 00000000`002d8840 000007fe`fe247da8 00000000`002f0090 : rpcrt4!Invoke+0x65 00000000`03cdeda0 000007fe`ffa12496 : 00000000`77859fc0 00000000`0000ffff 00000000`00000000 00000000`77859fd0 : rpcrt4!Ndr64StubWorker+0x61b 00000000`03cdf360 000007fe`fe220883 : 00000000`00000000 00000000`00000000 000007fe`fe253870 00000000`002de320 : rpcrt4!NdrStubCall3+0xb5 00000000`03cdf3c0 000007fe`fe220ccd : 00000000`00000001 00000000`00000000 00000000`02b96850 00000000`00000000 : ole32!CStdStubBuffer_Invoke+0x5b 00000000`03cdf3f0 000007fe`fe220c43 : 00000000`002f0090 00000000`002e07d4 00000000`00000000 000007fe`fe2371e0 : ole32!SyncStubInvoke+0x5d 00000000`03cdf460 000007fe`fe0da4f0 : 00000000`002f0090 00000000`002e6980 00000000`002f0090 000007fe`fe0d1b00 : ole32!StubInvoke+0xdb 00000000`03cdf510 000007fe`fe0ed551 : 00000000`00000000 ab08e781`00000001 00000000`002d6450 00000000`002de320 : ole32!CCtxComChnl::ContextInvoke+0x190 00000000`03cdf6a0 000007fe`fe22347e : 00000000`002e6980 00000000`00000000 00000000`002d8840 00000000`00000000 : ole32!STAInvoke+0x91 00000000`03cdf6f0 000007fe`fe22122b : 00000000`d0908070 00000000`002e6980 00000000`002ee330 00000000`002d8840 : ole32!AppInvoke+0x1aa 00000000`03cdf760 000007fe`fe223542 : 00000000`002f0000 00000000`00000400 00000000`00000000 000007fe`fe0bb3c4 : ole32!ComInvokeWithLockAndIPID+0x52b 00000000`03cdf8f0 000007fe`fe0ed42d : 00000000`002de320 00000000`00000000 00000000`002d7fc8 00000000`002f0000 : ole32!ComInvoke+0xae 00000000`03cdf920 000007fe`fe0ed1d6 : 00000000`002e6980 00000000`002f0008 00000000`00000400 00000000`00000000 : ole32!ThreadDispatch+0x29 00000000`03cdf950 00000000`775c9bd1 : 00000000`00000000 00000000`00000000 00000000`00000000 53d9b361`91bf321e : ole32!ThreadWndProc+0xaa 00000000`03cdf9d0 00000000`775c98da : 00000000`03cdfb30 000007fe`fe0ed12c 000007fe`fe285780 00000000`00806d70 : user32!UserCallWinProcCheckWow+0x1ad 00000000`03cdfa90 000007fe`fe0ed0ab : 00000000`02cd0606 00000000`02cd0606 000007fe`fe0ed12c 00000000`00000000 : user32!DispatchMessageWorker+0x3b5 00000000`03cdfb10 000007fe`fe213e57 : 00000000`002e6980 00000000`00000000 00000000`002e6b60 000007fe`fe0d3032 : ole32!CDllHost::STAWorkerLoop+0x68 00000000`03cdfb70 000007fe`fe0c0106 : 00000000`002e6980 00000000`002d6350 00000000`00000000 00000000`00000000 : ole32!CDllHost::WorkerThread+0xd7 00000000`03cdfbb0 000007fe`fe0c0182 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ole32!CRpcThread::WorkerLoop+0x1e 00000000`03cdfbf0 00000000`7729652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a 00000000`03cdfc20 00000000`7782c541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd 00000000`03cdfc50 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d SYMBOL_STACK_INDEX: 2 SYMBOL_NAME: esint+2190 FOLLOWUP_NAME: MachineOwner MODULE_NAME: esint IMAGE_NAME: esint.dll DEBUG_FLR_IMAGE_TIMESTAMP: 5385f6ee STACK_COMMAND: .cxr 0000000000000000 ; kb ; ~4s; .ecxr ; kb FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_esint.dll!Unknown BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_DETOURED_esint+2190 WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/dllhost_exe/6_1_7600_16385/4a5bca54/ole32_dll/6_1_7601_17514/4ce7c92c/c0000005/0002c170.htm?Retriage=1 Followup: MachineOwner

 

We can see esint.dll pointed to by !analyze –v is from Citrix:

Loaded symbol image file: esint.dll
Image path: c:\program files (x86)\Citrix\system monitoring\Agent\edgesight\esint.dll
Image name: esint.dll
Timestamp:        Thu May 29 00:47:10 2014 (5385F6EE)
CheckSum:         00013F89
ImageSize:        00013000
File version:     5.4.16.19
Product version:  5.4.16.19
File flags:       0 (Mask 3F)
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

However I didn’t suspect this component was likely to be at fault.

Almost always dllhost.exe crashes are caused by 3rd party viewers/codecs/printer drivers/etc. So I carefully examined the output of lmv

and found this…

000007fe`dcc00000 000007fe`dd9ab000   npdf       (deferred)            
    Image path: C:\Program Files\Nitro\Pro 9\npdf.dll
    Image name: npdf.dll
    Timestamp:        Tue Jun 24 12:13:04 2014 (53A8DEB0)
    CheckSum:         00C96AB8
    ImageSize:        00DAB000
    File version:     9.5.19.13
    Product version:  3.9.0.0
    File flags:       28 (Mask 3F) Private Special
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

 

I then checked DllHost.exe on several machines, including the DllHost.exe generated when printing, and none of them had this DLL loaded, even though the application was installed.

So this suggested to me this DLL was loaded only on tasks related to this plugin.

However I had several hundred minidumps, did they all have this module loaded?

To check this I ran another quick PowerShell script against the output text files:

$files=Get-ChildItem -Path c:\support\minidumps -Filter *dllhost*.txt $hasNitro=0 $noNitro=0 ForEach ($file in $files) { $sr = New-Object System.IO.StreamReader($file.FullName) $text=$sr.ReadToEnd() if ($text.Contains("npdf")) { $hasNitro++ } else { $file.FullName $noNitro++ } }

 

Checking the value of HasNitro vs noNitro at the end, we saw 153 had the DLL loaded, 2 didn’t. Because these were different I output their filename for further manual analysis. (The final 2 were related to Photo Preview Handler )

What was it doing to crash? Due to various reasons we were restricted and weren’t able to talk to users to find out what they were doing leading up to crash . So I checked dllhost.exe on it’s own, and the DLLs and compared them further with the crash dumps. (Note when printing, you will also see a dllhost.exe launch – this will have different DLLs again)

image

All the crash dumps also had Microsoft Thumbnail Cache loaded

000007fe`f4010000 000007fe`f402f000   thumbcache   (deferred)            
    Mapped memory image file: C:\symbols\thumbcache.dll\4CE7C9D01f000\thumbcache.dll
    Image path: C:\Windows\System32\thumbcache.dll
    Image name: thumbcache.dll
    Timestamp:        Sun Nov 21 00:14:56 2010 (4CE7C9D0)
    CheckSum:         00022DBA
    ImageSize:        0001F000
    File version:     6.1.7601.17514
    Product version:  6.1.7601.17514
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     thumbcache.dll
    OriginalFilename: thumbcache.dll
    ProductVersion:   6.1.7601.17514
    FileVersion:      6.1.7601.17514 (win7sp1_rtm.101119-1850)
    FileDescription:  Microsoft Thumbnail Cache
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

So from this we can guess – the crashes occurred when building thumbnails of PDFs

From http://msdn.microsoft.com/en-us/library/windows/desktop/cc144118(v=vs.85).aspx we can see thumbnail preview handlers have GUID E357FCCD-A995-4576-B01F-234630154E96

Looking up .PDF in HKEY_CLASSES_ROOT we see (Default) is set to NitroPDF.Document9image

We then look up HKEY_CLASSES_ROOT\NitroPDF.Document.9

And sure enough we can see it has a thumbnail handler installed under ShellEx\{e357fccd-a995-4576-b01f-234630154e96}

Note: {8895b1c6-b41f-4c1c-a562-0d564250836f} is for the preview handler. http://msdn.microsoft.com/en-us/library/windows/desktop/cc144144(v=vs.85).aspx

image

We can remove the preview handler by deleting this registry key.

We contacted the vendor and they confirmed their product caused this issue, and the latest version of the product had fixed the bug.

Posted in Citrix, ProcExp, WinDbg | Tagged | Leave a comment

Case of the System.Net.WebClient (407) Proxy Authentication Required

When attempting to launch DebugDiag Analysis Engine ( http://www.microsoft.com/en-us/download/details.aspx?id=42933 )

I received the error message

Error Running Task
One or more errors occurred.

System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.

   at System.Net.WebClient.OpenRead(Uri address)

   at DebugDiag.Analysis.AutoUpdate.GetWebContent(String pathToLatestFiles)

   at DebugDiag.Analysis.AutoUpdate.IsHostProcessOutOfDate_http(String pathToLatestFiles, Boolean throwIfFileMissing)

   at DebugDiag.Analysis.AutoUpdateVM.CheckForUpdatesInternal()

   at DebugDiag.Analysis.AutoUpdateVM.CheckAndInstallUpdatesInternal()

   at System.Threading.Tasks.Task.Execute()

image

From Fiddler ( http://www.telerik.com/fiddler ) we can see: 407 proxy denied access http://aka.ms/debugdiagupdate

image

In first request we see:

image

This is pretty normal, but in a working application, we will see it retry and complete authentication. In this case it just stops.

The simple fix is to update <exename>.config file (C:\Program Files\DebugDiag\DebugDiag.Analysis.exe.config) by adding within the <Configuration></Configuration> section:

<system.net> <defaultProxy useDefaultCredentials="true" > </defaultProxy> </system.net>

image

After the change, in fiddler we see:

image

On the first tab request in the auth tabs we see same as before:

image

In the 2nd request:

image

The 3rd request:

image

 

And finally:

image

If we had wanted to get some more information, we can right click process in Task Manager when error is displayed and click Create Dump File

Loading in WinDbg and loading SOS by running

.loadby sos clr

Using

.foreach (ex {!dumpheap -type Exception -short}){.echo “********************************”;!pe ${ex} }

( http://blogs.msdn.com/b/tess/archive/2009/04/16/net-exceptions-quick-windbg-sos-tip-on-how-to-dump-all-the-net-exceptions-on-the-heap.aspx )

We can see

Exception object: 0000000003ddfd90
Exception type:   System.Net.WebException
Message:          The remote server returned an error: (407) Proxy Authentication Required.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    000000001F35CA30 000007FEED1ADDD2 System_ni!System.Net.WebClient.OpenRead(System.Uri)+0x2f2
    000000001F35EB90 000007FE8F57980F DebugDiag_Analysis!DebugDiag.Analysis.AutoUpdate.GetWebContent(System.String)+0x3f
    000000001F35EBF0 000007FE8F579794 DebugDiag_Analysis!DebugDiag.Analysis.AutoUpdate.IsHostProcessOutOfDate_http(System.String, Boolean)+0x174
    000000001F35EC60 000007FE8F57836D DebugDiag_Analysis!DebugDiag.Analysis.AutoUpdateVM.CheckForUpdatesInternal()+0x6d
    000000001F35ECC0 000007FE8F5781E1 DebugDiag_Analysis!DebugDiag.Analysis.AutoUpdateVM.CheckAndInstallUpdatesInternal()+0x11
    000000001F35ED00 000007FEEDBA407E mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x6e

Then looking at !eestack we can find this started from thread 9

0:000> ~9s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`76f318ca c3              ret
0:009> !clrstack
OS Thread Id: 0x1c6c (9)
        Child SP               IP Call Site
000000001f35e878 0000000076f318ca [HelperMethodFrame_1OBJ: 000000001f35e878] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
000000001f35e9a0 000007feedb795c4 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
000000001f35e9e0 000007feee3b9117 System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean)
000000001f35ea20 000007feec5b8e87 System.Windows.Threading.DispatcherOperation+DispatcherOperationEvent.WaitOne()
000000001f35ea90 000007feec5c1bb1 System.Windows.Threading.DispatcherOperation.Wait(System.TimeSpan)
000000001f35eae0 000007feec5ef266 System.Windows.Threading.Dispatcher.InvokeImpl(System.Windows.Threading.DispatcherOperation, System.Threading.CancellationToken, System.TimeSpan)
000000001f35ebb0 000007feec5e6997 System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
000000001f35ec50 000007feec7d6102 System.Windows.Threading.Dispatcher.Invoke(System.Delegate, System.Object[])
000000001f35eca0 000007fe8f579a66 DebugDiag.Analysis.AutoUpdateVM.TaskCompleted(System.Threading.Tasks.Task)
000000001f35ed00 000007feedba407e System.Threading.Tasks.Task.Execute()
000000001f35ed70 000007feedb5f8a5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001f35eed0 000007feedb5f609 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001f35ef00 000007feedba3e55 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
000000001f35ef90 000007feedba3c75 System.Threading.Tasks.Task.ExecuteEntry(Boolean)
000000001f35efd0 000007feedb0b82a System.Threading.ThreadPoolWorkQueue.Dispatch()
000000001f35f528 000007feeec3f713 [DebuggerU2MCatchHandlerFrame: 000000001f35f528]

On this thread we can:

0:009>!dso

000000001F35ECB8 0000000003de5648 System.AggregateException
000000001F35ECD0 0000000003c10168 System.Threading.ContextCallback
000000001F35ECF0 0000000003b9aaa0 System.Threading.ExecutionContext
000000001F35ED00 0000000003c56160 System.Threading.Tasks.Task
000000001F35ED08 0000000003c56160 System.Threading.Tasks.Task
000000001F35ED70 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35EE10 0000000003bdc068 System.Threading.Thread
000000001F35EE40 0000000003bdc068 System.Threading.Thread
000000001F35EE98 0000000003b61898 System.Threading.ThreadPoolWorkQueue
000000001F35EEA8 0000000003b9aaa0 System.Threading.ExecutionContext
000000001F35EEB0 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35EEB8 0000000003c10168 System.Threading.ContextCallback
000000001F35EF00 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35EF48 0000000003c563c8 System.Threading.Tasks.TplEtwProvider
000000001F35EF78 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35EF90 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35EFA8 0000000003b61898 System.Threading.ThreadPoolWorkQueue
000000001F35EFD0 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
000000001F35F008 0000000003bdc040 System.Threading.ThreadPoolWorkQueueThreadLocals
000000001F35F018 0000000003b61898 System.Threading.ThreadPoolWorkQueue
000000001F35F038 0000000003c56350 System.Threading.Tasks.ContinuationTaskFromTask
0:009> !do 0000000003de5648
Name:        System.AggregateException
MethodTable: 000007feedd17350
EEClass:     000007feed724cc8
Size:        168(0xa8) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007feedd0aee0  4000002        8        System.String  0 instance 0000000000000000 _className
000007feedd09460  4000003       10 …ection.MethodBase  0 instance 0000000000000000 _exceptionMethod
000007feedd0aee0  4000004       18        System.String  0 instance 0000000000000000 _exceptionMethodString
000007feedd0aee0  4000005       20        System.String  0 instance 0000000003de52b0 _message
000007feedd08ee8  4000006       28 …tions.IDictionary  0 instance 0000000000000000 _data
000007feedd0b110  4000007       30     System.Exception  0 instance 0000000003ddfd90 _innerException
000007feedd0aee0  4000008       38        System.String  0 instance 0000000000000000 _helpURL
000007feedd0b4c0  4000009       40        System.Object  0 instance 0000000000000000 _stackTrace
000007feedd0b4c0  400000a       48        System.Object  0 instance 0000000000000000 _watsonBuckets
000007feedd0aee0  400000b       50        System.String  0 instance 0000000000000000 _stackTraceString
000007feedd0aee0  400000c       58        System.String  0 instance 0000000000000000 _remoteStackTraceString
000007feedd0dc90  400000d       88         System.Int32  1 instance                0 _remoteStackIndex
000007feedd0b4c0  400000e       60        System.Object  0 instance 0000000000000000 _dynamicMethods
000007feedd0dc90  400000f       8c         System.Int32  1 instance      -2146233088 _HResult
000007feedd0aee0  4000010       68        System.String  0 instance 0000000000000000 _source
000007feedd0ed00  4000011       78        System.IntPtr  1 instance                0 _xptrs
000007feedd0dc90  4000012       90         System.Int32  1 instance       -532462766 _xcode
000007feedcf7828  4000013       80       System.UIntPtr  1 instance                0 _ipForWatsonBuckets
000007feedd72960  4000014       70 …ializationManager  0 instance 0000000003de57a8 _safeSerializationManager
000007feedd0b4c0  4000001        0        System.Object  0   shared           static s_EDILock
                                 >> Domain:Value  00000000000cf5d0:NotInit  <<
000007feed68fcc8  400002c       98 …ption, mscorlib]]  0 instance 0000000003de5808 m_innerExceptions
0:009> !do 0000000003de52b0
Name:        System.String
MethodTable: 000007feedd0aee0
EEClass:     000007feed673720
Size:        82(0x52) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:     
One or more errors occurred.
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007feedd0dc90  40000aa        8         System.Int32  1 instance               28 m_stringLength
000007feedd0c1c8  40000ab        c          System.Char  1 instance               4f m_firstChar
000007feedd0aee0  40000ac       18        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000000cf5d0:NotInit  <<
0:009> !do 0000000003ddfd90
Name:        System.Net.WebException
MethodTable: 000007feecdb0818
EEClass:     000007feeca10138
Size:        176(0xb0) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007feedd0aee0  4000002        8        System.String  0 instance 00000000042284b8 _className
000007feedd09460  4000003       10 …ection.MethodBase  0 instance 0000000000000000 _exceptionMethod
000007feedd0aee0  4000004       18        System.String  0 instance 0000000000000000 _exceptionMethodString
000007feedd0aee0  4000005       20        System.String  0 instance 0000000003ddfce0 _message
000007feedd08ee8  4000006       28 …tions.IDictionary  0 instance 0000000000000000 _data
000007feedd0b110  4000007       30     System.Exception  0 instance 0000000000000000 _innerException
000007feedd0aee0  4000008       38        System.String  0 instance 0000000000000000 _helpURL
000007feedd0b4c0  4000009       40        System.Object  0 instance 0000000003de19a0 _stackTrace
000007feedd0b4c0  400000a       48        System.Object  0 instance 0000000000000000 _watsonBuckets
000007feedd0aee0  400000b       50        System.String  0 instance 0000000000000000 _stackTraceString
000007feedd0aee0  400000c       58        System.String  0 instance 0000000000000000 _remoteStackTraceString
000007feedd0dc90  400000d       88         System.Int32  1 instance                0 _remoteStackIndex
000007feedd0b4c0  400000e       60        System.Object  0 instance 0000000000000000 _dynamicMethods
000007feedd0dc90  400000f       8c         System.Int32  1 instance      -2146233079 _HResult
000007feedd0aee0  4000010       68        System.String  0 instance 0000000000000000 _source
000007feedd0ed00  4000011       78        System.IntPtr  1 instance                0 _xptrs
000007feedd0dc90  4000012       90         System.Int32  1 instance       -532462766 _xcode
000007feedcf7828  4000013       80       System.UIntPtr  1 instance      7feed50b5d0 _ipForWatsonBuckets
000007feedd72960  4000014       70 …ializationManager  0 instance 0000000003ddfe80 _safeSerializationManager
000007feedd0b4c0  4000001        0        System.Object  0   shared           static s_EDILock
                                 >> Domain:Value  00000000000cf5d0:NotInit  <<
000007feece0cf90  4001b81       94         System.Int32  1 instance                7 m_Status
000007feecdb1098  4001b82       98 …m.Net.WebResponse  0 instance 0000000003ddcf20 m_Response
000007feecdc5170  4001b83       a0         System.Int32  1 instance                0 m_InternalStatus
0:009> !do 0000000003ddfce0
Name:        System.String
MethodTable: 000007feedd0aee0
EEClass:     000007feed673720
Size:        172(0xac) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:     
The remote server returned an error: (407) Proxy Authentication Required.
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007feedd0dc90  40000aa        8         System.Int32  1 instance               73 m_stringLength
000007feedd0c1c8  40000ab        c          System.Char  1 instance               54 m_firstChar
000007feedd0aee0  40000ac       18        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000000cf5d0:NotInit  <<


 

We can examine the disassembled function this way, both the IL code, and JIT’ed code with annotations:

0:009> !name2ee DebugDiag_Analysis DebugDiag.Analysis.AutoUpdate.GetWebContent
Module:      000007fe8f452f90
Assembly:    DebugDiag.Analysis.exe
Token:       000000000600007b
MethodDesc: 
000007fe8f5fac30
Name:        DebugDiag.Analysis.AutoUpdate.GetWebContent(System.String)
JITTED Code Address:
000007fe8f5797d0
0:009> !dumpil 000007fe8f5fac30
ilAddr = 0000000001103f08
IL_0000: ldnull
IL_0001: stloc.0
IL_0002: newobj System.Net.WebClient::.ctor
IL_0007: stloc.1
IL_0008: ldloc.1
IL_0009: ldarg.0
IL_000a: callvirt System.Net.WebClient::OpenRead
IL_000f: stloc.2
.try
{
  IL_0010: ldloc.2
  IL_0011: newobj System.IO.StreamReader::.ctor
  IL_0016: stloc.3
  .try
  {
    IL_0017: ldloc.3
    IL_0018: callvirt System.IO.TextReader::ReadToEnd
    IL_001d: stloc.0
    IL_001e: leave.s IL_002a
  } // end .try
  .finally
  {
    IL_0020: ldloc.3
    IL_0021: brfalse.s IL_0029
    IL_0023: ldloc.3
    IL_0024: callvirt System.IDisposable::Dispose
    IL_0029: endfinally
  } // end .finally
  IL_002a: leave.s IL_0036
} // end .try
.finally
{
  IL_002c: ldloc.2
  IL_002d: brfalse.s IL_0035
  IL_002f: ldloc.2
  IL_0030: callvirt System.IDisposable::Dispose
  IL_0035: endfinally
} // end .finally
IL_0036: ldloc.0
IL_0037: ret
0:009> !U -ehinfo -gcinfo 000007fe8f5797d0
Normal JIT generated code
DebugDiag.Analysis.AutoUpdate.GetWebContent(System.String)
Begin 000007fe8f5797d0, size 132
>>> 000007fe`8f5797d0 55              push    rbp
000007fe`8f5797d1 53              push    rbx
000007fe`8f5797d2 57              push    rdi
000007fe`8f5797d3 4883ec40        sub     rsp,40h
000007fe`8f5797d7 488d6c2420      lea     rbp,[rsp+20h]
000007fe`8f5797dc 488bf9          mov     rdi,rcx
000007fe`8f5797df 48896500        mov     qword ptr [rbp],rsp
00000013 interruptible
00000013 +rdi
000007fe`8f5797e3 33c0            xor     eax,eax
00000015 +rax
000007fe`8f5797e5 48894508        mov     qword ptr [rbp+8],rax
000007fe`8f5797e9 48894510        mov     qword ptr [rbp+10h],rax
000007fe`8f5797ed 488d0ddc12885d  lea     rcx,[System_ni+0x42aad0 (000007fe`ecdfaad0)]
000007fe`8f5797f4 e897946d5f      call    clr!JIT_NewCrossContext_Portable (000007fe`eec52c90)
000007fe`8f5797f9 488bd8          mov     rbx,rax
0000002c -rax +rbx
000007fe`8f5797fc 488bcb          mov     rcx,rbx
0000002f +rcx
000007fe`8f5797ff e824d3695d      call    System_ni+0x246b28 (000007fe`ecc16b28) (System_ni)
00000034 -rcx
000007fe`8f579804 488bd7          mov     rdx,rdi
00000037 -rdi +rdx
000007fe`8f579807 488bcb          mov     rcx,rbx
0000003a -rbx +rcx
000007fe`8f57980a e8a9d1695d      call    System_ni+0x2469b8 (000007fe`ecc169b8) (System_ni)
0000003f -rdx -rcx +rax
000007fe`8f57980f 48894510        mov     qword ptr [rbp+10h],rax
00000043 -rax +rbp+10
000007fe`8f579813 48c7451800000000 mov     qword ptr [rbp+18h],0
0000004b +rbp+18
EHHandler 1: FAULT CLAUSE BEGIN
000007fe`8f57981b 488d0dbe80795e  lea     rcx,[mscorlib_ni+0x6a18e0 (000007fe`edd118e0)]
000007fe`8f579822 e869946d5f      call    clr!JIT_NewCrossContext_Portable (000007fe`eec52c90)
00000057 +rax
000007fe`8f579827 488bd8          mov     rbx,rax
0000005a -rax +rbx
000007fe`8f57982a 488b5510        mov     rdx,qword ptr [rbp+10h]
0000005e +rdx
000007fe`8f57982e 488bcb          mov     rcx,rbx
00000061 +rcx
000007fe`8f579831 e8e2c94e5e      call    mscorlib_ni+0x3f6218 (000007fe`eda66218) (System.IO.StreamReader..ctor(System.IO.Stream), mdToken: 00000000060046af)
00000066 -rdx -rcx
000007fe`8f579836 48895d08        mov     qword ptr [rbp+8],rbx
0000006a -rbx +rbp+8
EHHandler 0: FAULT CLAUSE BEGIN
000007fe`8f57983a 488b4508        mov     rax,qword ptr [rbp+8]
0000006e +rax
000007fe`8f57983e 488b08          mov     rcx,qword ptr [rax]
00000071 -rax
000007fe`8f579841 488b4148        mov     rax,qword ptr [rcx+48h]
000007fe`8f579845 488b4d08        mov     rcx,qword ptr [rbp+8]
00000079 +rcx
000007fe`8f579849 4c8b5d08        mov     r11,qword ptr [rbp+8]
0000007d +r11
000007fe`8f57984d ff5028          call    qword ptr [rax+28h]
00000080 -r11 -rcx +rax
000007fe`8f579850 48894518        mov     qword ptr [rbp+18h],rax
00000084 -rax
EHHandler 0: FAULT CLAUSE END
000007fe`8f579854 488b4d08        mov     rcx,qword ptr [rbp+8]
00000088 -rbp+8 +rcx
000007fe`8f579858 4c8d1d596beeff  lea     r11,[000007fe`8f4603b8]
000007fe`8f57985f ff15536beeff    call    qword ptr [000007fe`8f4603b8]
00000095 -rcx
000007fe`8f579865 90              nop
EHHandler 1: FAULT CLAUSE END
000007fe`8f579866 48837d1000      cmp     qword ptr [rbp+10h],0
000007fe`8f57986b 7413            je      000007fe`8f579880
000007fe`8f57986d 488b4d10        mov     rcx,qword ptr [rbp+10h]
000000a1 -rbp+10 +rcx
000007fe`8f579871 4c8d1d486beeff  lea     r11,[000007fe`8f4603c0]
000007fe`8f579878 ff15426beeff    call    qword ptr [000007fe`8f4603c0]
000000ae -rcx
000007fe`8f57987e 6690            xchg    ax,ax
000007fe`8f579880 488b4518        mov     rax,qword ptr [rbp+18h]
000000b4 not interruptible
000000b4 -rbp+18
000007fe`8f579884 488d6520        lea     rsp,[rbp+20h]
000007fe`8f579888 5f              pop     rdi
000007fe`8f579889 5b              pop     rbx
000007fe`8f57988a 5d              pop     rbp
000007fe`8f57988b c3              ret
EHHandler 0: FAULT HANDLER BEGIN
000007fe`8f57988c 55              push    rbp
000007fe`8f57988d 53              push    rbx
000007fe`8f57988e 57              push    rdi
000007fe`8f57988f 4883ec30        sub     rsp,30h
000007fe`8f579893 488b6920        mov     rbp,qword ptr [rcx+20h]
000007fe`8f579897 48896c2420      mov     qword ptr [rsp+20h],rbp
000007fe`8f57989c 488d6d20        lea     rbp,[rbp+20h]
000000d0 interruptible
000007fe`8f5798a0 48837d0800      cmp     qword ptr [rbp+8],0
000007fe`8f5798a5 7417            je      000007fe`8f5798be
000007fe`8f5798a7 488d150a6beeff  lea     rdx,[000007fe`8f4603b8]
000007fe`8f5798ae 488b4508        mov     rax,qword ptr [rbp+8]
000000e2 +rax
000007fe`8f5798b2 803800          cmp     byte ptr [rax],0
000000e5 -rax
000007fe`8f5798b5 488b4d08        mov     rcx,qword ptr [rbp+8]
000000e9 +rcx
000007fe`8f5798b9 4c8bda          mov     r11,rdx
000007fe`8f5798bc ff12            call    qword ptr [rdx]
000000ee -rcx
000007fe`8f5798be 90              nop
000000ef not interruptible
000007fe`8f5798bf 4883c430        add     rsp,30h
000007fe`8f5798c3 5f              pop     rdi
000007fe`8f5798c4 5b              pop     rbx
000007fe`8f5798c5 5d              pop     rbp
000007fe`8f5798c6 c3              ret
EHHandler 0: FAULT HANDLER END
EHHandler 1: FAULT HANDLER BEGIN
000007fe`8f5798c7 55              push    rbp
000007fe`8f5798c8 53              push    rbx
000007fe`8f5798c9 57              push    rdi
000007fe`8f5798ca 4883ec30        sub     rsp,30h
000007fe`8f5798ce 488b6920        mov     rbp,qword ptr [rcx+20h]
000007fe`8f5798d2 48896c2420      mov     qword ptr [rsp+20h],rbp
000007fe`8f5798d7 488d6d20        lea     rbp,[rbp+20h]
0000010b interruptible
000007fe`8f5798db 48837d1000      cmp     qword ptr [rbp+10h],0
000007fe`8f5798e0 7417            je      000007fe`8f5798f9
000007fe`8f5798e2 488d15d76aeeff  lea     rdx,[000007fe`8f4603c0]
000007fe`8f5798e9 488b4510        mov     rax,qword ptr [rbp+10h]
0000011d +rax
000007fe`8f5798ed 803800          cmp     byte ptr [rax],0
00000120 -rax
000007fe`8f5798f0 488b4d10        mov     rcx,qword ptr [rbp+10h]
00000124 +rcx
000007fe`8f5798f4 4c8bda          mov     r11,rdx
000007fe`8f5798f7 ff12            call    qword ptr [rdx]
00000129 -rcx
000007fe`8f5798f9 90              nop
0000012a not interruptible
000007fe`8f5798fa 4883c430        add     rsp,30h
000007fe`8f5798fe 5f              pop     rdi
000007fe`8f5798ff 5b              pop     rbx
000007fe`8f579900 5d              pop     rbp
000007fe`8f579901 c3              ret

Posted in .NET, Fiddler, WinDbg | Tagged | Leave a comment

Case of the 30 minute Windows 7 Logon

After following a similar approach as below:

http://chentiangemalc.wordpress.com/2014/09/12/case-of-the-8-minute-windows-8-1-first-logon/

I still hadn’t found any obvious cause of slow logon, except that it was in Win Logon Init phase.

clip_image002

CPU, Storage, Memory, and Disk usage are all low:

clip_image002[7]

Expanding Generic Events, looking at Microsoft-Windows-GroupPolicy section we could see delays of 2,000 seconds apart. Another big clue was servers started with SYD prefix for SYDNEY, but these clients were located in Melbourne.

image

The Microsoft-Windows-GroupPolicy/Operational event log showed frequent delays, and repeatedly contacting the wrong domain controllers, and we also saw many delays in here. There is a ton of useful diagnostic info in this log, so in Domain Environments you should be well acquainted with it Smile

Using a quick PowerShell script we can check domain controllers, this machine has connected to, and sort from most popular to least popular.

$xpath=@' <QueryList> <Query Id="0" Path="Microsoft-Windows-GroupPolicy/Operational"> <Select Path="Microsoft-Windows-GroupPolicy/Operational">*[System[(EventID=5310)]]</Select> </Query> </QueryList> '@ $DCs=@{} $events=Get-WinEvent -LogName "Microsoft-Windows-GroupPolicy/Operational" -FilterXPath $xpath ForEach ($event in $events) { $DCs[$event.Properties[2].Value]++ } $DCs.GetEnumerator() | Sort-Object Value -Descending

Here we saw frequency of going to domain controller in another site. On top of this DFS was in use, and was resulting in unnecessary WAN traffic due to wrong Active Directory site detection.

image

The root cause was the Melbourne site had new subnets added to it, but the Active Directory Sites & Services had not been updated, so the machines were frequently in the wrong site. Once this was corrected Windows logon times improved to more normal levels.

Well normal for the amount of crap they were required to have running at logon to meet business requirements.

Posted in Active Directory, Group Policy, Windows 7, Windows Performance Recorder, Windows Performance Toolkit | Tagged , | Leave a comment

Case of the Windows 8.1 Audio Glitches

In this scenario a Windows 8.1 device, despite well spec’d hardware wise, would get buzzing/clicking audio glitches during audio playback, typically when playing web content. In addition DRM’d web content like Amazon Prime stuttered completely, however it ran fine in HD on the same hardware in a Windows 7 virtual machine. The issue occurred within IE11, Chrome and Firefox on the Windows 8.1 device.

A quick workaround was to restart the Windows Audio Service which immediately stopped the audio glitches. But they kept coming back.

Using Windows Performance Recorder ( http://chentiangemalc.wordpress.com/2014/09/12/case-of-the-8-minute-windows-8-1-first-logon/ ) we recorded a 30 second trace while issue was occurring with the following options

  • First Level Triage
  • CPU Usage
  • Disk I/O Activity
  • File I/O Activity
  • GPU Activity
  • Scenario Analysis – Audio Glitches
  • Scenario Analysis – Internet Explorer
  • Scenario Analysis – Minifilter I/O Activity

 

From the first review, we can see the system is not really very heavily utilized:

image

In the trace I first checked Mini Filter Delays, Timeline by Driver, Process, Thread. By default it showed Process ID, but I changed it to Process Name, and added Stack. I dragged Process Name, and Stack into the “Group By Column”

In this case we were reproducing issue in Chrome.exe – chrome.exe had the most events (after wprui.exe)

image

 

The most busy Mini-Filter driver was WdFilter.sys – the Microsoft antimalware file system filter driver.

image

 

Expanding the stack we end up with SiWinAcc.sys – which is a Silicon Image SATALink Windows Accelerator Driver

image

 

In Device I/O we also see this as the most popular driver. (after fltrmgr.sys) We also see

  • FltrMgr.sys has count of 94,929 events with duration of 13.55 seconds
  • SiWinAcc.sys has a count of 15,232 events with duration of 12.97 seconds
  •  

    image

    image

     

    Disabling this driver with http://live.sysinternals.com/autoruns.exe and restarting the system, could no longer reproduce the audio glitch.

    image

     

    In Windows Device Manager we can see we have Silicon Image SiI 3132 SATALink Controller – where we can look for updates/vendor to contact etc. In this case Silicon Image drive will be updated by the OEM manufacturer, so you need to contact who built your PC vendor/motherboard manufacturer/controller.

    image

     

    Afterwards we can see a change, when recording for about the same time period playing similar content:

    image

    To compare traces you can:

    1) Open good trace “baseline” first

    2) Open bad trace second

    Clicking Window – New Comparative Analysis View

     

    image

    Then we select our traces:

    image

    We then added our Device I/O again, POSITIVE numbers here indicate the item occurred more frequently in the 2nd trace then in the baseline (good trace) Negative numbers indicate activity occurred less.

    Recommend reading http://msdn.microsoft.com/en-us/library/windows/hardware/dn282270.aspx for more info on Comparative Analysis if you are not familiar with it.

    image

     

    If we want, we can then do deeper investigation to see why this driver is a problem, do we need it/etc.

    Posted in Windows 8.1, Windows Performance Recorder | Tagged | Leave a comment