On a group of Windows 7 x64 machine the TrustedInstaller service, listed in Windows Services as the “Windows Module Installer Service”, was failing to start:
We also saw evidence in the Application Event log with errors like:
Faulting application name: TrustedInstaller.exe, version: 6.1.7601.17514, time stamp: 0x4ce7989b
Faulting module name: ole32.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c92c
Exception code: 0xc0000005
Fault offset: 0x0000000000010c7b
Faulting process id: 0x1380
Faulting application start time: 0x01cd439a251e982e
Faulting application path: C:\WINDOWS\servicing\TrustedInstaller.exe
Faulting module path: C:\WINDOWS\system32\ole32.dll
Report Id: 62cd866a-af8d-11e1-8d31-0050568f668c
Faulting package full name: %14
We also saw events in System Event log such as The Windows Module Installer service terminated unexpectedly. It has done this 236 time(s).
The impact of this is pretty significant, among other things:
- You can’t install/remove Windows updates
- You can’t run sfc /scannow
Initially I was providing assistance to someone interstate so I requested a full application crash dump file. To automatically generate these set the following registry keys (Create any that don’t already exist)
Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
Add the following values (Customize as you like…)
- DumpFolder REG_EXPAND_SZ C:\CrashDumps
- DumpCount REG_DWORD 10
- DumpType REG_DWORD 2
This generates full application crash dumps in c:\crashdumps whenever a process crashes, and stores the last ten.
When we attempted to start the service a file TrustedInstaller.exe.<PID>.dmp was generated which we opened in WinDbg (from the Windows SDK)
Running !analyze –v we got some basic information:
ole32!CLSIDFromString+5fb
000007fe`fdb40c7b f30f7f4030 movdqu xmmword ptr [rax+30h],xmm0
EXCEPTION_RECORD: ffffffffffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 000007fefdb40c7b (ole32!CLSIDFromString+0x00000000000005fb)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000030
Attempt to write to address 0000000000000030
PROCESS_NAME: TrustedInstaller.exe
ADDITIONAL_DEBUG_TEXT:
Use ‘!findthebuild’ command to search for the target build information.
If the build information is available, run ‘!findthebuild -s ; .reload’ to set symbol path and load symbols.
MODULE_NAME: propsys
FAULTING_MODULE: 0000000076ed0000 ntdll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7c94a
ERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000001
EXCEPTION_PARAMETER2: 0000000000000030
WRITE_ADDRESS: 0000000000000030
FOLLOWUP_IP:
propsys!InitPropVariantFromStringVector+165
000007fe`fb4a68c1 8bd8 mov ebx,eax
DETOURED_IMAGE: 1
APP: trustedinstaller.exe
LAST_CONTROL_TRANSFER: from 000007fefdb319df to 000007fefdb40c7b
FAULTING_THREAD: 0000000000000e78
DEFAULT_BUCKET_ID: WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS
BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS_NULL_CLASS_PTR_WRITE
STACK_TEXT:
00000000`013bedc0 000007fe`fdb319df : 76726553`00000002 00000000`000e82f0 00000000`000f2dd0 000007fe`fdd0a1b0 : ole32!CLSIDFromString+0x5fb
00000000`013bee40 000007fe`fdb49514 : 00000000`00000000 6f6e6143`00000000 00000000`00000000 00000000`000da3b8 : ole32+0x19df
00000000`013bef50 000007fe`fb4a68c1 : 00000000`00000000 000007fe`fb50fe38 00000000`000f2dd0 00000000`00000000 : ole32!NdrOleInitializeExtension+0x934
00000000`013bef90 000007fe`fb4a6870 : 00000000`000f2dd0 000007fe`fb50fe08 000007fe`fb50fe08 000007fe`fb54b0f8 : propsys!InitPropVariantFromStringVector+0x165
00000000`013befd0 000007fe`fb4a853a : 00000000`000f2dd0 000007fe`fb50fe38 000007fe`00000001 000007fe`00000001 : propsys!InitPropVariantFromStringVector+0x114
00000000`013bf000 000007fe`fb4a84f3 : 00000000`00000000 000007fe`fb50fe08 000007fe`fb50fe08 000007fe`fe2da2de : propsys!VariantToPropVariant+0x162
00000000`013bf030 000007fe`fb4a1e32 : 00000000`80004005 000007fe`fb4a3c34 000007fe`fe2da2de 000007fe`fd48a95f : propsys!VariantToPropVariant+0x11b
00000000`013bf060 000007fe`fb4a207e : 00000000`00000000 00000000`80070057 00000000`80004005 00000000`013bf140 : propsys!PSCreateMemoryPropertyStore+0x6ae
00000000`013bf0a0 000007fe`fb4a220d : 00000000`013bf240 00000000`00000000 00000000`013bf140 00000000`013bf140 : propsys!PSCreateMemoryPropertyStore+0x8fa
00000000`013bf0f0 000007fe`fb4a3c5d : 00000000`013bf190 00000000`000d5a70 00000000`00000000 00000000`00000000 : propsys!PSGetPropertyDescription+0x39
00000000`013bf120 000007fe`fde4d46c : 000007fe`fe272b88 00000000`013bf190 00000000`00000001 ffffffff`ffffffff : propsys!PSCoerceToCanonicalValue+0x29
00000000`013bf160 000007fe`fde41926 : 00000000`000ca030 00000000`013bf1f0 00000000`00000000 00000000`000da301 : shell32!AssocCreateForClasses+0x530
00000000`013bf1d0 000007fe`fde4187c : 00004399`23fc272b 00000000`000ca030 00000000`00000000 00000000`000da3b8 : shell32!CommandLineToArgvW+0x3de
00000000`013bf210 000007fe`fde86df7 : 00000000`00000000 00000000`00000000 00004399`23fc3a2b 00000000`00000000 : shell32!CommandLineToArgvW+0x334
00000000`013bf2a0 000007fe`fd83191d : 78652e72`656c6c61 00000000`609834a8 00004399`23fc213e 00000000`6095153d : shell32!ILLoadFromStreamEx+0x2877
00000000`013bf320 00000000`60979085 : 00000000`000da3b8 00000000`00000001 ffffffff`fffffffe 00000000`60963194 : shlwapi!AssocQueryStringW+0x81
00000000`013bf3b0 00000000`6097930e : 00000000`00000000 00000000`00000000 00000000`013bf4a0 00000000`00000000 : AMAppHook!CAMDbgManager::GetExLogEnabled+0xa295
00000000`013bf400 00000000`60958ea4 : 00000000`013bf528 00000000`000da3b8 ffffffff`ffffffff 00000012`00de0008 : AMAppHook!CAMDbgManager::GetExLogEnabled+0xa51e
00000000`013bf450 00000000`6095cbed : 00000000`013bfa38 00000000`0039a8f8 00000000`609aa030 00000000`609c2778 : AMAppHook!CAMDbgManager::GetLogEnabled+0x2264
00000000`013bf490 00000000`76811bbc : 00000000`00000100 00000000`013bfa20 ffffffff`ffffff70 00000000`ff20227c : AMAppHook!CAMDbgManager::GetLogEnabled+0x5fad
00000000`013bf940 00000000`ff1ffe81 : 00000000`04000000 00000000`ff1e3cf8 00000000`00396618 00000000`003959c8 : kernel32!CreateProcessW+0x6c
00000000`013bf9b0 00000000`7680652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : TrustedInstaller+0x1fe81
00000000`013bff60 00000000`76efc521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`013bff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: propsys!InitPropVariantFromStringVector+165
FOLLOWUP_NAME: MachineOwner
IMAGE_NAME: propsys.dll
STACK_COMMAND: .ecxr ; kb ; ~4s; .ecxr ; kb
BUCKET_ID: WRONG_SYMBOLS
FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000005_propsys.dll!InitPropVariantFromStringVector
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/TrustedInstaller_exe/6_1_7601_17514/4ce7989b/ole32_dll/6_1_7601_17514/4ce7c92c/c0000005/00010c7b.htm?Retriage=1
Followup: MachineOwner
———
Now this is very good at telling us what the application did just before the crash, but in this scenario was not exactly pointing to the true culprit of the crash. Like a game of Chinese whispers, if somebody early on in the game corrupts the conversation, the last person is certain to get the original conversation wrong…and it’s not fair to blame them.
I wanted to check out the 3rd party modules loaded here so I ran command lmv
This produced three 3rd party modules:
00000000`0f000000 00000000`0f006000 Detoured (deferred)
Image path: C:\Windows\System32\Detoured.dll
Image name: Detoured.dll
Timestamp: Tue Jan 30 22:04:29 2007 (45BF263D)
CheckSum: 00005E0A
ImageSize: 00006000
File version: 0.0.0.0
Product version: 0.0.0.0
File flags: 0 (Mask 0)
File OS: 0 Unknown Base
File type: 0.0 Unknown
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
00000000`60950000 00000000`609d0000 AMAppHook (export symbols) AMAppHook.dll
Loaded symbol image file: AMAppHook.dll
Image path: c:\Program Files\AppSense\Application Manager\Agent\AMAppHook.dll
Image name: AMAppHook.dll
Timestamp: Thu Jul 14 00:31:36 2011 (4E1DAC48)
CheckSum: 000822B1
ImageSize: 00080000
File version: 8.3.119.0
Product version: 8.3.119.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0809.04b0
CompanyName: AppSense Ltd
ProductName: AppSense Application Manager
InternalName: AMAppHook
OriginalFilename: AMAppHook.dll
ProductVersion: 8.3.119.0
FileVersion: 8.3.119.0
SpecialBuild: 8.3.119.0
FileDescription: AppSense Application Manager Hook Utility Library
LegalCopyright: Copyright © 1999-2011 AppSense Ltd. All rights reserved.
LegalTrademarks: Copyright © 1999-2011 AppSense Ltd. All rights reserved.
Comments: Contact support@appsense.com
00000000`74770000 00000000`7478d000 EmLoader (deferred)
Image path: C:\Program Files\AppSense\Environment Manager\Agent\EmLoader.dll
Image name: EmLoader.dll
Timestamp: Sat Nov 12 02:23:17 2011 (4EBD3DE5)
CheckSum: 0001E7EB
ImageSize: 0001D000
File version: 8.2.125.0
Product version: 8.2.125.0
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04e4
CompanyName: AppSense Ltd
ProductName: AppSense Environment Manager
InternalName: EmLoader.dll
OriginalFilename: EmLoader.dll
ProductVersion: 8.2.125.0
FileVersion: 8.2.125.0
SpecialBuild: 8.2.125.0
FileDescription: Environment Manager Loader
LegalCopyright: Copyright © 1999-2011 AppSense Ltd. All rights reserved.
LegalTrademarks: Copyright © 1999-2011 AppSense Ltd. All rights reserved.
I was most interested in the Detoured.dll file as it contained no company name, product name, or file version.
We could also have seen in ProcMon this DLL if you filter on Operation is Load Image, but I prefer the WinDbg output here as all modules/version/company/file location are listed in one neat output.
More importantly we found our culprit, renaming the file TrustedInstaller started fine;
But now wanted to know, what was this file? What was it doing here?
In addition did we break anything? Well we did, other processes were broken now, like ipconfig.exe
(Well they still worked, they just threw this error dialog box)
If a 3rd party DLL is getting loaded on every EXE then probably we have something set in AppInit_DLLs (http://support.microsoft.com/kb/197571)
Here we find one of the other 3rd party DLLs loaded we saw in our crash dump
I removed the value from AppInit_DLL key, and renamed detours.dl_ back to detours.dll. Everything was working fine.
I then restarted the machine…
AppSense added the hook key back, and everybody was happy again. It seems the brief time TrustedInstaller works some Windows Updates installed, possibly this helped fix any remaining issues.
Final note, Detoured.dll opened in IDA with the strings view (or you could have just used http://live.sysinternals.com/strings.exe for this scenario)
We can see it is from Microsoft Research, available here http://www.microsoft.com/about/legal/en/us/intellectualproperty/iplicensing/programs/detours.aspx
We can also see it is version 2.1 (Version info wasn’t in the DLL version information) Detours 3.0, the latest version, offers support for 64-bit code on x64 and IA64 http://research.microsoft.com/en-us/projects/detours/
Hi, my name is Mark. I work for AppSense and I am the Technical Lead for AppSense Application Manager (the product identified in your blog post).
Firstly, thanks for performing such a thorough investigation and sharing this with the community – your post will provide people with the information they need to work around the issue should they see it in their environment.
(As background information in case you’re not familiar with AppSense Application Manager, it is a solution that controls application access and manages user rights. Essentially, it determines if an application is authorized to, or prohibited from, executing based on the context of the user, device, location, file owner or time and allows or denies the execution accordingly. If the file is authorized to run, it can also control with what rights or privileges it will execute with).
As you can see from the call stack, the AppSense Application Manager Hook calls AssocQueryStringW( ) to query file association. Without looking at the crash dump it is difficult for me to say for sure, however, it looks like either the information being passed to AssocQueryStingW( ) is invalid/corrupt or running this in the context of the trusted installer service is causing unexpected behaviour.
I have passed this onto our QA team to see if they can reproduce this issue in house. In the meantime, if you want anything from AppSense please don’t hesitate to contact me and I will be happy to help.
Thanks again,
Mark Willamson
Thanks for the response. I’ll send you a link to Process Dump & ProcMon log.
The AppSense guys should definitely upgrade to the latest version of Detours, as older versions have a number of problems.
You deserve an award for working this out.. spent an age trying to work this out in an AppSense powered environment before I found this post.
we had the same issue and we ended up MANUALY copying DLLs between server :)
http://www.techieshelp.com/faulting-application-name-trustedinstaller-exe-version-6-1-server-2008r2/