Case of the Totally Broken TrustedInstaller

On a group of Windows 7 x64 machine the TrustedInstaller service, listed in Windows Services as the “Windows Module Installer Service”, was failing to start:

image

We also saw evidence in the Application Event log with errors like:

Faulting application name: TrustedInstaller.exe, version: 6.1.7601.17514, time stamp: 0x4ce7989b
Faulting module name: ole32.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c92c
Exception code: 0xc0000005
Fault offset: 0x0000000000010c7b
Faulting process id: 0x1380
Faulting application start time: 0x01cd439a251e982e
Faulting application path: C:\WINDOWS\servicing\TrustedInstaller.exe
Faulting module path: C:\WINDOWS\system32\ole32.dll
Report Id: 62cd866a-af8d-11e1-8d31-0050568f668c
Faulting package full name: %14

image

We also saw events in System Event log such as The Windows Module Installer service terminated unexpectedly.  It has done this 236 time(s).

image

The impact of this is pretty significant, among other things:

  • You can’t install/remove Windows updates
  • You can’t run sfc /scannow

Initially I was providing assistance to someone interstate so I requested a full application crash dump file. To automatically generate these set the following registry keys (Create any that don’t already exist)

Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

Add the following values (Customize as you like…)

  • DumpFolder REG_EXPAND_SZ C:\CrashDumps
  • DumpCount REG_DWORD 10
  • DumpType REG_DWORD 2

This generates full application crash dumps in c:\crashdumps whenever a process crashes, and stores the last ten.

When we attempted to start the service a file TrustedInstaller.exe.<PID>.dmp was generated which we opened in WinDbg (from the Windows SDK)

Running !analyze –v we got some basic information:

ole32!CLSIDFromString+5fb
000007fe`fdb40c7b f30f7f4030      movdqu  xmmword ptr [rax+30h],xmm0

EXCEPTION_RECORD:  ffffffffffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 000007fefdb40c7b (ole32!CLSIDFromString+0x00000000000005fb)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000001
   Parameter[1]: 0000000000000030
Attempt to write to address 0000000000000030

PROCESS_NAME:  TrustedInstaller.exe

ADDITIONAL_DEBUG_TEXT: 
Use ‘!findthebuild’ command to search for the target build information.
If the build information is available, run ‘!findthebuild -s ; .reload’ to set symbol path and load symbols.

MODULE_NAME: propsys

FAULTING_MODULE: 0000000076ed0000 ntdll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce7c94a

ERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1:  0000000000000001

EXCEPTION_PARAMETER2:  0000000000000030

WRITE_ADDRESS:  0000000000000030

FOLLOWUP_IP:
propsys!InitPropVariantFromStringVector+165
000007fe`fb4a68c1 8bd8            mov     ebx,eax

DETOURED_IMAGE: 1

APP:  trustedinstaller.exe

LAST_CONTROL_TRANSFER:  from 000007fefdb319df to 000007fefdb40c7b

FAULTING_THREAD:  0000000000000e78

DEFAULT_BUCKET_ID:  WRONG_SYMBOLS

PRIMARY_PROBLEM_CLASS:  WRONG_SYMBOLS

BUGCHECK_STR:  APPLICATION_FAULT_WRONG_SYMBOLS_NULL_CLASS_PTR_WRITE

STACK_TEXT: 
00000000`013bedc0 000007fe`fdb319df : 76726553`00000002 00000000`000e82f0 00000000`000f2dd0 000007fe`fdd0a1b0 : ole32!CLSIDFromString+0x5fb
00000000`013bee40 000007fe`fdb49514 : 00000000`00000000 6f6e6143`00000000 00000000`00000000 00000000`000da3b8 : ole32+0x19df
00000000`013bef50 000007fe`fb4a68c1 : 00000000`00000000 000007fe`fb50fe38 00000000`000f2dd0 00000000`00000000 : ole32!NdrOleInitializeExtension+0x934
00000000`013bef90 000007fe`fb4a6870 : 00000000`000f2dd0 000007fe`fb50fe08 000007fe`fb50fe08 000007fe`fb54b0f8 : propsys!InitPropVariantFromStringVector+0x165
00000000`013befd0 000007fe`fb4a853a : 00000000`000f2dd0 000007fe`fb50fe38 000007fe`00000001 000007fe`00000001 : propsys!InitPropVariantFromStringVector+0x114
00000000`013bf000 000007fe`fb4a84f3 : 00000000`00000000 000007fe`fb50fe08 000007fe`fb50fe08 000007fe`fe2da2de : propsys!VariantToPropVariant+0x162
00000000`013bf030 000007fe`fb4a1e32 : 00000000`80004005 000007fe`fb4a3c34 000007fe`fe2da2de 000007fe`fd48a95f : propsys!VariantToPropVariant+0x11b
00000000`013bf060 000007fe`fb4a207e : 00000000`00000000 00000000`80070057 00000000`80004005 00000000`013bf140 : propsys!PSCreateMemoryPropertyStore+0x6ae
00000000`013bf0a0 000007fe`fb4a220d : 00000000`013bf240 00000000`00000000 00000000`013bf140 00000000`013bf140 : propsys!PSCreateMemoryPropertyStore+0x8fa
00000000`013bf0f0 000007fe`fb4a3c5d : 00000000`013bf190 00000000`000d5a70 00000000`00000000 00000000`00000000 : propsys!PSGetPropertyDescription+0x39
00000000`013bf120 000007fe`fde4d46c : 000007fe`fe272b88 00000000`013bf190 00000000`00000001 ffffffff`ffffffff : propsys!PSCoerceToCanonicalValue+0x29
00000000`013bf160 000007fe`fde41926 : 00000000`000ca030 00000000`013bf1f0 00000000`00000000 00000000`000da301 : shell32!AssocCreateForClasses+0x530
00000000`013bf1d0 000007fe`fde4187c : 00004399`23fc272b 00000000`000ca030 00000000`00000000 00000000`000da3b8 : shell32!CommandLineToArgvW+0x3de
00000000`013bf210 000007fe`fde86df7 : 00000000`00000000 00000000`00000000 00004399`23fc3a2b 00000000`00000000 : shell32!CommandLineToArgvW+0x334
00000000`013bf2a0 000007fe`fd83191d : 78652e72`656c6c61 00000000`609834a8 00004399`23fc213e 00000000`6095153d : shell32!ILLoadFromStreamEx+0x2877
00000000`013bf320 00000000`60979085 : 00000000`000da3b8 00000000`00000001 ffffffff`fffffffe 00000000`60963194 : shlwapi!AssocQueryStringW+0x81
00000000`013bf3b0 00000000`6097930e : 00000000`00000000 00000000`00000000 00000000`013bf4a0 00000000`00000000 : AMAppHook!CAMDbgManager::GetExLogEnabled+0xa295
00000000`013bf400 00000000`60958ea4 : 00000000`013bf528 00000000`000da3b8 ffffffff`ffffffff 00000012`00de0008 : AMAppHook!CAMDbgManager::GetExLogEnabled+0xa51e
00000000`013bf450 00000000`6095cbed : 00000000`013bfa38 00000000`0039a8f8 00000000`609aa030 00000000`609c2778 : AMAppHook!CAMDbgManager::GetLogEnabled+0x2264
00000000`013bf490 00000000`76811bbc : 00000000`00000100 00000000`013bfa20 ffffffff`ffffff70 00000000`ff20227c : AMAppHook!CAMDbgManager::GetLogEnabled+0x5fad
00000000`013bf940 00000000`ff1ffe81 : 00000000`04000000 00000000`ff1e3cf8 00000000`00396618 00000000`003959c8 : kernel32!CreateProcessW+0x6c
00000000`013bf9b0 00000000`7680652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : TrustedInstaller+0x1fe81
00000000`013bff60 00000000`76efc521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`013bff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  propsys!InitPropVariantFromStringVector+165

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  propsys.dll

STACK_COMMAND:  .ecxr ; kb ; ~4s; .ecxr ; kb

BUCKET_ID:  WRONG_SYMBOLS

FAILURE_BUCKET_ID:  WRONG_SYMBOLS_c0000005_propsys.dll!InitPropVariantFromStringVector

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/TrustedInstaller_exe/6_1_7601_17514/4ce7989b/ole32_dll/6_1_7601_17514/4ce7c92c/c0000005/00010c7b.htm?Retriage=1

Followup: MachineOwner
———

Now this is very good at telling us what the application did just before the crash, but in this scenario was not exactly pointing to the true culprit of the crash. Like a game of Chinese whispers, if somebody early on in the game corrupts the conversation, the last person is certain to get the original conversation wrong…and it’s not fair to blame them.

I wanted to check out the 3rd party modules loaded here so I ran command lmv

This produced three 3rd party modules:

00000000`0f000000 00000000`0f006000   Detoured   (deferred)            
    Image path: C:\Windows\System32\Detoured.dll
    Image name: Detoured.dll
    Timestamp:        Tue Jan 30 22:04:29 2007 (45BF263D)
    CheckSum:         00005E0A
    ImageSize:        00006000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        0.0 Unknown
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
00000000`60950000 00000000`609d0000   AMAppHook   (export symbols)       AMAppHook.dll
    Loaded symbol image file: AMAppHook.dll
    Image path: c:\Program Files\AppSense\Application Manager\Agent\AMAppHook.dll
    Image name: AMAppHook.dll
    Timestamp:        Thu Jul 14 00:31:36 2011 (4E1DAC48)
    CheckSum:         000822B1
    ImageSize:        00080000
    File version:     8.3.119.0
    Product version:  8.3.119.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0809.04b0
    CompanyName:      AppSense Ltd
    ProductName:      AppSense Application Manager
    InternalName:     AMAppHook
    OriginalFilename: AMAppHook.dll
    ProductVersion:   8.3.119.0
    FileVersion:      8.3.119.0
    SpecialBuild:     8.3.119.0
    FileDescription:  AppSense Application Manager Hook Utility Library
    LegalCopyright:   Copyright © 1999-2011 AppSense Ltd. All rights reserved.
    LegalTrademarks:  Copyright © 1999-2011 AppSense Ltd. All rights reserved.
    Comments:         Contact support@appsense.com
00000000`74770000 00000000`7478d000   EmLoader   (deferred)            
    Image path: C:\Program Files\AppSense\Environment Manager\Agent\EmLoader.dll
    Image name: EmLoader.dll
    Timestamp:        Sat Nov 12 02:23:17 2011 (4EBD3DE5)
    CheckSum:         0001E7EB
    ImageSize:        0001D000
    File version:     8.2.125.0
    Product version:  8.2.125.0
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04e4
    CompanyName:      AppSense Ltd
    ProductName:      AppSense Environment Manager
    InternalName:     EmLoader.dll
    OriginalFilename: EmLoader.dll
    ProductVersion:   8.2.125.0
    FileVersion:      8.2.125.0
    SpecialBuild:     8.2.125.0
    FileDescription:  Environment Manager Loader
    LegalCopyright:   Copyright © 1999-2011 AppSense Ltd. All rights reserved.
    LegalTrademarks:  Copyright © 1999-2011 AppSense Ltd. All rights reserved.

I was most interested in the Detoured.dll file as it contained no company name, product name, or file version.

We could also have seen in ProcMon this DLL if you filter on Operation is Load Image, but I prefer the WinDbg output here as all modules/version/company/file location are listed in one neat output.

More importantly we found our culprit, renaming the file TrustedInstaller started fine;

image

But now wanted to know, what was this file? What was it doing here?

In addition did we break anything? Well we did, other processes were broken now, like ipconfig.exe

image

(Well they still worked, they just threw this error dialog box)

If a 3rd party DLL is getting loaded on every EXE then probably we have something set in AppInit_DLLs (http://support.microsoft.com/kb/197571)

Here we find one of the other 3rd party DLLs loaded we saw in our crash dump

image

image

I removed the value from AppInit_DLL key, and renamed detours.dl_ back to detours.dll. Everything was working fine.

I then restarted the machine…

AppSense added the hook key back, and everybody was happy again. It seems the brief time TrustedInstaller works some Windows Updates installed, possibly this helped fix any remaining issues.

image

Final note, Detoured.dll opened in IDA with the strings view (or you could have just used http://live.sysinternals.com/strings.exe for this scenario)

image

We can see it is from Microsoft Research, available here http://www.microsoft.com/about/legal/en/us/intellectualproperty/iplicensing/programs/detours.aspx

We can also see it is version 2.1 (Version info wasn’t in the DLL version information) Detours 3.0, the latest version, offers support for 64-bit code on x64 and IA64 http://research.microsoft.com/en-us/projects/detours/

About Malcolm McCaffery

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in 64-bit, AppSense, Debugging, ProcMon, Windows 7 and tagged . Bookmark the permalink.

5 Responses to Case of the Totally Broken TrustedInstaller

  1. Hi, my name is Mark. I work for AppSense and I am the Technical Lead for AppSense Application Manager (the product identified in your blog post).

    Firstly, thanks for performing such a thorough investigation and sharing this with the community – your post will provide people with the information they need to work around the issue should they see it in their environment.

    (As background information in case you’re not familiar with AppSense Application Manager, it is a solution that controls application access and manages user rights. Essentially, it determines if an application is authorized to, or prohibited from, executing based on the context of the user, device, location, file owner or time and allows or denies the execution accordingly. If the file is authorized to run, it can also control with what rights or privileges it will execute with).

    As you can see from the call stack, the AppSense Application Manager Hook calls AssocQueryStringW( ) to query file association. Without looking at the crash dump it is difficult for me to say for sure, however, it looks like either the information being passed to AssocQueryStingW( ) is invalid/corrupt or running this in the context of the trusted installer service is causing unexpected behaviour.

    I have passed this onto our QA team to see if they can reproduce this issue in house. In the meantime, if you want anything from AppSense please don’t hesitate to contact me and I will be happy to help.

    Thanks again,
    Mark Willamson

  2. ericlaw1979 says:

    The AppSense guys should definitely upgrade to the latest version of Detours, as older versions have a number of problems.

  3. James says:

    You deserve an award for working this out.. spent an age trying to work this out in an AppSense powered environment before I found this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s