Case of ConfigMgr Setup Updates Failing–Error 0x80072efe

ConfigMgr setup successfully downloaded several thousand files during pre-requisite downloads, then failed with error:

An error has occurred while attempting to download or verify required

prerequisite components and Setup cannot continue. Review the

configmgrsetup.log file for further details.

image

Opening C:\configmgrsetup.log showed the following error:

INFO: Downloading http://go.microsoft.com/fwlink/?LinkID=808217 as MobileDevice.cab  $$<Configuration Manager Setup><03-20-2017 14:13:55.946-660><thread=3224 (0xC98)>
INFO: WinHttpQueryHeaders() in Download() returned OK (200)  $$<Configuration Manager Setup><03-20-2017 14:14:01.830-660><thread=3224 (0xC98)>
ERROR: _ReadDataAndWriteToFile() failed with 0x80072efe  $$<Configuration Manager Setup><03-20-2017 14:14:27.760-660><thread=3224 (0xC98)>
ERROR: Download() failed with 0x80072EFE  $$<Configuration Manager Setup><03-20-2017 14:14:27.776-660><thread=3224 (0xC98)>

The download always failed on same file. Re-running the download while running Fiddler

We see http://go.microsoft.com/fwlink/?LinkID=808217 downloads with result code 302, redirecting to URL http://download.microsoft.com/download/9/D/D/9DD75DA0-C7B4-48CE-B0C4-93ABAF7D5B24/MobileDevice.cab which fails with result code 504.

Fiddler shows the following info for the failing request:

http://download.microsoft.com/download/9/D/D/9DD75DA0-C7B4-48CE-B0C4-93ABAF7D5B24/MobileDevice.cab

HTTP/1.1 504 Fiddler – Receive Failure
Date: Mon, 20 Mar 2017 07:52:30 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 18:52:30.059

[Fiddler] ReadResponse() failed: The server did not return a complete response for this request. Server returned 19,871,449 bytes.                                                                                                                                                                                                                                                                                                                                                                                             

I repeated the download, this time without Fiddler, and running Wireshark capture. Using Fiddler’s File –> Import Sessions –> Packet Capture option, the trace this time showed MobileDevice.cab shows 200 response.

p1

Selecting this response in Fiddler, right clicking, and selecting Save –> Save Response –> Response Body resulted in a .CAB file that was corrupt and didn’t open.

Reviewing the Wireshark trace with a filter on the proxy server IP address ip.addr eq <proxy IP> we can see what goes wrong

The last packet received from proxy server is a TCP reset:

image

You can also see the failure at Windows API level easily using Rohitab API Monitor

For this I monitored Internet and Windows Application UI Development –> Dialog Boxes APIs for process SetupDL.exe. This process is created only when downloads start. By monitoring the Dialog Box APIs you can just search for the error message application displayed and work backwards to see events leading to error.

In this case WinHttpReadData fails with error 12030 “The Connection with the server was terminated abnormally”

image

Attempting to manually download file in Internet Explorer, Mozilla Firefox or Google Chrome from the same machine via the same proxy ended up with the same result – corrupt CAB file.

Checking the environment’s proxy.pac file I noticed there was about 6 different potential proxy servers listed.

I manually set a different proxy server….the download worked manually. Re-trying ConfigMgr update download succeeded. Further investigation would need to be done from proxy server side to identify root cause of problem.

If a fix to proxy server is not immediately available you can use the “Get the latest Configuration Manager updates” option on a machine directly connected to the internet with no proxy, to download the updates.

Posted in API Monitor, ConfigMgr, Fiddler, WireShark | Tagged | Leave a comment

Case of the Constant Google Chrome Crashes

Google chrome was constantly crashing on a Windows 7 device, even after disabling all extensions and running Chrome Cleanup Tool ( https://www.google.com/chrome/cleanup-tool/ )

Quick check of event logs using PowerShell pointed to a 3rd party module linked to crashes not only in Chrome, but other applications…

$faulting_module = @{}
$faulting_process = @{}
$faulting_process_module = @{}

Get-WinEvent -FilterHashTable @{ "LogName"="Application";"ProviderName"="Application Error"; "ID"="1000" } | ForEach {
    $faulting_process["$($_.Properties[0].Value)"]++
    $faulting_module["$($_.Properties[3].Value)"]++
    $faulting_process_module["$($_.Properties[0].Value) $($_.Properties[3].Value)"]++
     }
     
Write-Host "Faulting Processes"

$faulting_process.GetEnumerator() | Sort-Object -Descending -Property Value

Write-Host "Faulting Modules"

$faulting_module.GetEnumerator() | Sort-Object -Descending -Property Value

Write-Host "Faulting Process and Module"

$faulting_process_module.GetEnumerator() | Sort-Object -Descending -Property Value

This showed a count for each time a particular process has crashed, as well as each time a particular module was blamed for the crash.=

image

Searching the machine for CSRBthFtpShellExt.dll linked the component to a product Bluetooth Feature Pack 5.0 

Removing the application in Add/Remove programs eliminated Chrome crashes on the affected machine.

Next time we will look at why this product caused crashes in Chrome, Notepad, and Windows Explorer, among other applications…

Posted in Debugging | Tagged | Leave a comment

Case of the Lenovo Driver Download Fail–500 Internal Server Error

Needed to update the graphics driver on a Lenovo Carbon X1, quick google found the driver easy:

image

Click the link…

The page failed to load in any browser. Searching from pcsupport.lenovo.com home page also failed…

image

 

Checking with Fiddler I could see the URL http://support.lenovo.com/us/en/products/laptops-and-netbooks/thinkpad-x-series-laptops/thinkpad-x1-carbon-type-20a7-20a8 was triggering a 302 redirect to URL http://support.lenovo.com/au/en/products/laptops-and-netbooks/thinkpad-x-series-laptops/thinkpad-x1-carbon-type-20a7-20a8/20a8/s0qg00 which then redirected to the error page at https://pcsupport.lenovo.com/error.html?aspxerrorpath=/au/en/products/laptops-and-netbooks/thinkpad-x-series-laptops/thinkpad-x1-carbon-type-20a7-20a8/20a8/s0qg00

Returning to the home page, I changed country from “Australia” to “United States”

image

Driver pages loaded fine from then on…

Interestingly when I reverted country setting back to “Australia” the URL redirection had already been fixed, and the site redirected to the valid page http://support.lenovo.com/au/en/products/laptops-and-netbooks/thinkpad-x-series-laptops/thinkpad-x1-carbon-type-20a7-20a8

image

Posted in Fiddler, Windows 7 | Tagged | Leave a comment

Slow Citrix Reconnects from Windows Embedded Thin Client

Thin clients, based on Windows 7 Embedded, which I could only access remotely via VNC, were experiencing very slow Citrix reconnect times (15 seconds)

The thin clients had a 3rd party single sign on component, Imprivata, that replaced the normal Windows logon screen.

Reconnections from Linux thin client, or a Windows 10 laptop was much faster.

To quickly identify if single sign on was having an impact I uninstalled it and used web interface to launch Citrix.

Reconnection was still very slow, even launching from web interface…

To get a high level view of what was happening during connection process I used a simple ProcMon filter:

  • Include Operation is Process Start
  • Include Operation is Process Create
  • Include Operation is Process Exit
  • Event Class is Network

From this simple test we could see it was nearly 10 seconds from launch of CDViewer.exe before it even started network communications with the Citrix server.

Because CDViewer.exe is a .NET application I profiled it with a .NET profiler – in this case I used NProfiler (https://www.nprofiler.com/)

With ICA file downloaded, I launched CDViewer from NProfiler

image

This showed a large amount of time spent on JIT compilation:

image

Running this PowerShell script reduced reconnect time from web interface by nearly 10 seconds:

Measure-Command {
    $programDir = "C:\Program Files\Citrix"
    $ngenPath = "C:\windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe"
    Write-host "Obtaining executables from $programDir"
    $files = Get-ChildItem "$programDir\*" -Include *.exe,*.dll -Recurse
    ForEach ($file in $files)
    {
        Write-Host "Checking is $file is .NET assembly"
        try
        {
            #ngen'ing Dazzle.Config DLL caused crash
            if (!$file.Name.StartsWith("ms") -and !$file.Name.StartsWith("mfc") -and !$file.Name.Contains("DazzleConfig"))
            {
            [System.Reflection.AssemblyName]::GetAssemblyName($file.FullName)
            # .NET assembly... ngen it
            ".NET assembly detected, ngen'ing"
            $imagePath = '"' + $file.FullName + '"'                                                
            Start-Process -FilePath $ngenPath -ArgumentList "install", $imagePath  -Wait -NoNewWindow 
            }
        }
        catch
        {
            # not a .NET binary
        }
    }
}

However when single sign on software was re-added, reconnections were still slow.

To trace the logon process I used ProcMon and WPR in separate instances, remotely starting them via psexec  before logon started. The tools I copied into a folder C:\support on the thin client then ran these commands:

psexec \\<thin client ip>  -u username -p password cmd.exe
cd C:\support
procmon /backingfile:log.pml /accepteula /quiet
< reproduce issue >
procmon /terminate

wpr -start GeneralProfile -start CPU -start DiskIO -start FileIO -start Registry -start Network
<reproduce issue>
wpr -stop logon.etl "Slow Citrix Reconnect"

In many cases I like to have both the ProcMon and WPR trace as I find certain problems are more easily visualised in ProcMon, some better in Windows Performance Analyser.

From this we could identify several seconds were wasted during single sign on process. Imprivata provided a registry key which cut off about 2 seconds:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\SSOProvider\VDI]
"FastConnectBypass"=dword:00000001

Further analysis of traces in Windows Performance Analyzer identified high amount of activity with fbwf.sys (File Based Write Filter) during logon process.

Switching off the File Based Write Filter saved 3-5 seconds.

Uninstalling the File Based Write Filter then installing and enabling the Enhanced Write Filter the performance improvement was maintained.

Final optimisation was using AutoRuns (http://live.sysinternals.com/autoruns.exe) to review and disable any unnecessary start up items/services, which gained another second.

However there was still a 3 second delay from Citrix receiver launching before comms with Citrix service began.

From WPR traces this showed extensive registry activity during this period. This appeared to be caused by a number of things:

  • ConfMgr.dll (Citrix Receiver Configuration DLL) calling into msi.dll!GetProductInfoA and msi.dll!MsiEnumRelatedProductsA to identify installed software
  • vddvc0n.dll (Citrix Dynamic Virtual Channel Client) calling into mstscax.dll!RefreshDevicesToRedirect and mstscax.dll!GetDevicesToREdirect
Posted in Citrix | Tagged | Leave a comment

Encrypt Password In CTX1 Encoding With PowerShell

Encrypts the password in a reversible format, the format used in some Citrix applications.

Function Encrypt-Ctx1Password
{
    Param([String]$password)

        [System.Byte[]]$keys  = (0xA5,0x04,0x0F,0x41)
        [System.Byte[]]$pwd = [System.Text.Encoding]::Unicode.GetBytes($password)
        [System.Int32[]]$crypt1 = @()
        [System.Int32[]]$crypt2 = @()

        $crypt1 += ($keys[0] -bxor $pwd[0])

        for ($i = 0; $i -lt $pwd.Count-1;$i++)
        {
            $xor = ($crypt1[$i] -bxor $keys[0])
            $crypt1 += ($xor -bxor $pwd[$i+1])
        }
 
        $c = 0

        for ($i =0; $i -lt $crypt1.Length;$i++)
        {
            $ch = ($crypt1[$i] -shr $keys[1])
            $ch = ($ch -band $keys[2])
            $ch = ($ch + $keys[3])
            $crypt2 += $ch

            $ch = ($crypt1[$i] -band $keys[2])
            $ch = $ch + $keys[3]
            $crypt2 += $ch
        }

        [System.Text.ASCIIEncoding]::ASCII.GetString($crypt2)
}

Encrypt-Ctx1Password -password "Welcome123"
Posted in PowerShell | Tagged | Leave a comment

Start ProcMon During Boot on Machine You Can’t Logon

Sometimes you might want to run some tracing tools like ProcMon, xperf, wpr, netsh trace, etc on a machine that logon is not completing properly.

Often you might be able to use PowerShell remoting, WMI, or psexec to get these tools launched remotely.

But for scenarios where this doesn’t work I’ve created a simple Windows service that can be applied to an offline system via Windows PE

Once the machine boots it will start tracing for specified number of seconds until it automatically stops the trace.

You can then boot back into Windows PE to grab the traces.

View the readme.txt for instructions on usage.

The binaries and Visual Studio 2015 C# project is available for download here:

https://1drv.ms/u/s!AiFhB4fT6aiTgdoPPYYD7Ru8s-3KJw

Posted in Debugging, ProcMon, Windows Performance Recorder, Windows Performance Toolkit | Tagged | Leave a comment

Removing Integrity Check from a Setup.exe

We had an executable installer that had a forced reboot and no option to remove it via command line. We wanted to run it through a task sequence and allow the task sequence to handle the reboot.

Following the process below the reboot was removed easily https://chentiangemalc.wordpress.com/2015/06/09/removing-forced-reboot-from-an-exe/

However this resulted in error message

NSIS Error

Installer integrity check has failed. Common causes include
incomplete download and damaged media. Contact the
installer’s author to obtain a new copy.

More information at:
http://nsis.sf.net/NSIS_Error

image

To remove the integrity check we opened EXE in IDA Pro and used Search Text function to find the error message

image

Selecting this an hitting ‘X’ we find the code using this reference:

image

Jumping to this code we can see the program jumps to this error message via loc_403889 label, so we select it and hit ‘X’ to reveal all the conditional jumps that result in an integrity check failure.

image

We select each one replacing them with bytes of value 90 (nop) which will ensure the integrity check failure code will never be reached. This is performed using IDA’s Edit –> Patch Program –> Change Byte command

The jg and jz instructions are replaced with 6 nop instructions, while the jnz short instructions are replaced with 2 nop instructions.

image

Resultant code changes will look something like this:

From:

image

to:

image

From:

image

To:

image

This process is continued until there are no more jumps to the integrity check.

Installer now launches fine:

image

Of course being a NSIS installer we could have just added /NCRC to the command line…

Next time we will look at how to retain the integrity check, but modify the checksum in installer to match our patches.

Posted in Hack, Hacking, IDA | Leave a comment