Start ProcMon During Boot on Machine You Can’t Logon

Sometimes you might want to run some tracing tools like ProcMon, xperf, wpr, netsh trace, etc on a machine that logon is not completing properly.

Often you might be able to use PowerShell remoting, WMI, or psexec to get these tools launched remotely.

But for scenarios where this doesn’t work I’ve created a simple Windows service that can be applied to an offline system via Windows PE

Once the machine boots it will start tracing for specified number of seconds until it automatically stops the trace.

You can then boot back into Windows PE to grab the traces.

View the readme.txt for instructions on usage.

The binaries and Visual Studio 2015 C# project is available for download here:

https://1drv.ms/u/s!AiFhB4fT6aiTgdoPPYYD7Ru8s-3KJw

Posted in Debugging, ProcMon, Windows Performance Recorder, Windows Performance Toolkit | Tagged | Leave a comment

Removing Integrity Check from a Setup.exe

We had an executable installer that had a forced reboot and no option to remove it via command line. We wanted to run it through a task sequence and allow the task sequence to handle the reboot.

Following the process below the reboot was removed easily https://chentiangemalc.wordpress.com/2015/06/09/removing-forced-reboot-from-an-exe/

However this resulted in error message

NSIS Error

Installer integrity check has failed. Common causes include
incomplete download and damaged media. Contact the
installer’s author to obtain a new copy.

More information at:
http://nsis.sf.net/NSIS_Error

image

To remove the integrity check we opened EXE in IDA Pro and used Search Text function to find the error message

image

Selecting this an hitting ‘X’ we find the code using this reference:

image

Jumping to this code we can see the program jumps to this error message via loc_403889 label, so we select it and hit ‘X’ to reveal all the conditional jumps that result in an integrity check failure.

image

We select each one replacing them with bytes of value 90 (nop) which will ensure the integrity check failure code will never be reached. This is performed using IDA’s Edit –> Patch Program –> Change Byte command

The jg and jz instructions are replaced with 6 nop instructions, while the jnz short instructions are replaced with 2 nop instructions.

image

Resultant code changes will look something like this:

From:

image

to:

image

From:

image

To:

image

This process is continued until there are no more jumps to the integrity check.

Installer now launches fine:

image

Of course being a NSIS installer we could have just added /NCRC to the command line…

Next time we will look at how to retain the integrity check, but modify the checksum in installer to match our patches.

Posted in Hack, Hacking, IDA | Leave a comment

Misleading PowerShell Exceptions–Example #1

In PowerShell 5.0 on Windows 10 was looking at a script with function not working:

Function Test-Condition($condition)
{
    
    if ([System.String]::IsNullOrWhiteSpace($condition))
    { 
        return $true 
    }
    
    switch ($conditon.ToLower())
    {
        "true" { return $true }
        "false" { return $false }
    }

    return (Invoke-Expression $condition)
}

The exception thrown by Powershell was:

You cannot call a method on a null-valued expression.
At C:\Users\chentiangemalc\Documents\Scripts\Test.ps1:5 char:9
+     if ([System.String]::IsNullOrWhiteSpace($condition))
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

The exception is misleading because it points to the wrong line of code. The actual error is in the switch line of code, $condition was misspelled into a a non-existent variable, throwing the “cannot call a method on a null-valued expression” exception.

If the script had started with

Set-StrictMode -Version Latest

We would get a more accurate exception:

The variable ‘$conditon’ cannot be retrieved because it has not been set.
At C:\Users\chentiangemalc\Documents\Scripts\Test.ps1:11 char:13
+     switch ($conditon.ToLower())
+             ~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (conditon:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

Posted in Debugging, PowerShell | Tagged | Leave a comment

Check Validity and Download SSL Certs in PowerShell

Little utility I made to check a list of URLs, their SSL certs, including any URLs they redirect to.

The entire certificate chain for each host is also downloaded into %temp%\SslCerts folder

Script source files can be downloaded here: https://1drv.ms/u/s!Aj0HCnRwyKPhge1bfpIYIlTJ6IugVQ

The .cmd file included can be used to launch the script automatically.

Multiple URLs can be tested, provided in an PowerShell array. View the .cmd file for example.

Output of tool:

image

The script source:

# Check SSL certs by @chentiangemalc
# Source : https://chentiangemalc.wordpress.com
#
# - Runs through all URLs specified in URLs parameter, checks their cert, and downloads entire cert chain to %TEMP%\SslCerts in PCS7 format (.P7B)
# - If specified URLs redirect, the redirects will be followed
# - Outputs reasons for certificate failures
# - If report file parameter is specified log is generated
# - Specify UserAgent if desired 
#
# If URLs redirect back & forth to each other
[CmdletBinding()]
param(
    [string[]]$urls=@("http://www.google.com.au","https://www.nytimes.com"),
    [string]$ReportFile="",
    [string]$UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")

if (![string]::IsNullOrEmpty($reportfile)) { Start-Transcript $ReportFile }

$global:currentURI = $null
    
# using GLOBAL so we display SSL error info to console
$global:RemoteCertificateValidationCallback = [System.Net.Security.RemoteCertificateValidationCallback]{
    param(
	[object]$sender,
	[System.Security.Cryptography.X509Certificates.X509Certificate]$certificate,
	[System.Security.Cryptography.X509Certificates.X509Chain]$chain,
	[System.Net.Security.SslPolicyErrors]$sslPolicyErrors)
  
    # save certificate
    $certs = New-Object Security.Cryptography.X509Certificates.X509Certificate2Collection
    $chain.ChainElements | ForEach-Object {[void]$certs.Add($_.Certificate)}
    
    $sslCertPath = $env:TEMP + "\SslCerts"
    if (!(Test-Path $sslCertPath))
    {
        mkdir $sslCertPath
    }
    $certFilename = $sslCertPath + "\" + $uri.DnsSafeHost + ".p7b"
    Write-Host "Certificate saved to $certFilename"
    Set-Content -Path $certFilename -Value $certs.Export("pkcs7") -Encoding Byte

    if ($sslPolicyErrors -eq [System.Net.Security.SslPolicyErrors]::None)
    {
        Write-Host "SSL Certificate OK!" -ForegroundColor Green
    }
    else
    {
        Write-Host "SSL Certificate Errors!" -ForegroundColor Red
    }
   
    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateChainErrors))
    {
        Write-Host "- Remote Certificate Chain Errors" -ForegroundColor Red 
        ForEach ($status in $chain.ChainStatus) { Write-Host "- $($status.StatusInformation)" -ForegroundColor Red }
    }

    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateNameMismatch))
    {
        Write-Host "- Remote Certificate Name Mismatch" -ForegroundColor Red 
    }

    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateNotAvailable))
    {
        Write-Host "- Remote Certificate Not Available" -ForegroundColor Red 
    }

    # ignore invalid certificates
    return $true     
}

Function Check-SSLCertificate
{
    param([parameter(Mandatory=$true)][uri]$uri) 

    $hostname = $uri.DnsSafeHost
    $port = $uri.Port
    #Create a TCP Socket to the computer and a port number
    $tcpsocket = New-Object Net.Sockets.TcpClient($hostname, $port) 

    #test if the socket got connected
    if(!$tcpsocket)
    {
        Write-Error "Error Opening Connection: $port on $hostname Unreachable"
        exit 1
    }
    else
    {
        #Socket Got connected get the tcp stream ready to read the certificate
        write-host "Successfully Connected to $hostname on $port" -ForegroundColor Green -BackgroundColor Black
        $tcpstream = $tcpsocket.GetStream()
        Write-host "Reading SSL Certificate…." -ForegroundColor Yellow -BackgroundColor Black 
        #Create an SSL Connection 4
        $global:currentURi = $uri
        $sslStream = New-Object System.Net.Security.SslStream($tcpstream,$false,$global:RemoteCertificateValidationCallback)
        #Force the SSL Connection to send us the certificate
        $sslStream.AuthenticateAsClient($hostname) 

        #Read the certificate
        $certinfo = New-Object System.Security.Cryptography.x509certificates.x509certificate2($sslStream.RemoteCertificate)
    } 
    
    return $certinfo 
}

Function Write-URL($url)
{
    $uri = [System.Uri]$url
    Write-Host "$($uri.Scheme)://" -ForegroundColor Gray -NoNewLine
    Write-Host $uri.Host -ForegroundColor Yellow -NoNewLine
    Write-Host $uri.LocalPath -ForegroundColor Gray -NoNewLine
    Write-Host $uri.Query -ForegroundColor DarkGray -NoNewline
}
Function Get-HttpContent($url)
{
    try
    {
        Write-Host "Reading " -NoNewLine
        Write-URL $url
        Write-Host ""

        $request = [System.Net.WebRequest]::Create($url)
        $request.UserAgent = $UserAgent
        $request.AllowAutoRedirect = $false
      
        # 30 second timeout
        $request.Timeout = 30000
        $request.Method = "GET"
        $request.UseDefaultCredentials = $true 
        
        $response = $request.GetResponse()
        if ($response.StatusCode -eq "OK")
        {
            Write-Host "Status Code OK" -ForegroundColor Green
        }
        else
        {
            Write-Host "Status Code $($response.StatusCode)" -ForegroundColor Magenta
        }

        if ($response.StatusCode -eq "Ambiguous" -or 
            $response.StatusCode -eq "Found" -or 
            $response.StatusCode -eq "Redirect" -or 
            $response.StatusCode -eq "Moved" -or 
            $response.StatusCode -eq "MultipleChoices" -or 
            $response.StatusCode -eq "RedirectKeepVerb" -or 
            $response.StatusCode -eq "SeeOther" -or 
            $response.StatusCode -eq "TemporaryRedirect" -or 
            $response.StatusCode -eq "RedirectMethod" -or 
            $response.StatusCode -eq "MovedPermanently")
        {
            $redirectURL = $response.Headers["location"]
            # relative redirect
            if ($redirectURL.StartsWith("/"))
            {
                $redirectURL = $url + $redirectURL
            }

            Write-URL $url
            Write-Host " redirects to "
            Write-URL $redirectURL
            Write-Host ""
            Process-URL($redirectURL)

        }
        else
        {
            $stream = $response.GetResponseStream()
            $reader = [System.IO.StreamReader]$stream
            return $reader.ReadToEnd() 
        }
    }
    catch
    {
        Write-Host "FAILED. Error: $($_.Exception.InnerException.Message)" -ForegroundColor Red 
    }

}

Function Process-URL($url)
{
    $uri = [System.Uri]$url
    Write-Host "Checking certificate for " -NoNewLine
    Write-Host "$($uri.Scheme)://" -NoNewLine -Foregroundcolor Gray
    Write-Host "$($uri.Host)" -ForegroundColor Yellow
    if ($uri.Scheme -eq "https")
    {
       $certInfo = Check-SSLCertificate $uri
       Write-Host ""
       Write-Host "Certificate Summary for " -NoNewline
       Write-Host $($uri.Host) -ForegroundColor Yellow
       Write-Host $certInfo
       Write-Host ""
    }

    $content = Get-HttpContent($url)
    if (![String]::IsNullOrEmpty($content))
    {
        if ($content -match '<meta\s{1,}http-equiv(\s)?=(\s)?"Refresh".*url\s{0,}=(.*)"')
        {
            $redirectURL = $Matches[3].Trim()
            # relative redirect
            if ($redirectURL.StartsWith("/"))
            {
                $redirectURL = $url + $redirectURL
            }

            Write-URL $url
            Write-Host " redirects to "
            Write-URL $redirectURL
            Write-Host ""
            Process-URL($redirectURL)
        }
    }
}

ForEach ($url in $urls)
{
    Write-Host "**************************** " -NoNewLine -ForegroundColor Cyan
    Write-Host $URL -NoNewline -ForegroundColor Yellow
    Write-Host " ****************************" -ForegroundColor Cyan
    Write-Host ""
    Process-URL($url)
    Write-Host ""
}

if (![string]::IsNullOrEmpty($reportfile)) { Stop-Transcript } 
Posted in .NET, PowerShell | Tagged , | Leave a comment

Poor PAC File Performance–A WireShark Example

A problem I have come across repeatedly in large enterprises is poor web browsing performance due to over engineered PAC file. One of the most common causes of issues I’ve found is poor DNS resolution performance in conjunction with extensive DNS queries through the PAC file.

In this example the PAC file had 100+ IsInNet calls that resulted in very slow performance browsing. The affected site also suffered very poor DNS resolution performance. However removing the PAC file resulted in vastly improved browsing performance.

Some good guidelines for PAC files are here http://www.websense.com/content/support/library/web/v76/pac_file_best_practices/PAC_best_pract.aspx

Also a tool I built here PacDbg for analysing /debugging logic in PAC file https://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/

Using Fiddler and the statistics it collects also is useful in analysing PAC performance issues http://www.telerik.com/fiddler

In this case we took two WireShark traces browsing a set of websites:

  • Test #1 – PAC file
  • Test #2 – Proxy set by IP address

These were repeated several times and results were consistent. ipconfig /flushdns run before each test.

Graphs generated using WireShark’s StatisticsI/O Graph menu option.

Test #1 – PAC File – Slow

Display Filter: dns.time

  • Y AXIS: LOAD(Y field)
  • Y field: dns.time

DNS query response time ranged from 0.1 – 2.4 seconds

image

The frequency of DNS queries also analysed:

  • Display Filter: dns.flag.response eq 0
  • Y AXIS: Packet/s

image

Test #2 – Fast – Proxy set by IP address

DNS query response still slow, but much less frequent.

image

image

Posted in PacDbg, WireShark | Tagged | Leave a comment

Case of the Missing Printer Ports

A Citrix environment had both Windows Server 2008 R2 published desktops and Windows 7 SP1  VDIs

On the Windows 7 VDIs the ports tab on printers added via a print server was always empty.

image

We did a quick comparison between working/broken system using AutoRuns compare functionality, but there were so many differences it was going to be easier to use debugging tools to find the cause.

The UI for this print dialog is hosted in Windows Explorer

So I started with Rohitab API Monitor and enabled tracing of Documents and Printing – Printing – Prints Spooler APIs

image

Here we saw the call to winspool.drv!EnumPorts was succeeding. However pcReturned was 0

image

Based on the MSDN documentation for EnumPorts here we know pcReturned is the number of ports retrieved from the print server.

On the working Server 2008 R2 machines this same call to EnumPorts showed pcReturned with a value of 628.

Using Windows in-built packet capture netsh trace we took a packet capture while opening the Ports tab. On the working machine we saw a DNS query to find the print server, then some additional network traffic to/from the print server as the ports were retrieved.

On the broken Windows 7 machines there was no network traffic at all, not even a DNS query.

To investigate the network call this time we attached API monitor to the Print Spooler service, spoolsv.exe and monitored Network APIs

image

In this case GetAddrInfo failed with error No Such Host Known

#    Time of Day    Thread    Module    API    Return Value    Error    Duration
18    5:06:27.421 PM    3    spoolsv.exe    GetAddrInfoW ( “PRINTSERVER01”, 0x0000000000000000, 0x0000000003c9e458, 0x0000000003c9e450 )    11001    11001 = No such host is known.     0.0000168

However if I called GetAddrInfo with the same parameters from my own program,, on the same computer it worked fine, and obtained the correct IP address for the print server.

So I attached WinDbg to the spoolsv.exe process and set breakpoint on GetAddrInfo and Set Last Error then traced some instructions

bp ws2_32!GetAddrInfoW

However it wasn’t immediately obvious what was causing the failure to resolve the host name.

To get a better understanding of call flow I set up a break point on spoolsv!EnumPortsW and created a trace log of 100,000 instructions

bp spoolsv!EnumPortsW
.logopen C:\logs\trace.txt
t 100000
.logclose

Then I used some quick & dirty PowerShell to parse this in an easy to follow manner:

$data = Get-Content C:\log\trace.txt

$last =""

ForEach ($line in $data)
{
  if ($line.Contains("!") -and $line.Contains("+"))
  {
    $b=$line.Split('+')
    if ($b[0].Contains("!") -and !$b[0].Contains(" ") -and !$b[0].StartsWith("ntdll"))
    {
    if ($last -ne $b[0])
    {
        Write-Output $b[0]
        $last=$b[0]
        }
    }
  }
}

From this it was easy to identify 3rd party code was getting executed during EnumPorts call:

spoolsv!TNameResolutionCache::GetNetBiosInfo
msvcrt!free
ntdll!RtlFreeHeap
msvcrt!free
spoolsv!TNameResolutionCache::GetNetBiosInfo
netutils!NetApiBufferFree
netutils!MIDL_user_free
KERNELBASE!LocalFree
ntdll!RtlFreeHeap
KERNELBASE!LocalFree
netutils!MIDL_user_free
netutils!NetApiBufferFree
spoolsv!TNameResolutionCache::GetNetBiosInfo
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TString::~TString
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::Next
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::~TLink
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::Next
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::~TLink
spoolsv!TNameResolutionCache::AddName
spoolsv!ImpersonatePrinterClient
spoolsv!ImpersonationToken
KERNELBASE!GetLastError
spoolsv!ImpersonationToken
ADVAPI32!GetTokenInformationStub
KERNELBASE!GetTokenInformation
ntdll!ZwQueryInformationToken
KERNELBASE!GetTokenInformation
KERNELBASE!BaseSetLastNTError
ntdll!RtlNtStatusToDosError
ntdll!RtlNtStatusToDosErrorNoTeb
ntdll!RtlNtStatusToDosError
KERNELBASE!BaseSetLastNTError
ntdll!RtlSetLastWin32Error
KERNELBASE!BaseSetLastNTError
KERNELBASE!GetTokenInformation
ADVAPI32!GetTokenInformationStub
spoolsv!ImpersonationToken
ntdll!RtlSetLastWin32Error
spoolsv!ImpersonationToken
spoolsv!ImpersonatePrinterClient
ntdll!NtSetInformationThread
spoolsv!ImpersonatePrinterClient
ntdll!ZwClose
spoolsv!ImpersonatePrinterClient
spoolsv!TNameResolutionCache::AddName
spoolsv!_security_check_cookie
spoolsv!TNameResolutionCache::AddName
spoolsv!EnumPortsW
spoolsv!YEnumPorts
spoolsv!EnumPortsW
UpProv!InitializePrintProvidor

From the complete trace log we could see spoolsv!EnumPortsW had a line of the code that initially entered localspl!LocalEnumPorts

spoolsv!EnumPortsW+0xa4
call    qword ptr [rbp+150h] ss:00000000`02783390={localspl!LocalEnumPorts (000007fe`ebaa8e20)}

However the 2nd time around this call went into Citrix Universal Print Client code:

spoolsv!EnumPortsW+0xa4:
00000000`ff1fd508 ff9550010000    call    qword ptr [rbp+150h] ss:00000000`0279bf70=000007fedc7d5170
0:001> u 000007fedc7d5170
UpProv!InitializePrintProvidor+0x1240:
000007fe`dc7d5170 44894c2420      mov     dword ptr [rsp+20h],r9d
000007fe`dc7d5175 4c89442418      mov     qword ptr [rsp+18h],r8
000007fe`dc7d517a 89542410        mov     dword ptr [rsp+10h],edx
000007fe`dc7d517e 48894c2408      mov     qword ptr [rsp+8],rcx
000007fe`dc7d5183 4883ec78        sub     rsp,78h
000007fe`dc7d5187 4883bc248000000000 cmp   qword ptr [rsp+80h],0
000007fe`dc7d5190 7512            jne     UpProv!InitializePrintProvidor+0x1274 (000007fe`dc7d51a4)
000007fe`dc7d5192 b932000000      mov     ecx,32h

In WinDbg checked where this DLL originated…

0:002> lmvm upprov
Browse full module list
start             end                 module name
000007fe`dc7d0000 000007fe`dc8e6000   UpProv     (export symbols)       C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Loaded symbol image file: C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Image path: C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Image name: UpProv.dll
    Browse all global symbols  functions  data
    Timestamp:        Sun Sep 13 14:27:23 2015 (55F4FB2B)
    CheckSum:         0011AE23
    ImageSize:        00116000
    File version:     7.6.300.7024
    Product version:  7.6.300.7024
    File flags:       8 (Mask 3F) Private
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Citrix Systems, Inc.
    ProductName:      Citrix ICA Host
    InternalName:     UPPROV
    OriginalFilename: UPPROV.DLL
    ProductVersion:   7.6
    FileVersion:      7.6.300.7024
    FileDescription:  Citrix Universal Printing Provider
    LegalCopyright:   Copyright 2012-2015 Citrix Systems, Inc.

Renaming the upprov.dll and restarting Print Spooler service fixed the issue.

Further investigating the Universal Printing Client configuration we identified it had a Citrix Group Policy set to disable it :

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\PrintingPolicies]
"UpsHttpPort"=dword:00001f90
"UpsEnable"=dword:00000000

Restoring upprov.dll then setting UpsEnable to 1 (Enabled with fallback to Windows native remote printing) or 2 (Enabled with no fallback to Windows native remote printing) and restarting the Print Spooler service also fixed the issue.

Posted in API Monitor, Citrix, PowerShell, WinDbg | Tagged | Leave a comment

Decompile Compiled VBS EXE with WinDbg

In this case we are looking  at a 32-bit EXE with WinDbg (x86) from the Windows SDK.

This exact process may or may not work depending on how the script was compiled, but the technique will be similar for many different types of interpreted scripts that are “compiled” to EXE

First we open the executable in WinDbg with File –> Open Executable

We then run the following commands

bp vbscript!COleScript::AddNamedItem
g
bp oleaut32!SysAllocStringLen
g

Now EAX is pointing to the beginning of our decompiled script, which we can check with du @EAX

If this doesn’t point to any script, you can instead try creating a dmp file with .dump /ma <filename.dmp> then use http://live.sysinternals.com/strings.exe to parse the dmp, i.e. strings filename.dmp > out.txt and examine the output for decompiled script.

image

To write the script to a VBS file type the following additional commands:

pt
.writemem c:\support\decompiled.vbs @edi @edi+@edx

image

Open in Notepad or other editor, ensuring to select “UNICODE” format

image

Posted in Reverse Engineering, WinDbg | Tagged | Leave a comment