Slow Citrix Reconnects from Windows Embedded Thin Client

Thin clients, based on Windows 7 Embedded, which I could only access remotely via VNC, were experiencing very slow Citrix reconnect times (15 seconds)

The thin clients had a 3rd party single sign on component, Imprivata, that replaced the normal Windows logon screen.

Reconnections from Linux thin client, or a Windows 10 laptop was much faster.

To quickly identify if single sign on was having an impact I uninstalled it and used web interface to launch Citrix.

Reconnection was still very slow, even launching from web interface…

To get a high level view of what was happening during connection process I used a simple ProcMon filter:

  • Include Operation is Process Start
  • Include Operation is Process Create
  • Include Operation is Process Exit
  • Event Class is Network

From this simple test we could see it was nearly 10 seconds from launch of CDViewer.exe before it even started network communications with the Citrix server.

Because CDViewer.exe is a .NET application I profiled it with a .NET profiler – in this case I used NProfiler (https://www.nprofiler.com/)

With ICA file downloaded, I launched CDViewer from NProfiler

image

This showed a large amount of time spent on JIT compilation:

image

Running this PowerShell script reduced reconnect time from web interface by nearly 10 seconds:

Measure-Command {
    $programDir = "C:\Program Files\Citrix"
    $ngenPath = "C:\windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe"
    Write-host "Obtaining executables from $programDir"
    $files = Get-ChildItem "$programDir\*" -Include *.exe,*.dll -Recurse
    ForEach ($file in $files)
    {
        Write-Host "Checking is $file is .NET assembly"
        try
        {
            #ngen'ing Dazzle.Config DLL caused crash
            if (!$file.Name.StartsWith("ms") -and !$file.Name.StartsWith("mfc") -and !$file.Name.Contains("DazzleConfig"))
            {
            [System.Reflection.AssemblyName]::GetAssemblyName($file.FullName)
            # .NET assembly... ngen it
            ".NET assembly detected, ngen'ing"
            $imagePath = '"' + $file.FullName + '"'                                                
            Start-Process -FilePath $ngenPath -ArgumentList "install", $imagePath  -Wait -NoNewWindow 
            }
        }
        catch
        {
            # not a .NET binary
        }
    }
}

However when single sign on software was re-added, reconnections were still slow.

To trace the logon process I used ProcMon and WPR in separate instances, remotely starting them via psexec  before logon started. The tools I copied into a folder C:\support on the thin client then ran these commands:

psexec \\<thin client ip>  -u username -p password cmd.exe
cd C:\support
procmon /backingfile:log.pml /accepteula /quiet
< reproduce issue >
procmon /terminate

wpr -start GeneralProfile -start CPU -start DiskIO -start FileIO -start Registry -start Network
<reproduce issue>
wpr -stop logon.etl "Slow Citrix Reconnect"

In many cases I like to have both the ProcMon and WPR trace as I find certain problems are more easily visualised in ProcMon, some better in Windows Performance Analyser.

From this we could identify several seconds were wasted during single sign on process. Imprivata provided a registry key which cut off about 2 seconds:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\SSOProvider\VDI]
"FastConnectBypass"=dword:00000001

Further analysis of traces in Windows Performance Analyzer identified high amount of activity with fbwf.sys (File Based Write Filter) during logon process.

Switching off the File Based Write Filter saved 3-5 seconds.

Uninstalling the File Based Write Filter then installing and enabling the Enhanced Write Filter the performance improvement was maintained.

Final optimisation was using AutoRuns (http://live.sysinternals.com/autoruns.exe) to review and disable any unnecessary start up items/services, which gained another second.

However there was still a 3 second delay from Citrix receiver launching before comms with Citrix service began.

From WPR traces this showed extensive registry activity during this period. This appeared to be caused by a number of things:

  • ConfMgr.dll (Citrix Receiver Configuration DLL) calling into msi.dll!GetProductInfoA and msi.dll!MsiEnumRelatedProductsA to identify installed software
  • vddvc0n.dll (Citrix Dynamic Virtual Channel Client) calling into mstscax.dll!RefreshDevicesToRedirect and mstscax.dll!GetDevicesToREdirect
Posted in Citrix | Tagged | Leave a comment

Encrypt Password In CTX1 Encoding With PowerShell

Encrypts the password in a reversible format, the format used in some Citrix applications.

Function Encrypt-Ctx1Password
{
    Param([String]$password)

        [System.Byte[]]$keys  = (0xA5,0x04,0x0F,0x41)
        [System.Byte[]]$pwd = [System.Text.Encoding]::Unicode.GetBytes($password)
        [System.Int32[]]$crypt1 = @()
        [System.Int32[]]$crypt2 = @()

        $crypt1 += ($keys[0] -bxor $pwd[0])

        for ($i = 0; $i -lt $pwd.Count-1;$i++)
        {
            $xor = ($crypt1[$i] -bxor $keys[0])
            $crypt1 += ($xor -bxor $pwd[$i+1])
        }
 
        $c = 0

        for ($i =0; $i -lt $crypt1.Length;$i++)
        {
            $ch = ($crypt1[$i] -shr $keys[1])
            $ch = ($ch -band $keys[2])
            $ch = ($ch + $keys[3])
            $crypt2 += $ch

            $ch = ($crypt1[$i] -band $keys[2])
            $ch = $ch + $keys[3]
            $crypt2 += $ch
        }

        [System.Text.ASCIIEncoding]::ASCII.GetString($crypt2)
}

Encrypt-Ctx1Password -password "Welcome123"
Posted in PowerShell | Tagged | Leave a comment

Start ProcMon During Boot on Machine You Can’t Logon

Sometimes you might want to run some tracing tools like ProcMon, xperf, wpr, netsh trace, etc on a machine that logon is not completing properly.

Often you might be able to use PowerShell remoting, WMI, or psexec to get these tools launched remotely.

But for scenarios where this doesn’t work I’ve created a simple Windows service that can be applied to an offline system via Windows PE

Once the machine boots it will start tracing for specified number of seconds until it automatically stops the trace.

You can then boot back into Windows PE to grab the traces.

View the readme.txt for instructions on usage.

The binaries and Visual Studio 2015 C# project is available for download here:

https://1drv.ms/u/s!AiFhB4fT6aiTgdoPPYYD7Ru8s-3KJw

Posted in Debugging, ProcMon, Windows Performance Recorder, Windows Performance Toolkit | Tagged | Leave a comment

Removing Integrity Check from a Setup.exe

We had an executable installer that had a forced reboot and no option to remove it via command line. We wanted to run it through a task sequence and allow the task sequence to handle the reboot.

Following the process below the reboot was removed easily https://chentiangemalc.wordpress.com/2015/06/09/removing-forced-reboot-from-an-exe/

However this resulted in error message

NSIS Error

Installer integrity check has failed. Common causes include
incomplete download and damaged media. Contact the
installer’s author to obtain a new copy.

More information at:
http://nsis.sf.net/NSIS_Error

image

To remove the integrity check we opened EXE in IDA Pro and used Search Text function to find the error message

image

Selecting this an hitting ‘X’ we find the code using this reference:

image

Jumping to this code we can see the program jumps to this error message via loc_403889 label, so we select it and hit ‘X’ to reveal all the conditional jumps that result in an integrity check failure.

image

We select each one replacing them with bytes of value 90 (nop) which will ensure the integrity check failure code will never be reached. This is performed using IDA’s Edit –> Patch Program –> Change Byte command

The jg and jz instructions are replaced with 6 nop instructions, while the jnz short instructions are replaced with 2 nop instructions.

image

Resultant code changes will look something like this:

From:

image

to:

image

From:

image

To:

image

This process is continued until there are no more jumps to the integrity check.

Installer now launches fine:

image

Of course being a NSIS installer we could have just added /NCRC to the command line…

Next time we will look at how to retain the integrity check, but modify the checksum in installer to match our patches.

Posted in Hack, Hacking, IDA | Leave a comment

Misleading PowerShell Exceptions–Example #1

In PowerShell 5.0 on Windows 10 was looking at a script with function not working:

Function Test-Condition($condition)
{
    
    if ([System.String]::IsNullOrWhiteSpace($condition))
    { 
        return $true 
    }
    
    switch ($conditon.ToLower())
    {
        "true" { return $true }
        "false" { return $false }
    }

    return (Invoke-Expression $condition)
}

The exception thrown by Powershell was:

You cannot call a method on a null-valued expression.
At C:\Users\chentiangemalc\Documents\Scripts\Test.ps1:5 char:9
+     if ([System.String]::IsNullOrWhiteSpace($condition))
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

The exception is misleading because it points to the wrong line of code. The actual error is in the switch line of code, $condition was misspelled into a a non-existent variable, throwing the “cannot call a method on a null-valued expression” exception.

If the script had started with

Set-StrictMode -Version Latest

We would get a more accurate exception:

The variable ‘$conditon’ cannot be retrieved because it has not been set.
At C:\Users\chentiangemalc\Documents\Scripts\Test.ps1:11 char:13
+     switch ($conditon.ToLower())
+             ~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (conditon:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

Posted in Debugging, PowerShell | Tagged | Leave a comment

Check Validity and Download SSL Certs in PowerShell

Little utility I made to check a list of URLs, their SSL certs, including any URLs they redirect to.

The entire certificate chain for each host is also downloaded into %temp%\SslCerts folder

Script source files can be downloaded here: https://1drv.ms/u/s!Aj0HCnRwyKPhge1bfpIYIlTJ6IugVQ

The .cmd file included can be used to launch the script automatically.

Multiple URLs can be tested, provided in an PowerShell array. View the .cmd file for example.

Output of tool:

image

The script source:

# Check SSL certs by @chentiangemalc
# Source : https://chentiangemalc.wordpress.com
#
# - Runs through all URLs specified in URLs parameter, checks their cert, and downloads entire cert chain to %TEMP%\SslCerts in PCS7 format (.P7B)
# - If specified URLs redirect, the redirects will be followed
# - Outputs reasons for certificate failures
# - If report file parameter is specified log is generated
# - Specify UserAgent if desired 
#
# If URLs redirect back & forth to each other
[CmdletBinding()]
param(
    [string[]]$urls=@("http://www.google.com.au","https://www.nytimes.com"),
    [string]$ReportFile="",
    [string]$UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")

if (![string]::IsNullOrEmpty($reportfile)) { Start-Transcript $ReportFile }

$global:currentURI = $null
    
# using GLOBAL so we display SSL error info to console
$global:RemoteCertificateValidationCallback = [System.Net.Security.RemoteCertificateValidationCallback]{
    param(
	[object]$sender,
	[System.Security.Cryptography.X509Certificates.X509Certificate]$certificate,
	[System.Security.Cryptography.X509Certificates.X509Chain]$chain,
	[System.Net.Security.SslPolicyErrors]$sslPolicyErrors)
  
    # save certificate
    $certs = New-Object Security.Cryptography.X509Certificates.X509Certificate2Collection
    $chain.ChainElements | ForEach-Object {[void]$certs.Add($_.Certificate)}
    
    $sslCertPath = $env:TEMP + "\SslCerts"
    if (!(Test-Path $sslCertPath))
    {
        mkdir $sslCertPath
    }
    $certFilename = $sslCertPath + "\" + $uri.DnsSafeHost + ".p7b"
    Write-Host "Certificate saved to $certFilename"
    Set-Content -Path $certFilename -Value $certs.Export("pkcs7") -Encoding Byte

    if ($sslPolicyErrors -eq [System.Net.Security.SslPolicyErrors]::None)
    {
        Write-Host "SSL Certificate OK!" -ForegroundColor Green
    }
    else
    {
        Write-Host "SSL Certificate Errors!" -ForegroundColor Red
    }
   
    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateChainErrors))
    {
        Write-Host "- Remote Certificate Chain Errors" -ForegroundColor Red 
        ForEach ($status in $chain.ChainStatus) { Write-Host "- $($status.StatusInformation)" -ForegroundColor Red }
    }

    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateNameMismatch))
    {
        Write-Host "- Remote Certificate Name Mismatch" -ForegroundColor Red 
    }

    if ($sslPolicyErrors.HasFlag([System.Net.Security.SslPolicyErrors]::RemoteCertificateNotAvailable))
    {
        Write-Host "- Remote Certificate Not Available" -ForegroundColor Red 
    }

    # ignore invalid certificates
    return $true     
}

Function Check-SSLCertificate
{
    param([parameter(Mandatory=$true)][uri]$uri) 

    $hostname = $uri.DnsSafeHost
    $port = $uri.Port
    #Create a TCP Socket to the computer and a port number
    $tcpsocket = New-Object Net.Sockets.TcpClient($hostname, $port) 

    #test if the socket got connected
    if(!$tcpsocket)
    {
        Write-Error "Error Opening Connection: $port on $hostname Unreachable"
        exit 1
    }
    else
    {
        #Socket Got connected get the tcp stream ready to read the certificate
        write-host "Successfully Connected to $hostname on $port" -ForegroundColor Green -BackgroundColor Black
        $tcpstream = $tcpsocket.GetStream()
        Write-host "Reading SSL Certificate…." -ForegroundColor Yellow -BackgroundColor Black 
        #Create an SSL Connection 4
        $global:currentURi = $uri
        $sslStream = New-Object System.Net.Security.SslStream($tcpstream,$false,$global:RemoteCertificateValidationCallback)
        #Force the SSL Connection to send us the certificate
        $sslStream.AuthenticateAsClient($hostname) 

        #Read the certificate
        $certinfo = New-Object System.Security.Cryptography.x509certificates.x509certificate2($sslStream.RemoteCertificate)
    } 
    
    return $certinfo 
}

Function Write-URL($url)
{
    $uri = [System.Uri]$url
    Write-Host "$($uri.Scheme)://" -ForegroundColor Gray -NoNewLine
    Write-Host $uri.Host -ForegroundColor Yellow -NoNewLine
    Write-Host $uri.LocalPath -ForegroundColor Gray -NoNewLine
    Write-Host $uri.Query -ForegroundColor DarkGray -NoNewline
}
Function Get-HttpContent($url)
{
    try
    {
        Write-Host "Reading " -NoNewLine
        Write-URL $url
        Write-Host ""

        $request = [System.Net.WebRequest]::Create($url)
        $request.UserAgent = $UserAgent
        $request.AllowAutoRedirect = $false
      
        # 30 second timeout
        $request.Timeout = 30000
        $request.Method = "GET"
        $request.UseDefaultCredentials = $true 
        
        $response = $request.GetResponse()
        if ($response.StatusCode -eq "OK")
        {
            Write-Host "Status Code OK" -ForegroundColor Green
        }
        else
        {
            Write-Host "Status Code $($response.StatusCode)" -ForegroundColor Magenta
        }

        if ($response.StatusCode -eq "Ambiguous" -or 
            $response.StatusCode -eq "Found" -or 
            $response.StatusCode -eq "Redirect" -or 
            $response.StatusCode -eq "Moved" -or 
            $response.StatusCode -eq "MultipleChoices" -or 
            $response.StatusCode -eq "RedirectKeepVerb" -or 
            $response.StatusCode -eq "SeeOther" -or 
            $response.StatusCode -eq "TemporaryRedirect" -or 
            $response.StatusCode -eq "RedirectMethod" -or 
            $response.StatusCode -eq "MovedPermanently")
        {
            $redirectURL = $response.Headers["location"]
            # relative redirect
            if ($redirectURL.StartsWith("/"))
            {
                $redirectURL = $url + $redirectURL
            }

            Write-URL $url
            Write-Host " redirects to "
            Write-URL $redirectURL
            Write-Host ""
            Process-URL($redirectURL)

        }
        else
        {
            $stream = $response.GetResponseStream()
            $reader = [System.IO.StreamReader]$stream
            return $reader.ReadToEnd() 
        }
    }
    catch
    {
        Write-Host "FAILED. Error: $($_.Exception.InnerException.Message)" -ForegroundColor Red 
    }

}

Function Process-URL($url)
{
    $uri = [System.Uri]$url
    Write-Host "Checking certificate for " -NoNewLine
    Write-Host "$($uri.Scheme)://" -NoNewLine -Foregroundcolor Gray
    Write-Host "$($uri.Host)" -ForegroundColor Yellow
    if ($uri.Scheme -eq "https")
    {
       $certInfo = Check-SSLCertificate $uri
       Write-Host ""
       Write-Host "Certificate Summary for " -NoNewline
       Write-Host $($uri.Host) -ForegroundColor Yellow
       Write-Host $certInfo
       Write-Host ""
    }

    $content = Get-HttpContent($url)
    if (![String]::IsNullOrEmpty($content))
    {
        if ($content -match '<meta\s{1,}http-equiv(\s)?=(\s)?"Refresh".*url\s{0,}=(.*)"')
        {
            $redirectURL = $Matches[3].Trim()
            # relative redirect
            if ($redirectURL.StartsWith("/"))
            {
                $redirectURL = $url + $redirectURL
            }

            Write-URL $url
            Write-Host " redirects to "
            Write-URL $redirectURL
            Write-Host ""
            Process-URL($redirectURL)
        }
    }
}

ForEach ($url in $urls)
{
    Write-Host "**************************** " -NoNewLine -ForegroundColor Cyan
    Write-Host $URL -NoNewline -ForegroundColor Yellow
    Write-Host " ****************************" -ForegroundColor Cyan
    Write-Host ""
    Process-URL($url)
    Write-Host ""
}

if (![string]::IsNullOrEmpty($reportfile)) { Stop-Transcript } 
Posted in .NET, PowerShell | Tagged , | Leave a comment

Poor PAC File Performance–A WireShark Example

A problem I have come across repeatedly in large enterprises is poor web browsing performance due to over engineered PAC file. One of the most common causes of issues I’ve found is poor DNS resolution performance in conjunction with extensive DNS queries through the PAC file.

In this example the PAC file had 100+ IsInNet calls that resulted in very slow performance browsing. The affected site also suffered very poor DNS resolution performance. However removing the PAC file resulted in vastly improved browsing performance.

Some good guidelines for PAC files are here http://www.websense.com/content/support/library/web/v76/pac_file_best_practices/PAC_best_pract.aspx

Also a tool I built here PacDbg for analysing /debugging logic in PAC file https://chentiangemalc.wordpress.com/2013/09/30/pacdbg-custom-proxy-browser-set-proxy-cmd-line-tool/

Using Fiddler and the statistics it collects also is useful in analysing PAC performance issues http://www.telerik.com/fiddler

In this case we took two WireShark traces browsing a set of websites:

  • Test #1 – PAC file
  • Test #2 – Proxy set by IP address

These were repeated several times and results were consistent. ipconfig /flushdns run before each test.

Graphs generated using WireShark’s StatisticsI/O Graph menu option.

Test #1 – PAC File – Slow

Display Filter: dns.time

  • Y AXIS: LOAD(Y field)
  • Y field: dns.time

DNS query response time ranged from 0.1 – 2.4 seconds

image

The frequency of DNS queries also analysed:

  • Display Filter: dns.flag.response eq 0
  • Y AXIS: Packet/s

image

Test #2 – Fast – Proxy set by IP address

DNS query response still slow, but much less frequent.

image

image

Posted in PacDbg, WireShark | Tagged | Leave a comment