Case of the Missing Printer Ports

A Citrix environment had both Windows Server 2008 R2 published desktops and Windows 7 SP1  VDIs

On the Windows 7 VDIs the ports tab on printers added via a print server was always empty.

image

We did a quick comparison between working/broken system using AutoRuns compare functionality, but there were so many differences it was going to be easier to use debugging tools to find the cause.

The UI for this print dialog is hosted in Windows Explorer

So I started with Rohitab API Monitor and enabled tracing of Documents and Printing – Printing – Prints Spooler APIs

image

Here we saw the call to winspool.drv!EnumPorts was succeeding. However pcReturned was 0

image

Based on the MSDN documentation for EnumPorts here we know pcReturned is the number of ports retrieved from the print server.

On the working Server 2008 R2 machines this same call to EnumPorts showed pcReturned with a value of 628.

Using Windows in-built packet capture netsh trace we took a packet capture while opening the Ports tab. On the working machine we saw a DNS query to find the print server, then some additional network traffic to/from the print server as the ports were retrieved.

On the broken Windows 7 machines there was no network traffic at all, not even a DNS query.

To investigate the network call this time we attached API monitor to the Print Spooler service, spoolsv.exe and monitored Network APIs

image

In this case GetAddrInfo failed with error No Such Host Known

#    Time of Day    Thread    Module    API    Return Value    Error    Duration
18    5:06:27.421 PM    3    spoolsv.exe    GetAddrInfoW ( “PRINTSERVER01”, 0x0000000000000000, 0x0000000003c9e458, 0x0000000003c9e450 )    11001    11001 = No such host is known.     0.0000168

However if I called GetAddrInfo with the same parameters from my own program,, on the same computer it worked fine, and obtained the correct IP address for the print server.

So I attached WinDbg to the spoolsv.exe process and set breakpoint on GetAddrInfo and Set Last Error then traced some instructions

bp ws2_32!GetAddrInfoW

However it wasn’t immediately obvious what was causing the failure to resolve the host name.

To get a better understanding of call flow I set up a break point on spoolsv!EnumPortsW and created a trace log of 100,000 instructions

bp spoolsv!EnumPortsW
.logopen C:\logs\trace.txt
t 100000
.logclose

Then I used some quick & dirty PowerShell to parse this in an easy to follow manner:

$data = Get-Content C:\log\trace.txt

$last =""

ForEach ($line in $data)
{
  if ($line.Contains("!") -and $line.Contains("+"))
  {
    $b=$line.Split('+')
    if ($b[0].Contains("!") -and !$b[0].Contains(" ") -and !$b[0].StartsWith("ntdll"))
    {
    if ($last -ne $b[0])
    {
        Write-Output $b[0]
        $last=$b[0]
        }
    }
  }
}

From this it was easy to identify 3rd party code was getting executed during EnumPorts call:

spoolsv!TNameResolutionCache::GetNetBiosInfo
msvcrt!free
ntdll!RtlFreeHeap
msvcrt!free
spoolsv!TNameResolutionCache::GetNetBiosInfo
netutils!NetApiBufferFree
netutils!MIDL_user_free
KERNELBASE!LocalFree
ntdll!RtlFreeHeap
KERNELBASE!LocalFree
netutils!MIDL_user_free
netutils!NetApiBufferFree
spoolsv!TNameResolutionCache::GetNetBiosInfo
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TString::~TString
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::Next
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::~TLink
spoolsv!TNameResolutionCache::AddName
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::Next
spoolsv!NCoreLibrary::TList<TConnection>::~TList<TConnection>
spoolsv!NCoreLibrary::TLink::~TLink
spoolsv!TNameResolutionCache::AddName
spoolsv!ImpersonatePrinterClient
spoolsv!ImpersonationToken
KERNELBASE!GetLastError
spoolsv!ImpersonationToken
ADVAPI32!GetTokenInformationStub
KERNELBASE!GetTokenInformation
ntdll!ZwQueryInformationToken
KERNELBASE!GetTokenInformation
KERNELBASE!BaseSetLastNTError
ntdll!RtlNtStatusToDosError
ntdll!RtlNtStatusToDosErrorNoTeb
ntdll!RtlNtStatusToDosError
KERNELBASE!BaseSetLastNTError
ntdll!RtlSetLastWin32Error
KERNELBASE!BaseSetLastNTError
KERNELBASE!GetTokenInformation
ADVAPI32!GetTokenInformationStub
spoolsv!ImpersonationToken
ntdll!RtlSetLastWin32Error
spoolsv!ImpersonationToken
spoolsv!ImpersonatePrinterClient
ntdll!NtSetInformationThread
spoolsv!ImpersonatePrinterClient
ntdll!ZwClose
spoolsv!ImpersonatePrinterClient
spoolsv!TNameResolutionCache::AddName
spoolsv!_security_check_cookie
spoolsv!TNameResolutionCache::AddName
spoolsv!EnumPortsW
spoolsv!YEnumPorts
spoolsv!EnumPortsW
UpProv!InitializePrintProvidor

From the complete trace log we could see spoolsv!EnumPortsW had a line of the code that initially entered localspl!LocalEnumPorts

spoolsv!EnumPortsW+0xa4
call    qword ptr [rbp+150h] ss:00000000`02783390={localspl!LocalEnumPorts (000007fe`ebaa8e20)}

However the 2nd time around this call went into Citrix Universal Print Client code:

spoolsv!EnumPortsW+0xa4:
00000000`ff1fd508 ff9550010000    call    qword ptr [rbp+150h] ss:00000000`0279bf70=000007fedc7d5170
0:001> u 000007fedc7d5170
UpProv!InitializePrintProvidor+0x1240:
000007fe`dc7d5170 44894c2420      mov     dword ptr [rsp+20h],r9d
000007fe`dc7d5175 4c89442418      mov     qword ptr [rsp+18h],r8
000007fe`dc7d517a 89542410        mov     dword ptr [rsp+10h],edx
000007fe`dc7d517e 48894c2408      mov     qword ptr [rsp+8],rcx
000007fe`dc7d5183 4883ec78        sub     rsp,78h
000007fe`dc7d5187 4883bc248000000000 cmp   qword ptr [rsp+80h],0
000007fe`dc7d5190 7512            jne     UpProv!InitializePrintProvidor+0x1274 (000007fe`dc7d51a4)
000007fe`dc7d5192 b932000000      mov     ecx,32h

In WinDbg checked where this DLL originated…

0:002> lmvm upprov
Browse full module list
start             end                 module name
000007fe`dc7d0000 000007fe`dc8e6000   UpProv     (export symbols)       C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Loaded symbol image file: C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Image path: C:\Program Files\Citrix\Universal Print Client\UpProv.dll
    Image name: UpProv.dll
    Browse all global symbols  functions  data
    Timestamp:        Sun Sep 13 14:27:23 2015 (55F4FB2B)
    CheckSum:         0011AE23
    ImageSize:        00116000
    File version:     7.6.300.7024
    Product version:  7.6.300.7024
    File flags:       8 (Mask 3F) Private
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Citrix Systems, Inc.
    ProductName:      Citrix ICA Host
    InternalName:     UPPROV
    OriginalFilename: UPPROV.DLL
    ProductVersion:   7.6
    FileVersion:      7.6.300.7024
    FileDescription:  Citrix Universal Printing Provider
    LegalCopyright:   Copyright 2012-2015 Citrix Systems, Inc.

Renaming the upprov.dll and restarting Print Spooler service fixed the issue.

Further investigating the Universal Printing Client configuration we identified it had a Citrix Group Policy set to disable it :

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\PrintingPolicies]
"UpsHttpPort"=dword:00001f90
"UpsEnable"=dword:00000000

Restoring upprov.dll then setting UpsEnable to 1 (Enabled with fallback to Windows native remote printing) or 2 (Enabled with no fallback to Windows native remote printing) and restarting the Print Spooler service also fixed the issue.

About chentiangemalc

specializes in end-user computing technologies. disclaimer 1) use at your own risk. test any solution in your environment. if you do not understand the impact/consequences of what you're doing please stop, and ask advice from somebody who does. 2) views are my own at the time of posting and do not necessarily represent my current view or the view of my employer and family members/relatives. 3) over the years Microsoft/Citrix/VMWare have given me a few free shirts, pens, paper notebooks/etc. despite these gifts i will try to remain unbiased.
This entry was posted in PowerShell, API Monitor, Citrix, WinDbg and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s