Time Travel Debugging in WinDbg Preview

The functionality similar to Microsoft’s internal Time Travel Tracing Diagnostic Tool has now added into WinDbg Preview, available only from Windows Store here

This version of WinDbg requires Windows 10 Anniversary update or later, and Windows Store access.

Time Travel Debugging is a powerful feature, allowing you to record program execution. Then you can go backwards or forwards through this execution to hit breakpoints or  perform stack analysis.

Documentation is available here

Note: You must use Run As Administrator when launching WinDbg to use Time Travel tracing.

Currently only user mode process is supported, not kernel debugging.

To test this out a friend launched a batch file that was failing, but couldn’t work out why.

They launched the application with WinDbg and selected “Record Process with Time Travel Debugging

You can also use Attach to process and select option to record process with time travel debugging.

image

Once the trace was created, they sent me the trace, and unlike a normal dmp file, I can now set breakpoints, and move forwards and backwards through the program execution with g and g- command.

Here I used tt to set time travel position, bp to set a breakpoint on SetLastError, !positions to show position in time travel trace, !error @r8 to translate error code passed to SetLastError into a human readable message, k for stack trace, and g to continue.

0:000> !tt 0
Setting position to the beginning of the trace
Setting position: 11:0
(6e00.14c4): Break instruction exception – code 80000003 (first/second chance not available)
Time Travel Position: 11:0
ntdll!NtQueryVolumeInformationFile+0x14:
00007ffb`a6ca5cc4 c3              ret
0:000> bp ntdll!RtlSetLastWin32Error “!positions;!error @r8;k;g”
0:000> g
>Thread ID=0x14C4 – Position: 5F:4EF
Error code: (Win32) 0 (0) – The operation completed successfully.
# Child-SP          RetAddr           Call Site
00 0000004b`8c94f048 00007ffb`a60ca93a ntdll!RtlSetLastWin32Error
01 0000004b`8c94f050 00007ffb`a6091040 msvcrt!getptd_noexit+0x6e
02 0000004b`8c94f080 00007ffb`a60eaa56 msvcrt!_LocaleUpdate::_LocaleUpdate+0x20
03 0000004b`8c94f0b0 00007ff7`aa04ffc6 msvcrt!wcsicmp_l+0x26
04 0000004b`8c94f110 00007ff7`aa04f5b9 cmd!FindAndFix+0x156
05 0000004b`8c94f3a0 00007ff7`aa04fcb5 cmd!FindFixAndRun+0x99
06 0000004b`8c94f840 00007ff7`aa05cf85 cmd!Dispatch+0xa5
07 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x5205
08 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
09 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
0a 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
>Thread ID=0x14C4 – Position: 90:1F
Error code: (NTSTATUS) 0x569a0150 (1452933456) – <Unable to get error code text>
# Child-SP          RetAddr           Call Site
00 0000004b`8c94ea98 00007ffb`a3354357 ntdll!RtlSetLastWin32Error
01 0000004b`8c94eaa0 00007ff7`aa0497f5 KERNELBASE!CreateDirectoryW+0x3d867
02 0000004b`8c94eb90 00007ff7`aa049aa7 cmd!MdWork+0xe5
03 0000004b`8c94ee10 00007ff7`aa04995a cmd!LoopThroughArgs+0x13f
04 0000004b`8c94f370 00007ff7`aa04f7d3 cmd!eMkdir+0x1a
05 0000004b`8c94f3a0 00007ff7`aa04fcb5 cmd!FindFixAndRun+0x2b3
06 0000004b`8c94f840 00007ff7`aa05cf85 cmd!Dispatch+0xa5
07 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x5205
08 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
09 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
0a 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
>Thread ID=0x14C4 – Position: B6:5CB
Error code: (Win32) 0xce (206) – The filename or extension is too long.
# Child-SP          RetAddr           Call Site
00 0000004b`8c94e7b8 00007ffb`a56f2d6e ntdll!RtlSetLastWin32Error
01 0000004b`8c94e7c0 00007ffb`a32e7ecd KERNEL32!BasepNotifyLoadStringResource+0x7e
02 0000004b`8c94e810 00007ffb`a32e7c97 KERNELBASE!BaseDllFormatMessage+0x22d
03 0000004b`8c94e940 00007ff7`aa0542c9 KERNELBASE!FormatMessageW+0x37
04 0000004b`8c94e990 00007ff7`aa054191 cmd!FindMsg+0x6d
05 0000004b`8c94ea90 00007ff7`aa05439c cmd!PutMsg+0x81
06 0000004b`8c94eb50 00007ff7`aa059ab2 cmd!PutStdErr+0x2c
07 0000004b`8c94eb90 00007ff7`aa049aa7 cmd!_chkstk+0x1d32
08 0000004b`8c94ee10 00007ff7`aa04995a cmd!LoopThroughArgs+0x13f
09 0000004b`8c94f370 00007ff7`aa04f7d3 cmd!eMkdir+0x1a
0a 0000004b`8c94f3a0 00007ff7`aa04fcb5 cmd!FindFixAndRun+0x2b3
0b 0000004b`8c94f840 00007ff7`aa05cf85 cmd!Dispatch+0xa5
0c 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x5205
0d 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
0e 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
0f 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
>Thread ID=0x14C4 – Position: B6:5E5
Error code: (Win32) 0xce (206) – The filename or extension is too long.
# Child-SP          RetAddr           Call Site
00 0000004b`8c94e7b8 00007ffb`a56f2d76 ntdll!RtlSetLastWin32Error
01 0000004b`8c94e7c0 00007ffb`a32e7ecd KERNEL32!BasepNotifyLoadStringResource+0x86
02 0000004b`8c94e810 00007ffb`a32e7c97 KERNELBASE!BaseDllFormatMessage+0x22d
03 0000004b`8c94e940 00007ff7`aa0542c9 KERNELBASE!FormatMessageW+0x37
04 0000004b`8c94e990 00007ff7`aa054191 cmd!FindMsg+0x6d
05 0000004b`8c94ea90 00007ff7`aa05439c cmd!PutMsg+0x81
06 0000004b`8c94eb50 00007ff7`aa059ab2 cmd!PutStdErr+0x2c
07 0000004b`8c94eb90 00007ff7`aa049aa7 cmd!_chkstk+0x1d32
08 0000004b`8c94ee10 00007ff7`aa04995a cmd!LoopThroughArgs+0x13f
09 0000004b`8c94f370 00007ff7`aa04f7d3 cmd!eMkdir+0x1a
0a 0000004b`8c94f3a0 00007ff7`aa04fcb5 cmd!FindFixAndRun+0x2b3
0b 0000004b`8c94f840 00007ff7`aa05cf85 cmd!Dispatch+0xa5
0c 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x5205
0d 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
0e 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
0f 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
>Thread ID=0x14C4 – Position: 13B:625
Error code: (Win32) 0 (0) – The operation completed successfully.
# Child-SP          RetAddr           Call Site
00 0000004b`8c94ee08 00007ffb`a60ca93a ntdll!RtlSetLastWin32Error
01 0000004b`8c94ee10 00007ffb`a6091040 msvcrt!getptd_noexit+0x6e
02 0000004b`8c94ee40 00007ffb`a60e50a2 msvcrt!_LocaleUpdate::_LocaleUpdate+0x20
03 0000004b`8c94ee70 00007ffb`a60da39c msvcrt!woutput_l+0x72
04 0000004b`8c94f380 00007ffb`a60da301 msvcrt!vsnwprintf_l+0x8c
05 0000004b`8c94f3f0 00007ff7`aa051c39 msvcrt!vsnwprintf+0x11
06 0000004b`8c94f430 00007ff7`aa051c03 cmd!StringCchPrintfW+0x79
07 0000004b`8c94f460 00007ff7`aa0685c4 cmd!StringCchPrintfW+0x43
08 0000004b`8c94f4a0 00007ff7`aa05b6ed cmd!PrintPrompt+0x230
09 0000004b`8c94f7a0 00007ff7`aa04ddbc cmd!_chkstk+0x396d
0a 0000004b`8c94f800 00007ff7`aa04d6b6 cmd!Lex+0x47c
0b 0000004b`8c94f870 00007ff7`aa04d3f8 cmd!GeToken+0x26
0c 0000004b`8c94f8a0 00007ff7`aa05cf4b cmd!Parser+0x118
0d 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x51cb
0e 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
0f 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
10 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
TTD: End of trace reached.
(6e00.14c4): Break instruction exception – code 80000003 (first/second chance not available)
Time Travel Position: 15D:0
ntdll!NtDeviceIoControlFile+0x12:
00007ffb`a6ca5482 0f05            syscall

Now I could click the time travel position, or type cmd tt B6:5E5 to go position of interest. We could then set further breakpoints, or trace instruction by instruction use t command.

The great thing is you can also set breakpoints and go backwards (g-) or start again to explore alternative debugging approach.

Continuing on…

0:000> !tt B6:5E5
Setting position: B6:5E5
(6e00.14c4): Break instruction exception – code 80000003 (first/second chance not available)
Time Travel Position: B6:5E5
ntdll!RtlSetLastWin32Error:
00007ffb`a6c5a630 894c2408        mov     dword ptr [rsp+8],ecx ss:0000004b`8c94e7c0=000000ce
0:000> k
# Child-SP          RetAddr           Call Site
00 0000004b`8c94e7b8 00007ffb`a56f2d76 ntdll!RtlSetLastWin32Error
01 0000004b`8c94e7c0 00007ffb`a32e7ecd KERNEL32!BasepNotifyLoadStringResource+0x86
02 0000004b`8c94e810 00007ffb`a32e7c97 KERNELBASE!BaseDllFormatMessage+0x22d
03 0000004b`8c94e940 00007ff7`aa0542c9 KERNELBASE!FormatMessageW+0x37
04 0000004b`8c94e990 00007ff7`aa054191 cmd!FindMsg+0x6d
05 0000004b`8c94ea90 00007ff7`aa05439c cmd!PutMsg+0x81
06 0000004b`8c94eb50 00007ff7`aa059ab2 cmd!PutStdErr+0x2c
07 0000004b`8c94eb90 00007ff7`aa049aa7 cmd!_chkstk+0x1d32
08 0000004b`8c94ee10 00007ff7`aa04995a cmd!LoopThroughArgs+0x13f
09 0000004b`8c94f370 00007ff7`aa04f7d3 cmd!eMkdir+0x1a
0a 0000004b`8c94f3a0 00007ff7`aa04fcb5 cmd!FindFixAndRun+0x2b3
0b 0000004b`8c94f840 00007ff7`aa05cf85 cmd!Dispatch+0xa5
0c 0000004b`8c94f8d0 00007ff7`aa0568c9 cmd!_chkstk+0x5205
0d 0000004b`8c94f970 00007ffb`a56f2774 cmd!wil::details_abi::ProcessLocalStorage<wil::details_abi::ProcessLocalData>::~ProcessLocalStorage<wil::details_abi::ProcessLocalData>+0x289
0e 0000004b`8c94f9b0 00007ffb`a6c70d51 KERNEL32!BaseThreadInitThunk+0x14
0f 0000004b`8c94f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:000> bp cmd!_chkstk+0x1d32
0:000> g
Breakpoint 2 hit
Time Travel Position: D6:42
cmd!_chkstk+0x1d32:
00007ff7`aa059ab2 90              nop
0:000> t
Time Travel Position: D6:43
cmd!_chkstk+0x1d33:
00007ff7`aa059ab3 e943fdfeff      jmp     cmd!MdWork+0xeb (00007ff7`aa0497fb)
0:000> t
Time Travel Position: D6:44
cmd!MdWork+0xeb:
00007ff7`aa0497fb 8bdf            mov     ebx,edi
0:000> t
Time Travel Position: D6:45
cmd!MdWork+0xed:
00007ff7`aa0497fd 488b8d38010000  mov     rcx,qword ptr [rbp+138h] ss:0000004b`8c94edc8=0000019f56bf0780
0:000> du 0000019f56bf0780
0000019f`56bf0780  “C:\long\long\long\long\long\long”
0000019f`56bf07c0  “\long\long\long\long\long\long\l”
0000019f`56bf0800  “ong\long\long\long\long\long\lon”
0000019f`56bf0840  “g\long\long\long\long\long\long\”
0000019f`56bf0880  “long\long\long\long\long\long\lo”
0000019f`56bf08c0  “ng\long\long\long\long\long\long”
0000019f`56bf0900  “\long\long\long\long\long\long\l”
0000019f`56bf0940  “ong\long\long\long\long\long”

From this analysis we worked out the long file path that was being supplied in a mkdir command, and was able to modify it into a permitted size.

Posted in WinDbg | Leave a comment

Case of the Win10 HyperV Fail – Property ‘MaxInternalSize’ does not exist in class ‘Msvm_VirtualHardDiskSettingData’

Trying to copy a huge file out of a HyperVM late at night I naively went and right clicked “Mount” in Windows Explorer on the VHDX file. Unfortunately my VM had multiple checkpoints, which aren’t visible when browsing the VHDX. Realised my mistake then unmounted the VHDX. Unfortunately now the VM stopped booting…

Can’t access disk, opening the VM properties selecting disk and clicking Inspect I get error:

[Window Title]
Virtual Hard Disk Properties

[Main Instruction]
There was a problem with one of the command line parameters. Either ‘DESKTOP-RTTN04O’ could not be found, or ‘C:\Users\Public\Documents\Hyper-V\Virtual hard disks\Win7_31AB2236-D8AC-4CF9-8A75-2DC321590C8C.Avhdx’ is not a valid path.

[Content]
Property ‘MaxInternalSize’ does not exist in class ‘Msvm_VirtualHardDiskSettingData’.

[Close]

This VM had the “parent” and two subsequent checkpoints..

  • Parent = Win7.vhdx
  • 1st checkpoint = Win7_5A0F4E19-3212-4D0A-8A43-83853D468B0B.avhdx
  • 2nd checkpoint = Win7_31AB2236-D8AC-4CF9-8A75-2DC321590C8C.avhdx

To fix this I used the PowerShell cmd Set-VHD

Set-VHD "C:\Users\Public\Documents\Hyper-V\Virtual hard disks\Win7_31AB2236-D8AC-4CF9-8A75-2DC321590C8C.Avhdx" -ParentPath "C:\users\public\documents\Hyper-V\Virtual hard disks\Win7_5A0F4E19-3212-4D0A-8A43-83853D468B0B.avhdx"
Set-VHD "C:\users\public\documents\Hyper-V\Virtual hard disks\Win7_5A0F4E19-3212-4D0A-8A43-83853D468B0B.avhdx" -ParentPath "C:\users\public\documents\Hyper-V\Virtual hard disks\Win7.vhdx"

Unfortunately the final command failed:

Set-VHD : Failed to set new parent for the virtual disk.
There exists ID mismatch between the differencing virtual hard disk and the parent disk.

To resolve this error I added –ignoreidmismatch

Set-VHD "C:\users\public\documents\Hyper-V\Virtual hard disks\Win7_5A0F4E19-3212-4D0A-8A43-83853D468B0B.avhdx" -ParentPath "C:\users\public\documents\Hyper-V\Virtual hard disks\Win7.vhdx" -ignoreidmismatch

I reset my VM hard disk to use the 2nd checkpoint disk file, I received a warning data loss could occur if I continued, but I risked everything and continued anyway.  Clicking “Inspect Disk” now succeeded and my VM now booted fine, and seemed to have the recent data I had added into it…

Posted in Uncategorized | 2 Comments

Faster Where-Object In PowerShell

One of the worst features of PowerShell for me is extremely slow performance in loops, especially using items like Where-Object { $_.Name –eq “value” }

For performance critical loops I use inline C#

Here is an example to search data imported from CSV more quickly, to retrieve the relevant line of data:

$Source = @"

using System;
using System.Management.Automation;
namespace FastSearch
{
    
    public static class Search
    {
        public static object Find(PSObject[] collection, string column, string data)
        {
            foreach(PSObject item in collection)
            {
                if (item.Properties[column].Value.ToString() == data) { return item; }
            }

            return null;
        }
    }
}
"@ 

Add-Type -ReferencedAssemblies $Assem -TypeDefinition $Source -Language CSharp 

$data = Import-Csv "songwriters.csv"

$row = [FastSearch.Search]::Find($data,"Name","Bob Dylan")

In test CSV file to search the CSV data 30k times the C# version took a few minutes, using $data | Where-Object { Name –eq “<name>”} took over 12 hours.

Posted in Performance, PowerShell | Leave a comment

Export Large Number of Users from ConfigMgr to CSV with PowerShell

Had a scenario where I required export of 50k+ users from ConfigMgr to do some analysis.

However PowerShell cmdlet Get-CMUser just hung for an hour or more. Maybe it would eventually come back with a result, I don’t know…

So I tried using WMI query:

$SiteName="<site code>"
$SCCMServer="<configmgr server>"
$SCCMNameSpace="root\sms\site_$SiteName"
$result = Get-WmiObject -namespace $SCCMNameSpace -computer $SCCMServer -query "select * from sms_r_user"
$result | Export-Csv UserIDs.csv -NoTypeInformation

However this also hung for quite some time until eventually failing:

Get-WmiObject : Shutting down
At line:4 char:11
+ $result = Get-WmiObject -namespace $SCCMNameSpace -computer $SCCMServer -query ” …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-WmiObject], ManagementException
    + FullyQualifiedErrorId : GetWMIManagementException,Microsoft.PowerShell.Commands.GetWmiObjectCommand

 

To work around this, did a search filtering by first character of username. In this case I just looked for usernames starting with a-z or 0-9:

$SiteName="<site code>"
$SCCMServer="<configmgr server>"
$SCCMNameSpace="root\sms\site_$SiteName"

$result = @()
ForEach ($letter in "a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","0","1","2","3","4","5","6","7","8","9")
{
    Write-Host "Searching for usernames starting with $letter"
    $result += Get-WmiObject -namespace $SCCMNameSpace -computer $SCCMServer -query "select * from sms_r_user WHERE UserName LIKE '$letter%'"
}

$result | Export-Csv UserIDs.csv -NoTypeInformation

This worked perfectly and extracted all the users into CSV format within a few minutes.

Posted in ConfigMgr | Tagged | Leave a comment

Case of the Automatic Outlook Signature Configuration Failure

An environment was using logon batch file, that called a VBS to automatically configure Outlook signatures. The signature had a different image attached to the end of it dependent on current marketing campaign.

One day it stopped working, manually running the script showed the following error

outlookad.vbs(256, 3) Microsoft VBScript runtime error: Object required: ‘objRoot’

This line had code like this:

for each campaign in objRoot.childNodes

Searching backwards in the code we found objRoot was assigned by loading XML:

Set xmlDoc = CreateObject(“Msxml.DOMDocument”)
xmlDoc.load strLogonServer & “\netlogon\marketingCampaigns.xml”
Set objRoot = xmlDOc.documentElement

Added a quick to see if XML was loading:

Set xmlDoc = CreateObject(“Msxml.DOMDocument”)
if (xmlDoc.load(strLogonServer & “\netlogon\marketingCampaigns.xml”)) Then
    WScript.Echo “XML loaded OK!”
Else
    WScript.Echo “XML Failed to load!”
End If

Set objRoot = xmlDOc.documentElement

The script output “XML Failed to load!”

Checking the XML file we could see it had last modified date of a week ago, and had content like below:

<?xml version=”1.0″?>
<campaigns>
  <campaign>
     <applicableGroup>DL-Australia</applicableGroup>
     <imageFile>NEW-Email_Footer.jpg</imageFile>
     <url>
http://www.contoso.com/?utm_source=EmailSignature&utm_campaign=CWT2017</url>
  </campaign>
</campaigns>

The issue was the XML had unescaped ampersand added into the URL field … replacing & with &amp; fixed the issue.

Posted in Office | Tagged | Leave a comment

Case of the VBS Compilation Error–Invalid Character

Running a VBS script resulted in error Microsoft VBScript compilation error: Invalid character

image

The line of code apparently containing invalid character was:

    Const ForWriting = 2

Opening in a Hex Editor it was revealed the “spaces” before the word const were something else

image

 

Selecting the blank spaces, copying them, pasting into a notepad search/replace didn’t remove them.

Using HexEdit I search/replaced the relevant bytes with 09 (tab)

image

After this some instances of C2 or A0 were on their own still causing the error, these I replaced with 0x20 (space)

image

After this the script ran fine…

Posted in Uncategorized | Leave a comment

Decompress Exchange Offline Address Book LZX Files to OAB format

Program binaries can be downloaded here https://1drv.ms/u/s!AiFhB4fT6aiTgdwrR_OOsORLs2SaUQ

Visual Studio 2017 C++ project and source code available here

https://github.com/chentiangemalc/LzxToOab

In further blog post will explain how to extract address information from OAB file.

For usage information run LzxToOab.exe without parameters:

image

Posted in Uncategorized | Tagged | Leave a comment