I look after end-user computing devices – the laptops, desktops, mobile devices across corporate environments. One common theme for anyone providing IT support in this area should know something already: You really need to know how to lay on the evidence when something is not a desktop issue. I’ve heard many desktop support staff complain about this over the years. My view is get over it, that’s life, it happens. Learn to use tools that can find the root cause rapidly.
In this case we were looking at call logging application, used across multiple capital cities in Australia – Perth & Sydney. The users in Perth reported poor performance.
Their description of the issue, in my mind, straight away sounded like an application performance issue:
- The problem is intermittent
- When it occurs, everybody seems to be affected
- Opening pages goes from <1 second to 5 seconds or more
- Issue occurred on XP+IE7, XP+IE8 and Win7+IE8 machines
- When Sydney goes offline, our performance improves
However application support had responded:
We’ve checked the performance of our application we can’t find any issues.
And network team responded:
We’ve checked for packet drops and didn’t find any.
Onsite support in Perth had been kind enough to capture Wireshark logs of the issue. However there were no obvious network issues there – and as the traffic was HTTPS (i.e. encrypted) we couldn’t really get any performance data out of it. Note if it had been HTTP traffic (non-encrypted) we could have used Microsoft Network Monitor to convert the WireShark log to .CAP format, then process using the Virtual Roundtrip Analyzer http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=21462
It’s at this time application support came up with another idea:
Did you delete your web cache? We recommend clearing your cache once daily.
Let me translate this:
I have no idea what is causing this problem so here is some random thing you can do to make you feel better.
To me this was a bit illogical primarily because I would think clearing cache would introduce performance problems as data would have to be re-cached. If pages shouldn’t be cached the application should be telling the browser no-cache
It was evidence collecting time, bring on Fiddler (http://www.fiddler2.com) king of web-debugging proxies.
However we got the following error message when running Fiddler
15:11:40:3771 !WARNING: Fiddler has detected that system policy ProxySettingsPerUser = 0. Unless run as Administrator, Fiddler may not be able to capture traffic from Internet Explorer and other programs.”
This is because the machine had IE7, and IE7 by default used per-machine proxy settings. (http://support.microsoft.com/kb/555850) This can be changed either through group policy or setting the following registry key:
Value Name: ProxySettingsPerUser
Data Type: REG_DWORD (DWORD Value)
Value Data: (0 = whole machine, 1 = per user)
Then we enabled Decrypt HTTPS option in Fiddler and started capturing the traffic. What we found were examples like this:
So we saw two things:
- 7 seconds from Server Got Request to Server Begin Response, this is in line with the type of delay being experienced
- The page is set to no-cache, so it doesn’t get cached. So clearing cache does absolutely nothing.
Sending the Fiddler reports to application support we then received the message:
We’ve engaged the vendor and it seems there are some performance issues related to heavy customization of product. We are working with vendor to find a resolution.