Boxes and Arrows: Comments by Karl Groves
Comments by Karl Groves
All excellent points, though it is important for readers to understand that there may be extenuating circumstances which must be addressed for regulatory compliance on. A few years ago, I was E-Commerce Manager at a large credit union in the DC area. Compliance with NCUA (and other) regulations often seemed like a moving target, as credit unions often lag far behind banks in their ability to serve their members electronically. Though there were no firm guidelines at the time regarding cookies and remembering users, this would definitely be on the list of things I’d clarify with my compliance officer.
Jonathan – this would indeed be a response to the article you mentioned. More than that, it is a response to the entire sentiment that any data of value (for usability’s sake) can be gotten from log files. Typically, those who espouse such a practice misunderstand what is found in log files and how it gets there. Things like browser caching, network & proxy caching, and user behavior totally invalidate the statistics.
As I say at the top of the article: “I realize that some analytics software gather data by other means than parsing log files and may in fact contain some features meant to overcome one or more of the criticisms I outline throughout this article. I do not discuss such programs, primarily because there is little consistency between them and ultimately they are just as poor at gathering real usability data as analytic tools which parse log files.”
Regardless of how the data is captured, it is unrealistic to expect to glean usability data from analytics. Without knowing what task a specific user is attempting to do, there’s no way of judging whether they’ve actually done it. Analytics tools only record requests. They can’t record whether that request did or did not satisfy a user’s need – only the user themselves can tell you that.
No, it would not be reasonable to do so. First, how do you know those requests are being performed by actual people and not bots? If you attempt to say that you filter out bots, I will show you one I’ve written which identifies itself successfully as IE. Second, you say this content page is being requested many thousands of times. To most proponents of analytics, this would seem to indicate success, would it not? After all, analytics tools only report on successful requests, right? So if it is successfully being requested, why move it? Last, and always, without knowing why the user is doing something, there’s no way to determine whether the request was successful or not successful. Exactly what brand of analytics tool, through client-side means or server-side, does it tell you the user traveled to that page and then realized it was or was not what they were looking for? Without that information, any decision to modify the site due to analytics is purely driven by conjecture.
In addition, I’d like to see commentary on the ROI of doing things purposely “unusable”. For example, take the credit card swipe machines at point-of-sale in retail establishments. When a user swipes a Visa (or MC) debit card, most of these machines are programmed to ask for a PIN number first. However, there is no option presented to “Process as Credit Card”, only a numeric keypad, and “Enter” & “Cancel” buttons. The user is given the impression that their only choice is to enter their PIN or cancel the transaction and walk out of the store emptyhanded. But, if they hit “Cancel”, they can then process the transaction as a credit card. In fact, I’ve recently observed an invisible “Cancel” button, essentially locking the person into entering their PIN! Where is the ROI on this, you ask? Processing credit card transactions incurs a higher fee for the retailer than processing debit. For major retailers, purposely tricking people into entering their PIN gives them a massive ROI.
Will you be able to supply some data validating the quality of the generated architecture? Perhaps some data which tests the architecture on actual participants and perhaps even an A-B test with an architecture created through other means? If so, this could really be an amazing article.