JM Spate Web Analytics

Archive for April, 2010

Musings on metadata

Apr 15

Posted by Jess in Technical notes | No Comments

Years ago when I was working with earth scientists on a regular basis, a hydrologist asked me to take a look at a couple of his time series. A usually reliable rainfall-runoff model was refusing to calibrate properly, and when forced, producing results that were obviously nonsensical.

The problem was easy enough to spot. Of the two series of daily values, peaks were appearing in the runoff (river level) before the rainfall event was recorded. Conceptual models get upset when things like that happen.

A quick look at the meta data revealed the cause. The data wasn’t necessarily poor or even unrepresentative of real world behaviour, but the measurements were not taken in the same place or by the same person. Rainfall was measured every day at 9am, using a day defined from 0900 to 0859. Runoff was automatically logged at midnight every night, 0000 to 2359. You can see what happened from there.

Could be go back and re-collect a decade of data? No. Could we interpolate one series to fudge the definition of a day? Absolutely. Problem solved and we had a happy and useable conceptual model.

However, it’s worth noting that almost all small rural rainfall measurements are taken 9am to 9am, and almost all runoff values assume the day begins at midnight. The problem exists in many, many more datasets than we applied the correction algorithm to. All the subsequent modelling done with uncorrected rainfall-runoff data could have been improved in accuracy and utility had more people taken the time to interrogate the metadata and see that all was not well.

More recently I’ve noticed a similar issue crop up in web analytics. Instead of differing definitions of what constitutes a daily period, the trick is in the time zones used by various data collection devices and logs. If you’re seeing an inconsistency between two or more datasets or series, it could be something as simple as that.

Particularly in analytics, it might not be that simple (and often won’t be), but the first check you make should always be in the metadata. If you’ve got it, check it, and if you haven’t, consistency problems may well be the least of your worries!

Tags: ,

The significance of significant figures

Apr 7

Posted by Jess in Technical notes | No Comments

At this point in time, the use of significant figures is almost exclusively confined to scientific disciplines. It’s a pity, because these days so many businesses are relying in analytics to provide the basis for some serious decisions and the figures they have are rarely as accurate as they might appear.

The significant figures (or significant digits) from any number are those that contribute meaningfully once uncertainty is taken into account. At DrJess we like to get stuck in and really investigate uncertainty, but the first step is to realise it’s there.

Say Google Analytics says drjess.com/blog site has seen 1621483 unique viewers in the last month. A touch on the generous side, but in thought experiments we are allowed to be optimistic.  Neither the Goog nor almost any other free web analytics suites have much to say on the subject of accuracy, but presenting in any halfway scientific context a number with that many significant figures implies a very high confidence in its accuracy, percentage-wise.

Web analytics pros know that Google Analytics and its tag-based friends tend to under count traffic by rather more than 10%. More on the reasons why in another post, but let’s be generous and assume a maximum 10% error for the sake of a simple thought experiment. Applying that to my imaginary unique user count of somewhere between  1621483 and 1783631.3.

If I had to pick a single number to represent that range, it would be 1700000, not 1621483. Not only is it probably more accurate, it gives anyone looking at it a much better indication of what the uncertainty is.

Now, if I was going to base some serious and potentially very expensive business decisions based on analytics numbers, I’d want to know just how good my figures were. So would most people who want analytics done, if it occurred to them to wonder if precision might be lacking in the first place. With the over-precise numbers spat out by most analytics programs, the question is rarely posed unless inconsistencies crop up.

I’m not blaming Google. The simple truth is that given the choice between a web analytics program that spits out 1700000 and one that spits out 1621483, the overwhelming majority of users will perceive the latter as more accurate.

Legend states that the first surveyor of Mount Everest measured the height at 29000ft exactly. It was reported as 29002ft, supposedly because the surveyor didn’t think anyone would believe his rounder figure had a reasonable degree of precision.

That was back in 1856. Unfortunately it still seems we’re having this kind of problem. The only solution is to state clearly what the uncertainty is. Bring it out of hiding and discuss error, accuracy, and precision before making decisions.

Tags: , ,