Yet another piece of research that points up the difficulty of getting hard numbers on the web.

The problem is of course that users can, and do, delete cookies at will - supposedly about 30% of users delete cookies once a month. See “Cookies will get you fat” at Immeria, for a nice summary of the “research” and what to do about it

# Avoid providing hard numbers; trends and context are essential.

# Use the “visit” as the metric of choice, think in term of “opportunity”.

# The only instance where you should report on “unique users” is when you can use a real unique identifier such as a login.

which last bit I concur with.. the others are perhaps sidestepping the real issue, the underlying massive unreliability of the data, particularly if repeat visitation is something that is relied upon.

For the original report, look at the Comscore press release
and some earlier reports “Cookie Death Small Potatoes” and “Don’t Grieve for Cookies”.

There is even an innovative suggestion “Tacoda Tech Replaces Deleted Cookies” which seems to involve taking a hash (unique summation) of first party cookies to form a unique identifier .. pity that won’t work, because it appears that first and third party cookies are deleted about equally often.

Not surprising .. many people use a smart cookie cleaner .. for example, I use CrapCleaner frequently (but on no fixed schedule).

No such thing as a unique user - maybe a unique combination

And as Immeria most sensibly points out

The study from ComScore doesn’t cover the fact that more and more users are using multiple devices (home & company computer, PDA and other devices) and even multiple browsers (sometimes switching between Firefox and MSIE, both installed on the same computer). So the notion of “unique user” should be retitled to “unique device + browser”

Right. I personally use 2 desktop machines with three browsers on each, plus a laptop, plus ..

So, what does a statistician make of all this?

First of all, be wary, be very wary. This is not good data. We have data that is affected by either “missing at random”ness or “censoring” or both.

And it may not be “just” a matter of bias (inflated figures), it may be a matter of inflated variance. And that affects your significance tests . whether you can conclude that one campaign is better than another, whether there are real trends.

To work this through properly, we would have to postulate a reasonable DGP (data generating process) and run some simulations.

But here is the intuition.

Working with Visits

Suppose that we are running a campaign at time T(1) and have collected visitor/visit data at times T(0) (before the campaign) and T(2) (after the campaign).

At each of these three times we have an accurate measure of the number of visits (H), aka hits, from log files. Now we know that visits is a stochastic quantity, it has a random component due to .. just about anything, the weather, competitive activity, whatever …

And with a long enough prior history we might have a good estimate of the month-to-month variability of H, its variance.

So, we could, in theory, run a significance test of the change in H from T(1) to T(2) and find a weight of evidence for any change (in H(0) to H(2)) being adducable to the campaign rather than just random. OK, it’s a bit more complicated than this, but that is the general idea - what is the probability, the likelihood that the observed change was due to our action compared to it being just a natural outcome of a system that moves up and down and around?

Working With Visitors

But what if we were not satisfied with hit counting? There might be good logical reasons for this .. the idea of the campaign might be to get more “users” not more “uses”.. and if so, it is not much use if all the campaign did was to induce the existing pool of visitors to visit more often.

We want to expand V (the population of visitors) .. note that there is terminological sloppiness here : we really DO want to expand the population of visitors, but at best (even if there was no cookie death) we would be observing only an estimate of V .. think mark-recapture sampling, where you catch some salmon, tag them and throw them back. Then catch some more and see how many have been tagged.

I will continue to talk about V, visitors/month as if it was the real V, for convenience, but the real V is all the salmon, not just those we see.

Here is where it gets interesting. We want to estimate V from H. Now, some of the salmon did not lose their tags so we have an error-free (perhaps) count of those. From the remaining hits, not those caused by the tagged salmon but some of which are caused by previously tagged salmon who had lost their tags or adversarily removed them (the smarter ones?) and some by never tagged salmon, we need to estimate how many salmon caused those hits.

And here is where an extra stochastic parameter enters in. The hits per visitor (HPV) is a random variable, which we know only imperfectly, but which we have to use in order to convert our “left over hits” into visitors. Unfortunately HPV is likely to have a long tail, too.

And there is another unknown - the proportion of salmon who have lost their tags (killed their cookies). We can estimate it (perhaps by using external panel surveys), but it too is a random variate subject to uncertainty.

So, the calculation of the number of unique salmon (um, visitors) from those observed (hits) is one that involves uncertain parameters - hence the variance gets increased, and simple minded significance tests are most probably wrong.

OK, enough fishy stories