Australia not so long ago had its latest 4 yearly Census, and I completed it online (by e-Census they really mean Web Census, not -for instance- surveys by email, of which we have more to say elsewhere).

I was impressed with the professionalism. An excellent use of javascript and DHTML (which meant that redundant questions were not asked, and help appeared on demand), although as far as I could figure out they were not using XMLHTTPRequest technology, aka AJAX, for background transfers. And it seems as if they did not cater for javascript disabled browsers which is maybe OK in the context (they are the Census organization after all, and can force you to do it on paper if need be).

Technologically, full marks (reportedly, at a cost of a $9million contract with IBM). Lots to be learned from this about elegant information hiding for web surveys.

There is also another issue. The form was re-organized, such that the web form did not look much like the paper form and required quite a different mindset.

The web form asked the questions on a per-person basis. That is, it re-presented all the (relevant) questions for each person. The paper form is a grid .. persons across the top, questions down the side. The web form is clearly easier to use, and the paper form could have been redesigned to match that flow of questioning.

OK, they did a good job. It was easy to use, I have little doubt that the e-responses will be of high quality (higher than paper, fewer logical inconsistencies..). I suspect that most e-respondents felt that it saved them time.

Having a high profile organization such as the Australian Bureau of Statistics endorse e-datacollection is surely of some comfort to commercial e-survey service providers. Maybe if people get used to doing official e-surveys, they will enjoy and do a conscientious job of answering commercial e-surveys. Maybe.

And certainly the commercial e-surveys can be sexed up .. that is, use the javascript/DHTML/information hiding techniques exemplified by the ABS e-Census.

But the Census is the Census, and it has some legislative force behind it. Unless you are particularly bloody minded, you are unlikely to be overly cavalier about answering the questions, no matter what your private opinion of the quality/relevance/fatuousness of a particular question (or of the entire study) may be.

But a web survey from some private supplier?

Do people take it seriously?

Will they start answering randomly?

just move that mouse and click on anything?

So, let’s get some smarts about this.

Let’s assume that in the absence of face-to-face interaction (umm, interviewers?) , in the absence of the fear of legislated penalty, people are going to be pretty careless about how they answer. Probably not going to give too much of a damn if they do the right thing, are going to get bored pretty quickly, gunna want to move right on through to the end.

Sure, you can do smart things to make the process more enjoyable and better tailored to the particular e-respondent and his/her particular circumstances.

But that won’t eliminate the cavalier approach, and it won’t get us away from the fact that web based surveying is likely to yield lower quality data than more expensive data collection methods.

Web surveys are here to stay (the cost advantage, the immediacy) so we had better start think about dealing with data that is a lot dirtier than we would like.. samples that are as not nearly as representative as we would like, question responses that show clear evidence of fatigue and “could not care” effects.

How much can we trust web surveys? Seems to me that is the critical question.

• Obviously they are methodologically flawed.
• Obviously we need to do everything we can (javascript tricks to reduce load and boredom) to make the user experience better, and quality of answering less random
• Obviously we need to randomize the order of presentation of questions where we can
• Obviously we need to do what we can to get a well constructed stratified random sample, albeit of web users
• Obviously we need to use whatever data we have to help us weight the survey results to generalize to the population at large, not just the web user (or paid survey respondents) population

At the end of the day, are you going to trust this data and the analysis that flows from it?

Well, yes, to a degree.

All samples are bad.

All questions are silly/fatuous/unrealistic/semantically flawed.

All analyses suffer from unexplicated data dredging.

All reported results are colored by a desire to please.

There is no justice and no truth. Insisting on (excessive) rigour in data collection procedures diverts resources from rigour and creativity in framing the enquiry, and from time and mental energy in exploring the dataset and its implications.

Bad data, dumb questions, iffy samples : we are stuck with them.

So? So what? Just more sources of uncertainty.

Bad data is better than no data, creative analyses of awfully flawed datasets can still yield insights.

So, if we have to deal with “web surveys”, let’s make sure that they are conducted as well as possible and then get on with the job of seeing how this fuzzy and flawed data modifies our priors, what lessons and surprises and insights there may be.