OK, I don’t mean precisely that – just being a bit provocative.

But I do mean that

• simulation is often easier than working through theory
• simulation is often cleaner and more transparent.. it forces you to expose your assumptions, because those assumptions are what you have to write the code to
• you can use it in complex situations – for example, as an adjunct to significance testing, to answer the question “how likely is this outcome, or something more extreme, to have occurred by chance”
• it is often more robust, or more amenable to robustifying the conclusions to changes in assumptions
• it can be a LOT safer. I would not, for example, dream of doing a complex choice modelling study without building a simulated population that gave me simulated choices amongst the simulated alternatives, and then analysing those choices to build a model with which I can then simulate .. sort of like doing the whole study before even collecting one scrap of data

Simulation is not only relevant at the design stage, although that is perhaps the easiest way to think about it.. the traditional “how large a sample do I need…” question.

That is actually a very complex question because it depends on the uses to which you intend to put the data, and of that you may be unaware until you get the data.

But if we simulate the outcomes from surveys/experiments of different sizes under different assumptions about the underlying processes and about what it is you wish to measure then yes we can get some idea of what sorts of sample sizes will be reasonably safe given your objectives.

Of course there is a bit of work in this, and the answers may well not be to your liking, but at least the approach gets us away from the forced use of textbook formulae to make statements about “precision” or “power” which are likely to be very poor surrogates for what we are really interested in.

Evidence Based Medicine and the Dice Experiments

To get closer to this and see some real world examples, you might like to visit the Bandolier website at http://www.jr2.ox.ac.uk/bandolier/index.html

Bandolier, “Evidence Based Thinking about Health Care”, is part of the evidence based medicine movement (see also The Cochrane Collaboration or google on “evidence based medicine”) which I highly recommend to anyone with an interest in honest statistics and/or in evaluation of therapies and interventions.

The specific Bandolier article of interest is entitled “Winning the Lottery”, at http://www.jr2.ox.ac.uk/bandolier/band105/b105-1.html.

Specifically the section entitled

“How Much Information Is Enough?”

“While it is relatively easy to demonstrate that inadequate amounts of information can result in erroneous conclusions, the alternative question, how much information we need to avoid erroneous conclusions, is more difficult to answer.”

Read the article, and enjoy. It will give you an insight into the role of simulation in design and in “trust level” formation.