There seems to have been an upsurge, recently, in interest in “choice modelling”. Have a look at the couple of emails below.

Well, having done dozens of choice modelling studies, I am here to tell you it is not as simple as it might sound. I suppose one might say that of any research or analysis technique, but there are certainly “traps for young players” in this one.

It is definitely NOT the first cab off the rank if you are in a position of significant quantitative ignorance about what drives your market, it is not a magic answer machine, it is – in my humble opinion – the most difficult of all research procedures to get right, the most risky (in terms of failed studies, failure to live up to expectations, downright wrong answers).

It’s a bit like brain surgery .. perhaps it would be prudent to assess the risks and the experience of the practitioner, and do your background research, before proceeding.

Now, there is SOME good news.

I think that if your existing research base is very solid, that you have done all the basics (like a very solid U&A), have developed standardized segmentation procedures, understand how customers view your and competing brands and products and the dimensions that enter into the choice process, if you have analysed available macro market data (sales and price effects) and now have some precise objectives and some very specific areas of trade-off space that you wish to explore, and you are very confident with your chosen consultant, and have pretested the modelling procedures and tried the computer program that will form the decision framework .. then I think you should proceed.

Carefully.

It’s a great technique, the Rolls Royce of research .. under the right circumstances. But I would not use it for a fishing expedition.

Let’s look at these emails.

A conjoint course

OK, a few comments. Firstly, I have no reason to believe that this is/was not a fine course, professionally presented : my comments are generic, not directed at this course or the personnel.

First off, I’d like to know if the presenter had actually designed and conducted such a study in the real world. What looks simple in the lab often does not translate to the real world.. variables/attributes are not so simply defined. For example, how are you going to communicate the different levels of “average bandwidth” to respondents .. is this meaningful to them? Are “maximum bandwidth” and “average bandwidth” correlated in the real world, and if so what steps are going to be taken to reflect this in the design (or are we going to see a design in which an alternative can appear which has an illogical maximum bandwidth lower than the average bandwidth).

Note that the attributes are few in number, and crisp.

This is not usually the case in real life. Nor are attributes “independent” in real life.

To construct choice sets that are realistic and informative (from the modelling perspective ie not all data points are equally useful) requires some special techniques, not just as one will often hear touted “fractional factorial designs”.

I well remember one study which I had the pleasure of designing and analyzing in which price was a computable function of the attributes with some slack .. there simply was no other way to make the design useful or realistic : varying the price independent of attributes was not sensible. And so on.

The ideal product?

Constructing an optimum product from such a model is problematic. What are we optimizing? Market share? .. well, simply provide the cheapest most fully featured product.. essentially, the “ideal”, but a good way of going broke. This needs more thinking through. And care needs to be taken to ensure that the density of data points in the vicinity of the optimum is sufficient to ensure that the model is well behaved in that region.

One can protest that I am being too harsh on what is, after all, a toy example constructed for the purposes of exposition. But that is precisely the point .. it is not easy to scale up from toy to the real world problem that you are most likely to be facing.

Another email

It’s great to see some intellectual firepower being put behind these problems. Theoretical optimum designs is an area of great interest, as is the ability to recover segments from choice patterns (the problem of individual choice models, recently addressed in a Bayesian framework by Rossi et al in their latest and very recent book “Bayesian Statistics in Marketing”(or somesuch) – be warned, it is a heavy read).

And I really like the interest in near-optimal designs ..there is a lot that can be done, possibly extending to assigning balanced supersaturated fractions of a full design to individuals, then using iterative or Bayesian techniques to get better individual choice models, thence to segmentation.

Warning Bells?

BUT! Are warning bells sounding? They should be. If this is an area of active academic research, well really just the beginnings of an academic research program, then maybe it is in truth quite a hard area, and maybe we should not be too ambitious.

It’s fine to be a pioneer (if you like arrows in the back) , but let’s be realistic.

If the heavyweights are still working on it (well, just starting to think about working on it), maybe our applications of choice modelling should be modest, sober, prudent, unambitious, manageable, not over-promising and not overly complex. Above all, perhaps we should strive to avoid hubris, admit that there is no magic formula and use the techniques with caution and common sense.

OK, enough about the emails.

I really do hope you conduct a choice modelling study, because there is great potential there, but I also want you to be aware and informed.

So I put together these desiderata, this list of 42 gotchas and reality checks.

Enjoy.

Oh, and why 42?

Well it is, of course, the magic number – the answer to life, the universe and everything. Just like choice modelling .

42 Questions That a (prospective) Choice Modelling Client Should Ask

Do you know what a multinomial logit is.. and the role of the reference category?

Are you going to get a computer model, what is it going to look like?

Is it reductive –reducing every thing to an engineers formula?

Is it overly ambitious? .. or encouraging you to be so

Is it easily hijacked to answer every question under the sun?

Are you clear what sample sizes are required, particularly with clustering .. but they are large .. how large?

Are you asking too much of the model?

Are you aware that the model can perform poorly in the region of interest?

Choices are not efficient.. why not use judgements?

There are better things to do first – have you done them ?

I have yet to see a successful segmentation based on choices .. have you seen one?

Choice modelling assumes you know all the factors.. do you ?

Are you aware that the design is generally much too big so has to be split across respondents?.. this has implications

Model coefficients are not directly interpretable (eg for importance)

Be sure that the design is ok on quadratic and cross price effects

Conjoint often does pair wise comparison of attributes.

We can easily prove that the results (re importance of factor) are different between choice modelling and direct questioning are different, but which is correct?. And what are the units. How can we meaningfully compare the importance of two factors (eg uptime and bandwidth) which are qualitatively different, and measured on different scales. What does it mean to say that “bandwidth is more important than uptime” unless we reduce them to a common scale (say a percentage scale around a mean of 100).

Are we confident that the model will match known market shares, or that we will have good explanations if not? Do we have in place weighting mechanisms to deal with this (potentially unsettling) discrepancy .. or are we content to treat it as a laboratory experiment.

Are we happy that respondents can deal with choices between 2 complex putative products, with lots of attributes that differ. Have we got a design in place that ensures only a certain number of attributes can differ in any one choice pair?

How would you cope with the choice between two laptops configured as follows

a)	ACER 40gb hard drive, DVD burner, 2gb RAM, 14” screen, 2 hour battery life, $2400, Office included
b)	IBM, 80gb drive, CD/DVD reader, 500mb RAM,12” screen, 3 hour battery life, $1600, Open Office

(ie a mixture of better and worse attributes, and this is a SIMPLE case with only 8 attributes each of which is “crisp”/readily defined)