There’s a bit of a buzz around about “behavioural economics”, and a new book out “Predictably Irrational: The Hidden Forces That Shape Our Decisions”.

Amazon’s recommender system seems to think it is part of a genre, including (of course)

• Freakonomics, and
• Nudge: Improving Decisions About Health, Wealth, and Happiness
• Sway: The Irresistible Pull of Irrational Behavior

For more on the “Predictably Irrational” , you can look at Dan Ariely’s blog and and the somewhat interesting concept of “customer revenge”

Barnes and Noble have a few reviews (not all positive) but possibly worth reading at http://search.barnesandnoble.com/Predictably-Irrational/Dan-Ariely/e/9780061353239#TABS

However, I am interested more in the implications for research, modelling and marketing.

So I turned to the excellent whitepaper from Diamond Information and Analytics Services
Predictably Irrational Customers: Optimizing Choices for How People Really Buy, Not How We Think They Buy.

That is a marketing oriented paper, and concentrates on on-line marketing. There are some useful facts there, and a thoughtful investigation of the principles of “behavioral economics” and their implications.

Much of this is of interest to me because of my belief that we are, in practice, in research and statistical modelling, a long way from building models that reflect the reality of decision making processes.

See, for example, my recent post “A web influenced decision process” and on a somewhat similar theme (roughly summarizable as “how much of observed behaviour is readily predictable?”) “Explaining Sales Differences” and “Limits To Influence”.

The irrational pull of “free” : do we behave rationally “at the margin” ?
Does Irrationality Matter?

It is easy enough to demonstrate in the lab - as Ariely has done - that “free” is overvalued by us all.

So?

Does it matter?

If we are conducting a brand-price tradeoff study to fine tune existing pricing strategy and estimate cross-price sensitivities, we are hardly likely to offer a “free” alternative.

Or are we? If we did, somewhere in the design, embed an alternative which is free (or very close to), the data point might well be illuminating .. but we would certainly have trouble developing a model which encompassed that and the “normal range” of variation. The design of a choice modeling exercise itself involves tradeoffs.. where, over the utility surface do we want greatest precision? Almost certainly not at the zero price point.

Suppose we were interested in researching and modelling a “buy one, get one free” offer

Well, we already KNOW, or presume it to be probable, that “free” has an irrational value.. otherwise we would not be researching it. The research design for this could and would be quite complex, but the fact remains that we do NOT have to assume customer rationality in order to model choice.

Indeed it is arguable that interaction effects (the dependence of the effect of one variable on the level of another variable) are indications of “irrationality”. That means that the utility surface is not strictly linear and additive, hence, arguably, irrational.

One could go further and argue that “brand value” is an indicator of irrationality .. if I “would not buy brand XYZ at ANY price”, am I not being irrational?

So, irrationality comes as no great surprise to practitioners. Indeed, it is the expected ..“Though this be madness, yet there is method in it.”

That is why research and modelling needs to go beyond the simplistic.

Some Interesting Points from the DiamondAnalytics Paper

Traditional economics suggests that
consumers make rational, consistent
choices to obtain products or services that
best meet their needs at the least possible
cost. This suggests that presenting consumers
with a wide range of choices increases
their chances of obtaining this goal. In reality,
however, too much choice can become
a barrier in the consumer’s decision-making
process, along with other obstacles such as
increased complexity, uncertainty, and limited
information around a selection’s future value.

OK, this is common sense and is informed/validated by some quoted studies.

In a research/choice modelling context this translates to the still under-addressed question of how many items should be in a choice set, and on how many dimensions should they differ. My expectation is that confusion/ “purchase paralysis” arising from too many alternatives would lead to more “random” choice, a poorer model .. but this is still an open and serious question.

Later sections in the paper address the practical issues of “choice filtering”, meaning reducing the number of alternatives shown to a given online customer, with the choices specific to that customer .. there are numerous ways of implementing this (collaborative filtering, self segmentation etc)

And there is a good discussion on “framing” - points of reference whereby customers can understand the relative value of a product compared to alternatives.

2. Process uncertainty. Customers may
be dissuaded from completing a transaction
if a Web site does not clearly help them
place an order, determine when they will
receive it, understand when and how
much they will be billed, or feel that their
information is secure.

Much of this lies in the realms of common sense and good design. Web designs are researchable, if need be. Customers abandoning a transaction for any of the quoted reasons would, it seems to me, to be acting rationally.

The paper elaborates on risk.. process risk, price risk, satisfaction risk : all worth study imho.

3. Faulty discounting. If a Web site overemphasizes
the long-term benefits of
a product relative to its short-term appeal,
consumers might be dissuaded from
purchasing because they tend to incorrectly
and inconsistently discount future
benefits. Consumers value current over future
consumption, even when that future
consumption is considerably more valuable.

The example given in the paper is a familiar one .. a laboratory experiment of the form “would you rather have $1 today or $2 in 1 week” : well, people apply a high discount rate, higher than would be “rational”. One could probably fine tune the estimate of the discount rate in a realistic situation, through some smart research.

Well, there is much more in the paper, and I commend it.

Perhaps read it in conjunction with “Predictably Irrational” and, along the way, contemplate how well your research designs and models capture the knowable “irrational” phenomena, and how much they are still the prisoner of “rational” unimaginative linear utility functions.

For those of you interested in forecasting by “novel” methods (including particle swarm optimization), here is a competition to watch or indeed participate in .. I doubt I will have the time to do anything serious, but I have registered to get the data and I will have a look at it.

It is the 2008 Time Series Forecasting Competition for Computational Intelligence

The data is interesting .. daily cash withdrawals over two years from 111 ATM machines across England. It contains true zeros, and missing data (although I am unclear as to whether this is missing at random, or injected missingness, or how long the runs of missingness are), and by inspection it is very spiky. Quite why it should be so spiky I do not know .. some payday and local effects I suspect .. and I do not now what the units of observation are (pounds, thousands of pounds?). Perhaps one should treat this as discrete count data and fit Poisson hidden Markov models?

An idea of the thrust of the competition can be gained from a listing of the “acceptable” methods

* Feed forward Neural Networks (MLP etc.)
* Recurrent Neural Networks (TLRNN, ENN, ec.)
* Fuzzy Predictors
* Decision & Regression Trees
* Particle Swarm Optimisation
* Support Vector Regression (SVR)
* Evolutionary & Genetic Algorithms
* Composite & Hybrid approaches
* Others