I think your concern, which I may grossly paraphrase as “it’s all subjective anyway, so if the technique does not do a bit more about the heavy lifting of dealing with the fact that the results are subjective and a function of human interpretation, it is not of much use”, is well put and leads us into some areas where there is indeed some heavy lifting involved .. the nature of subjectivity, and how it may be studied (scientifically, and outside of the psychology context).

Before I get into that, let me say that I think that subjectivity is necessary, inevitable and inescapable.

Suppose we had some (data analysis) technique that produced unambiguous results .. everyone is clear and in agreement on the process, and what the outputs “mean”. If such a thing existed, then the results would be uninformative (unless they were secret, confined to a closed group and could not be replicated readily). What can one do with such a thing? .. the reaction is “ok, I see what this data is” .. followed by an appeal “well, now what do we do?”… and that appeal is usually to a human who is presumably possessed of some special “insights” ( guru, a master of arcana, a witch doctor). The answer from the guru has got to be something along the lines of “Well, I think …..”

Looking back on the data analysis work I have done over the years, I would say that the projects fell into a couple of main categories

1. Deliberate and well designed fishing expeditions.. we had a lot of data or we set out to get some really good data, but without a single express purpose or a single clear hypothesis to be tested : essentially we were ‘looking for insights’ , understanding ..(chance discovery, if you like).

2. Designed experiments or surveys, where there was a single key question and an immediately actionable answer required. I have in mind things like product line rationalization studies (should we kill this product, and if so what will it do to our revenue)

In both cases we were using models and maps and visual representations and examination of outliers ..but I would say that the first category came closest to “chance discovery”.

Did better techniques help in the process? yes, indeed.

Would someone else (in case 1) have come up with the same conclusions?

Well, that is an interesting question. Since in many cases the subjectivity entered into it from my choice of design, my choice of analysis techniques, my beliefs as to what the results meant and what one should therefore DO .. then I guess probably not. Or at least it would be hard to sort out the commonalities between my insights and someone else’s insights.

Getting back to the point about the scientific study of subjectivity, there is a long history of this dating from the work of William Stephenson, his “inverted factor analysis” and Q-methodology. There is a society ISSS the International Society for the Scientific Study of Subjectivity .. some links below, if you are interested.

The work started in psychology, spread to advertising, marketing and politics .. perhaps it could be extended to a serious study of “data analytics” and what determines what analysts find surprising, unusual, noteworthy, and likely to provide “chances” in a dataset.

There is still a long way to go, imho

references:

http://homepages.ihug.co.nz/~sai/Harm_wldview.html
http://facstaff.uww.edu/cottlec/QArchive/science.htm
http://www.imprint.co.uk/online/baars.html
http://www.qmethod.org/

http://en.wikipedia.org/wiki/William_Stephenson_(psychologist)

nice article. I’ve got the book and I am finishing it. I am reading carefully, article after article.

my impression is that the problem of chance discovery is the subjectivity imposed by a human “interpretation” of the data, required by the main idea. In other words, Chance Discovery shows several methods to classify and visualize data, but the human part of the job is still performed by the human being and it’s still subjective.

sounds reasonable to assume that our AI state-of-art can’t infer things or have great insights, but on the other hand, assuming that and leaving the heavy lifting to the human interpretation does not bring too much advance either, IMO.

I have to confess that I am quite disappointed by the book. It seems that except for some really nice articles, the whole theory sounds like something that you would sell for some marketing division, where people would play with a toy to have “insights”.

Maybe, someone can even have an insight using this theory, but then, one will realize that all it does is to show some graphs and that’s it.

It’s science, but I can’t say it’s useful. Maybe, it’s just not enough.

cheers

Kenji Shikida