Survivor's bias

imageDo you read business blogs in which the author repeatedly describes his failures?
No, because you want to adopt the successful experience, and not to learn about experience from a guy who is not successful.

However, the fact that you have been studying only the experience of successful people could present a big problem than it seems.

Here are some stories that will help you understand the seriousness of this problem.

Marks from the bullets

During the Second World War the British bombers were sent daily to bomb the Germans. Most of the planes did not return home, and those that came back were covered with marks from the bullets of German anti-aircraft guns and the fighter planes.

Wanting to increase the likelihood of returning crews home the British engineers have studied the location of the marks of bullets. Engineers believed that in those places where the most hits, it is necessary additionally to armor the aircraft. Of course, the regularity was found: a lot of marks were on the wings, tail and near the tail gun. On the cockpit and fuel tanks were few marks.
It is logical to conclude that you need to add armor to the places where the most marks. But this is wrong.

Planes with bullets in the cockpit and fuel tanks were not able to return home, and on the returning aircrafts the marks from bullets were found just in the sufficiently reinforced places. The important information was in shot down aircrafts and not that just have returned on the base.

That is the literal example of “survivor’s bias” - to draw conclusions based on data that is available or convenient, thereby systematically distort the received results.

Do not the majority of recommendations suffer in terms of business in the same way? You read about the successes, but what about those businesses that could not “go home?” ,Perhaps, as with shot down aircrafts, the failures contain more useful information than the successes.

Hiding other data

Scientific journals like to publish outstanding results. Therefore, the results of studies which have not leaded to the statistically significant results are not published. These results are thrown off or quietly shelved in the archive.
This practice is called the “filing effect” and this is particularly insidious form of survivor’s bias, because everything is transparent. This is well said by Peter Norvig:
When a published paper, said “statistically, this is possible in one case from twenty,” it is likely that these experiments were conducted twenty times, but the results of other studies simply have not been published.

Pharmaceutical companies use the effect of filing in order to deliberately distort the results in the desired direction. The problem became so acute that the magazines have called for a public database of research in order to prevent fraud:
For example, over 66% of studies on the effect of antidepressants in depressed children showed that the drugs are no better than sugar syrup, but the companies only published the results of studies with positive results.

If all the studies have been recorded, then the doctors would know that positive results are only part of all received.

Washington Post

Do not the majority of recommendations suffer in terms of business in the same way? Harvard Business School publishes studies only the successful businesses. Let us paraphrase Peter, but what if twenty other coffee shops had the same idea, the same choice of goods, and the same tenacity as at Starbucks, but they have not achieved success. What can information affect that what we know about the success of Starbucks?

Experimental evidence for extrasensory perception

Dr. Joseph Rhine used the rigor of experimental psychology to the study of paranormal phenomena in general and extrasensory perception in particular. He created a furor in the 1930s for his experiments to test people's ability to predict the order of the cards in a shuffled deck of Zener (cards with images of the circle, square, star and wavy lines).

In a typical experiment 500 people were tested on “strong telepathic abilities”. It was believed that human has this ability; if he guessed the order of the cards in 25-card deck is much more often than usual. Selected in the first stage were tested again and again part of them was eliminated. Let us suppose after the third test remained one person. It was believed that the actions of this man's statistically confirmed the presence of his extra-sensory perception.

In order to show that this conclusion is just another form of survivor’s bias, let us consider the following experiment. We believe that some people are “Eagles”. If they toss a coin, it fall “heads” more often than it should be on the theory of probability. So, we put a thousand people in the room and ask each toss a coin 10 times. Most likely, any Margaret throws ten “eagles” in a row! The probability that she will throw ten “eagles” in a row is 1 in 1024, so we decide that Margaret has extraordinary abilities.

In fact, in the last statement is all true, but it is confusing. The probability that Margaret will throw ten “eagles” in a row is 1 in 1024, but then we did another experiment. We asked a thousand of people toss the coins and we “found” Margaret among them.

The likelihood that someone in the crowd of thousands of people will throw ten eagles in a row, is as much as 62%! Due to the fact that so many people try to do the feat, usually some incredible things happen. Abilities of Margaret have nothing to do with it!

Specific examples of survivor’s bias in the business advisers

Up to this point we asked rhetorically whether it is possible that advices in business may be subject to serious survivor’s bias. Steven Levitt, co-author of free economy, investigated this question specifically.

He read “Good to Great” by Jim Collins, a book in which were investigated 11companies. These companies were from the mediocre that are in the tail, they became a sensation in the stock market. It was concluded that all these companies are inherent the “culture of discipline”. It has sold millions of copies of this book; therefore, it is a good example of books on the business advice.

One of the eleven “excellent” companies had been Fannie Mae and Steven Levitt was reading a book at the very moment when the company was dying under the influence of the financial crisis. He wondered, how are the rest of “excellent” companies?

It turned out that not very good. Fannie Mae actually was not the only example of complete collapse. The book is praised and now bankrupt company Circuit City.

Why do these companies have not continued to thrive? It turns out; Jim began preparing to write a book with a list of 1435 companies and then chose from the list eleven companies. This is nothing like the one described an experiment to prove the presence of extra-sensory perception!

In addition, Jim did not bother figuring out whether the remaining 1424 companies practice the “culture of discipline”. Maybe this is something which involved all the companies regardless of the success achieved.

Maybe this book is an exception to the rule. No, Steven studied another book from the 1980s – “In Search of Excellence” and found the same thing.

Steven came to the conclusion to which we sum up:
These books are about the business mostly retrospective: what did the company in order to be successful? The future is always difficult to predict, and understanding the past is valuable, but on the other hand, the implicit message in these books is that the principles which used data from the company, not only brought them success in the past, but it will succeed in the future.

The conclusion, as it turns out is untruth, so it also put under a question the original premise of these books, is not it?

So how do we make certain that some advice on the business do not suffer from the described bias? Of course, no way. But, nevertheless, the fact that we know about the existence of survivor’s bias means we are less likely to fall under the influence of this bias. So reading this article will bring some benefit.
In addition, let us give preference to the advice that will make us think and respond to themselves to the uncomfortable questions. Let us use the advices as an amplifier of our own thoughts, let’s do not take them for granted as the sermons.
Tags: advice, Bias
Pirat 30 august 2011, 12:41
Vote for this post
Bring it to the Main Page


Leave a Reply

Avaible tags
  • <b>...</b>highlighting important text on the page in bold
  • <i>..</i>highlighting important text on the page in italic
  • <u>...</u>allocated with tag <u> text shownas underlined
  • <s>...</s>allocated with tag <s> text shown as strikethrough
  • <sup>...</sup>, <sub>...</sub>text in the tag <sup> appears as a superscript, <sub> - subscript
  • <blockquote>...</blockquote>For  highlight citation, use the tag <blockquote>
  • <code lang="lang">...</code>highlighting the program code (supported by bash, cpp, cs, css, xml, html, java, javascript, lisp, lua, php, perl, python, ruby, sql, scala, text)
  • <a href="http://...">...</a>link, specify the desired Internet address in the href attribute
  • <img src="http://..." alt="text" />specify the full path of image in the src attribute