Here's a nice article that was in The Atlantic last month--I think its message is relevant to our discussions here:
Pushing yourself to listen to contrary opinions is the way to make better judgments.
In 1906, the British statistician and polymath Francis Galton attended a country fair at which the attendees were invited to estimate the weight of an ox. Out of curiosity, Galton borrowed the cards on which the guesses were written, took them home, and ran the numbers. To his surprise, the average of the 787 entries turned out to be almost exactly the precise weight of the animal. It was more accurate than even the individual estimates of butchers and farmers, who presumably had an eye for such things.
Galton published his findings in Nature, thus establishing the study of collective intelligence: the principle that groups of people can be smarter than even the smartest, most-expert individuals among them. Since then, research in multiple fields has consistently demonstrated that aggregating a variety of judgments frequently and sometimes dramatically increases accuracy, as individual errors cancel themselves out. Nearly 100 years after Galton attended the fair, a best-selling book popularized his idea as "the wisdom of crowds." Crowdsourced judgments have been used to improve medical diagnoses, scientific research, and economic forecasts.
Of course, crowds are far from infallible. Wise probably wouldn't be the word we'd use to describe the throng of amateur investors who recently inflated the price of GameStop shares. Stock-market bubbles contravene a crucial condition of a smart crowd: that there should be a diversity of judgments. When a group herds toward one answer, it effectively becomes like one very fallible individual, rather than a sum of different guesses made independently. The wisest crowds are composed of individuals who disagree with one another.
Rather remarkably, the logic of the wisdom of crowds seems to apply to individuals too. A quarter-century after Galton published his paper, the experimental psychologist John Ridley Stroop directed research subjects to handle a group of identical-looking objects and rank them by weight. Then he asked them to come back and do the same again. Some of them returned up to 50 times. (Perhaps they came to see it as a workout regimen.)
Because the objects looked identical, returning participants couldn't remember their previous rankings, so each new estimate was independent of the previous one.
Stroop found that the average of the group was more accurate than individual estimates, as Galton would have predicted, but he also discovered something new: The same person's average guess was better than each of his or her individual guesses. Since then, a body of research has established that individuals can boost the accuracy of judgments by aggregating their own different guesses. Each of us has an inner crowd, too, with a wisdom of its own.
This raises the question of how a wise inner crowd can be cultivated. Psychologists have investigated various methods. One, following Stroop, is to harness the power of forgetting. Reassuringly for those of us who are prone to forgetting, people with poor working memories have been shown to have a wiser inner crowd; their guesses are more independent of one another, so they end up with a more diverse set of estimates and a more accurate average. The same effect has been achieved by spacing the guesses out in time.
More sophisticated methods harness the mind's ability to inhabit different perspectives and look at a problem from more than one angle. People generate more diverse estimates when prompted to base their second or third guess on alternative assumptions; one effective technique is simply asking people to "consider the opposite" before giving a new answer. A fascinating recent study in this vein harnesses the power of disagreement itself. A pair of Dutch psychologists, Philippe Van de Calseyde and Emir Efendić, asked people a series of questions with numerical answers, such as the percentage of the world's airports located in the U.S.. Then they asked participants to think of someone in their life with whom they often disagreed-that uncle with whom they always argue about politics-and to imagine what that person would guess.
The respondents came up with second estimates that were strikingly different from their first estimate, producing a much more accurate inner crowd. The same didn't apply when they were asked to imagine how someone they usually agree with would answer the question, which suggests that the secret is to incorporate the perspectives of people who think differently from us. That the respondents hadn't discussed that particular question with their disagreeable uncle did not matter. Just the act of thinking about someone with whom they argued a lot was enough to jog them out of habitual assumptions.
To generate quantifiable results, these experiments are typically carried out on questions with single numerical answers (the U.S. has about 33 percent of the world's airports, by the way), but crowd judgments have been successfully used in many contexts, including the forecasts of climate-related events, the prediction of sports outcomes, and the identification of phishing websites. This implies something fundamental about the practice of making any judgment that hinges on a long list of uncertainties: The more diverse and multivoiced we can make our inner dialogue, the better our thinking will be. The route to wisdom is to internalize and integrate a rich variety of contrasting viewpoints.
The University of Pennsylvania psychology professor Philip Tetlock has conducted extensive research on the skill of prediction and has identified a class of people he calls "superforecasters": individuals who are consistently better at making predictions about world events than the public, and even than the experts in relevant fields.
Superforecasters do not have some magical gift inaccessible to the rest of us, but they have trained themselves, according to Tetlock and his co-author, Dan Gardner, to "constantly look for other views they can synthesize into their own." Tetlock compares this synthesis to the way your brain combines different images from two eyes to make an accurate picture of the world.
Shrewd decision makers understand this principle intuitively. It was one reason Abraham Lincoln assembled a "team of rivals" to fill his Cabinet. The historian Doris Kearns Goodwin says that Lincoln broke with the convention that "you should appoint people who think the way you do." He picked politicians with different party affiliations and ideological backgrounds in part so that "he had access to a wide range of opinions, which he realized would sharpen his own thinking."
Not all leaders share Lincoln's intellectual self-discipline, and even those who do have to make a concerted effort to elicit different viewpoints. In most companies, staff are wary of contradicting the boss. Reed Hastings, who founded Netflix, has said that his biggest mistake in business-a hastily reversed decision to split the company into two operations: one for DVDs, one for streaming-resulted in part from his lack of access to dissenting views. Only afterward did he find out that many of his employees secretly thought his brilliant idea was terrible.
Modern work culture puts such a premium on being a good team player that a crucial ingredient of good teams gets overlooked: open disagreement. "The culture at Netflix had been sending the message to our people that, despite all our talk about candor, differences of opinion were not always welcome," Hastings writes. "We now say that it is unacceptable and unproductive when you disagree with an idea and do not express that disagreement." For big decisions, Netflix executives are expected to actively seek out dissenting views from across the company before moving forward. They aren't required to act on anyone's advice-simply to hear it. It serves only to improve the decision maker's thinking.
Whether we're captains of industry or just of our own ship, we are all smarter when we have a more diverse inner crowd, which we can achieve by cultivating a cognitively diverse outer crowd. But that involves engaging with people we disagree with, an activity that, in truth, most people do not do naturally. We find it stressful, as demonstrated by the endless toxic arguments played out on social media. Consequently, most of us are tempted to restrict our discussion of contentious issues to people we know will already agree with us, and to shut out those who think differently. That's worrisome, because the evidence suggests that it will only dull our thinking-as individuals, as teams, as a society. Here's my worry: What if, one day, Van de Calseyde and Efendić ask people to think of someone with whom they frequently disagree, and the respondents come up blank?
I am not an expert in this area on the wisdom vs madness of crowds but my understanding is that crowds are only wiser when each individual makes a decision that is totally independent of the others. So, if one wants a crowd to determine how many jelly beans are in a jar, it is best to have everyone submit an estimate without knowing what the others have done. On the other hand, one gets into the madness of crowds when decisions are NOT independent and one person says something that influences others until there is a big crowd saying the same thing (e.g. buy a certain stock that has no intrinsic value). This is important in diagnosis when one may get second or multiple opinions. If everyone looks at what the first diagnosis is and who made it (perhaps a really well respected physicians), there is more likely group think (especially if people are rushing) than if other opinions are made truly independently (hard to do in practice as prior diagnosis and clinical reasoning noted in the records).
NOTICE TO RECIPIENT: If you are not the intended recipient of this e-mail, you are prohibited from sharing, copying, or otherwise using or disclosing its contents. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and permanently delete this e-mail and any attachments without reading, forwarding or saving them. v.173.295 Thank you.
Yes, Michael is absolutely right--and I think they make the point in the article also, that each decision must be made totally independently. Whether it is by different people or the same person on multiple occasions, it must be an independent judgement.
Otherwise you are getting into alliterative errors--in Radiology it has been described as "satisfaction of report" when a study containing an error is over-read by a second radiologist. The second (reviewing) radiologist can be biased by the respect they have for the first interpreting radiologist, and tends to repeat their error rather than correcting it. For double-reading to catch & correct errors, both reads must be completely independent and unaware of each other.
All the best,
Michael A. Bruno, M.D., M.S., F.A.C.R. Professor of Radiology & Medicine
Vice Chair for Quality & Patient Safety
Chief, Division of Emergency Radiology
Penn State Milton S. Hershey Medical Center ( (717) 531-8703 | 6 (717) 531-5737
This post seems to have gotten off track when focused on crowdsourcing. What I read here is that it recommends the decision maker (clinician) to open one's analysis to consider different points of view. And then reconstruct your conclusions recognizing the alternatives.
This is the bedrock of Critical Thinking and has been employed by "experts" and successful leaders in any field for years. They are often the first in line to critique their own observations.
As I see things, the medical industry seems to demand an answer from a clinician, rather than the right process leading to a good decision. Hence the difficulty with Shared Decision Making.
Dr. Bruno has done a great favor by bringing this to the forefront. It is time for the industry to change the medical diagnosis process paradigm by altering the decision making process (in less than straight forward cases) and demand that individual clinicians show consideration of the broader spectrum of reasoning and possibilities. My observation is that the industry leaders already do this; and it needs to be highlighted to become a routine process.