Except Those on Whom Your Lord Has Mercy

Back to the Scene of the Experiment

Five years had passed since the Stanford experiment on which we built the section on biased assimilation in our earlier article. That was the 1979 experiment in which supporters and opponents of the death penalty read the very same dossier, word for word, and came out further apart than they went in. In 1984 two of its three authors, Charles Lord and Mark Lepper, went back to the same scene. With them was a third researcher, Elizabeth Preston. This time they brought a new question: we know the disease, so is there a cure?

They ran the experiment again exactly as before: the same issue, the same two opposing studies. But they split the participants into three groups. The first was left to its own devices, and it reproduced the 1979 result unchanged. The second was given the advice preachers have offered in every age: be objective, be fair, do not let your leanings sway your judgement. The advice achieved nothing. That group came out just as biased as the first. The third group was not preached at. It was handed one specific procedure to apply at every step of the reading: ask yourself whether, if this same study, with this method and this sample, had produced a result on the other side of the issue, you would have judged it the same way. Biased assimilation vanished.

Stay with that difference a while, because the whole article turns on it. Good intentions on their own did nothing. A specific procedure did everything. The mould is not tamed by resolve and sermons. It is tamed by working methods laid in its path to intercept it before it takes hold. That is the promise we made at the end of "And They Will Not Cease to Differ", and this is the time to keep it.

A two-storey stone building with Roman arches and a red tile roof in the main quadrangle at Stanford University — Building 420 at Stanford University, known until 2020 as Jordan Hall, home to the university's psychology department. Photograph by Frank Schulenburg, Wikimedia Commons. Licence: CC BY-SA 4.0.

The First Hope Falls: Knowledge Is Not Enough

Before the prescriptions, one comfortable hope has to be brought down. The reader may be holding it, and perhaps the writer of these lines too. It runs like this: I have read the earlier article and I know my biases by name, so I am in the clear. No. In 2002 Emily Pronin, with Daniel Lin and Lee Ross, the third author of the 1979 experiment, published a study of a phenomenon they named the bias blind spot. They showed participants a set of well-known biases, each with its explanation. Most agreed that such biases are common among people. Each then rated himself as the least affected of anyone. They found something worse still. Teaching a person about a bias can make him more confident that he has escaped it, because he now counts himself, in his own estimation, among those who know.

Two researchers at Yale, the philosopher Tamar Gendler and the cognitive scientist Laurie Santos, gave this false hope a name: the G.I. Joe fallacy, after an American cartoon series that closed its episodes with the slogan "now you know, and knowing is half the battle". In matters of bias, they said, knowing is not half the battle, nor anywhere near it. A man who knows about an optical illusion still sees the two lines as unequal after learning that they are equal, exactly as he saw them before. A physician knows the mechanisms of addiction as a specialist knows them, and that knowledge alone has never saved anyone from the disease. So the cure we are after is not information to be memorised. It is procedures to be practised until they become habit — like a clinical skill, not like a lecture about the skill.

The Key: Motivation Before Procedure

All these procedures have a single key. If it does not turn in the lock, nothing that comes after it works. We went back and read Ziva Kunda's 1990 paper, on which we built the section on motivated reasoning in the earlier article. Its second half, which we did not present at the time, turns out to be a purely therapeutic one. Kunda divides the motives of thinking in two. There is a directional motivation, where a person wants one particular conclusion to prevail, and an accuracy motivation, where he wants to reach the truth whatever face it wears. She documents from the experiments that bias shrinks by a measurable amount whenever accuracy motivation gains the upper hand. The person slows down. He works through the evidence more deeply. He calls to mind arguments from both sides that used to slip out of his memory. Bias is not wiped away, but it withers once the aim changes.

What Philip Tetlock found in his accountability experiments carries the thread further. A person who knew he would have to answer for his opinion before an audience whose leanings he did not know in advance thought more precisely, from more angles, turning the possibilities over. A person who knew his audience's leanings leaned along with them, and his bias only grew tighter. Accuracy wakes up in a man when there is nothing to be gained by flattery.

Here the laboratory meets the oldest thing in our tradition. In the earlier article we settled, with Ibn al-Qayyim, that whim is the rust on the mirror of the heart. If that is so, then the first treatment is to clear the intention, not to sharpen the intellect. We saw there, in Keith Stanovich's figures, that intelligence clears nothing at all. The laboratory called this key accuracy motivation. Our own scholars called it, long ago, stripping the aim down to the pursuit of truth. Of a piece with it is al-Shafi'i's saying, as reported by Abu Nu'aym in Hilyat al-Awliya':

The virtues of al-Shafi'i

"I have never debated anyone without wishing him to be granted success, to be set right and to be helped, and to have God's care and protection over him. And I have never debated anyone without being indifferent to whether God made the truth plain on my tongue or on his."

Consider what that means. Here is a man who set his motive straight before he opened his mouth, so the mould was left with nothing to defend. There is no defeat in debate for one who wants nothing but the truth. We will return to the full sense of this when we come to the scales of revelation.

The Individual's Pharmacy

Once the key sits properly in the lock, here are the prescriptions for the individual. Every one of them has either been tested in a laboratory or borne out in a life:

The first: consider the opposite. You have already seen it stated in the opening scene. It is enough to make it a standing question you put to every piece of news that pleases or grieves you. If this same report, on the same authority, had come out in my opponent's favour, would I have accepted it this easily? And if what I now reject had come out in my favour, would I have rejected it this strictly? In 1980 Asher Koriat and two colleagues found another form of the prescription. Anyone who made himself write out the arguments against his own view — not the ones for it — before settling on a judgement saw his overconfidence drop noticeably. His sense of how right he was came closer to reality.

The second: Darwin's golden rule. In his autobiography Charles Darwin relates that for many years he held himself to a golden rule. Whenever he came across a published fact, an observation, or a thought that ran against his general results, he wrote it down at once, without delay. He said he did this because experience had taught him that such facts and opinions slip out of memory far faster than agreeable ones. We explained the cause of that slippage in "And They Will Not Cease to Differ". The agreeable argument arrives whole, lit up, and lodges in the memory. The opposing one arrives truncated, faded, and quick to evaporate. Darwin knew the cause because he had suffered it, so he made a splint for it out of paper and pen: a notebook in which contrary arguments in particular were written down. Your memory will keep the agreeable ones for you without being asked.

A handwritten page from Darwin's notebook, carrying a sketch of a branching tree headed by the words I think — Page 36 of Charles Darwin's Notebook B (1837), one of the transmutation notebooks. It carries the first sketch of a tree of descent, headed by his words "I think". From Wikimedia Commons — public domain.

The third: do not criticise an opponent until he is satisfied with your account of his position. The economist Bryan Caplan devised a test for this and called it the Ideological Turing Test. Can you state your opponent's position so well that, if he read it, he would take it for the work of one of his own side? If you cannot, you are not arguing with his doctrine. You are arguing with your mould of it. The philosopher Daniel Dennett, taking it from the game theorist Anatol Rapoport, set out the etiquette of criticism in four steps. Restate your opponent's position so clearly and so fairly that he says: I wish I had put it that way. Then name the points where you agree with him. Then say what you have learned from him. Only after that, and not before, may you criticise. All of this simply institutionalises the second half of Ibn al-Haytham's counsel, with which we closed "And They Will Not Cease to Differ": that you suspect your own suspicion as you suspect the text, so that you neither bear down on it unfairly nor let it off lightly.

The fourth: read your opponent in his own book, not from the mouths of his enemies. We have seen that selective exposure brings us the opposing view only in fragments, quoted by the man refuting it. The mould feeds on those fragments and grows fat. The perfect example came earlier, in "And They Will Not Cease to Differ": al-Biruni describing the religion of India from its own books, "without slandering the opponent".

The fifth: the one companion who disagrees. In Solomon Asch's famous experiments on conformity, when everyone else in the room settled on an answer that was plainly wrong, the subject went along with them about a third of the time, while his own eyes told him otherwise. When a single dissenter was seated among them, giving the right answer, compliance collapsed to a small fraction of what it had been. Your mind does not need a majority on its side. One honest voice is enough to break the unanimity of the room. A man who arranges his company and his feeds so that only agreement reaches him has torn the safety valve out of his own mind with his own hand. Keep among your intimates one sensible person who disagrees with you, and honour him. Listen to him as you would listen to a doctor.

The sixth: slow the judgement down and name the alternatives. This prescription comes from the heart of medicine. We mentioned premature closure in "And They Will Not Cease to Differ", and Pat Croskerry proposed a treatment for it that he called cognitive forcing strategies. It is a short pause before the file is closed, in which the doctor asks himself: what else could this be? What does my diagnosis fail to explain? And what is the worst thing I could be wrong about? These questions force the mind to reopen what the mould shut too early. We set out part of how the mind is trained in alternative hypotheses in "The Absent Evidence", and it can be consulted there. The prescription travels from the bedside to every judgement. Name three possible explanations before you settle on the first.

The Pharmacy of Groups

Yet however careful the individual is, he remains blind to his own blind spot. Pronin's study settled that. So a remedy had to come to him from outside: the group must build the opposition into its own structure.

The devil's advocate (advocatus diaboli). The Catholic Church grasped this early, and with remarkable practicality. In 1587 Pope Sixtus V established a formal office in canonisation cases, which people came to know by that name. Its holder's task, by virtue of the office and not by virtue of his opinion, was to attack the file of the candidate for sainthood. He cast doubt on the miracles, hunted for flaws, and pressed the case for refusal as hard as it would bear. No cause passed unless it had withstood an opponent whose whole work was to oppose it. When the reforms of 1983 cut back the adversarial role of the office, the number of those declared saints afterwards multiplied enough to catch the attention of historians. Look at what happened. An ancient institution built Ibn al-Haytham's counsel into the core of its procedure — an appointed opponent for everything under examination — before it had ever heard of Ibn al-Haytham. When it weakened the opponent, the door opened.

Adversarial collaboration. Daniel Kahneman offered scholars an alternative to the war of rebuttals. Do not write a reply to your opponent stacked with your own arguments. Propose instead that the two of you design a single experiment together, on terms both sides accept before its result appears, with an arbiter you agree on between you. Then publish the result under both your names, whatever it turns out to be. Better than the prescription itself is that he applied it to himself. With Gary Klein, his scientific opponent on the question of expert intuition, he wrote a joint paper in 2009 titled "Conditions for Intuitive Expertise". In it the two of them mapped out together where they agreed and where they did not. The reader came away with a map of the truth instead of a battle between two moulds.

The premortem. Gary Klein has another prescription, this one for decision-making committees, and he named it with something close to a medical term. The committee meets before the decision is approved, and its members are told: imagine that we went ahead with this decision, and that a year has now passed and it has failed disastrously. Let each of you write the story of that failure. What happened? The pens then speak the objections that the etiquette of the meeting, or the collective mould, had been silencing. Objecting is now a required task rather than a deviation from the group's enthusiasm. Of the same kind are the red teams set up by armies and agencies: a team whose official job is to think as the enemy thinks and to attack its own side's plan.

And the etiquette of disagreement among us. These structures are nothing new to a tradition that knew formal disputation as a craft with its own etiquette and conditions, the first of them the one we cited from al-Shafi'i: stripping the intention bare. Fairness requires saying, though, that in the ages of sectarian warfare the craft turned into its opposite. Disputations were held for victory, not for truth. What had been a cure became an example of the disease. Structure alone does not protect a man when the motive is corrupt, which is what we set out in the key.

The Dose, and the Conditions for Keeping It

Someone may say: I have understood, so I will throw open every door to those who disagree with me and plunge into their platforms. Not so fast. A medicine has a dose and a route of administration, and the right medicine given the wrong way is poison. In 2018 Christopher Bail and his colleagues ran a large experiment. They paid Twitter users from both American parties to follow, for a full month, an account that reposted the words of leading figures from the opposing camp. What did they find? One group hardened in its positions by a clear, measurable margin. The other did not moderate. Why? Because what reached them was not the opponent's strongest arguments in their calmest form. It was his daily provocations at their loudest. The mould fattens on the provocation of an enemy just as it fattens on the praise of a friend.

The prescription, then, is not a blanket “expose yourself to those who disagree.” It is this: expose yourself to your opponent's strongest arguments in their calmest form, in a well-made book of his own, or with a courteous interlocutor from among the level-headed on his side. What has succeeded in the experiments on reducing polarisation is exactly this quiet, direct kind. In the Broockman and Kalla study (2016), a single calm conversation on the doorstep, built on listening to the story of one human being from the other side, changed the positions of those who took part. The change was measurable, and its effect lasted for months. Not one of the shouting campaigns managed anything like it.

Has Anyone Been Cured?

One question keeps pressing on the reader: these are fine prescriptions, but has anyone been cured by them in a way that can be measured? Yes. Philip Tetlock and his colleagues set up the Good Judgment Project, a series of vast tournaments for forecasting world events — elections, markets, wars. Thousands took part over a span of years. A small minority emerged, whom he called superforecasters. They were right more often than the specialists, and their edge held steady year after year. It was even said that they outdid analysts who had access to classified information they themselves did not have. The researchers then examined what set this minority apart. It was not extraordinary intelligence, nor breadth of reading. We have already cited Stanovich to the effect that intelligence is no protection. What they found instead was a habit of mind they called actively open-minded thinking. A person who has it treats his belief as a hypothesis to be reviewed, not a fortress to be defended. He goes looking for the argument against him. He adjusts his estimates by small amounts whenever something new comes in. And he finds nothing shameful in saying, “I was wrong.” Here is the point for which we told the story: this habit is acquired by practice, exactly as clinical skill is acquired. The ordinary participants improved appreciably once they were trained in these methods. A cure is possible, and there are numbers to show it.

The Scales of Revelation

Before the weighing, we repeat what we settled in our previous article, “And They Will Not Cease to Differ,” so that the aim is not mistaken. Everything we have said concerns the human instrument of reasoning as it receives texts, reports and events. It does not concern revelation itself, nor the fixed principles of the religion. The fault we are treating lies in the mirrors, not in the light.

If we weigh the harvest of the laboratory in the scales of revelation, we find the whole prescription gathered into a single verse: “Say: I exhort you to one thing only — that you stand before God, in twos and singly, and then reflect” [Sabaʾ: 46]. The verse came down about a specific matter: a summons to Quraysh to examine their claim that their companion, may God bless him and grant him peace, was possessed. It is a practical prescription for one particular disputed question, not a piece of wisdom suspended in the air. Look at the order of its parts. First, “before God”: the motive is corrected and personal desire is driven out. Fourteen centuries later the laboratory testified that bias shrinks under this, and gave it a name — the accuracy motivation. Then, “in twos and singly”: withdrawal from the pressure of the massed crowd, where partisan feeling flares up and retreating from an opinion becomes impossible. We have already given Asch's experiments, in which the unanimity of the room carries a person along with an obvious error. Then, “and then reflect”: the act comes last, after its two conditions have been prepared. The verse placed the purifying of the motive and the isolating from pressure ahead of reflection itself. That is precisely what this article has arrived at, from its first line to its last. We stop here, on our rule for keeping such comparisons in check. We do not claim that the verse was revealed to establish twentieth-century psychology. It was revealed as guidance. It is enough for us to say that it set down the order of the cure first, and that the laboratory went on groping its way towards it for centuries.

Then, after all the procedures, a door remains that no procedure opens. We saw in the previous article that the persistence of disagreement is a standing law: “and they will not cease to differ.” But the verse does not stop there. It makes an exception: “except those on whom your Lord has mercy; and for that He created them” [Hud: 119]. Those who come out of the whirlpool of disagreement are the people of His mercy. Guidance in what is disputed is ascribed to Him, glorified be He, not to the intelligence of those who are guided: “so God guided the believers, by His leave, to the truth in what they had differed about” [al-Baqara: 213]. That is why the Prophet, may God bless him and grant him peace, taught us no technique in this matter. What he taught — in the report of Muslim from ʿAʾisha, may God be pleased with her — is that whoever rises for the night prayer should open it with this supplication:

Sahih Muslim

“O God, Lord of Gabriel, Michael and Israfil, Originator of the heavens and the earth, Knower of the unseen and the seen: You judge between Your servants concerning what they differ about. Guide me, by Your leave, to the truth in what is disputed. Indeed, You guide whom You will to a straight path.”

Do you see? The best of creation, the purest of them in heart, the furthest from personal desire, opens his night prayer by asking for guidance in exactly those places where people differ. Who are we, then, to imagine that the laboratory's prescriptions make this door unnecessary? Let the end of the treatment be what its beginning was: the correcting of one's direction. Procedures by day, and supplication by night.

Conclusion: The Boy Treats Himself

We began the previous article with a boy in Tus who noticed that every newborn is poured into the mould of his family. We end this article on the cure with the same boy, now a man treating himself with his own hand. Al-Ghazali did not stop, in al-Munqidh min al-Dalal (Deliverance from Error), at the loosening of the bond of imitation. Doubt on its own is demolition, not healing. After it he laid out for himself a method of examination. He reduced the classes of seekers in his day to four: the theologians, the philosophers, the Batinis and the Sufis. He then approached each class through its own books, not through the books of those refuting it. So exacting was he in setting out the doctrine of the philosophers that he wrote a separate book on it, Maqasid al-Falasifa (The Aims of the Philosophers), before writing his refutation in Tahafut al-Falasifa (The Incoherence of the Philosophers). Indeed, when the Maqasid was translated into Latin stripped of its preface, Latin readers took its author for one of the philosophers, expounding their own doctrine. Consider what he did. He read his opponent in his opponent's own book. He restated that doctrine until his statement of it was indistinguishable from the doctrine of its own adherents. Only then did he criticise. He passed the Ideological Turing Test eight centuries before Turing was born. Nor did he claim that his method had brought him through on its own. At the end of the journey he ascribed his deliverance to “a light that God cast into his breast.” In his life he joined the two ends of this article: a rigorous procedure, and an honest sense of need.

A manuscript page in Arabic in an old naskh hand, the last page of Deliverance from Error — The last page of al-Ghazali's al-Munqidh min al-Dalal, as reproduced at the front of Richard McCarthy's translation Freedom and Fulfillment (Boston, 1980) — Wikimedia Commons — public domain.

From the Stanford laboratory to Darwin's notebook, to the debating halls, to the prayer niche of the night vigil, the conclusion is one. The mould is not broken by unaided resolve, nor by intelligence. It is broken by a procedure placed in its path, by a motive corrected before it, and by a mercy asked for above them both. Our previous article stopped at “and they will not cease to differ.” Let this one stop at the completion of the verse — may God make us and you among them: “except those on whom your Lord has mercy.”

❖

The personal checklist: making decisions and weighing opinions

Set the accuracy motivation (before you begin): am I after support for my existing view, or the right answer wherever it lies, even against my wishes?
Consider the opposite: if this same argument, in this same context, told for the other side, would I accept it as readily?
Darwin's notebook (write the objection down at once): did I record the finding that went against me the moment I met it, before memory could drop it?
Force the mind onto alternatives (cognitive forcing): did I make myself frame three possible explanations or paths before settling on the first diagnosis or option?
The Ideological Turing Test: can I state my opponent's position so accurately and fairly that he would accept it himself, before I set about criticising it?
Control the dose of exposure: do I read my opponent in his calmest sources and soundest books, or meet him on platforms built for mud-slinging and provocation?
The dissenting friend's voice: did I bring into the review one level-headed person who disagrees by temperament, to break the illusion of my own consensus?

The checklist for meetings and running teams

Engineer individual thinking first (isolate the pressure of the group): collect notes and proposals from members in writing, one by one, before open discussion begins. That keeps members from deferring to the majority, or to whoever ranks highest.
Appoint a devil's advocate formally: give one member or one team the defined job of attacking the proposal and hunting for its holes. Objecting then carries no social cost.
Hold a premortem session: assume the plan has failed outright a year after launch. Have every member write the story of that failure and its causes, without reserve.
Design an adversarial collaboration: when a dispute between two departments or two views turns heated, agree in advance on a defined measure and a practical test. Both sides accept it beforehand, and both are bound by its result, whatever it turns out to be.
Open accountability: remind the team that the decision will be judged on how well it anticipated the risks and turned the possibilities over. The judge is a neutral party with no prior preference.
Update in small steps (actively open-minded thinking): treat the plan as a hypothesis, revised and improved in small increments with each new datum or indicator. Do not dig in behind the original position.