The Righteous Mind Challenge
In a recent essay in the evolutionary magazine This View of Life, I analyzed Sam Harris’s Moral Landscape Challenge, in which he offers to pay $10,000 to anyone who can convince him to change his mind and renounce his views. From the perspective of The Righteous Mind, it seems unlikely that anyone who is heavily invested in an idea, and who writes about it with high levels of certainty, can be persuaded to change merely by the force of an essay words written by a stranger. So I offered to pay Harris $10,000 if he changes his mind.
Please read that essay first. This blog entry just gives lots of additional details, particularly about the word-count analyses. I’ll update this entry as people ask me additional questions.
I) How did I choose books to analyze?
I started by analyzing the three big New Atheist books: The God Delusion, The End of Faith, and Breaking the Spell. I then chose 3 recent books that were written by scientists who seem focused on explaining religion, not condemning it. I chose my own book, plus the two books that I knew about which came out shortly after mine, on the origins and psychology of religion: Jesse Bering’s The Belief Instinct, and Ara Norenzayan’s Big Gods. (I also examined some older books, to be confident that I wasn’t cherry picking, and they too were almost all below 1.6, so I just picked the 3 most recent of all the books I analyzed.) To pick the right-wing writers, I chose three of the most prominent, and then went onto Amazon to see which of their books was the most reviewed, which I took as a proxy for most read, or more influential. That led me to Glenn Beck’s Common Sense, Sean Hannity’s Deliver us from Evil, and Ann Coulter’s Treason. But here too, I analyzed a bunch of other books by these and other authors, and found that they typically fell in the middle range, between 1.4 and 1.7. [see below, in section III-B]
II) How did I do the analyses?
I obtained plain text files of all the books (I also bought Kindle versions of all the books, to ensure that the authors would get the royalties they deserve). I checked the text files carefully to make sure there were no issues that would skew the word count, such as headers that repeated on every page. I stripped out all the front-matter before the first word of the main text, and all the end-matter after the last chapter. I kept in introductions and epilogues, but cut out acknowledgments, notes, appendices, and references. I then ran each text file through LIWC, using the built-in 2007 dictionary, outputting the Certain category. Here are all the words that LIWC scores when computing its “certain” score:
absolute, absolutely, accura*, all, altogether, always, apparent, assur*, blatant*, certain*, clear, clearly, commit, commitment*, commits, committ*, complete, completed, completely, completes, confidence, confident, confidently, correct*, defined, definite, definitely, definitive*, directly, distinct*, entire*, essential, ever, every, everybod*, everything*, evident*, exact*, explicit*, extremely, fact, facts, factual*, forever, frankly, fundamental, fundamentalis*, fundamentally, fundamentals, guarant*, implicit*, indeed, inevitab*, infallib*, invariab*, irrefu*, must, mustnt, must’nt, mustn’t, mustve, must’ve, necessar*, never, obvious*, perfect*, positiv*, precis*, proof, prove*, pure*, sure*, total, totally, true, truest, truly, truth*, unambigu*, undeniab*, undoubt*, unquestion*, wholly
III) What are the actual mean scores?
A) Here are the original results I posted.
|
Author |
Title |
LIWC Certain score |
| Harris | The End of Faith | 2.24 |
| Dennett | Breaking the Spell | 1.77 |
| Dawkins | The God Delusion | 1.7 |
| Beck | Common Sense | 1.56 |
| Hannity | Deliver us from Evil | 1.49 |
| Coulter | Treason | 1.49 |
| Bering | Belief Instinct | 1.56 |
| Norenzayan | Big Gods | 1.2 |
| Haidt | Righteous Mind | 1.15 |
Note that there are no standard deviations, and no error bars on the graph, because these numbers are not samples from a larger population. They are the exact measurement done on the total population of words in each book.
B) Analyses of additional books [added on 3/3/14]:
Rahul in the comments asked me to post analyses of some other random works, e.g., from Gutenberg.org, to give us more context within which to interpret the certainty scores I posted. I agreed that this was a good idea, and asked him to pick some texts from Gutenberg, ideally works of relatively modern non-fiction. Rahul obliged, and provided this list. I show each work with its LIWC certainty score in parentheses: Darwin: On the Origin of Species (2.04); United States Presidents’ Inaugural Speeches: From Washington to George W. Bush (2.18); Gandhi: My Experiments with Truth (1.68); Speeches & Letters of Abraham Lincoln (2.05); Albert Einstein: The Meaning of Relativity (1.81); Works of Martin Luther (2.26); G. K. Chesterton “Orthodoxy” (2.85). I was surprised to find a relatively high score for The Origin of Species, which did not fit with my general sense of Darwin’s careful writing. So I added in all of his other books, which turned out to score quite low on certainty: Darwin: Descent of Man (1.43); Darwin: Expression of emotions (1.32); Darwin: Voyage of the Beagle (1.31), so Darwin’s average certainty score across his four books was 1.52 . I do grant that there is variation within each author, depending on the writing task at hand. Ideally, one would analyze multiple books from each author.
To put this all together, I provide below the LIWC certainty scores of all the books I have now analyzed, including the 9 I presented in my original essay; the 7 suggested by Rahul, plus Darwin’s others; the right wing authors I did not show in my original analysis (e.g., Rush Limbaugh, Bill O’Reilly, Michael Savage, and Mark Levin); and two older books on religion by non-new-atheists (David Sloan Wilson and Scott Atran). I sort the table by declining avg score for each author, given that I have more than 1 work for 6 of the authors.
| Work analyzed in LIWC | Avg certainty for each author | Certainty score for each book | |
| chesterton.orthodoxy | 2.85 | 2.85 | |
| HARRIS AVG | 2.3 | ||
| harris.end of faith | 2.24 | ||
| harris.The Moral Landscape | 2.37 | ||
| luther.works of martin luther | 2.26 | 2.26 | |
| us.presidents.inaugural.speeches | 2.18 | 2.18 | |
| lincoln.speeches and letters | 2.05 | 2.05 | |
| einstein.the meaning of relativity | 1.81 | 1.81 | |
| dennett.breaking the spell | 1.77 | 1.77 | |
| dawkins.god delusion | 1.7 | 1.7 | |
| gandhi.my experiments with truth | 1.68 | 1.68 | |
| beck.common sense [right wing] | 1.66 | 1.66 | |
| wilson.darwins cathedral [not a new atheist] | 1.65 | 1.65 | |
| limbaugh.the way things ought to be [right wing] | 1.64 | 1.64 | |
| O’REILLY AVG [right wing] | 1.59 | ||
| oreilly.culture-warrior | 1.59 | ||
| OReilly.whos_looking_out_for_you | 1.6 | ||
| bering.the belief instinct | 1.56 | 1.56 | |
| COULTER AVG [right wing] | 1.56 | ||
| coulter.godless | 1.63 | ||
| coulter.treason | 1.49 | ||
| HANNITY AVG [right wing] | 1.53 | ||
| hannity.deliver us from evil | 1.49 | ||
| hannity.let freedom ring | 1.57 | ||
| DARWIN AVG | 1.52 | ||
| Darwin.descent_of_man | 1.43 | ||
| Darwin.expression_of_emotion | 1.32 | ||
| darwin.on the origin of species | 2.04 | ||
| Darwin.voyage_of_the_beagle | 1.31 | ||
| atran.in gods we trust [not a new atheist] | 1.43 | 1.43 | |
| savage.liberalism is a mental disorder [right wing] | 1.43 | 1.43 | |
| levin.liberty and tyranny [right wing] | 1.41 | 1.41 | |
| HAIDT AVG | 1.25 | ||
| haidt.happiness-hypothesis | 1.36 | ||
| Haidt.Righteous-Mind | 1.15 | ||
| Norenzayan.Big Gods | 1.21 | 1.21 |
IV) Miscellaneous Methodological Issues
A) Some people on Twitter have pointed out a possible confound in the analyses: the LIWC certainty dictionary contains roots like funtamentalis* and fact. If the New Atheists are talking ABOUT religious fundamentalists, or if Harris’s book The Moral Landscape is about values as facts, its unfair to give them points for using those words. I agree, those are false positives. So I will re-run all analyses using a very restricted dictionary, which uses only the most unambiguously dogmatic words, such as “always” and “never.” I propose using this subset of the LIWC dictionary: Absolute, absolutely, always, certainly, definitely, every, inevitab*, must, necessar*, never, obvious*, totally, undeniab*, undoubt*, unquestion*. Please comment below if you think I should cut any of those words, or add back any from the full list given in section 2.
[Text added 3/3/14]: Commenter Bianluca Barbetta, below, suggested cutting “absolute.” So I did that and then re-analyzed the original 9 books (plus Moral Landscape) using the restricted dictionary listed above. The resulting numbers are much lower, of course (because many fewer words are captured), but the basic picture that emerges is little changed. In declining order, the scores are: Harris-End of faith (.47); Beck-Common Sense (.47); Harris-Moral Landscape (.46); Dennett-Breaking The Spell (.39); Dawkins-God Delusion (.38); Hannity-Let Freedom Ring (.37); Coulter-Treason (.36); Bering-Belief instinct (.30); Haidt-Righteous Mind (.21); Norenzayan-Big Gods (.15). These scores show the percentage of all words used in each book that were in the list of words coded by the dictionary. For Harris and Glenn Beck, it’s nearly a half percent of all words.
B) To get an independent check on whether the effects I report are real and robust, I hope somebody will create large text files for each author I chose, composed of, say, 20 blog post available on the internet, and then run LIWC on those files. If someone does that, i”ll post a link to the results here.
C) [added on 2/18/14]: Many of the critical commenters below note that there are many ways of using words like “certain” or “certainly” which don’t indicate anything about the mindset of the author. They are right. LIWC is a simple word count program; it does not analyze words in context, and it does not control for negation (e.g., “it is not certain that…”). So there are many false positives. A LIWC does not by itself prove that the new atheists are more dogmatic than other groups of authors. But it can test one’s subjective impressions; it can add or subtract confidence in one’s impressions. When I read the New Atheist books, Dawkins and Harris sounded angry, whereas Dennett did not. But all three authors seemed to to me to use certainty formulations to an unusual degree. I then ran LIWC to measure both of those categories, and it turned out that the New Atheists were high on both (except for Dennett, who scores low on anger, confirming the impressions of many readers). But then I examined the LIWC output to see the words in context, and it was clear to me that Harris scored high on anger in large part because he is talking about violence and killing related to religion. That’s his subject matter, not his emotion. There were so many of these false positives that I decided it would not be fair or accurate to publish the anger findings. The certainty findings, however, hold up much better. Those word uses did seem to be capturing something about Harris’s prose style. In response to a commenter below, I opened up The End of Faith at random, to page 40, and found this passage:
“The basis of our spirituality surely consists in this: the range of possible human experience far exceeds the ordinary limits of our subjectivity. Clearly, some experiences can utterly transform a person’s vision of the world. Every spiritual tradition rests on the insight that how we use our attention, from moment to moment, largely determines the quality of our lives.”
Surely, clearly, and every? This is not the way most scientists write.
V) Additional Stuff:
A) Here is my entry in the Moral Landscape Challenge, where I say why I think Harris’s claims about morality are wrong.
B) [added on 2/18/14] What will it take for me to pay Harris the $10,000? I will send him a personal check (or donate to his foundation, whichever he prefers) if two conditions are met: 1) Harris pays someone the $10,000 of his own money for writing an essay that changed his views about morality, and 2) this payment is accompanied by an explicit acknowledgment that his “case for a scientific understanding of morality is mistaken,” as he puts it in the challenge itself. Harris and I both agree that a scientific understanding of how morality and moral judgment work, descriptively, is appropriate, and is proceeding well. (We both participated in an Edge.org project on The New Science of Morality). We agree that scientists can study morality just as they can study language, or sexuality, or color vision. The dispute between us arises over whether science can tell us what is in fact morally right and wrong, in the same way that science can tell us facts about the natural world. That is Harris’s most provocative and interesting claim in The Moral Landscape. If he renounces that claim, or in some other way says that his argument in The Moral Landscape was largely wrong, I’ll pay him the money. If he pays the money while admitting only a minor error, or conceding some peripheral points, that would not count as having changed his mind or accepted the refutation of his thesis.
C) [added 3/3/14] It is interesting that in Harris’s recent debate with Dennett over free will, Harris opens his rebuttal with a claim that seems on its face to support my basic argument:
The virtues of rational discourse are everywhere espoused, and yet witnessing someone relinquish a cherished opinion in real time is about as common as seeing a supernova explode overhead. The perpetual stalemate one encounters in public debates is annoying because it is so clearly the product of motivated reasoning, self-deception, and other failures of rationality—and yet we’ve grown to expect it on every topic, no matter how intelligent and well-intentioned the participants.
D) The full opening paragraph from Section I of Hume’s Enquiry, is worth reading:
DISPUTES with men, pertinaciously obstinate in their principles, are, of all others, the most irksome; except, perhaps, those with persons, entirely disingenuous, who really do not believe the opinions they defend, but engage in the controversy, from affectation, from a spirit of opposition, or from a desire of showing wit and ingenuity, superior to the rest of mankind. The same blind adherence to their own arguments is to be expected in both; the same contempt of their antagonists; and the same passionate vehemence, in inforcing sophistry and falsehood. And as reasoning is not the source, whence either disputant derives his tenets; it is in vain to expect, that any logic, which speaks not to the affections, will ever engage him to embrace sounder principles.
(Note that Hume is describing two different kinds of disputants – the pertinaciously obstinate kind, and the disingenuous kind. That is why he says “either disputant” in the last sentence. But since I’m only talking about one kind, I changed it to [such a].)
Note to commenters: You can be as critical as you want of my ideas, but any comments that use obscenity or insults will be deleted.
Max Haidt reads The Righteous Mind
When I wrote The Righteous Mind, I tried to keep the writing clear and simple. I don’ t like non-fiction that is ornate or academic in style. I got some positive feedback on the writing when my son Max, age 6 (almost 7) picked up the book and started reading. His comprehension is pretty close to zero, but still — it’s an easy book to pronounce! Here’s Max reading from p. 4-6 of the paperback, on nativism and empricism. At 43 seconds, Max pronounces “vague” as “vag-yoo,” and my wife laughs in the next room.
Look how far we’ve come… apart
I have an essay in today’s NYT on America’s growing polarization. Marc Hetherington and I show what’s happened over the last 50 years, not just in Congress but among citizens, in 3 simple graphs.The first one is below, from McCarty, Poole, and Rosenthal. It shows that things were really polarized in the late 19th century, after the civil war, then they got much less polarized, but it’s been a rapid climb upwards since the 1960s. We talk about why.
The next 2 graphs were made by Hetherington, and they are just as depressing. Click here to read the essay.
But we end on an optimistic note. There really are so many changes we could make to roll things back, perhaps to the level of polarization in the 1990s, which was much lower than it is today. But only if we push our leaders to make the changes. They will not make the changes themselves, because each change will probably favor one side or the other, so the disadvantaged side will fight like hell. But extreme pressure from outside, for a comprehensive package of reforms — a kind of good-government-Simpson-Bowles — might do it.
—————
Post-script: see Pete Ditto’s call for “domestic realpolitik” to be realistic about how we’re going to get things done.
Read MoreMoral psychology and the campaigns (on “The Cycle”)
I was a guest today on a new MSNBC show, The Cycle, which is interesting because it features one conservative and three liberals discussing the days issues in a friendly way — I love it every time I see models of cross-partisan amity and constructive disagreement.
We talked about the role that moral values play in the campaigns. The hosts wanted to talk about what makes swing voters decide, but I preferred to talk about what energizes the bases. We haven’t had an election where both sides aimed for the middle since 2000, when George W. Bush ran as a moderate compassionate conservative.
[Forgive my big speech error of saying “right” once when I clearly meant “left.”]
Visit NBCNews.com for breaking news, world news, and news about the economy
Read MoreDiscovering that the other side is not really so loathesome
Here’s a lovely essay from Michael Rubens, a former producer on The Daily Show with Jon Stewart. For years, Rubens job was to produce segments in which Daily Show correspondents interview and make fun of people with far-out views, mostly on the far right:
So imagine how irksome it was to have to deal with persons like that on a constant basis and discover that those persons, in person, generally weren’t loathsome persons after all. In fact, to my great consternation and disappointment, I often liked them.
Rubens describes a series of tea partiers, rapture-believers, and state representatives who push gun rights and squash gay rights. He hated what they stood for and wanted to find them despicable, but after working with them he found them to be as nice as anyone on his own side. Even the spokeswoman for the hate-spewing Westboro Baptist Church, turned out to be “warm and affable and lovely.” Here’s Rubens conclusion:
What I’m hoping the lesson is: People are complex and can hold different views and still be moral actors — essentially the message that Jon Stewart talked about during his Rally for Sanity. Maybe you already grasp that concept, because you have good friends or loving relatives with beliefs that are wildly divergent from your own. But I tend to think my experience is more typical: I lived in a little bubble surrounded by people who think more or less like me. And when I considered people with opposing viewpoints I would turn into a fabulist, concocting an entire narrative of who they were and what they were like — and what they were like was yucko. Because I was not really interacting with them. I just thought I was, because, hey, look, there they are on the TV, or there’s that guy’s post in the comments section. But that stuff doesn’t count. Meeting people counts. Talking counts.
So yes, I love to loathe people, but my “Daily Show” experience complicated all that and sort of spoiled my fun. When I’m exposed to views that I dislike, I try to remind myself of the human being behind those views and to cut that person some slack. I hope that they would do the same. I think we should all fight hard for what we believe in, but I’d like to put in a request for some general slack cutting – especially as we move deeper into what is sure to be a very heated campaign season.
Read More