The Righteous Mind Challenge
In a recent essay in the evolutionary magazine This View of Life, I analyzed Sam Harris’s Moral Landscape Challenge, in which he offers to pay $10,000 to anyone who can convince him to change his mind and renounce his views. From the perspective of The Righteous Mind, it seems unlikely that anyone who is heavily invested in an idea, and who writes about it with high levels of certainty, can be persuaded to change merely by the force of an essay words written by a stranger. So I offered to pay Harris $10,000 if he changes his mind.
Please read that essay first. This blog entry just gives lots of additional details, particularly about the word-count analyses. I’ll update this entry as people ask me additional questions.
I) How did I choose books to analyze?
I started by analyzing the three big New Atheist books: The God Delusion, The End of Faith, and Breaking the Spell. I then chose 3 recent books that were written by scientists who seem focused on explaining religion, not condemning it. I chose my own book, plus the two books that I knew about which came out shortly after mine, on the origins and psychology of religion: Jesse Bering’s The Belief Instinct, and Ara Norenzayan’s Big Gods. (I also examined some older books, to be confident that I wasn’t cherry picking, and they too were almost all below 1.6, so I just picked the 3 most recent of all the books I analyzed.) To pick the right-wing writers, I chose three of the most prominent, and then went onto Amazon to see which of their books was the most reviewed, which I took as a proxy for most read, or more influential. That led me to Glenn Beck’s Common Sense, Sean Hannity’s Deliver us from Evil, and Ann Coulter’s Treason. But here too, I analyzed a bunch of other books by these and other authors, and found that they typically fell in the middle range, between 1.4 and 1.7. [see below, in section III-B]
II) How did I do the analyses?
I obtained plain text files of all the books (I also bought Kindle versions of all the books, to ensure that the authors would get the royalties they deserve). I checked the text files carefully to make sure there were no issues that would skew the word count, such as headers that repeated on every page. I stripped out all the front-matter before the first word of the main text, and all the end-matter after the last chapter. I kept in introductions and epilogues, but cut out acknowledgments, notes, appendices, and references. I then ran each text file through LIWC, using the built-in 2007 dictionary, outputting the Certain category. Here are all the words that LIWC scores when computing its “certain” score:
absolute, absolutely, accura*, all, altogether, always, apparent, assur*, blatant*, certain*, clear, clearly, commit, commitment*, commits, committ*, complete, completed, completely, completes, confidence, confident, confidently, correct*, defined, definite, definitely, definitive*, directly, distinct*, entire*, essential, ever, every, everybod*, everything*, evident*, exact*, explicit*, extremely, fact, facts, factual*, forever, frankly, fundamental, fundamentalis*, fundamentally, fundamentals, guarant*, implicit*, indeed, inevitab*, infallib*, invariab*, irrefu*, must, mustnt, must’nt, mustn’t, mustve, must’ve, necessar*, never, obvious*, perfect*, positiv*, precis*, proof, prove*, pure*, sure*, total, totally, true, truest, truly, truth*, unambigu*, undeniab*, undoubt*, unquestion*, wholly
III) What are the actual mean scores?
A) Here are the original results I posted.
LIWC Certain score
|Harris||The End of Faith||2.24|
|Dennett||Breaking the Spell||1.77|
|Dawkins||The God Delusion||1.7|
|Hannity||Deliver us from Evil||1.49|
Note that there are no standard deviations, and no error bars on the graph, because these numbers are not samples from a larger population. They are the exact measurement done on the total population of words in each book.
B) Analyses of additional books [added on 3/3/14]:
Rahul in the comments asked me to post analyses of some other random works, e.g., from Gutenberg.org, to give us more context within which to interpret the certainty scores I posted. I agreed that this was a good idea, and asked him to pick some texts from Gutenberg, ideally works of relatively modern non-fiction. Rahul obliged, and provided this list. I show each work with its LIWC certainty score in parentheses: Darwin: On the Origin of Species (2.04); United States Presidents’ Inaugural Speeches: From Washington to George W. Bush (2.18); Gandhi: My Experiments with Truth (1.68); Speeches & Letters of Abraham Lincoln (2.05); Albert Einstein: The Meaning of Relativity (1.81); Works of Martin Luther (2.26); G. K. Chesterton “Orthodoxy” (2.85). I was surprised to find a relatively high score for The Origin of Species, which did not fit with my general sense of Darwin’s careful writing. So I added in all of his other books, which turned out to score quite low on certainty: Darwin: Descent of Man (1.43); Darwin: Expression of emotions (1.32); Darwin: Voyage of the Beagle (1.31), so Darwin’s average certainty score across his four books was 1.52 . I do grant that there is variation within each author, depending on the writing task at hand. Ideally, one would analyze multiple books from each author.
To put this all together, I provide below the LIWC certainty scores of all the books I have now analyzed, including the 9 I presented in my original essay; the 7 suggested by Rahul, plus Darwin’s others; the right wing authors I did not show in my original analysis (e.g., Rush Limbaugh, Bill O’Reilly, Michael Savage, and Mark Levin); and two older books on religion by non-new-atheists (David Sloan Wilson and Scott Atran). I sort the table by declining avg score for each author, given that I have more than 1 work for 6 of the authors.
|Work analyzed in LIWC||Avg certainty for each author||Certainty score for each book|
|harris.end of faith||2.24|
|harris.The Moral Landscape||2.37|
|luther.works of martin luther||2.26||2.26|
|lincoln.speeches and letters||2.05||2.05|
|einstein.the meaning of relativity||1.81||1.81|
|dennett.breaking the spell||1.77||1.77|
|gandhi.my experiments with truth||1.68||1.68|
|beck.common sense [right wing]||1.66||1.66|
|wilson.darwins cathedral [not a new atheist]||1.65||1.65|
|limbaugh.the way things ought to be [right wing]||1.64||1.64|
|O’REILLY AVG [right wing]||1.59|
|bering.the belief instinct||1.56||1.56|
|COULTER AVG [right wing]||1.56|
|HANNITY AVG [right wing]||1.53|
|hannity.deliver us from evil||1.49|
|hannity.let freedom ring||1.57|
|darwin.on the origin of species||2.04|
|atran.in gods we trust [not a new atheist]||1.43||1.43|
|savage.liberalism is a mental disorder [right wing]||1.43||1.43|
|levin.liberty and tyranny [right wing]||1.41||1.41|
IV) Miscellaneous Methodological Issues
A) Some people on Twitter have pointed out a possible confound in the analyses: the LIWC certainty dictionary contains roots like funtamentalis* and fact. If the New Atheists are talking ABOUT religious fundamentalists, or if Harris’s book The Moral Landscape is about values as facts, its unfair to give them points for using those words. I agree, those are false positives. So I will re-run all analyses using a very restricted dictionary, which uses only the most unambiguously dogmatic words, such as “always” and “never.” I propose using this subset of the LIWC dictionary: Absolute, absolutely, always, certainly, definitely, every, inevitab*, must, necessar*, never, obvious*, totally, undeniab*, undoubt*, unquestion*. Please comment below if you think I should cut any of those words, or add back any from the full list given in section 2.
[Text added 3/3/14]: Commenter Bianluca Barbetta, below, suggested cutting “absolute.” So I did that and then re-analyzed the original 9 books (plus Moral Landscape) using the restricted dictionary listed above. The resulting numbers are much lower, of course (because many fewer words are captured), but the basic picture that emerges is little changed. In declining order, the scores are: Harris-End of faith (.47); Beck-Common Sense (.47); Harris-Moral Landscape (.46); Dennett-Breaking The Spell (.39); Dawkins-God Delusion (.38); Hannity-Let Freedom Ring (.37); Coulter-Treason (.36); Bering-Belief instinct (.30); Haidt-Righteous Mind (.21); Norenzayan-Big Gods (.15). These scores show the percentage of all words used in each book that were in the list of words coded by the dictionary. For Harris and Glenn Beck, it’s nearly a half percent of all words.
B) To get an independent check on whether the effects I report are real and robust, I hope somebody will create large text files for each author I chose, composed of, say, 20 blog post available on the internet, and then run LIWC on those files. If someone does that, i”ll post a link to the results here.
C) [added on 2/18/14]: Many of the critical commenters below note that there are many ways of using words like “certain” or “certainly” which don’t indicate anything about the mindset of the author. They are right. LIWC is a simple word count program; it does not analyze words in context, and it does not control for negation (e.g., “it is not certain that…”). So there are many false positives. A LIWC does not by itself prove that the new atheists are more dogmatic than other groups of authors. But it can test one’s subjective impressions; it can add or subtract confidence in one’s impressions. When I read the New Atheist books, Dawkins and Harris sounded angry, whereas Dennett did not. But all three authors seemed to to me to use certainty formulations to an unusual degree. I then ran LIWC to measure both of those categories, and it turned out that the New Atheists were high on both (except for Dennett, who scores low on anger, confirming the impressions of many readers). But then I examined the LIWC output to see the words in context, and it was clear to me that Harris scored high on anger in large part because he is talking about violence and killing related to religion. That’s his subject matter, not his emotion. There were so many of these false positives that I decided it would not be fair or accurate to publish the anger findings. The certainty findings, however, hold up much better. Those word uses did seem to be capturing something about Harris’s prose style. In response to a commenter below, I opened up The End of Faith at random, to page 40, and found this passage:
“The basis of our spirituality surely consists in this: the range of possible human experience far exceeds the ordinary limits of our subjectivity. Clearly, some experiences can utterly transform a person’s vision of the world. Every spiritual tradition rests on the insight that how we use our attention, from moment to moment, largely determines the quality of our lives.”
Surely, clearly, and every? This is not the way most scientists write.
V) Additional Stuff:
A) Here is my entry in the Moral Landscape Challenge, where I say why I think Harris’s claims about morality are wrong.
B) [added on 2/18/14] What will it take for me to pay Harris the $10,000? I will send him a personal check (or donate to his foundation, whichever he prefers) if two conditions are met: 1) Harris pays someone the $10,000 of his own money for writing an essay that changed his views about morality, and 2) this payment is accompanied by an explicit acknowledgment that his “case for a scientific understanding of morality is mistaken,” as he puts it in the challenge itself. Harris and I both agree that a scientific understanding of how morality and moral judgment work, descriptively, is appropriate, and is proceeding well. (We both participated in an Edge.org project on The New Science of Morality). We agree that scientists can study morality just as they can study language, or sexuality, or color vision. The dispute between us arises over whether science can tell us what is in fact morally right and wrong, in the same way that science can tell us facts about the natural world. That is Harris’s most provocative and interesting claim in The Moral Landscape. If he renounces that claim, or in some other way says that his argument in The Moral Landscape was largely wrong, I’ll pay him the money. If he pays the money while admitting only a minor error, or conceding some peripheral points, that would not count as having changed his mind or accepted the refutation of his thesis.
The virtues of rational discourse are everywhere espoused, and yet witnessing someone relinquish a cherished opinion in real time is about as common as seeing a supernova explode overhead. The perpetual stalemate one encounters in public debates is annoying because it is so clearly the product of motivated reasoning, self-deception, and other failures of rationality—and yet we’ve grown to expect it on every topic, no matter how intelligent and well-intentioned the participants.
D) The full opening paragraph from Section I of Hume’s Enquiry, is worth reading:
DISPUTES with men, pertinaciously obstinate in their principles, are, of all others, the most irksome; except, perhaps, those with persons, entirely disingenuous, who really do not believe the opinions they defend, but engage in the controversy, from affectation, from a spirit of opposition, or from a desire of showing wit and ingenuity, superior to the rest of mankind. The same blind adherence to their own arguments is to be expected in both; the same contempt of their antagonists; and the same passionate vehemence, in inforcing sophistry and falsehood. And as reasoning is not the source, whence either disputant derives his tenets; it is in vain to expect, that any logic, which speaks not to the affections, will ever engage him to embrace sounder principles.
(Note that Hume is describing two different kinds of disputants – the pertinaciously obstinate kind, and the disingenuous kind. That is why he says “either disputant” in the last sentence. But since I’m only talking about one kind, I changed it to [such a].)
Note to commenters: You can be as critical as you want of my ideas, but any comments that use obscenity or insults will be deleted.