The Righteous Mind Challenge
In a recent essay in the evolutionary magazine This View of Life, I analyzed Sam Harris’s Moral Landscape Challenge, in which he offers to pay $10,000 to anyone who can convince him to change his mind and renounce his views. From the perspective of The Righteous Mind, it seems unlikely that anyone who is heavily invested in an idea, and who writes about it with high levels of certainty, can be persuaded to change merely by the force of an essay words written by a stranger. So I offered to pay Harris $10,000 if he changes his mind.
Please read that essay first. This blog entry just gives lots of additional details, particularly about the word-count analyses. I’ll update this entry as people ask me additional questions.
I) How did I choose books to analyze?
I started by analyzing the three big New Atheist books: The God Delusion, The End of Faith, and Breaking the Spell. I then chose 3 recent books that were written by scientists who seem focused on explaining religion, not condemning it. I chose my own book, plus the two books that I knew about which came out shortly after mine, on the origins and psychology of religion: Jesse Bering’s The Belief Instinct, and Ara Norenzayan’s Big Gods. (I also examined some older books, to be confident that I wasn’t cherry picking, and they too were almost all below 1.6, so I just picked the 3 most recent of all the books I analyzed.) To pick the right-wing writers, I chose three of the most prominent, and then went onto Amazon to see which of their books was the most reviewed, which I took as a proxy for most read, or more influential. That led me to Glenn Beck’s Common Sense, Sean Hannity’s Deliver us from Evil, and Ann Coulter’s Treason. But here too, I analyzed a bunch of other books by these and other authors, and found that they typically fell in the middle range, between 1.4 and 1.7. [see below, in section III-B]
II) How did I do the analyses?
I obtained plain text files of all the books (I also bought Kindle versions of all the books, to ensure that the authors would get the royalties they deserve). I checked the text files carefully to make sure there were no issues that would skew the word count, such as headers that repeated on every page. I stripped out all the front-matter before the first word of the main text, and all the end-matter after the last chapter. I kept in introductions and epilogues, but cut out acknowledgments, notes, appendices, and references. I then ran each text file through LIWC, using the built-in 2007 dictionary, outputting the Certain category. Here are all the words that LIWC scores when computing its “certain” score:
absolute, absolutely, accura*, all, altogether, always, apparent, assur*, blatant*, certain*, clear, clearly, commit, commitment*, commits, committ*, complete, completed, completely, completes, confidence, confident, confidently, correct*, defined, definite, definitely, definitive*, directly, distinct*, entire*, essential, ever, every, everybod*, everything*, evident*, exact*, explicit*, extremely, fact, facts, factual*, forever, frankly, fundamental, fundamentalis*, fundamentally, fundamentals, guarant*, implicit*, indeed, inevitab*, infallib*, invariab*, irrefu*, must, mustnt, must’nt, mustn’t, mustve, must’ve, necessar*, never, obvious*, perfect*, positiv*, precis*, proof, prove*, pure*, sure*, total, totally, true, truest, truly, truth*, unambigu*, undeniab*, undoubt*, unquestion*, wholly
III) What are the actual mean scores?
A) Here are the original results I posted.
|
Author |
Title |
LIWC Certain score |
| Harris | The End of Faith | 2.24 |
| Dennett | Breaking the Spell | 1.77 |
| Dawkins | The God Delusion | 1.7 |
| Beck | Common Sense | 1.56 |
| Hannity | Deliver us from Evil | 1.49 |
| Coulter | Treason | 1.49 |
| Bering | Belief Instinct | 1.56 |
| Norenzayan | Big Gods | 1.2 |
| Haidt | Righteous Mind | 1.15 |
Note that there are no standard deviations, and no error bars on the graph, because these numbers are not samples from a larger population. They are the exact measurement done on the total population of words in each book.
B) Analyses of additional books [added on 3/3/14]:
Rahul in the comments asked me to post analyses of some other random works, e.g., from Gutenberg.org, to give us more context within which to interpret the certainty scores I posted. I agreed that this was a good idea, and asked him to pick some texts from Gutenberg, ideally works of relatively modern non-fiction. Rahul obliged, and provided this list. I show each work with its LIWC certainty score in parentheses: Darwin: On the Origin of Species (2.04); United States Presidents’ Inaugural Speeches: From Washington to George W. Bush (2.18); Gandhi: My Experiments with Truth (1.68); Speeches & Letters of Abraham Lincoln (2.05); Albert Einstein: The Meaning of Relativity (1.81); Works of Martin Luther (2.26); G. K. Chesterton “Orthodoxy” (2.85). I was surprised to find a relatively high score for The Origin of Species, which did not fit with my general sense of Darwin’s careful writing. So I added in all of his other books, which turned out to score quite low on certainty: Darwin: Descent of Man (1.43); Darwin: Expression of emotions (1.32); Darwin: Voyage of the Beagle (1.31), so Darwin’s average certainty score across his four books was 1.52 . I do grant that there is variation within each author, depending on the writing task at hand. Ideally, one would analyze multiple books from each author.
To put this all together, I provide below the LIWC certainty scores of all the books I have now analyzed, including the 9 I presented in my original essay; the 7 suggested by Rahul, plus Darwin’s others; the right wing authors I did not show in my original analysis (e.g., Rush Limbaugh, Bill O’Reilly, Michael Savage, and Mark Levin); and two older books on religion by non-new-atheists (David Sloan Wilson and Scott Atran). I sort the table by declining avg score for each author, given that I have more than 1 work for 6 of the authors.
| Work analyzed in LIWC | Avg certainty for each author | Certainty score for each book | |
| chesterton.orthodoxy | 2.85 | 2.85 | |
| HARRIS AVG | 2.3 | ||
| harris.end of faith | 2.24 | ||
| harris.The Moral Landscape | 2.37 | ||
| luther.works of martin luther | 2.26 | 2.26 | |
| us.presidents.inaugural.speeches | 2.18 | 2.18 | |
| lincoln.speeches and letters | 2.05 | 2.05 | |
| einstein.the meaning of relativity | 1.81 | 1.81 | |
| dennett.breaking the spell | 1.77 | 1.77 | |
| dawkins.god delusion | 1.7 | 1.7 | |
| gandhi.my experiments with truth | 1.68 | 1.68 | |
| beck.common sense [right wing] | 1.66 | 1.66 | |
| wilson.darwins cathedral [not a new atheist] | 1.65 | 1.65 | |
| limbaugh.the way things ought to be [right wing] | 1.64 | 1.64 | |
| O’REILLY AVG [right wing] | 1.59 | ||
| oreilly.culture-warrior | 1.59 | ||
| OReilly.whos_looking_out_for_you | 1.6 | ||
| bering.the belief instinct | 1.56 | 1.56 | |
| COULTER AVG [right wing] | 1.56 | ||
| coulter.godless | 1.63 | ||
| coulter.treason | 1.49 | ||
| HANNITY AVG [right wing] | 1.53 | ||
| hannity.deliver us from evil | 1.49 | ||
| hannity.let freedom ring | 1.57 | ||
| DARWIN AVG | 1.52 | ||
| Darwin.descent_of_man | 1.43 | ||
| Darwin.expression_of_emotion | 1.32 | ||
| darwin.on the origin of species | 2.04 | ||
| Darwin.voyage_of_the_beagle | 1.31 | ||
| atran.in gods we trust [not a new atheist] | 1.43 | 1.43 | |
| savage.liberalism is a mental disorder [right wing] | 1.43 | 1.43 | |
| levin.liberty and tyranny [right wing] | 1.41 | 1.41 | |
| HAIDT AVG | 1.25 | ||
| haidt.happiness-hypothesis | 1.36 | ||
| Haidt.Righteous-Mind | 1.15 | ||
| Norenzayan.Big Gods | 1.21 | 1.21 |
IV) Miscellaneous Methodological Issues
A) Some people on Twitter have pointed out a possible confound in the analyses: the LIWC certainty dictionary contains roots like funtamentalis* and fact. If the New Atheists are talking ABOUT religious fundamentalists, or if Harris’s book The Moral Landscape is about values as facts, its unfair to give them points for using those words. I agree, those are false positives. So I will re-run all analyses using a very restricted dictionary, which uses only the most unambiguously dogmatic words, such as “always” and “never.” I propose using this subset of the LIWC dictionary: Absolute, absolutely, always, certainly, definitely, every, inevitab*, must, necessar*, never, obvious*, totally, undeniab*, undoubt*, unquestion*. Please comment below if you think I should cut any of those words, or add back any from the full list given in section 2.
[Text added 3/3/14]: Commenter Bianluca Barbetta, below, suggested cutting “absolute.” So I did that and then re-analyzed the original 9 books (plus Moral Landscape) using the restricted dictionary listed above. The resulting numbers are much lower, of course (because many fewer words are captured), but the basic picture that emerges is little changed. In declining order, the scores are: Harris-End of faith (.47); Beck-Common Sense (.47); Harris-Moral Landscape (.46); Dennett-Breaking The Spell (.39); Dawkins-God Delusion (.38); Hannity-Let Freedom Ring (.37); Coulter-Treason (.36); Bering-Belief instinct (.30); Haidt-Righteous Mind (.21); Norenzayan-Big Gods (.15). These scores show the percentage of all words used in each book that were in the list of words coded by the dictionary. For Harris and Glenn Beck, it’s nearly a half percent of all words.
B) To get an independent check on whether the effects I report are real and robust, I hope somebody will create large text files for each author I chose, composed of, say, 20 blog post available on the internet, and then run LIWC on those files. If someone does that, i”ll post a link to the results here.
C) [added on 2/18/14]: Many of the critical commenters below note that there are many ways of using words like “certain” or “certainly” which don’t indicate anything about the mindset of the author. They are right. LIWC is a simple word count program; it does not analyze words in context, and it does not control for negation (e.g., “it is not certain that…”). So there are many false positives. A LIWC does not by itself prove that the new atheists are more dogmatic than other groups of authors. But it can test one’s subjective impressions; it can add or subtract confidence in one’s impressions. When I read the New Atheist books, Dawkins and Harris sounded angry, whereas Dennett did not. But all three authors seemed to to me to use certainty formulations to an unusual degree. I then ran LIWC to measure both of those categories, and it turned out that the New Atheists were high on both (except for Dennett, who scores low on anger, confirming the impressions of many readers). But then I examined the LIWC output to see the words in context, and it was clear to me that Harris scored high on anger in large part because he is talking about violence and killing related to religion. That’s his subject matter, not his emotion. There were so many of these false positives that I decided it would not be fair or accurate to publish the anger findings. The certainty findings, however, hold up much better. Those word uses did seem to be capturing something about Harris’s prose style. In response to a commenter below, I opened up The End of Faith at random, to page 40, and found this passage:
“The basis of our spirituality surely consists in this: the range of possible human experience far exceeds the ordinary limits of our subjectivity. Clearly, some experiences can utterly transform a person’s vision of the world. Every spiritual tradition rests on the insight that how we use our attention, from moment to moment, largely determines the quality of our lives.”
Surely, clearly, and every? This is not the way most scientists write.
V) Additional Stuff:
A) Here is my entry in the Moral Landscape Challenge, where I say why I think Harris’s claims about morality are wrong.
B) [added on 2/18/14] What will it take for me to pay Harris the $10,000? I will send him a personal check (or donate to his foundation, whichever he prefers) if two conditions are met: 1) Harris pays someone the $10,000 of his own money for writing an essay that changed his views about morality, and 2) this payment is accompanied by an explicit acknowledgment that his “case for a scientific understanding of morality is mistaken,” as he puts it in the challenge itself. Harris and I both agree that a scientific understanding of how morality and moral judgment work, descriptively, is appropriate, and is proceeding well. (We both participated in an Edge.org project on The New Science of Morality). We agree that scientists can study morality just as they can study language, or sexuality, or color vision. The dispute between us arises over whether science can tell us what is in fact morally right and wrong, in the same way that science can tell us facts about the natural world. That is Harris’s most provocative and interesting claim in The Moral Landscape. If he renounces that claim, or in some other way says that his argument in The Moral Landscape was largely wrong, I’ll pay him the money. If he pays the money while admitting only a minor error, or conceding some peripheral points, that would not count as having changed his mind or accepted the refutation of his thesis.
C) [added 3/3/14] It is interesting that in Harris’s recent debate with Dennett over free will, Harris opens his rebuttal with a claim that seems on its face to support my basic argument:
The virtues of rational discourse are everywhere espoused, and yet witnessing someone relinquish a cherished opinion in real time is about as common as seeing a supernova explode overhead. The perpetual stalemate one encounters in public debates is annoying because it is so clearly the product of motivated reasoning, self-deception, and other failures of rationality—and yet we’ve grown to expect it on every topic, no matter how intelligent and well-intentioned the participants.
D) The full opening paragraph from Section I of Hume’s Enquiry, is worth reading:
DISPUTES with men, pertinaciously obstinate in their principles, are, of all others, the most irksome; except, perhaps, those with persons, entirely disingenuous, who really do not believe the opinions they defend, but engage in the controversy, from affectation, from a spirit of opposition, or from a desire of showing wit and ingenuity, superior to the rest of mankind. The same blind adherence to their own arguments is to be expected in both; the same contempt of their antagonists; and the same passionate vehemence, in inforcing sophistry and falsehood. And as reasoning is not the source, whence either disputant derives his tenets; it is in vain to expect, that any logic, which speaks not to the affections, will ever engage him to embrace sounder principles.
(Note that Hume is describing two different kinds of disputants – the pertinaciously obstinate kind, and the disingenuous kind. That is why he says “either disputant” in the last sentence. But since I’m only talking about one kind, I changed it to [such a].)
Note to commenters: You can be as critical as you want of my ideas, but any comments that use obscenity or insults will be deleted.
49 Comments
Trackbacks/Pingbacks
- The certainty of the “New Atheists” - […] …I took the full text of the three most important New Atheist books—Richard Dawkins’ The God Delusion, Sam Harris’s The End…
- Another Week Ends: Hoffman and Addiction, Parenting Confessionals, Harris v Haidt, Trite Apologies, Super Bowl Commercials and Transform(ers)ational Ministry | Mockingbird - […] with a reasoned essay of under 1000 words? I’ll put my money on Hume and issue my own challenge, The…
- And I’m losing control | AbsurdBeats - […] I mean, I’m not a social psychologist, and even fellow political scientists doubt the soc-sci cred of theorists like…
Since you already have the texts and LIWC, could you put up a table listing the Certain scores alongside some of the other cognitive processes (Insight, Causation, Discrepancy, Inhibition, Inclusive, Exclusive)? And also negative emotions (anxiety, anger, sadness)?
Differences in other areas would be informative if not necessarily explanatory.
Great idea. I’ll try to do this by March 3. [sorry for delay — slammed with teaching and travels]
OK, here is the table of all LIWC output variables, on the 9 books that i analyzed in my original post, plus I added in Harris’s “The Moral Landscape” since i had those numbers on hand. You should be able to download an Excel spreadsheet here:
http://righteousmind.com/wp-content/uploads/2014/03/LIWC.output.of_.all_.categories.xlsx
What I’d love to know is scores of some other random works. Say, Freakonomics or The Great Stagnation or Friedman’s works or Ayn Rand’s.
Or just some famous books from Gutenberg.org if that’s easiest.
Can you post such scores?
Great idea.
So that i don’t do any cherry picking, can you pick three books on project gutenberg? Those books are all past copyright, and some are quite old so language use may be different. Try to find some that are as modern and direct in their language as possible, to make for good comparisons with modern works of non-fiction.
Do pdf’s work too? Here’s a sampling, some are pdfs:
(1) On the Origin of Species by Charles Darwin
http://www.gutenberg.org/ebooks/8205
(2) United States Presidents’ Inaugural Speeches: From Washington to George W. Bush
http://www.gutenberg.org/ebooks/925
(3) Gandhi “My Experiments with Truth”
http://www.mathrubhumi.com/gandhiji/pdf/AUTOBIOGRAPHY.pdf
(4) Speeches & Letters of Abraham Lincoln
http://www.gutenberg.org/ebooks/14721
(5) Albert Einstein “The Meaning of Relativity”
http://www.gutenberg.org/ebooks/36276
(6) Works of Martin Luther
http://www.gutenberg.org/ebooks/31604
(7) G. K. Chesterton “Orthodoxy”
http://www.gutenberg.org/ebooks/130
ok, i have now posted those analyses; thanks so much for suggesting them, they were quite informative.
Thanks! Very interesting indeed. Dawkins doesn’t look so bad now after all.
Ironically, fitting that Chesterton on Orthodoxy wins the prize for certainty? 🙂
if he changes it, then changes it back, do you have to pay twice ?
quick method point : i hope the analysis checks for a nearby negation
(ie it is not clear… , it is not certain…)
Unfortunately, it does not. It just does a simple word count. I did look at the files after LIWC had coded them, to see if it was picking up mostly false positives. There are some of course, but it seemed to me that it was mostly true positives.
actually, looking at the (free) first ten pages
of harris’s book i see things like this:
“This is all we know for certain about the young man.”
which would score twice for ‘certainty’ but is really the opposite.
nearby is this:
“His actions leave no clue at all.”
Yes, ignoring negation, qualification, and discourse context are pretty big no-nos in corpus linguistics. As a result, this analysis is deeply flawed and could actually demonstrate the opposite of what is the case. You’ll need to subject the texts to identical sampling and verify that, say, the certainty indicators are expressive of certainty as opposed to characterizations (or critiques) of the certainty of others. And you’ll have to resist your own tendencies toward confirmation bias as a party with a vested interest.
also note, the mean and SD for the certainty score
http://www.liwc.net/comparedicts.php
is 1.3 +- 0.8
I think those standard deviations are what you get when you look at the scoring of the thousands of books. For each book, there is no standard deviation, it’s an exact measurement. So i can’t report any variance measures.
of course, my point was that all your scores are within 2 sigma from the mean. (ie not significant)
2 standard deviations would be a gigantic effect!
we rarely get differences between groups that are so large.
Mr. Haidt,
It was delight reading your article.
I have certain doubts about those who claim to be atheists and I am sure you may have come across this or you may have an answer for me. That is, some atheists base their perception of the worldview on purely their “moral” judgment, others say their belief is governed by “science” and yet another asks “why must we believe in any system as ‘believing’ only belong to the religious group (rather sarcastically)?” I appreciate diversity in views as it makes everything better (economics, sciences, art and etc.). However, when this group questions the belief system of a religious group, should they not also have to have a belief system from which they are questioning the view/belief of the latter group? Or, am I wrong in thinking like this?
Because Richard Dawkins falls back on biology for his belief since he cannot track back the origin of life beyond 4.2 or 4.5 billion years.
Lawrence Kraus falls back on physics and quantum theories to back up his view. Even though, quantum theories themselves have a lot of black holes with no answer at this point, but they are completely okay with believing in random chances (that they cannot see or test) but not happy with the “god principle” theists believe in.
Are we to accept “random chances” as scientific proof of something or anything?
What mindset does thinkers of this type fall under? Since they have no concrete system to believe in, how can they question a theory (god) that also lack as much scientific proof?
My belief is that of a middle ground – everything is based on an individual’s belief system. One shouldn’t try to impose one view on another, especially when both sides lack pure scientific backing. Therefore, one will pursue what will benefit them the most as an individual or for their group.
I would love to hear back your thought on this.
Sincerely,
Suresh.T
Don’t expect anything back from Haidt about this.
(First, sorry for my bad English, I hope it will be at least comprehensible)
I’m glad that you noticed my objection on Twitter, your analysis is not worthy of the books you’ve written and that I’ve enjoyed (even when I disagree). The damage you’ve done to Sam Harris with that graph surely deserves an apology to him.
About your new list, the word “absolute” have to be removed. Look at Harris book (all occurrences):
-…”these are two object for absolute, impersonal consciousness”…
– …often called “absolute” pacificism- that is, the belief tha violence…
– …and children who were robbed of life in absolute terror and confusion…
– …and yet, implicit in this approach (of someone else) lurks a claim that is not relative but absolute.
– The philosopher Peter Unger has made a persuasive case that a single dollar spent on anything but the absolute essentials…
– …Suicide bombing isn’t really wrong, in any absolute sense; it just seems so from the parochial perspective of Western culture.
– …absolute panic that such a proposition… (speaking about fear to his daughter to be harmed)
-… The fact that the universe is illuminated where you stand, (…), is an absolute mistery- rivaled only by the mystery…
– This absolute consciousness, when is purified of the I, no longer… (speaking about Buddhism and conciousness).
Those are all the occurrences (I hope), but removing absolute will do Glenn Beck a favor because for example he writes “common sense should tell us that the very notion of another “reserve fund” is absolute insanity”.
With those examples I will try to make a general point. This analysis will remain pointless even if we leave only always, never and must, because you have to see every word in his context, and still it will be an impossible task to decide because different arguments require different kind of certainties (If we speak about Noah ark, we can surely use the word NEVER. If we want to speak about how crashing airplanes in skycrapers for religion motives is bad you MUST condemn it and I’m sure that Glenn Beck will agree, but it’s not the argument of his book).
And look at “must”. We have 8 occurrencies in only one page in The end of Faith… just because Harris reports verbatim Deuteronomy 13:7-11.
Context is not relevant for this kind of analysis, specifically because a moderate person would rather prefer not to use strong words. Yes, there are cases when you “absolutely” have to, but when your speech is impregnated with such words, they show you prefer a certain brush over an another to paint the picture.
Context is where you offer you own subjective interpretation of what the context really is. An a rational person should understand that one can not escape the interpretation part, specially on one’s side. That’s why avoidance of strong certainty words is a sign of intellectual honesty.
Hi,
First, it would not be intellectual honesty, but maybe intellectual wariness, and that is not always the best choice in all the arguments one can make (and you seem to acknowledge that). Since Harris thinks that burqa, honor killing, faith wars or atzec human sacrifice are wrong, it would be dishonest from him to use phrases like “seems a bad thing to do” or “could be that they are a little mistaken” in “The end of Faith”. It’s good to be humble and cautios when you present a new research and your conclusions, it’s not so good when you condemn some gory parts of the Bible.
But, just to be clear, in “The end of Faith” Harris have to use the certainty Brush more often also because he has to describe the views of the other side he wants to criticize, in particular Islam fondamentalis*
It have to be stated clearly. Haidt study is misleading. He did the right right by criticizing directly Harris arguments.
Jonathan, I appreciate you attempt at bringing objective analysis to these works. However, corpus linguistics is a robust field with decades of traditions and techniques. It is not as simple as running LIWC across some texts. (LIWC is not particularly respected by serious corpus linguists. Several of your commenters have already pointed out some of LIWC’s flawed assumptions). I will add that frequencies (whether sampled or complete) are typically reported on a per million words scale, to account for different corpus sizes (this would allow researchers to compare your counts with, say, Mark Davies’ corpora at BYU, for example. Percentage alone skews the comparison). If you are serious about using these techniques, I strongly suggest you sign up for the free online course . Best of luck!
Chris, I agree, LIWC is crude, and a more sophisticated analysis, or qualitative analysis would be a necessary complement to my simple analyses. I hope a linguistics grad student will analyze these texts using better methods, and then weigh in here.
I’d say that “necessar*” may be worth taking out of the next analysis as well. The examples that come to mind typically involve people saying something is “not necessarily” true, or some variation of that. In fact, “necessarily” was used in the comments here to promote less doubt, not more. (Mark: “other areas would be informative if not necessarily explanatory.”)
Further, is there any way of accounting for combinations of words that may reflect less certainty? For example, I may say that “almost every” person thinks murder is wrong, or that it’s “not obvious” to me that God is necessary for objective morality.
good idea, i will take out “necessar*”, and “absolute” (as suggested by another commenter).
LIWC does not handle combinations of words without some clumsy work-arounds. but if you read the texts, you see that Harris is not scoring high because of his frequent negations of certainty words.
The problem is that removing “absolute” can do a favor to other authors, like in my Glenn Beck example. The same apply to words like fondamentalist and fact, that can be used to express certainty based on the style of the writer (there is an opinionist in Italy for example that it’s know to use/abuse the phrase “it’s a fact” that now will score very low in your new test). If we have an author that abuse “absolute” or “fondamental*”, but never use “clearly” we will miss that.
You’re wasting your time with this, and opening pages at random from a scientist of your caliber is not a thing that can help to save it.
That graph is the result of an uncaring analysis with an enormous amount of problems well pointed out by other commenters and it informs badly the people who will read it. It seems not fair in respect of Sam Harris, since people will only remember the graph, not your corrections in this post. Leading scientists should do better than this (And I criticized Harris too for this, for example in his discussion about guns where I think he used statistics in an improper way).
I’m done with this topic, but I sincerely hope that at least you regret to have published it.
Certainty words appear most frequently in the three books that are about certainty. The author’s certainty is not measured.
It was quite interesting reading your analysis of these works and the critiques of your analysis that have been posted in the comments. I have three questions like to ask (really different analysises I’d like to see) based on you’ve already posted.
First, how do these works compare to works in other fields (both fiction and non-fiction)?
Similarly, do these authors have works in others fields where we would expect to see more neutral writing? For example, isn’t one of the new atheist authors a biologist?
The purpose of the first question is to determine if there is a quirk in the topic or language that results in unusual levels of certainty, unless the author takes exceptional pains to avoid it. It also helps by establishing a better baseline for judging these values. (Though, one of the other commenters mentioned a value and error bar for texts in general, so that might might be worth considering.)
The second question is ment to see if the higher certainty is simply the result of personal writing style, regardless of topic, and if the author would talk about new circuit board designs or metamaterials in the same way.
My third question is a bit different. One of your “neutral” books has a score that is significantly higher than other two (in fact, it’s as high as the highest conservative book). Is there a it’s so much higher and does that mean anything for the selection of books or or validity of the technique/conclusion?
Another reader asked about comparisons to other texts; i asked him to suggest some that i can get online, and he did. I’ll have those analyzed within a week.
You are surely right that the topic can strongly influence the word choice, without revealing anything about the mind of the author. That’s why i chose non-new-atheists writing about religion. I too am curious as to whether Richard Dawkins early writing was different. I imagine it was, and that he only rises in certainty when he writes about religion, or other topics that are emotionally arousing.
It is trure that the Bering book was higher than the others. I didn’t cherry pick. I just picked the 3 books i was going to analyze, and then let the chips fall where they will. As a class, the New Atheists score highest, and the non-new-atheists score lowest, but the boundaries are not neat. We’ll learn more when i finish analyzing a bunch of other texts.
thanks for these great questions.
If anyone is interested, this is my response to “Why Sam Harris is Unlikely to Change his Mind”:
http://diavgeia.blogspot.gr/2014/02/why-sam-harris-shouldnt-change-his-mind.html
For those who will find it long-winded, I’ll give you the short version. I contacted LIWC and here’s our interaction:
Roger: “LIWC only claims to report the proportion of words in various categories. Beyond that it’s up to the person using the analysis to interpret and, if necessary, draw conclusions or make claims based on the results. In that, interpretation is always comparative – for example, do people who use a higher proportion of certainty words have consistently differently characteristics that people who use a lower proportion.”
Me: “So, to your example, is it an assumption to say that someone who uses more “certainty” words is exhibiting more certainty? From what I’ve understood, one would have to prove that this is the case and not simply accept it as a given, simply because there is a category in LIWC that’s called “certainty”?”
Roger: “Yes that’s correct. Increases in the number of certainty words don’t necessarily indicate more certainty just a higher proportion of that category’s word use.”
Tatiana, i fully agree, and have added a note above, in section IV-C. A LIWC analysis does not prove anything by itself. You must still interpret the results, and there are many reasons why one particular text might have so many false positives that it renders the analysis invalid. I read the words in context, and found that the anger analyses i had originally done were not valid, even though they supported my argument. I invite you to read a full chapter in, say, The End of Faith, and a full chapter in one of the non-new-atheist books, and mark all the certainty words — and the many other constructions that indicate certainty that LIWC missed. Then decide if the LIWC results are picking up a real signal, a real average difference, or if there’s some artifact that makes all 3 New Atheist books score higher on certainty than the other books.
Thank you for your reply, Jonathan, and for making that important update, but I’m getting more and more confused by your approach.
You’re encouraging me to simply read a chapter from each book and “trust” my “general impressions”? It doesn’t seem like you’re worried too much about my emotional biases taking over, though you obviously know you should be. After all, that was the point of your essay.
But, for the sake of the argument, let’s assume we’ve proven the “New Atheists” express more certainty themselves. Even then, there’s a more important problem with your claim, that I mentioned in my article and not here: What your essay translates into is that one is just as wrong to claim certainty whatever it is they’re talking about; whatever the scientific level of certainty for that claim.
So, the claims “There aren’t unicorns” and “There are unicorns” should be asserted with the same level of certainty, even though the probabilities behind those claims are 99.9999…% (never 100%) and 0.0000…1% respectively. Otherwise, we are all equally hypocritical.
“B) To get an independent check on whether the effects I report are real and robust, I hope somebody will create large text files for each author I chose, composed of, say, 20 blog post available on the internet, and then run LIWC on those files. If someone does that, i”ll post a link to the results here.”
i ran a word count on everything on your blog front page
3251 words, 44 ‘certain’ hits , hits/word= 0.0135
on harris’s blog:
5502 words, 30 ‘certain’ hits, score = 0.0054
i think your LIWC is a waste of time and money.
Do you mean you looked for all the certainty words on my recent blog entries, including the one where I listed the 80 or so “certain” words from the LIWC dictionary? That would be a nice example of an artifact that renders an analysis invalid. Try it on 15 full older blog entries, and compare to 10 entries from Harris. (you can also find blog entries from me at http://www.yourmorals.org/blog/author/jon/
I’m dumbfounded to see the number of people who are going nuts over the “absolutist words”! It’s so interesting that they miss (I believe) the larger point. Sam Harris is well-known for his certainty about the relevance of atheism and his discounting of the value of religion. It’s unfortunate that we can’t appreciate that many areas can contribute to our understanding of life and morality, without having to disparage others.
Susan, I would like you to understand what you just said: It’s self-evident Harris shows “certainty”, therefore it doesn’t matter that the logic used to prove it isn’t correct.
The most comprehensive literature that argues in favor of the prospect of using information technology to create artificial moral agents is that of Luciano Floridi (1999, 2002, 2003, 2010b, 2011b), and Floridi with Jeff W. Sanders (1999, 2001, 2004). Floridi (1999) recognizes that issues raised by the ethical impacts of information technologies strain our traditional moral theories. To relieve this friction he argues that what is needed is a broader philosophy of information (2002). After making this move, Floridi (2003) claims that information is a legitimate environment of its own and that has its own intrinsic value that is in some ways similar to the natural environment and in other ways radically foreign but either way the result is that information is on its own a thing that is worthy of ethical concern. Floridi (2003) uses these ideas to create a theoretical model of moral action using the logic of object oriented programming.
Professor Haidt,
No particular words necessarily indicate the author’s degree of certainty, because they can all be used to formulate arguments which the author is attempting to critique. More importantly, any words used to express certainty can also be used in expressions of uncertainty or general caution. For example, “Even though we can never be certain, . . . ” or “We must always be careful to avoid jumping to conclusions.” Then there are cases where certainty terms are used to make concessions: For example, “Obviously I might be wrong.”
If you don’t look at the language in context, you cannot draw any conclusions about how appropriate or inappropriate the expressions are. The raw scores you are offering cannot even be considered useful guides.
Mr. Streitfeld: You are absolutely right. There is no word that always indicates a mindset of certainty; there are many false positives. Using LIWC does require that one check the results in context, which I did. As I explain above, in a new note in section IV-C, when i checked the results in context for LIWCs coding of anger words, i lost confidence, because the new atheists talk about violence much more than do the non-atheists, and that is their subject matter, not their mindset. So those results were unreliable. But when we look at the certainty words, it was a different story. Of course you can find false positives in all works, but there was a much higher signal to noise ratio there. ACross all the New Atheist books, there are many more real expressions of certainty, often in ways that sound dogmatic, e.g., just opening up The End of Faith at random, I find this passage, on p. 40:
“The basis of our spirituality surely consists in this: the range of possible human experience far exceeds the ordinary limits of our subjectivity. Clearly, some experiences can utterly transform a person’s
vision of the world. Every spiritual tradition rests on the insight that how we use our attention, from moment to moment, largely determines the quality of our lives.” Surely, clearly, and every? This is not the way most scientists write.
Oh dear, Jonathan. Reviewing the context of words does not magically transform a meaningless quantitative analysis of text into a meaningful qualitative textual analysis. Am I correct in suspecting that you are not planning on publishing these results in a peer-reviewed journal?
At best, this sort of quantitative textual analysis can be used to support claims that an author is making specific stylistic choices. But you should know better than anyone that we are unlikely to learn anything reliable about a person’s psychology by analyzing their writing style. You’d be on firmer ground advocating for the psychological validity of graphology.
Even if we were to accept that quantitative textual analysis is a valid methodological approach to determining the personal psychology of an author (which I and other qualitative researchers obviously would not), then at the very least you should made some effort at triangulation and controls. Did you, for example, run the same analysis for words associated with UNcertainty? Did you run random words? Did you analyze any other writings the authors’ have produced, such as Harris and Dennett’s peer-reviewed journal articles?
What about construct validity? Does the idea of “dogma” really apply equally to all claims? I would personally argue that it is almost comically obvious (come certainty words for you) that it does not. Is belief in scientific claims is identical in all respects to belief in religious claims, for the purposes of comparative analysis?
Some sound suggestions for improving this study here. And while I share some of the anecdotal evidence regarding Dawkins, I respect that you have openly invited for constructive criticism on how to improve this study. I hope some of these suggestions are followed, because it’s such an interesting study question that could be fascinating by way of implication if the measure had any real meaning…
The random opening of one page to one passage a statistically significant pattern does not make. For this to have any meaning, you’d have to do this for each and every indicator word.
I just looked at Harris’ quote again. I think the confirmation bias that permeates this study from the get go is painfully obvious in your analysis of the statement. For example, following the word ‘clearly’ is the immediate qualification word ‘some,’ which would be an indication of more fluid and careful thinking– the opposite of dogmatic rigidity– and is to what the word ‘clearly’ is attached.
The most glaring confounding factor here, which really does effectively invalidate the implication of the study, is the inability to include context with the isolated word. The only way this approach can reasonably measure what it is intending to measure would be to manually look at every sentence where the keyword gets a hit and take in the surrounding context. It is perfectly reasonable that books more dedicated toward criticizing rigid thinking would include more indicator words toward rigidity and certainty. If I understand your metrics correctly, it also does not exclude where the author may be quoting an opponent in attempt to show the limitation and lack of fluidity in their thinking.
Lastly, I would recommend taking your own book out of the lineup, with the consideration that the appearance of a conflict of interest may be just as bad as an actual conflict of interest.
Suggestion: Wouldn’t it be more beneficial to create indicators that look not for certainty, rigidity, or absolutism, but which indicate fluidity, plasticity, and open-mindedness?
I agree that the most important problem (after limiting what kind of words that should be in the certainty dictionary) is the contex. Although, I don’t think it’s necessary to go through all examples, it should be enough to go through a statistically representative number of them. Until that is done, the book might as well be full of quotes about or the folly of certainty.
Joacim– I agree about utilising a statistically representative sample. That is, indeed, what they’re good for, and why they exist. Thanks. -shane
Hi Jonathan,
A bit late to this, but the very high score of Chesterton’s Orthodoxy surprised me, so I did a bit of spot checking. “Certain” is one of Chesterton’s favorite words, but the great majority of the time he uses it in the sense “A certain person” or “a certain idea” meaning “a particular person” or “a particular idea,” and not one “which of which one has intellectual certainty.” This may have generated a lot of false positives if you didn’t control for a certain usage of “certain.”