Religious knowledge: does Pew get it right?

Yesterday, the results of a study by the Pew Research Center made national news.  And ever-media 2.0-conscious Pew even added a mini-version of the quiz readers can take.  Blogged, tweeted, facebooked about, the new findings cascaded through the world of social media, gathering new interpretations and speculations like a rolling snowball. 

My curiosity -- or rather suspicion -- was raised almost immediately.  A quick multiple-choice test for "religious knowledge"?  Atheists and agnostics scoring higher as an aggregate than Evangelical Protestant Christians or Catholic Christians?  That leads by lightening-quick inference to tweets [links removed] like

@MotherJones: A new Pew study shows that atheists and agnostics have the best religious knowledge

Atheists and agnostics know more about religion than Christians. Read the Pew forum you'd be surprised

 Even to some self-scrutinizing:

Re: the Pew Survey, is it more important for Catholics to know or to live their faith?  The WP weighs in

 And plenty of self-congratulation:

took the Pew Forum's religion quiz. I'm an Episcopalian, and I got 100%.

Some "in-depth" analysis has been posted.  As a specialist in Philosophy of Religion, who taught and researched in Religious Studies, dealing primarily with first- and second-year undergraduates, I have to say three main things about the poll, the results, the interpretations, and the analysis.

First, to anyone who works with religion and keeps their eyes and ears open, it has been painfully evident for a long time that Americans in general -- whatever religious affiliation they may identify with --  are on the whole uninformed about religion and religions, quite often even about their own.  No big surprise or news there.

Second, much of the supposed analysis is merely using the results to push positions to which the interpreters are already (often publicly) committed.  Put in terms of argumentation theory, the "results" are merely furnishing additional premises to be inserted into already long-standing positions and their arguments.
The trouble is, the results don't actually support those projects, since they're not really solid findings with those sorts of implications.  And for this reason, the poll and its results becomes a wax nose, turned in any way its interpreter wants. Pieces by Ramdas Lamb, Aseem Shukla, and Susan K. Smith provide a few representative examples of the confirm-what-we-already-believe "analysis" going on.  Unfortunately and very disappointingly, so does Stephen Prothero's My Take: Why American public schools need religion courses -- and I say that as someone who largely agrees with Prothero's program, outlined in his Religious Literacy book. (the Pew study, by the way, spurred by Prothero's arguments)

Third, the construction of the poll, the presentation of the poll's results, the ideological construct of "religious knowledge" being bandied about and the assumptions that are built into it should be very troubling, for anyone who has a background in education, assessment, and particularly in religion.  The fact that they don't seem to be troubling is even more troubling, a sign of basic deficits of pretty basic-level critical thinking among our literate, web-savvy, educated classes.  The fact that this could be regarded and replicated throughout the manifold of public discourse as "findings" as providing "knowledge" about religious knowledge -- what?  is there a tumor in the collective cultural brain, blocking proper function and self-scrutiny?

A few other people are noting some analogous problems (e.g. Richard Amesbury -- I'm not pretending to carry out a comprehensive lit review here -- just mentioning some interesting pieces coming up in my Twitter feed). I'm  going to blend the second and third points above together and just throw out some additional reflections which might lead to further discussions via comments.

First off, the entire "poll" is a multiple-choice and fill-in-the-blank test. I don't often use the shrill all caps, but. . . A MULTIPLE CHOICE TEST!

Educators -- and anyone who has to rely on assessment results (used, e.g. in hiring) -- know that multiple choice tests tell us relatively little about whether a person actually knows much about a subject.  Even adding some fill-in-the-blanks -- where it is a single phrase, not a short answer requiring some thought -- does not make this a good instrument.  That is precisely why we educators use other instruments -- essay questions, rubrics, application assignments,  the list could go on and on -- to assess students' less-than-superficial "knowledge" of subjects.  I'll admit, I would use a multiple choice test or quiz as one instrument in a class for testing my students in a World Religions course, but I would not for a minute believe that it correlates to anything I could generalize and make further reliable inferences about.  Pew, incidentally, acknowledges some of these problems in a FAQ page.

Second, even putting aside just for a moment the problems with testing some ill-conceived category of "religious knowledge," what do the results actually tell us about people grouped by religious affiliation?  Every group, considered as a whole, did badly -- D range at best (Atheists/Agnostics 65.3%, Jewish 64%, Mormon 63.4%).  Most failed.  The lowest group aggregate scores were what I call in my classes Mid-Low Fs (Hispanic Catholic 36.3%, Black Protestant 45.9%, Nothing In Particular 47.5%).  But every group did poorly by any standard.  Nobody should be congratulating themselves about their group's performance.

Third, when you start looking more closely at the breakdown of the results, based on the subject material of the questions, very different pictures emerge.  Do atheists really know more about the Bible than Christians?  Of 7 questions (5 about the Old Testament/Tanakh, 3 about the New Testament -- actually), Christians as an aggregate got 4.2, Atheists 4.4.  Again, marginally better performance by Atheists.  They'd both get Ds in my class.  Now, split Christians into Protestants (traditionally Sola Scriptura) and Catholics (who are more liturgically centered), and the scores diverge:  Protestants score a 4.5, Catholics 3.4.

Look closer at the Protestants, and there's a split as well.  White Evangelical Protestants -- precisely the sort of people Atheists are going to be arguing with -- score a 5.1.  Mainline (read: liberal, social-justice focused) Protestants score a 3.9, just barely better than White Catholics. Black Protestants are in the middle, with a 4.4. -- the same number answered correctly as Atheists.

This could go on and on.  Catholics doing poorly on the question about the first book of the Bible?  Not surprising, since Catholics are mainly unengaged in the front of the culture war which pits Genesis against Darwin/Dennet/take your pick -- but Atheists and White Evangelical Protestants are engaged there.
Genesis is big stuff if you're obsessed with evolution. Catholics aren't.

Christians know more about their own religion than about other religions?  Really?  That's a surprise?  That indicates a lower knowledge of "religion"?  Or of "religions".  I'll let you in on a dirty little trade secret:  There's no consensus among philosophers, sociologists, historians, theologians, precisely what constitutes "religion," what makes something a "religion," or even whether we can rigorously talk about "religion" in general.  Determining whether someone knows about "religion" based on their recall of facts about other people's "religion" relies on a lot of unquestioned assumptions.

Fourth, all of this is besides the point.  These are aggregate scores.  Do they correlate --  --can you make inferences from them to actual people?  Is the average Atheist marginally better (though still poorly) informed than the average Protestant about religious topics?  There is some probability, but only some, and the confidence one places in these sorts of inferences, given the tenuousness of the concepts and constructs involved, ought to be very qualified.  But, instead, even someone like Prothero, who ought to know better, ask questions and make claims like these:

Who knows more about religion - the arch-atheist Christopher Hitchens or Islam basher Rev. Franklin Graham?  Most likely the unbeliever, according to a U.S. Religious Knowledge Survey released today by the Pew Forum on Religion & Public Life.

It's statistically interesting to note that in the poll, 2% of the population got 29-32 correct (earning an A), 10 % got 25-28 (earning a C+ to B).  These are the people who are better informed about religion -- at least from the vantage point of a multiple choice exam, which tests surface knowledge, easy to learn and to regurgitate, requiring almost no actual understanding of the topics in their complex reality.

Let's imagine placing bets on an intellectual cage match between Hitchens and that Graham, or better yet one of the other more recognizable Grahams?  Would the results of the Pew Survey help you at all in placing your bet?  Let's raise the intellectual caliber:  how about one between the guy who became the present pope, Joseph Ratzinger, and the philosopher Jurgen Habermas?  No, this survey would tell you nothing, because religious knowledge is something much more complex, integrative, and deep-rooted than the poll can assess, and because polling groups of self-identifiers can tell you very little about those who are intensely involved in a community, a movement, a religion.  Most likely, it can tell you very little even about those who have a fairly high and consistent involvement. 

As a parting shot:  I do agree with scholars like Prothero and their projects of increasing religious literacy and advocating teaching about religion (without teaching religion as proselytizing) in schools.  Even if I didn't consider it necessary for an education in today's global society where despite centuries of secularization religion is not going away, I'd be for that project just for the simple fact that religious illiteracy translates into cultural illiteracy.  Most of the interpretations of this Pew poll being proffered are symptoms of precisely those forms of illiteracy compounded by failures of Critical Thinking.