The Bibby Survey: Internal and External Validity

Today, I want to return to survey conducted by Reginald Bibby that allegedly shows that people who believe in God have better values than those who do not.

I want to begin by stressing some elements of this presentation.

Just Evaluations

I am not going to argue that, “Of course, atheists must be better people than theists. Therefore, the survey must be flawed. I only need to discover the flaws.”

Atheism is a morally neutral theory about the makeup of the universe. It is just like heliocentrism in that it has absolutely no moral implications – other than the implications that follow from the fact that the proposition, “At least one God exists” is almost certainly false. Just as heliocentrists can be good or bad, atheists can be good or bad.

Furthermore, each of us has a right to be judged according to our own behavior. Bigotry consists, in essence, in creating a ‘group’ category and condemning or praising individuals in the virtue of their membership in that group, regardless of individual contributions. Because I am a male, I am in a category that is responsible for over 90% of the crime in the United States. Yet, this gives nobody the right to accuse me of being a violent criminal. There is nothing in ‘male’ that implies ‘violent criminal’ and I have a right to be presumed innocent unless and until there is evidence (beyond a reasonable doubt) of my guilt.

I extend the same principle to others. “At least one god exists” is, itself, a morally neutral principle that does not allow us to categorize one who believes it as good or bad. That depends (in part) on what else they believe in addition to their belief that at least one God exits.

Some of those religious people prove their immorality when they use a study such as Bibby’s to denigrate all atheists. When they do this, they violate the moral principle mentioned above of judging each person on his or her own merits. They become bigots, who use broad categories to condemn whole groups of people regardless of individual characteristics. Whenever we find a theist making these types of claims, we find a theist who is immoral – one who sees God inspiring him to do things that promote injustice in the real world. We can make this claim and justify it, but we cannot justify the implication that, ‘Therefore, some other theist who we know nothing else about, must also be a bigot.”

Internal and External Validity

Another issue that I wish to discuss is the internal and external validity of Bibby’s survey. I have already mentioned that there is a significant problem with the survey in that Bibby has not given any evidence of having acquired peer review. Peer review is the process by which experts in the field judge whether the conclusions that the author asserts actually follow from the evidence provided, and whether the conclusions can be extrapolated beyond the list of subjects involved in the survey.

Upon further consideration, I find that Bibby’s study lacks both internal and external validity.

On the matter of internal validity, the press release associated with the study states:

A new examination of Canadians who believe in God and those who do not has found that believers are more likely to place high value on traits such as kindness, politeness, and generosity.

The study does not support this conclusion at all. The study supports the conclusion that theists, more than atheists, are more likely to report having certain values. However, the fact that one has reported having a particular value is not proof that one actually has it. A great many criminals protest that they are innocent. The fact that they report innocence is not proof that they are, in fact, innocent.

If the paper had been subject to peer review, this flaw would have likely been caught.

The study also contains a flaw that concerns its external validity – any claim that the findings in this study reflects the general population.

The survey reports that Bibby questioned 1600 people, of which 7% were atheists. This means that his atheist population consists of only 112 people, plus or minus three.

Statisticians know that the accuracy of a poll is determined by it the size of the sample. The smaller the sample, the more likely it is that the population is skewed and does not represent the population at large. In fact, standard proactice is for any survey to mention the error range. You may have noticed that news anchors discussing presidential campaigns will report how a survey has an accuracy of, say, “Plus or minus three percentage points”. In this example, if one candidate has 45% of the vote, the other has 42%, and the error is 3%, the race would be considered a statistical dead heat. The error does not allow a researcher to draw any meaningful conclusions.

Bibby does not provide us with any error bars. In doing so, Bibby is not even living up to the minimum of his professional standards. This gives us little reason to trust his data.

I want to stress that it is not a valid argument to say, “Bibby’s survey lacks internal and external validity; therefore, atheists are better people than this survey reports.” Reality might go the other way. Bibby might have accidentally stumbled upon the only good atheists in all of Canada, and overstated their virtue. The flaws in Bibby’s study simply allow us to know that he has not, in fact, supported the conclusions that he claims to have supported. He has told us almost nothing about atheists in the real world. Only a bigot who does not care about these matters will think that the survey provides good reason to call atheists inferior based on this study.

Of course, Bibby might have been very much aware of the fact that a substantial portion of his audience will care nothing about internal and external validity. They want a reason to denigrate and smear all atheists, and will gladly overlook any flaws in a survey that appears to give them the ammunition they so strongly desire.

I have mentioned the virtue of intellectual responsibility in my previous posts. Part of that virtue is making sure that people do not use one’s research for evil purposes. We have no basis to assume that Bibby is guilty of intentionally feeding a stereotype and promoting hatred and bigotry. However, we do have enough evidence to suggest that he was negligent in trying to prevent it – in trying to prevent his research from being misused. He should have anticipated the evil that others would have done with his work and at least included a warning that said, “These findings are not subject to large error bars, do not prove that theists actually hold these values but only that they claim to hold these values, and cannot fairly and justly be used to condemn a whole group of people.”

Failure to do so is itself a moral failing, and shows that Bibby, at least, if he is Christian, seems to be lacking some of the virtues that his survey tries to attach to Christians generally.

A member of the studio audience has asked for my comment on a decision in Maine to give birth control pills to middle-school students.

I have not commented on this story to date because I do not know what to think about it just yet. A conclusion requires having knowledge that I do not have.

However, there are some related topics that I can write about.

One of those topics is that it is extremely important to experiment with different options and to collect data on the efficacy of those options, so that we can make better plans in the future. The very reason why I am uncertain about what to say on this issue is because I do not know what the effects will be, and I find it difficult to predict. I hope, for the sake of the kids, that the experiment will have a favorable outcome. I do not pretend to the level of arrogance required to know what that outcome will be.