How to Do Research Critically
Video version!

When I worked as a librarian I was often asked to recommend a "good source" for requested types of information. The questions I got, and still get today, are of the nature, "How do you know a source is reliable?" "Who do you trust and why?" "How do we decide between authorities -- especially when they are both well-credentialed?"

The answer many people give is, "You can't." The answer others give is, "Just pick who you like best."

The answer I give is: You follow some basic research guidelines. This is the sort of stuff librarians learn in school -- how to select sources and decide which ones are reliable, and how to decide between them.

In this service I have some basic guidelines which can make things easier. Understand that none of these is a 100%, ironclad absolute -- there can be obvious exceptions; but the fewer of these expectations are met, the less likely it is you have something worth using in your hands.

  1. Check their credentials. I've beaten this one into the ground that you probably guessed it, but I'll beat it some more and some more. Most sources will have some kind of short biography giving a person's credentials. It should tell you what their education has been, give a highlight of publishing credits (you can look for someone's vitae if you want a full list; with many scholars, they won't fit on a book's cover or flyleaf), and maybe name some prominent associations. If these are either:
    1. Missing (the publisher may not have had room, or if your book is missing a paper cover, it may have been there -- but assuming otherwise...)
    2. Not relevant (i.e., a book on Biblical scholarship written by someone with a Ph. D. in aerodynamics!)
    3. Unclear (i.e., it says they "got a Ph. D. at Vanderbilt" but does not say in what)'ll need to think about that source a bit. Obviously such credentials are not required to be an author in the right (especially if one uses sources of such credit to compose a book) but they can help you decide whether an author is likely to have credibility.

  2. Check their sources. A respectable source should have some sort of bibliography and/or notes. Look at these carefully. Warning signs on this are:
    1. If you see the same sources used over and over, in other words, minimal variety in sources. This could indicate someone who hasn't done a lot of work themselves and is just copying one author uncritically (good example: Michael Martin using Wells in his "did Jesus exist" chapter, or Laurence Gardner using so much Barbara Theiring). Of course you'll need to consider other possibilities (is the author offering a refutation of or "update" to the source he uses?), but re-use of the same source repeatedly is often a danger sign.
    2. If you see incomplete citations or none at all -- if the source has no notes (just a bibliography), or bad ones (like The Hiram Key noting with just a title and author -- no page number) there is likely malfeasance and/or incompetence at work.

      Though note that this is something that may work on a sliding scale: You don't need a note to the claim, "The sky is blue"; higher up, scholars will not need a note for something that is a "given" in their profession or their circle.

    3. If you see sources that are overwhelmingly old -- unless the book is a review of history or something like that, high use of sources that aren't recent could be a sign of someone not doing critical research.
    4. A veteran reader pointed to a related caution, one not so easy to detect -- someone who appeals to the same sources, AND to old ones, may be doing the same thing in essence. Their newer sources may just be rehashes of the old ones, and may themselves be using the same source.

      Also be cautious of people who cite a source, then "piggyback" their own statements onto what the source says. An example comes from a reader, who saw a dissertation in which the author mentioned the Inquisition. He used a credentialed source to back up a certain claim. In the next sentence, which was made seem to be a direct continuation to what was said earlier, the author gave a standard claim about the Inquisition claiming over 500,000 victims.

    5. Finally, please note that "padding" -- listing sources in a bibliography that are not actually used, or are used sparingly -- is not uncommon. The great example of how Freke and Gandy made use of that "Dionysus on the cross" picture, while failing to report that one of their sources said it was declared a forgery by experts, speaks for itself.
  3. Any recommendations? These days a good source may have recommendations from other writers or from review publications on their cover. These can be good to check. If good comments come from a fairly broad ideological spectrum (as is usually the case with Ben Witherington's books) you're likely in good shape. Recommends from a narrow spectrum (i.e., John Shelby Spong being recommended by John Dominic Crossan) are less good worthwhile, but still good. Recommends from nobodies or relative nobodies in context (i.e., a movie star recommending Spong) don't help a bit.

    Recommends from professional reviewing sources like Publishers Weekly may be useful depending on the credentials of the reviewer or their experience. Watch out though, when the recommend seems too short, like Eisenman's use of Kirkus Review calling his book "fascinating reading" and nothing else (when the rest of the review was quite negative).

  4. Reviews. If you want to dig deep, check for reviews of the book in peer-reviewed publications or in general reviewing sources like Kirkus. The former will usually be of more use than the latter since they will be by peers.
  5. And what's not a good reason to prefer a source? It's sad to have to say this, but since people do think these are reasons a source is credible, they have to be mentioned:
    1. "It's biased towards a point of view." Every source is "biased" to a point of view, and the truth is always biased. I don't throw out a book just because it comes from Prometheus Press, and I don't expect Skeptics to throw out a book just because it came from Word Books. Which relates to:
    2. "It's published by a fundamentalist/atheist/cultic press." A caveat here is that this sometimes gives critical information about a writer, on certain levels. Acharya S obviously gives us a warning being published with "Steamshovel Press" (which also does stuff on UFOs and Atlantis). This is a reliable indicator at extremes, but not so much in the middle.
    3. "People on Amazon like it." Most Amazon reviews are done by everyday people and are a symptom of intellectual democracy. Be just as critical with them (if not more so) than anything else.
    4. "It's a best seller!" So was Mein Kampf in pre-Nazi Germany.
    5. "It's provocative/it opens your mind!" No comment.

    Once again, these are just some general guidelines; they are far from ironclad, and they are certainly no substitute for logical and critical thinking. In fact, you really can't use one without the other.

    Appendix: On Trusting Works of Fiction. A writer-in made the point that we are overwhelmed today by a new genre of fiction that purports to be fact: Dan Brown's Da Vinci Code is one, and there have been others such as McGowan's The Expected One, and Sierra's The Secret Supper.

    Obviously, there are little or no works of fiction -- even those that report truth -- that are going to pass the tests above. An author of such a worrk may (as Harriet Beecher Stowe did) produce companion volumes (non-fiction) that validate claims in their fictional work.

    While I don't wish to posit some organized effort to do so, it does appear that many fiction writers are taking advantage of at-large gullibility or desire for "change" and "diversity" to promulgate wayward, invalid points of view about things like the identity of Jesus and the history of Christianity. But as with other works, the same tests must be applied.