The Five Traditional Evaluation Criteria and How to Apply Them

Evaluating Sources

Five Basic Criteria for Evaluating Sources

If any one dimension of writing research papers can said to be “most important,” perhaps it is the task of critically evaluating source materials. Good research isn’t a matter of finding, cutting, and pasting someone else’s ideas into your papers so you can get a good grade. Rather it’s a matter of critical engagement with the ongoing “conversation” on a given topic. To do this, you’ll need to “hear” and understand what others are saying to form an idea of which “voices” transmit high quality information and which voices do not.

Traditionally, researchers have used five criteria for evaluating a source’s worth: accuracy, authority, objectivity, currency, and coverage. Let’s look at each of these more closely, considering practical steps you can take to ensure that you use only the highest quality information in your papers.

How accurate is the information? Just because it’s in print doesn’t automatically mean that it’s true or 100% accurate. It might surprise you to learn that even the Encyclopedia Brittanica contains numerous errors in fact (not to mention typos). Even books by internationally recognized experts issued by venerable publishers occasionally contain typos and other errors that can garble meaning or inadvertently falsify information. And the vast majority of printed materials are intended to make money or advocate an idea—two very good motives for supressing or distorting the truth. Unfortunately, when you’re new to a topic, it isn’t easy to know whether information in a text you’re reading is widely considered to be accurate. To develop some confidence in a potential source, ask yourself the following questions:

Are there obvious errors in fact (e.g. dates or names that you know are wrong) and/or a large number of typos and other editorial errors?

Do the facts, figures, and truth-claims check-out with similar information in other sources?

Verifying accuracy: When dealing with a new source, it’s good practice to test a representative handful of checkable facts against other sources. Pick a handful of names, dates, or numerical data and see whether you can confirm them in two other sources.

How authoritative is the information? Facts such as names, dates, and quantities can be checked for their accuracy, but how does one determine whether the conclusions an author draws from such facts are reasonable or that the way in which a writer uses such facts is credible? Researchers new to a topic are often forced to rely on an author’s reputation to decide how much weight to give a new idea, theory, or interpretation of established facts. But, beyond reputation, how an author presents his or her ideas can tell you quite a bit about the depth of his or her expertise and the quality of his or her thinking. To gauge the authority of the source, ask the following questions:

What training, education, or other qualifications does the author posses?

Are the author’s qualifications directly related to the area that he or she is writing about? That is, expertise in one area isn’t necessarily portable to another. A Nobel-laureate chemist, for example, is not necessarily a reliable authority on history or politics.

Does the author refer to the research of others? Does his or her work feature a bibliography and does that bibliography suggest broad and deep knowledge of the subject?

Does the author carefully make his or her case, presenting all sides of a position, even those that diverge from his or her own? Does he or she summarize the views of others fairly, quoting their positions in context?

How well does the author explain why his or her position differs from others? How well does the author anticipate reasonable questions about—and objections to—his or her own position?

How reputable is the publisher? Who would you trust more, the National Enquirer or the Los Angeles Times?

Evaluating Authority: Gauging a writer’s reputation is relatively easy to do. Actively seek information about an author’s qualifications. This information is often supplied by publishers of books and periodicals; if not, try “googling” your author’s name, consulting encyclopedias and Who’s Who guides, or contacting the author directly. While there are certainly exceptions, the most authoratative writers have doctoral-level training, or extensive personal experience, or a notable publication record on their subjects. Be on the look-out for authors with multiple publications—or whose names are mentioned in other people’s writing.

Other measures of a writer’s credibility include his or her institutional affiliation(s), whether or not he or she has received grant moneys or various awards for his or her work, and the reputation of the publisher that has presented is or her work. Obviously, university and research-lab affiliations are prestigious and suggest that the author has earned professional stature through years of dedicated study. Competitive awards and grants, likewise, suggest that the author’s past work has been judged superior. And, in general, a univerity-press imprint on a book suggests that the material in it has been subjected to at least one (and often several) rounds of review by experts in the field and been checked for accuracy by professional editors and fact-checkers.

One of the primary internal measures of an author’s credibility is whether or not he or she cites the work of other respected researchers. To put it crudely, if the text you’re examining has a fat bibliography, there’s a good chance that its author knows the subject well and therefore can rightly claim some measure of expertise.

How objective is the information? Ideally, all sources used in a research paper will be completely objective. That is, they will be dispassionate explanations of the known facts—and high quality sources strive to achieve this very ideal. Nevertheless, all communication acts, whether they are spoken during quiet conversations between friends or published in scientific journals, are to one degree or another perspectival, opinionated, and influenced by a wide range of motives. Thus, no matter how near to the ideal of objectivity a writer comes, a reader should always bear in mind that objectivity is, in fact, an ideal and not a perfectly realizable reality. Ask yourself the following question:

What were the author’s likely purposes when composing the text you’re reading?

Evaluating objectivity: The savvy researcher looks behind the well-turned phrase, rhetorical fireworks, and the impressive display of facts for the human being working the levers. Most of the time, a writer’s motives are, if not positively altruistic, at least benign. People publish to share the latest discoveries and information, to communicate feelings, and to sell products and services. But there are also more personal motives: people write to earn a paycheck, to advance their careers, to identify themselves with various belief systems and/or special-interest groups, to gather a following to themselves, or simply as a means of validating their lives. In short, even the objective-seeming scientist or faceless bureaucrat writes from very personal and, more or less, self-interested motives in addition to such lofty ideals as the pursuit of knowledge and truth. A good researcher recognizes that this is true of all of us—that our communications with one another proceed from a complex mixture of selfish and selfless motives and, therefore, strikes an attitude balanced between an uncritically open mind and a hard-headed skepticism.

How current is the information?  As a general rule, you should use the most currents sources avaiable to you. This is particularly true of information related to scientific and technical topics. What a source had to say in 1992 about the World Wide Web, for example, is likely to seem quaint by today's standards. What doctors knew in 1989 about, for another example, about the human genome bear almost no comparison with they knew about it by the year 2000. It’s vitally important to a researcher’s credibility to demonstrate that he or she is aware of the most recent information and opinion about a subject.

Evaluating currency: Look for publication dates when conducting your literature search and limit your list of potential sources to those written in the past 5-10 years for subjects in the sciences and the past 10-15 years in the humanities unless your research requires you to investigate the history of your subject.

How broad is the coverage? In general, if you are new to a subject, the first thing you need is some basic background on your subject. As you become more familiar with your subject, you’ll want to dig into the details of certain dimensions of your topic. To cite the example considered in the Research Process section of these pages, if you’re doing research on deforestation in the Amazon rainforest, you’ll need some general idea of how tropical rainforests—and especially the Amazonian rainforest—works. You’ll need such basic information as where, exactly, the Amazon rainforest is and what people and other lifeforms live there before even beginning to understand how rapid deforestation effects the forest and the species that depend upon it. Later, however, you’ll need more specific facts, figures, anecdotes, and examples for your paper.

Evaluating coverage: Examine the titles and publication information of poptential sources carefully. Look for sources that explicitly use such words as introduction, history, overview, and general in the title. Similarly, some publications are intended for a general audience while others cater to a specialized audience. Thus, an article titled “Winners and Losers in the Amazon” in the Atlantic Monthly is clearly intended for a general audience. On the other hand, an article titled “Downstream Effects of Erosion from Small-Scale Gold Mining on the Instream Habitat and Fish Community of a Small Neotropical Rainforest Stream” in the journal Conservation Biology is clearly intended for an audience of specialists.