Evaluating the authority, usefulness, and reliability of resources is a crucial step in conducting a review of the literature that effectively covers pertinent research and, thereby, demonstrates to the reader that you know what you're talking about. The process of evaluating scholarly materials also enhances your general skills and ability to:
Black, Thomas R. Evaluating Social Science Research: An Introduction. London: Sage, 1993.
The act of thinking critically about the validity and reliability of a research resource generally involves asking yourself a series of questions about the quality of both the item and the content of that item.
Evaluating the Source
Inquiring about the Author
What are the author's credentials, such as, institutional affiliation [where he or she works], educational background, past writings, or experience? Is the book or article written on a topic in the author's area of expertise? Has your instructor mentioned this author? Have you seen the author's name cited in other sources or bibliographies? Is the author associated with a reputable institution or organization? What are the basic values or goals of that organization or institution?
Inquiring about the Date of Publication
When was the source published? Is the source current or out-of-date for your topic?
Inquiring about the Edition or Revision
Is this a first edition of this publication or not? Further editions usually indicate a source has been revised and updated to reflect changes in knowledge, to include prior omissions, and to better harmonize the contents with the intended needs of its readers. If you are using a web source, do the pages indicate last revision dates?
Inquiring about the Publisher
Note the publisher. If the source is published by a university press, it is likely to be scholarly. Although the fact that a publisher is reputable does not necessarily guarantee quality, it does show that the publisher has a high regard for the source being published [their reputation as an academic publisher relies on it].
Inquiring about the Title of Journal
Is this a scholarly or a popular journal? This distinction is important because it indicates different levels of complexity in conveying ideas and the intended readership.
Evaluating the Content
What type of audience is the author addressing? Is the publication aimed at a specialized or a general audience? Is this source too elementary, too technical, too advanced, or just right for your needs?
Is the information covered considered to be fact, opinion, or propaganda? It is not always easy to separate fact from opinion. Facts can usually be verified; opinions, though they may be based on factual information, evolve from the interpretation of facts. Does the information appear to be valid and well-researched, or is it questionable and unsupported by evidence? Note errors or omissions. Are the ideas and arguments advanced more or less in line with other works you have read on the same topic?
Does the work update or clarify prior knowledge, substantiate other materials you have read, or add new information? Does it extensively or only marginally cover your topic? Does it provide a balanced perspective? If the item in question does not meet this criteria, you should review enough sources to obtain a variety of viewpoints.
Is the publication organized logically? Are the main points clearly presented? Do you find the text easy to read, or is it stilted or choppy? Is the author's argument repetitive?
In the case of books, locate critical reviews of the work in a database such as ProQuest Multiple. Is the review positive? Is the book under review considered a valuable contribution to the field? Do reviewers agree on the value or attributes of the book or are there strong differences of opinion? Does the reviewer mention other books that might be better? If so, locate these sources for more information on your topic.
Brand-Gruwela, Saskia and Marc Stadtlerb. “Solving Information-based Problems: Evaluating Sources and Information.” Learning and Instruction 2 (April 2011): 175-179; Barzilaia, Sarit and Anat Zohara. “Epistemic Thinking in Action: Evaluating and Integrating Online Sources.” Cognition and Instruction 30 (2012): 39-85; Critical Thinking. Academic Skills Centre. University of Canberra; Evaluating Sources. Lakeland Library Research Guides. Lakeland Community College; Evaluating Sources. Writing@CSU. Colorado State University; Evaluating Print Sources. The Writing Center. University of North Carolina; Evaluation During Reading. The Writing Lab and The OWL. Purdue University; Walraven, Amber, Saskia Brand-Gruwel, and Henny P.A. Boshuizen. “How Students Evaluate Information and Sources When Searching the World Wide Web for Information.” Computers and Education 52 (January 2009): 234–246
Web Content Requires Additional Methods of Evaluation
A report from the Stanford University Graduate School of Education found that students evaluating information that flows across social media channels or retrieved from online search engines like Google or Bing, have difficulty distinguishing advertisements from news articles or how to identity where the content came from. In general, the principles that guide your evaluation of print materials are the same that apply to evaluating online resources. However, unlike print materials that have certain features that help determine their scholarly integrity, the interactive and multimedia dynamics of online sources requires additional attention to the content in order to obtain confidence that what you are viewing is valid and credible.
Additional things to look for when considering using an online resource:
Evaluating Internet Information. Online Library Learning Center. University of Georgia; Evaluating Internet Sources: A Library Resource Guide. Olsen Library. Northern Michigan University; Evaluating Sources. Writing@CSU. Colorado State University; Evaluating Web Sites. Teaching and Learning Services, University of Maryland Libraries; Ostenson, Jonathan. “Skeptics on the Internet: Teaching Students to Read Critically.” The English Journal 98 (May, 2009): 54-59; Stanford History Education Group. "Evaluating Information: The Cornerstone of Civic Online Reasoning." Stanford, CA: Graduate School of Education, 2016; Writing from Sources: Evaluating Web Sources. The Reading/Writing Center. Hunter College.
Bias, whether done intentionally or not, occurs when a statement reflects a partiality, preference, or prejudice for or against an object, person, place, or idea. Listed below are problems to look for when determining if the source is biased.
NOTE: The act of determining bias in scholarly research is also an act of constant self-reflection. Everyone has biases. Therefore, it is important that you minimize the influence of your own biases by approaching the assessment of another person's research introspectively and with a degree of self-awareness.
ANOTHER NOTE: The idea of bias often carries negative connotations, even though the meaning of the term is not defined in that context. Be sure when you are evaluating a source that you do not automatically reject it as invalid if you detect bias. One way to do this is to substitute the idea of bias with the idea of perspective. Ask yourself, what type of perspective does this source bring to the investigation of the research problem? As Lesh points out, the goal of research is to engage with multiple sources for the purpose of acquiring multiple perspectives about the topic. As long as the source is rooted in fact-based evidence, you should not reject it as being biased, but rather, consider it as a potential source of perspective about the research problem.
"Availability Bias, Source Bias, and Publication Bias in Meta-Analysis." In Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. 3rd Edition. (London: SAGE Publications, 2015), pp. 513-551; "Bias." In Key Concepts in Social Research. Geoff Payne and Judy Payne. (London: SAGE Publications, 2004), pp. 28-31; Evaluating Sources. Lakeland Library Research Guides. Lakeland Community College; Fischer, Fritz. "Teaching Trump in the History Classroom." Journal of American History 108 (March 2022): 772-778; Lesh, Bruce A. “Why Won’t You Just Tell Us the Answer?” Teaching Historical Thinking in Grades 7-12. Portland, ME: Stenhouse Publishers, 2017; Podsakoff, Philip M. et al. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies.” Journal of Applied Psychology 88 (October 2003): 879-903; Stereotypes and Biased Language. The Writing Lab and The OWL. Purdue University; Bias in Survey Sampling. Stat Trek Online Tutorials; What is Availability Bias? InnovateUs.net.
The CRAPP Test
This stands for Currency, Relevance, Accuracy, Authority, and Purpose. It is an apronym developed within the field of librarianship as short-hand for remembering the essential actions associated with effectively assessing the usefulness of a source in relation to the research problem you are investigating. Each word relates to a set of questions you should ask yourself when determining the validity of a source. These are:
As described above, a thorough evaluation of sources can encompass more than this basic model. Nevertheless, if you remember anything about how to evaluate a source as you conduct a literature review, remember this approach. The CCRAP Test applied to any source produced in any format [e.g., text, online, statistical, multimedia].
CRAPP Test.net; Evaluating Information: Applying the CRAPP Test, Meriam Library. California State University, Chico; Esparrago-Kalidas, Adeva Jane. "The Effectiveness of CRAAP Test in Evaluating Credibility of Sources." International Journal of TESOL & Education 1 (2021): 1-14; Liu, Grace. "Moving Up the Ladder of Source Assessment: Expanding the CRAAP Test with Critical Thinking and Metacognition." College & Research Libraries News 82 (2021): 75; Muis, Krista Renee, Courtney A. Denton, and Adam Dubé. "Identifying CRAAP on the Internet: A Source Evaluation Intervention." Advances in Social Sciences Research Journal 9 (July 2022): 239-265.