Organizing Your Social Sciences Research Paper

Evaluating the authority, usefulness, and reliability of resources is a crucial step in conducting a review of the literature that effectively covers pertinent research and, thereby, demonstrates to the reader that you know what you're talking about. The process of evaluating scholarly materials also enhances your general skills and ability to:

1. Seek out alternative points of view and differing perspectives,
2. Identify possible bias in the work of others,
3. Distinguish between fact, fiction, and opinion,
4. Develop and strengthen your ability to distinguish between relevant and non-relevant content,
5. Draw cogent, well thought out conclusions, and
6. Synthesize information, extracting meaning through a deliberate process of interpretation and analysis.

Taken together, these benefits help promote critical thinking skills applied to doing good research. This includes rigorously questioning ideas and assumptions, promoting rational, open-minded, clear and informed reasoning, and supporting self-awareness in the use of reliable information for analyzing problems.

Black, Thomas R. Evaluating Social Science Research: An Introduction. London: Sage, 1993; Singh, Rajesh, and Shailendra Kumar. "Information Evaluation Skills of Social Science Researchers: An Assessment." Annals of Library and Information Studies 67 (2021): 222-230; Critical Thinking. City Centre Libraries, Technological University Dublin.

The act of thinking critically about the validity and reliability of a research resource generally involves asking yourself a series of questions about the quality of both the item and the content of that item.

Evaluating the Source

Inquiring about the Author
What are the author's credentials, such as, institutional affiliation [where he or she works], educational background, past writings, or experience? Is the book or article written on a topic in the author's area of expertise? Has your instructor mentioned this author? Have you seen the author's name cited in other sources or bibliographies? Is the author associated with a reputable institution or organization? What are the basic values or goals of that organization or institution?

Inquiring about the Date of Publication
When was the source published? Is the source current or out-of-date for your topic?

Inquiring about the Edition or Revision
Is this a first edition of this publication or not? Further editions usually indicate a source has been revised and updated to reflect changes in knowledge, to include prior omissions, and to better harmonize the contents with the intended needs of its readers. If you are using a web source, do the pages indicate last revision dates?

Inquiring about the Publisher
Note the publisher. If the source is published by a university press, it is likely to be scholarly. Although the fact that a publisher is reputable does not necessarily guarantee quality, it does show that the publisher has a high regard for the source being published [their reputation as an academic publisher relies on it].

Inquiring about the Title of Journal
Is this a scholarly or a popular journal? This distinction is important because it indicates different levels of complexity in conveying ideas and the intended readership.

Evaluating the Content

Intended Audience
What type of audience is the author addressing? Is the publication aimed at a specialized or a general audience? Is this source too elementary, too technical, too advanced, or just right for your needs?

Objectivity
Is the information covered considered to be fact, opinion, or propaganda? It is not always easy to separate fact from opinion. Facts can usually be verified; opinions, though they may be based on factual information, evolve from the interpretation of facts. Does the information appear to be valid and well-researched, or is it questionable and unsupported by evidence? Note errors or omissions. Are the ideas and arguments advanced more or less in line with other works you have read on the same topic?

Coverage
Does the work update or clarify prior knowledge, substantiate other materials you have read, or add new information? Does it extensively or only marginally cover your topic? Does it provide a balanced perspective? If the item in question does not meet this criteria, you should review enough sources to obtain a variety of viewpoints.

Writing Style
Is the publication organized logically? Are the main points clearly presented? Do you find the text easy to read, or is it stilted or choppy? Is the author's argument repetitive?

Evaluative Reviews
In the case of books, locate critical reviews of the work in a database such as ProQuest Multiple. Is the review positive? Is the book under review considered a valuable contribution to the field? Do reviewers agree on the value or attributes of the book or are there strong differences of opinion? Does the reviewer mention other books that might be better? If so, locate these sources for more information on your topic.

Brand-Gruwela, Saskia and Marc Stadtlerb. “Solving Information-based Problems: Evaluating Sources and Information.” Learning and Instruction 2 (April 2011): 175-179; Barzilaia, Sarit and Anat Zohara. “Epistemic Thinking in Action: Evaluating and Integrating Online Sources.” Cognition and Instruction 30 (2012): 39-85; Critical Thinking. Academic Skills Centre. University of Canberra; Evaluating Sources. Lakeland Library Research Guides. Lakeland Community College; Evaluating Sources. Writing@CSU. Colorado State University; Evaluating Print Sources. The Writing Center. University of North Carolina; Evaluation During Reading. The Writing Lab and The OWL. Purdue University; Walraven, Amber, Saskia Brand-Gruwel, and Henny P.A. Boshuizen. “How Students Evaluate Information and Sources When Searching the World Wide Web for Information.” Computers and Education 52 (January 2009): 234–246

Web Content Requires Additional Methods of Evaluation

A report from the Stanford University Graduate School of Education found that students evaluating information that flows across social media channels or retrieved from online search engines like Google or Bing, have difficulty distinguishing advertisements from news articles or how to identity where the content came from. In general, the principles that guide your evaluation of print materials are the same that apply to evaluating online resources. However, unlike print materials that have certain features that help determine their scholarly integrity, the interactive and multimedia dynamics of online sources requires additional attention to the content in order to obtain confidence that what you are viewing is valid and credible.

Additional things to look for when considering using an online resource:

• Source of the content is stated -- determine whether the content is original or borrowed, quoted, or imported from elsewhere. Note that content imported from another source via RSS feed can be difficult to identify, as this material can be incorporated into other content on the page without being appropriately labeled.
• Don't be fooled by an attractive, professional-looking presentation -- just because a site looks professional doesn't mean that it is. However, poorly organized web page designs or poorly written content is easy to recognize and can be a signal that you should carefully scrutinize the site's content.
• Site is currently being maintained -- check for last posting dates or last revised dates. Most scholarly websites show a date when the content was last posted or revised. Note that, if no date is indicated, this does not mean its content is invalid. However, it may indicate that the content is out-of-date and does not reflect current information about the topic.
• Links are relevant and appropriate, and are in working order -- a site with a lot of broken links is an indication of neglect and out-of-date content.
• Clearly states authorship -- if a site is produced anonymously, you cannot verify the legitimacy of its creator. Note that an author of a site can be either be a person or an organization.
• The site includes contact information -- if you have questions about the site, contact information is an important indicator that the site is well-maintained.
• Domain location in the site address (URL) is relevant to the focus of the material [e.g., .edu for educational or research materials; .org for non-profit organizations; .gov for government sites; .com for business sites]. Note that the domain is not necessarily a primary indicator of site content. For example, some authors post their content on blog or wiki platforms hosted by companies with .com addresses. Also note that the tilde (~) in the URL usually indicates a personal page.

Evaluating Internet Information. Online Library Learning Center. University of Georgia; Evaluating Internet Sources: A Library Resource Guide. Olsen Library. Northern Michigan University; Evaluating Sources. Writing@CSU. Colorado State University; Evaluating Web Sites. Teaching and Learning Services, University of Maryland Libraries; Ostenson, Jonathan. “Skeptics on the Internet: Teaching Students to Read Critically.” The English Journal 98 (May, 2009): 54-59; Stanford History Education Group. "Evaluating Information: The Cornerstone of Civic Online Reasoning." Stanford, CA: Graduate School of Education, 2016; Writing from Sources: Evaluating Web Sources. The Reading/Writing Center. Hunter College.

Detecting Bias

Bias, whether done intentionally or not, occurs when a statement reflects a partiality, preference, or prejudice for or against an object, person, place, or idea. Listed below are problems to look for when determining if the source is biased.

1. Availability Bias -- this is a tendency for people to overestimate probabilities of events related to memorable or dramatic occurrences [e.g., after 9/11, people took vacation by traveling by car rather than airplane even though, statistically, car travel is much more dangerous]. This form of bias in a research study can take the form of an example used to support author’s argument or the design a case study focused around a particular event. Unless the purpose of the study is to illuminate new understanding around a memorable or dramatic occurrence, be critical of studies that use this type of measurement to examine a research problem. A seemingly mundane or uneventful occurrence can be just as valid in developing solutions to a problem or advancing new knowledge.
2. Distortion or Stretching of the Facts -- this refers to the act of making issues, problems, or arguments appear more extreme by using misinformation or exaggerated and/or imprecise language to describe research outcomes [e.g., “Everyone agreed the policy was a complete disaster.” Who's everyone? How was data gathered to come to this conclusion? And, how does one specifically define something as a "disaster"? Is there sufficient evidence to support such a broad statement?]. Look for declarative statements that lack appropriate reference to supporting evidence or are follow up with detailed analysis.
3. Flawed Research Design -- bias can enter the narrative as a result of a poorly designed study; this may include a claim or generalization about the findings based upon too small a sample, manipulating statistics, omitting contrary conclusions from other studies, or failing to recognize negative results [results that do not support the hypothesis].
4. Lack of Citations -- it is acceptable to issue a broad declarative statement if it is clearly supported and linked to evidence from your study [e.g., "Testimony during Congressional hearings shows that the Department of Education is reluctant to act so teachers must do so"]. This problem refers to statements or information presented as fact that does not include proper citation to a source or to sources that support the researcher's position, or that are not statements explicitly framed as the author's opinion.
5. Misquoting a Source -- this is when an author rewords, paraphrases, or manipulates a statement, the information about a source is incomplete, or a quote is presented in such a way that it misleads or conveys a false impression. This is important when paraphrasing another author. If you cannot adequate summarize a specific statement, finding, or recommendation, use a direct quote to avoid any ambiguity.
6. Persuasive or Inflammatory Language -- using words and phrases intended to elicit a positive or negative response from the reader or that leads the reader to arrive at a specific conclusion [e.g., referring to one group in an armed conflict as “terrorists” and the other group as “peace-loving”].
7. Selective Facts -- taking information out of context or selectively choosing information that only supports the argument while omitting the overall context or vital supporting evidence.
8. Statistical Survey Bias -- this can take several forms so, if data is presented in a study that was gathered by the author(s), examine it critically for the following possible biases:
   • Measurement Error: this results from problems with the process by which data was gathered, such as, the use of leading questions that influence the response rate or that are biased toward what respondents believe is socially desirable because most people want to present themselves favorably. The only way to assess bias in these cases is to have access to the survey instrument used to gather data.
   • Sample Size: increasing the number of a sample, for example the number of people interviewed, does not necessarily decrease bias, but look to see if the sample used is representative of the population under study to ensure that any generalizations or conclusions from the interpretation of the data is valid.
   • Undercoverage: this refers to the method of data gathering that is a result of non-response to a survey because some subjects do not have the opportunity to participate. In looking at data, be sure to understand the percentage of non-responses to a survey or groups of people who were not included.
   • Voluntary Response: this bias occurs when respondents to a survey are self-selected, resulting in an overrepresentation of individuals who have strong opinions [e.g., data from a radio call-in show]. Be an especially critical reader of web-based surveys about controversial topics if the author(s) have not indicated how they interpreted thew data from voluntary surveys.

NOTE:  The act of determining bias in scholarly research is also an act of constant self-reflection. Everyone has biases. Therefore, it is important that you minimize the influence of your own biases by approaching the assessment of another person's research introspectively and with a degree of self-awareness.

ANOTHER NOTE:  The idea of bias often carries negative connotations, even though the meaning of the term is not defined in that context. Be sure when you are evaluating a source that you do not automatically reject it as invalid if you detect bias. One way to do this is to substitute the idea of bias with the idea of perspective. Ask yourself, what type of perspective does this source bring to the investigation of the research problem? As Lesh points out*, the goal of research is to engage with multiple sources for the purpose of acquiring multiple perspectives about the topic. As long as the source is rooted in fact-based evidence, you should not reject it as being biased, but rather, consider it as a potential source of perspective about the research problem.

* Lesh, Bruce A. “Why Won’t You Just Tell Us the Answer?” Teaching Historical Thinking in Grades 7-12. Portland, ME: Stenhouse Publishers, 2017.

"Availability Bias, Source Bias, and Publication Bias in Meta-Analysis." In Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. 3rd Edition. (London: SAGE Publications, 2015), pp. 513-551; "Bias." In Key Concepts in Social Research. Geoff Payne and Judy Payne. (London: SAGE Publications, 2004), pp. 28-31; Evaluating Sources. Lakeland Library Research Guides. Lakeland Community College; Fischer, Fritz. "Teaching Trump in the History Classroom." Journal of American History 108 (March 2022): 772-778; Podsakoff, Philip M. et al. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies.” Journal of Applied Psychology 88 (October 2003): 879-903; Stereotypes and Biased Language. The Writing Lab and The OWL. Purdue University; Bias in Survey Sampling. Stat Trek Online Tutorials; What is Availability Bias? InnovateUs.net.

The CRAPP Test

This stands for Currency, Relevance, Accuracy, Authority, and Purpose. It is an apronym developed within the field of librarianship as short-hand for remembering the essential actions associated with effectively assessing the usefulness of a source in relation to the research problem you are investigating. Each word relates to a set of questions you should ask yourself when determining the validity of a source. These are:

• Currency --  relates to the timeliness of the information.
  • When was the information published or posted online?
  • Has the information been revised or updated?
  • Does your topic require current information, or will older sources also support your study?
  • Are the links functional?
• Relevance -- the importance of the information in relation to your research needs.
  • Does the information relate to your topic or address your research question(s)?
  • Who is the intended audience?
  • Is the information at an appropriate level (i.e., not too elementary or advanced for your needs)?
  • Have you looked at a variety of sources before determining this is one you will use?
  • Would you be comfortable citing this source in your research paper?
• Accuracy -- relates to identifying and verifying the source of the information.
  • Who is the author, publisher, source, or sponsor?
  • What are the author's credentials or organizational affiliations?
  • Is there evidence of the author's qualifications to write about the topic?
  • Is there contact information, such as a publisher or email address?
  • Des the URL reveal anything about the author or source [e.g., .com .edu .gov .org .net, etc.]?
• Authority -- relates to the reliability, truthfulness, and correctness of the content.
  • Where does the information come from?
  • Is the information supported by evidence?
  • Has the information been reviewed by an editor or peer reviewed?
  • Can you verify any of the information in another source or from personal knowledge?
  • Does the language or narrative tone appear objective and unbiased?
  • Are there spelling, grammar, or typographical errors?
• Purpose -- relates to the reason the information exists.
  • What is the purpose of the information, i.e., is it to inform, teach, sell, entertain, or persuade?
  • Do the author(s) explain why the information has been studied?
  • Is the information fact, opinion, or propaganda?
  • Does the point of view appear objective and impartial?
  • Are there political, ideological, cultural, religious, institutional, or personal biases?

As described above, a thorough evaluation of sources can encompass more than this basic model. Nevertheless, if you remember anything about how to evaluate a source as you conduct a literature review, remember this approach. The CCRAP Test applied to any source produced in any format [e.g., text, online, statistical, multimedia].

CRAPP Test.net; Evaluating Information: Applying the CRAPP Test, Meriam Library. California State University, Chico; Esparrago-Kalidas, Adeva Jane. "The Effectiveness of CRAAP Test in Evaluating Credibility of Sources." International Journal of TESOL & Education 1 (2021): 1-14;  Liu, Grace. "Moving Up the Ladder of Source Assessment: Expanding the CRAAP Test with Critical Thinking and Metacognition." College & Research Libraries News 82 (2021): 75; Muis, Krista Renee, Courtney A. Denton, and Adam Dubé. "Identifying CRAAP on the Internet: A Source Evaluation Intervention." Advances in Social Sciences Research Journal 9 (July 2022): 239-265.