Organizing Your Social Sciences Research Paper

Citation tracking [a.k.a., citation analysis, citation tracing, or cited reference searching] refers to a deliberate, systematic method of measuring and evaluating the impact of research studies over time by counting the number of times that an author or publication has been cited in other works. Tracking citations helps identify leading scholars in a particular discipline and the impact of a particular study based upon an analysis of who has cited a particular study, how often a specific study has been cited by others over time, and by determining what disciplines are represented, either by the scope of the publication or by the author's affiliation, based on those subsequent citations.

Cited Reference Searching. University Libraries Research Guides, University of Washington; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Mandal, Nitai. "An Approach Towards Citation Tracking: Special Reference to Academic and Research Libraries." International Journal of Information Movement 2 (December 2017): 148-152.

Reasons to Track Citations as Part of Your Review of the Literature

• It can be an effective way of using a frequently cited "landmark" or groundbreaking article to find more recent, related publications that cite the original work.
• It can be an effective strategy of identifying leading scholars in a particular area of study either by how frequently their work has been cited or by the number of scholars who have subsequently cited the author's original work.
• It can be a useful means for evaluating a study's impact within a particular discipline based upon the number of times a research study has been cited by others. A frequently cited study may indicate that the research findings are unique, groundbreaking, insightful, or in some way, seen by scholars as problematic or controversial.
• It can be an efficient way to locate studies that critique or challenge groundbreaking research, which is important if you are seeking research to support your attempt to challenge long-held assumptions or to offer evidence that existing studies do not adequately address the research problem.
• It can be an effective means of determining the interdisciplinary impact of a particular study because you can identify the number of subsequent citations in publications by scholars in other areas of study. Research that has been cited in a variety of disciplines is also a strong indication of the research study's overall impact throughout the social sciences [and perhaps beyond].

NOTE: A common assignment in social sciences classes is to assign each student the responsibility for summarizing and discussing a journal article from the required readings of the class syllabus. Before you lead the class discussion of the study, copy and paste the article you are discussing into Google Scholar. This will likely reveal "Cited by" references [e.g., Cited by 102] to other works that have subsequently cited the original study. In so doing, you can add during your in-class discussion of the article a description of how other scholars have used it to support their own research and, by extension, the ways in which the original research influenced future studies of the topic. This is an effective way to understand the impact of prior research, and, taking this extra step as part of summarizing and discussing an assigned reading will impress your professor!

Keep in mind the following points, however, when using methods for tracking citations to expand the scope of the literature you want to review:

• Authors do not always use the same name throughout their careers [e.g., Jane Anne Smith or Jane A. Smith] so be sure you work from a complete and accurate list of an author's publications. A thorough literature review of an author should reveal any variations in their name. To this point,...
• The Web of Science citation database uses the APA [American Psychological Association] style for citing authors [last name and first initial only], so a J Smith could be John, Jeff, Jane, Julie, Jason, etc. Smith. Be sure to truncate the initial  by adding an asterisk [e.g., J*] to see a more complete list of authors, then locate a record on a topic you know the author writes about and click on that author to exclude articles written by other J Smith's. Fortunately, the database indexes more than the first author of a paper so, if a second or third author has an uncommon name, you can search the unusual name to locate the article and who has cited it instead of using an author's more common name.
• Citation services are primarily based on existing journal literature. If the author is cited in books, organizational research reports, government documents, or non-scholarly publications, the usefulness of your citation analysis is limited. In addition, citation databases such as Web of Science have limited coverage of articles published in scholarly open-access journals [peer-reviewed journals published freely online], although this problem is slowly improving. In this case, be sure to check the "cited by" references in Google Scholar as it often includes citations found in other publications besides scholarly journals.
• Google Scholar and most library research databases do an inadequate job citing to non-English language sources. English is generally considered the universal language of scholarly research, which some argue is a remnant of colonialism. As a result, this privileges English-born publications at the expense of research published in other languages, particularly if the study is written using non-Roman characters [e.g., Korean, Hindi, Russian, etc.]. This deficiency is slowly improving as private database vendors and, in particular Google Scholar, begin to include more non-English language studies in their systems. Nevertheless, always keep in mind that the number of citations to an original study may not represent its true impact among scholars in non-English speaking regions of the world, particularly if the study focuses on a non-English-speaking region or country. If needed, contact a librarian for help identifying studies published in languages other than English. A thorough review of the literature should include studies published in other languages if you are able to read those languages and the research directly relates to the problem you are investigating.

Bakkalbasi, Nisa. “Three Options for Citation Tracking: Google Scholar, Scopus and Web of Science.” Biomedical Digital Libraries 3 (2006): https://www.bio-diglib.com/content/pdf/1742-5581-3-7.pdf; Kirkpatrick, Andy. "English as the International Language of Scholarship: Implications for the Dissemination of 'Local’ Knowledge." In English as an International Language: Perspectives and Pedagogical Issues. Bu Sharifian, Farzad. (Buffalo, NY: Multilingual Matters, 2009), pp: 254-270; Lawrence, D. J. “Journal Citation Tracking and Journal Indexing.” Journal of Manipulative and Physiological Therapeutics 15 (September 1992): 415-417; Kloda, Lorie A. "Use Google Scholar, Scopus and Web of Science for Comprehensive Citation Tracking." Evidence Based Library and Information Practice 2 (2007): 87-90; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Weisbard, Phyllis Holman. “Citation Tracking: Citings and Sightings.” Feminist Collections: A Quarterly of Women's Studies Resources 32 (Winter 2011): 21-25;

Records found while searching Google Scholar and many library databases often include a "cited by" reference followed by a number [e.g., Cited by 143]. This number indicates, in general, how many times a study has been subsequently cited by other authors in other publications [I say "in general" because the total may include duplicate records, such as, an author citing their own work or citations to the same study in multiple types of publications].

However, the total number of times a journal article or other publication has been cited after it was published is more nuanced than just the number count. When reviewing "cited by" references as part of your review of the literature, consider the following interpretive principles that will help you determine if the source should be examined more carefully.

• An item that is cited many times indicates impact, but it does not indicate value. Citations are value-neutral. A study may be cited frequently because it added important new knowledge to the understanding of a research problem. However, a study could also be cited frequently because subsequent studies found the original study to be poorly designed or the author employed an imprecise methodological approach or the conclusions were speculative and perhaps misleading [also see Another Citing Tip text box below]. Use the date limitation function to review a sample of recent studies that have referenced the work and assess how the original source is currently being described. Based on this, determine how the frequently cited source may be relevant to your own research. Note that if subsequent research shows the original study to be flawed, you can reference the study in your paper as an example of why further research is needed.
• A source that is frequently cited is a subjective assessment of impact, not an objective measurement or formal statistical calculation. Should a study published ten years ago with 800 cited by references be considered a frequently cited source? How about a study published ten years ago that has over 2,000 cited by references? Is that source better than the other in terms of assessing whether it should be reviewed in detail? What about a source with only 120 cited by references? There are a variety of factors that contribute to whether a source has been cited frequently or not. Ultimately, the number of cited by references is relevant to you only in relation to the research problem being investigated. If the study directly informs your understanding of the topic, you should review the cited by references. If there are a manageable number, review them all; if there is a large number of references, focus on reviewing the most recently published sources [e.g. last three to five years].
• Once published, a recently published study can become cited by other authors very quickly. This is particularly the case if the study was published in a core research journal read by almost all the leading scholars of the discipline, it was published in a source read by researchers from a variety of disciplines, the study was written by a well-known and preeminent scholar, or the research has been written about in national newspapers and magazines, discussed during television news programs, or discussed and shared on social media sites. A way to discern impact over time is to count the number of recent citations to a study [e.g., the last five years]. If the study has been cited only a few times, this likely indicate that the study's impact is waning and scholars have moved on.
• A study may be considered foundational or groundbreaking if it continues to be cited frequently in the most recently published literature on the topic. A source published many years ago that continues to cited can indicate the research was the first to advance new knowledge and, therefore, the study either contributed significantly to understanding the research problem or the findings' ability to inform practice was particularly important or innovative. In most cases, there is an expectation that a study interpreted to be foundational or groundbreaking should be cited as the basis for supporting your overall review and assessment of prior research or to document when a research problem first emerged.
• Review the cited by references of relevant resources to determine a study's impact across disciplines. Cross-disciplinary cited by references can be an important indicator of a study's overall impact if it has been referenced in a variety of different investigative contexts or domains of applied practice. Consider reviewing these sources in more detail in support of your own review of the literature. Do this by comparing the disciplinary focus of the original source in relation to the disciplinary focus of the cited by references. For example, an article originally published in a social work journal that is cited in journals from sociology, education, and psychology demonstrates impact beyond the field of social work.
• Examine whether the source is referenced alone or always grouped with other sources. This act of interpreting impact can be challenging, but, in general, a study that is consistently cited with other sources, though rarely cited as a stand-alone study, may indicate that the research is not unique or distinctive in a way that stands out from the overall domain of prior research about a topic. For example, you find that Smith's original study is always referenced along with other authors doing similar work. This should not be viewed necessarily as negative because many scholars may have investigated a specific research problem. However, in interpreting the relevance of a particular source in relation to your own research, studies that have never been cited as a stand-alone source of prior research can be an additional way of determining if they would add value to your own evaluation of prior research.
• Cited by references listed in the first few years after the original study was published can add important insights. For example, the earliest studies to cite an original work can highlight the initial interests and potential biases of scholars at the time. In addition, they can reflect a growing awareness of the need to critically examine the research problem, they can highlight the application of prevailing theories and ideas at the time, and, in some cases, the earliest published studies citing the original source can help illuminate key intellectual struggles among scholars as they attempt to make sense of the emerging problem. All or any of these factors can help contextualize the way in which you use prior research to write about the background and history of a topic in the introduction or literature review sections of your paper.

Bornmann, Lutz and Hans‐Dieter Daniel. "What Do Citation Counts Measure? A Review of Studies on Citing Behavior." Journal of Documentation 64 (2008): 45-80; Hou, Jianhua and Da Ma. "How the High-Impact Papers Formed? A Study Using Data from Social Media and Citation." Scientometrics 125 (2020): 2597-2615; Leimu, Roosa and Julia Koricheva. "What Determines the Citation Frequency of Ecological Papers?" Trends in Ecology & Evolution 20 (2005): 28-32; Moed, Henk F. Citation Analysis in Research Evaluation. Information Science and Knowledge Management, vol. 9. Dordrecht, The Netherlands: Springer Science & Business Media, 2005; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641; Robinson, Larry M. and Roy Adler. "Measuring the Impact of Marketing Scholars and Institutions: An Analysis of Citation Frequency." Journal of the Academy of Marketing Science 9 (1981): 147-162.

Searching Cited By References Without Getting Lost

Tracking cited by references can be an effective strategy for identifying relevant, more recently published literature about a specific research problem. However, citation tracking is not a linear process, but rather, an iterative process comprising of multi-layered pathways of discovery. Not only can you identify citations that directly cite the original study, but those studies may also have cited by references that can be reviewed, then those studies can have cited by references that can be searched, and so on. While this process of traversing the literature by reviewing multiple layers of cited by references can be useful in discovering new sources that perhaps would have been hidden otherwise, one can easily get lost in the process. This is particularly true with following cited by references in Google Scholar because the content is so immense. A good strategy for not getting lost is to note a promising record that cites a source that has cited the original source so you can go back and review the cited by references of that study without worrying about losing track of where you are in the initial review process.

Hinde, Sebastian and Eldon Spackman. "Bidirectional Citation Searching to Completion: An Exploration of Literature Searching Methods." Pharmacoeconomics 33, (2015): 5-11; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641.

The "Replication Crisis" in the Social Sciences and Searching Cited By References

The replication crisis in academe refers to research showing that findings published in top journals often cannot be replicated in subsequent studies. The ability to replicate research is a core principle of the scientific method of discovery and verification. Replicating the research of prior studies increases the probability and, by extension, the confidence one has that a given finding is valid in a real world setting or is reliably applicable to professional practice. The problem, as noted by Prochazka, Fedoseeva, and Houdek (2021) is that, “…the replication crisis in the social sciences has showed that many of the published effects has been found by chance or due to the specific conditions of a study…Therefore, an effect found in a single study should be viewed with caution.”

But that’s not all. Nonreplicable studies tend to be cited more frequently than studies that have been successfully replicated. One reason for this, according to researchers, is that authors are often motivated to use an unorthodox approaches to studying a problem. This is because methodological unorthodoxy is often rewarded [intentionally or not] by institutions of higher education in the faculty promotion process and by journal publishers who want to be seen as on the cutting edge of research. Replicating another person’s research doesn’t garner the same attention towards obtaining prestigious grant funding or gaining tenure for the author as a “unique” study. As a result, this tendency can distort the total number count of citations in some studies, making it more difficult to discern a truly groundbreaking study from a “failed” nonreplicable study that has been cited simply because it’s found to be unique in some way.

To address this issue when searching cited by references, review studies that cite the original study shortly after it was published. Are authors attempting to replicate the research? If so, are they doing so successfully and building on this research or are they simply citing the work in their literature review section. In the end however, don't get too caught up in this "crisis"; always keep in mind that the purpose of citation tracking is to build a body of knowledge that supports investigating the research problem.

Blaszczynski, Alex and Sally M. Gainsbury. "Editor’s Note: Replication Crisis in the Social Sciences." International Gambling Studies 19 (2019): 359-361; Pashler, Harold, and Eric–Jan Wagenmakers. "Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?." Perspectives on Psychological Science 7 (2012): 528-530; Pridemore, William Alex, Matthew C. Makel, and Jonathan A. Plucker. "Replication in Criminology and the Social Sciences." Annual Review of Criminology 1 (2018): 19-38; Prochazka, Jakub, Yulia Fedoseeva, and Petr Houdek. "A Field Experiment on Dishonesty: A Registered Replication of Azar et al.(2013)." Journal of Behavioral and Experimental Economics 90 (2021): 101617; "Science’s New Replication Crisis: Research That Is Less Likely to Be True Is Cited More." SciTechDaily [University of California, San Diego, May 21, 2021]; Serra-Garcia, Marta, and Uri Gneezy. "Nonreplicable Publications are Cited More than Replicable Ones." Science Advances 7 (2021): 1-7.