Citation tracking refers to a systematic method of measuring and evaluating the impact of research studies over time. Tracking citations helps identify leading scholars in a particular discipline based upon an analysis of who has cited a particular study, how often a specific study has been cited by others, and by determining what disciplines are represented, either by the scope of the publication or by the author's affiliation, based on those subsequent citations.
Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Mandal, Nitai. "An Approach Towards Citation Tracking: Special Reference to Academic and Research Libraries." International Journal of Information Movement 2 (December 2017): 148-152.
Citation tracking can facilitate the review and evaluation of literature pertinent to a research problem for the following reasons:
Keep in mind the following points when tracking citations:
Bakkalbasi, Nisa. “Three Options for Citation Tracking: Google Scholar, Scopus and Web of Science.” Biomedical Digital Libraries 3 (2006): https://www.bio-diglib.com/content/pdf/1742-5581-3-7.pdf; Kirkpatrick, Andy. "English as the International Language of Scholarship: Implications for the Dissemination of 'Local’ Knowledge." In English as an International Language: Perspectives and Pedagogical Issues. Bu Sharifian, Farzad. (Buffalo, NY: Multilingual Matters, 2009), pp: 254-270; Lawrence, D. J. “Journal Citation Tracking and Journal Indexing.” Journal of Manipulative and Physiological Therapeutics 15 (September 1992): 415-417; Kloda, Lorie A. "Use Google Scholar, Scopus and Web of Science for Comprehensive Citation Tracking." Evidence Based Library and Information Practice 2 (2007): 87-90; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Weisbard, Phyllis Holman. “Citation Tracking: Citings and Sightings.” Feminist Collections: A Quarterly of Women's Studies Resources 32 (Winter 2011): 21-25;
Records found while searching Google Scholar and many library databases often include a "cited by" reference followed by a number [e.g., Cited by 143]. This number indicates, in general, how many times a study has been subsequently cited by other authors in other publications [I say "in general" because the total may include duplicate records, such as, an author citing their own work or citations to same study in multiple types of publications].
However, the total number of times a journal article or other publication has been cited after it was published is more nuanced than just the number count. When reviewing "cited by" references as part of your review of the literature, consider the following interpretive principles that will help you determine if the source should be examined more carefully.
Bornmann, Lutz and Hans‐Dieter Daniel. "What Do Citation Counts Measure? A Review of Studies on Citing Behavior." Journal of Documentation 64 (2008): 45-80; Hou, Jianhua and Da Ma. "How the High-Impact Papers Formed? A Study Using Data from Social Media and Citation." Scientometrics 125 (2020): 2597-2615; Leimu, Roosa and Julia Koricheva. "What Determines the Citation Frequency of Ecological Papers?" Trends in Ecology & Evolution 20 (2005): 28-32; Moed, Henk F. Citation Analysis in Research Evaluation. Information Science and Knowledge Management, vol. 9. Dordrecht, The Netherlands: Springer Science & Business Media, 2005; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641; Robinson, Larry M. and Roy Adler. "Measuring the Impact of Marketing Scholars and Institutions: An Analysis of Citation Frequency." Journal of the Academy of Marketing Science 9 (1981): 147-162.
Searching Cited By References Without Getting Lost
Tracking cited by references can be an effective strategy for identifying relevant, more recently published literature about a specific research problem. However, citation tracking is not a linear process, but rather, an iterative process comprising of multi-layered paths of discovery. Not only can you identify citations that directly cite the original study, but those studies may also have cited by references that can be reviewed, then those studies can have cited by references that can be searched, and so on. While this process of traveling through the literature by reviewing multiple layers of cited by references can be useful in discovering new sources that perhaps would have been hidden otherwise, one can easily get lost in the process. This is particularly true with following cited by references in Google Scholar because the content is so immense. A good strategy for not getting lost is to note a promising record that cites a source that has cited the original source so you can go back and review the cited by references of that study without worrying about losing track of where you are in the initial review process.
Hinde, Sebastian and Eldon Spackman. "Bidirectional Citation Searching to Completion: An Exploration of Literature Searching Methods." Pharmacoeconomics 33, (2015): 5-11; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641.
The "Replication Crisis" in the Social Sciences and Searching Cited By References
The replication crisis in academe refers to research showing that findings published in top journals often cannot be replicated in subsequent studies. The ability to replicate research is a core principle of the scientific method of discovery and verification. Replicating the research of prior studies increases the probability and, by extension, the confidence one has that a given finding is valid in a real world setting or is reliably applicable to professional practice. The problem, as noted by Prochazka, Fedoseeva, and Houdek (2021) is that, “…the replication crisis in the social sciences has showed that many of the published effects has been found by chance or due to the specific conditions of a study…Therefore, an effect found in a single study should be viewed with caution.”
But that’s not all. Nonreplicable studies tend to be cited more frequently than studies that have been successfully replicated. One reason for this, according to researchers, is that authors are often motivated to use an unorthodox approaches to studying a problem. This is because methodological unorthodoxy is often rewarded [intentionally or not] by institutions of higher education and journal publishers. Replicating another person’s research doesn’t garner the same attention towards obtaining prestigious grant funding or gaining tenure for the author as a “unique” study. As a result, this tendency can distort the total number count of citations in some studies, making it more difficult to discern a truly groundbreaking study from a “failed” nonreplicable study that has been cited simply because it’s found to be unique in some way. To address this issue when searching cited by references, review studies that cite the original study shortly after it was published. Are authors attempting to replicate the research? If so, are they doing so successfully and building on this research or are they simply citing the work in their literature review section. In the end however, don't get too caught up in this "crisis"; always keep in mind that the purpose of citation tracking is to build a body of knowledge that supports investigating the research problem.
Blaszczynski, Alex and Sally M. Gainsbury. "Editor’s Note: Replication Crisis in the Social Sciences." International Gambling Studies 19 (2019): 359-361; Pashler, Harold, and Eric–Jan Wagenmakers. "Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?." Perspectives on Psychological Science 7 (2012): 528-530; Pridemore, William Alex, Matthew C. Makel, and Jonathan A. Plucker. "Replication in Criminology and the Social Sciences." Annual Review of Criminology 1 (2018): 19-38; Prochazka, Jakub, Yulia Fedoseeva, and Petr Houdek. "A Field Experiment on Dishonesty: A Registered Replication of Azar et al.(2013)." Journal of Behavioral and Experimental Economics 90 (2021): 101617; "Science’s New Replication Crisis: Research That Is Less Likely to Be True Is Cited More." SciTechDaily [University of California, San Diego, May 21, 2021]; Serra-Garcia, Marta, and Uri Gneezy. "Nonreplicable Publications are Cited More than Replicable Ones." Science Advances 7 (2021): 1-7.