Organizing Your Social Sciences Research Paper

Citation tracking refers to a systematic method of measuring and evaluating the impact of research studies over time. Tracking citations helps identify leading scholars in a particular discipline based upon an analysis of who has cited a particular study, how often a specific study has been cited by others, and by determining what disciplines are represented, either by the scope of the publication or by the author's affiliation, based on those subsequent citations.

Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Mandal, Nitai. "An Approach Towards Citation Tracking: Special Reference to Academic and Research Libraries." International Journal of Information Movement 2 (December 2017): 148-152.

Citation tracking can facilitate the review and evaluation of literature pertinent to a research problem for the following reasons:

1. It can be an effective way of using a frequently cited "landmark" or groundbreaking article to find more recent, related articles that cite the original work.
2. It can be an effective strategy of identifying leading scholars in a particular area of study either by how frequently their work has been cited or by scholars who have subsequently cited the original work.
3. It can be a useful means for evaluating a study's impact within a particular discipline based upon the number of times a research study has been cited by others. A frequently cited study may indicate that the research findings are unique, groundbreaking, or, in some way, controversial.
4. It can be an effective way to locate studies that critique or challenge groundbreaking research, which is important if you are seeking research to support your attempt to challenge long-held assumptions or to offer evidence that existing studies do not adequately address the research problem.
5. It can be an effective means of determining the interdisciplinary value of a particular study because you can identify the number of subsequent citations in publications by scholars in other disciplines or fields of study. Studies that have been cited in a variety of disciplines is also a strong indication of the research study's overall impact throughout the social sciences [and perhaps beyond].

Keep in mind the following points when tracking citations:

• Authors do not always use the same name throughout their careers [e.g., Jane Anne Smith or Jane A. Smith] so be sure you work from a complete and accurate list of an author's publications. A thorough literature review of an author should reveal any variations in their name.
• The Web of Science citation database uses the APA [American Psychological Association] style for citing authors [last name and first initial only], so a J Smith could be John, Jeff, Jane, Julie, Jason, etc. Be sure to truncate the initial [adding an asterisk *] to see a more complete list of authors, then locate a record on a topic you know the author writes about and click on that author to exclude articles written by other J Smith's. Fortunately, the database indexes more than the first author of a paper so if a second or third author has an uncommon name, you can search the unusual name instead of using an author's common name.
• Citation services are primarily based on selected journal literature. If the author is cited in books, organizational research reports, government documents, or non-scholarly publications, the usefulness of your citation analysis is limited. In addition, citation databases such as Web of Science have limited coverage of articles published in scholarly open-access journals [peer-reviewed journals published freely on the web], although this problem is slowly improving. In this case, be sure to check the "cited by" references in Google Scholar as it often includes citations found in other publications besides scholarly journals.
• Google Scholar and most library research databases do an inadequate job citing to non-English language sources. English is generally considered the universal language of scholarly research. However, this privileges English-born publications at the expense of research published in other languages, particularly if the study is written using non-Roman characters [e.g., Korean, Hindi, Russian, etc.]. This deficiency is slowly getting better as private database vendors and, in particular Google Scholar, begin to include more non-English language studies in their systems. Nevertheless, always keep in mind that the number of citations to an original study may not represent its true impact among scholars in non-English speaking regions of the world. If needed, contact a librarian for help identifying foreign language studies that may cite the original work.

Bakkalbasi, Nisa. “Three Options for Citation Tracking: Google Scholar, Scopus and Web of Science.” Biomedical Digital Libraries 3 (2006): https://www.bio-diglib.com/content/pdf/1742-5581-3-7.pdf; Kirkpatrick, Andy. "English as the International Language of Scholarship: Implications for the Dissemination of 'Local’ Knowledge." In English as an International Language: Perspectives and Pedagogical Issues. Bu Sharifian, Farzad. (Buffalo, NY: Multilingual Matters, 2009), pp: 254-270; Lawrence, D. J. “Journal Citation Tracking and Journal Indexing.” Journal of Manipulative and Physiological Therapeutics 15 (September 1992): 415-417; Kloda, Lorie A. "Use Google Scholar, Scopus and Web of Science for Comprehensive Citation Tracking." Evidence Based Library and Information Practice 2 (2007): 87-90; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Weisbard, Phyllis Holman. “Citation Tracking: Citings and Sightings.” Feminist Collections: A Quarterly of Women's Studies Resources 32 (Winter 2011): 21-25;

Records found while searching Google Scholar and many library databases often include a "cited by" reference followed by a number [e.g., Cited by 143]. This number indicates, in general, how many times a study has been subsequently cited by other authors in other publications [I say "in general" because the total may include duplicate records, such as, an author citing their own work or citations to same study in multiple types of publications].

However, the total number of times a journal article or other publication has been cited after it was published is more nuanced than just the number count. When reviewing "cited by" references as part of your review of the literature, consider the following interpretive principles that will help you determine if the source should be examined more carefully.

• An item that is cited many times indicates impact, but it does not indicate value. Citations are value-neutral. A study may be cited frequently because it added important new knowledge to understanding the research problem. However, a study could also be cited frequently because subsequent research found the original study to be poorly designed or the author employed an imprecise methodological approach or the conclusions were speculative and perhaps misleading [also note Another Citing Tip below]. Use the date limitation function to review a sample of recent studies that have referenced the work and assess how the original source is currently being described. Based on this, determine how the frequently cited source may be relevant to your own research. Note that if subsequent research shows the original study to be flawed, you can reference the study in your paper as an example of why further research is needed.
• A source that is frequently cited is a subjective assessment of impact, not an objective measurement or formal statistical calculation. Should a study published ten years ago with 800 cited by references be considered a frequently cited source? How about a study published ten years ago that has over 2,000 cited by references? Is that source better than the other in terms of assessing whether it should be reviewed in detail? What about a source with only 120 cited by references? There are a variety of factors that contribute to whether a source has been cited frequently or not. Ultimately, the number of cited by references is relevant to you only in relation to the research problem being investigated. If the study directly informs your understanding of the topic, you should review the cited by references. If there are a manageable number, review them all; if there is a large number of references, focus on reviewing the most recently published sources.
• Once published, a recently published study can become cited by other authors very quickly. This is particularly the case if the study was published in a core research journal read by all the leading scholars of the discipline, it was published in a source read by researchers from a variety of disciplines, the study was written by a well-known and preeminent scholar, or the research has been written about in national newspapers and magazines, discussed on television news programs, or even talked about and shared on social media sites. A way to discern impact over time is to count the number of recent citations to a study [e.g., the last five years]. If the study has been cited only a few times, this likely indicate that the study's impact is waning and scholars have moved on. However,....
• A study may be considered foundational or groundbreaking if it continues to be cited frequently in the most recently published literature on the topic. A source published many years ago that authors continue to cite can indicate the research was the first to advance new knowledge and, therefore, the study either contributed significantly to understanding the research problem or the findings' ability to inform practice was particularly important or innovative. In most cases, there is an expectation that a study interpreted to be foundational or groundbreaking should be cited as the basis for supporting your overall review and assessment of prior research or to document when a research problem first emerged.
• Review the cited by references of relevant resources to determine a study's impact across disciplines. Cross-disciplinary cited by references can be an important indicator of a study's overall impact if it has been referenced in a variety of different investigative contexts or domains of applied practice. Consider reviewing these sources in more detail in support of your own review of the literature. Do this by comparing the disciplinary focus of the original source in relation to the disciplinary focus of the cited by references. For example, an article originally published in a social work journal that is cited in journals from sociology, education, and psychology demonstrates impact beyond the field of social work.
• Examine whether the source is referenced alone or always grouped with other sources. This act of interpretation can be challenging, but, in general, a study that is consistently cited with other sources, though rarely cited as a stand-alone study, may indicate that the research is not unique or distinctive in a way that stands out from the overall domain of prior research about the topic. For example, you find that Smith's original study is always referenced in relation to other authors doing similar work. This should not be viewed necessarily as negative because many scholars may have investigated a specific research problem. However, in interpreting the relevance of a particular source in relation to your own research, sources that have never been cited as a stand-alone source of prior research can be an additional way of determining their overall impact.
• Cited by references listed in the first few years after the original study was published can also add important insights. For example, the earliest studies to cite an original work can highlight the initial interests and potential biases of scholars at the time, they can reflect a growing awareness of the need to critically examine the research problem, they can highlight the application of prevailing theories and ideas at the time, and the earliest published studies citing the original source can help illuminate key intellectual struggles among scholars as they attempt to make sense of an emerging problem. All or any of these factors can help contextualize the way in which you use prior research to write about the background and history of a topic in the introduction or literature review sections of your paper.

Bornmann, Lutz and Hans‐Dieter Daniel. "What Do Citation Counts Measure? A Review of Studies on Citing Behavior." Journal of Documentation 64 (2008): 45-80; Hou, Jianhua and Da Ma. "How the High-Impact Papers Formed? A Study Using Data from Social Media and Citation." Scientometrics 125 (2020): 2597-2615; Leimu, Roosa and Julia Koricheva. "What Determines the Citation Frequency of Ecological Papers?" Trends in Ecology & Evolution 20 (2005): 28-32; Moed, Henk F. Citation Analysis in Research Evaluation. Information Science and Knowledge Management, vol. 9. Dordrecht, The Netherlands: Springer Science & Business Media, 2005; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641; Robinson, Larry M. and Roy Adler. "Measuring the Impact of Marketing Scholars and Institutions: An Analysis of Citation Frequency." Journal of the Academy of Marketing Science 9 (1981): 147-162.

Searching Cited By References Without Getting Lost

Tracking cited by references can be an effective strategy for identifying relevant, more recently published literature about a specific research problem. However, citation tracking is not a linear process, but rather, an iterative process comprising of multi-layered paths of discovery. Not only can you identify citations that directly cite the original study, but those studies may also have cited by references that can be reviewed, then those studies can have cited by references that can be searched, and so on. While this process of traveling through the literature by reviewing multiple layers of cited by references can be useful in discovering new sources that perhaps would have been hidden otherwise, one can easily get lost in the process. This is particularly true with following cited by references in Google Scholar because the content is so immense. A good strategy for not getting lost is to note a promising record that cites a source that has cited the original source so you can go back and review the cited by references of that study without worrying about losing track of where you are in the initial review process.

Hinde, Sebastian and Eldon Spackman. "Bidirectional Citation Searching to Completion: An Exploration of Literature Searching Methods." Pharmacoeconomics 33, (2015): 5-11; Mavodza, Judith. Citation Tracking in Academic Libraries: An Overview. Oxford, UK: Chandos Publishing, May 2016; Nicolaisen, Jeppe. "Citation Analysis." Annual Review of Information Science and Technology 41 (2007): 609-641.

The "Replication Crisis" in the Social Sciences and Searching Cited By References

The replication crisis in academe refers to research showing that findings published in top journals often cannot be replicated in subsequent studies. The ability to replicate research is a core principle of the scientific method of discovery and verification. Replicating the research of prior studies increases the probability and, by extension, the confidence one has that a given finding is valid in a real world setting or is reliably applicable to professional practice. The problem, as noted by Prochazka, Fedoseeva, and Houdek (2021) is that, “…the replication crisis in the social sciences has showed that many of the published effects has been found by chance or due to the specific conditions of a study…Therefore, an effect found in a single study should be viewed with caution.”

But that’s not all. Nonreplicable studies tend to be cited more frequently than studies that have been successfully replicated. One reason for this, according to researchers, is that authors are often motivated to use an unorthodox approaches to studying a problem. This is because methodological unorthodoxy is often rewarded [intentionally or not] by institutions of higher education and journal publishers. Replicating another person’s research doesn’t garner the same attention towards obtaining prestigious grant funding or gaining tenure for the author as a “unique” study. As a result, this tendency can distort the total number count of citations in some studies, making it more difficult to discern a truly groundbreaking study from a “failed” nonreplicable study that has been cited simply because it’s found to be unique in some way. To address this issue when searching cited by references, review studies that cite the original study shortly after it was published. Are authors attempting to replicate the research? If so, are they doing so successfully and building on this research or are they simply citing the work in their literature review section. In the end however, don't get too caught up in this "crisis"; always keep in mind that the purpose of citation tracking is to build a body of knowledge that supports investigating the research problem.

Blaszczynski, Alex and Sally M. Gainsbury. "Editor’s Note: Replication Crisis in the Social Sciences." International Gambling Studies 19 (2019): 359-361; Pashler, Harold, and Eric–Jan Wagenmakers. "Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?." Perspectives on Psychological Science 7 (2012): 528-530; Pridemore, William Alex, Matthew C. Makel, and Jonathan A. Plucker. "Replication in Criminology and the Social Sciences." Annual Review of Criminology 1 (2018): 19-38; Prochazka, Jakub, Yulia Fedoseeva, and Petr Houdek. "A Field Experiment on Dishonesty: A Registered Replication of Azar et al.(2013)." Journal of Behavioral and Experimental Economics 90 (2021): 101617; "Science’s New Replication Crisis: Research That Is Less Likely to Be True Is Cited More." SciTechDaily [University of California, San Diego, May 21, 2021]; Serra-Garcia, Marta, and Uri Gneezy. "Nonreplicable Publications are Cited More than Replicable Ones." Science Advances 7 (2021): 1-7.