Scholarly Impact Challenge: Day 2: Research Metrics

Join the Scholarly Impact Challenge to improve your research impact!

"Research metrics" are the methods and processes of counting and statistically analyzing information about research outputs. Metrics are commonly counted at the author level, article level, or journal level.

 

What are Research Metrics?

"Research metrics" are the methods and processes of counting and statistically analyzing information about research outputs. Metrics are commonly counted at the author level, article level, or journal level. Metrics are commonly collected and used by institutions during decisions on hiring, promotion, tenure, and retention for individual faculty; can be used by grant funding agencies when selecting grantees; or can be gathered by authors and used to help them select the most impactful venue for their publications or make other choices about how, when, and where to publish. Common metrics used in assessing academic research include:

  • Download counts. How many times has this output-- a paper, a data set, a conference poster, etc.-- been downloaded?
  • Citation counts. How many times has this paper, book chapter, or website been cited by other authors?
  • Journal Impact Factor. This single number is created by counting several data points about a journal's performance in a year, such as number of published papers per year and the citations to those papers in each year, performing specific mathematical calculations, and normalizing the data against other journals in the same field and in the same year. The creators of this calculation believe it assesses and expresses the relative impact of journals within a field for that single year. It is typically used by academics to assess the value of a single paper, but impact factor is not an article-level metric and this is an inappropriate use.

You can already identify some potential problems with using these metrics to determine whether to hire, promote, provide tenure, or retain an individual faculty member, or to use as a basis for grant funding decisions. Different fields have different numbers of readers and authors: an article about treatments for a common disease like diabetes may receive more downloads and citations than an article about plant tissue culture methods, simply because there are more health care professionals treating individuals with diabetes than people performing plant tissue culture. A citation to an article may be done when you agree with the article, or when you disagree: it is not a measurement of the quality of the research work. It is commonplace in some disciplines to cite your prior works in your own later works, and this practice increases your citation count (conversely, some academic fields do not allow self-citation).  A journal that publishes 50 issues a year and encourages authors to include 25 or more citations in each manuscript will have a higher impact factor than a journal publishing 12 issues per year and allowing only 10 citations per article.

The Metrics Toolkit was developed in 2020 to provide brief explanations of commonly used research metrics, how to find and calculate them, their limitations, and guidance on appropriate ways to use each metric in academia.

Basic Challenge: Understand Common Research Metrics

1. Review the Metrics in the Metrics Toolkit. Click on the names of five metrics that are related to the types of publications or scholarly outputs you create Explore the metric and try to understand how and why it would be used, along with one criticism. If you are not sure what to pick, five commonly-used metrics include:

2. Reflect on your own work: What academic publishing do you undertake and what outputs do you create? How are these activities counted using research metrics? Given what you have seen in this toolkit and your experiences, are metrics used appropriately, inappropriately, or both? Now that you know more about metrics, how might you change your practices regarding publishing, citing, or other metric-generating activities?

3. Optional: record your answers to the questions above; we'll collect them at the end of the Challenge.

Advanced Challenge: Critique the use of Research Metrics

1. Review the Metrics in the Metrics Toolkit. You are likely familiar with many metrics; review ones that are new to you. Be sure to consider these metrics, which are mentioned in the DORA statement:

2. Read the San Francisco Declaration on Research Assessment (DORA), a statement issued in 2012 by a group of scientific journal publishers calling for responsible selection and use of research metrics. It includes recommendations for concrete activities that funding agencies, institutions, publishers, and researchers could undertake to support responsible metric use.

3. Reflect on your experiences in academia with research metrics. Do you agree or disagree with the DORA statement? Have some or all of the DORA recommendations been adopted in your field and the institutions in which you have worked? How might you support the appropriate and responsible use of metrics, considering your roles as an author/researcher, peer reviewer, journal editor, or administrator?

4.Optional: record your answers to the questions above; we'll collect them at the end of the Challenge.

Want to learn more?

There is a lot more to research metrics than what is included on this guide. Explore the other guides created by USC librarians linked here for more information, and contact the creators of this Challenge with additional questions or to discuss metrics not included on these guides.

Day 3: Open Access

Research metrics try to measure the impact of an individual researchers' entire career by examining the reach of a single piece of scholarship:  how often has it been read, how often has it been re-used or cited, and the relative quality of the journal in which the article was published. Continue on to Day 3: Open Access to learn about how you can choose to publish journal articles using publishing methods that make your research outputs free for any reader in the world, increasing readership and the potential for your work to be re-used and cited.