Evaluating Information Sources: Impact Factors and Citation Counts

Tips on evaluating popular and scholarly articles, bias and propaganda in publishing, impact metrics and predatory publishing.

What are metrics?

How does the scientific community measure how "good" or "great" a journal or an author is? How do you determine the "impact" of an author's work? Should it be purely based on the number of times the article is cited? If not, how can we measure the "quality" of the research?

Several methods to calculate the impact of an article, journal, or author have been developed answer these questions. These calculations and statistical methods are called metrics. Be aware metrics are highly debated. The most popular metrics include: number of citations (journal or author), journal impact factor, and author h-index. There are hundreds of other metrics available, some better defined than others.

Journal Impact Factors

According to Journal Citation Reports (JCR), an impact factor is a ratio focusing on original research. 

Impact factor = # of citations to all items published in that journal in the past two years
(divided by)
# of articles and reviews published over those past two years referencing those citations

For example, if a journal has an impact factor of 2.5, this means in the indexed year each article published was cited on average 2.5 times in the previous two years in that journal.

Impact factor is used for journals only.

JCR only includes 12,000 journals and conference proceedings from over 3,300 publishers.

Author Impact / Citations

The most common metric to track an author's impact is ask how often they are cited. However, for the information to be accurate, a database must:

  • Have access to all articles an author publishes
  • Have the full text of all articles (letters dissertations, etc.) in the author's field and be able to read every reference (citing article) in every article

We may be able to get close, but

  • ​Web of Science searches only 10,000 "high impact" journals of the over 24,000 journals in their database
  • Google Scholar searches for everything it can find freely on the web
  • Specialized databases have limited access to journals (e.g., PsycINFO includes only 2,395 journals)
  • The indexed database, PubMed, has citations from over 35,000 journals - but no references

Thus, no one database can give an accurate count of how many times an individual article (much less the author) has been cited! 

Author H-index

The h-index attempts to measure both the productivity and impact of an author. H is the number of articles published by an author which have each been cited at least h times.  (E.g., an author published 4 papers that are cited 10, 6 ,5 and 2 times respectively; that author's h-index is 3.)

Recently, some databases (e.g. Google Scholar) use an h-index for the journal. In this case, h is the number of articles published that have been cited h times over a given time period (H5 = five years).

For more information, see: 

Author h-index Options

Other options for finding and/or creating author h-index lists.

Google Scholar -

  • Create in personal My Profile. Google then searches the web for a matching name. Add and subtract materials. The h-index is  publich and available only for those authors who have created profiles.

Scopus

  • Use the author search to select an author

Web of Science

  • Use the author search to select an author
  • See the box to the right for details

Author Citation Reports in Web of Science

In Web of Science, a report of the author's overall citation counts, h-index, and publications can be created. This will be limited to those citations that are available through Web of Science. (For Scopus and Google Scholar h-index option, see below.)

Step 1: Identify author citations in Web of Science

  1. Click +More on the Basic Search line, then on Author Search Author Search
  2. Enter the authors Last Name and first initial; Use + Add Author Name Variant to select specific variants; For multiple initials, initials or variants, the * can be used as a wildcard to find all of them
  3. Optionally, limit the search using Research Domain and/to one or more Organizations 
  4. Click Finish Search Finish Search button

Step 2: Verify publications

Review the results. If all author citations (and only those author’s citations) have been retrieved, go to Step 4. Use the Marked List function to create a comprehensive list of publications for an individual author.

  1. From the search results, select individual items or all items on a page, then click Add to Marked List; Repeat until all items have been reviewed
  2. Save the Marked List by registering for an individual account with Web of Science Marked List options

Step 3: Find and add missing publications to the Marked List

  • If Research Domain or Organization limits were used, use different options or try the search without limits
  • Search with the full name of the author: Use the All Databases option from the Select a database drop-down menu 
  • Search for individual known citations using: Author, Title and Publication Name 
  • From a known citation, click on the link to the author's name

Step 4: Run the Citation Report

Once you have search results that represent all available citations for an author, run the citation report. Note: If you used it, click Marked List and scroll down.

Click Create Citation Report   Create Citation Report

Export the report to an Excel file using the Export Data function. Unless all items are specified, only the citations on the web-page (default of 10) will be saved.
Export Data Option

In addition to the selected citations, the report will include two charts (Total publications by year and Sum of times cited per year) and the numbers for: Results found (citations), Sum of the times cited, Average citations per item, and h-index.

Note: A researcher has an index of h if h of their papers have been cited at least h times each.

For full instructions, download: 

Cited Articles = Confusing Statistics

The question "how many times has this article been cited?" has no simple answer. This number depends entirely on the database used and how the article references are searched by that database. Take, for example, for the article:

Age differences activity during emotion processing: reflections of age-related decline or increased emotion regulation? by Kaoru Nashiro, Michiko Sakaki, and Mara Mather.

On May 1, 2017:

  • In PubMed, this article was cited 31 times in PMC (an archival database, formerly PubMed Central containing 4.3 million full-text articles)
  • Google Scholar reported: Cited by 90 with 22 available versions (Google Scholar searches openly web accessible resources, collects duplicates as versions, and includes non-English citations)
  • Scopus stated Cited by 49 documents (a database of over over 21,500 peer-reviewed journals plus conference papers, books and patents)
  • Web of Science reported the article was 37 times cited (a database searching 10,000 high-impact journals and additional international conference proceedings)
  • ResearchGate reported 70 citations (a database of author uploaded articles)

Each database or search engine has its strengths and limitations. Some (e.g. Google Scholar and ResearchGate) include duplicate listings in their citation numbers. To identify all of the articles, letters, dissertations and books that cite a specific article, it may be necessary to search multiple databases and keep your own records.