Assessing Scholarly Productivity: The Numbers Game

Assessing Scholarly Productivity

To evaluate the work of scholars objectively, funding agencies and tenure committees may attempt to quantify both its quality and impact. Quantifying scholarly work is fraught with danger, but the current emphasis on assessment in academe suggests that such measures can only become more important. There are a number of descriptive statistics associated with scholarly productivity. These fall broadly into two categories: those that describe individual researchers and those that describe journals.

Rating Researchers

Raw Citation Counts

One way to measure the impact of a paper is to simply count how many times it has been cited by others. This can be accomplished by finding the paper in Google Scholar and noting the "Cited by" value beneath the citation. Such numbers may be added together, or perhaps averaged over a period of years, to provide an informal assessment of scholarly productivity. Better yet, use Google Scholar Citations to keep a running list of your publications and their "cited by" numbers. For more information on determining where, by whom, and how often an article has been cited, see IC Library's guide on Cited Reference Searching.

Article comparing Google Scholar, Scopus and World of Science and their pros and cons.

Google Scholar as a new data source for citation analysis.


The h-index, created by Jorge E. Hirsh of the University of California, San Diego, is described by its creator as follows:

A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np - h) papers have no more than h citations each.1

In other words, if I have an h-index of 5, that means that my five most-cited papers each have been cited five or more times. This can be visualized by a graph, on which each point represents a paper. The scholar's papers are ranked along the x-axis by decreasing number of citing papers, while the actual number of citing papers is shown by the point's position along the y-axis. The grey line represents the equality of paper rank and number of citating articles. The h-index is equal to the number of points above the grey line.

The value of h will depend on the database used to calculate it. 2 Thomson Reuter's Web of Science and Elsevier's Scopus (neither is available at IC) offer automated tools for calculating this value. In November of 2011, Google Scholar Citations became generally available. This will calculate h based on the Google Scholar database. An add-on for Firefox called the Scholar H-Index Calculator is also based on Google Scholar data.

Google Scholar Metrics includes lists of top-ranked journals by index in a variety of subject areas.

Comparisons of h are only valid within a discipline, since standards of productivity vary widely between fields. Researchers in the life sciences, for instance, will generally have higher h values than those in physics.1

A large number of modifications to the h-index have been proposed, many attempting to correct for factors such as length of career and co-authorship.

The Publish or Perish website also provides a way to calculate h-index based on Google Scholar reports. Note that each website is likely to give a different result because it indexes, and calculates based on, different sources.

Researchers at the National Institutes of Health have developed a method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. The beta version of iCite, can be used to calculate the Relative Citation Ratios of articles listed in PubMed.

Rating Journals

Rightly or wrongly, the quality of a paper is sometimes judged by the reputation of the journal in which it is published. Various metrics have been devised to describe the importance of a journal.

Impact Factor

The Impact Factor (IF) is a proprietary measure calculated annually by Thomson Reuters (formerly by ISI). This figure is based on how often papers published in a given journal in the preceding two years are cited during the current year. This number is divided by the number of "citable items" published by that journal during the preceding two years to arrive at the IF. Weaknesses of this metric include sensitivity to inflation caused by extensive self-citation within a journal and by single, highly-cited articles. For more information about the IF, see the essays of Dr. Eugene Garfield, founder of ISI. Determining a journal's IF requires access to Thomson Reuters Journal Citation Reports, not available at IC Library.


The eigenfactor is a more recent, and freely-available metric, devised at the University of Washington by Jevin West and Carl Bergstrom.3 Where the IF counts all citations to a given article as being equal, the eigenfactor weights citations based on the impact of the citing journal. Its creators assert that it can be viewed as "a rough estimate of how often a journal will be used by scholars." Eigenfactor values are freely avialable at

SCImago Journal Rank Indicator

The SCImago Journal Rank indicator (SJR) is another open-source metric.4 It uses an algorithm similar to Google's PageRank. Currently, this metric is only available for those journals covered in Elsevier's Scopus database. Values may be found at


Journal Article Acceptance Rates

Locating acceptance rates for individual journals or for specific disciplines can be difficult, yet is necessary information for promotion and tenure. Journals with lower article-acceptance rates are frequently considered to be more prestigious and more “meritorious.”

The method of calculating acceptance rates varies among journals.  Some journals use all manuscripts received as a base for computing this rate.  Other journals allow the editor to choose which papers are sent to reviewers and calculate the acceptance rate on those that are reviewed that is less than the total manuscripts received.  Also, many editors do not maintain accurate records on this data and provide only a rough estimate.  Furthermore, the number of people associated with a particular area of specialization influences the acceptance rate.  If only a few people can write papers in an area, it tends to increase the journal's acceptance rate. Some journals will include the acceptance rate in the “information for authors” area of the print journal or on the home pages for the journal.

Some sources to find journal acceptance rates are as follows:

Cabell's Directories of Publishing Opportunities -Ithaca Collge School of Business has a subscription which covers the following areas: management, marketing, accounting, economics and finance. Go to Your Access where you can either browse or search for a specific journal(s). Note:  On-campus access only as Ithaca College IP address is required.

MLA International Bibliography Choose Advanced Search, then Directory of Periodicals. You can then look up the periodical you are interested in.

American Psychological Association (APA) Journal Statistics and Operations Data -These PDFs provide information about manuscript rejection rates, circulation data, publication lag time, and other journal statistics

Association for the Advancement of Computers in Education (AACE) -submission review policy, acceptance rate and indices.


Cited Reference Searching

 How to Find Cited References.


Hirsch, J.E. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102, 16569 -16572 (2005).
Meho, L.I. & Yang, K. Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology 58, 2105-2125 (2007).
Bergstrom, C. Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research Libraries News 68, (2007).
González-Pereira, B., Guerrero-Bote, V.P. & Moya-Anegón, F. The SJR indicator: A new indicator of journals' scientific prestige.


Altmetrics are non-traditional, Social Web-based measures of the reach and influence of scholarly output. Some examples of altmetrics are downloads, bookmarks, blog posts, shares, and citation indexes. Altmetrics data can be used to explain the amount and type of online attention that items receive. According to the authors of Meaningful Metricsthese measures show the impact of a wider type of items beyond traditional research journals (software, data sets, and slide decks, for example) and can show areas of influence outside of disciplinary boundaries and formal academic communities (through citations in public policy documents and references in patents, learning objects, and press coverage, for instance).

Wiley now makes altmetrics available for their fully open access journals. Other scholarly publishers such as Elsevier and Sage also offer article-level altmetrics, including comments and shares made by readers via social media channels, blogs, newspapers, etc. Here's an example from the ScienceDirect database:

When used with traditional, citation-based bibliometrics, altmetrics can form a more complete and detailed picture of the impact of scholarship. They can sometimes indicate the potential impact of research on society or a field of study. Altmetrics can be especially useful in discplines where journal articles are not the primary output.

Facuty can use altmetrics to
  • Determine who, where, and how their research is being used
  • Quantify how their research is being used
  • Spot potential collaborators in their field
  • Show the influence of their work inside and outside their field of study and academia
  • Learn what others think of their research
  • Demonstrate successful outreach
  • Pinpoint and filter other sources of importance
  • Provide a context for their work
  • Solicit research funding
  • Identify publications to submit to

For Tenure and Promotion

The Leiden Manifesto for Research Metrics

Guide to Preparing a Dossier for Promotion or Tenure
University of Colorado Denver Medical School

Guidelines for Preparing and Reviewing Promotion and Tenure Dossiers 2017-18  
Indiana University-Purdue University Indianapolis

Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History
College Art Association and the Society of Architectural Historians 

Guidelines for Evaluating Work in Digital Humanities and Digital Media
Modern Language Association

In a CV
Recommendations for including altmetrics in a CV include providing contextual information, such as percentiles, maps, and qualitative data. Here's one example from the CV of Trevor A. Branch, Biology, University of Washington:

Additional examples can be found at What Are Altmetrics? and in the University of Maryland Bibliometrics and Altmetrics: Mesuring the Impact of Knowledge guide.

Altmetrics Tools

Provides a profile of the online impact of a researcher's work.

Altmetric Bookmarklet
Enables viewing article-level metrics.

PLOS Article Level Metrics 
Tracks item-level views, saves, citations, recommendations, and discussions of scholarly output.

Contact Us

picture of Calida Barboza

Calida Barboza

Electronic Resources Librarian
(607) 274-1892
picture of Jim Bondra

Jim Bondra

Business Librarian
(607) 274-1962
picture of Lisabeth Chabot

Lisabeth Chabot

College Librarian
(607) 274-3182
picture of Ron Gilmour

Ron Gilmour

Web Services Librarian
(607) 274-3674