Leading with ideas

HOW DO YOU MEASURE RESEARCH PRODUCTIVITY?
(VERY CAREFULLY)

By Andy Boynton, John and Linda Powers Family Dean

As deans, each of us has the task of not only building and improving a research culture but also measuring it. Needless to say, it’s a challenge. Just how do you put a number on intellectual output and the creation of knowledge that makes a difference?

We all have our ways. In the spirit of exchanging best practices, here’s my take on an eight-year experiment we’ve undertaken at the Carroll School of Management.

Interspersed with my comments will be pages from what we simply call “Research Reports.” These are three-page summaries compiled every spring, capturing the quantity and quality of a faculty member’s research. The reports give us a transparent way of evaluating the research, one that lends to comparison across positions and departments.

We’ve found these annual summaries to be very useful. They don’t explain everything about a professor’s research, and they shouldn’t, but they do offer a starting point for discussion. They're one measure—a good first approximation of research productivity and impact. Naturally the data points come into play when considering tenure and promotion, but in my mind the real value is that they begin a conversation.

In the pages displayed here, you’ll see the standard references to “Impact Factor” and “Article Influence.” As you know, both of these refer to the quality of journals in which our professors are published. “Impact” relates to how often the average article in these journals is cited; “influence” gives greater weight to citations in higher quality journals.

You’ll see references to “Financial Times,” specifically the top 50 journals used by the Financial Times in compiling its rankings of business schools from the standpoint of research output. Our measures also take account of how often a given journal’s articles are cited in Web of Science, and how often articles written specifically by our faculty are cited in Google Scholar.

In addition, metrics are influenced by whether a faculty member’s article is coauthored, and if so, how many collaborators there are (more about how we don’t necessarily penalize coauthorship in a moment).

Here is the first page of the Research Report that every faculty member receives. It shows how their statistics are calculated:

On the second page, the faculty member sees his or her research statistics and how they compare with colleagues in the department and school. I have the honor here of using as a sample the May 2016 Research Report for Jeffrey Pontiff, the James F. Cleary Chair in Finance.

As you’ll read in this edition of Carroll Capital, Jeff recently won the Journal of Finance’s 2016 Amundi Smith Breeden First Prize for his paper, “Does Academic Research Destroy Stock Return Predictability?” He collaborated with our former doctoral student R. David McLean, now at Georgetown University’s McDonough School of Business.

Research Statistics for Jeffrey Pontiff (Finance)

I alluded earlier to the issue of coauthorship. I say “issue” because there’s more than one way to account for articles coauthored by our professors. On the one hand, we want to encourage collaboration; on the other, we want to see signs of independent authorship as well. The upshot: we include some measures that (in effect) subtract points for coauthorship and others that don’t.

Finally, I return to the broader question of quantifying the value of knowledge. I strongly believe that everyone’s research is nuanced, and to understand it well, you have to get beneath the data. Still, the data can tell you a lot.

All of us have goals in this regard—at the Carroll School, we are focused heavily on producing high-quality, impactful research. We need a measuring system compatible with this goal, not necessarily the whole picture of scholarly output, but at least a snapshot. We keep tinkering every year with our measuring system to make sure we’re getting the clearest possible picture.

I would be thrilled to share more details about our approach with readers of Carroll Capital, and hear about your efforts to measure research accomplishments. Please feel free to send me an e-mail.

I leave you now with the third and final page of Jeff Pontiff’s Research Report. “WOS” refers to Web of Science, and the 50 journals ranked by the Financial Times are assigned a value of 1 (all others carry a value of 0).

List of Publications for Jeffrey Pontiff (Finance)