Gaming the Metrics: An Analytic Summary

Introduction

Gaming the Metrics: Misconduct and Manipulation in Academic Research is a collection of twenty-one essays edited by Mario Biagioli and Alexandra Lippman. The MIT Press published it in 2020 in its Infrastructures series. It is open access under a Creative Commons CC BY-NC-ND licence, which means anyone may read and share it free, but not sell it or change it. The full text is at the MIT Press. The book grew out of a 2016 meeting called "Gaming Metrics" at the University of California, Davis, and out of a UC Davis project on how scholars publish and share their work (p. 283).

The book is about what happens to research when it is judged by numbers. Those numbers are mostly counts of citations: how often other papers refer to a given paper, a researcher or a journal. The book is not about the three classic kinds of research fraud: making up data (fabrication), altering data (falsification) and copying other people's work (plagiarism). It is about newer tricks that work on the numbers around a paper rather than on the paper itself. These include fake peer reviews: peer review is the check in which other experts read a paper before a journal accepts it. They also include citation rings (groups who agree to cite each other), journals that exist to be listed rather than read, and universities that reshape themselves to climb league tables. The book also covers the people who expose all this.

The thesis: from "publish or perish" to "impact or perish"

Misconduct used to be blamed on the pressure to publish. In 1986 a cardiologist was caught reporting experiments that had never taken place. At one point he was producing an article every ten days (p. 1). That pressure is still there, but a second one has joined it. Researchers are now rewarded less for how much they publish than for the measured "impact" of what they publish, which in practice usually means how often it is cited. "Publish or perish" is merging with "impact or perish" (p. 1).

The editors set their claim apart from Goodhart's law. The law says that once a number becomes a target, people learn to game it, and it stops being a good measure. That kind of gaming bends what already exists. A school changes its definition of a successful student. A university keeps classes at nineteen students so they count as a "small class" in a ranking (p. 1). The editors describe something different. Some authors and editors now manufacture the very evidence the metric counts, such as citations to their own work or their own journal. The spread of metrics, they argue, does more than give people a motive. It "creates their conditions of possibility" (p. 1). In other words, these tricks only exist because the modern university runs on audits and scores.

A summary of the editors' Introduction

Post-production manipulation. The official American definition treats misconduct as false statements inside a publication. The new tricks happen around the publication: fake peer review, hacked journal databases, and citation rings among authors or among editors (p. 2). The editors call these "post-production" manipulations, because they come after the research itself is done. A paper can be perfectly sound and still get into a high-impact journal through fake reviews. The editors do not want these tricks treated as a lesser evil. They expect the idea of misconduct will have to be rethought, not just stretched to fit (pp. 2–3). Their short version: these are less "epistemic crimes", crimes against knowledge, than "bureaucratic crimes", crimes against the paperwork (p. 3).

What publication now means. In the seventeenth century, to publish meant to make something public by any means, including lectures and letters. By the nineteenth century, in science, it had narrowed to the printed journal article (p. 4). One thing stayed the same: people judged a claim by reading it. That is no longer needed. A journal's impact factor, a score based on how often that journal's recent articles have been cited on average, is attached to an article on the day it appears.

A publication is born evaluated, so to speak. (p. 4)

The paper still looks as it did in the 1980s. What has changed is its metadata, the information about the paper: its journal, authors, citations and so on. That information used to describe the article. Now it grades it (p. 5). The editors compare the new kind of evaluation to a library catalogue. It processes facts about a publication without reading it (p. 5).

Journals as mints. The impact factor cannot measure an article's impact when it is published, because at that moment there is none. It is a forecast based on earlier articles in the same journal, much like an insurer's table of life expectancy (p. 6). Yet it is treated as a measure of value, and the editors take the money comparison literally. Chinese universities pay cash bonuses tied to the impact factor, with papers in Nature and Science worth over $30,000. Melbourne Business School pays A$15,000 for each paper in a journal on a Financial Times list (p. 6). The impact factor works like the face value stamped on a coin. It lets articles be traded for jobs. Universities in turn trade their staff's articles for rankings, students and donors (p. 7). Its real purpose, the editors conclude, is to make a market possible (p. 7).

An unstable object. Traditional misconduct has been boiled down to three practices. Post-production manipulation can take as many forms as there are metrics and markets, and each new metric invites new gaming (p. 8). What the forms have in common is their aim: to manipulate the numbers used to judge academic work (p. 9).

Ways of gaming. The new gaming is done in groups. Editors agree to cite each other's journals. Authors form rings that write fake peer reviews for one another. Editing services arrange friendly reviewers for a fee. Universities massage their data for the rankings (p. 9).

• So-called predatory journals, which charge authors to publish while doing little or no real checking, and conferences where you pay to present. The editors see these as the bottom end of the same metrics economy. What they turn out are "publication tokens" rather than real publications (pp. 9–10).
• Journals citing themselves. The Journal of Biomolecular Structural Dynamics saw its impact factor jump from 1.1 to 5.0 between 2009 and 2010. It was then dropped from the Journal Citation Reports, the yearly list where impact factors are published (p. 10). Such removals rose from nine in 2007 to thirty-eight in 2013 (p. 11).
• A citation cartel, a group of journals that agree to cite one another to raise their scores. Eight Brazilian journals set one up in 2009. It was exposed in 2013, when four of them were among fourteen journals suspended (p. 11). One editor blamed the national agency CAPES. It judged graduate programmes partly by the impact factors of the journals their students published in (p. 11).
• SCIgen, a program that writes nonsense papers. Three MIT graduate students wrote it to expose fake conferences, and other people then used it in earnest. Springer and the IEEE had to withdraw more than 120 generated papers (pp. 11–12).
• Fake peer review. Authors suggested reviewers whose email addresses they controlled, so in effect they reviewed their own papers. By then Retraction Watch, a website that tracks papers withdrawn after publication (retracted), had reported more than six hundred such papers (p. 12).
• Fake impact-factor companies with names close to real ones (p. 13).
• Universities that aim straight at the rankings. Northeastern University climbed from 162nd in the U.S. News list in 1996 to 98th in 2006 and 44th in 2018 (p. 13). A note adds the case of King Abdulaziz University. It rose in a subject ranking after hiring highly cited mathematicians as part-time adjunct professors (p. 20, n. 19).

"Spammish." The editors argue that "fake" and "predatory" are the wrong words for much of this. Doubtful journals do publish articles, some of which get cited, and some of these journals are bought by mainstream publishers (p. 14). The constant attacks on them look to the editors like a way for publishers in the rich Global North to define themselves against a convenient villain (p. 15). Academics who publish in these journals are rarely fooled. The editors suggest the word "spammish" for practices that look like spam but are partly done by agreement. Authors can then present them either as legitimate or as a trick played on them, whichever suits them (pp. 15–16).

New watchdogs. Much of the new manipulation cannot be seen by a referee reading a manuscript. It shows up only in journal databases: in the wording of reviews, in how fast reviews come back, and in patterns of citation. That evidence is mostly private to publishers (p. 16). Retraction Watch, PubPeer, and blogs such as the now-closed Science Fraud are a new kind of watchdog. They have no legal power. They rely instead on many readers, often anonymous, and on publicity that can push institutions to act (p. 17). They also cut against the metrics. Instead of collecting data about papers, they read the data, images and text inside them closely (p. 17). And they often use humour, which blurs the line between "policemen" and "pranksters" (p. 17).

How the book is built

After the Introduction come four parts. Each has a short introduction by the editors.

• Part I, Beyond and Before Metrics (five chapters), goes back to where citation metrics came from and asks whether Goodhart's law is the right way to think about them.
• Part II, Collaborative Manipulations (four chapters), looks at gaming by groups and institutions: university rankings, predatory publishing in the Czech Republic, co-authorship, and drug-company papers secretly managed by the companies.
• Part III, Interventions: Notes from the Field (eight chapters), is written mostly by the watchdogs themselves: Retraction Watch, PubPeer, a blogger writing under a pen name, the inventor of a fake highly cited scientist, a scientist who planted fake editors on junk journals, a developer of altmetrics (scores based on online attention such as downloads and shares), and an adviser on publication ethics. A case study of the Voinnet affair sits among them.
• Part IV, Mimicry for Parody or Profit (four chapters), covers spam journals, counterfeits, a parody preprint server (a site where papers are posted before formal review) and the humour of the new watchdogs.

The chapters are short. Several are personal accounts rather than research papers, and the title of Part III says as much.

The editors and contributors

What follows comes from the book's list of contributors (pp. 285–286) and from what the chapters say about their authors. Posts are as given when the book came out (2020).

Mario Biagioli was a Distinguished Professor in the School of Law and the Department of Communication at UCLA (p. 285). The Introduction draws on his earlier writing on citation gaming and on how research is judged in the age of metrics (pp. 1, 20).

Alexandra Lippman was a visiting assistant professor at Pitzer College (p. 286). She co-wrote the Introduction and wrote the closing chapter, on humour and hoaxes.

The contributors come from the history and sociology of science, scientometrics (the measurement of science), accounting, law, philosophy, media studies and the laboratory sciences, and from the watchdogs themselves. Among them, at the time of publication:

• Alex Csiszar, historian of science at Harvard; Yves Gingras, historian and sociologist of science at the Université du Québec à Montréal; Michael Power, professor of accounting at the London School of Economics; Paul Wouters, professor of scientometrics and director of the Centre for Science and Technology Studies at Leiden; James Griesemer, philosopher at UC Davis (pp. 285–286).
• Barbara M. Kehm, of the Leibniz Center for Science and Society, Hannover; Sarah de Rijcke, professor of science, technology and innovation studies at Leiden; Tereza Stöckelová, of the Institute of Sociology of the Czech Academy of Sciences and Charles University; Daniele Fanelli, a fellow in quantitative methodology at the LSE; Sergio Sismondo, philosopher at Queen's University, Canada (pp. 285–286).
• Ivan Oransky, co-founder of Retraction Watch (p. 141), listed as distinguished writer in residence at NYU's Carter Journalism Institute; Boris Barbour, research director at the École des Neurosciences Paris Île de France and a board member of the PubPeer Foundation; Brandon M. Stell, co-founder of the PubPeer Foundation and a CNRS researcher; Catherine Guaspare, a sociologist at the CNRS; Emmanuel Didier, a founding member and permanent researcher at the joint CNRS-UCLA lab Epidapo; Paul S. Brookes, professor of anesthesiology and of pharmacology and physiology at the University of Rochester; Burkhard Morgenstern, professor of bioinformatics at Göttingen; Jennifer Lin, of Crossref, formerly of PLOS (p. 213); Elizabeth Wager, consultant and trainer in medical publishing and founder of Sideview (pp. 285–286).
• Finn Brunton, media scholar at NYU and author of a history of spam (p. 243); Marie-Andrée Jacob, professor of law at Leeds; Alessandro Delfanti, of culture and new media at the University of Toronto (pp. 285–286).

Chapter 14 is signed by Ike Antkare. This is an invented person. The book explains that "Ike Antkare" (I Can't Care) was created by Cyril Labbé as the author of computer-generated papers (p. 139; see also p. 84, on "Labbé's experiments with his fictional author", and p. 274). The chapter's own notes say the pen name hides self-citation and give Labbé's address for letters (p. 189, nn. 1–3). Labbé is listed as Maître de Conférences, roughly an associate professor, at the Université Grenoble Alpes (p. 286).

The sections