Read the whole book free at the MIT Press ↗

The first part goes back to where publication metrics came from. It is less interested in how one metric gets gamed than in the gaming that comes with bringing in any metric at all (p. 27). Metrics never stand still. Each one creates its own gaming. The gaming floods the market, the metric is revised, and a new round begins. The targets, the editors write, are always moving (p. 27). Several of these chapters grant that Goodhart's law is useful but find it too narrow. Academic metrics are not one number but many scores and rankings competing for attention (p. 27).

Csiszar shows that the sociologist Robert Merton warned about the gaming of citation counts before they were widely used (pp. 27–28). Gingras describes a "perfect storm" that turned the scientific paper from a piece of knowledge into a unit for keeping accounts (p. 28). Power and Wouters move from papers to institutions. Power shows that in the UK, research is now shaped by the need to prove its impact before it is even done. Wouters argues that it is hard to tell gaming the system apart from simply working well within it (p. 28). Griesemer ends the part by proposing to study metrics by hacking them on purpose (pp. 28–29).

Chapter 1. Alex Csiszar, "Gaming Metrics Before the Game: Citation and the Bureaucratic Virtuoso"

Csiszar takes aim at a common story. It says that gaming is an unforeseen side effect of metrics, which an innocent earlier age had no reason to expect. He argues that the story gets the history wrong. The worry that measuring science would change how scientists behave was part of the debate about citation metrics from the very start (pp. 31–32).

His evidence is a 1974 conference on "science indicators", meaning numbers that track the state of science. It was held at the Center for Advanced Study in the Behavioral Sciences at Stanford. The occasion was the first Science Indicators report from the US National Science Foundation. After a 1968 change in the law, Congress wanted hard numbers on the health of science (p. 32). The sociologist Robert K. Merton ran the meeting. Eugene Garfield attended. He owned the Institute for Scientific Information and had invented the Science Citation Index, a database of who cites whom (p. 33). Merton kept bringing the talk back to "goal displacement", an idea he had brought into sociology in 1940. It describes what happens when a means becomes an end in itself (pp. 33–34). He wrote to Garfield afterward with a warning. Whenever an indicator becomes part of a reward system, people will manipulate it until it no longer shows what it once did, and scientists would start arranging citations among themselves (p. 34).

Garfield took the warning calmly. If gaming showed up within ten or twenty years, he wrote, the index would have to be dropped as a measure of quality (p. 36). Merton then reassured him. The danger would only come if citation counts were used on a massive scale in hiring, promotion and grants, and that, he said, was "not in the cards" (p. 37). Csiszar's verdict is blunt: Merton was wrong (p. 37). Garfield had published his first ranking of journals in Science in 1972, and that year he began marketing the Journal Citation Reports, which became the home of the impact factor (p. 35). By 1981 his company was planning products designed for judging individuals and institutions (p. 37).

Critics at the time saw the risk too. A letter to New Scientist called the journal ranking "a pedantic dunghill", and Alex Comfort called rankings "pop charts for science" (p. 36).

Goodhart's law and goal displacement point in opposite directions. Goodhart's law is about the damage done to the measure. Merton's idea is about the damage done to what is measured: which projects people choose and how they carry them out (pp. 37–38). Both ideas, Csiszar adds, treat science like a company whose goals everyone agrees on. In a democracy there is no such agreement. So arguing about how to measure science is also arguing about why the public should trust and fund it (p. 38). He suggests turning the law around to describe the usual order of events in history:

only when a measure becomes a target is it widely taken up as a measure worth using. (p. 39)

Chapter 2. Yves Gingras, "The Transformation of the Scientific Paper: From Knowledge to Accounting Unit"

Three changes came together from the mid-1990s. First, the internet let journals exist only online. Second, journal publishing became concentrated in a few large firms answerable to the stock market. Subscription prices soared, and open access, where readers pay nothing, grew up in answer. Third, the evaluation of research turned to numbers (pp. 43–44). Each change had its own effects. Gingras's thesis is that their combination, a "perfect storm", made possible the rise in bad behaviour by scientists, editors and university managers (p. 44). He leaves research done for profit out of his model and points to Sismondo's chapter for that (p. 44).

He starts from the classic picture of how science works, taken from Merton and the French sociologist Bourdieu. A scientist publishes a discovery. Peer review lets it into the field. Other scientists cite it, which brings recognition, and recognition brings money for more research (pp. 45–46). In this cycle the paper is a "unit of knowledge" (p. 45). Peer review is the gate.

Once the paper becomes a unit for keeping accounts, the cycle changes (p. 47, fig. 2.2).

• The databases change. The Science Citation Index at first recorded only the first author, because that was enough to find a paper. Databases now record every author, because the data are used to judge people (p. 47). The Science Citation Index had the field to itself until the publisher Elsevier launched its rival database, Scopus, in 2004 (p. 47).
• Journals multiply. Open access moved the cost of publishing from libraries to authors. Online, a journal is cheap to start. Together these opened the door to journals that offer fast publication to people who struggle to get their work accepted elsewhere (pp. 47–48). Famous brands multiply too: since the 2000s Nature has started more than thirty journals bearing its name (p. 48).
• The impact factor takes the place of citations. It was first published in 1975 as a tool to help librarians choose journals. It became controversial in the mid-1990s, once it was used to judge researchers (p. 48). Citations to a paper take years to build up. Evaluators want speed, so the journal's score stands in for the paper's (p. 48). Pakistan, China, South Korea and Japan paid rewards tied directly to impact factors (p. 49).
• New ranking products appear. The Nature Index, launched in 2014, ranks institutions by their papers in sixty-eight journals. Seventeen of those journals belong to the Nature group itself (p. 50).
• Altmetrics. These count downloads, views and tweets. They measure something different from citations, and they come from an unknown public. Gingras calls them crude and poorly defined (pp. 50–51).

He ends on a careful note. The structure of science held steady for more than three centuries and has changed in twenty years. Its long-term effects cannot be judged yet (p. 51). He mentions the "slow science" movement as one response (p. 51).

Chapter 3. Michael Power, "Playing and Being Played by the Research Impact Game"

In 2014 the philosophy departments of Birmingham and Keele came joint top of a UK league table, with an average grade of 3.80. Oxford came tenth with 3.40 (p. 57). The table measured impact, and "impact" here meant social and economic benefit outside universities. In the Research Excellence Framework of 2014 (REF2014), the UK's national review of university research, universities sent in 6,975 impact case studies alongside 191,150 pieces of research (pp. 57–58). Twenty per cent of research funding depended on impact (pp. 58–59). Power asks how such results are produced, and why nobody finds them strange (p. 58).

The tool was the Impact Case Study (ICS). This was a story written to a fixed template, with a word limit for each section, and graded for "reach" and "significance" (p. 60). Power describes it as a new unit of accounting placed on top of the journal article (p. 59). Old forms of public work, such as advising government or commenting in the press, were renamed "pathways to impact". They were no longer taken to count as impact in themselves (p. 59). A rough rule took hold of one case study for every ten staff doing research, and a case study with a top grade was worth a great deal of money (pp. 59–60).

At the LSE, where Power worked, many academics relied on "solicited testimony". They asked people their research had influenced to say so in a letter or email (pp. 60–61). The evidence of impact, he notes, was not lying there waiting to be found. The researcher made it (p. 61). People worried that the method was overused, and the evaluators themselves found such evidence hard to judge (p. 61).

This is not just individuals gaming the system; it shows how the system works. Taken together, the rules form a "meta-game": a set of rules for producing a certain kind of truth, in the sense the philosopher Foucault gave the phrase (pp. 58, 62). Universities now employ impact officers, and grant applications ask for the expected impact in advance. Impact has flipped from a result measured afterward into a target set beforehand (p. 62). Even where no metrics go in, the templates prepare the ground for turning stories into scores (p. 63). Behind it all Power sees a "logic of auditability", an urge to produce a paper trail whether or not anyone checks it. REF evaluators said they had little time to check the sources (p. 64).

He closes with Plato and Aristotle. Their influence is so widespread that they could never have written a case study:

the more impact you actually have, the harder it is to account for it. (p. 65)

In such a system, being an intellectual and writing long books come to look like misbehaviour (p. 65).

Chapter 4. Paul Wouters, "The Mismeasurement of Quality and Impact"

Wouters picks up the problem raised in the editors' Introduction. When evaluation relies on stand-in numbers, it becomes hard to tell someone gaming an indicator from someone doing good research (p. 67). He agrees with the editors' overall picture but adds limits. Evaluation is messier than they suggest. Reading still happens in many assessments, and the impact factor is not the only number in use (p. 67). If the trend goes on, he warns, work that cannot be shown in numbers may move to amateurs or vanish (p. 67).

Peer review and citation counts are usually set up as opposites, one qualitative and one quantitative. In practice they are closely tied (p. 69). Citations are themselves choices made by authors, for reasons that are both intellectual and tactical and cannot be pulled apart (p. 69). The people who cite and the people who review often belong to the same community (pp. 69–70). And now that researchers know their reference lists affect other people's careers, citing has become more tactical (p. 70). Evaluators also look up scores on Web of Science, Scopus or Google Scholar out of habit, even when the rules discourage it. Researchers develop "folk citation theories", home-made ideas of how to read those scores (p. 70).

This matters for reform. The UK report The Metric Tide proposed five principles for "responsible research metrics": robustness, humility, transparency, diversity and reflexivity, that is, the habit of questioning one's own methods (p. 71). The Leiden Manifesto says numbers should support expert judgment, not replace it (p. 71). Wouters doubts this approach. "Informed peer review", the idea that experts and numbers can check each other, assumes two independent sources of evidence. They are not independent (p. 72). Peer review has its own faults. It can favour "old boys" networks, discriminate against women and minorities, and screen out radical work or work that crosses fields (p. 72). Numbers fail too. A high citation count can mean great work or a citation cartel (p. 72).

His proposal is to replace "informed peer review" with "contextualized judgment". This puts context first and drops the false choice between peer review and numbers (p. 73). Two questions should guide the design of any evaluation. How will it affect the creative work of making knowledge? And who controls what gets researched (p. 74)? Evaluation, he concludes, is a political question about what kind of society we want (p. 74).

Chapter 5. James Griesemer, "Taking Goodhart's Law Meta: Gaming, Meta-Gaming, and Hacking Academic Performance Metrics"

Griesemer, a philosopher, takes Goodhart's law further. The law does not only describe people reacting to targets. Like the Lucas critique in economics, which warns that a policy can change the very behaviour it was built on, it describes a policy that changes the causes behind what it measures (pp. 77–78). When metrics become standards, the goal becomes beating the standard, and the work itself becomes a side effect (p. 78). He cites Chamayou's point that the goal becomes writing articles, not doing research (p. 78). For that reason, he argues, it is hard to preach about "predatory" practices as if it were clear what honest practice looks like in such a world (p. 77). Behaviour that looks bad by the old standards may be ordinary adjustment to the new one. Moral judgment trails behind measurement (p. 79). Reform will not come from a better metric. A "basket" of metrics will be gamed as well. A standard has to be public to be fair, but secret to resist gaming (pp. 81, 84).

If Goodhart's law is true, gaming is the normal state of a system that has adapted well. So researchers should look for places where people fail to adapt. They should go further and hack the metrics deliberately, as an experiment (pp. 82–83). He even suggests that "predatory" journals can be seen as pioneers of hacking, if one sets aside their profit motive (p. 83). Scholars who study science should experiment on their own metrics, in the tradition of doctors who tested treatments on themselves (p. 84). He points to Labbé's Ike Antkare experiment as an example already under way (p. 84).

His concrete plan is a set of openly playful journals called PuLP, for "Public Library of Philosophy", a name suggested by Jonathan Eisen (pp. 84–85). PuLP would publish reviews of metrics, personal accounts of what metrics have done to people, and research on how science works. It would also hack metrics in the open. For example, it could run papers with five hundred authors that cite other PuLP journals, and so push its impact factor or its authors' h-index to any level it wanted (p. 85). (The h-index is a single number for a researcher: someone with an h-index of 20 has 20 papers that have each been cited at least 20 times.) Good scholarship in the journal would stop it being written off as junk (p. 86). The aim is not to get rid of metrics but to show how arbitrary they are, and to bring evaluation back to judging the work itself (pp. 85–86).

If sustaining the research enterprise requires hacking the metrics, let the games begin! (p. 86)