Read the whole book free at the MIT Press ↗

The third part gives "notes from the field" by and about the new watchdogs and pranksters (p. 137). The editors' introduction says that catching the new kinds of cheating is beyond the reach, the resources and perhaps the job of official bodies. One such body is the US Office of Research Integrity, a government office that deals with research misconduct (p. 137). Independent websites, bloggers and pranksters have filled the gap, and they face criticism and legal threats for it (p. 137).

Retraction Watch tells the stories behind retractions, the formal withdrawal of published papers (p. 137). PubPeer began as an online journal club, a group that meets to discuss papers. It now lets anyone discuss published papers in the open, and it finds many problems linked to the race to publish in high-impact journals (pp. 137–138). It works largely because it ignores metrics and reads the papers (p. 138). The Voinnet affair shows this new landscape at work (p. 138). Paul Brookes's blog chose mockery ("snark") and anonymity to force action, until his identity was revealed and he was threatened with lawsuits (p. 138). Cyril Labbé's "Ike Antkare" and Burkhard Morgenstern's fake editors are pranks with a point (p. 139). Jennifer Lin defends altmetrics, measures of online attention such as downloads and social-media shares. Elizabeth Wager proposes to stop using the journal article to report original research (pp. 139–140). The editors compare PubPeer's position to WikiLeaks: it has speed and reach, and it is open to the charge that it is not transparent itself (p. 138).

Chapter 10. Ivan Oransky, "Retraction Watch: What We've Learned and How Metrics Play a Role"

Oransky and Adam Marcus started Retraction Watch in August 2010. As journalists, they found that retraction notices were vague or even wrong, and that the stories behind them were interesting (p. 141). He notes, with some irony, that the blog has metrics of its own: 150,000 unique visitors a month and more than a hundred citations in the research literature (p. 141).

He describes how hard it is to correct the published record. A team led by the nutrition researcher David Allison found more than two dozen badly flawed papers, and in many cases the journals did nothing (pp. 141–142). Part of the reason is the shame attached to retraction. Part is metrics: publishing in certain journals is the way to grants and promotion, so authors resist admitting mistakes (p. 142). Whistle-blowers, the people who report wrongdoing, often come off worse than those they accuse.

the most common outcome for those who commit fraud is: a long career. (p. 142)

Retraction Watch allows anonymous comments, but it moderates them closely and posts no accusation it cannot back up (p. 142). In 2014, readers' comments on posts about four retractions led the University of Calgary to reopen a case, which brought five more retractions (pp. 142–143). Oransky stresses that the site is one part of a wider system, with PubPeer and others. The blog science-fraud.org closed under legal threats, and PubPeer, with help from the ACLU (the American Civil Liberties Union), won the right not to reveal who one of its commenters was (p. 143). Because he and Marcus are journalists rather than working scientists, they can criticize without fearing for their careers (p. 143).

Retractions grew from about 40 a year to about 400 between 2001 and 2010, while the number of papers grew by 44 per cent. They were near 700 in the 2015 financial year (p. 144). Even so, fewer than one paper in two thousand is retracted (below 0.05 per cent) (p. 144). More retractions do not necessarily mean more fraud. Detection has got better, just as a new screening test raises the number of diagnoses (p. 144). Image manipulation does seem to be increasing. Elisabeth Bik and colleagues found problem images in about one in twenty-five of more than twenty thousand papers (p. 144). He lists reasons for retraction. They include plagiarism, copying others' work (about 10 per cent); duplication, publishing the same work twice (about 15 per cent); honest error (about 20 per cent); and fake peer review, in which authors arrange to review their own papers, with more than six hundred papers retracted for it (pp. 144–145). Overall, two-thirds of retractions are due to misconduct (p. 145).

Metrics run through the details. A retraction takes about three years on average, a year longer than the period over which the impact factor counts citations (p. 145). Retracted papers go on being cited (p. 146). Journals with the highest impact factors retract the most, which he puts down mainly to the closer attention they get (p. 146). He ends with a hopeful use of citations. Authors who retract for fraud lose citations, but those who retract for an honest error, clearly explained, do not (p. 146).

Chapter 11. Boris Barbour and Brandon M. Stell, "PubPeer: Scientific Assessment Without Metrics"

Barbour and Stell, who run PubPeer, start with the sentence "She has a Nature paper!" and ask how often anyone then talks about what the paper actually said (p. 149). The obsession with where a paper appeared, they argue, is a sensible response to the way jobs and grants are handed out. Every metric draws attention away from the content, and a paper's true importance is hard to judge while it is new (p. 149). Top journals also push authors to present "revolutionary concepts". This leads to exaggeration, and to the neglect of solid work that moves a field forward step by step (pp. 149–150).

They argue that science's ability to correct itself is sick (p. 151). Replications, papers that repeat an experiment to see whether it holds up, are rare. Editors do not want to publish refutations of exciting papers they have published themselves, funders will not pay to redo experiments, and those who repeat them risk reprisals (pp. 150–151). They point to the large project that tried to repeat one hundred psychology experiments; fewer than half held up (p. 150). Letters to journals and journal comment systems have proved almost useless (p. 151).

PubPeer, launched almost five years before the chapter was written, gives one central place, separate from the journals, to discuss papers after they are published (pp. 151–152). The number of comments rose fast once strong anonymity was brought in (p. 152). Many comments pointed to plain misconduct in figures, such as duplicated bands in western blots, a standard lab image used to detect proteins. The authors conclude that a surprising number of researchers have built careers on doubtful practices (p. 152).

They defend anonymity. Comments must rest on public facts or reasoning that readers can check for themselves, hearsay is banned, and moderators enforce the rules (p. 152). Critics, they say, rarely name a single career unfairly damaged; in the high-profile cases they know of, the criticism was correct (pp. 152–153). Their policy is to put readers first. Patients' lives and public money matter more than ruffled feathers (p. 153).

authors who don't wish their work to be criticized or questioned are always free not to publish. (p. 153)

They see PubPeer as part of two wider trends: preprint servers, websites where researchers post papers before peer review, and full access to research data (pp. 153–154). Authors who publish weak work should expect to provide "after-sales service" (p. 154). They doubt that journals add much value, and point to physicists' use of arXiv, a free preprint server (pp. 154–155). Their final position is the most absolute in the book: all metrics are dangerous attempts to measure what cannot be measured, and none is likely to help in running science (p. 155).

Chapter 12. Catherine Guaspare and Emmanuel Didier, "The Voinnet Affair: Testing the Norms of Scientific Image Management"

Guaspare and Didier treat a scandal as a moment when unwritten rules get spelled out (p. 158). Their idea is that in the last twenty years science has changed, partly because of metrics, and the old rules have blurred. Photoshop and digital images raised new questions about how much editing of an image is acceptable. Bigger teams turned lab heads ("principal investigators") into managers (p. 158). At the same time a new set of watchdogs has made misconduct easier to see, and the scandals that follow set the rules again (p. 158). They refuse to treat "pressure" as an explanation on its own: it is a complaint, not an analysis (p. 158).

In 2014 Olivier Voinnet was a 43-year-old plant molecular biologist known for work on RNA interference, a way cells switch genes off. He was a researcher at the CNRS, France's national research agency, on secondment to ETH Zurich since 2010, with many prizes and about a hundred papers (pp. 159–160).

• September 2014. Images in ten of his papers were flagged as suspect on PubPeer (p. 160).
• 9 January 2015. His former supervisor David Baulcombe, the corresponding author (the one who answers for the paper) on half the flagged papers, posted statements; Retraction Watch reported the case the same day (p. 160).
• End of January. Thirty-five papers were under discussion, with 255 comments on PubPeer (p. 160).
• 18 February. First correction (p. 160).
• End of March and April. The newspaper Le Monde took up the story. The plant scientist Vicki Vance wrote that she had reviewed one paper three times and believed some of its data had been misrepresented; Plant Cell had published it in 2004 (pp. 160–161).
• April. The CNRS and ETH Zurich announced investigations (p. 161).
• 2 June. First retraction, of the Plant Cell paper (p. 161).
• 10 July. Both institutions published their findings (p. 162).

ETH found no invented data meant to change the results. It did find errors and manipulations in about twenty papers: "beautified" images, duplicated bands, loading controls (the reference band showing each sample was loaded evenly) shared between experiments, and tidied-up figures made for lab meetings that were used by mistake (pp. 162–163). Its press release was titled "Conducted Properly—Published Incorrectly" (p. 163). The CNRS, unlike ETH, judged it scientific misconduct (p. 163). Both held Voinnet responsible as head of the team (p. 163). ETH's report described his labs as fast-moving and high-pressure, and said speed was part of the problem (p. 163). Penalties followed. The CNRS suspended him for two years after his return from Zurich. ETH gave him a formal warning. The Swiss National Science Foundation stopped 1.25 million francs of funding and barred him for three years. EMBO took back his gold medal. He was also forced to give up his leading post at the CNRS laboratory in Strasbourg (p. 163). By late 2018 eight of his papers had been retracted and twenty-four corrected (p. 163; p. 165, n. 4).

The authors conclude that anonymous watchdogs with good links to the press can now start scandals that force institutions to act. Practices kept quiet in some laboratories built careers until they were exposed (p. 164). A postscript reports a 2016 inquiry into five papers from his former Strasbourg lab, which found evidence of deliberate fabrication. The CNRS and ETH cleared Voinnet of any unethical manipulation of data, but the CNRS held him responsible for repeated breaches of good scientific practice (p. 164).

Chapter 13. Paul S. Brookes, "Crossing the Line: Pseudonyms and Snark in Post-Publication Peer Review"

Brookes, a biomedical scientist, tells how he became a critic writing under a pen name. He noticed that a rival lab had used the same loading control for two unrelated experiments. He found more duplicated images in older papers and reported them to the US Office of Research Integrity (ORI). The ORI passed his complaints on to the researcher's institution with his name attached. Anonymity, he learned, has to be asked for (pp. 169–170). He decided that anything he did from then on would be anonymous (p. 170).

In 2011 a new kind of "snarky" blog appeared: sharp, mocking and often rude (p. 170). With one of these bloggers, he went through the work of a prominent cancer researcher who published about forty-five papers a year. They found suspect data in fifty-two of the ninety-two papers they checked, and in the end about eighty-five papers with problems, which they reported to the ORI (p. 170). Four years later there had been six corrections, no retractions and no action by the ORI (p. 171).

So he started science-fraud.org, writing as "Francis de Triusce", an anagram of "Science Fraudster" (p. 172). He used puns and rude language on purpose. He argues that the message, not the messenger, should count: a schoolchild who finds a real problem in a data set deserves the same hearing as a Nobel prize-winner (p. 172). Some of the cases he posted led to findings of misconduct (pp. 172–173). The blog ran from July to December 2012 before the legal threats came. In January 2013 someone found out who he was and emailed everyone he had written about, and his university, calling the blog a "hate site" (pp. 173–174). His university gave him no legal support. He paid his own lawyer and answered the threats, but the blog came to an end (p. 174).

He then turned the episode into data. He had 274 papers he had blogged about and about 220 similar ones he had not yet posted. The blogged papers were corrected or retracted at seven times the rate of the others, and the gap had not closed when he checked again (p. 174). He concludes that public, colourful, snarky criticism gets more results than polite flagging on a forum (pp. 174–175). And he adds:

real scientists do not need lawyers. (p. 175)

Chapter 14. Ike Antkare, "Ike Antkare, His Publications, and Those of His Disciples"

This chapter is signed by an invented author and written as a joke with a serious point. The book names Cyril Labbé as the person behind the pen name (pp. 139, 274; see also p. 189, nn. 1–3). It asks whether metrics can easily be faked, and answers yes: it has been done, and is probably being done now (p. 177).

The Dada Engine appeared on 1 April 1996 and produced nonsense about postmodernism that sounded convincing (p. 178). SCIgen followed in 2005, from a group at MIT. It is a program that strings computer-science jargon together, following grammar rules written by hand, to produce fake papers, and it was made to test conferences suspected of accepting anything (p. 178). It was later adapted to physics and mathematics (p. 179).

Ike Antkare was invented from scratch in 2010. A slightly modified SCIgen produced about a hundred papers, each one citing all the others, posted on a single web page. Google Scholar, Google's free search engine for academic papers, then included a paper only if it cited at least one paper already in its index. So, to get them included, each paper also cited an Antkare "PhD dissertation" that cited only real papers (pp. 181–183). Google's automatic crawler did the rest. According to Scholarometer, a tool that calculated citation scores, Antkare reached an h-index of ninety-four, meaning ninety-four papers each cited at least ninety-four times. That put him twenty-first among the most cited scientists, ahead of Einstein. On the hm-index, a version that adjusts for co-authors, he ranked sixth, and first in computer science (p. 181). His score later rose above one hundred, fell to thirty-two and ended at zero (pp. 183–184). A Spanish team ran a similar experiment (p. 184). The lesson drawn is that Google Scholar is open to "spam" and cannot be used to measure the reputation of researchers or institutions (p. 184).

Databases whose contents are checked by editors are not safe either. Once the fakes were detected, more than a hundred SCIgen papers disappeared from the databases of IEEE, the large engineering body. Some came from conferences that claimed to accept as few as 28 per cent of submissions (p. 185). One 2009 conference of 150 papers included four SCIgen papers and a duplicate (p. 185). A journalist who looked into it found that such papers were published mainly to meet publication targets set for Chinese academics (p. 185). SCIgen text has also been used to fill the fake websites of hijacked journals, sites that copy the name and look of a real journal to collect fees from authors (p. 186).

The chapter notes that the automatic screening at arXiv has never been reported to let a SCIgen paper through (p. 188). A method based on "intertextual distance", a measure of how alike two texts are in their wording, was taken up by publishers, and Springer funded SciDetect, software that spots generated papers (p. 188). The chapter ends by imagining bots that write papers and bots that review them, after which scientists can go on holiday (p. 189). Its reference list mixes real and fake references on purpose, as a test for citation-counting software (p. 189).

Chapter 15. Burkhard Morgenstern, "Fake Scientists on Editorial Boards Can Significantly Enhance the Visibility of Junk Journals"

Morgenstern, a bioinformatician (a scientist who uses computers to study biological data), argues that research cannot be measured by numbers at all (p. 201).

it is fundamentally impossible to measure research work quantitatively. (p. 201)

You can compare how many operations surgeons perform, he says. But you cannot compare the discovery of the citric acid cycle, a basic process by which cells make energy, with a proof in computer science, and their impact is no more comparable (pp. 201–202). Metrics, he argues, were brought in not to understand research but to sharpen competition, out of a belief that competition cures everything (p. 202). For that purpose, any number loosely linked to research will do, and "visibility", how much attention something gets, is the easiest to count (p. 202).

His experiments began in 2012, with spam from the OMICS Publishing Group naming him to the editorial board of Molecular Biology Journal for his reputation in a field he had never worked in. He had trained as a mathematician (p. 203). He applied as "Dr. Peter Uhnemann", a character a German satirical magazine had invented to troll a politician. The CV gave a "Department of Oximology" at a university named after a Disney inventor, and research interests such as "applied endoplutomomics" (pp. 204–205). He was accepted the next day. Someone even corrected the grammar of his absurd biography without noticing what it said (pp. 205–206). After Jonathan Eisen blogged the story, the journal's website crashed under the traffic, and OMICS sent Uhnemann a certificate (p. 207). Many other journals listed him too, and in March 2016 he was still on several boards (p. 208).

A second invented figure, "Dr. Hoss Cartwright", a character from the TV Western Bonanza, came from the "Ponderosa Institute of Bovine Research" with a postdoctoral post at "Cowboy University". OMICS's Journal of Primatology welcomed him (p. 209). Even mass emails with every recipient's address on show brought acceptances (pp. 209–210). The prank ended when the journal's editor-in-chief, Joseph Erwin, a real scientist, found out and told the publisher. Cartwright was removed, and Erwin resigned (p. 211).

The pranks did make these journals more visible, which by the logic of metrics counts as a good thing. But they made them visible as junk (pp. 202–203, 211). At least one scientist decided not to send a paper to such a journal after learning about its fictional editor (p. 208).

Chapter 16. Jennifer Lin, "Altmetrics Gaming: Beast Within or Without?"

Lin writes from inside the altmetrics effort, first at PLOS, an open-access publisher, and then at Crossref, the body that registers permanent links (DOIs) for scholarly papers (pp. 213, 222). PLOS started its Article-Level Metrics programme in 2009 so that each article could be judged on its own merits rather than by the journal it appeared in (p. 213). Around the same time, a group of researchers published the Altmetrics Manifesto (p. 214). "Altmetrics" (short for "alternative metrics") names a group of measures by what they are not, the formal citation in a research paper. It gathers page views, downloads, social-media shares, bookmarks in reference-manager software, mentions on Wikipedia and more (pp. 214–215). She sees it as a way to widen the meaning of "quality" beyond the impact factor, a journal's average citation rate (p. 215).

She calls the fear that altmetrics will be gamed reasonable but premature. Altmetrics play no formal part in funding or hiring, so for now there is little reason to game them (p. 216). They may also be harder to game, because a cheat would have to coordinate across many separate platforms (p. 216).

a threat far more conceptual than material. (p. 216)

Altmetrics data pass along a supply chain: the platforms where the activity happens, the services that gather it up, and the providers that deliver it. Patterns of use differ by field and change over time (p. 217). Before calling something gaming, one has to define what counts as unusual against what would normally be expected (p. 217). She sorts unusual data into four types: changes in normal behaviour, real bursts of attention, background noise from third parties such as bots and link farms (networks of sites built to create fake links), and deliberate manipulation. Only the last is gaming, and it requires intent (pp. 217–218, table 16.1).

To spot gaming she points to algorithms that track trends and sudden bursts, to cross-checking between sources, and to machine learning (pp. 219–220). On oversight, she argues that existing procedures for handling misconduct could cover altmetrics gaming (pp. 220–221). Her deeper point is to move the focus from hunting cheats to keeping the data sound, a responsibility everyone shares (p. 221). She describes Crossref's Event Data service, an open central record of the raw events behind altmetrics, as a way to standardize the data and allow independent checks. Alongside it she places the code of conduct for altmetrics providers from NISO, a US standards body (pp. 222–223).

Chapter 17. Elizabeth Wager, "Why We Could Stop Worrying About Gaming Metrics If We Stopped Using Journal Articles for Publishing Scientific Research"

Gaming arises, Wager argues, because the journal article does two jobs: it reports research, and it is used to measure researchers (p. 229). Her radical answer is to stop using it for the first job, at least for research that produces numerical data (p. 229). Her example is clinical trials, the studies that test treatments on patients.

Despite the pressure to publish, as many as half of clinical trials are never published, and the missing ones are mostly those with disappointing results (pp. 229–230). Many trials are only partly reported, because a three-thousand-word article has no room for the details (p. 230). Some studies find that around 60 per cent of published trials change or leave out the main outcome named in the protocol, the plan written before the trial began (pp. 230–231). Peer reviewers rarely see the protocol or the raw data, so they cannot catch selective reporting, wrong statistics or skilful fabrication (pp. 230–231). The format has changed little in 350 years (p. 231). Results also appear in many versions that are not linked to each other, such as abstracts, entries in trial registries (public databases of trials) and reports to regulators. These may contradict each other, notably on harmful side effects (p. 232).

Her alternative is a linked, structured report that computers can read, connecting the data, the protocol and the registry entry (pp. 232–233). It might be dull to read, she admits, but that is a small price for accuracy (p. 233). Funders could hold back part of a grant until the results are posted, and new forms of quality control would probably replace peer review before publication (p. 233). Once journals stop being the "currency" of science, better metrics could be built, including credit for data (p. 233). Journals would remain for commentary and for pulling findings together (p. 233).

The barriers to change, she writes, are

more social and cultural than technological. (p. 232)

Such change has been possible, and has been proposed, for almost twenty years (p. 233).