Is the current peer-review system really giving us the best possible e-cig science?

Bafflement, frustration, annoyance: anyone who’s been around the e-cigarette sector for a while is familiar with the emotions prompted by much of the science published in this area.

There is plenty of useful and well-constructed research drawing reasonable conclusions too, of course – the problem isn’t a complete absence of decent scientific work, it’s the way that much of it is crowded out by more questionable papers that start from dubious assumptions, or leap to conclusions, or simply don’t provide clear enough data to let us know exactly what they are doing.

Media reporting, and the media departments of academic institutions anxious to obtain coverage for their scientists, can make matters worse (indeed, my colleague Aidan Semmens blogs this week about one such case).

Of course, clearer-headed observers – some scientists, some not – do publish their own analyses of what’s wrong with such research. Occasionally, corrections get published alongside the original work. Very very occasionally, papers get withdrawn. Yet the damage is already done, in an area where policy is led much more by non-scientists’ perceptions of “the science” than by actual scientific consensus (as indeed is public opinion).

But is chipping away at a flawed edifice the only solution? The psychologist Stuart Ritchie of King’s College London has a more radical idea: do away with the current system of peer-reviewed papers in edited journals altogether.

Today’s journal-led norm for publishing science “comes with big problems”, argues Ritchie in The Guardian: “Paradoxically, the sacred status of a published, peer-reviewed paper makes it harder to get the contents of those papers right.”

The biggest issue, he says, is that of publication bias, the scientific world’s equivalent of media sensationalism: “Reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. So scientists go to great lengths to hype up their studies, lean on their analyses so they produce ‘better’ results, and sometimes even commit fraud in order to impress those all-important gatekeepers”, the editors and peer-reviewers.

 

Identifying and countering bad research

Subscribe to our newsletter

Join in to hear about news, events, and podcasts in the sector

"*" indicates required fields

Name*
This field is for validation purposes and should be left unchanged.

 

What are his alternatives? “Maybe the decision to publish could be made based only on the methodology of a study, rather than on its results” – something which, in the tobacco world, would focus attention on thoughtful experimental design (where it is often lacking) at the same time as reducing the incentive to provide Big Answers at all costs.

Or going even further, “maybe scientists could just publish all their research by default, and journals would curate, rather than decide, which results get out into the world”. In an ideal scenario, suggests Ritchie, instead of predigesting extensive work for relatively short papers published in journals, scientists would self-publish “notebooks” – websites containing all the data and analysis for a specific project.

This, too, would be welcome in the tobacco-alternatives world. Not only might it draw more attention to problematic issues such as methodology and definitional conventions, the availability of full data sets would enable third parties to conduct their own analyses – even where the “wrong” approach was taken by the original researchers. It might not eliminate bad tobacco research, but it could make it easier to identify and argue against.

Ritchie’s vision may be a dramatic one, but he is far from alone in seeing systemic shortcomings in journal-dominated science publishing, and looking for alternatives. The online research journal eLife, for example, retains some editorial control over content – it doesn’t accept everything that’s submitted – but it strives for a less competitive, more open model.

For example, it publicly distances itself from the “impact factor” concept (which gives journals a ranking of importance based on how often they are cited, and is widely regarded as putting pressure on scientists to get their work into the highest-ranked journals). It says “we do not artificially limit the number of articles we publish or have a set acceptance rate”, again to minimise the temptation to hype up. And, importantly, it invites public comments as part of peer review.

Journal publishing is not going to disappear overnight, if ever. It’s a very big business with many vested interests and, of course, a great track record in publishing important science as well as questionable science – it certainly shouldn’t be dismissed or demonised. But particularly in areas like tobacco, where there is so much political and media interest (and most of it headed in one direction), its weaknesses can be all too visible.

– Barnaby Page ECigIntelligence staff

Artwork: Mohamed Hassan

  • If you are interested in how science and the media can affect the vaping market, you can now download for free our full briefing “Warning on youth vaping ‘a desperate effort to maintain a crisis narrative’” using the form at the top of this page.

Barnaby Page

Editorial director
Before joining ECigIntelligence in early 2014 as one of its first employees, Barnaby had a 30-year career as a reporter and editor for newspapers, magazines and online services, working in Canada, the US and the Middle East as well as his current British location. He has edited publications covering fields including technology and the advertising industry, and was launch editor of the first large daily online news service in the British regional media. Barnaby also writes on classical music and film for a number of publications. Barnaby manages the editorial and reporting teams and works closely with the analyst teams, to ensure that all content meets high standards of quality and relevance. He also writes for the site occasionally, mostly on science-related issues, and is a member of the Association of British Science Writers.