Science journals and weak science

Mar 03, 2023

You may have seen some headlines this week that a widely used sweetener (erythritol) might increase the risk of thrombosis (blood clotting) and serious CVD events like strokes and heart attacks. The article itself was published in Nature Medicine which is a journal that has a reputation for publishing high quality science. I read through the methods of the paper and was like “yo, really?”, and was so annoyed by how weak it was (THEY DIDN’T EVEN MEASURE ERYTHRITOL INTAKES) I did a tweetorial on it:

The thing that most disappointed me was the trial which is mentioned in the manuscript. There are lots of methods researchers use to understand the relationship between diet/nutrition and health/disease, and each has their strengths and limitations. However, unarguably, if you really want to know whether erythritol causes thrombosis, you’d really want a study in humans, ideally a randomised control trial. The authors of this paper do have a trial registered, but it’s not randomised (which is weird anyway). Even weirder though is that they don’t even report their [non-randomised] clinical trial data in their manuscript.

Their registered trial says it’s going to test the effect of 30g erythritol on blood clotting in 20 people (the other 20 people are going to get 30g xylitol). Blood coagulation is not my area so I can’t speak to how valid their method of measuring blood clotting is from a technical standpoint. Nevertheless, in a paper which aims (and claims) to explore the effect of erythritol on CVD risk, in vivo outcome data seems like…the essential piece of the puzzle?

So, you’d think they’d publish this data, right? No. Instead (no idea why) they present pharmacokinetic data from 8 people – essentially showing that consumption of 30g of erythritol causes really high concentrations of erythritol in the blood. I mean, this is nice data [notwithstanding the fact that 30g in one go is a crazy high dose]. But again, why refer to a clinical trial you are running which is actually going to look at markers of thrombosis and then not include the data?

Their trial registration states that they are only recruiting 40 people in total. In our trials in type 2 diabetes we typically recruit 10-15 people per month. Since this publication was submitted in July 2022, and the trial was registered back in 2021, I am surprised this trial is not finished. To be fair to the authors, sometimes bureaucracy and other things can delay a trial by months, or even years. So maybe it’s not finished yet. The clinical trial registry says it is still recruiting. And recruitment for studies can be really, really hard. But I really question – given the necessity of in vivo trial data in determining “is this even relevant in humans?” - and the very weak observational data they have - why the authors wouldn’t wait for the clinical trial data to help answer their own speculative findings. Also…

WHY THE JIGGINS DIDN’T THE JOURNAL INSIST ON WAITING FOR THE CLINICAL TRIAL DATA?

man in black crew neck t-shirt — Photo by Usman Yousaf on Unsplash

Because of course, the ultimate goal of academic journals is to disseminate the highest quality science.

HAHAAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

Academic journals make higher profits than Microsoft, Google and Coca Cola, and the amount of cashola they’re making is going up all the time. Academic publishing is a business. Like any commercial publishers, they generate more revenue the more subscribers they have, the more clicks they have (advertising revenue is not insignificant), and loosely, the more attention the work they publish gets.

And what gets attention? SEXY/NEW/CONTROVERSIAL WORK, especially if it’s a food or food product which might be killing us. Seriously. The number of utterly flawed studies I see in the highest impact (supposed to be highest quality) journals is worrying.

Nutrition studies also seem to be readily picked up by the media, which can then drive members of the public to the journal site (and thereby enhancing the journal’s reputation and potential advertising revenue further).

But surely you’ve got checks and balances in this system because….

Academic institutions behind the research care about the quality of research they do, don’t they?

HAHAAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

Academic institutions* are businesses. They too have an interest in marketing themselves and their research and getting your institution in the press probably really helps.

But what about scientists – surely…

woman looking on microscope inside room — Photo by Trust "Tru" Katsande on Unsplash

Academic scientists have an incentive to produce the most careful, robust research they can?

HAHAAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

I am a fairly junior researcher so maybe I am wrong here, but based on what I have observed in my 10-15 year career, there are far more incentives for chasing sexy, novel findings – robust data or not - than there are for producing high-quality reproducible work.

For example: for scientists, a Nature Medicine/Cell Metabolism/New England Journal of Medicine publication gets you promotions. It massively increases your chances of getting funding. And funding gets you promotion. In other words, there’s a huge financial incentive to aim for these highest impact journals, and since they’re very interested in novel/controversial/sexy work, guess what lots of researchers do?

Don’t despair though!

landscape photography of person's hand in front of sun — Photo by Marc-Olivier Jodoin on Unsplash

I don’t want to give the impression that no one in academia cares about doing valuable work that actually helps solve genuine problems. At Oxford, I am surrounded by really smart, creative, visionary people who really want to make a difference in the world

And just from the institutions I have worked at (I had nothing to do with these studies because I am not that clever ha ha)– here are some recent academic discoveries that have literally transformed people’s lives:

Work by Professor Frances Ashcroft at the University of Oxford led to babies with neonatal diabetes (ND) being able to switch from daily insulin injections to simple tablets.
- Neonatal diabetes is a potentially devastating monogenic form of diabetes that develops within the first six months of life and can be severe. Prof Ashcroft and collaborators showed that the genetic defect in neonatal diabetes is one that could be fixed with a widely-used type 2 diabetes medication called sulphonylurea. Before their discovery, these babies were assumed to have type 1 diabetes and had to go on life-long insulin therapy. Sulphonylurea treatment has transformed the quality of life of these patients and their families including reducing the mental and motor developmental delay that affects about a fifth of neonatal diabetes patients.
Professor Waljit Dhillo at Imperial College London and his team carried out research on a hormone called kisspeptin which they believe could be a safer option for women undergoing IVF.
- About one in every 100 women undergoing IVF will develop a severe form of ovarian hyperstimulation syndrome (OHSS). In OHSS, the ovaries overreact – in most women symptoms are mild and the women recover, but severe OHSS can occur which can cause severe illness and even death. Prof Dhillo and his team have tested the hormone kisspeptin which stimulates production of eggs but in a more gentle way. It has been tested in a clinical trial and the first kisspeptin IVF baby was born in 2013. Work is ongoing to continue testing, and scaling up this potential treatment.
The RECOVERY trial – looking at the efficacy of medications to treat severe COVID19.
- Lots of folks during the pandemic were complaining about the lack of evidence for things because noone was doing any RCTs. Not true. In crazy quick time, the RECOVERY (Randomised Evaluation of COVID-19 therapy) project was set up to examine the effect of different drugs on COVID-19 recovery for hospitalised patients, and discovered that dexamethasone (a widely used steroid) and tocilizumab (a drug for arthritis) were able to dramatically reduce the risk of death for patients with severe COVID-19.
- The brilliance of this project was to ensure that as many patients as possible who needed hospital care were randomised to one of the treatment arms. You might think this is unethical, but the point is that in the early days of the pandemic, no one had any idea what worked and what didn’t. The RECOVERY trial played a critical role in helping us understand how to save people’s lives.

So there are good people doing good research, and there are lots of good people who are genuinely doing their best to counter the bad science that’s out there. I’ll try to follow up in future posts and give some general tips on how to tell if something/someone is legit or shady :-)

*I can only speak for the UK & USA here based on my experiences.

Dr Guess’s Blog

Discussion about this post