Does green tea extract lower fasting triglycerides?
Let's put the method back in the scientific method.
After I wrote about a trial on probiotics last week a few people contacted me with questions about the whole multiple comparisons thing.
It’s so hard nowadays to determine what is “true”, and that goes for academic publishing too. Maybe at some point “peer-reviewed publication” had some meaning. Not any more.
So I thought I would use this post to give you a hypothetical example of how and why scrutinising the methods of a publication is literally the only way of knowing whether the conclusions reflect what the study actually found.
There are multiple aspects of a method we could look at, but in clinical trials one of the most important is the primary outcome. In this post I’ll focus on why you need one, and why investigators need to tell you what it is ahead of time.
Primary Outcome or GTFO
Imagine you see this title and abstract in a peer-reviewed scientific journal:
Looks pretty good doesn't it? It's published in what looks like an academic journal, it's a randomised controlled trial, and they see a significant effect.
Should people take green tea extract (GTE) to lower their triglycerides now?
If I saw a paper like this published, the first thing I would do is go and check in the method section whether the trial is registered and whether there was a single defined primary outcome*. Why?
Because:
Any single statistical test will yield a false positive 5% of the time. And you will be running a statistical test (whether a t-test or ANCOVA etc) for every outcome you measure. So a sort of rule of thumb is that for every 20 outcomes you measure, one of them will change “significantly”.**
So let's imagine the investigators of this study actually measured 20 different outcomes and they get the following results:
Did the triglycerides lower significantly because green tea extract is an efficacious biological agent which can lower triglycerides, or is it a false positive borne from measuring so many outcomes? We don’t know.
We would have more confidence in this result if fasting triglycerides were defined as the single primary outcome in a clinical trial registry. Because by saying “by THIS [and only this] outcome will we judge the effectiveness of green tea extract” you are not running 20 statistical tests; you are running one. And if that one statistical test yields a positive result, we have greater confidence that it’s not a false positive.
A primary outcome is registered something like this:
The primary outcome has to be DEFINED. This includes the what, how and when of your measurement.
Defining what and how is critical. For example, is “glucose” one primary outcome? Depends on the study. You could do a meal tolerance test with glucose measured at frequent intervals, measure glucose via a continuous glucose monitor (CGM) and measure HbA1c in a single study. (The methods section should tell you all the things the investigators did). And in this case, “glucose” is not therefore one outcome; it could be >10 outcomes:
Fasting glucose during MTT
Fasting glucose on CGM
HbA1c
Peak glucose on MTT
30-minute glucose on MTT
60-minute glucose on MTT
Time to peak glucose on MTT
incremental area under the curve (iAUC) on the MTT
total area under the curve (tAUC) on the MTT
Mean glucose on CGM
Post-prandial glucose on the CGM
etc.
When is important because some trials may measure an outcome at 3, 6, 9 and 12 months. Well, that’s four statistical tests (at least) - so investigators must define the exact timeframe of when they are measuring their primary outcome.
This is why checking a clinical trial database (eg clinicaltrials.gov or ISRCTN), and looking for these details is always a key first step in assessing the quality of a trial.
Investigators should register their trials on databases such as these before they start recruiting so that they are putting on record their method and primary outcome ahead of time. This is supposed to prevent investigators from changing their method halfway through (potentially because they’re not getting the results they want to see…) or from not having or changing a primary outcome ahead of time (both as bad as each other).
Which brings me onto….HARK-ing
HARK-ing
If investigators don’t register a primary outcome, then there is a risk that what they will do is HARK-ing: Hypothesising After Results are Known. What this means is they run a bunch of tests, then based on whatever significant result(s) come back they then write their hypothesis, and write the paper up as though they were always planning to measure that thing ahead of time. Needless to say this is very poor practice***. Lol.
“We’re measuring changes in the gut microbiome in our study”
Just to end with this: The gut microbiome is comprised of around 8 phyla, 18 families, 23 classes, 38 orders, 59 genera and 109 species of microbes. If measuring 20 outcomes results in potentially one false positive, what do you think happens when we measure “changes in the microbiome” as an outcome?
Coming next post: “Why I laugh at most microbiome trials”. Hahahaha.
*In some studies it may be more appropriate to have joint primary outcomes, but the reason for having more than one has to be clearly justified in the protocol. The vast majority of studies should have just one.
**I am simplifying this so if any stats boffins read this blog please don’t yell at me.
***By the way, I didn’t always know this stuff. In my early papers I didn’t always do a great job of getting all these things right. Also when you start doing research sometimes you just get lost in the pressure of getting funding and publishing, that it’s easy to lose sight of why you are doing what you’re doing. Also to defend researchers a bit - there is now SO MUCH bureaucracy in research, so many forms to fill in, reports to write, hoops to jump through to get the 2638672 approvals you need to start, that even though you intend to register your trial properly and on time etc, it can get missed.
Each article makes us less dumb. Thank you Nicola!
love it. Anything that is not a clinical trial, with no primary outcome specified beforehand, is it liable to such biases?