Thursday, June 28, 2012

The secret life of trials....


When trials disagree...it can get ugly! But going into meta-analysis could help sort things out.

For a meta-analysis - a technique for combining the results of multiple trials - trials have to pretty much belong together. Differences might be responsible for contradictory results - including differences in the people in the trials, the way they were treated, or the way the trials were done. That's called heterogeneity. Too much of it, and the trials shouldn't be together. But heterogeneity isn't always a deal breaker.

Want to read more about heterogeneity in systematic reviews? Here's an article by Paul Glasziou and Sharon Sanders from Statistics in Medicine [PDF]. Or try the open learning materials from the Cochrane Collaboration.



Monday, June 18, 2012

Promising = over-hyped + under-tested


We've apparently been using the word "promising" to mean "showing signs of future excellence" since about 1600. I first wrote about the tendency of "promising" treatments to metamorphose into "disappointing" treatments in a BMJ piece about evidence based mistakes. Early results, after all, can't promise anything at all.

For all sorts of reasons, research findings are themselves over-positive - or as John Ioannidis put it, false. In another of his studies, he found that it happens to even the most highly cited clinical studies in the "best" journals: a third have been contradicted or turned out to have over-estimated benefits.

My cartoon graph depicts a cumulative meta-analysis: each new study is being absorbed into a summation of the evidence so far. With 4 studies, it's shifted from the "this helps" side of the ledger over to the "this harms" side. See more about cumulative analyses in this classic article.

Speaking of classic articles, here's another: in the BMJ's 2015 tongue-in-cheek christmas issue, "promising" was one of the positive hype words Christiaan Vinkers and colleagues analyzed in PubMed's abstracts - from 1974 to 2014.





"Novel" was another favorite - it was one of the words with an increase of up to 15,000%: "At this rate, we predict that the word 'novel' will appear in every record by the year 2123"! A bold, innovative take!

So is "promising" still increasing? Yep! It was 2.4% of 2014's PubMed records - and 2.8% in 2016.

"Promising" is a media and marketing staple, too. In the last 24 hours, you can find "promising" results in Google News for kidney cancer "options", a skin antibiotic test, candidates for new antibiotics, leishmaniasis treatment, skin infections, a "novel" anti-tumor DNA vaccine, marijuana for Hepatitis C...

Several initiatives aim to keep the media to account on this, story by story: Behind the Headlines, Germany's Median-Doktor.de, Japan's Media Doctor, and the US Health News Review.

Yet the breathless hype marches relentlessly on. On the plus side, at least words like "promising" mean that sometimes at least, marketing or optimism bias comes clearly labeled!



This post was updated on 25 February 2017, adding the study and data on the use of "promising" in PubMed.


Wednesday, June 13, 2012

Begging hopefully for less bias


From my guest blog post at Scientific American - Holy sacred cow!

Personal bias, wishful thinking plus biased research results is one recipe for a sacred cow. If more rigorous research results in a conflicting message, it could cause cognitive dissonance - and the less biased research often faces an uphill battle for acceptance.

And I also wrote about the importance of being just as rigorous about the claims we want to believe as those we're skeptical of here at the British Journal of Sports Medicine blog.

If you want to get better at critically assessing health claims, Smart Health Choices is a great place to start.

Wednesday, June 6, 2012

The trial acronymania menace


As if there's not enough for us to remember, we're supposed to remember endless acronyms for trials too now. There's even a wiki to help us keep them straight and a call for a register of trial acronyms to reduce multiple use of all the words ending in T!

Somewhere along the line this became marketing: not much equipoise in ACHIEVE, MIRACLE or PROMISE, eh? A study has classified this as a form of coercion. Ivan Oransky called for a HALT (Help Acronyms Leave Trials). If you're irritated by the next outbreak of trial acronymania or acronymesis you come across, you're not alone!


Another trial acronym here at Statistically Funny.....Meet the AGHAST Investigators!