A fascinating insight

Why is so much published research later contradicted (e.g. value of hormone replacement therapy and vitamin E)? One reason maybe that "high profile" research is statistically more likely to be a fluke. This article in JAMA in July showed that 16% of highly cited research studies were later contradicted and others were either never replicated or showed a magnitude of effect that was not replicated.

Now the letters in response to the article are out, and one goes a long way to explaining the paradox. It points out that research studies are analagous to diagnostic tests and p-values can be roughly translated into the false positive rate. But of course the false positive rate itself doesn't tell us much about how to interpret a positive test, what we really want to now are it's positive predictve value, which depends on the prior probability of the disease.

For non-medicos I apoligize if you can't follow, but I am not the one to explain medical statistics, but basically this means that EKG changes in a 70 year old diabetic smoker with known high cholesterol are a lot more likely to accurately diagnose a myocardial infarction than similar changes in an 18 year old triathlete.

Now in what situation is a test with a low false positive rate (i.e. a high specificity) likely to lead a clinician astray? When the a priori chance of the patient having the disease is very low.

Hear is where the clever insight comes in. By its very nature, highly publicized research has a low a priori chance of being correct. That a new antibiotic is effective for cellulitis is no big deal, that a bacteria causes ulcers is (though this was true, of course). So, studies that are truly novel are also more likely to be wrong, regardless of p value. Pretty cool.

Of course one problem with this Bayesian reasoning is that it is hard enough in clincial care (how exactly do I know if a patient has a 40 or 70% a priori chance of having asthma?) but completely impossible in research. As Colin West, the letter's author puts it

As noted by Davidoff,one solution to this quandary is to better understand Bayesian approaches to study analysis and interpretation. Until such approaches are rendered comprehensible to the average researcher, however, we should at least understand the limitations of the P value

The author of the original study even has a paper arguing that the majority of published research is wrong (Ioannidis JPA. Why most published research findings are false. PloS Med. 2005;2:e124.) but I can't get it to load on the PLOS website