Via Wolfgang Flamme, a reader of Stephen McIntyre's blog.
In his article,
has analyzed various things like the publication bias (attempts to get a signal even though there is no signal and select the findings that have found one); the reverse bias (less frequent; tendency to hide signals); the probability that a finding is published as a function of the previous probability estimate that the result is correct, and some other factors. Insert these variables into some matrix differential equations and look at the outcome.
The conclusion is quite clear: most published research findings are false. Well, let me mention that the analysis focuses on life sciences but the conclusions could probably apply to many other fields of science, too. Hopefully not all of them. The fields based on statistics and phenomenology are probably affected by the mechanisms mentioned by Ioannidis.
The author shows that the probability that a published finding is false increases (or is greater) if and when
- the sample size shrinks - obviously, the quality of statistics decreases
- the size of the "effect" in a given subfield becomes small; not surprisingly, it is harder to measure whether XY increases the risk of UV by 0.1%
- the fraction of relationships that are tested becomes small (e.g. when the total number of possible relationships becomes large)
- designs, outcomes, definitions, and analytical modes of the field are flexible
- financial and other interests and prejudices are high
- the scientific field is hot (studied by very many teams); this point may be surprising to many
I don't guarantee that Ioannidis' article is quite correct - don't forget that it is published and most published findings are false :-) - but it is definitely an interesting set of ideas and mechanisms that may deserve a further mathematical simulation.
Incidentally, you can look at the list of the factors that increase the probability of false publications and compare them with the "global climate change". It is not hard to see that this research "passes" all of these tests. It studies one number - the average temperature - in a very small number of years. The anthropogenic effect is smaller - in fact much smaller than the natural interannual variations. Among millions of factors that may influence the climate, the "experts" only test a few, especially the relation between the CO2 concentration and the average temperature. They can adjust their GCMs in millions of different ways. They have huge financial and political interest to have a certain kind of outcome. And finally, the field is "hot" and studied by many people; they admit it themselves and they are even proud about it - which is why we often hear the oxymoron "scientific consensus".
