Few Replication Studies in Environmental Science

A recent survey in PeerJ — a multidisciplinary bioscience journal — by one of its academic editors has found that only 0.023 per cent of the ecology and evolution literature involved successful replication studies. In absolute terms, this is nine replications out of 38,730 papers published in one access journals. The survey is here.

For more than a decade, science has been convulsed by the world-wide phenomenon known as the replication crisis. The term refers to the fact that most of the independent attempts to replicate the findings of previous published research have failed to do so, and when such failures have ben reported in the literature they have generally been ignored.

All the scientific disciplines have been affected by the crisis and most have tried to do something about it. As a result there has been an increase in the prevalence of published replication studies, albeit from a very low historical base.

Not so in the environmental sciences, it seems.

The PeerJ survey found that only two of the 11 original-replication study pairs had sufficient detail for the author to conduct a formal analysis of replication success. He concluded that two teams had correctly concluded their replication was successful, but disagreed that a third team had conducted a failed replication, as it had claimed.

Subsequently Ecology and Evolution published a survey of environmental scientists to gauge their understanding of and opinions about replication studies in their fields. The published paper is here.

The majority of respondents considered replication studies were important (97 per cent), not prevalent enough (91 per cent), worth funding even given limited resources (61 per cent), and suitable for publication in all journals (62 per cent).

There is a vast gap between this expressed enthusiasm and the prevalence of replication studies in the literature. For example, the prevalence rate, reported above, is three magnitudes below the median estimate of 10 per cent given by survey respondents. A gap of this magnitude suggests that either the respondent have faked the enthusiasm or are profoundly ignorant of the literature in their discipline.

For their part, the survey authors thought the gap might be due to the difficulties involved in getting funding for replication studies, conducting them, and having their results published.