Errors found in Nature and BMJ papers
3 June 2004
Nature News Service
A study highlighting statistical gaffes in scientific literature has brought renewed calls for vigilance among mathematically challenged researchers and journal editors.
Statistical tests are sometimes seen as a necessary evil by researchers, who fear their complexity but know that they are needed to test hypotheses. With this aversion in mind, biostatisticians Emili García-Berthou and Carles Alcaraz of the University of Girona, Spain, gauged the extent of statistical errors in four volumes of Nature from 2001 (409–412) and a sample of results in two BMJ volumes (322–323) from the same year.
The pair used three standard software packages to recalculate 'P values', the parameters by which researchers measure whether a result has statistical significance. Generally a P value of less than 0.05 is taken to be significant and unlikely to have resulted from chance. This may indicate, for example, that blood pressure in a patient group was reduced more by an active drug than by a placebo.
In the Nature and BMJ papers, each P value was calculated from two other parameters that are included in the papers. García-Berthou and Alcaraz recalculated the P values from these numbers, and found that their results differed from those published in more than 11% of cases. They also found small mistakes, such as rounding errors, in 38% of the Nature papers and 25% of the BMJ ones1.
In only 1 case out of 27 did an incorrect P value change a significant result to a non-significant one. But, although minor, some believe that the slip-ups expose a pervasive sloppiness towards statistics in published research. "There are small mistakes that may occasionally have big consequences," says Martin Bland, an expert in medical statistics at the University of York, UK.
Philip Campbell, the editor-in-chief of Nature, says the journal will take a closer look at the study's numbers before deciding whether remedial action is needed. He adds that Nature has amended its editing practices since the period covered by the study.
Richard Smith, editor of the BMJ, says that one way forward is for researchers
or journals to publish more raw data on the Internet, where others would
be able to check them.
Copyright © 2004, Nature News Service