On the optimistic performance evaluation of newly introduced bioinformatic methods

Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods”, but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods; better ability to fix bugs in a preferred method; and selective reporting of method variants. We quantitatively investigate this bias using a topical example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.

Download Full-text

On the optimistic performance evaluation of newly introduced bioinformatic methods

Genome Biology ◽

10.1186/s13059-021-02365-4 ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Stefan Buchka ◽

Alexander Hapfelmeier ◽

Paul P. Gardner ◽

Rory Wilson ◽

Anne-Laure Boulesteix

Keyword(s):

Data Analysis ◽

Selective Reporting ◽

Optimistic Bias ◽

Analysis Methods ◽

Normalization Methods ◽

Epigenetic Analysis ◽

Humanmethylation450k Beadchip ◽

Data Analysis Methods ◽

Selection Of ◽

Better Than

AbstractMost research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.

Download Full-text

Analysis of ionospheric electrodynamic parameters on mesoscales – a review of selected techniques using data from ground-based observation networks and satellites

Annales Geophysicae ◽

10.5194/angeo-29-467-2011 ◽

2011 ◽

Vol 29 (3) ◽

pp. 467-491 ◽

Cited By ~ 17

Author(s):

H. Vanhamäki ◽

O. Amm

Keyword(s):

Data Analysis ◽

Data Sets ◽

Practical Applications ◽

Analysis Methods ◽

Ionospheric Electrodynamics ◽

Simplifying Assumptions ◽

Using Data ◽

Measured Input ◽

Data Analysis Methods ◽

Selection Of

Abstract. We present a review of selected data-analysis methods that are frequently applied in studies of ionospheric electrodynamics and magnetosphere-ionosphere coupling using ground-based and space-based data sets. Our focus is on methods that are data driven (not simulations or statistical models) and can be used in mesoscale studies, where the analysis area is typically some hundreds or thousands of km across. The selection of reviewed methods is such that most combinations of measured input data (electric field, conductances, magnetic field and currents) that occur in practical applications are covered. The techniques are used to solve the unmeasured parameters from Ohm's law and Maxwell's equations, possibly with help of some simplifying assumptions. In addition to reviewing existing data-analysis methods, we also briefly discuss possible extensions that may be used for upcoming data sets.

Download Full-text