Evaluation of statistical tests for the detection of bias in meta-analyses with binary outcomes

Dr. G. Schwarzer, FDM, Institute of Medical Biometry and Medical Informatics
Dr. G. Antes, German Cochrane Centre, Freiburg
Prof. Dr. M. Schumacher, FDM, Institute of Medical Biometry and Medical Informatics

• Summary of the project

The impact of systematic reviews and meta-analysis as statistical method to combine individual trial results has rapidly grown in the medical field. The validity of a systematic review may be affected by several factors including subjectivity in the selection of trials, heterogeneity of trial results and various sources of bias. Especially, publication bias has been discussed as a source of major concern (1).

Statistical tests for the detection of bias in meta-analyses have been developed during the last decade. A common assumption of these tests is that effect estimates are normally distributed with known variances. The properties of these tests in meta-analyses with binary outcomes had not been examined in detail in the past.

We conducted a simulation study to investigate the properties of two commonly used tests for the detection of bias in meta-analysis with binary outcomes (2). Simulation results indicate an inflation of type I error rates for both tests when the data are sparse. Results get worse with increasing treatment effect and number of trials combined.

A new test for the detection of publication bias in meta-analysis with sparse binary data has been developed. The test statistic is based on observed and expected cell frequencies, and the variance of the observed cell frequencies. These quantities are utilized in a rank correlation test. Simulation results indicate that, in contrast to existing test procedures, the new test holds the prescribed significance level when data are sparse. However, the power of all tests is low in many situations of practical importance (3).

• Publications

(1) Schwarzer G, Galandi D, Antes G, Schumacher M
Meta-Analysen randomisierter klinischer Studien, Publikations-Bias und Evidence-Based Medicine (EBM), Informatik, Biometrie und Epidemiologie in Medizin und Biologie, 31, 2000, 1-21

(2) Schwarzer G, Antes G, Schumacher M
Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes, Statistics in Medicine, 21, 2002, 2465-77

(3) Schwarzer G, Antes G, Schumacher M
A Test for Publication Bias in Meta-Analysis With Sparse Binary Data, Statistics in Medicine, accepted, 2006.