A test for reporting bias in trial networks: simulation and case studies

BMC Medical Research Methodology

Philippe Ravaud²  Gilles Chatellier⁴  John PA Ioannidis¹  Ludovic Trinquart³ 

[1] Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA;Université Paris Descartes - Sorbonne Paris Cité, Paris, France;Department of Epidemiology, Columbia University Mailman School of Public Health, New York, NY, USA;Assistance Publique-Hôpitaux de Paris, Hôpital Européen Georges Pompidou, Unité de Recherche Clinique, Paris, France

关键词: Comparative effectiveness research;    Randomized controlled trials;    Test of bias;    Selective outcome reporting;    Publication bias;   

Others  :  1090821
DOI  :  10.1186/1471-2288-14-112

 received in 2014-04-02, accepted in 2014-09-18,  发布年份 2014

【 摘 要 】

Background

Networks of trials assessing several treatment options available for the same condition are increasingly considered. Randomized trial evidence may be missing because of reporting bias. We propose a test for reporting bias in trial networks.

Methods

We test whether there is an excess of trials with statistically significant results across a network of trials. The observed number of trials with nominally statistically significant p-values across the network is compared with the expected number. The performance of the test (type I error rate and power) was assessed using simulation studies under different scenarios of selective reporting bias. Examples are provided for networks of antidepressant and antipsychotic trials, where reporting biases have been previously demonstrated by comparing published to Food and Drug Administration (FDA) data.

Results

In simulations, the test maintained the type I error rate and was moderately powerful after adjustment for type I error rate, except when the between-trial variance was substantial. In all, a positive test result increased moderately or markedly the probability of reporting bias being present, while a negative test result was not very informative. In the two examples, the test gave a signal for an excess of statistically significant results in the network of published data but not in the network of FDA data.

Conclusion

The test could be useful to document an excess of significant findings in trial networks, providing a signal for potential publication bias or other selective analysis and outcome reporting biases.

【 授权许可】

   
2014 Trinquart et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

【 参考文献 】

• [1]McGauran N, Wieseler B, Kreis J, Schuler Y, Kolsch H, Kaiser T: Reporting bias in medical research - a narrative review. Trials 2010, 11:37. BioMed Central Full Text
• [2]Song F, Parekh S, Hooper L, Loke Y, Ryder J, Sutton A, Hing C, Kwok C, Pang C, Harvey I: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010, 14:1-193.
• [3]Hopewell S, Loudon K, Clarke M, Oxman A, Dickersin K: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009, 1:MR000006. doi:10.1002/14651858.MR000006.pub3
• [4]Dwan K, Altman D, Arnaiz J, Bloom J, Chan A, Cronin E, Decullier E, Easterbrook P, Von Elm E, Gamble C, Ghersi D, Ioannidis J, Simes J, Williamson P: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008, 3:3081.
• [5]Begg C, Mazumdar M: Operating characteristics of a rank correlation test for publication bias. Biometrics 1994, 50:1088-101.
• [6]Harbord R, Egger M, Sterne J: A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med 2006, 25:3443-57.
• [7]Macaskill P, Walter S, Irwig L: A comparison of methods to detect publication bias in meta-analysis. Stat Med 2001, 20:641-54.
• [8]Rücker G, Carpenter J, Schwarzer G: Detecting and adjusting for small-study effects in meta-analysis. Biom J 2011, 53:351-68.
• [9]Rücker G, Schwarzer G, Carpenter J: Arcsine test for publication bias in meta-analyses with binary outcomes. Stat Med 2008, 27:746-63.
• [10]Schwarzer G, Antes G, Schumacher M: Inflation of type i error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes. Stat Med 2002, 21:2465-77.
• [11]Schwarzer G, Antes G, Schumacher M: A test for publication bias in meta-analysis with sparse binary data. Stat Med 2007, 26:721-33.
• [12]Sterne J, Sutton A, Ioannidis J, Terrin N, Jones D, Lau J, Carpenter J, Rücker G, Harbord R, Schmid C, Tetzlaff J, Deeks J, Peters J, Macaskill P, Schwarzer G, Duval S, Altman D, Moher D, Higgins J: Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011, 343:4002.
• [13]Ioannidis J, Trikalinos T: The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 2007, 176(8):1091-1096.
• [14]Ioannidis J, Trikalinos T: An exploratory test for an excess of significant findings. Clin Trials 2007, 4:245-53.
• [15]Sterne J, Egger M, Moher D: Chapter 10: Addressing reporting biases. In Cochrane Handbook for Systematic Reviews of Interventions. Edited by Higgins JSG. Chichester, UK: The Cochrane Collaboration and John Wiley and Sons Ltd; 2011:297-333.
• [16]Chatzinasiou F, Lill C, Kypreou K, Stefanaki I, Nicolaou V, Spyrou G, Evangelou E, Roehr J, Kodela E, Katsambas A, Tsao H, Ioannidis J, Bertram L, Stratigos A: Comprehensive field synopsis and systematic meta-analyses of genetic association studies in cutaneous melanoma. J Natl Cancer Inst 2011, 103:1227-35.
• [17]Ioannidis J: Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry 2011, 68:773-80.
• [18]Kavvoura F, McQueen M, Khoury M, Tanzi R, Bertram L, Ioannidis J: Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to alzheimer’s disease. Am J Epidemiol 2008, 168:855-65.
• [19]Tsilidis K, Papatheodorou S, Evangelou E, Ioannidis J: Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. J Natl Cancer Inst 2012, 104:1867-78.
• [20]Tzoulaki I, Siontis K, Evangelou E, Ioannidis J: Bias in associations of emerging biomarkers with cardiovascular disease. JAMA Internal Med 2013, 173:664-71.
• [21]Ioannidis J: Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ 2009, 181:488-93.
• [22]Salanti G, Ades A, Ioannidis J: Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin Epidemiol 2011, 64:163-71.
• [23]Salanti G, Higgins J, Ades A, Ioannidis J: Evaluation of networks of randomized trials. Stat Methods Med Res 2008, 17:279-301.
• [24]Li T, Puhan M, Vedula S, Singh S, Dickersin K: Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 2011, 9:79. BioMed Central Full Text
• [25]Mills E, Ioannidis J, Thorlund K, Schunemann H, Puhan M, Guyatt G: How to use an article reporting a multiple treatment comparison meta-analysis. JAMA 2012, 308:1246-53.
• [26]Lee A: Review of mixed treatment comparisons in published systematic reviews shows marked increase since 2009. J Clin Epidemiol 2014, 67:138-43.
• [27]Trinquart L, Abbe A, Ravaud P: Impact of reporting bias in network meta-analysis of antidepressant placebo-controlled trials. PLoS One 2012, 7:35219.
• [28]Bafeta A, Trinquart L, Seror R, Ravaud P: Analysis of the systematic reviews process in reports of network meta-analyses: methodological systematic review. BMJ 2013, 347:3675.
• [29]Ntzani E, Rizos E, Ioannidis J: Genetic effects versus bias for candidate polymorphisms in myocardial infarction: case study and overview of large-scale evidence. Am J Epidemiol 2007, 165:973-84.
• [30]Pan Z, Trikalinos T, Kavvoura F, Lau J, Ioannidis J: Local literature bias in genetic epidemiology: an empirical evaluation of the chinese literature. PLoS Med 2005, 2:334.
• [31]Copas J, Shi J: Meta-analysis, funnel plots and sensitivity analysis. Biostatistics 2000, 1:247-62.
• [32]Copas J, Shi J: A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res 2001, 10:251-65.
• [33]Davey J, Turner R, Clarke M, Higgins J: Characteristics of meta-analyses and their component studies in the cochrane database of systematic reviews: a cross-sectional, descriptive analysis. BMC Med Res Methodol 2011, 11:160. BioMed Central Full Text
• [34]Turner R, Davey J, Clarke M, Thompson S, Higgins J: Predicting the extent of heterogeneity in meta-analysis, using empirical data from the cochrane database of systematic reviews. Int J Epidemiol 2012, 41(3):818-827.
• [35]Lloyd C: Estimating test power adjusted for size. J Stat Comput Simulat 2005, 75:921-33.
• [36]Ioannidis J: Clarifications on the application and interpretation of the test for excess significance and its extensions. J Math Psychol 2013, 57:184-87.
• [37]Turner E, Matthews A, Linardatos E, Tell R, Rosenthal R: Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008, 358:252-60.
• [38]Turner E, Knoepflmacher D, Shapley L: Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the us food and drug administration database. PLoS Med 2012, 9:1001189.
• [39]Ioannidis J: Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philos Ethics Humanit Med 2008, 3:14. BioMed Central Full Text
• [40]Fanelli D: Negative results are disappearing from most disciplines and countries. Scientometrics 2012, 90:891-904.
• [41]Ioannidis J: Why most published research findings are false. PLoS Med 2005, 2:124.
• [42]Tsilidis K, Panagiotou O, Sena E, Aretouli E, Evangelou E, Howells D, Salman R, MacLeod M, Ioannidis J: Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 2013, 11(7):1001609.
• [43]Fanelli D: How many scientists fabricate and falsify research? a systematic review and meta-analysis of survey data. PLoS One 2009, 4:5738.
• [44]Trinquart L, Chatellier G, Ravaud P: Adjustment for reporting bias in network meta-analysis of antidepressant trials. BMC Med Res Methodol 2012, 12:150. BioMed Central Full Text
• [45]Ioannidis J, Trikalinos T: Authors’ response to v johnson and y yuan. Clinical Trials 2007, 4:256-7.
• [46]Jackson D: Assessing the implications of publication bias for two popular estimates of between-study variance in meta-analysis. Biometrics 2007, 63:187-93.
• [47]Ioannidis J: Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract 2008, 14:951-7.
• [48]Rücker G: Network meta-analysis, electrical networks and graph theory. Res Synthesis Methods 2012, 3:312-24.
• [49]Mavridis D, Sutton A, Cipriani A, Salanti G: A fully bayesian application of the copas selection model for publication bias extended to network meta-analysis. Stat Med 2013, 32:51-66.
• [50]Chaimani A, Salanti G: Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions. Res Synthesis Methods 2012, 3:161-76.
• [51]Salanti G, Dias S, Welton N, Ades A, Golfinopoulos V, Kyrgiou M, Mauri D, Ioannidis J: Evaluating novel agent effects in multiple-treatments meta-regression. Stat Med 2010, 29:2369-83.