Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

BMC Medical Research Methodology

Mariska M Leeflang²  Lotty Hooft¹  Rob JPM Scholten¹  Eleanor Ochodo²  W Annefloor van Enst² 

[1] Current address: Dutch Cochrane Centre, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands;Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Center, Amsterdam, The Netherlands

关键词: Small study-effects;    Meta-analyses;    Funnel plot;    Diagnostic test accuracy;    Publication bias;   

Others  :  865404
DOI  :  10.1186/1471-2288-14-70

 received in 2014-02-10, accepted in 2014-05-06,  发布年份 2014

【 摘 要 】

Background

The validity of a meta-analysis can be understood better in light of the possible impact of publication bias. The majority of the methods to investigate publication bias in terms of small study-effects are developed for meta-analyses of intervention studies, leaving authors of diagnostic test accuracy (DTA) systematic reviews with limited guidance. The aim of this study was to evaluate if and how publication bias was assessed in meta-analyses of DTA, and to compare the results of various statistical methods used to assess publication bias.

Methods

A systematic search was initiated to identify DTA reviews with a meta-analysis published between September 2011 and January 2012. We extracted all information about publication bias from the reviews and the two-by-two tables. Existing statistical methods for the detection of publication bias were applied on data from the included studies.

Results

Out of 1,335 references, 114 reviews could be included. Publication bias was explicitly mentioned in 75 reviews (65.8%) and 47 of these had performed statistical methods to investigate publication bias in terms of small study-effects: 6 by drawing funnel plots, 16 by statistical testing and 25 by applying both methods. The applied tests were Egger’s test (n = 18), Deeks’ test (n = 12), Begg’s test (n = 5), both the Egger and Begg tests (n = 4), and other tests (n = 2). Our own comparison of the results of Begg’s, Egger’s and Deeks’ test for 92 meta-analyses indicated that up to 34% of the results did not correspond with one another.

Conclusions

The majority of DTA review authors mention or investigate publication bias. They mainly use suboptimal methods like the Begg and Egger tests that are not developed for DTA meta-analyses. Our comparison of the Begg, Egger and Deeks tests indicated that these tests do give different results and thus are not interchangeable. Deeks’ test is recommended for DTA meta-analyses and should be preferred.

【 授权许可】

   
2014 van Enst et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Dickersin K: The existence of publication bias and risk factors for its occurrence. JAMA 1990, 263:1385-1389.
• [2]Egger M, Juni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003, 7:1-76.
• [3]Ioannidis JP, Cappelleri JC, Sacks HS, Lau J: The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials 1997, 18:431-444.
• [4]Ioannidis JP: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998, 279:281-286.
• [5]Moher D, Fortin P, Jadad AR, Juni P, Klassen T, Le LJ, Liberati A, Linde K, Penna A: Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 1996, 347:363-366.
• [6]Sampson M, Platt R, StJohn PD, Moher D, Klassen TP, Pham B, Platt R, StJohn PD, Viola R, Raina P: Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol 2003, 56:943-955.
• [7]Sterne JA, Gavaghan D, Egger M: Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000, 53:1119-1129.
• [8]Thornton A, Lee P: Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol 2000, 53:207-216.
• [9]Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JP: Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011, 343:d4002.
• [10]Sterne JA, Egger M, Moher D: Adressing reporting bias; detecting repoting bias. In Cochrane Handbook for Systematic Reviews of Interventions. Edited by Higgins JPT, Green S. Oxford, United Kingdom: Wiley-Blackwell; 2009:310-324.
• [11]Begg CB, Mazumdar M: Operating characteristics of a rank correlation test for publication bias. Biometrics 1994, 50:1088-1101.
• [12]Egger M, Davey SG, Schneider M, Minder C: Bias in meta-analysis detected by a simple, graphical test. BMJ 1997, 315:629-634.
• [13]Web of Knowledge Edited by Thomson R. New York, USA: Thomson Reuters; 2014.
• [14]Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y: Analysing and Presenting Results. In Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Edited by Deeks JJ, Bossuyt PM, Gatsonis C. Oxford, United Kingdom: The Cochrane Collaboration; 2010:46-47.
• [15]Rifai N, Altman DG, Bossuyt PM: Reporting bias in diagnostic and prognostic studies: time for action. Clin Chem 2008, 54:1101-1103.
• [16]Korevaar DA, Ochodo EA, Bossuyt PM, Hooft L: Publication and Reporting of Test Accuracy Studies Registered in ClinicalTrials.gov. Clin Chem 2014, 60:651-659.
• [17]Brazzelli M, Lewis SC, Deeks JJ, Sandercock PA: No evidence of bias in the process of publication of diagnostic accuracy studies in stroke submitted as abstracts. J Clin Epidemiol 2009, 62:425-430.
• [18]Song F, Khan KS, Dinnes J, Sutton AJ: Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. Int J Epidemiol 2002, 31:88-95.
• [19]Deeks JJ, Macaskill P, Irwig L: The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005, 58:882-893.
• [20]Deville WL, Bezemer PD, Bouter LM: Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol 2000, 53:65-69.
• [21]Korevaar DA, van Enst WA, Spijker R, Bossuyt PM, Hooft L: Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARD. Evid Based Med 2014, 19:47-54.
• [22]R Core Team: R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2013.
• [23]Chang KC, Yew WW, Zhang Y: Pyrazinamide susceptibility testing in Mycobacterium tuberculosis: a systematic review with meta-analyses. Antimicrob Agents Chemother 2011, 55:4499-4505.
• [24]Chang MC, Chen JH, Liang JA, Lin CC, Yang KT, Cheng KY, Yeh JJ, Kao CH: Meta-analysis: comparison of F-18 fluorodeoxyglucose-positron emission tomography and bone scintigraphy in the detection of bone metastasis in patients with lung cancer. Acad Radiol 2012, 19:349-357.
• [25]Cheng X, Li Y, Xu Z, Bao L, Li D, Wang J: Comparison of 18 F-FDG PET/CT with bone scintigraphy for detection of bone metastasis: a meta-analysis. Acta Radiol 2011, 52:779-787.
• [26]Descatha A, Huard L, Aubert F, Barbato B, Gorand O, Chastang JF: Meta-analysis on the performance of sonography for the diagnosis of carpal tunnel syndrome. Semin Arthritis Rheum 2012, 41:914-922.
• [27]Dong MJ, Zhao K, Liu ZF, Wang GL, Yang SY, Zhou GJ: A meta-analysis of the value of fluorodeoxyglucose-PET/PET-CT in the evaluation of fever of unknown origin. Eur J Radiol 2011, 80:834-844.
• [28]Dym RJ, Burns J, Freeman K, Lipton ML: Is functional MR imaging assessment of hemispheric language dominance as good as the Wada test?: a meta-analysis. Radiology 2011, 261:446-455.
• [29]Gao P, Li M, Tian QB, Liu DW: Diagnostic performance of des-gamma-carboxy prothrombin (DCP) for hepatocellular carcinoma: a bivariate meta-analysis. Neoplasma 2012, 59:150-159.
• [30]Gargiulo P, Petretta M, Bruzzese D, Cuocolo A, Prastaro M, D'Amore C, Vassallo E, Savarese G, Marciano C, Paolillo S, Filardi PP: Myocardial perfusion scintigraphy and echocardiography for detecting coronary artery disease in hypertensive patients: a meta-analysis. Eur J Nucl Med Mol Imaging 2011, 38:2040-2049.
• [31]Glasgow SC, Bleier JI, Burgart LJ, Finne CO, Lowry AC: Meta-analysis of histopathological features of primary colorectal cancers that predict lymph node metastases. J Gastrointest Surg 2012, 16:1019-1028.
• [32]Gong X, Xu Q, Xu Z, Xiong P, Yan W, Chen Y: Real-time elastography for the differentiation of benign and malignant breast lesions: a meta-analysis. Breast Cancer Res Treat 2011, 130:11-18.
• [33]Hernaez R, Lazo M, Bonekamp S, Kamel I, Brancati FL, Guallar E, Clark JM: Diagnostic accuracy and reliability of ultrasonography for the detection of fatty liver: a meta-analysis. Hepatology 2011, 54:1082-1090.
• [34]Inaba Y, Chen JA, Bergmann SR: Carotid plaque, compared with carotid intima-media thickness, more accurately predicts coronary artery disease events: a meta-analysis. Atherosclerosis 2012, 220:128-133.
• [35]Kobayashi Y, Hayashino Y, Jackson JL, Takagaki N, Hinotsu S, Kawakami K: Diagnostic performance of chromoendoscopy and narrow band imaging for colonic neoplasms: a meta-analysis. Colorectal Dis 2012, 14:18-28.
• [36]Li BS, Wang XY, Ma FL, Jiang B, Song XX, Xu AG: Is high resolution melting analysis (HRMA) accurate for detection of human disease-associated mutations? A meta analysis. PLoS One 2011, 6:e28078.
• [37]Li R, Liu J, Xue H, Huang G: Diagnostic value of fecal tumor M2-pyruvate kinase for CRC screening: a systematic review and meta-analysis. Int J Cancer 2012, 131:1837-1845.
• [38]Lu Y, Chen YQ, Guo YL, Qin SM, Wu C, Wang K: Diagnosis of invasive fungal disease using serum (1– –>3)-beta-D-glucan: a bivariate meta-analysis. Intern Med 2011, 50:2783-2791.
• [39]Lundstrom LH, Vester-Andersen M, Moller AM, Charuluxananan S, L'hermite J, Wetterslev J: Poor prognostic value of the modified Mallampati score: a meta-analysis involving 177 088 patients. Br J Anaesth 2011, 107:659-667.
• [40]Luo YX, Chen DK, Song SX, Wang L, Wang JP: Aberrant methylation of genes in stool samples as diagnostic biomarkers for colorectal cancer or adenomas: a meta-analysis. Int J Clin Pract 2011, 65:1313-1320.
• [41]Manea L, Gilbody S, McMillan D: Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): a meta-analysis. CMAJ 2012, 184:E191-E196.
• [42]Mao R, Xiao YL, Gao X, Chen BL, He Y, Yang L, Hu PJ, Chen MH: Fecal calprotectin in predicting relapse of inflammatory bowel diseases: a meta-analysis of prospective studies. Inflamm Bowel Dis 2012, 18:1894-1899.
• [43]Marton A, Xue X, Szilagyi A: Meta-analysis: the diagnostic accuracy of lactose breath hydrogen or lactose tolerance tests for predicting the North European lactase polymorphism C/T-13910. Aliment Pharmacol Ther 2012, 35:429-440.
• [44]Mathews WC, Agmas W, Cachay E: Comparative accuracy of anal and cervical cytology in screening for moderate to severe dysplasia by magnification guided punch biopsy: a meta-analysis. PLoS One 2011, 6:e24946.
• [45]McInnes MD, Kielar AZ, Macdonald DB: Percutaneous image-guided biopsy of the spleen: systematic review and meta-analysis of the complication rate and diagnostic accuracy. Radiology 2011, 260:699-708.
• [46]Meader N, Mitchell AJ, Chew-Graham C, Goldberg D, Rizzo M, Bird V, Kessler D, Packham J, Haddad M, Pilling S: Case identification of depression in patients with chronic physical health problems: a diagnostic accuracy meta-analysis of 113 studies. Br J Gen Pract 2011, 61:e808-e820.
• [47]Mitchell AJ, Meader N, Pentzek M: Clinical recognition of dementia and cognitive impairment in primary care: a meta-analysis of physician accuracy. Acta Psychiatr Scand 2011, 124:165-183.
• [48]Onishi A, Sugiyama D, Kogata Y, Saegusa J, Sugimoto T, Kawano S, Morinobu A, Nishimura K, Kumagai S: Diagnostic accuracy of serum 1,3-beta-D-glucan for pneumocystis jiroveci pneumonia, invasive candidiasis, and invasive aspergillosis: systematic review and meta-analysis. J Clin Microbiol 2012, 50:7-15.
• [49]Papathanasiou ND, Boutsiadis A, Dickson J, Bomanji JB: Diagnostic accuracy of (1)(2)(3)I-FP-CIT (DaTSCAN) in dementia with Lewy bodies: a meta-analysis of published studies. Parkinsonism Relat Disord 2012, 18:225-229.
• [50]Plana MN, Carreira C, Muriel A, Chiva M, Abraira V, Emparanza JI, Bonfill X, Zamora J: Magnetic resonance imaging in the preoperative assessment of patients with primary breast cancer: systematic review of diagnostic accuracy and meta-analysis. Eur Radiol 2012, 22:26-38.
• [51]Qu X, Huang X, Wu L, Huang G, Ping X, Yan W: Comparison of virtual cystoscopy and ultrasonography for bladder cancer detection: a meta-analysis. Eur J Radiol 2011, 80:188-197.
• [52]Sadeghi R, Gholami H, Zakavi SR, Kakhki VR, Tabasi KT, Horenblas S: Accuracy of sentinel lymph node biopsy for inguinal lymph node staging of penile squamous cell carcinoma: systematic review and meta-analysis of the literature. J Urol 2012, 187:25-31.
• [53]Sadigh G, Carlos RC, Neal CH, Dwamena BA: Ultrasonographic differentiation of malignant from benign breast lesions: a meta-analytic comparison of elasticity and BIRADS scoring. Breast Cancer Res Treat 2012, 133:23-35.
• [54]Summah H, Tao LL, Zhu YG, Jiang HN, Qu JM: Pleural fluid soluble triggering receptor expressed on myeloid cells-1 as a marker of bacterial infection: a meta-analysis. BMC Infect Dis 2011, 11:280. BioMed Central Full Text
• [55]Sun W, Wang K, Gao W, Su X, Qian Q, Lu X, Song Y, Guo Y, Shi Y: Evaluation of PCR on bronchoalveolar lavage fluid for diagnosis of invasive aspergillosis: a bivariate metaanalysis and systematic review. PLoS One 2011, 6:e28467.
• [56]Takakuwa KM, Keith SW, Estepa AT, Shofer FS: A meta-analysis of 64-section coronary CT angiography findings for predicting 30-day major adverse cardiac events in patients presenting with symptoms suggestive of acute coronary syndrome. Acad Radiol 2011, 18:1522-1528.
• [57]Thosani N, Singh H, Kapadia A, Ochi N, Lee JH, Ajani J, Swisher SG, Hofstetter WL, Guha S, Bhutani MS: Diagnostic accuracy of EUS in differentiating mucosal versus submucosal invasion of superficial esophageal cancers: a systematic review and meta-analysis. Gastrointest Endosc 2012, 75:242-253.
• [58]Tomasson G, Grayson PC, Mahr AD, Lavalley M, Merkel PA: Value of ANCA measurements during remission to predict a relapse of ANCA-associated vasculitis–a meta-analysis. Rheumatology (Oxford) 2012, 51:100-109.
• [59]Trallero-Araguas E, Rodrigo-Pendas JA, Selva-O'Callaghan A, Martinez-Gomez X, Bosch X, Labrador-Horrillo M, Grau-Junyent JM, Vilardell-Tarres M: Usefulness of anti-p155 autoantibody for diagnosing cancer-associated dermatomyositis: a systematic review and meta-analysis. Arthritis Rheum 2012, 64:523-532.
• [60]Wang GL, Zhao K, Liu ZF, Dong MJ, Yang SY: A meta-analysis of fluorodeoxyglucose-positron emission tomography versus scintigraphy in the evaluation of suspected osteomyelitis. Nucl Med Commun 2011, 32:1134-1142.
• [61]Wang QB, Zhu H, Liu HL, Zhang B: Performance of magnetic resonance elastography and diffusion-weighted imaging for the staging of hepatic fibrosis: A meta-analysis. Hepatology 2012, 56:239-247.
• [62]Wang W, Li Y, Li H, Xing Y, Qu G, Dai J, Liang Y: Immunodiagnostic efficacy of detection of Schistosoma japonicum human infections in China: a meta analysis. Asian Pac J Trop Med 2012, 5:15-23.
• [63]Wang Y, Zhang C, Liu J, Huang G: Is 18 F-FDG PET accurate to predict neoadjuvant therapy response in breast cancer? A meta-analysis. Breast Cancer Res Treat 2012, 131:357-369.
• [64]Wu LM, Xu JR, Liu MJ, Zhang XF, Hua J, Zheng J, Hu JN: Value of magnetic resonance imaging for nodal staging in patients with head and neck squamous cell carcinoma: a meta-analysis. Acad Radiol 2012, 19:331-340.
• [65]Xu HB, Li L, Xu Q: Tc-99 m sestamibi scintimammography for the diagnosis of breast cancer: meta-analysis and meta-regression. Nucl Med Commun 2011, 32:980-988.
• [66]Xu W, Shi J, Zeng X, Li X, Xie WF, Guo J, Lin Y: EUS elastography for the differentiation of benign and malignant lymph nodes: a meta-analysis. Gastrointest Endosc 2011, 74:1001-1009.
• [67]Ying L, Hou Y, Zheng HM, Lin X, Xie ZL, Hu YP: Real-time elastography for the differentiation of benign and malignant superficial lymph nodes: a meta-analysis. Eur J Radiol 2012, 81:2576-2584.
• [68]Yu YH, Wei W, Liu JL: Diagnostic value of fine-needle aspiration biopsy for breast mass: a systematic review and meta-analysis. BMC Cancer 2012, 12:41. BioMed Central Full Text
• [69]Zhang L, Zong ZY, Liu YB, Ye H, Lv XJ: PCR versus serology for diagnosing Mycoplasma pneumoniae infection: a systematic review & meta-analysis. Indian J Med Res 2011, 134:270-280.
• [70]Harbord RM, Egger M, Sterne JA: A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med 2006, 25:3443-3457.
• [71]Hazem A, Elamin MB, Malaga G, Bancos I, Prevost Y, Zeballos-Palacios C, Velasquez ER, Erwin PJ, Natt N, Montori VM, Murad MH: The accuracy of diagnostic tests for GH deficiency in adults: a systematic review and meta-analysis. Eur J Endocrinol 2011, 165:841-849.
• [72]Singh B, Parsaik AK, Agarwal D, Surana A, Mascarenhas SS, Chandra S: Diagnostic accuracy of pulmonary embolism rule-out criteria: a systematic review and meta-analysis. Ann Emerg Med 2012, 59:517-520.
• [73]Parekh-Bhurke S, Kwok CS, Pang C, Hooper L, Loke YK, Ryder JJ, Sutton AJ, Hing CB, Harvey I, Song F: Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol 2011, 64:349-357.
• [74]Burkner PC, Doebler P: Testing for publication bias in diagnostic meta-analysis: a simulation study. Stat Med 2014.
• [75]de Vet HCW, Eisinga A, Riphagen II, Aertgeerts B, Pewsner D: Searching for Studies. In Cochrane Handbook for Systematic Reviews of Diagnosic Test Accuracy. 0.4 edition Edited by The Cochrane Collaboration. 2008.
• [76]Li CR, Liao CT, Liu JP: A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves. Stat Med 2008, 27:1762-1776.
• [77]Liu JP, Ma MC, Wu CY, Tai JY: Tests of equivalence and non-inferiority for diagnostic accuracy based on the paired areas under ROC curves. Stat Med 2006, 25:1219-1238.
• [78]DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med 2004, 141:477-478.