Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis

BMC Medical Research Methodology

Lisa Hartling¹  Humam Saltaji²  Maria Ospina³  Jorge Fuentes⁵  Susan Armijo-Olivo⁴ 

[1] Alberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada;School of Dentistry, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada;School of Public Health, University of Alberta, Institute of Health Economics, Edmonton, Alberta, Canada;Faculty of Rehabilitation Medicine, Department of Physical Therapy, University of Alberta, 3-48 Corbett Hall, Edmonton T6G 2G4, Canada;Department of Physical Therapy, Catholic University of Maule, Talca, Chile

关键词: Quality of reporting;    Risk of bias;    Critical appraisal;    Quality assessment;    Methodological quality;    Bias;   

Others  :  1091699
DOI  :  10.1186/1471-2288-13-116

 received in 2012-12-26, accepted in 2013-09-12,  发布年份 2013

【 摘 要 】

Background

Assessing the risk of bias of randomized controlled trials (RCTs) is crucial to understand how biases affect treatment effect estimates. A number of tools have been developed to evaluate risk of bias of RCTs; however, it is unknown how these tools compare to each other in the items included. The main objective of this study was to describe which individual items are included in RCT quality tools used in general health and physical therapy (PT) research, and how these items compare to those of the Cochrane Risk of Bias (RoB) tool.

Methods

We used comprehensive literature searches and a systematic approach to identify tools that evaluated the methodological quality or risk of bias of RCTs in general health and PT research. We extracted individual items from all quality tools. We calculated the frequency of quality items used across tools and compared them to those in the RoB tool. Comparisons were made between general health and PT quality tools using Chi-squared tests.

Results

In addition to the RoB tool, 26 quality tools were identified, with 19 being used in general health and seven in PT research. The total number of quality items included in general health research tools was 130, compared with 48 items across PT tools and seven items in the RoB tool. The most frequently included items in general health research tools (14/19, 74%) were inclusion and exclusion criteria, and appropriate statistical analysis. In contrast, the most frequent items included in PT tools (86%, 6/7) were: baseline comparability, blinding of investigator/assessor, and use of intention-to-treat analysis. Key items of the RoB tool (sequence generation and allocation concealment) were included in 71% (5/7) of PT tools, and 63% (12/19) and 37% (7/19) of general health research tools, respectively.

Conclusions

There is extensive item variation across tools that evaluate the risk of bias of RCTs in health research. Results call for an in-depth analysis of items that should be used to assess risk of bias of RCTs. Further empirical evidence on the use of individual items and the psychometric properties of risk of bias tools is needed.

【 授权许可】

   
2013 Armijo-Olivo et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

【 参考文献 】

• [1]Khan K, Ter Riet G, Popay J, Nixon J, Kleijnen J: Satge II. Conducting the Review. Phase 5 Study Quality Assessment. In Undertaking Systematic Reviews of research Effectiveness CRD’s Guidance for those carrying out or commissioning reviews. York: Centre for Reviews and Dissemination; 2001:1-20.
• [2]Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA: The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol 2001, 54(7):651-654.
• [3]Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001, 323(7303):42-46.
• [4]Higgins JPT, Altman DG, Goetzsche PC, Juni P, Moher D, Oxman AD, Savovicć J, Schulz KF, Weeks L, Sterne JAC: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011, 343(7829):d5928. 10.1136/bmj.d5928
• [5]Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 2008, 88(2):156-175.
• [6]Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995, 16(1):62-73.
• [7]Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S: Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil 2002, 83(12):1745-1752.
• [8]Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 2006, 59(12):1249-1256.
• [9]Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G: Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias tool and the Effectve Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 2012, 18(1):12-18.
• [10]Higgins J, Altman D: Chapter 8: Assessing risk of bias in included studies. In Cochrane Handbook for Systematic Reviews of Interventions version 50. Edited by Higgins J, Green S. Chichester, UK: John Wiley & Sons, Ltd; 2008.
• [11]Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007, 36(4):847-857.
• [12]Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ 2008, 336(7644):601-605.
• [13]Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP: Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009, 339(7728):1017.
• [14]Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998, 352(9128):609-613.
• [15]Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995, 273(5):408-412.
• [16]Assessing risk of bias in included studies. http://bmg.cochrane.org/assessing-risk-bias-included-studies webcite
• [17]Delgado-Rodriguez M, Llorca J: Bias. J Epidemiol Commun Health 2004, 58(8):635-641.
• [18]Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA: A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 2004, 4(1):22. BioMed Central Full Text
• [19]Dechartres A, Charles P, Hopewell S, Ravaud P, Altman DG: Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. J Clin Epidemiol 2011, 64(2):136-144.
• [20]Kunz R, Autti-Ramo I, Anttila H, Malmivaara A, Makela M: A systematic review finds that methodological quality is better than its reputation but can be improved in physiotherapy trials in childhood cerebral palsy. J Clin Epidemiol 2006, 59(12):1239-1248.
• [21]Streiner D, Norman G: Validity. In Health Measurements Scales. Edited by Streiner D, Norman G. Oxford: Oxford University Press; 2004:172-193.
• [22]Streiner D, Norman G: Reliability. In Health Measurements Scales. Edited by Streiner D, Norman G. Oxford: Oxford University Press; 2004:126-152.
• [23]Streiner D, Norman G: Measuring Change. In Health Measurements Scales. Edited by Streiner D, Norman G. Oxford: Oxford University Press; 2004:194-212.
• [24]Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, Bouter LM, de Vet HCW: Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007, 60(1):34-42.
• [25]Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J: Sources of variation and bias in studies of diagnostic accuracy: a systematic review. An Intern Med 2004, 140(3):189-202.
• [26]Sackett DL: Bias in analytic research. J Chronic Dis 1979, 32(1–2):51-68.
• [27]Viswanathan M, Berkman ND: Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol 2011, 65(2):163-178.
• [28]Hayden JA, Cotte P, Bombardier C: Evaluation of the quality of prognosis studies in systematic reviews. An Intern Med 2006, 144(6):427-437.
• [29]Lash TL, Fox MP, Fink AK: Multiple Bias Modeling: Applying Quantitatvie Bias Analysis to. Springer New York: Observational Epidemiologic Research; 2009.
• [30]Byrt T: How good is that agreement? Epidemiology (Cambridge, Mass) 1996, 7(5):561.
• [31]Jurgens T, Whelan AM, MacDonald M, Lord L: Development and evaluation of an instrument for the critical appraisal of randomized controlled trials of natural products. BMC Complem Altern Med 2009, 9:11. doi:10.1186/1472-6882-9-11 BioMed Central Full Text
• [32]Gerber AJ, Kocsis JH, Milrod BL, Roose SP, Barber JP, Thase ME, Perkins P, Leon AC: A quality-based review of randomized controlled trials of psychodynamic psychotherapy. Am J Psychiatr 2011, 168(1):19-28.
• [33]Kocsis JH, Gerber AJ, Milrod B, Roose SP, Barber J, Thase ME, Perkins P, Leon AC: A new scale for assessing the quality of randomized clinical trials of psychotherapy. Compr Psychiatr 2010, 51(3):319-324.
• [34]Cipriani A, Malvini L, Furukawa TA, Barbui C: Relationship between quality of reports of antidepressant randomized controlled trials and treatment estimates: Systematic review, meta-analysis, and meta-regression analysis. J Clin Psychopharmacol 2007, 27(4):352-356.
• [35]Moncrieff J, Churchill R, Colin Drummond D, McGuire H: Development of a quality assessment instrument for trials of treatments for depression and neurosis. Int J Meth Psychiatr Res 2001, 10(3):126-133.
• [36]Boutron I, Ravaud P, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, Nizard R: A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin Epidemiol 2005, 58(12):1233-1240.
• [37]Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996, 17(1):1-12.
• [38]De Vet HCW, De Bie RA, Van Der Heijden GJMG, Verhagen AP, Sijpkes P, Knipschild PG: Systematic reviews on the basis of methodological criteria. Physiotherapy 1997, 83(6):284-289.
• [39]Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998, 51(12):1235-1241.
• [40]Sherrington C, Herbert RD, Maher CG, Moseley AM: PEDro: A database of randomized trials and systematic reviews in physiotherapy. Man Ther 2000, 5(4):223-226.
• [41]Moseley AM, Herbert RD, Sherrington C, Maher CG: Evidence for physiotherapy practice: a survey of the Physiotherapy Evidence Database (PEDro). Aust J Physiother 2002, 48(1):43-49.
• [42]Van Tulder MW, Assendelft WJJ, Koes BW, Bouter LM, Bombardier C, Nachemson AL, Esmail R, Deyo RA, Shekelle PG, Bouter LM, et al.: Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for spinal disorders. Spine 1997, 22(20):2323-2330.
• [43]Van Tulder M, Furlan A, Bombardier C, Bouter L: Updated method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group. Spine 2003, 28(12):1290-1299.
• [44]Bizzini M, Childs JD, Piva SR: Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome. J Orthop Sports Phys Ther 2003, 33(1):4-20.
• [45]Chalmers TC, Smith H Jr, Blackburn B: A method for assessing the quality of a randomized control trial. Control Clin Trials 1981, 2(1):31-49.
• [46]Reisch JS, Tyson JE, Mize SG: Aid to the evaluation of therapeutic studies. Pediatrics 1989, 84(5):815-827.
• [47]Andrew E: Method for assessment of the reporting standard of clinical trials with roentgen contrast media. Acta Radiologica - Series Diagnosis 1984, 25(1):55-58.
• [48]Imperiale TF, McCullough AJ: Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. Ann Intern Med 1990, 113(4):299-307.
• [49]Detsky AS, Naylor CD, O’Rourke K, McGeer AJ, L’Abbe KA: Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol 1992, 45(3):255-265.
• [50]Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. JAMA 1994, 272(2):101-104.
• [51]Balas EA, Austin SM, Ewigman BG, Brown GD, Mitchell JA: Methods of randomized controlled clinical trials in health services research. Med Care 1995, 33(7):687-299.
• [52]Sindhu F, Carpenter L, Seers K: Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. J Adv Nurs 1997, 25(6):1262-1268.
• [53]Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Commun Health 1998, 52(6):377-384.
• [54]Nguyen QV, Bezemer PD, Habets L, Prahl-Andersen B: A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthodont 1999, 21(5):503-515.
• [55]Smith LA, Oldman AD, McQuay HJ, Moore RA: Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain. Pain 2000, 86(1–2):119-132.
• [56]Arrive L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, Monnier-Cholley L, Tubiana JM: A scale of methodological quality for clinical studies of radiologic examinations. Radiology 2000, 217(1):69-74.
• [57]Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002, 287(21):2801-2804.
• [58]Yates SL, Morley S, Eccleston C, Williams ACDC: A scale for rating the quality of psychological trials for pain. Pain 2005, 117(3):314-325.
• [59]Graham N, Haines T, Goldsmith CH, Gross A, Burnie S, Shahzad U, Talovikova E: Reliability of three assessment tools used to evaluate randomized controlled trials for treatment of neck pain. Spine 2012, 37(6):515-522.
• [60]Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, Rowe B: Applying the Risk of Bias tool in a systematic review of combination longacting betaagonists and inhaled corticosteroids for persistent asthma. PLoS Med 2011, 6(2):e17242.
• [61]Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG: Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 2010, 1-7.
• [62]Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM: Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2012. S0895-4356(12)00217-X. doi: 10.1016/j.jclinepi.2012.07.005
• [63]Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999, 282(11):1054-1060.
• [64]Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B: Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004, 328(7430):22-24.
• [65]Sanderson S, Tatt ID, Higgins JPT: Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007, 36(3):666-676.