Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm

BMC Medical Research Methodology

Peter Tugwell²  George Wells³  Janet Smylie¹  Elizabeth Kristjansson⁴  Kevin Brand⁵  Vivian Welch² 

[1] Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada;Centre for Global Health, Institute of Population Health, University of Ottawa, 1 Stewart Street, Ottawa, ON, K1N6N5, Canada;Clinical Epidemiology Unit, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, ON, Canada;School of Psychology, Centre for Global Health, Institute of Population Health, University of Ottawa, 1 Stewart Street, Ottawa, ON, K1N6N5, Canada;Telfer School of Management, University of Ottawa, Ottawa, ON, Canada

关键词: Socioeconomic status;    Sex and gender;    Health equity;    Applicability;    Systematic reviews;   

Others  :  1126349
DOI  :  10.1186/1471-2288-12-187

 received in 2012-01-21, accepted in 2012-12-05,  发布年份 2012

【 摘 要 】

Background

Systematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).

Methods

A health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.

Results

The proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.

Conclusions

Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.

【 授权许可】

   
2012 Welch et al.; licensee BioMed Central Ltd.

【 参考文献 】

• [1]Whitehead M: The concepts and principles of equity and health. Int J Health Serv 1992, 22:429-445.
• [2]Kelly MP, Morgan A, Exworthy M, Popay J, Tugwell P, Robinson V, Simpson S, Narayan T, Myer L, Houweling T, et al.: The social determinants of health: developing an evidence base for political action: final report to the world health organization commission on the social determinants of health. Geneva; 2007.
• [3]White M, Adams J, Heywood P: How and why do interventions that increase health overall widen inequalities with populations? In Social inequality and public health. 1st edition. Edited by Babones SJ. Bristol: The Policy Press, University of Bristol; 2009:65-82.
• [4]World Health Assembly: Resolution 5834 on the ministerial summit on health research. 2005, 5834.
• [5]Lavis JN, Davies HTO, Gruen RL: Working within and beyond the cochrane collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 2006, 1:21-33.
• [6]Mackenbach JP: Tackling inequalities in health: the need for building a systematic evidence base. J Epidemiol Community Health 2003, 57:162.
• [7]Thomas S, Fayter D, Misso K, Ogilvie D, Petticrew M, Sowden A, Whitehead M, Worthy G, Thomas S, Fayter D, et al.: Population tobacco control interventions and their effects on social inequalities in smoking: systematic review. [Review] [109 refs]. Tob Control 2008, 17:230-237.
• [8]Chopra M, Munro S, Lavis JN, Vist G, Bennett S: Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet 2008, 371:668-774.
• [9]Tsikata S, Robinson V, Petticrew M, Kristjansson B, Moher D, McGowan J, Shea B, Wells G, Tugwell P: Do cochrane systematic reviews contain useful information about health equity? [abstract]. XI cochrane colloquium: evidence, health care and culture; 2003 Oct 26 31. Barcelona, Spain; 2003:77.
• [10]Petticrew M, Whitehead M, Macintye SJ, Graham H, Egan M: Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health 2004, 58:811-816.
• [11]Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy 2005, 10:35-48.
• [12]Tugwell P, Petticrew M, Robinson V, Kristjansson E, Maxwell L: Cochrane and campbell collaborations, and health equity. Lancet 2006, 367:1128-1130.
• [13]Petticrew M, Tugwell P, Welch V, Ueffing E, Kristjansson E, Armstrong R, Doyle J, Waters E: Better evidence about wicked issues in tackling health inequities. J Public Health 2009, 31:453-456.
• [14]Evans T, Brown H: Road traffic crashes: operationalizing equity in the context of health sector reform. Inj Control Saf Promot 2003, 10:11-12.
• [15]Kavanagh J, Oliver S, Lorenc T: Reflections in developing and using PROGRESS-plus. Equity update (2). Ottawa; 2008:1-3.
• [16]US Department of Health and Human Services: Understanding and improving health. Washington, DC; 2000.
• [17]Marmot M, Friel S, Bell R, Houweling TA, Taylor S: Commission on social determinants of health: closing the gap in a generation: health equity through action on the social determinants of health. Lancet 2008, 372:1661-1669.
• [18]Sun X, Briel M, Walter SD, Guyatt GH: Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. Br Med J 2010, 340:C117.
• [19]Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH: Credibility of claims of subgroup effects in randomised controlled trials: systematic review. Br Med J 2012, 344:e1553.
• [20]Petticrew M, Tugwell P, Kristjansson E, Oliver S, Ueffing E, Welch V: Damned if you do, damned if you don't: subgroup analysis and equity. J Epidemiol Community Health 2011. epub ahead of print
• [21]Feinstein AR: Clinimetrics. New Haven: Yale University Press; 1987.
• [22]Feinstein AR: Feinstein AR. Clinical epidemiology: the architecture of clinical research. Philadelphia: Saunders; 1985:39-52. Philadelphia: Saunders; 1985
• [23]Dans AL, Dans LF, Guyatt GH, Richardson S: Users' Guides to the medical literature: XIV. How to decide on the applicability of clinical trial results to your patient. Evidence-based medicine working group.[see comment]. JAMA 1998, 279:545-549.
• [24]Greenaway C, Sandoe A, Vissandjee B, Kitai I, Gruner D, Ueffing E, Wobeser W, Menzies D, Schwartzman K: Tuberculosis: evidence review for newly arriving immigrants. Can Med Assoc J 2010, 183(12):E939-E951.
• [25]Welch V, Tugwell P, Wells GA, Kristjansson E, Petticrew M, McGowan J, DeMontigny J, Benkhalti M, Ueffing E: How effects on health equity are assessed in systematic reviews of interventions. Cochrane Database Syst Rev 2010, (12):MR000028.
• [26]Driedger SM, Gallois C, Sanders CB, Santesso N, Driedger SM, Gallois C, Sanders CB, Santesso N: Finding common ground in team-based qualitative research using the convergent interviewing method. Qual Health Res 2006, 16:1145-1157.
• [27]Welch V, Smylie JK, Kristjansson E, Brand K, Tugwell P, Wells GA: What is the role of systematic reviews in tackling health inequity?. University of Ottawa; 2010. PhD Thesis
• [28]Dans AL, Dans LF, Guyatt G: Applying results to individual patients. In Users' Guides to the medical literature: a manual for evidence-based clinical practice. 2nd edition. Edited by Guyatt G, Rennie D, Meade MO, Cook DJ. New York, USA: The McGraw-Hill Companies, Inc; 2008:273-289.
• [29]Fleiss JL: Measuring nominal scale agreement among many raters. Psychol Bull 1971, 76:378-382.
• [30]Cicchetti DV, Feinstein AR: High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 1990, 43:551-558.
• [31]Evans DB, Lim SS, Adam T, Edejer TT: WHO choosing interventions that are cost effective (CHOICE) millennium development goals team: evaluation of current strategies and future priorities for improving health in developing countries. BMJ 2005, 331:1457-1461.
• [32]Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P: Priorities in health. New York: Oxford University Press; 2006.
• [33]Mathers CD, Loncar D: Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med 2006, 3:3442.
• [34]Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L: OMERACT: an international initiative to improve outcome measurement in rheumatology. Trials 2007, 8:38. BioMed Central Full Text
• [35]Vidanapathirana J, Abramson MJ, Forbes A, Fairley C: Mass media interventions for promoting HIV testing. Cochrane Database Syst Rev 2005, (3):CD004775.
• [36]Main C, Thomas S, Ogilvie D, Stirk L, Petticrew M, Whitehead M, Sowden A: Population tobacco control interventions and their effects on social inequalities in smoking: placing an equity lens on existing systematic reviews. BMC Publ Health 2008, 8:178. BioMed Central Full Text
• [37]Bisson J, Andrew M: Psychological treatment of post-traumatic stress disorder (PTSD). Cochrane Database Syst Rev 2007, (3):CD003388.
• [38]Wright JM, Musini VM: First-line drugs for hypertension. Cochrane Database Syst Rev 2009. Art. No. CD001841
• [39]Riaz Y, Mehta JS, Wormald R, Evans JR, Foster A, Ravilla T, Snelligen T: Surgical interventions for age-related cataract. Cochrane Database Syst Rev 2006., (4) Art. No. CD001323
• [40]Demicheli V, Jefferson T, Rivetti A, Price D: Vaccines for measles, mumps and rubella in children. Cochrane Database Syst Rev (4) Art. No. CD004407
• [41]Arroll B, Elley CR, Fishman T, Goodyear-Smith FA, Kenealy T, Blashki G, Kerse N, McGillivray S: Antidepressants versus placebo for depression in primary care. Cochrane Database Syst Rev 2009., (3) Art. No CD007954
• [42]Sinclair D, Zani B, Donegan S, Olliaro P, Garner P: Artemisinin-based combination therapy for treating uncomplicated malaria. Cochrane Database Syst Rev 2009.
• [43]Shults RA, Nichols JL, Dinh-Zarr TB, Sleet DA, Elder RW: Effectiveness of primary enforcement safety belt laws and enhanced enforcement of safety belt laws: a summary of the guide to community preventive services systematic reviews. J Safety Res 2004, 35:189-196.
• [44]Ejemot RI, Ehiri JE, Meremikwu MM, Critchley JA: Hand washing for preventing diarrhoea. Cochrane Database Syst Rev 2008., (1) Art No. CD004265
• [45]Kraemer HC, Periyakoil VS, Noda A: Tutorial in biostatistics: kappa coefficients in medical research. Stat Med 2002, 21:2109-2129.
• [46]Vach W: The dependence of Cohen's kappa on the prevalence does not matter. J Clin Epidemiol 2005, 58:655-661.
• [47]Napoles-Springer A, Santoyo-Olsson J, O'Brien H, Stewart AL: Using cognitive interviews to develop surveys in diverse populations. Medical Care 2006, 44:s21-s30.
• [48]Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E, Bonnefoy J, Morgan A, Doohan E, Kelly MP: Assessing equity in systematic reviews: realising the recommendations of the commission on social determinants of health. BMJ 2010, 341:c4739.
• [49]Glasziou P, Guyatt GH, Dans AL, Dans LF, Straus S, Sackett DL: Applying the results of trials and systematic reviews to individual patients.[see comment]. ACP J Club 1998, 129:A15-A16.
• [50]National Health and Medical Research Council: How to use the evidence: assessment and application of scientific evidence. Canberra, Australia; 2000.
• [51]Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De AL, Truman BI, Hopkins DP, Mullen PD, Thompson RS, Woolf SH, et al.: Developing an evidence-based guide to community preventive services-methods. The task force on community preventive services. Am J Prev Med 2000, 18(Suppl):43.
• [52]Rothwell PM: Treating individuals 2. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet 2005, 365:176-186.
• [53]Wang S, Moss JR, Hiller JE: Applicability and transferability of interventions in evidence-based public health. Health Promot Int 2006, 21:76-83.
• [54]Green LW, Glasgow RE: Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. [Review] [91 refs]. Eval Health Prof 2006, 29:126-153.
• [55]Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001, 134:663-694.
• [56]Higgins JPT, Green S (Eds): Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011. Available from http://www.cochrane-handbook.org webcite
• [57]Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking.[see comment]. Lancet 2004, 364:1615-1621.
• [58]Dans AM, Dans L, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D: Assessing equity in clinical practice guidelines. [Review] [53 refs]. J Clin Epidemiol 2007, 60:540-546.
• [59]Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009, 339:b2700.
• [60]Tackling health inequalities in Europe: an integrated approach. Rotterdam; 2007. [EUROTHINE: final report]
• [61]Buendia-Rodriguez JA, Sanchez-Villamil JP, Buendia-Rodriguez JA, Sanchez-Villamil JP: Using systematic reviews for evidence-based health promotion: basic methodology issues. Revista de Salud Publica 2006, 8(Suppl 2):94-105.