Efficient clinical evaluation of guideline quality: development and testing of a new tool

BMC Medical Research Methodology

Saravana Kumar²  Zuzana Machotka²  Julie Luker³  Lucylynn Lizarondo²  Olivia Thorpe²  Kate Beaton²  Ellena King²  Steve Milanese²  Janine Margarita Dizon¹  Karen Grimmer² 

[1] College of Rehabilitation Sciences, University of Santo Tomas, St. Martin de Porres Building University of Santo Tomas Espana, Manila 1083, Philippines;International Centre for Allied Health Evidence (iCAHE), University of South Australia, City East Campus, School of Health Sciences, Centenary, GPO box 2471, Adelaide 5001, Australia;Florey Institute of Neurosciences & Mental Health, University of Melbourne, NHMRC Research Fellow, Melbourne, Victoria, Australia

关键词: iCAHE guideline quality checklist;    AGREE II instrument;    Psychometric testing;    Guideline quality assessment;   

Others  :  866344
DOI  :  10.1186/1471-2288-14-63

 received in 2013-11-07, accepted in 2014-04-22,  发布年份 2014

【 摘 要 】

Background

Evaluating the methodological quality of clinical practice guidelines is essential before deciding which ones which could best inform policy or practice. One current method of evaluating clinical guideline quality is the research-focused AGREE II instrument. This uses 23 questions scored 1–7, arranged in six domains, which requires at least two independent testers, and uses a formulaic weighted domain scoring system. Following feedback from time-poor clinicians, policy-makers and managers that this instrument did not suit clinical need, we developed and tested a simpler, shorter, binary scored instrument (the iCAHE Guideline Quality Checklist) designed for single users.

Methods

Content and construct validity, inter-tester reliability and clinical utility were tested by comparing the new iCAHE Guideline Quality Checklist with the AGREE II instrument. Firstly the questions and domains in both instruments were compared. Six randomly-selected guidelines on a similar theme were then assessed by three independent testers with different experience in guideline quality assessment, using both instruments. Per guideline, weighted domain and total AGREE II scores were calculated, using the scoring rubric for three testers. Total iCAHE scores were calculated per guideline, per tester. The linear relationship between iCAHE and AGREE II scores was assessed using Pearson r correlation coefficients. Score differences between testers were assessed for the iCAHE Guideline Quality Checklist.

Results

There were congruent questions in each instrument in four domains (Scope & Purpose, Stakeholder involvement, Underlying evidence/Rigour, Clarity). The iCAHE and AGREE II scores were moderate to strongly correlated for the six guidelines. There was generally good agreement between testers for iCAHE scores, irrespective of their experience. The iCAHE instrument was preferred by all testers, and took significantly less time to administer than the AGREE II instrument. However, the use of only three testers and six guidelines compromised study power, rendering this research as pilot investigations of the psychometric properties of the iCAHE instrument.

Conclusion

The iCAHE Guideline Quality Checklist has promising psychometric properties and clinical utility.

【 授权许可】

   
2014 Grimmer et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Qaseem A, Forland F, Macbeth F, OllenschlÃĪger G, Phillips S, van der Wees P: Guidelines International Network: Toward International Standards for Clinical Practice Guidelines. Ann Intern Med 2012, 156:525-531.
• [2]Institute of Medicine: Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Pr; 2011.
• [3]World Health Organization: Global Programme on Evidence for Health Policy. Geneva. World Health Organization; [http://whqlibdoc.who.int/hq/2003/EIP_GPE_EQC_2003_1.pdf webcite]
• [4]The National Institute for Health and Clinical Excellence: The Guidelines Manual (January 2009). London; [http://www.nice.org.uk/guidelinesmanual webcite]
• [5]Network SIG: SIGN 50: A Guideline Developer’s Handbook. Edinburgh, Scotland: Scottish Intercollegiate Guidelines Network; 2008. [http://www.sign.ac.uk/guidelines/fulltext/50/index.html webcite]
• [6]National Health and Medical Research Council Clinical Practice Guidelines Portal. [http://www.clinicalguidelines.gov.au webcite]
• [7]Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA: Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999, 282:1458-1465.
• [8]Michie S, Johnston M, Abraham C, Lawton R, Parker D: Walker A on behalf of the “Psychological Theory” Group: Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care 2005, 14:26-33.
• [9]Grol R, Wensing M: What drives change? Barriers to and incentives for achieving evidence-based practice. Med J 2004, 15(180):57-60.
• [10]Grimmer K, Lekkas P, Nyland L, Young A, Kumar S: Perspectives on research evidence and clinical practice: a survey of Australian physiotherapists. Physiotherapy Research International 2007, 12(3):147-161.
• [11]Metcalfe C, Lewin R, Wisher S, Perry S, Bannigan K, Klaber Moffett J: Barriers to implementing the evidence base in four NHS therapies. Physiotherapy 2001, 87:433-440.
• [12]Parahoo K: Barriers to, and facilitators of, research utilisation among nurses in Northern Ireland. Journal of Advanced Nursing 2000, 31:89-98.
• [13]McColl A, Smith H, White P: Field J (1998): General practitioners’ perceptions of the route to evidence based medicine: a questionnaire survey. BMJ 1998, 316:361-365.
• [14]Collaboration AGREE: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12:18-23.
• [15]Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna S, Littlejohns P, Makarski J, Zitzelsberger L for the AGREE Next Steps Consortium: AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010, 182:E839-E842.
• [16]MacDermid JC, Brooks D, Solway S, Switzer-McIntyre S, Brosseau L, Graham ID: Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res 2005, 5:18. BioMed Central Full Text
• [17]Wimpenny P, van Zelm R: Appraising and comparing pressure ulcer guidelines. Worldview Evid-Based Nurs 2007, 4(1):40-50.
• [18]International Centre for Allied Health Evidence (iCAHE). [http://www.unisa.edu.au/cahe webcite]
• [19]Maher CG, Sherrington C, Herbert RD, Moseley AM, Elkins M: Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials. Phys Ther 2003, 83(8):713-721.
• [20]Microsoft: Microsoft excel. Redmond WA, USA: Microsoft; 2010.
• [21]SAS: SAS statistical analysis software ©Version 9.4. Cary NC, USA: SAS; 2013.
• [22]Faul F, Erdfelder E, Buchner A, Lang AG: Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behav Res Methods 2009, 41:1149-1160.
• [23]Thompson HJ, Mauk K (Eds): Clinical practice guidelines series- care of patient with mild traumatic brain injury. American Association of Neuroscience Nurses (AANN) & Association of Rehabilitation Nurses (ARN); 2011.
• [24]Barbosa RR, Jawa J, Watters J, Knight A, Kerwin E, Winston R, Robert D, Barraco R, Tucker B, Bardes J, Rowell S: Eastern Association for the Surgery of (2012).Evaluation and management of mild traumatic brain injury: an Eastern Association for the Surgery of Trauma practice management guideline’. The Journal of Trauma and Acute Care Surgery 2012, 73(4):S307-S314.
• [25]Brain Trauma Foundation: Guidelines for the Acute Medical Management of Severe Traumatic Brain Injury in Infants, Children, and Adolescents. Paediatr Crit Care Med 2012, 31(suppl 1):S1-S82.
• [26]Golisz K: Occupational therapy practice guidelines for adults with traumatic brain injury’. American Occupational Therapy Association (AOTA) 2009, 1:258.
• [27]National Institute of Health & Clinical Evidence (NICE) 2007, ‘Head injury: Triage, assessment, investigation and early management of head injury in infants, children and adults’, Clinical Guideline 56.
• [28]Scottish Intercollegiate Guidelines Network SIGN 2013 Brain injury rehabilitation in adults. Edinburgh: SIGN; 2013. (SIGN publication no. 130). [March 2013]. Available from: http://www.sign.ac.uk webcite