Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine

BMC Medical Education

Elmer Villanueva⁴  Julie K Tilson¹  Paul Glasziou³  Rusli Bin Nordin²  Dragan Ilic⁵ 

[1] Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, USA;Jeffrey Cheah School of Medicine and Health Sciences, Monash University, Johor Bahru, Malaysia;Faculty of Health Sciences and Medicine, Bond University, Robina, Australia;Gippsland Medical School, Monash University, Churchill, Australia;Department of Epidemiology & Preventive Medicine, School of Public Health & Preventive Medicine, Monash University, Level 6, The Alfred Centre, 99 Commercial Rd, Melbourne VIC 3004, Australia

关键词: Medical students;    Assessment;    Evidence based medicine;   

Others  :  866694
DOI  :  10.1186/1472-6920-14-114

 received in 2014-01-24, accepted in 2014-06-02,  发布年份 2014

【 摘 要 】

Background

While a variety of instruments have been developed to assess knowledge and skills in evidence based medicine (EBM), few assess all aspects of EBM - including knowledge, skills attitudes and behaviour - or have been psychometrically evaluated. The aim of this study was to develop and validate an instrument that evaluates medical trainees’ competency in EBM across knowledge, skills and attitude.

Methods

The ‘Assessing Competency in EBM’ (ACE) tool was developed by the authors, with content and face validity assessed by expert opinion. A cross-sectional sample of 342 medical trainees representing ‘novice’, ‘intermediate’ and ‘advanced’ EBM trainees were recruited to complete the ACE tool. Construct validity, item difficulty, internal reliability and item discrimination were analysed.

Results

We recruited 98 EBM-novice, 108 EBM-intermediate and 136 EBM-advanced participants. A statistically significant difference in the total ACE score was observed and corresponded to the level of training: on a 0-15-point test, the mean ACE scores were 8.6 for EBM-novice; 9.5 for EBM-intermediate; and 10.4 for EBM-advanced (p < 0.0001). Individual item discrimination was excellent (Item Discrimination Index ranging from 0.37 to 0.84), with internal reliability consistent across all but three items (Item Total Correlations were all positive ranging from 0.14 to 0.20).

Conclusion

The 15-item ACE tool is a reliable and valid instrument to assess medical trainees’ competency in EBM. The ACE tool provides a novel assessment that measures user performance across the four main steps of EBM. To provide a complete suite of instruments to assess EBM competency across various patient scenarios, future refinement of the ACE instrument should include further scenarios across harm, diagnosis and prognosis.

【 授权许可】

   
2014 Ilic et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Sackett D, Straus S, Richardson W, Rosenberg W, Haynes R: Evidence-Based Medicine. Edinburgh: Churchill Livingstone; 2000.
• [2]Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J: Sicily statement on evidence-based practice. BMC Med Educ 2005, 5:1. BioMed Central Full Text
• [3]Tilson J, Kaplan S, Harris F, Hutchinson A, Ilic D, Niederman R, Potomkova J, Zwolsman S: Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ 2011, 11:78. BioMed Central Full Text
• [4]Shaneyfelt T, Baum K, Bell D, Feldstein T, Kaatz S, Whelan C, Green M: Instruments for evaluating education in evidence-based practice: a systematic review. JAMA 2006, 296:1116-1127.
• [5]Ramos K, Schafer S, Tracz S: Validation of the Fresno test of competence in evidence based medicine. BMJ 2003, 326:319-321.
• [6]Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ 2002, 325:1338-1341.
• [7]McCluskey A, Bishop B: The adapted Fresno test of competence in evidence-based practice. J Contin Educ Health Prof 2009, 29:119-126.
• [8]Tilson J: Validation of the modified Fresno test: assessing physical therapists’ evidence based practice knowledge and skills. BMC Med Educ 2010, 10:38. BioMed Central Full Text
• [9]Spencer J, Jordan R: Learner centred approaches in medical education. BMJ 1999, 318:1280-1283.
• [10]Ilic D: Assessing competency in evidence based practice: strengths and limitations of current tools in practice. BMC Med Educ 2009, 9:53. BioMed Central Full Text
• [11]Dinkevich E, Marksinson A, Ahsan S, Lawrence B: Effect of a brief intervention on evidence-based medicine skills of pediatric residents. BMC Med Educ 2006, 6:1. BioMed Central Full Text
• [12]Meats E, Heneghan C, Crilly M, Glasziou P: Evidence-based medicine teaching in UK medical schools. Med Teach 2009, 31:332-337.
• [13]Ilic D: Rationale for using OSCEs to assess student competency in evidence-based medicine. Educ Health 2010, 23:434.
• [14]Fliegel J, Frohna J, Mangrulkar R: A computer-based OSCE station to measure competence in evidence-based medicine skills in medical students. Acad Med 2002, 77:1157-1158.
• [15]Jackson R, Ameratunga S, Broad J, Connor J, Lethaby A, Robb G, Wells S, Glasziou P, Heneghan C: The GATE frame: critical appraisal with pictures. Evid Base Med 2006, 11:335-338.
• [16]Guyatt G, Rennie D, Meade M, Cook D: Users’ guide to the medical literature. A manual for evidence-based medicine. New York: McGraw Hill Education; 2008.
• [17]Jones J, Hunter D: Consensus methods for medical and health services research. BMJ 1995, 311:376-380.
• [18]Ilic D, Bin Nordin R, Glasziou P, Tilson J, Villanueva E: Implementation of a blended learning approach to teaching evidence based practice: a protocol for a mixed methods study. BMC Med Educ 2013, 13:170. BioMed Central Full Text
• [19]Varma S: Preliminary item statistics using point-biserial correlation and p-values. Available at; http://www.eddata.com/resources/publications/EDS_Point_Biserial.pdf webcite Accessed; 28/11/2013
• [20]Nunnally J: Psychometric theory. Michigan: McGraw-Hill; 1978.
• [21]George D, Mallery P: SPSS for Windows Step by Step: A Simple Guide and Reference. 11.0 Update (4th ed.). Boston: Allyn & Bacon; 2003.
• [22]Kline P: The Handbook of Psychological Testing. 2nd edition. London: Routledge; 2000.
• [23]Aiken L: Relationships between the item difficulty and discrimination indexes. Educ Psychol Meas 1979, 39:821-824.
• [24]Ebel R: Essentials of Educational Measurement. Englewood Cliffs, NJ: Prentice Hall; 1979.