Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting

BMC Medical Education

Berit Eika²  Ole Gotzsche¹  Dorte Guldbrand Nielsen¹ 

[1] Department of Cardiology, Aarhus University Hospital, Aarhus, Denmark;Center for Medical Education, Aarhus University, Aarhus, Denmark

关键词: Checklist;    Global rating;    Ultrasound;    Assessment;    Echocardiography;    Transthoracic echocardiography;   

Others  :  1138968
DOI  :  10.1186/1472-6920-13-47

 received in 2012-09-21, accepted in 2013-02-27,  发布年份 2013

【 摘 要 】

Background

Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.

Methods

The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.

Results

Validity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.

Conclusions

We designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

【 授权许可】

   
2013 Nielsen et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

【 参考文献 】

• [1]Quinones MA, Douglas PS, Foster E: ACC/AHA Clinical Competence Statement on Echocardiography: a report of the American College of Cardiology/American Heart Association/American College of Physicians-American Society of Internal Medicine task force on clinical competence. J Am Coll Cardiol 2003, 41:687-708.
• [2]Ryan T, Armstron WF, Khandheria BK: Task Force 4: Training in Echocardiography. J Am Coll Cardiol 2008, 51(3):361-367.
• [3]Popescu BA, Andrade MJ, Badano L, Fox KF, Flachskampf F, Lancellotti P, Varga A, Sicari R, Evangelista A, Nihoyannopoulos P, Zamorano J: European Association of Echocardiography recommendations for training, competence, and quality improvement in echocardiography. Eur J Echocardiogr 2009, 10:893-905.
• [4]Sanfilippo A, Bewick D, Chan KL, Cujec B, Dumesnil JG, Honos G, Munt B, Sasson Z, Tam J, Tomlinson C: Guidelines for the provision of echocardiography in Canada: Recommendations of a joint Canadian Cardiovascular Society/Canadian Society of Echocardiography Consensus Panel. Can J Cardiol 2005, 21(9):763-780.
• [5]Decoodt P, Gillebert TC: European Certification of Clinical Competence in Adult Echocardiography issued in Belgium. Acta Cardiol 1995, L(4):265-271.
• [6]The European Society of Cardiology Core Curriculum for the general cardiologist: The European Society of Cardiology Core Curriculum for the general cardiologist. 2008. http://www.escardio.org/education/coresyllabus/Pages/core-curriculum.aspx webcite
• [7]Yu E: The assessment of technical skills in a cardiology training program: is the ITER sufficient? Can J Cardiol 2000, 16(4):457-462.
• [8]Nair P, Siu SC, Sloggett CE, Biclar L, Sidhu RS, Yu EHC: Assessment of Technical and Interpretative Proficiency in Echocardiography. J Am Soc Echocardiogr 2006, 19(7):924-931.
• [9]Epstein RM: Assessment in Medical Education. N Engl J Med 2007, 356(4):387-396.
• [10]Fox KF, Popescu BA, Janiszewski S, Nihoyannopoulos P, Fraser AG, Pinto FJ: Report on the European Association of Echocardiography Accreditations in Echocardiography: December 2003 - September 2006. Eur J Echocardiogr 2007, 8:74-79.
• [11]van der Vleuten CPM: The Assessment of Professional Competence: Developments, Research and Practical Implications. Adv Health Sci Educ 1996, 1:41-67.
• [12]Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997, 84:273-278.
• [13]Faulkner H, Regehr G, Martin J, Reznick R: Validation of an Objective Structured Assessment of Technical Skills for Surgical Residents. Acad Med 1996, 71(12):1363-1365.
• [14]Goff BA, Lentz GM, Lee D, Houmard B, Mandel L: Development of an objective structured assessment of technical skills for obstetric and gynaecology residents. Obstet Gynecol 2000, 96:146-150.
• [15]Kishore TA, Pedro RN, Monga M, Sweet RM: Assessment of Validity of an OSATS for Cystoscopic and Ureteroscopic Cognitive and Psychomotor Skills. J Endourol 2008, 22(12):2707-2711.
• [16]Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB: Systemativ review of the literature on assessment, feedback and physicians' clinical performance: BEME Guide No.7. Medical Teacher 2006, 28(2):117-128.
• [17]Cook D, Beckham T: Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. Am J Med 2006, 199(2):166.e7-166.e16.
• [18]AERA, APA, NCME: Standards for educational and psychological testing. Washington, DC: American Psychological Association; 1999.
• [19]Streiner DL, Norman G: Health measurement scales. 4th edition. Oxford: Oxford University Press; 2008.
• [20]Beard J, Choksy S, Khan S: Assessment of operative competence during carotid endarterectomy. Br J Surg 2007, 94(6):726-30.
• [21]DCS: Anbefalinger for standardiseret minimumskrav for transthorakal ekkokardiografi hos voksne. In (Recommendations for standardized minimum demands for adult transthoracic echocardiography). Copenhagen: Danish Cardiology Society; 2008.
• [22]Evangelista A, Flachskampf F, Lancellotti P, Badano L, Aguilar R, Monaghan M, Zamorano J, Nihoyannopoulos P: European Association of Echocardiography recommendations for standardization of performance, digital storage and reporting of echocardiographic studies. Eur J Echocardiogr 2008, 9:438-448.
• [23]Setna Z, Jha V, Boursicot KAM, Roberts TE: Evaluating the utility of workplace-based assessment tools for specialty training. Best Pract Res Clin Obstet Gynaecol 2010, 24(6):767-82.
• [24]Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sørensen JL, Dalsgaard T, Ottesen B: Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc 2006, 20(9):1460-6.
• [25]Siddiqui NY, Stepp KJ, Lasch SJ, Mangel JM, Wu JM: Objective structure assessment of technical skills for repair of fourth-degree perineal lacerations. Am J Obstet Gynecol 2008, 199(6):676.e1-676.e6.
• [26]Datta V, Bann S, Mandalia M, Darzi A: The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg 2006, 192:372-378.
• [27]Goff BA, Lentz GM, Lee D, Fenner D, Morris J, Mandel L: Development of a bench station objective structured assessment of technical skills. Obstet Gynecol 2001, 98:412-416.
• [28]Eisenberg MJ, Rice S, Schiller NB: Guidelines for Physician Training in Advanced Cardiac Procedures: The Importance of Case Mix. J Am Coll Cardiol 1994, 23(7):1723-1725.
• [29]Norcini JJ: Peer assessment of competence. Med Educ 2003, 37(6):539-543.
• [30]Fried GM, Feldman LS: Objective Assessment of Technical Performance. World J Surg 2008, 32:156-160.
• [31]Hodges B, Regehr G, McNaughton NRT, Hanson M: OSCE checklists do not capture increasing levels of expertise. Acad Med 1999, 74:1129-1134.
• [32]Regehr G, Macrae H, Reznick R, Szalay D: Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 1998, 73(9):993-997.
• [33]Morgan PJ, Cleave-Hogg D, Guest CB: A Comparison of Global Ratings and Checklist Scores from an Undergraduate Assessment Using an Anesthesia Simulator. Acad Med 2001, 76(10):1053.
• [34]Maatsch JL, Huang RR, Downing SM, Munger BS: Examiner assessment of clinical performance: what do they tell us about clinical competence? Eval Program Plann 1987, 10:13-17.
• [35]Wilkinson JR, Crossly JGM, Wragg A, Mills P, Cowan G, Wade W: Implementing workplace-based assessment cross the medical specialties in the United Kingdom. Med Educ 2008, 42:364-373.