“Who writes what?” Using written comments in team-based assessment to better understand medical student performance: a mixed-methods study

BMC Medical Education

Nishan Sharma¹  Jonathan Samuel White¹ 

[1] Department of Surgery, Faculty of Medicine & Dentistry, University of Alberta, 10240 Kingsway Avenue, Edmonton, AB T5H 3V9, Canada

关键词: Team;    Clinical performance;    Qualitative;    Mixed-methods;    Clerkship;    Medical students;    Assessment;    Undergraduate;    Written comments;   

Others  :  1152992
DOI  :  10.1186/1472-6920-12-123

 received in 2012-06-09, accepted in 2012-12-04,  发布年份 2012

【 摘 要 】

Background

Observation of the performance of medical students in the clinical environment is a key part of assessment and learning. To date, few authors have examined written comments provided to students and considered what aspects of observed performance they represent. The aim of this study was to examine the quantity and quality of written comments provided to medical students by different assessors using a team-based model of assessment, and to determine the aspects of medical student performance on which different assessors provide comments.

Methods

Medical students on a 7-week General Surgery & Anesthesiology clerkship received written comments on ‘Areas of Excellence’ and ‘Areas for Improvement’ from physicians, residents, nurses, patients, peers and administrators. Mixed-methods were used to analyze the quality and quantity of comments provided and to generate a conceptual framework of observed student performance.

Results

1,068 assessors and 127 peers provided 2,988 written comments for 127 students, a median of 188 words per student divided into 26 “Areas of Excellence” and 5 “Areas for Improvement”. Physicians provided the most comments (918), followed by patients (692) and peers (586); administrators provided the fewest (91). The conceptual framework generated contained four major domains: ‘Student as Physician-in-Training’, ‘Student as Learner’, ‘Student as Team Member’, and ‘Student as Person.’

Conclusions

A wide range of observed medical student performance is recorded in written comments provided by members of the surgical healthcare team. Different groups of assessors provide comments on different aspects of student performance, suggesting that comments provided from a single viewpoint may potentially under-represent or overlook some areas of student performance. We hope that the framework presented here can serve as a basis to better understand what medical students do every day, and how they are perceived by those with whom they work.

【 授权许可】

   
2012 White and Sharma; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

【 参考文献 】

• [1]Pelling S, Kalen A, Hammar M, Wahlström O: Preparation for becoming members of health care teams: findings from a 5-year evaluation of a student interprofessional training ward. J Interprof Care 2011, 25:328-332.
• [2]Farmer EA, Beard JD, Dauphinee WD, LaDuca T, Mann KV: Assessing the performance of doctors in teams and systems. Med Educ 2002, 36:942-948.
• [3]Miller B, Moore D Jr, Stead W: Beyond Flexner: a new model for continuous learning in the health professions. Acad Med 2010, 85:266-272.
• [4]Harden RM, Dent JA: A Practical Guide for Medical Teachers. Edinburgh: Churchill Livingstone; 2009.
• [5]Norcini J, Burch V: Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007, 29:855-871.
• [6]Williams RG, Klamen DA, McGaghie WC: Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med 2003, 15:270-292.
• [7]Bing You RG, Greenberg LW, Wiederman BL, Smith CS: A randomized multicenter trial to improve resident teaching with written feedback. Teach Learn Med 1997, 9:10-3.
• [8]Plymale MA, Donnelly MB, Lawton J, Pulito AR, Mentzer RM: Faculty evaluation of surgery clerkship students: important components of written comments. Acad Med 2002, 77:S45-S47.
• [9]van Mook WNKA, Gorter SL, O'Sullivan H, Wass V, Schuwirth LW, van der Vleuten CPM: Approaches to professional behaviour assessment: Tools in the professionalism toolbox. Eur J Int Med 2009, 20:e153-e157.
• [10]Green M, Zick A, Thomas JX: Commentary: Accurate medical student performance evaluations and professionalism assessment: "Yes, we can!". Acad Med 2010, 85:1105-1107.
• [11]Rose M: Widening the lens on standardized patient assessment: what the encounter can reveal about the development of clinical competence. Acad Med 2001, 76:856-859.
• [12]Durning SJ, Hanson J, Gilliland W, McManigle JM, Waechter D, Pangaro LN: Using qualitative data from a program director's evaluation form as an outcome measurement for medical school. Mil Med 2010, 175:448-452.
• [13]Burford B, Illing J, Kergon C, Morrow G: User perceptions of multi-source feedback tools for junior doctors. Med Educ 2010, 44:165-176.
• [14]Hoffman K, Hosokawa M, Donaldson J: What criteria do faculty use when rating students as potential house officers? Med Teach 2009, 31:e412-e417.
• [15]Pulito AR, Donnelly MB, Plymale M, Mentzer RM: What do faculty observe of medical students’ clinical performance? Teach Learn Med 2006, 18:99-104.
• [16]Hasley PB, Arnold RM: Summative evaluation on the hospital wards. What do faculty say to learners? Adv in Health. Sci Educ 2008, 14:431-439.
• [17]Cohen GS, Blumberg P, Ryan NC, Sullivan PL: Do final grades reflect written qualitative evaluations of student performance? Teach Learn Med 1993, 5:10-15.
• [18]Mazor KM, Canavan C, Farrell M, Margolis MJ, Clauser BE: Collecting validity evidence for an assessment of professionalism: findings from think-aloud interviews. Acad Med 2008, 83:S9-12.
• [19]Sokol-Hessner L, Shea JA, Kogan JR: The Open-Ended Comment Space for Action Plans on Core Clerkship Students’ Encounter Cards: What Gets Written? Acad Med 2010, 85:S110-S114.
• [20]Littlefield JH, DaRosa DA, Paukert J, Williams RG, Klamen DL, Schoolfield JD: Improving resident performance assessment data: numeric precision and narrative specificity. Acad Med 2005, 80:489-495.
• [21]Canavan C, Holtman MC, Richmond M, Katsufrakis PJ: The Quality of Written Comments on Professional Behaviors in a Developmental Multisource Feedback Program. Acad Med 2010, 85:S106-S109.
• [22]White J, Sharma N: Using multi-source feedback to assess medical students learning on an interprofessional surgical healthcare team. Proceedings of the 14th Ottawa Conference on Assessment in the Healthcare Professions 2010, 368-369.
• [23]Sharma N, Cui Y, Leighton J, White J: Team-based assessment of medical students in a clinical clerkship is feasible and acceptable. Med Teach 2012, 34(7):555-561.
• [24]Schum TR, Krippendorf RL, Biernat KA: Simple feedback notes enhance specificity of feedback to learners. Ambul Pediatr 2003, 3:9-11.
• [25]White MD, Marsh EE: Content analysis: A flexible methodology. Library Trends 2006, 55:22-45.
• [26]Lockyer JM L, Clyman SG: Multisource feedback (360-degree evaluation). In Practical Guide to the Evaluation of Clinical Competence 1st edition. Edited by Holmboe ES, Hawkins RE M. 2008, 75-85.
• [27]Ginsburg S, Regehr G, Hatala R, McNaughton N, Frohna A, Hodges B, et al.: Context, conflict, and resolution: a new conceptual framework for evaluating professionalism. Acad Med 2000, 75:S6-S11.
• [28]Johnson D, Cujec B: Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med 1998, 26:1811-1816.
• [29]Wenrich MD, Carline JD, Giles LM, Ramsey PG: Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med 1993, 68:680-687.
• [30]Kaplan CB, Centor RM: The use of nurses to evaluate house officers' humanistic behavior. J Gen Intern Med 1990, 5:410-414.
• [31]Whitehouse A, Hassell A, Bullock A, Wood L, Wall D: 360 degree assessment (multisource feedback) of UK trainee doctors: Field testing of team assessment of behaviours (TAB). Med Teach 2007, 29:171-176.
• [32]Bullock AD, Hassell A, Markham WA, Wall DW, Whitehouse AB: How ratings vary by staff group in multi-source feedback assessment of junior doctors. Med Educ 2009, 43:516-520.
• [33]Ogunyemi D, Gonzalez G, Fong A, Alexander C, Finke D, Donnon T, et al.: From the eye of the nurses: 360-degree evaluation of residents. J Contin Educ Health Prof 2009, 29:105-110.
• [34]Nofziger AC, Naumburg EH, Davis BJ, Mooney CJ, Epstein RM: Impact of peer assessment on the professional development of medical students: a qualitative study. Acad Med 2010, 85:140-147.
• [35]Finn GM, Garner J: Twelve tips for implementing a successful peer assessment. Med Teach 2011, 33:443-446.
• [36]Cushing A, Abbott S, Lothian D, Hall A, Westwood OMR: Peer feedback as an aid to learning – What do we want? Feedback. When do we want it? Now! Med Teach 2011, 33:e105-e112.
• [37]Lyons O, Willcock H, Rees J: Let the patient teach: patient feedback will help prepare medical students for the changing healthcare world. Clin Teach 2009, 34:256.
• [38]Burford B, Bedi A, Morrow G, Kergon C: Collecting patient feedback in different clinical settings: problems and solutions. Clin Teach 2009, 6:259-264.
• [39]Braend AM, Gran SF, Frich JC, Lindbaek M: Medical students’ clinical performance in general practice Triangulating assessments from patients, teachers and students. Med Teach 2010, 32:333-339.
• [40]Gran SF, Braend AM, Lindbaek M: Triangulation of written assessments from patients, teachers and students: Useful for students and teachers? Med Teach 2010, 32:e552-e558.
• [41]Frohna A, Stern D: The nature of qualitative comments in evaluating professionalism. Med Educ 2005, 39:763-768.
• [42]Lye PS, Biernat KA, Bragg DS, Simpson DE: A pleasure to work with–an analysis of written comments on student evaluations. Ambul Pediatr 2001, 1:128-131.
• [43]Frank JRE: The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better care. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2005.
• [44]Ginsburg S, Gold W, Cavalcanti RB, Kurabi B, McDonald-Blumer H: Competencies plus: the nature of written comments on internal medicine residents' evaluation forms. Acad Med 2011, 86:S30-S34.
• [45]Hemmer PA, Hawkins R, Jackson JL, Pangaro LN: Assessing how well three evaluation methods detect deficiencies in medical students' professionalism in two settings of an internal medicine clerkship. Acad Med 2000, 75:167-173.