A short questionnaire to assess pediatric resident’s competencies: the validation process

Italian Journal of Pediatrics

Giorgio Perilongo²  Egidio Robusto¹  Eugenio Baraldi²  Silvia Carraro²  Silvia Bressan²  Pasquale Anselmi¹  Liviana Da Dalt² 

[1] Department FISPPA, University of Padua, Via Venezia 8, 35131 Padua, Italy;Paediatric Residency Program, Department of Woman’s and Child’s Health, University of Padua, Via Giustiniani, 3 – 35128 Padua, Italy

关键词: Medical residency;    Evaluation;    Pediatrics;    Resident;   

Others  :  825078
DOI  :  10.1186/1824-7288-39-41

 received in 2012-11-24, accepted in 2013-04-02,  发布年份 2013

【 摘 要 】

Background

In order to help assess resident performance during training, the Residency Affair Committee of the Pediatric Residency Program of the University of Padua (Italy) administered a Resident Assessment Questionnaire (ReAQ), which both residents and faculty were asked to complete. The aim of this article is to present the ReAQ and its validation.

Methods

The ReAQ consists of 20 items that assess the six core competencies identified by the Accreditation Council of Graduate Medical Education (ACGME). A many-facet Rasch measurement analysis was used for validating the ReAQ.

Results

Between July 2011 and June 2012, 211 evaluations were collected from residents and faculty. Two items were removed because their functioning changed with the gender of respondents. The step calibrations were ordered. The self evaluations (residents rating themselves) positively correlated with the hetero evaluations (faculty rating residents; Spearman’s ρ = 0.75, p < 0.001). Unfortunately, the observed agreement among faculty was smaller than expected (Exp = 47.1%; Obs = 41%), which indicates that no enough training to faculty for using the tool was provided.

Conclusions

In its final form, the ReAQ provides a valid unidimensional measure of core competences in pediatric residents. It produces reliable measures, distinguishes among groups of residents according to different levels of performance, and provides a resident evaluation that holds an analogous meaning for residents and faculty.

【 授权许可】

   
2013 Da Dalt et al.; licensee BioMed Central Ltd.

【 参考文献 】

• [1]Epstein RM: Assessment in medical education. N Engl J Med 2007, 356:387-396.
• [2]Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S: General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002, 21:103-111.
• [3]Schwartz A: Assessment in Graduate Medical Education: A Primer for Pediatric Program Directors. Chapel Hill: American Board of Pediatrics; 2011.
• [4]The pediatrics milestone project [https://www.abp.org/abpwebsite/publicat/milestones.pdf webcite]
• [5]Pulito A, Donnelly MB, Plymale M, Mentzer RM Jr: What do faculty observe of medical students’ clinical performance? Teach Learn Med 2006, 18:99-104.
• [6]Norman G: The long case versus observation structured clinical examination. BMJ 2002, 324:748-749.
• [7]Margolis MJ, Clauser BE, Cuddy MM, Ciccone A, Mee J, Harik P, Hawkins RE: Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study. Acad Med 2006, 81(Suppl 10):56-60.
• [8]Norcini JJ, Blank LL, Duffy FD, Fortna GS: The mini-CEX: a method to assessing clinical skills. Ann Intern Med 2003, 138:476-481.
• [9]Kogan JR, Holmboe ES, Hauer KE: Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009, 302:1316-1326.
• [10]Baker K: Determining resident clinical performance: getting beyond the noise. Anesthesiology 2011, 115:862-878.
• [11]Epstein RM, Hundered EM: Defining and assessing professional competence. JAMA 2002, 287:226-235.
• [12]Linacre JM: Many-Facet Rasch Measurement. Chicago: MESA Press; 1989.
• [13]Clauser BE, Ross LP, Nungester RJ, Clyman SG: An evaluation of the Rasch model for equating multiple forms of a performance assessment of physicians’ patient-management skills. Acad Med 1997, 72(Suppl 1):76-78.
• [14]Conrad KJ, Wright BD, McKnight P, McFall M, Fontana A, Rosenheck R: Comparing traditional and Rasch analyses of the Mississippi PTSD Scale: revealing limitations of reverse-scored items. J Appl Meas 2004, 5:15-30.
• [15]de Morton NA, Nolan JS: Unidimensionality of the Elderly Mobility Scale in older acute medical patients: different methods, different answers. J Clin Epidemiol 2011, 64:667-674.
• [16]Dreer LE, Berry J, Rivera P, Snow M, Elliott TR, Miller D, Little TD: Efficient assessment of social problem-solving abilities in medical and rehabilitation settings: a Rasch analysis of the Social Problem-Solving Inventory-Revised. J Clin Psychol 2009, 65:653-669.
• [17]Fisher WP, Vial RH, Sanders CV: Removing rater effects from medical clerkship evaluations. Acad Med 1997, 72:443-444.
• [18]Shen L, Yen J: Item dependency in medical licensing examinations. Acad Med 1997, 72:19-21.
• [19]Anselmi P, Vianello M, Robusto E: Positive associations primacy in the IAT: a many-facet Rasch measurement analysis. Exp Psychol 2011, 58:376-384.
• [20]Anselmi P, Vianello M, Robusto E: Preferring thin people does not imply derogating fat people. A Rasch analysis of the implicit weight attitude. Obesityin press
• [21]Iramaneerat C, Yudkowsky R, Myford C, Downing SM: Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement. Adv Health Sci Educ Theory Pract 2008, 13:479-493.
• [22]Lawson DM, Brailovsky C: The presence and impact of local item dependence on objective structured clinical examinations scores and the potential use of the polytomous, many-facet Rasch model. J Manipulative Physiol Ther 2006, 29:651-657.
• [23]Linacre JM: Facets Rasch Measurement Computer Program (Version 3.66.0) [Computer software]. Chicago: Winsteps.com; 2009.
• [24]Wright BD, Linacre JM: Reasonable mean-square fit values. Rasch Meas Trans 1994, 8:370.
• [25]Smith EV: Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. J Appl Meas 2007, 3:205-231.
• [26]Linacre JM: Winsteps (Version 3.68.0) [Computer software]. Chicago: Winsteps.com; 2009.
• [27]Linacre JM: Optimizing rating scale category effectiveness. J Appl Meas 2002, 3:85-106.
• [28]Fisher WP: Reliability statistics. Rasch Meas Trans 1992, 6:238.
• [29]Messick S: Validity. In Educational measurement. 3rd edition. Edited by Linn RL. New York: Macmillan; 1989:13-103.
• [30]Smith EV: Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. J Appl Meas 2001, 2:281-311.
• [31]Forrest M, Andersen B: Ordinal scale and statistics in medical research. BMJ 1986, 292:537-538.
• [32]Merbitz C, Morris J, Grip JC: Ordinal scales and foundations of misinference. Arch Phys Med Rehabil 1989, 70:308-312.
• [33]Kahler E, Rogausch A, Brunner E, Himmel W: A parametric analysis of ordinal quality-of-life data can lead to erroneous results. J Clin Epidemiol 2008, 61:475-480.
• [34]Fisher WP Jr: A research program for accountable and patient-centered health outcome measures. J Outcome Meas 1998, 2:222-239.
• [35]Grimby G, Tennant A, Tesio L: The use of raw scores from ordinal scales: time to end malpractice? J Rehabil Med 2012, 44:97-98.
• [36]Andrich D: Controversy and the Rasch model: a characteristic of incompatible paradigms? Med Care 2004, 42(Suppl 1):7-16.
• [37]Bond TG: Validity and assessment: a Rasch measurement perspective. Metodología de las Ciencias del Comportamiento 2004, 5:179-194.
• [38]Klass D: Assessing doctors at work – progress and challenges. N Engl J Med 2007, 356:414-415.
• [39]Ross S, Poth CA, Donoff MG, Papile C, Humphries P, Stasiuk S, Georgis R: Involving users in the refinement of the competency-based achievement system: an innovative approach to competency-based assessment. Med Teach 2012, 34:e143-147.