Validating a set of tools designed to assess the perceived quality of training of pediatric residency programs

Italian Journal of Pediatrics

Giorgio Perilongo²  Egidio Robusto¹  Eugenio Baraldi²  Silvia Carraro²  Sara Furlan¹  Pasquale Anselmi¹  Liviana Da Dalt² 

[1] Department FISPPA, University of Padua, Padua, Italy;Paediatric Residency Program, Department of Woman’s and Child’s Health, University of Padua, Via Giustiniani 3 – 35128, Padua, Italy

关键词: Teaching faculty;    Pediatric residency program;    Many-facet Rasch model;    Validation study;    Evaluation tools;   

Others  :  1135871
DOI  :  10.1186/s13052-014-0106-2

 received in 2014-07-26, accepted in 2014-12-18,  发布年份 2015

【 摘 要 】

Background

The Paediatric Residency Program (PRP) of Padua, Italy, developed a set of questionnaires to assess the quality of the training provided by each faculty member, the quality of the professional experience the residents experienced during the various rotations and the functioning of the Resident Affair Committee (RAC), named respectively: “Tutor Assessment Questionnaire” (TAQ), “Rotation Assessment Questionnaire” (RAQ), and RAC Assessment Questionnaire”. The process that brought to their validation are herein presented.

Method

Between July 2012 and July 2013, 51 residents evaluated 26 tutors through the TAQ, and 25 rotations through the RAQ. Forty-eight residents filled the RAC Assessment Questionnaire. The three questionnaires were validated through a many-facet Rasch measurement analysis.

Results

In their final form, the questionnaires produced measures that were valid, reliable, unidimensional, and free from gender biases. TAQ and RAQ distinguished tutors and rotations into 5–6 levels of different quality and effectiveness. The three questionnaires allowed the identification of strengths and weaknesses of tutors, rotations, and RAC. The agreement observed among judges was coherent to the predicted values, suggesting that no particular training is required for developing a shared interpretation of the items.

Conclusions

The work herein presented serves to enrich the armamentarium of tools that resident medical programs can use to monitor their functioning. A larger application of these tools will serve to consolidate and refine further the results presented.

【 授权许可】

   
2015 Da Dalt et al.; licensee BioMed Central.

【 参考文献 】

• [1]Weiss KB, Wagner R, Nasca TJ: Development, testing, and implementation of the ACGME Clinical Learning Environment Review (CLER) program. J Grad Med Educ 2012, 4:396-8.
• [2]Snell L, Tallett S, Haist S, Hays R, Norcini J, Prince K, et al.: A review of the evaluation of clinical teaching: new perspectives and challenges. Med Educ 2000, 34:862-70.
• [3]Ghosh AK, Beckman TJ, Cook DA, Erwin PJ, Mandrekar JN: How reliable are assessments of clinical teaching? A review of the published instruments. J Gen Intern Med 2004, 19:971-7.
• [4]Lombarts KM, Bucx MJ, Arah OA: Development of a system for the evaluation of the teaching qualities of anesthesiology faculty. Anesthesiology 2009, 111:709-16.
• [5]Fluit CR, Bolhuis S, Grol R, Laan R: Wensing: Assessing the quality of clinical teachers: a systematic review of content and quality of questionnaires for assessing clinical teachers. J Gen Intern Med 2010, 25:1337-45.
• [6]Arah OA, Hoekstra JBL, Bos AP, Lombarts KM: New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey. PLoS ONE 2011, 6:e25983.
• [7]Nation JG, Carmichael E, Fidler H, Violato C: The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: a psychometric analysis. Medical Teacher 2011, 33:e290-6.
• [8]van der Leeuw R, Lombarts K, Heineman MJ, Arah O: Systematic evaluation of the teaching qualities of obstet- rics and gynecology faculty: Reliability and validity of the SETQ tools. PLoS ONE 2011, 6:e19142.
• [9]Boerebach BC, Arah OA, Busch OR, Lombarts KM: Reliable and valid tools for measuring surgeons’ teaching performance: residents’ vs. self Evaluation. J Surgical Education 2012, 69:511-20.
• [10]Da Dalt L, Callegaro S, Mazzi A, Scipioni A, Lago P, Chiozza ML, et al.: A model of quality assurance and quality improvement for post-graduate medical education in Europe. Med Teach 2010, 32:e57-64.
• [11]Da Dalt L, Anselmi P, Bressan S, Carraro S, Baraldi E, Robusto E, et al.: A short questionnaire to assess pediatric resident’s competencies: the validation process. Ital J Pediatr 2013, 5:39-41.
• [12]American Board of Pediatrics. The General Pediatrics International In-training Examination (I-ITE). [http://abp-intl.org/intrainingexamination.html], (last access June 4, 2014).
• [13]Linacre JM: Many-facet Rasch measurement. MESA Press, Chicago, IL; 1989.
• [14]Anselmi P, Vianello M, Voci A, Robusto E: Implicit sexual attitude of heterosexual, gay and bisexual individuals: disentangling the contribution of specific associations to the overall measure. Plos One 2013, 8:e78990.
• [15]Anselmi P, Vianello M, Robusto E: Preferring thin people does not imply derogating fat people. A Rasch analysis of the implicit weight attitude. Obesity 2013, 21:261-5.
• [16]Fisher AG: The assessment of IADL motor skills: an application of many-facet Rasch analysis. Am J Occup Ther 1993, 47:319-29.
• [17]Heinemann AW, Linacre JM, Wright BD, Hamilton BB, Granger C: Prediction of rehabilitation outcomes with disability measures. Arch Phys Med Rehab 1994, 75:133-43.
• [18]Clauser BE, Ross LP, Nungester RJ, Clyman SG: An evaluation of the Rasch model for equating multiple forms of a performance assessment of physicians’ patient-management skills. Acad Med 1997, 39(Suppl 1):76-8.
• [19]Conrad KJ, Wright BD, McKnight P, McFall M, Fontana A, Rosenheck R: Comparing traditional and Rasch analyses of the Mississippi PTSD Scale: revealing limitations of reverse-scored items. J Appl Meas 2004, 39:15-30.
• [20]de Morton NA, Nolan JS: Unidimensionality of the Elderly Mobility Scale in older acute medical patients: different methods, different answers. J Clin Epidemiol 2011, 39:667-74. doi:10.1016/j.jclinepi.2010.09.004
• [21]Haley SM, McHorney CA, Ware JE Jr: Evaluation of the MOS SF-36 physical functioning scale (PF-10): I. unidimensionality and reproducibility of the Rasch item scale. J Clin Epidemiol 1994, 47:671-84.
• [22]Linacre JM: Facets Rasch measurement computer program (Version 3.71.3) [Computer software]. Winsteps.com, Chicago, IL; 2013.
• [23]Wright BD, Linacre JM: Reasonable mean-square fit values. Rasch Meas Trans 1994, 8:370.
• [24]Linacre JM: Winsteps (Version 3.68.0) [Computer software]. Winsteps.com, Chicago, IL; 2009.
• [25]Linacre JM: Optimizing rating scale category effectiveness. J Appl Meas 2002, 39:85-106.
• [26]Smith EV: Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. J Appl Meas 2001, 39:281-311.
• [27]Fisher WP: Reliability statistics. Rasch Meas Trans 1992, 39:238.
• [28]Messick S: Validity. In Educational measurement. 3rd edition. Edited by Linn RL. Macmillan, New York; 1989:13-103.
• [29]Forrest M, Andersen B: Ordinal scale and statistics in medical research. BMJ 1986, 39:537-8. doi:10.1136/bmj.292.6519.537
• [30]Merbitz C, Morris J, Grip JC: Ordinal scales and foundations of mis-inference. Arch Phys Med Rehabil 1989, 39:308-12.
• [31]Kahler E, Rogausch A, Brunner E, Himmel W: A parametric analysis of ordinal quality-of-life data can lead to erroneous results. J Clin Epidemiol 2008, 39:475-80.
• [32]Fluit C, Bolhuis S, Grol R, Ham M, Feskens R, Laan R, et al.: Evaluation and feedback for effective clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles. Med Teach 2012, 34:893-901.
• [33]Fluit CV, Bolhuis S, Klaassen T, DE Visser M, Grol R, Laan R, et al.: Residents provide feedback to their clinical teachers: reflection through dialogue. Med Teach 2013, 35:e1485-92.
• [34]Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ: Evaluating obstetrical residency programs using patient outcomes. JAMA 2009, 302:1277-83.