A german-language competency-based multisource feedback instrument for residents: development and validity evidence

BMC Medical Education

Daniel Stricker¹  Eva K. Hennel¹  Christoph Berendonk¹  Sören Huwendiek¹  Sigrid Harendza²  Ulrike Subotic³ 

[1] Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012, Bern, Switzerland;Department of Internal Medicine, University Medical Centre Hamburg-Eppendorf, Martinistr. 52, 20246, Hamburg, Germany;University Children’s Hospital Basel, Spitalstrasse 33, 4056, Basel, Switzerland;

关键词: Multisource feedback;    360-degree;    Workplace-based assessment;    Assessment;    Postgraduate training;    Continuous professional development;   

DOI  :  10.1186/s12909-020-02259-2

来源: Springer

【 摘 要 】

BackgroundIn medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.MethodsWe developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, “unable to comment” ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents’ learning goals and the progress as reported via MSF.ResultsOur resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician’s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach’s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.ConclusionsTo support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

【 授权许可】

CC BY