The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training

BMC Medical Education

Karyn Mossman²  Marcia Grant¹  Michael Frommer²  Annette Burgess²  Tyler Clark²  Chris Roberts² 

[1] General Practice Education and Training, 10 Rudd Street, GPO Box 2914, Canberra, ACT 2601, Australia;Sydney Medical School, University of Sydney, Sydney, NSW 2006, Australia

关键词: General practice;    Assessment centre;    Situational judgment test;    Multiple-mini-interview;    Assessment;    Postgraduate training;    Validity;    Selection;   

Others  :  1091912
DOI  :  10.1186/1472-6920-14-169

 received in 2014-01-24, accepted in 2014-06-19,  发布年份 2014

【 摘 要 】

Background

Entry into specialty training was determined by a National Assessment Centre (NAC) approach using a combination of a behavioural Multiple-Mini-Interview (MMI) and a written Situational Judgement Test (SJT). We wanted to know if interviewers could make reliable and valid decisions about the non-cognitive characteristics of candidates with the purpose of selecting them into general practice specialty training using the MMI. Second, we explored the concurrent validity of the MMI with the SJT.

Methods

A variance components analysis estimated the reliability and sources of measurement error. Further modelling estimated the optimal configurations for future MMI iterations. We calculated the relationship of the MMI with the SJT.

Results

Data were available from 1382 candidates, 254 interviewers, six MMI questions, five alternate forms of a 50-item SJT, and 11 assessment centres. For a single MMI question and one assessor, 28% of the variance between scores was due to candidate-to-candidate variation. Interviewer subjectivity, in particular the varying views that interviewer had for particular candidates accounted for 40% of the variance in scores. The generalisability co-efficient for a six question MMI was 0.7; to achieve 0.8 would require ten questions. A disattenuated correlation with the SJT (r = 0.35), and in particular a raw score correlation with the subdomain related to clinical knowledge (r = 0.25) demonstrated evidence for construct and concurrent validity. Less than two per cent of candidates would have failed the MMI.

Conclusion

The MMI is a moderately reliable method of assessment in the context of a National Assessment Centre approach. The largest source of error relates to aspects of interviewer subjectivity, suggesting enhanced interviewer training would be beneficial. MMIs need to be sufficiently long for precise comparison for ranking purposes. In order to justify long term sustainable use of the MMI in a postgraduate assessment centre approach, more theoretical work is required to understand how written and performance based test of non-cognitive attributes can be combined, in a way that achieves acceptable generalizability, and has validity.

【 授权许可】

   
2014 Roberts et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Roberts C, Togno JM: Selection into specialist training programs: an approach from general practice. Med J Aust 2011, 194(2):93-95.
• [2]Papadakis MA, Teherani A, Banach MA, Knettler TR, Rattner SL, Stern DT, Veloski JJ, Hodgson CS: Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med 2005, 353(25):2673-2682.
• [3]Patterson F, Ferguson E, Norfolk T, Lane P: A new selection system to recruit general practice registrars: preliminary findings from a validation study. BMJ 2005, 330(7493):711-714.
• [4]Provan JL, Cuttress L: Preferences of program directors for evaluation of candidates for postgraduate training. CMAJ 1995, 153(7):919-923.
• [5]Patterson F, Ashworth V, Zibarras L, Coan P, Kerrin M, O’Neill P: Evaluations of situational judgement tests to assess non-academic attributes in selection. Med Educ 2012, 46(9):850-868.
• [6]Ahmed H, Rhydderch M, Matthews P: Can knowledge tests and situational judgement tests predict selection centre performance? Med Educ 2012, 46(8):777-784.
• [7]Lievens F, Patterson F: The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J Appl Psychol 2011, 96(5):927-940.
• [8]Dore KL, Kreuger S, Ladhani M, Rolfson D, Kurtz D, Kulasegaram K, Cullimore AJ, Norman GR, Eva KW, Bates S, Reiter H: The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions. Acad Med 2010, 85(10):S2060-S2063. Supplement(RIME): Proceedings of the Forty-Ninth Annual Conference November 7-November 10, 2010
• [9]Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR: Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ 2009, 43(8):767-775.
• [10]Eva KW, Rosenfeld J, Reiter HI, Norman GR: An admissions OSCE: the multiple mini-interview. Med Educ 2004, 38(3):314-326.
• [11]Brownell K, Lockyer J, Collin T, Lemay JF: Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility. Med Teach 2007, 29(4):394-396.
• [12]Harris S, Owen C: Discerning quality: using the multiple mini-interview in student selection for the Australian National University Medical School. Med Educ 2007, 41(3):234-241.
• [13]Roberts C, Walton M, Rothnie I, Crossley J, Lyon P, Kumar K, Tiller D: Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. Med Educ 2008, 42(4):396-404.
• [14]Al Alwan I, Al Kushi M, Tamim H, Magzoub M, Elzubeir M: Health sciences and medical college preadmission criteria and prediction of in-course academic performance: a longitudinal cohort study. Adv Health Sci Educ Theory Pract 2012, 18(3):1-12.
• [15]Halpern N, Bentov-Gofrit D, Matot I, Abramowitz MZ: The effect of integration of non-cognitive parameters on medical students’ characteristics and their intended career choices. Isr Med Assoc J 2011, 13(8):488-493.
• [16]O’Brien A, Harvey J, Shannon M, Lewis K, Valencia O: A comparison of multiple mini-interviews and structured interviews in a UK setting. Med Teach 2011, 33(5):397-402.
• [17]Uijtdehaage S, Doyle L, Parker N: Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders. Acad Med 2011, 86(8):1032-1039.
• [18]Roberts C, Zoanetti N, Rothnie I: Validating a multiple mini-interview question bank assessing entry-level reasoning skills in candidates for graduate-entry medicine and dentistry programmes. Med Educ 2009, 43(4):350-359.
• [19]Dore KL, Hanson M, Reiter HI, Blanchard M, Deeth K, Eva KW: Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Acad Med 2006, 81(10 Suppl):S70-S73.
• [20]Rosenfeld JM, Reiter HI, Trinh K, Eva KW: A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ Theory Pract 2008, 13(1):43-58.
• [21]Hofmeister M, Lockyer J, Crutcher R: The acceptability of the multiple mini interview for resident selection. Fam Med 2008, 40(10):734-740.
• [22]Yen W, Hovey R, Hodwitz K, Zhang S: An exploration of the relationship between emotional intelligence (EI) and the Multiple Mini-Interview (MMI). Adv Health Sci Educ Theory Pract 2011, 16(1):59-67.
• [23]Colliver JA, Conlee MJ, Verhulst SJ: From test validity to construct validity … and back? Med Educ 2012, 46(4):366-371.
• [24]Reiter HI, Eva KW, Rosenfeld J, Norman GR: Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ 2007, 41(4):378-384.
• [25]Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM: Multiple mini-interviews: opinions of candidates and interviewers. Med Educ 2008, 42(2):207-213.
• [26]Patterson F, Baron H, Carr V, Plint S, Lane P: Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Med Educ 2009, 43(1):50-57.
• [27]Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A: A competency model for general practice: implications for selection, training, and development. Br J Gen Pract 2000, 50(452):188-193.
• [28]Reiter HI, Salvatori P, Rosenfeld J, Trinh K, Eva KW: The effect of defined violations of test security on admissions outcomes using multiple mini-interviews. Med Educ 2006, 40(1):36-42.
• [29]Eva KW, Reiter HI, Rosenfeld J, Norman GR: The ability of the multiple mini-interview to predict preclerkship performance in medical school. Academic 2004, 79(10):S40-S42.
• [30]Latham GP, Saari LM, Pursell ED, Campion MA: The situational interview. J Appl Psychol 1980, 65(4):422-427.
• [31]Crossley J, Russell J, Jolly B, Ricketts C, Roberts C, Schuwirth L, Norcini J: ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007, 41(10):926-934.
• [32]Spearman C: The proof and measurement of association between two things. Am J Psychol 1904, 15(1):72-101.
• [33]Axelson RD, Kreiter CD: Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ 2009, 43(12):1198-1202.
• [34]Eva KWP, Reiter HIMDM, Rosenfeld JP, Norman GRP: The relationship between Interviewers’ characteristics and ratings assigned during a multiple mini-interview. Acad Med 2004, 79(6):602-609.
• [35]Guiton G, Hodgson CS, Delandshere G, Wilkerson L: Communication skills in standardized-patient assessment of final-year medical students: a psychometric study. Adv Health Sci Educ Theory Pract 2004, 9(3):179-187.
• [36]Regehr G, Freeman R, Robb A, Missiha N, Heisey R: OSCE performance evaluations made by standardized patients: comparing checklist and global rating scores. Acad Med 1999, 74(10):S135-S137.
• [37]Tavares W, Eva KW: Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract 2013, 18(2):291-303.
• [38]Govaerts MJ, Van de Wiel MW, Schuwirth LW, Van der Vleuten CP, Muijtjens AM: Workplace-based assessment: raters’ performance theories and constructs. Adv Health Sci Educ Theory Pract 2013, 18(3):375-396.
• [39]Wood TJ: Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract 2013, 1-19. doi:10.1007/s10459-013-9453-9
• [40]Roberts C, Rothnie I, Zoanetti N, Crossley J: Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview? Med Educ 2010, 44(7):690-698.
• [41]Downing SM: Validity: on meaningful interpretation of assessment data. Med Educ 2003, 37(9):830-837.
• [42]Townsend PD, Christensen MG, Kreiter CD, zumBrunnen JR: Investigating the use of written and performance-based testing to summarize competence on the case management component of the NBCE part IV–National Practical Examination. Teach Learn Med 2009, 22(1):16-21.
• [43]Verhoeven BH, Hamers JGHC, Scherpbier AJJA, Hoogenboom RJI, Van Der Vleuten CPM: The effect on reliability of adding a separate written assessment component to an objective structured clinical examination. Med Educ 2000, 34(7):525-529.