The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools

BMC Medicine

Jonathan S Dowell⁴  Sandra Nicholson¹  Chris Dewberry²  I C McManus³ 

[1] Institute of Health Science Education, Queen Mary London, Turner Street, London E1 2AD, UK;Department of Organizational Psychology, Birkbeck, University of London, Malet Street, Bloomsbury, London WC1E 7HX, UK;Research Department of Clinical, Educational and Health Psychology, Division of Psychology and Language Sciences, University College London, Gower Street, London WC1E 6BT, UK;Undergraduate Medical Education, Ninewells Hospital and Medical School, Dundee, Scotland DD1 9SY, UK

关键词: Contextual measures;    Socio-economic factors;    UKCAT;    Aptitude tests;    Educational attainment;    Medical student selection;   

Others  :  855376
DOI  :  10.1186/1741-7015-11-244

 received in 2013-07-05, accepted in 2013-09-10,  发布年份 2013

【 摘 要 】

Background

Most UK medical schools use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the United Kingdom Clinical Aptitude Test (UKCAT), and its four sub-scales, along with measures of educational attainment, individual and contextual socio-economic background factors, as predictors of performance in the first year of medical school training.

Methods

A prospective study of 4,811 students in 12 UK medical schools taking the UKCAT from 2006 to 2008 as a part of the medical school application, for whom first year medical school examination results were available in 2008 to 2010.

Results

UKCAT scores and educational attainment measures (General Certificate of Education (GCE): A-levels, and so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on) were significant predictors of outcome. UKCAT predicted outcome better in female students than male students, and better in mature than non-mature students. Incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by sex (male students performing less well), ethnicity (non-White students performing less well), and a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level (irrespective of public or private sector) performing less well. Multilevel modeling showed no differences between medical schools in predictive ability of the various measures. UKCAT sub-scales predicted similarly, except that Verbal Reasoning correlated positively with performance on Theory examinations, but negatively with Skills assessments.

Conclusions

This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training. UKCAT has predictive validity as a predictor of medical school outcome, particularly in mature applicants to medical school. UKCAT offers small but significant incremental validity which is operationally valuable where medical schools are making selection decisions based on incomplete measures of educational attainment. The study confirms the validity of using all the existing measures of educational attainment in full at the time of selection decision-making. Contextual measures provide little additional predictive value, except that students from high attaining secondary schools perform less well, an effect previously shown for UK universities in general.

【 授权许可】

   
2013 McManus et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]McManus IC, Richards P, Winder BC, Sproston KA: Clinical experience, performance in final examinations, and learning style in medical students: prospective study. BMJ 1998, 316:345-350.
• [2]McManus IC, Smithers E, Partridge P, Keeling A, Fleming PR: A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. BMJ 2003, 327:139-142.
• [3]Bekhradnia B, Thompson J: Who Does Best at University?. London: Higher Education Funding Council England; 2002. http://webarchive.nationalarchives.gov.uk/20081202000732/http://hefce.ac.uk/Learning/whodoes/ webcite
• [4]Trapmann S, Hell B, Weigand S, Schuler H: Die Validität von Schulnoten zur Vorhersage des Studienerfolgs – eine Metaanalyse. Z Pädagog Psychol 2007, 21:11-27.
• [5]Cohen-Schotanus J, Muijtjens AM, Reinders JJ, Agsteribbe J, van Rossum HJ, van der Vleuten CP: The predictive validity of grade point average scores in a partial lottery medical school admission system. Med Educ 2006, 40:1012-1019.
• [6]Iramaneerat C: Predicting academic achievement in the medical school with high school grades. J Med Assoc Thai 2006, 89:1497-1505.
• [7]Coates H: Establishing the criterion validity of the Graduate Medical School Admissions Test (GAMSAT). Med Educ 2008, 42:999-1006.
• [8]Contreras R, Ubilla S, Ugalde H, Vicentini E: Comparative-study of the academic aptitude-test and final grading in Chilean medical-schools. Rev Med Chil 1984, 112:1033-1043.
• [9]Kadmon G, Kirchner A, Duelli R, Resch F, Kadmon M: Z Evid Fortbild Qual Gesundhwes. 2012, 106:125-130.
• [10]Trost G: The aptitude-test for medical-students has proved its worth. München Med Wochen 1982, 124:885-886.
• [11]Khan JS, Tabasum S, Mukhtar O: Comparison of pre-medical academic achievement, entrance test and aptitude test scores in admission selection process. J Pak Med Assoc 2013, 63:552-557.
• [12]Lia C, Cavaggioni G: The selection process in the faculty of medicine: the usefulness of a psychological and aptitude assessment. Clin Ter 2013, 164:39-42.
• [13]Garcia-Mangas JA, Viniegra-Velasquez L, Arellano-Lopez J, Garcia-Moreno J: Evaluation of the clinical aptitude in physicians of the first level of attention. Rev Med Inst Mexicano Seguro Soc 2005, 43:465-472.
• [14]McGregor A: Aptitude-tests for entry to Swiss medical faculties. Lancet 1994, 344:1424.
• [15]Collin VT, Violato C, Hecker K: Aptitude, achievement and competence in medicine: a latent variable path model. Adv Health Sci Educ 2009, 14:355-366.
• [16]Zeleznik C, Hojat M, Veloski J: Long-range predictive and differential validities of the scholastic aptitude-test in medical-school. EPM 1983, 43:223-232.
• [17]Wright SR, Bradley PM: Has the UK clinical aptitude test improved medical student selection? Med Educ 2010, 44:1069-1076.
• [18]Yates J, James D: The UK clinical aptitude test and clinical course performance at Nottingham: a prospective cohort study. BMC Med Educ 2013, 13:32. BioMed Central Full Text
• [19]Tiffin PA, Dowell JS, McLachlan JC: Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort. BMJ 2012, 344:e1805.
• [20]UKCAT: UKCAT: 2006 Annual report. Cambridge, UK: United Kingdom Clinical Aptitude Test; 2008. http://www.ukcat.ac.uk/App_Media/uploads/pdf/Annual%20Report%202006.pdf webcite
• [21]Emery JL, Bell JF: The predictive validity of the BioMedical admissions test for pre-clinical examination performance. Med Educ 2009, 43:557-564.
• [22]Hunter JE, Schmidt FL, Le H: Implications of direct and indirect range restriction for meta-analysis methods and findings. J Appl Psychol 2006, 91:594-612.
• [23]McManus IC, Ferguson E, Wakeford R, Powis D, James D: Predictive validity of the BioMedical Admissions Test (BMAT): an evaluation and case study. Med Teach 2011, 33:53-57.
• [24]Cassidy J: UKCAT among the pigeons. BMJ 2008, 336:691-692.
• [25]Cleland JA, French FH, Johnson PW: A mixed-methods study identifying and exploring medical students’ views of the UKCAT. Med Teach 2011, 3:249.
• [26]Hausknecht JP, Halpert JA, Di Paolo NT, Moriarity Gerrard MO: Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability. J Appl Psychol 2007, 92:373-385.
• [27]Credé M, Kuncel NR: Study habits, skills, and attitudes: the third pillar supporting collegiate academic performance. Perspect Psychol Sci 2008, 3:425-453.
• [28]Yates J, James D: The value of the UK clinical aptitude test in predicting pre-clinical performance: a prospective cohort study at Nottingham Medical School. BMC Med Educ 2010, 10:55. BioMed Central Full Text
• [29]Adam J, Bore M, McKendree J, Munro D, Powis D: Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective study. BMC Med Educ 2012, 12:69. BioMed Central Full Text
• [30]Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G: Does the UKCAT predict Year 1 performance in medical school? Med Educ 2009, 43:1203-1209.
• [31]Turner R, Nicholson S: Can the UK clinical aptitude test (UKCAT) select suitable candidates for interview? Med Educ 2011, 45:1041-1047.
• [32]Donnon T, Paolucci EO, Violato C: The predictive validity of the MCAT for medical school performance medical board licensing examinations: a meta-analysis of the published research. Acad Med 2007, 82:100-106.
• [33]Adam J, Dowell J, Greatrix R: Use of UKCAT scores in student selection by UK medical schools, 2006–2010. BMC Med Educ 2011, 11:98. BioMed Central Full Text
• [34]McManus IC, Dewberry C, Nicholson S, Dowell J: The UKCAT-12 Study: Technical Report. UKCAT Consortium: Nottingham, UK; 2012. http://www.ukcat.ac.uk/App_Media/uploads/pdf/UKCAT-TechnicalReport-March2012-WithBackgroundAndSummary-Sep2013v2.pdf webcite
• [35]McManus IC, Dewberry C, Nicholson S, Dowell JS, Woolf K, Potts HW: Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies. BMC Med 2013, 11:243.
• [36]Emery JL, Bell JF, Vidal Rodeiro CL: The BioMedical admissions test for medical student selection: issues of fairness and bias. Med Teach 2011, 33:62-71.
• [37]Johnson J, Hayward G: Expert Group Report for Award Seeking Admission to the UCAS Tariff: Scottish Highers and Advanced HIghers. Cheltenham: UCAS; 2008.
• [38]UKCAT: UKCAT: 2007 Annual Report. Cambridge: United Kingdom Clinical Aptitude Test; 2008. http://www.ukcat.ac.uk/App_Media/uploads/pdf/Annual%20Report%202007.pdf webcite
• [39]UKCAT: UKCAT 2008 Annual Report. Nottingham, UK: UKCAT; 2008.
• [40]Pearson VUE: Technical Report: Testing Interval: July 11 2006 - October 14,2006 (January 2007, Revised May 3, 2007). Chicago, IL: Pearson VUE; 2007.
• [41]Pearson VUE: Technical Report regarding the UK Clinical Aptitude Test Examination for the UKCAT consortium; Testing Interval: 18 June 2007–10 October 2007; January 2008. Chicago, IL: Pearson VUE; 2008.
• [42]Pearson VUE: Technical Reports; UK Clinical Aptitude Test Consortium; UKCAT examination; Teseting Interval: 7 July 2008–12 October 2008. Chicago, IL: Pearson VUE; 2009.
• [43]Office for National Statistics: The National Statistics Socio-economic classification: User Manual. Newport: National Statistics; 2005. http://www.ons.gov.uk/ons/guide-method/classifications/archived-standard-classifications/soc-and-sec-archive/the-national-statistics-socio-economic-classification--user-manual.pdf webcite
• [44]McLennan D, Barnes H, Noble M, Davies J, Garratt E, Dibben C: The English Indices of Deprivation 2010. London: Department for Communities and Local Government; 2011.
• [45]McManus IC, Richards P: Prospective survey of performance of medical students during preclinical years. BMJ 1986, 293:124-127.
• [46]Yates J: When did they leave, and why? A retrospective case study of attrition on the Nottingham undergraduate medical course. BMC Medical Education 2012, 12:43. BioMed Central Full Text
• [47]McManus IC: Medical School differences: beneficial diversity or harmful deviations? Qual Safe Health Care 2003, 12:324-325.
• [48]McManus IC, Woolf K, Dacre J, Paice E, Dewberry C: The academic backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the Specialist Register in UK medical students and doctors. BMC Medicine 2013, 11:242.
• [49]McManus IC, Elder AT, De Champlain A, Dacre JE, Mollon J, Chis L: Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC Med 2008, 6:5. BioMed Central Full Text
• [50]Partington R: The Predictive Effectiveness of Metrics in Admission to the University of Cambridge. Cambridge, Uk: University of Cambridge; 2011. http://www.admin.cam.ac.uk/offices/admissions/research/docs/prefective_effectiveness_of_metrics_in_admission.pdf webcite
• [51]Department for Education: GCSE and AS Level Results Comparison: Predictors of Getting a 2:1 Degree or Above. London: Department for Education; 2013. [http://www.gov.uk/government/publications/a-comparison-of-gcse-results-and-as-level-results-as-a-predictor-of-getting-a-21-or-above-at-university webcite]
• [52]Woolf K, Potts HW, McManus IC: The relationship between ethnicity and academic performance in UK-trained doctors and medical students: a systematic review and meta-analysis. BMJ 2011, 342:d901.
• [53]Anonymous: Schooling effects on higher education achievement. 2003. [http://www.hefce.ac.uk/pubs/hefce/2003/03_32.htm webcite]
• [54]Nagengast B, Marsh HW: Big fish in little ponds aspire more: mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. J Educ Psychol 2012, 104:1033-1053.
• [55]Anders J: The link between household income, university applications and university attendance. Fiscal Studies 2012, 33:185-210.
• [56]Sackett PR, Kuncel NR, Beatty AS, Rigdon JL, Shen W, Kiger TB: The role of socioeconomic status in SAT-grade relationships and in college admissions decisions. Psychol Sci 2012, 23:1000-1007.
• [57]McManus IC, Powis DA, Wakeford R, Ferguson E, James D, Richards P: Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. BMJ 2005, 331:555-559.
• [58]Ferguson E, James D, Madeley L: Factors associated with success in medical school and in a medical career: systematic review of the literature. BMJ 2002, 324:952-957.
• [59]Siu E, Reiter H: Overview: what's worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ 2009, 14:775.
• [60]Royal Commission: Royal Commission on Medical Education (The Todd Report), Cmnd 3569. London: Her Majesty’s Stationery Office; 1968.