Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine?

BMC Medical Education

Richard J Hift¹ 

[1] Clinical and Professional Practice Research Group, School of Clinical Medicine, University of KwaZulu-Natal, Durban 4013, South Africa

关键词: Multiple choice;    MEQ;    Mental models;    Essay;    Conceptual change;    Assessment;   

Others  :  1090101
DOI  :  10.1186/s12909-014-0249-2

 received in 2014-05-08, accepted in 2014-11-07,  发布年份 2014

【 摘 要 】

Background

Written assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity.

Discussion

An extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format.

Summary

The evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.

【 授权许可】

   
2014 Hift; licensee BioMed Central.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Siemens G: Connectivism: Learning as Network-Creation. [http://www.elearnspace.org/Articles/networks.htm]
• [2]Siemens G: Connectivism: A learning theory for the digital age. Int J Instr Technol Distance Learn 2005, 2:3-10.
• [3]Perkins DN, Salomon G: Learning transfer. In International Encyclopaedia of adult education and training. 2nd edition. Edited by Tuijnman AC. Pergamon Press, Tarrytown, NY; 1996:422-427.
• [4]Haskell EH: Transfer of learning: Cognition, Instruction, and Reasoning. Academic Press, New York; 2001.
• [5]Spelke E: Initial Knowledge: Six Suggestions. In Cognition on cognition. Edited by Mehler J, Franck S. The MIT Press, Cambridge, MA US; 1995:433-447.
• [6]Barnett SM, Ceci SJ: When and where do we apply what we learn? A taxonomy for far transfer. Psychol Bull 2002, 128:612-637.
• [7]Brown AL: Analogical Learning and Transfer: What Develops? In Similarity and Analogical Reasoning. Edited by Vosniadou S, Ortony A. Cambridge University Press, New York; 1989:369-412.
• [8]Gick ML, Holyoak KJ: Schema Induction and Analogical Transfer. Psychology Press, New York, NY US; 2004.
• [9]Bloom BS: The Cognitive Domain. In Taxonomy of Educational Objectives, Handbook I. David McKay Co Inc, New York; 1956.
• [10]Anderson LW, Krathwohl DR, Airasian PW, Cruikshank KA, Mayer RE, Pintrich PR, Raths J, Wittrock MC: A Taxonomy for Learning, Teaching, and Assessing: a revision of Bloom's Taxonomy of Educational Objectives. Longman, New York; 2001.
• [11]Anderson LW, Sosniak LA: Bloom's Taxonomy: A Forty-year Retrospective. In Ninety-third yearbook of the National Society for the Study of Education: Part II. Edited by Anderson LW, Sosniak LA. University of Chicago Press, Chicago IL; 1994.
• [12]Conklin J: A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. Educ Horiz 2005, 83:154-159.
• [13]Haladyna TM, Downing SM: A taxonomy of multiple-choice item-writing rules. Appl Meas Educ 1989, 2:37-51.
• [14]Haladyna TM: Developing and Validating Multiple-choice Test Items. Mahwah NJ: L. Erlbaum Associates; 1999.
• [15]Miller GE: The assessment of clinical skills/competence/performance. Acad Med 1990, 65:S63-S67.
• [16]Dreyfus HL, Dreyfus SE, Athanasiou T: Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. Free Press, New York; 1986.
• [17]Norcini JJ, Swanson DB, Grosso LJ, Webster GD: Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. Med Educ 1985, 19:238-247.
• [18]Taconnat L, Froger C, Sacher M, Isingrini M: Generation and associative encoding in young and old adults: The effect of the strength of association between cues and targets on a cued recall task. Exp Psychol 2008, 55:23-30.
• [19]Baddeley AD, Eysenck MW, Anderson M: Memory. Psychology Press, New York; 2010.
• [20]Karpicke J, Grimaldi P: Retrieval-based learning: a perspective for enhancing meaningful learning. Educ Psychol Rev 2012, 24:401-418.
• [21]Rohrer D, Pashler H: Recent research on human learning challenges conventional instructional strategies. Educ Res 2010, 39:406-412.
• [22]Smith MA, Roediger HL III, Karpicke JD: Covert retrieval practice benefits retention as much as overt retrieval practice. J Exp Psychol Learn Mem Cogn 2013, 39:1712-1725.
• [23]McDermott KB, Agarwal PK, D’Antonio L, Roediger HL III, McDaniel MA: Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. J Exp Psychol Appl 2014, 20:3-21.
• [24]Cutting MF, Saks NS: Twelve tips for utilizing principles of learning to support medical education. Med Teach 2012, 34:20-24.
• [25]Schuwirth LWT, Van der Vleuten CPM: General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach 2011, 33:783-797.
• [26]Van der Vleuten CP, Schuwirth LW: Assessing professional competence: from methods to programmes. Med Educ 2005, 39:309-317.
• [27]Schuwirth L, Colliver J, Gruppen L, Kreiter C, Mennin S, Onishi H, Pangaro L, Ringsted C, Swanson D, Van der Vleuten C, Wagner-Menghin M: Research in assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011, 33:224-233.
• [28]Schuwirth LWT, Van der Vleuten CPM: Programmatic assessment and Kane's validity perspective. Med Educ 2012, 46:38-48.
• [29]Case SM, Swanson DB: Constructing Written Test Questions for the Basic and Clinical Sciences. 3rd edition. National Board of Medical Examiners, Philadelphia; 2002.
• [30]Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, Roberts T: Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011, 33:206-214.
• [31]Shepard LA: The role of assessment in a learning culture. Educ Res 2000, 29:4-14.
• [32]Coburn CA, Yerkes RM: A study of the behavior of the crow corvus americanus Aud. By the multiple choice method. J Anim Behav 1915, 5:75-114.
• [33]Yerkes RM, Coburn CA: A study of the behavior of the pig Sus Scrofa by the multiple choice method. J Anim Behav 1915, 5:185-225.
• [34]Brown W, Whittell F: Yerkes' multiple choice method with human adults. J Comp Psychol 1923, 3:305-318.
• [35]Yerkes RM: A New method of studying the ideational behavior of mentally defective and deranged as compared with normal individuals. J Comp Psychol 1921, 1:369-394.
• [36]Davidson C: Davidson CN: Now You See It: How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn. Viking Press, New York; 2011.
• [37]Frederiksen JR, Collins A: A Systems Approach to Educational Testing. Technical Report No. 2. Center for Technology in Education, New York; 1990.
• [38]Guthrie JT: Testing higher level skills. J Read 1984, 28:188-190.
• [39]Nickerson RS: New directions in educational assessment. Educ Res 1989, 18:3-7.
• [40]Stratford P, Pierce-Fenn H: Modified essay question. Phys Ther 1985, 65:1075-1079.
• [41]Wass V, Van der Vleuten C, Shatzer J, Jones R: Assessment of clinical competence. Lancet 2001, 357:945.
• [42]Rotfield H: Are we teachers or job trainers? Acad Mark Sci Q 1998, 2:2.
• [43]Crocker L, Algina J: Introduction to Classical & Modern Test Theory. Holt, Rinehart and Winston, Inc., Fort Worth, TX; 1986.
• [44]Angoff W: Test reliability and effective test length. Psychometrika 1953, 18:1-14.
• [45]Palmer EJ, Devitt PG: Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ 2007, 7:49-49.
• [46]Feletti GI, Smith EK: Modified essay questions: Are they worth the effort? Med Educ 1986, 20:126-132.
• [47]Palmer EJ, Duggan P, Devitt PG, Russell R: The modified essay question: its exit from the exit examination? Med Teach 2010, 32:e300-e307.
• [48]Schuwirth LW, Van der Vleuten CPM: Different written assessment methods: what can be said about their strengths and weaknesses? Med Educ 2004, 38:974-979.
• [49]Lukhele R, Thissen D, Wainer H: On the relative value of multiple-choice, constructed response, and examinee-selected items on two achievement tests. J Educ Meas 1994, 31:234-250.
• [50]Wainer H, Thissen D: Combining multiple-choice and constructed-response test scores: toward a Marxist theory of test construction. Appl Meas Educ 1993, 6:103-118.
• [51]Facione PA: The California Critical Thinking Skills Test--College Level. Technical Report #1. Experimental Validation and Content Validity. California Academic Press, Millbrae CA; 1990.
• [52]Facione PA, Facione NC, Blohm SW, Giancarlo CAF: The California Critical Thinking Skills Test [Revised]. In Millbrae CA: California Academic Press; 2007.
• [53]Rodriguez MC: Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. J Educ Meas 2003, 40:163-184.
• [54]Falk B, Ancess J, Darling-Hammond L: Authentic Assessment in Action: Studies of Schools and Students at Work. Teachers College Press, United States of America; 1995.
• [55]Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDuca T, Lew S, Page GG, Southgate LH: The relationship between competence and performance: implications for assessing practice performance. Med Educ 2002, 36:901-909.
• [56]Wilkinson TJ, Frampton CM: Comprehensive undergraduate medical assessments improve prediction of clinical performance. Med Educ 2004, 38:1111-1116.
• [57]Baker EL: Standards for Educational and Psychological Testing. Sage Publications, Inc; 2012.
• [58]Eignor DR: The Standards for Educational and Psychological Testing. In APA Handbook of Testing and Assessment in Psychology, Vol 1: Test Theory and Testing and Assessment in Industrial and Organizational Psychology. Edited by Geisinger KF, Bracken BA, Carlson JF, Hansen J-IC, Kuncel NR, Reise SP, Rodriguez MC. American Psychological Association, Washington, DC, US; 2013:245-250.
• [59]Eignor DR: Standards for the development and use of tests: The Standards for Educational and Psychological Testing. Eur J Psychol Assess 2001, 17:157-163.
• [60]Downing SM: Validity: on the meaningful interpretation of assessment data. Med Educ 2003, 37:830.
• [61]Messick S: The interplay of evidence and consequences in the validation of performance assessments. Educ Res 1994, 23:13-23.
• [62]Kuechler WL, Simkin MG: Why is performance on multiple-choice tests and constructed-response tests Not more closely related? theory and an empirical test. Decis Sci J Innov Educ 2010, 8:55-73.
• [63]Norman GR, Smith EK, Powles AC, Rooney PJ: Factors underlying performance on written tests of knowledge. Med Educ 1987, 21:297-304.
• [64]Bacon DR: Assessing learning outcomes: a comparison of multiple-choice and short-answer questions in a marketing context. J Mark Educ 2003, 25:31-36.
• [65]Kastner M, Stangla B: Multiple choice and constructed response tests: Do test format and scoring matter? Procedia - Social and Behav Sci 2011, 12:263-273.
• [66]Nichols P, Sugrue B: The lack of fidelity between cognitively complex constructs and conventional test development practice. Educ Measurement: Issues Pract 1999, 18:18-29.
• [67]Bennett RE, Rock DA, Wang M: Equivalence of free-response and multiple-choice items. J Educ Meas 1991, 28:77-92.
• [68]Bridgeman B, Rock DA: Relationships among multiple-choice and open-ended analytical questions. J Educ Meas 1993, 30:313-329.
• [69]Thissen D, Wainer H: Are tests comprising both multiple-choice and free-response items necessarily less unidimensional. J Educ Meas 1994, 31:113.
• [70]Lissitz RW, Xiaodong H, Slater SC: The contribution of constructed response items to large scale assessment: measuring and understanding their impact. J Appl Testing Technol 2012, 13:1-52.
• [71]Traub RE, Fisher CW: On the equivalence of constructed- response and multiple-choice tests. Appl Psychol Meas 1977, 1:355-369.
• [72]Martinez ME: Cognition and the question of test item format. Educ Psychol 1999, 34:207-218.
• [73]Hee-Sun L, Liu OL, Linn MC: Validating measurement of knowledge integration in science using multiple-choice and explanation items. Appl Meas Educ 2011, 24:115-136.
• [74]Wilson M, Wang W-C: Complex composites: Issues that arise in combining different modes of assessment. Appl Psychol Meas 1995, 19:51-71.
• [75]Ercikan K, Schwartz RD, Julian MW, Burket GR, Weber MM, Link V: Calibration and scoring of tests with multiple-choice and constructed-response item types. J Educ Meas 1998, 35:137-154.
• [76]Epstein ML, Lazarus AD, Calvano TB, Matthews KA, Hendel RA, Epstein BB, Brosvic GM: Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. Psychological Record 2002, 52:187-201.
• [77]Schuwirth LWT, Van der Vleuten CPM: Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011, 33:478-485.
• [78]Bridgeman B, Morgan R: Success in college for students with discrepancies between performance on multiple-choice and essay tests. J Educ Psychol 1996, 88:333-340.
• [79]Bleske-Rechek A, Zeug N, Webb RM: Discrepant performance on multiple-choice and short answer assessments and the relation of performance to general scholastic aptitude. Assessment Eval Higher Educ 2007, 32:89-105.
• [80]Hakstian AR: The Effects of Type of Examination Anticipated on Test Preparation and Performance. J Educ Res 1971, 64:319.
• [81]Scouller K: The influence of assessment method on Students' learning approaches: multiple choice question examination versus assignment essay. High Educ 1998, 35:453-472.
• [82]Thomas PR, Bain JD: Contextual dependence of learning approaches: The effects of assessments. Human Learning: J Pract Res Appl 1984, 3:227-240.
• [83]Watkins D: Factors influencing the study methods of Australian tertiary students. High Educ 1982, 11:369-380.
• [84]Minbashian A, Huon GF, Bird KD: Approaches to studying and academic performance in short-essay exams. High Educ 2004, 47:161-176.
• [85]Yonker JE: The relationship of deep and surface study approaches on factual and applied test-bank multiple-choice question performance. Assess Eval Higher Educ 2011, 36:673-686.
• [86]Joughin G: The hidden curriculum revisited: a critical review of research into the influence of summative assessment on learning. Assess Eval Higher Educ 2010, 35:335-345.
• [87]Scouller KM, Prosser M: Students' experiences in studying for multiple choice question examinations. Stud High Educ 1994, 19:267.
• [88]Hadwin AF, Winne PH, Stockley DB, Nesbit JC, Woszczyna C: Context moderates students' self-reports about how they study. J Educ Psychol 2001, 93:477-487.
• [89]Birenbaum M: Assessment and instruction preferences and their relationship with test anxiety and learning strategies. High Educ 2007, 53:749-768.
• [90]Birenbaum M: Assessment preferences and their relationship to learning strategies and orientations. High Educ 1997, 33:71-84.
• [91]Smith SN, Miller RJ: Learning approaches: examination type, discipline of study, and gender. Educ Psychol 2005, 25:43-53.
• [92]Rabinowitz HK, Hojat M: A comparison of the modified essay question and multiple choice question formats: their relationship to clinical performance. Fam Med 1989, 21:364-367.
• [93]Paterson DG: Do new and old type examinations measure different mental functions? School Soc 1926, 24:246-248.
• [94]Schuwirth LW, Verheggen MM, Van der Vleuten CPM, Boshuizen HP, Dinant GJ: Do short cases elicit different thinking processes than factual knowledge questions do? Med Educ 2001, 35:348-356.
• [95]Tanner DE: Multiple-choice items: Pariah, panacea or neither of the above? Am Second Educ 2003, 31:27.
• [96]Cilliers FJ, Schuwirth LW, van der Vleuten CP: Modelling the pre-assessment learning effects of assessment: evidence in the validity chain. Med Educ 2012, 46:1087-1098.
• [97]Jonassen DH, Strobel J: Modeling for Meaningful Learning. In Engaged Learning with Emerging Technologies. Edited by Hung D. Springer, Amsterdam; 2006:1-27.
• [98]Derry SJ: Cognitive schema theory in the constructivist debate. Educ Psychol 1996, 31:163-174.
• [99]Kim MK: Theoretically grounded guidelines for assessing learning progress: cognitive changes in Ill-structured complex problem-solving contexts. Educ Technol Res Dev 2012, 60:601-622.
• [100]Mayer RE: Models for Understanding. Rev Educ Res 1989, 59:43-64.
• [101]Jonassen D, Strobel J, Gottdenker J: Model building for conceptual change. Interact Learn Environ 2005, 13:15-37.
• [102]Jonassen DH: Tools for representing problems and the knowledge required to solve them. Edited by Tergan S-O, Keller T. Berlin, Heidelberg: Springer; 2005:82–94.
• [103]Bogard T, Liu M, Chiang Y-H: Thresholds of knowledge development in complex problem solving: a multiple-case study of advanced Learners' cognitive processes. Educ Technol Res Dev 2013, 61:465-503.
• [104]Van Gog T, Ericsson KA, Rikers RMJP: Instructional design for advanced learners: establishing connections between the theoretical frameworks of cognitive load and deliberate practice. Educ Technol Res Dev 2005, 53:73-81.
• [105]Schmidt HG, Norman GR, Boshuizen HP: A cognitive perspective on medical expertise: theory and implication. Acad Med 1990, 65:611-621.
• [106]Schmidt HG, Rikers RMJP: How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ 2007, 41:1133-1139.
• [107]Norman G, Young M, Brooks L: Non-analytical models of clinical reasoning: the role of experience. Med Educ 2007, 41:1140-1145.
• [108]Ericsson KA, Prietula MJ, Cokely ET: The Making of an Expert. Harv Bus Rev 2007, 85:114-121.
• [109]Hoffman RR: How Can Expertise be Defined? Implications of Research From Cognitive Psychology. In Exploring Expertise. Edited by Williams R, Faulkner W, Fleck J. University of Edinburgh Press, Edinburgh; 1996:81-100.
• [110]Norman GR: Problem-solving skills, solving problems and problem-based learning. Med Educ 1988, 22:279-286.
• [111]Ifenthaler D, Seel NM: Model-based reasoning. Comput Educ 2013, 64:131-142.
• [112]Jonassen D: Using cognitive tools to represent problems. J Res Technol Educ 2003, 35:362-381.
• [113]Mayer RE, Wittrock MC: Problem-Solving Transfer. In Handbook of Educational Psychology. Edited by Berliner DC, Calfee RC. Macmillan Library Reference USA, New York, NY; 1996:47-62.
• [114]Zhang J, Norman DA: Representations in distributed cognitive tasks. Cogn Sci 1994, 18:87-122.
• [115]Simon HA: Information-Processing Theory of Human Problem Solving. In Handbook of Learning & Cognitive Processes: V Human Information. Edited by Estes WK. Lawrence Erlbaum, Oxford England; 1978:271-295.
• [116]Jensen JL, Woodard SM, Kummer TA, McDaniel MA: Teaching to the test…or testing to teach: exams requiring higher order thinking skills encourage greater conceptual understanding. Educ Psychol Rev 2014, 26:307-329.
• [117]Cohen-Schotanus J, Van der Vleuten CPM: A standard setting method with the best performing students as point of reference: practical and affordable. Med Teach 2010, 32:154-160.
• [118]Desjardins I, Touchie C, Pugh D, Wood TJ, Humphrey-Murto S: The impact of cueing on written examinations of clinical decision making: a case study. Med Educ 2014, 48:255-261.
• [119]Pretz JE, Naples AJ, Sternberg RJ: Recognizing, Defining, and Representing Problems. In The Psychology of Problem Solving. Edited by Davidson JE, Sternberg RJ. Cambridge University Press, New York, NY US; 2003:3-30.
• [120]Schuwirth LWT, Schuwirth LWT, Van der Vleuten CPM: ABC of learning and teaching in medicine: written assessment. BMJ: British Med J (International Edition) 2003, 326:643-645.