An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

BMC Medical Education

Fedde Scheele⁴  Didi DM Braat³  Izabela Jozwiak²  Marja GK Dijksterhuis¹ 

[1] Department of Obstetrics and Gynecology, Amphia Ziekenhuis, Langendijk 75, 4819 EV, Breda, The Netherlands;Management of Learning, Maastricht University School of Business and Economics, Tongersestraat 53, 6211 LM, Maastricht, The Netherlands;Department of Obstetrics and Gynecology, University Medical Center Nijmegen, Geert Groteplein 10, 6525 GA, Nijmegen, The Netherlands;School of Medical Sciences, Department of Health Systems Innovation and Education, VU University Medical Center, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands

关键词: Face-validity;    Reliability;    Construct validity;    Internet access;    Knowledge assessment;    Progress testing;    Postgraduate medical education;   

Others  :  1138521
DOI  :  10.1186/1472-6920-13-147

 received in 2013-04-11, accepted in 2013-10-31,  发布年份 2013

【 摘 要 】

Background

Yearly formative knowledge testing (also known as progress testing) was shown to have a limited construct-validity and reliability in postgraduate medical education. One way to improve construct-validity and reliability is to improve the authenticity of a test. As easily accessible internet has become inseparably linked to daily clinical practice, we hypothesized that allowing internet access for a limited amount of time during the progress test would improve the perception of authenticity (face-validity) of the test, which would in turn improve the construct-validity and reliability of postgraduate progress testing.

Methods

Postgraduate trainees taking the yearly knowledge progress test were asked to participate in a study where they could access the internet for 30 minutes at the end of a traditional pen and paper test. Before and after the test they were asked to complete a short questionnaire regarding the face-validity of the test.

Results

Mean test scores increased significantly for all training years. Trainees indicated that the face-validity of the test improved with internet access and that they would like to continue to have internet access during future testing. Internet access did not improve the construct-validity or reliability of the test.

Conclusion

Improving the face-validity of postgraduate progress testing, by adding the possibility to search the internet for a limited amount of time, positively influences test performance and face-validity. However, it did not change the reliability or the construct-validity of the test.

【 授权许可】

   
2013 Dijksterhuis et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

【 参考文献 】

• [1]Masters K: For what purpose and reasons do doctors use the internet: a systematic review. Int J Med Inform 2008, 77(1):4-16.
• [2]Podichetty VK, Booher J, Whitfield M, Biscup RS: Assessment of internet use and effects among healthcare professionals: a cross sectional survey. Postgrad Med J 2006, 82(966):274-279.
• [3]Shanahan M, Herrington A, Herrington J: Workplace culture and accessibility of the Internet for professional learning. In World Conference on Educational Multimedia, Hypermedia and Telecommunications 2009. Edited by Siemens G, Fulford C. Honolulu, HI, USA: AACE; 2009:4423-4432.
• [4]Heijne-penninga M, Kuks JBM, Schönrock-Adema J, Snijders TAB, Cohen-Schoatnus J: Open-book tests to complement assessment-programmes: analysis of open and closed-book tests. Adv Health Sci Educ 2008, 13(3):263-273.
• [5]Dijksterhuis MGK, Schuwirth LWT, Braat DDM, Scheele F: An exploratory study into the impact and acceptability of formatively used progress testing in postgraduate obstetrics and gynaecology. Pers Med Educ 2013, 1-16. Epub
• [6]Dijksterhuis MGK, Scheele F, Schuwirth LWT, Essed GGM, Nijhuis JG, Braat DDM: Progress testing in postgraduate medical education. Med Teach 2009, 31(10):e464-e468.
• [7]Van Der Vleuten CPM, Schuwirth LWT: Assessing professional competence: from methods to programmes. Med Educ 2005, 39(3):309-317.
• [8]Vleuten CPM: The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1996, 1(1):41-67.
• [9]Norman G, Vleuten C, Graaff E: Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Med Educ 1991, 25(2):119-126.
• [10]Bennett NL, Casebeer LL, Zheng S, Kristofco R: Information-seeking behaviors and reflective practice. J Contin Educ Health Prof 2006, 26(2):120-127.
• [11]McKibbon KA, Fridsma DB: Effectiveness of clinician-selected electronic information resources for answering primary care Physicians’ information needs. J Am Med Inform Assoc 2006, 13(6):653-659.
• [12]Eilertsen TV, Valdermo O: Open-book assessment: a contribution to improved learning? Studies Educ Eval 2000, 26(2):91-103.
• [13]Feller M: Open-book testing and education for the future. Studies Educ Eval 1994, 20(2):235-238.
• [14]Koutselini Ioannidou M: Testing and life-long learning: open-book and closed-book examination in a university course. Studies Educ Eval 1997, 23(2):131-139.
• [15]Van der Vleuten CP, Norman GR, De Graaff E: Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ 1991, 25(2):110-118.
• [16]Ravesloot C, van der Schaaf M, Haaring C, Kruitwagen C, Beek E, Ten Cate O, van Schaik J: Construct validation of progress testing to measure knowledge and visual skills in radiology. Med Teach 2012, 34(12):1047-1055.
• [17]Broyles IL, Cyr PR, Korsen N: Open book tests: assessment of academic learning in clerkships. Med Teach 2005, 27(5):456-462.
• [18]Agarwal PK, Karpicke JD, Kang SH, Roediger HL III, McDermott KB: Examining the testing effect with open and closed book tests. Appl Cogn Psychol 2007, 22:861-876.
• [19]Brightwell R, Daniel J, Stewart A: Evaluation: is an open book examination easier. Biosci Educ e-J 2004., 3(3)
• [20]Schuwirth LW, van der Vleuten CP: General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach 2011, 33(10):783-797.
• [21]Nevo B, Sfez J: Examinees’ feedback questionnaires. Ass Eval Higher Educ 1985, 10(3):236-248.
• [22]Downing SM: Face-validity of assessments: faith-based interpretations or evidence-based science. Med Educ 2006, 40(1):7-8.
• [23]La Rochelle JS, Durning SJ, Pangaro LN, Artino AR, van der Vleuten CP, Schuwirth L: Authenticity of instruction and student performance: a prospective randomised trial. Med Educ 2011, 45(8):807-817.
• [24]Boniface D: Candidates’ use of notes and textbooks during an open-book examination. Educ Res 1985, 27(3):201-209.
• [25]Hughes B, Wareham J, Joshi I: Doctors’ online information needs, cognitive search strategies, and judgments of information quality and cognitive authority: How predictive judgments introduce bias into cognitive search models. J Am Soc Inf Sci Technol 2009, 61(3):433-452.
• [26]Metzger MJ: Making sense of credibility on the Web: models for evaluating online information and recommendations for future research. J Am Soc Inf Sci Technol 2007, 58(13):2078-2091.