Evaluating health worker performance in Benin using the simulated client method with real children

Implementation Science

Michael S Deming¹  Marcel Lama²  Faustin Onikpo³  Alexander K Rowe⁴ 

[1] Parasitic Diseases Branch, Division of Parasitic Diseases and Malaria, Center for Global Health, Centers for Disease Control and Prevention, Mailstop A06, 1600 Clifton Road NE, Atlanta, GA, 30333, USA;Africare-Benin, Porto Novo, Benin;Direction Départementale de la Santé Publique de l′Ouémé et Plateau, Ministry of Public Health, Porto Novo, B.P. 139, Benin;Malaria Branch, Division of Parasitic Diseases and Malaria, Center for Global Health, Centers for Disease Control and Prevention, Mailstop A06, 1600 Clifton Road NE, Atlanta, GA, 30333, USA

关键词: Simulated clients;    Integrated management of childhood illness;    Health services research;    Methods;    Developing country;    Child health;    Benin;   

Others  :  813898
DOI  :  10.1186/1748-5908-7-95

 received in 2012-03-29, accepted in 2012-09-27,  发布年份 2012

【 摘 要 】

Background

The simulated client (SC) method for evaluating health worker performance utilizes surveyors who pose as patients to make surreptitious observations during consultations. Compared to conspicuous observation (CO) by surveyors, which is commonly done in developing countries, SC data better reflect usual health worker practices. This information is important because CO can cause performance to be better than usual. Despite this advantage of SCs, the method’s full potential has not been realized for evaluating performance for pediatric illnesses because real children have not been utilized as SCs. Previous SC studies used scenarios of ill children that were not actually brought to health workers. During a trial that evaluated a quality improvement intervention in Benin (the Integrated Management of Childhood Illness [IMCI] strategy), we conducted an SC survey with adult caretakers as surveyors and real children to evaluate the feasibility of this approach and used the results to assess the validity of CO.

Methods

We conducted an SC survey and a CO survey (one right after the other) of health workers in the same 55 health facilities. A detailed description of the SC survey process was produced. Results of the two surveys were compared for 27 performance indicators using logistic regression modeling.

Results

SC and CO surveyors observed 54 and 185 consultations, respectively. No serious problems occurred during the SC survey. Performance levels measured by CO were moderately higher than those measured by SCs (median CO – SC difference = 16.4 percentage-points). Survey differences were sometimes much greater for IMCI-trained health workers (median difference = 29.7 percentage-points) than for workers without IMCI training (median difference = 3.1 percentage-points).

Conclusion

SC surveys can be done safely with real children if appropriate precautions are taken. CO can introduce moderately large positive biases, and these biases might be greater for health workers exposed to quality improvement interventions.

Trial number

http://clinicaltrials.gov webcite Identifier NCT00510679

【 授权许可】

   
2012 Rowe et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

【 参考文献 】

• [1]Madden JM, Quick JD, Ross-Degnan D, Kafle KK: Undercover careseekers: simulated clients in the study of health provider behavior in developing countries. Soc Sci Med 1997, 45:1465-1482.
• [2]Watson MC, Norris P, Granas AG: A systematic review of the use of simulated patients and pharmacy practice research. Int J Pharm Practice 2006, 14:83-93.
• [3]Franco LM, Daly CC, Chilongozi D, Dallabetta G: Quality of case management of sexually transmitted diseases: comparison of the methods for assessing the performance of providers. Bull World Health Organ 1997, 75:523-532.
• [4]Leonard K, Masatu MC: Outpatient process quality evaluation and the Hawthorne effect. Soc Sci Med 2006, 63:2330-2340.
• [5]Roethlisberger FJ, Dickson WJ: Management and the worker: an account of a research program conducted by Western Electric Company, Hawthorne Works, Chicago. Cambridge: Harvard University Press; 1939.
• [6]Rowe AK, Lama M, Onikpo F, Deming MS: Health worker perceptions of how being observed influences their practices during consultations with ill children. Trop Doc 2002, 32:166-167.
• [7]Rowe SY, Olewe MA, Kleinbaum DG, McGowan JE, McFarland DA, Rochat R, et al.: The influence of observation and setting on community health workers’ practices. Int J Qual Health Care 2006, 18:299-305.
• [8]Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M: Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA 2000, 283:1715-1722.
• [9]Gove S for the WHO Working Group on Guidelines for Integrated Management of the Sick Child: Integrated management of childhood illness by outpatient health workers: technical basis and overview. Bull World Health Organ 1997, 75(Suppl 1):7-24.
• [10]Rowe AK, Onikpo F, Lama M, Osterholt DM, Rowe SY, Deming MS: A multi-faceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin. Am J Public Health 2009, 99:837-846.
• [11]Benin Ministry of Public Health: Prise en charge intégrée des maladies de l’enfant. Livret des tableaux. Porto-Novo, Benin; 2002.
• [12]Rowe AK, Onikpo F, Lama M, Cokou F, Deming MS: Management of childhood illness at health facilities in Benin: problems and their causes. Am J Public Health 2001, 91:1625-1635.
• [13]SAS Institute: SAS/STAT(R) 9.2 User’s Guide, Second Edition. 2012. Available on-line at http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_surveylogistic_a0000000364.htm webcite, accessed on May 21. 2012
• [14]Glassman PA, Luck J, O’Gara EM, Peabody JW: Using standardized patients to measure quality: evidence from the literature and a prospective study. Jt Comm J Qual Improv 2000, 26:644-653.
• [15]Ali J, Cherry RA, Williams JI: Trauma base knowledge and the effect of the trauma evaluation and management program among senior medical students in seven countries. J Trauma 2005, 59:43-48.
• [16]Brooks AJ, Phipson M, Potgieter A, Koertzen H, Boffard KD: Education of the trauma team: video evaluation of the compliance with universal barrier precautions in resuscitation. Eur J Surg 1999, 165:1125-1128.
• [17]Scherer LA, Chang MC, Meredith JW, Battistella FD: Videotape review leads to rapid and sustained learning. Am J Surg 2003, 185:516-520.
• [18]Birnbach DJ, Santos AC, Bourlier RA, Meadows WE, Datta S, Stein DJ, et al.: The effectiveness of video technology as an adjunct to teach and evaluate epidural anesthesia performance skills. Anesthesiology 2002, 96:5-9.
• [19]Weston CFM, Richmond P, McCabe MJ, Evans R, Evans R: Video recording of cardiac arrest management: an aid to training and audit. Resuscitation 1992, 24:13-15.
• [20]McCormick DP, Rassin GM, Stroup-Benham CA, Baldwin CD, Levine HG, Persaud DI, et al.: Use of videotaping to evaluate pediatric resident performance of health supervision examinations of infants. Pediatrics 1993, 92:116-120.
• [21]Arborelius E, Bremberg S: Prevention in practice. How do general practitioners discuss life-style issues with their patients? Patient Educ Couns 1994, 23:23-31.
• [22]Fossum B, Arborelius E: Patient-centred communication: videotaped consultations. Patient Educ Couns 2004, 54:163-169.
• [23]Harting J, van Assema P, van der Molen HT, Ambergen T, de Vries NK: Quality assessment of health counseling: performance of health advisors in cardiovascular prevention. Patient Educ Couns 2004, 54:107-118.
• [24]Holmström IM, Rosenqvist U: Misunderstandings about illness and treatment among patients with type 2 diabetes. J Adv Nurs 2005, 49:146-154.
• [25]Adomat R, Hicks C: Measuring nursing workload in intensive care: an observational study using closed circuit video cameras. J Adv Nurs 2003, 42:402-412.
• [26]Arborelius E, Timpka T: In what way may videotapes be used to get significant information about the patient–physician relationship? Med Teacher 1990, 12:197-208.
• [27]Rogers SC, Dudley NC, McDonnell W, Scaife E, Morris S, Nelson D: Lights, camera, action… spotlight on trauma video review: an underutilized means of quality improvement and education. Pediatr Emer Care 2010, 26:803-807.
• [28]Abdulhadi N, Al-Shafaee MA, Östenson C-G, Vernby Å, Wahlström R: Quality of interaction between primary health-care providers and patients with type 2 diabetes in Muscat. Oman: an observational study. BMC Family Practice 2006, 7:72.
• [29]Centers for Disease Control and Prevention: Health worker performance after training in integrated management of childhood illness — Western Province, Kenya, 1996–1997. Morb Mort Weekly Report 1998, 47:998-1001.