Design of an impact evaluation using a mixed methods model – an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi

BMC Health Services Research

Manuela De Allegri⁵  Thomas Bossert³  Don P Mathanga¹  Malabika Sarker⁵  Till Bärnighausen⁴  Paul Jacob Robyn²  Adamson S Muula¹  Stephan Brenner⁵ 

[1] Department of Community Health, University of Malawi, College of Medicine, Blantyre, Malawi;The World Bank, Washington, DC, USA;Department of Global Health and Population, Harvard School of Public Health, Boston, Massachusetts, United States of America;Wellcome Trust Africa Centre for Health and Population Studies, University of KwaZulu-Natal, Mtubatuba, South Africa;Institute of Public Health, Ruprecht-Karls-University, Heidelberg, Germany

关键词: Study design;    Performance-based incentives;    Impact evaluation;    Mixed methods;   

Others  :  1132684
DOI  :  10.1186/1472-6963-14-180

 received in 2014-02-07, accepted in 2014-04-07,  发布年份 2014

【 摘 要 】

Background

In this article we present a study design to evaluate the causal impact of providing supply-side performance-based financing incentives in combination with a demand-side cash transfer component on equitable access to and quality of maternal and neonatal healthcare services. This intervention is introduced to selected emergency obstetric care facilities and catchment area populations in four districts in Malawi. We here describe and discuss our study protocol with regard to the research aims, the local implementation context, and our rationale for selecting a mixed methods explanatory design with a quasi-experimental quantitative component.

Design

The quantitative research component consists of a controlled pre- and post-test design with multiple post-test measurements. This allows us to quantitatively measure ‘equitable access to healthcare services’ at the community level and ‘healthcare quality’ at the health facility level. Guided by a theoretical framework of causal relationships, we determined a number of input, process, and output indicators to evaluate both intended and unintended effects of the intervention. Overall causal impact estimates will result from a difference-in-difference analysis comparing selected indicators across intervention and control facilities/catchment populations over time.

To further explain heterogeneity of quantitatively observed effects and to understand the experiential dimensions of financial incentives on clients and providers, we designed a qualitative component in line with the overall explanatory mixed methods approach. This component consists of in-depth interviews and focus group discussions with providers, service user, non-users, and policy stakeholders. In this explanatory design comprehensive understanding of expected and unexpected effects of the intervention on both access and quality will emerge through careful triangulation at two levels: across multiple quantitative elements and across quantitative and qualitative elements.

Discussion

Combining a traditional quasi-experimental controlled pre- and post-test design with an explanatory mixed methods model permits an additional assessment of organizational and behavioral changes affecting complex processes. Through this impact evaluation approach, our design will not only create robust evidence measures for the outcome of interest, but also generate insights on how and why the investigated interventions produce certain intended and unintended effects and allows for a more in-depth evaluation approach.

【 授权许可】

   
2014 Brenner et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Kutzin J: Towards universal health care coverage: a goal-oriented framework for policy analysis. In Health, nutrition and population discussion paper. Edited by Preker AS. Washington D.C: The World Bank; 2000.
• [2]Gottret PE, Schieber G: Health financing revisited: a practitioner’s guide. Washington, D.C: The World Bank; 2006.
• [3]World Health Assembly Resolution 58.33: Sustainable health financing, universal coverage and social health insurance. Geneva: World Health Organization; 2005.
• [4]Laxminarayan R, Mills AJ, Breman JG, Measham AR, Alleyne G, Claeson M, Jha P, Musgrove P, Chow J, Shahid-Salles S, Jamison DT: Advancement of global health: key messages from the disease control priorities project. Lancet 2006, 367:1193-1208.
• [5]World Health Organization: The world health report 2010 - health systems financing: the path to universal coverage. Geneva: World Health Organization; 2010.
• [6]Chaudhury N, Hammer J, Kremer M, Muralidharan K, Rogers FH: Missing in action: teacher and health worker absence in developing countries. J Econ Perspect 2006, 20:91-116.
• [7]Hutchinson PL, Do M, Agha S: Measuring client satisfaction and the quality of family planning services: a comparative analysis of public and private health facilities in Tanzania, Kenya and Ghana. BMC Health Serv Res 2011, 11:203-219. BioMed Central Full Text
• [8]Gauthier B: PETS-QSDS in sub-Saharan Africa: a stocktaking study. In Report for the project measuring progress in public services delivery: 7 September 2006. Washington D.C: The World Bank; 2006.
• [9]Gupta S, Verhoeven M, Tiongson ER: Public spending on health care and the poor. Health Econ 2003, 12:685-696.
• [10]Musgrove P: Financial and other rewards for good performance or results: a guided tour of concepts and terms and a short glossary. In The World Bank health results innovation trust fund - results-based financing for health: 14 September 2010. Washington D.C: The World Bank; 2011.
• [11]Toonen J, van der Wal B: Results-based financing in healthcare: developing an RBF approach for healthcare in different contexts: the case of Mali and Ghana. Amsterdam: KIT Publishers; 2012.
• [12]Ergo A, Paina L, Morgan L, Eichler R: Creating stronger incentives for high-quality health care in low- and middle-income countries. In USAID maternal and child health integrated program. Washington D.C: USAID; 2012.
• [13]Eichler R, Levine R, Performance-Based Incentives Working Group: Performance incentives for global health - potential and pitfalls. Washington D.C: Center for Global Development; 2009.
• [14]Ireland M, Paul E, Dujardin B: Can performance-based financing be used to reform health systems in developing countries. Bull World Health Organ 2011, 89:695-698.
• [15]Meessen B, Soucat A, Sekabaraga C: Performance-based financing: just a donor fad or a catalyst towards comprehensive health-care reform? Bull World Health Organ 2011, 89:153-156.
• [16]Ranganathan M, Legarde M: Promoting healthy behaviours and improving health outcomes in low and middle income coutnries: a review of the impact of conditional cash transfer programmes. Prev Med 2012, 55:S95-S105.
• [17]Giuffrida A, Gravelle H, Roland M: Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. BMJ 1999, 319:94-98.
• [18]Basinga P, Gertler PJ, Binagwaho A, Soucat AL, Sturdy J, Vermeersch CM: Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet 2011, 377:1421-1428.
• [19]Rusa L, Ngirabega J de D, Janssen W, Van Bastelaere S, Porignon D, Vandenbulcke W: Performance-based financing for better quality of services in Rwandan health centres: 3-year experience. Trop Med Int Health 2009, 14:830-837.
• [20]Soeters R, Peerenboom PB, Mushagalusa P, Kimanuka C: Performance-based financing experiment improved health care in the democratic republic of Congo. Health Aff 2011, 30:1518-1527.
• [21]Gorter A, Ir P, Meessen B: Evidence review: results-based financing of maternal and newborn health care in low- and middle-income countries. In Report for the project Programme to Foster Innovation, Learning and Evidence in Health Programmes of the German Development Cooperation: study commissioned and funded by the German federal ministry for economic cooperation and development (BMZ) through the sector project PROFILE at GIZ - Deutsche Gesellschaft für Internationale Zusammenarbeit. Eschborn: German Health Practice Collection; 2013.
• [22]Lundberg M: Client satisfaction and perceived quality of primary health care in Uganda. In Are you being served? new tools for measuring service delivery. Edited by Amin S, Dasb J, Goldstein M. Washington D.C: The World Bank; 2008:313-341.
• [23]Kalk A, Paul FA, Grabosch E: ‘Paying for performance’ in Rwanda: does it pay off? Trop Med Int Health 2010, 15:182-190.
• [24]Songstad NG, Lindkvist I, Moland KM, Chimhutu V, Blystad A: Assessing performance enhancing tools: experiences with the open performance review and appraisal system (OPRAS) and expectations towards payment for performance (P4P) in the public health sector in Tanzania. Global Health 2012, 8:33-45. BioMed Central Full Text
• [25]Leonard KL, Masatu MC: Professionalism and the know-do gap: exploring intrinsic motivation among health workers in Tanzania. Health Econ 2010, 19:1461-1477.
• [26]Witter S, Fretheim A, Kessy FL, Lindahl AK: Paying for performance to improve the delivery of health interventions in low- and middle-income countries. Cochrane Database Syst Rev 2012, 2:1-81.
• [27]Fretheim A, Witter S, Lindahl AK, Olsen IT: Performance-based financing in low- and middle-income countries: still more questions than answers. Bull World Health Organ 2012, 90:559-559A.
• [28]National Statistical Office and ICF Macro: Malawi demographic and health survey 2010. Zomba: National Statistical Office; 2011.
• [29]Kongnyuy EJ, Mlava G, van den Broek N: Facility-based maternal death review in three districts in the central region of Malawi: an analysis of causes and characteristics of maternal deaths. Womens Health Issues 2009, 19:14-20.
• [30]Carlson C, Boivin M, Chirwa A, Chirwa S, Chitalu F, Hoare G, Huelsmann M, Ilunga W, Maleta K, Marsden A, Martineau T, Minett C, Mlambala A, von Massow F, Njie H, Olson IT: Malawi health SWAp mid-term review: summary report. In Norad collected reviews, vol 22. Malawi: Norad: Commissioned by the Ministry of Health; 2008.
• [31]Bowie C, Mwase T: Assessing the use of an essential health package in a sector wide approach in Malawi. Health Research Policy and Systems 2011, 9:4-13. BioMed Central Full Text
• [32]RBF4MNH Options Office: Inception report results based financing for maternal and neonatal health (RBF4MNH initiative) 2012–2014. Lilongwe: Reproductive Health Unit; 2012.
• [33]World Health Organization: Monitoring emergency obstetric care: a handbook. Geneva: World Health Organization; 2009.
• [34]Ronsmans C, Graham WJ: Maternal mortality: who, when, where, and why. Lancet 2006, 368:1189-1200.
• [35]Filippi V, Ronsmans C, Campbell OM, Graham WJ, Mills A, Borghi J, Koblinsky M, Osrin D: Maternal health in poor countries: the broader context and a call for action. Lancet 2006, 368:1535-1541.
• [36]World Health Organization: Everybody’s business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: World Health Organization; 2007.
• [37]Donabedian A: The quality of care: how can it be assessed? JAMA 1988, 260:1743-1748.
• [38]Conrad P, De Allegri M, Moses A, Larsson EC, Neuhann F, Müller O, Sarker M: Antenatal care services in rural Uganda missed opportunities for good-quality care. Qual Health Res 2012, 22:619-629.
• [39]Andersen RM: Revisiting the behavioral model and access to medical care: does it matter? J Health Soc Behav 1995, 36:1-10.
• [40]Aday LA, Andersen RM: Equity of access to medical care: a conceptual and empirical overview. Med Decis Making 1981, 19:4-27.
• [41]Creswell JW, Clark PVL: Designing and conducting mixed methods research. 2nd edition. Thousand Oaks: SAGE Publications; 2010.
• [42]Shadish WR, Cook TD, Campbell DT: Experimental and quasi-experimental designs for generalized causal inference. Belmont: Wadsworth Cengage Learning; 2001.
• [43]Glaser BG, Strauss AL: The discovery of grounded theory: strategies for qualitative research. Hawthorne: Aldine de Gruyter; 1999.
• [44]Peabody JW: Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA 2000, 283:1715-1722.
• [45]Sarker M, Schmid G, Larsson E, Kirenga S, De Allegri M, Neuhann F, Mbunda T, Lekule I, Müller O: Quality of antenatal care in rural southern Tanzania: a reality check. BMC Res Notes 2010, 2010(3):209-315.
• [46]Gearing RE, Mian IA, Barber J, Ickowicz A: A methodology for conducting retrospective chart review research in child and adolescent psychiatry. J Can Acad Child Adolesc Psychiatry 2006, 15:126-134.
• [47]O’Donnell O, Van Doorslaer E, Wagstaff A, Lindelow M: Analyzing health equity using household survey data: a guide to techniques and their implementation. In WBI learning resource series. Washington, D.C: The World Bank; 2008.
• [48]Donald SG, Lang K: Inference with difference-in-differences and other panel data. Rev Econ Stat 2007, 89:221-233.
• [49]Athey S, Imbens GW: Identification and inference in nonlinear difference-in-difference models. Econometrica 2006, 74:431-497.
• [50]Victora CG, Black RE, Boerma JT, Bryce J: Measuring impact in the millennium development goal era and beyond: a new approach to large-scale effectiveness evaluations. Lancet 2011, 377:85-95.
• [51]Shadish WR, Cook TD: The renaissance of field experimentation in evaluating interventions. Annu Rev Psychol 2009, 60:607-629.
• [52]Winship C, Morgan SL: The estimation of causal effects from observational data. Annu Rev Sociol 1999, 25:659-707.
• [53]Patton MQ: Qualitative research & evaluation methods. Thousand Oaks: SAGE Publications; 2002.
• [54]Bowen GA: Naturalistic inquiry and the saturation concept: a research note. Qual Res 2008, 8:137-152.
• [55]Tuckett AG: Qualitative research sampling: the very real complexities. Nurse Researcher 2004, 12:47-61.
• [56]Harachi TW, Abbott RD, Catalano RF, Haggerty KP, Fleming CB: Opening the black box: using process evaluation measures to assess implementation and theory building. Am J Community Psychol 1999, 27:711-731.
• [57]Oakley A, Strange V, Bonell C, Allen E, Stephenson J: Process evaluation in randomised controlled trials of complex interventions. BMJ 2006, 332:413-416.
• [58]QSR International Pty Ltd: NVivo qualitative data analysis software. 2012. Version 10
• [59]Bamberger M: The evaluation of international development programs: a view from the front. Am J Eval 2000, 21:95-102.
• [60]Beane CR, Hobbs SH, Thirumurthy H: Exploring the potential for using results-based financing to address non-communicable diseases in low- and middle-income countries. BMC Public Health 2013, 13:92-100. BioMed Central Full Text
• [61]Hurley R: Funding aid according to outcomes can improve health in poor countries, seminar hears. BMJ 2011, 342:d2322.
• [62]Bamberger M: Introduction to mixed methods in impact evaluation. In Impact evaluation guidance notes. Washington D.C: InterAction; 2012.
• [63]White H: Theory-based impact evaluation: principles and practice. In 3ie working papers. New Delhi: International Initiative for Impact Evaluation (3ie); 2009.
• [64]Chen H-T, Rossi PH: The theory-driven approach to validity. Evaluation and Program Planning 1987, 10:95-103.
• [65]World Health Organization: Country cooperation strategy brief Malawi. Geneva: World Health Organization; 2013.
• [66]Prytherch H, Kagone M, Aninanya GA, Williams JE, Kakoko DC, Leshabari MT, Ye M, Marx M, Sauerborn R: Motivation and incentives of rural maternal and neonatal health care providers: a comparison of qualitative findings from Burkina Faso, Ghana and Tanzania. BMC Health Serv Res 2013, 13:149-163. BioMed Central Full Text
• [67]Ssengooba F, McPake B, Palmer N: Why performance-based contracting failed in Uganda: an ‘open-box’ evaluation of a complex health system intervention. Soc Sci Med 2012, 75:377-383.
• [68]Observatory AHW: Human resources for health country profile - Malawi. Brazzaville: Republic of Congo; 2009.
• [69]Victora CG, Wagstaff A, Schellenberg JA, Gwatkin D, Claeson M, Habicht JP: Applying an equity lens to child health and mortality: more of the same is not enough. Lancet 2003, 362:233-241.
• [70]Population Unit, Ministry of Development Planning and Cooperation: RAPID: population and development in Malawi. Lilongwe: Ministry of Development Planning and Cooperation; 2010.
• [71]White H: A contribution to current debates in impact. Evaluation 2010, 16:153-164.
• [72]Morgan SL: Counterfactuals and causal inference: methods and principles for social research. New York: Cambridge University Press; 2007.
• [73]Puffer S, Torgerson D, Watson J: Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ 2003, 327:785-789.