Health systems performance assessment in low-income countries: learning from international experiences

Globalization and Health

Bart Criel³  Jean Macq⁵  Juliet Nabyonga-Orem¹  Freddie Ssengooba²  Valéria Campos da Silveira³  Christine Kirunga Tashobya⁴ 

[1] WHO Regional Uganda for Africa, P.O Box 6, Brazaville, Congo;Makerere School of Public Health, Mulago Hill, P.O Box 7062, Kampala, Uganda;Institute of Tropical Medicine Antwerp, Nationalestraat 155, Antwerp, B 2000, Belgium;Ministry of Health, 6 Lourdel Road Nakasero, P.O Box 7272, Kampala, Uganda;Institute of Research Health and Society, Catholic University of Louvain, Promenade de l’Alma, 31 bte B1.41.03, Brussels B-12000, Belgium

关键词: Uganda;    Low-income countries;    Performance assessment framework;    Health systems;   

Others  :  805058
DOI  :  10.1186/1744-8603-10-5

 received in 2013-08-22, accepted in 2014-02-03,  发布年份 2014

【 摘 要 】

Background

The study aimed at developing a set of attributes for a ‘good’ health system performance assessment (HSPA) framework from literature and experiences in different contexts and using the attributes for a structured approach to lesson learning for low-income countries (LICs).

Methods

Literature review to identify relevant attributes for a HSPA framework; attribute validation for LICs in general, and for Uganda in particular, via a high-level Ugandan expert group; and, finally, review of a selection of existing HSPA frameworks using these attributes.

Results

Literature review yielded six key attributes for a HSPA framework: an inclusive development process; its embedding in the health system’s conceptual model; its relation to the prevailing policy and organizational set-up and societal context; the presence of a concrete purpose, constitutive dimensions and indicators; an adequate institutional set-up; and, its capacity to provide mechanisms for eliciting change in the health system. The expert group contextualized these attributes and added one on the adaptability of the framework.

Lessons learnt from the review of a selection of HSPA frameworks using the attributes include: it is possible and beneficial to involve a range of stakeholders during the process of development of a framework; it is important to make HSPA frameworks explicit; policy context can be effectively reflected in the framework; there are marked differences between the structure and content of frameworks in high-income countries, and low- and middle-income countries; champions can contribute to put HSPA high on the agenda; and mechanisms for eliciting change in the health system should be developed alongside the framework.

Conclusion

It is possible for LICs to learn from literature and the experience of HSPA in other contexts, including HICs. In this study a structured approach to lesson learning included the development of a list of attributes for a ‘good’ HSPA framework. The attributes thus derived can be utilized by LICs like Uganda seeking to develop/adjust their HSPA frameworks as guidelines or a check list, while taking due consideration of the specific context. The review of frameworks from varied contexts, highlighted varied experiences which provide lessons for LICs.

【 授权许可】

   
2014 Tashobya et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Arah OA, Westert GP: Correlates of health and health care performance: applying the Canadian health indicators framework at the provincial-territorial level. BMC Health Services Research 2005, 5:76. BioMed Central Full Text
• [2]Loeb MJ: The current state of performance measurement in health care. International Journal for Quality in Health Care 2004, 16(Supplement 1):i5-i9.
• [3]Thier SO, Geljins CA: Perspective: Improving health: the reason performance measurement matters. Health Affairs 1998, 17(4):26-28.
• [4]Arah OA, Westert GP, Hurst J, Klazinga NS: A conceptual framework for the OECD health care quality improvement project. International Journal for Quality in Health Care 2006, 5-13.
• [5]McLoughlin V, Leatherman S, Fletcher M, Owen JW: Improving performance using indicators. Recent experiences in the United States, the United Kingdom and Australia. International Journal for Quality in Health Care 2001, 13(6):455-462.
• [6]Veillard J, Champagne F, Klazinga NS, Kazandjian V, Arah OA, Guisset AL: A performance assessment framework for hospitals: the WHO regional office for Europe PATH project. International Journal for Quality in Health Care 2005, 17(6):487-496.
• [7]Kalinichenko O, Amado CAF, Santos PS: Performance Assessment in Primary Health Care: A systematic review. 2013. Evora Portugal CEFAGE-UE Working Paper 2013/03 CEFAGE- UE Universidale Evora
• [8]Smith PC, Mossialos E, Papanicolas I, Leatherman S: Introduction. In Performance measurement for health system improvement: experiences, challenges and prospects. Edited by Smith PC, Mossialos E, Papanicolas I, Leatherman S. New York: Cambridge University Press; 2009:3-24.
• [9]Barron P, Day C, Monticelli F, Vermaak K, Okorafor O, Moodley K, Doherty T: The District Health Barometer 2005/06. Durban South Africa Health Systems Trust; 2006.
• [10]Ministry of Health: The Health Sector Strategic and Investment Plan 2010/11 to 2014/15. Kampala Uganda: Ministry of Health; 2010.
• [11]Yates R, Tashobya CK, Cruz OV, McPake B, Ssengooba F, Murindwa G, Lochoro P, Bataringaya J, Nazerali H, Omaswa F: The Uganda health system reforms: miracle or mirage? In Health system reforms in Uganda: processes and outputs. Edited by Tashobya CK, Ssengooba F, Cruz OV. London, United Kingdom: London School of Hygiene and Tropical Medicine; 2006.
• [12]World Health Organisation: Investing in Health for Economic Development. Geneva, Switzerland: Commission on Macroeconomics and Health, World Health Organisation; 2001.
• [13]Tashobya CK, Ogwal PO: The effort to achieve the millennium development goals in Uganda: reaching for the sky? Health Policy and Development 2004, 2(1):33-39.
• [14]World Health Organisation: Health System Performance Assessment: Debates, Methods, Empricism. Edited by Murray JLC, Evans DB. Geneva Switzerland: World Health Organisation; 2003.
• [15]Arah OA, Klazinga NS, Delnoij JMD, Ten Asbroek AHA, Clusters T: Conceptual framework for health system performance: a quest for effectiveness, quality and improvement. International Journal for Quality in Health Care 2003, 15(5):377-398.
• [16]Edward A, Kumar B, Kakah F, Salehi AS, Burnham G, Peters DH: Configuring Balanced Scorecards for measuring health system performance: evidence from 5 years evaluation in Afghanistan. PLoS Medicine 2011, 8(7):e1001066.
• [17]Tawfik-Shukor AR, Klazinga NS, Arah OA: Comparing health system performance and management approaches in the Netherlands and Ontario, Canada. BMC Health Services Research 2007., 7(25) doi:10.1186/1472-6963-1187-1125
• [18]Siddiqqi S, Masud IT, Nishtar S, Peters DH, Sabri B, Bile MK, Jama AM: Framework for assessing governance of the health system in developing countries: gateway to good governance. Health Policy 2009, 90:13-25.
• [19]Ministry of Health: Annual Health Sector Performance Report 2002/03. Kampala Uganda Ministry of Health Uganda; 2003.
• [20]Komakech I: The curse of league tables: how Uganda missed the point. Health Policy and Development 2005, 3(2):165-169.
• [21]Ministry of Health: Mid-term Review of the Health Sector Strategic Plan II. Kampala Uganda Ministry of Health Uganda; 2008.
• [22]Lalonde M: A new perspective on the health of Canadians: a working document. Ministry of Supply and Services Government of Canada: Ottawa Canada; 1981.
• [23]Murray JLC, Frenk J: A framework for assessing the performance of health systems. Bulletin of the World Health Organisation 2000, 78(6):717-731.
• [24]World Health Organisation: The World Health Report 2000: Improving Performance. Geneva Switzerland World Health Organisation; 2000.
• [25]World Health Organisation: Everybody's Business. Strengthening Health Systems to Improve Health Outcomes. Geneva Switzerland: World Health Organisation; 2007.
• [26]Van Olmen J, Criel B, Bhojani U, Marchal B, van Belle S, Chenge MF, Hoeree T, Pirard M, Van Damme W, Kegels G: The health systems dynamics framework. The intorduction of an analytical model for health system analysis and its application to two cases. Health Culture and Society 2012., 2(1) doi:10.5195/hcs.2012.5171
• [27]Atun R: Health systems, systems thinking and innovation. Health Policy and Planning 2012, 27:iv4-iv8.
• [28]McDaniel RR, Lanham HJ, Anderson RA: Implications of complex adaptive systems theory for the design of research on health care organisation. Health Care Management Review 2009, 34(2):191-199.
• [29]Sterman JD: Learning from evidence in a complex world. American Journal of Public Health 2006, 96(3):505-514.
• [30]Rouse WB: Health care as a complex adaptive system: implications for design and management. The Bridge Spring 2008., (2008)
• [31]Palomba CA, Banta TW: Assessment essentials: planning, implementing and improving assessment in higher education. San Fransisco CA: Jossey-Bass Publishers; 1999.
• [32]Sicotte C, Champagne F, Contadriopoloulos AP, Barnsley J, Beland F, Leggat SG, Denis JL, Bilodeau H, Langley A, Bremond M, et al.: A conceptual framework for the analysis of health care organsiations' performance. Health Services Management Research 1998, 11:24-28.
• [33]Parsons T: Social Systems and the Evolution of Action Theory. New York USA Free Press; 1977.
• [34]Quinn RE, Rohrbaugh J: A spatial model of effectiveness criteria: towards a competing values approach to organisational analysis. Management Science 1983, 29:363-377.
• [35]Oxford Dictionarieshttp://www.oxforddictionaries.com/definition/english/framework webcite. accessed February 10, 2014
• [36]Groene O, Klazinga NS, Kazandjian V, Lombrail P, Bartels P: The World Health Organisation Performance Assessment Tool for Quality Improvement in Hospitals (PATH): an analysis of the pilot implementation in 37 hospitals. International Journal for Quality in Health Care 2008, 20(3):155-161.
• [37]Chiu W, Yang C, Lin H, Chu T: Development and implementation of a nationwide healthcare quality indicator system in Taiwan. International Journal for Quality in Health Care 2007, 19(1):21-28.
• [38]El-Jardali F, Salehi AS, Ataya N, Jamal D: Design, implementation and scaling up of the balanced score card for Hospitals in Lebanon: policy coherence, and application of lessons for low and middle income countries. Health Policy 2011, 103:305-314.
• [39]Leggat SG, Narine L, Lemieux-Charles L, Barnsley J, Baker GR, Sicotte C, Champagne F, Bilodeau H: A review of organisational performance assessment in health care. Health Services Management Research 1998, 11:3-13.
• [40]Minvielle E, Sicotte C, Champagne F, Contadriopoloulos AP, Jeanet M, Preaubert N, Bourdil A, Richard C: Hospital performance: competing or shared values. Health Policy 2007., (2007) doi:10.1016/j.healthpol.2007.09.017
• [41]Weir E, d'Entremont N, Stalker S, Kurji K, Robinson V: Applying the balanced score card to local public health measurement: deliberation and decisions. BMC Public Health 2009, 9:127. doi:10.1186/1471-2458-9-127 BioMed Central Full Text
• [42]Braveman P, Starfield B, Geiger JH: World Health Report 2000: how it removes equity from the agenda for public health monitoring policy. BMJ 2001, 323:678-681.
• [43]AbouZahr C, Adjei S, Kanchanachitra C: From data to policy: good practices and cautionary tales. Lancet 2007, 369:1039-1046.
• [44]Handler A, Issel M, Turnock B: A conceptual framework to measure performance of the public health system. American Journal of Public Health 2001, 91(8):1235-1239.
• [45]Kruk ME, Freedman LP: Assessing health system performance in developing countries: a review of the literature. Health Policy 2008, 85:263-276.
• [46]Scutchfield FD, Bhandari WM, Lawhorn AN, Lamberth DC, Ingram CR: Public health performance. American Journal of Preventive Medicine 2009, 36(3):266-272.
• [47]Smith PC, Mossialos E, Papanicolas I, Leatherman S: Conclusions. In Performance measurement for health system improvement: experiences, challenges and prospects. Edited by Smith PC, Mossialos E, Papanicolas, S L I. New York USA: Cambrige University Press; 2009:675-706.
• [48]Hurst J, Jee-Hughes M: Performance measurement and performance management in OECD health systems. In OECD Labour Market and Social Policy Occasional Papers. OECD Publishing; 2001. http://www.oecd.ilibrary.org/content/workingpaper/78822407371 webcite
• [49]Ibrahim JE: Performance indicators for all perspectives. International Journal for Quality in Health Care 2001, 13(6):431-432.
• [50]Navarro V: Assessment of the World Health Report. Lancet 2000, 356:1598-1601.
• [51]Williams A: Science or marketing at WHO? A commentary on 'World Health Report 2000'. Health Economics 2000, 10(2):283-285.
• [52]Kirigia MJ, Kirigia GD: The essence of governance in health development. International Archives of Medicine 2011, 4:11. BioMed Central Full Text
• [53]Olafsdottir AE, Redpath DD, Pokhrel S, Allotey P: Health system performance in sub-Saharan Africa: governance, outcome, equity. BMC Public Health 2011, 11:237. BioMed Central Full Text
• [54]Eddy MD: Performance measurement: problems and solutions. Health Affairs 1998, 17(4):7-25.
• [55]Bierman S, Clark JP: Performance measurement and equity. BMJ 2007, 334:1333-1334.
• [56]Freeman T: Using performance indicators to improve health care quality in the public sector: a review of literature. Health Services Management Research 2002, 15:126-137.
• [57]McGlynn AE: Selecting common measures of quality and system performance. Medical Care 2003, 41(1, Supplement 1):i39-i47.
• [58]Donabedian A: Evaluating the quality of medical care. The Milbank Quarterly 2005, 83:691-729.
• [59]Kaplan RS, Norton DP: The balanced score card - measures that drive performance. Harvard Business Review 1992, 71-79.
• [60]Wong TS, Yin D, Bhattacharya O, Wang B, Liu L, Chen B: Developing a performance measurement framework and indicators for community health service facilities in urban China. BMC Family Practice 2010, 11:91. BioMed Central Full Text
• [61]McGlynn AE: The outcomes utility index. International Journal for Quality in Health Care 1998, 10(6):485-490.
• [62]Mannion R, Davies HTO: Reporting health care performance: learning from the past, prospects for the future. Journal of Evaluation in Clinical Practice 2001, 8(2):215-228.
• [63]Nutley S, Smith PC: League Tables for performance improvement in healthcare Journal for Health Services Research and. Policy 1998, 3:50-57.
• [64]Nandraj S, Khot A, Menon S, Brugha R: A stakeholder approach towards hospital accreditation in India. Health Policy and Planning 2001, 16(Supplement 2):70-79.
• [65]Sutherland K, Leatherman S: Professional regulation: does certification improve medical standards? BMJ 2006, 333:439-441.
• [66]Haffner JM, Williams SC, Koss GR, Tschurtz BA, Schmaltz SP, Loeb JM: The perceived impact of public reporting hospital performance data: interviews with hospital staff. International Journal for Quality in Health Care 2011, 23(6):697-704.
• [67]Leatherman S, McCarthy D: Public disclosure of health care performance reports: experience, evidence and issues for policy. International Journal for Quality in Health Care 1999, 11(2):93-98.
• [68]Shekelle PG: Public performance reporting on quality information. In Performance measurement for health system improvement: experiences, challenges and prospects. Edited by Smith PC, Mossialos E, Papanicolas I, Leatherman S. New York USA: Cambridge University Press; 2009.
• [69]Conrad DA: Incentives for healthcare performance improvement. In Performance measurement for health system improvement: experiences, challenges and prospects. Edited by Smith PC, Mossialos E, Papanicolas I, Leatherman S. New York USA: Cambridge University Press; 2009.
• [70]Witter S, Fretheim A, Kessy FL, Lindahl AK: Paying for performance to improve the delivery of health interventions in low and middle income countries. Cochrane Database of Systematic Reviews 2012, 2:CD007899.
• [71]Mannion R, Davies HTO: Payment for performance in health care. BMJ 2008, 336:306-308.
• [72]Day C, Barron P, Massyn N, Padarath A, English R: The District Health Barometer 2012. South Africa Health Systems Trust: Durban; 2012.
• [73]Ministry of Health: Holistic Assessment of performance in the health sector. Accra Ghana Ministry of Health Ghana; 2008.
• [74]Health Canada and the Public Agency of Canada: Evaluation of the Health Information Initiative 2007-2008 to 2011-2012. Ontario Canada: Evaluation Directorate of Health Canada & Public Health Agency of Canada; 2013. At http://www.cihi.ca/CIHI-ext-portal/pdf/internet/ROADMAP_EXSUMM_EN webcite
• [75]IBM: Evaluation of the Health Information Roadmap Initiative: Roadmap II and Raodmap II plus, Executive Summary. Ontario Canada: IBM; 2008.
• [76]Ten Asbroek AHA, Arah OA, Geelhoed J, Custers T, Delnoij JMD, Klazinga NS: Develoding a national performance indicator framework for the Dutch health system. International Journal for Quality in Health Care 2004, 16:i65-i71.
• [77]Barron P, Day C, Loveday M, Monticelli F: The District Health Barometer Year 1, January - December 2004. South Africa Health Systems Trust: Durban; 2005.
• [78]Ministry of Health: Independent Review Health Sector Programme of Work 2010, Final Report. Accra Ghana Ministry of Health Ghana; 2008.
• [79]Australia Institute of Health and Welfare: Australia's health in 2012: in brief. Canberra Australia Australia Institute of Health and Welfare; 2012.
• [80]Canada Institute for Health Information: Annual Report for 2003–04. Ottawa Ontario Canada Canada Institute for Health Information; 2004.
• [81]Fonn S, Padarath A: Editorial. In South Africa Health Review. Edited by Fonn S, Padarath A. Durban South Africa Health Systems Trust; 2010.
• [82]English R, Masilela T, Barron P, Schonfeldt A: Health Information Systems in South Africa. In South Africa Health Review 2011. Edited by Padarath A, English R. Durban South Africa Health Systems Trust; 2011.
• [83]Treasury Board of Canada Secretariat: TUp-front Multi-year funding: grant to the Canada Institute for Health Information. At http://www.hc-sc.gc.ca/ahc-asc/performance/eval/2013-hii-iis-eng.php webcite
• [84]Canada H: Healthy Canadians - a Federal Report on Comparable Health Indicators 2010. Health Canada: Ottawa Ontario Canada; 2011.
• [85]Persaud DD, Nestman L: The utilisation of systematic outcome mapping to improve performance management in health care. Health Services Management Research 2006, 19:264-276.
• [86]National Health Information Standards and Statistics Committee (NHISSC): The National Health Performance Framework. Canberra Australia NHISSC 2009.
• [87]Human Rights Watch: Stop Making Excuses, Accountability for Maternal Health Care in South Africa. New York USA Human Rights Watch; 2011.
• [88]The Lancet South Africa Series Executive Summary Core Group: Health in South Africa: Executive Summary for the Series. Edited by Lawn JE, Kinney MV. Cape Town South Africa: The Lancet Health in South Africa: Executive Summary for the Series; 2009.
• [89]Ministry of Health: Subnational (District) health system performance assessment report 2004. Kampala Uganda Ministry of Health Uganda and World Health Organisation Uganda Country Office; 2004.
• [90]Health Systems 20/20: The Health System Assessment Approach: A how to manual. 2012. At http://www.healthsystems2020.org/content/resource/detail/528 webcite Version 2.0
• [91]McPake B, Mills A: What can we learn from international comparisons of health systems and health reform? Bulletin of the World Health Organisation 2000, 78(6):811-820.
• [92]Rose R: Lesson drawing in public policy: a guide to learning across time and space. New Jersey Chatham House Publishers; 1993.
• [93]Millar J: Healthy Canadians in 2010? Canadian Medical Association Journal 2000, 162(13):1823-1824.
• [94]Hyder AA: Misunderstanding the World Health Report 2000. American Journal of Public Health 2002, 92(7):1054-1055.
• [95]Hyder AA: Applying WHO's health system performance assessment in Pakistan: critical reflections. Journal of College of Physicians and Surgeons of Pakistan 2003, 13(4):187-191.
• [96]Australia Institute of Health and Welfare: Australia's Health 2010. Canberra Australia Australia Institute of Health and Welfare; 2010.
• [97]Navarro V: The new conventional wisdom: an evaluation of the WHO Report. Health Systems: Improving Performance International Journal of Health Services 2001, 31(1):23-33.
• [98]Weinberg N: Using performance measures to identofy plans of action to improve care. Journal on Quality Improvement 2001, 27(12):683-688.
• [99]Smith PC: Measuring health system performance. European Journal of Health Economics 2002, 3:145-148.