Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis

Health Research Policy and Systems

Imelda Bates²  Garry Aslanyan¹  Alan Boyd³  Donald C Cole⁴ 

[1] TDR, World Health Organization, Geneva 27 CH-1211, Switzerland;Department of International Public Health, Liverpool School of Tropical Medicine, Liverpool L3 5QA, UK;Manchester Business School, University of Manchester, Manchester M13 9PL, UK;Dalla Lana School of Public Health, University of Toronto, Toronto, ON M5T 3M7, Canada

关键词: World health;    Research support;    Research personnel;    Evaluation studies;    Capacity building;   

Others  :  802459
DOI  :  10.1186/1478-4505-12-17

 received in 2014-01-22, accepted in 2014-03-31,  发布年份 2014

【 摘 要 】

Background

The monitoring and evaluation of health research capacity strengthening (health RCS) commonly involves documenting activities and outputs using indicators or metrics. We sought to catalogue the types of indicators being used to evaluate health RCS and to assess potential gaps in quality and coverage.

Methods

We purposively selected twelve evaluations to maximize diversity in health RCS, funders, countries, and approaches to evaluation. We explored the quality of the indicators and extracted them into a matrix across individual, institutional, and national/regional/network levels, based on a matrix in the ESSENCE Planning, Monitoring and Evaluation framework. We synthesized across potential impact pathways (activities to outputs to outcomes) and iteratively checked our findings with key health RCS evaluation stakeholders.

Results

Evaluations varied remarkably in the strengths of their evaluation designs. The validity of indicators and potential biases were documented in a minority of reports. Indicators were primarily of activities, outputs, or outcomes, with little on their inter-relationships. Individual level indicators tended to be more quantitative, comparable, and attentive to equity considerations. Institutional and national–international level indicators were extremely diverse. Although linkage of activities through outputs to outcomes within evaluations was limited, across the evaluations we were able to construct potential pathways of change and assemble corresponding indicators.

Conclusions

Opportunities for improving health RCS evaluations include work on indicator measurement properties and development of indicators which better encompass relationships with knowledge users. Greater attention to evaluation design, prospective indicator measurement, and systematic linkage of indicators in keeping with theories of change could provide more robust evidence on outcomes of health RCS.

【 授权许可】

   
2014 Cole et al.; licensee BioMed Central Ltd.

【 参考文献 】

• [1]Hanney SR, González-Block MA: Organising health research systems as a key to improving health: the World Health Report 2013 and how to make further progress. Health Res Policy Syst 2013, 11:47. BioMed Central Full Text
• [2]McKee M, Stuckler D, Basu S: Where there is no health research: what can be done to fill the global gaps in health research? PLoS Med 2012, 9:e1001209.
• [3]Tugwell P, Sitthi-Amorn C, Hatcher-Roberts J, Neufeld V, Makara P, Munoz F, Czerny P, Robinson V, Nuyens Y, Okello D: Health research profile to assess the capacity of low and middle income countries for equity-oriented research. BMC Public Health 2006, 6:151. BioMed Central Full Text
• [4]Whitworth J, Sewankambo NK, Snewin VA: Improving implementation: building research capacity in maternal, neonatal, and child health in Africa. PLoS Med 2010, 7:e1000299.
• [5]Fonn S: African PhD research capacity in public health: raison d'être and how to build it. In Global Forum Update on Research for Health. Edited by Malin S. London: Pro-Brook Publishers; 2005:80-83.
• [6]Ghaffar A, IJsselmuiden C, Zicker F (Eds): Changing Mindsets: Research Capacity Strengthening in Low- and Middle-Income Countries. COHRED, Global Forum for Health Research, and UNICEF/UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases (TDR); 2008. http://www.who.int/tdr/publications/documents/changing_mindsets.pdf webcite Accessed 26 Apr 2013
• [7]Bennett S, Paina L, Kim C, Agyepong I, Chunharas S, McIntyre D, Nachuk S: What must be done to enhance capacity for Health Systems Research? In Background Papers Commissioned by the Symposium Secretariat for the First Global Symposium on Health Systems Research. Montreux, Switzerland: World Health Organization; 2010:16-19. http://healthsystemsresearch.org/hsr2010/index.php?option=com_content&view=article&id=77&Itemid=140 webcite
• [8]Golenko X, Pager S, Holden L: A thematic analysis of the role of the organisation in building allied health research capacity: a senior managers’ perspective. BMC Health Serv Res 2012, 12:276. BioMed Central Full Text
• [9]Vogel I: Research Capacity Strengthening: Learning from Experience. UKCDS; 2011. http://www.ukcds.org.uk/sites/default/files/content/resources/UKCDS_Capacity_Building_Report_July_2012.pdf webcite Accessed 27 Dec 2013
• [10]Cole DC, Kakuma R, Fonn S, Izugbara C, Thorogood M, Bates I: Evaluations of health research capacity development: a review of the evidence. Am J Trop Med Hyg 2012, 87(5 suppl 1):801.
• [11]Wirrman-Gadsby E: Research capacity strengthening, donor approaches to improving and assessing its impact in low- and middle-income countries. Int J Health Plann Mgmt 2011, 26:89-106.
• [12]ESSENCE on Health Research: Planning, Monitoring and Evaluation Framework for Capacity Strengthening in Health Research. Geneva: ESSENCE on Health Research; 2011. http://www.who.int/tdr/publications/non-tdr-publications/essence-framework/en/index.html webcite Accessed 26 Apr 2013
• [13]Centre for Theory of Change: What is Theory of Change?. http://www.theoryofchange.org/what-is-theory-of-change/#2 webcite Accessed 18 Jan 2014
• [14]Boyd A, Cole DC, Cho D-B, Aslanyan G, Bates I: Frameworks for evaluating health research capacity strengthening: a qualitative study. Health Res Policy Syst 2013, 11:46. BioMed Central Full Text
• [15]Hales D, UNAIDS: An Introduction to Indicators. UNAIDS Monitoring and Evaluation Fundamentals. Geneva: UNAIDS; 2010:14. http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/8_2-Intro-to-IndicatorsFMEF.pdf webcite Accessed 26 Apr 2013
• [16]Organization for Economic Co-operation and Development: Development Assistance Committee Quality Standards for Development Evaluation. 2010. http://www.oecd.org/dac/evaluationofdevelopmentprogrammes/dcdndep/44798177.pdf webcite Accessed 26 Apr 2013
• [17]Banzi R, Moja L, Pistotti V, Facchini A, Liberati A: Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst 2011, 9:26. BioMed Central Full Text
• [18]Keown K, Van Eerd D, Irvin E: Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. J Continuing Education Health Professions 2008, 28:67-72.
• [19]Maessen O: Intent to Build Capacity through Research Projects: An Examination of Project Objectives, Abstract and Appraisal Documents. Ottawa: International Development Research Centre; 2005. http://www.idrc.ca/uploads/user-S/11635261571Intent_to_Build_Capacity_through_Research_Projects.pdf webcite Accessed 26 Apr 2013
• [20]Lincoln YS, Guba EG: Naturalistic Inquiry. Newbury Park, CA: Sage Publications; 1985.
• [21]Pope C, Ziebland S, Mays N: Qualitative research in health care: analysing qualitative data. BMJ 2000, 320:114-116.
• [22]Saini M, Shlonsk A: Systematic Synthesis of Qualitative Research: A Pocket Guide for Social Work Research Methods. Oxford: Oxford University Press; 2012:240.
• [23]Day R, Stackhouse J, Geddes N: Evaluating Commonwealth Scholarships in the United Kingdom: Assessing Impact in Key Priority Areas. London: Commonwealth Scholarship Commission in the UK; 2009. http://cscuk.dfid.gov.uk/wp-content/uploads/2011/03/evaluation-impact-key-report.pdf webcite Accessed 26 Apr 2013
• [24]Ransom J, Stackhouse J, Groenhout F, Geddes N, Darnbrough M: Evaluating Commonwealth Scholarships in the United Kingdom: Assessing Impact in the Health Sector. London: Commonwealth Scholarship Commission in the UK; 2010. http://cscuk.dfid.gov.uk/wp-content/uploads/2011/03/evaluation-impact-health-report.pdf webcite Accessed 26 Apr 2013
• [25]Podems D: Evaluation of the Regional Initiative in Science and Education. OtherWise; Cape Town: The Carnegie-IAS Regional Initiative in Science and Education (RISE); 2010.
• [26]Health Research for Action (HERA): Review of DANIDA-Supported Health Research in Developing Countries: Main Report Volume One and Annexes Volume Two. Reet: HERA; 2007. http://www.enrecahealth.dk/pdfs/20070309_DANIDA-Research_FINAL_report_volume_I.pdf webcite Accessed 26 Apr 2013
• [27]Mills A, Pairamanzi R, Sewankambo N, Shasha W: Mid-Term Evaluation of the Consortium for National Health Research, Kenya. Nairobi: Consortium for National Health Research; 2010. [made available by the funders]
• [28]van Velzen W, Kamal R, Meda N, Nippert I, Piot P, Sauer F: Independent External Evaluation Report of the European and Developing Countries Clinical Trials Partnership (EDCTP programme). The Hague: EDCTP; 2009. http://ec.europa.eu/research/health/infectious-diseases/poverty-diseases/doc/iee-report-edctp-programme_en.pdf webcite Accessed 26 Apr 2013
• [29]Sachdeva N, Jimeno C, Peebles D: Gender Evaluation of Governance Equity and Health Program: Final Report on Phase I; Institutional Assessment. Ottawa: International Development Research Centre; 2008.
• [30]Sachdeva N, Peebles D: Gender Evaluation of Governance Equity and Health Program: Final Report; Summary and Way Forward/Next Steps. Ottawa: International Development Research Centre; 2008.
• [31]Sachdeva N, Peebles D, Tezare K: Gender Evaluation of Governance Equity and Health Program: Final Report on Phase 2; Projects Review. Ottawa: International Development Research Centre; 2008.
• [32]Peebles D, Sachdeva N: Gender Evaluation of Governance Equity and Health Program: Final Report on Phase 3 – Gender Training Workshop. Ottawa: International Development Research Centre; 2008.
• [33]Zuckerman B, Wilson A, Viola C, Lal B: Evaluation of the Fogarty International Research Collaboration Awards (FIRCA) Program: Phase II Outcome Evaluation. Bethesda: National Institutes of Health; 2006.
• [34]Srivastava CV, Hodges A, Chelliah S, Zuckerman B: Evaluation of the Global Research Initiative Program (GRIP) for New Foreign Investigators. Bethesda: National Institutes of Health; 2009.
• [35]Agyepong I, Kitua A, Borleffs J: Mid-Term Evaluation Report, NACCAP Programmes First Call: INTERACT. The Hague: NACCAP; 2008.
• [36]Vullings W, Meijer I: NACCAP Mid-Term Review: Information Guide and Appendices. Amsterdam: Technopolis Group; 2009.
• [37]Erlandsson B, Gunnarsson V: Evaluation of HEPNet in SSA. Stockholm: Sida; 2005. http://www.sida.se/Publications/Import/pdf/sv/Evaluation-of-HEPNet-in-SSA.pdf webcite Accessed 26 Apr 2013
• [38]Minja H, Nsanzabana C: TDR Research Capacity Strengthening Programmes: Impact Assessment Study Review for the Period 2000–2008 (Draft). Basel: Swiss Tropical & Public Health Institute (SwissTPH); 2010.
• [39]Minja H, Nsanzabana C, Maure C, Hoffmann A, Rumisha S, Ogundahunsi O, Zicker F, Tanner M, Launois P: Impact of health research capacity strengthening in low- and middle-income countries: the case of WHO/TDR programmes. PLoS Negl Trop Dis 2011, 5:e1351.
• [40]Pederson JS, Chataway J, Marjanovic S: CARTA (Consortium for Advanced Research Training in Africa): The Second Year. Cambridge: RAND Europe; 2011. [made available through the funder and contractors]
• [41]Östlin P, Schrecker T, Sadana R, Bonnefoy J, Gilson L, Hertzman C, Kelly MP, Kjellstrom T, Labonté R, Lundberg O, Muntaner C, Popay J, Sen G, Vaghri Z: Priorities for research on equity and health: towards an equity-focused health research agenda. PLoS Med 2011, 8:e1001115.
• [42]Volkman TA, Dassin J, Zurbuchen M: (Eds): Origins, Journeys and Returns: Social Justice in International Higher Education. New York: NY Social Science Research Council & Columbia University Press; 2009.
• [43]Coen SE, Bottorff JL, Johnson JL, Ratner PA: A relational conceptual framework for multidisciplinary health research centre infrastructure. Health Res Policy Syst 2010, 8:29. BioMed Central Full Text
• [44]de Savigny D, Adam T: Systems Thinking for Health Systems Strengthening. Geneva: Alliance for Health Policy and Systems Research/WHO; 2009.
• [45]Kuruvilla S, Mays N, Pleasant A, Walt G: Describing the impact of health research: a research impact framework. BMC Health Serv Res 2006, 6:134. BioMed Central Full Text
• [46]Bates I, Boyd A, Smith H, Cole DC: A practical and systematic approach to organisational capacity strengthening in the health sector in Africa. Health Res Policy Syst 2014, 12:11. BioMed Central Full Text
• [47]Kok MO, Schuit AJ: Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst 2012, 10:21. BioMed Central Full Text
• [48]Kok MO, Rodrigues A, Silva A, de Haan S: The emergence and current performance of a health research system: lessons from Guinea Bissau. Health Res Policy Syst 2012, 10:5. BioMed Central Full Text
• [49]DfID, UK: Call for Proposals for Contract on Centre of Excellence on Impact Evaluation of International Development. https://supplierportal.dfid.gov.uk/selfservice/pages/public/supplier/publicbulletin/viewPublicNotice.cmd?bm90aWNlSWQ9NTgzNzA%3D webcite] Accessed 1 Nov 2014
• [50]US Department of State, (Bureau of Resource Management): Department of State Program Evaluation Policy. 2012. http://www.state.gov/s/d/rm/rls/evaluation/2012/184556.htm#A webcite Accessed 26 Apr 2013
• [51]Vasquez EE, Hirsch JS, Giang LM, Parker RG: Rethinking health research capacity strengthening. Global Public Health 2013, 8(Supp1):S104-S124.