A narrative review of research impact assessment models and methods

Health Research Policy and Systems

Sally Redman¹  Adrian E Bauman²  Andrew J Milat² 

[1] Sax Institute, Sydney, Level 2, 10 Quay, St Haymarket 2000, NSW, Australia;School of Public Health, University of Sydney, Level 2, Medical Foundation, Building, K25, Sydney 2006, NSW, Australia

关键词: Research returns;    Research impact;    Policy and practice impact;   

Others  :  1170917
DOI  :  10.1186/s12961-015-0003-1

 received in 2014-11-07, accepted in 2015-02-16,  发布年份 2015

【 摘 要 】

Background

Research funding agencies continue to grapple with assessing research impact. Theoretical frameworks are useful tools for describing and understanding research impact. The purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research.

Methods

The review involved keyword searches of electronic databases, including MEDLINE, CINAHL, PsycINFO, EBM Reviews, and Google Scholar in July/August 2013. Review search terms included ‘research impact’, ‘policy and practice’, ‘intervention research’, ‘translational research’, ‘health promotion’, and ‘public health’. The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact. The review was conducted in two phases: initially, abstracts were retrieved and assessed against the review criteria followed by the retrieval and assessment of full papers against review criteria.

Results

Thirty one primary studies and one systematic review met the review criteria, with 88% of studies published since 2006. Studies comprised assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies had an explicit focus on assessing impacts of health promotion or public health research and one had a specific focus on intervention research impact assessment. A total of 16 different impact assessment models were identified, with the ‘payback model’ the most frequently used conceptual framework. Typically, impacts were assessed across multiple dimensions using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The vast majority of studies relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and end-users of research.

Conclusions

Research impact assessment is a new field of scientific endeavour and there are a growing number of conceptual frameworks applied to assess the impacts of research.

【 授权许可】

   
2015 Milat et al.; licensee BioMed Central.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Anderson W, Papadakis E: Research to improve health practice and policy. Med J Aust 2009, 191(11/12):646-7.
• [2]Cooksey D: A review of UK health research funding. HMSO, London; 2006.
• [3]Health and Medical Research Strategic Review Committee: The virtuous cycle: working together for health and medical research. Commonwealth of Australia, Canberra; 1998.
• [4]National Health and Medical Research Council Public Health Advisory Committee: Report of the Review of Public Health Research Funding in Australia. NHMRC, Canberra; 2008.
• [5]Campbell DM: Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy 2009, 6:21. BioMed Central Full Text
• [6]Wells R, Whitworth JA: Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy 2007, 4:14. BioMed Central Full Text
• [7]Australian Government Australian Research Council: Excellence in Research in Australia 2012. Australian Research Council, Canberra; 2012.
• [8]Kuruvilla S, Mays N, Walt G: Describing the impact of health services and policy research. J Health Serv Res Policy 2007, 12(Suppl 1):S1. -23-31
• [9]Weiss AP: Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatr 2007, 164(2):206-14.
• [10]Bornmann L: Measuring the societal impact of research. Eur Mol Biol Organ 2012, 13(8):673-6.
• [11]Holbrook JB: Re-assessing the science–society relation: The case of the US National Science Foundation’s broader impacts merit review criterion (1997–2011). In Peer Review, Research Integrity, and the Governance of Science–Practice, Theory, and Current Discussions. Edited by Frodeman R, Holbrook JB, Mitcham C, Xiaonan H. People’s Publishing House and Dalian University of Technology, Dalian; 2012:328-62.
• [12]Holbrook JB, Frodeman R. Science’s social effects. Issues in Science and Technology. 2007. http://issues.org/23-3/p_frodeman-3/.
• [13]Banzi R, Moja L, Pistotti V, Facchini A, Liberati A: Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. health Res Policy Syst 2011, 9:26. BioMed Central Full Text
• [14]Boaz A, Fitzpatrick S, Shaw B: Assessing the impact of research on policy: A review of the literature for a project on bridging research and policy through outcome evaluation. Policy Studies Institute London, London; 2008.
• [15]Aymerich M, Carrion C, Gallo P, Garcia M, López-Bermejo A, Quesada M, et al.: Measuring the payback of research activities: a feasible ex-post evaluation methodology in epidemiology and public health. Soc Sci Med 2012, 75(3):505-10.
• [16]Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S: Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect 2012, 15(3):229-41.
• [17]Barker K, The UK: Research Assessment Exercise: the evolution of a national research evaluation system. Res Eval 2007, 16(1):3-12.
• [18]Boyack KW, Jordan P: Metrics associated with NIH funding: a high-level view. J Am Med Inform Assoc 2011, 18(4):423-31.
• [19]Buxton M, Hanney S, Morris S, Sundmacher L, Mestre-Ferrandiz J, Garau M, et al. Medical research: what’s it worth. Estimating the economic benefits from medical research in the UK. Report for MRC, Wellcome Trust and the Academy of Medical Sciences. 2008. http://www.wellcome.ac.uk/stellent/groups/corporatesite/@sitestudioobjects/documents/web_document/wtx052110.pdf.
• [20]Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al.: ‘Making evidence count’: A framework to monitor the impact of health services research. Aust J Rural Health 2012, 20(2):51-8.
• [21]Deloitte Access Economics: Extrapolated returns on investment in NHMRC medical research. Australian Society for Medical Research, Canberra; 2012.
• [22]Derrick GE, Haynes A, Chapman S, Hall WD: The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS One 2011, 6(4):e18521.
• [23]Franks AL, Simoes EJ, Singh R, Gray BS: Assessing prevention research impact: a bibliometric analysis. Am J Prev Med 2006, 30(3):211-6.
• [24]Graham KE, Chorzempa HL, Valentine PA, Magnan J: Evaluating health research impact: development and implementation of the Alberta Innovates–Health Solutions impact framework. Res Eval 2012, 21(5):354-67.
• [25]Canadian Institutes of Health Research: Developing a CIHR framework to measure the impact of health research. Canadian Institutes of Health Research, Ottawa; 2005.
• [26]Group of Eight: Excellence in innovation: research impacting our nation’s future – assessing the benefits. Australian Technology Network of Universities, Adelaide; 2012.
• [27]Hanney S: An assessment of the impact of the NHS Health Technology Assessment Programme. National Coordinating Centre for Health Technology Assessment, University of Southampton, Southampton; 2007.
• [28]Higher Education Funding Council for England: Panel criteria and working methods. Higher Education Funding Council for England, London; 2012.
• [29]Kalucy EC, Jackson-Bowers E, McIntyre E, Reed R: The feasibility of determining the impact of primary health care research projects using the Payback Framework. Health Res Policy Syst 2009, 7:11. BioMed Central Full Text
• [30]Kuruvilla S, Mays N, Pleasant A, Walt G: Describing the impact of health research: a Research Impact Framework. BMC Health Serv Res 2006, 6(1):134. BioMed Central Full Text
• [31]Kwan P, Johnston J, Fung AYK, Chong DSY, Collins RA, Lo SV: A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res 2007, 7(1):121. BioMed Central Full Text
• [32]Landry R, Amara N, Lamari M: Climbing the ladder of research utilization: Evidence from social science research. Sci Commun 2001, 22:396-422.
• [33]Lavis J, Ross S, McLeod C, Gildiner A: Measuring the impact of health research. J Health Serv Res Policy 2003, 8(3):165-70.
• [34]Laws R, King L, Hardy LL, Milat AJ, Rissel C, Newson R, et al.: Utilization of a population health survey in policy and practice: a case study. Health Res Policy Syst 2013, 11:4. BioMed Central Full Text
• [35]Liebow E, Phelps J, Van Houten B, Rose S, Orians C, Cohen J, et al.: Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environ Health Perspect 2009, 117(7):1147.
• [36]Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al.: Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst 2013, 11:5. BioMed Central Full Text
• [37]National Institutes of Health: Cost savings resulting from NIH research support. United States Department of Health and Human Services National Institute of Health, Bethesda, MD; 1993.
• [38]Ovseiko PV, Oancea A, Buchan AM: Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res 2012, 12:478. BioMed Central Full Text
• [39]Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA: Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev 2012, 36(2):218-23.
• [40]Spoth RL, Schainker LM, Hiller-Sturmhöefel S: Translating family-focused prevention science into public health impact: illustrations from partnership-based research. Alcohol Res Health 2011, 34(2):188.
• [41]Sullivan R, Lewison G, Purushotham AD: An analysis of research activity in major UK cancer centres. Eur J Cancer 2011, 47(4):536-44.
• [42]Taylor J, Bradbury-Jones C: International principles of social impact assessment: lessons for research? J Res Nurs 2011, 16(2):133-45.
• [43]Warner KE, Tam J: The impact of tobacco control research on policy: 20 years of progress. Tob Control 2012, 21(2):103-9.
• [44]Wooding S, Hanney S, Buxton M, Grant J: The returns from arthritis research. Volume 1: Approach analysis and recommendations. RAND Europe, Netherlands; 2004.
• [45]Buxton M, Hanney S: How can payback from health services research be assessed? J Health Serv Res Policy 1996, 1(1):35-43.
• [46]Higher Education Funding Council for England: Decisions on assessing research impact. Higher Education Funding Council for England, Bristol; 2011.
• [47]Grant J, Brutscher P-B, Kirk SE, Butler L, Wooding S. Capturing research impacts: a review of international practice. Documented Briefing. RAND Corporation; 2010. http://www.rand.org/pubs/documented_briefings/DB578.html.
• [48]Murphy KM, Topel RH: Measuring the gains from medical research: an economic approach. University of Chicago Press, Chicago; 2010.
• [49]Balas EA, Boren SA: Managing clinical knowledge for health care improvement. In Yearbook of Medical Informatics 2000: Patient-Centered Systems. Edited by Bemmel J, McCray AT. Schattauer Verlagsgesellschaft mbH, Stuttgart, Germany; 2000:65-70.