Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool

Health Research Policy and Systems

Simon Chapman³  Sally Redman⁵  Adrian E Bauman²  Andrew J Milat¹  Lucie Rychetnik⁴  Lesley King²  Robyn Newson²  Jacqueline Schroeder³  Gillian Cohen³ 

[1] Centre for Epidemiology and Evidence, NSW Ministry of Health, Level 7, 73 Miller St, North Sydney, NSW 2060, Australia;Prevention Research Collaboration, School of Public Health, level 6, Building D17, University of Sydney, Sydney, NSW 2006, Australia;School of Public Health, University of Sydney, Edward Ford Building (A27), Fisher Rd, Sydney, NSW 2006, Australia;School of Medicine, University of Notre Dame, 160 Oxford Street, Darlinghurst Sydney, NSW 2010, Australia;Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, NSW 2007, Australia

关键词: Research translation;    Research impact;    Policy;    Intervention research;   

Others  :  1133773
DOI  :  10.1186/1478-4505-13-3

 received in 2014-10-14, accepted in 2014-12-11,  发布年份 2015

【 摘 要 】

Background

There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003–2007) of intervention research funded by Australia’s National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts.

Methods

We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance.

Results

Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews.

Conclusions

While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

【 授权许可】

   
2014 Cohen et al.; licensee BioMed Central.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Denholm EM, Martin WJ: Translational research in environmental health sciences. Transl Res 2008, 151:57-58.
• [2]Erno-Kjolhede E, Hansson F: Measuring research performance during a changing relationship between science and society. Res Eval 2011, 20:131-143.
• [3]Frodeman R, Holbrook JB: Science’s social effects. Issues Sci Technol 2007, 23:28-30.
• [4]Bornmann L: Measuring the societal impact of research. Eur Mol Biol Organ J 2012, 13:673-676.
• [5]Martin BR: The research excellence framework and the ‘impact agenda’: are we creating a Frankenstein monster? Res Eval 2011, 20:247-254.
• [6]Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE: Knowledge translation of research findings. Implement Sci 2012, 7:1748-5908.
• [7]Holbrook JB: Re-assessing the science–society relation: The case of the US National Science Foundation’s broader impacts merit review criterion (1997–2011). In Peer Review, Research Integrity, and the Governance of Science – Practice, Theory, and Current Discussion. Edited by Frodeman RJ, Holbrook B, Mitcham C, Xiaonan H. Dalian: People’s Publishing House and Dalian University of Technology; 2012:328-362.
• [8]Henshall C: The impact of payback research: developing and using evidence in policy. Res Eval 2011, 20:257-258.
• [9]Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C, Bauman A, Redman S, Bennie J: Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Sys 2013, 11:5. BioMed Central Full Text
• [10]Glasgow RE, Emmons KM: How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health 2007, 28:413-433.
• [11]Bhattacharyya OK, Estey EA, Zwarenstein M: Methodologies to evaluate the effectiveness of knowledge translation interventions: a primer for researchers and health care managers. J Clin Epidemiol 2011, 64:32-40.
• [12]Brownson RC, Kreuter MW, Arrington BA, True WR: Translating scientific discoveries into public health action: how can schools of public health move us forward? Public Health Rep 2006, 121:97-103.
• [13]Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, Kinsman L: ‘Making evidence count’: a framework to monitor the impact of health services research. Aust J Rural Health 2012, 20:51-58.
• [14]Buxton M: The payback of ‘Payback’: challenges in assessing research impact. Res Eval 2011, 20:259-260.
• [15]Banzi R, Moja L, Pistotti V, Facchini A, Liberati A: Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst 2011, 9:26. BioMed Central Full Text
• [16]Donovan C: State of the art in assessing research impact: introduction to a special issue. Res Eval 2011, 20:175-179.
• [17]Grant J, Brutscher P-B, Kirk SE, Butler L, Wooding S: Capturing Research Impacts: A Review of International Practice. Documented Briefing. Santa Monica, CA: Rand Corporation; 2010.
• [18]Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M: The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003, 1:2. BioMed Central Full Text
• [19]Hanney S, Packwood T, Buxton M: Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation 2000, 6:137-160.
• [20]Graham KER, Chorzempa HL, Valentine PA, Magnan J: Evaluating health research impact: development and implementation of the Alberta innovates – health solutions impact framework. Res Eval 2012, 21:354-367.
• [21]Wooding S, Hanney S, Pollitt A, Buxton J: Project Retrosight. Understanding the Returns from Cardiovascular and Stroke Research: Policy Report. Cambridge: RAND Europe; 2011. [http://www.rand.org/pubs/monographs/MG1079.html webcite] Accessed Oct 2014
• [22]Group of Eight: Excellence in Innovation: Research Impacting our Nation’s Future - Assessing the Benefits. Adelaide: Australian Technology Network of Universities; 2012. [https://go8.edu.au/programs-and-fellowships/excellence-innovation-australia-eia-trial webcite] Accessed Oct 2014
• [23]Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, Department for Employment and Learning Northern Ireland: Research Excellence Framework Impact Pilot Exercise Findings of the Expert Panels. London: Research Excellence Framework; 2010. [http://www.ref.ac.uk/pubs/refimpactpilotexercisefindingsoftheexpertpanels/ webcite] Accessed Oct 2014
• [24]Buxton M, Hanney S: How can payback from health services research be assessed? J Health Serv Res 1996, 1:35-43.
• [25]Hanney SR, Grant J, Wooding S, Buxton MJ: Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK‘s ’Arthritis Research Campaign’. Health Res Policy Syst 2004, 2:4. BioMed Central Full Text
• [26]Hanney SR, Watt A, Jones TH, Metcalf L: Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: the case of Asthma UK. Allergy Asthma Clin Immunol 2013, 9:17. BioMed Central Full Text
• [27]Higher Education Funding Council for England: Decisions on Assessing Research Impact. London: Research Excellence Framework; 2011. [http://www.ref.ac.uk/pubs/2011-01/ webcite] Accessed Oct 2014
• [28]Higher Education Funding Council for England: Panel Criteria and Working Methods. London: Research Excellence Framework; 2012. [http://www.ref.ac.uk/pubs/2012-01/ webcite] Accessed Oct 2014
• [29]Hanney S, Buxton M, Green C, Coulson D, Raftery J: An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 2007, 11:iii-iv. ix-xi, 1–180
• [30]Johnston SC, Rootenberg JD, Katrak S, Smith WS, Elkins JS: Effect of a US National Institutes of Health programme of clinical trials on public health and costs. Lancet 2006, 367:1319-1327.
• [31]Oortwijn WJ, Hanney SR, Ligtvoet A, Hoorens S, Wooding S, Grant J, Buxton MJ, Bouter LM: Assessing the impact of health technology assessment in The Netherlands. Int J Technol Assess Health Care 2008, 24:259-269.
• [32]Pollitt A, Wooding S, Hanney S, Buxton M, Grant J: Project Retrosight. Understanding the Returns from Cardiovascular and Stroke Research: Methodology Report. Cambridge: RAND Europe; 2011. [http://www.rand.org/pubs/technical_reports/TR925.html webcite] Accessed Oct 2014
• [33]Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR: Evaluation of the impact of national breast cancer foundation-funded research. Med J Aust 2014, 200:214-218.
• [34]Grant J, Cottrell R, Cluzeau F, Fawcett G: Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study. BMJ (Clin Res Ed) 2000, 320:1107-1111.
• [35]Garland SM, Hernandez-Avila M, Wheeler CM, Perez G, Harper DM, Leodolter S, Tang GW, Ferris DG, Steben M, Bryan J: Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N Engl J Med 2007, 356:1928-1943.
• [36]Tay S-K: Cervical cancer in the human papillomavirus vaccination era. Curr Opin Obstet Gyn 2012, 24:3-7.
• [37]Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S: Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect 2012, 15:229-241.
• [38]Kwan P, Johnston J, Fung AY, Chong DS, Collins RA, Lo SV: A systematic evaluation of payback of publically funded health and health services research in Hong Kong. BMC Health Serv Res 2007, 7:121. BioMed Central Full Text