Protocol for the process evaluation of a complex intervention designed to increase the use of research in health policy and program organisations (the SPIRIT study)

Implementation Science

Gisselle Gallego¹  Tari Turner²  Carmen Huckel Schneider⁵  Denise O¿Connor⁴  Stacy Carter³  Sue Brennan⁴  Abby Haynes⁶ 

[1] Faculty of Health Science, University of Sydney, 75 East Street, Lidcombe 2141, NSW, Australia;World Vision Australia, 1 Vision Drive, Burwood East 3151, VIC, Australia;Centre for Values, Ethics and the Law in Medicine (VELiM), Medical Foundation Building, Sydney School of Public Health, University of Sydney, 92-94 Parramatta Road, Camperdown 2006, NSW, Australia;Australasian Cochrane Centre, School of Public Health and Preventive Medicine, Monash University, The Alfred Centre, 99 Commercial Road, Melbourne 3004, VIC, Australia;Menzies Centre for Health Policy, Victor Coppleson Building, University of Sydney, Camperdown 2006, NSW, Australia;Sax Institute, 235 Jones Street, Ultimo 2007, NSW, Australia

关键词: Framework analysis;    Developmental evaluation;    Qualitative methods;    Capacity building;    Organisational change;    Health policy;    Knowledge exchange;    Implementation;    Complex intervention;    Process evaluation;   

Others  :  1146435
DOI  :  10.1186/s13012-014-0113-0

 received in 2014-07-28, accepted in 2014-08-12,  发布年份 2014

【 摘 要 】

Background

Process evaluation is vital for understanding how interventions function in different settings, including if and why they have different effects or do not work at all. This is particularly important in trials of complex interventions in `real world¿ organisational settings where causality is difficult to determine. Complexity presents challenges for process evaluation, and process evaluations that tackle complexity are rarely reported. This paper presents the detailed protocol for a process evaluation embedded in a randomised trial of a complex intervention known as SPIRIT (Supporting Policy In health with Research: an Intervention Trial). SPIRIT aims to build capacity for using research in health policy and program agencies.

Methods

We describe the flexible and pragmatic methods used for capturing, managing and analysing data across three domains: (a) the intervention as it was implemented; (b) how people participated in and responded to the intervention; and (c) the contextual characteristics that mediated this relationship and may influence outcomes. Qualitative and quantitative data collection methods include purposively sampled semi-structured interviews at two time points, direct observation and coding of intervention activities, and participant feedback forms. We provide examples of the data collection and data management tools developed.

Discussion

This protocol provides a worked example of how to embed process evaluation in the design and evaluation of a complex intervention trial. It tackles complexity in the intervention and its implementation settings. To our knowledge, it is the only detailed example of the methods for a process evaluation of an intervention conducted as part of a randomised trial in policy organisations. We identify strengths and weaknesses, and discuss how the methods are functioning during early implementation. Using `insider¿ consultation to develop methods is enabling us to optimise data collection while minimising discomfort and burden for participants. Embedding the process evaluation within the trial design is facilitating access to data, but may impair participants¿ willingness to talk openly in interviews. While it is challenging to evaluate the process of conducting a randomised trial of a complex intervention, our experience so far suggests that it is feasible and can add considerably to the knowledge generated.

【 授权许可】

   
2014 Haynes et al.; licensee BioMed Central Ltd.

【 参考文献 】

• [1]Hanney SR, González-Block MA: Evidence-informed health policy: are we beginning to get there at last. Health Res Policy Syst 2009, 22:30. BioMed Central Full Text
• [2]Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M: Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health 2004, 58:811-816.
• [3]Elliott H, Popay J: How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health 2000, 54:461-468.
• [4]Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L: Increasing the use of evidence in health policy: practice and views of policy makers and researchers.Aust N Z Health Policy 2009, 6.
• [5]SUPPORT Tools for evidence-informed health Policymaking (STP) Health Res Policy Syst 2009, 7(Supply 1):S1-S18.
• [6]Moore G, Redman S, Haines M, Todd A: What works to increase the use of research in population health policy and programmes: a review. Evid Policy 2011, 7:277-305.
• [7]Dobbins M, Hanna SE, Ciliska D, Manske S, Cameron R, Mercer SL, O¿Mara L, DeCorby K, Robeson P: A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implement Sci 2009, 4:61. BioMed Central Full Text
• [8]Huckel-Schneider C, Campbell D, Milat A, Haynes A, Quinn E: What are the key organisational capabilities that facilitate research use in public health policy?Public Health Res Pract 2014, 1(1).
• [9]Supporting Policy In health with Research: an Intervention Trial (SPIRIT)¿protocol for a stepped wedge trial BMJ Open 2014, 4:e005293.
• [10]Century J, Rudnick M, Freeman C: A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval 2010, 31:199-218.
• [11]Harachi TW, Abbott RD, Catalano RF, Haggerty KP, Fleming CB: Opening the Black Box: using process evaluation measures to assess implementation and theory building. Am J Community Psychol 1999, 27:711-731.
• [12]Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS: Health services research - process evaluation in randomised controlled trials of complex interventions. Br Med J 2006, 332:413-416.
• [13]Ling T: Evaluating complex and unfolding interventions in real time. Evaluation 2012, 18:79-91.
• [14]Sanderson I: Evaluation in complex policy systems. Evaluation 2000, 6:433-454.
• [15]Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O¿Cathain A, Tinati T, Wight D, Baird J: Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health 2013, 68:101-102.
• [16]Audrey S, Holliday J, Parry-Langdon N, Campbell R: Meeting the challenges of implementing process evaluation within randomized controlled trials: the example of ASSIST (A Stop Smoking in Schools Trial). Health Educ Res 2006, 21:366-377.
• [17]Hulscher MEJL, Laurant MGH, Grol RPTM: Process evaluation on quality improvement interventions. Qual Saf Health Care 2003, 12:40-46.
• [18]Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013, 14:15. BioMed Central Full Text
• [19]Grimshaw JM, Zwarenstein M, Tetroe JM, Godin G, Graham ID, Lemyre L, Eccles MP, Johnston M, Francis JJ, Hux J: Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario printed educational message, OPEM) to improve referral and prescribing practices in primary care in Ontario Canada. Implement Sci 2007, 2:38. BioMed Central Full Text
• [20]Chen H-T: Practical program evaluation: assessing and improving planning, implementation, and effectiveness. Sage Publications, Thousand Oaks, CA; 2005.
• [21]Hasson H: Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci 2010, 5:9. BioMed Central Full Text
• [22]Borrelli B, Sepinwall D, Ernst D, Bellg AJ, Czajkowski S, Breger R, DeFrancesco C, Levesque C, Sharp DL, Ogedegbe G, Resnick B, Orwig D: A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol 2005, 73:852-860.
• [23]Dusenbury L, Brannigan R, Falco M, Hansen WB: A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res 2003, 18:237-256.
• [24]Durlak JA, Dupre EP: Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008, 41:327-350.
• [25]Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, Ogedegbe G, Orwig D, Ernst D, Czajkowski S: Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol 2004, 23:443-451.
• [26]Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S: A conceptual framework for implementation fidelity.Implement Sci 2007, 2(40). doi:4010.1186/1748-5908-2-40.
• [27]Weiss CH: The many meanings of research utilization. Public Adm Rev 1979, 39:426-431.
• [28]Weiss CH: Policy Research: Data, Ideas or Argument? In Social Sciences and Modern States: National Experiences and Theoretical Crossroads. Edited by Wagner P, Weiss CH, Wittrock B, Wollman H. Cambridge University Press, Cambridge; 1991.
• [29]Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004, 82:581-629.
• [30]Russell J, Greenhalgh T, Byrne E, McDonnell J: Recognizing rhetoric in health care policy analysis. J Health Serv Res Policy 2008, 13:40-46.
• [31]Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Abelson J: Examining the role of health services research in public policymaking. Milbank Q 2002, 80:125-154.
• [32]Hanney S, Gonzalez-Block M, Buxton M, Kogan M: The utilisation of health research in policy-making: concepts, examples and methods of assessment.Health Res Policy Syst 2003, 1(2).
• [33]Nutley SM, Walter I, Davies HTO: Using Evidence: How Research can Inform Public Services. The Policy Press, Bristol; 2007.
• [34]Waters E, Armstrong R, Swinburn B, Moore L, Dobbins M, Anderson L, Petticrew M, Clark R, Conning R, Moodie M, Carter R: An exploratory cluster randomised controlled trial of knowledge translation strategies to support evidence-informed decision-making in local governments (The KT4LG study). BMC Public Health 2011, 11:34. BioMed Central Full Text
• [35]Rashman L, Withers E, Hartley J: Organizational learning and knowledge in public service organizations: a systematic review of the literature. Int J Manag Rev 2009, 11:463-494.
• [36]Senge PM: Reflections on ¿a leader¿s new Work: Building Learning Organizations¿. In Knowledge Management: Classic and Contemporary Works. Edited by Morey D, Maybury MT, Thuraisingham B. MIT Press, Cambridge, Mass; 2002:53-60.
• [37]Cheetham G, Chivers G: How professionals learn in practice: an investigation of informal learning amongst people working in professions. J Eur Ind Train 2001, 25:247-292.
• [38]Simon HA: Bounded rationality and organizational learning. Organ Sci 1991, 2:125-134.
• [39]Knowles MS, Holton EF III, Swanson RA: The Adult Learner. Elsevier, Oxford; 2012.
• [40]Brown JS, Duguid P: Organizational learning and communities of practice: towards a unified view of working, learning, and innovation. Organ Sci 1991, 2:40-57.
• [41]Hawe P, Shiell A, Riley T: Theorising interventions as events in systems. Am J Community Psychol 2009, 43:267-276.
• [42]Best A, Holmes B: Systems thinking, knowledge and action: towards better models and methods. Evid Policy 2010, 6:145-159.
• [43]Shiell A, Hawe P, Gold L: Complex interventions or complex systems? Implications for health economic evaluation. Br Med J 2008, 336:1281-1283.
• [44]Bradley F, Wiles R, Kinmonth A-L, Mant D, Gantley M: Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP). BMJ 1999, 318:711-715.
• [45]O¿Cathain A, Goode J, Drabble SJ, Thomas KJ, Rudolph A, Hewison J: Getting added value from using qualitative research with randomized controlled trials: a qualitative interview study. Trials 2014, 15:215. BioMed Central Full Text
• [46]Patton M: Developmental evaluation: applying complexity concepts to enhance innovation and use. Guilford Press, New York; 2011.
• [47]Brennan K: Developmental Evaluation: An Approach to Evaluating Complex Social Change Initiatives. Presentation at Next Generation Evaluation Conference: Embracing Complexity, Connectivity and Change. FSG: Foundation Strategy Group, Stanford University; 2013.
• [48]Campbell D, Donald B, Moore G, Frew D: Evidence Check: knowledge brokering to commission research reviews for policy. Evid Policy 2011, 7:97-107.
• [49]Ritchie J, Spencer L, O¿Connor W: Qualitative Research Practice: A Guide for Social Science Students and Researchers. Sage, London; 2003.
• [50]Pope C, Ziebland S, Mays N: Analysing qualitative data. Br Med J 2000, 320:114-116.
• [51]QSR International Pty Ltd: NVivo Qualitative Data Analysis Software: Version 10. 2012.
• [52]Haynes A, Gillespie JA, Derrick GE, Hall WD, Sally R, Simon C, Heidi S: Galvanizers guides, champions, and shields: the many ways that policymakers Use public health researchers. Milbank Q 2011, 89:564-598.