Learning Evaluation: blending quality improvement and implementation research methods to study healthcare innovations

Implementation Science

Kurt C Stange²  Benjamin F Crabtree³  William L Miller⁵  L Miriam Dickinson⁶  Rose Gunn¹  Melinda M Davis¹  Deborah J Cohen¹  Bijal A Balasubramanian⁴ 

[1] Department of Family Medicine, Oregon Health & Science University, Portland, OR, USA;Departments of Family Medicine and Community Health, Epidemiology and Biostatistics, and Sociology, Case Western Reserve University, Cleveland, OH, USA;Department of Family Medicine and Community Health, Robert Wood Johnson Medical School, Rutgers University, New Brunswick, NJ, USA;Harold Simmons Comprehensive Cancer Center, UT Southwestern Medical Center, Dallas, TX, USA;Department of Family Medicine, Lehigh Valley Health Network, Allentown, PA, USA;Department of Family Medicine, University of Colorado School of Medicine, Aurora, CO, USA

关键词: Delivery of healthcare;    Evaluation;    Implementation science;    Quality improvement;   

Others  :  1139266
DOI  :  10.1186/s13012-015-0219-z

 received in 2014-08-22, accepted in 2015-02-17,  发布年份 2015

【 摘 要 】

Background

In healthcare change interventions, on-the-ground learning about the implementation process is often lost because of a primary focus on outcome improvements. This paper describes the Learning Evaluation, a methodological approach that blends quality improvement and implementation research methods to study healthcare innovations.

Methods

Learning Evaluation is an approach to multi-organization assessment. Qualitative and quantitative data are collected to conduct real-time assessment of implementation processes while also assessing changes in context, facilitating quality improvement using run charts and audit and feedback, and generating transportable lessons. Five principles are the foundation of this approach: (1) gather data to describe changes made by healthcare organizations and how changes are implemented; (2) collect process and outcome data relevant to healthcare organizations and to the research team; (3) assess multi-level contextual factors that affect implementation, process, outcome, and transportability; (4) assist healthcare organizations in using data for continuous quality improvement; and (5) operationalize common measurement strategies to generate transportable results.

Results

Learning Evaluation principles are applied across organizations by the following: (1) establishing a detailed understanding of the baseline implementation plan; (2) identifying target populations and tracking relevant process measures; (3) collecting and analyzing real-time quantitative and qualitative data on important contextual factors; (4) synthesizing data and emerging findings and sharing with stakeholders on an ongoing basis; and (5) harmonizing and fostering learning from process and outcome data. Application to a multi-site program focused on primary care and behavioral health integration shows the feasibility and utility of Learning Evaluation for generating real-time insights into evolving implementation processes.

Conclusions

Learning Evaluation generates systematic and rigorous cross-organizational findings about implementing healthcare innovations while also enhancing organizational capacity and accelerating translation of findings by facilitating continuous learning within individual sites. Researchers evaluating change initiatives and healthcare organizations implementing improvement initiatives may benefit from a Learning Evaluation approach.

【 授权许可】

   
2015 Balasubramanian et al.; licensee BioMed Central.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Institute of Medicine (IOM): Crossing the quality chasm: a new health system for the 21st century. National Academy Press, Washington, D.C; 2001.
• [2]Berwick DM, Nolan TW, Whittington J: The triple aim: care, health, and cost. Health Aff 2008, 27:759-69.
• [3]Joint principles of the patient-centered medical home [http://www.pcpcc.net]
• [4]Nutting PA, Crabtree BF, Miller WL, Stange KC, Stewart E, Jaen C: Transforming physician practices to patient-centered medical homes: lessons from the national demonstration project. Health Affairs (Project Hope) 2011, 30:439-445.
• [5]Kathol RG, de Gruy F, Rollman BL: Value-based financially sustainable behavioral health components in patient-centered medical homes. Ann Fam Med 2014, 12:172-175.
• [6]Academy for integrating behavioral health and primary care [http://integrationacademy.ahrq.gov]
• [7]SMAHSA-HRSA Center for Integrated Health Solutions [http://www.integration.samhsa.gov]
• [8]Balasubramanian BA, Cohen DJ, Clark EC, Isaacson NF, Hung DY, Dickinson LM, et al.: Practice-level approaches for behavioral counseling and patient health behaviors. Am J Prev Med 2008, 35:S407-413.
• [9]Cohen DJ, Balasubramanian BA, Isaacson NF, Clark EC, Etz RS, Crabtree BF: Coordination of health behavior counseling in primary care. Ann Fam Med 2011, 9:406-415.
• [10]Crabtree BF, Chase SM, Wise CG, Schiff GD, Schmidt LA, Goyzueta JR, et al.: Evaluation of patient centered medical home practice transformation initiatives. Med Care 2011, 49:10-16.
• [11]Davis M, Balasubramanian BA, Waller E, Miller BF, Green LA, Cohen DJ: Integrating behavioral and physical health care in the real world: early lessons from advancing care together. J Am Board Fam Med 2013, 26:588-602.
• [12]Glasgow RE, Emmons KM: How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health 2007, 28:413-433.
• [13]Balasubramanian BA, Chase SM, Nutting PA, Cohen DJ, Strickland PA, Crosson JC, et al.: Using learning teams for reflective adaptation (ULTRA): insights from a team-based change management strategy in primary care. Ann Fam Med 2010, 8:425-32.
• [14]Solberg LI, Kottke TE, Brekke ML, Magnan S, Davidson G, Calomeni CA, et al.: Failure of a continuous quality improvement intervention to increase the delivery of preventive services. A randomized trial. Effect Clin Pract 2000, 3:105-115.
• [15]Batalden PB, Davidoff F: What is “quality improvement” and how can it transform healthcare? Qual Saf Health Care 2007, 16:2-3.
• [16]Peek CJ, Cohen DJ, deGruy Iii FV: Research and evaluation in the transformation of primary care. Am Psychol 2014, 69:430-442.
• [17]Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S, Group SD: Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care 2008, 17(Suppl 1):i3-9.
• [18]Stange KC, Glasgow RE: Considering and reporting important contextual factors in research on the patient-centered medical home. Agency Healthcare Res Q. 2013, AHRQ Publication No. 13-0045-EF.
• [19]Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, Cronholm PF, Halladay JR, Driscoll DL, et al.: Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med 2013, 11:S115-S23.
• [20]ACT Advancing Care Together - creating systems of care for the whole person [http://www.advancingcaretogether.org/]
• [21]Blount A: Integrated primary care: organizing the evidence. Fam Syst Health 2003, 21:121-133.
• [22]Butler M, Kane RL, McAlpine D, Kathol RG, Fu S, Hagedorn H, Wilt T: Integration of mental heatlh/substance abuse and primary care. AHRQ Publication No. 09-E003. In Book Integration of mental heatlh/substance abuse and primary care. AHRQ Publication No. 09-E003. Agency for Healthcare Research & Quality, Rockville, MD; 2008.
• [23]Cohen DJ, Leviton LC, Isaacson N, Tallia AF, Crabtree BF: Online diaries for qualitative evaluation: gaining real-time insights. Am J Eval 2006, 27:163-184.
• [24]Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999, 89:1322-1327.
• [25]Perla RJ, Provost LP, Murray SK: The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf 2011, 20:46-51.
• [26]Benneyan JC, Lloyd RC, Plsek PE: Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003, 12:458-464.
• [27]Pawson R, Tilley N: Realistic evaluation. Sage, London, UK; 1997.
• [28]Shadish WR, Cook TD, Campbell DT: Experimental and quasi-experimental designs for generalized causal inference. New York, NY, Houghton Mifflin Boston; 2002.
• [29]Miller WL, Crabtree BF: The dance of interpretation. In Doing qualitative research. 2nd edition. Edited by Crabtree BF, Miller WL. Sage Publications, Inc, Thousand Oaks, CA; 1999:127-143.
• [30]Raudenbush SW, Bryk AS: Hierarchical linear models: applications and data analysis methods. 2nd edition. Sage Publications, Inc, Thousand Oaks, CA; 2002.
• [31]Jaen CR, Crabtree BF, Palmer RF, Ferrer RL, Nutting PA, Miller WL, et al.: Methods for evaluating practice change toward a patient-centered medical home. Ann Fam Med 2010, 8(Suppl 1):S9-20. S92
• [32]Roy-Byrne PP, Sherbourne CD, Craske MG, Stein MB, Katon W, Sullivan G, et al.: Moving treatment research from clinical trials to the real world. Psychiatr Serv (Washington, DC) 2003, 54:327-332.
• [33]Westfall JM, Mold J, Fagnan L: Practice-based research - ”Blue Highways” on the NIH roadmap. JAMA 2007, 297:403-406.
• [34]Woolf SH: The meaning of translational research and why it matters. JAMA 2008, 299:211-213.
• [35]Fetterman DM: Steps of empowerment evaluation: from California to Cape Town. Eval Program Plann 1994, 17:305-313.
• [36]Fetterman DM: Empowerment evaluation. Eval Pract 1994, 15:1-15.
• [37]Pawson R, Greenhalgh T, Harvey G, Walshe K: Realist review - a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005, 1:21-34.
• [38]Ovretveit J: Action evaluation of health programmes and changes - a handbook for a user-focused approach. Radcliffe Medical Press Ltd, Oxon, United Kingdom; 2002.
• [39]Batalden PB, Nelson EC, Roberts JS: Linking outcomes measurement to continual improvement: the serial “V” way of thinking about improving clinical care. Jt Comm J Qual Improv 1994, 20:167-80.
• [40]Edmondson A, Moingeon B: From organizational learning to the learning organization. Manag Learn 1998, 29:5-20.
• [41]Edmondson AC: Learning from failure in health care: frequent opportunities, pervasive barriers. Qual Saf Health Care 2004, 13:ii3-ii9.
• [42]Etheredge LM: A rapid-learning health system. Health Aff 2007, 26:26.
• [43]Agency for Healthcare Research & Quality. RFA-HS-14-009: Evaluation of AHRQ Initiative to accelerate the dissemination and implementation of PCOR findings into primary care (R01).