Mixed-method study of a conceptual model of evidence-based intervention sustainment across multiple public-sector service settings

Implementation Science

Mark J Chaffin²  Debra B Hecht⁴  Scott C Roesch⁵  Mark G Ehrhart⁵  Cathleen E Willging³  Amy E Green¹  Gregory A Aarons¹ 

[1] Department of Psychiatry, University of California, La Jolla, San Diego, CA, USA;School of Public Health, Georgia State University, Atlanta, GA, USA;Pacific Institute for Research and Evaluation, Albuquerque, NM, USA;Department of Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, OK, USA;Department of Psychology, San Diego State University, San Diego, CA, USA

关键词: Public sector;    Policy;    Organizational climate;    Organizational culture;    Leadership;    Implementation;    Evidence-based intervention;    Sustainment;   

Others  :  1139476
DOI  :  10.1186/s13012-014-0183-z

 received in 2014-11-19, accepted in 2014-11-24,  发布年份 2014

【 摘 要 】

Background

This study examines sustainment of an EBI implemented in 11 United States service systems across two states, and delivered in 87 counties. The aims are to 1) determine the impact of state and county policies and contracting on EBI provision and sustainment; 2) investigate the role of public, private, and academic relationships and collaboration in long-term EBI sustainment; 3) assess organizational and provider factors that affect EBI reach/penetration, fidelity, and organizational sustainment climate; and 4) integrate findings through a collaborative process involving the investigative team, consultants, and system and community-based organization (CBO) stakeholders in order to further develop and refine a conceptual model of sustainment to guide future research and provide a resource for service systems to prepare for sustainment as the ultimate goal of the implementation process.

Methods

A mixed-method prospective and retrospective design will be used. Semi-structured individual and group interviews will be used to collect information regarding influences on EBI sustainment including policies, attitudes, and practices; organizational factors and external policies affecting model implementation; involvement of or collaboration with other stakeholders; and outer- and inner-contextual supports that facilitate ongoing EBI sustainment. Document review (e.g., legislation, executive orders, regulations, monitoring data, annual reports, agendas and meeting minutes) will be used to examine the roles of state, county, and local policies in EBI sustainment. Quantitative measures will be collected via administrative data and web surveys to assess EBI reach/penetration, staff turnover, EBI model fidelity, organizational culture and climate, work attitudes, implementation leadership, sustainment climate, attitudes toward EBIs, program sustainment, and level of institutionalization. Hierarchical linear modeling will be used for quantitative analyses. Qualitative analyses will be tailored to each of the qualitative methods (e.g., document review, interviews). Qualitative and quantitative approaches will be integrated through an inclusive process that values stakeholder perspectives.

Discussion

The study of sustainment is critical to capitalizing on and benefiting from the time and fiscal investments in EBI implementation. Sustainment is also critical to realizing broad public health impact of EBI implementation. The present study takes a comprehensive mixed-method approach to understanding sustainment and refining a conceptual model of sustainment.

【 授权许可】

   
2014 Aarons et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Edmondson AC: Framing for learning: lessons in successful technology implementation. Calif Manage Rev 2003, 45:35-54.
• [2]Edmondson AC: Speaking up in the operating room: how team leaders promote learning in interdisciplinary action teams. J Manage Stud 2003, 40:1419-1452.
• [3]Massatti RR, Sweeney HA, Panzano PC, Roth D: The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Ment Hlth 2008, 35:50-65.
• [4]Romance NR, Vitale MR, Dolan M, Kalvaitis L: An empirical investigation of leadership and organizational issues associated with sustaining a successful school renewal initiative.Paper presented at the Annual Meeting of the American Educational Research Association April, 2002; New Orleans, LA.
• [5]Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Hlth 2011, 38:4-23.
• [6]Chambers DA, Azrin ST: Research and services partnerships: partnership: a fundamental component of dissemination and implementation research. Psychiatr Serv 2013, 64:509-511.
• [7]Chamberlain P, Brown CH, Saldana L: Observational measure of implementation progress in community based settings: the stages of implementation completion (SIC). Implement Sci 2011, 6:116-123.
• [8]Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F: Implementation Research: A Synthesis of the Literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network (FMHI Publication #231, Tampa; 2005.
• [9]Tabak RG, Khoong EC, Chambers DB, Brownson RC: Models in dissemination and implementation research: useful tools in public health services and systems research. Frontiers in Public Health Services and Systems Research 2013, 2:8.
• [10]Chambers DA, Glasgow RE, Stange KC: The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci 2013, 8:117.
• [11]Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004, 82:581-629.
• [12]Stirman SW, Kimberly J, Cook N, Calloway A, Castro F: The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci 2012, 7:12.
• [13]Klein KJ, Conn AB, Sorra JS: Implementing computerized technology: an organizational analysis. J Appl Psychol 2001, 86:811-824.
• [14]Pluye P, Potvin L, Denis JL: Making public health program last: conceptualizing sustainability. Eval Program Plann 2004, 24:121-133.
• [15]Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey R, Hensley M: Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Hlth 2011, 38:65-76.
• [16]Mendel P, Meredith L, Schoenbaum M, Sherbourne C, Wells K: Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Hlth 2008, 35:21-37.
• [17]Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B: State implementation of evidence-based practice for youths, part II: recommendations for research and policy. J Am Acad Child Adolesc Psychiatry 2008, 47:499-504.
• [18]Feldman S: The ‘millionaires tax’ and mental health policy in California. Health Aff 2009, 28:809-815.
• [19]Raghavan R, Bright CL, Shadoin AL: Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci 2008, 3:3-26.
• [20]Aarons GA, Ehrhart MG, Farahnak LR, Sklar M: Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health 2014, 35:255-274.
• [21]Aarons GA: Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am 2005, 14:255-271.
• [22]Aarons GA, Sommerfeld DH, Walrath-Greene CM: Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci 2009, 4:83-96.
• [23]Caldwell DF, O’Reilly CA III: The determinants of team-based innovation in organizations: the role of social influence. Small Group Res 2003, 34:497-517.
• [24]Willging CE, Lamphere L, Waitzkin H: Transforming administrative and clinical practice in a public behavioral health system: an ethnographic assessment of the context of change. Journal of Health Care for the Poor and the Underserved 2009, 20:873-890.
• [25]Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Shackelford J, Schoenwald SK: Statewide adoption and initial implementation of contingency management for substance abusing adolescents. J Consult Clin Psychol 2008, 76:556-567.
• [26]Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, Hurford MO, Hadley T, Barg F, Walsh LM, Adams D, Mandell D: Individual and organizational predictors of community therapists’ use of evidence-based practices in a large public mental health system.JAMA Psychiatry, in press.
• [27]Glisson C: The organizational context of children’s mental health services. Clin Child Fam Psych 2002, 5:233-253.
• [28]Aarons GA, Sawitzky AC: Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychol Serv 2006, 3:61-72.
• [29]Aarons GA: Mental health provider attitudes toward adoption of evidence-based practice: the evidence-based practice attitude scale (EBPAS). Ment Health Serv Res 2004, 6:61-74.
• [30]Ehrhart MG: Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Pers Psychol 2004, 57:61-94.
• [31]Schneider B, Ehrhart MG, Mayer DM, Saltz JL, Niles-Jolly K: Understanding organization-customer links in service settings. Acad Manage J 2005, 48:1017-1032.
• [32]Ehrhart MG, Aarons GA, Farahnak LR: Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci 2014, 9:157.
• [33]Aarons GA: Transformational and transactional leadership: association with attitudes toward evidence-based practice. Psychiatr Serv 2006, 57:1162-1169.
• [34]Frambach RT, Schillewaert N: Organizational innovation adoption: a multi-level framework of determinants and opportunities for future research. Journal of Business Research Special Issue: Marketing Theory in the Next Millennium 2002, 55:163-176.
• [35]Jacobs SR, Weiner BJ, Bunger AC: Context matters: measuring implementation climate among individuals and groups. Implement Sci 2014, 9:46.
• [36]Butterfoss FDB: Coalitions and Partnerships in Community Health. Jossey-Bass, Hoboken, NJ; 2007.
• [37]Wallerstein N, Duran B: The conceptual, historical, and practice roots of community based participatory research and related participatory traditions. In Community-based Participatory Research for Health. Edited by Minkler M, Wallerstein N. Jossey-Bass, San Francisco; 2003.
• [38]Wallerstein N, Duran B: Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health 2010, 100:S40-S46.
• [39]Wells K, Jones L: Research in community-partnered, participatory research. JAMA 2009, 302:320-321.
• [40]Widdus R: Public-private partnerships for health: their main targets, their diversity, and their future directions. Bull World Health Organ 2001, 79:713-720.
• [41]Garland AF, Hurlburt MS, Hawley KM: Examining psychotherapy processes in a services research context. Clin Psychol-Sci Pr 2006, 13:30-46.
• [42]Wells K, Miranda J, Bruce ML, Alegria M, Wallerstein N: Bridging community intervention and mental health services research. Am J Psychiatry 2004, 161:955-963.
• [43]Institute of Medicine [IOM]: Improving the Quality of Health Care for Mental and Substance-use Conditions. National Academies Press, Washington DC; 2006.
• [44]Chaffin M, Hecht D, Bard D, Silovsky JF, Beasley WH: A statewide trial of the SafeCare home-based services model with parents in Child Protective Services. Pediatrics 2012, 129:509-515.
• [45]Swenson CC, Brown EJ, Lutzker JR: Issues of maltreatment and abuse. In Personality Disorders in Childhood and Adolescence. Edited by Freeman A. John Wiley & Sons Inc, Hoboken, NJ; 2007:229-295.
• [46]Bousha DM, Twentyman T: Mother-child interactional style in abuse, neglect, and control groups: naturalistic observations in the home. J Abnorm Psychol 1984, 93:106-114.
• [47]Crittenden PM: Children’s strategies for coping with adverse home environments: an interpretation using attachment theory. Child Abuse Negl 1992, 16:329-343.
• [48]Egeland B, Sroufe A, Erickson M: The developmental consequence of different patterns of maltreatment. Child Abuse Negl 1983, 7:459-469.
• [49]Kent J: A follow-up study of abused children. J Pediatr Psychol 1976, 1:25-31.
• [50]Culp RE, Watkins RV, Lawrence H, Letts D, Kelly DJ, Rice ML: Maltreated children’s language and speech development: abused, neglected, and abused and neglected. First Language 1991, 11:377-389.
• [51]Fox L, Long SH, Langlois A: Patterns of language comprehension deficit in abused and neglected children. Journal of Speech & Hearing Disorders 1988, 53:239-244.
• [52]Gaudin JM: Short-term and long-term outcomes. In Neglected Children: Research, Practice and Policy. Edited by Dubowitz H. Sage, London; 1999:89-108.
• [53]Widom CS, White HR: Problem behaviors in abused and neglected children grown-up: prevalence and co-occurrence of substance abuse, crime and violence. Criminal Behavior & Mental Health 1998, 7:287-310.
• [54]U.S. Department of Health and Human Services, Administration for Children and Families, Administration on Children, Youth and Families: Child Maltreatment. Children’s Bureau, Washington, DC; 2010.
• [55]Lutzker JR, Bigelow KM, Doctor RM, Kessler ML: Safety, health care, and bonding within an ecobehavioral approach to treating and preventing child abuse and neglect. J Fam Violence 1998, 13:163-185.
• [56]Whitaker DJ, Lutzker JR, Self-Brown S, Edwards AE: Implementing an evidence-based program for the prevention of child maltreatment: the SafeCare® program. Report on Emotional & Behavioral Disorders in Youth 2008, 8:55-62.
• [57]Wright C, Catty J, Watt H, Burns T: A systematic review of home treatment services. Soc Psychiatry Psychiatr Epidemiol 2004, 39:789-796.
• [58]Fagen MC, Flay BR: Sustaining a school-based prevention program: results from the Aban Aya sustainability project. Health Educ Behav 2009, 36:9-23.
• [59]Ozer E, Cantor J, Cruz G, Fox B, Hubbard E, Moret L: The diffusion of youth-led participatory research in urban schools: the role of the prevention support system in implementation and sustainability. Am J Community Psychol 2008, 41:278-289.
• [60]Elias MJ, Zins JE, Graczyk PA, Weissberg RP: Implementation, sustainability and scaling up of social-emotional and academic innovations in public schools. School Psych Rev 2003, 32:303-319.
• [61]Foster EM, Kelsch CC, Kamradt B, Sosna T, Yang Z: Expenditures and sustainability in systems of care. J Emot Behav Disord 2001, 9:53-62.
• [62]Jacobs JA, Dodson EA, Baker EA, Deshpande AD, Brownson RC: Barriers to evidence-based decision making in public health: a national survey of chronic disease practitioners. Public Health Rep 2010, 125:736-742.
• [63]Grimes J, Kurns S, Tilly WDI: Sustainability: an enduring commitment to success. School Psych Rev 2006, 35:224-244.
• [64]Larsen T, Samdal O: Implementing second step: balancing fidelity and program adaptation. J Educ Psychol Consult 2007, 17:1-29.
• [65]Jansen M, Harting J, Ebben N, Kroon B, Stappers J, Van Engelshoven E, de Vries N: The concept of sustainability and the use of outcome indicators. A case study to continue a successful health counselling intervention. Fam Pract 2008, 25:32-37.
• [66]Baumann A, Hunsberger M, Blythe J, Crea M: Sustainability of the workforce: government policies and the rural fit. Health Policy 2008, 85:372-379.
• [67]Myers KM, Valentine JM, Melzer SM: Feasibility, acceptability, and sustainability of telepsychiatry for children and adults. Psychiatr Serv 2007, 58:1493-1496.
• [68]Aarons GA, Sommerfeld DH: Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. J Am Acad Child Psy 2012, 51:423-431.
• [69]Aarons GA, Sommerfeld DH, Hecht DB, Silovsky JF, Chaffin MJ: The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for a protective effect. J Consult Clin Psychol 2009, 77:270-280.
• [70]Aarons GA, Fettes DL, Sommerfeld DH, Palinkas LA: Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreat 2012, 17:67-79.
• [71]Hurlburt M, Aarons GA, Fettes DL, Willging C, Gunderson L, Chaffin MJ: Interagency collaborative team model for capacity building to scale-up evidence-based practice. Child Youth Serv Rev 2014, 39:160-168.
• [72]Aarons GA, Fettes DL, Hurlburt MS, Palinkas LA, Gunderson L, Willging CE, Chaffin MJ: Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice.J Clin Child Adolesc 2014, pp.1–14. Advance online publication.
• [73]Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Silovsky JF, Hecht DB, Chaffin MJ: Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci 2012, 7:1-9.
• [74]Waitzkin H, Williams R, Bock J, McCloskey J, Willging C, Wagner W: Safety-net institutions buffer the impacts of medicaid managed care: a multi-method assessment in a rural state. Am J Public Health 2002, 92:598-610.
• [75]Willging C, Waitzkin H, Wagner W: Medicaid managed care for mental health services in a rural state. J Health Care Poor Underserved 2005, 16:497-514.
• [76]Stufflebeam DL, Shinkfield AJ: Evaluation Theory, Models, and Application. Jossey-Bass, San Francisco, CA; 2007.
• [77]Patton MQ: Qualitative analysis and interpretation. In Qualitative Research and Evaluation Methods. 3rd edition. Sage Publications, Thousand Oaks, CA; 2002:431-539.
• [78]Miles MB, Huberman AM: Qualitative Data Analysis: An Expanded Sourcebook. Sage Publications, Inc., Thousand Oaks, CA; 1994.
• [79]Fraser D: QSR NVivo Reference Guide. QSR International, Melbourne; 2000.
• [80]Steckler A, Goodman RM, McLeroy KR, Davis S, Koch G: Measuring the diffusion of innovative health promotion programs. Am J Health Promot 1992, 6:214-225.
• [81]Kano M, Willging C, Rylko-Bauer B: Community participation in New Mexico’s behavioral health care reform. Med Anthropol Q 2009, 23:277-297.
• [82]Emerson RM, Fretz RI, Shaw LL: Writing Ethnographic Fieldnotes. University of Chicago, Chicago; 1996.
• [83]Schoenwald SK, Sheidow AJ, Letourneau EJ: Toward effective quality assurance in evidence-based practice: links between expert consultation, therapist fidelity, and child outcomes. J Clin Child Adolesc 2004, 33:94-104.
• [84]Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood K, Mayberg S, Green P: Assessing the Organizational Social Context (OSC) of mental health services: implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research Special Issue: Improving Mental Health Services 2008, 35:98-113.
• [85]Bass BM, Avolio BJ: MLQ: Multifactor Leadership Questionnaire (Technical Report). Center for Leadership Studies, Binghamton University, NY; 1995.
• [86]Aarons GA, Ehrhart MG, Farahnak LR: The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci 2014, 9:45.
• [87]Kraut RE, Rice RE, Cool C, Fish RS: Varieties of social influence: the role of utility and norms in the success of a new communication medium. Organ Sci 1998, 9:437-453.
• [88]Hellofs LL, Jacobson R: Market share and customers’ perceptions of quality: when can firms grow their way to higher versus lower quality? J Mark 1999, 63:16-25.
• [89]Aarons GA, McDonald EJ, Sheehan AK, Walrath-Greene CM: Confirmatory factor analysis of the evidence-based practice attitude scale (EBPAS) in a geographically diverse sample of community mental health providers. Adm Policy Ment Hlth 2007, 34:465-469.
• [90]Aarons GA, Cafri G, Lugo L, Sawitzky A: Expanding the domains of attitudes towards evidence-based practice: the Evidence-based Practice Attitude Scale-50. Adm Policy Ment Hlth 2010, 39:331-340.
• [91]Mancini JA, Marek LI: Sustaining community-based programs for families: conceptualization and measurement. Fam Relat 2004, 53:339-347.
• [92]Goodman RM, McLeroy KR, Steckler AB, Hoyle RH: Development of level of institutionalization. Health Educ Q 1993, 20:161-178.
• [93]Brown CH, Wang W, Kellam SG, Muthen BO, Petras H, Toyinbo P, Poduska J, Ialongo N, Wyman PA, Chamberlain P, Sloboda Z, MacKinnon DP, Windham A: the Prevention Science and Methodology Group: Methods for testing theory and evaluating impact in randomized field trials: intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug Alcohol Depend 2008, 95:74-104.
• [94]Hedeker D, Gibbons RD: Longitudinal Data Analysis. 1st edition. Wiley-Interscience, Hoboken, NJ; 2006.
• [95]McCulloch CE, Searle SR: Generalized, Linear, and Mixed Models. Wiley, New York; 2001.
• [96]Raudenbush SW, Bryk AS: Hierarchical Linear Models: Applications and Data Analysis Methods. Sage Publications, Thousand Oaks, CA; 2002.
• [97]Singer JD, Willet JB: Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press, New York; 2003.
• [98]Hedeker D: An introduction to growth modeling. In The Sage Handbook of Quantitative Methodology for the Social Sciences. Edited by Kaplan D. Sage Publications, Thousand Oaks, CA; 2004.
• [99]Creswell JW, Plano Clark VL: Designing and Conducting Mixed Methods Research. Sage Publications, Thousand Oaks, CA; 2007.
• [100]Tashakkori A, Teddlie C: Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. Int J Social Res Methodol 2003, 6:61-77.
• [101]Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J: Mixed method designs in implementation research. Adm Policy Ment Hlth 2011, 38:44-53.
• [102]Teddlie C, Tashakkori A: Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Handbook of Mixed Methods in Social and Behavioral Research. Sage Publications, Thousand Oaks, CA; 2003:3-50.
• [103]Phillips JJ: Return on Investment in Training and Performance Improvement Programs. Routledge, New York, NY; 2003.
• [104]McHugo GJ, Drake RE, Whitley R, Bond GR, Campbell K, Rapp CA, Goldman HH, Lutz WJ, Finnerty MT: Fidelity outcomes in the national implementing evidence-based practices project. Psychiatr Serv 2007, 58:1279-1284.
• [105]Kendall PC, Beidas RS: Smoothing the trail for dissemination of evidence-based practices for youth: flexibility within fidelity. Prof Psychol: Res Pract 2007, 38:13-20.