Context matters: measuring implementation climate among individuals and groups

Implementation Science

Alicia C Bunger²  Bryan J Weiner³  Sara R Jacobs¹ 

[1] Department of Health Policy and Management, Gillings School of Global Public Health, 1101 McGavran-Greenberg Hall, CB #7411, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599-7411, USA;College of Social Work, The Ohio State University, 325W Stillman Hall, 1947 College Road, Columbus, OH 43210, USA;Cecil G. Sheps Center for Health Services Research, 725 Martin Luther King Jr. Blvd., CB #7590, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599-7411, USA

关键词: Measurement of group level phenomenon;    Measurement of global constructs;    Organizational context;    Implementation climate;   

Others  :  803927
DOI  :  10.1186/1748-5908-9-46

 received in 2013-08-09, accepted in 2014-04-14,  发布年份 2014

【 摘 要 】

Background

It has been noted that implementation climate is positively associated with implementation effectiveness. However, issues surrounding the measurement of implementation climate, or the extent to which organizational members perceive that innovation use is expected, supported and rewarded by their organization remain. Specifically, it is unclear whether implementation climate can be measured as a global construct, whether individual or group-referenced items should be used, and whether implementation climate can be assessed at the group or organizational level.

Methods

This research includes two cross-sectional studies with data collected via surveys at the individual level. The first study assessed the implementation climate perceptions of physicians participating in the National Cancer Institute’s (NCI) Community Clinical Oncology Program (CCOP), and the second study assessed the perceptions of children’s behavioral health clinicians implementing a treatment innovation. To address if implementation climate is a global construct, we used confirmatory factor analysis. To address how implementation climate should be measured and at what level, we followed a five-step framework outlined by van Mierlo and colleagues. This framework includes exploratory factor analysis and correlations to assess differences between individual and group-referenced items and intraclass correlations, interrater agreements, and exploratory factor analysis to determine if implementation climate can be assessed at the organizational level.

Results

The confirmatory factor analysis demonstrated that implementation climate is a global construct consisting of items related to expectations, support and rewards. There are mixed results, however, as to whether implementation climate should be measured using individual or group-referenced items. In our first study, where physicians were geographically dispersed and practice independently, there were no differences based on the type of items used, and implementation climate was an individual level construct. However, in the second study, in which clinicians practice in a central location and interact more frequently, group-referenced items may be appropriate. In addition, implementation climate could be considered an organizational level construct.

Conclusions

The results are context-specific. Researchers should carefully consider the study setting when measuring implementation climate. In addition, more opportunities are needed to validate this measure and understand how well it predicts and explains implementation effectiveness.

【 授权许可】

   
2014 Jacobs et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 4.

Figure 3.

【 参考文献 】

• [1]Klein KJ, Sorra JS: The challenge of innovation implementation. Acad Manage Rev 1996, 21:1055-1080.
• [2]Greenhalgh T: Diffusion of innovations in health service organisations: a systematic literature review. Malden, Mass: BMJ Books/Blackwell Pub; 2005.
• [3]Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009, 4:50. BioMed Central Full Text
• [4]Holahan PJ, Aronson ZH, Jurkat MP, Schoorman FD: Implementing computer technology: a multiorganizational test of Klein and Sorra’s model. J Eng Technol Manag 2004, 21:31-50.
• [5]Dong LY, Neufeld DJ, Higgins C: Testing Klein and Sorra’s innovation implementation model: an empirical examination. J Eng Technol Manag 2008, 25:237-255.
• [6]Osei-Bryson KM, Dong LY, Ngwenyama O: Exploring managerial factors affecting ERP implementation: an investigation of the Klein-Sorra model using regression splines. Inf Syst J 2008, 18:499-527.
• [7]Damschroder LM, Goodrich DE, Robinson CH, Fletcher CE, Lowery JC: A systematic exploration of differences in contextual factors related to implementing the MOVE! weight management program in the VA: a mixed methods study. BMC Health Serv Res 2011, 11:248. BioMed Central Full Text
• [8]Damschroder LM, Lowery JC: Evaluation of a large-scale weight management program using the consolidated framework for implementation research. Implement Sci 2013, 8:51. BioMed Central Full Text
• [9]Teal R, Bergmire DM, Johnston M, Weiner BJ: Implementing community-based provider participation in research: an empirical study. Implement Sci 2012, 7:41. BioMed Central Full Text
• [10]Weiner BJ, Haynes-Maslow L, Kahwati LC, Kinsinger L, Campbell MK: Implementing the MOVE! Weight- management program in the Veterans health administration, 2007-2010: a qualitative study. Prev Chronic Dis 2012, 9:E16.
• [11]Helfrich CD, Weiner BJ, McKinney MM, Minasian L: Determinants of implementation effectiveness: exploring an adapted model for complex innovations. Med Care Res Rev 2007, 64:279-303.
• [12]Weiner BJ, Belden CM, Bergmire DM, Johnston M: The meaning and measurement of implementation climate. Implement Sci 2011, 6:78. BioMed Central Full Text
• [13]Klein KJ, Conn AB, Sorra JS: Implementing computerized technology: an organizational analysis. J Appl Psychol 2001, 86:811-824.
• [14]Glick WH: Conceptualizing and measuring organizational and psychological climate - pitfalls in multilevel research. Acad Manag Rev 1985, 10:601-616.
• [15]Glisson C, James LR: The cross-level effects of culture and climate in human service teams. J Organ Behav 2002, 23:767-794.
• [16]Glisson C, Green P, Williams NJ: Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl 2012, 36:621-632.
• [17]Glisson C, Green P: Organizational climate, services, and outcomes in child welfare systems. Child Abuse Negl 2011, 35:582-591.
• [18]Glisson C, Dukes D, Green P: The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse Negl 2006, 30:855-880.
• [19]Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA, Research Network on Youth Mental Health: The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implement Sci 2012, 7:56. BioMed Central Full Text
• [20]Klein KJ, Conn AB, Smith DB, Sorra JS: Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. J Appl Psychol 2001, 86:3-16.
• [21]Baltes BB, Zhdanova LS, Parker CP: Psychological climate: a comparison of organizational and individual level referents. Human Relations 2009, 62:669-700.
• [22]Minasian LM, Carpenter WR, Weiner BJ, Anderson DE, McCaskill-Stevens W, Nelson S, Whitman C, Kelaghan J, O’Mara AM, Kaluzny AD: Translating research into evidence-based practice: the National Cancer Institute Community Clinical Oncology Program. Cancer 2010, 116:4440-4449.
• [23]Kaluzny AD, Lacey LM, Warnecke R, Morrissey JP, Sondk EJ, Ford L: Accrual of patients to randomized clinical trials. Factors affecting cancer prevention and control research. Int J Technol Assess Health Care 1994, 10:506-516.
• [24]Carpenter WR, Fortune-Greeley AK, Zullig LL, Lee SY, Weiner BJ: Sustainability and performance of the National Cancer Institute's Community Clinical Oncology Program. Contemp Clin Trials 2012, 33:46-54.
• [25]McKinney MM, Weiner BJ, Carpenter WR: Building community capacity to participate in cancer prevention research. Cancer Control 2006, 13:295-302.
• [26]McKinney MM, Weiner BJ, Wang V: Recruiting participants to cancer prevention clinical trials: lessons from successful community oncology networks. Oncol Nurse Forum 2006, 33:951-959.
• [27]Cohen JA, Mannarino AP, Knudsen K: Treating sexually abused children: 1 year follow-up of a randomized controlled trial. Child Abuse Negl 2005, 29:135-145.
• [28]Deblinger E, Stauffer LB, Steer RA: Comparative efficacies of supportive and cognitive behavioral group therapies for young children who have been sexually abused and their nonoffending mothers. Child Maltreat 2001, 6:332-343.
• [29]Cary CE, McMillen JC: The data behind the dissemination: a systematic review of trauma-focused cognitive behavioral therapy for use with children and youth. Child Youth Serv Rev 2012, 34:748-757.
• [30]Kilo CM: A framework for collaborative improvement: lessons from the institute for healthcare improvement’s breakthrough series. Qual Manag Health Care 1998, 8:1-13.
• [31]Markiewicz J, Ebert L, Ling D, Amaya-Jackson L, Kisiel C: Learning Collaborative Toolkit. Los Angeles, CA, and Durham, NC: National Center for Child Traumatic Stress; http://www.nctsn.org/sites/default/files/assets/pdfs/lc/Module_1.pdf webcite
• [32]van Mierlo H, Vermunt JK, Rutte CG: Composing group-level constructs from individual-level survey data. Organ Res Methods 2009, 12:368-392.
• [33]StataCorp: Stata Statistical Software: Release 12. College Station, TX: StataCorp LP; 2009.
• [34]Norman GR, Streiner DL (Eds): Chapter 17: Path Analysis and Structural Equation Modeling In PDQ Statistics. 3rd edition. London: BC Decker Inc; 2003:156-176.
• [35]Hox JJ, Bechger TM: An introduction to structural equation modeling. Fam Sci Rev 2007, 11:354-373.
• [36]Schreiber JB, Stage FK, King J: Reporting structural equation modeling and confirmatory factor analysis results: a review. J Educ Res 2006, 99:323-337.
• [37]Lance CE, Butts MM, Michels LC: The sources of four commonly reported cutoff criteria - What did they really say? Organ Res Methods 2006, 9:202-220.
• [38]Bliese PD: Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Multilevel theory, research, and methods in organizations. Edited by Klein KJ, Kozlowski SWJ. San Francisco: Jossey-Bass; 2000:349-381.
• [39]Cohen A, Doveh E, Eick U: Statistical properties of the rwg(J) index of agreement. Psychol Methods 2001, 6:297-310.
• [40]Dunlap WP, Burke MJ, Smith-Crowe K: Accurate tests of statistical significance for RWG and average deviation interrater agreement indexes. J Appl Psychol 2003, 88:356-362.
• [41]Dansereau F, Yammarino FJ: Is more discussion about levels of analysis really necessary? When is such discussion sufficient? Leadership Quarterly 2003, 17:537-552.
• [42]Altmann R, Huff J, Baltes B, LaCost H, Young S, Parker C: Psychological and organizational climate perceptions: A field experiment of a contemporary distinction. Dallas, TX: The Thirteenth Annual Meeting of the Society for Industrial Organizational Psychology; 1998.
• [43]Castro SL: Data analytic methods for the analysis of multilevel questions: a comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling. Leadersh Q 2002, 13:69-93.
• [44]Gagne P, Hancock GR: Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivar Behav Res 2006, 41:65-83.