The practice of ‘doing’ evaluation: lessons learned from nine complex intervention trials in action

Implementation Science

Clare IR Chandler⁸  Catherine Goodman⁸  Virginia Wiseman⁸  Sarah G Staedke⁵  Bonnie Cundill⁴  David Schellenberg²  David G Lalloo⁶  Hugh Reyburn²  Eleanor Hutchinson⁸  Toby Leslie²  Shunmay Yeung⁸  Lasse S Vestergaard⁷  Jayne Webster²  Katia Bruxvoort⁸  Hilda Mbakilwa^1,10  Sham Lal²  Evelyn K Ansah³  Lindsay Mangham-Jefferies⁸  Deborah DiLiberto¹  Joanna Reynolds⁹ 

[1] Department of Medical Statistics, London School of Hygiene & Tropical Medicine, Keppel St, London WC1E 7HT, UK;Disease Control Department, London School of Hygiene & Tropical Medicine, Keppel St, London WC1E 7HT, UK;Dangme West District Health Directorate, Ghana Health Service, PO Box DD1, Dodowa, Ghana;Department of Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine, Keppel St, London WC1E 7HT, UK;Department of Clinical Research, London School of Hygiene & Tropical Medicine, Keppel St, London WC1E 7HT, UK;Department of Clinical Sciences, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK;Department of Infectious Diseases, Copenhagen University Hospital, Copenhagen K, Denmark;Department of Global Health and Development, London School of Hygiene & Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UK;Department of Social and Environmental Health Research, London School of Hygiene & Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UK;Joint Malaria Programme, Moshi, Tanzania

关键词: Trials;    Reflection;    Low-income setting;    Health service;    Behavioural interventions;    Evaluation;    Complex interventions;   

Others  :  800941
DOI  :  10.1186/1748-5908-9-75

 received in 2014-05-15, accepted in 2014-06-13,  发布年份 2014

【 摘 要 】

Background

There is increasing recognition among trialists of the challenges in understanding how particular ‘real-life’ contexts influence the delivery and receipt of complex health interventions. Evaluations of interventions to change health worker and/or patient behaviours in health service settings exemplify these challenges. When interpreting evaluation data, deviation from intended intervention implementation is accounted for through process evaluations of fidelity, reach, and intensity. However, no such systematic approach has been proposed to account for the way evaluation activities may deviate in practice from assumptions made when data are interpreted.

Methods

A collective case study was conducted to explore experiences of undertaking evaluation activities in the real-life contexts of nine complex intervention trials seeking to improve appropriate diagnosis and treatment of malaria in varied health service settings. Multiple sources of data were used, including in-depth interviews with investigators, participant-observation of studies, and rounds of discussion and reflection.

Results and discussion

From our experiences of the realities of conducting these evaluations, we identified six key ‘lessons learned’ about ways to become aware of and manage aspects of the fabric of trials involving the interface of researchers, fieldworkers, participants and data collection tools that may affect the intended production of data and interpretation of findings. These lessons included: foster a shared understanding across the study team of how individual practices contribute to the study goals; promote and facilitate within-team communications for ongoing reflection on the progress of the evaluation; establish processes for ongoing collaboration and dialogue between sub-study teams; the importance of a field research coordinator bridging everyday project management with scientific oversight; collect and review reflective field notes on the progress of the evaluation to aid interpretation of outcomes; and these approaches should help the identification of and reflection on possible overlaps between the evaluation and intervention.

Conclusion

The lessons we have drawn point to the principle of reflexivity that, we argue, needs to become part of standard practice in the conduct of evaluations of complex interventions to promote more meaningful interpretations of the effects of an intervention and to better inform future implementation and decision-making.

【 授权许可】

   
2014 Reynolds et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Pawson R: Evidence-Based Policy. A Realist Perspective. London: Sage; 2006.
• [2]Peterson S: Assessing the scale-up of child survival interventions. Lancet 2010, 375:530-531.
• [3]Medical Research Council: A Framework For The Development And Evaluation Of RCTs For Complex Interventions To Improve Health. London: Medical Research Council; 2000.
• [4]Medical Research Council: Developing And Evaluating Complex Interventions: New Guidance. London: Medical Research Council; 2008.
• [5]Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P: Framework for design and evaluation of complex interventions to improve health. BMJ 2000, 321:694-696.
• [6]Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new medical research council guidance. BMJ 2008, 337:979-983.
• [7]Nazareth I, Freemantle N, Duggan C, Mason J, Haines A: Evaluation of a complex intervention for changing professional behaviour: the evidence based out reach (EBOR) trial. J Health Serv Res Policy 2002, 7:230-238.
• [8]Bonell CP, Hargreaves J, Cousens S, Ross D, Hayes R, Petticrew M, Kirkwood BR: Alternatives to randomisation in the evaluation of public health interventions: design challenges and solutions. J Epidemiol Community Health 2011, 65:582-587.
• [9]Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R: Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference. J Epidemiol Community Health 2011, 65:576-581.
• [10]Munro A, Bloor M: Process evaluation: the new miracle ingredient in public health research? Qual Res 2010, 10:699-713.
• [11]Wells M, Williams B, Treweek S, Coyle J, Taylor J: Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials 2012, 13:95.
• [12]Saunders RP, Evans MH, Joshi P: Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005, 6:134-147.
• [13]Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team: Process evaluation in randomized controlled trials of complex interventions. BMJ 2006, 332:413-416.
• [14]Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013, 14:1-10.
• [15]Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, Baird J: Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Commun Health 2014, 68:101-102.
• [16]Donovan J, Little P, Mills N, Smith M, Brindle L, Jacoby A, Peters T, Frankel S, Neal D, Hamdy F: Improving design and conduct of randomised trials by embedding them in qualitative research: protecT (prostate testing for cancer and treatment) study. BMJ 2002, 325:766-769.
• [17]Murtagh M, Thomson R, May C, Rapley T, Heaven B, Graham R, Kaner E, Stobbart L, Eccles M: Qualitative methods in a randomised controlled trial: the role of an integrated qualitative process evaluation in providing evidence to discontinue the intervention in one arm of a trial of a decision support tool. Qual Saf Health Care 2007, 16:224-229.
• [18]Koivisto J: What evidence base? Steps towards the relational evaluation of social interventions. Evid Pol 2007, 3:527-537.
• [19]National Institute for Health Research: Clinical Trials Tool Kit. [http://www.ct-toolkit.ac.uk/ webcite]
• [20]Petryna A: Ethical variability: drug development and globalizing clinical trials. Am Ethnol 2005, 32:183-197.
• [21]Kelly AH, Ameh D, Majambere S, Lindsay S, Pinder M: ‘Like sugar and honey’: the embedded ethics of a larval control project in the Gambia. Soc Sci Med 2010, 70:1912-1919.
• [22]Switula D: Principles of good clinical practice (GCP) in clinical research. Sci Eng Ethics 2000, 6:71-77.
• [23]European Medicines Agency: ICH Harmonised Tripartite Guideline E6: Note for Guidance on Good Clinical Practice (CPMP/ICH/135/95). London: European Medicines Agency; 2002. [http://ichgcp.net/pdf/ich-gcp-en.pdf webcite]
• [24]Lang T, Chilengi R, Noor RA, Ogutu B, Todd JE, Kilama WL, Targett GA: Data safety and monitoring boards for African clinical trials. Trans R Soc Trop Med Hyg 2008, 102:1189-1194.
• [25]Lawton J, Jenkins N, Darbyshire J, Farmer A, Holman R, Hallowell N: Understanding the outcomes of multi-centre clinical trials: a qualitative study of health professional experiences and views. Soc Sci Med 2012, 74:574-581.
• [26]Chandler C, Jones C, Boniface G, Juma K, Reyburn H, Whitty C: Guidelines and mindlines: why do clinical staff over-diagnose malaria in Tanzania? A qualitative study. Malar J 2008, 7:53.
• [27]Whitty CJ, Chandler C, Ansah EK, Leslie T, Staedke S: Deployment of ACT antimalarials for treatment of malaria: challenges and opportunities. Malar J 2008, 7(Suppl 1):S7.
• [28]Mangham L, Cundill B, Ezeoke O, Nwala E, Uzochukwu B, Wiseman V, Onwujekwe O: Treatment of uncomplicated malaria at public health facilities and medicine retailers in south-eastern Nigeria. Malar J 2011, 10:155.
• [29]Godwin M, Ruhland L, Casson I, MacDonald S, Delva D, Birtwhistle R, Lam M, Seguin R: Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol 2003, 3:28.
• [30]Stake R: The Art of Case Study Research. London: Sage Publications Ltd.; 1995.
• [31]Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A: The case study approach. BMC Med Res Methodol 2011, 11:100.
• [32]Pope C, Ziebland S, Mays N: Qualitative research in health care. Analysing qualitative data. BMJ 2000, 320:114-116.
• [33]Biruk C: Seeing like a research project: producing ‘high-quality data’ in AIDS research in Malawi. Med Anthropol 2012, 31:347-366.
• [34]Helgesson C-F: From Dirty Data To Credible Scientific Evidence: Some Practices Used To Clean Data In Large Randomised Clinical Trials. In Medical Proofs, Social Experiments: Clinical Trials in Shifting Contexts. Edited by Will C, Moreira T. Farnham, UK: Ashgate; 2010:49-64.
• [35]Molyneux S, Kamuya D, Madiega PA, Chantler T, Angwenyi V, Geissler PW: Field workers at the interface. Dev World Bioeth 2013, 13:ii-iv.
• [36]Blaise P, Kegels G: A realistic approach to the evaluation of the quality management movement in health care systems: a comparison between European and African contexts based on Mintzberg’s organizational models. Int J Health Plann Manage 2004, 19:337-364.
• [37]Baer AR, Zon R, Devine S, Lyss AP: The clinical research team. J Oncol Pract 2011, 7:188-192.
• [38]Barry C, Britten N, Barber N, Bradley C, Stevenson F: Using reflexivity to optimize teamwork in qualitative research. Qual Health Res 1999, 9:26-44.
• [39]Lawton J, Jenkins N, Darbyshire J, Holman R, Farmer A, Hallowell N: Challenges of maintaining research protocol fidelity in a clinical care setting: a qualitative study of the experiences and views of patients and staff participating in a randomized controlled trial. Trials 2011, 12:108.
• [40]Audrey S, Holliday J, Parry-Langdon N, Campbell R: Meeting the challenges of implementing process evaluation within randomized controlled trials: the example of ASSIST (a stop smoking in schools trial). Health Educ Res 2006, 21:366-377.
• [41]Clarke D, Hawkins R, Sadler E, Harding G, Forster A, McKevitt C, Godfrey M, Monaghan J, Farrin A: Interdisciplinary health research: perspectives from a process evaluation research team. Qual Prim Care 2012, 20:179-189.
• [42]Lang TA, White NJ, Hien TT, Farrar JJ, Day NPJ, Fitzpatrick R, Angus BJ, Denis E, Merson L, Cheah PY, Chilengi R, Kimutai R, Marsh K: Clinical research in resource-limited settings: enhancing research capacity and working together to make trials less complicated. PLoS Negl Trop Dis 2010, 4:e619.
• [43]Speed C, Heaven B, Adamson A, Bond J, Corbett S, Lake AA, May C, Vanoli A, McMeekin P, Moynihan P, Rubin G, Steen IN, McColl E: LIFELAX - diet and LIFEstyle versus LAXatives in the management of chronic constipation in older people: randomised controlled trial. Health Technol Assess 2010, 14:1-251.
• [44]Krishnankutty B, Bellary S, Kumar NB, Moodahadu LS: Data management in clinical research: an overview. Indian J Pharmacol 2012, 44:168-172.
• [45]Brandt CA, Argraves S, Money R, Ananth G, Trocky NM, Nadkarni PM: Informatics tools to improve clinical research study implementation. Contemp Clin Trials 2006, 27:112-122.
• [46]Sanjek R (Ed): Fieldnotes: the making of anthropology. Ithaca, New York: Cornell University Press; 1990.
• [47]Finlay L: “Outing” the researcher: the provenance, process, and practice of reflexivity. Qual Health Res 2002, 12:531-545.
• [48]Barnes BR: The Hawthorne effect in community trials in developing countries. Int J Soc Res Methodol 2010, 13:357-370.
• [49]Will CM: The alchemy of clinical trials. Biosocieties 2007, 2:85-99.
• [50]Hesse-Biber S: Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qual Inq 2012, 18:876-889.
• [51]Hertz R: Introduction: Reflexivity And Voice. In Reflexivity And Voice. Edited by Hertz R. Thousand Oaks, C.A: SAGE Publications Inc; 1997:vii-xviii.
• [52]Gough B: Deconstructing Reflexivity. In Reflexivity: A Practical Guide for Researchers in Health and Social Sciences. Edited by Finlay L, Gough B. Oxford: Blackwell Publishing Company; 2003:21-36.
• [53]Moreira T, Will C: Conclusion: So What? In Medical Proofs, Social Experiments: Clinical Trials in Shifting Contexts. Edited by Will C, Moreira T. Farnham, UK: Ashgate; 2010:153-160.
• [54]Hawe P: The truth, but not the whole truth? Call for an amnesty on unreported results of public health interventions. J Epidemiol Community Health 2012, 66:285.
• [55]Marchal B, Dedzo M, Kegels G: A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Serv Res 2010, 10:24.