Fidelity, dosage, and quality of delivery in implementation of the Resilient intervention : The importance of a well-functioning support system for high-quality implementation

Abstract

Background and aim: By researching the complex phenomena of implementation fidelity, dosage, and quality of delivery in the intervention of the Resilient randomized controlled trial (RCT), the research in this thesis embarks on a multifaceted, multidimensional endeavor. Even though the science of implementation has progressed rapidly in educational sciences, there is still a research gap regarding the evaluation of the implementation of interventions. This thesis aims to reduce this research gap by exploring what works (and does not work) in the implementation of the Resilient intervention ROBUST (RI) through implementation and process evaluation (IPE). As part of curriculum renewal in Norway in autumn 2020, the interdisciplinary subject public health and life mastery was to be implemented in Norwegian schools. The Resilient project has sought to meet the needs of students, teachers, schools, and policy-makers by developing the ROBUST social and emotional learning (SEL) curriculum to aid this implementation process. The RI is an evidence-based preventive SEL intervention consisting of five core components, namely, social relations, mindfulness, problem solving, emotions, and growth mindset, conducted in real-world settings by non-researchers (teachers). The RI has been implemented in 24 lower secondary schools at the classroom level in 43 classrooms by 61 teachers, among 1108 students aged 13-15 years old in three municipalities. The overall theme of this thesis is fidelity, dosage, and quality of delivery of core components in the implementation of the RI, with an emphasis on the importance of a well-functioning support system for high-quality implementation. Methods: The complexity of the research questions necessitated the collection of several data sources; therefore, this thesis has a qualitatively driven convergent mixed methods parallel database design. This involves the collection of qualitative and quantitative data in a parallel manner, concurrently, at different time points throughout the IPE of the RI. As indicated, this thesis is qualitatively driven and involves open, non-participant observation, semi-structured focus group interviews, and semi-structured individual interviews and is supplemented with quantitative data from a digital survey with closed-ended questions from a web-based fidelity application (WFA). This thesis is article-based and consists of three studies. Study I used a qualitative multimethod approach with open, non-participant observation and a semi-structured focus group interview, with the aim of exploring facilitators’ fidelity to, and delivery of, planned preintervention training for teachers (TPT) as a precondition for teachers’ high-quality implementation in their own classrooms. Study II aimed to evaluate the usability of the WFA as a tool for collecting data on teachers’ fidelity to the planned implementation of the RI, driven by a qualitative multimethod involving a semi-structured focus group interview and semi-structured individual interviews. Study III completed the research project by complementing Study I and Study II with a quantitatively driven mixed methods approach, utilizing quantitative data from the WFA and semi-structured focus group interviews. Study III aimed to understand which factors may explain the variability in implementation quality across the fidelity and dosage of the five core components and the overall level of fidelity and dosage in the RI. The data were analyzed separately in each independent study and merged through points of integration and meta-inference at the concluding phase to draw conclusions from the discussion in this thesis. Results: Fidelity and implementation quality: The transition to digital delivery required adaptations, and fidelity to the planned structure was compromised as a result. Fidelity to core elements, delivery, and standardization in the RI support system contributed to the quality of the implementation of the TPT, especially in the digital environment. The efficient registration in the WFA, in addition to it being perceived to have a self-explanatory, logical structure, contributed to teachers’ fidelity to registration. Implementation quality was high on average for fidelity, meaning that the RI curricula were implemented with high quality despite registered variability. Dosage and implementation quality: The planned training dosage was too high for the digital format and led to fatigue and a decrease in motivation among participants and facilitators. The WFA was described as intuitive in providing a recognizable link between the lesson content and the resource book; thus, teachers found it easy to register. As a result, the high registration dosage provided a solid foundation for evaluating the implementation quality. Too much lesson content and too little time to implement all core components made it difficult for teachers to implement with high quality. Implementation quality is high on average for dosage, meaning that most of the RI curricula were implemented despite difficulties and registered variability. Quality of delivery of planned components in the support system and implementation quality: Facilitator’s delivery of the TPT was satisfactory, as the implementation plan was followed and all content was communicated to participants. However, the implementation quality was diminished, as the exercises and activities could not be modeled and practiced as planned through digital delivery. It was possible to follow teachers’ implementation delivery progress because the WFA is time-efficient and cost-effective. Teachers point out that the quality of facilitators’ delivery of TPT is a precondition for high-quality implementation in their own classrooms. Conclusions: The integration and meta-inferences of the findings show that the implementation quality of the RI is high in terms of both Fidelity and dosage, despite the unforeseen challenges brought about by the COVID-19 pandemic and the variability of fidelity and dosage in classrooms. The findings further show how a well-functioning support system positively impacted the facilitator’s delivery of the TPT, teachers’ delivery of intervention core components, and the overall implementation quality. Because the support system in the RI was well developed, the TPT withstood the transition to the digital environment. Additionally, the need to apply such diverse methodological approaches in this thesis to explore and understand the areas investigated confirms the complexity of implementation as a phenomenon. In conclusion, the qualitative focus on fidelity, dosage, and quality of delivery in the IPE of the RI stands out in the body of literature on implementation quality and SEL, providing valuable knowledge that is relevant to further research.