case study mixed method

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

70k Accesses

30 Citations

12 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

A Longitudinal Mixed Methods Case Study Investigation of the Academic, Athletic, Psychosocial and Psychological Impacts of Being of a Sport School Student Athlete

• Original Research Article
• Open access
• Published: 18 April 2024

Cite this article

You have full access to this open access article

• Ffion Thompson   ORCID: orcid.org/0000-0002-5515-7633 1 , 2 ,
• Fieke Rongen 3 ,
• Ian Cowburn 1 &
• Kevin Till 1 , 4  

278 Accesses

Explore all metrics

Sport schools are popular environments for simultaneously delivering education and sport to young people. Previous research suggests sport school involvement to have impact (i.e. the positive/negative, intended/unintended and long/short-term outcomes, results and effects) on student athlete’s holistic (i.e. academic, athletic, psychosocial and psychological) development. However, previous research is limited by (1) cross-sectional methods, (2) limited multidimensional assessments, (3) lack of consideration for athlete characteristics (e.g. sex) and (4) failure to evaluate how sport school features affect student-athlete impacts.

The study, using a mixed methods case study approach, aims to (1) longitudinally evaluate the impact of sport school involvement on the holistic development of student athletes, (2) evaluate the impact on holistic development by student-athlete characteristics and (3) explore the features and processes of the sport–school programme that drive/facilitate holistic impacts.

A longitudinal mixed methods design was employed across one full academic school year (33 weeks). Six data-collection methods (i.e. online questionnaire, physical fitness testing battery, academic assessment grades, log diaries, field notes/observation and timeline diagram/illustration) were used to assess the academic, athletic, psychosocial and psychological impacts for 72 student athletes from one sport school in the United Kingdom (UK).

Student athletes developed positive long-term holistic overall impacts (i.e. academically, athletically and personally), including maintaining stable and relatively high levels of sport confidence, academic motivation, general recovery, life skills, resilience and friends, family and free time scores. Despite positive impacts, juggling academic and sport workload posed challenges for student athletes, having the potential to lead to negative holistic impacts (e.g. fatigue, stress and injury). Positive and negative impacts were linked to many potential features and processes of the sport school (e.g. academic and athletic support services versus insufficient training load build-up, communication, coordination, flexibility and planning). Furthermore, when considering student-athlete characteristics, females had lower sport confidence, higher general stress and body image concerns and less general recovery than males and student athletes who played sport outside the school had lower general recovery.

Conclusions

This mixed method, longitudinal study demonstrated sport school involvement resulted in many positive academic (e.g. good grades), athletic (e.g. fitness development), psychosocial (e.g. enhanced confidence) and psychological (e.g. improved resilience) impacts attributed to the academic and athletic support services provided. However, juggling heavy academic and athletic workloads posed challenges leading to negative impacts including fatigue, pressure, stress and injury. Furthermore, holistic impacts may be sex dependent and further support may be required for female student athletes in sport school environments. Overall, these findings demonstrate the complex nature of combining education and sport commitments and how sport schools should manage, monitor and evaluate the features of their programme to maximise the holistic impacts of sport–school student athletes.

Similar content being viewed by others

The Impacts of Sports Schools on Holistic Athlete Development: A Mixed Methods Systematic Review

Ffion Thompson, Fieke Rongen, … Kevin Till

Time use and health and wellbeing outcomes of sport school students in Australia

Olivia Knowles, Paul B. Gastin & Peter Kremer

Students’ perceptions of learning environment and their leisure-time exercise in medical school: Does sport background matter?

Oksana Babenko, Amber Mosewich & Janelle Sloychuk

Avoid common mistakes on your manuscript.

1 Introduction

In response to the potential negative consequences associated with the intensification of youth sports programmes [e.g. 1 , 2 ] and the drive for a more holistic approach to youth athlete development [e.g. 3 ], there has been a cultural shift towards combining sport and education in supportive environments to appropriately prepare individuals for working life if they do not become professional athletes [ 4 ]. This type of approach is referred to as a ‘dual career’ (DC) approach (i.e. combining sporting pursuits alongside education or vocational endeavours). A DC approach has long been evident in the USA, where collegiate athletes pursue university education alongside elite performance in Olympic sports or before entering the draft system for professional sports (Ryba et al. 2015). However, it has recently become more prominent in the United Kingdom (UK, [ 5 ]). Morris et al. [ 6 ] further distinguishes between different dual career development environments (DCDEs; i.e. environments that support DC approaches) based on the different structures and approaches used to provide both athlete development and academic support.

One example of a DCDE that aims to cater for youth athletes’ holistic development is a sports school. Sport schools are a key environment for DC development in many countries and are considered an increasingly integral part of a nations’ elite sport performance strategy [ 7 ]. Sport schools aim to combine sport and education to offer student athletes considerable academic flexibility (e.g. adaptation of school and training schedules and lighter load by one subject) and athletic support (e.g. high-quality coaches and physiotherapy) [ 8 ]. Recently, Morris et al. [ 6 ] categorised two types of sport schools: sport-friendly and elite. Both sport-friendly schools and elite sport schools are situated in lower and upper general and vocational secondary education (i.e. International Standard Classification of Education level 2–5). However, unlike a sport-friendly school, an elite sport school has formal communication with a sport federation, often receiving funding [ 6 ].

While a DC approach holds promise for enhancing the development of school-aged athletes, it brings forth various potential challenges. These challenges include managing academic study and training alongside competition schedules, dealing with fatigue/lack of sleep and being forced to make personal sacrifices [ 9 , 10 , 11 ]. Consequently, despite the intention of sport schools to provide a platform for athletes to balance sport and education, the reality is that they introduce heightened demands, potentially subjecting student athletes to risks of burnout and injury, as identified in previous research on intensified youth sports (e.g. [ 1 , 12 ]).

The process of youth athletic development within a school is complex, as athletes experience psychological, physical and psychosocial growth in an environment where they are navigating competing sport, academic and social demands [ 13 ]. Consequently, sport school involvement will impact (i.e. the positive/negative, intended/unintended and long/short-term outcomes, results and effects) an individual’s holistic development across academic, athletic, psychosocial and psychological dimensions [ 3 , 14 ]. Recognizing the diverse and extensive potential impacts of DCDEs (such as sport schools), aligns with the overarching idea of examining student athletes holistically. This comprehensive perspective is vital in understanding and navigating the multifaceted impacts of sport schools on the developmental trajectory of individuals [ 3 , 14 ].

Increasingly, research has explored such impacts on holistic athlete development. A recent mixed methods systematic review [ 8 ] highlighted there are a multitude of immediate, short- and long-term positive (e.g. physical development, more stable levels of general health and well-being, status/popularity and life skills) and negative (e.g. lower higher education attainment, limited experience with ordinary life outside of competitive sport, high number of injuries and performance pressure) impacts associated with the athletic, academic, psychosocial and psychological development of sport school student athletes. However, this systematic review identified several limitations within the current evidence base, including: (1) limited research examining how sport-friendly school features are operationalised in different contexts (e.g. UK), (2) a failure to evaluate multi-dimensional domains of athlete impact, often focussing on one or two dimensions and (3) limited research evaluating how features affect athlete impacts (i.e. causal relationship between the characteristics and features of sport school and holistic athlete impacts).

Subsequently, two studies [ 15 , 16 ] assessed the impacts of a UK sport-friendly school on student athletes across all four domains of holistic athlete development (i.e. academic, athletic, psychosocial and psychological). Overall, the findings of both studies demonstrated a multitude of positive impacts associated with being a sport school student athlete but, also, impacts of concern. However, both studies were cross-sectional in nature (i.e. use of a single moment of measurement), where exposure and impacts were simultaneous. Consequently, these studies oppose the nature of ‘transition’ as a process and the dynamic nature of sport-friendly school environments. Therefore, longitudinal research designs are required to investigate student-athlete development or changes over time. Additionally, although Thompson et al. [ 15 , 16 ] provided a general overview of the features and multiple possible impacts of sport school involvement, it is important to note that not every athlete experienced every potential impact. Instead, impacts varied across individuals and were driven by their individual characteristics and experiences of sport school features over time. Sport schools would benefit from an approach that is aware of individual differences and how they may impact a student athlete’s journey. Accordingly, it is important to explore the specificity of athlete characteristics/variables (e.g. biological sex) as holistic impacts may vary considerably depending upon an athlete's sex, sport requirements and boarding status [ 17 , 18 , 19 ].

Finally, given the complex and dynamic nature of DC environments [ 20 ], where student athletes have to interact with coaches, programme culture and practices, research needs to explore the features and processes (i.e. the context-individual interactions) of sport-friendly school programmes that drive and facilitate positive and negative holistic impacts described by Thompson and colleagues [ 15 , 16 ]. Moreover, within the UK, there are substantially more sport-friendly schools, with only one identified example of an elite sport school found in Scotland [ 5 ]. Sport-friendly schools in the UK tend to be more independent than the systemic approach in other countries (e.g. Germany and Sweden [ 21 ]). In the UK, the development of a sport-friendly school is primarily a matter for individual schools and is often pursued as part of a strategy to create a distinct identity. As a result, it is important to investigate the individual context of a sport-friendly school within the UK as a case study.

Based on the above, this study, using a mixed methods longitudinal case study design, aims to (1) longitudinally evaluate the impact of sport-friendly school involvement on the holistic (i.e. academic, athletic, psychological and psycho-social) development of student athletes, (2) evaluate the impact on holistic development by athlete characteristics (i.e. sex, boarding status and external sport involvement) and (3) explore the features and processes of the sport-friendly school programme that drive/facilitate positive and negative holistic impacts.

2.1 Research Approach

This study was aligned with and guided by a critical realist (CR) perspective. In line with North’s [ 22 ] perspective on CR, this study was guided by the principles of developing theory (i.e. first understanding of sport schools impacts, then, second, developing an understanding of ‘how,’ ‘why’, ‘what’ and ‘for whom’). As such, the researcher first engaged in contextual description (aims 1 and 2), then, second, started to develop an understanding/explanation of how observed patterns were generated (aim 3). To help achieve the study aims, this study adopted a concurrent mixed methods approach (i.e. qualitative and quantitative data collected simultaneously [ 23 ]). This design aims to create mutually exclusive sets of data that inform each other [ 24 ]. Furthermore, the qualitative and quantitative data were analysed separately but then integrated to cross-validate findings. Finally, in line with the CR stance of establishing ‘how’, ‘why’, ‘what’ and ‘for whom’, Pawson and Tilley's [ 25 ] and Yin’s [ 26 ] guiding principles for an explorative case study approach were used.

2.2 Positionality of the Researchers

It is also important to acknowledge the collective roles of the researchers’ autobiographies, values and beliefs in describing, designing and interpreting the findings [ 27 ]. To acknowledge this, we consciously outline them to help appreciate and evaluate the results in nuanced ways [ 28 ]. The first author, F.T., collected the data and was lead on the analysis and writing. As the school’s lead strength and conditioning (SC) coach and a previous student athlete at a different sport-friendly school for 5 years, this would have inevitably shaped the primary researchers’ conceptions and influenced the study’s initial framing, design and analysis. Furthermore, the collective experiences of the remainder of the research team will have contributed to the interpretation of the data and shaping of the results. Combined, K.T., F.R. and I.C. have over 30 years of research and applied experience within athlete development systems.

2.3 Context of Study

One sport-friendly school (pseudonym ‘Nunwick High’) was selected for the study based on Morris et al.’s [ 6 ] definition of a sport-friendly school. The selection of ‘Nunwick High’ was information-oriented and opportunistic. ‘Nunwick High’ has 8 years of experience providing DC support through a performance sport pathway embedded within a UK independent school. ‘Nunwick High’ has eight performance sports as part of its performance programmes: athletics, basketball, cricket, football, hockey, netball, rugby and swimming, targeted at year groups 7–13 (aged 12–18 years). Each student athlete enrolled on ‘Nunwick High’ performance sport programme receives a place to study, train and, in some cases, live during their lower and upper secondary school years, including access to learning facilities, a sport science centre, a sport treatment centre, sport facilities, accommodation buildings and a canteen all in one proximity (single campus). Based on the information above, ‘Nunwick High’ represented an established and mature environment that should be a rich source of information.

2.4 Participants

Participants had to meet the following inclusion criteria: participate as a student athlete in one of the performance sport programmes within ‘Nunwick High’ and be aged 16 or above (years 12–13). Years 12–13 were chosen specifically, as during this stage student athletes are transitioning to a more intense and structured period of athletic development [ 29 , 30 ], and increased educational demands, with the consequence that the management of their DC, is a distinct concern. A total of 72 student athletes (mean age 17.29 ± 0.52 years, 48 male and 24 female) participated in the study. At baseline (T1) the student athletes had been attending and competing at ‘Nunwick High’ for an average of 1.2 ± 1.5 years (range from 2 weeks to 7 years). Out of the 72 student athletes, 31 were boarders (i.e. live at the school) and 41 were non-boarders, 31 played sport externally to the sport-friendly school and 41 only played sport for the sport-friendly school, representing the following sport: athletics ( n  = 4), cricket ( n  = 4), hockey ( n  = 12), netball ( n  = 9), football ( n  = 18), rugby ( n  = 15) and basketball ( n  = 10).

2.5 Study Design

A longitudinal mixed methods case study design was employed across one full academic school year (33 weeks). To engage in a comprehensive and holistic investigation of the impacts of being a sport-friendly school student athlete and the features and processes that drive/facilitate such impacts, six data-collection methods were utilised: (1) online questionnaire, (2) physical fitness testing battery, (3) academic assessment grades, (4) log diaries, (5) field notes/observation and (6) timeline diagram/illustration.

The online questionnaire occurred over five data collection periods (Q1, September; Q2, November/December; Q3, February; Q4, March; and Q5, May). The physical fitness testing battery occurred over three data collection periods (PFT1, September; PFT2, December; and PFT3, March/April). The academic assessment grades occurred across four data collection periods (A1, October; A2, December; A3, February; and A4, June). The log diary occurred over four data collection periods (L1, October; L2, December; L3, January; and L5, March). The observational research was ongoing throughout the whole academic year (33 weeks). Finally, the timeline diagram/illustration was collected once at the end of the academic year. Figure  1 provides an overview of the data collection timeline. The university sub-ethics committee granted this study (ref. 86728) with online informed assent and parental written consent obtained.

Overview of data collection points at the sport-friendly school

2.6 Measures

2.6.1 online questionnaire.

Data collection involved participants completing an online questionnaire (predicted completion time, 29 min) that provided a multi-dimensional assessment of holistic athlete impacts identified in previous literature [ 8 , 15 , 16 ]. The online questionnaire comprised of 12 domains (i.e. academic and sport workload, difficulty balancing sport and academics, academic support and satisfaction, injury and illness, rest and recovery, body image, family, free time and friends; sport competence; sport confidence; life skills, dual career motivation and resilience) as presented in Table  1 . The questionnaire was conducted in a quiet room, and student athletes were allowed sufficient breaks when required and were allowed to return to the questionnaire at a later time within the same day. Further, open-ended questions were used to help expand on responses to close-ended questions [ 31 ], providing further information on the features and processes that drove/facilitated specific impacts. All questionnaires were collected across all timepoints (T1–T5) apart from The Life Skills Scale for Sport (LSSS) questionnaire which was added in from T2 as the LSSS requires participants to rate how much their environmental exposure has taught them to perform the skills listed within the questionnaire and a baseline value was not appropriate. Completion rates: 97% for Q1, 90% for Q2, 94%for Q3, 93% for Q4 and 99% Q5.

2.6.2 Academic Assessments Grades

To assess educational attainment, termly academic subject assessment grades were extracted from the school administrative system. As all student athletes were in years 12–13, and grades were provided in the UK national curriculum grading format for Advanced level (A-level) and Business and Technology Education Council (BTEC) qualifications. To adequately compare BTEC and A-level grades, in addition to statistical purposes, academic assessment grades were converted to a number using a school grades translation matrix in Table  2 (similar to [ 58 ]). After conversion, an average of each individual’s subject score was calculated to get one overall academic assessment score for each student athletes. Completion rates: 96% for A1, 90% for A2, 94% for A3 and 94% for A4.

2.6.3 Physical Fitness Testing Battery

To assess physical development, a fitness testing battery which included; lower-body power, strength, speed and cardiovascular fitness tests were conducted in line with previous studies [ 59 ]. Speed was reported at 10 and 40 m distances [ 60 ], lower-body power was reported using countermovement jump (CMJ) height (m) and strength was reported using the isometric mid-thigh pull (IMTP) [ 61 , 62 ,– 63 ] peak force (kg) and relative peak force (kg −1 ) measures. The fitness testing battery was conducted over 2 weeks. In week 1, subjects performed measures of strength via the IMTP and power via the CMJ. In week 2, field-based measures of 10–40 m sprints were performed to measure acceleration and max velocity. On all testing days, the test causing the greatest strain on the neuromuscular system was performed first to enhance the reliability of all maximal testing procedures [ 64 ]. Completion rates: 97% for PFT1, 96% for PFT2 and 97% for PFT3.

2.6.4 Log Diary

Student athletes were asked to fill in a log diary across four timepoints in the academic year consisting of open-ended questions that explored the positive and negative holistic impacts and any features and processes of the sport-friendly school that caused, attributed or drove these impacts. Open-ended questions allowed the respondents to express opinions without being influenced by the researcher [ 65 ]. For example, student athletes were asked to reflect on the last month and outline the positive and negative impacts they had experienced on their athletic/physical, academic, psychosocial and psychological development. Furthermore, open-ended questions allowed respondents to include more contextual information, giving more feedback on the features and processes of the sport-friendly school programme that drove/facilitated positive student athlete holistic impacts [ 31 ]. For example, student athletes were asked to outline what caused, attributed or drove these impacts/outcomes to happen (e.g. what characteristics, features or processes?). Completion rates: 24% for L1, 42% for L2, 38% for L3 and 38% for L4.

2.6.5 Observational Field Notes

To achieve contextual sensitivity, emphasis was placed on participant observation of the daily lives of the student athletes in their natural setting as an essential method of data collection [ 66 ]. Over the 33-week academic term, the primary researcher completed observational field notes throughout each academic day relating to objective observations and conversations and subjective reflections of the actions, behaviours and interactions observed at ‘Nunwick High’ [ 67 , 68 ]. Observations were made from a holistic viewpoint, generally attuned to the broader context of the school, including context-individual interactions and processes between sport school features and holistic athlete impacts. Notes were also taken on specific coaching actions and behaviours, individual participant experiences and the interactions observed between student athletes, coaches and teachers. The observations enhanced the researcher’s understanding of the ‘Nunwick High’ context and student athletes’ holistic development [ 68 ].

2.6.6 Timeline Diagram/Illustration

At the end of the academic year, a convenience sample of 15 participants (mixture of sport and sex) were chosen to complete a timeline diagram/illustration visualising and displaying their personal experiences of the fluctuations in academic stress and sport workload across the academic year. Within the group, each individual was asked to draw a graph representing their academic stress and sport workload across different periods of the academic year (term 1 to term 6). In addition, they were asked to highlight the key academic assessment periods across this time period. After the student athletes completed their timeline, they described and discussed their diagrams as a group, providing personal explanations and rationale for the timelines they had drawn with the primary researcher who wrote down additional notes. Successively, findings (from both quantitative and qualitative data) were fed back to participants and an opportunity was given for participants to elaborate and provide more contextual information on the findings. The data were then integrated as part of the results, complementing and enriching the data generated in the TA [ 69 ]. Although the researcher made sure to keep the discussion on topic, as well as reiterate that there were no right or wrong viewpoints [ 70 ], the direction of the discussion was driven by the student athletes. This form of research has been used in previous studies [e.g., 71 ] and provided student athletes with a sense of engagement and ownership over the research process.

2.7 Data Analysis

2.7.1 aims 1 and 2 data analysis, 2.7.1.1 quantitative analysis.

To address research aims 1 and 2, two generalised mixed models were conducted using R (Version 4.1.3). The first model (addressing aim 1) assessed the changes in impacts across the school year (33 weeks). The change in score of each holistic variable was used as the dependent variable, with time (i.e. Q1–Q5, PFT1–PFT3 and A1–A3) added as the fixed factor. Individual participants and sport were used as covariates (random factors). The second model (addressing aim 2) considered the specificity of athlete characteristics. Each holistic variable was used as the dependent variable, with biological sex (female versus male), living status (border versus non-boarder) and external sport commitment (a student athlete who played sport externally to the sport-friendly school versus a student athlete who only played sport for the sport-friendly school) added as fixed factors. Individual participants and sport were again used as covariates (random factors). The p -value was set at 0.05. Injury and illness incidence rates were processed separately using Excel (Microsoft Office 2021) and described using percentages with frequencies due to being bi-nominal data.

2.7.1.2 Qualitative Analysis

Alongside the quantitative data, qualitative data were used to evaluate the impacts of sport-friendly school involvement. The data was coded using a largely deductive approach [ 72 ]. First, during the preparation phase, qualitative data was organised and managed into categories to be analysed together (i.e. log diaries, open-ended questionnaires and observation field notes and timeline diagram/illustration transcripts) and the primary researcher obtained a sense of the whole data through reading the transcripts several times. Next, during the organisational phase, data were generated through coding [ 73 ]. Our coding approach was deductive in nature as most codes were generated through the available systematic review [ 8 ] and the online questionnaire items (refer to Table  1 ). Inductive coding was used as new themes specific to the holistic impacts of student athletes and any specificity of athlete characteristics were identified during the coding process.

2.7.1.3 Triangulation

Given that quantitative and qualitative methods were used to investigate the same holistic student athlete impacts, the data for analysis were compatible for integration using the process of triangulation resulting in the creation of a number of themes [ 74 , 75 ]. As part of this process, the primary researcher compared the findings from the quantitative and qualitative analysis and considered where the findings from each method agree (converge), offer complementary information on the same issue (complementarity) or appear to contrast each other (discrepancy or dissonance) [ 75 ]. Subsequently, the assessment of convergency, complementary and discrepancy were discussed among the authors to (1) clarify interpretations of the findings and (2) determine the degree of agreement among researchers on triangulated findings [ 75 ]. Finally, after refining the themes, the primary researcher defined and named the themes.

2.7.2 Aim 3 Data Analysis

Aim 3 aimed to provide a more explanatory (i.e. seeking to explain the causes of phenomena) approach to research [ 76 ]. As such, Fryers’ [ 77 ] five-step CR approach to thematic analysis (TA) was used to analyse the qualitative data (i.e. log diaries, open-ended questionnaires, observation field notes and timeline diagram/illustration transcripts). As part of the first stage of TA, the primary author clearly outlined and refined the research aim and objective (i.e. explore the features and processes of the sport-friendly school programme that drive/facilitate positive and negative holistic impacts). In the second stage, the primary author immersed herself in the data by reading and re-reading texts to familiarise themselves with the findings and make notes on the initial thoughts and questions. Following familiarisation, stage three consisted of applying, developing and reviewing codes (step 3 [ 77 ]). Descriptive codes were applied to segments of qualitative text that were considered relevant to the research aims (e.g. features and processes of ‘Nunwick High’). Following the development of codes, step 4 entailed grouping all codes into themes [ 77 ]. Explanations were developed to suggest how particular features and processes of ‘Nunwick High’ produce the holistic impacts evidenced in the data (i.e., aims 1). Finally, within stage five [ 77 ], reflections on the overall analysis were discussed and reviewed among the research team, with a particular focus on checking the plausibility of the explanations against pre-existing evidence (i.e. in the data as well as existing theory).

2.8 Establishing Research Rigour

Following recent recommendations, Hirose and Creswell’s [ 78 ] six core quality criteria for mixed methods studies are proposed as useful in judging the rigour of the current study. First, the authors have outlined a clear rationale for the use and appropriateness of mixed methods methodology in this study (i.e. criteria 1). Second, throughout the design included specific quantitative (e.g. What are the impacts of sport school involvement on the physical development of athletes?), qualitative (e.g. Can you tell us about the balance between sport and school?) and mixed methods (e.g. How were changes in personal development brought about by the environment?) questions (i.e. criteria 2). Third, it has been clearly outlined which elements of data collection resulted in quantitative and qualitative data, as well as how each type of data was analysed. Furthermore, quantitative data are clearly presented in Table  3 , and qualitative data have been represented in direct quotes throughout the results (i.e. criteria 3). The mixed methods research design has been identified along with a diagram of data-collection moments (i.e. criterion 4). Fifth, the authors have clearly outlined how data-integration has taken place, this is then evidenced throughout the results and Fig.  3 captures a display of how findings have been integrated (i.e. criterion 5). The integration of data resulted in added value, as it allowed the authors to highlight similarities and differences between quantitative and qualitative findings throughout the results, providing a more nuanced understanding of the holistic impact of sport school involvement. Furthermore, the notion of meta-inferences (i.e. inferences that draw on both quantitative, qualitative and transcend both databases or what does it all mean together), fit very well with the CR stance of the study and the analytical process employed to formulate initial theories (i.e. explanations) as to how things worked within this sport school context (i.e. criterion 6). Finally, further in line with the CR philosophical underpinnings and aims [ 79 ], we also invite the reader to judge the findings presented in terms of their plausibility (i.e. do the offered explanations make sense, both in light of the presented data and the existing research literature) and utility (i.e. how well the research account offers predictions for likely outcomes and can be used to guide practical actions in the real world).

In line with the study’s aims, the results are presented in three higher-order themes: (3.1) longitudinal investigation of student-athlete holistic impacts, (3.2) specificity of athlete characteristics and (3.3) features and processes of the sport-friendly school program (i.e. what worked for whom and how).

3.1 Longitudinal Investigation of Student-Athlete Holistic Impacts

The triangulated holistic student-athlete impacts are presented below. Table 3 presents the quantitative statistical results for each impact at each timepoint. Furthermore, differences in student-athlete characteristics (i.e. sex, boarding and external sport) are presented. The descriptions below triangulate the quantitative and qualitative data within key themes to present the longitudinal holistic impacts.

3.1.1 Fluctuations in Academic and Sports Workload Over-time Culminate in a Variety of Impacts

Table 3 presents how sports training, competition frequency and the number of rest days changed across the academic year. Sport training and competition frequency significantly decreased in March and May (1.57–2.22 h/week and 0.49–1.14 competitions/week) compared with September–February (8.84–10.23 h/week and 1.86–2.26 competitions/week). Significantly more rest days were experienced during May (~ 2.00 per week) than in the other periods. This finding is supported by the student athletes’ timeline diagrams/illustrations whereby most student athletes’ sport workload was typically high across terms 1–4, with a drop off in terms 5 and 6. In contrast, for summer sports such as cricket and athletics, the highest sport workload appeared in term 6 when they were also doing their final academic examinations, as exemplified by a summer sport student athlete when talking about term 6: “I think for [summer sport] it is hard. We literally will have three games a week and two exams a week”.

Fluctuating patterns were also shown for academic hours (represented by hours spent in academic lessons plus hours doing home work) and number of lessons missed. Academic hours were significantly lower during November/December and February (23.6–26.7 h/week) and highest in September and May (~ 28 h/week), which coincided with the number of lessons missed (i.e. more lessons missed in November–February than September–May). Furthermore, when the student athletes were describing their timelines, they highlighted three time periods that could be considered the most stressful from an academic perspective: (1) the second week back after the Christmas break (mock exam week), (2) the final 2 weeks before Easter (final coursework deadlines) and (3) the whole of terms 5 and 6 (final academic examinations).

3.1.1.1 Periods of Difficulty Balancing Dual Demands and Changes in Stress and Recovery

Student athletes found balancing academic and sports workload significantly harder during November–March (3.22–3.34) and easiest during May (2.43). When student athletes were describing their timelines, they described a constant oscillation between periods of high academic stress (e.g. assessment time, mocks and exams) and high sport workload (e.g. busy fixture list, major tournaments and finals), with them often coinciding, resulting in increased stress and pressure.

“So, at the moment it is fine, but now gradually, academics are getting a lot more pressure on and the fixtures start to go like that again [demonstrated a steep incline with hand]. And then there is not really a break till March and by then should be absolutely ready for your A-levels and you are behind. Still revising some topics”.

Although student athletes’ general stress stayed stable across the academic year (no significant change across September to May), sport-specific stress levels varied across different time periods (highest in February and lowest in May). Regarding recovery, although general recovery stayed relatively stable across the academic year, sport-specific recovery was significantly lower in February compared with September (implying that student athletes were not recovering as well from sports during February compared with September).

3.1.1.2 Fatigue Accumulation, Culminating in Student-Athletic De-motivation

At the beginning of the academic year (September), student athletes were exposed to an immediate high academic and physical workload (i.e. 9.93 training hours/week and 28.1 academic hours/week). Additionally, from a physical fitness perspective, student athletes are physically less fit. Overall, the initial challenges (i.e. demanding schedule) and lack of physical fitness appeared to result in student athletes feeling fatigued, both mentally and physically at the start of the academic year. For example, a student athlete stated in their log diary 3 weeks into term 1:

“I’m keeping up with my school work but the workload is high due to not having free periods (because I play sport). I feel motivated to improve in both my academics and my sport. I am finding myself feeling more tired during the week but this is probably a combination of higher amounts of physical activity and not going to bed early enough.”

The feeling of fatigue was a common impact across the academic year. The student athletes frequently stated in their log diary that they were ‘always tired’, as exemplified by this student athlete: “I always want to sleep”. This impact was further exaggerated for student athletes with increased academic demands (e.g. undertaking four A-levels versus three), as exemplified by one student athlete’s log diary:

“The workload is high because I am taking 4 A-levels as well as doing my sport throughout the day- this means I have less time in school to complete work set and have to do the majority of it at home. This can build up and occasionally I find myself working until late which is leaving me feeling tired in the morning”.

Finally, there seemed to be an accumulative build-up of fatigue towards the end of each academic term and year. From a conversation with one of the coaches at the end of term 2, they stated: “This time of year everything changes. Kids getting tired, we are getting tired and boredom setting in”. The effect of fatigue on student athletes’ academic work was further elaborated on in a conversation with a student athlete: “I think there is enough time to do your work, it is just not enough time where you are not tired. You come home and you are knackered you don’t want to do work.” The student athletes described becoming demotivated during the end of term with a lack of physical development. “I plateaued. I started hating [sport]. I wasn’t improving, I was tired, I was stressed. To the point where I didn’t enjoy it”.

The feelings of de-motivation and mental and physical fatigue were further exaggerated in terms 5 and 6. A student athlete stated: “It is a bit burnout. You go, boom, boom, boom, boom, boom and now you just feel like flat”. By terms 5 and 6, student athletes appeared to have a lack of motivation and burnout for performance sports (consistent with student athletic motivation score, which was significantly lower in May), where student athletes wanted a period of unstructured training and time away from the performance environment.

“The last summer term with exams. I remember that first weekend after school finished, I literally couldn’t do anything else. I was so tired, like mentally and physically. And then I dunno, the feeling was awful”.

3.1.1.3 Immediate and Multiple Stresses

New student athletes at ‘Nunwick High’ experienced increased stress and pressure from an immediate intensive level of training and increased academic demands. In addition, they reported emotional and social stress from moving away from home, family and friends into a new environment. For example, from conversations with new student athletes who had transitioned into the school, they stated that they found the workload (both academic and physical) ‘a lot more’ than previous experiences. “You’re sort of chucked straight into it and expected to do everything basically. It is quite intense and a lot asked of you. Kind of have to do it and get it done”.

Then across the academic year at ‘Nunwick High’ there was evidence of three types of stressors: (1) Competitive stressors related to the demanding game schedules. “The upcoming matches that have been occurring have caused me to become more stressed”. (2) Organisational stressors from commitments to school sport balance.

“But I think sometimes, yeah it happens, but you are not enjoying it, it doesn’t become enjoyable it just becomes stressful. To go to a match and then come back and do your work. It is then not an enjoyable period”.

Finally, (3) Personal stressors when student athletes sacrifice social life for sport.

“I’m not as social as I was at the beginning of the year, I think this is due to the stress given by school. I feel as though I need to spend more time doing school work compared to socialising”.

However, contradictory to the personal stressors, student athletes’ friends and family and free time KIDSCREEN-27 Health Questionnaire scores stayed stable across the academic year.

3.1.1.4 Despite Challenges and Academic Pressure, Student Athletes Generally Achieved Good Academic Grades

As highlighted above, student athletes experienced challenges across the academic year (e.g. demanding schedule, fatigue and multiple stressors), in addition to academic pressure, as highlighted by a student athlete, “for me, academic pressure is a really big thing, because I am really scared, I am going to let it slip accidently”. Despite these challenges, overall, academic grades stayed relatively stable across the academic year (4.29–4.57), with only June significantly higher than October. This finding coincides with the fact that academic motivation also stayed stable across the academic year (4.78–4.88). This is further supported by the qualitative data which highlighted that student athletes at ‘Nunwick High’ generally achieved good academic grades. According to the log diaries, although some challenges around managing the multiple demands on their time were highlighted, most student athletes were generally happy with their academic development across the year: “I think I have developed academically in my exams; I have improved consistently throughout the year”.

3.1.2 Sport Performance Development and Well-Being Across the Year

As highlighted in the previous theme (3.1.1.2) student athletes are physically less fit at the beginning of the academic year. However, over time there were significant improvements in IMTP strength (123.3 kg September and 160.6 kg in March/April), CMJ height (34.3 cm in September and 36.5 cm in March/April) and 40 m max velocity (only September–December), whilst 10 m acceleration stayed stable.

Sport confidence was stable across the academic year (no significant change from September to May). However, there was a significant decrease in student athletes’ perceived sport competence during February (3.40) compared with September–December (3.59–3.58). Sport competence then recovered between February to May but not compared with September–December levels. This data contradicts the qualitative findings whereby student athletes largely expressed how being involved in the performance sports programme had resulted in them becoming better at their sport. They stated in their log diaries that they could see improvements in their physical, technical and tactical development and overall sporting performance across the academic year.

“My athletic development has gradually improved over time during all of the training and sessions. My physical development has improved slightly as well, especially with things like speed and size. My personal fitness has improved from the training and has encouraged me to do more out of the sessions”.

When evaluating injury and illness, injury incidence was higher than illness incidence. The greatest number of injuries occurred in November/December (47%), with the lowest injury incidence in May (23%). Illness incidence was highest in September (31%) and lowest in May (11%). Finally, although there was no significant difference or change in the student athletes’ EAT-26 score across the academic year and average scores were below 10, there were student athletes who scored ≥ 10, signifying disordered eating behaviour and attitude.

3.1.3 Personal Development

Student athletes also reported to have developed personally, although LSSS (3.51–3.56) and resilience (3.29–3.39) scores stayed stable across the academic year (no significant change across September–May). Through the qualitative data many student athletes emphasised they had developed a range of life skills and attitudes they could use both within and outside of sport. For example, they felt they had become more confident and developed their communication, social integration ability, social skills, work ethic, motivation, time-management skills, teamwork and leadership skills, in addition to becoming more independent, resilient, disciplined, mature and responsible adults. Student athletes highlighted these developments in their log diaries and through conversations with the primary researcher:

Student athlete 1: “Allowed me to develop my motivational skills. Training more and work in the gym helped to develop my social skills and my physical and mental abilities of perseverance during training and during my school work”.

Student athlete 2: “We talked about balancing a lot and if you are doing sport and academics, you kind of naturally build the skill of time-management and balancing stuff. I have got sport and A-levels as well, so I kind of have to think about time management as well. So, after I finish my sport, I know I need to go home and complete my prep. So, I kind of manage my day to get it all done”.

3.2 Specificity of Athlete Characteristics

There was no significant difference between boarders and non-boarders across all variables. For sex, females had significantly fewer weekly competitions than males. Female sport confidence scores and general recovery scores were significantly lower (− 0.68, − 0.47) and general stress and EAT-26 scores were significantly higher (+ 1.03, + 9.28) than males. Finally, females had significantly lower CMJ (− 10.49 cm) and IMTP (− 50.96 kg) and significantly slower 40 m max velocity (+ 0.93 s) scores than males. Internal sport-only student athletes had significantly less training (− 2.30 h/week) and competitions (− 1.42 number/week) and more rest days (0.52 number/week) per week compared with external sport student athletes. However, internal-only student athletes’ general recovery was significantly lower (− 0.32), which contradicts the qualitative findings where student athletes who played sports externally and for the sport-friendly school expressed feeling particularly fatigued and lacking rest and recovery.

3.3 Features and Processes of the Sport-Friendly School Program

While the primary aim of this study is to evaluate sports school holistic impacts, the third aim is to gain insight into the context-individual interactions underpinning them (i.e. features and processes). Accordingly, this section aims to provide a narrative overview describing insights into particular features and processes of ‘Nunwick High’.

3.3.1 Importance of Personal Motivation, Value of Education and Academic Support Services

Student athletes stated that they achieved good academic grades due to developing their personal motivation, organisational skills and commitment (i.e. hard work ethic, determination, self-motivation, developing a revision routine and creating a timetable of free time to balance workload), as highlighted in a student athletes log diary: “My work ethic and motivation have improved, which has caused me to work harder and put in more effort. I am not afraid to ask questions anymore to help me understand”.

Secondly, coach support was highlighted to assist student athletes’ academic development. There appeared to be flexibility with sports training and support from the coaches around the periods of high academic stress (i.e. student athletes were allowed to miss training sessions to do work), as exemplified by a student athlete: “Since my coaches have understood about me wanting to focus on my work, sometimes it has been helpful as I know that they support me”.

Finally, the student athletes received extra academic support. Teachers and fellow pupils provided extra tutoring (i.e. one-to-one help) in their own time. Teachers provided subject and revision clinics, and ‘Nunwick High’ had a learning development department. The extra academic support provided is demonstrated in the following quote from a student athlete’s log diary:

“Getting help from teachers—one-to-one help. Clinic revision—weekly revision after school to revise through any topics that I am not comfortable with. Microsoft Teams—online teams in which I can message my teachers directly whenever I am stuck”.

3.3.2 Performance Sports Program with Direct Sport-Related Practices, Staff and Support Services

‘Nunwick High’ was reported and observed as having high-quality facilities, fixtures, coaching staff and training partners. Student athletes had access to professional, high-quality facilities (e.g. a fully equipped gym, pool, indoor three-court sports hall and numerous astroturfs and grass fields). The performance sports program arranged high-level fixtures against top opposition (e.g. academy teams, high-level clubs and top sports schools). As a result, the student athletes were challenged technically, tactically, physically and psychologically against high-level opposition, as attributed by a student athlete in their log diary: “Recent fixtures, tournaments and matches have positively impacted my development, lifting to my maximum potential and pushing myself in court sessions”. Moreover, ‘Nunwick High’ employed high-quality coaches who could provide expert coaching, support and education to enhance the sporting development of the student athletes further. ‘Nunwick High’ was also described as attracting a big pool of talented student athletes providing high-quality training partners/teammates who acted as influential mentors—providing a high-quality training and learning environment where student athletes pushed their peers to be better and develop from one another. For example, a student athlete stated in their log diary:

“My skill and physical capabilities have improved drastically over the past month as the combination of regular strength and conditioning sessions as well as daily access to an indoor basketball court and high-quality players and coaching staff has driven me to become a completely different basketball player”.

As highlighted in the qualitative and quantitative data, the student athletes trained regularly across the year. As a result, the student athletes had more opportunities to practice, play and develop in their sport. For example, a student athlete stated they had ‘developed as a player’ and that this was due to ‘training every day and having games regularly’.

‘Nunwick High’ also had a multi-disciplinary sports staff as part of the performance sports program (i.e. SC, physiotherapist and nutritionist). The student athletes had designated and regular SC sessions within their school timetable, where the SC staff provided them with tailored and sport-specific physical development programs. Additionally, the SC staff provided additional athletic and physical development resources (e.g. cardiovascular fitness sessions, advice on recovery, mobility sessions) and put on recovery sessions (e.g. stretching/yoga). This support was deemed to positively support the student athlete’s athletic and physical development. For example, a student athlete, when answering in their log diary what was the driving factor for their improved athletic and physical development, stated:

“I have had a personal SC programme fitted to what will help me make the biggest impact on my sport; this has been essential for me and helped me to push hard, knowing that my interests are being taken care of and frequently adapted to fit my needs and any progress that I make”.

Whilst another student athlete stated the support available when injured:

“Due to an injury, I haven’t been able to train as often as normal on the pitch; however, the programme has still been able to help me develop during this time. I have had a lot of physio sessions which have helped me understand what is wrong with me, and the physio works closely with the SC staff, who are then able to provide me with stretches related to my injury as well as exercises that help my performance whilst taking into consideration my injury/limitations”.

3.3.3 Because the Environment Demanded It

The requirement to take accountability and responsibility, live away from home, and the busy schedule of sports and academics required student athletes to manage themselves effectively, become better at managing multiple demands and be disciplined.

“Time-management as well. You don’t necessarily get taught it. But you learn it by having such a busy schedule. You have to work out what to do when”.

The school strongly focused on giving the players accountability and responsibility for their academic and sport development. As described above, an environment was witnessed where the student athletes were given the relevant tools to help aid their sporting development and academic development. The student athletes were responsible for using these resources and maximising the opportunities in their own time.

“Environment where everything the athletes need is available to them (e.g. video from games, SC, yoga, extra sessions, academic support, pastoral care), but although the athletes are encouraged to utilise everything that is on offer to them, it is the athlete’s responsibility on how they use their time and if they utilise their time here effectively”. [Field note, 03/02/2022]

However, there was a lack of upskilling to allow student athletes to maximise their development, particularly in managing their time effectively. The student athletes felt that sometimes, the staff presumed they had the relevant skills without providing them with the tools to facilitate appropriate ownership of their development (i.e. feeling left to their own devices). For example, student athletes stated the following comments when talking about taking responsibility:

Student athlete 1: “I think they just expect you to be more organised, to be able to fit your sport in”.

Student athlete 2: “What we get offered here, most of us haven't been exposed to it before coming here. Then you are expected to know how to use it. When a lot of people don’t. So, then they don’t get the most out of as they can do”.

Furthermore, the additional work student athletes were expected to do in their own time (e.g. clip their own video) adds to their workload, providing further conflicts with their academic study and personal time.

“Yeah, like no one tells you to go and watch the video. But I like to watch it and see what happens and see why we lost to [team]. But then that is an hour, hour and a half of Thursday when the video comes out. So that is when I should be working”.

3.3.4 Lack of Organisation and Planning of Training Load

When the student athletes first joined the performance sport programme in the sixth form, they transitioned into an intensive level of training. There was no preseason at the sport-friendly school, so the student athletes were immediately exposed to a high physical workload. Furthermore, first-team fixtures were organised within the second week of the term. When asked if the student athletes liked having fixtures within the second week of term, there was an overwhelming ‘no’ feeling. Student athletes stated they were ‘not adequately prepared’ and ‘had not had enough training time together’.

Additionally, from a physical fitness perspective, when student athletes transitioned into ‘Nunwick High’, or returned at the beginning of the academic year, they felt physically ill prepared for the immediate, intense training load. Through pre-season physical fitness testing, the primary researcher observed student athletes coming back from the summer holidays with lower physical fitness levels than expected.

“Just completed 30–15 running fitness test with [sport]. Generally, the student athlete’s cardiovascular fitness scores are lower than I would expect them to be at the beginning of the year in comparison to normative, expected data for their sport”. [Field note, 07/09/2021]

Coaches, in conversation with the researcher, emphasised that at the beginning of the year ‘students were not fit enough’. A student athlete further highlighted this comment when talking about the initial start of term: “And also, our fitness isn’t as good as it would have been after training all the time at school. So, I think we are lot more unfit as a lot of us don’t train outside of school”.

Within an academic year at ‘Nunwick High’, no periodised planning, tapering or deload was scheduled within the performance sport programme across a term. The primary researcher observed a lack of balance between high training loads followed by intentional low training loads (i.e. deload/tapering weeks).

“Season at [Nunwick High] is very full on and intense the whole time. There is no periodised planning, tapering or deload week within the term or season. This is resulting in kids being exhausted by the last 2 weeks”. [Field note, 15/03/2022] “Sometimes, I think we over train. Like having 2 h on Wednesday, then another 2 h on Thursday. Then an hour on Monday, SC on Tuesday and another SC on Friday. With no recovery. You know, it is really intense”.

Despite the benefit of offering high-quality competition from a sporting development point of view, ‘Nunwick High’ appeared to enter every competition, league and cup and has an extensive list of friendly competitions. As a result, some sports teams had two (on the rare occasion, three) internal sports fixtures a week (not considering the fixtures some student athletes have externally outside of school), leading to potential fixture congestion. Based on observations and log diaries, the extensive fixture list appeared to put further pressure on student athletes academically, as they missed many lessons and were fatigued. As exemplified by a student athlete in their log diary, “Having regular away fixtures has caused me to miss multiple lessons every week and afterschool training has limited time to catch up on homework”.

Finally, the primary researcher observed a lack of collaboration with external sports schedules. For example, based on her observations, the primary researcher reflected:

“Given that match play required a longer period of recovery than training, the school coach on Thursday often incorporated recovery sessions. However, unaware of the school match the previous day, one student athlete’s club team continued with an unmodified training session, including one to two hours of technical training on a Thursday night. As a result, negating the benefits of the recovery sessions within the school. The student athlete returned to school training on Friday, 24 h after the match, with the school coach presuming the fatigue from the match had largely dissipated. From chatting to the student athlete, they stated that they did not actually have the chance to recover from the match on Wednesday and entered the weekend fixtures feeling fatigued, which he believed compromised his performance”. [Field Note, 17/02/2022]

3.3.5 Lack of Coordination and Program Flexibility Between Academic and Sports Timetables

Although academic support services were available and some academic staff provided extra academic support and understanding for the student athletes when needed, there appeared to be a need for more understanding from all teachers. For example, in a conversation with two athletes, they stated:

Student athlete 1: “If you miss a lesson, then they just send you the work and expect you to do it yourself”.

Student athlete 2: “Sometimes I don't think my academic teachers understand. They are like ‘again, really’. I am like; it doesn’t change just because I did it last week”.

Moreover, although there appeared to be flexibility with physical training and support from the coaches around periods of high academic stress, there was a lack of planning, co-operation and compromise with scheduling, with sports fixtures clashing with periods of high academic stress (apart from in term 6). For example, as highlighted in Sect.  3.1.1 student athletes described a constant oscillation between periods of high academic stress and periods of high sport workload with them often coinciding, resulting in increased stress and pressure (as depicted in Fig.  2 which summarises the general patterns observed in the student athletes’ timeline diagrams/illustrations).

“I think my timetable doesn’t match up. So that means I get assessments and work I miss because I go to matches, and then I am still going to the gym and stuff like that. So, it feels like there is no compromise, and when it comes to assessments, I still feel like I need to do the match”.

Overview of oscillations in academic stress and sport workload across the school year

Despite academic flexibility/support by some coaches and teachers, due to the conflicts between academic and sport schedules, student athletes often felt conflicted, pressured and guilty towards both coaches and teachers if they chose one endeavour more than the other and were often reminded of it. “[Coach] will mention past things you have done. Like, yeah, but you didn't come to this one either, and you didn't come to this one, and now you are missing this one. So yeah, like the build-up of guilt”. Additionally, there seemed to be a lack of understanding and a conflict between what is a priority for student athletes regarding internal and external training, with student athletes feeling scared to come forward if they were tired.

“I feel like if I said to [coach], I can’t train as well on Wednesday as I was training for [club] on Tuesday. Then he would like to quit [club]. And I don’t want to quit it. But I don’t think I could come forward and say I was tired because I trained last night. As he would say that I am disrupting [school sport]. You are here to play [school sport]”.

There was little evidence of direct communication and alignment between sport coaches and teachers, where they worked together to ensure that their schedules were appropriately adjusted and aligned to the student’s academic (deadlines and submissions) and sport (tournaments and cup competitions) load. Instead, student athletes explained that they were the ‘middle ground’ for communication between coaches and teachers.

“I think the only thing that is hard about it is communication between your teachers and the coaches as well. As obviously, the teachers will have their say and be like, ‘You do too many matches’”.

Finally, student athletes at ‘Nunwick High’ had varying academic demands, extra-curricular activities and sporting commitments. Moreover, the performance sport teams’ schedules varied weekly (e.g. a team may compete in three competitions 1 week and no competitions the following week). Despite this, there appeared to be an overall ‘one size fits all’ approach to the overall planning, with a lack of adaptation to individual student athletes’ varying commitments and between-week team schedules, causing further competing demands and stress.

“Yeah. I think when we had gym and dance. That was sort of like a commitment. [Coach] would know we would have it after school but still expect me to go 100%, even though the night before we would have had a full run-through and everything went wrong and duh duh duh, school production itself. So it is sort of, I understand you have all this other stuff, but it doesn’t give you an excuse not to go 100% in training”.

4 Discussion

To our knowledge, this study is the first to longitudinally evaluate (1) the impact of sport-friendly school involvement on the holistic development (i.e. academic, athletic, psychosocial and psychological) of student athletes, (2) the holistic impact according to the specificity of student athlete characteristics (i.e. sex, boarding status and external sport involvement) and (3) the features and processes of the sport-friendly school programme that drive/facilitate positive and negative holistic impacts.

Overall, mixed-method data demonstrated that over-time student athletes, achieved good academic grades, enhanced their all-round sporting performance and developed personally, demonstrating positive short-term and potential long-term positive impacts of sport-friendly school involvement. In addition, student athletes’ sport confidence, academic motivation, academic grades, general recovery, life skills, resilience and friends, family and free time scores remained stable and relatively high across the academic year. Potential features and processes of ‘Nunwick High’ that contributed to these positive impacts included: high-quality facilities, fixtures, training partners and coaching staff, high frequency and extra training, multi-disciplinary sport support staff (e.g. SC, physiotherapist and nutritionist), academic support services, and self-reported motivation and hard work ethic to engage with training and academics. Despite these positive benefits, the simultaneous pursuit of academic and athletic achievements provided challenges for student athletes across an academic year. Potential negative impacts found included: increased stress and pressure at the beginning of the academic year, immediate accumulation of fatigue (both mentally and physically), competitive, organisational and personal stressors, high injury rates, potential body image concerns, conflicting demands and feeling “left to their own devices”. Furthermore, student athletes’ experienced significant fluctuations in their sport and academic workload, rest, academic lessons missed, sport-specific stress and recovery, sport competence and student-athletic motivation scores across the academic year. Many of the potential challenges/negative impacts student athletes experienced seemed to be attributed to a lack of (1) gradual increase in training exposure (intensity, frequency and volume) at the beginning of the academic year, (2) coordination and consideration between academic and sport timetables, (3) collaboration with external sport schedules, (4) direct communication and alignment between the coaches and teachers, (5) program flexibility and (6) periodised planning, tapering or deload scheduled within the sport timetable. However, it is worth noting that individual characteristics shaped the sport school experience and its impact on the holistic development of student athletes. Biological sex and external sport commitments were shown to influence student-athlete holistic impacts, however boarding status did not. Figure  3 summarises the longitudinal holistic impacts of sport-friendly school involvement, including the program’s features/processes driving positive and negative impacts.

Summary of the longitudinal holistic impacts of sport-friendly school involvement and the potential features and processes that drive/facilitate positive and negative holistic impacts

4.1 Longitudinal Investigation of Student Athlete Holistic Impacts

4.1.1 immediate and intermediate risk and challenges.

The student athletes faced numerous challenges at the onset of the academic year (e.g. high physical training loads, frequent sport fixtures and psychosocial adjustments) aligned to existing research [ 8 ]. Longitudinal data suggested these continued throughout the academic year. The workload challenges are similar to previous research in sport schools [ 80 , 81 , 82 ] and youth sport [ 12 , 83 ] but providing sport and academic load simultaneously emphasises the challenge of combining student athletes workload with external sporting commitments. These workload challenges potentially contribute to various other impacts experienced by student athletes, such as increased rates of missed academic lessons, heightened susceptibility to injuries, and the ongoing struggle to effectively balance their athletic commitments with academic responsibilities. Consequently, this confluence of demands often results in elevated levels of fatigue, persistent feelings of tiredness, and heightened stress among student athletes. This explanation is plausible given previous literature (e.g., [ 84 ] and [ 85 ]) has emphasised that the time commitments associated with combining education alongside sports training were a crucial contributor to fatigue accumulation and stress.

The longitudinal data highlights, student athletes’ need to negotiate many fluctuating academic and sport demands and expectations across a school year, which are often conflicting [ 84 , 86 , 87 ]. In parallel, student athletes seemed to find the sport–academic balance easier when they had increased rest and reduced training/competitions. These findings are unsurprising, as fewer competing demands exist. Previous research similarly demonstrates that the commitment (i.e. time and effort) to sport coincide with youth athletes’ education [ 21 ] and competitions/training, resulting in youth athletes missing school for several days or even weeks/months a year [ 88 ], making balancing both sport and education challenging [ 9 , 10 , 11 ].

Finally, the qualitative data reveal a consistent cycle between periods of high academic stress, such as assessment times and exams, and periods of intense sports workload, such as busy fixture lists and major tournaments. These overlapping demands potentially contribute to three main categories of stress: competitive stress due to game schedules, organizational stress from balancing school and sports, and personal stress involving social sacrifices. This pattern is supported by the correlation between changes in student athletes’ training loads and their sport-specific stress levels throughout the academic year. Competitive, organisation and personal stressors are supported by Kristiansen and Stensrud’s [ 85 ] study, which found evidence of all three stressors among youth female handball sport school athletes.

4.1.2 Long-Term Positive Impact

This study suggests that despite the challenges (e.g. balancing both sporting and academic commitments) student athletes within sport schools can excel in both sport and academics. Student athletes maintained stable and high academic grades throughout the year, supported by the qualitative data. These findings are congruent with broader youth sport research, which has indicated that student athletes excel in education (e.g., [ 89 ]). However, these findings contradict previous sports school literature [ 81 , 90 ], which suggested that sport participation negatively affected student athletes’ academic success.

Regarding athletic impacts, physical fitness data also demonstrated enhanced strength, speed and power. These results align with Beckmann et al. [ 91 ] study, showing increased fitness measures in student athletes enrolled in a sport school over 5 years. While the student athletes’ sport competence scores dipped compared with baseline throughout the academic year, qualitative findings demonstrated that student athletes felt they became better athletes (technical, tactically and physically). These findings may be explained by student athletes perceiving themselves as getting better but also had enhanced (different) perceptions and judgment as to where their own skills lay in comparison to others. Over time, student athletes may enhance their capacity for self-reflection and the evaluation of their abilities in comparison to others (i.e. their self-evaluation becomes increasingly more accurate but also more negative [ 92 ]), which could influence the self-perceived ratings of their own sport competence.

Finally, although student athletes’ psychosocial scores did not improve across the school year, they were relatively high at baseline and remained stable. Qualitative data highlighted the development of life skills and attitudes applicable in and beyond sport, reinforcing this trend. Previous sport school literature [ 82 , 93 , 94 ] supports the idea that sport school involvement fosters qualities and skills applicable to various aspects of life. Furthermore, overall LSSS scores were similar to that of British youth sport [ 95 ] and sport high school [ 96 ] student athletes. As such, sport-friendly schools should continue to develop student athletes technical, tactical, physical and academic capabilities but additionally develop their personal, social and life skill capabilities [ 97 ], to ensure student athletes develop transferable skills for life beyond the sport-friendly school environment [ 98 ].

4.2 Specificity of Athlete Characteristics

Sex and external sport commitments were shown to influence student athlete holistic impacts, however boarding status did not. In accordance with O’Connor et al. [ 99 ], females demonstrated lower levels of sport confidence and perceived competence compared with males, along with higher general stress, lower general recovery and greater body image concerns. Literature suggests that youth athletes, particularly females, are becoming concerned about their body image at increasingly early ages [ 100 ] and body-related shame and guilt are increasing over time among female youth athletes [ 101 ]. Looking at the inter-relationship between variables, previous research has found a significant relationship between body image and sport-related variables (e.g. sport confidence [ 102 ]). Furthermore, Murray et al.’s [ 103 ] study found a significant association between higher body dissatisfaction and higher ratings of peer stress and lower self-esteem. Given the potential heightened vulnerability in females, further research should explore the holistic development of female student athletes in sport schools.

Student athletes (such as those at ‘Nunwick High’) often participate in multiple sport or for various teams within the same sport [ 33 , 104 ]. External sport involvement increased student athletes’ time commitments (more training hours and competitions and less rest), intensifying the competing demands between academic and athletic pursuits. The additional demands link with lower general recovery scores for external sport student athletes. Research demonstrates that student athletes with higher weekly training loads have higher recovery-stress states than student athletes with lower weekly loads [ 105 ]. Furthermore, the qualitative data highlighted further fatigue and recovery challenges amongst this group, exacerbated by unsynchronized schedules between external and internal sport commitments. Previous research supports this conclusion, which demonstrates the ‘tug of war’ scenario of various weekly sport commitments, which can result from separate and contrasting athlete-focused training plans and goals [ 33 , 104 ]. Collaborative management of training schedules among the various stakeholders (i.e. coaches) is crucial to prevent fatigue, overreaching and injury risks among this specific group [ 106 , 107 , 108 , 109 ], requiring aligned training aims, load management, fixture lists and flexible programming [ 33 ].

4.3 Features and Processes of the Sport-Friendly School Program

As DC environments are complex and dynamic, whereby student athletes have to interact with many features and processes of a sport school, this study aims to advance on existing research to understand what facilitated and drove the positive and negative impacts. This approach was a unique and novel aspect of this study resulting in five key findings as discussed below.

4.3.1 Importance of Personal Motivation, Value of Education and Academic Support Services

One clear positive impact was that student athletes’ academic performance was high and stable consistent with previous research [ 89 , 110 ]. These findings may be explained by the student athletes displaying stable and relatively high levels of academic motivation across the school year and personal attributes aligned to academic work (e.g. hard work, organisation skills and commitment). Research (e.g. [ 111 ]) supports the associations between individual traits (e.g. AM, educational goals and commitment) and academic achievement demonstrating that student athletes’ academic motivation is important to achieving academic success. Furthermore, academic performance may reflect the importance of the additional support offered by sport schools (e.g. extra tutoring, revision clinics and consistent check-ups from academic and sport staff) in protecting academic success [ 8 ]. Mentorship, monitoring and extra tutoring were some of the academic support services provided at ‘Nunwick High’, which are consistent with previous sport school literature [ 7 , 8 , 93 , 112 , 113 ] and recognised as essential for encouraging academic success [ 114 ]. Finally, coach support (e.g. flexibility with sport training and support around the periods of high academic stress) was highlighted to assist student athletes academic development. This result is similar to Knight and colleagues [ 115 ], who underscored the need for an athlete’s support network to consistently reinforce the importance of education and the value of maintaining a DC. Ensuring the support staff are on the same page and everyone’s expectations are aligned, eases tensions within the group and prevents the student athletes from feeling conflicted [ 115 ].

4.3.2 Performance Sport Program with Direct Sport-Related Practices, Staff and Support Services

The current study provides additional evidence of Thompson et al. [ 15 ] cross-sectional study, demonstrating that student athletes will improve their all-round sport performance across an academic year and this change may be facilitated by a multi-disciplinary sport staff, high quality facilities, fixtures, training partners and coaching staff, high frequency of training, individualised support and a positive team culture. High-quality coaches and multi-disciplinary teams (e.g. SC coaches, sports psychologists, nutritionists and physiotherapists) are raised in the wider literature as aiding talent development [ 116 , 117 , 118 , 119 ]. Accordingly, it seems plausible that sport-friendly school programmes should employ high-quality coaches and support sport staff to provide high-quality training programmes and sessions. However, whilst this study demonstrates the value of high-quality coaches and support staff, future research should explore how coaches achieved performance education and development in practice. Having high-level fixtures and training partners is supported by Henriksen’s research [ 120 ], which supports a culture where you foster competition between members of the same institution and challenge them externally. However, although frequent and additional training opportunities were deemed a positive in this study, future research should explore the workload of the sport-friendly school student athletes objectively and their subsequent correlation with rest, recovery and injury.

4.3.3 Lack of Organisation and Planning of Training Load

Student athletes at ‘Nunwick High’ attributed their initial hard transition partly to inadequate physical preparation. Likewise, student athletes in Andersson and Barker-Ruchti’s [ 80 ] study attributed the initial stress they experienced due to the lower level of physical training that had taken place in their previous club communities. ‘Nunwick High’ student athletes faced an immediate, intense training load (with no preseason), possibly contributing to a high November/December (T2) injury rate. Similar findings in prior research (e.g. [ 121 ]) noted increased injuries after school holidays (e.g. summer). These findings suggest that more careful consideration of return to training planning and monitoring of appropriate training loads may be warranted [ 122 , 123 ]. From a fatigue, illness and injury prevention perspective, student athletes (particularly those new to a performance sport program) may benefit from a gradual, sequential increase in intensity, frequency, and volume early in the academic year. Furthermore, student athletes may benefit from support to help them prepare for and cope with the challenges and changes of moving into or transitioning through the sport-friendly school environment [ 81 , 85 ].

A recurring ‘tiredness’ theme emerged among ‘Nunwick High’ student athletes, with subsequent mental and physical fatigue accumulation. Across an academic term, ‘Nunwick High’ lacked planned deloading or periodization, with no systematic high-to-low load transitions to facilitate recovery [ 104 ]. As such, the issue may not be the overall load buts its organisation and lack of external sport workload coordination [ 104 ]. Scantlebury et al. [ 33 ] highlighted that a failure to provide appropriate periods of recovery between training sessions and within programmes could lead to lowered training capacity [ 124 , 125 ] or increased incidence of injury, illness and overtraining [ 126 , 127 , 128 ]. Furthermore, the lack of periodised planning may explain the fact that ~ 30% of student athletes had sustained an injury. To provide a sufficient stimulus for progressive overload, student athletes need be exposed to periods of high training volume and/or intensity [ 2 , 129 ], reflected in the increase in physical fitness testing data. However, recovery must be implemented after periods of intensified or voluminous training to allow the athlete to dissipate fatigue, adapt and avoid maladaptive responses such as overuse injury [ 108 ]. Accordingly, in sport schools, planned high-load/low-load periods are crucial to facilitate recovery and adaptations [ 33 , including periodised tapering or deload weeks aligned with high academic stress periods (e.g. assessments or mock exams).

4.3.4 Lack of Coordination and Program Flexibility Between Academic and Sport Timetables

Competing demands can be stressful when activities across the school timetable are insufficiently coordinated [ 85 ]. ‘Nunwick High’ lacked coordination between academic and sport timetables (e.g. fixtures scheduled throughout high academic stress periods, where student athletes missed lessons). Although some academic staff offered extra support and coaches were somewhat flexible and supportive (although may subconsciously emphasise sport within their communication with student athletes), better program planning, communication and alignment between coaches and teachers are needed. Previous research has highlighted that flexibility and planning are key to managing student athletes’ schedules [ 33 ] and alignment between coaches and teachers is crucial [ 84 ]. Consequently, coaches and teachers should adopt an athlete-centred approach, coordinating to recognise periods of high academic stress (e.g. exams and coursework deadlines) and high sport workload (e.g. competitions, finals) before adjusting schedules to ensure student athletes can manage both demands [ 33 ]. However, this may be more difficult for some sport (e.g. summer sport, such as cricket), where timetable clashes may be unavoidable. Previous research supports such integrated efforts as critical features of successful talent development environments [ 20 , 115 ], alleviating tensions and helping prevent dual career demands conflict [ 115 ].

It appeared hard for practitioners within ‘Nunwick High’ to plan effective training loads, efficient recovery and sufficient academic time due to the ‘individualised chaos’ within and between studentathletes varying weekly schedules [ 130 ]. Qualitative and quantitative (95% CI) data confirmed this variability. The challenges of within and between youth-athlete variance in weekly training load has been previously shown [ 33 , 131 ]. Individual needs differ based on sport, academic path and circumstances [ 132 ]. Consequently, in addition to program flexibility, sport-friendly schools may consider monitoring sport school student athletes’ varying weekly schedules, coaches/teachers should monitor student athletes’ physical and academic loads (e.g. training/work diaries), wellness (e.g. daily wellness questionnaire [ 133 ], the profile of mood states questionnaire for adolescents [ 134 ]) and recovery states (e.g. perceived recovery scale [ 135 ]) on an individual basis.

4.3.5 Because the Environment Demanded It

A clear positive impact was that student athletes’ developed life skills and attitudes applicable in and beyond sport. The requirement to take accountability and responsibility, live away from home and balance the busy schedule of sport and academics enabled student athletes to manage themselves effectively (i.e. become better at managing multiple demands) and be disciplined. However, it is also important to acknowledge the skills required to negotiate these challenges (e.g. psychological characteristics and competencies [ 136 ]). As such, there appeared to be a need for upskilling to allow student athletes to maximise their development earlier, particularly when managing their time effectively. Collins and Macnamara [ 136 ] proposed that skills development in an appropriately challenging environment is a big factor in the pursuit of ‘super-champ’ status. As such, sport-friendly schools may consider educating the student athletes with essential skills that would aid the challenges they face during their time at the sport school (e.g. time-management skills, developing coping strategies, a programme focused on understanding the most efficient way to maximise their learning) to allow them to exploit their development by understanding the most efficient way to maximize their learning and balance the issues arising from their restricted time schedules [ 33 , 86 ].

4.4 Balance Between Optimising Experience and Appropriate Challenge

It is worth noting that while student athletes encountered many challenges throughout the school year (e.g. oscillations in stress and demanding schedules), longer-term they reported largely positive impacts, potentially preparing them for the multiple demands of being a professional athlete or adult in the future. Research emphasises the value of incorporating challenges into talent development pathways (e.g., [ 137 ] and [ 138 ]). Overcoming challenges is increasingly seen as favourable for aspiring student athletes [ 137 , 138 ] but developing skills to navigate these challenges (e.g. psychological characteristics and competencies) should be planned and managed too. As such, while helping manage some of the physical overloading and scheduling (e.g. to prevent harm through injury, stress and emotional/physical fatigue), helping coaches understand progressive tolerance to the stresses experienced and upskilling student athletes is clearly warranted, there may be a need for some of these challenges to develop long-term positive holistic impacts (i.e. where the immediate/short term negative impacts could have medium-longer term positive impacts). So, while potential recommendations within this study may help optimise the experience, they should be carefully considered regarding their impact on the student athletes’ development in other areas (e.g. resilience, independence and self-motivation). Consequently, future research needs to explore what short-term impacts and processes are needed for long-term positive impacts.

5 Limitations and Future Research

Although the longitudinal design, mixed methods approach (triangulation), and generalised mixed modelling analysis are key strengths, it is also important to be aware of the study’s limitations. Some would argue that due to the first-hand experiences of the primary author, they already had their preconceived ideas, potentially narrowing the analytic lens of the study. However, the quantitative statistical analysis alongside the use of critical friends and frequent peer-debriefing and reflection sessions among co-authors, to minimise any potential biases [ 69 ]. Self-reported measures introduce another limitation, including the potential influence of social desirability. Moreover, different questionnaires were necessary to capture diverse impacts, potentially impacting response quality due to the questionnaire’s length [ 139 ]. However, the questionnaire was conducted in a quiet room, student athletes were allowed sufficient breaks when required and were allowed to return to the questionnaire at a later time within the same day. Furthermore, while participant concerns might not have been openly expressed in front of an institution member, the primary author's rapport with student athletes and staff fostered positive interactions, emphasising confidentiality and encouraging open, honest responses. Finally, in the academic year, term 6 was only 3 weeks long, and most upper-sixth student athletes had already left after final exams, leading to the decision to omit the online questionnaire during this term. Despite this, observational research covered the full 33 weeks, with the timeline diagram conducted at the study’s conclusion, though the lack of log diary assessment in terms 5 and 6 is a limitation.

While this study offers an initial insight into sport–school student athletes’ holistic impacts and trajectories, future research could explore this further using longitudinal methods, such as Cobley et al. [ 140 ], tracking the comprehensive development of select youth players and employing different statistical techniques such as multivariate latent growth models (e.g. [ 141 ]). Moreover, while this study provides an initial insight into how individual characteristics shape the sport school experience and its impact on the holistic development of youth athletes, further research is needed to gain a more in-depth understanding. For example, exploring additional individual characteristics like sport-by-sport analysis, age, injury status and training cycles could further enrich understanding. Finally, while preliminary discussions about potential correlations between impacts were included (e.g. academic attainment and AM), these relationships lack statistical exploration, necessitating further modelling and investigation of direct impact relationships.

6 Conclusions

Overall, ‘Nunwick High’ student athletes developed positive long-term holistic impacts (i.e. academically, athletically and personally), including maintaining stable and relatively high levels of sport confidence, academic motivation, general recovery, life skills, resilience and friends, family and free time scores. Development was generally attributed to the sport school’s athletic and academic support services and personal traits of the student athletes and staff. Moreover, accountability, responsibility, independence and navigating busy schedules fostered crucial life skills. Despite positive impacts, juggling academic and sport workload posed challenges for student athletes, potentially leading to negative holistic impacts (e.g. fatigue, pressure, stress, injury and lessons missed). These issues were linked to insufficient training load build-up, communication, coordination, flexibility and planning. While addressing physical overloading and coach understanding is important, future research should evaluate other environments and explore what short-term impacts are needed for long-term positive impacts.

Additionally, individual characteristics (e.g. biological sex) influenced sport school impact. Females had lower sport confidence, higher general stress and body image concerns and less general recovery compared with males. This vulnerability warrants detailed research on female student athletes. Furthermore, engagement in external sport introduces additional time and workload commitments, prompting sport schools to collaborate with broader sporting partners to harmonise student athletes’ training schedules and create coordinated athlete-focused training plans and goals. In summary, these findings demonstrate the complex nature of combining education and sport commitments and how sport schools should manage, monitor and evaluate the features of their programme to maximise the holistic impacts of sport–school student athletes.

Bergeron MF, et al. International Olympic Committee consensus statement on youth athletic development. Br J Sport Med. 2015;49(13):843–51.

Article   Google Scholar  

Lloyd RS, et al. National Strength and Conditioning Association position statement on long-term athletic development. J Strength Condition Res. 2016;30(6):1491–509.

Wylleman P, Rosier N. Holistic perspective on the development of elite athletes in Sport and exercise psychology research. Amsterdam: Elsevier; 2016. p. 269–88.

Book   Google Scholar  

Cartigny E, et al. Mind the gap: a grounded theory of dual career pathways in sport. J Applied Sport Psychol. 2021;33(3):280–301.

Cartigny E, Morris R. A taxonomy of dual career development environments in the United Kingdom. Sport Exerc Psychol Rev. 2020;16(2):24–38.

Morris R, et al. A taxonomy of dual career development environments in European countries. Eur Sport Manag Quarter. 2021;21(1):134–51.

Kristiansen E, Houlihan B. Developing young athletes: the role of private sport schools in the Norwegian sport system. Int Rev Sociol Sport. 2017;52(4):447–69.

Thompson F, et al. The impacts of sports schools on holistic athlete development: a mixed methods systematic review. Sports Med. 2022;2022:1–39.

CAS   Google Scholar  

Burden SA, Tremayne P, Marsh, HW. Impact of an elite sport lifestyle on educational choices and career outcomes. in Self-concept, motivation and identity, where to from here? In: Proceedings of the Third International Biennial SELF Research Conference. 2004.

O’Neill M, Allen B, Calder AM. Pressures to perform: An interview study of Australian high performance school-age athletes’ perceptions of balancing their school and sporting lives. Perform Enhance Health. 2013;2(3):87–93.

Pummell B, Harwood C, Lavallee D. Jumping to the next level: a qualitative examination of within-career transition in adolescent event riders. Psychol Sport Exerc. 2008;9(4):427–47.

Jayanthi NA, et al. Health consequences of youth sport specialization. J Athl Train. 2019;54(10):1040–9.

Article   PubMed   PubMed Central   Google Scholar  

McKay CD, Cumming SP, Blake T. Youth sport: friend or foe? Best Pract Res Clin Rheumatol. 2019;33(1):141–57.

Article   PubMed   Google Scholar  

Wylleman P, Reints A, De-Knop P. A developmental and holistic perspective on athletic career development in Managing high performance sport. London: Routledge; 2013. p. 191–214.

Google Scholar  

Thompson F, et al. A case study of the features and holistic athlete impacts of a UK sports-friendly school: student-athlete, coach and teacher perspectives. PLoS ONE. 2022;17(11):e0278401.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Thompson F, et al. What is it like to be a sport school student-athlete? A mixed method evaluation of holistic impacts and experiences. PLoS ONE. 2023;18(11):e0289265.

Bronfenbrenner U. Making human beings human: bioecological perspectives on human development. Thousand Oaks: Sage; 2005.

Gagné F. Transforming gifts into talents: the DMGT as a developmental theory. High Abil Stud. 2004;15(2):119–47.

Lerner RM. Diversity in individual context relations as the basis for positive development across the life span: a developmental systems perspective for theory, research, and application. In: The 2004 Society for the Study of Human Development Presidential Address, in Special Issue: Risk and Resilience in Human Development. Psychol Press, London; 2018. pp. 327–346.

Henriksen K. The ecology of talent development in sport: a multiple case study of successful athletic talent development environments in Scandinavia. Syddansk Universitet. Det Sundhedsvidenskabelige Fakultet; 2010.

De Bosscher V, De Knop P, Vertonghen J. A multidimensional approach to evaluate the policy effectiveness of elite sport schools in Flanders. Sport Soc. 2016;19(10):1596–621.

North J. Sport coaching research and practice: Ontology, interdisciplinarity and critical realism. London: Routledge; 2017.

Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res. 1991;40(2):120–3.

Article   CAS   PubMed   Google Scholar  

Dawadi S, Shrestha S, Giri RA. Mixed-methods research: a discussion on its types, challenges, and criticisms. J Pract Studies Educ. 2021;2(2):25–36.

Pawson R, Tilley N. An introduction to scientific realist evaluation, evaluation for the 21st century: a handbook. Thousand Oaks: SAGE Publications Inc; 1997.

Yin RK. Validity and generalization in future case study evaluations. Evaluation. 2013;19(3):321–32.

Danermark B. et al. Generalization, scientific inference and models for an explanatory social science. In: Explaining society: critical realism in the social sciences; 1997. pp. 73–114.

Wynn-Jr D, Williams CK. Principles for conducting critical realist case study research in information systems. MIS Quarter. 2012;2012:787–810.

Bloom B, Sosniak LA. Developing talent in young people. USA: Random House Publishing Group; 1985.

Wylleman P, Lavallee D. A developmental perspective on transitions faced by athletes. Dev Sport Exerc Psychol: A Lifespan Perspect. 2004;2004:507–27.

Reja U, et al. Open-ended vs close-ended questions in web questionnaires. Develop Appl Stat. 2003;19(1):159–77.

Hendricks S, et al. Rating of perceived challenge as a measure of internal load for technical skill performance. Br J Sports Med. 2019;53(10):611–3.

Scantlebury S, et al. Navigating the complex pathway of youth athletic development: challenges and solutions to managing the training load of youth team sport athletes. Strength Condition J. 2020;42(6):100–8.

Corrado L, et al. Motivation for a dual-career: Italian and Slovenian student-athletes. Kinesiol Sloven. 2012;18(3):47–56.

Lupo C, et al. Motivation towards dual career of European student-athletes. Eur J Sport Sci. 2015;15(2):151–60.

Faria L, Vieira PN. Moderating role of negative career thoughts in the relationship between motivation and burnout in portuguese dual career athletes. J Educ Society Behav Sci. 2022;35(11):106–17.

Fortes PC, Rodrigues, G, Tchantchane A. Investigation of academic and athletic motivation on academic performance among university students. 2010. https://ro.uow.edu.au/dubaipapers/233 .

Guidotti F, et al. Validation of the Italian version of the student athletes’ motivation toward sport and academics questionnaire. Sport Sci Health. 2013;9(2):51–8.

Kellmann M, Kallus KW. Recovery-stress questionnaire for athletes: user manual. Hum Kinet. 2001;2001:124–56.

Nugent FJ, Comyns TM, Warrington GD. Effects of increased training volume during a ten-day training camp on competitive performance in national level youth swimmers. J Sports Med Phys Fitness. 2017;58(12):1728–34.

PubMed   Google Scholar  

Rongen F, et al. Psychosocial outcomes associated with soccer academy involvement: longitudinal comparisons against aged matched school pupils. J Sports Sci. 2020;38(11–12):1387–98.

Vierimaa M, et al. Positive youth development: a measurement framework for sport. Int J Sports Sci Coach. 2012;7(3):601–14.

Cox RH, Martens MP, Russell WD. Measuring anxiety in athletics: the revised competitive state anxiety inventory–2. J Sport Exerc Psychol. 2003;25(4):519–33.

Fernández-Fernández J, et al. Psychophysiological stress responses during training and competition in young female competitive tennis players. Int J Sports Med. 2015;36(01):22–8.

Garner DM, et al. The eating attitudes test: psychometric features and clinical correlates. Psychol Med. 1982;12(4):871–8.

Lee S, et al. Screening Chinese patients with eating disorders using the Eating Attitudes Test in Hong Kong. Int J Eating Disord. 2002;32(1):91–7.

Mintz LB, O’Halloran MS. The eating attitudes test: validation with DSM-IV eating disorder criteria. J Personal Assess. 2000;74(3):489–503.

Article   CAS   Google Scholar  

Garner DM, Rosen LW, Barry D. Eating disorders among athletes: research and recommendations. Child Adolesc Psychiatr Clin N Am. 1998;7(4):839–57.

Rosendahl J, et al. Dieting and disordered eating in German high school athletes and non-athletes. Scand J Med Sci Sports. 2009;19(5):731–9.

Boyadjieva S, Steinhausen HC. The Eating Attitudes Test and the Eating Disorders Inventory in four Bulgarian clinical and nonclinical samples. Int J Eating Disord. 1996;19(1):93–8.

Buddeberg-Fischer B, et al. Epidemiology of eating behaviour and weight distribution in 14–to 19-year-old Swiss students. Acta Psychiatr Scand. 1996;93(4):296–304.

Aschenbrenner K. et al. Störungen des Essverhaltens bei gymnasiasten und studenten. PPmP-Psychother· Psychosom Med Psychol. 2004; 54 (06):T1–T13.

The KIDSCREEN Group Europe. The KIDSCREEN questionnaires quality of life questionnaires for children and adolescents handbook. Lengerich: Pabst Science Publishers; 2006.

Robitail S, et al. Validation of the European proxy KIDSCREEN-52 pilot test health-related quality of life questionnaire: first results. J Adolesc Health. 2006;39(4):596.

Cronin LD, Allen J. Development and initial validation of the Life Skills Scale for Sport. Psychol Sport Exerc. 2017;28:105–19.

Smith BW, et al. The brief resilience scale: assessing the ability to bounce back. Int J Behav Med. 2008;15(3):194–200.

Sorkkila M, et al. The role of resilience in student-athletes’ sport and school burnout and dropout: a longitudinal person-oriented study. Scand J Med Sci Sports. 2019;29(7):1059–67.

The Training of Young Athletes (TOYA) study: TOYA and Education. Sports Council, London: Sports Council. 1993.

Sawczuk T, et al. Between-day reliability and usefulness of a fitness testing battery in youth sport athletes: Reference data for practitioners. Meas Physic Educ Exerc Sci. 2018;22(1):11–8.

Cronin JB, Templeton RL. Timing light height affects sprint times. J Strength Cond Res. 2008;22(1):318–20.

Sawczuk T, et al. Relationships between training load, sleep duration, and daily well-being and recovery measures in youth athletes. Pediatr Exerc Sci. 2018;30(3):345–52.

Beckham G, et al. Relationships of isometric mid-thigh pull variables to weightlifting performance. J Sports Med Phys Fitness. 2013;53(5):573–81.

CAS   PubMed   Google Scholar  

Darrall-Jones JD, et al. Reliability and usefulness of linear sprint testing in adolescent rugby union and league players. J Strength Cond Res. 2016;30(5):1359–64.

Harman EA, et al. Effects of two different eight-week training programs on military physical performance. J Strength Cond Res. 2008;22(2):524–34.

Foddy M, Crundall I. A field study of social comparison processes in ability evaluation. Br J Soc Psych. 1993;32(4):287–305.

Creswell JW. Steps in conducting a scholarly mixed methods study. DBER Speaker Series, 48. 2013.

Phillippi J, Lauderdale J. A guide to field notes for qualitative research: context and conversation. Qual Health Res. 2018;28(3):381–8.

Patton MQ. Two decades of developments in qualitative inquiry: a personal, experiential perspective. Qual Soc Work. 2002;1(3):261–83.

Smith B, McGannon KR. Developing rigor in qualitative research: problems and opportunities within sport and exercise psychology. Int Review Sport Exerc Psychol. 2018;11(1):101–21.

Sparkes AC, Smith B. Qualitative research methods in sport, exercise and health: from process to product. London: Routledge; 2013.

Assidiqqi MG, et al. A descriptive study of the achievement timeline of volleyball athletes at State University of Malang. J SPORTIF: J Penelitian Pembelajaran. 2022;8(3):37–60.

Terry G, et al. Thematic analysis. Sage Handb Qual Res Psychol. 2017;2:17–37.

Roller MR, Lavrakas PJ. Applied qualitative research design: a total quality framework approach. New York: Guilford Publications; 2015.

Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398–405.

Farmer T, et al. Developing and implementing a triangulation protocol for qualitative health research. Qual Health Res. 2006;16(3):377–94.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches. Thousand Oaks: Sage publications; 2017.

Fryer T. A critical realist approach to thematic analysis: producing causal explanations. J Crit Realism. 2022;2022:1–20.

Hirose M, Creswell JW. Applying core quality criteria of mixed methods research to an empirical study. J Mix Methods Res. 2023;17(1):12–28.

Ronkainen NJ, Wiltshire G. Rethinking validity in qualitative sport and exercise psychology research: a realist perspective. Int J Sport Exerc Psychol. 2021;19(1):13–28.

Andersson R, Barker-Ruchti N. Career paths of Swedish top-level women soccer players. Soccer Soc. 2019;20(6):857–71.

Romar J-E. An analysis of Finnish skiing school students’ academic education and athletic success. Acta Gym. 2012;42(1):35–41.

Stambulova NB, et al. Searching for an optimal balance: dual career experiences of Swedish adolescent athletes. Psychol Sport Exerc. 2015;21:4–14.

Post EG, et al. The association of sport specialization and training volume with injury history in youth athletes. Am J Sports Med. 2017;45(6):1405–12.

Cosh S, Tully PJ. Stressors, coping, and support mechanisms for student athletes combining elite sport and tertiary education: Implications for practice. Sport Psychol. 2015;29(2):120–33.

Kristiansen E, Stensrud T. Young female handball players and sport specialisation: how do they cope with the transition from primary school into a secondary sport school? Br J Sports Med. 2017;51(1):58–63.

Cosh S, Tully PJ. “All I have to do is pass”: a discursive analysis of student athletes’ talk about prioritising sport to the detriment of education to overcome stressors encountered in combining elite sport and tertiary education. Psychol Sport Exerc. 2014;15(2):180–9.

McGillivray D, Fearn R, McIntosh A. Caught up in and by the beautiful game: a case study of Scottish professional footballers. J Sport Soc Issues. 2005;29(1):102–23.

Borggrefe C, Cachay K. “Dual Careers”: the structural coupling of elite sport and school exemplified by the German Verbundsysteme. Eur J Sport Soc. 2012;9(1–2):57–80.

Hartmann, D. High school sports participation and educational attainment: Recognizing, assessing, and utilizing the relationship. In: Report to the LA84 Foundation; 2008. pp. 130–8.

van Rens FE, Elling A, Reijgersberg N. Topsport Talent Schools in the Netherlands: a retrospective analysis of the effect on performance in sport and education. Int Rev Sociol Sport. 2015;50(1):64–82.

Beckmann, J. Chancen und Risiken: vom Leben im Verbundsystem von Schule und Leistungssport: Psychologische, soziologische und sportliche Leistungsaspekte. 2006. Sportverl. Strauß.

Rongen F, et al. Psycho-social maturation and the implications for coaching children. Routledge Handb Coach Children Sport. 2022;2022:274.

Henriksen K, Stambulova N, Roessler KK. Riding the wave of an expert: a successful talent development environment in kayaking. Sport Psychol. 2011;25(3):341–62.

Ronkainen NJ, et al. Superwomen? Young sporting women, temporality and learning not to be perfect. Int Rev Sociol Sport. 2021;56(8):1137–53.

Cronin LD, Allen J. Examining the relationships among the coaching climate, life skills development and well-being in sport. Int J Sports Sci Coach. 2018;13(6):815–27.

Duz S, Aslan TV. The effect of sport on life skills in high school students. Asian J Educ Train. 2020;6(2):161–8.

Commission E. EU guidelines on dual careers of athletes: recommended policy actions in support of dual careers in high-performance sport. In: Sport Unit, European Commission, Education, Culture and Sport Brussels. 2012.

Rongen F, et al. Do youth soccer academies provide developmental experiences that prepare players for life beyond soccer? A retrospective account in the United Kingdom. Sport Exerc Perform Psychol. 2021;2021:359–80.

O’Connor D, et al. Positive youth development and gender differences in high performance sport. J Sports Sci. 2020;38(11–12):1399–407.

Davison TE. Body image and psychological, social, and sexual functioning. London: Deakin University; 2002.

Sabiston C, et al. Changes in body-related self-conscious emotions over time among youth female athletes. Body Image. 2020;32:24–33.

Soulliard ZA, et al. Examining positive body image, sport confidence, flow state, and subjective performance among student athletes and non-athletes. Body Image. 2019;28:93–100.

Murray K, Rieger E, Byrne D. The relationship between stress and body satisfaction in female and male adolescents. Stress Health. 2015;31(1):13–23.

Phibbs PJ, et al. We know they train, but what do they do? Implications for coaches working with adolescent rugby union players. Int J Sports Sci Coach. 2017;12(2):175–82.

Hartwig TB, Naughton G, Searl J. Load, stress, and recovery in adolescent rugby union players during a competitive season. J Sports Sci. 2009;27(10):1087–94.

Bahr, R. Demise of the fittest: are we destroying our biggest talents? In: BMJ Publishing Group Ltd and British Association of Sport and Exercise Medicine; 2014. p. 1265–1267.

Booth M, Orr R, Cobley S. Call for coordinated and systematic training load measurement (and progression) in athlete development: a conceptual model with practical steps. In: BMJ Publishing Group Ltd and British Association of Sport and Exercise Medicine; 2017. p. 559–560.

Matos NF, Winsley RJ, Williams CA. Prevalence of nonfunctional overreaching/overtraining in young English athletes. Med Sci Sports Exerc. 2011;43(7):1287–94.

Seiler S. What is best practice for training intensity and duration distribution in endurance athletes. Int J Sports Physiol Perform. 2010;5(3):276–91.

Umbach PD, et al. Intercollegiate athletes and effective educational practices: winning combination or losing effort? Res High Educ. 2006;47(6):709–33.

Comeaux E, Harrison CK. A conceptual model of academic success for student–athletes. Educ Res. 2011;40(5):235–45.

Kristiansen E. Walking the line: how young athletes balance academic studies and sport in international competition. Sport Soc. 2017;20(1):47–65.

Brown S. Learning to be a ‘goody-goody’: ethics and performativity in high school elite athlete programmes. Int Rev Sociol Sport. 2016;51(8):957–74.

Capranica L, Millard-Stafford ML. Youth sport specialization: how to manage competition and training? Int J Sports Physiol Perform. 2011;6:4.

Knight CJ, Harwood CG, Sellars PA. Supporting adolescent athletes’ dual careers: the role of an athlete’s social support network. Psychol Sport Exerc. 2018;38:137–47.

Johnson MB, et al. “Hard work beats talent until talent decides to work hard”: coaches’ perspectives regarding differentiating elite and non-elite swimmers. Int J Sports Sci Coach. 2008;3(3):417–30.

Holt NL, Morley D. Gender differences in psychosocial factors associated with athletic success during childhood. Sport Psychol. 2004;18:2.

Morgan TK, Giacobbi PR. Toward two grounded theories of the talent development and social support process of highly successful collegiate athletes. Sport Psychol. 2006;20(3):295–313.

Li C, Wang CJ, Pyun DY. Talent development environmental factors in sport: a review and taxonomic classification. Quest. 2014;66(4):433–47.

Henriksen K, Knight C, Araújo D. Talent development environments in the Routledge International encyclopedia of sport and exercise psychology. London: Routledge; 2020. p. 658–70.

Price R, et al. The Football Association medical research programme: an audit of injuries in academy youth football. Br J Sports Med. 2004;38(4):466–71.

DiFiori JP, et al. Overuse injuries and burnout in youth sports: a position statement from the American Medical Society for Sports Medicine. Br J Sports Med. 2014;48(4):287–8.

Gabbett TJ, et al. The relationship between workloads, physical performance, injury and illness in adolescent male football players. Sports Med. 2014;44(7):989–1003.

Brink MS, Kersten AW, Frencken WG. Understanding the mismatch between coaches’ and players’ perceptions of exertion. Int J Sports Physiol Perform. 2017;12:4.

Gabbett TJ. The training—injury prevention paradox: should athletes be training smarter and harder? Br J Sports Med. 2016;50(5):273–80.

Brink MS, et al. Coaches’ and players’ perceptions of training dose: not a perfect match. Int J Sports Physiol Perform. 2014;9(3):497–502.

Haddad M, Padulo J, Chamari K. The usefulness of session rating of perceived exertion for monitoring training load despite several influences on perceived exertion. Int J Sports Physiol Perform. 2014;9(5):882–3.

Kenttä G, Hassmén P, Raglin J. Training practices and overtraining syndrome in Swedish age-group athletes. Int J Sports Med. 2001;22(06):460–5.

Lloyd RS, et al. Long-term athletic development-part 1: a pathway for all youth. J Strength Condition Res. 2015;29(5):1439–50.

Phibbs PJ, et al. Organized chaos in late specialization team sports: weekly training loads of elite adolescent rugby union players. J Strength Condition Res. 2018;32(5):1316–23.

Phibbs PJ, et al. The organised chaos of English adolescent rugby union: influence of weekly match frequency on the variability of match and training loads. Eur J Sport Sci. 2018;18(3):341–8.

Ko B-G, et al. The construction of sports talent identification models. Int J Appl Sports Sci. 2003;15(2):64–84.

McLean BD, et al. Neuromuscular, endocrine, and perceptual fatigue responses during different length between-match microcycles in professional rugby league players. Int J Sports Physiol Perform. 2010;5(3):367–83.

Oliver JL, Lloyd RS, Meyers RW. Training elite child athletes: promoting welfare and well-being. Strength Condition J. 2011;33(4):73–9.

Laurent CM, et al. A practical approach to monitoring recovery: development of a perceived recovery status scale. J Strength Condition Res. 2011;25(3):620–8.

Collins D, MacNamara A. Talent development: a practitioner guide. Cambridge: Routledge; 2017.

Collins D, MacNamara Á. The rocky road to the top: why talent needs trauma. Sports Med. 2012;42:907–14.

Collins DJ, Macnamara A. Making champs and super-champs—current views, contradictions, and future directions. Front Psychol. 2017;2017:823.

Sharma H. How short or long should be a questionnaire for any research? Researchers dilemma in deciding the appropriate questionnaire length. Saudi J Anaesth. 2022;16(1):65–8.

Cobley SP, et al. Variable and changing trajectories in youth athlete development: further verification in advocating a long-term inclusive tracking approach. J Strength Condition Res. 2014;28(7):1959–70.

Ivarsson A, et al. The predictive ability of the talent development environment on youth elite football players’ well-being: a person-centered approach. Psychol Sport Exerc. 2015;16:15–23.

Download references

Author information

Authors and affiliations.

Room G07, Cavendish Hall, Carnegie School of Sport, Leeds Beckett University, Headingley Campus, Leeds, LS6 3QS, UK

Ffion Thompson, Ian Cowburn & Kevin Till

Queen Ethelburga’s College, York, UK

Ffion Thompson

School of Science and Technology, Nottingham Trent University, Nottingham, UK

Fieke Rongen

Leeds Rhinos Rugby League Club, Leeds, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ffion Thompson .

Ethics declarations

Approval was obtained (29/07/2021) from the ethics committee of Leeds Beckett University (Ref. 86728). The study was performed in accordance with the standards of ethics outlined in the Declaration of Helsinki.

Consent for Publication

Consent for publication was obtained electronically from all participants, parents and gatekeeper.

Availability of Data and Materials

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

No sources of funding were used to assist in the preparation of this article.

Conflict of Interest

The authors (Ffion Thompson, Fieke Rongen, Ian Cowburn, and Kevin Till) all declared no potential conflicts of interest concerning the research, the content, authorship, and/or publication of the review.

Authors' Contributions

The first author, FT, collected the data and was the lead on the analysis and writing. FR, IC and KT contributed to the conception and design of the study, drafting and critically revising the manuscript, and approval for publication. All authors read and approved the final manuscript.

Consent to Participate

Online informed assent and parental consent were obtained from all participants.

Code Availability

Not applicable.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Thompson, F., Rongen, F., Cowburn, I. et al. A Longitudinal Mixed Methods Case Study Investigation of the Academic, Athletic, Psychosocial and Psychological Impacts of Being of a Sport School Student Athlete. Sports Med (2024). https://doi.org/10.1007/s40279-024-02021-4

Download citation

Accepted : 11 March 2024

Published : 18 April 2024

DOI : https://doi.org/10.1007/s40279-024-02021-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research

• Research article
• Open access
• Published: 12 May 2020

A mixed methods case study investigating how randomised controlled trials (RCTs) are reported, understood and interpreted in practice

• Ben E. Byrne   ORCID: orcid.org/0000-0002-2183-8166 1 ,
• Leila Rooshenas 1 ,
• Helen S. Lambert 2 &
• Jane M. Blazeby 1 , 3 , 4  

BMC Medical Research Methodology volume  20 , Article number:  112 ( 2020 ) Cite this article

3120 Accesses

2 Citations

10 Altmetric

Metrics details

While randomised controlled trials (RCTs) provide high-quality evidence to guide practice, much routine care is not based upon available RCTs. This disconnect between evidence and practice is not sufficiently well understood. This case study explores this relationship using a novel approach. Better understanding may improve trial design, conduct, reporting and implementation, helping patients benefit from the best available evidence.

We employed a case-study approach, comprising mixed methods to examine the case of interest: the primary outcome paper of a surgical RCT (the TIME trial). Letters and editorials citing the TIME trial’s primary report underwent qualitative thematic analysis, and the RCT was critically appraised using validated tools. These analyses were compared to provide insight into how the TIME trial findings were interpreted and appraised by the clinical community.

23 letters and editorials were studied. Most authorship included at least one academic (20/23) and one surgeon (21/23). Authors identified wide-ranging issues including confounding variables or outcome selection. Clear descriptions of bias or generalisability were lacking. Structured appraisal identified risks of bias. Non-RCT evidence was less critically appraised. Authors reached varying conclusions about the trial without consistent justification. Authors discussed aspects of internal and external validity covered by appraisal tools but did not use these methodological terms in their articles.

Conclusions

This novel method for examining interpretation of an RCT in the clinical community showed that published responses identified limited issues with trial design. Responses did not provide coherent rationales for accepting (or not) trial results. Findings may suggest that authors lacked skills in appraisal of RCT design and conduct. Multiple case studies with cross-case analysis of other trials are needed.

Peer Review reports

It is widely recognised that clinical practice is often not in line with the best available evidence. This is the so-called ‘gap’ between research and practice [ 1 , 2 ]. Best evidence predominantly comes from well designed and conducted randomised controlled trials (RCTs) [ 3 ]. However, RCTs are often complex and challenging. Surgical RCTs present specific issues with recruitment, blinding of patients and surgeons, and intervention standardisation [ 4 ]. Many of these issues have been clarified with methodological research [ 5 , 6 , 7 , 8 , 9 , 10 ]. Such work has led to improvements in trial quality over time [ 11 , 12 ]. However, the gap between trials and implementation of their results in practice persists [ 13 ], potentially compromising patient care and wasting resources. Reasons for the disconnect are myriad.

Trial findings that report putative evidence for a change in clinical practice may not be implemented because of poor conduct and reporting [ 14 ], limitations in generalisation and applicability [ 15 ], cost, and unacceptability of new interventions. Clinical culture may emphasise the importance of experience over evidence [ 16 ], and some clinicians may have limited numeracy skills required to understand and apply quantitative results from trials [ 17 ]. Appropriate understanding of RCTs is critical to implementation and of vital importance to clinicians, researchers and funders. We have previously described a novel approach to explore understanding and interpretation of RCT evidence, by examining writings about individual surgical trials [ 18 ]. The present study aims to apply this new method to a single case study: the TIME (Traditional Invasive versus Minimally invasive Esophagectomy) RCT [ 19 ]. The purpose is to better understand how this trial has been interpreted and to illustrate the potential of this novel approach.

The methodology used in this study has been described in detail elsewhere [ 18 ] and will be summarised here. The approach represents a form of case-study research, comprising mixed methods analysis of documentary evidence relating to a published RCT [ 20 ]. Case-study approaches have been defined in various ways and used across numerous disciplines. Their central tenet is to explore an event or phenomenon in depth and in its natural context [ 21 ]. The ‘real-world context’ in this study was the landscape of published articles that interpreted, appraised and discussed implementation of the TIME trial’s findings. Our approach aligned with Stake’s ‘instrumental case-study’ [ 22 ], using a particular case (the TIME RCT’s outcomes paper) to gain a broader appreciation of the issue or phenomenon of interest (in this case, interpretation and appraisal of RCTs in the clinical community, and implications for implementation). We conducted qualitative analysis of selected published articles citing this RCT’s primary report and compared this with structured critical appraisal of the RCT using established tools. We also sought to demonstrate the utility of this novel approach, which we intend to apply in future case studies.

Identify and analyse articles citing a trial

Purposefully select a major surgical rct.

An index RCT was identified and summarised as the case of interest. We sought a highly cited trial report, published in a high-impact journal within the last 10 years. The TIME trial [ 19 ], comparing open and minimally invasive surgical access for removal of oesophageal cancer, was selected as it met these criteria and was within our area of expertise.

Identify and systematically sample articles citing the RCT

All articles citing this RCT were identified using Web of Science and Scopus citation tracking tools. Letter, editorial and discussion article types were included. On-line comments were identified using the Altmetric.com bookmarklet. Non-English language articles were excluded. Searches were conducted in October 2017.

Undertake in-depth qualitative analysis and identify relevant themes

Included articles were thematically analysed using the constant comparison technique, adopted from grounded theory [ 23 , 24 ]. Articles were read in detail, with no a priori coding framework. Text was considered against the research topic, which focused on understanding how the authors interpreted, appraised and/or applied the findings of the trial. New findings or interpretations were continuously related to existing findings to develop the data set as a whole (i.e. the constant comparison technique). Coding was not constrained by pre-defined boundaries defining relevance. Rather, this was guided by the content of the articles being analysed. During analysis, it transpired that understanding authors’ interpretations of the RCT required examination of their discussion of evidence from other studies. Therefore, other articles cited by the authors were sought to determine the types of evidence being referenced. The designs of these additional studies were ascertained based on the descriptions in those articles (rather than our assessment).

Analysis was performed by BEB and LR. BEB is a senior surgical trainee and postdoctoral researcher with previous experience of qualitative research. LR is a Lecturer in Qualitative Health Science with an interest in trial recruitment issues, implementation of trial evidence, and experience of working on multiple surgical RCTs. Both researchers work within a department with expertise in trials methodology and have detailed knowledge in this field which is likely to have influenced their identification and coding of relevant themes.

Two rounds of double coding of five articles were performed by BEB and LR. Further coding was conducted by BEB and reviewed among the team to revise coded themes. Descriptive data on authorship and origins of the articles were collected.

Summarise validity and reporting of the RCT

The RCT was assessed by BEB using a range of critical appraisal tools commonly used to appraise RCTs. These included two of the most commonly used tools to assess RCTs: one examining trial reporting in a broad sense (Consolidated Standards of Reporting Trials for Non-Pharmacological Treatments (CONSORT-NPT) [ 5 ]), and another focusing on internal validity as commonly assessed in systematic reviews of trials (the updated Cochrane Risk of Bias Tool (ROBT 2.0) [ 7 ]). In addition, the Pragmatic Explanatory Continuum Indicator Scale (PRECIS-2) tool [ 8 ] was included, to examine domains associated with the broad applicability and utility of the trial, and the Context and Implementation of Complex Interventions (CICI) framework [ 25 ] was included on an exploratory basis to identify broader contextual factors that could be relevant. JMB contributed to assessment during piloting of the tools and in discussion with BEB where there was uncertainty.

Broad comparison of all results to develop deeper understanding of how trials are understood and relationship with trial quality

The results of both qualitative analysis and structured critical appraisal were considered side-by-side, with the overall aim of better understanding how other authors’ interpretations of the TIME trial compared with the critical appraisal guided by the above tools. The qualitative analysis of the authors’ interpretations was conducted before the structured critical appraisal to ensure the coding/themes were grounded in authors’ writings, rather than our experience of conducting the structured appraisals. The final step aimed to draw together both analyses, to see whether authors discussing the trial raised concerns across similar domains to the areas covered by the critical appraisal tools, or whether their topics of discussion addressed other considerations.

Ethical considerations

This study involved secondary use of publicly available written material and did not require ethical review.

Patient and public involvement

Patients and members of the public were not involved in any aspect of the design of this study.

Summary of index RCT

The TIME trial was a two-group, multicentre randomised trial comparing a minimally invasive approach to the surgical removal of oesophageal cancer with an open approach to the abdomen and chest. It was conducted in five centres across four European countries from 2009 to 2011 and is summarised in Table  1 .

Characteristics of articles

Searches identified 26 articles, and 23 were included (exclusions: an incorrectly classified case report and two articles in German). Summary characteristics are provided in Table  2 . Most articles (18/23, 78%) originated from Europe or the United States. The majority (20/23, 87%) included at least one author holding an academic position; 18/23 (78%) included at least one professor or associate professor (as defined within their own institution). Nearly all included at least one consultant or trainee surgeon (21/23, 91%).

Altmetric.com identified several references to the TIME trial, detailed in Table  3 . Only one, part of the British Medical Journal blog series, included text discussing the trial, rather than simply restating its results or directing readers to the study report.

Themes identified

Qualitative analysis resulted in description of three key themes: identification of wide-ranging issues with the RCT; limited appraisal of non-RCT studies; and variable recommendations for future practice and research. Codes linking quotes to articles and bibliographic data are provided in supplementary Table  1 .

Identification of wide-ranging issues with the RCT

Authors extensively discussed and critiqued several features of trial design and conduct. These included the population, intervention and outcomes of the trial.

If the author’s primary outcome was focused on pulmonary infection, perhaps other patient associated inclusion / exclusion criteria may have been of value. These would include patients with poor pulmonary function parameters … patients with major organ disease … and recent history of prior malignancy. (E2).

In the present [TIME] trial, the difference between minimally invasive and open oesophagectomy was maximised with a purely thoracoscopic (prone position) and laparoscopic technique. (E1).

The primary outcome … was pulmonary infection within the first 2 weeks after surgery and during the whole stay in hospital. This cannot be considered as the relevant primary outcome with reference to the decision problem outline by the authors … (E5).

Beyond these basic trial design parameters, authors of the citing articles also highlighted important confounding variables.

Many non-studied variables, including malnutrition, previous and current smoking, pulmonary comorbidities, functional status, and clinical TNM (tumour, node, metastasis) staging, have all been shown to strongly affect the primary endpoint of this trial – postoperative pulmonary infection. (L2).

Several correspondents suggest that lower rates of respiratory infection might have been achieved by use of alternative strategies for preoperative preparation, patient positioning, ventilator settings, anaesthetic agents, or postoperative care. (L6).

The articles also covered other potential problems with the trial, such as sample size and learning curve effects.

The sample size for sufficient statistical power for major morbidity, survival, total morbidity and other similarly important outcomes may actually be larger. (E2).

The inclusion criteria for participating surgeons appears to have the performance of a minimum of only 10 MIOs and this low level of experience may be reflected in relatively high conversion rate of 13%. (E4).

Only one article (E2) made clear statements praising aspects of the trial:

‘…The protocols for the RCT appear sound with randomization, intention to treat, PICO … and bias elimination.’

The next sentence of this article balanced these positive comments with discussion of limits due to the lack of blinding and other potential confounding variables.

Limited appraisal of non-RCT studies

Authors often cited other types of evidence in the same field to support their views without discussing their methodological limitations. Types of evidence included single-surgeon series, non-randomised comparative studies, systematic reviews (SRs) and meta-analyses (MAs).

Luketich et al. , one of the earlier pioneers of MIE, reported their extensive experience of 1033 consecutive patients undergoing MIE with acceptable lymph node resection, postoperative outcomes, and a 1.7% mortality rate. (L8).

In a population-based national study, … the incidence of pneumonia was 18.6% after open oesophagectomy and 19.9% after minimally invasive oesophagectomy … (L3).

Although systematic reviews and a large comparative study of minimally invasive oesophagectomy have not shown this technique to be beneficial as compared with open oesophagectomy, some meta-analyses have suggested specific advantages. (E1).

The existing SRs and MAs were discussed in relation to the intervention and its outcomes, without directly relating them to the TIME trial itself. The implications for authors’ impressions of the TIME trial findings were generally unclear.

There was limited appraisal of these SRs and MAs, especially when contrasted with discussion of the TIME trial. Several authors referred to the large, single-surgeon series of MIO by Luketich, but only one author described limits of this single-institution non-comparative study.

We must not rely on the limitation of single-institution studies and historical data. This procedure must be broadly applicable and not the domain of a few experts for it to become the new gold standard. (E12).

A few others highlighted the limits of other study designs, but there was a striking disparity in the level of critique, when compared with that of the TIME trial.

In their systematic review … Uttley et al. correctly conclude that due to factors such as selection bias, sufficient evidence does not exist to suggest the MIO is either equivalent to or superior to open surgery. (E6).

All these studies however, concede that due to a lack of feasible evidence by way of prospective randomized controlled trials (RCT), no definitive statement of MIE ‘superiority’ over standard open techniques can be made. (E2).

Although several authors referred to the existing SRs and MAs, none reported the design of the included primary studies, which were largely retrospective and non-randomised.

Variable recommendations for future practice and research

The authors had differing interpretations and recommendations for implementation based on the TIME trial. Some articles discussed issues with the trial and did not make recommendations for future practice, in some cases asking for additional information to better understand or interpret the trial.(L1, L3–5) For example, one simply wrote that the authors ‘have several concerns’, before reporting differences in outcomes between TIME and other studies, and describing practice in their own institution. (L1) Others reported that more work was required, such as further analysis of long-term results of patients included in TIME, or called for further trials in different patient populations.

However, the main issue which this study [TIME] does not address is that of long-term survival. … If the authors can indeed demonstrate at least equivalent long-term oncological outcome for MIO and open oesophagectomy, then this paper should provide an impetus for driving forward the widespread adoption of MIO. (E4).

Of interest will be whether similar results can be repeated in patients in Asia, with mainly squamous cell cancers that are proximally located. … The substantial benefit shown in this trial [TIME] … might encourage investigators to do further randomised studies at other centres. If these results can be confirmed in other settings, minimally invasive oesophagectomy could truly become the standard of care. (E1).

One article (E6) considered the evidence for MIO, discussed this against methodological aspects of a colorectal trial evaluating a minimally invasive approach, before restating the findings of TIME, opining that:

‘This study confirms that RCT [sic] for open versus MIO is indeed possible, but further larger trials are required.’

Later in that article, the authors suggested extensive control of wide-ranging aspects of perioperative care would be important for future trials.

Authors of three articles (E7, E9, E11) suggested that the available evidence was enough for increasing adoption of MIO.

…The available evidence increasingly favors a prominent role for minimally invasive approaches in the management of esophageal cancer. Endoscopic therapies and minimally invasive approaches offer at least equivalent oncologic outcomes, with reduced complications and improved quality of life compared with maximal surgery. (E11).

We are close to a situation in which one can argue that MIE is ready for prime time in the curative treatment of invasive esophageal cancer. If we critically analyse the level and grading of evidence, the current situation concerning MIE and hybrid MIE is far better than was the case when laparoscopic cholecystectomy, anti-reflux surgery, and bariatric surgery were introduced into clinical practice. (E9).

No authors called for the cessation of MIO, although one referred to some centres stopping ‘their MIE [minimally invasive esophagectomy] program due to safety reasons’. (E13).

Assessment of RCT using validated tools

The TIME trial results and protocol papers [ 19 , 26 ] were examined to assess the trial and its reporting. Assessment using CONSORT-NPT demonstrated reporting shortfalls in several areas (full notes in supplementary Table  2 ). These included: lack of information on adherence of care providers and patients to the treatment protocol; discrepancies between the primary outcomes proposed in the protocol (3 pulmonary outcomes) and the trial report (one pulmonary result); no information on interim analyses or stopping criteria; a lack of information regarding statistical analysis to allow for clustering of patients by centre; and absence of discussion of the trial limitations or generalisability.

Risk of bias was assessed as shown in Table  4 . Overall, the TIME trial was considered at high risk of bias.

Assessment using the PRECIS-2 tool is shown in Table  5 . Overall, TIME had features in keeping with a more pragmatic rather than explanatory trial. This suggested a reasonable degree of applicability and usefulness to wider clinical practice.

Application of the CICI framework highlighted several higher-level considerations relevant to the applicability of the TIME trial not described in the protocol or study report (see Table  6 ). These included lack of detail on the setting, as well as epidemiological and socio-economic information.

Overall, these tools suggested that TIME had several limitations. These included issues with standardisation and monitoring of intervention adherence, lack of blinding, failure to use hierarchical analysis and a lack of information on provider volume. The risk of bias was high, limiting confidence attributing outcomes to the allocated interventions. Broad applicability was considered reasonable, though study utility was compromised by a short-term clinical outcome, rather than longer term or patient-reported outcomes. While TIME may have provided early evidence for benefit of MIO to reduce pulmonary infection within 2 weeks of surgery, the appraisal suggested more evidence was needed before considering wider adoption of MIO.

Broad comparison of all results to develop deeper understanding

We considered the findings from the qualitative analysis in relation to those of the critical appraisal. In doing so, broad domains of internal and external validity seemed a useful system to bring together results of both analyses. While the ROBT was described by its creators as focused on internal validity, the PRECIS-2 and CICI tools were not described in terms of validity. Rather, their authors referred to applicability and reproducibility in other settings, which may also be described as external validity. CONSORT-NPT is a tool focused on reporting of trials, and its authors referred to both domains, with some duplication of factors covered in the other tools. However, authors of the articles included in the qualitative analysis did not adopt such methodological terminology when expressing concerns about these aspects of the index RCT’s conduct or reporting.

Robust internal validity allows confident attribution of treatment effects to the experimental intervention. The ROBT identified high risk of bias in the TIME trial. Qualitative analysis revealed discussion of various aspects relevant to internal validity. For example, several authors discussed differences in patient positioning and anaesthetic techniques. These confounding variables may have introduced systematic differences in care between groups, aside from the allocated intervention, resulting in bias. However, the article authors did not articulate the implications of their concerns in such terms and did not consider whether these problems rendered the trial fatally biased.

Sound external validity suggests similar treatment effects may be achieved by other clinicians in other settings for other patients. Pragmatic trials have broad applicability, with wide inclusion criteria, and patient-centred outcomes. The PRECIS-2 describes domains relevant to this applicability. TIME had several features of a pragmatic trial, suggesting relatively broad applicability. The qualitative analysis showed authors were concerned about these issues. For example, several discussed the appropriateness and utility of 2-week and in-hospital pulmonary infection rates as the primary outcome measure. However, authors did not directly relate such concerns to external validity or generalisability, to reach a conclusion about whether the trial should influence practice.

While many authors identified issues relevant to internal and external validity, the lack of clear explanation of their implications meant it was difficult to determine whether they thought the trial justified a change in practice. This contrasts with the structured assessments, which defined clear problems with the trial and limits to its usefulness.

This study presents the first application and results of a new method to generate insights into how evidence from a trial was understood, contextualised and related to practice. Qualitative analysis of letters and editorials, largely written by academic surgeons, documented extensive discussion of problems with the trial, but without clear formulation of the implications of these concerns for its internal or external validity and applicability. These authors reached a variety of conclusions about the implications of the trial for surgical practice. A separate assessment using structured tools defined specific weaknesses in trial methodology. Whilst this new approach yielded useful findings in this single case study, the method should be further tested using multiple trials and cross-case analyses. The initial findings based on this single case study suggest a need to clarify standards against which a trial may be assessed to guide decisions about its role in changing practice, and potentially also to guide efforts to influence practitioners to implement change if appropriate. Within this, our findings suggest a need to focus efforts on educating surgeons about trial design and quality, which may contribute to implementation science-based efforts to inform clinical decision-making and implementation of trial results.

This study contributes to the wider literature showing that evidence does not speak for itself. New evidence is often considered alongside competing bodies of existing evidence that may support different ideas, theories or interventions [ 27 , 28 ]. When a study is published, this new evidence is assimilated into the wider scientific context. Its strengths, weaknesses and overall contribution are debated and disputed. Through the lens of Latour’s actor-network theory [ 29 , 30 ], the new trial can be considered a novel actor within the wider network of actors that includes other trials and studies of the intervention, as well as the consumers of this evidence. Those commenting on the trial have an important role in how different features of the trial are identified, discussed and debated, and how its findings are framed. This agency may be influenced by their own clinical experience, education, skill set, work environment and colleagues, amongst other factors. Given these complexities, it is not surprising to find that different authors reached different conclusions about the TIME trial.

The way authors of the included articles used and appraised different types of study raises questions about how the hierarchy of evidence, and the primacy of the RCT, is applied to routine clinical practice. We found extensive criticism of the TIME trial. Article authors described several limitations relating to its population, intervention, associated co-interventions and confounding variables, as well as the outcomes selected. Certainly, the authors presented valid criticisms that limited the trial’s validity, as identified by structured critical appraisal. Over recent years, trials methodologists have worked to better understand and optimise many such aspects of trial conduct. The development of the CONSORT reporting standards promotes detailed description of key methods, such as random sequence generation and allocation concealment, that allow critical judgements about internal validity to be made [ 5 ]. The growth of pragmatic trials, featuring wide inclusion criteria, conducted across multiple sites, with clinically meaningful outcomes, reflects a concerted effort to improve applicability or external validity of RCTs [ 8 , 31 ]. It may never be possible to conduct a ‘perfect’ trial, but improvements in the rigor and transparency of design hopefully ensure that RCTs can provide sufficiently robust evidence that is useful to the broad population of patients and clinicians within a healthcare system. Whether these developments, designed to address valid criticisms of RCTs, are widely understood outside the sphere of trials methodologists is unclear.

Conversely, the authors of the included articles were far less critical of non-RCT evidence. For example, several authors referred to the single-surgeon case-series of Luketich [ 32 ]. Only one author discussed its limitations for generalisation. Surgical skill and performance vary [ 33 ]; what is possible for a single surgeon cannot be generalised to what is usual for most. Similarly, authors cited systematic reviews and meta-analyses without clear description of the original study designs. Evidence synthesis cannot eliminate biases in retrospective, non-randomised studies using statistical techniques. Failure to clearly articulate limitations of these different studies may support our contention that the authors lacked appropriate appraisal skills. Alternatively, it may suggest bias in favour of the intervention, such that the authors understood, but did not want to articulate its limitations.

While RCTs have not been toppled from their position at the top of the hierarchy of evidence about the efficacy of interventions, developments in other areas have seen increasingly sophisticated use of observational data to better understand the effects of treatments. Researchers have taken advantage of increasing availability of vast quantities of genetic data. In epidemiology, the concept of Mendelian randomisation has been used to try and unpick causal relationships from non-causative correlations [ 34 ]. At the patient level, genetic testing of different types of cancer has allowed targeting of treatments according to cellular sensitivities [ 35 ]. The development of such markers by which to tailor treatment have led to proposals of an idealised future whereby individual treatments are entirely personalised according to a panel of markers that accurately predict treatment response and prognosis. These different research approaches are inevitably competing for resources and intellectual priority. However, as has been argued by Backmann, for these other study types to take priority, “what needs to be shown is not only that RCTs might be problematic …, but that other methods such as cohort studies actually have better external validity.” [ 36 ]

Evidence-based medicine aims to apply the best available evidence to individual patients [ 37 ]. This aim, by its very nature, creates a disconnect between evidence from RCTs, which are aggregated studies of groups of patients to determine average effects, and clinical decision-making at the individual level [ 38 ]. This could be considered to represent an insurmountable ‘get-out’ clause, whereby a clinician may always justify deviation from ‘the evidence’ due to differences between the patient in front of them and those included in the relevant study. It may also prove very difficult to allow the theory-based weight of a journal article to over-ride an individual clinician’s personal lived experience of different interventions and their efficacy. This may be particularly problematic in surgical practice [ 16 ] where the practitioner is usually physically connected with the intervention. This may increase the importance attached to experience, even if that experience is at odds with large-scale studies. We do not disagree that clinicians must treat individual patients according to their specific condition and their wishes. However, it may be considered that aggregate practice, across a surgeon’s cases or across a department, should fall roughly in line with an appropriate body of suitably valid and relevant evidence.

Implementation science research has illuminated many factors affecting implementation beyond knowledge of the evidence. Damschroder et al. described the Consolidated Framework for Implementation Research (CFIR) to identify real-world constructs influencing implementation, relating to the intervention, individuals, organisations and systems [ 39 ]. These included ‘evidence strength and quality’ as well as ‘knowledge and beliefs about the intervention’, constructs readily identified within the present study. Their framework also highlights many other important factors such as cost, patient needs and resources, peer pressure, external policies and incentives, and organisational culture. Surgical research has demonstrated wide variation in practice, even in the presence of high quality evidence [ 40 ], and the broad range of factors affecting implementation of interventions, such as Enhanced Recovery After Surgery [ 41 ]. Our approach may contribute as another tool to understand barriers and facilitators to evidence implementation. It may prove particularly useful in conjunction with other methods such as interviews and observations, informed by a relevant framework, such as the Theoretical Domains Framework [ 42 , 43 ].

The early promise of our new method needs further work to conduct multiple case studies of different RCTs to allow cross-case analyses and a more thorough understanding of how RCTs are interpreted and appraised in the landscape of written commentaries. Examination of further case-studies may also inform refinements to the methods. For example, further analyses may indicate recurring themes across case-studies, which may in turn contribute towards a priori coding criteria and more efficient approaches to analyses (e.g. framework analysis [ 44 ]). It will also be important to include assessment of how each trial is situated in the wider context of relevant evidence, across study types. For individual trials, combined qualitative and structured analyses may determine the extent to which that RCT is flawed and requires further evaluation in a more methodologically sound study. Alternatively, it may demonstrate that the problem in bridging the gap between evidence and practice resides in the competition between different bodies of evidence, comprised of different types of study, and appropriate understanding of their strengths and weaknesses, as well as their applicability to practice. Work should also be undertaken to investigate how contemporary practice may have changed alongside publication of such articles, to investigate the relationship between what is written about the trial, and clinical practice as delivered.

While this study has shown the potential of this new method, its strengths and limitations must be considered. Rigorous analysis using robust qualitative methods and double coding by experienced researchers was undertaken. The articles examined were written without knowledge that they would be analysed in this manner, limiting bias this could introduce. The use of multiple tools to assess the index RCT created a broad overview of its strengths and weaknesses. The most important study limitation was that we did not directly explore authors’ understandings and interpretations, so underlying understanding of the key issues was inferred, rather than directly scrutinised. Failure to articulate is not the same as a lack of understanding. Further, we did not ask authors their motivations to publish their articles, an activity with its own significance. In addition, this study attempted to provide insights into the authors understanding and interpretation of the trial, and it does not purport to be an assessment of practice itself, which would benefit from other approaches to investigation (e.g. qualitative observations, interviews, quantitative procedure rate analyses). This study applied our new method to a single, surgical RCT. The issues identified may be particular to that intervention, specialty, or trial design; further case studies are required to determine broader relevance.

This study has successfully applied a new method to better understand how clinicians and academics understand evidence from a surgical RCT - the TIME trial. It identified discussion of many issues with the trial, but the authors who cited the trial did not specifically articulate the implications of these issues in terms of its internal and external validity. The authors reached a wide range of conclusions, ranging from further evaluation of the intervention, to widespread adoption. Structured appraisal of TIME suggested that the trial was at high risk of bias with limited generalisability. Further application of this method to multiple trials will allow cross-case analyses to determine whether the issues identified are similar across other trials and yield information to better understand how this type of evidence is interpreted and related to practice. This approach may be complemented by other data, such as in-depth interviews. This may reveal genuine flaws in trial design that limit application, or that other issues such as poor understanding or competing non-clinical factors impede the translation of evidence into practice. We hope that this work may help existing efforts to close the research-practice gap, and help ensure that patients receive the best care, based upon the highest level of evidence.

Availability of data and materials

The dataset upon which this work is based consists of articles already available within the published literature.

Abbreviations

• Randomised controlled trial

Traditional Invasive versus Minimally invasive Esophagectomy

CONsolidated Standards Of Reporting Trials for Non-Pharmacological Treatments

PRagmatic Explanatory Continuum Indicator Scale

Context and Implementation of Complex Interventions

Risk Of Bias Tool

Minimally Invasive Oesophagectomy

Systematic Review

Meta-Analysis

Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA, et al. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. Br Med J. 1998;317:465–8 Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.317.7156.465 .

Article   CAS   Google Scholar  

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.

Article   Google Scholar  

Oxford Centre for Evidence-Based Medicine. Levels of evidence. 2009 [cited 2018 Sep 25]. Available from: http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/ .

Google Scholar  

Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien P-A, Reeves BC, et al. Challenges in evaluating surgical innovation. Lancet. 2009;374:1097–104. Available from. https://doi.org/10.1016/S0140-6736(09)61086-2 .

Article   PubMed   PubMed Central   Google Scholar  

Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P. Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Ann Intern Med. 2008;148:295–309. Available from. https://doi.org/10.7326/0003-4819-148-4-200802190-00008 .

Article   PubMed   Google Scholar  

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Br Med J. 2011;343:d5928 Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.d5928 .

Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10:CD201601.

Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. Br Med J. 2015;350:h2147 Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.h2147 .

McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:1–8.

Donovan JL, Rooshenas L, Jepson M, Elliott D, Wade J, Avery K, et al. Optimising recruitment and informed consent in randomised controlled trials: the development and implementation of the quintet recruitment intervention (QRI). Trials. 2016;17:1–11. Available from. https://doi.org/10.1186/s13063-016-1391-4 .

Antoniou SA, Andreou A, Antoniou GA, Koch OO, Köhler G, Luketina RR, et al. Volume and methodological quality of randomized controlled trials in laparoscopic surgery: assessment over a 10-year period. Am J Surg. 2015;210:922–9.

Ali UA, van der Sluis PC, Issa Y, Habaga IA, Gooszen HG, Flum DR, et al. Trends in worldwide volume and methodological quality of surgical randomized controlled trials. Ann Surg. 2013;258:199–207.

Kristensen N, Nymann C, Konradsen H. Implementing research results in clinical practice - the experiences of healthcare professionals. BMC Health Serv Res. 2016;16:48. Available from. https://doi.org/10.1186/s12913-016-1292-y .

Blencowe NS, Boddy AP, Harris A, Hanna T, Whiting P, Cook JA, et al. Systematic review of intervention design and delivery in pragmatic and explanatory surgical randomized clinical trials. Br J Surg. 2015;102:1037–47.

Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365:82–93.

Orri M, Farges O, Clavien P-A, Barkun J, Revah-Lévy A. Being a surgeon - the myth and the reality. A meta-synthesis of surgeons’ perspectives about factors affecting their practice and well-being. Ann Surg. 2014;260:721–9 Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00000658-201411000-00002 .

Garcia-Retamero R, Cokely ET, Wicki B, Joeris A. Improving risk literacy in surgeons. Patient Educ Couns. 2016;99:1156–61. Available from. https://doi.org/10.1016/j.pec.2016.01.013 .

Byrne BE, Rooshenas L, Lambert H, Blazeby JM. Evidence into practice: protocol for a new mixed-methods approach to explore the relationship between trials evidence and clinical practice through systematic identification and analysis of articles citing randomised controlled trials. BMJ Open. 2018;8:e023215 Available from: http://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2018-023215 .

Biere SSAY, van Berge Henegouwen MI, Maas KW, Bonavina L, Rosman C, Garcia JR, et al. Minimally invasive versus open oesophagectomy for patients with oesophageal cancer: a multicentre, open-label, randomised controlled trial. Lancet. 2012;379:1887–92.

Yin RK. Case study research and applications: design and methods. Sixth: SAGE Publications; 2018.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100.

Stake RE. The art of case study research. Thousand Oaks: SAGE Publications; 1995.

Glaser BG, Strauss AL. The discovery of grounded theory: strategies for qualitative research. Observations. Aldine Transaction; 1967. Available from: http://www.amazon.com/dp/0202302601 .

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101 [cited 2014 May 25]. Available from: http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa .

Pfadenhauer L, Rohwer A, Burns J, Booth A, Lysdahl KB, Hofmann B, et al. Guidance for the assessment of context and implementation in Health Technology Assessments (HTA) and systematic reviews of complex interventions: the Context and Implementation of Complex Interventions (CICI) framework. 2016. Available from: https://www.integrate-hta.eu/wp-content/uploads/2016/02/Guidance-for-the-Assessment-of-Context-and-Implementation-in-HTA-and-Systematic-Reviews-of-Complex-Interventions-The-Co.pdf .

Biere SSAY, Maas KW, Bonavina L, Garcia JR, van Berge Henegouwen MI, Rosman C, et al. Traditional invasive vs. minimally invasive esophagectomy: a multi-center, randomized trial (TIME-trial). BMC Surg. 2011;11:1–7. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=emed10&AN=71483010%5Cnhttp://nt2yt7px7u.search.serialssolutions.com/?sid=OVID:Embase&genre=article&id=pmid:&id=doi:10.1007%2Fs00464-012-2198-3&issn=0930-2794&volume=26&issue=1&spage=S40&.

Fitzgerald L, Ferlie E, Wood M, Hawkins C. Interlocking interactions, the diffusion of innovations in health care. Hum Relat. 2002;55:1429–49.

Fitzgerald L, Ferlie E, Wood M, Hawkins C. Evidence into practice? An exploratory analysis of the interpretation of evidence. In: Mark AL, Dopson S, editors. Organisational Behaviour in Health Care: The Research Agenda. Palgrave Macmillan; 1999.

Latour B. Reassembling the social: an introduction to actor-network theory. Oxford: Oxford University Press; 2005..

Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010;10:1–11.

Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375:454–63.

Luketich JD, Pennathur A, Awais O, Levy RM, Keeley R, Shende M, et al. Outcomes after minimally invasive esophagectomy: review of over 1000 patients. Ann Surg. 2012;256:95–103.

Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369:1434–42 [cited 2014 Jul 23]. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24106936 .

Ebrahim S, Ferrie JE, Smith GD. The future of epidemiology: methods or matter? Int J Epidemiol. 2016;45:1699–716.

Jackson SE, Chester JD. Personalised cancer medicine. Int J Cancer. 2015;137:262–6.

Backmann M. What’s in a gold standard? In defence of randomised controlled trials. Med Health Care Philos. 2017;20:513–23.

Evidence-based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. J Am Med Assoc. 1992;268:2420–5 Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00006534-201412000-00040 .

Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q. 2004;82:661–87.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Urbach DR, Baxter NN. Reducing variation in surgical care. Br Med J. 2005;330:1401–2.

Gramlich LM, Sheppard CE, Wasylak T, Gilmour LE, Ljungqvist O, Basualdo-Hammond C, et al. Implementation of Enhanced Recovery After Surgery: a strategy to transform surgical care across a health system. Implement Sci. 2017;12:67.

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Heal Care. 2005;14:26–33.

Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:1–17.

Ritchie J, Spencer L. In: Bryman A, Burgess RG, editors. Qualitative data analysis for applied policy research. Routledge: Anal Qual data; 1994.

Chapter   Google Scholar  

Download references

Acknowledgements

We would like to thank Cath Borwick, Information Specialist at the University of Bristol for her help developing the literature search strategy, advising on the available tools and highlighting the full range of resources available for this study.

B E Byrne is supported by the National Institute for Health Research. Jane Blazeby is a NIHR Senior Investigator. This work was undertaken with the support of the MRC ConDuCT-II (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures) Hub for Trials Methodology Research (MR/K025643/1) and the NIHR Bristol Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol (BRC-1215-20011) and support from the Royal College of Surgeons of England Bristol Surgical Trials Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The funders had no role in developing the protocol.

Author information

Authors and affiliations.

Centre for Surgical Research, Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Clifton, Bristol, BS8 2PS, UK

Ben E. Byrne, Leila Rooshenas & Jane M. Blazeby

Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK

Helen S. Lambert

MRC ConDuCT-II Hub, Bristol Medical School, University of Bristol, Bristol, UK

Jane M. Blazeby

NIHR Bristol Biomedical Research Centre, University Hospitals Bristol NHS Foundation Trust, Bristol, UK

You can also search for this author in PubMed   Google Scholar

Contributions

BEB and JMB conceived the study. BEB, LR, HL and JMB developed the protocol and refined the study design. BEB and LR conducted the qualitative analysis. BEB prepared a preliminary draft manuscript. JMB, LR and HL extensively revised the manuscript. All authors have approved the final manuscript.

Corresponding author

Correspondence to Ben E. Byrne .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1 table s1..

Identifying codes and bibliographic information on all citing articles included in analysis. Table S2. CONSORT-NPT checklist with notes on TIME trial.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Byrne, B.E., Rooshenas, L., Lambert, H.S. et al. A mixed methods case study investigating how randomised controlled trials (RCTs) are reported, understood and interpreted in practice. BMC Med Res Methodol 20 , 112 (2020). https://doi.org/10.1186/s12874-020-01009-8

Download citation

Received : 26 November 2019

Accepted : 06 May 2020

Published : 12 May 2020

DOI : https://doi.org/10.1186/s12874-020-01009-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Translational medical research
• Health services research
• Evidence-based medicine

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Study protocol
• Open access
• Published: 20 August 2013

A mixed methods multiple case study of implementation as usual in children’s social service organizations: study protocol

• Byron J Powell 1 ,
• Enola K Proctor 1 ,
• Charles A Glisson 2 ,
• Patricia L Kohl 1 ,
• Ramesh Raghavan 1 , 3 ,
• Ross C Brownson 1 , 4 ,
• Bradley P Stoner 5 , 6 ,
• Christopher R Carpenter 7 &
• Lawrence A Palinkas 8  

Implementation Science volume  8 , Article number:  92 ( 2013 ) Cite this article

24k Accesses

29 Citations

Metrics details

Improving quality in children’s mental health and social service settings will require implementation strategies capable of moving effective treatments and other innovations ( e.g ., assessment tools) into routine care. It is likely that efforts to identify, develop, and refine implementation strategies will be more successful if they are informed by relevant stakeholders and are responsive to the strengths and limitations of the contexts and implementation processes identified in usual care settings. This study will describe: the types of implementation strategies used; how organizational leaders make decisions about what to implement and how to approach the implementation process; organizational stakeholders’ perceptions of different implementation strategies; and the potential influence of organizational culture and climate on implementation strategy selection, implementation decision-making, and stakeholders’ perceptions of implementation strategies.

Methods/design

This study is a mixed methods multiple case study of seven children’s social service organizations in one Midwestern city in the United States that compose the control group of a larger randomized controlled trial. Qualitative data will include semi-structured interviews with organizational leaders ( e.g ., CEOs/directors, clinical directors, program managers) and a review of documents ( e.g ., implementation and quality improvement plans, program manuals, etc.) that will shed light on implementation decision-making and specific implementation strategies that are used to implement new programs and practices. Additionally, focus groups with clinicians will explore their perceptions of a range of implementation strategies. This qualitative work will inform the development of a Web-based survey that will assess the perceived effectiveness, relative importance, acceptability, feasibility, and appropriateness of implementation strategies from the perspective of both clinicians and organizational leaders. Finally, the Organizational Social Context measure will be used to assess organizational culture and climate. Qualitative, quantitative, and mixed methods data will be analyzed and interpreted at the case level as well as across cases in order to highlight meaningful similarities, differences, and site-specific experiences.

This study is designed to inform efforts to develop more effective implementation strategies by fully describing the implementation experiences of a sample of community-based organizations that provide mental health services to youth in one Midwestern city.

Peer Review reports

Children in the U.S. continue to receive substandard mental health and child welfare services [ 1 – 4 ], partly because we do not understand how to effectively integrate evidence-based treatments (EBTs) into ‘real world’ service settings. Evidence-based treatments are seldom implemented, and when they are, problems with implementation can severely diminish their impact [ 5 ]. To improve the quality of care for children, EBTs will need to be complemented by evidence-based approaches to implementation [ 6 ]. Thus, the National Institutes of Health and the Institute of Medicine have prioritized efforts to identify, develop, refine, and test implementation strategies [ 7 , 8 ], which are defined as ‘systematic intervention processes to adopt and integrate evidence-based health innovations into usual care’ [ 9 ].

State of the evidence for implementation strategies

While the health and mental health literatures describe many potentially promising implementation strategies [ 9 ], the evidence of their effectiveness remains imperfect [ 10 – 13 ]. Most strategies deliver only modest effect sizes [ 10 ], and are effective under some, but not all, conditions [ 14 ]. Passive strategies, such as disseminating educational materials and continuing education courses, may be useful in increasing knowledge, but are generally not sufficient to change provider behavior [ 15 – 18 ]. Training approaches that incorporate ongoing supervision and consultation can lead to therapist behavior change [ 15 , 18 ], but it is increasingly recognized that strategies need to move beyond focusing solely on provider level factors such as knowledge and expertise [ 19 – 21 ]. Indeed, implementing EBTs with fidelity does not always improve outcomes [ 22 ], suggesting that other barriers to quality service provision must also be addressed [ 23 ]. Implementation is a complex, multi-level process, and existing theoretical and empirical work suggests that ‘best practices’ in implementation would involve the planned use of multiple strategies to address barriers to change that can emerge at all levels of the implementation context [ 9 , 20 , 21 , 24 – 28 ]. There are a number of strategies that extend beyond the provider level [ 9 ]; however, in social services research, there are very few randomized studies that test the effectiveness of multi-level implementation strategies (for one exception, see [ 23 ]). More research is needed to develop effective ways of tailoring strategies to target implementation barriers [ 29 ] and to develop innovative strategies that are efficient, cost-effective, and robust or readily adaptable [ 30 ].

The need for a better understanding of implementation as usual

Implementation scientists cannot develop these strategies ‘in a vacuum’ [ 31 ]; they must possess a thorough understanding of the service systems and organizational contexts in which these strategies will (hopefully) be adopted [ 32 ]. Hoagwood and Kolko warn that ‘it is difficult and perhaps foolhardy to try to improve what you don’t understand’ [ 31 ], and note that program implementers and services researchers are often unable to anticipate implementation challenges largely because the context of service delivery has not been adequately described. In other words, there is a need for a better understanding of usual care settings, and in particular, what constitutes ‘implementation as usual’.

Garland et al. acknowledge that ‘studies that “simply” characterize existing practice may not be perceived as innovative or exciting compared to studies that test new innovations’ [ 33 ]. However, these studies are ‘a necessary complement – if not precursor’ – to studies that will strengthen knowledge on the implementation of EBTs [ 31 ]. Indeed, an increased understanding of implementation as usual has the potential to identify leverage points for implementation, specify targets for improvement, and generate useful insights into the types of implementation processes that are likely to be successful in the real world.

At present, very little is known about the implementation processes that occur in usual care [ 31 , 33 , 34 ]. This highlights the need for descriptive studies that define the range and context of current implementation processes in relation to what is known about ‘best implementation practice’ [ 35 ], which (for the purpose of this study) is characterized as the planned use of multiple strategies to address barriers to change at various levels [ 20 , 26 , 28 , 36 ]. The current study addresses this need by leveraging a control group of a larger implementation trial that is not receiving an active implementation intervention. Using control groups to examine implementation as usual may yield critical information that can be used to improve the development of implementation strategies. This approach maximizes the use of research funding, illuminates implementation processes within control conditions that may be helpful in understanding the results of larger trials, and ultimately, avoids treating control conditions as ‘black boxes’ that are assumed to have no ‘action’ related to treatment and implementation decisions and processes. The last point constitutes a considerable advantage over studies that focus solely on outcomes obtained by control groups thought to represent ‘usual care’ without generating rich descriptions of what actually occurs in these settings. This study will describe four elements of these organizations that may play a role in determining implementation, service system, and clinical outcomes [ 37 ]: patterns of implementation strategy use, implementation decision-making, perceptions of implementation strategies, and organizational social context.

Implementation strategy patterns

There is a paucity of descriptive data pertaining to basic contextual elements of implementation such as organizational operations, staffing patterns, and electronic technologies for tracking service visits in usual care settings [ 31 ]. Even less is known about implementation strategy patterns in children’s social service organizations. One exception is Schoenwald and colleagues’ examination of organizations’ use of training, supervision and evaluation [ 34 ]. Encouragingly, they found that training and supervisory practices were more or less ‘in line’ with the typical procedures in an effectiveness trial. However, there has yet to be a study that maps a fuller range of potential implementation strategies that extends beyond commonly used strategies such as training and supervision [ 9 ]. Thus, very little is known about the types of strategies employed, the frequency and intensity at which they are used, and the conceptual domains and levels of the implementation context that they target.

Organizational decision-making related to implementation processes

Organizational leaders face tremendous challenges when it comes to determining which treatments will be implemented in their settings and how they will be implemented. As Ferlie notes, ‘implementation process is often emergent, uncertain, and affected by the local context and features of action’ [ 38 ]. It would be ideal if organizational leaders would base their decisions upon the latest theoretical and empirical findings; however, little is written about how organizational leaders approach implementation decision-making. In particular, we need to know more about whether and how organizational leaders use research related to management and implementation, and the conditions under which they may be more likely to use research [ 38 ]. Furthermore, there is a need for more insight into the types ( e.g ., summaries of implementation barriers and facilitators, reviews of implementation strategies), formats ( e.g ., statistical or narrative summaries), and sources ( e.g ., academics, peers from other organizations) of information that organizational leaders find most valuable when making decisions about how to implement EBTs. This will highlight the ways in which implementation research could be made more accessible to organizational leaders, and could inform the development of decision aids that could facilitate the identification, selection, and tailoring of implementation strategies.

Stakeholders’ perceptions of the characteristics of implementation strategies

The characteristics of interventions may play a large role in determining whether or not they are adopted and sustained in the real world [ 26 , 39 , 40 ]. Rogers’ diffusion of innovations theory suggests that innovative treatment models will not likely be adopted unless they are: superior to treatment as usual; compatible with agency practices; no more complex than existing services; easy to try (and reject if it fails); and likely to produce tangible results recognizable by authorities [ 40 , 41 ]. Other potentially influential characteristics of interventions specified in theoretical models include the intervention source ( i.e ., the legitimacy of the source and whether it was internally or externally developed), evidence strength and quality, adaptability, design quality and packaging, and costs [ 26 ]. While these characteristics are often considered in relation to clinical interventions, they also readily apply to implementation strategies. In fact, a better understanding of stakeholders’ perceptions of implementation strategies may facilitate the process of identifying, developing, and selecting strategies that will be feasible and effective in the real world.

Influence of organizational culture and climate on implementation processes

The conceptual and empirical literatures have underscored the importance of organizational factors such as culture and climate in facilitating or impeding the uptake of innovations [ 24 , 26 , 42 – 44 ]. ‘Organizational culture’ is what makes an organization unique from others, including its core values and its organizational history of adapting with successes and failures [ 42 ]. It involves not only values and patterns related to products and services, but also how individuals within an organization treat and interact with one another [ 42 ]. Glisson and colleagues write, ‘Culture describes how the work is done in the organization and is measured as the behavioral expectations reported by members of the organization. These expectations guide the way work is approached and socialize new employees in the priorities of the organization’ [ 43 ]. Thus, culture is passed on to new employees and is conceptualized as a rather stable construct that is difficult to change. ‘Organizational climate’ is formed when employees have shared perceptions of the psychological impact of their work environment on their own well-being and functioning in the organization [ 43 ].

More constructive or positive organizational cultures and climates are associated with more positive staff morale [ 45 ], reduced staff turnover [ 46 ], increased access to mental health care [ 47 ], improved service quality and outcomes [ 45 , 48 , 49 ], greater sustainability of new programs [ 46 ], and more positive attitudes toward EBTs [ 50 ]. Yet, it is less clear how culture and climate relate to implementation processes. Knowing more about this relationship would inform efforts to facilitate organizational change. For example, it may be that organizations with poor cultures and climates require more intensive implementation support in order to develop well-coordinated implementation plans that address relevant determinants of practice [ 51 ].

This mixed methods multiple case study addresses these gaps in knowledge related to implementation contexts and processes in children’s social service organizations through the following aims:

Aim 1: To identify and characterize the implementation strategies used in community-based children’s social service settings;

Aim 2: To explore how organizational leaders make decisions about which treatments and programs to implement and how to implement them;

Aim 3: To assess stakeholders’ (organizational leaders’ and clinicians’) perceptions of the effectiveness, relative importance, acceptability, feasibility and appropriateness of implementation strategies; and

Aim 4: To examine the relationship between organizational context (culture and climate) and implementation strategy selection, implementation decision-making, and perceptions of implementation strategies.

Aim 1 will rely upon semi-structured interviews with organizational leaders (management and clinical directors) and document review to yield rich descriptions of the implementation strategies employed by seven agencies. This data will be compared to ‘best practices’ in implementation derived from existing theoretical and empirical work [ 11 , 13 , 15 , 18 , 36 ] to inform future work developing strategies in areas that are currently poorly addressed. It will also allow researchers and administrators to build upon ‘practice-based evidence’ and the strengths of ‘positive deviants’ ( i.e ., organizations that are consistently effective in implementing change despite a myriad of implementation barriers) [ 52 , 53 ].

Aim 2 will also use semi-structured interviews with organizational leaders and document review to generate new knowledge about how agency leaders use evidence and other sources of information to make decisions about implementation. Learning more about the type of information that organizational leaders seek, the sources they look to for that information, and the conditions under which they seek that information, may inform future work to make implementation science findings more accessible and ensure that implementation decision-making is based upon the best available theoretical and empirical knowledge in the field.

Aim 3 will utilize focus groups and an online survey to ensure that future work to develop and test implementation strategies will be informed by stakeholders’ (organizational leaders’ and clinicians’) perceptions about the types of strategies that are likely to be effective in the real world.

Aim 4 will examine how organizational social context (culture and climate) facilitates or hinders implementation by linking the data about strategy selection, implementation decision-making, and stakeholders’ perceptions of implementation strategies to organizations’ scores on a standardized measure of culture and climate [ 43 ].

Guiding conceptual frameworks

The proposed study is informed by two conceptual frameworks: the consolidated framework for implementation research CFIR [ 26 ] and Grol and Wensing’s implementation of change model [ 36 ]. These models will be integrated in all stages of the research process, including conceptualization ( e.g ., selecting implementation processes on which to focus), data collection ( e.g ., using components of the conceptual models as interview questions and probes), analysis ( e.g ., determining how comprehensively organizations are addressing constructs essential to implementation success, comparing ‘implementation as usual’ to ‘best practices’), and dissemination ( e.g ., framing findings conceptually so that they will be comparable to other implementation studies).

The CFIR was developed for the purpose of serving as a common reference to the many constructs that have been identified as important to implementation success [ 26 ]. It identifies five major domains related to implementation, including: intervention characteristics, the outer setting, the inner setting, the characteristics of the individuals involved, and the process of implementation. Detailed definitions of the 39 constructs included in the CFIR can be found in the supplementary materials associated with that article [ 26 ]. It captures the complex, multi-level nature of implementation, and suggests that successful implementation may necessitate the use of an array of strategies that target multiple levels of the implementation context [ 9 ]. The CFIR has informed the semi-structured interview guide (see Additional file 1 ) by specifying specific probes for eliciting descriptions of implementation strategies across various ‘levels’. It will also be used to assess the comprehensiveness of organizations’ approaches to implementation. For example, an organization that focuses only on the ‘characteristics of individuals’ while neglecting other domains such as ‘intervention characteristics’ or the ‘inner setting’ would have a less comprehensive approach to implementation than an organization that addresses all three (or more) of those domains.

Grol and Wensing’s implementation of change model informs this research by specifying a process of implementation that begins with identifying problems or gaps in care, identifying ESTs or other best-practices, carefully planning the implementation effort, developing a proposal with targets for improvement or change, analyzing current performance, developing implementation strategies, executing the implementation plan, and continuously evaluating and (if necessary) adapting the plan [ 36 ]. The model provides a structure and a process to implementation that the CFIR lacks. It also emphasizes an important aspect of implementation ‘best practice, ’ namely, that while implementation processes may be complex, necessitating iterative and flexible approaches [ 54 , 55 ], they should be planned and deliberate rather than haphazard. The implementation of change model has also informed the development of the interview guide informing Aims 1 and 2.

This study employs a mixed methods multiple case study design, in which each participating organization (n = 7) is conceptualized as a ‘case’ [ 56 , 57 ]. Case studies are particularly helpful in understanding the internal dynamics of change processes, and including multiple cases capitalizes on organizational variation and permits an examination of how contextual factors influence implementation [ 58 ]. Leaders in the field have emphasized the importance of using case study and other mixed methods observational designs to develop a more nuanced, theoretically informed understanding of change processes [ 59 – 64 ]. The study relies upon the ‘sequential collection and analysis of qualitative and quantitative data, beginning with qualitative data, for the primary purpose of exploration and hypothesis generation, ’ or a QUAL → quan approach [ 64 ]. This serves the primary function of ‘development, ’ as collecting qualitative data in Aims 1 to 3 affords the opportunity to examine the impact of organizational context in Aim 4 [ 64 ]. It serves the secondary function of ‘convergence’ by using quantitative and qualitative data to answer the same questions in Aim 3 [ 64 ].

The study will be conducted in the control arm of a U.S. National Institute of Mental Health funded randomized controlled trial (RCT) [ 65 ] of the Availability, Responsiveness, and Continuity (ARC) organizational implementation strategy [ 23 , 49 , 66 ], which affords a unique opportunity to study implementation as usual. The sample includes seven children’s social service organizations in a Midwestern city that reflect the characteristics of children’s mental health service providers nationwide [ 34 ] in that they are characterized by nonprofit organizational structures, they employ therapists that have master’s and bachelor’s degrees, and are comprised of a predominantly social work staff.

All participating organizations may not be currently implementing EBTs; however, they will likely be able to discuss strategies they have used to implement other clinical programs, services, or treatment models [ 46 ]. Thus, we will maintain an inclusive stance toward the types of programs and practices that organizations are implementing. This is warranted given that the primary scientific objective is to learn more about the processes and contexts of implementation rather than the particulars of implementing a specific EBT or class of EBTs.

While sampling logic should not be used in multiple case study research [ 57 , 67 ], seven cases are expected to be enough to ‘replicate’ findings across cases [ 57 ]. Yin writes that each ‘case’ (organization) is in essence treated as a separate study that either predicts similar results (literal replication) or predicts contrasting results but for anticipatable reasons (theoretical replication) [ 57 ]. In the present study, organizations with the worst cultures and climates may be expected to demonstrate similar implementation processes and perceptions of strategies ( i.e ., literal replication), whereas organizations with more positive cultures and climates may embrace a much different set of implementation processes and perceptions of strategies ( i.e ., theoretical replication).

Data collection

The proposed study will rely upon qualitative data from semi-structured interviews (Aims 1, 2, and 4), document review (Aims 1, 2, and 4), and focus groups (Aim 3). Additionally, quantitative data from a project-specific survey being developed (described below) and the Organizational Social Context (OSC) measure [ 43 ] will be used to accomplish Aims 3 and 4 respectively (see Table  1 ).

Qualitative data collection

Semi-structured interviews.

Semi-structured interviews will be conducted with organizational leaders ( e.g ., management and clinical supervisors) from each participating organization. The interviews will explore the implementation strategies their agencies have employed within the past year (Aim 1) and their approach to implementation decision-making (Aim 2). Interviews will be conducted by the lead author and will be structured by an interview guide (Additional file 1 ) informed by a review of implementation strategies [ 9 ] and the guiding conceptual models [ 26 , 36 ]. Specifically, the interview guide contains questions and prompts that will encourage participants to consider the implementation strategies that their organization has employed at multiple levels of the implementation context as specified by the CFIR [ 26 ] and the Powell et al. taxonomy [ 9 ] ( e.g ., asking if their organization used strategies related to the intervention, the policy or inter-organizational level, and the organization’s structure and functioning in addition to more commonly considered individual-level and process-level strategies). Through the process of snowball sampling [ 68 ], each participant will be asked to identify other employees who possess the requisite knowledge and experience to inform the study’s objectives. It is estimated that each organization will identify between three and five key informants, resulting in approximately 21 to 35 total interviews. Many agencies may not have more than this number of individuals who have direct knowledge of the use of implementation strategies [ 69 ], and more importantly, the decision-making processes surrounding implementation.

Guest and colleagues emphasize that very small samples can yield complete and accurate information as long as the respondents have the appropriate amount of expertise in the domain of inquiry [ 70 ]. Further, a main benefit of the multiple case study design is obtaining different sources of information that will be used to triangulate the interview data [ 57 , 64 ]. Interviews will last 60 to 90 minutes and will be digitally recorded. Immediately following each interview, the interviewer will complete field notes that will capture the main themes of the interview and any information that is pertinent to the study aims [ 71 , 72 ]. Interviews and field notes will be transcribed, and entered into NVivo, version 10, for data analysis.

Document review

The study will also involve a review of publically available and organization-provided documents. Organizational leaders will be asked to provide access to any documents that describe and formalize implementation processes. For example, these processes may be captured in notes from a board meeting in which the implementation of a new program or practice was discussed, or in an organization’s response to a request for proposals that seeks funding for a particular training or implementation related resource. Other documents may include (but are certainly not limited to) formal implementation or quality improvement plans, annual reports, and program manuals. These sources will serve to augment or triangulate interview respondents’ descriptions of implementation strategies and decision-making processes. With permission from the organizations, potentially useful documents will be obtained and entered into NVivo, version 10, for analysis.

Focus groups interviews

Focus groups involving approximately four to eight clinicians (or direct care staff members) will be conducted in each participating organization to capture the depth and nuances of their perceptions of strategies. The number of participants per focus group is consistent with Barbour’s recommendation of a minimum of three or four participants and a maximum of eight [ 73 ]. The number of focus groups (one per agency) is appropriate because the relatively homogenous population ( e.g ., clinicians at a given agency) and the structured and somewhat narrow scope of inquiry reduces the number of individuals needed to reach saturation [ 70 ]. Further, the quantitative data will serve to triangulate the focus group data [ 57 , 64 ], reducing the need for a larger sample size. The focus groups will be conducted by the first author and a research assistant. The interview will be guided by a structured interview guide (Additional file 2 ) informed by a conceptual taxonomy of implementation outcomes [ 74 ]. Participants will be asked to discuss the implementation strategies that they have used at their organization, and the facilitator(s) will record each strategy mentioned on a whiteboard so that all participants can see the running list. Additional strategies drawn from the literature may be listed if the participants focus on a relatively narrow range of strategies. Participants will then be asked to reflect upon the effectiveness, acceptability, feasibility, and appropriateness of the listed strategies. Although the primary purpose of the focus group interviews is to assess participants’ perceptions of various implementation strategies, it is also possible that these individuals will provide information about implementation strategies used at their organization that were not captured in the semi-structured interviews with organizational leaders.

Each focus group will last approximately 60 to 90 minutes and will be digitally recorded. As with the individual interviews, the interviewer will complete field notes following the focus groups that will document the main themes of the session and any observations pertinent to the study aims. The interviews and the field notes will be transcribed and entered into NVivo, version 10, for analysis.

Quantitative survey data

Survey of stakeholders’ perceptions of implementation strategies.

A project-specific self-administered web-based survey will be developed to assess stakeholders’ perceptions and experiences with specific implementation strategies. The implementation strategies included in the survey will be generated from the qualitative work in Aims 1, 2, and 3 and a published ‘menu’ that describes 68 distinct implementation strategies [ 9 ]. In order to ensure a relatively low burden to respondents, it is unlikely that more than 40 strategies will be included. Decisions about the inclusion of strategies will be driven by the qualitative analysis ( i.e ., using the strategies mentioned by organizational leaders and clinicians), while attempts will be made to include strategies that address a number of different targets as specified in the CFIR [ 26 ].

It should also be noted that the Powell and colleagues’ compilation includes a number of strategies that could not be reasonably adopted by the participants of this study ( e.g ., ‘centralize technical assistance’) [ 9 ], and those strategies will be eliminated. The survey will also be informed by a conceptual taxonomy of implementation outcomes [ 74 ] and other existing surveys drawn from implementation science measures collections [ 75 , 76 ]. In addition to basic demographic questions, stakeholders will be asked whether or not they have experienced each included implementation strategy (yes or no) and will then rate each strategy (using a Likert-style scale) on the following dimensions: ‘effectiveness’ and ‘relative importance’ ( i.e ., How well did it work and how important was it relative to other strategies?), ‘acceptability’ ( i.e ., How agreeable, palatable, or satisfactory is the strategy?), ‘feasibility’ ( i.e. , the perception that the strategy has been or could be successfully used within a given setting), and ‘appropriateness’ ( i.e ., the perceived fit, relevance, or compatibility of the strategy with the setting). This survey will be administered via an email with a link to the online survey, and will be pilot tested to ensure face-validity and ease of use.

Organizational social context (OSC) survey

The OSC is a standardized measure that assesses organizational culture, climate, and work attitudes (the latter of which is not being used for the current study) using 105 Likert-style items [ 43 ]. Culture is assessed in terms of an organization’s level of ‘rigidity’ (centralization, formalization), ‘proficiency’ (responsiveness, competence), and ‘resistance’ (apathy, suppression). The ‘best’ organizational cultures are highly proficient and not very rigid or resistant, while the ‘worst’ cultures are not very proficient and are highly rigid and resistant to change or new ideas. Climate is assessed with three second-order factors: ‘engagement’ (personalization, personal accomplishment), ‘functionality’ (growth and achievement, role clarity, cooperation), and ‘stress’ [ 43 ]. The ‘best’ organizational climates are described as being highly engaged, highly functional, and low in stress [ 43 ]. Cronbach’s alphas for the OSC subscales (rigidity, proficiency, resistance, stress, engagement, functionality) range from 0.78 to 0.94. The OSC will be administered on site, and a research assistant will assure respondents that their responses will remain confidential.

Data analysis

Qualitative data analysis.

Qualitative data from semi-structured interviews, document review, and focus groups will be imported and analyzed (separately) in NVivo using qualitative content analysis [ 77 – 80 ], which has been used successfully in similar studies [ 81 – 83 ]. Content analysis enables a theory driven approach, and an examination of both manifest ( i.e ., the actual words used) and latent ( i.e ., the underlying meaning of the words) content [ 72 ]. Accordingly, analysis will be informed by the guiding conceptual models, with additional patterns, themes, and categories being allowed to emerge from the data [ 72 , 84 ]. The first author and a doctoral student research assistant will independently co-code a sample of the transcripts to increase reliability and reduce bias [ 72 , 85 ]. Both coders will participate in a frame-of-reference training to ensure a common understanding of the core concepts related to the research aims [ 82 ]. Disagreements will be discussed and resolved through consensus. Initially, the coders will review the transcripts to develop a general understanding of the content. ‘Memos’ will be generated to document initial impressions and define the parameters of specific codes. Next, the data will be condensed into analyzable units (text segments), which will be labelled with codes based on a priori ( i.e ., derived from the interview guide or guiding theories) or emergent themes that will be continually refined and compared to each other. For instance, the implementation of change model [ 36 ] will be used to develop a priori codes such as ‘identifying programs and practices’ or ‘planning’ related to implementation decision-making. The CFIR [ 26 ] will be used in a similar fashion by contributing a priori codes that will serve to distinguish different types of implementation strategies, such as strategies that focus on the ‘inner setting’ or the ‘outer setting’. Finally, the categories will be aggregated into broader themes related to implementation strategy patterns, implementation decision-making, and stakeholders’ perceptions of strategies.

The use of multiple respondents is intentional, as some individuals may be more or less knowledgeable about their organization’s approach to implementation; however, it is possible that participants from a given agency may not endorse the use of the same strategies [ 86 ]. The approach to handling such ‘discrepancies’ will be one of inclusion, in that each unique strategy endorsed will be recorded as ‘in use’ at that agency (for an example of this approach, see Hysong et al. [ 82 ]). If participants’ responses regarding strategies vary widely within a given organization, it may be indicative of a lack of a coherent or consistent strategy [ 86 ]. The use of mixed methods and multiple sources of data will allow us to make sense of reported variation in strategy use by affording the opportunity to determine the extent to which these sources of data converge [ 57 , 64 , 86 , 87 ]. The use of multiple respondents and different sources of data also reduces the threat of bias that is sometimes associated with the collection of retrospective accounts of phenomena such as business strategy [ 69 ].

Quantitative data analysis

The developed survey capturing stakeholders’ perceptions of implementation strategies will yield descriptive data that will augment the qualitative data from semi-structured interviews, document review, and focus groups. In the cross-case analysis, this data will be compared to determine differences and similarities between cases. Data will also be pooled across all seven cases to reveal an overall picture of strategy use, as well as perceived effectiveness, relative importance, acceptability, feasibility, and appropriateness of implementation strategies.

Results from the OSC measure will be analyzed and interpreted in consultation with its developer according to procedures described by Glisson et al . [ 43 ]. Scoring will be completed at the University of Tennessee’s Children’s Mental Health Services Research Center, including the generation of internal reliability estimates (alpha), agreement indices for organizational unit profiles, and t-scores for culture and climate. The resulting organizational profiles can be compared to norms from a nationwide sample of 1,154 clinicians in 100 mental health clinics, which affords the opportunity to determine the generalizability of study findings beyond the selected sites. The OSC data will serve to characterize the organizations’ culture and climate in individual case descriptions. Additionally, organizations will be stratified by their OSC profiles in order to differentiate more positive cultures (highly proficient and not very rigid or resistant) and climates (highly engaged, highly functional, low stress) from less positive cultures (low proficiency, highly rigid and resistant) and climates (low engagement and functionality, high stress) [ 43 ]. Qualitative results will then be categorized according to those OSC profiles to determine whether strategy patterns, approaches to decision-making, and perceptions of strategies vary by organizational culture and climate.

Mixed methods analysis

As previously mentioned, the structure of this study is QUAL → quan, meaning that qualitative methods precede quantitative and that they are predominant [ 64 , 88 ]. This serves the primary function of ‘development,’ as collecting qualitative data in Aims 1 to 3 affords the opportunity to examine the impact of organizational context in Aim 4. It also serves the function of ‘convergence’ by using quantitative and qualitative data to answer the same question in Aim 3 [ 64 ].

The processes of ‘mixing’ the qualitative and quantitative data flow directly from these functions. To serve the function of ‘development, ’ the quantitative data on organizational social context [ 43 ] is connected with the qualitative and quantitative results from Aims 1 to 3 regarding implementation strategy use, implementation decision-making, and stakeholder perceptions of implementation strategies [ 64 ]. Assuming there is a meaningful relationship between organizational social context and the data from Aims 1 to 3, this can be shown in a joint display [ 88 ] that categorizes the themes emerging from the qualitative and quantitative data based upon the OSC profiles [ 43 ] as described above. For example, a separate table may be used to show how implementation strategy patterns differ based upon organizational social context. Examples of this approach can be found in Killaspy et al . [ 89 ] and Hysong et al . [ 82 ], and are also detailed in Creswell and Plano-Clark’s methods book [ 88 ].

To serve the function of ‘convergence, ’ the qualitative data and quantitative data will be merged in order to answer the same question, which for Aim 3 is, ‘What are implementation stakeholders’ perceptions of implementation strategies’? These data are merged for the purpose of triangulation; in this case, to use the quantitative data from the stakeholder perceptions survey to validate and confirm the qualitative findings from the focus-group interviews. Once again, this process can be depicted through a table placing qualitative themes side by side with the quantitative findings to show the extent to which the data converges [ 88 ].

It is worth noting that the approaches to ‘mixing’ qualitative and quantitative data will be used at both the case-level and the cross-case level (as described below).

Cross-case analysis

A primary benefit of a multiple case study is the ability to make comparisons across cases. The proposed study will utilize cross-case synthesis [ 57 ], which treats individual cases as separate studies that are then compared to identify similarities and differences between the cases. This will involve creating word tables or matrices that will display the data according to a uniform framework [ 57 , 84 ]. For example, data from the first three aims (strategy patterns, implementation decision-making, and stakeholder perceptions) will be categorized based upon their OSC profiles [ 43 ] in Aim 4. This approach will be used to compare across cases for each of the proposed aims, allowing for meaningful similarities, differences, and site-specific experiences to emerge from the data [ 56 , 57 ].

Limitations

A number of limitations should be considered. There is some concern that the organizations in the sample will not be comparable since they will not all be implementing the same programs and practices. There are several protections against this danger. First, while there is evidence to suggest that specific programs and practices will require unique implementation strategies e.g ., [ 90 ], implementation strategies can also be viewed as more general components of an organization’s infrastructure [ 34 ]. In fact, this view of implementation strategies may become more salient as we begin to shift the focus away from implementing solitary practices and toward fostering evidence-based systems and “learning organizations” capable of implementing a number of EBTs well [ 91 ]. Obtaining descriptive data about the types of implementation strategies that organizations are currently using is a first step toward determining which strategies may need to be routinized in organizations and systems of care. Second, while they may not all be implementing the same interventions, the organizations in this sample are comparable in terms of client need, service provision, funding requirements, and other external or ‘outer setting’ factors [ 26 ]. Third, programs and practices can be compared in meaningful ways based upon their characteristics [ 39 , 40 , 92 ].

The cross-sectional nature of the data will not reveal how implementation processes change over time. Additionally, recall bias may limit the accuracy of participants’ memories of implementation processes. The use of multiple informants and data sources ( i.e ., triangulation) will increase the validity of findings and minimize the threat of this bias [ 57 , 58 ].

A final challenge is the lack of existing surveys that can assess stakeholder perceptions of strategies; however, the web-based survey will be informed by theories related to the intervention characteristics associated with increased adoption [ 26 , 39 , 40 ], related surveys [ 75 ], a taxonomy of implementation outcomes [ 74 ], and other emerging measurement models e.g ., [ 93 ].

Trial status

The Institutional Review Board at Washington University in St. Louis has approved all study procedures. Recruitment and data collection for this study began in March of 2013.

Improving the quality of children’s social services will require ‘making the right thing to do, the easy thing to do’ [ 94 ] by providing organizational leaders and clinicians with the tools they need to provide evidence-based care. In order for this to be accomplished, there is much we need to know about the approaches to implementation that routinely occur, the ‘on the ground’ perspectives of organizational stakeholders regarding the types of implementation strategies that are likely to work, and the ways in which organizational context impacts implementation processes. This study represents a novel approach to studying implementation as usual in the control group of an implementation RCT. By shedding light on ‘implementation as usual’ in children’s social service settings, this study will inform efforts to develop and tailor strategies, propelling the field toward the ideal of evidence-based implementation.

Abbreviations

Availability Responsiveness and Continuity

Consolidated Framework for Implementation Research

Evidence-Based Treatments

National Research Service Award

National Institute of Mental Health

Organizational Social Context

randomized controlled trial

qualitative (dominate method)

quantitative (subordinate method).

Garland AF, Brookman-Frazee L, Hurlburt MS, Accurso EC, Zoffness RJ, Haine-Schlagel R, Ganger W: Mental health care for children with disruptive behavior problems: a view inside therapists’ offices. Psychiatr Serv. 2010, 61: 788-795.

Article   PubMed   PubMed Central   Google Scholar  

Kohl PL, Schurer J, Bellamy JL: The state of parent training: Program offerings and empirical support. Fam Soc. 2009, 90: 247-254.

Article   Google Scholar  

Raghavan R, Inoue M, Ettner SL, Hamilton BH: A preliminary analysis of the receipt of mental health services consistent with national standards among children in the child welfare system. Am J Public Health. 2010, 100: 742-749.

Zima BT, Hurlburt MS, Knapp P, Ladd H, Tang L, Duan N, Wallace P, Rosenblatt A, Landsverk J, Wells KB: Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. J Am Acad Child Adolesc Psychiatry. 2005, 44: 130-144.

Article   PubMed   Google Scholar  

Durlak JA, DuPre EP: Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008, 41: 327-350.

Grol R, Grimshaw JM: Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv. 1999, 25: 503-513.

CAS   PubMed   Google Scholar  

Institute of Medicine: Initial National Priorities for Comparative Effectiveness Research. 2009, Washington, DC: The National Academies Press

Google Scholar  

Dissemination and implementation research in health (R01). http://grants.nih.gov/grants/guide/pa-files/PAR-13-055.html ,

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL: A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012, 69: 123-157.

Grimshaw JM, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, Vale L: Toward evidence-based quality improvement: evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006, 21 (2): S14-20.

PubMed   PubMed Central   Google Scholar  

Grol R, Wensing M, Eccles M: Improving patient care: The implementation of change in clinical practice. 2005, Edinburgh: Elsevier

Powell BJ, Proctor EK, Glass JE: A systematic review of implementation strategies in mental health service settings. 2011, Washington: Seattle

Straus S, Tetroe J, Graham ID: Knowledge translation in health care: Moving from evidence to practice. 2009, Hoboken, NJ: Wiley-Blackwell

Book   Google Scholar  

The Improved Clinical Effectiveness through Behavioural Research Group (ICEBeRG): Designing theoretically-informed implementation interventions. Implement Sci. 2006, 1: 1-8.

Beidas RS, Kendall PC: Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010, 17: 1-30.

Davis DA, Davis N: Educational interventions. Knowledge translation in health care: Moving from evidence to practice. Edited by: Straus S, Tetroe J, Graham ID. 2009, Oxford, UK: Wiley-Blackwell, 113-123.

Herschell AD, McNeil CB, Urquiza AJ, McGrath JM, Zebell NM, Timmer SG, Porter A: Evaluation of a treatment manual and workshops for disseminating, Parent–child Interaction Therapy. Adm Policy Ment Health. 2009, 36: 63-81.

Herschell AD, Kolko DJ, Baumann BL, Davis AC: The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clin Psychol Rev. 2010, 30: 448-466.

Flanagan ME, Ramanujam R, Doebbeling BN: The effect of provider- and workflow-focused strategies for guideline implementation on provider acceptance. Implement Sci. 2009, 4: 1-10.

Solberg LI, Brekke ML, Fazio CJ, Fowles J, Jacobsen DN, Kottke TE, Mosser G, O’Connor PJ, Ohnsorg KA, Rolnick SJ: Lessons from experienced guideline implementers: attend to many factors and use multiple strategies. Journal on Quality Improvement. 2000, 26: 171-188.

CAS   Google Scholar  

Wensing M, Bosch M, Grol R: Selecting, tailoring, and implementing knowledge translation interventions. Knowledge Translation in health care: Moving from evidence to practice. Edited by: Straus S, Tetroe J, Graham ID. 2009, Oxford, UK: Wiley-Blackwell, 94-113.

Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Miranda J, Bearman SK, Daleiden EL, Ugueto AM, Ho A, Martin J, Gray J, Alleyne A, Langer DA, Southam-Gerow MA, Gibbons RD: Health TRN on YM: Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. Arch Gen Psychiatry. 2012, 69: 274-282.

Glisson C, Schoenwald S, Hemmelgarn A, Green P, Dukes D, Armstrong KS, Chapman JE: Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010, 78: 537-550.

Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011, 38: 4-23.

Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA: Getting research findings into practice: closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. Br Med J. 1998, 317: 465-468.

Article   CAS   Google Scholar  

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 1-15.

Shortell SM: Increasing value: a research agenda for addressing the managerial and organizational challenges facing health care delivery in the United States. Medical Care Research Review. 2004, 61: 12S-30S.

Solberg LI: Guideline implementation: what the literature doesn’t tell us. Journal on Quality Improvement. 2000, 26: 525-537.

Baker R, Cammosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, Robertson N: Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010, CD005470-10.1002/14651858.CD005470.pub2.

Mittman BS: Criteria for peer review of D/I funding applications. 2010, Missouri: St. Louis

Hoagwood KE, Kolko DJ: Introduction to the special section on practice contexts: a glimpse into the nether world of public mental health services for children and families. Adm Policy Ment Health. 2009, 36: 35-36.

Proctor EK, Rosen A: From knowledge production to implementation: research challenges and imperatives. Res Soc Work Pract. 2008, 18: 285-291.

Garland AF, Bickman L, Chorpita BF: Change what? Identifying quality improvement targets by investigating usual mental health care. Adm Policy Ment Health. 2010, 37: 15-26.

Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, Glisson C, Rolls-Reutz J: A survey of the infrastructure for children’s mental health services: implications for the implementation of empirically supported treatments (ESTs). Adm Policy Ment Health. 2008, 35: 84-97.

Fixsen DL, Blase KA, Naoom SF, Wallace F: A National Plan of Implementation Research. 2005, Chapel Hill, N. C.: National Implementation Research Network

Grol R, Wensing M: Effective implementation: a model. Improving patient care: The implementation of change in clinical practice. Edited by: Grol R, Wensing M, Eccles M. 2005, Edinburgh: Elsevier, 41-57.

Proctor EK, Landsverk J, Aarons GA, Chambers DA, Glisson C, Mittman BS: Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009, 36: 24-34.

Ferlie E: Organizational interventions. Knowledge translation in health care: Moving from evidence to practice. Edited by: Straus S, Tetroe J, Graham ID. 2009, Oxford, UK: Wiley-Blackwell, 144-150.

Grol R, Bosch MC, Hulscher MEJ, Eccles MP, Wensing M: Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q. 2007, 85: 93-138.

Rogers EM: Diffusion of Innovations. 2003, New York: Free Press, 5

Fraser MW, Richman JM, Galinsky MJ, Day SH: Intervention Research: Developing Social Programs. 2009, Oxford: New York

Aarons GA, Horowitz JD, Dlugosz LR, Ehrhart MG: The role of organizational processes in dissemination and implementation research. Dissemination and implementation research in health: Translating science to practice. Edited by: Brownson RC, Colditz GA, Proctor EK. 2012, New York: Oxford University Press, 128-153.

Chapter   Google Scholar  

Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P: Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008, 35: 98-113.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004, 82: 581-629.

Glisson C: Assessing and changing organizational culture and climate for effective services. Res Soc Work Pract. 2007, 17: 736-747.

Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S, Green P: Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Adm Policy Ment Health. 2008, 35: 124-33.

Glisson C, Green P: The effects of organizational culture and climate on the access to mental health care in child welfare and juvenile justice systems. Adm Policy Ment Health. 2006, 33: 433-48.

Glisson C, Hemmelgarn A: The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse Negl. 1998, 22: 401-21.

Article   CAS   PubMed   Google Scholar  

Glisson C, Hemmelgarn A, Green P, Williams NJ: Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child & Adolescent Psychiatry. 2013, 52: 493-500.

Aarons GA, Sawitzky AC: Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychol Serv. 2006, 3: 61-72.

Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, Baker R, Eccles MP: A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013, 8: 1-11.

Bradley EH, Curry LA, Ramanadhan S, Rowe L, Nembhard IM, Krumholz HM: Research in action: using positive deviance to improve quality of health care. Implement Sci. 2009, 4: 1-11.

Pascale R, Sternin J, Sternin M: The Power of Positive Deviance: How Unlikely Innovators Solve the World’s Toughest Problems. 2010, Boston, MA: Harvard Business Press

Aarons GA, Palinkas LA: Implementation of evidence-based practice in child welfare: service provider perspectives. Administrative Policy in Mental Health & Mental Health Services Research. 2007, 34: 411-419.

Pressman JL, Wildavsky A: Implementation: How Great Expectations in Washington Are Dashed in Oakland; or, Why It’s Amazing That Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes. 1984, Berkeley: University of California Press, 3

Stake RE: Multiple Case Study Analysis. 2005, New York: Guilford Press

Yin RK: Case Study Research: Design and Methods. 2009, Thousand Oaks, CA: Sage, [ Applied Social Research Methods Series ], 4

Wensing M, Eccles M, Grol R: Observational evaluations of implementation strategies. Improving patient care: The implementation of change in clinical practice. Edited by: Grol R, Wensing M, Eccles M. 2005, Edinburgh: Elsevier, 248-255.

Aarons GA, Fettes DL, Sommerfeld DH, Palinkas LA: Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreat. 2012, 17: 67-79.

Berwick DM: The science of improvement. JAMA. 2008, 299: 1182-1184.

Eccles MP, Armstrong D, Baker R, Cleary K, Davies H, Davies S, Gasziou P, Ilott I, Kinmonth ALL, Leng G, Logan S, Marteau T, Michie S, Rogers H, Rycroft-Malone J, Sibbald B: An implementation research agenda. Implement Sci. 2009, 4: 1-7.

Institute of Medicine: The State of Quality Improvement and Implementation Research: Workshop Summary. 2007, Washington, DC: The National Academies Press

Landsverk J, Brown CH, Reutz JR, Palinkas LA, Horwitz SM: Design elements in implementation research: a structured review of child welfare and child mental health studies. Adm Policy Ment Health. 2011, 38: 54-63.

Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J: Mixed methods designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research. 2011, 38: 44-53.

Glisson C, Proctor EK: Testing an Organizational Implementation Strategy in Children’s Mental Health. 2009, National Institute of Mental Health (R01 MH084855), http://projectreporter.nih.gov/project_info_description.cfm?aid=8460576&icde=17345134&ddparam=&ddvalue=&ddsub=&cr=1&csb=default&cs=ASC ,

Glisson C, Hemmelgarn A, Green P, Dukes D, Atkinson S, Williams NJ: Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child & Adolescent Psychiatry. 2012, 51: 780-787.

Small ML: “How many cases do I need?”: on science and the logic of case selection in field based research. Ethnography. 2009, 10: 5-38.

Marshall MN: Sampling for qualitative research. Fam Pract. 1996, 13: 522-525.

Golden BR: The past is the past - or is it? The use of retrospective accounts as indicators of past strategy. Acad Manage J. 1992, 35: 848-860.

Guest G, Bunce A, Johnson L: How many interviews are enough?: an experiment with data saturation and variability. Field Methods. 2006, 18: 59-82.

Denzin NK, Lincoln YS: The SAGE Handbook of Qualitative Research. 2011, Thousand Oaks, CA: Sage, 4

Bernard HR: Research Methods in Anthropology: Qualitative and Quantitative Approaches. 2011, Lanham, Maryland: AltaMira Press, 5

Barbour R: Doing Focus Groups. 2007, Thousand Oaks, CA: Sage

Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey R, Hensley M: Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2011, 38: 65-76.

Rabin BA, Purcell P, Naveed S, Moser RP, Henton MD, Proctor EK, Brownson RC, Glasgow RE: Advancing the application, quality and harmonization of implementation science measures. Implement Sci. 2012, 7: 1-11.

SIRC Measures Project. http://www.seattleimplementation.org/sirc-projects/sirc-instrument-project/ ,

Forman J, Damschroder L: Qualitative content analysis. Empirical methods for bioethics: A primer. Edited by: Jacoby L, Siminoff LA. 2008, Amsterdam: Elsevier, 11: 39-62. Advances in Bioethics

Hsieh H-F, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005, 15: 1277-1288.

Kohlbacher F: The use of qualitative content analysis in case study research. Forum: Qualitative Social Research. 2006, 7: http://www.qualitative-research.net/index.php/fqs/article/view/75/154 ,

Mayring P: Qualitative content analysis. Forum: Qualitative Social Research. 2000, 1: http://www.qualitative-research.net/index.php/fqs/article/view/1089/2386 ,

Forsner T, Hansson J, Brommels M, Wistedt AA, Forsell Y: Implementing clinical guidelines in psychiatry: a qualitative study of perceived facilitators and barriers. BMC Psychiatry. 2010, 10: 1-10.

Hysong SJ, Best RG, Pugh JA: Clinical practice guideline implementation strategy patterns in veterans affairs primary care clinics. Health Serv Res. 2007, 42: 84-103.

Magnabosco JL: Innovations in mental health services implementation: a report on state-level data from the U.S. evidence-based practices project. Implement Sci. 2006, 1: 1-11.

Miles MB, Huberman AM: Qualitative Data Analysis. 1994, Thousand Oaks, CA: Sage, 2

Krippendorff K: Content Analysis: An Introduction to Its Methodology. 2003, Thousand Oaks, CA: Sage Publications, 2

Bowman C, Ambrosini V: Using single respondents in strategy research. Br J Manag. 1997, 8: 119-131.

Voss C, Tsikriktsis N, Frohlich M: Case research in operations management. International Journal of Operations & Production Management. 2002, 22: 195-219.

Creswell JW, Plano Clark VL: Designing and Conducting Mixed Methods Research. 2011, Thousand Oaks, CA: Sage, 2

Killaspy H, Johnson S, Pierce B, Bebbington P, Pilling S, Nolan F, King M: Successful engagement: a mixed methods study of the approaches of community treatment and community mental health teams in the REACT trial. Soc Psychiatry Psychiatr Epidemiol. 2009, 44: 532-540.

Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco J, Rapp CA, Ganju V, Goldman HH: The state policy context of implementation issues for evidence-based practices in mental health. Psychiatr Serv. 2007, 58: 914-921.

Chambers DA: Forward. Dissemination and implementation research in health: Translating science to practice. Edited by: Brownson RC, Colditz GA, Proctor EK. 2012, New York: Oxford University Press, vii–x

Scheirer MA: Linking sustainability research to intervention types. Am J Public Health. 2013, 103 (4): e73-e80.

Cook JM, O’Donnell C, Dinnen S, Coyne JC, Ruzek JI, Schnurr PP: Measurement of a model of implementation for health care: toward a testable theory. Implement Sci. 2012, 7: 1-15.

Clancy CM, Slutsky JR: Guidelines for guidelines: we’ve come a long way. CHEST. 2007, 132: 746-747.

Download references

Acknowledgements

Funding for this study has been provided by the National Institute of Mental Health (NIMH) through a National Research Service Award (NRSA) Individual Pre-Doctoral Fellowship (NIMH F31 MH098478; Powell, PI), the Doris Duke Charitable Foundation through a Fellowship for the Advancement of Child Well-Being (administered by Chapin Hall at the University of Chicago), the Fahs-Beck Fund for Research and Experimentation at the New York Community Trust, and the larger randomized clinical trial that is providing the sample of organizations and measure of OSC (NIMH R01 MH084855; Glisson, PI). This project was also made possible by training support from an NIMH NRSA Institutional Pre-Doctoral Fellowship (NIMH T32 MH19960; Proctor, PI) and a National Institutes of Health Pre-Doctoral Institutional Training Fellowship through the Washington University School of Medicine (NIH TL1 RR024995, UL1 RR024992; Polonsky, PI). The protocol was strengthened by the receipt of feedback on preliminary versions presented at the NIMH-funded Seattle Implementation Research Conference on October 13, 2011, and Knowledge Translation Canada’s Summer Institute funded by the Canadian Institute for Health Research on June 5, 2012.

Author information

Authors and affiliations.

Brown School, Washington University in St. Louis, Campus, Box 1196, One Brookings Drive, St. Louis, MO, 63130, USA

Byron J Powell, Enola K Proctor, Patricia L Kohl, Ramesh Raghavan & Ross C Brownson

College of Social Work, University of Tennessee, Knoxville, TN, 37996, USA

Charles A Glisson

Department of Psychiatry, Washington University School of Medicine, St. Louis, MO, 63130, USA

Ramesh Raghavan

Department of Surgery and Siteman Cancer Center, Washington University School of Medicine, MO, St. Louis, 63130, USA

Ross C Brownson

Department of Anthropology, Washington University in St. Louis, MO, St. Louis, 63130, USA

Bradley P Stoner

Department of Internal Medicine, Division of Infectious Diseases, Washington University School of Medicine, St. Louis, MO, 63130, USA

Department of Emergency Medicine, Washington University School of Medicine, St. Louis, MO, 63130, USA

Christopher R Carpenter

School of Social Work, University of Southern California, Los Angeles, California, 90089, USA

Lawrence A Palinkas

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Byron J Powell .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

BJP is the principal investigator of the study. BJP generated the idea and designed the study, drafted the manuscript, and approved all changes. EKP is the primary mentor on BJP’s F31 award from the National Institute of Mental Health and the award from the Doris Duke Charitable Foundation. CAG is the principal investigator and EKP is the co-principal investigator of the ARC RCT that provides the context for the current study. CAG is the developer of the OSC survey, and he and his colleagues from the Children’s Mental Health Services Research Center at the University of Tennessee, Knoxville will assist with the analysis and interpretation of that data. EKP, CAG, PLK, RR, RCB, BPS, CRC, and LAP provided input into the design of the study. All authors reviewed and provided feedback for this manuscript. The final version of this manuscript was vetted and approved by all authors.

Electronic supplementary material

Additional file 1: semi-structured interview guide. (pdf 112 kb), additional file 2: focus group interview guide.(pdf 77 kb), rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Powell, B.J., Proctor, E.K., Glisson, C.A. et al. A mixed methods multiple case study of implementation as usual in children’s social service organizations: study protocol. Implementation Sci 8 , 92 (2013). https://doi.org/10.1186/1748-5908-8-92

Download citation

Received : 08 July 2013

Accepted : 19 August 2013

Published : 20 August 2013

DOI : https://doi.org/10.1186/1748-5908-8-92

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation strategies
• Mental health
• Children and adolescents
• Mixed methods
• Multiple case study

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

A Longitudinal Mixed Methods Case Study Investigation of the Academic, Athletic, Psychosocial and Psychological Impacts of Being of a Sport School Student Athlete

Affiliations.

• 1 Room G07, Cavendish Hall, Carnegie School of Sport, Leeds Beckett University, Headingley Campus, Leeds, LS6 3QS, UK. [email protected].
• 2 Queen Ethelburga's College, York, UK. [email protected].
• 3 School of Science and Technology, Nottingham Trent University, Nottingham, UK.
• 4 Room G07, Cavendish Hall, Carnegie School of Sport, Leeds Beckett University, Headingley Campus, Leeds, LS6 3QS, UK.
• 5 Leeds Rhinos Rugby League Club, Leeds, UK.
• PMID: 38635007
• DOI: 10.1007/s40279-024-02021-4

Background: Sport schools are popular environments for simultaneously delivering education and sport to young people. Previous research suggests sport school involvement to have impact (i.e. the positive/negative, intended/unintended and long/short-term outcomes, results and effects) on student athlete's holistic (i.e. academic, athletic, psychosocial and psychological) development. However, previous research is limited by (1) cross-sectional methods, (2) limited multidimensional assessments, (3) lack of consideration for athlete characteristics (e.g. sex) and (4) failure to evaluate how sport school features affect student-athlete impacts.

Objectives: The study, using a mixed methods case study approach, aims to (1) longitudinally evaluate the impact of sport school involvement on the holistic development of student athletes, (2) evaluate the impact on holistic development by student-athlete characteristics and (3) explore the features and processes of the sport-school programme that drive/facilitate holistic impacts.

Methods: A longitudinal mixed methods design was employed across one full academic school year (33 weeks). Six data-collection methods (i.e. online questionnaire, physical fitness testing battery, academic assessment grades, log diaries, field notes/observation and timeline diagram/illustration) were used to assess the academic, athletic, psychosocial and psychological impacts for 72 student athletes from one sport school in the United Kingdom (UK).

Results: Student athletes developed positive long-term holistic overall impacts (i.e. academically, athletically and personally), including maintaining stable and relatively high levels of sport confidence, academic motivation, general recovery, life skills, resilience and friends, family and free time scores. Despite positive impacts, juggling academic and sport workload posed challenges for student athletes, having the potential to lead to negative holistic impacts (e.g. fatigue, stress and injury). Positive and negative impacts were linked to many potential features and processes of the sport school (e.g. academic and athletic support services versus insufficient training load build-up, communication, coordination, flexibility and planning). Furthermore, when considering student-athlete characteristics, females had lower sport confidence, higher general stress and body image concerns and less general recovery than males and student athletes who played sport outside the school had lower general recovery.

Conclusions: This mixed method, longitudinal study demonstrated sport school involvement resulted in many positive academic (e.g. good grades), athletic (e.g. fitness development), psychosocial (e.g. enhanced confidence) and psychological (e.g. improved resilience) impacts attributed to the academic and athletic support services provided. However, juggling heavy academic and athletic workloads posed challenges leading to negative impacts including fatigue, pressure, stress and injury. Furthermore, holistic impacts may be sex dependent and further support may be required for female student athletes in sport school environments. Overall, these findings demonstrate the complex nature of combining education and sport commitments and how sport schools should manage, monitor and evaluate the features of their programme to maximise the holistic impacts of sport-school student athletes.

© 2024. The Author(s).

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.27(7); 2022 Nov

Using mixed-methods in evidence-based nursing: a scoping review guided by a socio-ecological perspective

Associated data.

Supplemental Material for Using mixed-methods in evidence-based nursing: a scoping review guided by a socio-ecological perspective by a socio-ecological perspective by Lieu Thompson and Nataliya Ivankova in Journal of Research in Nursing

Increased pressure for evidence-based practice in nursing necessitates that researchers use effective approaches. Mixed-methods research (MMR) has potential to improve the knowledge and implementation of evidence-based nursing (EBN) by generating outcome-based and contextually-focused evidence.

To identify methodological trends in how MMR is used in EBN research.

Searches were completed in PubMed, CINAHL, and Google Scholar using the terms “nursing”, “mixed-methods”, and “evidence-based”. Seventy-two articles using MMR to address EBN and published 2000–2021 were reviewed across content themes and methodological domains of the Socio-Ecological Framework for MMR.

Mixed-methods research has been used to study how EBN strategies are perceived, developed and assessed, and implemented or evaluated. A few studies provided an MMR definition reflecting the methods perspective, and the dominant MMR rationale was gaining a comprehensive understanding of the issue. The leading design was concurrent, and half of studies intersected MMR with evaluation, action/participatory, and/or case-study approaches. Research quality was primarily assessed using criteria specific to quantitative and qualitative approaches.

Conclusions

Mixed-methods research has great potential to enhance EBN research by generating more clinically useful findings and helping nurses understand how to identify and implement the best available research evidence in practice.

Introduction

There has been increased pressure from diverse stakeholders for healthcare professionals to utilise evidence-based practices (EBP), which integrate research evidence, patient preference, and clinical expertise to provide quality patient care ( Breimaier et al., 2015 ; Gorsuch et al., 2020 ; Melnyk et al., 2018 ; Sackett et al., 1996 ). Many studies have shown that EBP improves patient safety and clinical results and reduces healthcare costs and variation in patient outcomes ( Black et al., 2015 ; Laibhen-Parkes et al., 2018 ). It is particularly important that nurses use EBP as they make up the largest group of healthcare professionals and play a major role in improving the safety and quality of care. Therefore, it is not surprising that the Institute of Medicine has identified EBP as a core competency of nursing ( American Nurses Association, 2015 ).

Nevertheless, nurses’ use of EBP remains inconsistent ( Breimaier et al., 2015 ; Laibhen-Parkes et al., 2018 ), and they continue to have difficulty implementing EBP knowledge and skills in practice ( Camargo et al., 2018 ; Gorsuch et al., 2020 ). Longstanding barriers to nurses’ use of EBP include a lack of access to research-based evidence and educational tools, lack of authority and organisational support to change clinical practice, and lack of time to implement new ideas ( Black et al., 2019 ; Gorsuch et al., 2020 ). These factors constrain nurses’ EBP knowledge and competence, which can lead to ineffective practices that jeopardise patient safety and well-being ( Black et al., 2015 ; Camargo et al., 2018 ). U.S. national surveys have found that nurses do not feel competent in any of the 24 competencies necessary to implement EBP ( Melnyk et al., 2018 ) and that nurse leaders lack competencies in several basic steps in the EBP process ( Harper et al., 2017 ). Given the high stakes of poor quality of care, it is imperative to leverage research strategies that can fully illuminate the complex challenges of EBP use in nursing care contexts.

Evidence-based nursing (EBN) has been defined as “the conscientious, explicit and judicious use of theory-derived, research-based information in making decisions about care delivery…in consideration of individual needs and preferences” ( Ingersoll, 2000 : 152). Although randomised controlled trials have been considered the gold standard of evidence, context, and experience of nursing care require the use of multiple methods that can generate both contextualised and outcome-oriented forms of evidence ( Ingersoll, 2000 ). Mixed-methods research (MMR) that integrates quantitative and qualitative approaches is becoming increasingly used in nursing research to address a wide range of health care issues ( Bressan et al., 2017 ; Halcomb and Hickman, 2015 ; Shorten and Smith, 2017 ; Younas et al., 2019 ). There has been a steady rise in MMR studies in nursing journals, and some nursing journals have published special issues devoted to MMR. For instance, the Journal of Research in Nursing’s June 2017 special issue highlighted how MMR can generate findings that are more readily adopted in health care practice ( Lesser, 2017 ).

Mixed-methods research has been recognised to have potential to improve the knowledge base for EBN by capitalising on the MMR advantages to generate both outcome-based and contextually focused evidence ( Breimaier et al., 2015 ; Flemming, 2007 ; Mathieson et al., 2018 ). Qualitative research, as part of an MMR approach, can inform the design and conduct of intervention effectiveness studies ( Flemming, 2007 ), secure patients’ and providers’ perspectives on EBP adoption and implementation ( Barbour, 2000 ), and provide the context for evaluating EBP in nursing ( Ailinger, 2003 ). Despite these advantages of MMR for optimising EBN practice, quantitative approaches continue to dominate EBN research ( Kidd and Twycross, 2019 ; Noble and Shorten, 2018 ). Recent reviews of EBN articles found that qualitative approaches were used in only 15–20% of studies, and MMR approaches were used in one study ( Kidd and Twycross, 2019 ; Noble and Shorten, 2018 ). Other review articles have displayed similar research designs with an emphasis on randomised controlled trials ( Adiewere et al., 2018 ).

To better understand how MMR can support EBN, we conducted a scoping literature review to identify methodological trends in how nursing researchers use MMR to address EBN problems. The review was guided by a comprehensive Socio-Ecological Framework for MMR ( Plano Clark and Ivankova, 2016 ) that shapes researchers’ decisions when applying MMR in EBN studies.

Methodology

Conceptual framework.

The Socio-Ecological Framework for MMR ( Plano Clark and Ivankova, 2016 ) aims to provide an understanding of how different MMR methodological components and study contexts influence researchers’ approaches to designing and implementing MMR studies and places the MMR process in the centre of the framework. Figure 1 presents the framework as consisting of five methodological domains including MMR definitions, rationales, designs, quality, and MMR intersection with other approaches and designs nested within three hierarchical layers representing the influences of personal, interpersonal, and social contexts on the MMR process.

Socio-ecological framework for mixed-methods research.

**Reprinted from Plano Clark and Ivankova (2016) with permission of SAGE Publications.

Literature search

Electronic searches were conducted in two prominent nursing research databases, PubMed and CINAHL, to identify empirical journal articles addressing various aspects of EBP in nursing. The review was limited to English-language studies published between 2000 and 2021. The search terms used were “nursing”, “mixed-method”, and “evidence-based”. The abstracts of identified articles were screened to determine their relevance to this review, and full texts were obtained for articles deemed as relevant. The full article texts were examined to determine their eligibility for inclusion in the review. The bibliographies of these articles were also examined to identify additional relevant studies.

The PRISMA diagram for the study selection process is presented in Figure 2 . Of the 262 articles identified, 85 duplicates were identified and excluded. Forty-eight articles were excluded because they were reviews, proposals, or commentaries, and 57 articles were excluded because they did not use MMR or address EBN. This resulted in a total of 72 articles for inclusion in this review.

PRISMA diagram. Source: Based on Moher et al. (2009) .

The selected articles were grouped into three content themes based on the aspect of EBN examined. In cases where an article reflected more than one theme, the content and research objectives of the article were used to determine the most appropriate classification. Articles were then analysed using the five methodological domains of the Socio-Ecological Framework for MMR to identify methodological trends in the use of MMR within and across themes.

Three themes emerged across the 72 reviewed studies: (1) perspectives on EBN strategies, (2) development and assessment of EBN strategies, and (3) implementation or evaluation of EBN strategies. The first theme describes the perspectives of different stakeholders (e.g., practitioners, patients, educators, researchers, and managers) on EBP and its role in nursing (e.g., beliefs, behaviours, and barriers/facilitators). The second theme refers to the development and assessment of EBN strategies such as interventions, practice guidelines, and measurement instruments. The third theme addresses the implementation and evaluation of EBN strategies in practice. Table S1 shows the distribution of the articles across the three themes. The most common theme was perspectives ( n = 31, 43%) followed by implementation/evaluation ( n = 30, 42%) and development and assessment ( n = 11, 15%). The findings for each content theme and methodological domain of the Socio-Ecological Framework for MMR are discussed next and summarised in Table 1 .

Results by content themes and methodological domains.

Defining MMR

Four major perspectives on defining MMR - method, methodology, philosophy, and community of research practice - were reported in the MMR literature ( Plano Clark and Ivankova, 2016 ). These perspectives reflect different views on what constitutes MMR and what aspects of mixing are emphasised in the MMR process. In our review, only seven (9%) articles included a definition of MMR. Most articles ( n = 6, 8%) defined MMR from a methods perspective, which implies mixing quantitative and qualitative methods of data collection and analysis within a single study. This perspective on MMR was observed mostly among development and assessment ( n = 2, 18%) and implementation/evaluation ( n = 3, 10%) articles. O’Brien et al. (2012) applied “quantitative and qualitative methods” (p. 2, development and assessment theme) to identify predictors of participant attrition and home visit completion…in a… nurse-family partnership programme. Nordsteien et al. (2017) used “quantitative data collection and analysis...supported by qualitative data” (p. 24, implementation/evaluation theme) to evaluate the influence of a collaborative library-faculty teaching intervention on nursing students’ use of evidence-based research tools.

The methodology definition of MMR, which supports mixing qualitative and qualitative approaches throughout the entire research process, was only reflected in Strandberg et al.’s (2014) perspectives article. The authors used “quantitative and subsequent qualitative approaches” (p. 57) to examine how nurses understand the concept of research utilisation. Six studies (8%) provided a citation to an underlying methodological source instead of defining MMR. The general absence of an MMR definition in the reviewed articles aligns with existing literature indicating that the reporting of MMR approaches in nursing research is incomplete and inconsistent, and that this significantly limits nurses’ ability to understand and utilise MMR evidence in clinical practice ( Bressan et al., 2017 ).

Rationales for MMR

Rationales for MMR are the arguments that researchers make to justify their decision to use MMR in a single study. A wide range of rationales have been discussed in the literature indicating the extensive applicability of MMR to address a variety of complex problems including EBP ( Ivankova et al., 2018 ; Plano Clark and Ivankova, 2016 ; Shorten and Smith, 2017 ). About half of the reviewed studies ( n = 42, 54%) stated rationales for using MMR to address the research purpose. Thirteen articles (17%) provided separate rationales for using quantitative and qualitative methods in the study, and this was mostly present within the perspectives ( n = 7, 23%) and implementation/evaluation ( n = 5, 14%) themes. Three major rationales for using MMR were identified in the reviewed articles: (1) gaining a comprehensive understanding of the issue, (2) using a qualitative approach to gain a deeper understanding of quantitative results, and (3) strengthening validity (see Table 2 for examples of rationales across three themes).

Examples of rationales for mixed-methods research.

The most frequently stated rationale was gaining a comprehensive understanding of the issue ( n = 19, 24%), and it was observed mostly in the articles focused on EBN implementation / evaluation ( n = 12, 34%). The second most common rationale, using a qualitative approach to gain a deeper understanding of quantitative results, ( n = 17, 22%), was noted primarily within the perspectives ( n = 9, 29%) and development and assessment ( n = 3, 25%) themes. The least stated rationale, using MMR to strengthen the validity of results ( n = 6, 8%), was most commonly used within the development and assessment ( n = 2, 17%) and implementation/evaluation ( n = 3, 9%) themes. In addition to the three major reasons for using MMR, some articles provided EBN-focused rationales. Horwood et al. (2021) stated that “real world evaluation based on mixed-methods including routine data” (p. 9, implementation/evaluation theme) was needed to test the feasibility and acceptability of implementing a nurse-led, telephone management service for patients diagnosed with chlamydia or gonorrhea.

MMR designs

Three core mixed-methods designs have been advanced in the MMR literature: a concurrent Quan + Qual and two sequential Quan → Qual and Qual →Quan ( Creswell and Plano Clark, 2018 ; Plano Clark and Ivankova, 2016 ). A concurrent Quan + Qual design, in which quantitative and qualitative components are implemented independently and both sets of results are combined to produce integrated conclusions, was the dominant design in the reviewed studies. It was used in two-thirds of studies across themes ( n = 52, 67%) and in over half of studies within each theme. Parsons et al. (2021) integrated quantitative survey results with qualitative interview findings to… determine the acceptability and feasibility of a referral and case management intervention ( development and assessment theme). Eaton et al. (2015) explored the EBP beliefs and behaviours of nurses who provide cancer pain management by collecting and analysing survey and interview data separately, and then interpreting both sets of results together ( perspectives theme).

Among sequential designs, in which one study phase is completed first and its findings inform the next phase, Quan → Qual design ( n = 15, 19%) was observed more often than Qual → Quan ( n = 11, 14%) design. Quan → Qual design was used equally across the themes, accounting for seven (20%) implementation/evaluation articles, two (19%) perspectives articles, and two (17%) development and assessment articles. De La Rue-Evans et al. (2013) conducted qualitative interviews to determine when and why nurses performed specific activities and then used the findings to inform the implementation and evaluation of new guidelines for preventing sleep disturbances among patients with traumatic brain injury ( implementation/evaluation theme). Lam and Schubert (2019) used quantitative survey results on organisational drivers of EBP to guide qualitative interviews exploring factors impacting nursing students’ understanding of EBP and information-seeking behaviours ( perspectives theme)

Compared to Quan→ Qual design, Qual→ Quan design was used differently among the themes. The design prevailed within the perspectives ( n = 7, 23%) theme and was equally common within the implementation/evaluation ( n = 3, 9%) and development and assessment ( n = 1, 8%) themes. Dale et al. (2005) used qualitative interview data to develop a quantitative survey measuring perceived versus actual barriers and facilitators to protocol uptake ( perspectives theme). De La Rue-Evans et al. (2013) analysed qualitative interview data to inform the implementation and quantitative evaluation of new sleep hygiene guidelines ( implementation/evaluation theme).

MMR and other approaches

Mixed-methods research has methodological flexibility to intersect or meaningfully combine with another design or methodology to form complex designs ( Creswell and Plano Clark, 2018 ; Plano Clark and Ivankova, 2016 ). Such intersection allows for addressing multifaceted research problems by using MMR to enhance another design or approach. In this review, intersecting MMR with another approach was observed in about one-third of studies ( n = 25, 35%) and mostly in implementation/evaluation ( n = 25, 50%) and perspectives ( n = 9, 29%) studies. Intersecting with evaluation approaches was most common, occurring in 12 (15%) studies. This is not surprising since EBN employs evaluation to continuously test and refine practices to improve patients’ and clinician’ outcomes. Using MMR with evaluation was most commonly noted in implementation/evaluation ( n = 9, 27%) and development and assessment ( n = 1, 9%) studies. Amacher et al. (2016) embedded an MMR design within an evaluation methodology to assess the satisfaction of patients and providers with a fall prevention programme ( implementation/evaluation theme). Parsons et al. (2021) embedded MMR design in a process evaluation to develop and refine an intervention to promote earlier return to work among staff with common mental health disorders ( development and assessment theme).

Some studies intersected MMR with action/participatory or case study approaches. Each approach was observed in six studies (8%) and only in perspectives and implementation/evaluation studies. Mixed-methods research with action/participatory approaches was noted in three studies (10%) in each theme, Breimaier et al. (2015) collected and analysed quantitative and qualitative data using participatory action research to assess the effectiveness of a fall-prevention guideline in an acute care hospital setting ( implementation/evaluation theme). Combining MMR with case study research was noted twice as often in the perspectives theme ( n = 4, 13%) than in the implementation/evaluation theme ( n = 2, 7%). Russell et al. (2019) used data from quantitative clinical records and qualitative interviews to construct case studies on eight family practices describing factors affecting their uptake of an intervention to prevent vascular disease ( perspectives theme).

MMR quality

Mixed-methods research quality are the decisions that researchers make about how to assess the quality of an MMR study ( Creswell and Plano Clark, 2018 ; Plano Clark and Ivankova, 2016 ). Among the reviewed articles, the leading strategy was separately reporting the quality of quantitative and qualitative study components ( n = 69, 88%), which occurred mostly within the development and assessment ( n = 12, 100%) and perspectives ( n = 29, 94%) themes. Gifford et al. (2012) engaged multiple investigators in a quantitative randomised controlled trial and used research-based guides for qualitative interviews to pilot an intervention to promote guideline adherence ( development and assessment theme). Lin et al. (2020) administered a previously validated quantitative survey, collected qualitative data until data saturation was reached, and maintained audit trails and memos throughout the research process ( implementation/evaluation ).

Quality assurance was discussed in some studies for only the quantitative component ( n = 4, 6%) or the qualitative component ( n = 3, 4%). These articles focused exclusively on EBN perspectives or implementation/evaluation . Miller et al. (2018) used a quantitative instrument shown to have “superior sensitivity and specificity” (p.91) in accurately identifying alcohol misuse in comparable target populations ( implementation/evaluation theme). Ersek and Jablonski (2014) employed multiple investigators to develop and confirm qualitative themes on barriers and facilitators to protocol adoption ( implementation/evaluation theme). No studies discussed quality assurance for the overall MMR process, which is not surprising since quality criteria for MMR studies remain one of the most debated topics ( Plano Clark and Ivankova, 2016 ; Tashakkori et al., 2021 ).

This paper synthesised 72 empirical MMR articles in EBN to explore how researchers employ MMR within and across three content themes addressing various aspects of EBN: stakeholder perspectives on EBN, development and assessment of EBN strategies, and implementation/evaluation of EBN strategies. Our review was guided by five methodological domains of the Socio-Ecological Framework for MMR including MMR definitions, rationales, designs, quality, and MMR intersection with other approaches and designs. The findings suggest that this framework is a useful tool for identifying methodological trends in EBN research and understanding how EBN researchers approach MMR, justify the choice of MMR, and design and implement MMR to address a variety of EBN issues.

Summary of methodological trends

Most studies in this review did not provide a definition or citation for MMR, and the definitions provided overwhelmingly reflected the methods perspective. In contrast, most articles reported a rationale for using MMR. Gaining a comprehensive understanding of the issue was the most frequently cited rationale, particularly in the studies aimed at evaluating EBN practices. Some EBN-specific rationales were also noted that emphasised the advantages of using MMR for addressing clinical questions within a context.

Another clear trend is the dominant use of a concurrent Quan + Qual design, which is consistent with the noted popularity of this design in health science research due to its relative time efficiency ( Curry and Nunez-Smith, 2015 ; Ivankova and Kawamura, 2010 ). Meanwhile, evaluation, action/participatory, and case study were the primary approaches that embedded MMR to form complex designs, and study quality was assessed using criteria traditionally associated with quantitative and qualitative approaches rather than MMR-specific criteria. To better understand these trends in MMR use in EBN, it is important to examine the contexts that may have influenced how the researchers designed, conducted and reported MMR ( Plano Clark and Ivankova, 2016 ).

The influence of MMR contexts

According to the Socio-Ecological Framework for MMR, three types of study contexts influence a researcher’s decision for how to apply MMR in a study: personal, interpersonal, and social. Personal contexts include researchers’ background knowledge, philosophical assumptions, and use of theoretical models. Interpersonal contexts incorporate relations with study participants, research teams, and editors/reviewers of the journals that publish MMR. Social contexts include institutional structures, disciplinary conventions, and societal priorities related to promoting MMR ( Plano Clark and Ivankova, 2016 ). These contexts directly and indirectly influence the study process and the use of MMR and likely played a role in how MMR was applied to address EBN issues.

Researchers’ focus on EBN along with their knowledge of and an adopted worldview on MMR may have influenced their perspectives on MMR, rationales for using MMR as a methodology of choice, and use of quality criteria associated with either quantitative or qualitative approaches. It is not surprising that EBN researchers elected to use MMR since it can provide a more complete understanding of EBN issues ( Shorten and Smith, 2017 ). The tendency to define MMR as the mixing of different methods is consistent with how researchers design and report MMR in health sciences ( Curry and Nunez-Smith, 2015 ; Wisdom et al., 2012 ).

At the interpersonal level, the interdisciplinary nature of most research teams, availability of resources and access to study participants may have affected methodological decisions about the type and sequence of quantitative and qualitative data collection and analysis ( Creswell and Plano Clark, 2018 ) resulting in the dominant use of concurrent Quan + Qual design. Concurrent designs often associated with time constraints to complete funded research capitalise on teamwork and the skills each team member brings into an MMR project ( Curry et al., 2012 ). The diversity of research skills also likely facilitated intersecting MMR with other approaches and designs and provided opportunities for more informed discussions of quality considerations related to different study components.

The influence of social contexts is evident in the adoption of MMR in nursing research ( Halcomb and Hickman, 2015 ). Evidence-based nursing authors may expect readers to be familiar with MMR so feel no need to define or describe it. In contrast, it is possible that the authors expected some pushback regarding their choice of MMR, so they felt the need to provide a rationale for using it. Support from universities and funding agencies, which is evident from authors’ affiliations in most studies may have made concurrent designs more likely due to budget constraints and improved access to participants (e.g., patients and health care providers) through existing academic networks.

Implications for using MMR in EBN research and practice

Mixed-methods research has the potential to advance knowledge of EBN and its impact on EBN outcomes. In the traditional hierarchy of evidence that clinicians and researchers often rely on, evidence from quantitative research designs such as clinical trials ranks as the strongest form of evidence ( Melnyk and Fineout-Overholt, 2019 ). This creates a dilemma for researchers debating whether to use qualitative and MMR approaches to generate evidence to include in nursing curricula as well as for nurses aiming to interpret and apply MMR findings in practice. It also increases the likelihood that nursing researchers and educators are more familiar with quantitative methodologies compared to qualitative and mixed-methods methodologies and thus, need more comprehensive guidance on the strengths of MMR ( Bressan et al., 2017 ).

Nurse researchers can help address this dilemma by clearly explaining their approaches to MMR, rationales for using MMR to address the EBN problems, decisions about MMR designs, and criteria for assessing MMR study quality. Doing so may mitigate the continued dominance of quantitative methodologies in EBN ( Kidd and Twycross, 2019 ; Noble and Shorten, 2018 ) by encouraging researchers to consider less traditional evidence hierarchies when designing their studies and by illustrating the feasibility of applying MMR to a range of EBN problems and contexts. The emphasis on methodological pluralism that characterises MMR also encourages researchers to use multiple methods and different data sources to produce alternative types of evidence on important antecedents and outcomes of EBN care that may not be apparent in quantitative data. For example, a researcher can use quantitative data to draw generalisations about the prevalence of adherence to nursing guidelines and use qualitative data to develop transferable findings on nurses' and doctors’ perceived barriers to EBP as in Storm-Versloot et al. (2012) .

Advancing MMR application in EBN research may subsequently result in the use of research designs that yield more clinically and contextually relevant study designs given that the traditional hierarchy of evidence does not fit all clinical questions ( Melnyk and Fineout-Overholt, 2019 : 192). It can also improve the quality of nursing care by increasing the likelihood that nurses make clinical decisions that consider the needs of patients and clinicians, a key component of EBN highlighted in this review that may not be reflected in quantitative research evidence.

Limitations

This review has several limitations. The selected articles are primarily from two prominent nursing databases, and the search terms used may have influenced the resulting pool of papers. Another limitation is the subjectivity involved in classifying articles into mutually exclusive themes and methodological content domains. Additionally, it was necessary to draw inferences based on the provided information in cases of ambiguity.

This review provides insight into the variety of MMR approaches nursing researchers use to generate new types of evidence in support of EBN practice. Mixed-methods research has significant potential to enhance EBN research aimed at improving patient care and outcomes by producing more clinically useful findings and helping nurses understand how to identify and implement the available research evidence in practice. We hope that this paper encourages nurses and policymakers searching for effective strategies to apply MMR-generated evidence by illustrating the ways in which MMR has been used to inform the development, implementation, and evaluation of EBN strategies.

Key points for policy, practice, and/or research

• • Mixed-methods research has the potential and utility to advance knowledge of EBN research by providing a multifaceted understanding of complex EBN issues.
• • The Socio-Ecological Framework for MMR can facilitate an understanding of the varied ways in which EBN researchers apply MMR to design studies addressing different aspects of EBN.
• • Using MMR can help nurses and policymakers develop and implement strategies to facilitate the translation of research into real-world improvement in patient safety and quality care.

Supplemental Material

Lieu Thompson is a doctoral candidate in Health Services Administration at the University of Alabama at Birmingham. She has interdisciplinary training in quantitative and qualitative methodologies and strategic management research.

Nataliya V Ivankova is a Professor at the University of Alabama at Birmingham. She is an applied research methodologist working at the intersection of mixed-methods, qualitative, community-based and translational research.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval: Ethical approval was not needed to conduct this literature review.

Supplemental material: Supplemental material for this article is available online.

Lieu Thompson https://orcid.org/0000-0003-2721-5215

• Ailinger RL. (2003) Contributions of qualitative research to evidence-based practice in nursing . Revista Latino-Americana de Enfermagem 11 ( 3 ): 275–279. [ PubMed ] [ Google Scholar ]
• Adiewere P, Gillis RB, Imran Jiwani S, et al. (2018) A systematic review and meta-analysis of patient education in preventing and reducing the incidence or recurrence of adult diabetes foot ulcers (DFU) . Heliyon 4 ( 5 ): e00614. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Amacher AE, Nast I, Zindel B, et al. (2016) Experiences of general practitioners, home care nurses, physiotherapists and seniors involved in a multidisciplinary home-based fall prevention programme: A mixed method study . BMC Health Services Research 16 . [ PMC free article ] [ PubMed ] [ Google Scholar ]
• American Nurses Association (2015) Nursing: Scope and Standards of Practice . 3rd ed. Silver Spring, MD: Nursesbooks.org. [ Google Scholar ]
• Barbour RS. (2000) The role of qualitative research in broadening the ‘evidence base’ for clinical practice . Journal of Evaluation in Clinical Practice 6 ( 2 ): 155–163. [ PubMed ] [ Google Scholar ]
• Black TA, Balneaves GL, Garossino HC, et al. (2015) Promoting evidence-based practice through a research training program for point-of-care clinicians . The Journal of Nursing Administration 45 ( 1 ): 14–20. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Black A, Ali S, Baumbusch J, et al. (2019) Practice-based nursing research: evaluation of clinical and professional impacts from a research training programme . Journal of Clinical Nursing 28 ( 13–14 ), 2681–2687. [ PubMed ] [ Google Scholar ]
• Breimaier HE, Halfens RJG, Lohrmann C, et al. (2015) Effectiveness of multifaceted and tailored strategies to implement a fall-prevention guideline into acute care nursing practice: a before-and-after, mixed-method study using a participatory action research approach . BMC Nursing 14 , 18( 1 ). [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bressan V, Bagnasco A, Aleo G, et al. (2017) Mixed-methods research in nursing – a critical review . Journal of Clinical Nursing 26 ( 19–20 ): 2878–2890. [ PubMed ] [ Google Scholar ]
• Camargo FC, Iwamoto HH, Galvão CM, et al. (2018) Competences and barriers for the evidence-based practice in nursing: an integrative review . Revista brasileira de enfermagem 71 ( 4 ): 2030–2038. [ PubMed ] [ Google Scholar ]
• Creswell JW, Plano Clark VL. (2018) Designing and Conducting Mixed Methods Research . 3rd ed. Thousand Oaks, CA: SAGE. [ Google Scholar ]
• Curry L, Nunez-Smith M. (2015) Mixed Methods in Health Sciences Research: A Practical Primer . Thousand Oaks, CA: SAGE. [ Google Scholar ]
• Curry LA, O’Cathain A, Plano Clark VL, et al. (2012) The role of group dynamics in mixed methods health sciences research teams . Journal of Mixed Methods Research 6 ( 1 ) 5–20. [ Google Scholar ]
• Dale S, Levi C, Ward J, et al. (2015) Barriers and enablers to implementing clinical treatment protocols for fever, hyperglycaemia, and swallowing dysfunction in the Quality in Acute Stroke Care (QASC) Project‐ A mixed methods study . Worldviews on Evidence‐Based Nursing 12 ( 1 ): 41–50. [ PubMed ] [ Google Scholar ]
• De La Rue-Evans L, Nesbitt K, Oka RK. (2013) Sleep hygiene program implementation in patients with traumatic brain injury . Rehabilitation Nursing 38 ( 1 ): 2–10. [ PubMed ] [ Google Scholar ]
• Eaton LH, Meins AR, Mitchell PH, et al. (2015) Evidence-based practice beliefs and behaviors of nurses providing cancer pain management: A mixed-methods approach . Oncology Nursing Forum 42 ( 2 ): 165–173. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ersek M, Jablonski A. (2014) A mixed-methods approach to investigating the adoption of evidence-based pain practices in nursing homes . Journal of Gerontological Nursing 40 ( 7 ): 52–60. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Flemming K. (2007) The knowledge base for evidence-based nursing: a role for mixed methods research? Advances in Nursing Science 30 ( 1 ): 41–51. [ PubMed ] [ Google Scholar ]
• Gifford WA, Davies BL, Graham ID, et al. (2013) Developing leadership capacity for guideline use: A pilot cluster randomized control trial . Worldviews on Evidence-Based Nursing 10 ( 1 ): 51–65. [ PubMed ] [ Google Scholar ]
• Gorsuch CPF, Gallagher Ford L, Koshy Thomas B, et al. (2020) Impact of a formal educational skill-building program based on the ARCC model to enhance evidence-based practice competency in nurse teams . Worldviews on Evidence-Based Nursing 17 ( 4 ): 258–268. [ PubMed ] [ Google Scholar ]
• Halcomb E, Hickman L. (2015) Mixed methods research . Nursing Standard: Promoting Excellence in Nursing Care 29 ( 32 ): 41–47. [ PubMed ] [ Google Scholar ]
• Harper MG, Gallagher-Ford L, Warren JI, et al. (2017) Evidence-based practice and U.S. healthcare outcomes: findings from a national survey with nursing professional development practitioners . Journal for Nurses in Professional Development 33 ( 4 ): 170–179. [ PubMed ] [ Google Scholar ]
• Horwood J, Brangan E, Manley P, et al. (2020) Management of chlamydia and gonorrhoea infections diagnosed in primary care using a centralised nurse-led telephone-based service: mixed methods evaluation . BMC Family Practice 21 ( 1 ): 265. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ingersoll GL. (2000) Evidence-based nursing: what it is and what it isn’t . Nursing Outlook 48 ( 4 ): 151–152. [ PubMed ] [ Google Scholar ]
• Ivankova NV, Herbey, Roussel LA, et al. (2018) Theory and practice of using mixed methods in translational research: a cross-disciplinary perspective . International Journal of Multiple Research Approaches 10 : 356–372. [ Google Scholar ]
• Ivankova NV, Kawamura Y. (2010) Emerging trends in the utilization of integrated designs in the social, behavioral, and health sciences. In: Tashakkori A, Teddlie C. (eds). SAGE Handbook of Mixed Methods in Social and Behavioral Research . 2nd ed. Thousand Oaks, CA: SAGE, 581–612. Available at: 10.4135/9781506335193 (accessed 27 Oct 2021). [ CrossRef ] [ Google Scholar ]
• Kidd L, Twycross A. (2019) EBN perspectives: nursing issues . Evidence Based Nursing 22 ( 4 ): 101–104. [ PubMed ] [ Google Scholar ]
• Laibhen-Parkes N, Kimble LP, Melnyk BM, et al. (2018) An adaptation of the original Fresno test to measure evidence-based practice competence in pediatric bedside nurses . Worldviews on Evidence-Based Nursing 15 ( 3 ): 230–240. [ PubMed ] [ Google Scholar ]
• Lam CK, Schubert C. (2019) Evidence-Based Practice Competence in Nursing Students: An Exploratory Study With Important Implications for Educators . Worldviews on Evidence-Based Nursing 16 ( 2 ): 161–168. [ PubMed ] [ Google Scholar ]
• Lin F, Marshall AP, Gillespie B, et al. (2020) Evaluating the implementation of a multi-component intervention to prevent surgical site infection and promote evidence-based practice . Worldviews on Evidence-Based Nursing 17 ( 3 ): 193–201. [ PubMed ] [ Google Scholar ]
• Lesser J. (2017) Guest editorial . Journal of Research in Nursing , 22 ( 4 ): 261–263. [ Google Scholar ]
• Mathieson A, Grande G, Luker K, et al. (2018) Strategies, facilitators and barriers to implementation of evidence-based practice in community nursing: a systematic mixed-studies review and qualitative synthesis . Primary Health Care Research and Development 20 : e6-e6. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Melnyk BM, Gallagher-Ford L, Zellefrow C, et al. (2018) The first U.S. study on nurses’ evidence-based practice competencies indicates major deficits that threaten healthcare quality, safety, and patient outcomes . Worldviews on Evidence Based Nursing 15 ( 1 ): 16–25. [ PubMed ] [ Google Scholar ]
• Melnyk BM, Fineout-Overholt E. (2019) Critically appraising qualitative evidence for clinical decision making. In: Melnyk BM, Fineout-Overholt E. (eds) Evidence-based Practice in Nursing & Healthcare: A Guide to Best Practice . 4th ed.. Philadelphia: Wolters Kluwer, 192. [ Google Scholar ]
• Miller LB, Brennan-Cook J, Turner B, et al. (2018) Utilizing an evidence-based alcohol screening tool for identification of alcohol misuse . Journal of Addictions Nursing 29 ( 2 ): 90–95. [ PubMed ] [ Google Scholar ]
• Moher D, Liberati A, Tetzlaff J, et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Plos Medicine 6 ( 7 ): e1000097. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Noble H, Shorten A. (2018) Nursing issues . Evidence Based Nursing 21 ( 1 ): 4–6. [ PubMed ] [ Google Scholar ]
• Nordsteien A, Horntvedt MT, Syse J, et al. (2017) Use of research in undergraduate nursing students’ theses: a mixed methods study . Nurse Education Today 56 : 23–28. [ PubMed ] [ Google Scholar ]
• O’Brien R, Moritz P, Luckey D, et al. (2012) Mixed methods analysis of participant attrition in the nurse-family partnership . Prevention Science 13(3): 219–228. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Parsons V, Juszczyk D, Gilworth G, et al. (2021) A case management occupational health model to facilitate earlier return to work of NHS staff with common mental health disorders: a feasibility study . Health Technology Assessment 25(12): 1–94. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Russell G, Lane R, Parker S, et al. (2019) Preventive evidence into practice: What factors matter in a facilitation intervention to prevent vascular disease in family practice ? BMC Family Practice 20(1): 113. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Plano Clark V, Ivankova N. (2016) Mixed Methods Research: A Guide to the Field . Thousand Oaks, CA: SAGE. [ Google Scholar ]
• Sackett DL, Rosenberg WMC, Gray JAM, et al. (1996) Evidence based medicine: what it is and what it isn’t . British Medical Journal 312 : 71–72. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shorten A, Smith J. (2017) Mixed methods research: expanding the evidence base . Evidence Based Nursing 20 ( 3 ): 74–75. [ PubMed ] [ Google Scholar ]
• Storm-Versloot MN, Knops AM, Ubbink DT, et al. (2012) Long-term adherence to a local guideline on postoperative body temperature measurement: mixed methods analysis . Journal of Evaluation in Clinical Practice 18 ( 4 ): 841–847. [ PubMed ] [ Google Scholar ]
• Strandberg E, Catrine Eldh A, Forsman H, et al. (2014) The concept of research utilization as understood by Swedish nurses: Demarcations of instrumental, conceptual, and persuasive research utilization . Worldviews on Evidence-Based Nursing 11 ( 1 ): 55–64. [ PubMed ] [ Google Scholar ]
• Tashakkori A, Johnson RB, Teddlie C. (2021) Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences . 2nd ed. Thousand Oaks, CA: SAGE. [ Google Scholar ]
• Wisdom JP, Cavaleri MA, Onwuegbuzie AJ, et al. (2012) Methodological reporting in qualitative, quantitative, and mixed methods health services research articles . Health Services Research 47 ( 2 ): 721–745. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Younas A, Pedersen M, Tayaben JL, et al. (2019) Review of mixed-methods research in nursing . Nursing Research 68 ( 6 ): 464–472. [ PubMed ] [ Google Scholar ]

• Reference Manager
• Simple TEXT file

People also looked at

Original research article, application of mixed reality navigation technology in primary brainstem hemorrhage puncture and drainage surgery: a case series and literature review.

• 1 Department of Neurosurgery, Chongqing Emergency Medical Center, Chongqing University Central Hospital, Chongqing, China
• 2 Pre-hospital Emergency Department, Chongqing Emergency Medical Center, Chongqing University Central Hospital, Chongqing, China
• 3 Qinying Technology Co., Ltd., Chongqing, China

Objective: The mortality rate of primary brainstem hemorrhage (PBH) is high, and the optimal treatment of PBH is controversial. We used mixed reality navigation technology (MRNT) to perform brainstem hematoma puncture and drainage surgery in seven patients with PBH. We shared practical experience to verify the feasibility and safety of the technology.

Method: We introduced the surgical procedure of brainstem hematoma puncture and drainage surgery with MRNT. From January 2021 to October 2022, we applied the technology to seven patients. We collected their clinical and radiographic indicators, including demographic indicators, preoperative and postoperative hematoma volume, hematoma evacuation rate, operation time, blood loss, deviation of the drainage tube target, depth of implantable drainage tube, postoperative complications, preoperative and 1-month postoperative GCS, etc.

Result: Among seven patients, with an average age of 56.71 ± 12.63 years, all had underlying diseases of hypertension and exhibited disturbances of consciousness. The average evacuation rate of hematoma was 50.39% ± 7.71%. The average operation time was 82.14 ± 15.74 min, the average deviation of the drainage tube target was 4.58 ± 0.72 mm, and the average depth of the implantable drainage tube was 62.73 ± 0.94 mm. Among all seven patients, four patients underwent external ventricular drainage first. There were no intraoperative deaths, and there was no complication after surgery in seven patients. The 1-month postoperative GCS was improved compared to the preoperative GCS.

Conclusion: It was feasible and safe to perform brainstem hematoma puncture and drainage surgery by MRNT. The technology could evacuate about half of the hematoma and prevent hematoma injury. The advantages included high precision in dual-plane navigation technology, low cost, an immersive operation experience, etc. Furthermore, improving the matching registration method and performing high-quality prospective clinical research was necessary.

Introduction

Primary brainstem hemorrhage (PBH) is spontaneous brainstem bleeding associated with hypertension unrelated to cavernous hemangioma, arteriovenous malformation, and other diseases. Hypertension is the leading risk factor for PBH, and other elements include anticoagulant therapy, cerebral amyloid angiopathy, et al. PBH is the deadliest subtype of intracerebral hemorrhage (ICH), accounting for 6%–10% of all ICH with an annual incidence of approximately 2–4/100,000 people [ 1 – 3 ]. The clinical characteristics of PBH are acute onset, rapid deterioration, poor prognosis, and high mortality (30%–90%) [ 1 , 4 , 5 ].

The inclusion criteria of previous ICH research all excluded PBH, such as STICH and MISTIE trials. There is no clear evidence for the optimal treatment of PBH, and the view of surgical treatment has noticeable regional differences. European and North American countries generally believe that severe disability or survival in a vegetative state is a high mental and economic burden for PBH patients and their families. These countries do not favor surgical treatment. However, many PBH surgical treatments have been carried out in China, Japan, and South Korea. Surgical treatment methods, surgical effects, monitoring methods, and complications have been investigated, and much experience has been accumulated.

In 1998, Korean scholars performed the first craniotomy to evacuate the brainstem hematoma [ 6 ]. However, in 1989, the Japanese scholar Takahama performed stereotactic brainstem hematoma aspiration surgery [ 7 ]. In our opinion, microsurgery craniotomy requires high electrophysiological monitoring and surgical skills, and these limitations are not conductive to popularization. Minimally invasive surgery has the characteristics of a simple operation, minimally invasive, and short operation time, and it is believed to reduce the damage to critical brainstem structures and protect brainstem function as much as possible. More and more minimally invasive treatments have been adopted to improve the precision of PBH puncture, including stereotactic frameworks, robotic-assisted navigation systems, 3D printing techniques, and even laser combined with CT navigation techniques.

Mixed reality navigation technology (MRNT) is based on virtual and augmented reality development. The technology uses CT images to construct a 3D head model and design an individual hematoma puncture trajectory. The actual environmental position is captured by a camera during surgery and was fused with 3D head model synchronously. MRNT not only display the model image combined with actual environment but also navigate the puncture trajectory in real time, allowing the surgeon to precisely control puncture angle and depth to achieve a perfect procedure. This technology makes the head utterly transparent during the surgery and brings an immersive experience to the surgeon.

MRNT has broad application prospects. However, it is still in its infancy, and its application in neurosurgery has rarely been reported. Furthermore, there is no report on application of MRNT in the surgical treatment of PBH. In this study, we used MRNT to perform brainstem hematoma puncture and drainage surgery in seven patients with PBH to share practical experience to verify the feasibility and safety of the technology.

Materials and methods

General information.

With the approval of the Ethics Committee of the Chongqing Emergency Medical Center, we included seven patients diagnosed with PBH from January 2021 to October 2022. All underwent brainstem hematoma puncture and drainage surgery with MRNT under general anesthesia. Indications for surgery were patients who 1) were 18–80 years of age; 2) had hematoma volume greater than 5 mL and less than 15 mL; 3) had a diameter of the hematoma greater than 2 cm; 4) had hematoma deviating toward one side or the dorsal side; 5) had GCS less than 8; and 6) had surgery within 6–24 h after onset. Family members were informed and signed the consent form [ 8 ]. Exclusion criteria were patients who had 1) brainstem hemorrhage caused by cavernous hemangioma, arteriovenous malformation, and other diseases; 2) GCS >12; 3) bilateral pupil dilation; 4) unstable vital signs; 5) severe underlying disease; or 6) coagulation dysfunction.

Mixed reality navigation technology (MRNT)

All patients preparing for surgery were required to wear sticky analysis markers in the parieto-occipital region and undergo a CT scan before surgery. CT image scanning was performed with a 64-slice CT scanner (Lightspeed VCT 6, General Electric Company, United States of America). The image parameters included in the exposure were 3 mAS, the thickness was 5mm, and the image size was 512 × 512. The DICOM data were analyzed to construct the 3D model of the hematoma and head, and the volume of brainstem preoperative hematoma was calculated using software (Medical Modeling and Design System). In addition, the hematoma puncture trajectory was designed according to the constructed head model.

After general anesthesia, the sticky analysis markers were replaced with bone nail markers, keeping the same position [ 9 ]. Based on the principle of near-infrared optical navigation, the camera captured the actual space position in real-time, fused it with the markers of the 3D head model (HSCM3D DICOM), and transmitted the information to the wearable device (HoloLens). During surgery, the camera continuously tracked the position of the puncture needle to achieve navigation function. In short, the image processing software matched and fused information from camera systems and wearable device through multiple markers. When controlling the movement of surgical tools, the software also processed the dynamic tool position data and fused it with the virtual model through wireless transmission.

Surgical procedures

Hydrocephalus patients were first treated with external ventricular drainage (EVD), and the frontal Kocher point was selected as the cranial entry point. The procedures were cutting the skin, drilling the skull, cutting the dura mater, puncturing in the direction of the plane of binaural connection, fixing the drainage tube, and suturing it layer by layer.

The patient was placed in a prone position with the head frame fixed. The puncture point was 2 cm below the transverse sinus and 3 cm lateral to the midline of the hematoma side. After cutting the skin, the muscle was separated. The dura mater was cut through a drilled hole. Wearing HoloLens, the surgeon synchronously observed actual head structure and fused puncture trajectory from multiple angles and used dual-plane navigation technology [ 9 ] for hematoma puncture. After watching that the drainage tube was in place, the puncture needle was removed, and a 5 mL empty syringe was connected for suction. The drainage tube was fixed and sutured layer by layer. The head CT was reviewed immediately after the surgery, and the decision whether to inject urokinase according to the drainage tube’s position and the residual hematoma volume. Urokinase was injected from a drainage tube for 2-3 w units every 12 h, usually 4–6 times, and kept for 1.5 h before opening the tube. The retention time of the drainage tube was no more than 72 h after the surgery. The surgical procedure to apply MRNT is shown in Figure 1 .

Figure 1 . Surgical procedure for brainstem hematoma puncture and drainage surgery with MRNT (A) Patients were required to wear sticky analysis markers in the parieto-occipital region. (B) The camera captured the real space position of the calibration plate, puncture needle, and head. (C) Wearing HoloLens, the surgeon viewed the two planes of the image. (D) MRNT displays the model image and the actual environment synchronously, allowing the surgeon to perform precise surgery. (E) The real-time navigation of MRNT showed that the puncture needle was close to the hematoma target. (F) The surgeon was aspirating the hematoma.

Clinical and radiographic indicators

The indicators for analysis included: demographic indicators, preoperative and postoperative hematoma volume, hematoma evacuation rate, operation time, blood loss, deviation of the drainage tube target, depth of implantable drainage tube, postoperative complications, and preoperative and 1-month postoperative GCS, etc.

The deviation of the drainage tube target was defined as the distance between the tip of the drainage tube and the planned puncture hematoma target. The deviation calculation was done with the BLENDER 2.93.3 software, which used the 3D global coordinate system to visualize the distance.

The head CT examination was reviewed within 24 h after surgery, and the postoperative hematoma volume was measured by non-operators using previous software (Medical Modeling and Design System). Hematoma evacuation rate = (preoperative hematoma volume - postoperative hematoma volume)/preoperative hematoma volume.

Statistical analysis

All statistical analyses were performed with SPSS (version 21, IBM, Chicago, IL, United States). Quantitative variables are presented as means ± standard deviations. The normality of quantitative variables was assessed through the Kolmogorov-Smirnov test. If the distribution was found to be normal, paired t -test were performed. The categorical variables are presented as percentages and tested by χ2 or Fisher’s test. A p -value less than 0.05 was considered statistically significant.

From January 2021 to October 2022, seven patients were diagnosed with PBH and underwent brainstem hematoma puncture and drainage surgery with MRNT. A summary of the demographic and clinical characteristics of the patients was provided in Table 1 . Among the seven patients, five were men, with an average age of 56.71 ± 12.63 years (37–74 years). The seven cases had underlying hypertension, and four cases had diabetes. The average time from onset to admission was 4.2 ± 1.47 h. Seven patients had prominent disturbances of consciousness, four required ventilator assistance, and three had a high fever.

Table 1 . Demographic and clinical characteristics of seven patients.

According to the brainstem hematoma classification advocated by Chung [ 10 ], 2 cases belonged to small unilateral tegmental type, 4 cases belonged to basal-tegmental type, and other 1 case belonged to bilateral tegmental type. The average volume of preoperative brainstem hematoma was 8.47 ± 2.22 mL (range, 5.45–12.2 mL), the average volume of postoperative brainstem hematoma was 4.16 ± 1.17 mL (range, 3.14–5.95 mL), and the differences were significant. The average hematoma evacuation rate was 50.39% ± 7.71% (range, 41.65%–63.23%). Four of the seven patients underwent EVD first (57.1%), and one underwent EVD 2 days after hematoma puncture and drainage surgery. The average operation time was 82.14 ± 15.74 min, the average blood loss was 32.2 ± 8.14 mL, the average deviation of the drainage tube target was 4.58 ± 0.72 mm (range, 3.36–5.32 mm), and the average depth of the implantable drainage tube was 62.73 ± 0.94 mm (range, 61.42–64.23 mm). Three patients were injected with urokinase after surgery, and the average retention time of the drainage tube was 53.56 ± 7.83 h.

There were no intraoperative deaths in seven patients. Two patients had slight intraoperative fluctuations in vital signs. The most common postoperative comorbidity was pneumonia (7/7, 100%), followed by gastrointestinal bleeding (5/7, 71.43%). There were no rebleeding incidents, ischemic stroke, intracranial infection, or epilepsy within 2 weeks after surgery. The preoperative high fever symptoms were relieved after surgery. Only one patient died due to pneumonia 12 days after surgery, one patient gave up 20 days after surgery. Two patients were conscious and three patients were still in a coma 1 month after surgery.

The average preoperative GCS was 6.57 ± 1.51, and the average postoperative GCS was 10.00 ± 2.83 1 month after surgery. The improvement was statistically significant. The representative cases are shown in Figure 2 and Figure 3 .

Figure 2 . The representative case 2 (A) Preoperative CT showed PBH in the axial, sagittal, and coronal planes. (B) The 3D model constructed from CT images showed hematoma and designed the puncture trajectory from the axial, sagittal, and coronary positions. (C) Postoperative CT of the axial plane showed that the drainage tube location was precise. The yellow circle indicated the tip of the drainage tube. (D) Fusion of preoperative and postoperative 3D model showed that the preoperative hematoma volume was 5.45 mL, the postoperative hematoma volume was 3.18 mL, the hematoma evacuation rate was 41.65%, the deviation of the target drainage tube was 4.22 mm, and the depth of the implantable drainage tube was 63.42 mm.

Figure 3 . The representative case 5. (A) Preoperative CT showed PBH in the axial, sagittal, and coronal planes. (B) The 3D model constructed from CT images showed hematoma, lateral ventricular, and a designed puncture trajectory from axial, sagittal, and coronary positions. (C) Postoperative CT of the axial plane showed that the drainage tube location was precise. The yellow circle indicated the tip of the drainage tube. (D) Fusion of the preoperative and postoperative 3D model showed that the preoperative hematoma volume was 10.21 mL, the postoperative hematoma volume was 5.95 mL, the hematoma evacuation rate was 41.72%, the deviation of the drainage tube target was 3.36 mm. The depth of the implantable drainage tube was 61.84 mm.

The brainstem is small, deep in the skull, and includes the midbrain, pons, and medulla oblongata. The brainstem is the center of life, controlling respiration, heart rate, blood pressure, and body temperature. About 60%–80% of PBH occurs in the pons due to the rupture of the perforating vessels of the basilar artery [ 1 , 2 ]. Hypertension is one of the most common causes of severe cerebrovascular disease. By causing mechanical and chemical damage to essential structures in the brainstem, such as the nucleus clusters and the reticular system, the hematoma quickly induces clinical symptoms such as coma, central hyperthermia, tachycardia, abnormal pupils, and hypotension. The prognosis is extremely poor, which presents a challenge to existing treatment methods.

The conservative treatment strategy for PBH is mainly related to the hypertensive treatment strategy for ICH [ 11 ]. Since the primary damage of PBH is irreversible, surgical treatment is believed to relieve mechanical compression of the hematoma and prevent secondary injury, improving prognosis [ 1 , 12 , 13 ]. However, there have been some controversies about surgical treatment. Due to the high mortality and disability rate of PBH, it is necessary to strictly evaluate the indications for surgery. Indications for surgery proposed by Shresha included a hematoma volume greater than 5 mL, a relatively concentrated hematoma, GCS less than 8, progressive neurological dysfunction, and uneventful vital signs, particularly requiring ventilatory assistance [ 14 ]. Huang established a brainstem hemorrhage scoring system and suggested patients with a score of 2–3 might benefit from surgical treatment. A score of 4 was a contraindication to surgical treatment [ 15 ]. A review of 10 cohort studies showed that the patients in the surgical group were 45–65 years old, unconscious, with a GCS of 3–8, and the hematoma volume was approximately 8 mL. The surgical group had a better prognosis and lower mortality than the conservative treatment group. The research also suggested that older age and coma were not contraindications for brainstem hemorrhage surgery [ 16 ]. According to the Chinese guidelines for brainstem hemorrhage, we specified the following surgical indications: age 18–80 years old, hematoma volume greater than 5 mL and less than 15 mL, hematoma diameter greater than 2 cm, hematoma deviated to one side or the dorsal side, GCS less than 8, surgery performed within 6–24 h after onset, and family consent [ 8 ].

The surgical treatments for PBH included microscopic craniotomy to evacuate the hematoma, which removed the hematoma as much as possible, performed hemostasis, and removed the fourth ventricular hematoma to smooth the circulation of cerebrospinal fluid. However, this technology required various intraoperative monitoring methods and proficient surgical skills. The most widely chosen method was stereotactic hematoma puncture and drainage surgery. To achieve precise puncture of the brainstem hematoma, surgeons had used invasive stereotaxic frames [ 17 ], robot-assisted navigation systems [ 18 ], the 3D printing technology navigation method [ 19 ], and laser combined with CT navigation technology [ 13 ]. The above techniques had shortcomings, including invasive placement positioning framework, the risk of skull bleeding and infection, expensive costs of robot-assisted and neuronavigation systems, the lengthy procedure of 3D printing technology, etc.

We innovatively used MRNT to perform brainstem hematoma puncture and drainage surgery. Our team used this technology to successfully perform intracranial foreign body removal [ 20 ] and minimally invasive puncture surgery for deep ICH, with a deviation of the drainage tube target of 5.76 ± 0.80 mm [ 9 ]. Based on previous experience and technical improvement, we applied technology to perform brainstem hematoma puncture and drainage surgery. The average volume of preoperative brainstem hematoma was 8.47 ± 2.22 mL, postoperative brainstem hematoma was 4.16 ± 1.17 mL, and the average hematoma evacuation rate was 50.39% ± 7.71%, which prevented hematoma primary compression and secondary injury. The surgical procedure under general anesthesia took an average of 82.14 ± 15.74 min, the average target deviation was 4.58 ± 0.72 mm, and the average depth of the implantable drainage tube was 62.73 ± 0.94 mm. The depth of the drainage tube was longer than that in the application of deep ICH, which required higher precision. Moreover, we found MRNT was safe in seven patients.

A comparison of the precision of augmented reality technology, mixed reality technology, and traditional stereotactic methods have been discussed in previous literature. Van Doormaal et al. conducted a holographic navigation study using augmented reality technology. They found that the fiducial registration error was 7.2 mm in a plastic head model, and the fiducial registration error was 4.4 mm in three patients [ 21 ]. A meta-analysis was conducted to systematically review the accuracy of augmented reality neuronavigation and compare it with conventional infrared neuronavigation. In 35 studies, the average target registration error of 2.5 mm in augmented reality technology was no different from that of 2.6 mm in traditional infrared navigation [ 22 ]. Moreover, In the study of neuronavigation using mixed reality technology, the researchers received a target deviation range of 4–6 mm [ 23 – 25 ].

The augmented reality technology application scenarios mainly involve intracranial tumors and rarely involve ICH. Qi et al. used mixed reality navigation technology to perform ICH surgery. They also used markers for point registration and image fusion. The results showed that the occipital hematoma puncture deviation was 5.3 mm due to the prone and supine position, and the deviation in the basal ganglia was 4.0 mm [ 26 ]. Zhou et al. also presented a novel multi-model mixed reality navigation system for hypertensive ICH surgery. The results of the phantom experiments revealed a mean registration error of 1.03 mm. The registration error was 1.94 mm in clinical use, which showed that the system was sufficiently accurate and effective for clinical application [ 27 ]. A summary of the deviations in the application of MR or AR was provided in Table 2 .

Table 2 . Reported cases of deviations in the application of MR or AR in neurosurgery.

In addition to precision puncture and hematoma drainage, surgical treatment of PBH also required further discussion on the timing of surgery, external ventricular drainage, and fibrinolytic drugs. Shrestha et al. found that surgical treatment within 6 h after onset was associated with a good prognosis [ 14 ]. The ultra-early operation alleviated the hematoma mass effect and reduced secondary injury. In particular, for patients with a severe condition, early hematoma aspiration could immediately eliminate harmful effects and prevent worse clinical outcomes [ 17 ] However, many primary hospitals are not equipped with PBH surgical treatment abilities. Patients have to waste a lot of time in the transfer process, which is a big challenge in clinical treatment. PBH can also cause cerebrospinal fluid circulation disorder that induces patients to become unconscious. External ventricular drainage is beneficial in improving cerebrospinal fluid circulation, managing intracranial pressure, and facilitating patient recovery [ 17 ]. In our study, external ventricular drainage was performed in five cases of seven patients. Previous research investigating the effects of rtPA on ICH and ventricular hemorrhage by MISTIE and CLAEA demonstrated that fibrinolytic drug administration did not increase the risk of hemorrhage [ 30 – 33 ]. Currently, there is no evidence and consensus to verify the effects of the thrombolytic drug used in PBH. We also found that urokinase did not increase the risk of bleeding and improve drainage efficiency, as reported in previous literature [ 13 , 18 ].

Compared with the expensive neuronavigation system, mixed reality navigation technology was an independent research and development project, the equipment of the technology was simple, and the cost was low. The effect of the technology met the clinical application of intracerebral hemorrhage surgery, and was beneficial to popularization for primary hospital.

There were also some limitations in our technology. Firstly, in order to introduce our innovative mixed reality navigation technology earlier and faster, we reported few cases, so there are not enough data to verify the advancement of the technology. At present, it was difficult to perform a cohort study because of the small number of patients enrolled. We plan to carry out clinical study with other centers in the future. Secondly, navigation technology was mainly based on point-matching technology, which enabled the fusion of the image model with the actual space through markers. Implementing invasive markers in the skull might carry potential risks of bleeding or infection. Moreover, the procedure required CT examinations before surgery, which delayed surgery time, and increased costs. Some researchers proposed the face registration plan, but the target deviation of the face registration was higher than that of the point registration, and the clinical practicability was poor [ 34 ]. Clinical practice must explore a precise, simple, fast, and noninvasive matching and fusion innovative solution.

It was feasible and safe to perform brainstem hematoma puncture and drainage by MRNT. Early minimally invasive precise surgery could prevent hematoma primary and secondary injury, and improve the prognosis of patients with PBH. The advantages included high precision in dual-plane navigation technology, low cost, an immersive operation experience, etc. Furthermore, improving the matching registration method and performing high-quality prospective clinical research was necessary.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Ethics statement

The studies involving humans were approved by Ethics Committee of the Chongqing Emergency Medical Center. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

XT: Writing–original draft, Data curation, Software. YaW: Writing–original draft. GT: Conceptualization, Project administration, Writing–original draft. YiW: Investigation, Resources, Software, Writing–original draft. WX: Resources, Formal Analysis, Writing–original draft, Writing–review and editing. YL: Methodology, Writing–original draft. YD: Writing–review and editing. PC: Writing–review and editing, Conceptualization, Writing–original draft.

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This study was financially supported by the Fundamental Research Funds for the Central Universities (2022CDJYGRH-015) and Medical Research Project of Science and Technology Bureau and Health Commission, Chongqing, China (2023MSXM076).

Conflict of interest

Author YiW was employed by Qinying Technology Co., Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. Chen P, Yao H, Tang X, Wang Y, Zhang Q, Liu Y, et al. Management of primary brainstem hemorrhage: a review of outcome prediction, surgical treatment, and animal model. Dis Markers (2022) 2022:1–8. doi:10.1155/2022/4293590

CrossRef Full Text | Google Scholar

2. Chen D, Tang Y, Nie H, Zhang P, Wang W, Dong Q, et al. Primary brainstem hemorrhage: a review of prognostic factors and surgical management. Front Neurol (2021) 12:727962. doi:10.3389/fneur.2021.727962

PubMed Abstract | CrossRef Full Text | Google Scholar

3. van Asch CJ, Luitse MJ, Rinkel GJ, van der Tweel I, Algra A, Klijn CJ. Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis. Lancet Neurol (2010) 9:167–76. doi:10.1016/s1474-4422(09)70340-0

4. Behrouz R. Prognostic factors in pontine haemorrhage: a systematic review. Eur Stroke J (2018) 3:101–9. doi:10.1177/2396987317752729

5. Balci K, Asil T, Kerimoglu M, Celik Y, Utku U. Clinical and neuroradiological predictors of mortality in patients with primary pontine hemorrhage. Clin Neurol Neurosurg (2005) 108:36–9. doi:10.1016/j.clineuro.2005.02.007

6. Hong JT, Choi SJ, Kye DK, Park CK, Lee SW, Kang JK. Surgical outcome of hypertensive pontine hemorrhages: experience of 13 cases. J Korean Neurosurg Soc (1998) 27:59–65.

Google Scholar

7. Takahama H, Morii K, Sato M, Sekiguchi K, Sato S. Stereotactic aspiration in hypertensive pontine hemorrhage: comparative study with conservative therapy. No Shinkei Geka (1989) 17:733–9.

PubMed Abstract | Google Scholar

8. Chen L, Chen T, Mao G, Chen B, Li M, Zhang H, et al. Clinical neurorestorative therapeutic guideline for brainstem hemorrhage (2020 China version). J Neurorestoratology (2020) 8:232–40. doi:10.26599/jnr.2020.9040024

9. Peng C, Yang L, Yi W, Yidan L, Yanglingxi W, Qingtao Z, et al. Application of fused reality holographic image and navigation technology in the puncture treatment of hypertensive intracerebral hemorrhage. Front Neurosci (2022) 16:850179. doi:10.3389/fnins.2022.850179

10. Chung CS, Park CH. Primary pontine hemorrhage: a new CT classification. Neurology (1992) 42(4):830–4. doi:10.1212/wnl.42.4.830

11. Greenberg SM, Ziai WC, Cordonnier C, Dowlatshahi D, Francis B, Goldstein JN, et al. 2022 guideline for the management of patients with spontaneous intracerebral hemorrhage: a guideline from the American heart association/American stroke association. Stroke (2022) 53:e282–e361. doi:10.1161/str.0000000000000407

12. Balami JS, Buchan AM. Complications of intracerebral haemorrhage. Lancet Neurol (2012) 11:101–18. doi:10.1016/s1474-4422(11)70264-2

13. Wang Q, Guo W, Zhang T, Wang S, Li C, Yuan Z, et al. Laser navigation combined with XperCT technology assisted puncture of brainstem hemorrhage. Front Neurol (2022) 13:905477. doi:10.3389/fneur.2022.905477

14. Shrestha BK, Ma L, Lan Z, Li H, You C. Surgical management of spontaneous hypertensive brainstem hemorrhage. Interdiscip Neurosurg (2015) 2:145–8. doi:10.1016/j.inat.2015.06.005

15. Huang K, Ji Z, Sun L, Gao X, Lin S, Liu T, et al. Development and validation of a grading Scale for primary pontine hemorrhage. Stroke (2017) 48:63–9. doi:10.1161/strokeaha.116.015326

16. Zheng WJ, Shi SW, Gong J. The truths behind the statistics of surgical treatment for hypertensive brainstem hemorrhage in China: a review. Neurosurg Rev (2022) 45:1195–204. doi:10.1007/s10143-021-01683-2

17. Du L, Wang JW, Li CH, Gao BL. Effects of stereotactic aspiration on brainstem hemorrhage in a case series. Front Surg (2022) 9:945905. doi:10.3389/fsurg.2022.945905

18. Zhang S, Chen T, Han B, Zhu W. A retrospective study of puncture and drainage for primary brainstem hemorrhage with the assistance of a surgical robot. Neurologist (2023) 28:73–9. doi:10.1097/nrl.0000000000000445

19. Wang Q, Guo W, Liu Y, Shao W, Li M, Li Z, et al. Application of a 3D-printed navigation mold in puncture drainage for brainstem hemorrhage. J Surg Res (2020) 245:99–106. doi:10.1016/j.jss.2019.07.026

20. Li Y, Huang J, Huang T, Tang J, Zhang W, Xu W, et al. Wearable mixed-reality holographic navigation guiding the management of penetrating intracranial injury caused by a nail. J Digit Imaging (2021) 34:362–6. doi:10.1007/s10278-021-00436-3

21. van Doormaal TPC, van Doormaal JAM, Mensink T. Clinical accuracy of holographic navigation using point-based registration on augmented-reality glasses. Oper Neurosurg (Hagerstown) (2019) 17:588–93. doi:10.1093/ons/opz094

22. Fick T, van Doormaal JAM, Hoving EW, Willems PWA, van Doormaal TPC. Current accuracy of augmented reality neuronavigation systems: systematic review and meta-analysis. World Neurosurg (2021) 146:179–88. doi:10.1016/j.wneu.2020.11.029

23. Incekara F, Smits M, Dirven C, Vincent A. Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Neurosurg (2018) 118:e422–7. doi:10.1016/j.wneu.2018.06.208

24. McJunkin JL, Jiramongkolchai P, Chung W, Southworth M, Durakovic N, Buchman CA, et al. Development of a mixed reality platform for lateral skull base anatomy. Otol Neurotol (2018) 39:e1137–42. doi:10.1097/mao.0000000000001995

25. Li Y, Chen X, Wang N, Zhang W, Li D, Zhang L, et al. A wearable mixed-reality holographic computer for guiding external ventricular drain insertion at the bedside. J Neurosurg (2018) 1–8. doi:10.3171/2018.4.JNS18124

26. Qi Z, Li Y, Xu X, Zhang J, Li F, Gan Z, et al. Holographic mixed-reality neuronavigation with a head-mounted device: technical feasibility and clinical application. Neurosurg Focus (2021) 51:E22. doi:10.3171/2021.5.focus21175

27. Zhou Z, Yang Z, Jiang S, Zhuo J, Zhu T, Ma S. Surgical navigation system for hypertensive intracerebral hemorrhage based on mixed reality. J Digit Imaging (2022) 35:1530–43. doi:10.1007/s10278-022-00676-x

28. Zhu T, Jiang S, Yang Z, Zhou Z, Li Y, Ma S, et al. A neuroendoscopic navigation system based on dual-mode augmented reality for minimally invasive surgical treatment of hypertensive intracerebral hemorrhage. Comput Biol Med (2022) 140:105091. doi:10.1016/j.compbiomed.2021.105091

29. Hou Y, Ma L, Zhu R, Chen X, Zhang J. A low-cost iPhone-assisted augmented reality solution for the localization of intracranial lesions. PLoS One (2016) 11(7):e0159185. doi:10.1371/journal.pone.0159185

30. Hanley DF, Thompson RE, Rosenblum M, Yenokyan G, Lane K, McBee N, et al. Efficacy and safety of minimally invasive surgery with thrombolysis in intracerebral haemorrhage evacuation (MISTIE III): a randomised, controlled, open-label, blinded endpoint phase 3 trial. Lancet (2019) 393:1021–32. doi:10.1016/s0140-6736(19)30195-3

31. Hanley DF, Lane K, McBee N, Ziai W, Tuhrim S, Lees KR, et al. Thrombolytic removal of intraventricular haemorrhage in treatment of severe stroke: results of the randomised, multicentre, multiregion, placebo-controlled CLEAR III trial. Lancet (2017) 389:603–11. doi:10.1016/s0140-6736(16)32410-2

32. Montes JM, Wong JH, Fayad PB, Awad IA. Stereotactic computed tomographic-guided aspiration and thrombolysis of intracerebral hematoma: protocol and preliminary experience. Stroke (2000) 31:834–40. doi:10.1161/01.str.31.4.834

33. Vespa P, McArthur D, Miller C, O'Phelan K, Frazee J, Kidwell C, et al. Frameless stereotactic aspiration and thrombolysis of deep intracerebral hemorrhage is associated with reduction of hemorrhage volume and neurological improvement. Neurocrit Care (2005) 2:274–81. doi:10.1385/ncc:2:3:274

34. Mongen MA, Willems PWA. Current accuracy of surface matching compared to adhesive markers in patient-to-image registration. Acta Neurochir (Wien) (2019) 161:865–70. doi:10.1007/s00701-019-03867-8

Keywords: primary brainstem hemorrhage, mixed reality navigation technology, brainstem hematoma puncture and drainage surgery, neuronavigation, deviation

Citation: Tang X, Wang Y, Tang G, Wang Y, Xiong W, Liu Y, Deng Y and Chen P (2024) Application of mixed reality navigation technology in primary brainstem hemorrhage puncture and drainage surgery: a case series and literature review. Front. Phys. 12:1390236. doi: 10.3389/fphy.2024.1390236

Received: 23 February 2024; Accepted: 26 March 2024; Published: 17 April 2024.

Reviewed by:

Copyright © 2024 Tang, Wang, Tang, Wang, Xiong, Liu, Deng and Chen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Yongbing Deng, [email protected] ; Peng Chen, [email protected]

† These authors share first authorship

This article is part of the Research Topic

Multi-Sensor Imaging and Fusion: Methods, Evaluations, and Applications – Volume II