articles on mixed method research

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

113k Accesses

91 Citations

13 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

Mixed Methods

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism. Run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected] , [email protected]

Affiliation Department of Psychology and Education, Universitat Oberta de Catalunya, Barcelona, Spain

Roles Conceptualization, Methodology, Validation, Writing – review & editing

Affiliation Department of Education, Universidad del Norte, Barranquilla, Colombia

Roles Conceptualization, Formal analysis, Methodology, Validation, Writing – review & editing

Affiliation Department of Management, Universidad de Alicante, Alicante, Spain

Roles Conceptualization, Validation, Writing – review & editing

Affiliation Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), University College London, London, United Kingdom

Roles Conceptualization, Supervision, Writing – review & editing

Affiliation Centre d’Estudis Sociologics sobre la Vida Quotidiana i el Treball (Sociological Research Centre on Everyday Life and Work—QUIT), Universitat Autonoma de Barcelona, Barcelona, Spain

• Sergi Fàbregues, 
• Elsa Lucia Escalante-Barrios, 
• José Francisco Molina-Azorin, 
• Quan Nha Hong, 
• Joan Miquel Verd

• Published: July 9, 2021
• https://doi.org/10.1371/journal.pone.0252014
• Peer Review
• Reader Comments

Recent growth and institutionalization in the field of mixed methods research has provided fertile ground for a wide range of thoughtful criticism of how this research approach has been developed and conceptualized by some members of the mixed methods community. This criticism reflects the increasing maturity of the field as well as the different theoretical perspectives and methodological practices of researchers in different disciplines. While debates addressing these criticisms are likely to lead to valuable insights, no empirical studies have been carried out to date that have investigated researchers’ critical views on the development and conceptualization of mixed methods research. This study examines the criticisms of the mixed methods field raised by a cross-national sample of researchers in education, nursing, psychology, and sociology. We carried out a secondary analysis of semi-structured interviews with 42 researchers and identified 11 different criticisms, which we classified in four domains: essence of mixed methods, philosophy, procedures, and politics. The criticisms related to the procedures domain were equally distributed among the four disciplines, while those related to the essence, philosophy and politics domains were more common among sociologists. Based on our findings, we argue that the divergence of views on foundational issues in this field reflects researchers’ affiliation to different communities of practice, each having its own principles, values, and interests. We suggest that a greater awareness of this divergence of perspectives could help researchers establish effective collaboration and anticipate potential challenges when working with researchers having different methodological approaches.

Citation: Fàbregues S, Escalante-Barrios EL, Molina-Azorin JF, Hong QN, Verd JM (2021) Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views. PLoS ONE 16(7): e0252014. https://doi.org/10.1371/journal.pone.0252014

Editor: Adam T. Perzynski, The MetroHealth System and Case Western Reserve University, UNITED STATES

Received: July 24, 2020; Accepted: May 7, 2021; Published: July 9, 2021

Copyright: © 2021 Fàbregues et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly as participants did not give consent for their transcripts to be shared in this manner. Since the consent statement approved by the Institutional Review Board (IRB) of the Universitat Autònoma de Barcelona, and signed by the participants, did not include the provision that data would be made publicly available, we do not have participant consent to share this data. Also, the content is sensitive, and participants could be identified. Requests for anonymized data can be made to the Principal Investigator of the study, Sergi Fàbregues ( [email protected] ) or the Institutional Review Board (IRB) of the Universitat Autònoma de Barcelona ( [email protected] ).

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Since the 1990s, mixed methods research as a distinct methodology has seen vigorous institutionalization [ 1 , 2 ] with the launch of a specialized journal ( Journal of Mixed Methods Research ) in 2007, the establishment of the Mixed Methods International Research Association (MMIRA) in 2013, and the publication of a stream of handbooks, textbooks, and articles on the philosophy and practice of mixed methods. Mixed methods research is increasingly used in a wide range of disciplines, particularly in the social, educational, behavioral and health sciences [ 3 – 6 ]. Several scholars have argued that mixed methods inquiry can help researchers arrive at a more complete understanding of research problems, develop more robust quantitative instruments, and integrate several worldviews in a single research study [ 7 , 8 ].

A clear indication of the institutionalization of mixed methods research as a field is the expansion of the mixed methods community. Tashakkori, Johnson and Teddlie [ 9 ] have characterized this community as a group of scholars who share similar backgrounds, methodological orientations, philosophical assumptions, and views on research and practice. As distinct from researchers using only qualitative or quantitative approaches, these scholars often share similarities in training, research background, and professional affiliation. Those authors’ view of the mixed methods community is consistent with Thomas Kuhn’s preferred definition of paradigms, as cited by Morgan [ 10 ]. In disagreement with the view of paradigms as incompatible epistemological stances, Morgan [ 10 ], following Kuhn [ 11 ], advanced a more integrative notion of paradigms focused on shared beliefs and joint actions in a community of researchers. Denscombe [ 12 ] took this perspective one step further by proposing the notion of communities of practice , an idea originally developed by the educationalist Étienne Wenger. This notion brings us closer to a definition of paradigms as shared beliefs able to accommodate the diversity of perspectives and approaches that currently exists within the mixed methods community [ 12 , 13 ].

While the institutionalization of mixed methods as a field has helped to formalize and clarify research practices [ 14 ], this trend has also led to some criticism of the ways in which this approach has been developed and conceptualized by some members of the mixed methods community [ 15 , 16 ]. The critiques of the mixed methods field have already been summarized in two overviews published in the early 2010s. In the second edition of the Handbook , Tashakkori and Teddlie [ 17 ] outlined four frequent criticisms raised in the mixed methods literature, including overreliance on typologies and the higher status of quantitative versus qualitative research. One year later, Creswell [ 18 ] described some of the same criticisms in a summary of 11 key controversies. The emergence of this criticism testifies to the increasing maturity of the field and its progress towards what Creswell and Plano Clark [ 7 ] define as the period of reflection and refinement in mixed methods research. In their view, the mixed methods community should be honored that it has attracted critical attention and it should stimulate debate around the issues raised. Therefore, it is crucially important to address these criticisms in greater detail because such engagement will lead to valuable insights that could lay the basis for further discussion needed to ensure the healthy development of the field. Furthermore, addressing those criticisms is essential to enhance researchers’ understanding of the complexity of the mixed methods field and to provide them with the awareness needed to deal with tensions that might emerge when working in teams with researchers subscribing to different methodological viewpoints [ 19 , 20 ].

Most of the criticisms of mixed methods so far have been formulated in the literature by several scholars specialized in theoretical aspects of methodology. However, it would be useful to find out whether other researchers share these criticisms and whether they may have formulated others. Scholars with an interest in mixed methods research come from different academic disciplines that embody different theoretical and methodological perspectives. As Plano Clark and Ivankova [ 3 ] argue, these differences are highly likely to influence the ways in which those scholars view mixed methods as well as the questions they might raise regarding current ideas in the field. Therefore, it would be useful to examine the ways in which researchers’ criticisms differ according to discipline.

The aim of the present study is to examine the criticisms of the mixed methods field raised by a cross-national sample of 42 researchers working in the disciplines of education, nursing, psychology, and sociology. We report a secondary analysis of semi-structured interviews originally conducted to describe how researchers operationalize and conceptualize the quality of mixed methods research [ 14 ]. The contribution of this article is twofold: (a) it is, to our knowledge, the first study based on an empirical approach to examining researchers’ critical views on the development and conceptualization of mixed methods research, and (b) it enhances our understanding of the ways in which these critical views may be associated with different academic disciplines.

This article reports a secondary analysis of data originally collected in a multiple-case study of the quality of mixed methods based on semi-structured interviews with researchers in the disciplines of education, nursing, psychology, and sociology. In line with Heaton’s [ 21 ] definition of secondary analysis — which he calls supplementary analysis — as an “in-depth investigation of an emergent issue or aspect of the data which was not considered or fully addressed in the primary study” [ 21 ], in this article we re-analyze the original interview data in order to address the following two research questions (RQs): (RQ1) What criticisms of the mixed methods field are made by researchers in education, nursing, psychology, and sociology? and (RQ2) What differences and similarities can be identified in the criticisms reported by researchers working in different disciplines? In the following subsections, we provide a brief description of the sampling and data collection methods used in the original study, and of the procedures used in the secondary analysis of data. A more detailed explanation of procedures followed in the original study can be found in Fàbregues, Paré, and Meneses [ 22 ].

The original study

Sampling and recruitment of participants..

The disciplines of education, nursing, psychology, and sociology were selected for four main reasons: (1) professionals working in these disciplines contribute a relatively high proportion of mixed methods empirical articles and other methodological publications, (2) a considerable number of prevalence studies and methodological reviews on the use of mixed methods in subfields of these disciplines have been published, and (3) these disciplines are characterized by their clear disciplinary boundaries, and this characteristic offers the possibility of gaining useful comparative insights [ 6 ]. Criterion and maximum variation sampling were used to select the researchers who participated in the study [ 23 ]. In the criterion sampling, participants fulfilled two inclusion criteria: (1) they had carried out research primarily in one of the four disciplines mentioned above, and (2) they had contributed to at least one methodological publication on mixed methods research. Participant identification started with a systematic search for methodological publications on mixed methods published in English during or after 2003. Selected publications fulfilled the definitions of mixed methods suggested by either Creswell and Tashakkori [ 24 ] or Johnson, Onwuegbuzie, and Turner [ 25 ]. A number of characteristics of the first authors’ profiles were extracted, including the field of expertise, the country of affiliation, and the job title. To ensure heterogeneity of the sample, maximum variation sampling was applied to authors meeting the two inclusion criteria. An iterative approach was used to recruit 11 participants for each discipline. Sample size was based on recommendations found in the literature [ 26 ]. Potential participants were contacted using a prioritized list until a total of 44 participants had been recruited.

Data collection.

Data collection involved semi-structured interviews. Questions from the interview guide were focused on the following topics: (a) participants’ research background and methodological expertise, (b) participants’ conceptualization of mixed methods research, and (c) how participants perceived the quality of the mixed methods approach in practice. Interviews were conducted using Skype, telephone and, in two cases, e-mail correspondence. Interviews were audio-recorded and transcribed, and average interview length was 49 minutes. We carried out a member-checking process by sending back to participants the transcriptions and summaries of key points of the interviews to confirm that the data accurately represented their views. At this stage, we also gave participants an opportunity to clarify or expand the statements they made during the interview.

Trustworthiness.

Four strategies were used to enhance the trustworthiness of the original study. First, as explained above, all participants member-checked their transcribed interviews and summaries to confirm accuracy. Second, peer-debriefing was carried out during data collection by one researcher working together with another researcher familiar with mixed methods research who was not included in the sample. Third, an audit trail was used to record the decisions made during the study and to help researchers to reflect on the influence on the study findings of their own assumptions and disciplines. Fourth, the decisions taken during the analysis and interpretation of the data, as well as the disagreements arising during this stage, were discussed by the researchers until a consensus was reached.

The secondary data analysis

While the original study aimed to examine researchers’ views on the conceptualization and operationalization of the quality of mixed methods research, the aim of this secondary analysis of the same data is to examine researchers’ critical views of commonly accepted concepts and practices in the mixed methods field. Ethical approval for secondary data analysis was included in the ethics application for the original study, which was approved by the Institutional Review Board of the Universitat Autònoma de Barcelona. Participants signed an informed consent before the interview. Of the 44 researchers who participated in the original study, two did not consent to the subsequent use of their interview data for a secondary analysis. Therefore, the information provided by these two researchers was not used in the present study. Table 1 shows the characteristics of the 42 participants finally included in this secondary analysis.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0252014.t001

Qualitative content analysis as described by Graneheim & Lundman [ 27 ] was used to carry out the secondary analysis of the interview data. This form of analysis is especially appropriate when, as in this study, researchers are interested in systematically describing only the topics of interest indicated by the research questions, and not in obtaining a holistic overview of all of the data [ 28 ]. The data analysis was carried out in three stages using NVivo 12 for Mac (QSR International Pty Ltd, Victoria, Australia). In the first stage, the interview transcripts were read thoroughly in order to extract the sections of text in which the participants raised criticisms of the mixed methods field. These sections of text constituted the unit of analysis. In the second stage, the extracted sections were divided into meaning units, which were subsequently condensed, abstracted and labelled with codes. Each code included a description of the meaning of the code, an indicator to identify its presence in the data, and an example of a passage coded as belonging to that code. In the third stage, the codes were compared for similarities and differences, and clustered into several categories. The underlying meaning of these categories was then examined and formulated into themes. These themes represented the study participants’ criticisms of mixed methods as a field, which were the focus of RQ1. Decisions made in this phase of the study, along with any disagreements, were discussed by the researchers until a consensus was reached.

In order to answer RQ2, a multiple correspondence analysis [ 29 – 31 ] was carried out. This technique is a non-inferential form of statistical analysis designed to analyze the multivariate association of categorical variables by generating a representation of the underlying structure of a dataset. Since the statistical requirements of multiple correspondence analysis (i.e., sampling, linearity, and normality) are highly flexible, this method is especially suited for examining qualitative interview data transformed into quantitative data [ 32 ]. The output of the multiple correspondence analysis is a scatterplot representing the spatial grouping of categories and participants. The distances between plotted points represent the degree of similarity in the patterns of participants’ responses. Multiple correspondence analysis was used to examine the relationship between the participants’ discipline and the themes relating to criticisms. To perform this analysis, we used the NVivo “matrix coding query” function to generate a matrix in which binary codes related to criticisms identified in the qualitative content analysis (the mention or failure to mention the criticism) were displayed in the columns, while the 42 participants were displayed in the rows. The matrix output was exported to XLSTAT Version 2018.1 (Addinsoft, Paris, France), which was used to perform the analysis, using the binary codes for the criticisms as active variables and the participants’ discipline as supplementary variables. Following the recommendations of Bazeley [ 32 ], after carrying out the multiple correspondence analysis, we checked the results against the qualitative data to verify the interpretation of the statistical analysis.

RQ1; What criticisms of the mixed methods field are made by researchers in education, nursing, psychology, and sociology?

Eleven criticisms of how some members of the mixed methods community have developed and conceptualized this research approach were identified in 27 of the 42 interviews included in this secondary analysis. These criticisms were then grouped into the four domains used by Creswell [ 33 ] to map the landscape of mixed methods research: (1) the essence of mixed methods research (definitions and nomenclature), (2) philosophy (philosophical assumptions and paradigmatic stances), (3) procedures ( methods and techniques for carrying out mixed methods research), and (4) politics (justification of the use of mixed methods research). Each of these four domains and the corresponding criticisms are discussed in the following sections with the support of verbatim quotes from the interviews. Table 2 shows the criticisms for each domain and the number of participants making each criticism.

https://doi.org/10.1371/journal.pone.0252014.t002

Domain 1: The essence of mixed methods research.

Criticism 1 : The accepted definition of mixed methods research takes into account only the mixing of both quantitative and qualitative methods . Some participants objected that the most common definition of mixed methods research that usually prevails in the literature conceives the approach as being limited to the use of quantitative and qualitative methods. These participants believe that the field should adopt a broader definition that would also include the mixing of methods within the same tradition in a single design, that is, the combination of two or more quantitative methods or qualitative methods:

“(…) for me mixed methods is not only mixing qualitative and quantitative methods , but it could also be qualitative + qualitative or quantitative + quantitative methods” (Sociologist #4).

These participants argued that the current definition of mixed methods implies that mixing the two distinct families of methods is often the only appropriate approach while in fact this definition obviates the contingent nature of research. Certain research questions might be better answered by using a combination of methods from a single tradition. Furthermore, combining two methods from the same tradition can be as valuable and as challenging as combining two methods from different traditions. One participant used the term “pressure” to describe the feeling that he was obliged to mix quantitative and qualitative methods even when this approach was not the most appropriate one:

“ … combining methods isn’t just a matter of combining quantitative and qualitative methods . You can combine different methods that are both qualitative or both quantitative and that’s , that’s valuable in it- itself , and I am worried about the kind of pressure to combine quantitative and qualitative as if that would always be appropriate” (Sociologist #2).

Criticism 2 : The terminology used in mixed methods reflects a lack of agreement among its proponents . Several participants noted the lack of clear agreement on the terminology generally used to describe the concepts and procedures that pertain to mixed methods research. They also cited a tendency to use multiple definitions for the same term and different terms to refer to similar notions. One participant cited as problematic the use of several different terms (e.g., legitimation, validity, rigor) to refer to the quality of mixed methods research:

“I would like to see a word that’s used by as many people as possible to describe that [quality]… . But , you know , I , I just think if we , everybody continues to use different terms , that could be problematic” (Educationalist #8).

According to this participant, while synonymous terms might add some precision when used to describe the complexities of implementing mixed methods research, their use can also generate confusion, especially among reviewers, editors and researchers who are trying to familiarize themselves with the field:

“…it just gets to the point where if everyone has a different definition , then how useful is that ? And that gets confusing for those who review manuscripts , or editors , when people are using in different ways that exact same term” (Educationalist #8).

Participants suggested two possible reasons for this lack of agreement. First, the tendency among some scholars to consider that mixed methods researchers should be able to use whatever terminology they may find convenient. Second, the desire of some authors to claim priority for the terminology that defines a particular method or typology. In order to resolve this lack of agreement, participants suggested that members of the mixed methods community should work towards building a greater consensus on terminology:

“There needs to be a common language” (Educationalist #9).

Criticism 3 : Mixed methods research is not a new type of methods practice . Some participants noted the tendency in the literature to present mixed methods as a new type of research practice that emerged during the past three decades. They pointed out that the use of mixed methods has a prior history that considerably predates the time when it became formalized as a research field. These participants cited examples of studies in sociology by Jahoda and Zeisel (Marienthal study of unemployment) and fieldwork in anthropology by Margaret Mead, both dating from the early 20th century. While these studies had an influence on methodology in the social sciences on account of the ways in which they creatively combined multiple quantitative and qualitative data sources, they have been generally overlooked in the mixed methods literature:

“I don’t particularly think that [mixed methods research has allowed us to answer research questions which were left unanswered in the past] but what I do think is that , you know , do remember as well that mixed methods research does actually have a long history in Sociology” (Sociologist #7).

Domain 2: Philosophy.

Criticism 4 : Mixed methods research is not a third paradigm . A considerable number of participants argued against the idea of characterizing mixed methods research as a third paradigm. They found two major faults with this characterization. First, it relies on the idea of mixed methods research as an approach that is distinct from quantitative and qualitative methodologies. In the view of these participants, mixed methods approaches do not rely on singular elements that are distinct in their nature, philosophy, or procedures:

“So no , I think , ultimately , I’m probably , I’m not really convinced that is a distinct methodology (…) So I worry when , when the idea of something that’s very special about mixed methods is given a lot , is given too much primacy” (Sociologist #3). “I don’t think it’s helpful to see it as a separate approach in terms of actually conducting , you know , planning and conducting , the research…I certainly think it’s stretching it to see it as a different , as a separate paradigm…I think the whole idea of ‘paradigm’ is a little bit difficult” (Sociologist #5).

Second, the conceptualization of mixed methods as a paradigm presupposes a strong link between epistemology and method, that is, the identification of the use of mixed methods with a particular epistemological or ontological view, whereas, in fact, these are separate entities. Attaching epistemological and ontological assumptions to mixed methods research would weaken its functionality and creative potential:

“…if we restrict mixed methods to only one paradigm then we’re bottlenecking mixed methods into a certain area , and we restrict the functionality of it” (Nurse, #10).

Criticism 5 : Current discussions of mixed methods research conceive quantitative and qualitative research as separate paradigms . Related to the previous criticism, a number of participants noted that current conceptualizations of mixed methods take for granted the nature of the quantitative and qualitative approaches, conceiving them as separate paradigms based on particular philosophical assumptions, thus reinforcing the conventional divide between them and accentuating their differences:

“…the whole purpose , of course , of mixed methods is that it’s , that’s a paradigm , but I’m not convinced it is because it still draws on those conventional traditional paradigms…I find that’s likely less helpful because again it starts from the assumption that there is a strong division between qualitative and quantitative research” (Educationalist #2).

These participants stated that the mixed methods literature may have uncritically incorporated the methodological “rules” (conventions) that were dominant in the 1980s by associating qualitative research with the constructivist paradigm and quantitative research with the positivist paradigm. This linkage between philosophy and method may have been a result of the process of formalization of the methodology carried out by the “second generation of mixed methods researchers” (from 1980s to present), while the “first generation” (i.e., from the 1900s to 1980s) might not have had a philosophical problem:

“I think the biggest problem that mixed methods research is in right now is having adopted , without reflection the rules that were established in the mid 80’s on , on paradigms in quali and quanti…we have these pillars , these quali-quanti pillars and we’re working on these rooms…All these classical studies [from the first generation] had no problems in doing quali-quanti , it was only the attempt to formalize it which has actually created these , these problems” (Sociologist #6).

According to this participant, the association of the quantitative and qualitative approaches with particular epistemological stances contradicts the very nature of the mixed methods approach: if such philosophical and methodological differences between quantitative and qualitative research really existed, then the integration of the two approaches would not be possible:

“…the big problem with having adopted this [association] , on the one hand , it actually makes mixed methods impossible . So , it is not possible within one single design to argue that your da- , that there is a single and objectifiable reality out there , on the other hand , and there are multiple or no reality , there’s no reality” (Sociologist #6).

In the view of another participant, part of the mixed methods literature may have accentuated the differences between the two methodologies by representing their characteristics in different columns in a table, while ignoring the existence of methods that incorporate features of both approaches (e.g., survey containing both open and closed ended questions or qualitative studies that include descriptive statistics):

“I know what’s been recently suggested in the literature (…) I’m not even sure that I would say that we should have drawn a line between qualitative and quantitative as firmly as we have . A lot of the qualitative work that I do includes descriptive statistics” (Educationalist #7).

Criticism 6 : Superficiality of pragmatism . Some participants argued that authors in the mixed methods community sometimes characterize the notion of pragmatism in a superficial way by reducing it to merely eclecticism and confusing it with “practicalism”. In this way, these authors advocate a “what works” approach which may be useful when justifying the integration of the quantitative and qualitative methods, but this attitude distorts the nature of pragmatism by failing to consider its underlying theoretical and philosophical assumptions:

“…they tend to think that pragmatism is just the practicalities , and it’s just the technicalities…” (Educationalist #2).

One participant noted that the feebly argued debates on pragmatism to date may have led the mixed methods community to undervalue the important contribution this paradigm has made to the philosophical basis of empirical inquiry:

“I’ve probably never in my life seen such weak debates on pragmatism as I have in mix- , in the mixed methods debate . I mean if I think of this fabulous contribution that , that pragmatism as a philosophical discipline has made” (Sociologist #6).

Moreover, another participant observed that many researchers in the field have acquired their knowledge of pragmatism mainly from the descriptions of the mixed methods paradigm found in the literature, whereas a sound basis for pragmatism in mixed methods research practice would require consulting the seminal papers on pragmatism, such as those by John Dewey, Charles Sanders Pierce or William James:

“… from what I’ve read anything about pragmatism that’s in a mixed methods paper does tend to be superficial…you have to go right back to the original authors of pragmatism and I think sometimes when we speak about pragmatism in mixed methods research , students particularly ten- tend , maybe just read some articles in pragmatism and think they know about this , but I think it is important to go right back to Dewey and James and Pierce” (Nurse, #6).

Criticism 7 : Mixed methods research aligns with positivism . A few participants noted that some members of the mixed methods community tend to accord a higher status to the quantitative component because they consider that it is more objective and more closely embodies the scientific method. In their view, some researchers regard the qualitative component as mainly a supplement to the quantitative component. Consequently, researchers may fail to appreciate the added value that may be gained by using mixed methods research:

“qualitative research [is often used] to almost to kind of flesh out the , the , the quantitative aspects , so it’s a kind of embellishment rather than seeing it as something that might challenge some of the quantitative findings or might contribute to , to ultimately rephrasing the research question or to reanalyzing the , the quantitative data” (Sociologist #5).

Domain 3: Procedures.

Criticism 8 : Limitations of typologies . A number of participants criticized the tendency of some authors to present mixed methods designs and procedures from a typological perspective. Typologies are used in the mixed methods literature as classifications of methodological features, such as the timing and priority of the quantitative and qualitative components and the stage at which integration is carried out [ 7 , 9 ]. In the view of those participants, typologies are presented in the literature in a way that is excessively mechanical and prescriptive, unnecessarily simplifying the process of carrying out a mixed methods study by suggesting that a successful implementation of a mixed methods design can be carried out only by following a predefined set of steps:

“I’m arguing against approaches that I think are too sort of mechanical in the sense of laying out : ‘Ok , here’s categories A , B , C , D and E , and here are the rules for applying them . And if you just follow the rules , then you’ll be ok’” (Educationalist #1).

Participants noted that this approach entails four problems. First, in order to adapt their approach to the research questions that they need to answer, researchers may need to modify the guidelines suggested in the literature. Therefore, guidelines for the use of mixed methods designs should be only “guiding principles” that are adaptable to varying circumstances and able to take into account the interactions between the different elements of the design. One of these participants stated the following:

“I think , I mean , I started off by using…the sort of prescription…and it’s only when you start to get delve more into mixed methods… So , it’s , it’s really , I think [they should be] just guiding principles” (Nurse, #5).

Second, the typologies may curtail the creativity of researchers by restricting them to a series of predefined models that are considered the “correct” ways of combining quantitative and qualitative methods. As expressed by the following participant:

“…researchers are using mixed methods in such creative ways , it’s like , it’s just , when you read these designs and they can be just so , so different and they just don’t fit into , you know , the typologies” (Nurse, #3).

Third, rather than being empirically generated by examining how mixed methods research is actually carried out in practice, these typologies are the highly formalized result of a list of ideal designs formulated by mixed methods theorists, as noted by this participant:

“There were basically two different approaches [to the development of typologies] and the one that was most common was the sort of develop very formal systems…The opposite of that was Bryman who went out and interviewed qualitative researchers about what they did…he talked to people about what they really did rather than coming up with formal systems” (Sociologist #1).

Fourth, the existing typologies are too extensive, which makes them difficult for inexperienced researchers to apply, as we can see in the following quote:

“…there must be forty or fifty different designs associated with mixed methods and I think it’s , you know , I think that’s confusing to people and it’s … in some way I think it becomes irrelevant” (Psychologist #9).

Criticism 9 : Procedures described in the literature are not aligned with mixed methods practice . Some participants mentioned occasional discrepancies between the procedures explained in textbooks and articles and the implementation and reporting of mixed methods in practice, which may not always conform to published guidelines and typologies.

“you , you open any textbook…and the rules that are proposed there are broken every day very successfully by researchers who , who actually conduct the research…the practice and the debate need to run parallel and they probably , right now I think they are a bit too separate from each other” (Sociologist #3).

Participants attributed this disjuncture to the fact that a few influential authors probably lacked sufficient practical experience in using mixed methods. These authors may have tended to suggest methodological guidelines “from their desk” without testing them in practice or reviewing the empirical work of other researchers:

“…there might be a gap that , that a lot of researchers talk about using mixed methods , but I’m not sure if they actually do it in practice” (Psychologist #2).

Furthermore, participants also noted that in some cases those authors may have placed greater emphasis on the philosophical and theoretical basis of mixed methods than on describing the techniques involved in implementing mixed methods research:

“…people spend far too much time talking about epistemology , most of those discussions are actually very simple , but people make them very complicate…I think…technical questions about how you work with the data and what it means [are more important]” (Educationalist #6).

Domain 4: Politics.

Criticism 10 : Mixed methods research is not better than monomethod research . Some participants pointed out a tendency among some members of the mixed methods community to consider this type of research to be inherently superior to monomethod research:

“…thinking about the , the papers that I’ve reviewed have been for the kind of applied end journals…I think the main issue for me has been in terms of , you know , the purpose of using mixed methods; that I think there’s a tendency to slip into thinking that more is necessarily better” (Sociologist #9).

Participants noted that to attribute a higher status to mixed methods research is wrong because this view could lead to the oversimplification of other approaches, which would undermine their prestige. Furthermore, participants argued that a mixed methods approach is not always the best research option and a fully integrated design may not be the most appropriate. What really determines the suitability of an approach or a design is the research question of a study, so that a monomethod design is sometimes the most appropriate.

Criticism 11 : Homogenization of mixed methods research . A few participants criticized a tendency in the mixed methods field to homogenize terminology and procedures. In their view, some members of the community have tried to develop a “mixed methods way of doing things” which would be acceptable to all researchers and would require them to write in a particular way using particular terminologies and strategies:

“They’re , they’re trying to develop a language , they’re trying to develop an approach , a strategy that , that is going to be acceptable by all mixed methods researchers , which really is , is unacceptable” (Educationalist #2).

This attitude towards homogenization of mixed methods research could hinder the advancement of the field since it promotes a uniform approach, suppresses intellectual disputes and ignores the diversity of approaches and attitudes regarding mixed methods found in the literature. As one participant argued, to find space for legitimate difference in the field is very difficult due to the protectionist attitude of some prominent authors who are interested in propagating their own ideas rather than incorporating the ideas of other authors:

“I fear that there’s , among , among those who have some prominence , there are some who would be very eager to protect their own turf and not wanting to come together for some kind of joint effort” (Educationalist #3).

RQ2: What differences and similarities can be identified in the criticisms reported by researchers working in different disciplines?

Of the 27 participants who raised criticisms, ten were sociologists, eight were educationalists, five were nurses, and four were psychologists. Multiple correspondence analysis was used to analyze the differences, depending on their discipline, in the types of criticisms the participants raised. Fig 1 shows the multiple correspondence analysis map for the first two axes. Highly associated categories are plotted near to one another on the basis of their loading to the corresponding axes, while the least associated categories are plotted far from one another. Therefore, the axes should be interpreted based on the grouping seen on the map of the relative positions of the categories, which are expressed by the magnitude of the coordinates. The measures known as eigenvalues indicate how much of the categorical information is explained by each dimension. Higher eigenvalues indicate a greater amount of variance of the variables in that dimension [ 29 , 30 ].

Note: The Yes label indicates that the criticism was mentioned, while the No label indicates that the criticism was not mentioned.

https://doi.org/10.1371/journal.pone.0252014.g001

On Axis 1, which accounts for most of the variation in the data (eigenvalue of 92.4%), the yes categories of the criticisms associated with the domains of philosophy, politics and the essence of mixed methods are plotted on the right-hand side of the map, while the no categories of the same domains are on the left-hand side of the map. As shown in the figure, sociologists were more likely to formulate criticisms associated with the three abovementioned domains while psychologists and nurses were less likely to formulate criticisms associated with any of these three domains. Educationalists were less inclined than sociologists to formulate criticisms associated with those domains, as indicated by the proximity of the education category to the centroid (the center of the axis). Finally, the fact that the yes and no categories of the procedures domain are plotted on Axis 2, which has very low explanatory power (eigenvalue of 7.6%), reveals that participants in the four disciplines showed no relevant differences in formulating this criticism.

Summary of findings

The aim of this study, based on a secondary analysis of interview data, was to describe the criticisms of mixed methods as a field raised by a sample of researchers in the disciplines of education, nursing, psychology, and sociology. Overall, the findings revealed a number of criticisms related to several different issues. These criticisms were initially identified by the first author in the data analysis phase of the original study, which was focused on the conceptualization and operationalization of quality in mixed methods research. The criticisms were particularly relevant since they were unprompted, that is, they were spontaneously given by the participants in response to questions about another subject, rather than to questions about criticisms. Furthermore, the original study was based on a broad and diverse sample of participants; it included a few researchers from the disciplines of sociology and education known for their criticisms as well as a considerable number of researchers whose publications seemed to be neutral on the subject of criticisms of the mixed methods field. To this latter group of researchers, the interviews afforded an opportunity to express their disagreement with some predominant notions in the mixed methods field which they otherwise might not have published. Another key element of this study is the examination of the relationship between the discipline of the participants and the frequency and type of criticisms they made. This has been possible since the sample is relatively balanced in terms of the number of participants from each of the four disciplines included in the study.

In response to RQ1, participants raised a total of eleven unprompted critical remarks, categorized in the following four domains: the essence of mixed methods (three criticisms), philosophy (four criticisms), procedures (two criticisms), and politics (two criticisms). Nine of the eleven critical comments have been previously mentioned in the literature. For instance, on the essence of mixed methods domain, some authors have alluded to problems such as the narrow definition of mixed methods research [ 15 , 34 – 36 ], the lack of agreement on the terminology used [ 16 , 37 ] and the inappropriateness of considering mixed methods a new methodology [ 35 , 38 , 39 ]. In the philosophical domain, several authors have criticized the dominance of a positivist approach to mixed methods research in some disciplines [ 37 , 40 , 41 ] while some authors have pointed out that considering as a separate or distinct paradigm can lead to an artificial separation of the quantitative and qualitative approaches [ 42 – 45 ]. In the procedures domain, a number of authors cited the problems inherent in conceptualizing mixed methods designs typologically, since such a view is restrictive and unable to reflect the variety of mixed methods designs used in practice [ 38 , 46 , 47 ]. Finally, in the politics domain, a few authors have criticized a tendency, in some of the literature, to homogenize the field [ 48 ], while others have critically noted the occasional adoption of a universalist position based on the idea that the mixed methods approach is inherently superior to monomethod research [ 44 , 47 ]. However, we also identified two criticisms not previously mentioned in the literature: the excessively superficial characterizations of pragmatism (criticism 6) found occasionally in the mixed methods literature and the description of procedures that are not necessarily in line with research practice (criticism 9).

Regarding criticism 6, some participants noted a tendency on the part of some researchers in the field to cite, when writing about pragmatism, what other mixed methods researchers had written about this paradigm rather than citing foundational writings, such as those by John Dewey, William James, or Charles Sanders Peirce. In fact, none of the most influential and most frequently cited textbooks on mixed methods research in the four disciplines we studied cites any work by key authors in the pragmatist tradition. Therefore, it could be useful to learn whether these authors’ highly synthetic explanation of foundational knowledge leads inexperienced researchers to only weakly engage with this paradigm, or, on the other hand, whether this simplification might help them to grasp the basic principles of pragmatism more quickly while leading them to consult first-hand the foundational writings.

With respect to criticism 9, participants reported that researchers do not always follow in practice all of the procedures described in mixed methods textbooks. This disjuncture between textbook guidance and research practice has been described in several methodological reviews of the use of mixed methods in the four disciplines we included in our study. Features such as explicitly stating the mixed methods design used, reporting mixed methods research questions, or explicitly stating the limitations associated with the use of a mixed methods design are regarded by some authors as key characteristics of mixed methods studies (Creswell and Plano Clark, 2018; Plano Clark and Ivankova, 2016; Onwuegbuzie and Corrigan, 2014; O’Cathain et al, 2008). However, Bartholomew & Lockard (2018) reported that very few of the studies included in their review of the use of mixed methods in psychotherapy explicitly stated the mixed methods design used (13%) or reported mixed methods research questions (29%). Additionally, Bressan et al. (2017) and Irvine et al. (2021), in their reviews on mixed methods in nursing, found that most of the studies they included failed to report the limitations associated with the use of a mixed methods design. Therefore, it could be of great interest to study whether the omission of these characteristic features of mixed methods studies reflects the researchers’ view that these features are unimportant, or whether they are unfamiliar with reporting standards. Identifying this latter criticism is a particularly relevant finding of this study, since the intimate context of the interview might have led to the expression of subjective judgments that otherwise might not have come to light (i.e., the participant’s perception that some authors may not habitually carry out empirical research).

With respect to RQ2, we found relevant differences in the type of criticisms raised across disciplines. In fact, one of our key findings is that criticisms in the procedures domain were equally distributed across the four disciplines, while criticisms in the essence of mixed methods, philosophy and politics domains were clearly more common in sociology. First, these findings are consistent with statements made by Plano Clark and Ivankova [ 3 ] regarding the ways in which the sociocultural context of researchers — including the discipline in which they work — can shape their beliefs, knowledge and even experiences with regards to mixed methods. Indeed, the greater number of criticisms made by sociologists categorized in three of the four domains shows how disciplinary conventions might affect how researchers think about mixed methods and judge the acceptability of certain predominant conceptualizations. According to several authors [ 49 – 51 ], critique is a foundational and distinct feature of the discipline of sociology. Therefore, the generalized tendency among sociologists to question traditional assumptions about the order of the world and to detach themselves from predominant belief systems and ideologies might help to explain why many sociologists in our sample criticized ideas such as the conceptualization of quantitative and qualitative research as separate entities and the consideration of mixed methods as inherently better than monomethod research.

Theoretical implications of the study

The findings of our study highlight several differences in opinion in the mixed methods field previously identified by authors such as Greene [ 52 ], Tashakkori and Teddlie [ 17 ], Leech [ 53 ] and Maxwell, Chmiel, and Rogers [ 54 ], among others. Those authors showed that, owing to differences in philosophical and theoretical stances and their adherence to different research cultures, researchers in this field sometimes disagree on foundational issues such as nomenclature, the need for consensus, and the definition of mixed methods research. In our study, the participants, particularly those in the field of sociology, made several criticisms about how some foundational and philosophical aspects of mixed methods research have been conceptualized by the mixed methods community, including how mixed methods has been defined and accorded status as a third paradigm. Furthermore, our findings showed contradictory criticisms formulated by the participants as a group: while some researchers criticized the lack of a consensus in the field on the terminology used to describe mixed methods research (criticism 2), others criticized a tendency by some authors to homogenize terminology (criticism 11).

This divergence of views is consonant with the notion of communities of practice suggested by Denscombe [ 12 ]. Departing from Kuhn’s notion of paradigms as “shared beliefs among the members of a specialty area” (as cited by Morgan [ 10 ]), in Denscombe’s view, the broader mixed methods community is a paradigm encompassing a conglomerate of multiple research communities shaped by the principles, values and interests prevailing in their disciplines and research orientations. In line with that author’s view that methodological decisions and viewpoints “will be shaped by a socialization process involving the influence of peers” [ 12 ], our findings suggest that the disciplinary community of our participants is likely to have informed their criticisms of the mixed methods field. Although this divergence of views suggests that complete agreement and unhindered communication among researchers is not possible [ 55 ], Ghiara [ 56 ] argued that, in Kuhn’s view, some form of communication is always possible; and furthermore, conflicting viewpoints can be reconciled to a certain extent. Similarly, Johnson [ 57 ] has argued that this diversity of critical voices, rather than being a problem, merely indicates that “reality is likely plural”—there is no single set of ontological assumptions underlying mixed methods—and, furthermore, knowledge is articulated on the basis of “multiple standpoints and strategies for learning about our world” that can be reconciled to some degree. An example of the healthy coexistence of divergent viewpoints within the mixed methods community can be found in the Journal of Mixed Methods Research , the leading journal in the field, described in its webpage as a “primary forum for the growing community of international and multidisciplinary scholars of mixed methods research”. The journal publishes a wide range of manuscripts, including articles revealing approaches to mixed methods research that rest on divergent foundational and philosophical perspectives.

The desire for inclusion of divergent viewpoints should not lead researchers to ignore the challenges posed by this divergence. Due to the interdisciplinary nature of mixed methods research, Curry et al. [ 58 ] argue that mixed methods teams often include researchers with different methodological backgrounds and propensities. Occasionally, these differences may pose challenges for establishing effective collaboration and for efficiently integrating research methods. A greater awareness of multiple perspectives on mixed methods research, including divergent critical views like those reported in our study, could help researchers better anticipate difficulties that might present themselves in the course of working with researchers who hold differing viewpoints. Furthermore, as Maxwell, Chmiel, and Rogers [ 54 ] have suggested, a better understanding on the part of mixed methods researchers of the perspectives of others in the field who embrace a differing approach should facilitate the process of integrating quantitative and qualitative methods in studies where different ontological positions coincide. In a similar vein, such an understanding could also help overcome a form of methodological tokenism described by Hancock, Sykes and Verma [ 59 ]. This can occur when mixed methods researchers fail to attend to, and therefore align, the distinct epistemological and ontological premises that underlie the methodological orientations that are integrated in a mixed methods design.

Furthermore, the recognition of mixed methods researchers’ divergence of views should be an integral part in any effort to design and implement a curriculum for mixed methods research. Plano Clark and Ivankova [ 13 ] have pointed out that any lack of clarity concerning the existing disagreements about foundational elements of the mixed methods approach could be confusing to researchers inexperienced in this field. Therefore, it is essential that courses and workshops on mixed methods research take note of these criticisms. This last point is particularly important since the topic of critical viewpoints is not included in any of the mixed methods syllabus exemplars published in the literature, including those by Earley [ 60 ], Christ [ 61 ], and Ivankova and Plano Clark [ 13 ].

Limitations, strengths, and possibilities for future research

Our findings are subject to a few limitations. First of all, the interviews were carried out by Skype and telephone. While these two forms of communication allowed us to interview participants residing in various locations around the world, they limited the possibilities for building the sort of rapport that might have encouraged some participants to elaborate more in their responses. To minimize this limitation, participants were given the opportunity in the member-checking phase to add additional insights to their initial statements. Second, the transferability of our findings is limited by our decision to include only four disciplines while excluding the views of researchers working in other disciplines that also have a high prevalence of mixed methods studies, such as medicine, business, and information science. From our findings alone, it is not possible to infer how frequent these criticisms are and what types of criticisms may be more prevalent in each of the disciplines. Third, multiple correspondence analysis is an exploratory method not appropriate for testing hypotheses or statistical significance. In other words, the method is designed to describe associations between categorical variables rather than to make predictions about a population [ 31 ]. Therefore, in light of the limitations of multiple correspondence analysis for drawing deeper inferences, the findings regarding RQ2 should be considered provisional and subject to further investigation. Finally, since this study is based on a secondary analysis, the interview questions did not specifically prompt the participants to bring up criticisms since the questions were focused on participants’ views related to the quality of mixed methods research. If we had specifically prompted participants to report their own criticisms, it is likely that more critical opinions would have been gathered.

While the use of secondary data entailed certain limitations, it also conferred some advantages. Participants’ critical statements were entirely spontaneous since they were not explicitly solicited. This spontaneity probably helped to reduce the social desirability bias, that is, a presumed tendency for respondents to dissimulate their own critical views in a way that might seem professionally and socially more acceptable. Furthermore, despite the limitations of multiple correspondence analysis, this method allowed us to generate a parsimonious visual representation of the underlying patterns of relationships among the criticisms and the disciplines. This representation helped us to improve our interpretation of the qualitative findings and to identify useful leads for carrying our further analysis. A further strength is that the study included a broad sample of participants in terms of their geographic location, academic position, seniority, and methodological expertise. This diversity probably also afforded us access to a wider range of views. Finally, a key strength of our study is that, to our knowledge, it is the first empirical study that has addressed the topic of criticisms of mixed methods as a field.

This study represents a step towards a better understanding of some current criticisms of mixed methods research. However, further research will be needed to confirm and expand our findings. Future research based on a larger and more diverse sample of mixed methods researchers could extend the scope of our research questions and help researchers generalize from our findings. Such studies might help us discover whether researchers from other disciplines share the criticisms made by the participants in our study and whether those researchers harbor other criticisms of their own. Other analytical tools could be used to examine in greater detail how the circumstances and attributes of researchers — including their disciplinary background, methodological expertise and paradigmatic viewpoints — influence the way these scholars formulate their criticisms of the mixed methods field. Finally, future research is needed to examine the critical accounts of researchers less experienced in methodological writing and probably less exposed to current theoretical debates and developmental issues in the field.

Acknowledgments

The authors would like to acknowledge the help of Dick Edelstein in editing the final manuscript. The first draft of this paper was written while the first author was a visiting scholar in the Mixed Methods Program of the University of Michigan, directed by Michael D. Fetters, John W. Creswell, and Timothy D. Guetterman. This author would like to thank these three scholars for their support and, particularly Michael D. Fetters for his invaluable mentorship over the past few years.

• 1. Hesse-Biber S. Navigating a turbulent research landscape: Working the boundaries, tensions, diversity, and contradictions of multimethod and mixed methods inquiry. In: S Hesse-Biber, Johnson RB, editors. The Oxford handbook of multimethod and mixed methods research inquiry. New York, NY: Oxford University Press; 2015. p. xxxiii–liii.
• View Article
• Google Scholar
• 3. Plano Clark VL, Ivankova NV. Mixed methods research: A guide to the field. Thousand Oaks, CA: Sage; 2016.
• PubMed/NCBI
• 6. Ivankova NV, Kawamura Y. Emerging trends in the utilization of integrated designs in the social, behavioral, and health sciences. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 581–612.
• 7. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks, CA: Sage; 2018.
• 8. Fetters MD. The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Thousand Oaks, CA: Sage; 2020.
• 9. Tashakkori A, Johnson RB, Teddlie C. Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. 2nd ed. Thousand Oaks, CA: Sage; 2021.
• 11. Kuhn TS. The structure of scientific revolutions. 4th ed. Chicago: The University of Chicago Press; 2012.
• 17. Tashakkori A, Teddlie C. Epilogue: Current developments and emerging trends in integrated research methodology. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 803–26.
• 18. Creswell JW. Controversies in mixed methods research. In: Denzin NK, Lincoln Y, editors. The SAGE handbook of qualitative research. 4th ed. Thousand Oaks, CA: Sage; 2011. p. 269–83.
• 21. Heaton J. Reworking qualitative data. London: Sage; 2004.
• 23. Patton MQ. Qualitative research & evaluation methods. 4th ed. Thousand Oaks, CA: Sage; 2014.
• 28. Schreier M. Qualitative content analysis in practice. London: Sage; 2012.
• 29. Greenacre M. Correspondence analysis in practice. 3rd ed. Boca Raton, FL: Chapman & Hall/CRC; 2017.
• 30. Le Roux B, Rouanet H. Multiple correspondence analysis. Thousand Oaks, CA: Sage; 2010.
• 31. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis: A global perspective. 7th ed. Upper Saddle River, NJ: Pearson Education; 2010.
• 32. Bazeley P. Integrating analyses in mixed methods research. London: Sage; 2018.
• 33. Creswell JW. Mapping the developing landscape of mixed methods research. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 45–68.
• 39. Pelto PJ. Mixed methods in ethnographic research: Historical perspectives. New York, NY: Routledge; 2017.
• 43. Niglas K. The multidimensional model of research methodology: An integrated set of continua. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 215–36.
• 52. Greene JC. Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass; 2007.
• 53. Leech NL. Interviews with the early developers of mixed methods research. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 253–72.
• 54. Maxwell JA, Chmiel M, Rogers SE. Designing integration in multimethod and mixed methods research. In: Hesse-Biber S, Johnson RB, editors. The Oxford handbook of multimethod and mixed methods research inquiry. New York: Oxford University Press; 2015. p. 223–39.

ORIGINAL RESEARCH article

A mixed methods research study of parental perception of physical activity and quality of life of children under home lock down in the covid-19 pandemic.

• 1 Transdisciplinary Research Center in Psychology, Autonomous University of the State of Morelos, Cuernavaca, Mexico
• 2 Institute of Psychology and Special Education, Department of Applied Psychology, University Center for Health Sciences, University of Guadalajara, Guadalajara, Mexico
• 3 Facultad de Ciencias Sociales, Universidad Europea de Canarias, La Orotava, Spain
• 4 Faculty of Psychology and Speech Therapy, Department of Clinical Psychology, Psychobiology and Methodology, University of La Laguna, Santa Cruz de Tenerife, Spain

Household confinement due to the rapid spread of the pandemic caused by COVID-19 has brought very significant changes, such as the forced stay-at-home of children due to the closure of schools. This has meant drastic changes in the organization of daily life and restrictions on their activities, including exercise, which could affect the quality of life of the children due to its importance. In order to study the relationship between physical activity and psychological well-being of minors, a study has been carried out with Mixed Methods Research, combining survey methodology with transversal design with qualitative methodology using discourse analysis. A total of 234 parents of minors in Spain and several Spanish-speaking countries in America participated. The instrument was a questionnaire in Google Forms, which included the Kidscreen-27 quality of life scale. The results show significant differences in both the type of physical activity and its frequency due to age, and differences in parents’ perception of whether their children’s physical activity levels were sufficient or not, both on the health, mood and school subscales, and in the categorization of open responses referring to concerns due to the pandemic, analyzed with the ALCESTE technique. The relationship between physical activity of children and adolescents and quality of life is clearly concluded.

Introduction

At the end of December 2019, the first evidence appeared in Wuhan, China, that a new lethal viral disease had emerged, for which no vaccine or specific medication was available. In March the disease became a pandemic and a large majority of countries, either with specific regulations or through recommendations to the population, established confinement and social distance as the possible solution to prevent further spread of the disease, to avoid saturation of hospitals and curb the lethality of the virus. On March 10th, the global situation with regard to COVID-19 was 113,702 confirmed cases (4,125 new) and 4,012 deaths (203 new) ( World Health Organization [WHO], 2020a ). On December 29, the number of confirmed cases worldwide was to over 79 million, with a cumulative death toll of over 1.7 million ( World Health Organization [WHO], 2020b ).

In the field of Psychology, theoretical formulations have been made to explain the reasons why COVID-19 evolved so rapidly and was so widely spread. Urzúa et al. (2020) point out three factors: (a) illusory optimism; (b) inadequate perception of absence of contingencies produced by the population’s behavior; (c) optimistic risk perception. Vera-Villarroel (2020) has stated that physical and mental health are closely linked, and explains the expansion of the pandemic based on three psychological processes: cognitive, with the population having irrational beliefs about the disease and illusory optimism; emotional, with feelings such as fear, stress and anger; behavioral, with exposure and risk behaviors. The author points out that these factors must be considered in the intervention to save lives.

Several studies have shown the risk that social isolation caused by the pandemic implies not only for the most exposed groups (health workers), but also for the mental health of the general population. Problems of anxiety ( Chew et al., 2020 ; Holmes et al., 2020 ; Wang and Zhao, 2020 ), stress and psychological distress have been reported, both during and even after the biodisaster ( Liu D. et al., 2020 ). Along the same lines, the narrative review conducted by Huarcaya-Victoria (2020) points out three types of problems for the general population: health anxiety, depression and stress. Rajkumar (2020) groups the problems derived from the pandemic into four sections: general population, health workers, vulnerable people and therapeutic strategies and interventions. The author emphasizes the need to study the effect of the situation generated by the pandemic on children and adolescents.

A particularly vulnerable group in this whole situation is children and adolescents. Although results in children for Coronavirus-19 disease are still inconsistent. Changes produced in their environment since COVID-19, such as the restrictions that home isolation and not being able to access the main areas of socialization ( Socías et al., 2020 ), with risks such as stress from both them and their parents, since COVID-19 can cause psychological alterations in children such as those caused by other stressors ( Espada et al., 2020 ; Socías et al., 2020 ).

Certain factors can have effects not only during confinement but also afterward, such as the disappearance of healthy habits like attending classes, which have been replaced by unhealthy behaviors, such as sedentary lifestyles, inappropriate diets, excessive use of screens which can produce, in addition to weight gain, physical problems ( Brazendale et al., 2017 ; Wu et al., 2017 ). From this follows the importance of understanding the effects that a wide variety of personal and contextual factors ( Holgado-Tello et al., 2010 ) can have on children and adolescents and their interaction in the way they experience physical activity and sports during the pandemic situation. Other risks that have been highlighted, depending on age, include substance abuse, accommodation issues and overcrowding and change and disruption of social networks ( Holmes et al., 2020 ). It is expected that, after confinement, in most cases these problems will disappear ( Barlett et al., 2020 ), although some may persist after the situation generated by the pandemic has passed ( Espada et al., 2020 ). Space restrictions and not being able to go outside are especially important in childhood for the proper development of playing, which is essential for its maturation process ( García-Serrano and García-Fernández, 2015 ).

In view of the difficult situation experienced, the population has been provided with recommendations, some of which have been generated by institutions to support their citizens ( Socías et al., 2020 ). These guidelines have many points in common: maintaining routines, being active, supporting minors, carrying out social activities, in short, maintaining a normal life in safety ( Liu J. J. et al., 2020 ). The support of parents is important, who can strengthen family ties and meet the needs of children through appropriate parenting styles ( Wang et al., 2020 ). The need for physical exercise is also stressed ( Holmes et al., 2020 ; Mera-Mamián et al., 2020 ; Romero et al., 2020 ). Physical exercise plays a relevant role both on a physical level ( Vidarte Claros et al., 2011 ) and in mental processes ( Ramírez et al., 2004 ; Zhou et al., 2020 ) as well as on a psychological level ( Berger and Motl, 2000 ; Biddle and Mutrie, 2001 ; Tessier et al., 2007 ; Anderson and Brice, 2011 ). In particular, there is clear evidence of the contribution of physical activity to psychological well-being ( Molina-García et al., 2007 ; Jiménez et al., 2008 ; Romero et al., 2009 ; Fernández Ozcorta et al., 2015 ).

The relationship between physical activity and well-being linked to the quality of life has been the subject of multiple investigations in recent years, which have also emphasized its influence on the general health of the various sectors of the population ( Schwartzmann, 2003 ; Bize et al., 2007 ; Anokye et al., 2012 ). In particular, different studies have highlighted the association between high levels of physical activity, or the practice of sports, and the quality of life in children and adolescents ( Anokye et al., 2012 ; Marker et al., 2018 ; Luna et al., 2019 ).

Likewise, recent reviews of studies on interventions focused on the promotion of sports practices and their impact on issues such as mental health, self-esteem, anxiety levels, and perception of well-being in children and adolescents, underline the benefits of this kind of activities for the general health of this population in particular, showing that physical-sport education pilot programs might promoted significant improvements in specific indicators of subjective well-being and emotional intelligence of participating adolescents’ groups ( Bermejo-Cantarero et al., 2017 ; Luna et al., 2019 ).

The lack of physical activity is a widely reported public health problem ( Bermejo-Cantarero et al., 2017 ). For this reason, evaluation that focuses on the relationships between physical activity and health-related quality of life is an important focus of research in this field. On the other hand, there is little research aimed at exploring parents’ knowledge and perceptions of their children’s physical activity, their ideas about its importance and impact on the way they experience diverse dimensions of a stressful life ( Gallego-Méndez et al., 2020 ; Spinelli et al., 2020 ; Yarımkaya and Esentürk, 2020 ) particularly during the Coronavirus outbreak. Exploring these issues, including the different perspectives of persons involved in families’ life ( Izquierdo-Sotorrío et al., 2016 ), could help provide recommendations and support programs for parents to guide their children’s physical activity.

In the case of children and adolescents, physical activity has important benefits: it promotes growth and enhances both physical development ( Rosa et al., 2018 ) and psychomotor, cognitive and social development, and generally favors all body systems: metabolism of carbohydrates and lipids, control of blood pressure, decreases the risk of type 2 diabetes and improves body composition ( Camargo Lemos and Ortiz Dallos, 2010 ).

Physical activity also favors psychological factors: it helps to build a balanced self-concept and improves self-perception, mood, self-image, physical self-concept, perception of health and life satisfaction, and intellectual function ( Camargo Lemos and Ortiz Dallos, 2010 ; Reigal-Garrido et al., 2012 ; Rosa, 2015 ).

The home quarantine imposed by the COVID-19 may make physical activity more difficult, and as we have seen in the studies reviewed, this leads to a decline in the quality of life of children and adolescents. Quality of life (QoL) is understood as personal satisfaction (or dissatisfaction) with the cultural or intellectual conditions in which an individual lives. Health is one of the domains of quality of life, this domain comprises not only physical health but also psychological health, as well as the interaction that people have with others and with the community ( Ravens-Sieberer et al., 2005 ). For this research, we are interested in reviewing the quality of life, based on the assessment of the well-being perceived by parents.

Given that the collection of high quality data is a priority in order to understand the psychological effects that the quarantine may have produced in the population, and that there is an urgent need to discover, analyze and evaluate the psychological interventions that could alleviate the problems generated and minimize the risks that could occur in the mental health of society ( Holmes et al., 2020 ), the aim of this research is to analyze parents’ perceptions of their children’s quality of life in relation to observed physical activity in the conditions of staying in the housing due to the pandemic situation due to the COVID-19. It hypothesizes the existence of greater quality of life perceived by parents who consider their children to be sufficiently physically active.

In this sense, we try to find out if there is any difference in quality of life between children of different ages and sex in the conditions of staying in the housing due to the pandemic situation due to the COVID-19 as perceived by mothers and fathers. In addition, it is investigated whether the characteristics of the housing (the space) conditioned the perception of the parents about their children doing more or less physical activity, and whether there are differences between the age and the type of physical exercise done. It is also interesting to know the relationship between the level of physical activity and psychological well-being.

Materials and Methods

Methodology and design.

This is a non-experimental design. Mixed methodology was used (Mixed Methods Research, MMR; Johnson and Onwuegbuzie, 2004 ; Denscombe, 2008 ). The data was collected through a cross-sectional design with survey methodology, using an ex post facto design, and there are open questions that allow a qualitative analysis.

To determine the differences in physical activity, three independent variables were considered: age (children, adolescents), sex (male, female), as well as a third variable, grouping parents according to their opinion about the physical activity developed by their children in confinement (sufficient, insufficient). The dependent variables used has been the different scales that make up the KIDSCREEN test, which therefore requires multivariate analysis.

Participants

A total of 234 participants responded to the survey. The average age was 42.82 (SD = 7.10), with a range between 24 and 65. More mothers (203) than fathers (30) participated, and only one of the informants was guardian of the minors, relative in charge of the child. Table 1 presents the data regarding age (values corresponding to the percentiles 25, 50, 75, and over 75) and educational level. The procedure for selecting the sample was one of convenience.

Table 1. Parents’ age and educational level.

Parents and caregivers were asked to think of one of their children when answering the questionnaires. In this way, for the data analysis, they were grouped by the ages of the children, the largest group being children between 8 and 11 years old, 125 (52 female) and 109 adolescents between 12 and 17 (54 female).

The countries of origin of the participants were mainly Spain (134, 57.3%), and Mexico (86, 36.9%) and others American countries (Panamá, Colombia, Argentina, and Chile; 13, 5.8%).

Most families (230, 98.3%) reported not having been victims of the coronavirus. Only four families had a confirmed patient in the family unit, and in four other cases there was a suspicion that a family member had the disease.

In the questionnaire, a question was included about family and housing conditions. Most of the sample lived in the same dwelling with up to four family members (167, 71.4%), while it was less frequent for the family size to be greater than four (67, 28.6%). The average number of rooms, discounting common services, such as kitchen, living room and bathroom, was 3 (113 of the participants, 48.3%), with a range between 1 and 10 rooms. Most of the dwellings have at least one exterior space (177 of the participants, 75.6%).

Instruments

A questionnaire was designed to obtain data on parents’ perceptions of their children’s physical activity, some specific data on the type of housing during their child’s confinement. This questionnaire consists of 18 questions (15 closed, 3 open-ended) distributed in the following categories: (1) descriptive data of the participants (6 items); (2) family and housing conditions (5 items); (3) issues related to the situation produced by the COVID-19 pandemic (3 items); (4) complaints and needs caused by the situation produced by the COVID-19 pandemic (4 items) (see Supplementary Data Sheet 1 ). At the end of the questionnaire it was mentioned that if they wanted to ask for the results of the research they could leave their e-mail. All questions were marked as mandatory in the Google form, so there was no room for incomplete or missing data.

For the HRQoL measure, the Kidscreen-27 Parent Questionnaire ( Ravens-Sieberer et al., 2005 ). Spanish version was used, once the authorization for its use in this study was requested and obtained. This is a questionnaire that assesses health-related quality of life. This questionnaire was used because it provides a parameter to contrast the perception of psychological and health well-being in the child population with the physical activity observed by the parents. It consists of 27 items, which are answered in a Likert-type scale of five alternatives (from nothing to very much), structured in five scales: physical activity (4 items), mood (7 items), family life (7 items), friends (4 items), and school (4 items), and a single question about your child’s general state of health in the last week. The test is filled in by parents, for children and adolescents between the ages of 8 and 18. The original authors ( Ravens-Sieberer et al., 2005 ) offer evidence of the factorial validity of the test and its reliability in all the subscales of the test, in terms of internal consistency, with the total Cronbach’s Alpha value equal to 0.82. With our data, a similar Alpha of 0.831 has been obtained.

The questionnaires were assembled in electronic format with the Google Forms application. It was sent out by email and through social networks (Whatsapp, Facebook, and Twitter) to contacts in different educational associations, using the snowball technique. It was sent during the month of May 2020 (it can be defined as the first period of confinement). Only one of both parents was asked to answer the questionnaire with one of their children in mind (in case they have two or more), and who was in the age range of 8–17 years. The time required to fill in the questionnaire was 15 to 20 min.

At the time of data collection, all participants (regardless of country) were in the same conditions of confinement, leaving the home only for essential activities, with restrictions on going to school, physical activities or recreation outside the home.

As far as ethical aspects are concerned, the Commission on Ethics in Research and Animal Welfare of the University of La Laguna (CEIBA) was asked to authorize the study, which was granted (Registration Number: CEIBA2020-0396). In the questionnaire, the corresponding information for the participants was set out in the Organic Law 3/2018, of December 5th, on Personal Data Protection and guarantee of digital rights ( BOE, 2018 ), guaranteeing the anonymity and confidentiality of the data.

Data Analysis

The relationship between parental consideration of physical activity sufficiency and having or not having outdoor space in the home was calculated using the V of Cramer.

To check the absence of univariate outliers, we used Tukey’s test that takes as reference the difference in interquartile range, considering a slight outlier at 1.5 times this distance, and extreme when it is at three times that distance. To determine the existence of multivariate outliers, the Mahalanobis distance was calculated.

Regarding quality of life, it was analyzed in two ways taking three independent variables: age, sex and parents’ assessment. Since the quality of life variable, measured by Kidscreen, is split into several scales, it requires a multivariate approach, so three MANOVAs were carried out, one according to each independent variable studied. All quantitative analyses were conducted with the SPPS program, v.21.

For the qualitative analysis, the phenomenological discourse analysis method was used, which identifies the meanings of language, through lexical analysis using the ALCESTE software (in French: Analyse des Lexèmes Coocurrents dans les Enoncés Simples d’un Texte ) ( Reinert, 2003 ). This program facilitates the analysis of linguistic materials that generally arise in social research, such as answers to open-ended questions in questionnaires, in-depth interviews or answers based on projective techniques ( De Alba, 2004 ). The ALCESTE methodology consists of three stages: the construction of the data matrix, the classification of the context units (statements) and the description of the classes ( Gil et al., 1994 ). The methodology focuses on the statistical distribution of word succession, taking into account only the simultaneous presence of several words in the same statement. In this way, classes are identified as semantic fields, represented in trees or dendograms. In the ALCESTE method, the initial text is broken down into elementary contextual units (ECUs), which approximately match the size of a sentence.

The statistical analysis, although limited to explain in detail the meaning of a text, allows the elaboration of a “cartography” of the lexical worlds chosen by the speaker to express himself and, therefore, of the reference systems from which he constructs his way of seeing reality ( Gil et al., 1994 ; Reinert, 2003 ).

Quantitative Analysis

Physical activity.

In order to know if there is a relation between the participant’s perception of the sufficiency or not of the physical activity developed by his or her child and the space dedicated to exercises, these variables were analyzed, considering in the household conditions whether there was no outdoor space to carry out activities or if, on the contrary, there was. The results are shown in Table 2 . There is significant dependence between both variables (V of Cramer = 0.146; p = 0.026).

Table 2. Perception of adequacy of physical activity and space for it.

Elimination of Outsiders

Eleven extreme univariate cases were eliminated and none multivariate by Mahalanobis distance, with the criterion of probabilities less than 0.001.

Psychological Well-Being by Age and Sex

Most parents consider their child’s health to be excellent (88, 39.5%) or very good (114, 51.1%), while only 21 (9.4%) rate it as “fair.”

The group was divided into two ages: from 8 to 11 (children) and 12 and older (adolescents). Table 3 shows the descriptive statistics.

Table 3. Age and sex level descriptive statistics.

To know if there are differences by sex and age, a MANOVA was calculated, which was for sex (Wilk’s λ = 0.949, F 5 . 215 = 2.3, p = 0.046, Partial η 2 = 0.051), and for age (Wilk’s λ = 0.843, F 5 . 215 = 8.034, p = 0.001, Partial η 2 = 0.157) nor for interaction (Wilk’s λ = 0.982, F 5 . 215 = 0.796, p = 0.554, Partial η 2 = 0.018). Individual ANOVA results are only significant for the variable age in the health scale ( F 1 , 219 = 7.692, p = 0.006, Partial η 2 = 0.034), with a small effect size and in the friend one ( F 1 , 219 = 28.421, p < 0.001, Partial η 2 = 0.115), with a large effect size.

Physical Activity and Well-Being

In order to assess whether the children developed adequate physical activity, the parents were asked whether they considered it sufficient or insufficient. A total of 146 considered it to be insufficient and 77 sufficient. The informants were divided into two groups according to this variable and it was analyzed whether there were significant differences in their assessment of the psychological well-being of the children. Table 4 presents the mean values and standard deviations of each welfare scale.

Table 4. Descriptive statistics of physical activity and well-being.

The result of the MANOVA was significant (Wilk’s λ = 0.743, F 5 , 217 = 15.001, p < 0.001, Partial η 2 = 0.257). Individual ANOVA results are only significant for the health scale ( F 1 , 223 = 64.821, p < 0.001, Partial η 2 = 0.227), with a large effect size.

Qualitative Analysis

In order to find out the perceptions that families have regarding different aspects of stay-at-home confinement, both required by law and recommended, four open-ended questions were analyzed by ALCESTE, separating into two samples parents who considered that their children were getting enough exercise and those who thought it was insufficient: (a) Explain why you say you have sufficient or insufficient physical activity; (b) How did your child live it?; and (c) What or who does your child miss?

Analysis of the Question “Explain Why You Have Sufficient or Insufficient Physical Activity”

The analysis of ALCESTE, for the group of parents who consider that their children have sufficient physical activity (see Figure 1 ), the results are grouped into three classes, which explain 66% of textual units. The first class is linked to the link between the second and third classes. The most representative class is 1, as it groups the largest number of EUs. The details of the analysis, in terms of class name, UCEs grouped and percentage involved, most representative word and examples, are presented in Table 5 .

Figure 1. Dendogram corresponding to the question “Explain why you have sufficient physical activity.”

Table 5. Analysis of the question “Explain why you have sufficient or insufficient physical activity.”

The reasons given by parents for considering that their children could not get enough physical activity are more dispersed, as they have been grouped into six clases (see Figure 2 ). In this case, there are two groupings: on the one hand, class 2 connects with the union of classes 5 and 6, while class 1 connects with the link between classes 3 and 4. Classes 1, 5, and 6 are related to the impossibility of doing either exercise or sports that they did before the pandemic, while the difficulties of the other set of classes go in the direction of lack of space and the need to go outside.

Figure 2. Dendogram corresponding to the question “Explain why you have insufficient physical activity.”

Analysis of the Question “How Did Your Child Live Not Being Able to go Out on the Street?”

The analysis of the group that considers that their son or daughter has had enough activity explains 51% of the text corpus. The dendogram is shown in Figure 3 .

Figure 3. Dendogram of the question “How did your child live not being able to go outside?” Sufficient physical activity.

On the other hand, in the group of parents who consider the activity performed by their children insufficient, although it explains only 27% of the corpus, extracting only two classes ( Figure 4 ).

Figure 4. Dendogram of the question: “How did your child live not being able to go outside?” Insufficient physical activity.

Table 6 shows the detail of the classes, in terms of their name, number of UCEs they group, percentage of the corpus they explain and the most representative word, as well as representative examples of each class. In both groups, a distinction is made between positive aspects, of being at home, or pointing out some kind of problem.

Table 6. Information from the analysis to the question “How did your child live not being able to go out on the street?”

Analysis of the Question “What or Who Does Your Child Miss?”

The analysis of this question, for the group of parents who consider that the physical activity developed by their son or daughter is sufficient, gives two classes, which explain 65% of the textual units, that is, an average relevance of the treatment (see Figure 5 ). These are two antagonistic classes: the second is the one that groups the most textual units (71.70%), where it is clear that the child misses both the extended family and the people in his or her school environment. The first class includes those who responded that they have not missed anything or anyone and is quite homogeneous: they do not miss anyone (see Table 7 ).

Figure 5. Dendogram of the answers: What or who does your child miss? Sufficient physical activity.

Table 7. Information from the analysis to the question “What or who does your child miss?”

In the case of parents who feel that their son or daughter does not get enough physical activity, there are six classes, with a grouping of classes on a ladder: from class 1 to 4 are connected individually, linking class 5 with 6.

It explains 72% of the textual units, which means that the relevance of the treatment is high. They are presented in Figure 6 .

Figure 6. Dendogram of the answers: “What or who does your child miss?” Insufficient physical activity.

The first thing to note is that the data collection was done in the months of April and early May, at the time of the most severe confinement, as in Spain, Mexico, Panama, and Argentina ( BBC, 2020 ). It is true that the regulatory conditions regarding confinement have differed in the countries where the participants in this research live, in some cases, such as Spain, being obligatory by the State of Alarm, while in other countries governments strongly recommended avoiding going out and staying at home. This has meant 24-hour family life, with parents having to telework and children being taught online. The possibilities of exercising under these conditions have been very limited, which can have important effects on the psychological well-being of the minors.

As far as the health of their children is concerned, a large majority consider it to be good or very good. Furthermore, taking into account the five scales of quality of life in relation to age levels (children and adolescents), parents value the health of their children more the younger they are. In contrast, differences in contact with friends score higher for adolescents.

Parents’ perceptions of their children’s quality of life significant differences are observed with respect to sex at the global level, which is not maintained in the scales separately, but they do differ by age on two of the instrument’s scales: health, where younger children score higher, and friends, with the opposite result, as would be expected: adolescents score significantly higher on this scale.

Physical activity is conditioned by the type of housing. The results show that when there is no outdoor space to develop physical activity, parents find that exercise performed by their children is insufficient more often.

The objective of this study, to establish whether there is a relationship between physical activity and psychological quality of life in the conditions of confinement at home from the parents’ perspective, has been clearly corroborated, both in quantitative and qualitative analyses, finding differences between the two established groups of participants: those who considered that their children could develop sufficient physical exercise versus those who thought it was insufficient. Divergences are shown in both groups at the quantitative and qualitative levels.

With respect to the quality of life instrument, there are significant differences between the overall scores of the two groups; however, significant differences are only found in the health scale; when parents consider that the physical activity developed by their children is insufficient, lower scores are obtained in that scale. These results support the hypothesis of a positive relationship between quality of life and physical activity.

The differences found between the two groups of parents (those who consider their children’s physical activity sufficient and those who do not) in the quantitative analyses are also verified in the qualitative ones. The second group of parents shows more dispersion in the open responses given, as well as greater concern.

Thus, in the first open question analyzed qualitatively, “ Explain why you have sufficient or insufficient physical activity ,” the discourse of some parents differs significantly, as it is obvious, since the reasons they give for the physical activity done by their children being sufficient must be differentiated from those who consider it to be insufficient. In the latter, two perspectives are clearly distinguished in the two branches that appear in the dendogram: lack of space or impossibility of doing the exercise they would like to do. Moreover, it also confirms what has already been commented, that is, how there is a relationship between physical space and the facilities of households to exercise is related to the satisfaction or dissatisfaction with the physical activity performed by their children.

The second question, centered on their child’s experiences of not being able to go out, parents who feel their children have enough physical activity, report that their children experienced the lock down positively. On the other hand, in the other group there is a division of opinions: one part considers that their children lived the lock down without problems, but others think that their children lived it with stress, being this last one the most representative class. It confirms again a greater decline in the quality of life of their children for this group.

Finally, in the question relating to whether their child misses something or someone, there is greater variability among the children whose parents consider they do not have enough activity, since the answers are grouped together in one more class, where there is content where school life is missing.

The limitations of this work are about convenience samples, since there is no guarantee of absence of selection bias. However, having included several countries, all of them with a significant restriction on going out of the house, it gives indications of cross validity. This unusual development of the pandemic has evened out the differences between nations in a common struggle against an unprecedented biological crisis.

As far as the uncertainty of living under what has come to be called the new normality, together with the certainty that the threat of the pandemic is not over and that outbreaks, more or less virulent, may occur, it is particularly relevant to carry out research on mental health and psychological well-being, in order to be able to foresee more precisely the actions to be taken, knowing the dangers involved. Holmes et al. (2020) point out how important it is to accumulate experience based on the evidence that has provided the lessons learned so that those in power can coordinate measures that will damage the lives of citizens as little as possible, especially those who are most vulnerable. In this regard, since children are a vulnerable sector of the population, knowledge of their reactions and how they have been affected is particularly relevant. For future research, this could also include children’s self-report, comparing their perception with their mothers and fathers’s ( Izquierdo-Sotorrío et al., 2016 ). As a general recommendation in the light of the data collected, emphasizing the importance of exercise in guaranteeing the psychological well-being of minors is vital and must be conveyed to the population.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by The University of La Laguna’s Ethics Committee of Research and Animal Welfare has approved this research (Registration Number: CEIBA2020-0396). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

ÁB, GL-A, MV, and DC-S had participated in theoretical review. ÁB, ER-N, GL-A, DC-S, and MV had participated in research design and instrument. ÁB had participated in the data analysis. ÁB, ER-N, GL-A, DC-S, and MV had participated in discussion. ÁB, ER-N, GL-A, DC-S, MV, and TA had participated in the study planning, writing, and revision of the article. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors thank all the families that have participated in this research.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.649481/full#supplementary-material

Anderson, R. J., and Brice, S. (2011). The mood-enhancing benefits of exercise: memory biases augment the effect. Psychol. Sport Exerc. 12, 79–82. doi: 10.1016/j.psychsport.2010.08.003

CrossRef Full Text | Google Scholar

Anokye, N. K., Trueman, P., Green, C., Pavey, T. G., and Taylor, R. S. (2012). Physical activity and health related quality of life. BMC Public Health 12:624. doi: 10.1186/1471-2458-12-624

PubMed Abstract | CrossRef Full Text | Google Scholar

Barlett, J. D., Griffin, J., and Thomson, D. (2020). Resources for Supporting Children’s Emotional Well-Being During the COVID-19 Pandemic. Available online at: https://www.childtrends.org/publications/resources-for-supporting-childrens-emotional-well-being-during-the-covid-19-pandemic [Accessed March 19, 2020]

Google Scholar

BBC (2020). Coronavirus: Qué tan Rápidamente se Expande el Covid-19 Por América Latina (y Cómo se Compara con Otras Regiones Del Mundo) [Coronavirus: How Fast Covid-19 is Spreading in Latin America (and How it Compares to Other Regions of the World)]. Available online at: https://www.bbc.com/mundo/noticias-america-latina-51802906 [Accessed June 20, 2020]

Berger, B. G., and Motl, R. W. (2000). Exercise and mood: a selective review and synthesis of research employing the Profile of mood states. J. Appl. Sport Psychol. 12, 69–92. doi: 10.1080/10413200008404214

Bermejo-Cantarero, A., Álvarez-Bueno, C., Martínez-Vizcaino, V., García-Hermoso, A., Torres-Costoso, A. I., and Sánchez-López, M. (2017). Association between physical activity, sedentary behavior, and fitness with health related quality of life in healthy children and adolescents: a protocol for a systematic review and meta-analysis. Medicine 96:e6407. doi: 10.1097/MD.0000000000006407

Biddle, S. J. H., and Mutrie, N. (2001). Psychology of Physical Activity: Determinants, Well-Being and Interventions. London: Routledge.

Bize, R., Johnson, J. A., and Plotnikoff, R. C. (2007). Physical activity level and health-related quality of life in the general adult population: a systematic review. Prev. Med. 45, 401–415. doi: 10.1016/j.ypmed.2007.07.017

BOE (2018). BOE (2018, n° 294 of December 6th) Boletín Oficial del Estado. Available online at: https://www.boe.es/boe/dias/2018/12/06/pdfs/BOE-S-2018-294.pdf [Accessed June 20, 2020]

Brazendale, K., Beets, M. W., Weaver, R. G., Pate, R. R., Turner-McGrievy, G. M., Kaczynski, A. T., et al. (2017). Understanding differences between summer vs. school obesogenic behaviors of children: the structured days hypothesis. Int. J. Behav. Nutr. Phys. Act. 14:100.

Camargo Lemos, D. M., and Ortiz Dallos, C. J. (2010). Actividad física en niños y adolescentes: determinantes y medición. Rev. Univ. Ind. Santander. Salud. 42, 153–165.

Chew, Q. H., Wei, K. C., Vasoo, S., Chua, H. C., and Sim, K. (2020). Narrative synthesis of psychological and coping responses towards emerging infectious disease outbreaks in the general population: practical considerations for the COVID-19 pandemic. Singapore Med. J. 61, 1–31. doi: 10.11622/smedj.2020046

De Alba, M. (2004). El método ALCESTE y su aplicación al estudio de las representaciones sociales del espacio urbano: el caso de la ciudad de México. Papers Soc. Representations 13, 1.1–1.20.

Denscombe, M. (2008). A Research Paradigm for the Mixed Methods Approach. J. Mix. Methods Res. 2, 270–283. doi: 10.1177/1558689808316807

Espada, J. P., Orgilés, M., Piqueras, J. A., and Morales, A. (2020). Las buenas prácticas en la atención psicológica infanto-juvenil ante el COVID-19. Clín. Salud. 31, 109–113. doi: 10.5093/clysa2020a14

Fernández Ozcorta, E. J., Almagro Torres, B. J., and Sáenz-López Buñuel, P. (2015). Inteligencia emocional percibida y el bienestar psicológico de estudiantes universitarios en función del nivel de actividad física. Cult. Cienc. Deporte 10, 31–39. doi: 10.12800/ccd.v10i28.513

Gallego-Méndez, J., Pérez-Gómez, J., Calzada-Rodríguez, J. I., Denche-Zamorano, ÁM., Mendoza-Muñoz, M., Carlos-Vivas, J., et al. (2020). Relationship between health-related quality of life and physical activity in children with hyperactivity. Int. J. Environ. Res. Public Health 17:2804. doi: 10.3390/ijerph17082804

García-Serrano, P., and García-Fernández, J. A. (2015). “La rueda del hámster. Crítica a la especialización y confinamiento infantil en el espacio público urbano,” in Proceedings of the 2nd Conference on Urban Processes. Facultad de CC Políticas y Sociología , (Madrid: Universidad Complutense de Madrid).

Gil, J., García, E., and Rodríguez, E. (1994). Una metodología para el análisis estadístico de datos textuales: el programa Alceste. Rev. Investig. Educ. 24, 107–124.

Holgado-Tello, F., Navas, L., López, M., and García, T. (2010). A structural model of goal orientation in sports: personal and contextual variables. Span. J. Psychol. 13, 257–266. doi: 10.1017/S1138741600003838

Holmes, E. A., O’Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., et al. (2020). Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry 7, 547–560. doi: 10.1016/S2215-0366(20)30168-1

Huarcaya-Victoria, J. (2020). Consideraciones sobre la salud mental en la pandemia de COVID-19. Rev. Peru. Med. Exp. Salud Pública 37, 327–334. doi: 10.17843/rpmesp.2020.372.5419

Izquierdo-Sotorrío, E., Holgado-Tello, F. P., and Carrasco, M. Á (2016). Incremental validity and informant effect from a multi-method perspective: assessing relations between parental acceptance and children’s behavioral problems. Front. Psychol. 7:664. doi: 10.3389/fpsyg.2016.00664

Jiménez, M. G., Martínez, P., Miró, E., and Sánchez, A. I. (2008). Bienestar psicológico y hábitos saludables: ¿están asociados a la práctica de ejercicio físico? Inter. J. Clin. Health Psychol. 8, 185–202.

Johnson, R. B., and Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educ. Res. 33, 14–26. doi: 10.3102/0013189X033007014

Liu, D., Ren, Y., Yan, F., Li, Y., Xu, X., Yu, X., et al. (2020). Psychological impact and predisposing factors of the coronavirus disease 2019 (COVID-19) pandemic on general public in China. Lancet Psychiatry doi: 10.2139/ssrn.3551415 [Epub ahead of print].

Liu, J. J., Bao, Y., Huang, X., Shi, J., and Lu, L. (2020). Mental health considerations for children quarantined because of COVID-19. Lancet Child Adolesc. Health 4, 347–349.

Luna, P., Guerrero, J., and Cejudo, J. (2019). Improving adolescents’ subjective well-being, trait emotional intelligence and social anxiety through a programme based on the sport education model. Int. J. Environ. Res. Public Health 16:1821. doi: 10.3390/ijerph16101821

Marker, A. M., Steele, R. G., and Noser, A. E. (2018). Physical activity and health-related quality of life in children and adolescents: a systematic review and meta-analysis. Health Psychol. 37, 893–903. doi: 10.1037/hea0000653

Mera-Mamián, A. Y., Tabares González, E., Montoya-González, S., Muñoz Rodríguez, D., and Monsalve, F. (2020). Recomendaciones prácticas para evitar el desacondicionamiento físico durante el confinamiento por la pandemia asociada a COVID-19. Universidad Salud. 22, 166–177. doi: 10.22267/rus.202202.188

Molina-García, J., Castillo, I., and Pablos, C. (2007). Bienestar psicológico y práctica deportiva en universitarios. Motricidad. Eur. J. Hum. Mov. 18, 79–91.

Rajkumar, R. P. (2020). COVID-19 and mental health: a review of the existing literature. Asian J. Psychiatry 52:102066. doi: 10.1016/j.ajp.2020.102066

Ramírez, R., Vinaccia, S., and Suárez, G. R. (2004). El impacto de la actividad física y el deporte sobre la salud, la cognición, la socialización y el rendimiento académico: una revisión teórica. Rev. Estudios Soc. 18, 67–75. doi: 10.7440/res18.2004.06

Ravens-Sieberer, U., Gosch, A., Rajmil, L., Erhart, M., Bruil, J., Dür, W., et al. (2005). KIDSCREEN-52 Medida de la calidad de vida de los niños y adolescentes. Rev. Exp. Farmacoeconomía Resultados Invest. 5, 353–364.

Reigal-Garrido, R., Videra García, A., Parra Flores, J. L., and Juárez Ruiz de Mier, R. (2012). Actividad Físico Deportiva, Autoconcepto Físico y Bienestar Psicológico en la Adolescencia. Retos: Nuevas tendencias en Educación Física, Deporte y Recreación. 22, 19–23.

Reinert, M. (2003). Alceste Users’ Manual.(English Version). Touluse: Image.

Romero, A., García-Mas, A., and Brustad, R. (2009). Estado del arte y perspectiva actual del concepto de bienestar psicológico en psicología del deporte. Rev. Latinoam. Psicol. 41, 335–347.

Romero, C. C., del Mazo Fuente, A., Besada, M. D., and Hernández, M. D. L. O. R. (2020). Algunas aportaciones de la orientación educativa en red durante el confinamiento por la pandemia de COVID-19. Rev. AOSMA, N° Extra 28, 33–49. Available online at: https://dialnet.unirioja.es/descarga/articulo/7381631.pdf

Rosa, A. (2015). Actividad Física y Autoconcepto: Una Revisión Teórica Aplicada al Ámbito Escolar. Available online at: https://www.efdeportes.com/efd202/actividad-fisica-y-autoconcepto.htm (accessed December 31, 2020).

Rosa, A., García-Cantó, E., and Carrillo, P. J. (2018). Percepción de salud, actividad física y condición física en escolares. Cuad. Psicol. Deport. 18, 179–189.

Schwartzmann, L. (2003). Calidad de vida relacionada con la salud: aspectos conceptuales. Cienc. Enferm. 9, 09–21. doi: 10.4067/S0717-95532003000200002

Socías, C. O., Ballester Brage, L. L., and Nevot-Caldentey, L. L. (2020). Apoyo Familiar Ante el Covid-19 en España [Family Support Against Covid-19 in Spain]. Available online at: https://doi.org/10.1590/SciELOPreprints.297 [Accessed March 3, 2020].

Spinelli, M., Lionetti, F., Pastore, M., and Fasolo, M. (2020). Parents’ stress and children’s psychological problems in families facing the COVID-19 outbreak in Italy. Front. Psychol. 11:1713. doi: 10.3389/fpsyg.2020.01713

Tessier, S., Vuillemin, A., Bertrais, S., Boini, S., Le Bihan, E., Oppert, J. M., et al. (2007). Association between leisure-time physical activity and health-related quality of life changes over time. Prev. Med. 44, 202–208. doi: 10.1016/j.ypmed.2006.11.012

Urzúa, A., Vera-Villarroel, P., Caqueo-Urízar, A., and Polanco-Carrasco, R. (2020). La Psicología en la prevención y manejo del COVID-19. Aportes desde la evidencia inicial. Terapia Psicol. 38, 103–118.

Vera-Villarroel, P. (2020). Psicología y covid-19: un análisis desde los procesos psicológicos básicos. Panam. J. Neuropsychol. 14, 10–18.

Vidarte Claros, J. A., Vélez, C., Sandoval, C., and Alfonso, M. L. (2011). Actividad física: estrategia de promoción de la salud. Rev. Hacia Promoción Salud. 16, 202–218.

Wang, C., and Zhao, H. (2020). The impact of COVID-19 on anxiety in Chinese university students. Front. Psychol. 11:1168. doi: 10.3389/fpsyg.2020.01168

Wang, G., Zhang, Y., Zhao, J., Zhang, F., and Jiang, F. (2020). Mitigate the effects of home confinement on children during the COVID-19 outbreak. Lancet 395, 945–947. doi: 10.1016/S0140-6736(20)30547-X

World Health Organization [WHO] (2020a). Coronavirus Disease 2019 (COVID-19) Situation Report – 50. Available online at: https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200310-sitrep-50-covid-19.pdf?sfvrsn=55e904fb_2 . [Accessed Mars 10, 2020]

World Health Organization [WHO] (2020b). Weekly Epidemiological Update - 29 December 2020. Available online at: https://www.who.int/publications/m/item/weekly-epidemiological-update—29-december-2020 [Accessed December 31, 2020]

Wu, X. Y., Han, L. H., Zhang, J. H., Luo, S., Hu, J. W., and Sun, K. (2017). The influence of physical activity, sedentary behavior on health-related quality of life among the general population of children and adolescents: a systematic review. PLoS One 12:e0187668. doi: 10.1371/journal.pone.0187668

Yarımkaya, E., and Esentürk, O. K. (2020). Promoting physical activity for children with autism spectrum disorders during Coronavirus outbreak: benefits, strategies, and examples. Int. J. Dev. Disabil. 1–6. doi: 10.1080/20473869.2020.1756115

Zhou, F., Xi, X., and Qin, C. H. (2020). Regular open-skill exercise generally enhances attentional resources related to perceptual processing in young males. Front. Psychol. 11:941. doi: 10.3389/fpsyg.2020.00941

Keywords : home lock down, physical activity, quality of life, pandemic, mixed methods research

Citation: López-Aymes G, Valadez MD, Rodríguez-Naveiras E, Castellanos-Simons D, Aguirre T and Borges Á (2021) A Mixed Methods Research Study of Parental Perception of Physical Activity and Quality of Life of Children Under Home Lock Down in the COVID-19 Pandemic. Front. Psychol. 12:649481. doi: 10.3389/fpsyg.2021.649481

Received: 04 January 2021; Accepted: 16 February 2021; Published: 15 March 2021.

Reviewed by:

Copyright © 2021 López-Aymes, Valadez, Rodríguez-Naveiras, Castellanos-Simons, Aguirre and Borges. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Elena Rodríguez-Naveiras, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

70k Accesses

30 Citations

12 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 11 April 2024

“Being an informal caregiver – strengthening resources”: mixed methods evaluation of a psychoeducational intervention supporting informal caregivers in palliative care

• Tabea Theißen   ORCID: orcid.org/0000-0002-0277-2544 1   na1 ,
• Anneke Ullrich   ORCID: orcid.org/0000-0002-1759-4461 1   na1 ,
• Karin Oechsle   ORCID: orcid.org/0000-0002-5183-641X 1 ,
• Julia Wikert   ORCID: orcid.org/0000-0002-7622-9923 1 , 2 ,
• Carsten Bokemeyer   ORCID: orcid.org/0000-0001-6071-7810 1 &
• Aneta Schieferdecker   ORCID: orcid.org/0009-0008-7354-753X 1  

BMC Palliative Care volume  23 , Article number:  95 ( 2024 ) Cite this article

97 Accesses

1 Altmetric

Metrics details

Informal caregivers are key support for patients with progressive incurable diseases. However, their own needs often remain unmet. Therefore, we developed, manualised and implemented the intervention “Being an informal caregiver – strengthening resources” aiming to support and empower informal caregivers by addressing relevant information-related, physical, psychological and social needs.

In this pilot study, we evaluated the acceptance and experiences with this psychoeducational intervention. The study was conducted over two years (2019-2021). Informal caregivers were recruited from the University Medical Centre Hamburg-Eppendorf and the metropolitan region of Hamburg, Germany. The intervention was aimed at adult persons who identified themselves as an informal caregiver to an adult patient with a progressive incurable cancer and non-cancer disease. For the evaluation we used a mixed methods approach, combining a longitudinal questionnaire survey (pre-intervention, after each module, 3-months follow-up) and semi-structured interviews post-intervention. Quantitative data were analysed using descriptive statistics and a paired t -Test, interviews were analysed based on the qualitative content analysis according to Mayring. Results were triangulated using a convergent triangulation design.

Of 31 informal caregivers who received the intervention, 25 returned the follow-up questionnaire and 20 informal caregivers were interviewed. Triangulated results showed a high satisfaction with the implementation of the intervention. Of a broad range of subjective benefits, gaining knowledge, self-awareness and self-efficacy were most apparent. Informal caregivers reported improved preparedness, awareness of own needs as well as confidence regarding handling own emotions and interacting with the ill person. However, implementing the learned skills into daily life can be challenging due to internal and external factors. Motivations and challenges for participating as well as potential for improvement were identified.

Conclusions

This pilot study showed an overall positive evaluation and several subjective benefits of the psychoeducational intervention “Being an informal caregiver – strengthening resources”. Further research is needed to measure the efficacy of this intervention on informal caregivers’ outcomes. Therefore, a multicentre randomized prospective study is planned.

Peer Review reports

Introduction

Informal caregivers (ICs) are key support for patients with progressive incurable cancer and non-cancer diseases [ 1 , 2 , 3 ]. At the same time, ICs are affected by the patients' disease with own burden and needs: Studies demonstrate various psychological, social, physical and economic burden as well as health system challenges of these ICs [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ]. ICs psychosocial burden may even exceed that of the patients [ 4 , 11 , 12 ] and increases with the patients’ disease progression and nearing death [ 12 , 13 , 14 , 15 , 16 ]. This is reflected in a high prevalence of anxiety (32% - 47%) and depression (20% - 41%) in ICs [ 5 , 6 , 17 , 18 , 19 ]. Unmet needs are associated with significantly higher levels of ICs psychological and overall caregiver burden [ 20 , 21 , 22 ]. Most common unmet needs are informational needs, especially those related to the patients’ disease and its treatment as well as care issues [ 23 , 24 ]. Furthermore, ICs physical, psychological, financial, spiritual and social unmet needs are well described [ 23 ]. Being an IC means enormous responsibility and many ICs feel unprepared at the practical and the emotional levels of caring for a patient with progressive incurable disease [ 25 ]. Empirical evidence shows that feeling prepared is associated with decreased IC burden [ 25 , 26 , 27 ]. Therefore, it is paramount to improve ICs knowledge and competence to facilitate preparedness, support role recognition and increase confidence in ICs [ 28 ].

Increasingly, international studies assess interventions for ICs aiming at addressing their burden and needs during palliative care [ 1 , 29 ]. Previous interventions were structured in different ways (psychoeducational interventions, behavioural or skills trainings and palliative care team interventions), delivered at group or individual level, and focused different target groups (ICs, IC-patient dyads or patients plus their families) [ 1 ]. Some studies have already shown that interventions can have positive effects on self-efficacy and preparedness [ 30 , 31 , 32 , 33 ] as well as on psychological burden [ 29 , 34 , 35 ] in ICs. However, many studies are pilot or feasibility studies and results are inconclusive [ 1 , 29 ]. Meta-analyses show mixed results regarding ICs psychological burden, including distress, anxiety and depression, and quality of life [ 36 , 37 , 38 ]. Focus on heterogeneous populations, heterogeneous interventions contents and implementations as well as numerous or unspecified primary end points make it difficult to draw conclusions or to recommend one particular intervention [ 1 ].

In Germany, evaluated interventions for ICs in palliative care are limited. The 22-hours existential behavioural therapy of Fegg et al. demonstrated significant effects on anxiety, depression and quality of life [ 39 ]. However, the 2-hours short-term therapy showed no significant impact on ICs outcome [ 40 ]. The educational initiative “Last Aid Course” (LAC) by Georg Bollig aims to teach the public about the fundamentals of palliative care. It has been implemented in 20 different countries, including Germany [ 41 ]. LAC can lead to the promotion of death literacy in communities and is feasible in a web-based format [ 42 , 43 ]. To date, available data on the effects of LAC on ICs are preliminary, especially in Germany [ 44 , 45 ].

Because of this gap in evaluated interventions for ICs of patients with progressive incurable cancer and non-cancer diseases in Germany, we developed, manualised and implemented the psychoeducational intervention “Being an informal caregiver – strengthening resources” and conducted a mixed methods pilot study. The aim of this study was to evaluate ICs acceptance of and experiences with the intervention, including participation, motivation and for participating, satisfaction with the implementation, subjective benefits and potential for improvement.

Intervention

Design of the intervention.

The aim of the newly developed psychoeducational intervention “Being an informal caregiver – strengthening resources” was to support and empower ICs of patients with incurable progressive diseases by increasing their knowledge, self-awareness and self-efficacy. Self-awareness is the ability to focus on oneself and how the own actions, thoughts or emotions align with internal standards [ 46 ] and self-efficacy is the belief in one’s ability to confidently deal with a specific situation [ 47 ]. We wanted to develop an intervention that would be as efficient as possible, but also practical in many places. At the same time, it has to be appealing and helpful for many and ICs should be able, to share their experiences with others in a group setting.

The intervention setting was an ongoing group program, including six multidisciplinary modules (90 min. duration each; table 1 ). Each week on the same day (Monday 5pm), one of the six modules was offered. ICs could join the program at any time and in any order as well as individually decide which modules to attend. The contents of the modules covered the physical, psychological, social and spiritual dimensions of palliative care. While most of them are directly visible in the topics of the modules (e.g. physical in module 1 and 5, table 1 ), others can be found all over the modules. For instance, spirituality: seeing and validating the ICs as individuals, encouraging and supporting them to approach emotions such as grief, loss and fears as well as finding and using resources. The modules were offered by palliative care professionals with expertise in different fields (physician, nurse, grief counsellor, social worker, physiotherapist, psychologist). To ensure the intervention fidelity, weekly implementation of the modules was accompanied and moderated by one member of the research team (psychologist).

The intervention was offered from November 2019 until November 2021. Initially, it was conducted in a face-to-face format. Due to the COVID-19 pandemic, it was paused from end of March until June 2020. During that time, the intervention and evaluation components were adjusted into a web-based format. From July 2020, the intervention was offered in a web-based format as videoconference

Development of the intervention

The intervention was based on the curriculum for developing training programs in palliative care (KoMPaC) by the German Association for Palliative Medicine [ 48 ]. Additionally, experts of the study team reviewed IC-directed interventions in the context of palliative care. Palliative care experts were involved in the planning process to increase practical relevance of the intervention. In the end, six standardised evidence-based modules were developed. For each module, learning objectives, contents, and materials, including presentations and handouts, were developed and manualised. Handouts for ICs included further recommendations, like information about support services and recommended reading. The materials were iteratively revised through feedback from palliative care experts during the development plus throughout the study, as we asked participating ICs and palliative care experts for feedback, including those who delivered the intervention. There were regular discussions and exchanges between the palliative care experts and the moderator. We used the TIDieR (Template for Intervention Description and Replication) Checklist (Suppl. File 1 ) to report on the intervention [ 49 ].

Study design

This prospective interventional mixed methods study was conducted between November 2019 and November 2021. It combines a quantitative study part, including a longitudinal questionnaire survey at baseline (pre-intervention), after each module and at follow-up three months after baseline as well as a qualitative study part with semi-structured interviews post- intervention. As shown in Fig.  1 , the six modules were offered as an ongoing program, one module per week. After six weeks, the program started from the beginning, as illustrated by the two arrows forming a cycle. ICs could join the intervention at any time and in any order.

The mixed methods design approach was applied to triangulate the study, as mixed methods designs have been increasingly used in palliative care research to enrich findings [ 50 ]. Quantifying the ICs experience solely by questionnaires might be weak in understanding reasons behind their experiences with the intervention. Therefore, qualitative research was used for a deeper insight into the ICs subjective experience. To combine the approaches, we used a convergent triangulation method. The convergent triangulation design is a mixed methods design in which quantitative and qualitative data are collected and analysed separately and then results from both databases are compared and merged for interpretation [ 51 ].

The ethics committee of the Medical Association in Hamburg, Germany approved the study (June 04, 2019; Reference: PV7009).

Participants

The intervention was addressed to ICs of patients with a progressive incurable disease. All individuals who identified themselves as a primary IC could participate. ICs are characterised as people providing unpaid care to someone with whom they have a personal relationship with (family, friend or another close person) [ 52 ]. We did not specify further criteria of informal caregiving (like duration, kind or effort of taking care). ICs were recruited from the University Medical Centre Hamburg-Eppendorf and the metropolitan region of Hamburg, Germany, via information material (flyers and posters). We used a wide approach to reach as many ICs as possible (e.g. medical practices, pharmacies, supermarkets). Interested ICs were asked to contact the study team, who informed about the study and checked eligibility. Inclusion criteria were: being an adult primary IC (≥18 years) to an adult patient with a progressive incurable disease and cognitive capacity for giving fully informed consent and completing the questionnaires. Written informed consent was obtained from all participating ICs. Within the informed consent, ICs were asked, if they would agree to be contacted for the interview later on. All ICs, who participated in an interview, gave informed consent.

Data collection

Procedures of data collection.

ICs, who registered for the intervention, received the baseline questionnaire (pre-intervention) and the 3-months follow-up questionnaire by mail. In addition, they received a questionnaire directly after each module. In the face-to-face format, questionnaires were handed out and filled in on-site. After web-based modules, ICs received a link for the online survey.

Participants for the interviews were selected using a purposive sampling (criteria: number of participated modules: at least 5 vs. less, format of participation: web-based vs. face-to-face). Two female researchers (AS: physician, TT: psychologist), who were not known by the participants, interviewed the ICs via telephone. Interviews were planned to last 45 min., were audio-recorded with permission and transcribed verbatim. Besides, field notes were made during the interviews.

Measurements

Sociodemographic and medical variables.

ICs informed about their age, gender, current occupation, school education, relationship to the patient, as well as the patient’s disease, place of residence and care level.

Process data

Data regarding the intervention format, module and date of participation were prospectively documented.

Satisfaction with the implementation of the intervention

The Health Education Impact Questionnaire program scale (heiQ-program) was used to assess the quality of the intervention implementation [ 53 ]. It includes 9 items, each rated on a 6-point Likert scale (0 “strongly disagree” to 5 “strongly agree”). The heiQ-program was reported on a single item level. Reliability and validity have been supported for the German Version [ 54 ]. The internal consistency of the heiQ-program in this study was satisfying (Cronbach’s alpha .89; original heiQ .08 or higher). Using the heiQ format, we developed two additional items, asking ICs whether their expectations had been met and whether the subject matter provided was clear and easy to understand. The two additional items were the result of testing the questionnaire with two ICs during the development phase of the intervention in order to check its completeness and comprehensibility. The heiQ-program plus the two additional items were used at follow-up as well as after each module.

Subjective benefits

Due to the lack of suitable instruments, we used eight self-developed items to investigate ICs perceived benefits of participating. Items were rated in accordance with the heiQ-program (0 “strongly disagree” to 5 “strongly agree”). For example, ICs indicated whether they have more answers to important questions, know where to find support as an IC and feel more confident. These items were used at follow-up as well as after each module. Additionally, ICs rated how helpful the intervention was (scale: 1 “not helpful at all” to 10 “extremely helpful”) at follow-up.

Caregiver burden

ICs subjective burden of care was evaluated using the validated German short version of the Burden Scale for Family Caregivers (BSFC-s) [ 55 ]. The 10 items are rated on a 4-point Likert scale (0 “strongly disagree” to 3 “strongly agree”). The score ranges from 0 to 30, with higher scores indicating greater burden (low: 0-5, medium: 6-14, high 15-30). The BSFC-s was used at baseline and follow-up. Cronbach’s alpha in this study was satisfying (baseline .90; follow-up .86; original BSFC-s .92 [ 55 ]).

Interview guide

A semi-structured interview guide (Supplement File 2 ) was used for deeper insight into the ICs experience with the intervention. Topics are displayed in Table 2. . The guide was developed based on current literature as well as practical clinical knowledge, following the approach by Helfferich [ 56 ]. During implementation, no need for further modification of the guide emerged.

Data analysis

Descriptive statistics were calculated including the frequencies, percentages, means ( M ) and standard deviations ( SD ). The heiQ-program plus the two additional items were analysed on a single item level with ratings of 4 (“agree”) and 5 (“strongly agree”) being considered as satisfaction with the intervention. IC burden was analysed using the mean total score of the BSFC-s. The authors of the BSFC-s used EM algorithm for imputing missing values [ 57 ]. Due to our small sample size, we refrained from that and decided on replacing the mean score on an individual level [ 58 ]. Therefore, scores for respondents with one or two missing items were calculated by imputing the mean score for the missing items based on items completed by that individual. A paired t -Test was conducted to investigate changes in ICs burden before and after the intervention. All conditions required to conduct a t-test were met [ 59 ]. All statistical analyses were computed using IBM SPSS Statistics version 27.0 [ 60 ] and a significance level of α < 0.05.

Interviews were iteratively analysed based on the qualitative content analysis according to Mayring [ 61 ], using the software program MAXQDA (2020). The analysis took place using inductive coding techniques. A preliminary coding template was developed based on the line by line coding of a subset of transcripts (50% of the data material) by one researcher (AS). Categories and subcategories were regularly discussed in a series of meetings by the multi-professional research team. As no new (sub-) categories emerged after the tenth interview, all interviews were coded using the final coding template in accordance to Mayring [ 61 ]. To ensure intra- and interrater reliability, two researchers (TT, AS) each coded the data of five transcripts independently. The comparison showed high agreement between the two researchers; differences were resolved in discussions within the research team (TT, AS, AU).

Convergent triangulation design

In accordance to Creswell and Clark [ 51 ], quantitative and qualitative data were collected and analysed parallel. Afterwards the two sets of results were merged and jointly interpreted in a narrative discussion. Throughout a series of meetings (TT, AS, AU, KO), data was compared and contrasted. We looked for common concepts across the results and compared quantitative and qualitative results side-by-side for each concept to identify in what ways they confirm, disconfirm or expand each other. We used the guidelines for Good Reporting of A Mixed Methods Study (GRAMMS) Checklist [ 62 ] (Supplement File 3 ).

Characteristics of participants and participation

Of 36 ICs registered for the intervention, five dropped out before baseline. In one case, the patient died before the IC started the intervention; in a further four cases, the reasons for dropout remain unknown, as the ICs could not be contacted despite several attempts. The remaining 31 ICs completed the baseline questionnaire and were included in the intervention, with 25 ICs returning the follow-up questionnaire. Therefore, six ICs were lost to follow-up for reasons unknown. Afterwards, 20 of the 31 ICs were interviewed (duration in min.: M =31.8, SD =12.1, 15-62). The mean age of participating ICs ( N =31) was 51.4 years and 90% ( n =28) of ICs were female. In most cases, the person they cared for was living at home ( n =25, 81%), was their spouse ( n =20, 65%) and had an underlying cancer disease (n=21, 68%; Table 3 ).

Table 4 summarizes the characteristics of participation. Of 31 ICs, 14 (45%) completed the intervention (=at least 5/6 modules). However, this is more a description of the use of the intervention than a quality indicator, as ICs were able to select modules according to their needs. The average number of attended modules was 3.6 out of 6 ( SD =2.0, range 1-6). Due to the pandemic, 19 (61%) of the ICs participated web-based. The average duration of participation was 15.5 weeks ( SD =4.5, range 1-27). In relative terms, module 2 ( Getting prepared: Information about social and legal issues ) and module 5 ( Strategies for handling changes in the disease progression ) were attended most commonly, by 21 ICs (68%) each.

Triangulated results of the evaluation

In the following, triangulated results are reported. The triangulation showed that in some parts qualitative analysis gave results in greater detail, in other parts quantitative analysis was more informative and sometimes both parts were complementary. Qualitative findings are displayed in Fig.  2 . In supplement file 4 , the detailed category system is provided and explanations and illustrative quotes for each subcategory have been added. Quantitative data are summarized in Tables 5 , 6 and 7 , data on module level in Supplement File 5 . Tables and figures are shown where the corresponding results are reported.

Category system of the qualitative evaluation. Abbreviations: ICs, informal caregivers; a dual role = being a person who supports and cares for the ill person at the same time as being a person with own needs

Intention of participating

In the qualitative data, ICs reported expectations and motivations for participating in the intervention (Fig.  2 ). One motivational factor was the explicit focus of the intervention on ICs, as their role as ICs, associated burden and needs are often overlooked:

“Everyone naturally cares about the ill person and asks about him, how he is going, but no one asks about how I am.” (IC 04)

ICs also expected to receive information about specific topics, e.g. care issues, legal aspects and support services. They believed that an increased knowledge on these topics could help them being more prepared and confident. For example, one IC wanted to acquire information about what she might face in future:

„Information, gaining knowledge (...) I just wanted to know: what to expect and who to ask if I don't know what to do.” (IC 24)

Likewise, ICs expected to learn how to deal with potentially burdensome topics (e.g. grief), own emotions and fears, as well as finding a better way of understanding and dealing with the ill person. They hoped to strengthen their resources by improving self-care and self-efficacy. Furthermore, ICs participated in the intervention to get in contact with others concerned, to share experiences and feel less isolated.

The evaluation of the heiQ-program showed medium to high satisfaction with the implementation of the intervention (Table 5 ). Overall, 8 of the 9 original heiQ-program items were rated as “agree” by at least 76% of ICs. Satisfaction was particularly visible for module 1 ( Hands-on care: tips and strategies for providing care at home ), for detailed data see Supplement file 5 . At follow-up, all ICs ( n =25, 100%) reported that they have trusted the given information and felt that the subject matter provided was clear and easy to understand. Interview data refined that ICs trusted the given information as they had trust in the institution delivering the intervention:

„A program at an institution that I consider trustworthy, namely a university hospital (...), and I have the opportunity to inform myself about the broad spectrum of issues, that's a great chance for me.” (IC 24)

On average, ICs rated the whole intervention as very helpful ( M =8.2, SD =1.6, range: 3-10 on a scale of 1 “not helpful at all” – 10 “extremely helpful”; data not shown). Overall, 92% ( n =23) would recommend the intervention to other ICs.

ICs described feeling encouraged by the palliative care experts delivering the intervention, as besides receiving new information, they felt validated in their prior knowledge and experiences. They also reported gaining confidence over the course of the intervention, through the way the course was implemented and the way the experts act.

“ At the beginning I was skeptical, maybe a little bit afraid, because of the emotional issues, but in the course it was okay, I felt really well taken care of.” (IC11)

Subjective benefits of the intervention

Changes on cognitive, emotional and behavioural level.

Qualitative data revealed that subjective benefits of the intervention can be described on three levels: cognitive, emotional and behavioural.

On the cognitive level, beneficial changes due to the intervention included increase in knowledge, self-reflection and self-awareness:

„Due to the intervention, I realised that I should take care of myself more often.“ (IC 04)

On the emotional level, beneficial changes included a sense of encouragement (e.g. to start therapy), strength, gratitude, feeling less guilty (e.g. when doing self-care activities), and better acceptance of one’s own emotionality:

„(…) to gain the feeling that it is okay to feel a certain way and also to accept oneself with one's own fears and worries.“ (IC 33)

Further, the intervention had led to goal directed behaviour and concrete actions in ICs everyday life. For instance, including more self-care into daily life, significant changes in dealing with the ill person and use of support services (e.g. psychotherapist, palliative care specialist):

„We immediately looked for a local palliative care physician, here where we live (...) we go there every four weeks.“ (IC 24)

Topics of subjective benefits

Several perceived benefits of participating in the intervention could be identified in quantitative (Table 6 ) and qualitative data (Fig.  2 ). Quantitative results appeared similarly across all modules (for detailed data see Supplement File 6 ).

Gaining knowledge

Quantitative analysis demonstrated a subjective benefit of the intervention regarding the ICs knowing more about support services for ICs ( n =18, 75%) and having more answers to important questions ( n =17, 71%; Table 6 ). Qualitative data support these findings, as ICs reported that the intervention helped them through information about topics such as legal aspects, care issues, disease progression and medication, and support services. Thus, ICs felt better prepared and relieved:

„I learned (...) who I can contact and also how it works when he dies, who will take care of him and that the funeral home will do a lot. That was a real benefit for me.“ (IC 05)

Having more knowledge was also described as a good starting point for discussions with the ill person as well as with others concerned. Realising the principles and goals of palliative care led to relief and more acceptance of help, as one participant reported:

“I realized (…) that palliative care does not just start when the end of life is very near, but even before that (...) that was a very important realisation for me.“ (IC 28)

Self-awareness and -efficacy

Quantitative data showed that the intervention affected the ICs self-awareness and –efficacy (Table 5 ). Over half of the ICs reported on being more aware of their own needs ( n =15, 65%) and feeling more confident in handling their own feelings ( n =15, 63%). ICs also indicated to feel more confident interacting with the ill person ( n =11, 48%) and to be better at knowing ( n =9, 38%) and assessing ( n =11, 46%) their own capacities. Qualitative data revealed insights into how self-awareness and self-efficacy were improved. For example, the intervention had helped ICs reflecting upon finitude of life and their role as an IC, and thus to accept the situation and accompanying emotions. ICs reported being more aware of their own needs and the importance of self-care:

„I learned (...) that I can work on myself (...). When I have a problem or fears I look at what does me good, (..) and that I do gymnastic exercises without having a bad conscience.“ (IC 18)

They felt recognized and valued in their unique burdens and needs as ICs:

„It was definitely very useful and valuable for me to know (...) that there are people who think about us, the ICs. That not only patients are in such a difficult situation, but also we, the ICs.“  (IC 31)

Furthermore, ICs reported positive changes in dealing with the ill person by feeling more confident in communicating feelings:

„What I remember best is dealing with my own fears, worries and needs, because that is what has burdened me most. (…) I gained more self-confidence in dialogue with my wife to talk about these things.” (IC 28)

Potential burden

Despite the sensitive topics of the intervention, in quantitative data, most of the ICs ( n =19, 83%) reported no further emotional distress due to participating. Sharing experiences with other persons in similar situations was described as very helpful (e.g. sense of common):

„The feeling of not being alone (…), [to know] this happens to many people (…), that alone has helped me. (…) To release a little pressure (...) to see that other people are not doing well either when they see their loved ones will die.” (IC 16)

Subjective burden of informal caregiving

Though most ICs reported subjective benefits, there was no significant change in the ICs burden as measured by the BSFC-s ( t (21)=-1.65, p =.115, N =22; data not shown). At baseline, ICs on average showed a high burden of caregiving ( M =16.4, SD =6.8, range 4-30). Three months later, the mean score dropped to a medium degree of burden ( M =14.1, SD =7.4, range 4-29).

Implementing the learned skills into daily life

Supporting factors.

The most supporting factors (Table 7 ) for implementation to daily life were the ICs themselves and their motivation ( n =23, 92%), other affected persons ( n =12, 48%), information from TV, books or brochures ( n =11, 46%), physicians or other healthcare professionals and the ill person ( n =10, 42%). Thus, the ill person can be a hindering and supporting factor, which is also seen in the interviews. For example, one IC described improvements in her mother’s health as helpful for application of the newly acquired knowledge:

„(…) my mother is getting fitter, which makes everything that I've learned a bit easier. Bit by bit, the overload (...) becomes less, which gives room for (...) for example self-care, making the implementation of the learned (...) easier.“ (IC 07)

In addition to the quantitative data, qualitative data revealed intervention-related factors to support implementation to daily life. Practical contents with specific tips and tutorials as well as additionally provided materials and recommendations were reported as particularly helpful, as one IC stated:

„The contents were simply so practical, from my point of view it was immediately usable.“ (IC 34)

Hindering factors

At 3-months follow-up, internal and external factors were identified that hindered ICs implementing what they have learned into daily life (Table 8 ). Most ICs ( n =15, 60%) stated “lack of time, stress ” as a barrier, followed by their impaired own well-being ( n =13, 54%), their job ( n =13, 52%) and the ill person ( n =10, 42%). These factors aligned with those reported in the interviews, with qualitative data illuminating the underlying mechanisms. For example, one IC described facing resistance as she wanted to talk to her partner about the power of attorney and the living will:

„It couldn’t work out to start a conversation with my husband. (...) He always gives me the answer: I don't want to think about it.” (IC 18)

Moreover, ICs named their own habits and lack of motivation as hindering factors, both in quantitative ( n =10, 40%; n =3, 12%) and qualitative data:

„My own laziness, my own weaker self, and that I still don't have a living will although I was completely enthusiastic about it.“ (IC 02)

Challenges of participating

Qualitative data revealed relevant challenges of participating in the intervention (Fig.  2 ). There were structural challenges like technical problems and lack of time, due to work, childcare or taking care of the ill person, as this quote illustrates:

„ I worked in a full-time job, was in the hospital every day, and (...) took care of his cat. The time pressure (...) made it difficult for me.“ (IC 05)

Further, ICs described the group heterogeneity as demanding as they are facing different issues and questions due to various ages, relationships to the ill person and therefore different times in their life in general (e.g. newly married). Thereof, diverging needs and burdens could be derived. For instance, in one session most participants were older and looked back on a long-lasting relationship with the ill person. One younger IC reported facing different issues than the other participants, because of her specific stage of life:

„We were faced with questions about starting a family and how to go on with life planning, and the older women, (…) the nature of accompaniment is different, after you've spent your whole life together.” (IC 33)

Despite this challenge, the IC was still able to identify herself with the other participants in other aspects:

“There were definitely individual comments from the other participants that made me think: yes, that's exactly how I've felt too.” (IC 33)

Before starting the intervention, ICs recognized emotional challenges stemming from their uncertainty how they would deal with particular topics, from questioning their ability to seek and accept help, and from concerns about emotions possibly evoked by the intervention. As one IC narrated:

„For me it was a challenge to go there, because all this is very emotional and I knew I would most likely cry a lot.“ (IC 11)

Despite such concerns, most of the ICs (n=19, 83%; Table 5 ) reported no further emotional distress after participating in the intervention.

Potential for improvement

In the interviews, ICs were asked on how the intervention could be improved. While some ICs perceived the contents and the information density of the individual modules as sufficient, others requested adjustments, as illustrated by the following quotes:

„What I missed a bit (...) [is the topic] how the ill person behaves and how I deal with it.“  (IC 18) „Sometimes [it was] too much content for too little time.” (IC 07)

ICs heterogeneous needs also became apparent regarding the exchange among the participants. Some desired more opportunity for sharing experiences, for instance by closed group meetings and greater groups, as one participant stated:

„Personally, I think it would be (...) good to have a closed group, to get to know each other and to learn more from the others, and thereby benefit.“ (IC 22)

At the same time, others appreciated the focus of the intervention on providing information and transferring of knowledge, as they explicitly did not seek for a support group.

Other named potentials for improvement were having more opportunities to ask questions and getting more practical tips, like checklists or exercises. Furthermore, ICs suggestions for improvement referred to the timeframe: different time of day, different day of the week or rhythm (e.g. every 2 weeks).

Regarding getting information about support services like this intervention, ICs wished to be informed by the professional caregivers:

„I would have liked a nurse or a physician to talk to me personally about this [the intervention], so that you could feel (...) individually addressed .“ (IC 21)

Apart from the indications for possible improvements, other ICs stated to see no potential improvement:

"I really don't know what could be improved." (IC 11)

Moreover, ICs showed appreciation and gratefulness for the intervention and experienced the acquired knowledge as lasting:

"What I find wonderful is that it is so sustainable. It is not as if you participated (...) and forgot about it, I still benefit from it. (...) It [the intervention] has really been like a gift for me." (IC 32)

In this study, we evaluated a newly developed psychoeducational intervention named “Being an informal caregiver – strengthening resources”. The intervention aims to support and empower ICs of patients with incurable progressive cancer and non-cancer diseases by addressing relevant information-related, as well as physical, social and psychological needs. Main findings relate to the overall positive evaluation of the intervention with high satisfaction regarding its implementation, as well as various subjective benefits on the cognitive, emotional and behavioural level. The most apparent benefits were gaining knowledge, self-awareness, which can be seen in an increased focus on and awareness of their own needs as ICs, as well as an increased self-efficacy, as the ICs felt more confident in handling their situation. It was evident that the ICs felt seen and validated as a result of the intervention. Nevertheless, several factors were identified, which could support or hinder the implementation of the learned skills into daily life. Furthermore, motivations and challenges for participating were identified and opportunities for improving the intervention were discovered.

The ICs knowledge gain is reflected in the qualitative and quantitative data in many facets, for example in relation to care issues, disease progression, medication, legal aspects and support services, and therefore represents a major benefit of the intervention. This gain of knowledge in turn led to a feeling of preparedness and relief. It is well known that preparedness is not only needed with regard to practical aspects of caregiving, but also on the emotional level, since ICs have to cope with a broad range of emotions – from burden during caregiving to loss of the loved one [ 25 ]. To address these needs, we implemented the specific modules own needs and emotions , self-care and own physical well-being as well as grief and loss . ICs reported that the intervention had helped them reflecting on their role as an IC, the difficult situation they are confronted with as well as being more aware of their own needs and feeling more confident in dealing with the ill person. These benefits are reflected in changes on cognitive, emotional and on behavioural levels. Changes in the ICs everyday life, regarding themselves (e.g. increased self-care) as well as their interaction with the ill person (e.g. starting difficult conversations) can be found. Despite ICs difficulties of acknowledging and disclosing their own needs and accepting help [ 28 ], our psychoeducational intervention seems to succeed in its key goals. To increase the ICs preparedness, self-care and resilience can have a positive effect on ICs outcomes like caregiver burden [ 25 , 26 , 27 , 63 ]. Psychoeducational, psychotherapeutic and mindfulness-based approaches might have a greater impact on ICs outcomes than other approaches [ 38 ]. However, further research is needed measuring the efficacy of our intervention on ICs outcomes, like psychological burden and quality of life.

Despite potentially burdensome topics of the intervention, ICs reported no additional emotional distress caused by intervention participation. On contrary, due to the positive experience, ICs felt more confident during the intervention. ICs had trust in information, considered the content clear and easy to understand, and difficult topics handled well. The fact that each module was provided by a palliative care professional experienced in this specific field as well as the professional, trustworthy environment in which the intervention was offered (university medical centre) might support this.

A major challenge of our study was the inclusion of ICs. In this case, the circumstances of the COVID-19 pandemic could be responsible, but low recruitment is a well-known problem in interventions supporting ICs and is reflected in numerous studies, also in the pre-pandemic era [ 24 , 64 ]. In our study, most common barriers were lack of time and taking care of the ill person. These were also identified as hindering factors for implementing the learned skills into daily life. ICs feel highly responsible for the patients, put them in centre of care [ 65 , 66 ] and feel uncomfortable leaving them even though they desire to take a break [ 28 ]. Our module-based intervention with short sessions and the opportunity to attend modules as required and possible could be a promising way to face this problem. Furthermore, web-based interventions might be useful for the overburdened population of ICs. Studies show that a web-based format of intervention is feasible [ 43 ], but comparisons between web-based and face-to-face interventions are missing. In our study, some ICs reported that they were only able to participate because the intervention was web-based, while others stated that they lacked personal contact and would prefer a face-to-face format. In both formats, ICs named potential challenges (web-based: e.g. technology, face-to-face: e.g. mobility and time), indicating a broad range of individual constraints, capabilities and needs among this heterogeneous group. Our study was not designed to answer this question and due to the small sample size, we are unable to provide any details or conclusion on the distinction between face-to-face and web-based participating. Future research on the acceptance of different intervention formats and their effects on ICs outcomes is required.

Furthermore, the group heterogeneity was described as challenging. In our study, ICs named the different ages and relationship to the ill person in the group as demanding, as it might have an impact on their individual needs and burdens. The heterogeneity is also shown in diverging expectations of participating, as some IC were specifically looking for exchange with other persons concerned, whereas others were seeking information and not sharing personal experiences. Due to the diversified group setting, heterogeneity is inevitable. For future implementations, this could be addressed at the beginning of a module, to emphasise the unifying aspect of being an IC and simultaneously the individuality of each participant. The high proportion of ICs of patients with a cancer disease might be an indication that ICs of non-cancer diseases did not feel addressed. When advertising the intervention, special attention should be paid to the interventions’ contents and aims, as well as to whom it is addressed.

From this study as well as other studies [ 24 , 64 ] it is apparent, that there is a gap between ICs needs and their use of support services. Thus, further research is needed on overcoming this gap and improving participation. Our results show that IC wish to be informed about support services by their professional caregiver. For clinical practice this could mean that all professional caregivers - whether physicians, nurses or psychologists - should be aware of ICs needs, available support services and interventions, and actively offer them to their patients and ICs.

Strengths and limitations

Our study has several strengths and limitations. One strength is the broad approach to ICs through the public recruitment strategy, the inclusion of ICs from both cancer and non-cancer patients and the possibility of early participation, not just when death is imminent or palliative care has already been implemented. Another advantage of our intervention is that it is offered in an open group setting, which allows for wider dissemination by reaching more ICs, in contrast to interventions that are designed as one-to-one contact by visit or telephone. Further, the intervention was manualised, which facilitates the implementation in other settings. By combining quantitative and qualitative methods to evaluate the intervention, we gained deeper insight into the ICs experiences of participating.

However, the generalizability of the results is limited due to the monocentric design of our study. Despite the broad recruitment approach, we only achieved a small sample size. We do not know which ICs did not participate to participate and for what reasons. It may also be that those ICs, who did participate, were in relatively good health and did not have too much caring work at that time. In addition, due to possible selection bias (mostly female, working and cancer patient ICs), our results may be less applicable to male ICs, those not working/retired and non-cancer patient ICs. Furthermore, there is no knowledge about the sustainability of the gained subjective benefits, as the evaluation ended with the 3-months follow-up. Transferability of the intervention might also be challenging, as the implementation requires experts from various professional groups, which might be difficult for smaller medical centres. However, the experts for the intervention do not have to be part of the same team or organization, which may be an advantage in terms of expert recruitment. In order to reduce resource requirements, the intervention could be offered three to four times a year, for example, instead of a continuous weekly program.

Strengthening ICs is important because they feel unprepared and the tasks and role as an IC entail a variety of burdens. This mixed methods pilot study shows an overall positive evaluation of the intervention “Being an informal caregiver – strengthening resources”, in terms of both its content and its implementation. Important goals pursued by a psychoeducational intervention, like improvement of knowledge, self-awareness and self-efficacy, were achieved, with positive changes reported at cognitive, emotional and behavioural levels. This manualised intervention has the potential to have a positive impact on ICs. A multicentre randomized prospective study is planned to measure the efficacy of the intervention on ICs outcomes.

Nevertheless, it was a major challenge to include ICs in our intervention, which is a common problem in interventions supporting ICs of patients with progressive incurable diseases. It is particularly important to develop strategies to effectively disseminate available support services and increase participation, including that of non-cancer ICs. In our study, ICs emphasized the key role of professional caregivers in informing about support services. Therefore, professionals have to be equipped with relevant information about support services and about when to refer ICs to such services, also in non-cancer settings. Furthermore, new intervention formats, such as web-based formats, might lower the threshold for participation in this burdened population. Future research of our study group aims to examine the acceptance of different intervention formats and their effects on participation.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due to reasons of privacy but are available from the corresponding author (TT) on reasonable request.

Abbreviations

Informal caregiver

Coronavirus Disease 2019

Last Aid Course

Curriculum for developing training programs in palliative care

Template for Intervention Description and Replication Checklist

Health Education Impact Questionnaire program scale

Burden Scale for Family Caregivers

standard deviation

Good Reporting of A Mixed Methods Study Checklist

Alam S, Hannon B, Zimmermann C. Palliative Care for Family Caregivers. J Clin Oncol. 2020;38(9):926–36.

Article   PubMed   Google Scholar  

Kitko L, McIlvennan CK, Bidwell JT, Dionne-Odom JN, Dunlay SM, Lewis LM, et al. Family caregiving for individuals with heart failure: a scientific statement from the american heart association. Circulation. 2020;141(22):e864–78.

D'Alvano G, Buonanno D, Passaniti C, De Stefano M, Lavorgna L, Tedeschi G, et al. Support needs and interventions for family caregivers of patients with Amyotrophic Lateral Sclerosis (ALS): A Narrative review with report of telemedicine experiences at the time of COVID-19 pandemic. Brain Sci. 2021;12(1):49.

Nipp RD, El-Jawahri A, Fishbein JN, Gallagher ER, Stagl JM, Park ER, et al. Factors associated with depression and anxiety symptoms in family caregivers of patients with incurable cancer. Ann Oncol. 2016;27(8):1607–12.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ullrich A, Ascherfeld L, Marx G, Bokemeyer C, Bergelt C, Oechsle K. Quality of life, psychological burden, needs, and satisfaction during specialized inpatient palliative care in family caregivers of advanced cancer patients. BMC Palliat Care. 2017;16(1):31.

Article   PubMed   PubMed Central   Google Scholar  

Oechsle K, Ullrich A, Marx G, Benze G, Heine J, Dickel LM, et al. Psychological burden in family caregivers of patients with advanced cancer at initiation of specialist inpatient palliative care. BMC Palliat Care. 2019;18(1):102.

Chong E, Crowe L, Mentor K, Pandanaboyana S, Sharp L. Systematic review of caregiver burden, unmet needs and quality-of-life among informal caregivers of patients with pancreatic cancer. Support Care Cancer. 2022;31(1):74.

Zhang Y, Zhang S, Liu C, Chen X, Ding Y, Guan C, Hu X. Caregiver burden among family caregivers of patients with advanced cancer in a palliative context: a mixed-method study. J Clin Nurs. 2023;32(21–22):7751–64.

Schutz SE, Walthall HE. What are the needs and experiences of caregivers of people with heart failure? A qualitative study. Heart Lung. 2022;54:42–8.

Article   CAS   PubMed   Google Scholar  

Schischlevskij P, Cordts I, Gunther R, Stolte B, Zeller D, Schroter C, et al. Informal caregiving in Amyotrophic Lateral Sclerosis (ALS): A high caregiver burden and drastic consequences on caregivers' lives. Brain Sci. 2021;11(6):748.

Braun M, Mikulincer M, Rydall A, Walsh A, Rodin G. Hidden morbidity in cancer: spouse caregivers. J Clin Oncol. 2007;25(30):4829–34.

Grunfeld E, Coyle D, Whelan T, Clinch J, Reyno L, Earle CC, et al. Family caregiver burden: results of a longitudinal study of breast cancer patients and their principal caregivers. CMAJ. 2004;170(12):1795–801.

Northouse LL, Mood DW, Montie JE, Sandler HM, Forman JD, Hussain M, et al. Living with prostate cancer: patients’ and spouses’ psychosocial status and quality of life. J Clin Oncol. 2007;25(27):4171–7.

Kim Y, Given BA. Quality of life of family caregivers of cancer survivors: across the trajectory of the illness. Cancer. 2008;112(11 Suppl):2556–68.

Burridge LH, Barnett AG, Clavarino AM. The impact of perceived stage of cancer on carers’ anxiety and depression during the patients’ final year of life. Psychooncology. 2009;18(6):615–23.

Tang ST, Chang WC, Chen JS, Wang HM, Shen WC, Li CY, Liao YC. Course and predictors of depressive symptoms among family caregivers of terminally ill cancer patients until their death. Psychooncology. 2013;22(6):1312–8.

Hudson PL, Thomas K, Trauer T, Remedios C, Clarke D. Psychological and social profile of family caregivers on commencement of palliative care. J Pain Symptom Manage. 2011;41(3):522–34.

Trevino KM, Prigerson HG, Maciejewski PK. Advanced cancer caregiving as a risk for major depressive episodes and generalized anxiety disorder. Psychooncology. 2018;27(1):243–9.

Gotze H, Brahler E, Gansera L, Schnabel A, Gottschalk-Fleischer A, Kohler N. Anxiety, depression and quality of life in family caregivers of palliative cancer patients during home care and after the patient’s death. Eur J Cancer Care (Engl). 2018;27(2):e12606.

Hwang SS, Chang VT, Alejandro Y, Osenenko P, Davis C, Cogswell J, et al. Caregiver unmet needs, burden, and satisfaction in symptomatic advanced cancer patients at a Veterans Affairs (VA) medical center. Palliat Support Care. 2003;1(4):319–29.

Sharpe L, Butow P, Smith C, McConnell D, Clarke S. The relationship between available support, unmet needs and caregiver burden in patients with advanced cancer and their carers. Psychooncology. 2005;14(2):102–14.

Schrank B, Ebert-Vogel A, Amering M, Masel EK, Neubauer M, Watzke H, et al. Gender differences in caregiver burden and its determinants in family members of terminally ill cancer patients. Psychooncology. 2016;25(7):808–14.

Wang T, Molassiotis A, Chung BPM, Tan JY. Unmet care needs of advanced cancer patients and their informal caregivers: a systematic review. BMC Palliat Care. 2018;17(1):96.

Ullrich A, Marx G, Bergelt C, Benze G, Zhang Y, Wowretzko F, et al. Supportive care needs and service use during palliative care in family caregivers of patients with advanced cancer: a prospective longitudinal study. Support Care Cancer. 2021;29(3):1303–15.

Bilgin A, Ozdemir L. Interventions to improve the preparedness to care for family caregivers of cancer patients: a systematic review and meta-analysis. Cancer Nurs. 2022;45(3):E689–705.

Karabulutlu EY, Turan GB, Yanmis S. Evaluation of care burden and preparedness of caregivers who provide care to palliative care patients. Palliat Support Care. 2022;20(1):30–7.

Kim Y, Carver CS, Spiegel D, Mitchell HR, Cannady RS. Role of family caregivers’ self-perceived preparedness for the death of the cancer patient in long-term adjustment to bereavement. Psychooncology. 2017;26(4):484–92.

Harrop E, Byrne A, Nelson A. “It’s alright to ask for help”: findings from a qualitative study exploring the information and support needs of family carers at the end of life. BMC Palliat Care. 2014;13:22.

Alshakhs S, Park T, McDarby M, Reid MC, Czaja S, Adelman R, et al. Interventions for family caregivers of patients receiving palliative/hospice care at home: a scoping review. J Palliat Med. 2023;27(1):112–27.

Havyer RD, van Ryn M, Wilson PM, Griffin JM. The effect of routine training on the self-efficacy of informal caregivers of colorectal cancer patients. Support Care Cancer. 2017;25(4):1071–7.

Hendrix CC, Bailey DE Jr, Steinhauser KE, Olsen MK, Stechuchak KM, Lowman SG, et al. Effects of enhanced caregiver training program on cancer caregiver’s self-efficacy, preparedness, and psychological well-being. Support Care Cancer. 2016;24(1):327–36.

Holm M, Arestedt K, Carlander I, Furst CJ, Wengstrom Y, Ohlen J, Alvariza A. Short-term and long-term effects of a psycho-educational group intervention for family caregivers in palliative home care - results from a randomized control trial. Psychooncology. 2016;25(7):795–802.

Hudson P, Trauer T, Kelly B, O’Connor M, Thomas K, Summers M, et al. Reducing the psychological distress of family caregivers of home-based palliative care patients: short-term effects from a randomised controlled trial. Psychooncology. 2013;22(9):1987–93.

McMillan SC, Small BJ. Using the COPE intervention for family caregivers to improve symptoms of hospice homecare patients: a clinical trial. Oncol Nurs Forum. 2007;34(2):313–21.

Dionne-Odom JN, Azuero A, Lyons KD, Hull JG, Tosteson T, Li Z, et al. Benefits of Early Versus Delayed Palliative Care to Informal Family Caregivers of Patients With Advanced Cancer: Outcomes From the ENABLE III Randomized Controlled Trial. J Clin Oncol. 2015;33(13):1446–52.

Candy B, Jones L, Drake R, Leurent B, King M. Interventions for supporting informal caregivers of patients in the terminal phase of a disease. Cochrane Database Syst Rev. 2011;6:CD007617.

Google Scholar  

Dhumal T, Siddiqui ZA, Kelley GA, Harper F, Kelly KM. Systematic review and meta-analysis of randomized controlled trials of interventions addressing caregiver distress and burden among cancer caregivers. PEC Innov. 2023;2:100145.

Chow R, Mathews JJ, Cheng EY, Lo S, Wong J, Alam S, et al. Interventions to improve outcomes for caregivers of patients with advanced cancer: a meta-analysis. J Natl Cancer Inst. 2023;115(8):896–908.

Fegg MJ, Brandstatter M, Kogler M, Hauke G, Rechenberg-Winter P, Fensterer V, et al. Existential behavioural therapy for informal caregivers of palliative patients: a randomised controlled trial. Psychooncology. 2013;22(9):2079–86.

Kuhnel MB, Marchioro L, Deffner V, Bausewein C, Seidl H, Siebert S, Fegg M. How short is too short? A randomised controlled trial evaluating short-term existential behavioural therapy for informal caregivers of palliative patients. Palliat Med. 2020;34(6):806–16.

Bollig G, Brandt F, Ciurlionis M, Knopf B. Last aid course. An education for all citizens and an ingredient of compassionate communities. Healthcare (Basel). 2019;7(1):19.

Macaden L, Broadfoot K, Carolan C, Muirhead K, Neylon S, Keen J. Last aid training online: Participants' and facilitators' perceptions from a mixed-methods study in rural Scotland. Healthcare (Basel). 2022;10(5):918.

Bollig G, Meyer S, Knopf B, Schmidt M, Hayes Bauer E. First Experiences with online last aid courses for public palliative care education during the COVID-19 pandemic. Healthcare (Basel). 2021;9(2):172.

Zelko E, Bollig G. Report from the Third International Last Aid Conference: cultural diversity in palliative care. Palliat Care Soc Pract. 2023;17:26323524231166932.

PubMed   PubMed Central   Google Scholar  

Giehl C, Chikhradze N, Bollig G, Vollmar HC, Otte I. "I needed to know, no matter what i do, I won't make it worse"-eExpectations and experiences of last aid course participants in Germany-A qualitative pilot study. Healthcare (Basel). 2023;11(4):592.

Wicklund RA, Duval S. Opinion change and performance facilitation as a result of objective self-awareness. J Exp Soc Psychol. 1971;7(3):319–42.

Article   Google Scholar  

Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.

Becker D, Kamp T. Kompetenzbasierte berufsgruppenunabhängige Matrix zur Erstellung von Curricula für die Weiterbildung curricularer Bildungsinhalte in Palliative Care/Palliativmedizin (KoMPaC). Bonn: Pallia Med; 2017.

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

Fabregues S, Hong QN, Escalante-Barrios EL, Guetterman TC, Meneses J, Fetters MD. A methodological review of mixed methods research in palliative and end-of-life care (2014-2019). Int J Environ Res Public Health. 2020;17(11):3853.

Creswell JW, Clark VPL. Designing and Conducting Mixed Methods Research. London: SAGE Publications, Inc; 2017.

Roth DL, Fredman L, Haley WE. Informal caregiving and its impact on health: a reappraisal from population-based studies. Gerontologist. 2015;55(2):309–19.

Osborne RH, Elsworth GR, Whitfield K. The Health Education Impact Questionnaire (heiQ): an outcomes and evaluation measure for patient education and self-management interventions for people with chronic conditions. Patient Educ Couns. 2007;66(2):192–201.

Schuler M, Musekamp G, Faller H, Ehlebracht-Konig I, Gutenbrunner C, Kirchhof R, et al. Assessment of proximal outcomes of self-management programs: translation and psychometric evaluation of a German version of the Health Education Impact Questionnaire (heiQ). Qual Life Res. 2013;22(6):1391–403.

Graessel E, Berth H, Lichte T, Grau H. Subjective caregiver burden: validity of the 10-item short version of the Burden Scale for Family Caregivers BSFC-s. BMC Geriatr. 2014;14:23.

Helfferich C. Leitfaden- und Experteninterviews. In: Bauer N, Blasius J, editors. Methoden der empirischen Sozialforschung Wiesbaden: Springer VS; 2014. p. 559-74.

Pendergrass A, Malnis C, Graf U, Engel S, Graessel E. Screening for caregivers at risk: Extended validation of the short version of the Burden Scale for Family Caregivers (BSFC-s) with a valid classification system for caregivers caring for an older person at home. BMC Health Serv Res. 2018;18(1):229.

Wirtz M. On the problem of missing data: How to identify and reduce the impact of missing data on findings of data analysis. Rehabilitation (Stuttg). 2004;43(2):109–15.

CAS   PubMed   Google Scholar  

Bortz J. Statistik für Human- und Sozialwissenschaftler. Heidelberg: Springer; 2010. p. 117–36.

Book   Google Scholar  

IBM Corp. Released 2020. IBM SPSS statistics for windows, version 27.0. Armonk: IBM Corp.

Mayring P. Qualitative Inhaltsanalyse: Grundlagen und Techniken. Weinheim: Beltz; 2022.

O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services Research. J Health Serv Res Policy. 2008;13(2):92–8.

van Roij J, Brom L, Sommeijer D, van de Poll-Franse L, Raijmakers N, eQuiPe study g. Self-care, resilience, and caregiver burden in relatives of patients with advanced cancer: results from the eQuiPe study. Support Care Cancer. 2021;29(12):7975-84.

Dionne-Odom JN, Applebaum AJ, Ornstein KA, Azuero A, Warren PP, Taylor RA, et al. Participation and interest in support services among family caregivers of older adults with cancer. Psychooncology. 2018;27(3):969–76.

Totman J, Pistrang N, Smith S, Hennessey S, Martin J. “You only have one chance to get it right”: A qualitative study of relatives’ experiences of caring at home for a family member with terminal cancer. Palliat Med. 2015;29(6):496–507.

Harrop E, Morgan F, Byrne A, Nelson A. “It still haunts me whether we did the right thing”: a qualitative analysis of free text survey data on the bereavement experiences and support needs of family caregivers. BMC Palliat Care. 2016;15(1):92.

Download references

Acknowledgements

We thank all participating ICs, without whom this study would not have been possible. Further, we gratefully acknowledge the staff of the Palliative Care Unit, Department of Oncology, Haematology and BMT, University Medical Centre Hamburg-Eppendorf for their partnership in research and for their dedicated work. We especially thank all palliative care experts participating in the development and implementation of the intervention and Miriam Albrecht, our student assistant in this study.

Open Access funding enabled and organized by Projekt DEAL. This study received funding by the “Förderverein der Palliativversorgung am UKE e.V” within the Full Professorship for Palliative Care with Focus on Informal Caregivers Research. The funders of the study had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Tabea Theißen and Anneke Ullrich contributed equally to this manuscript and share first authorship.

Authors and Affiliations

Palliative Care Unit, Department of Oncology, Haematology and BMT, University Medical Centre Hamburg-Eppendorf, Martinistr. 52, 20246, Hamburg, Germany

Tabea Theißen, Anneke Ullrich, Karin Oechsle, Julia Wikert, Carsten Bokemeyer & Aneta Schieferdecker

Department of Palliative Medicine, LMU University Hospital, Munich, Germany

Julia Wikert

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: AU, KO. Methodology: AU, KO. Validation: AU. Formal analysis: TT, AS, AU. Investigation: TT, JW, AS, KO, AU. Resources: KO, CB. Data Curation: TT, AU, JW, AS. Writing – Original Draft: TT, AU, AS. Writing – Review & Editing: TT, AU, KO, JW, CB, AS. Visualization: TT. Supervision: AU, KO. Project administration: TT, JW. Funding Acquisition: KO, CB.

Corresponding author

Correspondence to Tabea Theißen .

Ethics declarations

Ethics approval and consent to participate.

The ethics committee of the General Medical Council of Hamburg, Germany, approved the study protocol (reference number PV7009).

All participants provided written informed consent before taking part in the study, including study participation, data analysis and publication.

The study was conducted in accordance with the Helsinki Declaration of 1975 as revised in 2008.

Consent for publication

Not applicable

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., supplementary material 5., supplementary material 6., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Theißen, T., Ullrich, A., Oechsle, K. et al. “Being an informal caregiver – strengthening resources”: mixed methods evaluation of a psychoeducational intervention supporting informal caregivers in palliative care. BMC Palliat Care 23 , 95 (2024). https://doi.org/10.1186/s12904-024-01428-0

Download citation

Received : 06 December 2023

Accepted : 04 April 2024

Published : 11 April 2024

DOI : https://doi.org/10.1186/s12904-024-01428-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Informal caregivers
• Palliative care
• Psychoeducational intervention
• Pilot study
• Empowerment

BMC Palliative Care

ISSN: 1472-684X

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 16 April 2024

Client satisfaction and contributing factors towards sexual and reproductive health services delivery system among youth at Family Guidance Association of north Ethiopia (FGAE) clinics, 2023: mixed method study

• Niguss Cherie 1 ,
• Yawkal Tsega 2 ,
• Anissa Mohammed 3 ,
• Zinet Abegaz 1 ,
• Abel Endawkie 3 ,
• Yeshimebet Ali Dawed 4 &
• Natnael Kebede 5  

BMC Health Services Research volume  24 , Article number:  473 ( 2024 ) Cite this article

Metrics details

The Family Guidance Association of Ethiopia (FGAE) operates as a non-governmental organization dedicated to offering family planning and reproductive health services to the Ethiopian population. The gap in the study regarding client satisfaction and contributing factors towards sexual and reproductive health services for youth at FGAE clinics highlights the need for a comprehensive investigation to fill this void. By conducting a mixed-method study, this research aims to provide a holistic understanding of the factors influencing client satisfaction in the delivery of sexual and reproductive health services to youth at FGAE clinics. The added value of this study lies in its potential to offer valuable insights and recommendations for improving service delivery systems and enhancing client satisfaction levels, ultimately contributing to the overall well-being and health outcomes of youth in North Ethiopia. Therefore study aimed to asses Client Satisfaction and Contributing Factors towards in sexual and reproductive health services delivery system among youth at Family Guidance Association of North Ethiopia (FGAE) Clinics, 2023.

The study was conducted within the clinics of the Family Guidance Association of Ethiopia (FGAE), spanning Dessie, Kombolcha, and Woldia city administrations, involving a participant cohort of 416 clients. Facility-based concurrent type mixed method study design both quantitative and qualitative techniques were applied. Quantitative research employed a simple random sampling technique and conversely, the qualitative study utilized a heterogeneous type of purposive sampling strategy to recruit participants The collected data underwent a rigorous process of entry, cleaning, and coding using Epi-Data 4.6 software, followed by analysis in STATA V17. Descriptive statistics and binary logistic regression were employed to highlight the impact of independent variables on the dependent variable. A more comprehensive examination was provided through multivariable logistic regression. Crude and adjusted odds ratios, along with a 95% confidence interval, were computed, with significance set at a p-value ≤ 0.05.

Nearly more than half of the clients 194 (47.8%) came to receive family planning services followed by maternal and child health 107 (26.4%). Sixty patients (14.8%) didn’t receive all the services they wanted or came for. Half of the participants 30 (50%) raised the unavailability of the service as a reason for not taking the service followed by not having enough time in the clinic 12 (20%). About 65.52% (60.74-70.00%) of the participants were satisfied with the Sexual and Reproductive Health services provided by the clinics of FGAE in northeast Ethiopia. Clients in the age group of 25–34 (AOR = 2.04; 95%CI: 1.11–3.72). Clients who had primary and secondary education (AOR = 2.49; 95%CI: 1.03–6.02) and (AOR = 3.05; 95%CI: 1.25–7.49) respectively. Clients who responded that physicians show respect (AOR = 5.59; 95%CI: 1.89–16.49). clients who received an explanation about the side effects of the utilized methods and follow-up dates (AOR = 4.59;95%CI:1.68–12.53) and (AOR = 2.89;95%CI:1.53–5.49) respectively.

The proportion of client satisfaction with Client Satisfaction in the Services delivery system at Family Guidance Association of Ethiopia (FGAE) Clinics was low as compared to the previous study. Age group 25–34 years, primary and secondary education, showing respect, explaining side-effects and follow-up visits were significant associated factors of client service satisfaction. Enhancing service delivery at Family Guidance Association of Ethiopia (FGAE) Clinics by targeting specific areas identified in the study. Strategies should focus on improving communication regarding side effects, ensuring respectful interactions, and prioritizing follow-up visits, particularly for clients aged 25–34 with primary and secondary education backgrounds.

Peer Review reports

The Family Guidance Association of Ethiopia (FGAE) is a non-governmental, volunteer-based, not-for-profit organization dedicated to providing comprehensive reproductive health services across various regions of Ethiopia. Over the past five decades, FGAE’s initiatives have evolved significantly, transforming from a single-room clinic staffed by one nurse into a network of 46 integrated sexual and reproductive health (SRH) service delivery facilities. This network includes one Maternal and Child Health (MCH) Center, seven higher Model Clinics, 14 Medium Clinics, and 10 Confidential (Sex-workers Friendly) SRH Clinics. Additionally, FGAE operates 15 youth centers, more than 478 franchised clinics, and over 500 Primary Health Facilities (PHFs) throughout the country, effectively covering almost all regions. These endeavors are coordinated and managed by eight Area Offices strategically positioned across the nation [ 1 , 2 ]. Consumer satisfaction and perceived quality of outpatient health services, and analysis of consumers’ satisfaction factors will provide valuable insights into healthcare services and consumer satisfaction dynamics [ 3 ]. Addressing customer satisfaction and user satisfaction in healthcare services remains a critical focus area in both marketing and healthcare research. It enhances the ongoing efforts to enhance customer and user satisfaction in healthcare services through various strategies and interventions [ 4 , 5 ].

The satisfaction of clients in sexual and reproductive health (SRH) services serves as a pivotal indicator, gauging the level of contentment a client derives from the services provided by healthcare professionals. Consequently, it mirrors the disparity between the anticipated service and the experience, as perceived by the client [ 6 ]. The satisfaction of clients is an essential component of quality and hence is expected to improve through better compliance with the service, at the same time, satisfied clients will generate demand in the community and assist in the recruitment of new clients who can use the services [ 7 ].

It has been reported in the meta-analysis that, clients’ satisfaction with family planning services in Ethiopia was 56.78% [ 6 ]. Despite high client satisfaction with FP services, the 2016 Ethiopian Demographic and Health Survey (EDHS) reported that over a third of women had stopped using FP within a year [ 8 ]. To address this issue, the Ethiopian government has launched a health transformation program aimed at increasing FP use to 55% by 2020 and minimizing the unmet need for FP use to 10% but, progress has been slow since mini-EDHS (2019) found the contraceptive rate to be 41% [ 9 ].

Client satisfaction is also an essential determinant of service uptake and continuation because satisfied clients are more likely to revisit the service pass on a positive message to others and continue the use of particular family planning methods [ 6 , 10 ]. It also signals other aspects of quality of care including structural and process issues of quality of care in FP services. It reflects the perception of healthcare consumers on the quality of care on existing health services [ 11 ]. To achieve the target, it is obvious that different health institutions and actors’ involvement shall be important. To contribute to this national effort DMC, DCC, KMSRHS, and WMSRHS are also playing a critical role. The gap in the study regarding client satisfaction and contributing factors towards sexual and reproductive health services for youth at FGAE clinics highlights the need for a comprehensive investigation to fill this void. By conducting a mixed-method study, this research aims to provide a holistic understanding of the factors influencing client satisfaction in the delivery of sexual and reproductive health services to youth at FGAE clinics. The added value of this study lies in its potential to offer valuable insights and recommendations for improving service delivery systems and enhancing client satisfaction levels, ultimately contributing to the overall well-being and health outcomes of youth in North Ethiopia. Therefore study aimed to asses Client Satisfaction and Contributing Factors towards in sexual and reproductive health services delivery system among at youth at Family Guidance Association of north Ethiopia (FGAE) Clinics, 2023.

Methods and material

Study area and period.

The study will be conducted at FGAE SRH clinics found in Dessie, Kombolcha, and Woldia city administrations which are located in the Eastern Amhara region 401 Km, away from Addis Ababa. Dessie lies 401 km North of Addis Ababa. In the population projection for 2021, the entire population of Dessie was 296,536. A total of 60,315 were women of reproductive age group (15 to 49 years old) of which a total of 9,993 were expected to become pregnant in the year. The city has six hospitals, two of which are government and the other four of which are private. There are also 8 public health centers and 2 Non-Governmental Organizations (NGO) health institutions. The study was completed from August– October 2023.

Study design and population

Facility-based concurrent type mixed method study design both quantitative (cross-sectional study design) and qualitative (descriptive study design) techniques were applied. The source population for the quantitative study were all clients of FGAE and selected clients of sexual and reproductive health service utilizers at medium clinics of FGAE were study populations. The study participants for the qualitative study were all purposively selected health workers who have adequate experience or are adequately involved in the program and youth who receive services from the Family Guidance Association of Ethiopia’s Northeast Area office in the east Amhara region, Ethiopia.

Inclusion and exclusion criteria

All clients aged 15–49 years who utilized services were included and those clients who were in critical pain during the data collection period were excluded from the study.

Sample size determination

The sample size for the quantitative study was calculated using the single population proportion formula for the second, third, and fourth objectives and the double population proportion formula was further calculated for the second objective as well. The sample size for the qualitative study was determined based on the level of saturation. Saturation was considered reached when further addition of participants to the study did not yield any new perspectives or information [ 12 ]. The intended number of participants for this study was 10 Key Informant Interviews (KII) and 4 Focus Group Discussions (FGD). The overall client satisfaction in Ethiopia among family planning services was 56.78 [ 3 ]. Therefore, with a 5% margin of error and a 95% confidence interval, the sample size was calculated to be 378. Adding a 10% non-response rate, the final sample size was determined to be 416. The sample size for factors with a 95% confidence interval for significant variables from previous research could be calculated using open epi software (Table  1 ).

Sampling techniques and procedures

The sample size was distributed proportionally for each FGAE clinic based on the monthly client flow. Consequently, according to the northeast branch of FGAE, there were 31,035 clients in DMC, 9,587 in DCC, 8,675 in KMSRHS, and 10,048 in WMSRHS annually. Therefore, the number of client flows per data collection period (10 days) was estimated to be 162, 51, 46, and 157 in DMC, DCC, KMSRH, and WMSRHS, respectively. As a result, clients were recruited systematically by calculating the sampling interval for each clinic. Hence, every 5th client was interviewed after the first client was recruited using a simple random sampling technique. The sampling techniques for the qualitative study were a heterogeneous type of purposive sampling strategy to recruit the participants. The heterogeneous characteristics of participants were maintained by considering sex and educational level.

Data collection tools and measurement

The questionnaire was adapted from different literature and pre-tested in a similar setting other than the FGAE clinics of DCC, DMC, WMSRHS, and KMSRHS. Client satisfaction was assessed with exit interviews using twelve Likert-scaled question items. Each item had 5 points, with one denoting “1” for strongly disagree, two for “2” disagree, three for “3” not sure, four for “4” agree, and five for “5” strongly agree. Finally, the mean was computed, and by classifying responses as above/equal and below the mean, they were categorized into “Satisfied (coded as 1)” and “Not satisfied (coded as 0),” respectively. The questionnaire contained the socio-demographic factors of the clients, health service organization experience, reproductive health history, provider competence, and attitude of clients towards SRH service. Besides, the service delivery system of SRH at FGAE was assessed by reviewing strategic and action plans, organizational charts, staffing plans, staff lists, staff job descriptions, and evaluation tools, human resource management manual, financial management and administration, and annual implementation reports. Data were collected with face-to-face interviews with clients, health service providers, and managers, and document review.

The independent variables included socio-demographic factors (age, residence, marital status, educational level, religion, ethnicity, occupation), health facility-related factors (frequency of visit, opening time convenience, privacy maintained during counseling and procedure, cleanliness of the clinic, waiting time, and rooms having posters with key messages of family planning), the information given, and provider-related factors (appointments made for follow-up, clinical staff show respect, providers explain how the method works, providers demonstrate how to use the method, providers describe the possible side-effects, providers describe what to do when a problem occurs, and providers describe the possibility of changing the method when there is a complication) and other interpersonal characteristics. The data collection technique for the qualitative study was key informant interviews and focus group discussions.

Key Informant Interviews involved people focusing on a list of issues regarding a topic with which interviewees had first-hand knowledge to explore coping strategies for the identified factors hindering service uptake in the DMC, DCC, Kombolcha, and Woldia Medium SRH Clinics. A total of 10 key informant interviews were conducted in the study area. Four focus group discussions were conducted with clients who received services from the Family Guidance Association of Ethiopia’s Northeast Area office. The setting was selected by the participants and was convenient and suitable for sitting circularly to allow face-to-face interaction and recording with minimal external disturbance. Three facilitators, including the principal investigator and recorder/note-taker, were assigned for each FGD, with clear roles defined before each session. The principal investigator led the discussion, asked all questions specified in the focus group guide, kept the discussion on track, encouraged participation from all attendees, and took short notes and memos.

Data quality assurance

T he study was collected by 6 BSc nurses working at FGAE clinics. The data collection tool was pre-tested on 5% of the total sample individuals at the youth center. In addition, training was provided for data collectors. Supervisors followed and supervised the overall data collection activities. The quality of the qualitative study was checked by trustworthiness based on Lincoln and Guba’s criteria of credibility, dependability, conformability, and transferability [ 13 ]. The principal investigator ensured credibility by pretesting the interview guide in a similar context, using probing and multiple data sources, and inviting participants to review the transcription. Phone calls were made to build rapport, and preconceptions were bracketed to minimize bias. Conformability was ensured by recording all participant activities during interviews, preserving audio-taped interviews, and involving qualitative researchers in peer debriefing for data analysis and interpretations. Dependability was achieved through accurate documentation, frequent checks for spelling errors, and including all relevant documents in the final report, ensuring transparency. Transferability was achieved by providing a clear description of the study setting, sample, and data collection procedures and seeking input from qualitative experts through peer debriefing.

Data-analysis procedures

Data were entered, cleaned, and coded into Epi-Data 4.6 software and then exported to STATA V17 software for analysis. Simple descriptive statistics such as frequencies mean and standard deviations were done. Following bivariable binary logistic regression, variables with p-values less than 0.25 were selected for inclusion in the multivariable logistic regression model. This step aims to identify independent predictors of the outcome variable while controlling for potential confounders a result, a crude and adjusted odds ratio with a 95% confidence interval was calculated. A p-value of less than or equal to 0.05 was considered significant. The Hosmer and Lemeshow test demonstrated a well-fitted model with a p-value of 0.07, indicating satisfactory agreement between observed and predicted values. Additionally, the absence of multicollinearity suggests that the independent variables in the model are not highly correlated, enhancing the reliability of the regression analysis results. For the qualitative, Audio-recorded interviews were transcribed verbatim and translated to English by the principal investigators, and the translated data were further checked for accuracy by another independent reader. The thematic analysis approach was used to analyze the data. The principal investigators read and re-read the transcriptions several times and listened to the audio-taped interview repeatedly to provide a sense of integrity and understand the meaning of the experiences from the participant’s viewpoint. Each meaning unit was labeled with a code representing its content by open coding, and then similar codes were organized into categories. Atlas. ti software version 7 was used to facilitate data analysis. Two independent coders participated in coding. Categories were peer-reviewed and checked by the co-lead author, and final categories and themes were created. Lastly, the report was written based on categories and predefined themes for presenting the discoveries of the study. Quotes were used to highlight each category and show association with each theme. The findings were triangulated with quantitative findings.

Socio-demographic characteristics of study participants

Among the 416 eligible participants, 406 clients participated in the study, reflecting a response rate of 97.6%. Of these, the majority were female, constituting 361 individuals (88.9%), while over half, 209 individuals (51.5%), were married. The average age of the clients was 28.1 years (SD: 6.9), with 202 individuals (49.7%) falling within the 25–34 years age group. Additionally, 364 participants (89.6%) hailed from urban areas. Furthermore, 129 individuals (31.8%) had attained a college education or higher, while 112 (27.6%) identified as housewives. (Table  2 ). Nearly half of the clients, specifically 194 individuals (47.8%), sought family planning services, while maternal and child health services were sought by 1078 clients (26.4%). Sixty patients (14.8%) were unable to access all the services they required or came for. Among them, half of the participants, totaling 30 individuals (50%), cited service unavailability as the primary reason, followed by insufficient time in the clinic, mentioned by 12 individuals (20%).

Facility related factors

Over 60% of the clients, totaling 252 individuals (62.1%), made multiple visits to the clinic. A vast majority, 389 clients (95.8%), found the clinic’s opening hours convenient, while 382 clients (94.1%) were satisfied with its cleanliness. On average, clients waited 33.5 min in the waiting room (SD: 27.8), with the majority (70.2%) waiting for less than 30 min. Appointment dates were provided on follow-up cards for 321 clients (79.1%). Additionally, healthcare providers explained and demonstrated the use of sexual and reproductive health services to 270 clients (68.2%) and 288 clients (71.3%), respectively. (Table  3 ).

Client satisfaction level towards SRH services in FGAE clinics Northeast area

About 65.52% (60.74-70.00%) of the participants were satisfied with the Sexual and Reproductive Health services provided by the clinics of FGAE in the northeast area office. Moreover, 396(97.54%), 387(95.32%), and 393(96.80%) of the respondents reported that the working hours of the clinics were convenient, the compassionate care, and the consultation time with the health care provider was sufficient to fulfill their needs, respectively. However, about 34(8.37%), 35(8.62%), and 121(29.80%) of the participants stated that their privacy was not secure during the consultation, the service payment was not affordable, and did not get the opportunity to choose family planning (FP) methods in the clinics, respectively (Table  4 ).

Client satisfaction level and facility-related factors by types of FGAE clinics

A noteworthy variation was observed among the different clinics. While WMSRHC had the highest percentage of satisfied clients at 83.58%. On the contrary.KMSRHC and WMSRHC stood out with overwhelmingly positive attitudes at 94.8% and 88.1%, respectively. On the other hand, DCC and DMC had a higher percentage of clients with negative attitudes, indicating a potential area for improvement in these clinics. The analysis of facility-related factors revealed positive trends across all clinics. Opening time convenience, working hour convenience, privacy during procedures, and cleanliness of the clinic were reported as satisfactory by a significant majority of clients in each clinic. Notably, WMSRHC consistently scored high in all these factors, indicating a well-rounded positive facility experience (Table  5 ).

The gap concerning client satisfaction in the clinic appears to be primarily attributed to the narrow and overcrowded waiting areas, particularly in services such as Antenatal Care (ANC) and family planning, as highlighted in key informant interviews and discussions with clinic management. However, confidentiality and privacy do not seem to pose significant challenges to the clinics.

A participant of KII said “I think all the health care providers including the janitors and securities keep the privacy of clients, since the SRH service needs confidentiality. As a client, some of them may be afraid to use the services here, however, we reassure the clients to keep their secrets” (KII, KMSRHC).

The clinic’s waiting area, particularly in services like Antenatal Care (ANC) and family planning, is identified as a potential gap in client satisfaction due to overcrowding and narrow space, as indicated in interviews with key informants and discussions with clinic management. However, confidentiality and privacy issues do not seem to present significant challenges to the clinics.

The clinics used different service promotion strategies to the community through demand creation and use of ICC/BCC materials. But to reach a large community there is a need for large-scale promotion through mass media (radio, TV) and social media (facebook, telegram, and TickTok) to reach more young technology-based generation. There is also a need for promotion banners at different areas of the town and town entry doors to all clinics.

Participants of KII said” We have a demand creation team in the community, but the promotion is not enough to reach the population to advertise the services. We promote the clinic services through demand creation. We have also posted out said on the clinic and other ICC/BCC materials. We have no media promotion. Better also use promotion banners at different areas of the town and town entry Gates. We use Demand creators, meetings, and IEC/BCC, but do not use media like radio, TV, and other social media communications (KII, DCC, DMC, WMSRH, and KMSRHC).

Another KII participant reported that ” currently, the number of clients is very low, because, we are not making promotions well through radio or on the street with microphone. Some clients said that they do not know about the FGA availability in the town. But after the clients come to the clinic, we provide leaflets. I think the clinic is not accessible to the clients, this is because, after we moved from the privies place at “KUTEBA” to the current place, the client’s flow decreased significantly. I think the main reason for this decrement is changing the site, and not promoting well about the new site” (KII, KMSRHC).

Client dissatisfaction primarily stems from the absence of new service components, such as C/S delivery and infertility services, along with extended service waiting times. Additionally, the unavailability of essential services like delivery, Postnatal Care (PNC), pharmacy services, and diagnostic tests in the laboratory can diminish client satisfaction. Other discomforts for clients include overcrowded waiting areas, excessive service integration, lack of accessibility features like ramps for disabled persons at the clinic’s entrance, and weak service integration, particularly in immunization services such as growth monitoring and cold chain management issues.

The KII participant said, “Clients need new service components to attract clients like C/S delivery, infertility services, decrease client overcrowding and service waiting time, relief the crowding at service payment area, child growth monitoring and promotion and improve child health service room with decoration to attract clients”(KII, DMC). Another KII participant said “I think, the unavailability of the different services like delivery, PNC, immunization, Pharmacy etc, may reduce the satisfaction of the clients. If these services had been available, customer satisfaction could have been better (KII, KMSRHC).

Participants of KII said that “Common reasons to decrease client satisfaction are unavailability of essential drugs, low numbers of health professionals, and low salary of the employees, long service waiting time and overcrowding of clients at payment area, professional do not wear identification badge among clinic staffs, not wear of uniforms among guards and cleaners(DMC, DCC, WMSR ).

The main logistic challenges were market fluctuation, lack of logistics on the market, Performa process and increased cost of drugs and supplies, shortage of family planning choices (LAFP, COC), condoms, HIV test kits, and STI management kits, vaccine fridge, and cervical cancer screening materials.

A participant of KII said “ There is insufficiency of some inputs like HIV kits, clients go back with no services, lack of CBC machine, shortage of reagents and chemistry machine not working. Medical equipment and materials shortage and not availability is the barrier in this clinic to start services based on clients need ” (KII, WMSRHC).

To enable this clinic to provide quality services, it should be funded sufficiently, fulfill necessary logistics, be filled with enough health professionals as per the standards, and Increase health professionals’ satisfaction as well.

A participant of KII reported that” The clinic mainly delivers SRH services, but the community demands additional medical and investigation services from the clinic. There is a need for a separate room for radiography and medical services, a need for gynecology and obstetrics specialists, renovation of the clinic building like painting, classroom expansion, improvement staff carrier, need for X-ray machine and service needs attention to improve client satisfaction(KII, DMC).

Another KII said “To enable this clinic to provide quality services, it should be funded sufficiently, filled with enough health professionals, Increase health professionals satisfaction, service expansion like delivery, PNC, immunization, Pharmacy etc, may increase the client satisfaction ”KII, WMSRHC, KMSRHC).

Attitude towards Family Guidance Association of Ethiopia (FGAE) clinics service

The majority of respondents, accounting for 78.57%, expressed a strongly agree with the belief that clients should utilize health services for SRH. About 73.15% of participants strongly agree that youth should be aware of the importance of SRH services. A gender-related perspective emerged in the data, with 40.15% disagreeing that only females should use health services for SRH. Concerns about the judgmental nature of health providers were evident, with 42.12% disagreeing that health providers exhibit judgmental behavior. Positive responses were observed regarding confidentiality assurance, with 58.13% strongly agreeing that health providers ensure the confidentiality of youth (Table  6 ).

The mean attitude score towards the services provided by the Family Guidance Association of Northeast Ethiopia (FGAE) Clinics was 13.62 ± SD 3.61. Approximately half of the participants, constituting 50.7% (CI: 45.9-55.6%), demonstrated a favorable attitude towards FGAE Clinics, highlighting a positive reception of the services. Conversely, the remaining 49.3% (CI: 44.1-54.1%) exhibited a negative attitude, indicating a discernible proportion of individuals who may have reservations or concerns regarding the services provided by FGAE Clinics.

Factors associated with client satisfaction in FGAE clinic services

In the binary logistic regression, age, sex, educational status, occupation, residence, frequency of visit, respect is shown for clients, comfort with the cleanness of the clinic, availability of messages in the waiting room about SRH services, demonstration of a method for clients, explaining the side effects of the methods and what to do when a problem occurs, longer waiting times and explaining follow-up time were significantly associated with satisfaction with p-value < 0.25.

In the multivariable logistic regression, only age, 25–34 years, primary and secondary education, showing respect, and explaining side-effects and follow-up visits become important factors for service satisfaction. Clients in the age group of 25–34 were 2.04 times more satisfied than clients in the age group of 15–24 (AOR = 2.04; 95%CI: 1.11–3.72).clients who had primary and secondary education were 2.49 and 3.05 times more satisfied than those clients who were unable to read and write respectively (AOR = 2.49; 95%CI: 1.03–6.02) and (AOR = 3.05; 95%CI: 1.25–7.49) respectively. Clients who responded that physicians show respect were 5.59 times more satisfied than their counterparts (AOR = 5.59; 95%CI: 1.89–16.49). clients who received an explanation about the side effects of the utilized methods and follow-up dates were 4.59 and 2.89 times more satisfied than their counterparts respectively (AOR = 4.59;95%CI:1.68–12.53) and (AOR = 2.89;95%CI:1.53–5.49) respectively (Table  7 ).

According to the findings, the proportion of client satisfaction was 65.52% (95%CI: 60.74-70.00%) in northeast Ethiopia FGAE clinics services. One study in Nigeria found a similar level of client satisfaction with sexual and reproductive health services, with a proportion of 67% among youth accessing services at public health facilities [ 14 ]. This finding is lower than a previous study conducted in India [ 15 ]. In contrast, a study in Kenya reported a higher proportion of client satisfaction, with 75% of youth expressing satisfaction with sexual and reproductive health services provided by non-governmental organizations (NGOs) [ 16 ]. The possible difference may be due to the cultural context in India may have different expectations and standards for client satisfaction compared to the northeast Ethiopia FGAE clinics. Cultural factors such as communication styles, attitudes toward healthcare, and expectations of service quality can influence client satisfaction.

In this study Clients in the age group of 25–34 were more satisfied than clients in the age group of 15–24. In a previous study conducted in India, it was found that clients in the age group of 25–34 were more satisfied with healthcare services compared to clients in the age group of 18–24 [ 15 ]. One possible justification for the statement could be that the younger age group (25–34) in this study had access to more advanced and improved healthcare services. This could have resulted in higher satisfaction levels among the younger clients in this study. Additionally, it is possible that the younger age group in this study had different expectations and preferences for healthcare services, which were better met by the healthcare providers, leading to their higher satisfaction levels.

Moreover, the study found that clients who had primary and secondary education were more satisfied than those clients who were unable to read and write. This finding is consistent with previous studies, conducted in India [ 17 ], and Ethiopia [ 18 ]. One possible justification for this finding could be that clients with higher levels of education may have better health literacy and understanding of their healthcare needs and services. They may also be more empowered to advocate for their health and communicate effectively with healthcare providers. In contrast, a study conducted in Nigeria found no significant association between education level and client satisfaction [ 19 ]. The possible discrepancy may be cultural or societal factors may play a role in shaping perceptions of healthcare and satisfaction with services. Further research is needed to explore these potential factors and better understand the relationship between education level and client satisfaction across different contexts.

Furthermore, Clients who responded that physicians show respect were more satisfied than their counterparts. A previous study conducted in the United States also found that clients who perceived their physicians as respectful were more satisfied with their healthcare experience [ 20 ]. This suggests that the way physicians interact with their clients can have a significant impact on their satisfaction with healthcare services. Showing respect to clients can help build trust and rapport, which can lead to better communication and more positive healthcare outcomes. Additionally, clients who feel respected may be more likely to adhere to treatment plans and follow-up care recommendations, which can further improve their satisfaction with healthcare services. Therefore, healthcare providers should strive to show respect to their clients as part of their overall approach to providing high-quality care. Finally, clients who received an explanation about the side effects of the utilized methods and follow-up dates were more satisfied than their counterparts. This is similar to the previous studies conducted in Japan [ 21 ] and in the United States [ 22 , 23 ]. One possible justification by explaining the potential side-effects and follow-up dates, physicians are empowering their clients to make informed choices about their treatment and care. When clients are informed about potential side effects and know what to expect in terms of follow-up care, they may feel more in control of their health outcomes and have a better understanding of the treatment process. This, in turn, can lead to higher levels of satisfaction with healthcare services.

Practical implication of the study

The study on client satisfaction and contributing factors towards sexual and reproductive health services delivery system among youth at Family Guidance Association provides valuable insights that can have profound scientific contributions to the field of public health. By understanding the factors that influence client satisfaction with these services, healthcare providers can tailor their services to better meet the needs of youth, ultimately improving health outcomes and promoting positive behaviours. This research can also inform policy decisions and program development aimed at enhancing the delivery of sexual and reproductive health services to youth, ensuring that they have access to high-quality care that meets their specific needs. Overall, this study has the potential to drive significant advancements in the field of sexual and reproductive health, ultimately benefiting the health and well-being of young people.

Conclusions

In summary, the proportion of client satisfaction was low in northeast FGAE clinic services. Age group 25–34 years, primary and secondary education, showing respect, explaining side-effects and follow-up visits were significant associated factors of client service satisfaction. Client dissatisfaction arose from a lack of new service components like C/S delivery, insufficient infertility services, and prolonged waiting times. The absence of services such as delivery, PNC, Pharmacy EPI, and a robust laboratory could diminish client satisfaction. Uncomfortable clinic conditions included crowded waiting areas, over-integration of services, accessibility issues for disabled persons, and weak integration of services in immunization, impacting areas like growth monitoring and cold chain management. Enhancing service delivery at Family Guidance Association of Ethiopia (FGAE) Clinics by targeting specific areas identified in the study. Strategies should focus on improving communication regarding side effects, ensuring respectful interactions, and prioritizing follow-up visits, particularly for clients aged 25–34 with primary and secondary education backgrounds.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Anti-Natal Care

Behavioural Communication Change

Cease Ran Section

Dessie Confidential Clinic

Dessie model Clinic

Expanded program on Immunization

Focused Group Discussion

Information Communication Change

Key Informant Interview

Kombolcha Medium SRH Clinics

Out Patient Department

Woldia Medium SRH Clinics SRH:Sexual and Reproductive Health

Post-Natal Care

Ali M, Mekonnen W, Tekalegn Y. Magnitude and factors associated with intrauterine contraceptive device method utilization among clients attending family guidance association clinics in Addis Ababa. Ethiopia. 2019;16:327.

Google Scholar  

Teshome SB, Hoebink P. Aid, ownership, and coordination in the health sector in Ethiopia. Dev Stud Res. 2018;5(sup1):S40–55.

Article   Google Scholar  

Peyrot M, Cooper PD, Schnapf D. Consumer satisfaction and perceived quality of outpatient health services. Mark Health Serv. 1993;13(1):24.

CAS   Google Scholar  

Ali J, Jusoh A, Idris N, Nor KM. Healthcare service quality and patient satisfaction: a conceptual framework. Int J Qual Reliab Manage. 2023.

Lu W, Hou H, Ma R, Chen H, Zhang R, Cui F, et al. Influencing factors of patient satisfaction in teleconsultation: a cross-sectional study. Technol Forecast Soc Chang. 2021;168:120775.

Geta T, Awoke N, Lankrew T, Elfios E, Israel E. Prevalence and associated factors of client satisfaction with family planning service among family planning users in Ethiopia: a systematic review and meta-analysis. BMC Womens Health. 2023;23(1):151.

Article   PubMed   PubMed Central   Google Scholar  

Birhanu Z, Assefa T, Woldie M, Morankar S. Determinants of satisfaction with health care provider interactions at health centers in central Ethiopia: a cross-sectional study. BMC Health Serv Res. 2010;10(1):1–12.

CSA. Ethiopia Demographic and Health Survey (2016) [Available from: https://dhsprogram.com/pubs/pdf/FR328/FR328.pd .

CSA/ICF. Ethiopian Mini Demographic and Health Survey 2019, Addis Ababa and Rockville: Ethiopian Public Health Institute (EPHI) and DHS Program ICF, 2019 [Available from https://dhsprprogram.com/pubs/pdf/PR120/PR120.pdf .

Argago TG, Hajito KW, Kitila SB. Clients satisfaction with family planning services and associated factors among family planning users in Hossana town public health facilities, South Ethiopia: facility-based cross-sectional study. Int J Nurs Midwifery. 2015;7(5):74–83.

Tessema GA, Mahmood MA, Gomersall JS, Assefa Y, Zemedu TG, Kifle M, et al. Client and facility-level determinants of quality of care in family planning services in Ethiopia: multilevel modeling. PLoS ONE. 2017;12(6):e0179167.

Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.

Lincoln YS, Guba EG. Naturalistic inquiry: sage; 1985.

Wright KO, Oluwole E, Adeniran A, Kuyinu Y, Goodman O, Odusanya O. Youth-friendly health services in a rural community of Lagos, Nigeria: are the youths receptive? Int J Adolesc Med Health. 2016;29(3):20150095.

Jain A, Mishra N, Pandey C. A study to assess patient satisfaction in the outpatient department of a tertiary care hospital in north India. Int J Community Med Public Health. 2016;3(1):328–34.

Kitui LN. Youth Friendly Health Services in Kenya: characteristics, knowledge, attitude. Practices, and Experiences: JKUAT-COHES; 2023.

Martinussen PE, Rydland HT. Is a decentralized health policy associated with better self-rated health and health services evaluation? A comparative study of European countries. Int J Health Policy Manage. 2021;10(2):55.

Geberu DM, Biks GA, Gebremedhin T, Mekonnen TH. Factors of patient satisfaction in adult outpatient departments of the private wing and regular services in public hospitals of Addis Ababa, Ethiopia: a comparative cross-sectional study. BMC Health Serv Res. 2019;19(1):1–13.

Chirdan O, Lar L, Afolaranmi T, Inalegwu E, Igoh C, Adah G. Client satisfaction with maternal health services comparison between public and private hospitals in Jos Nigeria. Jos J Med. 2013;7(1):1–9.

Depositions LFP. The doctor-patient relationship and malpractice. Arch Intern Med. 1994;154:1365–70.

Walker MS, Ristvedt SL, Haughey BH. Patient care in multidisciplinary cancer clinics: does attention to psychosocial needs predict patient satisfaction? Psycho-Oncology: J Psychol Social Behav Dimensions Cancer. 2003;12(3):291–300.

Aiken LH, Sermeus W, Van den Heede K, Sloane DM, Busse R, McKee M et al. Patient safety, satisfaction, and quality of hospital care: cross-sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ. 2012;344.

Arraras JI, Illarramendi JJ, Viudez A, Ibáñez B, Lecumberri MJ, de la Cruz S, et al. Determinants of patient satisfaction with care in a Spanish oncology day hospital and its relationship with quality of life. Psycho-oncology. 2013;22(11):2454–61.

Article   PubMed   Google Scholar  

Wogu D, Lolaso T, Meskele M. Client satisfaction with family planning services and associated factors in Tembaro District, Southern Ethiopia. Open Access J Contracept. 2020:69–76.

Download references

Acknowledgements

We are grateful to thank Wollo University, the College of Medicine, and the Health Sciences Ethical Review Committee. Also, we want to gratify study participants and data collectors.

This investigation received no specific grant from a funding agency in the public commercial or for-profit sectors.

Author information

Authors and affiliations.

Department of Reproductive and Family Health, School of Public Health, College of Medicine and Health Sciences, Wollo University, Dessie City, Ethiopia

Niguss Cherie & Zinet Abegaz

Department of Health System and Management, School of Public Health, College of Medicine and Health Sciences, Wollo University, Dessie City, Ethiopia

Yawkal Tsega

Department of Epidemiology and Biostatistics, School of Public Health, Colleges of Medicine and Health Science, Wollo University, Dessie City, Ethiopia

Anissa Mohammed & Abel Endawkie

Department of Public Health Nutrition, School of Public Health, College of Medicine and Health Sciences, Wollo University, Dessie, Ethiopia

Yeshimebet Ali Dawed

Department of Health Promotion, School of Public Health, College of Medicine and Health Sciences, Wollo University, Dessie City, Ethiopia

Natnael Kebede

You can also search for this author in PubMed   Google Scholar

Contributions

Niguss Cherie, Yawkal Tsega, Anissa Mohammed, Zinet Abegaz, Abel Endawkie, Yeshimebet Ali Dawed, and Natnael Kebede, Dessie Ethiopia were contributed to Conceptualization, Investigation, Methodology, Formal Analysis, and software, Writing– review and Editing, and Writing–original draft.

Corresponding author

Correspondence to Natnael Kebede .

Ethics declarations

Ethical approval and consent to participate.

Ethical approval was obtained from the ethical review committee of the School of Public Health, College of Medicine and Health Sciences at Wollo University. After explaining the purpose of the study, written informed consent was obtained from participants before data collection. They were informed that participating in the study was voluntary and their right to withdraw from the study at any time during the interview was assured. For this purpose, a one-page consent letter was attached as a cover page of each questionnaire stating the general objective of the study and issues of confidentiality. Permission to undertake the study was performed at all levels. Caretakers were given detailed information about the purpose of the study, data collection procedures, and possible risks/ discomforts and benefits of participating in the study through the consent process. Informed written consent was obtained from all caretakers whose children participated in the study. In this case, caretakers were the families of each student and were presented as caretakers. A child was included in the study only if the caretakers agreed with the child. Despite the caretakers, consent, a child’s decision not to participate in the study was respected. All methods and materials were performed according to the guidelines.

Consent to Publish

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Cherie, N., Tsega, Y., Mohammed, A. et al. Client satisfaction and contributing factors towards sexual and reproductive health services delivery system among youth at Family Guidance Association of north Ethiopia (FGAE) clinics, 2023: mixed method study. BMC Health Serv Res 24 , 473 (2024). https://doi.org/10.1186/s12913-024-10874-8

Download citation

Received : 03 December 2023

Accepted : 15 March 2024

Published : 16 April 2024

DOI : https://doi.org/10.1186/s12913-024-10874-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Client satisfaction
• Contributing factors
• Sexual and Reproductive Health Services

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 16 April 2024

Development and validation of the Cannabis Exposure in Pregnancy Tool (CEPT): a mixed methods study

• Kathleen H. Chaput 1 , 2 ,
• Carly A. McMorris 3 ,
• Amy Metcalfe 1 , 2 ,
• Catherine Ringham 4 ,
• Deborah McNeil 2 , 5 ,
• Shaelen Konschuh 6 ,
• Laura J. Sycuro 1 , 6 &
• Sheila W. McDonald 2 , 7  

BMC Pregnancy and Childbirth volume  24 , Article number:  280 ( 2024 ) Cite this article

Metrics details

Evidence of associations between prenatal cannabis use (PCU) and maternal and infant health outcomes remains conflicting amid broad legalization of cannabis across Canada and 40 American states. A critical limitation of existing evidence lies in the non-standardized and crude measurement of prenatal cannabis use (PCU), resulting in high risk of misclassification bias. We developed a standardized tool to comprehensively measure prenatal cannabis use in pregnant populations for research purposes.

We conducted a mixed-methods, patient-oriented tool development and validation study, using a bias-minimizing process. Following an environmental scan and critical appraisal of existing prenatal substance use tools, we recruited pregnant participants via targeted social media advertising and obstetric clinics in Alberta, Canada. We conducted individual in-depth interviews and cognitive interviewing in separate sub-samples, to develop and refine our tool. We assessed convergent and discriminant validity internal consistency and 3-month test–retest reliability, and validated the tool externally against urine-THC bioassays.

Two hundred fifty four pregnant women participated. The 9-item Cannabis Exposure in Pregnancy Tool (CEPT) had excellent discriminant (Cohen’s kappa = -0.27–0.15) and convergent (Cohen’s kappa = 0.72–1.0) validity; as well as high internal consistency (Chronbach’s alpha = 0.92), and very good test–retest reliability (weighted Kappa = 0.92, 95% C.I. [0.86–0.97]). The CEPT is valid against urine THC bioassay (sensitivity = 100%, specificity = 82%).

The CEPT is a novel, valid and reliable measure of frequency, timing, dose, and mode of PCU, in a contemporary sample of pregnant women. Using CEPT (compared to non-standardized tools) can improve measurement accuracy, and thus the quality of research examining PCU and maternal and child health outcomes.

Peer Review reports

Amidst legalization and regulation of recreational cannabis in Canada in 2018, and legalization of medicinal and/or recreational use in 40 American States, prenatal cannabis use (PCU) is rising [ 1 , 2 ]. Despite recent studies showing associations between PCU and adverse maternal, infant, and child outcomes, such as pregnancy anemia, preterm birth, small for gestational age, placental abruption, neonatal intensive-care unit (NICU) admission, and intrapartum stillbirth [ 3 , 4 , 5 , 6 ], the evidence remains conflicting [ 7 , 8 , 9 , 10 , 11 , 12 ]. A critical limitation of published studies is a high risk of misclassification bias resulting from a lack of standardized measurement of PCU across adequate domains, including frequency, dose, modes, timing of use in pregnancy, and second-hand smoke and vapour. There is an urgent need for high-quality cannabis-related health research, and pregnant individuals and infants have been identified as priority populations [ 9 , 10 , 13 ]. Improved measurement of PCU in research is a key component to improving the quality of the evidence.

Current PCU measurement options available for research include administrative data collected during routine prenatal care, substance use disorder (SUD) screening tools, non-validated questionnaires, and biological tests. Administrative data is problematic for research use because pregnant people are known to under-report prenatal substance use to physicians, due to fears of judgement and/or being reported to child services [ 14 , 15 ]. Further, PCU screening is not standardized practice, occurs variably, and is seen as low-priority for the majority of obstetricians [ 16 ]. While Canadian studies using administrative data have reported PCU prevalence between 2 and 3% [ 2 , 3 , 4 ], emerging evidence from an anonymous Canadian survey indicates an 11% prevalence of PCU [ 17 ]. In a US study only 36% of women with cannabis-positive urine tests had reported their use to a care provider [ 18 ], indicating that the majority of those using cannabis prenatally may be misclassified in administrative data studies. Because those whose infants are at higher risk of PCU-related outcomes may also be less likely to report their PCU due to being younger, and socioeconomically disadvantaged, the current evidence may substantially underestimate the impacts of PCU on infant health outcomes.

While self-administered research questionnaires can garner more accurate reporting of substance (e.g. alcohol) use in pregnancy than screening in clinical settings [ 19 , 20 ], non-standardized survey questions have limited utility for measurement of PCU, as they can unintentionally convey perceived bias against PCU. They often identify cannabis as an illicit drug and do not differentiate between medicinal and recreational use, which may increase response bias, as the stigma of recreational use in pregnancy is higher, and people may be more willing to disclose cannabis use if they can attest that it is for medicinal purposes [ 18 , 21 ]. Survey questions are problematic for studying nuanced associations with maternal and infant health outcomes due to inconsistent assessment of frequency and timing of use, including changing patterns through pregnancy, and often lack dose measurement, or use subjective dose-terminology [ 9 , 10 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ]. Further, most lack measurement of potentially important consumption modes aside from smoking (vapourized, edible, topical, second-hand) [ 22 , 23 , 24 ]. Standardized SUD screening tools aim to detect a diagnosable SUD, and do not measure patterns PCU throughout pregnancy [ 29 ]. Many screen for alcohol misuse alone [ 30 , 31 , 32 , 33 ], or combine all drugs into a single category [ 29 ] preventing the separate evaluation of cannabis-related health outcomes. Biological (urine/blood/saliva) cannabis-screeners exist, but are limited to detection within 1–5 weeks of use, or up to 30–60 days in exceptional circumstances after high-dose long-term use, depending on individual metabolism and test cut-off levels [ 34 , 35 , 36 , 37 , 38 ]. Given that pregnancy is a 40-week period, the utility of these tests is limited. Biological samples are also resource-intensive and stigmatizing to collect, limiting their utility for prospective research.

Our study developed and validated a novel PCU measurement tool, that addresses the limitations of current measurement methods, using a prospective patient-oriented approach to identify patient-perceived stigma, and reduce perceived sources of response bias, using a six-step, peer-reviewed process [ 39 ].

We recruited pregnant participants who used cannabis prior to or during pregnancy, between 08/2019 and 04/2020 for the mixed-methods tool development phase and an external validation cohort between 04/2022 and 12/2022. We used social media advertising targeted to women aged 18–45 years, residing in Alberta, with listed interests or group memberships related to pregnancy, parenting, and/or cannabis, and posted gender-neutral recruitment ads in an online trans-gender parent support group. Study recruitment letters were also mailed to patients who visited Alberta Health Services (AHS) clinics for pregnancy-related care in the preceding six months, identified using pregnancy-related codes in the National Ambulatory Care Reporting System (NACRS)(Appendix A). We included participants meeting target criteria who were < 36 weeks’ gestation at intake. Our target development sample size of 150 participants was sufficient to detect a Cronbach’s alpha of ≥ 0.9, with 95% confidence for test–retest reliability on a tool that contains up to 15 items [ 39 ], and our external convenience sample of 85 participants was feasible for conducting urine tetrahydrocannabinol (THC) bioassays with available resources.

Step 1 qualitative interviews

We conducted individual in-depth interviews (IDIs) with 8 participants who used cannabis prenatally, and 2 who used previously but not in pregnancy, purposively selected from the full sample (Fig.  1 ). Two research assistants with qualitative interview training conducted telephone interviews at a time chosen by the participant, about views and experiences with cannabis use in general, and during pregnancy. Prior to interviews, research staff contacted participants twice to discuss study details, including confidentiality, and establish a trusting relationship, by disclosing their own connections to the study topic, emphasizing a non-judgmental approach, and acknowledging all experiences shared were important. We recorded and transcribed interviews verbatim, and used deductive thematic analysis to extract pre-determined themes of: language around cannabis and its use; perceptions of stigma and judgement, and their relationships to truthful disclosure of use; patterns of use in pregnancy (timing, frequency of use, typical dose); motivations for use; and forms of cannabis used. Two team members experienced in qualitative methods coded salient content that corresponded to the pre-determined themes, collapsed codes into broader themes using constant comparison technique, discussion and consensus. Themes were then reported back to the qualitative participants via email for member-checking of the relevance and appropriateness to ensure truth value.

A : Study flow diagram: development phase (08/2019 to 04/2020). 1 B ) External Validation phase (04/2022 to 12/2022)

Step 2 devising items

We devised constructs for the tool, and item wording, to draft the tool based on strengths and shortcomings identified in existing SUD tools and published survey questions (Table  1 ), and on themes identified from interviews. We eliminated double-barreled questions, ambiguous wording and ensured a 6th grade reading level.

Step 3 Cognitive interviewing and bias reduction

Schwartz and Oyserman [ 42 ] propose five stages of cognition required to accurately self-report behaviour, each of which are susceptible to bias: 1. question understanding, 2. recalling relevant behaviour, 3. inference & estimation, 4. mapping answer onto response options, and 5. answer editing. Cognitive interviews (CIs) assess the cognitive processing of each item and its response options by a respondent as they read and respond to the tool. To identify points of bias at all five stages of cognition, we conducted individual CIs with an additional sub-sample of participants from the full sample, in which respondents were asked to think aloud, and share impressions, understanding, and reasoning related to each of the five stages of cognition, as we administered the newly developed tool [ 43 ]. CI participants were recruited sequentially via social media advertising. We iteratively revised items according to participant feedback prior to each subsequent interview, until no new suggestions for revision were made in two consecutive interviews (after interview 17).

Step 4 content validation

We then formatted the refined items into the CEPT online tool, compared to our critical appraisal of existing tools to ensure it captured all domains of measurement that are critical to prospective research cannabis in pregnancy, including timing, multiple modes of consumption, dose per use and frequency of use.

Step 5 convergent and discriminant validation

We then administered the finalized CEPT, along with the SURP-P [ 44 ] and 4Ps + [ 30 ] SUD screening tools via electronic questionnaire, to our remaining sample of 150 women. We measured concurrent validity of CEPT responses against detailed cannabis use information revealed during the interviews using Cohen’s weighted kappa. There is strong evidence that a high degree of truth value can be achieved with rigorous qualitative interview techniques [ 45 ]. We assessed discriminant validity of CEPT responses against SURP-P and 4Ps + tools using Cohen’s kappa. We calculated internal consistency on all CEPT cannabis consumption items using Chronbach’s alpha, acknowledging that it measures multiple constructs of cannabis exposure (i.e. any use, frequency, timing, dose, mode and reasons), rather than a single construct. However, we anticipated internal consistency among the CEPT items, as a person indicating use should have non-zero responses for dose, mode frequency and reasons for use. We then re-administered the tool to all development-phase participants ( n  = 150) 3 months later to assess test–retest reliability using a weighted Cohen’s kappa (Fig.  1 a, b).

Step 6 external validation

In an additional external sample of 84 pregnant participants, we validated CEPT responses against urine bioassay measurements of 11-nor-9-carboxy-Δ 9 - THC, the most abundant THC metabolite (Fig.  2 ). Participants provided urine samples in sterile collection containers that were shipped frozen to our laboratory by pre-paid courier for analysis, within 24 h of completing an online questionnaire including the CEPT. We stored samples at -80°c until analysis. 2ml aliquots were taken from thawed samples, centrifuged and diluted (10x) with ultrapure water and assayed in duplicate using a 96-strip-well, THC Metabolite ELISA Kit (catalogue # 701570, Cayman Chemicals™, United States of America) according to manufacturer’s protocol, by team members blinded to CEPT results. No freeze–thaw cycles were allowed, and the lowest threshold of THC positivity detectable by the kits with 88% sensitivity (0.072ηg/ml) was used to classify those with PCU versus those without [ 40 ].

Cognitive interviews—bias reduction for the five stages of cognition. References:

Our sample included 254 pregnant women (including those who used cannabis in pregnancy or in the past, and those who’d never used cannabis), 170 in the development phase and 84 in the external validation cohort. Despite efforts to recruit gender-diverse participants, none enrolled in the study. Specific sub-samples participated in various steps (Fig.  1 ). Table 2 summarizes participant characteristics at enrollment. Other sociodemographic characteristics of our sample were similar to the overall maternal population in Canada [ 46 , 47 , 48 ]. (Suppl. Figure  1 ).

Qualitative interviews

We completed qualitative data collection after 10 interviews, when we reached thematic saturation (no new themes emerged). Summaries of deductive themes and illustrative quotes are presented in Table  3 .

Interviews informed bias-minimizing language and wording, tool structure, and appropriate response options for frequency dose and reasons for use. Themes drove the terminology and language used in the tool preamble and questions, guided tool structuring including inclusion of specific items (e.g. reasons for use) and response options, and determined the method of dose measurement. While legalization was perceived to have reduced stigma around cannabis use in general, perceptions of stigma against prenatal use were prevalent and thus important for consideration to encourage accurate disclosure. Several participants noted that including a response option to disclose cannabis consumption that occurred only prior to pregnancy recognition was crucial, and noted if this option was not present, they would not report use, even if they had consumed cannabis prior to pregnancy recognition . A challenging aspect of cannabis consumption measurement is identifying dose. IDI results identified a reliable method of categorizing approximate dose per use (i.e. comparing amounts to common objects, like food items or coins). Approximate THC/CBD content can be inferred based on mean THC content of dried cannabis available on the contemporary market (24%) [ 49 ], or the labeled concentration of products as reported by participants.(Supplementary file 2 ).

Cognitive interviews

We completed cognitive interviews with a separate sub-sample of 17 participants to assess and minimize points of bias through participant-led refinement (Fig.  2 ). This resulted in 9 sequential iterations of our initial draft tool. Perceived sources of bias at all five stages of cognition were identified, and changes made based on participant feedback.

Question understanding

Most draft-tool questions were well understood; however, some changes were made to improve clarity.

Recalling relevant behaviour

All participants indicated they were accurately able to recall details of first-hand cannabis consumption, including frequency, trimester of consumption, reasons, modes, and amounts per use. Nearly all participants (93%) indicated they were able to accurately recall the details of second-hand cannabis smoke or vapour exposure, aside from brief outdoor exposures. We amended the second-hand exposure question to include exposure while in the same room as the user.

Inference & estimation

Participants did not express concerns about inference or estimation on items measuring any consumption/exposure, or frequency, timing or reasons for use. Dose questions were adjusted to address perceived ambiguity and aid with estimation (Fig.  2 ).

Mapping answers onto response options

Several participants noted problems with initial dose-per-use options, increments for some product types were deemed too large for use in pregnancy, and we refined categories to align with appropriate ranges and increments.

Answer editing

No participants expressed the need to edit responses once the above clarifications and response-option edits had been made. Participants agreed the tool was non-judgemental, appropriate, and acceptable to them in pregnancy, and that it would elicit truthful responses, confirming face and content validity from the participant perspective.

The final CEPT measures weeks of gestation, second-hand exposure, partner use, trimester(s) of use, frequency, reasons, modes of consumption, and dose per use for each mode indicated. Frequency items repeat for each trimester, and dose items for each mode of use indicated. (Supplement 2 ).

Validity and reliability

Concurrent validity was excellent, with agreement between IDI participant CEPT responses and use reported in IDIs, ranging from 80 to 100%, and kappa values ranging from substantial (0.72) to perfect (1.0) [ 50 ] (Table  4 ). The timing of use construct showed the lowest level of agreement, which was expected, given that the second administration of the CEPT was at a later point in pregnancy. Use will be reported in more trimesters as a pregnancy progresses. A greater proportion of participants (40%) reported third-trimester use on the online CEPT, compared with IDIs (30%), which occurred 5–6 weeks prior, as many were not yet in the third trimester at the time of IDI.

Discriminant validation indicated poor agreement between two pregnancy SUD screening tools (4ps+ and the SURP-p) [ 33 ], with weighted Kappa values ranging from -0.31 to 0.36 indicating that the CEPT measures different constructs from those on the existing tools. (Table  5 ).

Reliability testing showed excellent internal consistency (Chronbach’s alpha = 0.92) and substantial to near-perfect Kappa values (0.71–0.99) for test–retest reliability (Table  6 ). Although some patterns of use may be expected to change throughout pregnancy, the strong agreement between early and late pregnancy responses on the CEPT support that recall of cannabis consumption using this tool is reliable up to delivery.

CEPT-reported cannabis use was valid against urine-THC bioassay with 100% sensitivity, and 82% specificity, indicating that it has promise as an improved measure of PCU for research purposes (Table  7 ). All participants with positive urine bioassay disclosed that their last cannabis use was within 1 week of the urine sample being collected, indicating that the time elapsed since last use was the main driver of lower specificity.

The CEPT addresses the measurement limitations faced by previously published studies of PCU and maternal and infant health, which are highly susceptible to misclassification bias, have inconsistent findings, and are rated moderate at best by the US National Academies of Science Engineering and Medicine [ 10 , 41 ]. It offers researchers a measurement option that has initially shown strong validity and reliability, that accounts for frequency, modes, reasons and estimated dose-per-use, and separately measures CBD and THC. The CEPT measures the frequency of use in each trimester separately to capture changing patterns of PCU. This enables an estimate of the total number of uses throughout pregnancy, based on the number of months, weeks or days in the given trimester (which can be adjusted for gestational age at delivery), and can then be multiplied by the estimated dose per use to generate an estimated total exposure over the pregnancy interval. It also measures frequency of second-hand exposure in each trimester, in addition to partner’s cannabis use. The CEPT thus enables a more complete picture of PCU patterns and a more nuanced estimate of total exposure over pregnancy than currently published studies have been able to capture. The patient-oriented methods we used are a strength; qualitative interviews can reveal aspects of health behaviour that contrast with the researcher’s underlying assumptions, that can interfere with the five stages of cognition leading to biased response [ 39 , 42 ]. Prenatal alcohol use studies indicate that non-disclosure bias for prenatal substance use varies according to participant perceptions, and that question wording and structure informed by patient-oriented designs can improve validity [ 20 , 51 ]. Further, the language, tone, and perceived intent of the tool are critical to non-biased response. Our qualitative interviews guided us in reducing perceived judgemental or stigmatizing language in our tool. The cognitive interviews further reduced sources of bias. While we may never be able to completely eliminate PCU reporting bias our patient-oriented development process was chosen because it is crucial for minimising perceived stigma, and ensuring a much lower probability of bias than the methods of measurement used in previous studies, including self-selection for biological samples, which do not allow the participant to explain their reasons for use, nor to judge the researachers’ motivations.

Although there remains no feasible gold-standard measure of prenatal cannabis consumption across the entire gestational period, the CEPT represents a useful tool for researchers to augment the quality and expand the scope of longitudinal research into the health outcomes associated with prenatal cannabis exposure. Our results support that it minimizes self-report bias, and its nuanced measurement of multiple dimensions of cannabis consumption may also reduce misclassification of very low exposures, allow for assessment of potential dose–response relationships, and enable the identification of critical windows of fetal exposure in future studies, that were not possible with previous crude measures.

Limitations

The CEPT is designed to measure behaviours over pregnancy, rather than to detect a condition or health state. Where medical screening tools can be validated against diagnostic tests or interview, validating a measure of behaviour is more complex. A limitation of our study is the lack of a true gold-standard measure of PCU for validation, which was financially infeasible for this study, as it requires multiple bioassays of at least weekly serial urine samples throughout gestation. However, we have preliminarily validated CEPT responses against a biological reference-standard, showing excellent sensitivity and high specificity. While we could not attain a true biological gold-standard in our study, the validation we conducted against single bioassays, and in-depth qualitative interviews remains rigorous. Biological levels of THC metabolite cannot be falsified, and the qualitative methods we employed result in high credibility and truth-value for qualitative results [ 52 ]. Further, interviews allowed for comparison of binary cannabis use as well as PCU patterns (modes, frequency, timing) that cannot be validated with a biological test. Although our study sample was adequate to detect a Cronbach’s alpha of ≥ 0.9 on a tool with up to 15 items, we acknowledge that our external bioassay validation sample ( n  = 84) was small, and differences in maternal age, marital status and household income between the development and validation smaples were noted. Future validation studies should include larger samples to confirm findings, and should explore whether the estimated dosage measured by the CEPT correlates to quantitative biological THC and CBD metabolite levels. Further, psychometric testing of the CEPT is recommended in future studies. It is also important to note that our tool and the validation conducted are limited to English-speaking individuals, and any translations will require further validation.

PCU and its associated health outcomes have been identified as priorities for research in Canada and the U.S. following cannabis legalization [ 9 ]. We recommend the CEPT as a rigorous, feasible, patient-oriented health research tool for measuring PCU. The use of the CEPT as a standardized measure of PCU in future studies can contribute substantial new knowledge about the implications of timing, dose, frequency, and modes of exposure for maternal, fetal, infant and child health, accounting for varying patterns of consumption and the strength and diversity of cannabis products available on the contemporary legal market. The CEPT has the potential to significantly improve measurement accuracy and thus the quality of research in this area, which can in turn inform evidence-based education, prevention and health policy to mitigate potential health risks.

Availability of data and materials

Quantitative data can be made available in accordance with the ethics approval for the study, on reasonable request to the corresponding author.

Abbreviations

Prenatal Cannabis Use

Tetrahydrocannabinol

Young-Wolff KC, Sarovar V, Tucker LY, Avalos LA, Alexeeff S, Conway A, et al. Trends in marijuana use among pregnant women with and without nausea and vomiting in pregnancy, 2009–2016. Drug Alcohol Depend. 2019;196:66–70.

Article   PubMed   PubMed Central   Google Scholar  

Corsi DJ, Hsu H, Weiss D, Fell DB, Walker M. Trends and correlates of cannabis use in pregnancy: a population-based study in Ontario, Canada from 2012 to 2017. Can J Public Heal Can Sante Publique. 2019;110:76–84.

Luke S, Hutcheon J, Kendall T. Cannabis use in pregnancy in British Columbia and selected birth outcomes. J Obstet Gynaecol Can. 2019;41:1311.

Article   PubMed   Google Scholar  

Corsi DJ, Walsh L, Weiss D, Hsu H, El-Chaar D, Hawken S, et al. Association between self-reported prenatal cannabis use and maternal, perinatal, and neonatal outcomes. JAMA - J Am Med Assoc. 2019;322:145.

Article   Google Scholar  

El Marroun H, Tiemeier H, Steegers EAP, Jaddoe VWV, Hofman A, Verhulst FC, et al. Intrauterine cannabis exposure affects fetal growth trajectories: the generation R study. J Am Acad Child Adolesc Psychiatry. 2009;48(12):1173–81.

El Marroun H, Hudziak JJ, Tiemeier H, Creemers H, Steegers EAP, Jaddoe VWV, et al. Intrauterine cannabis exposure leads to more aggressive behavior and attention problems in 18-month-old girls. Drug Alcohol Depend. 2011;118:470.

Stickrath E. Marijuana use in pregnancy: an updated look at marijuana use and its impact on pregnancy. Clin Obstet Gynecol. 2019;62:185.

Metz TD, Stickrath EH. Marijuana use in pregnancy and lactation: a review of the evidence. Am J Obstetr Gynecol. 2015;213:761–78.

Porath-Waller AJ. Clearing the Smoke on Cannabis Series: Maternal Cannabis Use during Pregnancy. Can Cent Subst Abus. 2015;2:1–8.

Board on Population Health and Public Health Practice, Health and Medicine Division, National Academies of Sciences Engineering and Medicine. The Health Effects of Cannabis and Cannabinoids: The Current State of Evidence and Recommendations for Research. The Health Effects of Cannabis and Cannabinoids: The Current State of Evidence and Recommendations for Research. 2017. 440 p. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28182367

Torres CA, Medina-Kirchner C, O’Malley KY, Hart CL. Totality of the evidence suggests prenatal cannabis exposure does not lead to cognitive impairments: a systematic and critical review. Front Psychol. 2020;11:816.

Chaput KH, Lebel CA, McMorris CA. Commentary: Totality of the Evidence Suggests Prenatal Cannabis Exposure Does Not Lead to Cognitive Impairments: A Systematic and Critical Review. Front Psychol. 2020;11:817.

Center for Behavioral Health Statistics and Quality. Results from the 2012 National Survey on Drug Use and Health. 2013.

Roberts SCM, Nuru-Jeter A. Women’s perspectives on screening for alcohol and drug use in prenatal care. Women’s Heal Issues. 2010;20(3):193–200.

Ernhart CB, Morrow-Tlucak M, Sokol RJ, Martier S. Underreporting of alcohol use in pregnancy. Alcohol Clin Exp Res. 1988;12(4):506–11.

Article   CAS   PubMed   Google Scholar  

Holland CL, Nkumsah MA, Morrison P, Tarr JA, Rubio D, Rodriguez KL, et al. “Anything above marijuana takes priority”: Obstetric providers’ attitudes and counseling strategies regarding perinatal marijuana use. Patient Educ Couns. 2016;99(9):1446–51.

Kaarid KP, Vu N, Bartlett K, Patel T, Sharma S, Honor RD, et al. Assessing the prevalence and correlates of prenatal cannabis consumption in an urban Canadian population: a cross-sectional survey. Can Med Assoc Open Access J. 2021;9(2):E703--E710. Available from: https://www.cmajopen.ca/content/9/2/E703

Chang JC, Holland CL, Tarr JA, Rubio D, Rodriguez KL, Kraemer KL, et al. Perinatal illicit drug and marijuana use: An observational study examining prevalence, screening, and disclosure. Am J Heal Promot. 2017;31(1):35–42.

Hannigan JH, Chiodo LM, Sokol RJ, Janisse J, Ager JW, Greenwald MK, et al. A 14-year retrospective maternal report of alcohol consumption in pregnancy predicts pregnancy and teen outcomes. Alcohol. 2010;44(7–8):583–94.

Chiodo LM, Sokol RJ, Delaney-Black V, Janisse J, Hannigan JH. Validity of the T-ACE in pregnancy in predicting child outcome and risk drinking. Alcohol. 2010/01/06. 2010;44(7–8):595–603. Available from: https://pubmed.ncbi.nlm.nih.gov/20053522

Rotermann M, Pagé MM. Prevalence and correlates of non-medical only compared to self-defined medical and non-medical cannabis use, Canada, 2015. Heal Reports. 2018;29(7):3–13.

Google Scholar  

Singer LT, Arendt R, Fagan J, Minnes S, Salvator A, Bolek T, et al. Neonatal visual information processing in cocaine-exposed and non-exposed infants. Infant Behav Dev. 1999;22:1.

Fried PA, Watkinson B. 36- and 48-month neurobehavioral follow-up of children prenatally exposed to marijuana, cigarettes, and alcohol. J Dev Behav Pediatr. 1990;11(2):49-58.

Day NL, Richardson GA, Goldschmidt L, Robles N, Taylor PM, Stoffer DS, et al. Effect of prenatal marijuana exposure on the cognitive development of offspring at age three. Neurotoxicol Teratol. 1994;16:169.

el Marroun H, Tiemeier H, Jaddoe VWV, Hofman A, Mackenbach JP, Steegers EAP, et al. Demographic, emotional and social determinants of cannabis use in early pregnancy: the generation R study. Drug Alcohol Depend. 2008;98:218.

Office of Research and Surveillance. Canadian Tobacco Alcohol and Drugs (CTADS): 2015 summary. CTADS 2015. 2017. Available from: https://www.canada.ca/en/health-canada/services/canadian-tobacco-alcohol-drugs-survey/2015-summary.html

Coleman-Cowger VH, Schauer GL, Peters EN. Marijuana and tobacco co-use among a nationally representative sample of US pregnant and non-pregnant women: 2005–2014 National survey on drug use and health findings. Drug Alcohol Depend. 2017;177:130.

Leos-Toro C, Rynard V, Hammond D. Prevalence of problematic cannabis use in Canada: Cross-sectional findings from the 2013 Canadian Tobacco, alcohol and drugs survey. Can J Public Heal. 2017;108(5–6):e516–22.

Gotman N, Kershaw T, Rounsaville BJ, Forray A, Howell HB, Yonkers KA. Screening for prenatal substance use. Obstet Gynecol. 2010;116(4):827–33.

Chasnoff IJ, Wells AM, McGourty RF, Bailey LK. Validation of the 4P’s Plus© screen for substance use in pregnancy validation of the 4P’s Plus. J Perinatol. 2007;27(12):744–8.

Ondersma SJ, Svikis DS, Lebreton JM, Streiner DL, Grekin ER, Lam PK, et al. Development and preliminary validation of an indirect screener for drug use in the perinatal period. Addiction. 2012;107(12):2099–106.

Harris SK, Knight JR, Van Hook S, Sherritt L, Brooks TL, Kulig JW, et al. Adolescent substance use screening in primary care: validity of computer self-administered versus clinician-administered screening. Subst Abus. 2016;37(1):197–203.

Yonkers KA, Gotman N, Kershaw T, Forray A, Howell HB, Rounsaville BJ. Screening for prenatal substance use: Development of the substance use risk profile-pregnancy scale. Obstet Gynecol. 2010;116(4):827–33.

James WH, Moore DD. Examining the relationship between gender and drug-using behaviors in adolescents: the use of diagnostic assessments and biochemical analyses of urine samples. J Drug Educ. 2005;29(3):235–49.

Mura P, Kintz P, Papet Y, Ruesch G, Piriou A. Evaluation of six rapid tests for screening of cannabis in sweat, saliva and tears. Acta Clin Belg. 1999;53(Suppl 1):35–8.

PubMed   Google Scholar  

Pippins JR, Brawarsky P, Jackson RA, FuentesAfflick E, Haas JS. Association of Breastfeeding with Maternal Depressive Symptoms. J Women’s Heal. 2006;15(6):754–62.

Riley KJ, Lu NT, Taylor BG. Drug screening: a comparison of urinalysis results from two independent laboratories. J Drug Issues. 2000;30(1):171–86.

Earleywine M. Understanding Marijuana: A New Look at the Scientific Evidence. Understanding Marijuana: A New Look at the Scientific Evidence. 2010;1–341.

Streiner DL, Norman GR. Health Measurement Scales: A practical guide to their development and use. Health Measurement Scales: A Practical Guide to their Development and Use. 2008;1-452

Biomedical C. PATHFINDER THC Elisa kit booklet.

The National Academies of Science Engineering. The health effects of Cannabis and Cannabinoids. Committee’s conclusions. The National Academies of Science Engineering and Medicine. Natl Acad Press. 2018; Available from: http://www.dictionary.com/browse/weight

Schwarz N, Oyserman D. Asking questions about behavior: Cognition, communication, and questionnaire construction. Am J Eval. 2001;22(2):127–60.

Willis G, Lessler J. Question Appraisal System (QAS 99 Manual). National Cancer Institute. 1999;1-25.

Coleman-Cowger VH, Oga EA, Peters EN, Trocin K, Koszowski B, Mark K. Comparison and validation of screening tools for substance use in pregnancy: a cross-sectional study conducted in Maryland prenatal clinics. BMJ Open. 2018;8(2):e020248.

Sale J, Brazil K. A strategy to identify critical appraisal criteria for primary mixed methods research. Qual Quan. 2004;38:351–65.

Canada. PHA of. Data Tables - The Maternity Experiences Survey (MES) 2006 - 2007. 2012. 2009. Available from: http://www.phac-aspc.gc.ca/rhs-ssg/pdf/tab-eng.pdf

Teubert D, Pinquart M, Pinheiro MIS, Haase VG, Del Prette A, Amarante CLD, et al. NIH Public Access. Eur J Psychotraumatol. 2015;15(1):No-Pagination Specified. Available from: http://www.repositorio.ufc.br/bitstream/riufc/7744/1/2014-DIS-ASMGRANGEIRO.pdf http://www.files/5407/GRANGEIRO,2014.pdf http://www.files/5404/rocha,marinatesesysr.pdf http://www.linkinghub.elsevier.com/retrieve/pii/S0890856716302167 http://www.files/5400/ache

No Title. 2019. 2019. Available from: https://www.nbs.go.tz/index.php/en/census-surveys/population-and-housing-census/178-tanzania-total-population-by-district-regions-2016-2017

ElSohly MA, Mehmedic Z, Foster S, Gon C, Chandra S, Church JC. Changes in cannabis potency over the last 2 decades (1995–2014): analysis of current data in the United States. Biol Psychiatry. 2016;79:613.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159–74. Available from: http://www.jstor.org/stable/2529310

Sokol RJ, Martier SS, Ager JW. The T-ACE questions: Practical prenatal detection of risk-drinking. Am J Obstet Gynecol. 1989;160(4):863–70.

Lincoln Y, Guba NJ. Naturalistic inquiry, vol. 1. Beverly Hills: Sage Publications; 1985.

Download references

Acknowledgements

This study was funded by the Alberta Children’s Hospital Research Institute and the Alberta Children’s Hospital Foundation.

Author information

Authors and affiliations.

Department of Obstetrics and Gynecology, Cumming School of Medicine, University of Calgary, 2500 Unievrsity Drive NW, Calgary, AB, Canada

Kathleen H. Chaput, Amy Metcalfe & Laura J. Sycuro

Department of Community Health Sciences, Cumming School of Medicine University of Calgary, 1403 29 Street NW, Calgary, AB, T2N 2T9, Canada

Kathleen H. Chaput, Amy Metcalfe, Deborah McNeil & Sheila W. McDonald

Werklund School of Education, School and Child Psychology, University of Calgary, Calgary, Canada

Carly A. McMorris

School of Nursing, Thomson Rivers University, 40 College Way, Kamloops, BC, Canada

Catherine Ringham

Maternal Newborn Child and Youth Strategic Clinical Network, Alberta Health Services, Edmonton, Canada

Deborah McNeil

Department of Microbiology, Immunology and Infectious Diseases, Cumming School of Medicine, University of Calgary, Calgary, Canada

Shaelen Konschuh & Laura J. Sycuro

Research and Innovation Population, Public, and Indigenous Health, Alberta Health Services, Edmonton, Canada

Sheila W. McDonald

You can also search for this author in PubMed   Google Scholar

Contributions

KC contributed to conception and design of the work, and the acquisition of data, KC, SK, and LS conducted data analysis, and KC, CM, AM, CR, DM, SM contributed to interpretation of data. KC drafted the work and all authors substantively revised it. All authors have approved the submitted version (and any substantially modified version that involves the author's contribution to the study), and have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Kathleen H. Chaput .

Ethics declarations

Ethics approval and consent to participate.

This study was performed in accordance with the Declaration of Helsinki and was approved by the Conjoint Health Research Ethics Board at the University of Calgary (REB19-0670), and written informed consent was obtained from all participants.

Consent for publication

N/A No details, images, or videos relating to an individual person are included in this manuscript.

Competing Interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Chaput, K.H., McMorris, C.A., Metcalfe, A. et al. Development and validation of the Cannabis Exposure in Pregnancy Tool (CEPT): a mixed methods study. BMC Pregnancy Childbirth 24 , 280 (2024). https://doi.org/10.1186/s12884-024-06485-0

Download citation

Received : 26 May 2023

Accepted : 05 April 2024

Published : 16 April 2024

DOI : https://doi.org/10.1186/s12884-024-06485-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Measurement
• Reliability

BMC Pregnancy and Childbirth

ISSN: 1471-2393

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 16.4.2024 in Vol 8 (2024)

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Authors of this article:

Original Paper

• Nicola Luigi Bragazzi 1, 2, 3 , MPH, MD, PhD   ; 
• Sergio Garbarino 2, 4 , MD, PhD  

1 Human Nutrition Unit, Department of Food and Drugs, University of Parma, Parma, Italy

2 Department of Neuroscience, Rehabilitation, Ophthalmology, Genetics and Maternal/Child Sciences, University of Genoa, Genoa, Italy

3 Laboratory for Industrial and Applied Mathematics, Department of Mathematics and Statistics, York University, Toronto, ON, Canada

4 Post-Graduate School of Occupational Health, Università Cattolica del Sacro Cuore, Rome, Italy

Corresponding Author:

Nicola Luigi Bragazzi, MPH, MD, PhD

Human Nutrition Unit

Department of Food and Drugs

University of Parma

Via Volturno 39

Parma, 43125

Phone: 39 0521 903121

Email: [email protected]

Background: Adequate sleep is essential for maintaining individual and public health, positively affecting cognition and well-being, and reducing chronic disease risks. It plays a significant role in driving the economy, public safety, and managing health care costs. Digital tools, including websites, sleep trackers, and apps, are key in promoting sleep health education. Conversational artificial intelligence (AI) such as ChatGPT (OpenAI, Microsoft Corp) offers accessible, personalized advice on sleep health but raises concerns about potential misinformation. This underscores the importance of ensuring that AI-driven sleep health information is accurate, given its significant impact on individual and public health, and the spread of sleep-related myths.

Objective: This study aims to examine ChatGPT’s capability to debunk sleep-related disbeliefs.

Methods: A mixed methods design was leveraged. ChatGPT categorized 20 sleep-related myths identified by 10 sleep experts and rated them in terms of falseness and public health significance, on a 5-point Likert scale. Sensitivity, positive predictive value, and interrater agreement were also calculated. A qualitative comparative analysis was also conducted.

Results: ChatGPT labeled a significant portion (n=17, 85%) of the statements as “false” (n=9, 45%) or “generally false” (n=8, 40%), with varying accuracy across different domains. For instance, it correctly identified most myths about “sleep timing,” “sleep duration,” and “behaviors during sleep,” while it had varying degrees of success with other categories such as “pre-sleep behaviors” and “brain function and sleep.” ChatGPT’s assessment of the degree of falseness and public health significance, on the 5-point Likert scale, revealed an average score of 3.45 (SD 0.87) and 3.15 (SD 0.99), respectively, indicating a good level of accuracy in identifying the falseness of statements and a good understanding of their impact on public health. The AI-based tool showed a sensitivity of 85% and a positive predictive value of 100%. Overall, this indicates that when ChatGPT labels a statement as false, it is highly reliable, but it may miss identifying some false statements. When comparing with expert ratings, high intraclass correlation coefficients (ICCs) between ChatGPT’s appraisals and expert opinions could be found, suggesting that the AI’s ratings were generally aligned with expert views on falseness (ICC=.83, P <.001) and public health significance (ICC=.79, P =.001) of sleep-related myths. Qualitatively, both ChatGPT and sleep experts refuted sleep-related misconceptions. However, ChatGPT adopted a more accessible style and provided a more generalized view, focusing on broad concepts, while experts sometimes used technical jargon, providing evidence-based explanations.

Conclusions: ChatGPT-4 can accurately address sleep-related queries and debunk sleep-related myths, with a performance comparable to sleep experts, even if, given its limitations, the AI cannot completely replace expert opinions, especially in nuanced and complex fields such as sleep health, but can be a valuable complement in the dissemination of updated information and promotion of healthy behaviors.

Introduction

An adequate amount of good, restorative sleep is of paramount importance for both individual and public health [ 1 , 2 ]: from an individual standpoint, it helps maintain optimal physical and mental health, facilitating cognitive function, ensuring well-being, and mitigating the risks associated with chronic diseases [ 3 ]. In the context of public health, sleep’s impact is profound and multifaceted as well, being a pivotal element in driving the economy, ensuring public safety, and managing health care expenditures. The strategic addressing of sleep-related issues not only alleviates the global burden of disease but also ameliorates the economic strain associated with it [ 4 , 5 ].

The promotion of healthy sleep patterns and the intervention in sleep-related disorders emerge as vital strategies, paving the way for the enhancement of overall societal well-being, boosting productivity, and fostering social cohesion [ 6 ]. Such initiatives can yield substantial benefits, at both the individual and community levels, thereby underscoring the role of innovative tools, including digital ones—spanning from dynamic websites to sleep trackers and mobile apps—in promoting and providing education on sleep health [ 7 ].

The internet offers a vast, versatile, easily accessible, and cost-effective platform for disseminating up-to-date information about sleep, reaching diverse populations, raising public awareness about the importance of sleep, and providing personalized guidance on sleep health and related topics. People can access the latest findings and recommendations to make informed decisions about their sleep habits, with telemedicine and web-based consultations with sleep experts becoming increasingly popular. The digital realm can enable individuals to monitor their sleep patterns, engaging them in continuous learning about sleep health, and facilitating self-awareness and behavioral changes to improve sleep quality [ 8 ].

In the era of generative conversational artificial intelligence (AI) [ 9 , 10 ], characterized by disruptive technological transformation, the importance of sleep health promotion and education becomes even more relevant [ 11 ]: conversational AI-based platforms and agents, such as chatbots, can provide instant responses to sleep-related queries, making information readily available at any time. This real-time accessibility can help individuals seeking answers about sleep health, who can receive personalized advice and recommendations based on an individual’s specific sleep patterns and concerns. However, besides being accessible and tailored, this information should also be accurate [ 12 ].

There are only a few studies that have assessed sleep-related knowledge of conversational AI-based chatbots, such as ChatGPT-4, which was found very recently to successfully pass the sleep medicine certification board examinations [ 13 ] and be conversant in sleep disorders, such as obstructive sleep apnea syndrome [ 14 - 16 ].

On the other hand, conversational AI may contribute to disseminating “factual errors, nonsense, fabricated sources, and dangerous advice” and, thus, spreading biomedical misinformation, including sleep-related misinformation [ 17 ]. Therefore, our study was conducted to verify the accuracy of a popular prototype of conversational AI, ChatGPT, in addressing queries concerning sleep health and, in particular, sleep-related myths. These can be defined as widely held “false beliefs about sleep” that “lack an evidence base” and “can persist despite contradicting scientific evidence, potentially impairing” and even degrading population health, by promoting the adoption of unhealthy behaviors and lifestyles, the identification of which “can inform efforts to promote population sleep health” [ 18 ].

A list of 20 sleep-related myths, as defined above, was taken from a previously published study [ 18 ]. This list was compiled using internet searches of popular press and scientific literature and leveraging a Delphi process that involved 10 sleep experts from the fields of sleep medicine and research. Experts were recruited by convenience sampling, after being identified through literature searches (using PubMed). To be considered an expert, they were required to have published 20 papers that were cited by 20 or more different peer-reviewed sources, and at least one of these publications had to be tagged with the “Medical Subject Headings” “sleep” along with either “circadian rhythms,” “neuroscience,” or “psychiatry.” A total of 20 individuals who fulfilled these requirements were reached out to and out of these 20 experts 10 took part in this study. The Delphi process consisted of selecting and refining myths and was conducted in 3 stages: initially, focus groups were held (phase 1); this was followed by a period of email-based feedback for editing, adding, or removing myths (phase 2); finally, closed-ended surveys were used (phase 3), during which experts assessed the myths. The 20 myths were, then, categorized along six domains: namely, (1) “sleep duration” (n=6), (2) “sleep timing” (n=1), (3) “behaviors during sleep” (n=4), (4) “daytime behaviors that relate to sleep” (n=2), (5) “pre-sleep behaviors” (n=5), and (6) “brain function and sleep” (n=2). Besides providing feedback, experts had to rate myths on 2 dimensions: falseness and public health significance using a 5-point Likert scale from 1 (“not at all false” or “not at all significant”) to 5 (“extremely false” or “extremely significant”) [ 18 ].

It should be noted that, while some of these myths are patently false (such as the statement “during sleep, the brain is not active,” which belongs to the “brain function and sleep” domain), other statements such as “lying in bed with your eyes closed is almost as good as sleeping” (belonging to the “behaviors during sleep” domain) contain some elements of truth and other sleep experts may disagree in labeling them as complete misinformation or myths. Indeed, whether “waking rest” and other “resting states” may confer benefits almost as good as deep rest is debated [ 19 , 20 ]: these concepts challenge the conventional dichotomy of sleep and wakefulness conceived as binary and distinct states and suggest that periods of quiet reflection during wakefulness, characterized by a lack of effortful, focused thought and the absence of distracting stimuli, can also contribute significantly to mental rejuvenation, memory consolidation, hormonal regulation, cellular repair, and emotion regulation [ 21 - 23 ]. Altogether, these resting states, defined also as “offline states,” including eyes-closed rest, daydreaming, mind wandering, or inattentive states, represent approximately half of our waking hours [ 24 , 25 ]. However, the existing scholarly literature reports scarce, contrasting, or even negative findings [ 26 ], that warrant further research and suggest that these concepts are not yet well established and are continuously evolving [ 19 , 20 ].

Sleep is, indeed, a complex, nonlinear process, and sometimes, our perception of how well we sleep or even whether we are asleep or awake can be incorrect. For instance, “sleep state misperception,” also known as “paradoxical insomnia,” is a condition where individuals believe they are awake for most of the night, despite actually sleeping for a normal duration. This disorder is characterized by a significant discrepancy between perceived and actual sleep time, often associated with personality traits like neuroticism and altered brain activity during sleep, though its causes and prevalence remain under investigation [ 27 - 30 ].

Ethical Considerations

Full ethical clearance was waived for this study, as this study is a purely observational study with responses generated by AI (ChatGPT) and secondary analysis of research data, consisting of anonymous or deidentified study data [ 18 ].

Statistical Analysis

ChatGPT (version 4) was asked both to determine if these 20 sleep-related disbeliefs were true or false and to appraise them using the 5-point Likert scale. To determine if there is a statistical difference between the 6 abovementioned domains in terms of the distribution of true, false, and other categories of responses, a chi-square test of independence was used. This test helped us determine if the differences in proportions across the different domains were statistically significant. Means were also reported for the overall score (along with their SDs) and broken down according to each domain.

In terms of accuracy, the sensitivity and the positive predictive value of ChatGPT in categorizing the sleep-related statements as false were computed.

Finally, ChatGPT’s ratings of falseness and public health significance of sleep-related myths were compared with those provided by sleep experts. The degree of agreement was measured, in terms of consistency, using the interrater reliability analysis, computing the intraclass correlation coefficients (ICCs) [ 31 ].

All statistical analyses were carried out using SPSS (version 28 for Windows; IBM Corp). P values less than .05 were considered statistically significant.

Qualitative Analytical Approach

A qualitative comparative analysis was also conducted. Initially, responses from ChatGPT and summary responses from the sleep experts [ 18 ] were subjected to a line-by-line comparison to identify similarities and differences in content, style, and complexity of information provided. Then, responses were scrutinized to identify themes, concepts, or categories that were entered in a matrix to have a clear snapshot of where ChatGPT and the experts aligned or diverged in their discussions and to make emerging patterns of alignment and divergence between them. This phase was crucial for understanding how ChatGPT’s training data correlated or not with the current consensus among experts and enabled the identification of gaps in ChatGPT’s knowledge base.

ChatGPT’s Falseness Quantitative Analysis of Sleep-Related Myths

Overall, ChatGPT labeled 45% (n=9) of the statements as “false,” while a further 40% (n=8) of the items were deemed as “generally false.” Of note, concerning the remaining statements, 5% (n=1) and 10% (n=2) of them were considered “true” and “not (entirely or necessarily) true or false,” respectively. In terms of domain, half of the items related to “sleep duration” were considered “false” (n=3), with the remaining half percent being deemed “generally false” (n=3). The statement concerning “sleep timing” was labeled as “generally false” (n=1). Further, 75% (n=3) of the items related to “behaviors during sleep” were correctly identified as “false,” while the remaining 25% (n=1) were classified as “generally false.” All the statements concerning “daytime behaviors related to sleep” were considered “generally false” (n=2). When assessing the accuracy of items concerning “pre-sleep behaviors,” half of them were properly labeled as “false” (n=2), whereas 16.67% (n=1) of the statements were considered “generally false,” with a further 16.67% (n=1) being “not entirely true or false” and the remaining 16.67% (n=1) being even considered “true.” Finally, concerning “brain function and sleep,” half of the statements were correctly appraised as “false” (n=1), with the remaining half being labeled as “not necessarily true or false” (n=1). Further details are reported in Table 1 and Multimedia Appendix 1 .

The various response categories did not vary depending on the domain of sleep-related myths ( χ 2 15 =14.60, P =.48).

On the 5-point Likert scale, the degree of falseness was computed at 3.45 (SD 0.87), according to ChatGPT’s estimates. The highest scores were recorded for “brain function and sleep” (4.00, SD 1.41), “sleep timing” (4.00, single item), and “sleep duration” (3.67, SD 0.98), while the “behavioral domains” scored the lowest. More in detail, “behaviors during sleep” yielded a value of 3.25 (SD 0.50), followed by “pre-sleep behaviors” (3.20, SD 0.84) and “daytime behaviors that relate to sleep” (3.00, SD 1.41). Further details are presented in Table 1 and Multimedia Appendix 2 .

Based on these data, ChatGPT demonstrated an overall sensitivity of 85% and a positive predictive value of 100% in categorizing the statements as “false.”

Quantitative Comparison of ChatGPT’s and Expert Ratings on the Falseness of Sleep-Related Myths

When comparing with sleep experts, a good interrater agreement could be found between ChatGPT’s categorization of statements and expert rating on their falseness. Statements categorized by ChatGPT as “false” and “generally false” were those that received the highest scores by the experts (4.25 and 3.97, respectively), whereas those judged by the AI as “true” and “not true or false” received the lowest scores by the experts (3.75 and 3.44, respectively), as shown in Figure 1 . From a more quantitative standpoint, the association yielded an ICC value of 0.83 ( P <.001), when ChatGPT was asked to rate the degree of falseness of the statement on the 5-point Likert scale ( Figure 2 ).

ChatGPT’s Public Health Significance Quantitative Analysis of Sleep-Related Myths

The overall score was 3.15 (SD 0.99). “Sleep timing” was the domain scoring the highest (4.00, single item), followed by “sleep duration” (3.33, SD 1.37) and “behaviors during sleep” (3.25, SD 0.96). “Daytime behaviors that relate to sleep” scored the lowest (2.50, SD 0.71), while both “brain function and sleep” and “pre-sleep behaviors” yielded a mean score of 3.00 (SD 1.41, and SD 0.71, respectively). Further details are presented in Table 2 .

Quantitative Comparison of ChatGPT’s and Expert Ratings on the Public Health Significance of Sleep-Related Myths

Similar trends to those observed for the expert appraisals of the falseness of sleep-related myths could be reported for the expert rating on their public health significance. Items labeled by ChatGPT as “false” and “generally false” corresponded to a score of 3.37 and 3.04, respectively, while statements appraised by the AI as “true” and “not true or false” scored the lowest (2.71 and 2.07, respectively), as pictorially represented in Figure 3 .

When comparing the ratings on public health significance provided by ChatGPT with those by the experts, an ICC of 0.79 could be computed ( P =.001), as shown in Figure 4 .

Qualitative Comparison and Thematic Analysis of ChatGPT’s and Sleep Experts’ Responses

The qualitative comparison between ChatGPT’s and sleep experts’ appraisals of sleep-related myths revealed both similarities and differences across the various misconceptions.

In general, both ChatGPT and sleep experts clarified and, in some instances, strongly refuted these misconceptions, highlighting the importance of adequate sleep for overall well-being, while acknowledging individual differences in sleep needs and patterns, and emphasizing that sleep requirements can vary from person to person. However, differences in responses could be noted: ChatGPT adopted a more conversational and accessible style and provided a more generalized view, focusing on broad concepts, while experts sometimes used technical jargon, providing evidence-based explanations to debunk false beliefs about sleep. Sleep experts tended to provide a more detailed assessment, often including specific medical and physiological contexts, referencing studies, and focusing both on individual (clinical) and public health implications, while a population health perspective was generally missing in ChatGPT’s responses.

Specifically concerning “sleep duration myths,” both ChatGPT and sleep experts clarified the misconceptions, but experts provided a more medically oriented focus, delving into the biological and physiological underpinnings more deeply, while ChatGPT emphasized sleep-related flexibility and individual variability. Further, sleep experts provided a more nuanced appraisal of risks and the evidence (or lack thereof) supporting them, focusing not only on temporary or short-term effects but offering detailed perspectives on long-term consequences and recovery too. Similarly, concerning “sleep-related timing myths,” both ChatGPT and sleep experts emphasized the role of circadian rhythms and the importance of sleep timing, but experts gave a more technical assessment focusing on specific physiological processes. Regarding “behaviors during sleep” myths and “daytime sleep-related behaviors” myths, sleep experts delved into physiological, cultural, and habitual aspects and potential adverse outcomes. Concerning “pre-sleep behaviors myths,” sleep experts focused more on the neurophysiological impacts and provided more evidence-based assessments, with more nuanced explanations of the causes and effects underlying sleep phenomena and disturbances. Finally, regarding “brain function and sleep myths,” sleep experts gave a more detailed explanation of brain functions, highlighting the complexity of sleep research and the presence of controversial topics and conflicting results. Further details are reported in Table S1 in Multimedia Appendix 3 .

Generative Conversational AI and Sleep-Related Myths

AI-driven platforms and agents, including chatbots, can provide sleep-related information and education, ensuring that a diverse global audience can access valuable and customized resources on sleep health. This tailored guidance can be particularly beneficial for improving sleep quality, in that AI-driven chatbots and digital assistants can provide ongoing support and reminders for healthy sleep habits, promoting consistent behavior change over time. However, information disseminated using digital tools must be accurate and reliable.

According to a few studies, the internet can be a useful platform for enhancing sleep-related health literacy, but can also contribute to spreading misinformation, often including commercial biases and incorrect and misleading content. Robbins et al [ 32 ] evaluated the understandability, quality of information, and presence of misinformation in popular YouTube videos about sleep compared to those featuring credible experts. The top YouTube videos on sleep or insomnia and 5 expert-led videos were analyzed for clarity and understanding using established tools. Sleep medicine experts agreed on instances of misinformation and commercial bias, with about 67% (n=14/21) of popular videos showing evidence of commercial bias, unlike the expert videos. Misinformation was more prevalent in popular videos that averaged 8.2 million (SD 2.2) views, significantly higher than the expert videos’ 0.3 million (SD 0.2 million) views. Most YouTube videos were found to have clickbait and be appealing to shorter attention spans, having engaging content, good visual quality, and being highly relatable to viewers. All this highlights the issue of misinformation and bias in widely viewed sleep or insomnia videos on YouTube and other web-based platforms, suggesting the urgent need for combating digital sleep-related misinformation [ 33 ].

ChatGPT is anticipated to play a key role in sleep health promotion and education, enhancing public perceptions of the importance of sleep in daily life and its impact on human health. This analysis demonstrates the potential for AI tools like ChatGPT to provide health information, in particular in the arena of sleep medicine. Considering the overall distribution of responses provided by ChatGPT, a high proportion of sleep-related myths (n=17/20, 85% of the statements) was correctly identified as either false or generally false, suggesting that ChatGPT is aligned with scientific evidence. However, the categorization of some statements as “true” or “not necessarily true or false” indicates ChatGPT’s ability to recognize and label scientific items as accurate can be still improved.

In general, ChatGPT has a good, scholarly understanding of several crucial aspects of sleep health, spanning from sleep duration and timing to behaviors during sleep, while it demonstrates some limitations in the field of sleep hygiene, and in the understanding of sleep-related occupational and public health implications.

Moreover, addressing sleep myths involves a nuanced exploration of sleep-related topics: our qualitative analysis on how common misconceptions are clarified by both AI platforms such as ChatGPT and sleep experts shows a good alignment, though some statements are approached from different angles. From a qualitative comparative perspective, ChatGPT tends to provide more pragmatic advice and tips, emphasizing the importance of regular sleep schedules and practices, even if in the context of a certain degree of flexibility in sleep systems, and the impact of individual behaviors on sleep quality. This approach often includes general recommendations based on a broad understanding of sleep science, aiming to correct misunderstandings such as the notion that less sleep can be habitually sufficient or that lying in bed with eyes closed substitutes for genuine sleep. In contrast, sleep experts delve deeper into the medical and physiological specifics, offering a more detailed assessment that considers individual health conditions, genetic predispositions, and the long-term health risks associated with disrupted sleep patterns. They might focus on the precise effects of sleep deprivation on cognitive function, the specific dangers of certain presleep behaviors, or the complex, nonlinear relationship between sleep stages and overall health. A major difference between ChatGPT and sleep experts is that only the latter have mentioned the public and occupational aspects of sleep, while the former has focused more on the individual level. The dialogue between these perspectives can enrich our understanding of sleep, blending practical guidance with in-depth scientific insights to debunk myths and promote healthier sleep practices across diverse populations.

However, as previously mentioned, some errors by ChatGPT in correctly classifying myths as false statements underscore the current limitations of AI: users should be aware of the shortcomings of AI-based tools in interpreting complex, evolving fields like sleep science and sleep health. ChatGPT’s classifications are not definitive statements of truth but rather reflections of current knowledge and interpretations, which are constantly evolving. In summary, the categorizations by ChatGPT provide an interesting insight into how AI tools process and present information on complex health topics such as sleep, emphasizing the importance of contextual understanding and the ongoing development of AI capabilities in health education.

Implications and Future Directions

ChatGPT’s ability to debunk sleep-related myths has several important implications, both in the field of sleep health and in the context of AI in health care and information dissemination. The ability of ChatGPT to accurately debunk sleep-related myths can significantly contribute to enhancing public health education, including sleep health education. Providing reliable information can help correct widespread misconceptions about sleep, which is vital given the importance of sleep for overall health. ChatGPT can also serve as a tool for supporting health care professionals, helping them to stay abreast of the latest advancements, quickly verify information, and provide evidence-based advice to their patients, potentially improving the quality of sleep health advice given.

Moreover, AI-based platforms such as ChatGPT can make sleep health information more accessible to a broader audience and can offer personalized advice based on individual queries, which is difficult to achieve through traditional health education methods.

This study indicates that AI can be a reliable source of health information. However, it also highlights the need for ongoing evaluation to ensure accuracy, especially in areas with nuanced and complex information, such as sleep health. More in detail, this study suggests that while AI tools such as ChatGPT can be highly effective, they should not replace expert opinion but rather complement it. This is particularly important in complex fields where contextual understanding and professional judgment are crucial. There is a need for continuous learning and updating: AI systems must continuously learn and upgrade their knowledge base to ensure the information they provide stays current with the latest scientific findings and expert consensus.

Moreover, ChatGPT’s ability to identify and correct false information is particularly relevant in an era where misinformation can spread rapidly on the web [ 34 - 36 ]. This capability can play a significant role in public health initiatives. On an individual level, accurate AI-driven advice on sleep health can directly contribute to the prevention of diseases, including sleep-related disorders, which are often linked to chronic diseases such as obesity, diabetes, and cardiovascular issues. By debunking myths and offering personalized guidance on healthy sleep practices, these tools can play a pivotal role in enhancing individual wellness, mental health, and overall quality of life. In the broader context of occupational and public health, provided that the above-mentioned shortcomings of ChatGPT in these fields are properly addressed, the dissemination of reliable sleep-related information via AI platforms can aid in the formulation of more informed public health policies and initiatives. By increasing the general population’s understanding of the importance of sleep, these tools can contribute to a reduction in health care costs associated with sleep disorders and their comorbidities. Furthermore, the implications for public safety are significant. Improved sleep health, guided by AI-based tools, can lead to decreased incidences of accidents and errors attributed to sleep deprivation, such as those in high-risk professions (eg, transportation, health care, etc). This would not only enhance the safety of the individuals in these roles but also safeguard the broader community. Thus, the integration of AI in sleep health education and promotion aligns with broader public health and safety goals, offering a proactive approach to mitigating risks associated with poor sleep and promoting a healthier, safer society.

Finally, this study opens the door for similar applications of AI in other areas of health and wellness, suggesting a potential for AI tools to become more integrated into various aspects of health care delivery, provided that ethical and practical considerations in addressing misinformation and biases are taken into full account. As previously mentioned, there is a need to constantly monitor and improve AI systems to prevent the spread of misinformation and reduce biases in the information provided. Further, in the context of digital health tools, ensuring the privacy and security of user data is paramount, especially when personal health information is involved, underscoring the need for regulatory and ethical oversight in the use of AI in health care to ensure that these tools are used responsibly and for the benefit of individual, occupational, and public health.

Strengths and Limitations

This study has some strengths, including its novelty, methodological rigor, and reproducibility. On the other hand, it suffers from several limitations that should be properly acknowledged: future studies should investigate other AI-based tools, such as Google Bard. Not all digital assistants and chatbots have demonstrated efficacy in improving health- and sleep-related behaviors [ 37 , 38 ]. It should be, indeed, considered that each AI-enhanced platform, being trained on different knowledge bases, has specific technical features and capabilities, and, therefore, some AI-based tools may exhibit lower sleep-related knowledge and literacy, demonstrating less capability of correctly identifying the sleep-related statements as false. As such, this implies that monitoring of the AI system should be tool-specific.

Conclusions

In the present digital era, the synergy of generative conversational AI and sleep health promotion has the potential to positively impact individual, occupational, and public health by providing easy access to evidence-based information and support. This study’s findings demonstrate the potential of AI tools such as ChatGPT in enhancing public health education, particularly in debunking myths and disseminating accurate information related to sleep health. While promising, it is important to use these tools as supplements to, rather than replacements for, sleep expert opinion and to maintain strict standards of accuracy, privacy, and ethical use.

Acknowledgments

Only the text included in Multimedia Appendices 1 and 2 was generated by generative AI.

Data Availability

All data are available within this paper’s text and in Multimedia Appendices 1 - 3 .

Conflicts of Interest

None declared.

ChatGPT full replies to sleep-related false myths in terms of true/false.

ChatGPT full replies to sleep-related false myths in terms of rating on the 5-point Likert scale.

Qualitative comparison between ChatGPT’s and sleep experts’ appraisals of the falseness and public health significance of sleep-related false.

Verbatim transcription from the interaction with ChatGPT about the truthfulness or falseness of twenty sleep-related false myths.

• Mukherjee S, Patel SR, Kales SN, Ayas NT, Strohl KP, Gozal D, et al. American Thoracic Society ad hoc Committee on Healthy Sleep. An official American Thoracic Society statement: the importance of healthy sleep. Recommendations and future priorities. Am J Respir Crit Care Med. 2015;191(12):1450-1458. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grandner MA. Sleep, health, and society. Sleep Med Clin. 2017;12(1):1-22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garbarino S, Lanteri P, Bragazzi NL, Magnavita N, Scoditti E. Role of sleep deprivation in immune-related disease risk and outcomes. Commun Biol. 2021;4(1):1304. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garbarino S, Sannita WG. Poor sleeping has underrepresented medical, healthcare, and social costs? Eur J Intern Med. 2017;38:e15-e16. [ CrossRef ] [ Medline ]
• Garbarino S, Lanteri P, Durando P, Magnavita N, Sannita WG. Co-morbidity, mortality, quality of life and the healthcare/welfare/social costs of disordered sleep: a rapid review. Int J Environ Res Public Health. 2016;13(8):831. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Albakri U, Drotos E, Meertens R. Sleep health promotion interventions and their effectiveness: an umbrella review. Int J Environ Res Public Health. 2021;18(11):5533. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Germain A, Markwald RR, King E, Bramoweth AD, Wolfson M, Seda G, et al. Enhancing behavioral sleep care with digital technology: study protocol for a hybrid type 3 implementation-effectiveness randomized trial. Trials. 2021;22(1):46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chu HC, Liu YM, Kuo FR. A mobile sleep-management learning system for improving students' sleeping habits by integrating a self-regulated learning strategy: randomized controlled trial. JMIR mHealth uHealth. 2018;6(10):e11557. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li H, Zhang R, Lee YC, Kraut RE, Mohr DC. Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being. NPJ Digit Med. 2023;6(1):236. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang Y, Li N, Chen L, Wu M, Meng S, Dai Z, et al. Guidelines, consensus statements, and standards for the use of artificial intelligence in medicine: systematic review. J Med Internet Res. 2023;25:e46089. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bragazzi NL, Guglielmi O, Garbarino S. SleepOMICS: how big data can revolutionize sleep science. Int J Environ Res Public Health. 2019;16(2):291. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alter IL, Chan K, Lechien J, Rameau A. An introduction to machine learning and generative artificial intelligence for otolaryngologists-head and neck surgeons: a narrative review. Eur Arch Otorhinolaryngol. 2024.:1-9. [ CrossRef ] [ Medline ]
• Cheong RCT, Pang KP, Unadkat S, Mcneillis V, Williamson A, Joseph J, et al. Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google Bard. Eur Arch Otorhinolaryngol. 2024;281(4):2137-2143. [ CrossRef ] [ Medline ]
• Bilal M, Jamil Y, Rana D, Shah HH. Enhancing awareness and self-diagnosis of obstructive sleep apnea using AI-powered Chatbots: the role of ChatGPT in revolutionizing healthcare. Ann Biomed Eng. 2024;52(2):136-138. [ CrossRef ] [ Medline ]
• Mira FA, Favier V, Dos Santos Sobreira Nunes H, de Castro JV, Carsuzaa F, Meccariello G, et al. Chat GPT for the management of obstructive sleep apnea: do we have a polar star? Eur Arch Otorhinolaryngol. 2024;281(4):2087-2093. [ CrossRef ] [ Medline ]
• Campbell DJ, Estephan LE, Mastrolonardo EV, Amin DR, Huntley CT, Boon MS. Evaluating ChatGPT responses on obstructive sleep apnea for patient education. J Clin Sleep Med. 2023;19(12):1989-1995. [ CrossRef ] [ Medline ]
• Monteith S, Glenn T, Geddes JR, Whybrow PC, Achtyes E, Bauer M. Artificial intelligence and increasing misinformation. Br J Psychiatry. 2024;224(2):33-35. [ CrossRef ] [ Medline ]
• Robbins R, Grandner MA, Buxton OM, Hale L, Buysse DJ, Knutson KL, et al. Sleep myths: an expert-led study to identify false beliefs about sleep that impinge upon population sleep health practices. Sleep Health. 2019;5(4):409-417. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ong JC. Waking rest: a game changer or a name changer? Sleep. 2019;42(10):zsz172. [ CrossRef ] [ Medline ]
• Humiston GB, Tucker MA, Summer T, Wamsley EJ. Resting states and memory consolidation: a preregistered replication and meta-analysis. Sci Rep. 2019;9(1):19345. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lamp A, Cook M, Smith RNS, Belenky G. Exercise, nutrition, sleep, and waking rest? Sleep. 2019;42(10):zsz138. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wamsley EJ. Memory consolidation during waking rest. Trends Cogn Sci. 2019;23(3):171-173. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tucker MA, Humiston GB, Summer T, Wamsley E. Comparing the effects of sleep and rest on memory consolidation. Nat Sci Sleep. 2020;12:79-91. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wamsley EJ. Offline memory consolidation during waking rest. Nat Rev Psychol. 2022;1(8):441-453. [ CrossRef ]
• Humiston GB, Wamsley EJ. A brief period of eyes-closed rest enhances motor skill consolidation. Neurobiol Learn Mem. 2018;155:1-6. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Richter J, Seffen A, Benedict T, Gast A. No evidence of consolidation of evaluative conditioning during waking rest and sleep. Cogn Emot. 2021;35(5):844-858. [ CrossRef ] [ Medline ]
• Rezaie L, Fobian AD, McCall WV, Khazaie H. Paradoxical insomnia and subjective-objective sleep discrepancy: a review. Sleep Med Rev. 2018;40:196-202. [ CrossRef ] [ Medline ]
• Castelnovo A, Ferri R, Punjabi NM, Castronovo V, Garbazza C, Zucconi M, et al. The paradox of paradoxical insomnia: a theoretical review towards a unifying evidence-based definition. Sleep Med Rev. 2019;44:70-82. [ CrossRef ] [ Medline ]
• Valko PO, Hunziker S, Graf K, Werth E, Baumann CR. Sleep-wake misperception. A comprehensive analysis of a large sleep lab cohort. Sleep Med. 2021;88:96-103. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stephan AM, Siclari F. Reconsidering sleep perception in insomnia: from misperception to mismeasurement. J Sleep Res. 2023;32(6):e14028. [ CrossRef ] [ Medline ]
• Liljequist D, Elfving B, Roaldsen KS. Intraclass correlation—a discussion and demonstration of basic features. PLoS One. 2019;14(7):e0219854. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Robbins R, Epstein LJ, Iyer JM, Weaver MD, Javaheri S, Fashanu O, et al. Examining understandability, information quality, and presence of misinformation in popular YouTube videos on sleep compared to expert-led videos. J Clin Sleep Med. 2023;19(5):991-994. [ CrossRef ] [ Medline ]
• Robbins R. Musings on messages and mediums: how communication and marketing techniques can aid in sleep health promotion. Sleep Health. 2023.:1-4. [ CrossRef ] [ Medline ]
• Gutiérrez-Cirlos C, Carrillo-Pérez DL, Bermúdez-González JL, Hidrogo-Montemayor I, Carrillo-Esper R, Sánchez-Mendiola M. ChatGPT: opportunities and risks in the fields of medical care, teaching, and research. Gac Med Mex. 2023;159(5):372-379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xue J, Zhang B, Zhao Y, Zhang Q, Zheng C, Jiang J, et al. Evaluation of the current state of Chatbots for digital health: scoping review. J Med Internet Res. 2023;25:e47217. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buchanan T. Why do people spread false information online? The effects of message and viewer characteristics on self-reported likelihood of sharing social media disinformation. PLoS One. 2020;15(10):e0239666. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dhinagaran DA, Sathish T, Soong A, Theng YL, Best J, Tudor Car L. Conversational agent for healthy lifestyle behavior change: web-based feasibility study. JMIR Form Res. 2021;5(12):e27956. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Singh B, Olds T, Brinsley J, Dumuid D, Virgara R, Matricciani L, et al. Systematic review and meta-analysis of the effectiveness of chatbots on lifestyle behaviours. NPJ Digit Med. 2023;6(1):118. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 22.12.23; peer-reviewed by H Aghayan Golkashani, Y Chen; comments to author 23.02.24; revised version received 25.02.24; accepted 14.03.24; published 16.04.24.

©Nicola Luigi Bragazzi, Sergio Garbarino. Originally published in JMIR Formative Research (https://formative.jmir.org), 16.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Acta Inform Med
• v.30(1); 2022 Mar

Mixed Methodology of Scientific Research in Healthcare

Emina smajic.

1 Agram Polyclinic, Sarajevo, Bosnia and Herzegovina

Dijana Avdic

2 Faculty of Health Studies, Sarajevo, Bosnia and Herzegovina

Aleksandra Pasic

3 Clinical Biochemistry with Immunology, Sarajevo, Clinical Center University of Sarajevo, Bosnia and Herzegovina

Alden Prcic

4 General Hospital “Prim dr. Abdulah Nakas”, Sarajevo, Bosnia and Herzegovina

Maja Stancic

Background:.

Scientific research is usually classified as quantitative or qualitative. However, methodologists are increasingly emphasizing the integration of qualitative and quantitative data as the center of mixed methods (mix methodologies). Mixed research method implies the use of different research methods, ie. quantitative and qualitative methods in one study.

The aim of this review paper is to present the purpose of using a mixed methodology in health research.

The relevant articles were searched from online data sources including PubMed and Google Scholar.

This approach to the use of mixed methods creates opportunities for a deeper study of various problems. The purpose of using mixed research methods is to obtain valid answers to research questions, however the researcher may still have different reasons or purposes for which he wants to strengthen the research study and its conclusions by applying mixed methods. The use of mixed scientific methodology is widely used in the field of health outcomes and should not be limited to a closed list of possible methodological options.

Conclusion:

Recently, there has been an increase in the number of scientific studies in healthcare that use mixed research methods. The advantage of applying this scientific method is that through the triangulation of data obtained by different (quantitative / qualitative) approaches, we get a deeper and more complete picture of the phenomenon in health care that we observe.

1. BACKGROUND

In health science research, there is a priority to develop new methodologies to improve the quality and scientific strength of data leading to an extraordinary increase in methodological diversity. This diversity reflects the nature of public health problems, such as differences between populations, age groups, ethnic groups and cultures, poor adherence to treatments considered effective, behavioral factors contributing to disability and health, and translational needs for health research. Diversity also signals a growing acceptance of qualitative and social science research, the formation of interdisciplinary research teams, and the use of multilevel approaches to research complex health issues such as patient attitudes and cultural and social models of disease and health (1) .

2. OBJECTIVE

The retrieved articles were reviewed by the authors and the results are presented along with the relevant discussion

4.1. MIXED METHODOLOGY

Scientific research is usually classified as quantitative or qualitative. However, methodologists are increasingly emphasizing the integration of qualitative and quantitative data as the center of mixed methods (mix methodologies). Integration is a deliberate process by which the researcher combines quantitative and qualitative approaches in the study. Quantitative and qualitative data then become interdependent in solving questions and hypotheses. Mixed research method implies the use of different research methods, quantitative and qualitative methods in one study ( 2 , 3 ). Research on mixed methods should be distinguished from multi-method research (method-combination) in which either multiple qualitative approaches or only multiple quantitative approaches are combined (4) .

The most accepted definition of a mixed research method is that it is a research in which a researcher or team of researchers combines elements of a qualitative and quantitative approach to research (use of qualitative and quantitative perspectives, data collection, analysis, inference techniques) to understand and support research. As we see in the definition, the use of both quantitative and qualitative methods in a single study (or series of related studies) is crucial, unlike the use of combined methods that combine two or more quantitative or two or more qualitative research methods ( 2 , 4 ).

The basic premise of using mixed research methods is that some research issues can be addressed more comprehensively than using either quantitative or qualitative methods. The issues that benefit most from the design of mixed methods tend to be broad and complex, with multiple aspects that each can have. Mixed research methods can exploit the strengths and weaknesses of both approaches and can be particularly useful when addressing complex, multifaceted issues such as health service interventions and living with chronic diseases (2) . This approach to the use of mixed methods creates opportunities for a deeper study of various problems (5) .

4.2. PURPOSE OF USING MIXED RESEARCH

The purpose of using mixed research methods is to obtain valid answers to research questions, however the researcher may still have different reasons or purposes for which he wants to strengthen the research study and its conclusions by applying mixed methods. The purpose classification of mixed research methods was first introduced in 1989 by Greene, Caracelli, and Graham, based on an analysis of published studies of mixed methods. This classification is still used and we have a total of five “purposes” for why a mixed methodology is used in research (4) . Classification of the purpose of using mixed methodology:

• Complementarity. Using data obtained from one method to illustrate the results of another method.
• Development. Using the results of one method to develop or inform about the use of another method.
• Initiation. We can use the results of different methods to search for areas of non-compliance in certain areas in order to create new insights.
• Expansion. Examining different aspects of a research question, with each aspect justifying different methods.
• Triangulation. Use of data obtained by both methods to support the findings (2) .

In the last 28 years, this classification has been supplemented by several other authors. So in 2006 Bryman compiled a list of more specific rationales for the use or purpose of mixed research methods. Bryman’s classification decomposes the categorization of Greene et al. on several aspects and adds a number of additional aspects (3) .

Bryman’s addition to the classification of Greene et al.

Credibility. It refers to suggestions that the application of both methodological approaches improves the integrity of the results.

Context. Refers to cases where the combination is justified in terms of qualitative research that provides contextual understanding, along with generalized, externally valid results, or broad relationships between the variables identified by the survey.

Illustration. It refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bone” “dry” quantitative findings.

Usefulness or improvement of usefulness of results. It refers to the suggestion, which is more likely to be highlighted among articles with an applied focus, that combining the two approaches will be more useful for practitioners.

Confirmation and discovery. It involves the use of qualitative data to generate hypotheses and the use of quantitative research to test them within a single project.

Diversity of views. This includes two slightly different explanations–namely, combining the perspective of researchers and participants through quantitative and qualitative research, and discovering the relationship between variables through quantitative research, while revealing meanings among research participants through qualitative research (3) .

4.3. RESEARCH DESIGN

Research designs are procedures for collecting, analyzing, interpreting, and reporting data in research studies. They represent different models for doing research, and these models have distinct names and procedures associated with them (6) .

The four major types of mixed methods designs are the Triangulation Design, the Embedded Design, the Explanatory Design, and the Exploratory Design (6) .

4.4. HEALTH APPLICATION

There is a wide range of methods used to collect both quantitative and qualitative data. And the research question and the necessary data are the main determinants of the methods used. To a lesser extent, the choice of methods may be influenced by feasibility. Method priority refers to the emphasis on each method in the study. For example, a study may be predominantly quantitative with a small qualitative component or vice versa. Alternatively, both quantitative and qualitative methods and data may be equally weighted. The emphasis on each methodological component of the study will be driven mainly by the research question, research team skills and feasibility. Finally, researchers must decide when each method will be used in research (2) . By analyzing the research methods and research designs used, Bryman suggests that on the quantitative side, structured interview and questionnaire research within cross-sectional design predominates, while on the qualitative side, semi-structured interviews within cross-sectional design predominate (7) . A key feature of mixed-method research is its methodological pluralism, which often results in research that provides broader perspectives than those offered by monomethodal designs (8) .

The use of mixed scientific methodology is widely used in the field of health outcomes and should not be limited to a closed list of possible methodological options, but should be seen as a framework for a specific research issue to be addressed using quantitative and qualitative components (data and / or methods) , when quantitative and qualitative components are articulated intentionally and prospectively in a well-defined, pre-specified research design and as a framework for meta-inference (9) .

The importance of applying mixed research methods can be reflected in the trend of “measuring / analysing what is important” for patients and the treatment outcomes reported by patients are increasingly used in clinical care and research. However, a recent review of studies documenting the development of outcome measures reported by patients highlights that only 11% of them were developed actually asking patients which outcomes are important to them. This emphasizes the importance of applying mixed qualitative and quantitative methods in health research to ensure a focus on the priorities identified by the patient, scientific rigor, and improved patient outcomes (9) . Mixed methods are also an increasingly accepted approach used to investigate organizational phenomena in health care (10) .

The application of the mix methodology is considered a significant contribution to health science. By combining quantitative and qualitative data in the same study, health researchers can reap the benefits of each approach while minimizing their disadvantages. In practice, this endeavor facilitates research by health science researchers on the complex and multifactorial nature of human health and disease. Researchers using the mixed method approach for the first time can easily feel overwhelmed by uncertainty about the philosophical foundations of the method, as well as the multitude of typologies of mixed method research. Although further research and debate are warranted, health researchers seeking solutions to real problems are increasingly embracing pragmatism as a paradigm of choice (11) .

Author’s contribution:

All authors were involved in preparation of this article. Final proofreading was made by first author.

Financial support and sponsorship:

Conflicts of interest:.

There are no conflicts of interest.