research methodology in medical education

Advertisement

• Previous Issue
• Previous Article
• Next Article

Clarifying the Research Purpose

Methodology, measurement, data analysis and interpretation, tools for evaluating the quality of medical education research, research support, competing interests, quantitative research methods in medical education.

Submitted for publication January 8, 2018. Accepted for publication November 29, 2018.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

John T. Ratelle , Adam P. Sawatsky , Thomas J. Beckman; Quantitative Research Methods in Medical Education. Anesthesiology 2019; 131:23–35 doi: https://doi.org/10.1097/ALN.0000000000002727

Download citation file:

• Ris (Zotero)
• Reference Manager

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Image: J. P. Rathmell and Terri Navarette.

There has been an explosion of research in the field of medical education. A search of PubMed demonstrates that more than 40,000 articles have been indexed under the medical subject heading “Medical Education” since 2010, which is more than the total number of articles indexed under this heading in the 1980s and 1990s combined. Keeping up to date requires that practicing clinicians have the skills to interpret and appraise the quality of research articles, especially when serving as editors, reviewers, and consumers of the literature.

While medical education shares many characteristics with other biomedical fields, substantial particularities exist. We recognize that practicing clinicians may not be familiar with the nuances of education research and how to assess its quality. Therefore, our purpose is to provide a review of quantitative research methodologies in medical education. Specifically, we describe a structure that can be used when conducting or evaluating medical education research articles.

Clarifying the research purpose is an essential first step when reading or conducting scholarship in medical education. 1   Medical education research can serve a variety of purposes, from advancing the science of learning to improving the outcomes of medical trainees and the patients they care for. However, a well-designed study has limited value if it addresses vague, redundant, or unimportant medical education research questions.

What is the research topic and why is it important? What is unknown about the research topic? Why is further research necessary?

What is the conceptual framework being used to approach the study?

What is the statement of study intent?

What are the research methodology and study design? Are they appropriate for the study objective(s)?

Which threats to internal validity are most relevant for the study?

What is the outcome and how was it measured?

Can the results be trusted? What is the validity and reliability of the measurements?

How were research subjects selected? Is the research sample representative of the target population?

Was the data analysis appropriate for the study design and type of data?

What is the effect size? Do the results have educational significance?

Fortunately, there are steps to ensure that the purpose of a research study is clear and logical. Table 1   2–5   outlines these steps, which will be described in detail in the following sections. We describe these elements not as a simple “checklist,” but as an advanced organizer that can be used to understand a medical education research study. These steps can also be used by clinician educators who are new to the field of education research and who wish to conduct scholarship in medical education.

Steps in Clarifying the Purpose of a Research Study in Medical Education

Literature Review and Problem Statement

A literature review is the first step in clarifying the purpose of a medical education research article. 2 , 5 , 6   When conducting scholarship in medical education, a literature review helps researchers develop an understanding of their topic of interest. This understanding includes both existing knowledge about the topic as well as key gaps in the literature, which aids the researcher in refining their study question. Additionally, a literature review helps researchers identify conceptual frameworks that have been used to approach the research topic. 2  

When reading scholarship in medical education, a successful literature review provides background information so that even someone unfamiliar with the research topic can understand the rationale for the study. Located in the introduction of the manuscript, the literature review guides the reader through what is already known in a manner that highlights the importance of the research topic. The literature review should also identify key gaps in the literature so the reader can understand the need for further research. This gap description includes an explicit problem statement that summarizes the important issues and provides a reason for the study. 2 , 4   The following is one example of a problem statement:

“Identifying gaps in the competency of anesthesia residents in time for intervention is critical to patient safety and an effective learning system… [However], few available instruments relate to complex behavioral performance or provide descriptors…that could inform subsequent feedback, individualized teaching, remediation, and curriculum revision.” 7  

This problem statement articulates the research topic (identifying resident performance gaps), why it is important (to intervene for the sake of learning and patient safety), and current gaps in the literature (few tools are available to assess resident performance). The researchers have now underscored why further research is needed and have helped readers anticipate the overarching goals of their study (to develop an instrument to measure anesthesiology resident performance). 4  

The Conceptual Framework

Following the literature review and articulation of the problem statement, the next step in clarifying the research purpose is to select a conceptual framework that can be applied to the research topic. Conceptual frameworks are “ways of thinking about a problem or a study, or ways of representing how complex things work.” 3   Just as clinical trials are informed by basic science research in the laboratory, conceptual frameworks often serve as the “basic science” that informs scholarship in medical education. At a fundamental level, conceptual frameworks provide a structured approach to solving the problem identified in the problem statement.

Conceptual frameworks may take the form of theories, principles, or models that help to explain the research problem by identifying its essential variables or elements. Alternatively, conceptual frameworks may represent evidence-based best practices that researchers can apply to an issue identified in the problem statement. 3   Importantly, there is no single best conceptual framework for a particular research topic, although the choice of a conceptual framework is often informed by the literature review and knowing which conceptual frameworks have been used in similar research. 8   For further information on selecting a conceptual framework for research in medical education, we direct readers to the work of Bordage 3   and Irby et al. 9  

To illustrate how different conceptual frameworks can be applied to a research problem, suppose you encounter a study to reduce the frequency of communication errors among anesthesiology residents during day-to-night handoff. Table 2 10 , 11   identifies two different conceptual frameworks researchers might use to approach the task. The first framework, cognitive load theory, has been proposed as a conceptual framework to identify potential variables that may lead to handoff errors. 12   Specifically, cognitive load theory identifies the three factors that affect short-term memory and thus may lead to communication errors:

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Intrinsic load: Inherent complexity or difficulty of the information the resident is trying to learn ( e.g. , complex patients).

Extraneous load: Distractions or demands on short-term memory that are not related to the information the resident is trying to learn ( e.g. , background noise, interruptions).

Germane load: Effort or mental strategies used by the resident to organize and understand the information he/she is trying to learn ( e.g. , teach back, note taking).

Using cognitive load theory as a conceptual framework, researchers may design an intervention to reduce extraneous load and help the resident remember the overnight to-do’s. An example might be dedicated, pager-free handoff times where distractions are minimized.

The second framework identified in table 2 , the I-PASS (Illness severity, Patient summary, Action list, Situational awareness and contingency planning, and Synthesis by receiver) handoff mnemonic, 11   is an evidence-based best practice that, when incorporated as part of a handoff bundle, has been shown to reduce handoff errors on pediatric wards. 13   Researchers choosing this conceptual framework may adapt some or all of the I-PASS elements for resident handoffs in the intensive care unit.

Note that both of the conceptual frameworks outlined above provide researchers with a structured approach to addressing the issue of handoff errors; one is not necessarily better than the other. Indeed, it is possible for researchers to use both frameworks when designing their study. Ultimately, we provide this example to demonstrate the necessity of selecting conceptual frameworks to clarify the research purpose. 3 , 8   Readers should look for conceptual frameworks in the introduction section and should be wary of their omission, as commonly seen in less well-developed medical education research articles. 14  

Statement of Study Intent

After reviewing the literature, articulating the problem statement, and selecting a conceptual framework to address the research topic, the final step in clarifying the research purpose is the statement of study intent. The statement of study intent is arguably the most important element of framing the study because it makes the research purpose explicit. 2   Consider the following example:

This study aimed to test the hypothesis that the introduction of the BASIC Examination was associated with an accelerated knowledge acquisition during residency training, as measured by increments in annual ITE scores. 15  

This statement of study intent succinctly identifies several key study elements including the population (anesthesiology residents), the intervention/independent variable (introduction of the BASIC Examination), the outcome/dependent variable (knowledge acquisition, as measure by in In-training Examination [ITE] scores), and the hypothesized relationship between the independent and dependent variable (the authors hypothesize a positive correlation between the BASIC examination and the speed of knowledge acquisition). 6 , 14  

The statement of study intent will sometimes manifest as a research objective, rather than hypothesis or question. In such instances there may not be explicit independent and dependent variables, but the study population and research aim should be clearly identified. The following is an example:

“In this report, we present the results of 3 [years] of course data with respect to the practice improvements proposed by participating anesthesiologists and their success in implementing those plans. Specifically, our primary aim is to assess the frequency and type of improvements that were completed and any factors that influence completion.” 16  

The statement of study intent is the logical culmination of the literature review, problem statement, and conceptual framework, and is a transition point between the Introduction and Methods sections of a medical education research report. Nonetheless, a systematic review of experimental research in medical education demonstrated that statements of study intent are absent in the majority of articles. 14   When reading a medical education research article where the statement of study intent is absent, it may be necessary to infer the research aim by gathering information from the Introduction and Methods sections. In these cases, it can be useful to identify the following key elements 6 , 14 , 17   :

Population of interest/type of learner ( e.g. , pain medicine fellow or anesthesiology residents)

Independent/predictor variable ( e.g. , educational intervention or characteristic of the learners)

Dependent/outcome variable ( e.g. , intubation skills or knowledge of anesthetic agents)

Relationship between the variables ( e.g. , “improve” or “mitigate”)

Occasionally, it may be difficult to differentiate the independent study variable from the dependent study variable. 17   For example, consider a study aiming to measure the relationship between burnout and personal debt among anesthesiology residents. Do the researchers believe burnout might lead to high personal debt, or that high personal debt may lead to burnout? This “chicken or egg” conundrum reinforces the importance of the conceptual framework which, if present, should serve as an explanation or rationale for the predicted relationship between study variables.

Research methodology is the “…design or plan that shapes the methods to be used in a study.” 1   Essentially, methodology is the general strategy for answering a research question, whereas methods are the specific steps and techniques that are used to collect data and implement the strategy. Our objective here is to provide an overview of quantitative methodologies ( i.e. , approaches) in medical education research.

The choice of research methodology is made by balancing the approach that best answers the research question against the feasibility of completing the study. There is no perfect methodology because each has its own potential caveats, flaws and/or sources of bias. Before delving into an overview of the methodologies, it is important to highlight common sources of bias in education research. We use the term internal validity to describe the degree to which the findings of a research study represent “the truth,” as opposed to some alternative hypothesis or variables. 18   Table 3   18–20   provides a list of common threats to internal validity in medical education research, along with tactics to mitigate these threats.

Threats to Internal Validity and Strategies to Mitigate Their Effects

Experimental Research

The fundamental tenet of experimental research is the manipulation of an independent or experimental variable to measure its effect on a dependent or outcome variable.

True Experiment

True experimental study designs minimize threats to internal validity by randomizing study subjects to experimental and control groups. Through ensuring that differences between groups are—beyond the intervention/variable of interest—purely due to chance, researchers reduce the internal validity threats related to subject characteristics, time-related maturation, and regression to the mean. 18 , 19  

Quasi-experiment

There are many instances in medical education where randomization may not be feasible or ethical. For instance, researchers wanting to test the effect of a new curriculum among medical students may not be able to randomize learners due to competing curricular obligations and schedules. In these cases, researchers may be forced to assign subjects to experimental and control groups based upon some other criterion beyond randomization, such as different classrooms or different sections of the same course. This process, called quasi-randomization, does not inherently lead to internal validity threats, as long as research investigators are mindful of measuring and controlling for extraneous variables between study groups. 19  

Single-group Methodologies

All experimental study designs compare two or more groups: experimental and control. A common experimental study design in medical education research is the single-group pretest–posttest design, which compares a group of learners before and after the implementation of an intervention. 21   In essence, a single-group pre–post design compares an experimental group ( i.e. , postintervention) to a “no-intervention” control group ( i.e. , preintervention). 19   This study design is problematic for several reasons. Consider the following hypothetical example: A research article reports the effects of a year-long intubation curriculum for first-year anesthesiology residents. All residents participate in monthly, half-day workshops over the course of an academic year. The article reports a positive effect on residents’ skills as demonstrated by a significant improvement in intubation success rates at the end of the year when compared to the beginning.

This study does little to advance the science of learning among anesthesiology residents. While this hypothetical report demonstrates an improvement in residents’ intubation success before versus after the intervention, it does not tell why the workshop worked, how it compares to other educational interventions, or how it fits in to the broader picture of anesthesia training.

Single-group pre–post study designs open themselves to a myriad of threats to internal validity. 20   In our hypothetical example, the improvement in residents’ intubation skills may have been due to other educational experience(s) ( i.e. , implementation threat) and/or improvement in manual dexterity that occurred naturally with time ( i.e. , maturation threat), rather than the airway curriculum. Consequently, single-group pre–post studies should be interpreted with caution. 18  

Repeated testing, before and after the intervention, is one strategy that can be used to reduce the some of the inherent limitations of the single-group study design. Repeated pretesting can mitigate the effect of regression toward the mean, a statistical phenomenon whereby low pretest scores tend to move closer to the mean on subsequent testing (regardless of intervention). 20   Likewise, repeated posttesting at multiple time intervals can provide potentially useful information about the short- and long-term effects of an intervention ( e.g. , the “durability” of the gain in knowledge, skill, or attitude).

Observational Research

Unlike experimental studies, observational research does not involve manipulation of any variables. These studies often involve measuring associations, developing psychometric instruments, or conducting surveys.

Association Research

Association research seeks to identify relationships between two or more variables within a group or groups (correlational research), or similarities/differences between two or more existing groups (causal–comparative research). For example, correlational research might seek to measure the relationship between burnout and educational debt among anesthesiology residents, while causal–comparative research may seek to measure differences in educational debt and/or burnout between anesthesiology and surgery residents. Notably, association research may identify relationships between variables, but does not necessarily support a causal relationship between them.

Psychometric and Survey Research

Psychometric instruments measure a psychologic or cognitive construct such as knowledge, satisfaction, beliefs, and symptoms. Surveys are one type of psychometric instrument, but many other types exist, such as evaluations of direct observation, written examinations, or screening tools. 22   Psychometric instruments are ubiquitous in medical education research and can be used to describe a trait within a study population ( e.g. , rates of depression among medical students) or to measure associations between study variables ( e.g. , association between depression and board scores among medical students).

Psychometric and survey research studies are prone to the internal validity threats listed in table 3 , particularly those relating to mortality, location, and instrumentation. 18   Additionally, readers must ensure that the instrument scores can be trusted to truly represent the construct being measured. For example, suppose you encounter a research article demonstrating a positive association between attending physician teaching effectiveness as measured by a survey of medical students, and the frequency with which the attending physician provides coffee and doughnuts on rounds. Can we be confident that this survey administered to medical students is truly measuring teaching effectiveness? Or is it simply measuring the attending physician’s “likability”? Issues related to measurement and the trustworthiness of data are described in detail in the following section on measurement and the related issues of validity and reliability.

Measurement refers to “the assigning of numbers to individuals in a systematic way as a means of representing properties of the individuals.” 23   Research data can only be trusted insofar as we trust the measurement used to obtain the data. Measurement is of particular importance in medical education research because many of the constructs being measured ( e.g. , knowledge, skill, attitudes) are abstract and subject to measurement error. 24   This section highlights two specific issues related to the trustworthiness of data: the validity and reliability of measurements.

Validity regarding the scores of a measurement instrument “refers to the degree to which evidence and theory support the interpretations of the [instrument’s results] for the proposed use of the [instrument].” 25   In essence, do we believe the results obtained from a measurement really represent what we were trying to measure? Note that validity evidence for the scores of a measurement instrument is separate from the internal validity of a research study. Several frameworks for validity evidence exist. Table 4 2 , 22 , 26   represents the most commonly used framework, developed by Messick, 27   which identifies sources of validity evidence—to support the target construct—from five main categories: content, response process, internal structure, relations to other variables, and consequences.

Sources of Validity Evidence for Measurement Instruments

Reliability

Reliability refers to the consistency of scores for a measurement instrument. 22 , 25 , 28   For an instrument to be reliable, we would anticipate that two individuals rating the same object of measurement in a specific context would provide the same scores. 25   Further, if the scores for an instrument are reliable between raters of the same object of measurement, then we can extrapolate that any difference in scores between two objects represents a true difference across the sample, and is not due to random variation in measurement. 29   Reliability can be demonstrated through a variety of methods such as internal consistency ( e.g. , Cronbach’s alpha), temporal stability ( e.g. , test–retest reliability), interrater agreement ( e.g. , intraclass correlation coefficient), and generalizability theory (generalizability coefficient). 22 , 29  

Example of a Validity and Reliability Argument

This section provides an illustration of validity and reliability in medical education. We use the signaling questions outlined in table 4 to make a validity and reliability argument for the Harvard Assessment of Anesthesia Resident Performance (HARP) instrument. 7   The HARP was developed by Blum et al. to measure the performance of anesthesia trainees that is required to provide safe anesthetic care to patients. According to the authors, the HARP is designed to be used “…as part of a multiscenario, simulation-based assessment” of resident performance. 7  

Content Validity: Does the Instrument’s Content Represent the Construct Being Measured?

To demonstrate content validity, instrument developers should describe the construct being measured and how the instrument was developed, and justify their approach. 25   The HARP is intended to measure resident performance in the critical domains required to provide safe anesthetic care. As such, investigators note that the HARP items were created through a two-step process. First, the instrument’s developers interviewed anesthesiologists with experience in resident education to identify the key traits needed for successful completion of anesthesia residency training. Second, the authors used a modified Delphi process to synthesize the responses into five key behaviors: (1) formulate a clear anesthetic plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize one’s limits. 7 , 30  

Response Process Validity: Are Raters Interpreting the Instrument Items as Intended?

In the case of the HARP, the developers included a scoring rubric with behavioral anchors to ensure that faculty raters could clearly identify how resident performance in each domain should be scored. 7  

Internal Structure Validity: Do Instrument Items Measuring Similar Constructs Yield Homogenous Results? Do Instrument Items Measuring Different Constructs Yield Heterogeneous Results?

Item-correlation for the HARP demonstrated a high degree of correlation between some items ( e.g. , formulating a plan and modifying the plan under changing conditions) and a lower degree of correlation between other items ( e.g. , formulating a plan and identifying performance improvement opportunities). 30   This finding is expected since the items within the HARP are designed to assess separate performance domains, and we would expect residents’ functioning to vary across domains.

Relationship to Other Variables’ Validity: Do Instrument Scores Correlate with Other Measures of Similar or Different Constructs as Expected?

As it applies to the HARP, one would expect that the performance of anesthesia residents will improve over the course of training. Indeed, HARP scores were found to be generally higher among third-year residents compared to first-year residents. 30  

Consequence Validity: Are Instrument Results Being Used as Intended? Are There Unintended or Negative Uses of the Instrument Results?

While investigators did not intentionally seek out consequence validity evidence for the HARP, unanticipated consequences of HARP scores were identified by the authors as follows:

“Data indicated that CA-3s had a lower percentage of worrisome scores (rating 2 or lower) than CA-1s… However, it is concerning that any CA-3s had any worrisome scores…low performance of some CA-3 residents, albeit in the simulated environment, suggests opportunities for training improvement.” 30  

That is, using the HARP to measure the performance of CA-3 anesthesia residents had the unintended consequence of identifying the need for improvement in resident training.

Reliability: Are the Instrument’s Scores Reproducible and Consistent between Raters?

The HARP was applied by two raters for every resident in the study across seven different simulation scenarios. The investigators conducted a generalizability study of HARP scores to estimate the variance in assessment scores that was due to the resident, the rater, and the scenario. They found little variance was due to the rater ( i.e. , scores were consistent between raters), indicating a high level of reliability. 7  

Sampling refers to the selection of research subjects ( i.e. , the sample) from a larger group of eligible individuals ( i.e. , the population). 31   Effective sampling leads to the inclusion of research subjects who represent the larger population of interest. Alternatively, ineffective sampling may lead to the selection of research subjects who are significantly different from the target population. Imagine that researchers want to explore the relationship between burnout and educational debt among pain medicine specialists. The researchers distribute a survey to 1,000 pain medicine specialists (the population), but only 300 individuals complete the survey (the sample). This result is problematic because the characteristics of those individuals who completed the survey and the entire population of pain medicine specialists may be fundamentally different. It is possible that the 300 study subjects may be experiencing more burnout and/or debt, and thus, were more motivated to complete the survey. Alternatively, the 700 nonresponders might have been too busy to respond and even more burned out than the 300 responders, which would suggest that the study findings were even more amplified than actually observed.

When evaluating a medical education research article, it is important to identify the sampling technique the researchers employed, how it might have influenced the results, and whether the results apply to the target population. 24  

Sampling Techniques

Sampling techniques generally fall into two categories: probability- or nonprobability-based. Probability-based sampling ensures that each individual within the target population has an equal opportunity of being selected as a research subject. Most commonly, this is done through random sampling, which should lead to a sample of research subjects that is similar to the target population. If significant differences between sample and population exist, those differences should be due to random chance, rather than systematic bias. The difference between data from a random sample and that from the population is referred to as sampling error. 24  

Nonprobability-based sampling involves selecting research participants such that inclusion of some individuals may be more likely than the inclusion of others. 31   Convenience sampling is one such example and involves selection of research subjects based upon ease or opportuneness. Convenience sampling is common in medical education research, but, as outlined in the example at the beginning of this section, it can lead to sampling bias. 24   When evaluating an article that uses nonprobability-based sampling, it is important to look for participation/response rate. In general, a participation rate of less than 75% should be viewed with skepticism. 21   Additionally, it is important to determine whether characteristics of participants and nonparticipants were reported and if significant differences between the two groups exist.

Interpreting medical education research requires a basic understanding of common ways in which quantitative data are analyzed and displayed. In this section, we highlight two broad topics that are of particular importance when evaluating research articles.

The Nature of the Measurement Variable

Measurement variables in quantitative research generally fall into three categories: nominal, ordinal, or interval. 24   Nominal variables (sometimes called categorical variables) involve data that can be placed into discrete categories without a specific order or structure. Examples include sex (male or female) and professional degree (M.D., D.O., M.B.B.S., etc .) where there is no clear hierarchical order to the categories. Ordinal variables can be ranked according to some criterion, but the spacing between categories may not be equal. Examples of ordinal variables may include measurements of satisfaction (satisfied vs . unsatisfied), agreement (disagree vs . agree), and educational experience (medical student, resident, fellow). As it applies to educational experience, it is noteworthy that even though education can be quantified in years, the spacing between years ( i.e. , educational “growth”) remains unequal. For instance, the difference in performance between second- and third-year medical students is dramatically different than third- and fourth-year medical students. Interval variables can also be ranked according to some criteria, but, unlike ordinal variables, the spacing between variable categories is equal. Examples of interval variables include test scores and salary. However, the conceptual boundaries between these measurement variables are not always clear, as in the case where ordinal scales can be assumed to have the properties of an interval scale, so long as the data’s distribution is not substantially skewed. 32  

Understanding the nature of the measurement variable is important when evaluating how the data are analyzed and reported. Medical education research commonly uses measurement instruments with items that are rated on Likert-type scales, whereby the respondent is asked to assess their level of agreement with a given statement. The response is often translated into a corresponding number ( e.g. , 1 = strongly disagree, 3 = neutral, 5 = strongly agree). It is remarkable that scores from Likert-type scales are sometimes not normally distributed ( i.e. , are skewed toward one end of the scale), indicating that the spacing between scores is unequal and the variable is ordinal in nature. In these cases, it is recommended to report results as frequencies or medians, rather than means and SDs. 33  

Consider an article evaluating medical students’ satisfaction with a new curriculum. Researchers measure satisfaction using a Likert-type scale (1 = very unsatisfied, 2 = unsatisfied, 3 = neutral, 4 = satisfied, 5 = very satisfied). A total of 20 medical students evaluate the curriculum, 10 of whom rate their satisfaction as “satisfied,” and 10 of whom rate it as “very satisfied.” In this case, it does not make much sense to report an average score of 4.5; it makes more sense to report results in terms of frequency ( e.g. , half of the students were “very satisfied” with the curriculum, and half were not).

Effect Size and CIs

In medical education, as in other research disciplines, it is common to report statistically significant results ( i.e. , small P values) in order to increase the likelihood of publication. 34 , 35   However, a significant P value in itself does necessarily represent the educational impact of the study results. A statement like “Intervention x was associated with a significant improvement in learners’ intubation skill compared to education intervention y ( P < 0.05)” tells us that there was a less than 5% chance that the difference in improvement between interventions x and y was due to chance. Yet that does not mean that the study intervention necessarily caused the nonchance results, or indicate whether the between-group difference is educationally significant. Therefore, readers should consider looking beyond the P value to effect size and/or CI when interpreting the study results. 36 , 37  

Effect size is “the magnitude of the difference between two groups,” which helps to quantify the educational significance of the research results. 37   Common measures of effect size include Cohen’s d (standardized difference between two means), risk ratio (compares binary outcomes between two groups), and Pearson’s r correlation (linear relationship between two continuous variables). 37   CIs represent “a range of values around a sample mean or proportion” and are a measure of precision. 31   While effect size and CI give more useful information than simple statistical significance, they are commonly omitted from medical education research articles. 35   In such instances, readers should be wary of overinterpreting a P value in isolation. For further information effect size and CI, we direct readers the work of Sullivan and Feinn 37   and Hulley et al. 31  

In this final section, we identify instruments that can be used to evaluate the quality of quantitative medical education research articles. To this point, we have focused on framing the study and research methodologies and identifying potential pitfalls to consider when appraising a specific article. This is important because how a study is framed and the choice of methodology require some subjective interpretation. Fortunately, there are several instruments available for evaluating medical education research methods and providing a structured approach to the evaluation process.

The Medical Education Research Study Quality Instrument (MERSQI) 21   and the Newcastle Ottawa Scale-Education (NOS-E) 38   are two commonly used instruments, both of which have an extensive body of validity evidence to support the interpretation of their scores. Table 5 21 , 39   provides more detail regarding the MERSQI, which includes evaluation of study design, sampling, data type, validity, data analysis, and outcomes. We have found that applying the MERSQI to manuscripts, articles, and protocols has intrinsic educational value, because this practice of application familiarizes MERSQI users with fundamental principles of medical education research. One aspect of the MERSQI that deserves special mention is the section on evaluating outcomes based on Kirkpatrick’s widely recognized hierarchy of reaction, learning, behavior, and results ( table 5 ; fig .). 40   Validity evidence for the scores of the MERSQI include its operational definitions to improve response process, excellent reliability, and internal consistency, as well as high correlation with other measures of study quality, likelihood of publication, citation rate, and an association between MERSQI score and the likelihood of study funding. 21 , 41   Additionally, consequence validity for the MERSQI scores has been demonstrated by its utility for identifying and disseminating high-quality research in medical education. 42  

Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007. 2  

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The NOS-E is a newer tool to evaluate the quality of medication education research. It was developed as a modification of the Newcastle-Ottawa Scale 43   for appraising the quality of nonrandomized studies. The NOS-E includes items focusing on the representativeness of the experimental group, selection and compatibility of the control group, missing data/study retention, and blinding of outcome assessors. 38 , 39   Additional validity evidence for NOS-E scores includes operational definitions to improve response process, excellent reliability and internal consistency, and its correlation with other measures of study quality. 39   Notably, the complete NOS-E, along with its scoring rubric, can found in the article by Cook and Reed. 39  

A recent comparison of the MERSQI and NOS-E found acceptable interrater reliability and good correlation between the two instruments 39   However, noted differences exist between the MERSQI and NOS-E. Specifically, the MERSQI may be applied to a broad range of study designs, including experimental and cross-sectional research. Additionally, the MERSQI addresses issues related to measurement validity and data analysis, and places emphasis on educational outcomes. On the other hand, the NOS-E focuses specifically on experimental study designs, and on issues related to sampling techniques and outcome assessment. 39   Ultimately, the MERSQI and NOS-E are complementary tools that may be used together when evaluating the quality of medical education research.

Conclusions

This article provides an overview of quantitative research in medical education, underscores the main components of education research, and provides a general framework for evaluating research quality. We highlighted the importance of framing a study with respect to purpose, conceptual framework, and statement of study intent. We reviewed the most common research methodologies, along with threats to the validity of a study and its measurement instruments. Finally, we identified two complementary instruments, the MERSQI and NOS-E, for evaluating the quality of a medical education research study.

Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312–9.

Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7–166. e116.

Franenkel JR, Wallen NE, Hyun HH: How to Design and Evaluate Research in Education. 9th edition. New York, McGraw-Hill Education, 2015.

Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research. 4th edition. Philadelphia, Lippincott Williams & Wilkins, 2011.

Irby BJ, Brown G, Lara-Alecio R, Jackson S: The Handbook of Educational Theories. Charlotte, NC, Information Age Publishing, Inc., 2015

Standards for Educational and Psychological Testing (American Educational Research Association & American Psychological Association, 2014)

Swanwick T: Understanding medical education: Evidence, theory and practice, 2nd edition. Wiley-Blackwell, 2013.

Sullivan GM, Artino Jr AR: Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013; 5(4):541–2.

Sullivan GM, Feinn R: Using effect size—or why the P value is not enough. Journal of graduate medical education. 2012; 4(3):279–82.

Tavakol M, Sandars J: Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical teacher. 2014; 36(10):838–48.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

Review article: medical education research: an overview of methods

Affiliation.

• 1 Department of Anesthesiology of The Ottawa Hospital & The University of Ottawa Skills and Simulation Centre, University of Ottawa, Ottawa, ON, Canada.
• PMID: 22215522
• DOI: 10.1007/s12630-011-9635-y

Purpose: This article provides clinician-teachers with an overview of the process necessary to move from an initial idea to the conceptualization and implementation of an empirical study in the field of medical education. This article will allow clinician-teachers to become familiar with educational research methodology in order to a) critically appraise education research studies and apply evidence-based education more effectively to their practice and b) initiate or collaborate in medical education research.

Source: This review uses relevant articles published in the fields of medicine, education, psychology, and sociology before October 2011.

Principal findings: The focus of the majority of research in medical education has been on reporting outcomes related to participants. There has been less assessment of patient care outcomes, resulting in informing evidence-based education to only a limited extent. This article explains the process necessary to develop a focused and relevant education research question and emphasizes the importance of theory in medical education research. It describes a range of methodologies, including quantitative, qualitative, and mixed methods, and concludes with a discussion of dissemination of research findings. A majority of studies currently use quantitative methods. This article highlights how further use of qualitative methods can provide insight into the nuances and complexities of learning and teaching processes.

Conclusions: Research in medical education requires several successive steps, from formulating the correct research question to deciding the method for dissemination. Each approach has advantages and disadvantages and should be chosen according to the question being asked and the specific goal of the study. Well-conducted education research should allow progression towards the important goal of using evidence-based education in our teaching and institutions.

Publication types

• Cooperative Behavior
• Education, Medical / methods*
• Evidence-Based Medicine / education
• Research / education*
• Research Design*

• Open access
• Published: 04 March 2024

Analyzing the application of mixed method methodology in medical education: a qualitative study

• Abdulaziz Ibrahim Alhassan 1 , 2 , 3  

BMC Medical Education volume  24 , Article number:  225 ( 2024 ) Cite this article

649 Accesses

Metrics details

Interest in mixed methods methodology within medical education research has seen a notable increase in the past two decades, yet its utilization remains less prominent compared to quantitative methods. This study aimed to investigate the application and integration of mixed methods methodology in medical education research, with a specific focus on researchers’ perceptions, strategies, and readiness, including the necessary skills and expertise. This study adheres to the COREQ guidelines for reporting qualitative research.

Faculty members from King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), Saudi Arabia, across its three campuses in Riyadh, Jeddah, and Al Ahsa, participated in this study during the 2021–2022 academic year. We conducted 15 in-depth, one-on-one interviews with researchers who had previously used mixed methods in their medical education research. Theoretical saturation was reached with no refusals or dropouts. Data were collected using a semi-structured interview guide developed from literature review and mixed methods guidelines. Thematic analysis was employed to analyze the data, enabling a comprehensive understanding of the participants’ perspectives.

The thematic analysis of the interviews yielded three key themes. The first theme, ‘Understanding and Perceptions of Mixed Methods in Medical Education Research,’ delved into researchers’ depth of knowledge and conceptualization of mixed methods. The second theme, ‘Strategies and Integration in Mixed Methods Implementation,’ explored how these methodologies are applied and the challenges involved in their integration. The final theme, ‘Mastery in Mixed Methods: Prerequisites and Expert Consultation in Research,’ highlighted the gaps in readiness and expertise among researchers, emphasizing the importance of expert guidance in this field.

Findings indicate a varied understanding of mixed methods among participants. Some lacked a comprehensive grasp of its application, while others perceived mixed methods primarily as a means to enhance the publication prospects of their studies. There was a general lack of recognition of mixed methods as a guiding methodology for all study aspects, pointing to the need for more in-depth training and resources in this area.

Peer Review reports

The mixed methods methodology has been used within the social sciences for nearly 100 years. As early as 1938, mixed-methods methodology was used in an anthropological study conducted in Southeast Asia [ 1 ]. The use of mixed methods has become strongly engrained within the social sciences as students are routinely educated about mixed methods methodology and various numbers of textbooks include discussions and guidelines about how to conduct mixed methods research [ 2 ]. In medicine and medical education, the mixed methods methodology is relatively new with attention being given to the methodology only within the past 20 years [ 3 ]. Even with a growing interest in mixed methods research within medicine, the use of this methodology has been met with on-going debate about its usefulness, how it should be used, and whether it is a distinct research methodology or whether it is just a combination of qualitative and quantitative research methods [ 4 ].

The reason that mixed methods is still relatively new and not heavily utilized within medicine may be because this methodology requires a broader range of skills and more time and effort to carry out as compared to quantitative studies [ 5 ]. In addition, medical researchers and practitioners have historically believed that quantitative research and a reliance on objective results and findings is more appropriate than qualitative measures that are perceived to be based on purely subjective analysis [ 6 ]. The idea that useful information and findings are only available from statistical analysis has likely led to the idea that mixed methods do not provide more useful data and a broader understanding of issues and problems.

If the desire exists to increase the understanding and use of mixed methods methodology in medicine and medical research, then it is necessary to understand the level of awareness and knowledge that exists about mixed methods among medical education researchers. If medical education researchers do not understand mixed methods methodology, then they are less likely to use it. However, medical education researchers may understand mixed methods methodology, but view it as being inferior to quantitative methods. The purpose of this study is to examine the use of mixed method methodology in medical education research, the level of awareness about the reason to use mixed method methodology, how it is utilized, and its characteristics. By examining the knowledge that medical education researchers possess about mixed methods methodology and how they use the methodology, it may be possible to better understand where gaps in knowledge exist, and, in turn, increase the use of mixed methods methodology among medical education researchers. The importance of this study is that rather than examining why mixed methods methodology is still used less in medicine than the social sciences, the focus is on what medical education researchers know about mixed methods. With the information gained from this study, it will be possible to better understand the actions that may need to be taken in order for medical education researchers to more actively use mixed methods methodology in medical education.

Defining mixed methods methodology

It may seem unnecessary to define mixed methods methodology for researchers who are already interested, and likely knowledgeable, about this research methodology. However, in a study in which the goal is to understand the knowledge that exists among medical education researchers about mixed methods methodology, it is appropriate to briefly discuss the different definitions that exist. Qualitative and quantitative research methods are often discussed as being in opposition to each other with quantitative research being used to test hypotheses and qualitative research being used to investigate the lived experiences and social interactions of individuals and groups [ 7 ].

From a broad perspective, mixed methods methodology has been defined as a research methodology that combines elements of qualitative and quantitative methods to gain more depth and understanding of a problem or topic [ 8 ]. However, Creswell & Plano Clark (2011) explained that the mixed methods methodology is more than just combining qualitative and quantitative methods [ 9 ]. Instead, mixed methods is the use of both qualitative and quantitative methods in a single study in a way that builds on each other to answer a given research question. In this regard, mixed methods methodology is not about using 50% qualitative and 50% quantitative methods in a single study. Instead, mixed methods methodology involves the use of qualitative and quantitative methods based on what is needed and appropriate to answer a research question.

Mixed methods methodology has also been argued to be distinct and separate from other research methods [ 10 ]. Rather than solely being a combination of qualitative and quantitative methods, mixed methods methodology has its own vocabulary and techniques with the goal of not only answering research questions, but also applying scientific findings to actual practice [ 9 ]. The mixed methods methodology is a distinct third research methodology separate from qualitative and quantitative methodology that involves the integration of quantitative and qualitative data in order to gain a more in-depth understanding of problems and issues [ 11 ]. In this regard, the mixed methods methodology extends beyond discovering whether variables are related to each other or whether certain conditions are found to exist in relation to specific phenomena. Instead, mixed methods is about taking the results of scientific study and applying them to practice, such as applying the results of medical studies to medical practice.

Why is mixed methods limited in medical research

There are several reasons for the limited use of the mixed methods methodology within medicine and medical research. These reasons are noted in Table  1 . One of the reasons for the limited use of mixed methods in medicine is because medical research has traditionally been focused on objective findings and the idea that there should be a separation between objective factors and subjective feelings and opinions [ 4 ]. Quantitative research has often been viewed as being related to a single truth that can be discovered by analyzing numerical data while qualitative research involves many truths based on the perceptions and ideas of those who are being studied [ 12 ]. The idea that qualitative research methods yield many truths has been viewed as inappropriate or not specific enough within the realm of medicine.

In addition, conducting mixed methods research is more complex than conducting quantitative research [ 5 ]. Sawatsky et al. (2019) argued that even with advanced training in mixed methods methodology, it can be difficult for medical clinicians to understand and engage in mixed methods research [ 13 ]. The result of the difficulty and lack of understanding in conducting mixed methods research can lead to medical research in which sociologists and other social scientists conducting the qualitative portion of a mixed methods study and medical researchers conducting the quantitative portion [ 14 ].

Curry et al. (2013) also argued that another reason for the lack of mixed methods research in medicine is because of the lack of information and instruction about how to conduct mixed methods studies within journals pertaining to biomedical and health services [ 15 ]. The lack of information and guidance provided to medical researchers has resulted in a lack of use of mixed methods methodology.

Benefits of mixed methods research in medicine

A question that might arise is why mixed methods research is important or useful within medical research. If medical research is about finding the connections between variables or discovering why certain conditions might be related to specific outcomes, then quantitative research might be most appropriate. One of the arguments that has been made about the importance of mixed methods methodology in medical research is the ability to expand the results of scientific studies so they can be used in actual practice [ 16 ]. Mixed method studies can provide information and insights that lead to more robust assessments and practice and increased competency within medical education than is possible through quantitative research alone [ 6 ].

Thistlehwaite et al. (2012) argued that quantitative studies are no longer enough in terms of evidence-based practice for medical education [ 17 ]. Instead, the idea of evidence must be expanded for the benefit of both teaching medical information and findings and engaging in the process of learning medical information. The combination of quantitative and qualitative findings can lead medical practitioners to better implement interventions to patients that consider the social issues that influence patient actions and interactions with medicine and medical conditions [ 18 ]. In this regard, the use of mixed methods methodology can allow practitioners to use medical research in actual practice to improve patient care. Glasgow et al. (2012) argued that even with the advantages of methodological approaches such as mixed methods methodology, future healthcare practitioners and scientists are not being trained in how to use those methodologies to impact actual practice [ 19 ].

Mixed methods in medical education

While mixed methods have been gradually gaining traction in medical research, their application in medical education is distinct yet equally vital. Medical education encompasses a broad range of areas including evaluation, assessment, faculty development, curriculum design, and teaching methods. Mixed methods research offers a nuanced approach to understanding these complex and multifaceted aspects. For instance, it allows for a comprehensive evaluation of educational interventions by quantitatively measuring their outcomes while qualitatively exploring the experiences and perceptions of learners and educators [ 20 ]. This dual approach is essential in a field where both measurable outcomes and subjective experiences are crucial.

Bridging quantitative and qualitative paradigms in education

In medical education, the integration of mixed methods can bridge the gap between quantitative assessments of educational efficacy and qualitative insights into educational experiences. This integration is particularly important in areas like curriculum development and faculty training, where understanding both the effectiveness and the experiential aspects can lead to more informed and holistic improvements [ 21 ].

Identification of the gap

The gap in mixed methods utilization.

Despite the potential benefits, there’s a noticeable gap in the effective application of mixed methods in medical education. A significant portion of medical education researchers comes from a background strongly rooted in quantitative methods, often leading to a preference for, or over-reliance on, these methods. The challenge arises when researchers, familiar with qualitative and quantitative methods individually, attempt to conduct mixed methods research without a comprehensive understanding of its unique nature [ 20 ].

Misconceptions and challenges

The misconception that familiarity with both qualitative and quantitative methods automatically equates to proficiency in mixed methods can lead to suboptimal research designs. Mixed methods research is not simply a juxtaposition of two methodologies; it requires an understanding of how to blend these approaches cohesively, with its own strategies, theoretical frameworks, and specific types of research questions [ 22 ]. Researchers might utilize mixed methods due to the availability of both data types or under the mistaken belief that it inherently enhances the study, without a clear rationale for its use. This can result in poorly integrated results where quantitative and qualitative findings are reported separately without meaningful synthesis [ 22 ].

Study objective

This study aims to investigate the application and integration of mixed methods methodology in medical education research. It focuses on understanding researchers’ perceptions, their strategies in applying mixed methods, and their readiness in terms of skills and expertise required for effective implementation. The objective is to uncover how mixed methods are currently being used in medical education research, identify the gaps in knowledge and application, and provide insights into how these gaps can be bridged for more robust and effective educational research.

Methodology

Study design.

A qualitative research method utilizing one-on-one in-depth interviews with the researchers, who previously used mixed methods research as part of their medical educational research, was used to achieve the objectives of the study. A qualitative approach was considered appropriate specifically phenomenology, to investigate the application and integration of mixed methods methodology in medical education research and to fully understand the reasons and rationale of using a mixed methods approach by the study participants. In this study, it allowed for an in-depth exploration of how researchers in medical education perceive, experience, and interpret mixed methods methodology. By focusing on their personal narratives, phenomenology provided a framework for understanding the complexities and nuances in their perspectives, which is central to addressing the research objectives.The interviews were conducted through the main author (AA), utilizing a constructed semi-structured interview guide composed of open-ended questions formulated after a review of the literature and considering the guidelines for using appropriate mixed method methodology.

The study was conducted at King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), Saudi Arabia in 2021–2022 academic year. The university has three campuses situated in three different cities, Riyadh, Jeddah, and Al Ahsa. The university provides both undergraduate and postgraduate programs. The university requires the students to complete research projects within their educational programs supervised by faculty members. The university also encourages and supports its faculty members to conduct their own research as part of their expected academic roles.

Study population

Inclusion criteria: all researchers who had conducted mixed method methodology research, including faculty members and postgraduate students were included.

Exclusion Criteria: any researcher who had conducted mixed methods research that was not in the field of medicine nor medical education was excluded.

Sample size

Considering that the nature of utilizing mixed methods methodology is not very common compared to using quantitative or qualitative studies alone, an initial sample size considered to generate sufficient data was between six and twelve researchers. However, the sample size was guided by theoretical saturation which was reached after 15 participants with no refusal nor drop outs during the study.

Sampling technique

A purposive sampling technique was used. The researcher began interviewing faculty members in the field of medical education who taught postgraduate medical educational students, graduate students in the medical education program, and any university faculty member.

Data collection

The interview guide included a series of questions developed based on a literature review to analyze and determine the proper utilization and application of the mixed method methodology design. The researcher followed the interview guide, consisting of 15 open-ended questions. The guide consisted of questions on demographic data such as gender, years of experience, researcher background, academic standing, and whether the participants supervised any students in case they were faculty members. The interview guide was piloted by five expert members in medical education to ensure clarity and coherence of the questions.

All participants were asked to sign a consent form, and the researcher explained the aims of the study, the value of their input to the study, and that their participation was voluntary, and that they had the right to withdraw from the interview at any time during the data collection. The interviews were conducted face-to-face in English and audio recorded for quality purposes of data collection and analysis. The interviews lasted from 40 to 60 min in duration. For validity and credibility, the data was checked by two expert members. Once data saturation was obtained, the interviews were discontinued. The transcripts were returned to participants for comments and concretion.

Data analysis

The interviews were imported in NVIVO (QSR international) version 12 for analysis. AA transcribed the interviews verbatim, and AA verified the interview text in consultation with the expert member check NA. A thematic analysis was utilized in which the interview text was read multiple times to become familiar with the content of the data, and then the most relevant and appropriate codes were assigned by the coders AA and NA, summarizing the meaning of the participant expressions in phrases and lines. This was followed by logical editing of code names. Following this, the codes explaining a common pattern in the data were aligned under candidate themes. The themes were finalized, and the coded data with code names and participant references was exported to a word document. The themes were finalized in consultation with AA and the text were read again for potential inconsistencies in coding or theme assignment before interpretation and editing were performed to summarize the findings from the analysis.

Reflexivity

In conducting this research, I, Abdulaziz Alhassan, have been mindful of my positionality as an assistant professor with expertise in mixed methods methodology, which might influence the research process and interpretation of findings. My experience in chairing evaluation and educational program units and teaching postgraduate courses at KSAU-HS has both informed my understanding and necessitated a conscious effort to maintain objectivity.

To ensure the trustworthiness of the research, I employed several strategies. Firstly, triangulation was used by combining different data sources and perspectives, interviewing a diverse range of principal investigators at KSAU-HS. This approach adds depth and breadth to the findings, ensuring that they are not solely reflective of my interpretations. Secondly, member checking was conducted, where findings were shared with some participants for validation. This process ensured that the interpretations accurately reflected their experiences and perceptions, thereby maintaining the integrity of the data. Lastly, reflective journaling played a crucial role throughout the study. I maintained a research journal to document personal reflections, potential biases, and decisions made during the research process. This practice was instrumental in continuously evaluating and mitigating any personal biases that might have influenced the study.

The research was conducted in an environment familiar to me– KSAU-HS. While this familiarity provided a rich contextual understanding, it also required careful attention to avoid bias. My interactions with participants were conducted professionally, with an emphasis on eliciting their views and experiences without imposing my own perspectives.

By acknowledging my background and actively engaging in reflexivity, I aimed to enhance the credibility and reliability of the research while being transparent about my influence on the study.

Our analysis identified three themes from the data which were distinct and unique, explaining the construct of the perceptions of using mixed methods in research related to medical education. These themes included:: understanding and perceptions of mixed methods in medical education research, strategies and integration in mixed methods implementation, and mastery in mixed methods: prerequisites and expert consultation in research. No modifications were provided by the participants’ feedback on the findings.

Theme 1: understanding and perceptions of mixed methods in medical education research

The participants offered a range of definitions, associating them with what they believed was comprised of mixed methods research. A few participants believed that the presence of quantitative and qualitative methods in a single study comprised of a mixed methods design. Most believed that it was a design of research that merges a quantitative part with a qualitative part. Others believed that since it is comprised of two components of quantitative and qualitative research, the qualitative research is an inductive part of the research while quantitative research is more of a deductive mode of inquiry; whereas the data that originates from both types of research is called mixed methods design. Some participants believed that the mixing of data during the analytic part consisted of mixed methods research. Most had a clear understanding of the composition of mixed methods research.

Mixed methods is the presence of quantitative and qualitative studies in a single study. 5-Female .

Usually, it involves mixing quantitative and qualitative methodologies in one research project. 12-Female .

It is a combination of exploratory and explanatory methods. It is transferring one piece of data to another or merging data together. 9-Male .

It’s when we analyze quantitative data, and we want to explore more data qualitatively.

Participants understood that they required a strong justification and a need to implement a mixed methods approach to answer their research questions. Some believed that when quantitative data were not sufficient to explain a certain phenomenon or construct, qualitative data can add to it. They believed that adding a qualitative component offered a better understanding of the answers to their research questions particularly inquiring about participants’ perceptions, feelings, and reflections about a particular topic. Others argued that using a mixed methods approach gives a strong backbone to the findings from the studies, especially when investigating a new concept. Others simply believed that conducting mixed methods research would increase their chances of publishing their research in higher ranked journals. Moreover, the participants were also aware that they could not use a mixed methods approach for just any type of research question, as this carries its own limitations in conducting research, including the factors of time, resources, target population, and settings.

Because in medical education, we cannot quantify everything in numbers since medical education has many aspects that have perceptions, emotions, feelings, and reflections to generalize it. 14-Female .

I also found that the mixed methods methodology is frequently used in medical education research and publications especially when you are studying a phenomenon and you want to see the impact of that phenomena. 12-Female .

It depends on research question, we shouldn’t use mixed methods methodology just because of the availability of both quantitative and qualitative data. 15-Male .

Theme 2: strategies and integration in mixed methods implementation

When inquired about if the participants clearly stated the objectives or the research questions being addressed by the mixed methods research, they were divided in their responses, where some clearly identified and stated their objectives while others did not.

I had three research questions to address the aim of the study, a quantitative, a qualitative and a mixed method methodology question. 12-Female .

One question is sufficient in mixed methods methodology to be answered by the quantitative and qualitative study. 14-Female .

Participants were sometimes advised to explore their research questions in more detail, at times more complex than what could be answered through a simplistic design, leading them to consider a mixed-methods research design. Some explained how they searched for literature on their topics, and from there they were inspired to employ a mixed methods design. A participant explained that a mixed methods approach was necessary in her case because she believed it added value to the reliability, validity, and generalizability of the data. Other participants used mixed methods research because they needed to investigate patterns in their samples that quantitative analysis alone could not answer, such as perceptions about a specific phenomenon.

(I added) a qualitative study to get more in-depth data and which will explain what we are going to find in the quantitative data. I also started reading similar studies to see which method is best for such studies, and I found out that mixed methods methodology was the most suitable one. 12-Female .

I usually use mixed methods methodology studies when I need to explore perceptions, experiences, and challenges because first I must identify these challenges and experiences and then further explore them in-depth. 5-Female .

For most participants, the mixed methods approach was driven by the research questions that they formulated at the outset of planning their research. For some, it was part of the theoretical frameworks they wanted to use in their research, requiring a comprehensive approach. Others just found at a later stage of their research, to employ a mixed methods approach, whereas for others, the available data required them to use this approach.

So, my research questions in the qualitative part were guided by these components, like assignments, clinical evaluations, and exams. 12-Female .

So, it depends on the cognitive or theoretical framework that serves your purpose. 10-Male .

Most participants in our study acknowledged the need for clarity about using an appropriate study design at the outset. A participant said that if the design was wrong at the beginning, it would lead to poor data and half-answered research questions. Participants talked about considering how the data sets would be analyzed in a mixed methods design. They were particularly aware of the need for integration of the findings in the discussion part of their report.

If you have good quantitative and qualitative parts in your study, that doesn’t mean you will be able to mix the results together for a mixed methods methodology study. 10-Male .

The merging of results will also occur when I discuss the results in the discussion part of the study. 15-Male .

When asked about how they prioritized a component in their mixed methods design, participants talked about the way they selected the components. Most believed that both the components supported each other, but they had to consider the factors that could affect the decision about which part of the study could be started first, including the factors of feasibility and resource availability, starting with the quantitative to address the research questions. However, for studies for which participants were required to implement an exploratory design, they mostly started with a qualitative design followed by a quantitative one, which was supported by items from the preliminary analysis of qualitative data. However, a participant said that explanatory designs start with quantitative rather than qualitative designs. Some participants who did not employ a sequential procedure used both components simultaneously. Nevertheless, the participants were aware of the logical sequence of the components of the mixed methods research.

It is very important to prioritize data sets to select the appropriate design. 13- Male .

We still need to determine the time of data collection of one data set over the other even if the data sets have equal priority. 10-Male .

Despite of weather the design is sequential or concurrent, you must decide which data set you will start with, quantitative or qualitative and must give the rationale behind it. 6-Male .

We can prioritize the data in triangulation based on the phenomena that we have and what the relevant data and resources are that we will explore. 5-Female .

If they are not depending on each other then I will see other factors that could affect which one can be started first, such as feasibility and available resources to start with the quantitative to answer the first or second objective then I will do this. 1-Female .

Integration and making sense of the data originating from two different sources of data was another major challenge for the study participants. They talked about situations where they attempted to reduce the data, but simultaneously, they were afraid of losing the richness in their data, which they believed was essential to capture the nuances in the data. A participant talked about the importance of merging by saying that it must be considered at the outset when the data is being collected and later when analysis begins. Nevertheless, most participants, except some, valued that merging was one of the most important aspects of mixed-method studies, as it added more weight and credibility to the study. They were able to give examples of how they merged their data in the discussion sections of their reports. However, many participants believed that their understanding of the methods of merging in such studies was limited; this is the reason, for instance, that a participant explained that he drafted the findings from two sets of data separately within a single report without performing merging of results.

Then I will report it in the discussion part, the relation between both results. 12-Female .

The quantitative and qualitative studies should be linked together and merged to get the mixed methods methodology results. 10-Male .

We didn’t merge the quantitative and qualitative results. I kept them separately. 7-Male .

I stated the quantitative and qualitative results independently. 4-Male .

Data will be integrated in the results and in the discussion parts.

The study should have quantitative results, qualitative results and mixed methods methodology results. 12-Female .

I integrated the data in the results part and these merged results were discussed in the discussion part. 3-Female .

I had no specific technique to present the MMM results. 2-Male .

Yes. I did a table that shows my results for qualitative and quantitative studies and the mixed methods methodology results. 3-Female .

I displayed my mixed methods methodology results using tables. I had to interpret some words into numbers.

Technique to merge data depends on the design used and depends on the researcher. 5-Female .

Theme 3: mastery in mixed methods: prerequisites and expert consultation in research

The participants talked about several issues pertaining to the prerequisites for conducting a mixed methods study. Experience in qualitative research techniques was valued as crucial prerequisites for a researcher’s readiness to use a mixed methods research approach. For the participants, resources, and feasibility, apart from a strong rationale, some experience, and knowledge, were one of the main prerequisites for conducting the qualitative component of the mixed methods research.

The qualitative section specifies the type of qualitative method you will employ focus groups, interviews, formal or informal interviews, and whether you have sufficient manpower resources to conduct this interview. 1-Female .

You must have experience of conducting both quantitative and qualitative studies and good exposure to have a well-structured mixed methods methodology study. 5-Female .

Participants also talked about how using mixed methods research was a challenge for them and how they upgraded their skills to accomplish their complex projects. It required some to do extra reading before they initiated their projects. For many, it meant getting upskilled in qualitative research methods, as they already had adequate training in quantitative skills. In doing so, they believed that access to resources and resourceful people who could impart qualitative skills in their institutions was essential to their projects. Some participants, therefore, indicated the need for formal training in mixed methods research and qualitative techniques. This is because they believed that these types of studies needed to be designed perfectly, since if the design is wrong at the beginning, the data may not be rich enough to answer the research questions.

You must have enough knowledge about the mixed methods methodology studies before doing one. 5-Female .

I personally took workshops in conducting qualitative studies because I only had a background in quantitative studies. 3-Female .

You must have training in mixed methods methodology and exposure to conduct mixed studies. 5-Female .

It is very important to have an expert in mixed methods methodology even if we have experienced researchers in quantitative and qualitative studies. 13-Male .

Participants discussed how important it was to consult an expert with a mixed research methods background, especially at the start. They believed it would enhance the quality of the study because, even if the researcher has a good background in both quantitative and qualitative studies, they still required guidance throughout the study in analysis, writing results, and discussion because it is different than quantitative and qualitative studies. The participants insisted that even if a researcher has a background in both quantitative and qualitative research, it is always better to consult an expert with experience in mixed methods research.

It is very important to have an expert in the field of mixed methods methodology because he will be extremely helpful in integrating the data together. 15-Male .

I think we need experts in mixed methods methodology, even if the researcher has a good background in both quantitative and qualitative studies because you will need them to guide you through the study in analysis part, writing results and discussion parts because it is different than quantitative and qualitative studies. 12-Female .

They did, however, believe that it was difficult to find qualitative researchers who were knowledgeable about advising on mixed methods research. They argued that the supervisors should be aware of the mixed methods design so that they are able to advise on the appropriateness of conducting studies with advanced study designs. Some believed that, ideally, the researcher should have a supervisor with a sound background in mixed methods research because it is not enough that the researcher has either a background in quantitative or qualitative research alone. Whereas others thought that specialist mixed methods researchers may not be needed in situations where a person who is expert equally in quantitative and qualitative approaches.

It is better to have an expert in mixed methods methodology look at the themes and see how you will analyze your data. 3-Female .

We don’t need to involve a mixed methods methodology expert while doing the research if we have researchers with good experience in both quantitative and qualitative studies. 14-Female .

Our study’s findings indicate a diverse range of understandings of mixed methods methodology among medical education researchers. Some of the participants understood mixed methods methodology, when it should be used, and how it should be implemented. However, the responses from several of the participants indicated a lack of understanding about mixed methods methodology utilization. From the responses provided by the participants, three important issues were identified that need to be discussed. These issues are related to how mixed methods methodology is defined, the reason to use mixed methods methodology as opposed to only using a qualitative or quantitative methodology, and the process of presenting the results of a methods study. Table  2 lists the problems identified in the data analyzed from the participants. Each of these issues are important because of the indication that some of the participants did not fully understand mixed methods methodology and how to select and use the methodology to achieve the best outcomes for medical research.

The simplistic definitions and explanations provided by some participants highlight a fundamental challenge. Some of the participants defined mixed methods methodology as simply being the combined use of both qualitative and quantitative methods in a single research study. In this regard, the participants understood the composition of mixed methods methodology, but did not have a more in-depth understanding that mixed methods methodology is about bringing together the inductive nature of a qualitative methodology and the deductive nature of a quantitative methodology to examine a research problem more broadly [ 9 ].

Furthermore, the participants did not explain that mixed methods methodology can be treated as a distinctive methodology on its own separate from qualitative and quantitative methodologies [ 10 ]. Instead, the participants perceived that the mixed methods methodology is solely about bringing qualitative and quantitative methods together. However, some of the participants stated that by bringing both qualitative and quantitative components together, it was possible to gain a more in-depth understanding of the feelings or perceptions of the people being studied.

Some of the participants also believed that conducting a mixed-methods study involved combining qualitative and quantitative data during the analysis. From this perspective, the mixed methods methodology was not viewed as a specific methodology that guided all the work that was performed from formulating research questions to determining the best methodology to use to answer those questions [ 8 ]. Instead, a mixed methods approach was only thought of as something to be considered when engaging in data analysis.

One other aspect of the responses provided by the participants about how they defined and perceived the mixed methods methodology was that some of the participants seemed to view this methodology as a means of getting published in higher ranking journals. Some of the participants stated that they recognized that mixed methods studies were more desirable for some journals because of the ability to gain broader and more in-depth data for analysis. In this way, the use of mixed methods was perceived to be a utility to increase the potential to have studies published in more respectable journals rather than as a distinct research methodology.

Appropriately using mixed methods methodology

Another issue that was identified in the responses provided by the participants was that some of the participants did not understand the appropriate use of mixed methods methodology. As has already been discussed, some of the participants believed that mixed methods methodology should be used to increase the potential of having a study published in a higher-ranking journal. However, other participants described using mixed methods methodology to gain a better understanding of the perceptions and feelings of participants. While mixed methods methodology can be used to gain a more in-depth understanding of a problem, it is not always appropriate to use this methodology even if the goal is to understand the perceptions or feelings of participants.

Instead, the use of qualitative methods can allow a researcher to gain data about the perceptions and beliefs of participants in relation to a problem or phenomenon. Using a qualitative methodology over mixed methods methodology is more appropriate in many circumstances when the goal is to understand the perceptions and beliefs of a group of individuals. The mixed methods methodology is not appropriate in all situations in which the goal is to gain data about the perceptions of participants. In this regard, there was a lack of understanding among some of the participants about when it is appropriate to use a qualitative methodology and when it is appropriate to use mixed methods methodology.

An important aspect of the misunderstanding among some of the participants about when to use mixed methods methodology was that the mixed methods methodology should be used when the problem being investigated and the type of data that are needed to address a research problem warrant its use. One of the participants stated that mixed methods methodology was appropriate when investigating a new concept. The investigation of a new concept might warrant the use of mixed methods methodology, but the use of a qualitative methodology or a quantitative methodology may be more appropriate depending on the research problem being examined and the goals of the study.

Presenting the results of a mixed methods study

The third issue that was identified in the responses provided by the participants was about how the results of a mixed methods study should be presented. The participants generally explained that the results of the qualitative and quantitative portions of a mixed methods study should be presented independently of each other. This is problematic because the participants viewed the qualitative and quantitative aspects of a mixed methods study to be independent of each other rather than as being used together to gain a deeper understanding of a research problem. There was a lack of understanding that in a mixed methods study, the qualitative and quantitative data that are collected should be merged and presented together in a way that answered the research questions for which the data were collected [ 23 ].

On a broader level, the idea that the qualitative and quantitative data in a mixed methods study would be presented independently of each other leads back to the issue of correctly using the mixed methods methodology as part of the design of a study. A mixed methods study should be conducted so that the qualitative and quantitative data are used together to address the research problem being investigated. The qualitative and quantitative data that are collected should not be treated as being independent of each other. If the qualitative and quantitative data are independent of each other, then mixed methods methodology has not been fully used to gain an in-depth understanding of an issue or phenomenon [ 2 ]. Instead, two research methods were used and brought together separately.

Finally, a mixed methods study does not require that half of the data presentation be used to present the quantitative data results and half of the presentation be used to present the qualitative data results. Instead, the data should be presented together in a way that is appropriate for the study that was conducted. In some studies, this might mean that more qualitative data are presented or that more quantitative data are presented. The important issue is to merge the qualitative and quantitative data and present them as one complete unit [ 9 ]. The lack of understanding among the participants about how to appropriately present the results of a mixed methods study also demonstrates that while some of the participants may have understood the idea of mixed methods methodology, they lacked a complete understanding of how to carry out a mixed methods study in terms of presenting the data. In this regard, the participants did not demonstrate a full understanding of the entire process of conducting research using the mixed methods methodology.

Based on the methods used to conduct this study and the findings of the study, several recommendations can be made both for future research and for practice in medical education. One of the recommendations for medical education practice is that training needs to be provided in medical education about mixed methods methodology. While training in mixed methods methodology is common in other disciplines, it is still relatively new in medical education. Increasing the training that students receive about mixed methods methodology would help to increase the general knowledge that exists of the methodology within medical education and its use among medical researchers.

Another recommendation for medical education is to change how research methodologies are presented to students. In medical education, it is still common for quantitative research to be presented as more appropriate for medical research [ 6 ]. For the use of mixed methods methodology to increase within medicine, future researchers in medicine need to be trained that quantitative methodology is not the only appropriate methodology for medical studies. However, they also need to be trained that mixed methods methodology is one methodology that can be used, but it should not be used in all situations.

In terms of future research, one recommendation is for future research to be conducted on how mixed methods methodology is being used in published medical education research. It would be useful to examine if mixed methods methodology is being used appropriately in published medical studies. By examining published mixed methods studied in medical education, an examination could occur about how mixed methods methodology is used and whether it is used in a way that integrates qualitative and quantitative methods.

Contribution to mixed methods research

The results of this study are useful as a contribution to mixed methods research because more has been learned about the perceptions and knowledge that exist about mixed methods among medical education faculty. Mixed methods methodology is still relatively underutilized as compared to quantitative methods within medical research and medical education. Based on the findings of this study, medical faculty lack a strong knowledge about how to implement mixed methods methodology and why it should be used as compared to other methodologies. The practical contribution that is made is that medical faculty need more training in mixed methods methodology, and future medical researchers and faculty need to receive more training in mixed methods methodology as part of their medical education.

This study aimed to investigate the application and integration of mixed methods methodology in medical education research, with a focus on researchers’ perceptions, strategies, and readiness. The findings reveal a varied understanding of mixed methods among participants. While some researchers could articulate basic definitions, their comprehension often did not extend to a strategic or integrated use throughout the research process. This suggests a gap between recognizing mixed methods as a concept and effectively applying it in practice, encompassing both qualitative and quantitative approaches cohesively.

Researchers generally acknowledged the importance of mixed methods in medical education research but displayed a limited grasp of how to strategically implement and integrate these methods. This indicates a need for enhanced educational strategies in medical education to foster a more profound understanding and skillful application of mixed methods research.

The study highlights that readiness in terms of skills and expertise for conducting mixed methods research is not yet fully developed among medical education researchers. There is a growing interest in mixed methods, but it is often seen as a means to an end, such as achieving publication, rather than as a comprehensive methodology suited to certain types of research questions.

The limitation of the study lies in its small and non-representative sample, which challenges the generalizability of the findings. The focus was mainly on perceptions rather than the actual execution of mixed methods research. Future research should thus explore how medical education researchers apply mixed methods in real-world research settings, assessing the alignment between their theoretical understanding and practical application.

These findings are crucial for understanding the current state of mixed methods research in medical education and point towards the need for more targeted training and resources. By bridging the gap between theoretical knowledge and practical application, the quality and efficacy of mixed methods research in medical education can be significantly enhanced.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on responsible request.

-Pelto PJ. What is so new about mixed methods? Qual Health Res. 2015;25(6):734–45. https://doi.org/10.1177/1049732315573209 .

Article   PubMed   Google Scholar  

-Timans R, Wouters P, Heilbron J. Mixed methods research: what it is and what it could be. Theory Soc. 2019;48(2):193–216. https://doi.org/10.1007/s11186-019-09345-5 .

Article   Google Scholar  

-Schifferdecker KE, Reed VA. Using mixed methods research in medical education: Basic guidelines for researchers. Med Educ. 2009;43(7):637–44. https://doi.org/10.1111/j.1365-2923.2009.03386.x .

-Maudsley G. Mixing it but not mixed-up: mixed methods research in medical education (a critical narrative review). Med Teach. 2011;33(2):e92–e104. https://doi.org/10.3109/0142159x.2011.542523 .

-Boet S, Sharma S, Goldman J, Reeves S. Review article: medical education research: an overview of methods. Can J Anesthesia/Journal Canadien d’anesthésie. 2012;59(2):159–70. https://doi.org/10.1007/s12630-011-9635-y .

-Hoang NS, Lau JN. A call for mixed methods in competency-based medical education. Acad Med. 2018;93(7):996–1001. https://doi.org/10.1097/acm.0000000000002205 .

-Bordage G. Moving the Field forward: going beyond quantitative–qualitative*. Acad Med. 2007;82(Suppl):126–S128. https://doi.org/10.1097/acm.0b013e31813e661d .

-Venkatesh V, Brown S, Sullivan Y. Guidelines for conducting mixed-methods research: an extension and illustration. J Association Inform Syst. 2016;17(7):435–94. https://doi.org/10.17705/1jais.00433 .

-Creswell JW, Clark VL. Designing and conducting mixed methods research. SAGE; 2011.

-Lavelle E, Vuk J, Barber C. Twelve tips for getting started using mixed methods in medical education research. Med Teach. 2013;35(4):272–6. https://doi.org/10.3109/0142159x.2013.759645 .

-Baškarada S, Koronios A. A philosophical discussion of qualitative, quantitative, and mixed methods research in social science. Qualitative Res J. 2018;18(1):2–21. https://doi.org/10.1108/qrj-d-17-00042 .

-Leppink J. Revisiting the quantitative–qualitative-mixed methods labels: Research questions, developments, and the need for replication. J Taibah Univ Med Sci. 2017;12(2):97101. https://doi.org/10.1016/j.jtumed.2016.11.008 .

-Sawatsky AP, Ratelle JT, Beckman TJ. Qualitative research methods in medical education. Anesthesiology. 2019;131:14–22. https://doi.org/10.1097/ALN.0000000000002728 .

-O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qual Health Res. 2008;18(11):1574–85. https://doi.org/10.1177/1049732308325535 .

-Curry LA, Krumholz HM, O’Cathain A, Clark P, Cherlin VL, E., Bradley EH. Mixed methods in biomedical and health services research. Circulation: Cardiovasc Qual Outcomes. 2013;6(1):119–23. https://doi.org/10.1161/CIRCOUTCOMES.112.967885 .

-Christenson JD, Gutierrez DM. Using qualitative, quantitative, and mixed methods research to promote family therapy with adolescents in residential settings. Contemp Family Ther. 2016;38(1):52–61. https://doi.org/10.1007/s10591-016-9374-x .

-Thistlethwaite J, Davies H, Dornan T, Greenhalgh T, Hammick M, Scalese R. (2012). What is evidence? Reflections on the AMEE symposium, Vienna, August 2011. Medical Teacher , 34 (6), 454–457. https://doi.org/10.3109/0142159x.2012.671561 .

-Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2007;28(1):413–33. https://doi.org/10.1146/annurev.publhealth.28.021406.144145 .

-Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National institutes of health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102(7):1274–81. https://doi.org/10.2105/ajph.2012.300755 .

Article   PubMed   PubMed Central   Google Scholar  

-Schifferdecker KE, Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ. 2009;43(7):637–44.

-Lavelle E, Vuk J, Barber C. Twelve tips for getting started using mixed methods in medical education research. Med Teach. 2013;35(4):272–6.

-Plano Clark VL. Mixed methods research. J Posit Psychol. 2017;12(3):305–6.

-Regnault A, Willgoss T, Barbic S. Towards the use of mixed methods inquiry as best practice in health outcomes research. J Patient-Reported Outcomes. 2018;2(1):1–4. https://doi.org/10.1186/s41687-018-0043-8 .

Download references

Acknowledgements

We want to express our appreciation to the academic staff involved in this research. Additionally, our thanks go to the King Abdullah International Medical Research Center for their support.

There is no funding to report.

Author information

Authors and affiliations.

Department of Medical Education, College of Medicine, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia

Abdulaziz Ibrahim Alhassan

King Abdullah International Medical Research Center, Riyadh, Saudi Arabia

College of Medicine, Department of Medical Education King Saud bin, Abdulaziz University for Health Sciences (KSAU-HS), KSAU-HS- Riyadh, 11481, +966-11- 4299999, Riyadh, P.O. Box 3660, Saudi Arabia

You can also search for this author in PubMed   Google Scholar

Contributions

This study was conducted by one author and he contributed to all steps of the study including data collection and analysis, result interpretation and writing the final manuscript.

Corresponding author

Correspondence to Abdulaziz Ibrahim Alhassan .

Ethics declarations

Ethics approval and consent to participate.

The study procedures adhered to the prescribed guidelines and regulations. Voluntary participation was emphasized, granting participants the freedom to withdraw at any research stage without providing reasons. Informed consent, encompassing study details, objectives, and benefits, was secured from all participants. Anonymity assurance was provided, and participant-identifying information was securely stored. Ethical approval, granted by the Institutional Review Board of King Abdullah International Medical Research Center, National Guard Health Affairs, Riyadh, Saudi Arabia (Study Number: NRC21R/491/11), validated the study’s ethical compliance.

Consent for publication

Not Applicable.

Competing of interest

The author declares that he has no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Alhassan, A.I. Analyzing the application of mixed method methodology in medical education: a qualitative study. BMC Med Educ 24 , 225 (2024). https://doi.org/10.1186/s12909-024-05242-3

Download citation

Received : 15 April 2023

Accepted : 28 February 2024

Published : 04 March 2024

DOI : https://doi.org/10.1186/s12909-024-05242-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mixed methods methodology
• Medical education research

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Educ Eval Health Prof

An overview of ethnography in healthcare and medical education research

Leigh goodson.

Oklahoma State University Center for Health Sciences, Tulsa, OK, USA.

Matt Vassar

Research in healthcare settings and medical education has relied heavily on quantitative methods. However, there are research questions within these academic domains that may be more adequately addressed by qualitative inquiry. While there are many qualitative approaches, ethnography is one method that allows the researcher to take advantage of relative immersion in order to obtain thick description. The purpose of this article is to introduce ethnography, to describe how ethnographic methods may be utilized, to provide an overview of ethnography's use in healthcare and medical education, and to summarize some key limitations with the method.

INTRODUCTION

Researchers from the social science disciplines are able to take advantage of a wide array of research methodologies. Such methodologies range from the traditional quantitative approaches within the positivist tradition to the qualitative approaches premised upon the ideologies of constructivism. A third alternative, mixed methods designs, have also gained popularity in recent years. Inherent to each of these approaches are different, and sometimes opposing, philosophical and epistemological views. These views dictate the nature of the research design. On one hand, positivists focus on studying that which can be directly observed and confirmed by the senses. Such research seeks to test theory-based, testable hypotheses while remaining objective and value neutral. On the other hand, constructivists believe that knowledge is socially constructed and situated within a particular context. Since each context is unique and will have different perspectives, the world has many different meanings - none of which may be more valid than another. Hence, research designs are primarily aimed at describing the context or group of interest.

Creswell [ 1 ] cites these three types of designs: qualitative, quantitative, and mixed methods. A mixed method is a combination of qualitative and quantitative. These two methods are at opposite ends of the continuum and the continuum is the level of measurement: nominal, ordinal, interval, and ratio. The type of research method used is referring to the method used to collect the data. Researchers using quantitative methods are testing theories through examining statistical relationships between variables. The data is measured numerically with an ordinal, interval, or ratio scale. Qualitative methods use a nominal scale. That is, they may use numbers to label categories but it is only for the purpose of sorting information. The number is simply a label. Qualitative methods aim to explore a culture or group of individuals to understand more about the social or human problem this group experiences. Qualitative methods are often open ended or participatory in nature. Qualitative research, whether standing alone or in a mixed method, adds rich information to any investigation otherwise not discoverable. With quantitative methods, investigators rely on literature and past surveys to target the proper information. Qualitative research allows for variable discovery. That is, the results may glean something never before addressed in similar research. The use of qualitative methods is essential to get closer to exhaustive information on any given topic or population. Oftentimes, the qualitative research leads the researcher to further quantitative investigation. These are not competing but rather complimentary methods.

Qualitative research is often overlooked as an option when considering the methodological approach to a research question. This is especially true in academic domains such as medicine where evidence-based practice has emerged as a popular treatment philosophy based largely on the quantitative research tradition. However, qualitative research can provide rich information otherwise not discovered through quantitative approaches. Ethnography is one qualitative approach that involves relative submersion into the setting to be studied, and is an appropriate methodology for a wide variety of research topics within healthcare and medical education. While, to some extent, ethnography has been applied in healthcare settings and in the medical education environment, we feel that there is a general lack of research employing this methodology. This opinion has also been expressed by Leung [ 2 ]. At times, deficits in particular research methodologies within various academic domains may be attributed to a general lack of knowledge regarding the methodology itself or ways in which the method may be applied. Therefore, the purpose of this introductory paper is to explain ethnographic methodology, discuss how the method may be used, provide a discussion of ethnography's use in healthcare and medical education, and briefly summarize some key limitations with the method. Since ethnography is a method not easily summarized in a single paper, we are writing a series of articles to follow which will address specific aspects related to conducting ethnographic research.

We begin by describing ethnography, synthesizing the works of Leung [ 2 ], Savage [ 3 ], LeCompte and Schensul [ 4 ], Pope [ 5 ], and Atkinson and Pugsley [ 6 ]. Leung [ 2 ], for example, discusses ethnography as a social research method occurring in natural settings characterized by learning the culture of the group under study and experiencing their way of life before attempting to derive explanations of their attitudes or behavior. The culturally based approach can be related to ethnicity, nationality, gender, regional origin, occupation, generation, or in healthcare the focus might be a particular pathology such as cancer, HIV, heart disease, or diabetes. Ethnographies are normally conducted in a single setting, and data collection is largely dependent upon participant observation and interviews. Savage [ 3 ] notes that ethnography may also require historical research prior to beginning actual field work. In terms of time considerations, ethnography is a research method characterized by long-term fieldwork since thick description of the participants and setting may only be acquired from sufficient exposure to them. Savage [ 3 ] further notes that ethnography is not used for developing generalized conclusions but rather studying a specific group of people regarding a specific topic and drawing conclusions only about what was studied.

According to LeCompte and Schensul [ 4 ], there are seven defining characteristics of ethnography. These include: 1) being carried out in a natural setting, not in a laboratory; 2) involving intimate, face-to-face interaction with participants; 3) presenting an accurate reflection of participant perspectives and behaviors; 4) utilizing inductive, interactive, and recursive data collection to build local cultural theories; 5) using both qualitative and quantitative data; 6) framing all human behavior within a sociopolitical and historical context; and 7) using the concept of culture as a lens through which to interpret study results.

Finally, in order to fully understand ethnographic methodology, it is necessary to briefly describe the fundamental ideas and guiding principles of ethnography which have been derived across the many academic disciplines that make use of its application. Atkinson and Pugsley [ 6 ] nicely summarize these ideas which have been succinctly stated in Table 1 .

Fundamental ideas underlying ethnography

USES OF ETHNOGRAPHY

Ethnography is a useful qualitative approach to address a particular type of research question. LeCompte and Schensul [ 4 ] suggest that ethnography should be used to:

• Define a problem when the problem is not yet clear.
• Define a problem when it is complex and embedded in multiple systems or sectors.
• Indentify participants when the participants, population sectors, stakeholders, or the boundaries of the study population are not yet known or identified.
• Clarify the range of settings where a problem or situation currently occurs when not all of the possible settings are fully identified, known, or understood.
• Explore the factors associated with a problem in order to indentify, understand, and address them either though research or intervention studies, when they are not known.
• Document a process.
• Identify and describe unexpected or unanticipated outcomes.
• Design measures that match the characteristics of the target population, clients, or community participants when existing measures are not a good fit or need to be adapted.
• Answer questions that cannot be addressed with other methods or approaches.
• Ease the access of clients to the research process and its products.

Consider the following example by Perry et al. [ 7 ] on the use of ethnography to address a research question.

Summary : Many individuals must deal with loved ones who are too disabled to care for themselves. As pressure continues to keep healthcare costs down, many patients stay fewer days in the hospital after an injury or illness. Perry et al [ 7 ]. described the challenges associated with releasing patients from the hospital before they are fully recovered. Nurses often become the supervisor of the care rather than the primary caregiver. This trend brings to light potential conflicts between supervisor and caregiver. Perry et al. acknowledge the many nursing scholars who contend we must see culture as part of the patient plan. In response to these issues, Perry et al. used ethnography to examine the process of resisting vulnerability as it relates to health care professionals and health system structure. The purpose of the Perry study was to describe the family's view when a family member is in the hospital. The research team concluded families of diverse backgrounds needed to work harder to resist vulnerability in the healthcare system. Method use : Perry et al. used feminist ethnography methods. This ethnographic method begins with the individual's experiences and broadens their view from there to better understand the decision making process the families of non-Caucasian backgrounds. Two studies were conducted: one for the in-hospital experience and the second after the patient was discharged. Data : In ethnography, data collection can sometimes change along the course of the study. A conversation with one study participant may lead to generation of further questions. This happened in the Perry study. Both family members and healthcare providers were interviewed for this study. Once data was collected, it was then coded, and reviewed for accurate interpretation. Report : The management of vulnerability by the family was the focus of this study. Families felt pulled toward vulnerability when they perceived a lack of two way communication from the healthcare providers. When families felt heard by the healthcare providers, they felt a resistance to vulnerability. With communication as a focal point in the vulnerability of the patient, language was identified as important to the perception of the families. Perry et al. emphasized the recognition of cultural and linguistic differences when working with families to care for the patient.

In this case, ethnography was a useful method in addressing the research question. In order to fully understand the family's perspective regarding premature hospital release as well as the responsibility of such caregiving, the researchers needed to understand the complex nature of these phenomena. Ethnography was chosen since it allows for thick description of these underlying issues.

In general, a deep understanding to a research question is nicely achieved through the use of ethnography. Rice and Ezzy [ 8 ] contend that there are several benefits to conducting an ethnography: diverse cultures are better understood through a deep understanding of why people behave in certain ways and have certain beliefs, it is a strategy for the development of grounded theory, and it best addresses the needs of humans.

ETHNOGRAPHIC STUDIES IN HEALTHCARE AND MEDICAL EDUCATION

While cultural understanding is the most cited use of ethnography, there are many specific settings where the method is quite valuable. One of those areas is healthcare. The vast numbers of variables in the clinical setting lend themselves well to be analyzed in a more open ended approach rather than answering questions from a survey or pulling archived data from the hospital and clinic databases [ 8 ]. The patient/nurse relationship is the key to the success of healthcare. Better understanding those dynamics allows decision makers to proceed with more reliable and pertinent information. Getting to the root issues about patient care rather than tracking behavior leads to real solutions rather than a trial and error process.

Campbell [ 9 ] also suggests the use of qualitative research in health care. A large field in itself, health policy would benefit from the complimentary methods of qualitative analysis. Further suggested was meta-ethnography as a qualitative compliment to quantitative meta-analysis [ 9 ]. Through a study involving diabetic care, Campbell [ 9 ] concluded a meta-ethnography was possible and advantageous. The issue of non-compliance was studied with the benefit of seven different studies. Campbell [ 9 ] further suggested methodology within the studies included needed to be better documented a suggestion for further development of meta-ethnography.

Garro [ 10 ] nicely summarizes the uses of ethnography in health care decisions. It begins with the most basic of decisions as to whether or not to seek health care. Garro [ 10 ] goes on to cite several examples of use in medicine. Women deciding to go to the hospital to have their baby generally go too early, hence requiring them to go home and come back again. The Scrimshaw and Souza study cited by Garro [ 10 ] goes on to describe a dynamic where healthcare providers assumed patients understood the instructions the same way as the healthcare provider. The result and further action item was to redesign instructions in two different languages to accommodate the different interpretive tendencies of the patients. Cultural factors continue to be a major factor in the delivery of healthcare. Patients make decisions through this lens creating a dynamic in the healthcare system with a wide variance of options for the various cultures served.

Garro [ 10 ] finally concludes her review with an example leading to an interesting conclusion. Two Mexican towns were studied to determine decision making patterns as they relate to healthcare. The towns were thought to have similar beliefs and the interviewing techniques for these two towns were different. Again here, we see ethnography can be tailored to the population being studied. The flexibility is very useful. The two techniques yielded similar results in terms of the medical knowledge of the two communities. The difference in the two communities was access. The town having greater access to healthcare sought medical attention at twice the rate of the town lacking access. The Garro study indicated access to healthcare was more indicative than illness beliefs when determining how and when a patient sought care [ 10 ].

Van der Geest and Finkler [ 11 ] suggest the use of ethnography in the hospital setting. Hospitals are often cultures within themselves. And, while some can be very similar, the community of the hospital is often unique. Because hospitals reflect dominant culture and belief systems, the care in each hospital can be different based on the cultural influences. This is not clear to the naked eye. From the outside, hospitals look and operate similarly. The patient care and decision making processes can vary widely. Rice and Ezzy [ 8 ] suggest that, through ethnography, behaviors are understood and used to treat the patient through means that fit the needs of the patient. The benefits brought by the ethnography are understanding of the social and cultural backgrounds of the patients and how health behaviors differ across groups. Savage [ 3 ] cites useful ethnographies in health care looking at various issues from cultural differences among clinic attendees to the clinical reasoning differences among physician specialists.

Ethnography has been used in medical education for more than 50 years. Two landmark studies conducted in the United States show earlier uses of ethnography. The first, entitled The Student Physician , was a collection of research from several medical schools. Within this collection, Fox's ethnographic project on medical innovations in clinical settings or the enthographic account of uncertainty in medical knowledge has influenced much subsequent research [ 6 , 12 ].

Boys in White , the second ethnographic account, focused on medical student culture. In Atkinson and Pugsley's [ 6 ] account, this study drew "an explicit parallel between student cultures in universities and shop-floor cultures in workplace settings. Workers and students alike established shared perspectives on their shared problems and collective responses to shared demands" (p. 233). Medical students were found to more effectively manage these demands by using selective negligence (only learning vital information). More recent uses of ethnography in medical education have been noted and will be discussed in our series of subsequent articles as exemplars of ethnographic research.

LIMITATIONS OF ETHNOGRAPHY

Sample size is a limitation of the ethnography method. The time required being involved in participant observation and conducting long interviews greatly limits the sample size. Unlike a scan sheet used for a common survey, ethnography is laborious and detailed in the collection of data.

It is difficult to generalize with the ethnography method. When researching a certain culture, the results cannot necessarily be generalized to other populations. Because the results are based on the cultural responses, the outcome of the study cannot be applied beyond where the study was conducted. For instance, if a best practice ethnography was conducted for the Emergency Room in a hospital, the same best practices may not be applied to the hospital on the other side of town. The best practices for hospital number one were developed based on the population using that hospital and the administration running that hospital.

Subjectivity is certainly a limitation of ethnography. The interpretation of the cultural experience will vary among researchers. There is not a list of answers from which to choose but rather the use of notes made by the investigator and later interpreted and categorized by the investigator. The entire project is subject to the processes and interpretations developed by the researcher and the research team. Not so with quantitative research. With quantitative, there is a measurable response from which to draw conclusions. It is objective.

Funders are often reluctant to fund such projects. Some feel this type of research can lack generalisability [ 3 ]. Another issue for funders is the great expense. It is much less expensive to conduct research with a survey than to immerse an investigator into a culture for a given period of time to extract extensive information.

Finally, the biggest challenge an ethnographer may encounter is the acceptance of the culture. Should the researcher choose ethnography, they must be accepted temporarily within the culture in order to gather accurate information. Hence the need to conduct thorough reviews prior to beginning the ethnography. Additionally, the development of key informants is essential.

Qualitative research, while time consuming, is an excellent tool to investigate differences among cultures, genders, professions, and geographic regions, to name a few. It provides a rich collection of information for the investigator. Oftentimes, it is the source of determining variables for further research and can therefore serve as the launching pad for larger original studies. While we have seen some use of qualitative research within healthcare and medical education, it is underrepresented compared to its quantitative counterpart. Of these qualitative approaches, ethnography is a method quite amenable to medicine, and the application of ethnography to healthcare is widely supported. Health behaviors and differences in healthcare delivery are not necessarily detected in a quantitative study. The use of ethnography allows the decision maker to have a better understanding of the patient and the healthcare delivery team.

This article is available from: http://jeehp.org/