list of research limitations

21 Research Limitations Examples

Research limitations refer to the potential weaknesses inherent in a study. All studies have limitations of some sort, meaning declaring limitations doesn’t necessarily need to be a bad thing, so long as your declaration of limitations is well thought-out and explained.

Rarely is a study perfect. Researchers have to make trade-offs when developing their studies, which are often based upon practical considerations such as time and monetary constraints, weighing the breadth of participants against the depth of insight, and choosing one methodology or another.

In research, studies can have limitations such as limited scope, researcher subjectivity, and lack of available research tools.

Acknowledging the limitations of your study should be seen as a strength. It demonstrates your willingness for transparency, humility, and submission to the scientific method and can bolster the integrity of the study. It can also inform future research direction.

Typically, scholars will explore the limitations of their study in either their methodology section, their conclusion section, or both.

Research Limitations Examples

Qualitative and quantitative research offer different perspectives and methods in exploring phenomena, each with its own strengths and limitations. So, I’ve split the limitations examples sections into qualitative and quantitative below.

Qualitative Research Limitations

Qualitative research seeks to understand phenomena in-depth and in context. It focuses on the ‘why’ and ‘how’ questions.

It’s often used to explore new or complex issues, and it provides rich, detailed insights into participants’ experiences, behaviors, and attitudes. However, these strengths also create certain limitations, as explained below.

1. Subjectivity

Qualitative research often requires the researcher to interpret subjective data. One researcher may examine a text and identify different themes or concepts as more dominant than others.

Close qualitative readings of texts are necessarily subjective – and while this may be a limitation, qualitative researchers argue this is the best way to deeply understand everything in context.

Suggested Solution and Response: To minimize subjectivity bias, you could consider cross-checking your own readings of themes and data against other scholars’ readings and interpretations. This may involve giving the raw data to a supervisor or colleague and asking them to code the data separately, then coming together to compare and contrast results.

2. Researcher Bias

The concept of researcher bias is related to, but slightly different from, subjectivity.

Researcher bias refers to the perspectives and opinions you bring with you when doing your research.

For example, a researcher who is explicitly of a certain philosophical or political persuasion may bring that persuasion to bear when interpreting data.

In many scholarly traditions, we will attempt to minimize researcher bias through the utilization of clear procedures that are set out in advance or through the use of statistical analysis tools.

However, in other traditions, such as in postmodern feminist research , declaration of bias is expected, and acknowledgment of bias is seen as a positive because, in those traditions, it is believed that bias cannot be eliminated from research, so instead, it is a matter of integrity to present it upfront.

Suggested Solution and Response: Acknowledge the potential for researcher bias and, depending on your theoretical framework , accept this, or identify procedures you have taken to seek a closer approximation to objectivity in your coding and analysis.

3. Generalizability

If you’re struggling to find a limitation to discuss in your own qualitative research study, then this one is for you: all qualitative research, of all persuasions and perspectives, cannot be generalized.

This is a core feature that sets qualitative data and quantitative data apart.

The point of qualitative data is to select case studies and similarly small corpora and dig deep through in-depth analysis and thick description of data.

Often, this will also mean that you have a non-randomized sample size.

While this is a positive – you’re going to get some really deep, contextualized, interesting insights – it also means that the findings may not be generalizable to a larger population that may not be representative of the small group of people in your study.

Suggested Solution and Response: Suggest future studies that take a quantitative approach to the question.

4. The Hawthorne Effect

The Hawthorne effect refers to the phenomenon where research participants change their ‘observed behavior’ when they’re aware that they are being observed.

This effect was first identified by Elton Mayo who conducted studies of the effects of various factors ton workers’ productivity. He noticed that no matter what he did – turning up the lights, turning down the lights, etc. – there was an increase in worker outputs compared to prior to the study taking place.

Mayo realized that the mere act of observing the workers made them work harder – his observation was what was changing behavior.

So, if you’re looking for a potential limitation to name for your observational research study , highlight the possible impact of the Hawthorne effect (and how you could reduce your footprint or visibility in order to decrease its likelihood).

Suggested Solution and Response: Highlight ways you have attempted to reduce your footprint while in the field, and guarantee anonymity to your research participants.

5. Replicability

Quantitative research has a great benefit in that the studies are replicable – a researcher can get a similar sample size, duplicate the variables, and re-test a study. But you can’t do that in qualitative research.

Qualitative research relies heavily on context – a specific case study or specific variables that make a certain instance worthy of analysis. As a result, it’s often difficult to re-enter the same setting with the same variables and repeat the study.

Furthermore, the individual researcher’s interpretation is more influential in qualitative research, meaning even if a new researcher enters an environment and makes observations, their observations may be different because subjectivity comes into play much more. This doesn’t make the research bad necessarily (great insights can be made in qualitative research), but it certainly does demonstrate a weakness of qualitative research.

6. Limited Scope

“Limited scope” is perhaps one of the most common limitations listed by researchers – and while this is often a catch-all way of saying, “well, I’m not studying that in this study”, it’s also a valid point.

No study can explore everything related to a topic. At some point, we have to make decisions about what’s included in the study and what is excluded from the study.

So, you could say that a limitation of your study is that it doesn’t look at an extra variable or concept that’s certainly worthy of study but will have to be explored in your next project because this project has a clearly and narrowly defined goal.

Suggested Solution and Response: Be clear about what’s in and out of the study when writing your research question.

7. Time Constraints

This is also a catch-all claim you can make about your research project: that you would have included more people in the study, looked at more variables, and so on. But you’ve got to submit this thing by the end of next semester! You’ve got time constraints.

And time constraints are a recognized reality in all research.

But this means you’ll need to explain how time has limited your decisions. As with “limited scope”, this may mean that you had to study a smaller group of subjects, limit the amount of time you spent in the field, and so forth.

Suggested Solution and Response: Suggest future studies that will build on your current work, possibly as a PhD project.

8. Resource Intensiveness

Qualitative research can be expensive due to the cost of transcription, the involvement of trained researchers, and potential travel for interviews or observations.

So, resource intensiveness is similar to the time constraints concept. If you don’t have the funds, you have to make decisions about which tools to use, which statistical software to employ, and how many research assistants you can dedicate to the study.

Suggested Solution and Response: Suggest future studies that will gain more funding on the back of this ‘ exploratory study ‘.

9. Coding Difficulties

Data analysis in qualitative research often involves coding, which can be subjective and complex, especially when dealing with ambiguous or contradicting data.

After naming this as a limitation in your research, it’s important to explain how you’ve attempted to address this. Some ways to ‘limit the limitation’ include:

• Triangulation: Have 2 other researchers code the data as well and cross-check your results with theirs to identify outliers that may need to be re-examined, debated with the other researchers, or removed altogether.
• Procedure: Use a clear coding procedure to demonstrate reliability in your coding process. I personally use the thematic network analysis method outlined in this academic article by Attride-Stirling (2001).

Suggested Solution and Response: Triangulate your coding findings with colleagues, and follow a thematic network analysis procedure.

10. Risk of Non-Responsiveness

There is always a risk in research that research participants will be unwilling or uncomfortable sharing their genuine thoughts and feelings in the study.

This is particularly true when you’re conducting research on sensitive topics, politicized topics, or topics where the participant is expressing vulnerability .

This is similar to the Hawthorne effect (aka participant bias), where participants change their behaviors in your presence; but it goes a step further, where participants actively hide their true thoughts and feelings from you.

Suggested Solution and Response: One way to manage this is to try to include a wider group of people with the expectation that there will be non-responsiveness from some participants.

11. Risk of Attrition

Attrition refers to the process of losing research participants throughout the study.

This occurs most commonly in longitudinal studies , where a researcher must return to conduct their analysis over spaced periods of time, often over a period of years.

Things happen to people over time – they move overseas, their life experiences change, they get sick, change their minds, and even die. The more time that passes, the greater the risk of attrition.

Suggested Solution and Response: One way to manage this is to try to include a wider group of people with the expectation that there will be attrition over time.

12. Difficulty in Maintaining Confidentiality and Anonymity

Given the detailed nature of qualitative data , ensuring participant anonymity can be challenging.

If you have a sensitive topic in a specific case study, even anonymizing research participants sometimes isn’t enough. People might be able to induce who you’re talking about.

Sometimes, this will mean you have to exclude some interesting data that you collected from your final report. Confidentiality and anonymity come before your findings in research ethics – and this is a necessary limiting factor.

Suggested Solution and Response: Highlight the efforts you have taken to anonymize data, and accept that confidentiality and accountability place extremely important constraints on academic research.

13. Difficulty in Finding Research Participants

A study that looks at a very specific phenomenon or even a specific set of cases within a phenomenon means that the pool of potential research participants can be very low.

Compile on top of this the fact that many people you approach may choose not to participate, and you could end up with a very small corpus of subjects to explore. This may limit your ability to make complete findings, even in a quantitative sense.

You may need to therefore limit your research question and objectives to something more realistic.

Suggested Solution and Response: Highlight that this is going to limit the study’s generalizability significantly.

14. Ethical Limitations

Ethical limitations refer to the things you cannot do based on ethical concerns identified either by yourself or your institution’s ethics review board.

This might include threats to the physical or psychological well-being of your research subjects, the potential of releasing data that could harm a person’s reputation, and so on.

Furthermore, even if your study follows all expected standards of ethics, you still, as an ethical researcher, need to allow a research participant to pull out at any point in time, after which you cannot use their data, which demonstrates an overlap between ethical constraints and participant attrition.

Suggested Solution and Response: Highlight that these ethical limitations are inevitable but important to sustain the integrity of the research.

For more on Qualitative Research, Explore my Qualitative Research Guide

Quantitative Research Limitations

Quantitative research focuses on quantifiable data and statistical, mathematical, or computational techniques. It’s often used to test hypotheses, assess relationships and causality, and generalize findings across larger populations.

Quantitative research is widely respected for its ability to provide reliable, measurable, and generalizable data (if done well!). Its structured methodology has strengths over qualitative research, such as the fact it allows for replication of the study, which underpins the validity of the research.

However, this approach is not without it limitations, explained below.

1. Over-Simplification

Quantitative research is powerful because it allows you to measure and analyze data in a systematic and standardized way. However, one of its limitations is that it can sometimes simplify complex phenomena or situations.

In other words, it might miss the subtleties or nuances of the research subject.

For example, if you’re studying why people choose a particular diet, a quantitative study might identify factors like age, income, or health status. But it might miss other aspects, such as cultural influences or personal beliefs, that can also significantly impact dietary choices.

When writing about this limitation, you can say that your quantitative approach, while providing precise measurements and comparisons, may not capture the full complexity of your subjects of study.

Suggested Solution and Response: Suggest a follow-up case study using the same research participants in order to gain additional context and depth.

2. Lack of Context

Another potential issue with quantitative research is that it often focuses on numbers and statistics at the expense of context or qualitative information.

Let’s say you’re studying the effect of classroom size on student performance. You might find that students in smaller classes generally perform better. However, this doesn’t take into account other variables, like teaching style , student motivation, or family support.

When describing this limitation, you might say, “Although our research provides important insights into the relationship between class size and student performance, it does not incorporate the impact of other potentially influential variables. Future research could benefit from a mixed-methods approach that combines quantitative analysis with qualitative insights.”

3. Applicability to Real-World Settings

Oftentimes, experimental research takes place in controlled environments to limit the influence of outside factors.

This control is great for isolation and understanding the specific phenomenon but can limit the applicability or “external validity” of the research to real-world settings.

For example, if you conduct a lab experiment to see how sleep deprivation impacts cognitive performance, the sterile, controlled lab environment might not reflect real-world conditions where people are dealing with multiple stressors.

Therefore, when explaining the limitations of your quantitative study in your methodology section, you could state:

“While our findings provide valuable information about [topic], the controlled conditions of the experiment may not accurately represent real-world scenarios where extraneous variables will exist. As such, the direct applicability of our results to broader contexts may be limited.”

Suggested Solution and Response: Suggest future studies that will engage in real-world observational research, such as ethnographic research.

4. Limited Flexibility

Once a quantitative study is underway, it can be challenging to make changes to it. This is because, unlike in grounded research, you’re putting in place your study in advance, and you can’t make changes part-way through.

Your study design, data collection methods, and analysis techniques need to be decided upon before you start collecting data.

For example, if you are conducting a survey on the impact of social media on teenage mental health, and halfway through, you realize that you should have included a question about their screen time, it’s generally too late to add it.

When discussing this limitation, you could write something like, “The structured nature of our quantitative approach allows for consistent data collection and analysis but also limits our flexibility to adapt and modify the research process in response to emerging insights and ideas.”

Suggested Solution and Response: Suggest future studies that will use mixed-methods or qualitative research methods to gain additional depth of insight.

5. Risk of Survey Error

Surveys are a common tool in quantitative research, but they carry risks of error.

There can be measurement errors (if a question is misunderstood), coverage errors (if some groups aren’t adequately represented), non-response errors (if certain people don’t respond), and sampling errors (if your sample isn’t representative of the population).

For instance, if you’re surveying college students about their study habits , but only daytime students respond because you conduct the survey during the day, your results will be skewed.

In discussing this limitation, you might say, “Despite our best efforts to develop a comprehensive survey, there remains a risk of survey error, including measurement, coverage, non-response, and sampling errors. These could potentially impact the reliability and generalizability of our findings.”

Suggested Solution and Response: Suggest future studies that will use other survey tools to compare and contrast results.

6. Limited Ability to Probe Answers

With quantitative research, you typically can’t ask follow-up questions or delve deeper into participants’ responses like you could in a qualitative interview.

For instance, imagine you are surveying 500 students about study habits in a questionnaire. A respondent might indicate that they study for two hours each night. You might want to follow up by asking them to elaborate on what those study sessions involve or how effective they feel their habits are.

However, quantitative research generally disallows this in the way a qualitative semi-structured interview could.

When discussing this limitation, you might write, “Given the structured nature of our survey, our ability to probe deeper into individual responses is limited. This means we may not fully understand the context or reasoning behind the responses, potentially limiting the depth of our findings.”

Suggested Solution and Response: Suggest future studies that engage in mixed-method or qualitative methodologies to address the issue from another angle.

7. Reliance on Instruments for Data Collection

In quantitative research, the collection of data heavily relies on instruments like questionnaires, surveys, or machines.

The limitation here is that the data you get is only as good as the instrument you’re using. If the instrument isn’t designed or calibrated well, your data can be flawed.

For instance, if you’re using a questionnaire to study customer satisfaction and the questions are vague, confusing, or biased, the responses may not accurately reflect the customers’ true feelings.

When discussing this limitation, you could say, “Our study depends on the use of questionnaires for data collection. Although we have put significant effort into designing and testing the instrument, it’s possible that inaccuracies or misunderstandings could potentially affect the validity of the data collected.”

Suggested Solution and Response: Suggest future studies that will use different instruments but examine the same variables to triangulate results.

8. Time and Resource Constraints (Specific to Quantitative Research)

Quantitative research can be time-consuming and resource-intensive, especially when dealing with large samples.

It often involves systematic sampling, rigorous design, and sometimes complex statistical analysis.

If resources and time are limited, it can restrict the scale of your research, the techniques you can employ, or the extent of your data analysis.

For example, you may want to conduct a nationwide survey on public opinion about a certain policy. However, due to limited resources, you might only be able to survey people in one city.

When writing about this limitation, you could say, “Given the scope of our research and the resources available, we are limited to conducting our survey within one city, which may not fully represent the nationwide public opinion. Hence, the generalizability of the results may be limited.”

Suggested Solution and Response: Suggest future studies that will have more funding or longer timeframes.

How to Discuss Your Research Limitations

1. in your research proposal and methodology section.

In the research proposal, which will become the methodology section of your dissertation, I would recommend taking the four following steps, in order:

• Be Explicit about your Scope – If you limit the scope of your study in your research question, aims, and objectives, then you can set yourself up well later in the methodology to say that certain questions are “outside the scope of the study.” For example, you may identify the fact that the study doesn’t address a certain variable, but you can follow up by stating that the research question is specifically focused on the variable that you are examining, so this limitation would need to be looked at in future studies.
• Acknowledge the Limitation – Acknowledging the limitations of your study demonstrates reflexivity and humility and can make your research more reliable and valid. It also pre-empts questions the people grading your paper may have, so instead of them down-grading you for your limitations; they will congratulate you on explaining the limitations and how you have addressed them!
• Explain your Decisions – You may have chosen your approach (despite its limitations) for a very specific reason. This might be because your approach remains, on balance, the best one to answer your research question. Or, it might be because of time and monetary constraints that are outside of your control.
• Highlight the Strengths of your Approach – Conclude your limitations section by strongly demonstrating that, despite limitations, you’ve worked hard to minimize the effects of the limitations and that you have chosen your specific approach and methodology because it’s also got some terrific strengths. Name the strengths.

Overall, you’ll want to acknowledge your own limitations but also explain that the limitations don’t detract from the value of your study as it stands.

2. In the Conclusion Section or Chapter

In the conclusion of your study, it is generally expected that you return to a discussion of the study’s limitations. Here, I recommend the following steps:

• Acknowledge issues faced – After completing your study, you will be increasingly aware of issues you may have faced that, if you re-did the study, you may have addressed earlier in order to avoid those issues. Acknowledge these issues as limitations, and frame them as recommendations for subsequent studies.
• Suggest further research – Scholarly research aims to fill gaps in the current literature and knowledge. Having established your expertise through your study, suggest lines of inquiry for future researchers. You could state that your study had certain limitations, and “future studies” can address those limitations.
• Suggest a mixed methods approach – Qualitative and quantitative research each have pros and cons. So, note those ‘cons’ of your approach, then say the next study should approach the topic using the opposite methodology or could approach it using a mixed-methods approach that could achieve the benefits of quantitative studies with the nuanced insights of associated qualitative insights as part of an in-study case-study.

Overall, be clear about both your limitations and how those limitations can inform future studies.

In sum, each type of research method has its own strengths and limitations. Qualitative research excels in exploring depth, context, and complexity, while quantitative research excels in examining breadth, generalizability, and quantifiable measures. Despite their individual limitations, each method contributes unique and valuable insights, and researchers often use them together to provide a more comprehensive understanding of the phenomenon being studied.

Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative research , 1 (3), 385-405. ( Source )

Atkinson, P., Delamont, S., Cernat, A., Sakshaug, J., & Williams, R. A. (2021).  SAGE research methods foundations . London: Sage Publications.

Clark, T., Foster, L., Bryman, A., & Sloan, L. (2021).  Bryman’s social research methods . Oxford: Oxford University Press.

Köhler, T., Smith, A., & Bhakoo, V. (2022). Templates in qualitative research methods: Origins, limitations, and new directions.  Organizational Research Methods ,  25 (2), 183-210. ( Source )

Lenger, A. (2019). The rejection of qualitative research methods in economics.  Journal of Economic Issues ,  53 (4), 946-965. ( Source )

Taherdoost, H. (2022). What are different research approaches? Comprehensive review of qualitative, quantitative, and mixed method research, their applications, types, and limitations.  Journal of Management Science & Engineering Research ,  5 (1), 53-63. ( Source )

Walliman, N. (2021).  Research methods: The basics . New York: Routledge.

Chris Drew (PhD)

Dr. Chris Drew is the founder of the Helpful Professor. He holds a PhD in education and has published over 20 articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education. [Image Descriptor: Photo of Chris]

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 5 Top Tips for Succeeding at University
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 50 Durable Goods Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 100 Consumer Goods Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 30 Globalization Pros and Cons

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

• Privacy Policy

Buy Me a Coffee

Home » Limitations in Research – Types, Examples and Writing Guide

Limitations in Research – Types, Examples and Writing Guide

Table of Contents

Limitations in Research

Limitations in research refer to the factors that may affect the results, conclusions , and generalizability of a study. These limitations can arise from various sources, such as the design of the study, the sampling methods used, the measurement tools employed, and the limitations of the data analysis techniques.

Types of Limitations in Research

Types of Limitations in Research are as follows:

Sample Size Limitations

This refers to the size of the group of people or subjects that are being studied. If the sample size is too small, then the results may not be representative of the population being studied. This can lead to a lack of generalizability of the results.

Time Limitations

Time limitations can be a constraint on the research process . This could mean that the study is unable to be conducted for a long enough period of time to observe the long-term effects of an intervention, or to collect enough data to draw accurate conclusions.

Selection Bias

This refers to a type of bias that can occur when the selection of participants in a study is not random. This can lead to a biased sample that is not representative of the population being studied.

Confounding Variables

Confounding variables are factors that can influence the outcome of a study, but are not being measured or controlled for. These can lead to inaccurate conclusions or a lack of clarity in the results.

Measurement Error

This refers to inaccuracies in the measurement of variables, such as using a faulty instrument or scale. This can lead to inaccurate results or a lack of validity in the study.

Ethical Limitations

Ethical limitations refer to the ethical constraints placed on research studies. For example, certain studies may not be allowed to be conducted due to ethical concerns, such as studies that involve harm to participants.

Examples of Limitations in Research

Some Examples of Limitations in Research are as follows:

Research Title: “The Effectiveness of Machine Learning Algorithms in Predicting Customer Behavior”

Limitations:

• The study only considered a limited number of machine learning algorithms and did not explore the effectiveness of other algorithms.
• The study used a specific dataset, which may not be representative of all customer behaviors or demographics.
• The study did not consider the potential ethical implications of using machine learning algorithms in predicting customer behavior.

Research Title: “The Impact of Online Learning on Student Performance in Computer Science Courses”

• The study was conducted during the COVID-19 pandemic, which may have affected the results due to the unique circumstances of remote learning.
• The study only included students from a single university, which may limit the generalizability of the findings to other institutions.
• The study did not consider the impact of individual differences, such as prior knowledge or motivation, on student performance in online learning environments.

Research Title: “The Effect of Gamification on User Engagement in Mobile Health Applications”

• The study only tested a specific gamification strategy and did not explore the effectiveness of other gamification techniques.
• The study relied on self-reported measures of user engagement, which may be subject to social desirability bias or measurement errors.
• The study only included a specific demographic group (e.g., young adults) and may not be generalizable to other populations with different preferences or needs.

How to Write Limitations in Research

When writing about the limitations of a research study, it is important to be honest and clear about the potential weaknesses of your work. Here are some tips for writing about limitations in research:

• Identify the limitations: Start by identifying the potential limitations of your research. These may include sample size, selection bias, measurement error, or other issues that could affect the validity and reliability of your findings.
• Be honest and objective: When describing the limitations of your research, be honest and objective. Do not try to minimize or downplay the limitations, but also do not exaggerate them. Be clear and concise in your description of the limitations.
• Provide context: It is important to provide context for the limitations of your research. For example, if your sample size was small, explain why this was the case and how it may have affected your results. Providing context can help readers understand the limitations in a broader context.
• Discuss implications : Discuss the implications of the limitations for your research findings. For example, if there was a selection bias in your sample, explain how this may have affected the generalizability of your findings. This can help readers understand the limitations in terms of their impact on the overall validity of your research.
• Provide suggestions for future research : Finally, provide suggestions for future research that can address the limitations of your study. This can help readers understand how your research fits into the broader field and can provide a roadmap for future studies.

Purpose of Limitations in Research

There are several purposes of limitations in research. Here are some of the most important ones:

• To acknowledge the boundaries of the study : Limitations help to define the scope of the research project and set realistic expectations for the findings. They can help to clarify what the study is not intended to address.
• To identify potential sources of bias: Limitations can help researchers identify potential sources of bias in their research design, data collection, or analysis. This can help to improve the validity and reliability of the findings.
• To provide opportunities for future research: Limitations can highlight areas for future research and suggest avenues for further exploration. This can help to advance knowledge in a particular field.
• To demonstrate transparency and accountability: By acknowledging the limitations of their research, researchers can demonstrate transparency and accountability to their readers, peers, and funders. This can help to build trust and credibility in the research community.
• To encourage critical thinking: Limitations can encourage readers to critically evaluate the study’s findings and consider alternative explanations or interpretations. This can help to promote a more nuanced and sophisticated understanding of the topic under investigation.

When to Write Limitations in Research

Limitations should be included in research when they help to provide a more complete understanding of the study’s results and implications. A limitation is any factor that could potentially impact the accuracy, reliability, or generalizability of the study’s findings.

It is important to identify and discuss limitations in research because doing so helps to ensure that the results are interpreted appropriately and that any conclusions drawn are supported by the available evidence. Limitations can also suggest areas for future research, highlight potential biases or confounding factors that may have affected the results, and provide context for the study’s findings.

Generally, limitations should be discussed in the conclusion section of a research paper or thesis, although they may also be mentioned in other sections, such as the introduction or methods. The specific limitations that are discussed will depend on the nature of the study, the research question being investigated, and the data that was collected.

Examples of limitations that might be discussed in research include sample size limitations, data collection methods, the validity and reliability of measures used, and potential biases or confounding factors that could have affected the results. It is important to note that limitations should not be used as a justification for poor research design or methodology, but rather as a way to enhance the understanding and interpretation of the study’s findings.

Importance of Limitations in Research

Here are some reasons why limitations are important in research:

• Enhances the credibility of research: Limitations highlight the potential weaknesses and threats to validity, which helps readers to understand the scope and boundaries of the study. This improves the credibility of research by acknowledging its limitations and providing a clear picture of what can and cannot be concluded from the study.
• Facilitates replication: By highlighting the limitations, researchers can provide detailed information about the study’s methodology, data collection, and analysis. This information helps other researchers to replicate the study and test the validity of the findings, which enhances the reliability of research.
• Guides future research : Limitations provide insights into areas for future research by identifying gaps or areas that require further investigation. This can help researchers to design more comprehensive and effective studies that build on existing knowledge.
• Provides a balanced view: Limitations help to provide a balanced view of the research by highlighting both strengths and weaknesses. This ensures that readers have a clear understanding of the study’s limitations and can make informed decisions about the generalizability and applicability of the findings.

Advantages of Limitations in Research

Here are some potential advantages of limitations in research:

• Focus : Limitations can help researchers focus their study on a specific area or population, which can make the research more relevant and useful.
• Realism : Limitations can make a study more realistic by reflecting the practical constraints and challenges of conducting research in the real world.
• Innovation : Limitations can spur researchers to be more innovative and creative in their research design and methodology, as they search for ways to work around the limitations.
• Rigor : Limitations can actually increase the rigor and credibility of a study, as researchers are forced to carefully consider the potential sources of bias and error, and address them to the best of their abilities.
• Generalizability : Limitations can actually improve the generalizability of a study by ensuring that it is not overly focused on a specific sample or situation, and that the results can be applied more broadly.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

How to Write Limitations of the Study (with examples)

This blog emphasizes the importance of recognizing and effectively writing about limitations in research. It discusses the types of limitations, their significance, and provides guidelines for writing about them, highlighting their role in advancing scholarly research.

Updated on August 24, 2023

No matter how well thought out, every research endeavor encounters challenges. There is simply no way to predict all possible variances throughout the process.

These uncharted boundaries and abrupt constraints are known as limitations in research . Identifying and acknowledging limitations is crucial for conducting rigorous studies. Limitations provide context and shed light on gaps in the prevailing inquiry and literature.

This article explores the importance of recognizing limitations and discusses how to write them effectively. By interpreting limitations in research and considering prevalent examples, we aim to reframe the perception from shameful mistakes to respectable revelations.

What are limitations in research?

In the clearest terms, research limitations are the practical or theoretical shortcomings of a study that are often outside of the researcher’s control . While these weaknesses limit the generalizability of a study’s conclusions, they also present a foundation for future research.

Sometimes limitations arise from tangible circumstances like time and funding constraints, or equipment and participant availability. Other times the rationale is more obscure and buried within the research design. Common types of limitations and their ramifications include:

• Theoretical: limits the scope, depth, or applicability of a study.
• Methodological: limits the quality, quantity, or diversity of the data.
• Empirical: limits the representativeness, validity, or reliability of the data.
• Analytical: limits the accuracy, completeness, or significance of the findings.
• Ethical: limits the access, consent, or confidentiality of the data.

Regardless of how, when, or why they arise, limitations are a natural part of the research process and should never be ignored . Like all other aspects, they are vital in their own purpose.

Why is identifying limitations important?

Whether to seek acceptance or avoid struggle, humans often instinctively hide flaws and mistakes. Merging this thought process into research by attempting to hide limitations, however, is a bad idea. It has the potential to negate the validity of outcomes and damage the reputation of scholars.

By identifying and addressing limitations throughout a project, researchers strengthen their arguments and curtail the chance of peer censure based on overlooked mistakes. Pointing out these flaws shows an understanding of variable limits and a scrupulous research process.

Showing awareness of and taking responsibility for a project’s boundaries and challenges validates the integrity and transparency of a researcher. It further demonstrates the researchers understand the applicable literature and have thoroughly evaluated their chosen research methods.

Presenting limitations also benefits the readers by providing context for research findings. It guides them to interpret the project’s conclusions only within the scope of very specific conditions. By allowing for an appropriate generalization of the findings that is accurately confined by research boundaries and is not too broad, limitations boost a study’s credibility .

Limitations are true assets to the research process. They highlight opportunities for future research. When researchers identify the limitations of their particular approach to a study question, they enable precise transferability and improve chances for reproducibility. 

Simply stating a project’s limitations is not adequate for spurring further research, though. To spark the interest of other researchers, these acknowledgements must come with thorough explanations regarding how the limitations affected the current study and how they can potentially be overcome with amended methods.

How to write limitations

Typically, the information about a study’s limitations is situated either at the beginning of the discussion section to provide context for readers or at the conclusion of the discussion section to acknowledge the need for further research. However, it varies depending upon the target journal or publication guidelines. 

Don’t hide your limitations

It is also important to not bury a limitation in the body of the paper unless it has a unique connection to a topic in that section. If so, it needs to be reiterated with the other limitations or at the conclusion of the discussion section. Wherever it is included in the manuscript, ensure that the limitations section is prominently positioned and clearly introduced.

While maintaining transparency by disclosing limitations means taking a comprehensive approach, it is not necessary to discuss everything that could have potentially gone wrong during the research study. If there is no commitment to investigation in the introduction, it is unnecessary to consider the issue a limitation to the research. Wholly consider the term ‘limitations’ and ask, “Did it significantly change or limit the possible outcomes?” Then, qualify the occurrence as either a limitation to include in the current manuscript or as an idea to note for other projects. 

Writing limitations

Once the limitations are concretely identified and it is decided where they will be included in the paper, researchers are ready for the writing task. Including only what is pertinent, keeping explanations detailed but concise, and employing the following guidelines is key for crafting valuable limitations:

1) Identify and describe the limitations : Clearly introduce the limitation by classifying its form and specifying its origin. For example:

• An unintentional bias encountered during data collection
• An intentional use of unplanned post-hoc data analysis

2) Explain the implications : Describe how the limitation potentially influences the study’s findings and how the validity and generalizability are subsequently impacted. Provide examples and evidence to support claims of the limitations’ effects without making excuses or exaggerating their impact. Overall, be transparent and objective in presenting the limitations, without undermining the significance of the research. 

3) Provide alternative approaches for future studies : Offer specific suggestions for potential improvements or avenues for further investigation. Demonstrate a proactive approach by encouraging future research that addresses the identified gaps and, therefore, expands the knowledge base.

Whether presenting limitations as an individual section within the manuscript or as a subtopic in the discussion area, authors should use clear headings and straightforward language to facilitate readability. There is no need to complicate limitations with jargon, computations, or complex datasets.

Examples of common limitations

Limitations are generally grouped into two categories , methodology and research process .

Methodology limitations

Methodology may include limitations due to:

• Sample size
• Lack of available or reliable data
• Lack of prior research studies on the topic
• Measure used to collect the data
• Self-reported data

The researcher is addressing how the large sample size requires a reassessment of the measures used to collect and analyze the data.

Research process limitations

Limitations during the research process may arise from:

• Access to information
• Longitudinal effects
• Cultural and other biases
• Language fluency
• Time constraints

The author is pointing out that the model’s estimates are based on potentially biased observational studies.

Final thoughts

Successfully proving theories and touting great achievements are only two very narrow goals of scholarly research. The true passion and greatest efforts of researchers comes more in the form of confronting assumptions and exploring the obscure.

In many ways, recognizing and sharing the limitations of a research study both allows for and encourages this type of discovery that continuously pushes research forward. By using limitations to provide a transparent account of the project's boundaries and to contextualize the findings, researchers pave the way for even more robust and impactful research in the future.

Charla Viera, MS

See our "Privacy Policy"

Ensure your structure and ideas are consistent and clearly communicated

Pair your Premium Editing with our add-on service Presubmission Review for an overall assessment of your manuscript.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Limitations of the Study
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. Study limitations are the constraints placed on the ability to generalize from the results, to further describe applications to practice, and/or related to the utility of findings that are the result of the ways in which you initially chose to design the study or the method used to establish internal and external validity or the result of unanticipated challenges that emerged during the study.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Theofanidis, Dimitrios and Antigoni Fountouki. "Limitations and Delimitations in the Research Process." Perioperative Nursing 7 (September-December 2018): 155-163. .

Importance of...

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and have your grade lowered because you appeared to have ignored them or didn't realize they existed.

Keep in mind that acknowledgment of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgment of a study's limitations also provides you with opportunities to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and discuss how they possibly impacted your results. Note that descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is generally less relevant in qualitative research if explained in the context of the research problem.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but provide cogent reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe a need for future research based on designing a different method for gathering data.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian! In cases when a librarian has confirmed that there is little or no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design ]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to the accuracy of what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency, but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, data, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this needs to be described. Also, include an explanation why being denied or limited access did not prevent you from following through on your study.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, event, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. NOTE :   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias. For example, if a previous study only used boys to examine how music education supports effective math skills, describe how your research expands the study to include girls.
• Fluency in a language -- if your research focuses , for example, on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic or to speak with these students in their primary language. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods. Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings!

After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitations of your study. Inflating the importance of your study's findings could be perceived by your readers as an attempt hide its flaws or encourage a biased interpretation of the results. A small measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated. Or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may very well be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Lewis, George H. and Jonathan F. Lewis. “The Dog in the Night-Time: Negative Evidence in Social Research.” The British Journal of Sociology 31 (December 1980): 544-558.

Yet Another Writing Tip

Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgment about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Boddy, Clive Roland. "Sample Size for Qualitative Research." Qualitative Market Research: An International Journal 19 (2016): 426-432; Huberman, A. Michael and Matthew B. Miles. "Data Management and Analysis Methods." In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444; Blaikie, Norman. "Confounding Issues Related to Determining Sample Size in Qualitative Research." International Journal of Social Research Methodology 21 (2018): 635-641; Oppong, Steward Harrison. "The Problem of Sampling in qualitative Research." Asian Journal of Management Sciences and Education 2 (2013): 202-210.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: Apr 5, 2024 1:38 PM
• URL: https://libguides.usc.edu/writingguide

Join thousands of product people at Insight Out Conf on April 11. Register free.

Insights hub solutions

Analyze data

Uncover deep customer insights with fast, powerful features, store insights, curate and manage insights in one searchable platform, scale research, unlock the potential of customer insights at enterprise scale.

Featured reads

Inspiration

Three things to look forward to at Insight Out

Tips and tricks

Make magic with your customer data in Dovetail

Four ways Dovetail helps Product Managers master continuous product discovery

Events and videos

© Dovetail Research Pty. Ltd.

How to present limitations in research

Last updated

30 January 2024

Reviewed by

Limitations don’t invalidate or diminish your results, but it’s best to acknowledge them. This will enable you to address any questions your study failed to answer because of them.

In this guide, learn how to recognize, present, and overcome limitations in research.

• What is a research limitation?

Research limitations are weaknesses in your research design or execution that may have impacted outcomes and conclusions. Uncovering limitations doesn’t necessarily indicate poor research design—it just means you encountered challenges you couldn’t have anticipated that limited your research efforts.

Does basic research have limitations?

Basic research aims to provide more information about your research topic. It requires the same standard research methodology and data collection efforts as any other research type, and it can also have limitations.

• Common research limitations

Researchers encounter common limitations when embarking on a study. Limitations can occur in relation to the methods you apply or the research process you design. They could also be connected to you as the researcher.

Methodology limitations

Not having access to data or reliable information can impact the methods used to facilitate your research. A lack of data or reliability may limit the parameters of your study area and the extent of your exploration.

Your sample size may also be affected because you won’t have any direction on how big or small it should be and who or what you should include. Having too few participants won’t adequately represent the population or groups of people needed to draw meaningful conclusions.

Research process limitations

The study’s design can impose constraints on the process. For example, as you’re conducting the research, issues may arise that don’t conform to the data collection methodology you developed. You may not realize until well into the process that you should have incorporated more specific questions or comprehensive experiments to generate the data you need to have confidence in your results.

Constraints on resources can also have an impact. Being limited on participants or participation incentives may limit your sample sizes. Insufficient tools, equipment, and materials to conduct a thorough study may also be a factor.

Common researcher limitations

Here are some of the common researcher limitations you may encounter:

Time: some research areas require multi-year longitudinal approaches, but you might not be able to dedicate that much time. Imagine you want to measure how much memory a person loses as they age. This may involve conducting multiple tests on a sample of participants over 20–30 years, which may be impossible.

Bias: researchers can consciously or unconsciously apply bias to their research. Biases can contribute to relying on research sources and methodologies that will only support your beliefs about the research you’re embarking on. You might also omit relevant issues or participants from the scope of your study because of your biases.

Limited access to data : you may need to pay to access specific databases or journals that would be helpful to your research process. You might also need to gain information from certain people or organizations but have limited access to them. These cases require readjusting your process and explaining why your findings are still reliable.

• Why is it important to identify limitations?

Identifying limitations adds credibility to research and provides a deeper understanding of how you arrived at your conclusions.

Constraints may have prevented you from collecting specific data or information you hoped would prove or disprove your hypothesis or provide a more comprehensive understanding of your research topic.

However, identifying the limitations contributing to your conclusions can inspire further research efforts that help gather more substantial information and data.

• Where to put limitations in a research paper

A research paper is broken up into different sections that appear in the following order:

Introduction

Methodology

The discussion portion of your paper explores your findings and puts them in the context of the overall research. Either place research limitations at the beginning of the discussion section before the analysis of your findings or at the end of the section to indicate that further research needs to be pursued.

What not to include in the limitations section

Evidence that doesn’t support your hypothesis is not a limitation, so you shouldn’t include it in the limitation section. Don’t just list limitations and their degree of severity without further explanation.

• How to present limitations

You’ll want to present the limitations of your study in a way that doesn’t diminish the validity of your research and leave the reader wondering if your results and conclusions have been compromised.

Include only the limitations that directly relate to and impact how you addressed your research questions. Following a specific format enables the reader to develop an understanding of the weaknesses within the context of your findings without doubting the quality and integrity of your research.

Identify the limitations specific to your study

You don’t have to identify every possible limitation that might have occurred during your research process. Only identify those that may have influenced the quality of your findings and your ability to answer your research question.

Explain study limitations in detail

This explanation should be the most significant portion of your limitation section.

Link each limitation with an interpretation and appraisal of their impact on the study. You’ll have to evaluate and explain whether the error, method, or validity issues influenced the study’s outcome and how.

Propose a direction for future studies and present alternatives

In this section, suggest how researchers can avoid the pitfalls you experienced during your research process.

If an issue with methodology was a limitation, propose alternate methods that may help with a smoother and more conclusive research project. Discuss the pros and cons of your alternate recommendation.

Describe steps taken to minimize each limitation

You probably took steps to try to address or mitigate limitations when you noticed them throughout the course of your research project. Describe these steps in the limitation section.

• Limitation example

“Approaches like stem cell transplantation and vaccination in AD [Alzheimer’s disease] work on a cellular or molecular level in the laboratory. However, translation into clinical settings will remain a challenge for the next decade.”

The authors are saying that even though these methods showed promise in helping people with memory loss when conducted in the lab (in other words, using animal studies), more studies are needed. These may be controlled clinical trials, for example. 

However, the short life span of stem cells outside the lab and the vaccination’s severe inflammatory side effects are limitations. Researchers won’t be able to conduct clinical trials until these issues are overcome.

• How to overcome limitations in research

You’ve already started on the road to overcoming limitations in research by acknowledging that they exist. However, you need to ensure readers don’t mistake weaknesses for errors within your research design.

To do this, you’ll need to justify and explain your rationale for the methods, research design, and analysis tools you chose and how you noticed they may have presented limitations.

Your readers need to know that even when limitations presented themselves, you followed best practices and the ethical standards of your field. You didn’t violate any rules and regulations during your research process.

You’ll also want to reinforce the validity of your conclusions and results with multiple sources, methods, and perspectives. This prevents readers from assuming your findings were derived from a single or biased source.

• Learning and improving starts with limitations in research

Dealing with limitations with transparency and integrity helps identify areas for future improvements and developments. It’s a learning process, providing valuable insights into how you can improve methodologies, expand sample sizes, or explore alternate approaches to further support the validity of your findings.

Get started today

Go from raw data to valuable insights with a flexible research platform

Editor’s picks

Last updated: 21 December 2023

Last updated: 16 December 2023

Last updated: 6 October 2023

Last updated: 17 February 2024

Last updated: 5 March 2024

Last updated: 19 November 2023

Last updated: 15 February 2024

Last updated: 11 March 2024

Last updated: 12 December 2023

Last updated: 6 March 2024

Last updated: 10 April 2023

Last updated: 20 December 2023

Latest articles

Related topics, log in or sign up.

Get started for free

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker

APA Citation Generator

MLA Citation Generator

Chicago Citation Generator

Vancouver Citation Generator

• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Present the Limitations of the Study Examples

What are the limitations of a study?

The limitations of a study are the elements of methodology or study design that impact the interpretation of your research results. The limitations essentially detail any flaws or shortcomings in your study. Study limitations can exist due to constraints on research design, methodology, materials, etc., and these factors may impact the findings of your study. However, researchers are often reluctant to discuss the limitations of their study in their papers, feeling that bringing up limitations may undermine its research value in the eyes of readers and reviewers.

In spite of the impact it might have (and perhaps because of it) you should clearly acknowledge any limitations in your research paper in order to show readers—whether journal editors, other researchers, or the general public—that you are aware of these limitations and to explain how they affect the conclusions that can be drawn from the research.

In this article, we provide some guidelines for writing about research limitations, show examples of some frequently seen study limitations, and recommend techniques for presenting this information. And after you have finished drafting and have received manuscript editing for your work, you still might want to follow this up with academic editing before submitting your work to your target journal.

Why do I need to include limitations of research in my paper?

Although limitations address the potential weaknesses of a study, writing about them toward the end of your paper actually strengthens your study by identifying any problems before other researchers or reviewers find them.

Furthermore, pointing out study limitations shows that you’ve considered the impact of research weakness thoroughly and have an in-depth understanding of your research topic. Since all studies face limitations, being honest and detailing these limitations will impress researchers and reviewers more than ignoring them.

Where should I put the limitations of the study in my paper?

Some limitations might be evident to researchers before the start of the study, while others might become clear while you are conducting the research. Whether these limitations are anticipated or not, and whether they are due to research design or to methodology, they should be clearly identified and discussed in the discussion section —the final section of your paper. Most journals now require you to include a discussion of potential limitations of your work, and many journals now ask you to place this “limitations section” at the very end of your article. 

Some journals ask you to also discuss the strengths of your work in this section, and some allow you to freely choose where to include that information in your discussion section—make sure to always check the author instructions of your target journal before you finalize a manuscript and submit it for peer review .

Limitations of the Study Examples

There are several reasons why limitations of research might exist. The two main categories of limitations are those that result from the methodology and those that result from issues with the researcher(s).

Common Methodological Limitations of Studies

Limitations of research due to methodological problems can be addressed by clearly and directly identifying the potential problem and suggesting ways in which this could have been addressed—and SHOULD be addressed in future studies. The following are some major potential methodological issues that can impact the conclusions researchers can draw from the research.

Issues with research samples and selection

Sampling errors occur when a probability sampling method is used to select a sample, but that sample does not reflect the general population or appropriate population concerned. This results in limitations of your study known as “sample bias” or “selection bias.”

For example, if you conducted a survey to obtain your research results, your samples (participants) were asked to respond to the survey questions. However, you might have had limited ability to gain access to the appropriate type or geographic scope of participants. In this case, the people who responded to your survey questions may not truly be a random sample.

Insufficient sample size for statistical measurements

When conducting a study, it is important to have a sufficient sample size in order to draw valid conclusions. The larger the sample, the more precise your results will be. If your sample size is too small, it will be difficult to identify significant relationships in the data.

Normally, statistical tests require a larger sample size to ensure that the sample is considered representative of a population and that the statistical result can be generalized to a larger population. It is a good idea to understand how to choose an appropriate sample size before you conduct your research by using scientific calculation tools—in fact, many journals now require such estimation to be included in every manuscript that is sent out for review.

Lack of previous research studies on the topic

Citing and referencing prior research studies constitutes the basis of the literature review for your thesis or study, and these prior studies provide the theoretical foundations for the research question you are investigating. However, depending on the scope of your research topic, prior research studies that are relevant to your thesis might be limited.

When there is very little or no prior research on a specific topic, you may need to develop an entirely new research typology. In this case, discovering a limitation can be considered an important opportunity to identify literature gaps and to present the need for further development in the area of study.

Methods/instruments/techniques used to collect the data

After you complete your analysis of the research findings (in the discussion section), you might realize that the manner in which you have collected the data or the ways in which you have measured variables has limited your ability to conduct a thorough analysis of the results.

For example, you might realize that you should have addressed your survey questions from another viable perspective, or that you were not able to include an important question in the survey. In these cases, you should acknowledge the deficiency or deficiencies by stating a need for future researchers to revise their specific methods for collecting data that includes these missing elements.

Common Limitations of the Researcher(s)

Study limitations that arise from situations relating to the researcher or researchers (whether the direct fault of the individuals or not) should also be addressed and dealt with, and remedies to decrease these limitations—both hypothetically in your study, and practically in future studies—should be proposed.

Limited access to data

If your research involved surveying certain people or organizations, you might have faced the problem of having limited access to these respondents. Due to this limited access, you might need to redesign or restructure your research in a different way. In this case, explain the reasons for limited access and be sure that your finding is still reliable and valid despite this limitation.

Time constraints

Just as students have deadlines to turn in their class papers, academic researchers might also have to meet deadlines for submitting a manuscript to a journal or face other time constraints related to their research (e.g., participants are only available during a certain period; funding runs out; collaborators move to a new institution). The time available to study a research problem and to measure change over time might be constrained by such practical issues. If time constraints negatively impacted your study in any way, acknowledge this impact by mentioning a need for a future study (e.g., a longitudinal study) to answer this research problem.

Conflicts arising from cultural bias and other personal issues

Researchers might hold biased views due to their cultural backgrounds or perspectives of certain phenomena, and this can affect a study’s legitimacy. Also, it is possible that researchers will have biases toward data and results that only support their hypotheses or arguments. In order to avoid these problems, the author(s) of a study should examine whether the way the research problem was stated and the data-gathering process was carried out appropriately.

Steps for Organizing Your Study Limitations Section

When you discuss the limitations of your study, don’t simply list and describe your limitations—explain how these limitations have influenced your research findings. There might be multiple limitations in your study, but you only need to point out and explain those that directly relate to and impact how you address your research questions.

We suggest that you divide your limitations section into three steps: (1) identify the study limitations; (2) explain how they impact your study in detail; and (3) propose a direction for future studies and present alternatives. By following this sequence when discussing your study’s limitations, you will be able to clearly demonstrate your study’s weakness without undermining the quality and integrity of your research.

Step 1. Identify the limitation(s) of the study

• This part should comprise around 10%-20% of your discussion of study limitations.

The first step is to identify the particular limitation(s) that affected your study. There are many possible limitations of research that can affect your study, but you don’t need to write a long review of all possible study limitations. A 200-500 word critique is an appropriate length for a research limitations section. In the beginning of this section, identify what limitations your study has faced and how important these limitations are.

You only need to identify limitations that had the greatest potential impact on: (1) the quality of your findings, and (2) your ability to answer your research question.

Step 2. Explain these study limitations in detail

• This part should comprise around 60-70% of your discussion of limitations.

After identifying your research limitations, it’s time to explain the nature of the limitations and how they potentially impacted your study. For example, when you conduct quantitative research, a lack of probability sampling is an important issue that you should mention. On the other hand, when you conduct qualitative research, the inability to generalize the research findings could be an issue that deserves mention.

Explain the role these limitations played on the results and implications of the research and justify the choice you made in using this “limiting” methodology or other action in your research. Also, make sure that these limitations didn’t undermine the quality of your dissertation .

Step 3. Propose a direction for future studies and present alternatives (optional)

• This part should comprise around 10-20% of your discussion of limitations.

After acknowledging the limitations of the research, you need to discuss some possible ways to overcome these limitations in future studies. One way to do this is to present alternative methodologies and ways to avoid issues with, or “fill in the gaps of” the limitations of this study you have presented.  Discuss both the pros and cons of these alternatives and clearly explain why researchers should choose these approaches.

Make sure you are current on approaches used by prior studies and the impacts they have had on their findings. Cite review articles or scientific bodies that have recommended these approaches and why. This might be evidence in support of the approach you chose, or it might be the reason you consider your choices to be included as limitations. This process can act as a justification for your approach and a defense of your decision to take it while acknowledging the feasibility of other approaches.

P hrases and Tips for Introducing Your Study Limitations in the Discussion Section

The following phrases are frequently used to introduce the limitations of the study:

• “There may be some possible limitations in this study.”
• “The findings of this study have to be seen in light of some limitations.”
•  “The first is the…The second limitation concerns the…”
•  “The empirical results reported herein should be considered in the light of some limitations.”
• “This research, however, is subject to several limitations.”
• “The primary limitation to the generalization of these results is…”
• “Nonetheless, these results must be interpreted with caution and a number of limitations should be borne in mind.”
• “As with the majority of studies, the design of the current study is subject to limitations.”
• “There are two major limitations in this study that could be addressed in future research. First, the study focused on …. Second ….”

For more articles on research writing and the journal submissions and publication process, visit Wordvice’s Academic Resources page.

And be sure to receive professional English editing and proofreading services , including paper editing services , for your journal manuscript before submitting it to journal editors.

Wordvice Resources

Proofreading & Editing Guide

Writing the Results Section for a Research Paper

How to Write a Literature Review

Research Writing Tips: How to Draft a Powerful Discussion Section

How to Captivate Journal Readers with a Strong Introduction

Tips That Will Make Your Abstract a Success!

APA In-Text Citation Guide for Research Writing

Additional Resources

• Diving Deeper into Limitations and Delimitations (PhD student)
• Organizing Your Social Sciences Research Paper: Limitations of the Study (USC Library)
• Research Limitations (Research Methodology)
• How to Present Limitations and Alternatives (UMASS)

Article References

Pearson-Stuttard, J., Kypridemos, C., Collins, B., Mozaffarian, D., Huang, Y., Bandosz, P.,…Micha, R. (2018). Estimating the health and economic effects of the proposed US Food and Drug Administration voluntary sodium reformulation: Microsimulation cost-effectiveness analysis. PLOS. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002551

Xu, W.L, Pedersen, N.L., Keller, L., Kalpouzos, G., Wang, H.X., Graff, C,. Fratiglioni, L. (2015). HHEX_23 AA Genotype Exacerbates Effect of Diabetes on Dementia and Alzheimer Disease: A Population-Based Longitudinal Study. PLOS. Retrieved from https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001853

What are the limitations in research and how to write them?

Learn about the potential limitations in research and how to appropriately address them in order to deliver honest and ethical research.

It is fairly uncommon for researchers to stumble into the term research limitations when working on their research paper. Limitations in research can arise owing to constraints on design, methods, materials, and so on, and these aspects, unfortunately, may have an influence on your subject’s findings.

In this Mind The Graph’s article, we’ll discuss some recommendations for writing limitations in research , provide examples of various common types of limitations, and suggest how to properly present this information.

What are the limitations in research?

The limitations in research are the constraints in design, methods or even researchers’ limitations that affect and influence the interpretation of your research’s ultimate findings. These are limitations on the generalization and usability of findings that emerge from the design of the research and/or the method employed to ensure validity both internally and externally. 

Researchers are usually cautious to acknowledge the limitations of their research in their publications for fear of undermining the research’s scientific validity. No research is faultless or covers every possible angle. As a result, addressing the constraints of your research exhibits honesty and integrity .

Why should include limitations of research in my paper?

Though limitations tackle potential flaws in research, commenting on them at the conclusion of your paper, by demonstrating that you are aware of these limitations and explaining how they impact the conclusions that may be taken from the research, improves your research by disclosing any issues before other researchers or reviewers do . 

Additionally, emphasizing research constraints implies that you have thoroughly investigated the ramifications of research shortcomings and have a thorough understanding of your research problem. 

Limits exist in any research; being honest about them and explaining them would impress researchers and reviewers more than disregarding them. 

Remember that acknowledging a research’s shortcomings offers a chance to provide ideas for future research, but be careful to describe how your study may help to concentrate on these outstanding problems.

Possible limitations examples

Here are some limitations connected to methodology and the research procedure that you may need to explain and discuss in connection to your findings.

Methodological limitations

Sample size.

The number of units of analysis used in your study is determined by the sort of research issue being investigated. It is important to note that if your sample is too small, finding significant connections in the data will be challenging, as statistical tests typically require a larger sample size to ensure a fair representation and this can be limiting. 

Lack of available or reliable data

A lack of data or trustworthy data will almost certainly necessitate limiting the scope of your research or the size of your sample, or it can be a substantial impediment to identifying a pattern and a relevant connection.

Lack of prior research on the subject

Citing previous research papers forms the basis of your literature review and aids in comprehending the research subject you are researching. Yet there may be little if any, past research on your issue.

The measure used to collect data

After finishing your analysis of the findings, you realize that the method you used to collect data limited your capacity to undertake a comprehensive evaluation of the findings. Recognize the flaw by mentioning that future researchers should change the specific approach for data collection.

Issues with research samples and selection

Sampling inaccuracies arise when a probability sampling method is employed to choose a sample, but that sample does not accurately represent the overall population or the relevant group. As a result, your study suffers from “sampling bias” or “selection bias.”

Limitations of the research

When your research requires polling certain persons or a specific group, you may have encountered the issue of limited access to these interviewees. Because of the limited access, you may need to reorganize or rearrange your research. In this scenario, explain why access is restricted and ensure that your findings are still trustworthy and valid despite the constraint.

Time constraints

Practical difficulties may limit the amount of time available to explore a research issue and monitor changes as they occur. If time restrictions have any detrimental influence on your research, recognize this impact by expressing the necessity for a future investigation.

Due to their cultural origins or opinions on observed events, researchers may carry biased opinions, which can influence the credibility of a research. Furthermore, researchers may exhibit biases toward data and conclusions that only support their hypotheses or arguments.

The structure of the limitations section 

The limitations of your research are usually stated at the beginning of the discussion section of your paper so that the reader is aware of and comprehends the limitations prior to actually reading the rest of your findings, or they are stated at the end of the discussion section as an acknowledgment of the need for further research.

The ideal way is to divide your limitations section into three steps: 

1. Identify the research constraints; 

2. Describe in great detail how they affect your research; 

3. Mention the opportunity for future investigations and give possibilities. 

By following this method while addressing the constraints of your research, you will be able to effectively highlight your research’s shortcomings without jeopardizing the quality and integrity of your research.

Present your research or paper in an innovative way

If you want your readers to be engaged and participate in your research, try Mind The Graph tool to add visual assets to your content. Infographics may improve comprehension and are easy to read, just as the Mind The Graph tool is simple to use and offers a variety of templates from which you can select the one that best suits your information.

Subscribe to our newsletter

Exclusive high quality content about effective visual communication in science.

About Jessica Abbadia

Jessica Abbadia is a lawyer that has been working in Digital Marketing since 2020, improving organic performance for apps and websites in various regions through ASO and SEO. Currently developing scientific and intellectual knowledge for the community's benefit. Jessica is an animal rights activist who enjoys reading and drinking strong coffee.

Content tags

Research Limitations: A Comprehensive Guide

Embarking on a research journey is an exciting endeavor, but every study has its boundaries and constraints. Understanding and transparently acknowledging these limitations is a crucial aspect of scholarly work. In this guide, we'll explore the concept of research limitations, why they matter, and how to effectively address and navigate them in your academic endeavors.

1. Defining Research Limitations:

• Definition: Research limitations are the constraints or shortcomings that affect the scope, applicability, and generalizability of a study.
• Inherent in Research: Every research project, regardless of its scale or significance, possesses limitations.

2. Types of Research Limitations:

• Methodological Limitations: Constraints related to the research design, data collection methods, or analytical techniques.
• Sampling Limitations: Issues associated with the representativeness or size of the study sample.
• Contextual Limitations: Restrictions stemming from the specific time, place, or cultural context of the study.
• Resource Limitations: Constraints related to time, budget, or access to necessary resources.

3. Why Acknowledge Limitations?

• Transparency: Acknowledging limitations demonstrates transparency and honesty in your research.
• Robustness of Findings: Recognizing limitations adds nuance to your findings, making them more robust.
• Future Research Directions: Addressing limitations provides a foundation for future researchers to build upon.

4. Identifying Research Limitations:

• Reflect on Methodology: Consider the strengths and weaknesses of your research design, data collection methods, and analysis.
• Examine Sample Characteristics: Evaluate the representativeness and size of your study sample.
• Consider External Factors: Assess external factors that may impact the generalizability of your findings.

5. How to Address Limitations:

• In the Methodology Section: Clearly articulate limitations in the methodology section of your research paper.
• Offer Solutions: If possible, propose ways to mitigate or address identified limitations.
• Future Research Suggestions: Use limitations as a springboard to suggest areas for future research.

6. Common Phrases to Express Limitations:

• "This study is not without limitations."
• "One limitation of our research is..."
• "It is important to acknowledge the constraints of this study, including..."

7. Examples of Addressing Limitations:

• Example 1 (Methodological): "While our survey provided valuable insights, the reliance on self-reported data introduces the possibility of response bias."
• Example 2 (Sampling): "The small sample size of our study limits the generalizability of our findings to a broader population."
• Example 3 (Resource): "Due to budget constraints, our research was limited to a single geographical location, potentially impacting the external validity."

8. Balancing Strengths and Limitations:

• Emphasize Contributions: Highlight the contributions and strengths of your research alongside the limitations.
• Maintain a Positive Tone: Discuss limitations objectively without undermining the significance of your study.

9. Feedback and Peer Review:

• Seek Feedback: Share your research with peers or mentors to gain valuable insights.
• Peer Review: Embrace the feedback received during the peer-review process to enhance the robustness of your work.

10. Continuous Reflection:

• Throughout the Research Process: Continuously reflect on potential limitations during the entire research process.
• Adjust as Needed: Be willing to adjust your approach as you encounter unforeseen challenges.

Conclusion:

Understanding and effectively addressing research limitations is a hallmark of rigorous and responsible scholarship. By openly acknowledging these constraints, you not only enhance the credibility of your work but also contribute to the broader academic discourse. Embrace the nuances of your research journey, navigate its limitations thoughtfully, and pave the way for future investigations.

Related Guides

Research Limitations & Delimitations

What they are and how they’re different (with examples)

By: Derek Jansen (MBA) | Expert Reviewed By: David Phair (PhD) | September 2022

If you’re new to the world of research, you’ve probably heard the terms “ research limitations ” and “ research delimitations ” being thrown around, often quite loosely. In this post, we’ll unpack what both of these mean, how they’re similar and how they’re different – so that you can write up these sections the right way.

Overview: Limitations vs Delimitations

• Are they the same?
• What are research limitations
• What are research delimitations
• Limitations vs delimitations

First things first…

Let’s start with the most important takeaway point of this post – research limitations and research delimitations are not the same – but they are related to each other (we’ll unpack that a little later). So, if you hear someone using these two words interchangeably, be sure to share this post with them!

Research Limitations

Research limitations are, at the simplest level, the weaknesses of the study, based on factors that are often outside of your control as the researcher. These factors could include things like time , access to funding, equipment , data or participants . For example, if you weren’t able to access a random sample of participants for your study and had to adopt a convenience sampling strategy instead, that would impact the generalizability of your findings and therefore reflect a limitation of your study.

Research limitations can also emerge from the research design itself . For example, if you were undertaking a correlational study, you wouldn’t be able to infer causality (since correlation doesn’t mean certain causation). Similarly, if you utilised online surveys to collect data from your participants, you naturally wouldn’t be able to get the same degree of rich data that you would from in-person interviews .

Simply put, research limitations reflect the shortcomings of a study , based on practical (or theoretical) constraints that the researcher faced. These shortcomings limit what you can conclude from a study, but at the same time, present a foundation for future research . Importantly, all research has limitations , so there’s no need to hide anything here – as long as you discuss how the limitations might affect your findings, it’s all good.

Research Delimitations

Alright, now that we’ve unpacked the limitations, let’s move on to the delimitations .

Research delimitations are similar to limitations in that they also “ limit ” the study, but their focus is entirely different. Specifically, the delimitations of a study refer to the scope of the research aims and research questions . In other words, delimitations reflect the choices you, as the researcher, intentionally make in terms of what you will and won’t try to achieve with your study. In other words, what your research aims and research questions will and won’t include.

As we’ve spoken about many times before, it’s important to have a tight, narrow focus for your research, so that you can dive deeply into your topic, apply your energy to one specific area and develop meaningful insights. If you have an overly broad scope or unfocused topic, your research will often pull in multiple, even opposing directions, and you’ll just land up with a muddy mess of findings .

So, the delimitations section is where you’ll clearly state what your research aims and research questions will focus on – and just as importantly, what they will exclude . For example, you might investigate a widespread phenomenon, but choose to focus your study on a specific age group, ethnicity or gender. Similarly, your study may focus exclusively on one country, city or even organization. As long as the scope is well justified (in other words, it represents a novel, valuable research topic), this is perfectly acceptable – in fact, it’s essential. Remember, focus is your friend.

Need a helping hand?

Conclusion: Limitations vs Delimitations

Ok, so let’s recap.

Research limitations and research delimitations are related in that they both refer to “limits” within a study. But, they are distinctly different. Limitations reflect the shortcomings of your study, based on practical or theoretical constraints that you faced.

Contrasted to that, delimitations reflect the choices that you made in terms of the focus and scope of your research aims and research questions. If you want to learn more about research aims and questions, you can check out this video post , where we unpack those concepts in detail.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

17 Comments

Good clarification of ideas on how a researcher ought to do during Process of choice

Thank you so much for this very simple but explicit explanation on limitation and delimitation. It has so helped me to develop my masters proposal. hope to recieve more from your site as time progresses

Thank you for this explanation – very clear.

Thanks for the explanation, really got it well.

This website is really helpful for my masters proposal

Thank you very much for helping to explain these two terms

I spent almost the whole day trying to figure out the differences

when I came across your notes everything became very clear

thanks for the clearly outlined explanation on the two terms, limitation and delimitation.

Very helpful Many thanks 🙏

Excellent it resolved my conflict .

I would like you to assist me please. If in my Research, I interviewed some participants and I submitted Questionnaires to other participants to answered to the questions, in the same organization, Is this a Qualitative methodology , a Quantitative Methodology or is it a Mixture Methodology I have used in my research? Please help me

How do I cite this article in APA format

Really so great ,finally have understood it’s difference now

Getting more clear regarding Limitations and Delimitation and concepts

I really appreciate your apt and precise explanation of the two concepts namely ; Limitations and Delimitations.

thank you for this, very helpful to researchers

Very good explained

Great and clear explanation, after a long confusion period on the two words, i can now explain to someone with ease.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

How to Identify Limitations in Research

4-minute read

• 7th March 2022

Whether you’re a veteran researcher with years of experience under your belt or a novice to the field that’s feeling overwhelmed with where to start, you must understand that every study has its limitations. These are restrictions that arise from the study’s design, or the methodology implemented during the testing phase. Unfortunately, research limitations will always exist due to the subjective nature of testing a hypothesis. We’ve compiled some helpful information below on how to identify and accept research limitations and use them to your advantage. Essentially, we’ll show you how to make lemonade (a brilliant piece of academic work ) from the lemons you receive (the constraints your study reveals).

Research Limitations

So, let’s dive straight in, shall we? It’s always beneficial (and good practice) to disclose your research limitations . A common thought is that divulging these shortcomings will undermine the credibility and quality of your research. However, this is certainly not the case— stating the facts upfront not only reinforces your reputation as a researcher but also lets the assessor or reader know that you’re confident and transparent about the results and relevance of your study, despite these constraints.

Additionally, it creates a gap for more research opportunities, where you can analyze these limitations and determine how to incorporate or address them in a new batch of tests or create a new hypothesis altogether. Another bonus is that it helps readers to understand the optimum conditions for how to apply the results of your testing. This is a win-win, making for a far more persuasive research paper .

Now that you know why you should clarify your research limitations, let’s focus on which ones take precedence and should be disclosed. Any given research project can be vulnerable to various hindrances, so how do you identify them and single out the most significant ones to discuss? Well, that depends entirely on the nature of your study. You’ll need to comb through your research approach, methodology, testing processes, and expected results to identify the type of limitations your study may be exposed to. It’s worth noting that this understanding can only offer a broad idea of the possible restrictions you’ll face and may potentially change throughout the study.

We’ve compiled a list of the most common types of research limitations that you may encounter so you can adequately prepare for them and remain vigilant during each stage of your study.

Sample Size:

It’s critical that you choose a sample size that accurately represents the population you wish to test your theory on. If a sample is too small, the results cannot reliably be generalized across a large population.

Methodology:

The method you choose before you commence testing might seem effective in theory, but too many stumbling blocks during the testing phase can influence the accuracy and reliability of the results.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

Collection of Data:

The methods you utilize to obtain your research—surveys, emails, in-person interviews, phone calls—will directly influence the type of results your study yields.

Age of Data:

The nature of the information—and how far back it goes—affects the type of assumptions you can make. Extrapolating older data for a current hypothesis can significantly change the outcome of your testing.

Time Constraints:

Working within the deadline of when you need to submit your findings will determine the extent of your research and testing and, therefore, can heavily impact your results. Limited time frames for testing might mean not achieving the scope of results you were originally looking for.

Limited Budget:

Your study may require equipment and other resources that can become extremely costly. Budget constraints may mean you cannot acquire advanced software, programs, or travel to multiple destinations to interview participants. All of these factors can substantially influence your results.

So, now that you know how to determine your research limitations and the types you might experience, where should you document it? It’s commonly disclosed at the beginning of your discussion section , so the reader understands the shortcomings of your study before digging into the juicy bit—your findings. Alternatively, you can detail the constraints faced at the end of the discussion section to emphasize the requirements for the completion of further studies.

We hope this post will prepare you for some of the pitfalls you may encounter when conducting and documenting your research. Once you have a first draft ready, consider submitting a free sample to us for proofreading to ensure that your writing is concise and error-free and your results—despite their limitations— shine through.

Share this article:

Post A New Comment

Got content that needs a quick turnaround? Let us polish your work. Explore our editorial business services.

3-minute read

What Is a Content Editor?

Are you interested in learning more about the role of a content editor and the...

The Benefits of Using an Online Proofreading Service

Proofreading is important to ensure your writing is clear and concise for your readers. Whether...

2-minute read

6 Online AI Presentation Maker Tools

Creating presentations can be time-consuming and frustrating. Trying to construct a visually appealing and informative...

What Is Market Research?

No matter your industry, conducting market research helps you keep up to date with shifting...

8 Press Release Distribution Services for Your Business

In a world where you need to stand out, press releases are key to being...

How to Get a Patent

In the United States, the US Patent and Trademarks Office issues patents. In the United...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Writing Limitations of Research Study — 4 Reasons Why It Is Important!

It is not unusual for researchers to come across the term limitations of research during their academic paper writing. More often this is interpreted as something terrible. However, when it comes to research study, limitations can help structure the research study better. Therefore, do not underestimate significance of limitations of research study.

Allow us to take you through the context of how to evaluate the limits of your research and conclude an impactful relevance to your results.

Table of Contents

What Are the Limitations of a Research Study?

Every research has its limit and these limitations arise due to restrictions in methodology or research design.  This could impact your entire research or the research paper you wish to publish. Unfortunately, most researchers choose not to discuss their limitations of research fearing it will affect the value of their article in the eyes of readers.

However, it is very important to discuss your study limitations and show it to your target audience (other researchers, journal editors, peer reviewers etc.). It is very important that you provide an explanation of how your research limitations may affect the conclusions and opinions drawn from your research. Moreover, when as an author you state the limitations of research, it shows that you have investigated all the weaknesses of your study and have a deep understanding of the subject. Being honest could impress your readers and mark your study as a sincere effort in research.

Why and Where Should You Include the Research Limitations?

The main goal of your research is to address your research objectives. Conduct experiments, get results and explain those results, and finally justify your research question . It is best to mention the limitations of research in the discussion paragraph of your research article.

At the very beginning of this paragraph, immediately after highlighting the strengths of the research methodology, you should write down your limitations. You can discuss specific points from your research limitations as suggestions for further research in the conclusion of your thesis.

1. Common Limitations of the Researchers

Limitations that are related to the researcher must be mentioned. This will help you gain transparency with your readers. Furthermore, you could provide suggestions on decreasing these limitations in you and your future studies.

2. Limited Access to Information

Your work may involve some institutions and individuals in research, and sometimes you may have problems accessing these institutions. Therefore, you need to redesign and rewrite your work. You must explain your readers the reason for limited access.

3. Limited Time

All researchers are bound by their deadlines when it comes to completing their studies. Sometimes, time constraints can affect your research negatively. However, the best practice is to acknowledge it and mention a requirement for future study to solve the research problem in a better way.

4. Conflict over Biased Views and Personal Issues

Biased views can affect the research. In fact, researchers end up choosing only those results and data that support their main argument, keeping aside the other loose ends of the research.

Types of Limitations of Research

Before beginning your research study, know that there are certain limitations to what you are testing or possible research results. There are different types that researchers may encounter, and they all have unique characteristics, such as:

1. Research Design Limitations

Certain restrictions on your research or available procedures may affect your final results or research outputs. You may have formulated research goals and objectives too broadly. However, this can help you understand how you can narrow down the formulation of research goals and objectives, thereby increasing the focus of your study.

2. Impact Limitations

Even if your research has excellent statistics and a strong design, it can suffer from the influence of the following factors:

• Presence of increasing findings as researched
• Being population specific
• A strong regional focus.

3. Data or statistical limitations

In some cases, it is impossible to collect sufficient data for research or very difficult to get access to the data. This could lead to incomplete conclusion to your study. Moreover, this insufficiency in data could be the outcome of your study design. The unclear, shabby research outline could produce more problems in interpreting your findings.

How to Correctly Structure Your Research Limitations?

There are strict guidelines for narrowing down research questions, wherein you could justify and explain potential weaknesses of your academic paper. You could go through these basic steps to get a well-structured clarity of research limitations:

• Declare that you wish to identify your limitations of research and explain their importance,
• Provide the necessary depth, explain their nature, and justify your study choices.
• Write how you are suggesting that it is possible to overcome them in the future.

In this section, your readers will see that you are aware of the potential weaknesses in your business, understand them and offer effective solutions, and it will positively strengthen your article as you clarify all limitations of research to your target audience.

Know that you cannot be perfect and there is no individual without flaws. You could use the limitations of research as a great opportunity to take on a new challenge and improve the future of research. In a typical academic paper, research limitations may relate to:

1. Formulating your goals and objectives

If you formulate goals and objectives too broadly, your work will have some shortcomings. In this case, specify effective methods or ways to narrow down the formula of goals and aim to increase your level of study focus.

2. Application of your data collection methods in research

If you do not have experience in primary data collection, there is a risk that there will be flaws in the implementation of your methods. It is necessary to accept this, and learn and educate yourself to understand data collection methods.

3. Sample sizes

This depends on the nature of problem you choose. Sample size is of a greater importance in quantitative studies as opposed to qualitative ones. If your sample size is too small, statistical tests cannot identify significant relationships or connections within a given data set.

You could point out that other researchers should base the same study on a larger sample size to get more accurate results.

4. The absence of previous studies in the field you have chosen

Writing a literature review is an important step in any scientific study because it helps researchers determine the scope of current work in the chosen field. It is a major foundation for any researcher who must use them to achieve a set of specific goals or objectives.

However, if you are focused on the most current and evolving research problem or a very narrow research problem, there may be very little prior research on your topic. For example, if you chose to explore the role of Bitcoin as the currency of the future, you may not find tons of scientific papers addressing the research problem as Bitcoins are only a new phenomenon.

It is important that you learn to identify research limitations examples at each step. Whatever field you choose, feel free to add the shortcoming of your work. This is mainly because you do not have many years of experience writing scientific papers or completing complex work. Therefore, the depth and scope of your discussions may be compromised at different levels compared to academics with a lot of expertise. Include specific points from limitations of research. Use them as suggestions for the future.

Have you ever faced a challenge of writing the limitations of research study in your paper? How did you overcome it? What ways did you follow? Were they beneficial? Let us know in the comments below!

Frequently Asked Questions

Setting limitations in our study helps to clarify the outcomes drawn from our research and enhance understanding of the subject. Moreover, it shows that the author has investigated all the weaknesses in the study.

Scope is the range and limitations of a research project which are set to define the boundaries of a project. Limitations are the impacts on the overall study due to the constraints on the research design.

Limitation in research is an impact of a constraint on the research design in the overall study. They are the flaws or weaknesses in the study, which may influence the outcome of the research.

1. Limitations in research can be written as follows: Formulate your goals and objectives 2. Analyze the chosen data collection method and the sample sizes 3. Identify your limitations of research and explain their importance 4. Provide the necessary depth, explain their nature, and justify your study choices 5. Write how you are suggesting that it is possible to overcome them in the future

Excellent article ,,,it has helped me big

This is very helpful information. It has given me an insight on how to go about my study limitations.

Good comments and helpful

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• Diversity and Inclusion
• Trending Now

The Silent Struggle: Confronting gender bias in science funding

In the 1990s, Dr. Katalin Kariko’s pioneering mRNA research seemed destined for obscurity, doomed by…

• Promoting Research

Plain Language Summary — Communicating your research to bridge the academic-lay gap

Science can be complex, but does that mean it should not be accessible to the…

Addressing Barriers in Academia: Navigating unconscious biases in the Ph.D. journey

In the journey of academia, a Ph.D. marks a transitional phase, like that of a…

• Manuscripts & Grants
• Reporting Research

Unraveling Research Population and Sample: Understanding their role in statistical inference

Research population and sample serve as the cornerstones of any scientific inquiry. They hold the…

• Manuscript Preparation
• Publishing Research

Research Problem Statement — Find out how to write an impactful one!

What Is a Research Problem Statement? A research problem statement is a clear, concise, and…

How to Develop a Good Research Question? — Types & Examples

5 Effective Ways to Avoid Ghostwriting for Busy Researchers

Top 5 Key Differences Between Methods and Methodology

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

Research Limitations

It is for sure that your research will have some limitations and it is normal. However, it is critically important for you to be striving to minimize the range of scope of limitations throughout the research process.  Also, you need to provide the acknowledgement of your research limitations in conclusions chapter honestly.

It is always better to identify and acknowledge shortcomings of your work, rather than to leave them pointed out to your by your dissertation assessor. While discussing your research limitations, don’t just provide the list and description of shortcomings of your work. It is also important for you to explain how these limitations have impacted your research findings.

Your research may have multiple limitations, but you need to discuss only those limitations that directly relate to your research problems. For example, if conducting a meta-analysis of the secondary data has not been stated as your research objective, no need to mention it as your research limitation.

Research limitations in a typical dissertation may relate to the following points:

1. Formulation of research aims and objectives . You might have formulated research aims and objectives too broadly. You can specify in which ways the formulation of research aims and objectives could be narrowed so that the level of focus of the study could be increased.

2. Implementation of data collection method . Because you do not have an extensive experience in primary data collection (otherwise you would not be reading this book), there is a great chance that the nature of implementation of data collection method is flawed.

3. Sample size. Sample size depends on the nature of the research problem. If sample size is too small, statistical tests would not be able to identify significant relationships within data set. You can state that basing your study in larger sample size could have generated more accurate results. The importance of sample size is greater in quantitative studies compared to qualitative studies.

4. Lack of previous studies in the research area . Literature review is an important part of any research, because it helps to identify the scope of works that have been done so far in research area. Literature review findings are used as the foundation for the researcher to be built upon to achieve her research objectives.

However, there may be little, if any, prior research on your topic if you have focused on the most contemporary and evolving research problem or too narrow research problem. For example, if you have chosen to explore the role of Bitcoins as the future currency, you may not be able to find tons of scholarly paper addressing the research problem, because Bitcoins are only a recent phenomenon.

5. Scope of discussions . You can include this point as a limitation of your research regardless of the choice of the research area. Because (most likely) you don’t have many years of experience of conducing researches and producing academic papers of such a large size individually, the scope and depth of discussions in your paper is compromised in many levels compared to the works of experienced scholars.

You can discuss certain points from your research limitations as the suggestion for further research at conclusions chapter of your dissertation.

My e-book,  The Ultimate Guide to Writing a Dissertation in Business Studies: a step by step assistance  offers practical assistance to complete a dissertation with minimum or no stress. The e-book covers all stages of writing a dissertation starting from the selection to the research area to submitting the completed version of the work within the deadline. John Dudovskiy

Stating the Obvious: Writing Assumptions, Limitations, and Delimitations

During the process of writing your thesis or dissertation, you might suddenly realize that your research has inherent flaws. Don’t worry! Virtually all projects contain restrictions to your research. However, being able to recognize and accurately describe these problems is the difference between a true researcher and a grade-school kid with a science-fair project. Concerns with truthful responding, access to participants, and survey instruments are just a few of examples of restrictions on your research. In the following sections, the differences among delimitations, limitations, and assumptions of a dissertation will be clarified.

Delimitations

Delimitations are the definitions you set as the boundaries of your own thesis or dissertation, so delimitations are in your control. Delimitations are set so that your goals do not become impossibly large to complete. Examples of delimitations include objectives, research questions, variables, theoretical objectives that you have adopted, and populations chosen as targets to study. When you are stating your delimitations, clearly inform readers why you chose this course of study. The answer might simply be that you were curious about the topic and/or wanted to improve standards of a professional field by revealing certain findings. In any case, you should clearly list the other options available and the reasons why you did not choose these options immediately after you list your delimitations. You might have avoided these options for reasons of practicality, interest, or relativity to the study at hand. For example, you might have only studied Hispanic mothers because they have the highest rate of obese babies. Delimitations are often strongly related to your theory and research questions. If you were researching whether there are different parenting styles between unmarried Asian, Caucasian, African American, and Hispanic women, then a delimitation of your study would be the inclusion of only participants with those demographics and the exclusion of participants from other demographics such as men, married women, and all other ethnicities of single women (inclusion and exclusion criteria). A further delimitation might be that you only included closed-ended Likert scale responses in the survey, rather than including additional open-ended responses, which might make some people more willing to take and complete your survey. Remember that delimitations are not good or bad. They are simply a detailed description of the scope of interest for your study as it relates to the research design. Don’t forget to describe the philosophical framework you used throughout your study, which also delimits your study.

Limitations

Limitations of a dissertation are potential weaknesses in your study that are mostly out of your control, given limited funding, choice of research design, statistical model constraints, or other factors. In addition, a limitation is a restriction on your study that cannot be reasonably dismissed and can affect your design and results. Do not worry about limitations because limitations affect virtually all research projects, as well as most things in life. Even when you are going to your favorite restaurant, you are limited by the menu choices. If you went to a restaurant that had a menu that you were craving, you might not receive the service, price, or location that makes you enjoy your favorite restaurant. If you studied participants’ responses to a survey, you might be limited in your abilities to gain the exact type or geographic scope of participants you wanted. The people whom you managed to get to take your survey may not truly be a random sample, which is also a limitation. If you used a common test for data findings, your results are limited by the reliability of the test. If your study was limited to a certain amount of time, your results are affected by the operations of society during that time period (e.g., economy, social trends). It is important for you to remember that limitations of a dissertation are often not something that can be solved by the researcher. Also, remember that whatever limits you also limits other researchers, whether they are the largest medical research companies or consumer habits corporations. Certain kinds of limitations are often associated with the analytical approach you take in your research, too. For example, some qualitative methods like heuristics or phenomenology do not lend themselves well to replicability. Also, most of the commonly used quantitative statistical models can only determine correlation, but not causation.

Assumptions

Assumptions are things that are accepted as true, or at least plausible, by researchers and peers who will read your dissertation or thesis. In other words, any scholar reading your paper will assume that certain aspects of your study is true given your population, statistical test, research design, or other delimitations. For example, if you tell your friend that your favorite restaurant is an Italian place, your friend will assume that you don’t go there for the sushi. It’s assumed that you go there to eat Italian food. Because most assumptions are not discussed in-text, assumptions that are discussed in-text are discussed in the context of the limitations of your study, which is typically in the discussion section. This is important, because both assumptions and limitations affect the inferences you can draw from your study. One of the more common assumptions made in survey research is the assumption of honesty and truthful responses. However, for certain sensitive questions this assumption may be more difficult to accept, in which case it would be described as a limitation of the study. For example, asking people to report their criminal behavior in a survey may not be as reliable as asking people to report their eating habits. It is important to remember that your limitations and assumptions should not contradict one another. For instance, if you state that generalizability is a limitation of your study given that your sample was limited to one city in the United States, then you should not claim generalizability to the United States population as an assumption of your study. Statistical models in quantitative research designs are accompanied with assumptions as well, some more strict than others. These assumptions generally refer to the characteristics of the data, such as distributions, correlational trends, and variable type, just to name a few. Violating these assumptions can lead to drastically invalid results, though this often depends on sample size and other considerations.

Click here to cancel reply.

You must be logged in to post a comment.

Copyright © 2024 PhDStudent.com. All rights reserved. Designed by Divergent Web Solutions, LLC .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Perspect Med Educ
• v.8(4); 2019 Aug

Limited by our limitations

Paula t. ross.

Medical School, University of Michigan, Ann Arbor, MI USA

Nikki L. Bibler Zaidi

Study limitations represent weaknesses within a research design that may influence outcomes and conclusions of the research. Researchers have an obligation to the academic community to present complete and honest limitations of a presented study. Too often, authors use generic descriptions to describe study limitations. Including redundant or irrelevant limitations is an ineffective use of the already limited word count. A meaningful presentation of study limitations should describe the potential limitation, explain the implication of the limitation, provide possible alternative approaches, and describe steps taken to mitigate the limitation. This includes placing research findings within their proper context to ensure readers do not overemphasize or minimize findings. A more complete presentation will enrich the readers’ understanding of the study’s limitations and support future investigation.

Introduction

Regardless of the format scholarship assumes, from qualitative research to clinical trials, all studies have limitations. Limitations represent weaknesses within the study that may influence outcomes and conclusions of the research. The goal of presenting limitations is to provide meaningful information to the reader; however, too often, limitations in medical education articles are overlooked or reduced to simplistic and minimally relevant themes (e.g., single institution study, use of self-reported data, or small sample size) [ 1 ]. This issue is prominent in other fields of inquiry in medicine as well. For example, despite the clinical implications, medical studies often fail to discuss how limitations could have affected the study findings and interpretations [ 2 ]. Further, observational research often fails to remind readers of the fundamental limitation inherent in the study design, which is the inability to attribute causation [ 3 ]. By reporting generic limitations or omitting them altogether, researchers miss opportunities to fully communicate the relevance of their work, illustrate how their work advances a larger field under study, and suggest potential areas for further investigation.

Goals of presenting limitations

Medical education scholarship should provide empirical evidence that deepens our knowledge and understanding of education [ 4 , 5 ], informs educational practice and process, [ 6 , 7 ] and serves as a forum for educating other researchers [ 8 ]. Providing study limitations is indeed an important part of this scholarly process. Without them, research consumers are pressed to fully grasp the potential exclusion areas or other biases that may affect the results and conclusions provided [ 9 ]. Study limitations should leave the reader thinking about opportunities to engage in prospective improvements [ 9 – 11 ] by presenting gaps in the current research and extant literature, thereby cultivating other researchers’ curiosity and interest in expanding the line of scholarly inquiry [ 9 ].

Presenting study limitations is also an ethical element of scientific inquiry [ 12 ]. It ensures transparency of both the research and the researchers [ 10 , 13 , 14 ], as well as provides transferability [ 15 ] and reproducibility of methods. Presenting limitations also supports proper interpretation and validity of the findings [ 16 ]. A study’s limitations should place research findings within their proper context to ensure readers are fully able to discern the credibility of a study’s conclusion, and can generalize findings appropriately [ 16 ].

Why some authors may fail to present limitations

As Price and Murnan [ 8 ] note, there may be overriding reasons why researchers do not sufficiently report the limitations of their study. For example, authors may not fully understand the importance and implications of their study’s limitations or assume that not discussing them may increase the likelihood of publication. Word limits imposed by journals may also prevent authors from providing thorough descriptions of their study’s limitations [ 17 ]. Still another possible reason for excluding limitations is a diffusion of responsibility in which some authors may incorrectly assume that the journal editor is responsible for identifying limitations. Regardless of reason or intent, researchers have an obligation to the academic community to present complete and honest study limitations.

A guide to presenting limitations

The presentation of limitations should describe the potential limitations, explain the implication of the limitations, provide possible alternative approaches, and describe steps taken to mitigate the limitations. Too often, authors only list the potential limitations, without including these other important elements.

Describe the limitations

When describing limitations authors should identify the limitation type to clearly introduce the limitation and specify the origin of the limitation. This helps to ensure readers are able to interpret and generalize findings appropriately. Here we outline various limitation types that can occur at different stages of the research process.

Study design

Some study limitations originate from conscious choices made by the researcher (also known as delimitations) to narrow the scope of the study [ 1 , 8 , 18 ]. For example, the researcher may have designed the study for a particular age group, sex, race, ethnicity, geographically defined region, or some other attribute that would limit to whom the findings can be generalized. Such delimitations involve conscious exclusionary and inclusionary decisions made during the development of the study plan, which may represent a systematic bias intentionally introduced into the study design or instrument by the researcher [ 8 ]. The clear description and delineation of delimitations and limitations will assist editors and reviewers in understanding any methodological issues.

Data collection

Study limitations can also be introduced during data collection. An unintentional consequence of human subjects research is the potential of the researcher to influence how participants respond to their questions. Even when appropriate methods for sampling have been employed, some studies remain limited by the use of data collected only from participants who decided to enrol in the study (self-selection bias) [ 11 , 19 ]. In some cases, participants may provide biased input by responding to questions they believe are favourable to the researcher rather than their authentic response (social desirability bias) [ 20 – 22 ]. Participants may influence the data collected by changing their behaviour when they are knowingly being observed (Hawthorne effect) [ 23 ]. Researchers—in their role as an observer—may also bias the data they collect by allowing a first impression of the participant to be influenced by a single characteristic or impression of another characteristic either unfavourably (horns effect) or favourably (halo effort) [ 24 ].

Data analysis

Study limitations may arise as a consequence of the type of statistical analysis performed. Some studies may not follow the basic tenets of inferential statistical analyses when they use convenience sampling (i.e. non-probability sampling) rather than employing probability sampling from a target population [ 19 ]. Another limitation that can arise during statistical analyses occurs when studies employ unplanned post-hoc data analyses that were not specified before the initial analysis [ 25 ]. Unplanned post-hoc analysis may lead to statistical relationships that suggest associations but are no more than coincidental findings [ 23 ]. Therefore, when unplanned post-hoc analyses are conducted, this should be clearly stated to allow the reader to make proper interpretation and conclusions—especially when only a subset of the original sample is investigated [ 23 ].

Study results

The limitations of any research study will be rooted in the validity of its results—specifically threats to internal or external validity [ 8 ]. Internal validity refers to reliability or accuracy of the study results [ 26 ], while external validity pertains to the generalizability of results from the study’s sample to the larger, target population [ 8 ].

Examples of threats to internal validity include: effects of events external to the study (history), changes in participants due to time instead of the studied effect (maturation), systematic reduction in participants related to a feature of the study (attrition), changes in participant responses due to repeatedly measuring participants (testing effect), modifications to the instrument (instrumentality) and selecting participants based on extreme scores that will regress towards the mean in repeat tests (regression to the mean) [ 27 ].

Threats to external validity include factors that might inhibit generalizability of results from the study’s sample to the larger, target population [ 8 , 27 ]. External validity is challenged when results from a study cannot be generalized to its larger population or to similar populations in terms of the context, setting, participants and time [ 18 ]. Therefore, limitations should be made transparent in the results to inform research consumers of any known or potentially hidden biases that may have affected the study and prevent generalization beyond the study parameters.

Explain the implication(s) of each limitation

Authors should include the potential impact of the limitations (e.g., likelihood, magnitude) [ 13 ] as well as address specific validity implications of the results and subsequent conclusions [ 16 , 28 ]. For example, self-reported data may lead to inaccuracies (e.g. due to social desirability bias) which threatens internal validity [ 19 ]. Even a researcher’s inappropriate attribution to a characteristic or outcome (e.g., stereotyping) can overemphasize (either positively or negatively) unrelated characteristics or outcomes (halo or horns effect) and impact the internal validity [ 24 ]. Participants’ awareness that they are part of a research study can also influence outcomes (Hawthorne effect) and limit external validity of findings [ 23 ]. External validity may also be threatened should the respondents’ propensity for participation be correlated with the substantive topic of study, as data will be biased and not represent the population of interest (self-selection bias) [ 29 ]. Having this explanation helps readers interpret the results and generalize the applicability of the results for their own setting.

Provide potential alternative approaches and explanations

Often, researchers use other studies’ limitations as the first step in formulating new research questions and shaping the next phase of research. Therefore, it is important for readers to understand why potential alternative approaches (e.g. approaches taken by others exploring similar topics) were not taken. In addition to alternative approaches, authors can also present alternative explanations for their own study’s findings [ 13 ]. This information is valuable coming from the researcher because of the direct, relevant experience and insight gained as they conducted the study. The presentation of alternative approaches represents a major contribution to the scholarly community.

Describe steps taken to minimize each limitation

No research design is perfect and free from explicit and implicit biases; however various methods can be employed to minimize the impact of study limitations. Some suggested steps to mitigate or minimize the limitations mentioned above include using neutral questions, randomized response technique, force choice items, or self-administered questionnaires to reduce respondents’ discomfort when answering sensitive questions (social desirability bias) [ 21 ]; using unobtrusive data collection measures (e.g., use of secondary data) that do not require the researcher to be present (Hawthorne effect) [ 11 , 30 ]; using standardized rubrics and objective assessment forms with clearly defined scoring instructions to minimize researcher bias, or making rater adjustments to assessment scores to account for rater tendencies (halo or horns effect) [ 24 ]; or using existing data or control groups (self-selection bias) [ 11 , 30 ]. When appropriate, researchers should provide sufficient evidence that demonstrates the steps taken to mitigate limitations as part of their study design [ 13 ].

In conclusion, authors may be limiting the impact of their research by neglecting or providing abbreviated and generic limitations. We present several examples of limitations to consider; however, this should not be considered an exhaustive list nor should these examples be added to the growing list of generic and overused limitations. Instead, careful thought should go into presenting limitations after research has concluded and the major findings have been described. Limitations help focus the reader on key findings, therefore it is important to only address the most salient limitations of the study [ 17 , 28 ] related to the specific research problem, not general limitations of most studies [ 1 ]. It is important not to minimize the limitations of study design or results. Rather, results, including their limitations, must help readers draw connections between current research and the extant literature.

The quality and rigor of our research is largely defined by our limitations [ 31 ]. In fact, one of the top reasons reviewers report recommending acceptance of medical education research manuscripts involves limitations—specifically how the study’s interpretation accounts for its limitations [ 32 ]. Therefore, it is not only best for authors to acknowledge their study’s limitations rather than to have them identified by an editor or reviewer, but proper framing and presentation of limitations can actually increase the likelihood of acceptance. Perhaps, these issues could be ameliorated if academic and research organizations adopted policies and/or expectations to guide authors in proper description of limitations.

• The role of limitations in research: why they are important
• How to Organize Limitations of a Research Study

What are the Limitations of a Study (Research)?

Why and where to include limitations in my research paper, common limitations of the researchers.

• Limited Access to Information

Time Limits

Conflicts on biased views and personal issues, different types, 1. research design limitations, 2. impact limitations, 3. data or statistical limitations, how to structure your research limitations correctly, how to set your research limitations, formulation of your objectives and aims, implementation of your data collection methods, what are sample sizes, lacking previous studies in the same field, scope of discussions, concluding thoughts.

When completing a study or any other important work, there are different details that you should include to present its comprehensive and clear description. Sometimes you might even need to hire a thesis writer to help you with the whole writing process. Don’t underrate the section with limitations in research . It plays a big role in the entire process. Some students find it difficult to write this part, while others are reluctant to include it in their academic papers. Don’t underestimate the significance of limitations in research to provide readers with an accurate context of your work and enough data to evaluate the impact and relevance of your results. What is the best way to go about them? Keep reading to find out more.

Every research has its limitations. These limitations can appear due to constraints on methodology or research design. Needless to say, this may impact your whole study or research paper. Most researchers prefer to not discuss their study limitations because they think it may decrease the value of their paper in the eyes of the audience.

Remember that it’s quite important to show your study limitations to your audience (other researchers, editors of journals, and public readers). You need to notice that you know about these limitations and about the impact they may have. It’s important to give an explanation of how your research limitations can affect the conclusions and thoughts drawn from your research. 

In this guide, you can read useful tips on how to write limitations on your future research. Read great techniques on making a proper limitations section and see examples to make sure you have got an idea of writing your qualitative research limitations. You need to understand that even if limitations show the weaknesses of your future research, including them in your study can make your paper strengthen because you show all the problems before your readers will discover them by themselves. 

Apart from this, when the author points out the study limitations, it means that you have researched all the weak sides of your study and you understand the topic deeply. Needless to say, all the studies have their limitations even if you know how to make research design properly. When you’re honest with your readers, it can impress people much better than ignoring limitations at all.

Every research has certain limitations, and it’s completely normal, but you need to minimize their range of scope in the process. Provide your acknowledgment of them in the conclusion. Identify and understand potential shortcomings in your work.

When discussing limitations in research, explain how they impact your findings because creating their short list or description isn’t enough. Your research may have many limitations. Your basic goal is to discuss the ones that relate to the research questions that you choose for a specific academic assignment.

Limitations of your qualitative research can become clear to your readers even before they start to read your study. Sometimes, people can see the limitations only when they have viewed the whole document. You have to present your study limitations clearly in the Discussion section of a researh paper . This is the final part of your work where it’s logical to place the limitations section. You should write the limitations at the very beginning of this paragraph, just after you have highlighted the strong sides of the research methodology. When you discuss the limitations before the findings are analyzed, it will help to see how to qualify and apply these findings in future research.   

Limitations related to the researcher must also be written and shown to readers. You have to provide suggestions on decreasing these limitations in both your and future studies.

Limited Access to Information 

Your study may involve some organizations and people in the research, and sometimes you may get problems with access to these organizations. Due to this, you need to redesign and rewrite your study. You need to explain the cause of limited access to your readers.

Needless to say, all the researchers have their deadlines when they need to complete their studies. Sometimes, time constraints can affect your research negatively. If this happened, you need to acknowledge it and mention a need for future research to solve the main problem. 

Some researchers can have biased views because of their cultural background or personal views. Needless to say, it can affect the research. Apart from this, researchers with biased views can choose only those results and data that support their main arguments. If you want to avoid this problem, pay your attention to the problem statement and proper data gathering.

Before you start your study or work, keep in mind that there are specific limitations to what you test or possible research results. What are their types? There are different types that students may encounter and they all have unique features, including:

• Research design limitations,
• Impact limitations,
• Data or statistical limitations.

Specific constraints on your population research or available procedures may affect the final outcomes or results that you obtain.

Even if your research has excellent stats and a strong design, it may suffer from the impact of such factors as:

• The field is conductive to incremental findings,
• Being too population-specific.
• A strong regional focus.

In some cases, it’s impossible to collect enough data or enrollment is very difficult, and all that under-powers your research results. They may stem from your study design. They produce more issues in interpreting your findings.

There are strict rules to structure this section of your academic paper where you need to justify and explain its potential weaknesses. Take these basic steps to end up with a well-structured section:

• Announce to identify your research limitations and explain their importance,
• Reflect to provide the necessary depth, explain their nature, and justify your study choices,
• Look forward to suggest how it’s possible to overcome them in the future.

They walk your readers through this section. You need them to make it clear to your target audience that you recognize potential weaknesses in your work, understand them, and can point effective solutions.

No one is perfect. It means that your work isn’t beyond possible flaws, but you need to use them as a great opportunity to overcome new challenges and improve your knowledge. In a typical academic paper, research limitations can relate to these points:

• Formulation of your objectives and aims,
• Implementation of your data collection methods,
• Sample sizes,
• Lack of previous studies in your chosen area,
• The scope of discussions.

Learn to determine them in each one.

Your work has certain shortcomings if you formulate objectives and aims in a very broad manner. What to do in this case? Specify effective methods or ways to narrow your formulation of objectives and aims to increase the level of your study focus.

If you don’t have a lot of experience in collecting primary data, there’s a certain risk that the implementation of your methods has flaws. It’s necessary to acknowledge that.

They depend on the nature of your chosen problem and their significance is bigger in quantitative studies, unlike the qualitative ones. If your sample size is very small, statistical tests will fail to identify important relationships or connections within a particular data set. How to solve this problem? State that other researchers need to base the same study on a larger sample size to end up with more accurate results. To find more information on how to identify a resesrch problem , check our guide. 

Writing a literature review is a key step in any scientific work because it helps students determine the scope of existing studies in the chosen area. Why should you use the literature review findings? They are a basic foundation for any researcher who must use them to achieve a set of specific objectives or aims. What if there are no previous works? You may face this challenge if you choose an evolving or current problem for your study or if it’s very narrow.

Feel free to include this point as a shortcoming of your work, no matter what your chosen area is. Why? The main reason is that you don’t have long years of experience in writing scientific papers or completing complex studies. That’s why the depth and scope of your discussions can be compromised in different levels compared to scholars with a lot of expertise. Include certain points from limitations in research. Use them as suggestions for the future.

Any research suffers from specific limitations that range from common flaws to serious problems in design or methodology dissertation has. The ability to set these shortcomings plays a huge role in writing a successful academic paper and earning good grades. What if you lack it? Turn to our professional thesis writers and get their expert consultation on thesis or research paper.

What comes first, the research design or research problem selection? Read on this guide from our dissertation writing service if you are struggling to answer this question. Any research paper is based on the hypothesis, datum, and methodology. These things though are not written down in the instruct...

The methodology is an important part of your dissertation. It describes a broad philosophical underpinning to your chosen research methods, either quantitative or qualitative, to explain to readers your approach better. Make sure that you’re clear about an academic basis for your choice of research ...

Students have to complete different writing assignments, and some of them are utterly complex. Every assignment has the central idea or problem, which is supposed to be discussed and analyzed during the entire work. It’s called a thesis statement. The main objective of the statement is to explain to...

Organizing Academic Research Papers: Limitations of the Study

• Purpose of Guide
• Design Flaws to Avoid
• Glossary of Research Terms
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Executive Summary
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tertiary Sources
• What Is Scholarly vs. Popular?
• Qualitative Methods
• Quantitative Methods
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Annotated Bibliography
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• How to Manage Group Projects
• Multiple Book Review Essay
• Reviewing Collected Essays
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Research Proposal
• Acknowledgements

The limitations of the study are those characteristics of design or methodology that impacted or influenced the application or interpretation of the results of your study. They are the constraints on generalizability and utility of findings that are the result of the ways in which you chose to design the study and/or the method used to establish internal and external validity.

Importance of...

Always acknowledge a study's limitations. It is far better for you to identify and acknowledge your study’s limitations than to have them pointed out by your professor and be graded down because you appear to have ignored them.

Keep in mind that acknowledgement of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgement of a study's limitations also provides you with an opportunity to demonstrate to your professor that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitiations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the findings and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Structure: How to Structure the Research Limitations Section of Your Dissertation . Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in your paper.

Here are examples of limitations you may need to describe and to discuss how they possibly impacted your findings. Descriptions of limitations should be stated in the past tense.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but to offer reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe the need for future research.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, consult with a librarian! In cases when a librarian has confirmed that there is a lack of prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design]. Note that this limitation can serve as an important opportunity to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need in future research to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing self-reported data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to take what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data contain several potential sources of bias that should be noted as limitations: (1) selective memory (remembering or not remembering experiences or events that occurred at some point in the past); (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, or documents and, for whatever reason, access is denied or otherwise limited, the reasons for this need to be described.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single research problem, the time available to investigate a research problem and to measure change or stability within a sample is constrained by the due date of your assignment. Be sure to choose a topic that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, or thing is viewed or shown in a consistently inaccurate way. It is usually negative, though one can have a positive bias as well. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places and how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. Note that if you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating bias.
• Fluency in a language -- if your research focuses on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students, for example, and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic. This deficiency should be acknowledged.

Brutus, Stéphane et al. Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations. Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods . Powerpoint Presentation. Regent University of Science and Technology.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as a pilot study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in later studies.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study  is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to reframe your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to  the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't ask a particular question in a survey that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in any future study. A underlying goal of scholarly research is not only to prove what works, but to demonstrate what doesn't work or what needs further clarification.

Brutus, Stéphane et al. Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations. Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. Limitations are not Properly Acknowledged in the Scientific Literature. Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed . January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation . Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings! After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitiations of your study. Inflating of the importance of your study's findings in an attempt hide its flaws is a big turn off to your readers. A measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated, or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Yet Another Writing Tip

A Note about Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgement about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Huberman, A. Michael and Matthew B. Miles. Data Management and Analysis Methods. In Handbook of Qualitative Research. Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: Jul 18, 2023 11:58 AM
• URL: https://library.sacredheart.edu/c.php?g=29803
• QuickSearch
• Library Catalog
• Databases A-Z
• Publication Finder
• Course Reserves
• Citation Linker
• Digital Commons
• Our Website

Research Support

• Ask a Librarian
• Appointments
• Interlibrary Loan (ILL)
• Research Guides
• Databases by Subject
• Citation Help

Using the Library

• Reserve a Group Study Room
• Renew Books
• Honors Study Rooms
• Off-Campus Access
• Library Policies
• Library Technology

User Information

• Grad Students
• Online Students
• COVID-19 Updates
• Staff Directory
• News & Announcements
• Library Newsletter

My Accounts

• Interlibrary Loan
• Staff Site Login

FIND US ON  

Scientific Research and Methodology : An introduction to quantitative research and statistics

9 study design limitations.

So far, you have learnt to ask a RQ and designs studies. In this chapter , you will learn to identify:

• limitations to internally valid.
• limitations to externally valid.
• limitations to ecologically valid.

9.1 Introduction

The type of study and the study design determine how the results of the study should be interpreted. Ideally, a study would be perfectly externally and internally valid; in practice this is very difficult to achieve. Practically every study has limitations. The results of a study should be interpreted in light of these limitations. Limitations are not necessarily problems .

Limitations generally can be discussed through three components:

• Internal validity (Sect. 3.8 ): Discuss any limitations to internal validity due to the study design (such as identifying possible confounding variables). This is related to the effectiveness of the study within the sample (Sect. 9.2 ).
• External validity (Sect. 3.9 ): Discuss how well the sample represents the intended population. This is related to the generalisability of the study to the intended population (Sect. 9.3 ).
• Ecological validity : Discuss how well the study methods, materials and context approximate the real situation being studied. This is related to the practicality of the results to real life (Sect. 9.4 ).

All these issues should be addressed when considering the study limitations.

Almost every study has limitations. Identifying potential limitations, and discussing the likely impact they have on the interpretation of the study results, is important and ethical.

Example 9.1 Delarue et al. ( 2019 ) discuss studies where subjects rate the taste of new food products. They note that taste-testing studies should (p. 78):

... allow generalizing the conclusions obtained with a consumer sample [...] to the general targeted population [i.e., external validity]... tests should be reliable in terms of accuracy and replicability [i.e., internal validity].

However, even with good internal and external validity, these studies often result in a 'high rate of failures of new launched products'. That is, the studies do not replicate the real world, and so lack ecological validity .

9.2 Limitations: internal validity

Internal validity refers to the extent to which a cause-and-effect relationship can be established in a study, eliminating other possible explanations (Sect. 3.8 ). A discussion of the limitations of internal validity should cover, as appropriate: possible confounding variables; the impact of the Hawthorne, observer, placebo and carry-over effects; the impact of any other design decisions.

If any of these issues are likely to compromise internal validity, the implications on the interpretation of the results should be discussed. For example, if the participants were not blinded, this should be clearly stated, and the conclusion should indicate that the individuals in the study may have behaved differently than usual (the Hawthorne effect).

Example 9.2 (Study limitations) A study ( Axmann et al. 2020 ) randomly allocated Ugandan farmers to receive, or not receive, hybrid maize seeds to improve internal validity. One potential threat to internal validity was that farmers receiving the hybrid seeds could share their seeds with their neighbours.

Hence, the researchers contacted the \(75\) farmers allocated to receive the hybrid seeds; none of the contacted farmers reported selling or giving seeds to other farmers. This extra step increased the internal validity of the study.

Maximizing internal validity in observational studies is more difficult than in experimental studies (e.g., random allocation is not possible). The internal validity of experimental studies involving people is often compromised because people must be informed that they are participating in a study.

Example 9.3 (Internal validity) In a study of the hand-hygiene practices of paramedics ( Barr et al. 2017 ) , self -reported hand-hygiene practices were very different than what was reported by peers . That is, how people self-report their behaviours may not align with how they actually behave, which influence the internal validity of the study.

A study evaluated using a new therapy on elderly men, and listed some limitations of their study:

... the researcher was not blinded and had prior knowledge of the research aims, disease status, and intervention. As such, these could all have influenced data recording [...] The potential of reporting bias and observer bias could be reduced by implementing blinding in future studies. --- Kabata-Piżuch et al. ( 2021 ) , p. 10

9.3 Limitations: external validity

External validity refers to the ability to generalise the findings made from the sample to the entire intended population (Sect.  3.9 ). For a study to be externally valid, it must first be internally valid: if the study of not effective in the sample studied (i.e., internally valid), the results may not apply to the intended population either.

External validity refers to how well the sample is likely to represent the intended population in the RQ.

If the population is Alaskans, then the study is externally valid if the sample is representative of Alaskans. The results do not have to apply to people in the rest of the United States (though this can be commented on, too). The intended population is Alaskans .

External validity depends on how the sample was obtained. Results from random samples (Sects.  5.4 to  5.8 ) are likely to generalise to the population and be externally valid. (The analyses in this book assume all samples are simple random samples .) Furthermore, results from approximately representative samples (Sect.  5.9 ) may generalise to the population and be externally valid if those in the study are not obviously different than those not in the study.

Example 9.4 (External validity) A New Zealand study ( Gammon et al. 2012 ) identified (for well-documented reasons) a population of interest: 'women of South Asian origin living in New Zealand' (p. 21). The women in the sample were 'women of South Asian origin [...] recruited using a convenience sample method throughout Auckland' (p. 21).

The results may not generalise to the intended population ( all New Zealand women) because all the women in the sample came from Auckland, and the sample was not a random sample.

Example 9.5 (Using biochar) A study of growing ginger using biochar ( Farrar et al. 2018 ) used one farm at Mt Mellum, Australia. The results may only generalise to growing ginger at Mt Mellum, but since ginger is usually grown in similar types of climates and soils, the results may apply to other ginger farms also.

9.4 Limitations: ecological validity

The likely practicality of the study results in the real world should also be discussed. This is called ecological validity .

Definition 9.1 (Ecological validity) A study is ecologically valid if the study methods, materials and context closely approximate the real situation of interest.

Studies don't need to be ecologically valid to be useful; much can be learnt under special conditions, as long as the potential limitations are understood when applying the results to the real world. The ecological validity of experimental studies may be compromised because the experimental conditions are sometimes artificially controlled (for good reason).

Example 9.6 (Ecological validity) Consider a study to determine the proportion of people that buy coffee in a reusable cup. People could be asked about their behaviour . This study may not be ecologically valid, as how people act may not align with how they say they will act.

An alternative study could watch people buy coffees at various coffee shops, and record what people do in practice. This second study is more likely to be ecologically valid , as real-world behaviour is observed.

A study observed the effect of using high-mounted rear brake lights ( Kahane and Hertz 1998 ) , which are now commonplace. The American study showed that such lights reduced rear-end collisions by about \(50\) %. However, after making these lights mandatory, rear-end collisions reduced by only \(5\) %. Why?

9.5 Study types and limitations

Experimental studies, in general, have higher internal validity than observational studies, since more of the study design in under the control of the researchers; for example, random allocation of treatments is possible to minimise confounding.

Only well-conducted experimental studies can show cause-and-effect relationships.

However, experimental studies may suffer from poor ecological validity; for instance, laboratory experiments are often conducted under controlled temperature and humidity. Many experiments also require that people be told about being in a study (due to ethics), and so internal validity may be comprised (the Hawthorne effect).

Example 9.7 (Retrofitting) In a study of retro-fitting houses with energy-saving devices, Giandomenico, Papineau, and Rivers ( 2022 ) found large discrepancies in savings for observational studies ( \(12.2\) %) and experimental studies ( \(6.2\) %). The authors say that 'this finding reinforces the importance of using study designs with high internal validity to evaluate program savings' (p. 692).

9.6 Chapter summary

The limitations in a study need to be identified, and may be related to:

• internal validity (effectiveness): how well the study is conducted within the sample, isolating the relationship of interest.
• external validity (generalisability): how well the sample results are likely to apply to the intended population.
• ecological validity (practicality): how well the results may apply to the real-world situation.
• the type of study.

9.7 Quick review questions

Are the following statements true or false ?

• When interpreting the results of studies, the steps taken to maximize internal validity should be evaluated TRUE FALSE
• If studies are not externally valid, then they are not useful. TRUE FALSE
• When interpreting the results of studies, the steps taken to maximize external validity do not need to be evaluated TRUE FALSE
• When interpreting the results of studies, ecological validity is about the impact of the study on the environment. TRUE FALSE

9.8 Exercises

Selected answers are available in App.  E .

Exercise 9.1 A research study examined how people can save energy through lighting choices ( Gentile 2022 ) . The study states (p. 9) that the results 'are limited to the specific study and cannot be easily projected to other similar settings'.

What type of validity is being discussed here?

Exercise 9.2 Fill the blanks with the correct word: internal , external or ecological .

When interpreting the results of studies, we consider the practicality ( internal external ecological validity), the generalizability ( internal external ecological validity) and the effectiveness ( internal external ecological validity).

Exercise 9.3 A student project at the university where I work posed the RQ:

Among university students on-campus, is the percentage of word retention higher in male students than female students?

When discussing external validity , the students stated:

We cannot say whether or not the general public have better or worse word retention compared to the students that we will be studying.

Why is the statement not relevant in a discussion of external validity?

Exercise 9.4 Researchers conducted an experimental study ( Yeh et al. 2018 ) to 'determine if using a parachute prevents death or major traumatic injury when jumping from an aircraft'.

The researchers randomised \(23\) volunteers into one of two groups: wearing a parachute, or wearing an empty backpack. The response variable was a measurement of death or major traumatic injury upon landing. From the study, death or major injury was the same in both groups (0% for each group). However, the study used 'small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps' (p. 1).

Comment on the internal, external and ecological validity.

Exercise 9.5 A study examined how well hospital patients sleep at night ( Delaney et al. 2018 ) . The researchers state that 'convenience sampling was used to recruit patients' (p. 2). Later, the researchers state (p. 7):

while most healthy individuals sleep primarily or exclusively at night, it is important to consider that patients requiring hospitalization will likely require some daytime nap periods. This study looks at sleep only in the night-time period \(22\) : \(00\) -- \(07\) : \(00\) h, without the context of daytime sleep considered.

Discuss these issues using the language introduced in this chapter.

Exercise 9.6 A study ( Botelho et al. 2019 ) examined the food choices made when subjects were asked to shop for ingredients to make a last-minute meal. Half were told to prepare a 'healthy meal', and the other half told just to prepare a 'meal'. The authors stated (p. 436):

Another limitation is that results report findings from a simulated purchase. As participants did not have to pay for their selection, actual choices could be different. Participants may also have not behaved in their usual manner since they were taking part in a research study, a situation known as the Hawthorne effect.

What type of limitation is being discussed?

Limitations Section

This guide will discuss the core concepts of study limitations and provide the foundations for how to formulate this section in an academic research paper.

Scientific research is an imperfect process. The core aspect of research, to investigate research questions, on topics both known and unknown, inherently includes an element of risk. These include human error, barriers to data gathering, limited resources, and bias. Researchers are encouraged to discuss the limitations of their work to enhance the process of research, as well as to allow readers to gain an understanding of the study’s framework and value.

The limitations of a study are defined as any characteristics, traits, actions, or influences that could impact the research process , and therefore its findings . Types of limitations can differ significantly, ranging from internal aspects, such as flaws in design and methodology, to external influences that a researcher was unable to control. A study may have several limitations that impact how its findings withstand validity tests, the generalizability of conclusions, or the appropriateness of the study design in a specific context.

Importance of Discussing Limitations

Many new researchers fear openly and clearly stating the limitations of their studies as they worry it will undermine the validity and relevance of their work for readers and other professionals in the field. That is not the case , as a statement of study limitations allows the reader to better understand the conditions of the study and challenges that the researcher has encountered . Not including this section, or leaving out vital aspects, which can address anything from sampling to the specific research methodology, can be detrimental to the general research field as it establishes an incomplete and potentially fallacious depiction of the research. Within academia, it is expected that all studies have limitations to some extent. Including this section demonstrates a comprehensive and holistic understanding of the research process and topic by the author.

A discussion of limitations should be a subjective learning process that assesses the magnitude, and critically evaluates the extenuating impact of the said limitations. This leads to the importance of stating limitations as it creates opportunities for both the original author and other researchers to improve the quality and validity of any future studies. Including limitations is based on the core principle of transparency in scientific research, with the purpose to maintain mutual integrity and promote further progress in similar studies.

Descriptions of Various Limitations

• Sample size or profile – sampling is one of the most common limitations mentioned by researchers. This is often due to the difficulty of finding a perfect sample that both fits the size parameters and necessary characteristics of the study to ensure generalizability of results. Various sampling techniques are also open to error and bias, which may potentially influence outcomes. Sometimes researchers are faced with limitations in selecting samples and resort to selective picking of participants or, the opposite, including irrelevant people in the general pool to reach the necessary total.
• Availability of information or previous research – generally, studies are based on previous knowledge or theoretical concepts on a specific topic. This provides a strong foundation for developing both the design and research problem for the investigation. However, there are instances where research is done on relatively specific topics, or is very progressive. Therefore, a lack of knowledge or other previous studies may limit the scope of the analysis, lead to inaccuracies in the author’s arguments, and present an increased margin for error in many aspects of the research and methodology.
• Methodology errors – the complexity of modern research leads to potential limitations in methodology. Most often, it is regarding data collection and analysis, as these aspects can strongly influence outcomes. Data collection techniques differ and, although fitting for the study design, present strong limitations in terms of privacy, distractions, or inappropriate levels of detail.
• Bias – a potential limitation that can affect all researchers. This is a limitation that researchers attempt to avoid by ensuring there are no conflicts of interest, lack of any emotional or prejudiced attitudes towards the topic, and establishing a level of oversight by referring to an ethics committee and peer-review procedures. As humans, it is inherent that bias will be present to some extent. However, it is the responsibility of the researcher to remain objective and attempt to control any potential bias or inaccuracies throughout every stage of the research process.

Structuring and Writing Limitations in Research Paper

The limitation section should be written in such a way that it demonstrates that the author understands the core concepts of bias, confounding, and analytical self-criticism . It is not necessary to highlight every single limitation, but rather the ones that have a direct impact on the study results or the research problem. The thought process of the researcher should be presented, explaining the pros and cons of any decisions made and the circumstances which have led to the limitation. Structuring the limitations should be done in a fourfold approach:

• Identify and describe the limitation. This should be done through the use of professional terminology and accompanying definitions when necessary. The explanation of the limitation should be brief and precise to ensure that readers have a clear grasp of the issue, as well as being able to follow the author’s pattern of thought.
• Outline the potential influence or impact that the limitation may have on the study. This consists of elements such as the likelihood of occurrence, the magnitude of impact, and the general direction that a specific limitation has driven the study findings. It is generally accepted that some limitations will have a more profound influence than others. Therefore, it is vital to highlight the impact of the limitation so that readers can decide which issues to consider when examining the topic as limitations with a null value bias are less dangerous.
• Discuss alternative approaches to the specific limitations , or the research question in general. A justification should be provided by the author to support the particular approach and methodology selected in the specific study and why it was warranted within the context of any limitations. If possible, persuasive evidence should be provided and alternative decisions discussed to some extent. This demonstrates transparency of thought and reassures readers that despite potential limitations, the selected approach was the best alternative for the current research on the topic within the field of study.
• Describe techniques to minimize any risks resulting from the limitations. This may include reference to previous research and suggestions on the improvement of design and analysis.

Limitations are an inherent part of any research study. Therefore, it is generally accepted in academia to acknowledge various limitations as part of the research process. Issues may vary, ranging from sampling and literature review, to methodology and bias. However, there is a structure for identifying these elements, discussing them, and offering insight or alternatives on how limitations can be mitigated. This not only enhances the process of the research but also helps readers gain a comprehensive understanding of a study’s conditions.

Unfortunately, your browser is too old to work on this site.

For full functionality of this site it is necessary to enable JavaScript.

• Open access
• Published: 06 April 2024

Tools for assessing the methodological limitations of a QES—a short note

• Heid Nøkleby   ORCID: orcid.org/0009-0008-9844-7279 1 ,
• Heather Melanie R. Ames 1 ,
• Lars Jørun Langøien 1 &
• Christine Hillestad Hestevik 1  

Systematic Reviews volume  13 , Article number:  103 ( 2024 ) Cite this article

Metrics details

The increasing prevalence and application of qualitative evidence syntheses (QES) in decision-making processes underscore the need for robust tools to assess the methodological limitations of a completed QES. This commentary discusses the limitations of three existing tools and presents the authors’ efforts to address this gap. Through a simple comparative analysis, the three tools are examined in terms of their coverage of essential topic areas. The examination finds that existing assessment tools lack comprehensive coverage, clarity, and grounding in qualitative research principles. The authors advocate for the development of a new collaboratively developed evidence-based tool rooted in qualitative methodology and best practice methods. The conclusion emphasizes the necessity of a tool that can provide a comprehensive judgement on the methodological limitations of a QES, addressing the needs of end-users, and ultimately enhancing the trustworthiness of QES findings in decision-making processes.

Peer Review reports

As qualitative evidence syntheses (QES) are becoming more common and increasingly used in decision-making processes [ 1 , 2 , 3 , 4 , 5 ], there is a need for a tool to assess the methodological limitations of a complete QES. This methodological assessment tool could help users to understand the trust they can place in the findings of a QES and help to interpret further use. In our work, this type of assessment tool would primarily be useful when an existing QES is found that answers a commissioner’s question. In this case, we need to be able to assess the methodological limitations of the completed QES to make a judgement for the commissioner on the extent to which the findings can be trusted and used to suit their purposes. We refer to an assessment tool, and not a checklist, as a deeper methodological understanding of the limitations of a QES is needed to assess the synthesis and how its methodological limitations impact on further use. We believe that the scoring or ranking which are the products of a checklist would not allow for a deep enough evaluation of and reflection around the methodological limitations of the QES and how they relate to the context and question that the QES is going to be used in and for.

The foundation for the discussion in this commentary was a teaching experience the team had in 2022. A request for course content was how to assess the methodological limitations of a QES. We recently had an in-house discussion about the three tools we had identified as options for assessing the methodological limitations of a QES. All three tools are in beta or preliminary versions. We discovered that the assessment tools are not easily accessible. Knowledge of their existence and whereabouts is necessary to locate them. All tools have been developed to meet an internal need (fit for purpose). None of them have been developed through best practice methods [ 6 , 7 ]. The three tools are:

Tool 1 : Criteria for assessing how well a qualitative evidence syntheses (systematic reviews of qualitative studies) was conducted , a tool developed by Lewin and colleagues in 2012 [ 8 , 9 ]

Tool 2: a prototype assessment tool based on AMSTAR 2 [ 10 ], Measurement Appraisal Checklist to Assess Qualitative Evidence Syntheses (MACAQuES) by Booth and colleagues from 2019 [ 11 ]

Tool 3: Review template for qualitative evidence synthesis (QES) , developed by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) based on the ENTREQ reporting guidance [ 12 ] in 2023, is published but is still marked as “under development” [ 13 ]

We wanted to expand our in-house discussion further for teaching purposes. To do this, first, the authors compared the QES methodological assessment tools in a table and through discussion. Next, we incorporated them into an introductory course on QES methods delivered in October 2022. During the course, we had students reflect over any topics or questions they felt were missing from the existing tools based on the course content. Finally, we reflected on the student feedback and our experiences to assess and conclude that none of the tools fully met our needs.

In this short note, we aim to briefly present and compare items across the three assessment tools we identified and describe what we believe to be their strengths and limitations.

Three assessment tools

We have compared the three QES assessment tools (see Table  1 ). An x was placed in the table if an item was mentioned in a question or a prompt.

Seven of the eighteen topic areas are covered in all three tools (review question, inclusion criteria, literature search, methodological assessment of the included studies, analysis/synthesis, findings and reflexivity). Four of the topic areas are covered by two tools; a description of the excluded studies is covered in tool 1 and tool 2. Planning/protocol, conflict of interest, and confidence in the findings are covered in tool 2 and tool 3. However, tool 1 was published before the use of GRADE CERQual was implemented, so it is not surprising that that topic area is missing. Six topic areas are covered only by one assessment tool; tool 2 asks users to think through patient involvement, the description of the included studies, data extraction/coding, and dissemination bias. Tool 3 asks users to reflect on researchers’ competence, screening, and other.

Tool 1 has considerably fewer topic areas but includes prompt questions to help the user think through the topic areas. Tool 2 also provides prompt questions or items of note that users should consider when thinking through the topic area. Tool 3 is accompanied by a user guide.

All three tools require experience with and knowledge of qualitative research. This knowledge is needed to interpret the items/questions in a “qualitative manner” to ensure that methodological limitations relevant to qualitative research are assessed. Many questions are not explicitly formulated, meaning that the end user needs to understand qualitative research principles and practices to interpret and apply them. For example, a detailed knowledge around searching and where relevant qualitative evidence is located [ 13 ], a knowledge of which synthesis method is appropriate for which type of question [ 8 , 9 , 11 , 13 ], and a knowledge of the QES authors background, experience, and competence [ 13 ]. Finally, the tools raise concepts that may be new to some researchers such as the concept of the impact of dissemination bias in primary qualitative research and its implications on QES findings [ 11 ].

Need for collaboration in developing a new evidence-based methodological assessment tool for QES

Based on our comparison of the three assessment tools, we think there is a need to systematically search for map and assess existing tools. If there is not an existing tool which has been developed in an evidence-based way, then a tool should be considered. Ideally, the end goal would be to develop a new assessment tool that is based on the principles of qualitative research and qualitative evidence syntheses using best practice methods for assessment tool development. The development of the new tool should follow best practice methods so that it reflects all items relevant for the assessment of a completed QES, is based on qualitative methodology, and addresses the needs of the end user—being able to assess the limitations of a completed QES.

This process should be a collaborative effort within the QES community. The first step would be a systematic search for existing tools and the identification of relevant principles in these tools. Additional principles should be gathered from focus groups. This exploratory step would be followed by a Delphi process where stakeholders could come to an agreement on the principles that should be included in a future tool. After the consensus process has been completed, an assessment tool could begin to be developed and user tested.

Recently, this process of collaboratively developing an evidence-based tool for the assessment of the methodological limitations of primary qualitative studies included in a QES (CAMELOT) has been completed [ 5 , 14 , 15 , 16 ]. The CAMELOT project followed the same process we describe above, involving a large number of relevant stakeholders in a collaborative process to determine what was important to include and how the tool could be used. CAMELOT, along with other previous [ 17 , 18 , 19 , 20 , 21 , 22 ] and ongoing [ 23 ] projects that have used the same methodology, lead us to believe that this process would lead to an evidence-based assessment tool for the assessment of the methodological limitations of a QES. We believe that the development of a tool for assessing the methodological limitations of a qualitative evidence synthesis is needed.

In conclusion, we believe that none of the QES methodological assessment tools covered all of the areas that were raised by students as well as our reflections from working in the field. We found that the tools did not seem to be clearly grounded in qualitative research methods (for example words or expressions common in quantitative research were used). We also found that they could not provide a comprehensive/complete judgement on the methodological limitations of a QES that we could present to a commissioner or use to make a decision as critical areas or items were missing that we feel should be considered. We believe that the development of a tool for assessing the methodological limitations of a qualitative evidence synthesis is needed.

Availability of data and materials

Not applicable.

Lewin S, Glenton C, Lawrie TA, Downe S, Finlayson KW, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 2–using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):1–18.

Article   Google Scholar  

Downe S, Finlayson KW, Lawrie TA, Lewin SA, Glenton C, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 1–using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):1–12.

Glenton C, Lewin S, Lawrie TA, Barreix M, Downe S, Finlayson KW, et al. Qualitative evidence synthesis (QES) for guidelines: paper 3–using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Syst. 2019;17(1):1–14.

Flemming K, Noyes J. Qualitative evidence synthesis: where are we at? Int J Qual Methods. 2021;20:1609406921993276.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

Article   PubMed   PubMed Central   Google Scholar  

Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6:1–9.

Lewin S. Criteria for assessing how well a qualitative evidence syntheses (systematic reviews of qualitative studies) was conducted; EPOC resources for review authors. Oslo: Norwegian Institute of Public Health; 2018.

Google Scholar  

Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Booth AftCQaIMG. Prototype - Measurement Appraisal Checklist to Assess Qualitative Evidence Syntheses (QES) (MACAQuES). Sheffield: University of Sheffield; 2019.

Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):1–8.

SBU. Granskningsmall för kvalitativa evidenssynteser (QES) (Tool to assess methodological limitations of qualitative evidence synthesis). Stockholm: SBU; 2023.

Munthe-Kaas H, Bohren MA, Glenton C, Lewin S, Noyes J, Tunçalp Ö, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implement Sci. 2018;13(1):25–32.

Munthe-Kaas HM, Sommer I, Noyes J, Cooper S, Garside R, Hannes K, et al. Development of the CAMELOT approach for considering methodological limitations of qualitative research in the context of GRADE-CERQual and qualitative evidence syntheses – protocol (version 1). Geneve: Zenodo; 2023.

Munthe-Kaas AHB, Sommer I, Cooper S, Garside R, Hannes K, Noyes J. Developing CAMELOT for assessing methodological limitations of qualitative research for inclusion in qualitative evidence syntheses. Submitted to Cochrane Evidence Synthesis and Methods. 2024.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;2021:372.

Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–12.

Article   PubMed   Google Scholar  

France E, Ring N, Noyes J, Maxwell M, Jepson R, Duncan E, et al. Protocol-developing meta-ethnography reporting guidelines (eMERGe). BMC Med Res Methodol. 2015;15:1–14.

France EF, Cunningham M, Ring N, Uny I, Duncan EA, Jepson RG, et al. Improving reporting of meta-ethnography: the eMERGe reporting guidance. BMC Med Res Methodol. 2019;19(1):1–13.

Cunningham M, France EF, Ring N, Uny I, Duncan EA, Roberts RJ, et al. Developing a reporting guideline to improve meta-ethnography in health research: the eMERGe mixed-methods study. Health Serv Deliv Res. 2019;7(4):1–116.

Svendsen C, Whaley P, Vist GE, Husøy T, Beronius A, Di Consiglio E, et al. Protocol for designing INVITES-IN, a tool for assessing the internal validity of in vitro studies. Evid Based Toxicol. 2023;1(1):2232415.

Download references

Acknowledgements

Thanks to the authors of the tools for sharing information.

This work was funded by the Norwegian Institute of Public Health.

Author information

Authors and affiliations.

Norwegian Institute of Public Health, Oslo, Norway

Heid Nøkleby, Heather Melanie R. Ames, Lars Jørun Langøien & Christine Hillestad Hestevik

You can also search for this author in PubMed   Google Scholar

Contributions

All four authors (HN, HA, LJL, CH) participated in planning, giving, or evaluating the initial teaching experience. HA had the idea to the commentary. HN made the comparison between the three assessment tools and sketched the text, and HA, LJL, and CH revised it. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Heid Nøkleby .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Nøkleby, H., Ames, H.M.R., Langøien, L.J. et al. Tools for assessing the methodological limitations of a QES—a short note. Syst Rev 13 , 103 (2024). https://doi.org/10.1186/s13643-024-02511-6

Download citation

Received : 10 January 2024

Accepted : 07 March 2024

Published : 06 April 2024

DOI : https://doi.org/10.1186/s13643-024-02511-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative evidence synthesis
• Assessment tool
• Methodological limitations

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

ORIGINAL RESEARCH article

Glossary of healthcare pathways: a methodological approach involving a transdisciplinary team in public health.

• 1 Health, Systemic, Process, UR 4129 Research Unit, University Claude Bernard Lyon 1, University of Lyon, Lyon, France
• 2 Presage Institute, University Jean Monnet, Saint-Etienne, France
• 3 CIC 1408 Inserm, CHU of Saint-Etienne, Saint-Etienne, France
• 4 Laboratory Interpsy, UR4432, University of Lorraine, Nancy, France
• 5 Hospices Civils of Lyon, Lyon, France
• 6 CHU Grenoble-Alpes-Voiron, Voiron, France

Introduction: The healthcare pathway is at the heart of public health organization concerns, but communication between the various players can be an obstacle. This work, produced by a French transdisciplinary team, offers a methodological approach based on formalized consensus to elaborate a glossary of healthcare pathways. A two-steps procedure was elaborated, including a double rounded Delphi method to formalize expert consensus, and two groups of experts: a workgroup and a review group.

Methods: The workgroup provided a list of words or expressions that, in their opinion, described, evaluated or compared the healthcare pathways for patients, caregivers or regulators. The review group checked this list and added or deleted words or expressions. Then, definitions were added by the workgroup based into account three dimensions: official, academic and from the field. The review group validated the definitions and provided complementary proposals if needed.

Results: After pooling the list of words proposed by each of the six members of the working group, 417 words/expressions were ranked. After the two rounds of evaluation, 294 words/expressions were rated “appropriate” and were analyzed by the review group. This group, after two rounds of evaluation, agreed on 263 words/expressions that were transmitted to the working group who defined them. These definitions were rated by the review group. The first round of evaluation established 195 definitions as being appropriated whereas 68 definitions were amended by the review group.

Conclusion: This glossary supports transdisciplinary communication, reduces the extent of variations in practice and optimizes decision-making. International debate on all aspects might be strengthened by an improved understanding of the concept of health pathway.

1 Introduction

The definition of care pathway was proposed by Vanhaecht et al. ( 1 ) as “a complex intervention for the mutual decision-making and organization of care processes for a well-defined group of patients during a well-defined period” ( 1 ). In 2005, the European Pathway Association defined the care pathway as a methodology for the mutual decision-making and organization of care for a well-defined group of patients during a well-defined period ( 2 ). To characterize a care pathway, it is necessary: (i) to include an explicit statement of the goals and key elements of care based on evidence, best practice, and patients’ expectations and their characteristics; (ii) to facilitate the communication among the team members, with patients and families; (iii) to coordinate the care process by coordinating the roles and sequencing the activities of the transdisciplinary care team, patients and their relatives; (iv) to document, monitor, and evaluate variances and outcomes; and (v) to identify the appropriate resources.

A care pathway aims to enhance the quality of care across the continuum by improving risk-adjusted patient outcomes, promoting patient safety, increasing patient satisfaction, and optimizing the use of resources. This definition involves interdisciplinarity and optimal collaboration between healthcare professionals, patients and researchers. Nancarrow et al. identified 10 characteristics underpinning effective interdisciplinary team work ( 3 ) such as communication strategy and structures, appropriate resources and procedures, appropriate skill mix, clarity of vision, quality and outcomes of care, and respecting and understanding roles.

In practice, healthcare teams vary dramatically in their structures and effectiveness in ways that can damage team processes and patient outcomes ( 4 ). But how can researchers from various disciplines work together on the understanding of health pathways if they do not use the same semantics? As an example of prior work, a glossary of culture in epidemiology was produced ( 5 ) to address three primary classes of challenges; definitional, theoretical and methodological, hypothesizing that culture was a determinant of population differences in health and well-being.

In France, as in many other countries, the pathway approach is at the heart of public health organization concerns. This organization has emerged as a result of the growing increase in chronic diseases and the ageing of the population in developed countries. It aims to standardize the evolution of treatments, reduce the pressure on the healthcare system and the problems related to the areas that require attention by decision-makers ( 6 ). In theory, the pathway approach has a number of benefits: (i) reducing the length of the care production process, (ii) enhancing the cohesion of care, (iii) lowering the risk of errors, (iv) reducing the cost of the care production process, (v) enhancing the patient and professional satisfaction ( 7 , 8 ). Nevertheless, in practice, the introduction of the care pathway concept in the field of public health still falls short of its potential, particularly in terms of promoting transdisciplinarity between healthcare professionals ( 9 ). Thus, an obstacle could be the lack of common semantic between the various players. This article, produced by a French transdisciplinary team, offers a glossary to enrich and contribute to this collective dynamic.

The aim of this study was to present a replicable methodology to define key concepts and terms relevant to health pathway, taking into account three dimensions: theoretical, institutional and applied. To our knowledge, this work represents the first initiative driven by a transdisciplinary team composed of healthcare professionals and academic researchers with a replicable and formalized methodology of expert consensus, based on two expert groups. The result of this study was edited in a French glossary intended for a wide audience of healthcare professionals, students and researchers in the field of the healthcare pathways.

2 Materials and methods

2.1 study design.

A two-steps procedure was specifically elaborated for the purpose of this study, including a double rounded Delphi method ( 10 , 11 ) to formalize expert consensus, and two groups of experts: a workgroup and a review group.

The first step consisted in choosing the words or expressions to be included in the glossary ( Figure 1 ); the work group had to provide a list of words or expressions that, in their opinion, described, evaluated or compared the healthcare pathways for the benefit of patients, caregivers or regulators. The review group checked the list and added or deleted words or expressions.

Figure 1 . First step of the procedure specifically elaborated for the selection of the words/expressions of the glossary of health pathways.

The second step consisted in providing the definitions from the approved list of words and expressions ( Figure 2 ); the workgroup had to search for the most appropriate definitions for each word or expression. The review group validated the definitions and provided complementary proposals if required.

Figure 2 . Second step of the procedure specifically elaborated for the definitions of the word/expressions of the glossary of health pathways.

2.2 Groups of experts

2.2.1 workgroup.

The workgroup was composed of six volunteers, associated or academic researchers of the laboratory “Health Systemic Process” (P2S, Parcours Santé Systémique of the University Claude Bernard, University Lyon 1); a practitioner specialized in neurology, a psychologist, a nurse specialized in tertiary prevention, a pharmacist specialized in health-economy, a specialist of oral health and an epidemiologist-methodologist. The workgroup aimed to constitute a list of words or expressions and to provide definitions.

2.2.2 Review group

The review group was composed of seven volunteers, associated or academic researchers of the laboratory “Health Systemic Process” (P2S, Parcours Santé Systémique of the University of Lyon 1); a pharmacist, a specialist of public health, a specialist of pathway for children in difficulties, a specialist of educational science, two specialists of health promotion and prevention and a specialist in oncology pathways. The review group aimed to check and amend the list and the definitions of words or expressions elaborated by the workgroup.

2.3 Doubled rounded Delphi and rating analysis

The consensus methods are defined as a way to synthesize information and compare contradictory opinions, with the aim to define the degree of agreement within a group of selected individuals. The design of this study was based on the Delphi consensus method to obtain a final, unique, convergent opinion of the group, as described by the recommendations of the Department for good professional practice of the Haute Autorité de Santé (HAS, French high authority for health) ( Figure 3 ) ( 10 ).

Figure 3 . Formalized expert consensus.

Each member of the workgroup established a list of words or expressions which were potentially related to the characterization or the evaluation of the health/care/life pathway. Then, the members pooled all their proposals and organized a first round of rating to determine if the words/expressions should be included in the glossary. Each word/expression was rated from 1 (totally disagree) to 9 (totally agree) with 5 to express indecision. The ratings were analyzed according to the rules described in Table 1 . If a word/expression was deemed appropriate, then it was included in the glossary. If a word/expression was deemed inappropriate, then it was excluded. If the word/expression was considered to be uncertain or when a value was missing, the members of the workgroup discussed and rated it for a second round. If an agreement was still not reached, then the word/expression was rejected.

Table 1 . Conditions for obtaining an agreement between experts according to median value and distribution of the quotations.

2.4 Structure of the definitions

The glossary was composed of all the words and expressions retained by the workgroup and the review group: each of them was defined in several sections organized as follows:

– Etymology: this section stated the origin of words and the way in which their meanings changed throughout history. For expressions, the etymology of each individual word was associated. French dictionaries constituted the main etymological source.

– Current dictionary definition: this section provided the definition from the most recent version of the French language dictionary ( Larousse ) for each word defined independently.

– Definition from an official source: when applicable. Sources in this section were extracted from official sources, such as public, governmental or institutional websites, from legal texts, such as health programs. When the definition retained was a legal text, the definition was verified by a lawyer.

– Academic definition: sources included works from the national academy of medicine, the University Press of France or the National Library of France. Also, French scientific articles published in peer-reviewed journals were used.

– Definition from the field: this section reported definitions from patient or health professional associations, hospital centers or regional agencies for health.

– Associated terms: this section referred to other words/expressions related to the words/expressions defined in the glossary.

3.1 Result of the process leading to the list of words/expressions to be defined

After pooling the list of words proposed by each of the six members of the working group, 417 words/expressions were ranked. After the first round of evaluation, 77 of them were rated as “appropriate,” 327 as “uncertain” and 13 as “inappropriate.” After discussion between the experts of the working group, the second round of evaluation permitted the elimination of 110 words/expressions rated as “inappropriate.” The remaining 294 “appropriate” words/expressions composed the list analyzed by the reviewers ( Figure 4 ).

Figure 4 . Flowchart of the selection of the words/expressions.

The review group rated each of these 294 words/expressions according to the formalized expert consensus ( Figure 3 ). During the first round, they rated 250 words/expressions as “appropriate,” 39 as “uncertain” and deleted five words/expressions. During the second round, 13 of the 39 “uncertain” were discussed until agreement. Finally, a list of 263 words/expressions were given to the working group who defined them ( Figure 4 ).

3.2 Results of the process leading to the list of definitions associated to words/expressions

The definitions of the 263 words/expressions were rated by the review group. The first round of evaluation established 195 definitions as being appropriated, and 68 definitions as requiring amendment. The 68 definitions were amended by the review group ( Figure 5 ).

Figure 5 . Flowchart of the validation of definitions by the review group.

3.3 Example of definition

Table 2 presents an example of a definition which was translated from French to English.

Table 2 . Example of the definition of “Health pathway” translated from French to English.

4 Discussion

4.1 main results.

The outcome of this collective work is a glossary about the field of health pathways composed of 263 words/expressions defined by different sources. The creation of a glossary on healthcare pathways offers several advantages and addresses various needs in the healthcare field. Firstly, it allows for the definition of specific terms as the field of care pathways can be complex and involve the use of specialized terminology. A glossary provides definitions for technical and medical terms, thus facilitating understanding among different actors. Second, by establishing an official list of terms and their definitions, it contributes to the standardization of terminology used in the field of care pathways, promoting communication among different healthcare actors, which is essential for ensuring quality care and effective coordination. Third, it enhances transdisciplinary communication and understanding among various actors in healthcare pathways (physicians, nurses, social workers, therapists, etc.). Fourth, it serves as a reference for academic and clinical research, as well as for teaching in healthcare institutions and universities.

The glossary is aimed at several audiences: students from the health sector, amongst which PhD candidates; trainees and interns working with research laboratories; the medical, paramedical and scientific professionals collaborating on the management of particular diseases or patient population; the patients and their caregivers wishing to learn more or to improve their level of literacy concerning the health pathways.

In order to disseminate this glossary and make it accessible, the communication medium could be a book or an e-book from a publisher, a dedicated website, etc. However, whatever format you choose, the words will be classified in alphabetical order. A table of content will be placed upstream. For the paper format, the page number corresponding to each definition will be indicated. For the digital format, links will be created so that users can click on the word and access it directly. In addition, in this format, for each definition, if words are defined in the glossary, clicking on them will allow direct access to their definition.

The strengths of this study are the panel of the experts from the research team with several levels of expertise and seniority, who volunteered to participate in the elaboration of the glossary, and the replicable and formalized methodology of expert consensus.

4.2 Limitations

This study has several limitations. Firstly, although the methodology used to create this glossary is replicable and can be used in all countries, it is specific to the French healthcare system. Some words/expressions do not exist in other countries or the definitions are not universal and transferable to other countries. Second, the working group and the review group were set up on a voluntary basis, so the working group included only one practitioner specialized in neurology. However, other laboratory members were interviewed from time to time. Third, experts cannot be sure that all the words/expressions related to health pathways are embedded in the glossary (non-exhaustiveness) and that some words belong to common language without specific definition (examples: personal medical record or medical management). Fourth, as any work that produces up-to-date data, another limitation is to keep updated definitions, especially those relating to legal texts.

4.3 Comparison with prior work

Direct benefits on the research team were observed. After months of being close to one another, the 13 members have gone to know each other personally. This enabled them to pool their specific expertise, to debate about the scope and perimeter of the concepts, to finally find a consensus and speak the same language. On a human level, this project has been a unifying force for the members who learned from their differences. This work has given rise to other research projects currently underway.

Other initiatives have also used the Delphi formalized expert consensus to elaborate a glossary of health terms. For example, a Consensus Paper on Terminology for use in the treatment of conservative spinal deformities based on the Delphi method was used to reach a preliminary consensus before the meeting, where the terms that still needed further clarification were discussed ( 12 ). Also, an international consensus of experts was found using a Delphi study technique in 3 rounds to assess agreement and then resolve disagreement on Hidradenitis Suppurativa definitions among international experts ( 13 ). A face-to-face consensus meeting was held between Delphi survey rounds two and three in the elaboration of the semantics in the active surveillance for men with localized prostate cancer. Bruinsma et al. presented results of this research project ( 14 ) and North et al. the glossary of neurostimulation terminology ( 15 ).

5 Conclusion

This glossary supports transdisciplinary communication, reduce the extent of variations in practice and optimize decision-making. International debate on all aspects might be strengthened by an improved understanding of the concept of health pathway.

It is the first time, to our knowledge, a research team has elaborated a glossary of the health pathway, with the approved methodology of expert consensus. The glossary gained consensus across a panel of senior and professional experts and represents a transdisciplinary work. Future work is needed to address issues such as updating the definitions, and to develop international consensus about the words/expressions and definitions.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

LF: Investigation, Methodology, Writing – original draft. EV: Investigation, Writing – review & editing. H-MS: Investigation, Writing – review & editing. MG: Investigation, Writing – review & editing. CK: Investigation, Writing – review & editing. CC: Investigation, Writing – review & editing. AD-B: Investigation, Writing – review & editing. SR: Investigation, Writing – review & editing. CD: Conceptualization, Supervision, Writing – review & editing. GM: Conceptualization, Supervision, Investigation, Writing – review & editing. FC: Conceptualization, Investigation, Supervision, Writing – original draft.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

The authors thank all the members of the laboratory “Health Systemic Process” (P2S, UR4129) whose valuable efforts and long-term commitment significantly contributed to the success of this project.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. Vanhaecht, K, De Witte, K, and Sermeus, W. The impact of clinical pathways on the organisation of care processes. PhD dissertation, Belgium: KU Leuven; 2007

PubMed Abstract | Google Scholar

2. Kinsman, L, Rotter, T, James, E, Snow, P, and Willis, J. What is a clinical pathway? Development of a definition to inform the debate. BMC Med . (2010) 8:31. doi: 10.1186/1741-7015-8-31

PubMed Abstract | Crossref Full Text | Google Scholar

3. Nancarrow, SA, Booth, A, Ariss, S, Smith, T, Enderby, P, and Roots, A. Ten principles of good interdisciplinary team work. Hum Resour Health . (2013) 11:19. doi: 10.1186/1478-4491-11-19

4. West, MA, and Lyubovnikova, J. Illusions of team working in health care. J Health Organ Manag . (2013) 27:134–42. doi: 10.1108/14777261311311843

5. Hruschka, DJ, and Hadley, C. A glossary of culture in epidemiology. J Epidemiol Community Health . (2008) 62:947–51. doi: 10.1136/jech.2008.076729

6. Aspland, E, Gartner, D, and Harper, P. Clinical pathway modelling: a literature review. Health Syst . (2019) 10:1–23. doi: 10.1080/20476965.2019.1652547

7. Hall, R, Belson, D, Murali, P, and Dessouky, M. Modeling patient flows through the healthcare system In: RW Hall, editor. Patient flow: Reducing delay in healthcare delivery . International Series in Operations Research & Management Science . Boston, MA: Springer US (2006). 1–44.

Google Scholar

8. Schrijvers, G, van Hoorn, A, and Huiskes, N. The care pathway: concepts and theories: an introduction. Int J Integr Care . (2012) 12:e192. doi: 10.5334/ijic.812

9. Jabbour, M, Newton, AS, Johnson, D, and Curran, JA. Defining barriers and enablers for clinical pathway implementation in complex clinical settings. Implement Sci . (2018) 13:139. doi: 10.1186/s13012-018-0832-8

10. HAS/department for good professional practice. Development of good practice guidelines “formal consensus” method. (2015). Available at: https://www.has-sante.fr/upload/docs/application/pdf/2018-03/good_practice_guidelines_fc_method.pdf

11. Nasa, P, Jain, R, and Juneja, D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol . (2021) 11:116–29. doi: 10.5662/wjm.v11.i4.116

12. Grivas, TB, de Mauroy, JC, Négrini, S, Kotwicki, T, Zaina, F, Wynne, JH, et al. Terminology - glossary including acronyms and quotations in use for the conservative spinal deformities treatment: 8th SOSORT consensus paper. Scoliosis . (2010) 5:23. doi: 10.1186/1748-7161-5-23

13. Frew, JW, Lowes, MA, Goldfarb, N, Butt, M, Piguet, V, O’Brien, E, et al. Global harmonization of morphological definitions in hidradenitis Suppurativa for a proposed glossary. JAMA Dermatol . (2021) 157:449–55. doi: 10.1001/jamadermatol.2020.5467

14. Bruinsma, SM, Roobol, MJ, Carroll, PR, Klotz, L, Pickles, T, Moore, CM, et al. Expert consensus document: semantics in active surveillance for men with localized prostate cancer - results of a modified Delphi consensus procedure. Nat Rev Urol . (2017) 14:312–22. doi: 10.1038/nrurol.2017.26

15. North, RB, Lempka, SF, Guan, Y, Air, EL, Poree, LR, Shipley, J, et al. Glossary of Neurostimulation terminology: a collaborative neuromodulation foundation, Institute of Neuromodulation, and international neuromodulation society project. Neuromodulation . (2022) 25:1050–8. doi: 10.1016/j.neurom.2021.10.010

Keywords: public health, care pathway, glossary, Delphi consensus method, dictionarie, transdisciplinary communication, transdisciplinary research

Citation: Fraticelli L, Verot E, Späth H-M, Genton MC, Kempf C, Clement C, Darlington-Bernard A, Roy S, Dussart C, Mick G and Carrouel F (2024) Glossary of healthcare pathways: a methodological approach involving a transdisciplinary team in public health. Front. Public Health . 12:1347774. doi: 10.3389/fpubh.2024.1347774

Received: 01 December 2023; Accepted: 20 March 2024; Published: 05 April 2024.

Reviewed by:

Copyright © 2024 Fraticelli, Verot, Späth, Genton, Kempf, Clement, Darlington-Bernard, Roy, Dussart, Mick and Carrouel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Laurie Fraticelli, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Open access
• Published: 24 November 2021

Immune checkpoint inhibition for pancreatic ductal adenocarcinoma: limitations and prospects: a systematic review

• Hong-Bo Li 1 ,
• Zi-Han Yang 1 &
• Qing-Qu Guo   ORCID: orcid.org/0000-0003-3181-6865 1  

Cell Communication and Signaling volume  19 , Article number:  117 ( 2021 ) Cite this article

5192 Accesses

23 Citations

2 Altmetric

Metrics details

Pancreatic cancer is an extremely malignant tumor with the lowest 5-year survival rate among all tumors. Pancreatic ductal adenocarcinoma (PDAC), as the most common pathological subtype of pancreatic cancer, usually has poor therapeutic results. Immune checkpoint inhibitors (ICIs) can relieve failure of the tumor-killing effect of immune effector cells caused by immune checkpoints. Therefore, they have been used as a novel treatment for many solid tumors. However, PDAC is not sensitive to monotherapy with ICIs, which might be related to the inhibitory immune microenvironment of pancreatic cancer. Therefore, the way to improve the microenvironment has raised a heated discussion in recent years. Here, we elaborate on the relationship between different immune cellular components in this environment, list some current preclinical or clinical attempts to enhance the efficacy of ICIs by targeting the inhibitory tumor microenvironment of PDAC or in combination with other therapies. Such information offers a better understanding of the sophisticated tumor-microenvironment interactions, also providing insights on therapeutic guidance of PDAC targeting.

Video Abstract

As early as 2014, some researchers predicted that by 2030 pancreatic cancer would surpass breast and colorectal cancer as the second-largest tumor-related fatal disease [ 1 ]. PDAC is the most common histological subtype [ 2 ] and has a mortality rate almost equal to its incidence rate. In 2019, 45,750 of the 56,770 newly diagnosed pancreatic cancer patients in the USA will eventually die from the disease (American Cancer Society. Cancer Facts and Figures 2019; American Cancer Society: New York, NY, USA, 2019). There are three main reasons for poor prognosis of PDAC: (1) lack of specific tumor markers and early screening methods leads to late diagnosis; (2) distant metastasis occurs early, and patients often lose the opportunity for surgery; and (3) pancreatic cancer, as a “cold” tumor, has a poor response to radiotherapy and chemotherapy [ 3 ]. Surgical resection combined with chemoradiotherapy may prolong the overall survival (OS) time of patients with localized disease; however, its effect on patients with advanced-stage is still unsatisfactory [ 4 ]. Hence, considering the fact that existing treatment cannot completely cure pancreatic cancer, patients urgently need a more effective treatment. ICIs can block the co-inhibitory signaling pathway in tumor cells and promote immune-mediated tumor cell clearance [ 5 ]. ICIs have also been proved to be effective against a variety of solid tumors, including melanoma [ 6 , 7 , 8 ], as well as lung [ 9 , 10 , 11 ], renal [ 12 , 13 ] and bladder [ 14 , 15 ] cancer. However, this new treatment seems not to be entirely effective for pancreatic cancer, at least with monotherapy [ 16 ], which might be related to the unique immunosuppressive tumor microenvironment (TME) of PDAC. Therefore, reversing the silent TME to make tumors sensitive to ICI therapy may be a new effective treatment for PDAC. Therefore, in order to better apply ICI drugs to the clinical treatment of PDAC patients, we try to describe the research progress of ICI drugs in clinical and laboratory, and some assumptions of reversing the immunosuppressive pancreatic TME by targeting immune cells and small molecules, and provide some future directions to improve the therapeutic efficacy for later researchers.

The mainly immunosuppressive cells in the TME of PDAC

Regulatory t cells (tregs).

Tregs were first identified by Sakaguchi [ 17 ] and are indispensable for the maintenance of normal immune tolerance, and their deficiency leads to many autoimmune diseases [ 18 ]. However, Tregs in the TME often aggravate immunosuppression and hinder immunotherapy [ 19 ]. Tregs exert their immunosuppressive effect through two completely different pathways, tumor-intrinsic and tumor-extrinsic pathways [ 19 ]. The internal regulatory function of Tregs mainly depends on the secretion of immunosuppressive cytokines, such as transforming growth factor-β (TGF-β), interleukin (IL)-10, IL-35, and depleted IL-2, which downregulates proliferation of effector T cells. Thus, the killing effect of effector T cells on tumor cells is reduced [ 20 , 21 , 22 , 23 , 24 , 25 , 26 ]. For the extrinsic pathway, immune checkpoint molecules, such as cytotoxic T lymphocyte-associated protein (CTLA-4), are expressed on the membrane of Tregs [ 27 ]. These molecules have a high affinity for CD80/CD86 on effector cells [ 19 ]. The binding of these two receptors can lead to the following results. Firstly, competitive inhibition of the binding of the CD28 receptor to B7 on the surface of traditional T cells can inhibit the activation of T cells [ 19 ]. Secondly, indoleamine 2,3-dioxygenase (IDO) can be induced by Tregs, the rate-limiting enzyme for tryptophan metabolism to kynurenine, and further leads to the apoptosis of effector T cells caused by tryptophan deficiency [ 28 ]. Other evidence also shows that Treg affects the function of effector T cells. For example, in the tumor model of Treg removed mice, the immunosuppression of tumor-infiltrating CD8 + cells was relieved [ 29 ]. Finally, the binding of CTLA-4 and B7 also downregulates the number of B7 receptors on the surface of dendritic cells (DCs), which further hinders the inhibitory effect of functional T cells on the immune response [ 30 ]. In addition to CTLA-4, Tregs in the TME overexpress many other immunosuppressive molecules, including glucocorticoid-induced TNFR-related protein (also known as TNFRSF18), lymphocyte-activation gene (LAG)3 protein, T cell immunoglobulin mucin receptor 3 (TIM, also known as HAVCR2), OX40 (also known as TNFRSF4), programmed cell death protein (PD)-1, and inducible T-cell co-stimulator (ICOS) [ 31 , 32 , 33 ]. The complex interaction between these molecules and other components in the TME makes Tregs become a barrier in the process of immune recognition and elimination of tumor cells by effector cells (Additional file 4 ).

Myeloid-derived suppressor cells (MDSCs)

In the tumor, trauma, and other pathological states, bone marrow progenitor cells and immature myeloid cells cannot differentiate into normal mature granulocytes but form a kind of immature heterogeneous cells [ 34 ], namely MDSCs. MDSCs can be divided into two subtypes, polymorphonuclear (PMN)-MDSCs and monocytic-MDSC (M-MDSC). These two subtypes have immunosuppressive effects, and M-MDSCs have stronger immunosuppressive ability [ 35 ]. Once these immunosuppressive cells are recruited into the TME, they can be activated by the surrounding vascular endothelial growth factor (VEGF). Furthermore, activated MDSCs can produce more VEGF, which is a positive feedback process [ 36 ]. Activated MDSCs mediate immunosuppression in the TME, mainly through the consumption of amino acids necessary for the proliferation of immune cells, as well as the release of reactive oxidants such as induced NO synthase and NAPDH oxidase 2, and ultimately affect the activity of effector T cells [ 37 , 38 ]. An experiment conducted by Stromnes et al. [ 39 ] showed that depletion of granulocytic MDSC (GR MDSC) in PDA models in vivo and in vitro could increase the internal accumulation of activated CD8 + T cells and apoptosis of tumor epithelial cells.In addition, MDSCs can induce Treg proliferation by secreting IL-2 and TGF-β to mediate immunosuppression indirectly [ 40 ]. MDSCs can also upregulate the expression of programmed death-ligand (PD-L)1 [ 41 ] and promote the proliferation of Tregs, which is regulated by IL-10 secreted by activated T cells in the TME [ 40 ].

Tumor-associated macrophages (TAMs)

TAMs, one of the immune cell populations abundant in TME, are referred to the macrophages in the tumor stroma, which can be divided into two types, the M1 and M2, according to their phenotypes and functions in the view of macrophage polarization [ 42 ]. In general terms, M1 macrophages are pro-inflammatory with antitumor properties, while M2 macrophages are anti-inflammatory with both antitumor and protumor properties in TME [ 43 ]. However, the M2 type deviation with protumor effects is predominant in PDAC. Monocytes entering the TME under the influence of chemotaxis differentiate into TAMs [ 44 ]. TAMs exert their immunosuppressive function mainly through the expression of ligands or receptors. Like MDSCs, TAMs also express arginase-1 and IDO, leading to depletion of essential amino acids for T cell proliferation. In addition, TAMs overexpress PD-1, PD-L1, and HLA, which are the ligands of inhibitory receptors such as CD94 and IL-2/4. The binding of PD-1 to PD-L1 contributes to immune escape and T-cell depletion, while the binding of HLA to the inhibitory receptor IL-2/4 on the surface of T cells can directly inhibit the proliferation of T cells [ 45 , 46 ]. IL-10, TGF-1, and prostaglandin E2 in the TME can inhibit the expression of MHC-II molecules on the surface of macrophages, so the macrophages cannot effectively present tumor antigen to T cells and hinder the specific killing reaction of T cells against tumor cells [ 46 , 47 ]. In addition, Beavis et al . found that TAMs can overexpress CD73 and CD39 ectoenzymes and generate pericellular adenosine, which finally causes the suppression of Teff via activation of the adenosine A2A receptor [ 48 ], suggesting that TAMs play an important role in the immunosuppressive pancreatic TME.

Pancreatic satellite cells (PSCs)

The most significant difference between pancreatic cancer and other solid tumors is that there are 80%–90% matrix components in pancreatic cancer tissue [ 47 ]. In health, PSCs are responsible for maintaining the homeostasis of the extracellular matrix proteins. Recent studies have implied its potential immune functions in normal. Apte et al . found that quiescent PSCs can phagocytize damaged pancreatic parenchymal cells, and this can delay the progression of early pancreatic disease. However, a variety of small molecules, including cytokines such as hyperglycemia, endothelin 1, COX-2, galectin 1, and fibrinogen, can activate PSCs through the paracrine pathway [ 49 ]. Activated PSCs secrete many matrix proteins containing type I collagen, which plays a crucial role in the formation of the dense extracellular matrix of pancreatic cancer. The dense extracellular matrix constitutes a physical barrier in the TME of pancreatic cancer, blocks the infiltration of T and B lymphocytes, and affects the recognition and elimination of tumor cell antigens by these lymphocytes. In addition to participating in the formation of pancreatic cancer inhibitory TME, PSCs can also secrete various inhibitory cytokines to participate in the regulation of immune cells. For example, granulocyte–macrophage colony-stimulating factor (GM-CSF) and IL-6 secreted by PSCs can directly inhibit the migration and invasion of cytotoxic T cells and reduce the number of cytotoxic T cells in the TME. In contrast, IL-6 can induce MDSCs into the TME and exert an immunosuppressive effect [ 50 , 51 ]. Galectin-1, another cytokine highly expressed in PSCs, plays an essential role in maintaining the immunosuppressive TME and T-cell depletion [ 52 ]. PSCs significantly increase the number of other immunosuppressive cells in the PDAC microenvironment, including MDSCs, M2-TAMs, and Tregs, and decrease the number of effector T cells, natural killer (NK) cells, and M1 TAMs. Also, Ene-Obong et al. [ 53 ] reported that in KPC (Pdx-1-Cre; LSL-KrasG12D/ + ; LSL-Trp53R172H/ +) mice, PSCs sequester antitumor CD8 + T cells around nonadjacent regions in the stroma, resulting in the dysfunction of CD8 + cells to infiltrate into the pancreatic tumor epithelial cells. Therefore, PSCs aggravate the immunosuppression of PDAC [ 54 ]. However, another study showed that depletion of carcinoma associated fibroblasts would induce the process of immunosuppression and accelerates pancreas cancer, suggesting that PSC may have antitumor properties at the same time [ 55 ]. Thus, the exact role of PSC in Pancreatic TME remains ambiguous. (The mechanism of endothelial cell entry into the pancreatic cancer microenvironment and its immunosuppressive effect is summarized in Additional file 1 .)

Interacting between different compounds of PDAC microenvironment

As the core of the tumor microenvironment in pancreatic cancer, PDAC cells can directly or indirectly inhibit the immune function of T cells [ 56 ], and the direct effects include the secretion of inhibitory cytokines such as IDO and TGF-β, which directly inhibit the proliferation of T cells. The indirect inhibitory effect is related to the interaction of various immunosuppressive cell components in the tumor microenvironment of PDAC, PDAC cells can promote the proliferation and activation of MDSC by secreting GM-CSF, and MDSC can continue to secrete cytokines such as IDO, IL-10, TGF-β, Arg-1, iNOS to inhibit the cell activity and immune effect of T cells. TAMs can also receive cytokines such as CSF-1, Bag-3, TGF-β, and IL-10 secreted by PDAC and then secrete inhibitory cytokines such as Arg-1, TGF-β, and IL-10 similar to MDSCs, resulting in T cell immune dysfunction. As the representative cells negatively regulate the body’s immune function, Tregs are also regulated by TGF-β and IL-10 secreted by PDAC cells. All of these reflect the core role of PDAC cells in the immunosuppressive tumor microenvironment of pancreatic cancer. PSC cells not only promote the accumulation of extracellular matrix and participate in the interstitial components of PDAC, but also secrete IL-6 and GM-CSF to promote the proliferation of PDAC cells. At the same time, IL-6 can also induce MDSC cells to enter the tumor microenvironment and activate [ 57 ] (Fig.  1 ).

The relationship between different cellular components in the tumor microenvironment of PDAC

Research progress on IC molecules

Ctla-4 and pd-1.

CTLA-4 and PD-1, as two well-known IC molecules, have completely different mechanisms for T cell immune regulation. Both CTLA-4 and CD28 are expressed on the surface of Tregs [ 58 ]. CD28 plays an active role in the activation of T cells by binding to B7 ligands (B7-1/2) on the surface of antigen-presenting cells (APCs) [ 59 ]. CTLA-4, as the first discovered IC molecule [ 60 ], can also bind to B-7 ligands with higher affinity. Thus, CTLA-4 can competitively inhibit the binding of the CD28 receptor to B-7 ligands and further block the vital signal transmission of T-cell activation, leading to an immunosuppressive effect [ 61 , 62 ]. However, unlike CTLA-4, PD-1 is widely expressed on the surface of activated lymphocytes (B or T cells), NK cells, and many other immune cells [ 63 ]. As a member of the CD28 superfamily [ 64 ], PD-1 directly inhibits T-cell activation by binding to its two ligands (PD-L1 and PD-L2) [ 65 , 66 ], which are widely expressed on the surface of tumor cells and many immune cells [ 67 ]. The combination between PD-1 and its ligands can induce apoptosis of lymphocytes and finally cause the immune escape of the tumor [ 64 ]. Once PD-1 binds to PD-L1, this signal will generate a positive feedback loop to inhibit T cell activation by recruiting SHP2 tyrosine phosphatase. At the same time, this pathway dephosphorylates CD28 and weakens the TCR signal [ 68 ]. Other studies have shown that even as two independent immunosuppressive molecules, PD-1 and PD-L1 can independently inhibit the activity of T lymphocytes and reduce the ability of these tumor killer cells to infiltrate the TME [ 69 , 70 ].

As a critical negative regulator, CTLA-4 limits immune responses of T cells to PDAC cells under the circumstances of pancreatic cancer, which provides a potential treatment option, that is, the CTLA-4 blockade. It is widely acknowledged that by using anti-CTLA-4 antibodies, ipilimumab for instance, the silent immune responses will be restored. The immune system comes back online, followed by tumor regression. In a similar way but with distinct mechanisms of action, by targeting PD-1 using anti-PD-1 antibodies such as nivolumab, tumor regression can also be achieved in cancer patients [ 71 ]. In addition, many studies have shown that the high expression of immune checkpoints on the surface of endothelial cells is related to the poor prognosis of PDAC patients, Cloutier et al . confirmed by immunohistochemistry that upgrade the expression of PD-L1 will lead to inferior prognosis ( P  = 0.0367), Gao Jin et al . revealed that the expression of PD-L1 was related to the T stage of PDAC. The research showed that the positive rate of PD-L1 in patients with PDAC in the T3-T4 stage was much higher than that in patients in the T1-T2 stage [ 72 , 73 , 74 ]. Therefore, targeting such IC molecules as immunotherapy strategies provides insights on the immune regulation of TME in PDAC.

LAG-3 (CD223)

MHC receptors on the surface of APCs bind to T-cell receptors (TCRs) and play an active role in the activation and proliferation of T cells. LAG-3 has a high affinity for MHC class II, which prevents the same MHC molecule from contacting TCRs, thus indirectly hindering TCR signal transduction immune response [ 75 ]. LAG-3 is expressed on CD4 + T cells, CD8 + T cells, Tregs, NK cells, and B cells. The wide expression of LAG-3 not only reduces the activity of CD4 + T cells, but also weakens the ability of cytotoxic T cells to eliminate tumor cells. Moreover, LAG-3 can promote the immunosuppressive activity of Tregs through the secretion of TGF-β, IL-10, and other immunosuppressive molecules [ 76 ].

TIM-3 is a member of the Tim gene family [ 77 ]. Pu-Ji et al . found that the expression of TIM-3 was significantly higher in pancreatic cancer than in healthy pancreas tissue, according to the result of immunohistochemical analysis of patient samples. Similar to PD-1, Tim-3 exerts its immunosuppressive effect by binding with the ligands on the effective immune cells and in a variety of solid tumors, including pancreatic cancer, TIM-3, and PD-1 co-expressed on TILs, resulting in poor clinical prognosis [ 78 ]. Its ligands include protein ligands such as galectin-9 [ 79 ], carcinoembryonic antigen cell adhesion molecule 1 [ 80 ], high mobility group box 1 [ 81 ] and non-protein ligand phosphatidylserine [ 82 ]. Interferon (IFN)-γ can promote NK cell activity and enhance antigen presentation in favor of the recognition and killing of tumor cells by lymphocytes. TIM-3 is expressed chiefly on IFN-γ-producing CD4 + T cells (T helper 1 cells) [ 83 ]. By binding with various ligands, TIM-3 can induce CD4 + T-cell depletion and reduction of IFN-γ, indirectly inhibiting the activation of immune cells [ 84 ].

T cell immunoglobulin and ITIM domain (TIGIT)

Yu et al . [ 85 ] first discovered that TIGIT could inhibit T-cell activation as an IC in 2009. TIGIT directly inhibits T-cell activation by directly combining with its ligands CD155 and CD112to transmit inhibitory signals. At the same time, TIGIT can competitively inhibit the binding of CD266 or CD96 with CD155 and CD112, reducing the active signal to T cells [ 86 , 87 , 88 ]. The binding of TIGIT with CD155, in turn, induces the phosphorylation of CD155 and release of IL-10, which prevents T cells’ activation [ 84 ].

V-domain Ig-containing suppressor of T-cell activation (VISTA)

VISTA, which is a member of the B7 family, is homologous with PD-L1 [ 89 ]. VISTA is highly expressed in PDAC and in endothelial cells and immune cells such as T cells [ 90 ]. Expression of VISTA on T cells can inhibit the proliferation and activation of T cells. In addition, Jorge et al . in 2019 found that VISTA is highly expressed in CD68 + macrophages of PDAC and plays an important role in the reduction of cytokine production by T cells in metastatic pancreatic tumors [ 91 ], and Blando et al. [ 92 ] found that VISTA was highly expressed in the pancreatic stromal area and diminishes cytokine production by T cells.

In 2001, Chapoval et al. [ 93 ] first found that B7-H3 (also called CD276) can play a positive role in promoting T-cell activation and IFN-γ secretion. However, later studies showed that B7-H3, as a member of the B7 family, acts more as a negative regulator to inhibit the immune response of T cells [ 94 , 95 ]. B7-H3 is widely expressed on the surface of a variety of activated immune cells, including T cells, NK cells, and APCs [ 88 ]. Although no receptor of B7-H3 has been found, its effect on inhibiting T cells and NK cells has been confirmed [ 96 ].

BTLA (CD272)

As a member of the CD28 superfamily [ 97 ], BTLA (CD272) is expressed on the surface of T cells, B cells, and MDSCs [ 98 ]. BTLA can compete with two TNF family members, LIGHT and lymphotoxin-α (CD160), to bind their ligand, herpesvirus entry mediator (HVEM). CD160, like BTLA, inhibits the activation of T cells after binding with HVEM, while LIGHT promotes the activation of T cells [ 88 ]. The combination of BTLA with HVEM inhibits the activation and proliferation of CD4 + /CD8 + cancer-specific T cells by promoting the phosphorylation of immunoreceptor tyrosine-based inhibition motifs (ITIMs) and Srchomology 2 (SH2) domain-containing phosphatase 1 (SHP-1)/SHP-2 association [ 99 ].

Peripheral TCR profiling correlated with responses of ICIs

The TCR is a polymorphic receptor that is essential for the development and the peripheral maturation and activation of T cells. CTLA-4, as a TCR expressed on CD4 + and CD8 + T cells, competitively inhibits the CD28 co-stimulation, thus inhibiting T cell activation, while PD-1 acts in a distinct manner by preventing CD8 +

T cells from interacting with the target cell. Inhibition of the above two pathways restores the ability of T cells, having them engage and destroy the targets. The development of ICIs to tackle the immune suppression problem improves the efficacy of cancer treatment. Additionally, it brings us a whole new angle to view and assess how the use of ICIs would lead to alterations of the peripheral TCR diversity. Advancements in TCR sequencing and the use of bioinformatic tools allow us to measure the heterogeneity of the T cells, or TCR repertoires [ 100 ]. In PDAC patients, a previous study had measured large shifts in TCR repertoire when ICIs involved, which has also been used as predictors of clinical outcome [ 101 ]. For instance, anti-CTLA-4 antibodies and anti-PD-1 antibodies both achieve an optimal therapeutic effect in PDAC patients, but each method has different effects on the peripheral TCR repertoire, more specifically, demonstrating a diversification indicated by a change in clonality [ 101 ]. Therefore, the evaluation of peripheral TCR repertoire would be a promising direction to elaborate the response of ICIs and further elucidate the rationales of other potential treatments (Fig.  2 ).

The pairing relationship between various immune checkpoints and their corresponding receptors

Research progress on treatment with ICIS

Research progress of ici monotherapy.

Ipilimumab, an inhibitor of CTLA-4, can improve the prognosis of patients with malignant melanoma and is approved by the USA and Europe for clinical use [ 6 , 102 ]. Research on ipilimumab for pancreatic cancer has also been underway for several years. In 2010, Richard et al . used ipilimumab as monotherapy for locally advanced or metastatic PDAC. However, of the 27 patients who participated in the experiment, only two with locally advanced disease showed mild efficacy, and the rest of the patients progressed rapidly and died soon thereafter [ 16 ]. In a phase II clinical trial (NCT02527434) started in 2015, another CTLA-4 inhibitor, tremelimumab, was used as monotherapy for PDAC. Unfortunately, the average OS and average progression-free survival (PFS) of 20 patients in this clinical trial were 4.14 and 1.84 mo, respectively, which was lower than the patients treated with chemotherapy (8.3 and 4.3 mo). In 2012, Julie et al . conducted a clinical trial of PD-L1 antibody monotherapy for a variety of advanced solid tumors, among which 14 patients with PDAC showed no objective responses [ 103 ]. After that, two phase I/II clinical trials using anti-PD-1 and anti-PD-L1 as monotherapy for PDAC did not achieve satisfactory results [ 104 , 105 ]. A phase II clinical trial utilizing another antibody against PD-1, nivolumab, for treatment of metastatic pancreatic cancer is still in progress. In addition to CTLA-4, PD-1, and PD-L1 antibodies, some new IC antibodies have also been used in clinical trials for the treatment of PDAC. However, recent studies have shown that monotherapy with novel ICIs does not significantly improve the condition of cancer patients either. There are two main reasons that lead to the poor efficacy of ICI monotherapy. The first one is that the immunosuppressive pancreatic cancer microenvironment and dense stromal impede the infiltration of effector T cells. Second, the Subtype as MSI-high (MSI-H) or mismatch-repair-deficient (dMMR), which has been confirmed to be effective for ICI drugs, seems quite rare in pancreatic cancer [ 106 ]. Although ICI monotherapy for solid tumors, including PDAC, is not effective, these studies have revealed the unique natural immunosuppressive TME of pancreatic cancer. Therefore, how to improve the efficacy of ICI drugs in PDAC, which is called an immune desert by scholars, has become a hot spot in recent years.

Targeting different components of the TME may enhance the efficacy of ICIs

Failure of monotherapy makes people realize that the use of a single ICI cannot change the immunosuppressive TME of PDAC [ 107 ]. Therefore, researchers expect to reverse the inhibitory TME and increase the efficacy of ICIs by targeting different components of the TME, including MDSCs, TAMs, Tregs, and PSCs. MDSCs have been widely studied in recent years as important components of the immunosuppressive TME. Inhibition of C-X-C motif chemokine receptor (CXCR)2 can directly prevent infiltration of MDSCs into the TME and mediate infiltration of T cells. Thus, the combined use of CXCR2 and PD-1 inhibitors significantly prolongs OS in a mouse model of PDAC [ 108 ]. Apolipoprotein A-I mimetic peptide L-4F can inhibit the differentiation and activation of MDSCs by downregulating the signal transducer and activator of transcription (STAT)3 signaling pathway of MDSCs. L-4F has the potential to be used as an adjunctive drug for ICI treatment [ 109 ]. In 2016, Huang et al. [ 110 ] found that ltp-1, another inhibitor of the STAT3 signaling pathway, can inhibit the growth of pancreatic cancer in vivo and in vitro. It will be interesting to establish whether ltp-1 can enhance ICI therapy. In addition to the STAT3 signaling pathway, MDSCs are also regulated by CD200 and colony stimulating factor 1 receptor (CSF1R). Antagonists of CD200 and CSF1R can reduce the proliferation and activation of MDSCs and inhibit the growth of a PDAC model in vivo by combination with ICIs [ 111 , 112 ]. Another study showed that the combination of CSF1R inhibitors and CXCR2 blockers significantly inhibited proliferation and activation of TAMs and MDSCs and enhanced the therapeutic effect of ICIs on solid tumors [ 113 ]. In addition to targeting CSF1R, disruption of the galectin-9/dectin 1 axis can also reverse the immunosuppressive TME caused by M2-TAMs. Zhou et al . found that exosomes based on bone marrow mesenchymal stem cells can significantly enhance the efficacy of targeted therapy and downregulate the number of M2-TAMs and Tregs in the TME. In the future, the combination of this new biological therapy and ICIs is worth pursuing [ 114 ]. For the other two immunosuppressive components, Tregs have a higher expression of C–C chemokine receptor (CCR)4. CCR4 antibody can induce apoptosis of Tregs. However, the combination of CCR4 antibody and ICIs durvalumab or tremelimumab did not improve the prognosis of patients with advanced solid tumors in a phase II clinical trial. The reason for this is not known and may be related to drug dose [ 115 ]. Growth of TAMs/cancer-associated fibroblasts (CAFs) can be directly inhibited by blocking the PAK1 [ 116 ] pathway or using xl888 (a heat shock protein 90 inhibitor) [ 117 ]. Thus, these two novel therapeutic methods targeting PSCs not only improve T-cell proliferation and infiltration, but also significantly improve the efficacy of ICIs as adjuvants. The small molecule glutamine analog 6-diazo-5-oxo-L-norleucine enhances infiltration of CD8 + T cells through downregulation of dense extracellular matrix, which has been proved to have a synergistic effect with PD-1 receptor blockers [ 118 ]. By targeting the small molecules secreted by these cellular components, the efficacy of ICIs can also be improved. For example, CXC chemokine ligand (CXCL)12, secreted by CAFs, can induce tumor-cell evasion of immune surveillance by inhibiting the activation of T cells. By inhibiting CXCR4, some authors have discovered that the activity of CD3 + T cells can be restored in a synergistic manner with anti-PD-1 drugs in vitro or in vivo [ 119 , 120 ]. Another study designed to inhibit galectin-1 secreted by PSCs improved the ability of CD4 + and CD8 + T cells to infiltrate the TME. The authors speculated that the infiltration of functional T cells into the TME is the key factor in ensuring the efficacy of immunotherapy [ 121 ]. By reconstituting the TME of PDAC, ICIs can more easily eliminate immunosuppression, which also provides a new possibility for combination therapy with ICIs in the future.

Combination of ICIs and traditional chemoradiotherapy

Chemotherapy with FOLFIRINOX (fluorouracil, leucovorin, irinotecan, and oxaliplatin) and gemcitabine combined with nanoparticle albumin-bound paclitaxel (nab-paclitaxel) remained the first-line treatment for PDAC in 2020 [ 122 ]. These drugs exert antitumor effects mainly by affecting the process of tumor cell replication and proliferation. In addition to the traditional cytotoxic effects, they can enhance the therapeutic effect of ICIs on PDAC by enhancing the antigenicity of tumor cells and targeting some inhibitory components in the TME [ 123 ]. For example, gemcitabine can downregulate the proportion of MDSCs, Tregs, and TGF-β in the TME of PDAC and increase the number of effector T cells infiltrating the TME [ 124 ]. 5-Fluorouracil can reduce the number of activated MDSCs, improve the ability of effector T cells to produce IFN-γ, and promote the efficacy of immunotherapy [ 125 ].

The efficacy of the combination of chemotherapy and ICIs has also been confirmed in vivo and in vitro. In a preclinical model of PDAC, the combination of gemcitabine and anti-PD-L1 induced a complete response [ 126 ]. Moreover, this combination therapy has been proved to enhance the immune response by increasing the proportion of M1 macrophages and effector T cells in a murine model of liver metastasis [ 127 ]. Besides in vitro experiments, some clinical trials have also confirmed that the combination of chemotherapy and ICIs can improve the prognosis of patients with PDAC. In a recent study, Ma et al. [ 128 ] found that patients treated with chemotherapy and ICIs had higher OS and PFS than those treated with chemotherapy alone. A phase Ib clinical trial (NCT01473940) also proved that the combination of ipilimumab and gemcitabine could achieve a better prognosis in PDAC patients [ 129 ]. Albumin paclitaxel can further improve the prognosis of PDAC patients. In a phase Ib/II clinical trial (NCT02331251), 17 patients who received gemcitabine, albumin paclitaxel, and PD-1 receptor blocker pembrolizumab had an average PFS of 9.1 mo and an average OS of 15 mo [ 130 ].

Similar to chemotherapy, PDAC cells are also resistant to radiotherapy owing to the barrier formed by dense matrix. However, the combination of radiotherapy and ICIs can still improve the prognosis of patients, which may be due to the following reasons. First, the tumor antigens on the cells can be exposed by the radiation-mediated tumor cell killing, which is presented by MHC class I and recognized and eliminated by cytotoxic T cells [ 131 ]. Second, Valkenburg et al . recently found that radiotherapy can reconstruct the matrix stromal components in the TME. Therefore, the immunosuppressive TME of PDAC is changed, which is more conducive to the efficacy of ICIs [ 132 ]. This theory is also be supported by the results of some clinical trials. Azad et al . used (12, 5 × 3,20 Gy) high-dose radiotherapy combined with ICIs to treat PDAC. Radiotherapy increased the number of activated T cells and upregulated the ratio of CD8:Tregs [ 133 ]. The combination of radiotherapy and IDO inhibitors can reduce T-cell depletion, which has a synergistic role with ICI treatment [ 134 ]. Many other preclinical and clinical trials have also proved that through combination with radiotherapy, ICIs are more likely to exert their effect of contact immunosuppression and lead to inhibition of tumor growth in vitro and in vivo [ 135 , 136 , 137 , 138 ]. Finally, another ongoing phase 2 clinical trial (NCT04361162), which combined nivolumab, ipilimumab, and radiotherapy, was conducted in 30 patients with metastatic, microsatellite stable pancreatic cancer. This study started in March 2020 and is currently in progress.

Combination therapy of two or three antibodies

The combination of two or three immunosuppressants has been shown to improve the prognosis of patients with pancreatic cancer. Rafeal et al . confirmed that PD-1 blocker as a supplement to CTLA-4 blocker might alleviate the immune resistance effect of monotherapy and improve the OS of PC patients [ 139 ]. The combination of PD-1 and PD-L1 blockers also has a better curative effect by inducing more effector T cells into the TME and generating memory T cells with the function of preventing tumor recurrence [ 140 , 141 ]. In four patients who received durvalumab combined with tremelimumab, the average OS was increased to 7.18 mo, significantly higher than the mean OS with monotherapy or existing first-line treatment (NCT02527434). A phase 2 clinical trial from 2019 yielded similar results: the objective response rate was 3.1% for patients receiving combination therapy of durvalumab and tremelimumab, compared with no response for patients receiving monotherapy [ 142 ]. Another ongoing phase 2 clinical trial, which combines nivolumab, ipilimumab, and radiotherapy, in 30 patients with metastatic, microsatellite stable pancreatic cancer (NCT04361162), started in March 2020. The combination of novel ICIs with antibodies against PD-1, PD-L1, and CTLA-4 has also achieved some success. As a functional monoclonal antibody with LAG-3, TSR-033 can improve the efficacy of PD-1 monotherapy in patients with pancreatic cancer. The combination of LAG-3 and PD-1 receptor antagonists can also enhance the proliferation and infiltration of effector T cells, reversing the immune resistance of the tumor [ 143 , 144 ]. Similar to LAG-3, the anti-Tim-3 monoclonal antibody (clone m6903) can block the binding of Tim-3 with its three inhibitory ligands. However, Tim-3 receptor blocker monotherapy had no effect on a mouse model of melanoma. Survival can be improved by administering anti-Tim-3 monoclonal antibody and PD-1 receptor antagonist simultaneously [ 145 , 146 ]. Chauvin et al. [ 147 ] have shown that the combination of TIGIT and PD-1 receptor blockers can increase the antitumor activity of CD8 + T cells in patients with advanced melanoma and improve prognosis. Ongoing clinical trials include a phase 1/2clinical study (NCT01928394) of combined nivolumab and ipilimumab in solid tumors, including PDAC. In general, the combination of two or three kinds of ICIs may improve the prognosis of PC patients. Additional table files show more information on clinical trials in detail. (Additional file 3 supplements and summarizes the efficacy and tolerability of ICI treatment in clinical studies with existing results in Additional file 2 .)

The “Achilles’ heel” of ICI drugs

Although in some animal experiments and clinical trials, immune checkpoint inhibitors have shown some ability to reverse PDAC immune resistance. But overall, the OS and PFS of most PDAC patients did not achieve significant improvement from this treatment. In addition to the specific immune resistance of PDAC that has been mentioned above, as a new anti-tumor treatment method in recent years, ICIs drug itself has many limitations. First, its immune-related adverse effects (irAEs) as delayed toxicity towards some specific organs of the human body, and this specifical adverse effect seems irrelevant to dose, which means that a lower dose cannot effectively reduce its adverse effects [ 148 ]. Second, in addition to the conventional adverse effects of drugs, hyperprogression, which is described by Lancet magazine as the “Achilles’ heel” of immune checkpoint inhibitor treatment, not only increases the mortality of patients in the early stage of immunotherapy but also becomes an uncertain factor on the road to pursue “precision immunotherapy”. Hyperprogression refers to the phenomenon that the degree of disease progression at a rate that far more exceeds than the normal course of this disease in the early stage of treatment. During this period, the degree of tumor progression (volume, speed) and mortality of patients are greatly improved. It is worth mentioning that the super progress phenomenon is not the “patent” of immune checkpoint inhibitor treatment, but its incidence has been greatly improved compared with other treatment methods such as chemotherapy [ 149 , 150 ]. Although there are few reports on the hyperprogression of immunotherapy for PDAC, we should pay enough attention to it in future research.

Immunotherapy is changing our traditional concept of cancer treatment, and even has become a first-line therapeutic drug in some solid tumors such as non-small cell lung cancer. By relieving the inhibitory effect on T cells, ICIs drugs are expected to tackle problems that cannot be solved by conventional therapy. However, ICIS monotherapy did not effectively improve the prognosis of PDAC patients, which is thought to be related to the suppressive TME and dense extracellular matrix of pancreatic cancer. In addition, we lack effective biomarkers to monitor drug efficacy and guide our selection of drugs. With the progress of research, multi-drug combination therapy seems to bring a glimmer of dawn for PDAC patients. The combined use of chemotherapy, radiotherapy, other immunotherapies including CAR-T and tumor vaccine with ICIs drugs, has improved the therapeutic efficacy of PDAC. However, as a non-immunogenic tumor, the efficacy of ICIs is still limited by the fact that T cells cannot be effectively activated in the TME of pancreatic cancer. For different clinical patients, their tumor antigenicity may have individual differences. The reasonable classification of this population may help us to find the best combination partner of ICI drugs. Some studies have proposed the concept of “immune score” to evaluate the effectiveness of immunotherapy by combining clinicopathological basis with gene sequencing. Peripheral blood TCR profiling also provides a new possibility for early efficacy prediction of ICI. Considering that the immune response is dynamic and changes over time, we need to establish more effective predictors of ICIS regimen treatment response, which may include TIL, IC molecular expression, and many other emerging biomarkers, so as to more effectively and confidently apply ICI drugs to the clinical treatment of PDAC patients. To sum up, we put forward some future directions for improving the efficacy of ICI drugs on PDAC: (1) Increase the initial activation number of T cells, increase the number of tumor-infiltrating T cells and reduce the depletion of T cells; (2) Find more effective biomarkers that can predict the efficacy in a more precise way; (3) Individualized treatment of PDAC patients and monitoring the efficacy in order to find the best combination of ICI drugs.

Availability of data and materials

We declared that materials described in the manuscript, including all relevant raw data, will be freely available to any scientist wishing to use them for non-commercial purposes without breaching participant confidentiality.

Abbreviations

• Pancreatic ductal adenocarcinoma
• Immune checkpoint inhibitors
• Tumor microenvironment

Myeloid-derived suppressor cells

Tumor-associated macrophages

Pancreatic satellite cells

Overall survival

Progression-free survival

Programmed cell death protein 1

Cytotoxic T-lymphocyte-associated protein 4

Rahib L, et al. Projecting cancer incidence and deaths to 2030: the unexpected burden of thyroid, liver, and pancreas cancers in the United States. Cancer Res. 2014;74(11):2913–21.

Article   CAS   PubMed   Google Scholar  

Henriksen A, et al. Checkpoint inhibitors in pancreatic cancer. Cancer Treat Rev. 2019;78:17–30.

Wolfgang CL, et al. Recent progress in pancreatic cancer. CA Cancer J Clin. 2013;63(5):318–48.

Article   PubMed   PubMed Central   Google Scholar  

Bliss LA, et al. Outcomes in operative management of pancreatic cancer. J Surg Oncol. 2014;110(5):592–8.

Article   PubMed   Google Scholar  

Darvin P, et al. Immune checkpoint inhibitors: recent progress and potential biomarkers. Exp Mol Med. 2018;50(12):1–11.

Hodi FS, et al. Improved survival with ipilimumab in patients with metastatic melanoma. N Engl J Med. 2010;363(8):711–23.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ribas A, et al. Pembrolizumab versus investigator-choice chemotherapy for ipilimumab-refractory melanoma (KEYNOTE-002): a randomised, controlled, phase 2 trial. Lancet Oncol. 2015;16(8):908–18.

Robert C, et al. Pembrolizumab versus ipilimumab in advanced melanoma. N Engl J Med. 2015;372(26):2521–32.

Rizvi NA, et al. Activity and safety of nivolumab, an anti-PD-1 immune checkpoint inhibitor, for patients with advanced, refractory squamous non-small-cell lung cancer (CheckMate 063): a phase 2, single-arm trial. Lancet Oncol. 2015;16(3):257–65.

Garon EB, et al. Pembrolizumab for the treatment of non-small-cell lung cancer. N Engl J Med. 2015;372(21):2018–28.

Reck M, et al. Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancer. N Engl J Med. 2016;375(19):1823–33.

Motzer RJ, et al. Nivolumab for metastatic renal cell carcinoma: results of a randomized phase II trial. J Clin Oncol. 2015;33(13):1430–7.

Weinstock M, McDermott D. Targeting PD-1/PD-L1 in the treatment of metastatic renal cell carcinoma. Ther Adv Urol. 2015;7(6):365–77.

Powles T, et al. MPDL3280A (anti-PD-L1) treatment leads to clinical activity in metastatic bladder cancer. Nature. 2014;515(7528):558–62.

Powles T, et al. Efficacy and safety of durvalumab in locally advanced or metastatic urothelial carcinoma: updated results from a phase 1/2 open-label study. JAMA Oncol. 2017;3(9):e172411.

Royal RE, et al. Phase 2 trial of single agent Ipilimumab (anti-CTLA-4) for locally advanced or metastatic pancreatic adenocarcinoma. J Immunother. 2010;33(8):828–33.

Sakaguchi S, et al. Immunologic self-tolerance maintained by activated T cells expressing IL-2 receptor alpha-chains (CD25). Breakdown of a single mechanism of self-tolerance causes various autoimmune diseases. J Immunol (1950). 1995;155(3):1151–64.

Article   CAS   Google Scholar  

Zou W. Regulatory T cells, tumour immunity and immunotherapy. Nat Rev Immunol. 2006;6(4):295–307.

Oleinika K, et al. Suppression, subversion and escape: the role of regulatory T cells in cancer progression. Clin Exp Immunol. 2013;171(1):36–45.

Taylor A, et al. Mechanisms of immune suppression by interleukin-10 and transforming growth factor-beta: the role of T regulatory cells. Immunology. 2006;117(4):433–42.

Sakaguchi S, et al. Regulatory T cells and immune tolerance. Cell. 2008;133(5):775–87.

Chinen T, et al. An essential role for the IL-2 receptor in Treg cell function. Nat Immunol. 2016;17(11):1322–33.

Shimizu J, Yamazaki S, Sakaguchi S. Induction of tumor immunity by removing CD 25+ CD 4+ T cells: a common basis between tumor immunity and autoimmunity. J Immunol (1950). 1999;163(10):5211–8.

Shitara K, Nishikawa H. Regulatory T cells: a potential target in cancer immunotherapy: regulatory T cells in cancer immunity. Ann N Y Acad Sci. 2018;1417(1):104–15.

Takeuchi Y, Nishikawa H. Roles of regulatory T cells in cancer immunity. Int Immunol. 2016;28(8):401–9.

Bauer CA, et al. Dynamic Treg interactions with intratumoral APCs promote local CTL dysfunction. J Clin Investig. 2014;124(6):2425–40.

Walunas TL, et al. CTLA-4 can function as a negative regulator of T cell activation. Immunity (Cambridge, Mass). 1994;1(5):405.

CAS   Google Scholar  

Alegre M-L, et al. Modulation of tryptophan catabolism by regulatory T cells. Nat Immunol. 2003;4(12):1206–12.

Budhu S, et al. Blockade of surface-bound TGF-beta on regulatory T cells abrogates suppression of effector T cell function in the tumor microenvironment. Sci Signal. 2017;10(494):eaak9702.

Jang J-E, et al. Crosstalk between regulatory T cells and tumor-associated dendritic cells negates anti-tumor immunity in pancreatic cancer. Cell Rep (Cambridge). 2017;20(3):558–71.

Kim J-H, Kim BS, Lee S-K. Regulatory T cells in tumor microenvironment and approach for anticancer immunotherapy. Immune Netw. 2020;20(1):e4–e4.

De Simone M, et al. Transcriptional landscape of human tissue lymphocytes unveils uniqueness of tumor-infiltrating T regulatory cells. Immunity (Cambridge, Mass). 2016;45(5):1135–47.

Google Scholar  

Sasidharan Nair V, Elkord E. Immune checkpoint inhibitors in cancer therapy: a focus on T-regulatory cells. Immunol Cell Biol. 2018;96(1):21–33.

Gabrilovich DI, Ostrand-Rosenberg S, Bronte V. Coordinated regulation of myeloid cells by tumours. Nat Rev Immunol. 2012;12(4):253–68.

Pergamo M, Miller G. Myeloid-derived suppressor cells and their role in pancreatic cancer. Cancer Gene Ther. 2017;24(3):100–5.

Kusmartsev S, et al. Oxidative stress regulates expression of VEGFR1 in myeloid cells: link to tumor-induced immune suppression in renal cell carcinoma. J Immunol. 2008;181(1):346–53.

Fulton SA, et al. Inhibition of major histocompatibility complex II expression and antigen processing in murine alveolar macrophages by mycobacterium bovis BCG and the 19-kilodalton mycobacterial lipoprotein. Infect Immun. 2004;72(4):2101–10.

Nagaraj S, Gabrilovich DI. Myeloid-derived suppressor cells as regulators of the immune system. Nat Rev Immunol. 2009;9(3):162–74.

Stromnes IM, et al. Targeted depletion of an MDSC subset unmasks pancreatic ductal adenocarcinoma to adaptive immunity. Gut. 2014;63(11):1769–81.

Huang BO, et al. Gr-1+CD115+ immature myeloid suppressor cells mediate the development of tumor-induced t regulatory cells and T-cell anergy in tumor-bearing host. Cancer Res (Chicago, Ill). 2006;66(2):1123–31.

Pinton L, et al. Activated T cells sustain myeloid-derived suppressor cell-mediated immune suppression. Oncotarget. 2016;7(2):1168–84.

Yang S, Liu Q, Liao Q. Tumor-associated macrophages in pancreatic ductal adenocarcinoma: origin, polarization, function, and reprogramming. Front Cell Dev Biol. 2020;8:607209.

Lankadasari MB, et al. TAMing pancreatic cancer: combat with a double edged sword. Mol Cancer. 2019;18(1):48.

Grimshaw MJ, Balkwill FR. Inhibition of monocyte and macrophage chemotaxis by hypoxia and inflammation–a potential mechanism. Eur J Immunol. 2001;31(2):480–9.

Biswas SK, Allavena P, Mantovani A. Tumor-associated macrophages: functional diversity, clinical significance, and open questions. Semin Immunopathol. 2013;35(5):585–600.

Caux C, et al. A milestone review on how macrophages affect tumor growth. Cancer Res (Chicago, Ill). 2016;76(22):6439–42.

Ruffell B, Coussens LM. Macrophages and therapeutic resistance in cancer. Cancer Cell. 2015;27(4):462–72.

Beavis PA, et al. Adenosine receptor 2A blockade increases the efficacy of anti-PD-1 through enhanced antitumor T-cell responses. Cancer Immunol Res. 2015;3(5):506–17.

Apte MV, Pirola RC, Wilson JS. Pancreatic stellate cells: a starring role in normal and diseased pancreas. Front Physiol. 2012;3:344.

Hamada S, et al. IL-6/STAT3 plays a regulatory role in the interaction between pancreatic stellate cells and cancer cells. Dig Dis Sci. 2016;61(6):1561–71.

Tang D, et al. Apoptosis and anergy of T cell induced by pancreatic stellate cells-derived galectin-1 in pancreatic cancer. Tumor Biol. 2015;36(7):5617–26.

Tang D, et al. High expression of Galectin-1 in pancreatic stellate cells plays a role in the development and maintenance of an immunosuppressive microenvironment in pancreatic cancer. Int J Cancer. 2012;130(10):2337–48.

Ene-Obong A, et al. Activated pancreatic stellate cells sequester CD 8+ T cells to reduce their infiltration of the juxtatumoral compartment of pancreatic ductal adenocarcinoma. Gastroenterology. 2013;145(5):1121–32.

Li C, et al. Pancreatic stellate cells promote tumor progression by promoting an immunosuppressive microenvironment in murine models of pancreatic cancer. Pancreas. 2020;49(1):120–7.

Ozdemir BC, et al. Depletion of carcinoma-associated fibroblasts and fibrosis induces immunosuppression and accelerates pancreas cancer with reduced survival. Cancer Cell. 2015;28(6):831–3.

Jonescheit H, et al. Influence of indoleamine-2,3-dioxygenase and its metabolite kynurenine on γδ T cell cytotoxicity against ductal pancreatic adenocarcinoma cells. Cells (Basel, Switzerland). 2020;9(5):1140.

Fan JQ, et al. Current advances and outlooks in immunotherapy for pancreatic ductal adenocarcinoma. Mol Cancer. 2020;19(1):32.

Wang CJ, et al. Cutting edge: cell-extrinsic immune regulation by CTLA-4 expressed on conventional T cells. J Immunol. 2012;189(3):1118–22.

Camacho LH. CTLA-4 blockade with ipilimumab: biology, safety, efficacy, and future considerations. Cancer Med. 2015;4(5):661–72.

Leach DR, Krummel MF, Allison JP. Enhancement of antitumor immunity by CTLA-4 blockade. Science. 1996;271(5256):1734–6.

Skelton RA, et al. Overcoming the resistance of pancreatic cancer to immune checkpoint inhibitors. J Surg Oncol. 2017;116(1):55–62.

Chen L. Co-inhibitory molecules of the B7-CD28 family in the control of T-cell immunity. Nat Rev Immunol. 2004;4(5):336–47.

Keir ME, et al. PD-1 and its ligands in tolerance and immunity. Annu Rev Immunol. 2008;26:677–704.

Okazaki T, Honjo T. PD-1 and PD-1 ligands: from discovery to clinical application. Int Immunol. 2007;19(7):813–24.

Zhu B, et al. Targeting the upstream transcriptional activator of PD-L1 as an alternative strategy in melanoma therapy. Oncogene. 2018;37(36):4941–54.

Flies DB, Chen L. The new B7s: playing a pivotal role in tumor immunity. J Immunother (1997). 2007;30(3):251–60.

Latchman YE, et al. PD-L1-deficient mice show that PD-L1 on T cells, antigen-presenting cells, and host tissues negatively regulates T cells. Proc Natl Acad Sci PNAS. 2004;101(29):10691–6.

Yokosuka T, et al. Programmed cell death 1 forms negative costimulatory microclusters that directly inhibit T cell receptor signaling by recruiting phosphatase SHP2. J Exp Med. 2012;209(6):1201–17.

Davoodzadeh Gholami M, et al. Exhaustion of T lymphocytes in the tumor microenvironment: significance and effective mechanisms. Cell Immunol. 2017;322:1–14.

Feng M, et al. PD-1/PD-L1 and immunotherapy for pancreatic cancer. Cancer Lett. 2017;407:57–65.

Sharma P, Allison JP. The future of immune checkpoint therapy. Science. 2015;348(6230):56–61.

Lu SW, et al. IL-20 antagonist suppresses PD-L1 expression and prolongs survival in pancreatic cancer models. Nat Commun. 2020;11(1):4611.

Tessier-Cloutier B, et al. Programmed cell death ligand 1 cut-point is associated with reduced disease specific survival in resected pancreatic ductal adenocarcinoma. BMC Cancer. 2017;17(1):618.

Gao HL, et al. The clinicopathological and prognostic significance of PD-L1 expression in pancreatic cancer: a meta-analysis. Hepatobiliary Pancreat Dis Int. 2018;17(2):95–100.

Goldberg MV, Drake CG. LAG-3 in cancer immunotherapy. Curr Top Microbiol Immunol. 2011;344:269–78.

CAS   PubMed   PubMed Central   Google Scholar  

Long L, et al. The promising immune checkpoint LAG-3: from tumor microenvironment to cancer immunotherapy. Genes Cancer. 2018;9(5–6):176–89.

Meyers JH, et al. The TIM gene family regulates autoimmune and allergic diseases. Trends Mol Med. 2005;11(8):362–9.

Jones RB, et al. Tim-3 expression defines a novel population of dysfunctional T cells with highly elevated frequencies in progressive HIV-1 infection. J Exp Med. 2008;205(12):2763–79.

Zhu C, et al. The Tim-3 ligand galectin-9 negatively regulates T helper type 1 immunity. Nat Immunol. 2005;6(12):1245–52.

Huang YH, et al. CEACAM1 regulates TIM-3-mediated tolerance and exhaustion. Nature. 2015;517(7534):386–90.

Li G, Liang X, Lotze MT. HMGB1: the central cytokine for all lymphoid cells. Front Immunol. 2013;4:68–68.

Nakayama M, et al. Tim-3 mediates phagocytosis of apoptotic cells and cross-presentation. Blood. 2009;113(16):3821–30.

Monney L, et al. Th1-specific cell surface protein Tim-3 regulates macrophage activation and severity of an autoimmune disease. Nature. 2002;415(6871):536–41.

Saka D, et al. Mechanisms of T-cell exhaustion in pancreatic cancer. Cancers (Basel). 2020;12(8):2274.

Yu X, et al. The surface protein TIGIT suppresses T cell activation by promoting the generation of mature immunoregulatory dendritic cells. Nat Immunol. 2009;10(1):48–57.

Stanietsky N, et al. The interaction of TIGIT with PVR and PVRL2 inhibits human NK cell cytotoxicity. Proc Natl Acad Sci U S A. 2009;106(42):17858–63.

Dougall WC, et al. TIGIT and CD96: new checkpoint receptor targets for cancer immunotherapy. Immunol Rev. 2017;276(1):112–20.

Qin S, et al. Novel immune checkpoint targets: moving beyond PD-1 and CTLA-4. Mol Cancer. 2019;18(1):155.

Ni L, Dong C. New checkpoints in cancer immunotherapy. Immunol Rev. 2017;276(1):52–65.

Wang L, et al. VISTA, a novel mouse Ig superfamily ligand that negatively regulates T cell responses. J Exp Med. 2011;208(3):577–92.

Blando J, et al. Comparison of immune infiltrates in melanoma and pancreatic cancer highlights VISTA as a potential target in pancreatic cancer. Proc Natl Acad Sci PNAS. 2019;116(5):1692–7.

Blando J, et al. Comparison of immune infiltrates in melanoma and pancreatic cancer highlights VISTA as a potential target in pancreatic cancer. Proc Natl Acad Sci U S A. 2019;116(5):1692–7.

Chapoval AI, et al. B7–H3: a costimulatory molecule for T cell activation and IFN-gamma production. Nat Immunol. 2001;2(3):269–74.

Prasad DVR, et al. Murine B7–H3 is a negative regulator of T cells. J Immunol (1950). 2004;173(4):2500–6.

Suh WK, et al. The B7 family member B7–H3 preferentially down-regulates T helper type 1-mediated immune responses. Nat Immunol. 2003;4(9):899–906.

Yang S, Wei W, Zhao Q. B7–H3, a checkpoint molecule, as a target for cancer immunotherapy. Int J Biol Sci. 2020;16(11):1767–73.

Ceeraz S, Nowak EC, Noelle RJ. B7 family checkpoint regulators in immune regulation and disease. Trends Immunol. 2013;34(11):556–63.

Han P, et al. An inhibitory Ig superfamily protein expressed by lymphocytes and APCs is also an early marker of thymocyte positive selection. J Immunol (1950). 2004;172(10):5931–9.

Owada T, et al. Activation-induced accumulation of B and T lymphocyte attenuator at the immunological synapse in CD4+ T cells. J Leukoc Biol. 2010;87(3):425–32.

Russano M, et al. Liquid biopsy and tumor heterogeneity in metastatic solid tumors: the potentiality of blood samples. J Exp Clin Cancer Res. 2020;39(1):95.

Hopkins AC, et al. T cell receptor repertoire features associated with survival in immunotherapy-treated pancreatic ductal adenocarcinoma. JCI Insight. 2018;3(13):e122092.

Article   PubMed Central   Google Scholar  

Torphy RJ, Zhu Y, Schulick RD. Immunotherapy for pancreatic cancer: barriers and breakthroughs. Ann Gastroenterol Surg. 2018;2(4):274–81.

Brahmer JR, et al. Safety and activity of anti-PD-L1 antibody in patients with advanced cancer. N Engl J Med. 2012;366(26):2455–65.

Herbst RS, et al. Predictive correlates of response to the anti-PD-L1 antibody MPDL3280A in cancer patients. Nature. 2014;515(7528):563–7.

Patnaik A, et al. Phase I study of pembrolizumab (MK-3475; anti-PD-1 monoclonal antibody) in patients with advanced solid tumors. Clin Cancer Res. 2015;21(19):4286–93.

Eso Y, Seno H. Current status of treatment with immune checkpoint inhibitors for gastrointestinal, hepatobiliary, and pancreatic cancers. Ther Adv Gastroenterol. 2020;13:1756284820948773.

Kraman M, et al. Suppression of antitumor immunity by stromal cells expressing fibroblast activation protein-alpha. Science. 2010;330(6005):827–30.

Steele CW, et al. CXCR2 inhibition profoundly suppresses metastases and augments immunotherapy in pancreatic ductal adenocarcinoma. Cancer Cell. 2016;29(6):832–45.

Peng M, et al. Apolipoprotein A-I mimetic peptide L-4F suppresses granulocytic-myeloid-derived suppressor cells in mouse pancreatic cancer. Front Pharmacol. 2020;11:576.

Huang HL, et al. LTP-1, a novel antimitotic agent and Stat3 inhibitor, inhibits human pancreatic carcinomas in vitro and in vivo. Sci Rep. 2016;6:27794.

Zhu Y, et al. CSF1/CSF1R blockade reprograms tumor-infiltrating macrophages and improves response to T-cell checkpoint immunotherapy in pancreatic cancer models. Cancer Res. 2014;74(18):5057–69.

Zhu Y, et al. CSF1/CSF1R blockade reprograms tumor-infiltrating macrophages and improves response to T-cell checkpoint immunotherapy in pancreatic cancer models. Cancer Res (Chicago, Ill). 2014;74(18):5057–69.

Kumar V, et al. Cancer-associated fibroblasts neutralize the anti-tumor effect of CSF1 receptor blockade by inducing PMN-MDSC infiltration of tumors. Cancer Cell. 2017;32(5):654-668.e5.

Zhou W, et al. Pancreatic cancer-targeting exosomes for enhancing immunotherapy and reprogramming tumor microenvironment. Biomaterials. 2021;268:120546.

Zamarin D, et al. Mogamulizumab in combination with durvalumab or tremelimumab in patients with advanced solid tumors: a phase I study. Clin Cancer Res. 2020;26(17):4531–41.

Wang K, et al. Inhibition of PAK1 suppresses pancreatic cancer by stimulation of anti-tumour immunity through down-regulation of PD-L1. Cancer Lett. 2020;472:8–18.

Zhang Y, et al. Heat shock protein-90 inhibition alters activation of pancreatic stellate cells and enhances the efficacy of PD-1 blockade in pancreatic cancer. Mol Cancer Ther. 2021;20(1):150–60.

Sharma NS, et al. Targeting tumor-intrinsic hexosamine biosynthesis sensitizes pancreatic cancer to anti-PD1 therapy. J Clin Investig. 2020;130(1):451–65.

Feig C, et al. Targeting CXCL12 from FAP-expressing carcinoma-associated fibroblasts synergizes with anti–PD-L1 immunotherapy in pancreatic cancer. Proc Natl Acad Sci PNAS. 2013;110(50):20212–7.

Bockorny B, et al. BL-8040, a CXCR4 antagonist, in combination with pembrolizumab and chemotherapy for pancreatic cancer: the COMBAT trial. Nat Med. 2020;26(6):878–85.

Orozco CA, et al. Targeting galectin-1 inhibits pancreatic cancer progression by modulating tumor-stroma crosstalk. Proc Natl Acad Sci U S A. 2018;115(16):E3769–78.

Christenson ES, Jaffee E, Azad NS. Current and emerging therapies for patients with advanced pancreatic ductal adenocarcinoma: a bright future. Lancet Oncol. 2020;21(3):e135–45.

Zheng Y, et al. Using chemo-drugs or irradiation to break immune tolerance and facilitate immunotherapy in solid cancer. Cell Immunol. 2015;294(1):54–9.

Eriksson E, et al. Gemcitabine reduces MDSCs, tregs and TGFβ-1 while restoring the teff/treg ratio in patients with pancreatic cancer. J Transl Med. 2016;14(1):282–282.

Vincent J, et al. 5-Fluorouracil selectively kills tumor-associated myeloid-derived suppressor cells resulting in enhanced T cell-dependent antitumor immunity. Cancer Res. 2010;70(8):3052–61.

Nomi T, et al. Clinical significance and therapeutic potential of the programmed death-1 ligand/programmed death-1 pathway in human pancreatic cancer. Clin Cancer Res. 2007;13(7):2151–7.

Ho TTB, et al. Combination of gemcitabine and anti-PD-1 antibody enhances the anticancer effect of M1 macrophages and the Th1 response in a murine model of pancreatic cancer liver metastasis. J Immunother Cancer. 2020;8(2):e001367.

Ma J, et al. Immune checkpoint inhibitors combined with chemotherapy for the treatment of advanced pancreatic cancer patients. Cancer Immunol Immunother. 2020;69(3):365–72.

Kalyan A, et al. Ipilimumab and gemcitabine for advanced pancreas cancer: a phase Ib study. J Clin Oncol. 2016;34(15_suppl):e15747–e15747.

Article   Google Scholar  

Weiss GJ, et al. Phase Ib/II study of gemcitabine, nab-paclitaxel, and pembrolizumab in metastatic pancreatic adenocarcinoma. Investig New Drugs. 2018;36(1):96–102.

Reits EA, et al. Radiation modulates the peptide repertoire, enhances MHC class I expression, and induces successful antitumor immunotherapy. J Exp Med. 2006;203(5):1259–71.

Valkenburg KC, de Groot AE, Pienta KJ. Targeting the tumour stroma to improve cancer therapy. Nat Rev Clin Oncol. 2018;15(6):366–81.

Azad A, et al. PD-L1 blockade enhances response of pancreatic ductal adenocarcinoma to radiotherapy. EMBO Mol Med. 2017;9(2):167–80.

Liu M, et al. IDO inhibitor synergized with radiotherapy to delay tumor growth by reversing T cell exhaustion. Mol Med Rep. 2020;21(1):445–53.

CAS   PubMed   Google Scholar  

Agrawal V, Benjamin KT, Ko EC. Radiotherapy and immunotherapy combinations for lung cancer. Curr Oncol Rep. 2020;23(1):4.

Vrankar M, Kern I, Stanic K. Prognostic value of PD-L1 expression in patients with unresectable stage III non-small cell lung cancer treated with chemoradiotherapy. Radiat Oncol. 2020;15(1):247.

Deng L, et al. Irradiation and anti-PD-L1 treatment synergistically promote antitumor immunity in mice. J Clin Investig. 2014;124(2):687–95.

Twyman-Saint Victor C, et al. Radiation and dual checkpoint blockade activate non-redundant immune mechanisms in cancer. Nature. 2015;520(7547):373–7.

Winograd R, et al. Induction of T-cell immunity overcomes complete resistance to PD-1 and CTLA-4 blockade and improves survival in pancreatic carcinoma. Cancer Immunol Res. 2015;3(4):399–411.

Burrack AL, et al. Combination PD-1 and PD-L1 blockade promotes durable neoantigen-specific T cell-mediated immunity in pancreatic ductal adenocarcinoma. Cell Rep. 2019;28(8):2140-2155.e6.

Zheng NN, et al. Combining protein arginine methyltransferase inhibitor and anti-programmed death-ligand-1 inhibits pancreatic cancer progression. World J Gastroenterol. 2020;26(26):3737–49.

O’Reilly EM, et al. Durvalumab with or without tremelimumab for patients with metastatic pancreatic ductal adenocarcinoma: a phase 2 randomized clinical trial. JAMA Oncol. 2019;5(10):1431–8.

Ghosh S, et al. TSR-033, a novel therapeutic antibody targeting LAG-3, enhances T-cell function and the activity of PD-1 blockade in vitro and in vivo. Mol Cancer Ther. 2019;18(3):632–41.

Burova E, et al. Preclinical development of the anti-LAG-3 antibody REGN3767: characterization and activity in combination with the anti-PD-1 antibody cemiplimab in human PD-1xLAG-3 -Knockin mice. Mol Cancer Ther. 2019;18(11):2051–62.

Zhang D, et al. Identification and characterization of M6903, an antagonistic anti-TIM-3 monoclonal antibody. Oncoimmunology. 2020;9(1):1744921.

Saleh R, Toor SM, Elkord E. Targeting TIM-3 in solid tumors: innovations in the preclinical and translational realm and therapeutic potential. Expert Opin Ther Targets. 2020;24(12):1251–62.

Chauvin JM, et al. TIGIT and PD-1 impair tumor antigen-specific CD8(+) T cells in melanoma patients. J Clin Investig. 2015;125(5):2046–58.

de Miguel M, Calvo E. Clinical challenges of immune checkpoint inhibitors. Cancer Cell. 2020;38(3):326–33.

Brower V. Hyperprogressive disease with anti-PD-1 and anti-PD-L1. Lancet Oncol. 2016;17(12):e527.

Kanjanapan Y, et al. Hyperprogressive disease in early-phase immunotherapy trials: clinical predictors and association with immune-related toxicities. Cancer. 2019;125(8):1341–9.

Download references

Acknowledgements

This work was supported by grants from Science and Technology Department of Zhejiang Province, China (Grant No. LY17H160012).

Author information

Authors and affiliations.

Department of Gastrointestinal Surgery, The Second Affiliated Hospital of Zhejiang University School of Medicine, 88 Jiefang Road, Hangzhou, 310009, Zhejiang Province, China

Hong-Bo Li, Zi-Han Yang & Qing-Qu Guo

You can also search for this author in PubMed   Google Scholar

Contributions

HBL and ZHY wrote the initial draft. QQG reviewed the literature and revised the paper. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Qing-Qu Guo .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the Medical Ethics Committee of the Second Affiliated Hospital, College of Medicine, Zhejiang University.

Consent for publication

Informed written consent was obtained from the patient for publication of this case, including all individual details and accompanying images. The written consent form is available for review upon request.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

. The mechanism of endothelial cell entry into the pancreatic cancer microenvironment and its immunosuppressive effect.

Additional file 2

. Detailed information of the ICI treatment in the existing clinical studies.

Additional file 3

. Summary and supplement of the ICI treatment in the existing clinical studies.

Additional file 4

. Complex tumor microenvironment prevents Tregs from responding in TME when ICIs involved.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Li, HB., Yang, ZH. & Guo, QQ. Immune checkpoint inhibition for pancreatic ductal adenocarcinoma: limitations and prospects: a systematic review. Cell Commun Signal 19 , 117 (2021). https://doi.org/10.1186/s12964-021-00789-w

Download citation

Received : 14 May 2021

Accepted : 16 September 2021

Published : 24 November 2021

DOI : https://doi.org/10.1186/s12964-021-00789-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Cell Communication and Signaling

ISSN: 1478-811X

• General enquiries: [email protected]

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

https://www.nist.gov/glossary-term/23046

Exclusion With Limitations

The result of a comparison between two hair samples in which the characteristics of the questioned hair differ from those present in the known hair sample, and therefore the donor of the known sample cannot be included as a possible source of the questioned hair. DiscussionThis result is reached in a compara- tive hair examination when differences are noted in the macroscopic or microscopic characteristics between the ques- tioned and known hairs; however, the differences are insufficient for an absolute exclusion of a person as a possible source. This could be due to the natural variation that occurs in hairs as a biological specimen, the effect that time or environment can have upon a hair, or the reference sample does not capture the complete variation of the individual's hair.