research articles process

• Visit the University of Nebraska–Lincoln
• Apply to the University of Nebraska–Lincoln
• Give to the University of Nebraska–Lincoln

Search Form

Overview of research process.

The Research Process

Anything you write involves organization and a logical flow of ideas, so understanding the logic of the research process before beginning to write is essential. Simply put, you need to put your writing in the larger context—see the forest before you even attempt to see the trees.

In this brief introductory module, we’ll review the major steps in the research process, conceptualized here as a series of steps within a circle, with each step dependent on the previous one. The circle best depicts the recursive nature of the process; that is, once the process has been completed, the researcher may begin again by refining or expanding on the initial approach, or even pioneering a completely new approach to solving the problem.

Identify a Research Problem

You identify a research problem by first selecting a general topic that’s interesting to you and to the interests and specialties of your research advisor. Once identified, you’ll need to narrow it. For example, if teenage pregnancy is your general topic area, your specific topic could be a comparison of how teenage pregnancy affects young fathers and mothers differently.

Review the Literature

Find out what’s being asked or what’s already been done in the area by doing some exploratory reading. Discuss the topic with your advisor to gain additional insights, explore novel approaches, and begin to develop your research question, purpose statement, and hypothesis(es), if applicable.

Determine Research Question

A good research question is a question worth asking; one that poses a problem worth solving. A good question should:

• Be clear . It must be understandable to you and to others.
• Be researchable . It should be capable of developing into a manageable research design, so data may be collected in relation to it. Extremely abstract terms are unlikely to be suitable.
• Connect with established theory and research . There should be a literature on which you can draw to illuminate how your research question(s) should be approached.
• Be neither too broad nor too narrow. See Appendix A for a brief explanation of the narrowing process and how your research question, purpose statement, and hypothesis(es) are interconnected.

Appendix A Research Questions, Purpose Statement, Hypothesis(es)

Develop Research Methods

Once you’ve finalized your research question, purpose statement, and hypothesis(es), you’ll need to write your research proposal—a detailed management plan for your research project. The proposal is as essential to successful research as an architect’s plans are to the construction of a building.

See Appendix B to view the basic components of a research proposal.

Appendix B Components of a Research Proposal

Collect & Analyze Data

In Practical Research–Planning and Design (2005, 8th Edition), Leedy and Ormrod provide excellent advice for what the researcher does at this stage in the research process. The researcher now

• collects data that potentially relate to the problem,
• arranges the data into a logical organizational structure,
• analyzes and interprets the data to determine their meaning, 
• determines if the data resolve the research problem or not, and
• determines if the data support the hypothesis or not.

Document the Work

Because research reports differ by discipline, the most effective way for you to understand formatting and citations is to examine reports from others in your department or field. The library’s electronic databases provide a wealth of examples illustrating how others in your field document their research.

Communicate Your Research

Talk with your advisor about potential local, regional, or national venues to present your findings. And don’t sell yourself short: Consider publishing your research in related books or journals.

Refine/Expand, Pioneer

Earlier, we emphasized the fact that the research process, rather than being linear, is recursive—the reason we conceptualized the process as a series of steps within a circle. At this stage, you may need to revisit your research problem in the context of your findings. You might also investigate the implications of your work and identify new problems or refine your previous approach.

The process then begins anew . . . and you’ll once again move through the series of steps in the circle.

Continue to Module Two

Appendix C - Key Research Terms

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research Research Tools and Apps

Research Process Steps: What they are + How To Follow

There are various approaches to conducting basic and applied research. This article explains the research process steps you should know. Whether you are doing basic research or applied research, there are many ways of doing it. In some ways, each research study is unique since it is conducted at a different time and place.

Conducting research might be difficult, but there are clear processes to follow. The research process starts with a broad idea for a topic. This article will assist you through the research process steps, helping you focus and develop your topic.

Research Process Steps

The research process consists of a series of systematic procedures that a researcher must go through in order to generate knowledge that will be considered valuable by the project and focus on the relevant topic.

To conduct effective research, you must understand the research process steps and follow them. Here are a few steps in the research process to make it easier for you:

Step 1: Identify the Problem

Finding an issue or formulating a research question is the first step. A well-defined research problem will guide the researcher through all stages of the research process, from setting objectives to choosing a technique. There are a number of approaches to get insight into a topic and gain a better understanding of it. Such as:

• A preliminary survey
• Case studies
• Interviews with a small group of people
• Observational survey

Step 2: Evaluate the Literature

A thorough examination of the relevant studies is essential to the research process . It enables the researcher to identify the precise aspects of the problem. Once a problem has been found, the investigator or researcher needs to find out more about it.

This stage gives problem-zone background. It teaches the investigator about previous research, how they were conducted, and its conclusions. The researcher can build consistency between his work and others through a literature review. Such a review exposes the researcher to a more significant body of knowledge and helps him follow the research process efficiently.

Step 3: Create Hypotheses

Formulating an original hypothesis is the next logical step after narrowing down the research topic and defining it. A belief solves logical relationships between variables. In order to establish a hypothesis, a researcher must have a certain amount of expertise in the field. 

It is important for researchers to keep in mind while formulating a hypothesis that it must be based on the research topic. Researchers are able to concentrate their efforts and stay committed to their objectives when they develop theories to guide their work.

Step 4: The Research Design

Research design is the plan for achieving objectives and answering research questions. It outlines how to get the relevant information. Its goal is to design research to test hypotheses, address the research questions, and provide decision-making insights.

The research design aims to minimize the time, money, and effort required to acquire meaningful evidence. This plan fits into four categories:

• Exploration and Surveys
• Data Analysis
• Observation

Step 5: Describe Population

Research projects usually look at a specific group of people, facilities, or how technology is used in the business. In research, the term population refers to this study group. The research topic and purpose help determine the study group.

Suppose a researcher wishes to investigate a certain group of people in the community. In that case, the research could target a specific age group, males or females, a geographic location, or an ethnic group. A final step in a study’s design is to specify its sample or population so that the results may be generalized.

Step 6: Data Collection

Data collection is important in obtaining the knowledge or information required to answer the research issue. Every research collected data, either from the literature or the people being studied. Data must be collected from the two categories of researchers. These sources may provide primary data.

• Questionnaire

Secondary data categories are:

• Literature survey
• Official, unofficial reports
• An approach based on library resources

Step 7: Data Analysis

During research design, the researcher plans data analysis. After collecting data, the researcher analyzes it. The data is examined based on the approach in this step. The research findings are reviewed and reported.

Data analysis involves a number of closely related stages, such as setting up categories, applying these categories to raw data through coding and tabulation, and then drawing statistical conclusions. The researcher can examine the acquired data using a variety of statistical methods.

Step 8: The Report-writing

After completing these steps, the researcher must prepare a report detailing his findings. The report must be carefully composed with the following in mind:

• The Layout: On the first page, the title, date, acknowledgments, and preface should be on the report. A table of contents should be followed by a list of tables, graphs, and charts if any.
• Introduction: It should state the research’s purpose and methods. This section should include the study’s scope and limits.
• Summary of Findings: A non-technical summary of findings and recommendations will follow the introduction. The findings should be summarized if they’re lengthy.
• Principal Report: The main body of the report should make sense and be broken up into sections that are easy to understand.
• Conclusion: The researcher should restate his findings at the end of the main text. It’s the final result.

LEARN ABOUT: 12 Best Tools for Researchers

The research process involves several steps that make it easy to complete the research successfully. The steps in the research process described above depend on each other, and the order must be kept. So, if we want to do a research project, we should follow the research process steps.

QuestionPro’s enterprise-grade research platform can collect survey and qualitative observation data. The tool’s nature allows for data processing and essential decisions. The platform lets you store and process data. Start immediately!

LEARN MORE         FREE TRIAL

MORE LIKE THIS

Best 15 Behavior Analytics Tools to Explore Your User Actions

Apr 8, 2024

Top 7 Concept Testing Tools to Elevate Your Ideas in 2024

AI Question Generator: Create Easy + Accurate Tests and Surveys

Apr 6, 2024

Top 17 UX Research Software for UX Design in 2024

Apr 5, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Uncategorized
• Video Learning Series
• What’s Coming Up
• Workforce Intelligence

• Privacy Policy

Buy Me a Coffee

Home » Research Process – Steps, Examples and Tips

Research Process – Steps, Examples and Tips

Table of Contents

Research Process

Definition:

Research Process is a systematic and structured approach that involves the collection, analysis, and interpretation of data or information to answer a specific research question or solve a particular problem.

Research Process Steps

Research Process Steps are as follows:

Identify the Research Question or Problem

This is the first step in the research process. It involves identifying a problem or question that needs to be addressed. The research question should be specific, relevant, and focused on a particular area of interest.

Conduct a Literature Review

Once the research question has been identified, the next step is to conduct a literature review. This involves reviewing existing research and literature on the topic to identify any gaps in knowledge or areas where further research is needed. A literature review helps to provide a theoretical framework for the research and also ensures that the research is not duplicating previous work.

Formulate a Hypothesis or Research Objectives

Based on the research question and literature review, the researcher can formulate a hypothesis or research objectives. A hypothesis is a statement that can be tested to determine its validity, while research objectives are specific goals that the researcher aims to achieve through the research.

Design a Research Plan and Methodology

This step involves designing a research plan and methodology that will enable the researcher to collect and analyze data to test the hypothesis or achieve the research objectives. The research plan should include details on the sample size, data collection methods, and data analysis techniques that will be used.

Collect and Analyze Data

This step involves collecting and analyzing data according to the research plan and methodology. Data can be collected through various methods, including surveys, interviews, observations, or experiments. The data analysis process involves cleaning and organizing the data, applying statistical and analytical techniques to the data, and interpreting the results.

Interpret the Findings and Draw Conclusions

After analyzing the data, the researcher must interpret the findings and draw conclusions. This involves assessing the validity and reliability of the results and determining whether the hypothesis was supported or not. The researcher must also consider any limitations of the research and discuss the implications of the findings.

Communicate the Results

Finally, the researcher must communicate the results of the research through a research report, presentation, or publication. The research report should provide a detailed account of the research process, including the research question, literature review, research methodology, data analysis, findings, and conclusions. The report should also include recommendations for further research in the area.

Review and Revise

The research process is an iterative one, and it is important to review and revise the research plan and methodology as necessary. Researchers should assess the quality of their data and methods, reflect on their findings, and consider areas for improvement.

Ethical Considerations

Throughout the research process, ethical considerations must be taken into account. This includes ensuring that the research design protects the welfare of research participants, obtaining informed consent, maintaining confidentiality and privacy, and avoiding any potential harm to participants or their communities.

Dissemination and Application

The final step in the research process is to disseminate the findings and apply the research to real-world settings. Researchers can share their findings through academic publications, presentations at conferences, or media coverage. The research can be used to inform policy decisions, develop interventions, or improve practice in the relevant field.

Research Process Example

Following is a Research Process Example:

Research Question : What are the effects of a plant-based diet on athletic performance in high school athletes?

Step 1: Background Research Conduct a literature review to gain a better understanding of the existing research on the topic. Read academic articles and research studies related to plant-based diets, athletic performance, and high school athletes.

Step 2: Develop a Hypothesis Based on the literature review, develop a hypothesis that a plant-based diet positively affects athletic performance in high school athletes.

Step 3: Design the Study Design a study to test the hypothesis. Decide on the study population, sample size, and research methods. For this study, you could use a survey to collect data on dietary habits and athletic performance from a sample of high school athletes who follow a plant-based diet and a sample of high school athletes who do not follow a plant-based diet.

Step 4: Collect Data Distribute the survey to the selected sample and collect data on dietary habits and athletic performance.

Step 5: Analyze Data Use statistical analysis to compare the data from the two samples and determine if there is a significant difference in athletic performance between those who follow a plant-based diet and those who do not.

Step 6 : Interpret Results Interpret the results of the analysis in the context of the research question and hypothesis. Discuss any limitations or potential biases in the study design.

Step 7: Draw Conclusions Based on the results, draw conclusions about whether a plant-based diet has a significant effect on athletic performance in high school athletes. If the hypothesis is supported by the data, discuss potential implications and future research directions.

Step 8: Communicate Findings Communicate the findings of the study in a clear and concise manner. Use appropriate language, visuals, and formats to ensure that the findings are understood and valued.

Applications of Research Process

The research process has numerous applications across a wide range of fields and industries. Some examples of applications of the research process include:

• Scientific research: The research process is widely used in scientific research to investigate phenomena in the natural world and develop new theories or technologies. This includes fields such as biology, chemistry, physics, and environmental science.
• Social sciences : The research process is commonly used in social sciences to study human behavior, social structures, and institutions. This includes fields such as sociology, psychology, anthropology, and economics.
• Education: The research process is used in education to study learning processes, curriculum design, and teaching methodologies. This includes research on student achievement, teacher effectiveness, and educational policy.
• Healthcare: The research process is used in healthcare to investigate medical conditions, develop new treatments, and evaluate healthcare interventions. This includes fields such as medicine, nursing, and public health.
• Business and industry : The research process is used in business and industry to study consumer behavior, market trends, and develop new products or services. This includes market research, product development, and customer satisfaction research.
• Government and policy : The research process is used in government and policy to evaluate the effectiveness of policies and programs, and to inform policy decisions. This includes research on social welfare, crime prevention, and environmental policy.

Purpose of Research Process

The purpose of the research process is to systematically and scientifically investigate a problem or question in order to generate new knowledge or solve a problem. The research process enables researchers to:

• Identify gaps in existing knowledge: By conducting a thorough literature review, researchers can identify gaps in existing knowledge and develop research questions that address these gaps.
• Collect and analyze data : The research process provides a structured approach to collecting and analyzing data. Researchers can use a variety of research methods, including surveys, experiments, and interviews, to collect data that is valid and reliable.
• Test hypotheses : The research process allows researchers to test hypotheses and make evidence-based conclusions. Through the systematic analysis of data, researchers can draw conclusions about the relationships between variables and develop new theories or models.
• Solve problems: The research process can be used to solve practical problems and improve real-world outcomes. For example, researchers can develop interventions to address health or social problems, evaluate the effectiveness of policies or programs, and improve organizational processes.
• Generate new knowledge : The research process is a key way to generate new knowledge and advance understanding in a given field. By conducting rigorous and well-designed research, researchers can make significant contributions to their field and help to shape future research.

Tips for Research Process

Here are some tips for the research process:

• Start with a clear research question : A well-defined research question is the foundation of a successful research project. It should be specific, relevant, and achievable within the given time frame and resources.
• Conduct a thorough literature review: A comprehensive literature review will help you to identify gaps in existing knowledge, build on previous research, and avoid duplication. It will also provide a theoretical framework for your research.
• Choose appropriate research methods: Select research methods that are appropriate for your research question, objectives, and sample size. Ensure that your methods are valid, reliable, and ethical.
• Be organized and systematic: Keep detailed notes throughout the research process, including your research plan, methodology, data collection, and analysis. This will help you to stay organized and ensure that you don’t miss any important details.
• Analyze data rigorously: Use appropriate statistical and analytical techniques to analyze your data. Ensure that your analysis is valid, reliable, and transparent.
• I nterpret results carefully : Interpret your results in the context of your research question and objectives. Consider any limitations or potential biases in your research design, and be cautious in drawing conclusions.
• Communicate effectively: Communicate your research findings clearly and effectively to your target audience. Use appropriate language, visuals, and formats to ensure that your findings are understood and valued.
• Collaborate and seek feedback : Collaborate with other researchers, experts, or stakeholders in your field. Seek feedback on your research design, methods, and findings to ensure that they are relevant, meaningful, and impactful.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

Research Questions – Types, Examples and Writing...

Research Procedures

• Open Access
• First Online: 28 March 2023

Cite this chapter

You have full access to this open access chapter

• Ivan Buljan   ORCID: orcid.org/0000-0002-8719-7277 3  

Part of the book series: Collaborative Bioethics ((CB,volume 1))

5103 Accesses

This chapter offers a guide on how to implement good research practices in research procedures, following the logical steps in research planning from idea development to the planning of analysis of collected data and data sharing. This chapter argues that sound research methodology is a foundation for responsible science. At the beginning of each part of the chapter, the subtitles are formulated as questions that may arise during your research process, in the attempt to bring the content closer to the everyday questions you may encounter in research. We hope to stimulate insight into how much we can predict about a research study before it even begins. Research integrity and research ethics are not presented as separate aspects of research planning, but as integral parts that are important from the beginning, and which often set the directions of research activities in the study.

• Research plan
• Research question
• Study design
• Measurement
• Protocol registration
• Reproducibility

You have full access to this open access chapter,  Download chapter PDF

Similar content being viewed by others

Ethical Issues in Research Methods

Research Integrity: Responsible Conduct of Research

What this chapter is about, case scenario: planning research.

This hypothetical scenario was adapted from a narrative about the process of poor research planning and its consequences. The original case scenario is developed by the Members of The Embassy of Good Science and is available at the Embassy of Good Science . The case is published under the Creative Commons Attribution-ShareAlike License, version 4.0 (CC BY-SA 4.0).

Professor Gallagher is a leader of a research project on moral intuitions in the field of psychology. She is working on the project with Dr. Jones, a philosopher, and Mr. Singh, a doctoral student. Although she has little experience in the matter, Dr. Jones is put as the principal investigator in the study design and analysis of the two experiments, while Mr. Singh prepares materials and conducts the experiments.

After the first experimental study, Mr. Singh sends the results to Dr. Jones for analysis. After some time, eager to enter the results in his thesis, Singh asks Dr. Jones about the results of the study. She admits that she forgot to formulate the hypothesis before data analysis, and now the results can be interpreted as confirmatory, regardless of the direction. They decide to formulate a hypothesis that will result in a positive finding.

Mr. Singh and Dr. Jones present the results to Dr. Gallagher, who is satisfied and proceeds with paper writing. In the second study, Dr. Jones formulates multiple hypotheses before the study begins. Mr. Singh conducts the study and sends the results to Dr. Jones. She performs the analysis by trying to find only significant differences between groups. Finally, to achieve significance, she excludes participants over 60 years from the analysis and while presenting the results, admits that to Prof Gallagher. Prof Galagher is happy about the results and proceeds with the paper writing, while Mr. Singh enters the results in his dissertation.

Before Mr. Singh has the public defense of his dissertation, one of the internal reviewers notices that some data has been excluded from the second study and only significant results were reported. She invites Mr. Singh for an examination board meeting during which MR Singh admits that the data has been excluded and that in the first study hypothesis was formulated after the results were known.

Questions for You

Why is hypothesizing after the results are known, as described in the first study, considered problematic?

What was wrong about reporting only significant results in Study 2?

How would you improve the entire research process described in the scenario?

Good research practice from the European Code of Conduct for Research Integrity:

Researchers take into account the state-of-the-art in developing research ideas.

Researchers make proper and conscientious use of research funds.

What to Do First When You Have an Idea?

It is difficult to come up with a good research idea, and if you struggle to come up with a new research direction, that is perfectly fine. Creative processes are the highest form of learning and developing an idea requires significant cognitive effort. In some cases, you may have an epiphany, where you would suddenly come up with a great idea for your research project. This is something popularized by stereotypes about scientists as eccentric figures who come up with brilliant ways of tackling things using only their intelligence and intuition. However, scientific work resembles ore mining. It takes a tremendous effort to read relevant scientific literature, communicate with your peers, plan, and, in some cases, attempt and fail before you even start digging for gold. As in a mine, you will need to dig a lot of rocks before you come across diamonds and gold.

Usually, the most important decisions are made before digging even begins. To decide where you will start mining, you start with the exploration of the terrain. In research, this means knowing your field of study. You may read an interesting piece in the scientific literature or listen to a presentation at a conference and then think of a hypothesis whose testing will answer an interesting and important question in your research field. On the other hand, sometimes you have to adjust your research interest so that they fit the specific aims of grant funding calls. It does not matter what the source of the idea is, there are always two things to consider when developing research ideas: the current state of the field and the resources available to you. Good research practice is to consider the state of the art in developing your research ideas and make the proper use of research funds. This does not mean that you are not allowed to develop research ideas if they address a research topic that has been neglected. It is the responsibility of a researcher to combine the best of the “old” evidence with new research developments. It is important to keep in mind that research is not performed in a vacuum and that the funds and resources provided by public or private funders are given with an expectation of an honest answer to a specific research question. The main responsibility for the proper use of research funds is on the researcher, and this is overseen by funders during and at the end of the proposal. Another recommendation refers to the use of state-of-the-art information as a basis for your research. The control system in this case is other scientists who read or evaluate your research, and who will recognize outdated research results.

Let’s get back to the analogy of the mine for a moment. If you are paid to dig in the mine, you are expected to find important ore. In our case, a research funder is an employer, and the researchers are workers who need to go down the mine and get their hands dirty in the search for new true information. If you are set to dig a deep hole in the ground with the possibility of finding gold and diamonds, but you do not get any guarantee that you will find them unless you chose an appropriate place in a specific period, you would probably spend a lot of time planning and trying to decide where to start digging, what to do when specific problems arise and to avoid ending with a huge number of worthless rocks instead of gold and diamonds. The process is similar to research planning since a significant amount of the research process can be defined before data collection begins. As valuable as it can be, a research idea is just a thought which needs to be translated into research practice to gain its full impact.

How to Formulate a Good Research Question?

Research is performed to answer a specific question. The research process can be observed as a complex tool that, if used properly, can give a clear answer to a posed question. The research question is the compass of the research process (or the mine if we continue with our mine analogy) since it determines the steps of the research process. It translates into specific research aims and, consequently, into testable research hypotheses. Formulation of a research question is a skill that develops over time, a skill that can be learned. Your research question should have a FINER structure, which stands for: F easible, I nteresting, N ovel, E thical and R elevant. Although initially developed as a set of recommendations for quantitative research, FINER recommendations can be applied to formulating a research question in any given field of science.

The feasibility of a research aim is often defined by time restrictions and funding because research is often burdened by deadlines and output requirements set by the funders. F easibility is also affected by the availability of technology, geographical restrictions, availability of participants, or availability of collaborators. If one considers all those factors, it is obvious that research interests play only a small part in the formulation of a research question. Ask yourself: What research can be published in an excellent journal if you have limited funds and only 1 year for research, with limited access to a specific technology? (Today, highly specialized experts may be a greater problem than the technology in question). You might experience that the formulation of the research question is mostly defined by non-research factors, because, in the end, it is better to have a completed than never-finished research.

There are other elements of the research question that are as important as feasibility. The first one to consider is E thics, which affects all parts of the research process due to its broad nature. If research is not ethical, then it should not be conducted. In a mining analogy, ethics is training and safety, which helps you to protect others and yourself during the entire process. To get back to the best research practices, researchers should make proper use of research funds and fulfill the basic research aim – the benefit to society. This also implies treating members of that society with respect, respecting their privacy and dignity, and being honest and transparent about the research process and results. Therefore, when determining the feasibility of a research study, ethics aspects are the first to consider, along with the objective factors of time, cost, and manpower.

I nterest, N ovelty, and R elevance from the FINER guidance are the elements of the research question that increase the chances of getting funding or the chances for a journal publication, and they are closely aligned. Regardless of the audience (researchers, publishers, non-experts), research should be new to be interesting and relevant. However, doing research just for the novelty’s sake is analogous to the digger who starts digging a new mine every couple of days. It gives you the thrill of a new beginning, but you have not dug deep enough to get to the real results. Relevance, defined in this context as a significant add-on to the current knowledge, can be assessed with a high probability of success by a thorough search for available evidence. The main aim of that process is to identify research or practice gaps that can be filled to improve general knowledge.

Interest is related to the principal internal motivation of an individual to pursue research goals. The interest to pursue research aims is difficult to assess. When planning research, do you consider that research is interesting to you, your peers, potential users, or all three? Probably the last, but here is the catch. Interest is the most subjective part of research planning. Research planning could be understood as a balance between your interest and all other factors that affect the research outcome. A good research idea is often the compromise between objective possibilities and a desire to make a research discovery. If the research idea is interesting but extremely difficult (or even impossible) to conduct in given circumstances, you will end up frustrated. On the other hand, if you decide to perform research based solely on convenience (because it is something for which is easy to get funded or someone is offering you a research topic you are not interested in), it will be very difficult to stay motivated to complete the study.

The more structured your research question is, the easier it is to determine which research design is best to test the hypothesis and statistical analysis is more straightforward. Let’s look at several examples of research questions in biomedical research: Are psychedelics more effective in the treatment of psychosis than the standard treatment? What are the opinions of young fathers on exclusive breastfeeding of their spouses? Which percentage of the population has suffered from post-COVID-19 syndrome? Intuitively, for each of posed research questions, we would try to find answers differently. In cases of comparison of treatment methods and assessment of population percentage, we could express the results quantitatively, e.g., we could state explicitly how much the psychedelics treatment is better compared to standard methods in terms of days of remission or everyday functionality or an explicit number of people in the sample who had COVID-19-related symptoms. On the other hand, the answers to the question about the opinions of young fathers about exclusive breastfeeding are not straightforward or numerical, but more textual and descriptive. It is an example of the research question that would be more suitable for qualitative research. Qualitative and quantitative study designs answer different types of research questions and are therefore suitable for different situations. It is important to carefully consider and choose the most appropriate study design for your research question because only then can you get valid answers.

To conclude, research question development is the crucial factor in setting research direction. Although framed as a single sentence, it defines numerous parts of the research process, from research design to data analysis. On the other hand, non-research factors also have an equal role in research questions and need to be considered.

Literature Search

In a literature search, researchers go through the relevant information sources to systematically collect information, i.e. foreground knowledge, about a specific research phenomenon and/or procedure. While research information is readily available online not only to researchers but to the whole public, the skill of systematic literature search and critical appraisal of evidence is a specific research skill. A literature search is closely tied with the development of the research aim, because you may want to change it after you read about previous research.

When doing a literature search, you must be careful not to omit previous studies about the topic. Here we have two directions that must be balanced. The first one is to do a very precise search to find specific answers, and the other one is to perform a wide, sensitive search that will include many synonyms and combinations of words to discover articles that related to a specific term. Both of those approaches have their advantages and disadvantages: a precise search is less time-consuming and retrieves a small number of studies. However, it may omit important results, so you may end up performing studies for which we already have established conclusions. This creates waste in research because you will spend time and resources, and involve participants in unnecessary work, which would be unethical. You may also miss citing important studies. On the other hand, if you perform a search that is too wide, you will spend a lot of time filtering for useful articles, which leaves less time for doing research.

Researchers design, carry out, analyze and document research in a careful and well-considered manner.

Researchers report their results in a way that is compatible with the standards of the discipline and, where applicable, can be verified and reproduced.

What Is the Optimal Study Design for My Research?

Study designs are one of the main heuristics related to the reader’s perception of the credibility of research information. Also, different study designs give answers to different research questions. It is intuitively easy to understand that different approaches should be taken if the question is about the percentage of infected people in the population vs about which drug is the most effective in the treatment of the disease. The roughest categorization of the study designs is observational and experimental (Box 3.1 ). However, in different scientific areas, even that type of categorization is not enough, since study designs can be theoretical, as in physics or mathematics, or critical, as in humanities, and those types of research will not be covered in this chapter.

Box 3.1 Types of Study Designs

Observational study designs :.

Case study / case series / qualitative study : All three types of study designs take into account a small number of participants and examine the phenomenon of interest in-depth but cannot make generalizations about the entire population.

Case-control study : Individuals with a certain outcome or disease are selected and then information is obtained on whether the subjects have been exposed to the factor under investigation more frequently than the carefully selected controls. This approach is quick and cost-effective in the determination of factors related to specific states (e.g., risk factors), but it relies too much on records and/or self-report, which may be biased.

Cross-sectional study : Best study design for determining the prevalence and examination of relationships between variables that exist in the population at a specific time. Although it is simple to perform, and relatively cheap, it is susceptible to various types of bias related to participant selection, recall bias, and potential differences in group sizes.

Cohort study : Participants are followed over a certain period (retrospectively or prospectively) and data are compared between exposed and unexposed groups to determine predictive factors for the phenomenon of interest.

Experimental study designs :

Randomized controlled trial (RCT) : Participants are allocated to treatment or control groups using randomization procedures to test the strength of the interventions.

Quasi-experimental trial : Participants are allocated to treatment or control groups to test the strengths of the interventions, but there is no randomization procedure.

For some research areas (e.g. health sciences, social sciences), there is another type of research often referred to as evidence synthesis, or literature review. The literature review is a review of evidence-based on a formulated research question and elements. They differ in their scope and methodology (Box 3.2 ).

Box 3.2 Most Common Types of Review

Systematic review : A type of review that searches systematically for, appraises, and synthesizes research evidence, often adhering to guidelines on the conduct of a review.

Scoping review : Type of review which serves as a preliminary assessment of the potential size and scope of available research literature to identify the nature and extent of research evidence (usually including ongoing research).

Meta-analysis : Statistical synthesis of the results from quantitative studies to provide a more precise effect of the results.

Rapid review : A type of review that assesses what is already known about a policy or practice issue, by using systematic review methods to quickly search and critically appraise existing research to inform practical steps.

Umbrella review : Specific type of review that searches and assesses compiling evidence from multiple reviews into one accessible and usable document. Focuses on broad conditions or problems for which there are competing interventions and highlights reviews that address these interventions and their results.

How to Assess which Study Design Is Most Suitable for Your Research Question?

Based on the research aim, one may already get a hint about which study design will be applied, since different study designs give answers to different research questions. However, very often a research question is not so straightforward. Sometimes the research aim could be to determine whether category X is superior to category Y, related to the specific outcome. In those cases, one must determine what the core outcome of the study is (e.g., testing of the effectiveness of two interventions, the scores on current differences between two groups, or the changes over time between different groups), and then it is not difficult to determine the study type in question. In principle, a single research question can be answered with a single study design. However, what we can also use are substitute study designs that can give approximate answers to the question we are asking but will never give as clear an answer as the appropriate design. For example, if we want to explore the reasons early-career researchers seek training in research integrity using a survey approach, we could list all possible answers and say to participants to choose everything that applies to them. The more appropriate study design would be to use a qualitative approach instead because in the survey approach the assumption is that we already know most of the reasons. The survey approach gives us the answer which answer is the most frequent of all. It is a subtle, but important difference. Similarly, although we can test causation using a cohort approach, the evidence for causation is never strong enough in a cohort study as it would be in an experimental study, simply because in a cohort study the researcher does not have control over the independent variable. For example, if we would test the effects of alcohol uptake on the occurrence of cancer, we would compare participants who drink versus those who do not drink to determine the incidence of cancer and make the conclusion about the association between alcohol and cancer. However, the true study design for testing the causation is the randomized controlled trial, where participants are randomized into the interventional and control group, the researcher can give an exact amount of alcohol based on persons’ weight, over a specific period, and in the end, compare the incidence between two groups. However, that type of study would not be an ethical study, so it is not possible to do it. So, there are subtle, but important differences which answer whether can specific and good formulated research questions can be tested and answered fully with only one study design, but due to the various reasons (time restrictions, ethics, cost-benefit analysis) we often use substitute study designs.

When describing people involved in the research process, researchers often refer to them as “participants” or “respondents” (in the case of surveys). A more precise term would be to name the group based on the population they are drawn from (children, people with specific diseases, or people from a specific geographical area). The appropriate term to use would be “participants”, since people are willingly involved in the research process, and the generation of new findings depends on them. Being a participant in a research process means that a person has willingly entered into a research, without any real or imagined coercion, possesses respect and interest for the research topic, with the understanding that positive aspects of research findings encompass the research situation and contribute to general knowledge. This would be a definition of an ideal participant and the researcher should avoid a situation where the participants are coerced to enter research, whether by situational factors or personal reasons because that will probably result in a decrease in motivation for participation and lower quality of research findings. To act ethically and to improve the quality of the research you have to inform participants about the reasons for the study, its purpose, research procedure, their rights, and expected outcomes. A potential pitfall in the research process can happen if all information were not given to participants at the beginning of a research. On the other hand, if a participant enters willingly into the trial, but possesses no real interest in the research topic, it may also affect the motivation for participation in research, because those participants may consider the topic irrelevant and not take the research process seriously (it is easy to imagine a situation where teenagers in a classroom willingly decide to take the survey and participate in research about personality traits, but quickly lose interest after the second page of the questionnaire). All those things are not reflected in the research report but may have an enormous influence on the research findings. Therefore, it is important to define the population of interest and try to motivate participants by providing them with all information before the research begins. Some additional ways to increase participant retention are financial rewards or similar incentives. There are several sampling strategies used when approaching participants for a study (Box 3.3 ).

Box 3.3 Most Common Sampling Methods

Simple random sampling : Each member of the defined population has an equal chance of being included in the study. The sampling is often performed by a coin toss, throwing dice, or (most commonly) using a computer program.

Stratified random sampling : The population of interest is first divided into strata (subgroups) and then we perform random sampling from each subgroup. In this way, the sample with better reflects the target population in specific (relevant) characteristics.

Cluster random sampling : In cluster sampling, the parts of the population (subgroups) are used as sampling units instead of individuals.

Systematic sampling : Participants are selected by equal intervals set before the data collection begins (e.g., every third of every fifth participant who enters the hospital).

Convenience sampling : Participants are approached based on availability. This is perhaps the most common sampling method, especially for survey research.

Purposive sampling : This is the most common approach in qualitative study designs. Researchers choose participants (or they define their characteristics in detail), based on their needs since participants with those special characteristics are the research topic.

It is difficult to give clear criteria on when to stop collecting data. In the case of pre-registered studies, the stopping rule is defined in the protocol. Examples include time restrictions (e.g. 1 month), or the number of participants (e.g. after collecting data on 100 participants). If the research protocol has not been pre-registered, then the stopping rule should be explained in detail in the publication, with reasons. In the latter case, it is never completely clear if the stopping happened after researchers encountered the desired result or if it has been planned. The practice of stopping after you collect sufficient data to support your desired hypothesis is highly unethical since it can lead to biased findings. Therefore, the best way of deciding to terminate the data collection is to pre-register your study, or at least define the desired number of participants by performing sample size calculation before the study begins and pre-registering your study. More about pre-registration and biases which it eliminates will be said later in the chapter.

Ethics of the Sample Size: Too Small and Too Big Samples

A common problem in sampling is that researchers often determine the desired number of participants in a study. The problem is that the response rate is always lower than 100% (in survey research it is often around 15–20%), and a certain percentage of participants drops out of research, resulting in a sample size significantly lower than initially planned. The sample used in research can be too small, and there is a possibility that you will not find a true effect between groups, and in that case, you would make a type II error. The reason is that in small-scale studies the error margin is big, and you would need an extremely large effect size to reach statistical significance. On the other hand, in cases of a big sample, the problem is different. If you have big samples, even small effects will be statistically significant, but the effect size may be negligible. The reason is that within big samples, the margin of error is small, and consequently, every difference is statistically significant. Once again, the proper solution (practically and ethically) for this issue is to calculate the minimum sample size needed to determine the desired difference between groups to avoid the issues with small samples and report effect sizes also, to avoid issues related to (too) big samples.

What We Can and What Cannot Measure?

When it comes to measuring in research, that part is mostly associated with statistical analysis of research data. The principal thing in statistical analysis is to determine the nature of the main outcome variable. In qualitative research (e.g. interview, focus group) or a systematic review without meta-analysis, statistical analysis is not necessary. On the other hand, for quantitative studies (a term often used for mostly case-control, cross-sectional, cohort, and interventional studies) the most important part of the research plan is to define the outcome which can be measured.

In general, there are two types of variables: qualitative and quantitative. When it comes to statistical analysis of qualitative variables (in a statistical context you will encounter the terms nominal and ordinal variables), we can do only basic functions, like counting and comparing the proportions between different groups, but we are not able to calculate mean or standard deviation, because those variables do not possess numerical characteristics. Examples of qualitative variables in research can be the number of surviving patients in a group at the end of the trial, self-reported socioeconomic status as a demographic characteristic, or any binary (yes/no) question in a questionnaire. In some cases, qualitative variables may be coded with numbers, but that does not make them quantitative. A good example is jersey numbers where numbers serve only as a label and not as a measure of quantity (e.g. if you have team player numbers 2, 4, 6, you probably will not state that the average jersey number is 4 because the very concept of the “average” jersey number is absurd). On the other hand, for quantitative variables, differences between numbers indicate the differences in value (e.g. if you say that person X is 1.80 m high, you know that that person is taller than person Y who is 1.70 m tall). You can also calculate different statistical parameters, like mean and median, and dispersion measures, which gives you a more flexible approach in the choice of statistical tests, especially those tests for differences between groups. On the other hand, applying statistical tests would mean that you are more familiar with statistics, which sometimes may present a problem for less (and more) experienced researchers.

When Is the Time to Consult with a Statistician (and Do You Have to)?

Some (lucky) researchers possess sufficient knowledge to perform data analysis themselves. They usually do not need to rely on somebody else to do the statistical analysis for their study. For everybody else, statistical analysis is a crossroad where one needs to decide on including a person with statistical knowledge in a research team or to learn statistical analyses by themselves. The usual process is that the research team defines the research aim, spends time collecting data, collects data, and then tries to find a statistician who will analyse the data. If we keep in mind that research often has high stakes (e.g. doctoral diploma) and researchers are under a great time and financial pressure, the decision to include a statistician is sound and logical, but is it really necessary? The inclusion of a statistician in research when the data are already collected is similar to the situation when you give a cook an already finished stew and ask him/her how it can be improved. The cook may help with the decorations and give some spice which would make the food look and taste better but cannot change the essence of the food since it is already cooked. It is the same with data. The golden rule of statistics is “garbage in, garbage out”, referring to a situation where poorly collected data or data of poor quality will give rise to wrong conclusions. Researchers should know statistics, not only because of the statistical analysis but because statistical reasoning is important in the formulation of measurable research aims. Therefore, statistical analysis is an important part of responsible research and begins with the formulation of the research aim. Statistical experts should be included in the study at that point.

Statisticians usually analyse data based on the initially set research aim. They send back the results of the data analysis to the research team, and they all together (in an ideal scenario) write the manuscript. The dataset remains in the possession of the principal researcher and the paper is published in a journal. Many journals and funders require that the data are publicly available so that anyone can use it, respecting the FAIR principles. Keeping that in mind, the process when somebody else is doing statistical analysis for you requires an enormous level of trust for statisticians, because they can do analysis wrong but you may never know it. Unless, of course, someone else analyses publicly available data and sees the error. In that case, you are also responsible for the analysis because it does not matter that you did not perform it. In some cases, this may lead to the retraction of the paper, which consequently may lead to certain consequences for you (especially if the articles are the basis for a doctoral thesis). If you are willing to put trust in someone to do data analysis, that is perfectly fine, just be aware of this risk, and remember that people make mistakes, very often unintentionally, and therefore a double check by a third party would be recommended.

On the other hand, if you are willing to learn how to do statistical analysis, the good news is that today there are lots of resources to help you. The first thing about statistics you need to know is that you do not need to know all statistics to do statistics. The only knowledge about statistics and statistical programs you need is the one that would help you do the analysis of your research aim and test the research hypothesis. To do that, you will have to see the data you have and search online for ways to analyze a specific problem. You can use tutorials of the statistical program that simultaneously give instructions about the statistical principles and procedures for analysis. Today, most of those programs have online videos and detailed tutorials. Some of those programs are user-friendly and free (e.g., JAMOVI or JASP ), some are commercial (e.g., SPSS, Statistica), and some are less user-friendly but free and available (e.g., R programming language ). If you are a beginner, use a more user-friendly program that has detailed instructions and try to do the statistical analysis by yourself. It is expected that you will make errors, so it would be good if someone more experienced looked at the results and provides feedback on your first attempts.

There are many tutorials on how to do statistical analysis, but far less on how to do proper data entry, which is the preparation of data for statistical analysis. Usually, the data entry table is made in a computer program that provides a tabular view of the data (e.g., Microsoft Excel). The golden rule is that each column represents a variable collected in research, by the order it was collected in the research and that each row represents the unit of the analysis (usually participant, text, article, or any other unit). In a separate sheet or a document, there should be a codebook that contains information about each level of each variable in the dataset, in a way that a person who is not familiar with research can understand the nature of the variable. The codebook should always accompany the dataset, so if the dataset is shared publicly, the codebook should also be shared. The rule of thumb for the data entry is that textual variables are entered as texts and quantitative variables as numbers, and textual variables can later be coded with numbers if necessary. The table for data entry should be made before the research begins, and it is good to seek help from a statistician when defining that, too.

Researchers publish results and interpretations of research in an open, honest, transparent, and accurate manner, and respect the confidentiality of data or findings when legitimately required to do so.

Preregistration of Research Findings

Pre-registration refers to the presentation of the research plan before the research begins. This process serves as the quality control mechanism because it prevents a change in the research hypothesis and methodology to fit the data collected. Pre-registration of research findings should be done after the research has been approved by the ethics committee. There are various registries, some of which are more discipline-specific (e.g., ClinicalTrials.gov for clinical studies) while others are open to different disciplines and study designs (e.g., Open Science Framework ). For the pre-registration of a study, one should clearly define all steps related to the research aim, methods, planned analysis, and planned use of data. Pre-registration of data is nothing more than the public sharing of a research plan. However, even that relatively simple procedure helps eliminate specific biases and decreases the probability of unethical behavior. Pre-registration eliminates the problem of h ypothesizing a fter the r esults are k nown (so-called HARKing) because you need to state your hypothesis publicly before the research begins. Pre-registration should be done before the actual research begins, since you may have already collected the data and modified your hypothesis so that it fits your data (this is called PARKing – p re-registering a fter the r esults are k nown), which should be avoided since it is not a true pre-registration.

Why is pre-registration good for research? When a study is pre-registered, researchers will follow the research plan and planned analysis and will not alter the study protocol and statistical analysis unless there is a valid and strong reason for protocol modification. Many journals today require that studies are pre-registered and that research data are shared. It is recommended to pre-register not only the study aim and methods, planned analysis, but also planned impact, data use, and authorship. When pre-registering authorship, you make clear from the beginning of the study the roles and expectations of each member of the research team. If during the research process some changes happen with the study protocol, those should be clearly explained and pointed out in the final publication, because deviations from the protocol can sometimes bring suspicion in the interpretation of the results if they are not reported. Pre-registration can be peer-reviewed and some problems, which would affect the final interpretation of the results, can be addressed even before the study begins. Finally, when pre-registered, you have the evidence that it was you who came up first with a specific research idea.

One problem that pre-registration cannot prevent is research spin or exaggeration in the scope of study results. Even if data have been carefully collected and properly analyzed, the interpretation of the results is up to the researcher. You should be honest (and modest) when interpreting the results of your study, by stating the true magnitude of your results and putting them in the context of the previous studies.

After the research has been published, the data used in research should be made available to everyone who wants to use them, since data sharing helps research replication and evidence synthesis. You can read more about data sharing in the chapter on Data Management and the chapter on Publication and Dissemination.

With this knowledge in mind, how would you improve the research procedure from the case scenario at the beginning of this chapter?

Author information

Authors and affiliations.

Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia

Ivan Buljan

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ivan Buljan .

Editor information

Editors and affiliations.

School of Medicine, University of Split, Split, Croatia

Ana Marusic

If You Want to Learn More

The embassy of good science.

Replicability

AllTrials campaign: https://embassy.science/wiki/Theme:0bb5e4f7-9336-4ca8-92e3-c506413d1450

Forensic statistics to detect data fabrication: https://embassy.science/wiki/Theme:467f5cf6-d41f-42a0-9b19-76556579845d

Pre-registration of animal study protocols

Prospective registration of clinical trials

Statistical pre-registration

Data driven hypothesis without disclosure (“HARKing”)

Insufficiently reported study flaws and limitations

Spin of research results .

Published Articles

Cooke A, Smith D, Booth A (2012) Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res 22(10):1435–1443. https://doi.org/10.1177/1049732312452938

Grant MJ, Booth A (2009) A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J 26(2):91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Kerr NL (1998) HARKing: hypothesizing after the results are known. Personal Soc Psychol Rev 2(3):196–217. https://doi.org/10.1207/s15327957pspr0203_4

Lipowski EE (2008) Developing great research questions. Am J Health Syst Pharm 65(17):1667–1670. https://doi.org/10.2146/ajhp070276

Onwuegbuzie AJ, Leech NL (2006) Linking research questions to mixed methods data analysis procedures. Qual Rep 11(3):474–498

Rau JL (2004) Searching the literature and selecting the right references. Respir Care 49(10):1242–1245

Richardson WS, Wilson MC, Nishikawa J, Hayward RS (1995) The well-built clinical question: a key to evidence-based decisions. ACP J Club 123(3):A12–A13

Yamada Y (2018) How to crack pre-registration: toward transparent and open science. Front Psychol 9:1831. https://doi.org/10.3389/fpsyg.2018.01831

Hulley SB. Designing clinical research: Lippincott Williams & Wilkins; 2007.

University of Oxford. The Centre for Evidence-Based Medicine .

HealthKnowledge .

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Designing Clinical Research. 4th ed. Philadelphia: Lippincott Williams and Wilkins; 2013. p. 14–22.

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and permissions

Copyright information

© 2023 The Author(s)

About this chapter

Buljan, I. (2023). Research Procedures. In: Marusic, A. (eds) A Guide to Responsible Research. Collaborative Bioethics, vol 1. Springer, Cham. https://doi.org/10.1007/978-3-031-22412-6_3

Download citation

DOI : https://doi.org/10.1007/978-3-031-22412-6_3

Published : 28 March 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-22411-9

Online ISBN : 978-3-031-22412-6

eBook Packages : Biomedical and Life Sciences Biomedical and Life Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is a Research Design | Types, Guide & Examples

What Is a Research Design | Types, Guide & Examples

Published on June 7, 2021 by Shona McCombes . Revised on November 20, 2023 by Pritha Bhandari.

A research design is a strategy for answering your   research question  using empirical data. Creating a research design means making decisions about:

• Your overall research objectives and approach
• Whether you’ll rely on primary research or secondary research
• Your sampling methods or criteria for selecting subjects
• Your data collection methods
• The procedures you’ll follow to collect data
• Your data analysis methods

A well-planned research design helps ensure that your methods match your research objectives and that you use the right kind of analysis for your data.

Table of contents

Step 1: consider your aims and approach, step 2: choose a type of research design, step 3: identify your population and sampling method, step 4: choose your data collection methods, step 5: plan your data collection procedures, step 6: decide on your data analysis strategies, other interesting articles, frequently asked questions about research design.

• Introduction

Before you can start designing your research, you should already have a clear idea of the research question you want to investigate.

There are many different ways you could go about answering this question. Your research design choices should be driven by your aims and priorities—start by thinking carefully about what you want to achieve.

The first choice you need to make is whether you’ll take a qualitative or quantitative approach.

Qualitative research designs tend to be more flexible and inductive , allowing you to adjust your approach based on what you find throughout the research process.

Quantitative research designs tend to be more fixed and deductive , with variables and hypotheses clearly defined in advance of data collection.

It’s also possible to use a mixed-methods design that integrates aspects of both approaches. By combining qualitative and quantitative insights, you can gain a more complete picture of the problem you’re studying and strengthen the credibility of your conclusions.

Practical and ethical considerations when designing research

As well as scientific considerations, you need to think practically when designing your research. If your research involves people or animals, you also need to consider research ethics .

• How much time do you have to collect data and write up the research?
• Will you be able to gain access to the data you need (e.g., by travelling to a specific location or contacting specific people)?
• Do you have the necessary research skills (e.g., statistical analysis or interview techniques)?
• Will you need ethical approval ?

At each stage of the research design process, make sure that your choices are practically feasible.

Prevent plagiarism. Run a free check.

Within both qualitative and quantitative approaches, there are several types of research design to choose from. Each type provides a framework for the overall shape of your research.

Types of quantitative research designs

Quantitative designs can be split into four main types.

• Experimental and   quasi-experimental designs allow you to test cause-and-effect relationships
• Descriptive and correlational designs allow you to measure variables and describe relationships between them.

With descriptive and correlational designs, you can get a clear picture of characteristics, trends and relationships as they exist in the real world. However, you can’t draw conclusions about cause and effect (because correlation doesn’t imply causation ).

Experiments are the strongest way to test cause-and-effect relationships without the risk of other variables influencing the results. However, their controlled conditions may not always reflect how things work in the real world. They’re often also more difficult and expensive to implement.

Types of qualitative research designs

Qualitative designs are less strictly defined. This approach is about gaining a rich, detailed understanding of a specific context or phenomenon, and you can often be more creative and flexible in designing your research.

The table below shows some common types of qualitative design. They often have similar approaches in terms of data collection, but focus on different aspects when analyzing the data.

Your research design should clearly define who or what your research will focus on, and how you’ll go about choosing your participants or subjects.

In research, a population is the entire group that you want to draw conclusions about, while a sample is the smaller group of individuals you’ll actually collect data from.

Defining the population

A population can be made up of anything you want to study—plants, animals, organizations, texts, countries, etc. In the social sciences, it most often refers to a group of people.

For example, will you focus on people from a specific demographic, region or background? Are you interested in people with a certain job or medical condition, or users of a particular product?

The more precisely you define your population, the easier it will be to gather a representative sample.

• Sampling methods

Even with a narrowly defined population, it’s rarely possible to collect data from every individual. Instead, you’ll collect data from a sample.

To select a sample, there are two main approaches: probability sampling and non-probability sampling . The sampling method you use affects how confidently you can generalize your results to the population as a whole.

Probability sampling is the most statistically valid option, but it’s often difficult to achieve unless you’re dealing with a very small and accessible population.

For practical reasons, many studies use non-probability sampling, but it’s important to be aware of the limitations and carefully consider potential biases. You should always make an effort to gather a sample that’s as representative as possible of the population.

Case selection in qualitative research

In some types of qualitative designs, sampling may not be relevant.

For example, in an ethnography or a case study , your aim is to deeply understand a specific context, not to generalize to a population. Instead of sampling, you may simply aim to collect as much data as possible about the context you are studying.

In these types of design, you still have to carefully consider your choice of case or community. You should have a clear rationale for why this particular case is suitable for answering your research question .

For example, you might choose a case study that reveals an unusual or neglected aspect of your research problem, or you might choose several very similar or very different cases in order to compare them.

Data collection methods are ways of directly measuring variables and gathering information. They allow you to gain first-hand knowledge and original insights into your research problem.

You can choose just one data collection method, or use several methods in the same study.

Survey methods

Surveys allow you to collect data about opinions, behaviors, experiences, and characteristics by asking people directly. There are two main survey methods to choose from: questionnaires and interviews .

Observation methods

Observational studies allow you to collect data unobtrusively, observing characteristics, behaviors or social interactions without relying on self-reporting.

Observations may be conducted in real time, taking notes as you observe, or you might make audiovisual recordings for later analysis. They can be qualitative or quantitative.

Other methods of data collection

There are many other ways you might collect data depending on your field and topic.

If you’re not sure which methods will work best for your research design, try reading some papers in your field to see what kinds of data collection methods they used.

Secondary data

If you don’t have the time or resources to collect data from the population you’re interested in, you can also choose to use secondary data that other researchers already collected—for example, datasets from government surveys or previous studies on your topic.

With this raw data, you can do your own analysis to answer new research questions that weren’t addressed by the original study.

Using secondary data can expand the scope of your research, as you may be able to access much larger and more varied samples than you could collect yourself.

However, it also means you don’t have any control over which variables to measure or how to measure them, so the conclusions you can draw may be limited.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

As well as deciding on your methods, you need to plan exactly how you’ll use these methods to collect data that’s consistent, accurate, and unbiased.

Planning systematic procedures is especially important in quantitative research, where you need to precisely define your variables and ensure your measurements are high in reliability and validity.

Operationalization

Some variables, like height or age, are easily measured. But often you’ll be dealing with more abstract concepts, like satisfaction, anxiety, or competence. Operationalization means turning these fuzzy ideas into measurable indicators.

If you’re using observations , which events or actions will you count?

If you’re using surveys , which questions will you ask and what range of responses will be offered?

You may also choose to use or adapt existing materials designed to measure the concept you’re interested in—for example, questionnaires or inventories whose reliability and validity has already been established.

Reliability and validity

Reliability means your results can be consistently reproduced, while validity means that you’re actually measuring the concept you’re interested in.

For valid and reliable results, your measurement materials should be thoroughly researched and carefully designed. Plan your procedures to make sure you carry out the same steps in the same way for each participant.

If you’re developing a new questionnaire or other instrument to measure a specific concept, running a pilot study allows you to check its validity and reliability in advance.

Sampling procedures

As well as choosing an appropriate sampling method , you need a concrete plan for how you’ll actually contact and recruit your selected sample.

That means making decisions about things like:

• How many participants do you need for an adequate sample size?
• What inclusion and exclusion criteria will you use to identify eligible participants?
• How will you contact your sample—by mail, online, by phone, or in person?

If you’re using a probability sampling method , it’s important that everyone who is randomly selected actually participates in the study. How will you ensure a high response rate?

If you’re using a non-probability method , how will you avoid research bias and ensure a representative sample?

Data management

It’s also important to create a data management plan for organizing and storing your data.

Will you need to transcribe interviews or perform data entry for observations? You should anonymize and safeguard any sensitive data, and make sure it’s backed up regularly.

Keeping your data well-organized will save time when it comes to analyzing it. It can also help other researchers validate and add to your findings (high replicability ).

On its own, raw data can’t answer your research question. The last step of designing your research is planning how you’ll analyze the data.

Quantitative data analysis

In quantitative research, you’ll most likely use some form of statistical analysis . With statistics, you can summarize your sample data, make estimates, and test hypotheses.

Using descriptive statistics , you can summarize your sample data in terms of:

• The distribution of the data (e.g., the frequency of each score on a test)
• The central tendency of the data (e.g., the mean to describe the average score)
• The variability of the data (e.g., the standard deviation to describe how spread out the scores are)

The specific calculations you can do depend on the level of measurement of your variables.

Using inferential statistics , you can:

• Make estimates about the population based on your sample data.
• Test hypotheses about a relationship between variables.

Regression and correlation tests look for associations between two or more variables, while comparison tests (such as t tests and ANOVAs ) look for differences in the outcomes of different groups.

Your choice of statistical test depends on various aspects of your research design, including the types of variables you’re dealing with and the distribution of your data.

Qualitative data analysis

In qualitative research, your data will usually be very dense with information and ideas. Instead of summing it up in numbers, you’ll need to comb through the data in detail, interpret its meanings, identify patterns, and extract the parts that are most relevant to your research question.

Two of the most common approaches to doing this are thematic analysis and discourse analysis .

There are many other ways of analyzing qualitative data depending on the aims of your research. To get a sense of potential approaches, try reading some qualitative research papers in your field.

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A research design is a strategy for answering your   research question . It defines your overall approach and determines how you will collect and analyze data.

A well-planned research design helps ensure that your methods match your research aims, that you collect high-quality data, and that you use the right kind of analysis to answer your questions, utilizing credible sources . This allows you to draw valid , trustworthy conclusions.

Quantitative research designs can be divided into two main categories:

• Correlational and descriptive designs are used to investigate characteristics, averages, trends, and associations between variables.
• Experimental and quasi-experimental designs are used to test causal relationships .

Qualitative research designs tend to be more flexible. Common types of qualitative design include case study , ethnography , and grounded theory designs.

The priorities of a research design can vary depending on the field, but you usually have to specify:

• Your research questions and/or hypotheses
• Your overall approach (e.g., qualitative or quantitative )
• The type of design you’re using (e.g., a survey , experiment , or case study )
• Your data collection methods (e.g., questionnaires , observations)
• Your data collection procedures (e.g., operationalization , timing and data management)
• Your data analysis methods (e.g., statistical tests  or thematic analysis )

A sample is a subset of individuals from a larger population . Sampling means selecting the group that you will actually collect data from in your research. For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

In statistics, sampling allows you to test a hypothesis about the characteristics of a population.

Operationalization means turning abstract conceptual ideas into measurable observations.

For example, the concept of social anxiety isn’t directly observable, but it can be operationally defined in terms of self-rating scores, behavioral avoidance of crowded places, or physical anxiety symptoms in social situations.

Before collecting data , it’s important to consider how you will operationalize the variables that you want to measure.

A research project is an academic, scientific, or professional undertaking to answer a research question . Research projects can take many forms, such as qualitative or quantitative , descriptive , longitudinal , experimental , or correlational . What kind of research approach you choose will depend on your topic.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, November 20). What Is a Research Design | Types, Guide & Examples. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/methodology/research-design/

Is this article helpful?

Shona McCombes

Other students also liked, guide to experimental design | overview, steps, & examples, how to write a research proposal | examples & templates, ethical considerations in research | types & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Stages in the research process

Affiliation.

• 1 Faculty of Health, Social Care and Education, Anglia Ruskin University, Cambridge, England.
• PMID: 25736674
• DOI: 10.7748/ns.29.27.44.e8745

Research should be conducted in a systematic manner, allowing the researcher to progress from a general idea or clinical problem to scientifically rigorous research findings that enable new developments to improve clinical practice. Using a research process helps guide this process. This article is the first in a 26-part series on nursing research. It examines the process that is common to all research, and provides insights into ten different stages of this process: developing the research question, searching and evaluating the literature, selecting the research approach, selecting research methods, gaining access to the research site and data, pilot study, sampling and recruitment, data collection, data analysis, and dissemination of results and implementation of findings.

Keywords: Clinical nursing research; nursing research; qualitative research; quantitative research; research; research ethics; research methodology; research process; sampling.

• Data Collection / methods
• Pilot Projects
• Research Design / standards*
• Research Personnel / education*
• United Kingdom

The Research Process

Research methods outline the whole process that a researcher undertakes in collecting, analyzing, presenting and interpreting findings about a phenomena. Scientific methodology guides the researcher when carrying out research. The systematic approach provides a logical sequence for a researcher to follow when seeking to obtain objective data that can form the basis for authentic conclusions. Research methods in scientific research provides a step-by-step process comprising of seven steps.

• Identifying the research problem

Research starts with identifying the problem to be investigated. Identifying the research problem not only highlights the gap to be filled, but also the research methods, approaches, and techniques suitable to collect data for solving the research problem.

• Reviewing literature

Literature review provides a basis for foundational knowledge about problem area. It also identifies the research methods that were used in previous studies and the suitability of those research methods to the studies.

• Defining Terms and Concepts

It is necessary to define terms and concepts to enhance the clarity of the phenomenon under investigation. Defining the terms and concepts makes the scope more manageable. This enhances the ease of collecting data using the predetermined research methods.

• Defining the Population

The research problem and purpose are crucial in identifying the specific group of people, objects and organizations that will be involved in the study. The necessity of defining the population is to ensure that the researcher maintains focus on a particular context of the study. The nature of the population is also critical in identifying the research methods and tools to be adopted.

• Developing the research design

The research design is the overall plan of the study. As a road map for the entire study, the research design specifies who will participate in the study, how, when, and where data will be collected. To carry out the study effectively, there is need to carefully think through all steps of the study and outline a clear plan of research methods. The research design has several elements that must be explained in the research methods section.

• Collecting data

This stage involves engaging in actual collection of data that will help the researcher answer the research questions. Based on the research methods guidelines commonly referred to as the research onion , there are various strategies and data collection instruments that can be adopted depending on the nature of the research question, target population, and context of study.

• Analysis data

The foregoing six steps culminate in this final stage of data collection. Data collected must be analyzed to make meaning of it. Research methods section of the study specifies how data will be analyzed. Data analysis can take the qualitative or quantitative approach. Qualitative data can be analyzed thematically through coding and theming. Quantitative data may require use of some statistical software such as SPSS, R, and Eviews. Hypotheses are rejected or not rejected depending on the statistical significance of the findings while research questions are answered from the themes created from qualitative data.

stratford-blog

Publishing a research article, publishing a journal article-critical steps.

Cancel reply

Your comment ...

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Author Desk

• Author Guidelines
• Publication Charges
• Modes of Payment
• Review Process
• Ethics and Malpractice
• Online Submissions
• Procedure of Publication Process
• Copyright Agreement

DOI NO. 10.53819

Journal Indexing

Download Files

• Manuscript Template
• Journals Articles

Author’s Copyright

The author retains the copyright of the published manuscript.

2023 – 2024 IMPACT FACTOR

Availability of the published manuscript.

The published manuscript is available in both Online and in Print. Authors requiring hard-copy print of the issue in which their paper appears can make orders and this will be processed on demand.

• Peer review guide
• Submission guide
• Online Submission
• Journal Publication process
• Book Publication Process
• Business & Management
• Journal of Procurement & Supply Chain
• Journal of Finance and Accounting
• Journal of Strategic Management
• Journal of Entrepreneurship & Project Management
• Journal of Marketing and Communication
• Journal of Economics
• Social Sciences
• Journal of Hospitality & Tourism Management
• Journal of Human Resource & Leadership
• Journal of Sociology, Psychology & Religious Studies
• Journal of Public Policy & Governance
• Journal of Education
• Information Technology
• Journal of Information and Technology
• Agriculture
• Journal of Agriculture
• Medicine & Healthcare
• Journal of Medicine, Nursing & Public Health
• Journal of pharmacy & Biochemistry
• Life Sciences
• Journal of Biological Sciences
• Editorial board
• Journal System
• Journals Store
• Books store
• Submission: [email protected]

You cannot copy or paste content on this page.

• Open access
• Published: 08 March 2022

WHO-led consensus statement on vaccine delivery costing: process, methods, and findings

• Ann Levin 1 ,
• Laura Boonstoppel 2   na1 ,
• Logan Brenzel 3   na1 ,
• Ulla Griffiths 4   na1 ,
• Raymond Hutubessy 5   na1 ,
• Mark Jit 6   na1 ,
• Vittal Mogasale 7   na1 ,
• Sarah Pallas 8   na1 ,
• Stephen Resch 9   na1 ,
• Christian Suharlim 9   na1 &
• Karene Hoi Ting Yeung 5   na1  

BMC Medicine volume  20 , Article number:  88 ( 2022 ) Cite this article

2529 Accesses

10 Citations

2 Altmetric

Metrics details

Differences in definitions and methodological approaches have hindered comparison and synthesis of economic evaluation results across multiple health domains, including immunization. At the request of the World Health Organization’s (WHO) Immunization and Vaccines-related Implementation Research Advisory Committee (IVIR-AC), WHO convened an ad hoc Vaccine Delivery Costing Working Group, comprising experts from eight organizations working in immunization costing, to address a lack of standardization and gaps in definitions and methodological guidance. The aim of the Working Group was to develop a consensus statement harmonizing terminology and principles and to formulate recommendations for vaccine delivery costing for decision making. This paper discusses the process, findings of the review, and recommendations in the Consensus Statement.

The Working Group conducted several interviews, teleconferences, and one in-person meeting to identify groups working in vaccine delivery costing as well as existing guidance documents and costing tools, focusing on those for low- and middle-income country settings. They then reviewed the costing aims, perspectives, terms, methods, and principles in these documents. Consensus statement principles were drafted to align with the Global Health Cost Consortium costing guide as an agreed normative reference, and consensus definitions were drafted to reflect the predominant view across the documents reviewed.

The Working Group identified four major workstreams on vaccine delivery costing as well as nine guidance documents and eleven costing tools for immunization costing. They found that some terms and principles were commonly defined while others were specific to individual workstreams. Based on these findings and extensive consultation, recommendations to harmonize differences in terminology and principles were made.

Conclusions

Use of standardized principles and definitions outlined in the Consensus Statement within the immunization delivery costing community of practice can facilitate interpretation of economic evidence by global, regional, and national decision makers. Improving methodological alignment and clarity in program costing of health services such as immunization is important to support evidence-based policies and optimal resource allocation. On the other hand, this review and Consensus Statement development process revealed the limitations of our ability to harmonize given that study designs will vary depending upon the policy question that is being addressed and the country context.

Peer Review reports

Immunization has been shown to provide a high return on investment across low- and middle-income countries [ 1 ]. Nevertheless, disparities in immunization access persist within and between countries. With the launch of the new Immunization Agenda 2030 [ 2 ], many low- and middle-income countries (LICs and MICs) are considering introducing new vaccines or vaccine-related technologies, life-course immunization programs, and improving the effectiveness and efficiency of their immunization programs. To determine the feasibility of doing so, estimation of vaccine procurement and delivery costs is of considerable interest to policymakers, program managers, researchers, and other stakeholders concerned with improving immunization programs. In particular, results from delivery cost studies can help countries in decision-making and planning on introducing new infant and life-course vaccines and technologies, preparation of budgets and financing for rollout of vaccines, and evaluation of alternative service delivery approaches.

Recent reviews of immunization delivery cost literature identified a lack of standardization in methods and reporting, making cross-study comparison and synthesis difficult [ 3 ]. These discrepancies limit the interpretability and utility of immunization cost study evidence for immunization program decision-making. In light of these challenges, the World Health Organization (WHO) Immunization and Vaccines-related Implementation Research Advisory Committee (IVIR-AC) recommended at their March 2018 meeting that the WHO Guidance on Vaccine Delivery Costing be updated [ 4 ]. An ad hoc Working Group comprising vaccine delivery costing experts Footnote 1 was therefore convened by the WHO secretariat to review guidance documents and tools on vaccine delivery costing, focused on low- and middle-income country settings. This initial review found that several groups were already developing methodological guidance to address the disparate definitions and approaches in the field, which partly address the original IVIR-AC’s request. In March 2019, IVIR-AC modified its request to instead review guidance documents and costing tools, assess their similarities and differences, and identify gaps in guidance [ 5 ]. In addition, the Working Group recommended that a Consensus Statement be developed to harmonize the differences in costing terminology and principles for groups working in vaccine delivery costing.

For the purpose of this paper, vaccine delivery costing is defined as “costs associated with delivering immunizations to target populations, exclusive of vaccine procurement costs” [ 6 ].

This paper describes the history and process involved to develop the Consensus Statement on Vaccine Delivery Costing, the methods used and the findings of the review of guidance documents and costing tools, and terms and principles as well as recommendations agreed upon by the Working Group.

Process of developing the consensus statement

The consultation process of coming to agreement on a Consensus Statement included setting up a time-limited Working Group of staff of organizations working in vaccine delivery costing (who are also the authors of this paper), conducting a review of guidance documents, costing tools, and other documents and the costing terms, methods, and principles used in these, agreeing upon the costing terminologies and principles, making recommendations to harmonize their differences, writing the text of the Consensus Statement and Annexes (Additional file 1 ). Figure 1 shows a timeline of the meetings and activities that led to the development of the Consensus Statement.

Timeline of the consultation process to develop a Consensus Statement (CS) on vaccine delivery costs

In March 2018, the IVIR-AC initiated the process and requested that WHO update its guidance for conducting vaccine delivery costing in LICs and MICs so that methods used in costing tools and guidance documents could be standardized among WHO and other organizations.

As a follow-up, the WHO secretariat set up a Working Group of Experts comprising staff of organizations conducting research and policy advice on vaccine delivery costing in LICs and MICs to ensure that no parallel efforts were taking place. The initial Working Group comprised of technical experts from the Bill & Melinda Gates Foundation (BMGF), International Vaccine Institute (IVI), Levin & Morgan LLC, UNICEF, and WHO, and two members of IVIR-AC. The group noted that there are several ongoing workstreams conducting cost studies and developing guidance documents and/or costing tools, with different purposes and approaches to costing. In addition, some of these workstreams had developed guidance documents specific to their approach, which were already in the public domain. Thus, a review of these would be required to determine if an additional vaccine delivery costing guidance would be necessary. The Working Group also suggested that a presentation be made at the IVIR-AC meeting in March 2019 to present the findings on the different workstreams to determine the next steps.

In March 2019, the WHO team and the BMGF-funded ThinkWell project (Immunization Costing Action Network [ICAN]) presented to IVIR-AC on findings from the discussion with the Working Group [ 2 ]. Footnote 2 IVIR-AC recommended that an in-person workshop meeting be held with other groups working on vaccine delivery costing so that a consensus could be reached on the best way to standardize costing terms and principles.

In July 2019, WHO and the BMGF convened a meeting with eleven experts from different organizations and institutions in immunization economics during an International Health Economics Association (iHEA) meeting in Basel, Switzerland. The Working Group was expanded to include technical experts from other organizations involved in vaccine delivery costing such as Harvard (Expanded Programme on Immunization Costing [EPIC] studies) and the United States Centers for Disease Control and Prevention (CDC). Based on the subject matter knowledge and professional experience of the Working Group members, the different purposes of the workstreams were discussed and a matrix of costing tools listing out the characteristics of each was created. The Working Group agreed that it would be useful to develop a Consensus Statement that presents the different purposes of each workstream, a review of existing vaccine delivery costing guidance documents and tools, and agreed-upon costing terms and principles.

As a follow-up from the meeting in Basel, from August 2019 to March 2020, an analysis of guidance documents and tools was conducted for each of the four workstreams identified by the Working Group. The Group identified similarities and differences in costing methods, terms and principles among the approaches and in guidance documents, and gaps where further guidance was needed.

Based on these findings, the WHO team developed a proposed draft Consensus Statement report with recommendations for costing terms and principles that could be adhered to for future vaccine delivery costing work and accompanying annexes that summarized the findings from the review on costing terms, costing principles, and methods for vaccine delivery costing. After extensive consultation within the Working Group and several rounds of written revisions to reach consensus on the statement, the findings and recommendations were presented to IVIR-AC in September 2020. The IVIR-AC commended the process to create the Consensus Statement (Additional file 1 ) [ 7 ].

Review of vaccine delivery costing guidance documents and tools

The first step was to conduct a landscape analysis of the organizations involved in vaccine delivery costing and their workstreams, and the available guidance documents and tools on vaccine delivery costs. This landscape analysis was conducted through discussions between the Working Group members during teleconferences and an in-person meeting as well as internet searches of websites of organizations working in the field (e.g., ICAN, Immunization Economics) between August 2019 and March 2020. It was not a systematic literature review and did not aim to include general health service costing tools and guidance documents beyond those with known use for costing immunization in LICs and MICs. However, the analysis built on the recent systematic review and reporting guidance for immunization costing studies conducted by some working group member organizations [ 3 ].

The second step was to compare the characteristics of the guidance documents and tools for immunization costing identified in terms of (1) how costing terms were defined in the guidance documents and costing tools; (2) whether data collection, sampling, and analysis were described in the guidance documents; and (3) whether costing principles were specified in guidance documents.

To review the costing terms in the guidance documents, the definitions were extracted from the source documents and entered into a table so that similarities and differences could be compared qualitatively and recommendations could be made for harmonized definitions for key terms. The costing principles and the guidance text, including on data collection, sampling, and analysis, were also compared and entered into a table by workstreams to assess the similarities, differences, and gaps. To do so, the costing principles in the guidance documents were compared to the ones in the checklist in the Global Health Cost Consortium (GHCC) [ 8 ] that has become a normative reference standard for global health costing work. These principles are similar to those found in the CHEERS checklist [ 9 ]. Recommendations were then made for harmonized principles in the Consensus Statement.

Existing immunization delivery costing workstreams

The Working Group identified four major current workstreams on vaccine delivery costing in LICs and MICs. These include the following: (1) retrospective routine immunization (multiple vaccines) cross-sectional costing, (2) retrospective single-vaccine costing, (3) new vaccine introduction cost projection, and (4) national immunization program cost projection (Fig. 2 ). Although the workstreams had involvement from particular organizations at the time of the review, they are defined by their different objectives and corresponding methodologies and constitute a typology of immunization delivery costing work to which other organizations and practitioners beyond those listed contribute.

Major current workstreams in vaccine delivery costing identified by the Working Group. Note: 2YL, 2nd Year of Life; BMGF, Bill & Melinda Gates Foundation; C4P, Cervical Cancer Prevention and Control Costing; CDC, United States Centers for Disease Control and Prevention; CHOLTOOL, Oral Cholera Vaccine Costing Tool; cMYP, comprehensive multi-year plan; EPIC, Expanded Programme on Immunization Costing; ICAN, Immunization Costing Action Network; IVI, International Vaccine Institute; MVICT, Malaria Vaccine Immunization Costing Tool; SIICT, Seasonal Influenza Immunization Costing Tool; TCVCT, Typhoid Conjugate Vaccine Costing Tool; VTIA, Vaccine Technology Costs and Health Impact Assessment Tool; WHO, World Health Organization

The first workstream is focused on estimating retrospective (i.e., already incurred) routine immunization cross-sectional costs of service delivery units at a single point in time for multiple vaccines delivered through the routine immunization program. These analyses focus on estimating routine immunization costs incurred at the facility, district, and higher administrative levels in the health system. Such analyses typically estimate unit costs (cost per dose, cost per person, or cost per fully immunized person [FIP]). Some examples of this work include the EPIC studies [ 10 ] and other work by institutes, such as the Harvard T.H. Chan School of Public Health, Wits University, Curatio Foundation, PAHO, ThinkWell, UNICEF, Johns Hopkins University, and PATH. The objectives of research within this workstream are to develop benchmarks for costs to be used in future studies, to analyze variation in unit costs, and to compare the findings with data from other costing studies [ 4 ].

The second workstream is to estimate retrospective costs for a specific vaccine or campaign, typically using incremental costing. That is, it usually aims to measure the value of additional resources employed to introduce a new vaccine or conduct a vaccination campaign. This is often done through data collection at a single point in time (post-campaign or post-introduction) with reference to documents and recall by key informants to estimate which resource use was specifically incremental. Examples of such studies include those conducted by groups such as EPIC, ThinkWell, CDC, and IVI. This workstream includes retrospective cost studies of vaccine implementation using vaccine-specific costing tools (e.g., Cervical Cancer Prevention and Control Costing [C4P], Oral Cholera Vaccine Costing Tool [CHOLTOOL], Malaria Vaccine Immunization Costing Tool [MVICT], Seasonal Influenza Immunization Costing Tool [SIICT], and Typhoid Conjugate Vaccine Costing Tool [TCVCT]). These studies yield results that will assist countries with comparing budgeted amounts to actual implementation resource use, budgeting for future immunization activities, and conducting cost-effectiveness analyses that compare the incremental resource use for a specific vaccine introduction or campaign with its incremental health impacts.

The third workstream is focused on estimating new vaccine introduction costs through projection of the value of resources or ingredients (e.g., time, equipment, training, and vaccines) needed for vaccine introduction, typically using incremental costing for a specific period. Data for these analyses are obtained through interviews with program managers and facility visits to obtain current information on personnel time, supplies, equipment, and other resources as well as retrospective cost data from other vaccine introduction. These analyses are often conducted using costing tools, including some of the same tools used for retrospective single-vaccine costing (e.g., C4P, CHOLTOOL, MVICT, SIICT, and TCVCT). These studies produce cost estimates that will assist countries with planning and decision-making on new vaccines during the introduction period.

The fourth workstream is projection of immunization program costs. Some costing tools used to produce these estimates include the comprehensive multi-year plan (cMYP), 2nd Year of Life (2YL), and OneHealth tool where the activities of a national program and related cost is entered for a baseline year and then the future years are projected. These analyses are an integral part of strategic planning for budgeting and resource mobilization over a specific period of time such as 5 years. Whereas work under the first three workstreams may produce estimates of financial, economic, or undepreciated financial costs, projections under the fourth workstream are intended to estimate undepreciated financial costs (i.e., undiscounted monetary outlays).

In practice, projects may combine elements of multiple workstreams (e.g., retrospective single vaccine costing in one country may be used to help inform estimates of new vaccine introduction costs for a different vaccine or delivery strategy).

Existing guidance documents on vaccine delivery costing

Table 1 shows the nine guidance documents on vaccine delivery costing identified by the Working Group. Some of these provide guidance for more than one type of costing. Three are for estimation of retrospective routine immunization cross-sectional costs, five are for estimation of retrospective single-vaccine costs, five are for projection of new vaccine introduction costs, and one for projection of immunization program costs. The list of costing tools for vaccine delivery identified by the Working Group is shown in Additional file 1 : Table A2b.

Table 2 shows a comparison of costing term definitions among the various guidance documents. It shows that among the different guidance documents, definitions are generally similar but have differences in wording, e.g., vaccine delivery cost, economic cost, start-up/ introduction cost, and prospective cost. Also, some terms (retrospective costing, cost projections, bottom-up and top-down costing) are only defined in the Global Health Costing Consortium reference case. Note that some guidance documents have been grouped together since they were developed by the same teams; i.e., (i) EPIC documents and (ii) WHO vaccine-specific costing tool user manuals.

Figure 3 shows the percentage of guidance documents with definitions of individual costing terms. As can be seen, most documents had definitions of financial cost, economic cost, capital cost, recurrent cost, incremental cost, and vaccine delivery cost, and about half of these defined start-up/introduction cost. Fewer than half of the guidance documents had definitions of perspective, micro-costing (ingredients costing), full costing, retrospective costing, or cost projection.

Percentage of guidance documents with definitions of costing terms ( N = 9)

Several gaps were noted from the review. Most guidance documents did not go into detail about some methodological decision points in costing, such as how the choice of perspective will affect which costs are included as financial costs, which may limit the comparability of such costs across studies. For example, if a payer or provider perspective is used, the organizations included in the study definition as “payers” or “providers” will determine whose monetary outlays are considered as financial costs. If a donor (e.g., Gavi) provides funding to a UNICEF country office for social mobilization for a new vaccine introduction, expenditures of those funds will be included as financial costs only if the study perspective is defined as including UNICEF (e.g., a provider perspective defined as all partners “providing” the new vaccine introduction activities, or a health sector perspective including all health sector partners); however, if the study is conducted from a perspective that does not include UNICEF (e.g., a provider perspective defined as only the government “providing” the new vaccine introduction activities, or a government perspective), these resources from UNICEF would not be counted as financial costs but only as economic costs as an in-kind contribution from UNICEF.

Also, most guidance documents did not address whether to include economic costs of existing capital such as equipment or building space, or how to make or assess assumptions for slackness (i.e., available unused capacity) of existing capital goods. Also, vaccine delivery costing definitions differ on whether actual vaccine product costs should be included or not. If not, which specific aspects of the vaccine product costs should be excluded (e.g., vaccine only, diluent, syringes, safety boxes, freight, and insurance). For financial costs, the guidance review suggested whether to include existing personnel costs will depend on whether the costing is incremental or full.

Figure 4 shows the percentage of documents that recommended key costing principles (details in Additional file 1 : Table A3). As can be seen, most guidance documents recommended principles on stating objectives, defining units, describing time horizon, methods and data sources, and annualizing capital costs, while less than half recommend specifying the perspective, scope, sampling, data collection timing, discount rates, shadow prices, exploring variation, analyzing uncertainty, and methods of communicating results.

Percentage of costing principles recommended by guidance documents ( N = 9)

In Table 3 , the recommendations of guidance documents on data collection and analysis are disaggregated by workstream. While guidance is given on some aspects in all documents, in other cases, no guidance is provided. Specifically, guidance is given on data collection for all of the workstreams with the exception of projection of new vaccine introduction costs.

Recommended costing terminology and principles

After reviewing the definitions of costing terms, the following definitions of costing terms are recommended:

Vaccine delivery costs

Costs associated with delivering immunization programs to target populations, exclusive of vaccine costs.

Vaccine cost

At a minimum includes the cost of the vaccine and diluent (if applicable); the analysis should include accounting for wastage rates; the analyst should specify whether this also includes injection supplies (syringes), international shipment, insurance, and customs/duties.

Financial cost

Monetary outlays, with straight-line depreciation for capital goods; does not include opportunity costs for use of resources or donated goods and services from sources other than the payer(s) defined in the analysis. Definition is dependent on perspective since monetary outlays are specific to the payer(s) defined in the analysis.

Economic cost

The value of all resources utilized, regardless of the source of financing. Includes opportunity costs for use of existing resources and any donated goods or services from any source. Capital costs are annualized and discounted.

Undepreciated financial cost

Financial costs without depreciation of capital costs (note: such costs have been termed “initial investment” in some costing tools and referred to as fiscal costs in previous analyses.)

Recurrent cost

Value of resources that last less than one year. Start-up activity costs may include recurrent costs.

Capital cost

Value of resources lasting more than one year such as equipment, buildings, and trainings. Start-up activity costs may include capital costs.

Incremental cost

Cost of adding a new service/intervention or a package of services/interventions over and above an existing program; inclusion of existing resources will depend on assumptions made about excess capacity (i.e., whether resources are underemployed; if there are no slack resources (e.g., all personnel time is fully allocated before the addition of the new service/intervention), then their use for the new service or intervention incurs an opportunity cost that should be included—either by measurement or assumption).

Baseline cost as well as the additional/incremental cost of the new intervention, including vaccine cost.

Cost projection

Estimation of future costs of both recurrent and capital inputs.

Prospective data collection

Direct observation of resource use during intervention implementation; i.e., data are collected concurrently with intervention implementation.

Retrospective data collection

Data collection after resource use is completed.

Start-up cost

Cost of initial one-time programmatic activities. Examples may include initial micro-planning, initial training activities, and initial sensitization/social mobilization/information, education, and communication (IEC); does not include routine or repeated programmatic activities such as refresher training or annual microplanning. Start-up activities may include both recurrent and capital costs; they are defined by the non-repeating nature of the activity, not the type of input.

Micro-costing

Focuses on granular accounting of input prices and quantities; disaggregates costs of particular output into specific goods and services consumed.

Bottom-up costing

Measures input quantities at the client (e.g., per vaccination administered) or activity level.

Top-down costing

Divides overall program cost or expenditures, often including those at administrative levels above service delivery level, by number of outputs to calculate unit cost.

Perspective

The point of view considered for costs (and benefits, if included) in a costing study, by whom the costs were incurred. Payers are the disbursing agents for a good or service, and may differ from the original source of funding. A provider perspective includes costs incurred by health service providers (can be limited to the government), a payer perspective includes costs to the payer(s), such as government or an external partner, while the societal perspective includes all costs incurred by providers as well as clients.

Shared cost

Shared resources that are not used only for immunization, but also for other productive activities.

The recommended costing principles include the following.

Definitions of terms used in studies of vaccine delivery costing should conform closely to the recommended definitions in this Consensus Statement.

The study scope in terms of its purpose, audience, target population, time horizon, and service/output should be clearly stated. It should also state whether data collection will be prospective or retrospective, and whether the analysis will be retrospective or a cost projection.

The perspective of the cost estimation should be stated and justified.

Types of costs to be generated should be clearly defined in terms of start-up/introduction or non-start-up/introduction (sometimes called operating costs), recurrent and capital, undepreciated financial, financial or economic, and incremental or full. Capital costs should be appropriately annualized and depreciated for financial and economic costs and the discount rate justified.

The scope of the inputs to be estimated should be defined, justified, and if needed referenced. For example, do the costs include national and sub-national costs or only facility-level service delivery costs? Are non-immunization costs included?

The “units” in the unit costs for strategies, services, and interventions should be defined, e.g., cost per dose administered.

If incremental costing is conducted, any assumptions made regarding existing health system capacity should be described (see GHCC reference case, pg. 64).

The selection of the data sources, including any adjustments to price data (e.g., inflation or currency conversion) should be described and referenced.

The methods for estimating the quantity of inputs should be described—whether top-down or bottom-up, methods of allocation, use of shadow prices and the opportunity cost of time, and methods for excluding research and evaluation costs.

Costs should be mapped and reported as either inputs or activities:

Resource inputs include, for example, personnel time, vaccines, injection and safety supplies, vehicles, fuel, per diem and travel allowances, cold chain equipment, stationery, laboratory equipment, and buildings;

Program activities include, for example, vaccine procurement, service delivery, training, micro-planning, social mobilization, and advocacy and communication, monitoring and evaluation, surveillance, adverse event following immunization monitoring, and supervision.

Some boundaries around costs included in the analysis may be employed to keep the costing scope feasible and will depend on the purpose of the costing study, with the rationale for any exclusions provided; use discretion about including one-time costs that are unique or unlikely to be replicated or transferable across settings (for example, new vaccine launches with the President). Clarify definition and threshold for including or excluding small costs that have expected small contribution (e.g., <$25) to total costs in aggregate across all sampled units, such as the use of existing office supplies by health facility staff.

The sampling strategy employed should aim for internal and external validity of the data Footnote 3 . Sampling strategy should be stated, described, and justified, depending on the workstream and costing objectives. Sampling of different service delivery units is desirable as it provides a more representative picture of costs and highlights cost variation and cost drivers for a strategy or vaccine.

Variation in the cost of the intervention by site/organization, sub-population, or by other drivers of heterogeneity should be explored and reported for retrospective analyses when possible.

The uncertainty around the cost estimates should be appropriately characterized when feasible, (e.g., sensitivity analyses; ranges of results for different input parameter scenarios for cost projections; mean and standard deviation for non-representative samples with multiple units; and confidence intervals or credible intervals for retrospective analyses).

Inclusion and exclusion criteria: “stopping rules” Footnote 4 should be defined, explaining which costs are included and the respective rationale.

Cost estimates should be communicated clearly and transparently to enable decision-makers to interpret and use the results relevant to the original policy and/or programmatic question.

The lack of standardization in terminology, implementation, and principles for vaccine delivery costing has resulted in difficulties in making comparisons among studies, reducing the potential for synthesis of economic evidence across studies for immunization program policy, planning, budgeting, and implementation. As noted earlier, governments need to know the cost of vaccine delivery in order to make decisions on introducing new infant and life-course vaccines, budgeting, and for making improvements in service delivery. The review indicates that existing guidance documents differ somewhat in the inclusion and definitions, of costing terms and costing principles that are recommended, reflecting in part differences in the aims and scope of the costing study.

The review of guidance documents and tools on vaccine delivery costing and iterative discussions among the Working Group members revealed considerable agreement among the different groups working in vaccine delivery costing. Most of the documents made the distinction between economic and financial costs as well as recurrent and capital costs. However, fewer went into detail about the perspective to choose, definition of some costing terms such as start-up costs, micro-costing, and bottom-up/top-down costing, and in some cases, recommended approaches for data collection and analyses. The review also identified gaps in guidance for some analyses, e.g., such as how perspective affects financial costs calculation.

The review revealed that different workstreams focus on distinct aspects of immunization costing with different purposes. These require different types of data collection and analyses. For example, retrospective costing of vaccination focuses on estimating actual resource use, benchmarking of costs, and investigation of variation at the facility and other levels. Cost projections, on the other hand, focus on estimation of (typically incremental) costs to assist in decision-making, preparation of budgets, and evaluating different approaches to a new technology, vaccine, or service delivery strategy.

The process to achieve a consensus statement of vaccine delivery costing methods was facilitated by having extensive consultations with different organizations conducting this work. It also was facilitated by conducting reviews of the guidance documents and costing tools so that similarities, differences, and gaps could be identified. Other strengths of the process include broad and ongoing engagement of experts across various workstreams, including members of the Immunization Economics Community of Practice [ 18 ], as well as dedicated support for facilitation, review, and write-up.

The process to develop a consensus statement provides lessons for developing agreement among other organizations and researchers on types of research methods and tools in other study areas. It requires the potential to bring together organizations working on similar research and then having the time and resources to develop consensus. In addition, it is useful to have some teleconferences and in-person (or virtual meetings with break-out sessions) meetings to have sufficient time to come to consensus.

One limitation of the exercise was that a systematic review was not conducted and some guidelines and costing tools may have been missed. More engagement of country-level practitioners and data and analysis experts outside of those directly involved in the workstreams, as well as a systematic literature search for any methodology documents beyond those known to the workstream participants, would have strengthened the process.

The work on immunization costing is extensive but some gaps were identified. The guidance documents, mostly user manuals for costing tools and the 2002 WHO guidance on introducing new vaccines for cost projections of new vaccines, are not sufficiently detailed regarding data collection and analyses. That is, these do not include instructions on methods of data collection and sampling and analysis methods, when required. Researchers that have piloted the costing tools have also noted that the manuals need to provide more instructions on perspective (see [ 19 ], for example). For example, there is a need for more guidance on how to treat perspective when there is more than one source of financing of vaccines, how to handle slack, etc. As a result, it would be useful to add to current user manuals or develop a new guidance document for cost projections for both single vaccines, multiple vaccines, and immunization programs.

This review and Consensus Statement development process revealed the limitations of our ability to harmonize given that study designs will vary depending upon the policy question that is being addressed and the country context. The Working Group hopes that the consensus statement will contribute to the development of costing guidelines and tools for new vaccines (single or multiple) and immunization programs that are better aligned in terms of definitions, methods, and reporting.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its additional file.

The experts in the Working Group were from the World Health Organization, UNICEF, US Centers for Disease Control and Prevention, the Bill & Melinda Gates Foundation, Harvard T.H. Chan School of Public Health, International Vaccine Institute, ThinkWell, and the London School of Hygiene and Tropical Medicine.

Two presentations were made: (1) the WHO team’s Ann Levin presented on WHO/IVI/PATH’s work conducting vaccine delivery cost projections with costing tools and the lack of standardization with other workstreams; and (2) ThinkWell’s Annette Ozaltin, representing the BMGF portfolio, presented their work on vaccine costing and the repository of vaccine delivery costs known as the Vaccine Cost Catalogue.

Internal validity refers to the extent of systematic bias in an estimate while external validity is the extent to which the cost estimate can be directly applied to other programmatic setting. (GHCC, pg. A15–A16).

A “stopping rule” defines and explains which costs are included, and how the line is drawn between inclusions and exclusions. (GHCC reference case, pg. B-2)

Abbreviations

2nd Year of Life

Bill & Melinda Gates Foundation

Cervical Cancer Prevention and Control Costing

United States Centers for Disease Control and Prevention

Oral Cholera Vaccine Costing Tool

Comprehensive multi-year plan

• Consensus statement

Expanded Programme on Immunization Costing

Fully immunized person

Global Health Cost Consortium

Immunization Costing Action Network

Information, education, and communication

International Health Economics Association

International Vaccine Institute

Immunization and Vaccines-related Implementation Research Advisory Committee

Low-income country

Middle-income country

Malaria Vaccine Immunization Costing Tool

Seasonal Influenza Immunization Costing Tool

Typhoid Conjugate Vaccine Costing Tool

Vaccine Technology Costs and Health Impact Assessment Tool

World Health Organization

Johns Hopkins University, International Vaccine Access Center. Methodology report: decade of vaccines economics (DOVE). Return on investment analysis. Medford: Immunization Economics; 2019. https://static1.squarespace.com/static/556deb8ee4b08a534b8360e7/t/5d56d54c6dae8d00014ef72d/1565971791774/DOVE-ROI+Methodology+Report+16AUG19.pdf . Accessed March 2020.

Google Scholar  

WHO. Immunization agenda 2030: a global strategy to leave no one behind; 2019.

Vaughan K, Ozaltin A, Moi F, Griffiths UK, Mallow M, Brenzel L. 2020 reporting gaps in immunization costing studies: recommendations for improving the practice. Vaccine X. 2020;5:100069.

Article   Google Scholar  

WHO 2018. Immunization and vaccine-related implementation research advisory committee (IVIR-AC): executive summary, 6-8 march 2018, vol. 93. Chamonix: WHO Weekly Epidemiological Record, No 24; 2018. p. 345–56.

WHO 2019. Immunization and Vaccine-related Implementation Research Advisory Committee (IVIR-AC) recommendations. WHO Wkly Epidemiol Record No 24. 2019;94:225–32.

Immunization Costing Action Network (ICAN). Immunization delivery costs in low- and middle-income countries: a methodology note for the systematic review, cost catalogue, and analytics. Washington, DC: ThinkWell; 2018.

WHO, 2020. Immunization and Vaccine-related Implementation Research Advisory Committee (IVIR-AC): summary and recommendations, September 2019. WHO Wkly Epidemiol Record No 49. 2020;95:609–28.

Vassall A, Sweeney S, Kahn J, Gomez G, Bollinger L, Marseille E, et al. Reference case for estimating the costs of Global Health services and interventions; 2017.

Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. BMJ. 2013;346:f1049. https://doi.org/10.1136/bmj.f1049 .

Article   PubMed   Google Scholar  

ImmunizationEconomics.org. 2021. EPIC projects. The Bill & Melinda Gates Foundation. https://immunizationeconomics.org/epic . Accessed 24 Jan 2022.

Brenzel L. 2013. Common approach for the costing and financing analyses of routine immunization and new vaccine introduction costs. Working Paper. Bill & Melinda Gates Foundation.

Harvard T. H. Chan School of Public Health. 2019. How to cost immunization programs: a practical guide on primary data collection and analysis.

International Vaccine Institute and WHO, Choltool: Planning and Costing User’s Guide. Seoul, South Korea.

WHO. Guidelines for estimating costs of introducing new vaccines into the national immunization system. Geneva: Switzerland; 2002.

WHO, WHO Cervical Cancer Prevention and Control Costing Tool (C4P) User’s Guide. Geneva, Switzerland. https://cdn.who.int/media/docs/default-source/immunization/hpv/who-cervical-cancer-prevention-and-control-costing.pdf?sfvrsn=b56c33cb_7 . Accessed 1 Feb 2022.

World Health Organization. Immunization Costing & Financing: a tool and user guide for comprehensive multi-year planning (cMYP). Geneva: WHO; 2006.

WHO, WHO Seasonal Influenza Immunization Costing Tool: Planning and Costing User’s Guide. Geneva, Switzerland.

ImmunizationEconomics.org. 2021. Community of practice. The Bill & Melinda Gates Foundation. https://immunizationeconomics.org/community . Accessed 24 Jan 2022.

Pallas S, Ahmeti A, Morgan W, Preza I, Nelaj E, Ebama M, et al. Program cost analysis of influenza vaccination of health care workers in Albania. Vaccine. 2020;38(2):220–7. https://doi.org/10.1016/j.vaccine.2019.10.027.Epub 2019 Oct 25.

Download references

Acknowledgements

We thank Xiao Xian Huang and Stéphane Verguet, two Working Group members, for their contributions to and review of the consensus statement. We also thank Shuoning Huang for her coordination and contribution to the first draft of the consensus statement. We acknowledge comments received on earlier versions of the consensus statement from Taiwo Abimbola, Anna Hidle, Timothy Brennan, Nelly Mejia, and Carlo Davila Payan of the CDC.

The views in this manuscript are those of the authors in their individual capacities and do not represent the official positions of the authors’ organizations.

The WHO provided funding for the consultant that led the working group.

Author information

Laura Boonstoppel, Logan Brenzel, Ulla Griffiths, Raymond Hutubessy, Mark Jit, Vittal Mogasale, Sarah Pallas, Stephen Resch, Christian Suharlim, and Karene Hoi Ting Yeung contributed equally and are listed in alphabetical order.

Authors and Affiliations

Levin & Morgan LLC, Bethesda, USA

ThinkWell, Geneva, Switzerland

Laura Boonstoppel

Bill & Melinda Gates Foundation, Seattle, USA

Logan Brenzel

UNICEF, New York, USA

Ulla Griffiths

Department of Immunization, Vaccines and Biologicals, World Health Organization, Geneva, Switzerland

Raymond Hutubessy & Karene Hoi Ting Yeung

London School of Hygiene & Tropical Medicine, London, UK

International Vaccine Institute, Seoul, South Korea

Vittal Mogasale

Centers for Disease Control and Prevention, Atlanta, USA

Sarah Pallas

Harvard T.H. Chan School of Public Health, Boston, USA

Stephen Resch & Christian Suharlim

You can also search for this author in PubMed   Google Scholar

Contributions

AL reviewed the guidance documents and costing tools and was the lead author. All authors contributed to the data interpretation and critical revision of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Raymond Hutubessy .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. .

Consensus Statement on Vaccine Delivery Costs which includes a review of vaccine delivery cost guidance documents and costing tools as well as a consensus statement on the terminology and methodological principles to be used for vaccine delivery costing. It includes two figures and 6 tables. Figure S1. Major current workstreams in vaccine delivery costing identified by working group. Figure A1 – Timeline for developing a Consensus Statement on Vaccine Delivery Costs. Table A2a. List of guidelines by publication year, target interventions, and purposes. Table A2b. List of costing tools for vaccine delivery or immunization program. Table A3. Definitions of costing terms in guidance documents. Table A4. Comparison of costing principles among guidance. Table A5. Characteristics of costing workstreams. Table 6A. Data sources, sampling and characterization of uncertainty, and terminology by workstreams.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Levin, A., Boonstoppel, L., Brenzel, L. et al. WHO-led consensus statement on vaccine delivery costing: process, methods, and findings. BMC Med 20 , 88 (2022). https://doi.org/10.1186/s12916-022-02278-4

Download citation

Received : 18 October 2021

Accepted : 27 January 2022

Published : 08 March 2022

DOI : https://doi.org/10.1186/s12916-022-02278-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Delivery cost
• Immunization

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

ORIGINAL RESEARCH article

Challenges in facilitators' professional development -complexities within a scaling up process as reasons for dropout provisionally accepted.

• 1 Ruhr University Bochum, Germany
• 2 University of Duisburg-Essen, Germany

The final, formatted version of the article will be published soon.

In-service courses for teachers are often provided by so called facilitators who function as teacher trainers and whose tasks in the education system are multifaceted. The professional development (PD) of facilitators is of great importance, and the complex process of scaling up might lead to unexpected constraints, and influence the effectiveness of a program. In general, more research is needed concerning facilitators’ PD with respect to their different roles and functions. In this paper, a project will be presented that focused on qualifying facilitators for the topic ‘inclusive mathematics’ and accompanied the process of scaling up. In the end, unexpectedly, five out of 15 facilitators did not finish the program. Therefore, the challenges and concrete reasons for dropout were investigated in detail. The paper will present a case analysis, where Facilitators’ multifaceted roles, functions, and tasks emerged as the central category. Moreover, conclusions for professionalization programs and research will be derived and discussed.

Keywords: Facilitators, professional development (PD), scaling up, inclusive mathematics, Reasons for dropout

Received: 20 Oct 2023; Accepted: 09 Apr 2024.

Copyright: © 2024 Rolka and Scherer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Prof. Katrin Rolka, Ruhr University Bochum, Bochum, Germany

People also looked at

Solar Eclipse 2024: What to Know as the Eclipse Passes Over

T he first total solar eclipse to pass over the U.S. since 2017 took place Monday, as the moon scooted between Earth and the sun, casting a shadow that plunged parts of North America into darkness. In a swath of the continent known as the path of totality—where the sun was totally covered—temperatures were expected to drop, and some animals were expected to go gaga as day turned to night.

“The first time I saw one was life-changing and mind-blowing,” said C. Alex Young, a National Aeronautics and Space Administration solar astrophysicist. “I feel like there’s this window that opens up that allows me to see our star in a way that normally we can’t experience with our own eyes.”

What time is the eclipse?

The eclipse began in the U.S. just before 12:30 p.m. local time in Texas, ending in northeastern Maine around 4:40 p.m. local time.

In Austin, Texas, the total eclipse started at 1:36 p.m. local time. Indianapolis was in total darkness beginning at 3:06 p.m. local time, while Buffalo, N.Y., completely lost sight of the sun at 3:18 p.m. local time.

Astronomers had suggested checking Timeanddate.com to see when the total eclipse began and ended in a specific area.

Where is the path of totality?

It changes from eclipse to eclipse, but this time, the path was a roughly 115-mile-wide band stretching from central Mexico to Newfoundland, Canada—passing through more than a dozen U.S. states in between.

Austin, Dallas, Indianapolis, Cleveland, Buffalo and Rochester, N.Y., were among the cities in the path of totality. Most of the rest of the continent saw a partial eclipse.

What happens during a total solar eclipse?

In the path of totality, the celestial spectacle starts as a partial eclipse, as the moon slowly obscures more of the sun over a period of time.

Just before totality, points of light appear around the edges of the moon. Scientists call these Baily’s beads, after one of the first eclipse chasers. They are the result of sunlight moving through valleys on the lunar surface and quickly disappear, leaving one final bright spot resembling a diamond on a ring. Once that disappears, the sun takes on the appearance of a black disc.

“It’s not like a sunset where it’s bright in one direction and dark in another direction,” said Jay Anderson, a Canadian meteorologist and eclipse chaser. “For the eclipse, you’re in the middle of a big shadow.”

The moon on Monday completely blocked the sun for as long as 4½ minutes. Then the process reversed, with the sun slowly re-emerging.

How can you watch a solar eclipse safely?

Skywatchers needed to use eclipse glasses, which consist of solar filters that block out light from the sun, during the event. The only time it’s safe to look directly at the sun is during totality, when the moon completely blocks the bright orb. Even 1% of the sun’s surface is 10,000 times brighter than a full moon and dangerous to view without the right equipment.

Gazing at the sun without protection can cause what is known as eclipse blindness, or retinal burns, when nerve tissue at the back of the eye is damaged. The retina has no pain receptors, so viewers are unaware when this damage occurs.

What causes a total solar eclipse?

The sun is 400 times the moon’s diameter, yet about 400 times farther away, resulting in the two appearing nearly the same size in the sky. So when the moon passes in just the right spot between Earth and the sun, the star is blocked from view.

How often do total solar eclipses occur?

These events happen somewhere on the planet every year or two. The geometry of the Earth’s orbit, moon’s orbit and their relative positions in relationship to the sun make it so that, on average, the same spot on Earth only experiences a total solar eclipse once every 375 years.

Some parts of the U.S. are lucky. Small areas in Missouri, Illinois and Kentucky experienced totality in 2017 and got to experience it again in 2024.

How long do total eclipses last?

There is no standard length for a total eclipse, though there is always one place where totality lasts the longest. For Monday’s eclipse, the maximum duration of totality was four minutes, 28 seconds near Torreón, Mexico. The longest total eclipses on record have exceeded seven minutes.

“The average length is about 3½ minutes,” Anderson said. “The shortest I’ve seen is 18 seconds.”

The length depends on how close the moon is to Earth, and how far the sun is from Earth on eclipse day. The farther away the sun, the smaller it appears in the sky, and the more easily it can be covered by the moon. The closer the moon, the bigger it appears, and the longer its disc will cover the sun.

The nearness of the moon also affects the shadow it projects onto the planet. The path of totality for this eclipse was wider than the one from 2017, which was roughly 70 miles wide.

How can cloud cover affect your eclipse experience?

Clouds can conceal a total eclipse but don’t completely thwart the experience. Even if the sight of the moon fully blocking the sun is obscured, the temperature drop, changing winds, impacts on animals and darkening that accompany totality are noticeable.

“There was still a visceral experience about it, there was still a sensory experience that’s different than anything you would experience in your lifetime,” according to Young, who said he has experienced a couple of total eclipses that were clouded out.

Historical data about typical cloud coverage this time of year in North America suggested Mexico and the southern U.S. were the most likely spots along the path of totality to be cloud-free. But the latest forecasts had shown Texas and many other parts of the path blanketed in clouds.

Cloud cover can change over the course of the day and during the eclipse itself, according to Patricia Reiff, a professor of physics and astronomy at Rice University in Houston.

“Some clouds begin to thin, and even those thin clouds become more transparent as totality approaches, because there’s not as much sunlight scattered inside them,” Reiff said, adding that she has experienced eclipses in which the sky opened up just for totality before the clouds closed in again.

What science can be done during a total solar eclipse?

Researchers can use these events to improve their understanding of difficult-to-study parts of the sun’s upper atmosphere, known as the corona. The superhot corona is so faint compared with the sun’s surface that it can’t be seen from Earth unless the light from the sun is totally blocked. While scientists can build instruments that mimic solar eclipses, they don’t measure up to the real thing, experts say.

The National Center for Atmospheric Research planned to use a jet to follow the path of totality and observe infrared light from the corona. The corona is a source of solar wind—a stream of charged particles spewed into the solar system that can affect our navigation and communication systems, satellites and power grids by causing space weather above our planet. By better understanding the corona, scientists hope to improve our ability to predict dangerous space weather, said Paul Bryans, a project scientist at the center.

NASA also planned to launch small rockets to study how an electrically charged part of Earth’s atmosphere known as the ionosphere changes in the sun’s absence during an eclipse. Radio and GPS signals from satellites and ground-based systems travel through the ionosphere, so changes there could affect technology.

What happens to animals during an eclipse?

Anecdotal evidence suggests that, as the moon slowly blocks out the sun, most animals tend to switch to their nighttime routines, but the study of animal behavior during these events has been limited, according to Adam Hartstone-Rose, a professor of biological sciences at North Carolina State University.

During the 2017 eclipse, Hartstone-Rose said he and his colleagues watched more than a dozen species at the Riverbanks Zoo in Columbia, S.C., during totality. The group noticed lorikeets flocked together and flew to their nighttime roost, giraffes started galloping in their enclosure, a sedentary Komodo dragon started running around its cage, and Galapagos tortoises began mating.

“Obviously, we can’t know exactly what animals are thinking,” he said, “but many species had a reaction that we think that we relate to anxiety.”

When is the next total solar eclipse?

Viewers in Spain and Iceland will experience the next totality on Aug. 12, 2026. Less than a year after that, skywatchers in parts of North Africa, Saudi Arabia and Spain will also glimpse a total solar eclipse.

The continental U.S.’s next totality experience won’t occur until 2044, when a total eclipse passes over Montana and North Dakota. A 2045 total eclipse will herald a longer path of totality, cutting from Florida to California.

According to scientists, eventually Earth will stop experiencing total eclipses. More than 50 years ago, Apollo astronauts left laser reflectors on the moon’s surface to help determine how far away it is from Earth. Observations show the moon is moving away at a rate of about 1.5 inches a year. As the moon recedes, how large it appears in the sky shrinks, so in about 600 million years the moon will be far enough away that it will appear too small to fully cover the sun, Bryans said.

How do you photograph a solar eclipse?

Believe it or not, a smartphone mounted to a tripod is sufficient to photograph totality. Don’t forget to turn off the autoflash, and consider doing a panorama shot.

To snap a close-up of the sun’s darkened disc, opt for a powerful telephoto lens.

Eclipse chasers and scientists recommend spending most of a total eclipse looking up or observing the world around you rather than taking photos.

“What I’ve sometimes done during totality is set up my camera, or my cellphone, on a tripod, turned it on video and just hit record and left it alone,” Young said. “My recommendation has always been to just put your camera down and enjoy.”

To photograph a partial eclipse before and after totality, be sure to purchase a solar filter for your lens. (Remove it during totality.) Never use a camera—or binoculars or a telescope—without a filter because these devices concentrate a lot of light into your eyes and cause injury, even if you have eclipse glasses on, according to NASA.

Other tips for seeing a total solar eclipse?

Totality plunges parts of the world into a darkness akin to a full moonlit night. Astronomers say you are able to see planets such as Venus and Jupiter.

To get a better experience, Young said to avoid city lights as you would if you were watching a meteor shower.

“If there’s a place that normally has really good stars, then that’s going to be a good place to see the eclipse,” Reiff said.

This explanatory article may be periodically updated.

Write to Aylin Woodward at [email protected]

WCG Launches Site Feasibility Assessment App

WCG has kicked off a new application on its ClinSphere platform designed to accelerate and transform the site feasibility assessment process.

The new app, Total Feasibility, speeds up development of site feasibility questionnaires, providing more accurate site matching, fewer redundancies and quicker site response times. The app also features real-time tracking and analyses of site responses to help inform trial expansion and protocol changes.

“By streamlining the questionnaires from which sponsors and CROs harness information from all users, we decrease the burden on sites in answering redundant questions,” says Cristin MacDonald, WCG’s vice president of client delivery. “This promotes better collaboration among sponsors, sites and investigators and provides a more efficient process for sponsors to select the best-matched sites for their studies.”

Learn more about Total Feasibility here .

Upcoming Events

Magi 2024: the clinical research conference, effective root cause analysis and capa investigations for drugs, devices and clinical trials, 2024 avoca quality consortium summit, featured products.

Surviving an FDA GCP Inspection: Resources for Investigators, Sponsors, CROs and IRBs

Best Practices for Clinical Trial Site Management

Featured stories.

Thought Leadership: Remote Patient Monitoring Gives New View of Safety in Cardiac Clinical Trials

Ask the Experts: Applying Quality by Design to Protocols

Clinical Trials Need Greater Representation of Obese Patients, Experts Say

FDA IT Modernization Plan Prioritizes Data-Sharing, AI, Collaboration and More

Standard operating procedures for risk-based monitoring of clinical trials, the information you need to adapt your monitoring plan to changing times..

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Med Libr Assoc
• v.106(4); 2018 Oct

A systematic approach to searching: an efficient and complete method to develop literature searches

Associated data.

Creating search strategies for systematic reviews, finding the best balance between sensitivity and specificity, and translating search strategies between databases is challenging. Several methods describe standards for systematic search strategies, but a consistent approach for creating an exhaustive search strategy has not yet been fully described in enough detail to be fully replicable. The authors have established a method that describes step by step the process of developing a systematic search strategy as needed in the systematic review. This method describes how single-line search strategies can be prepared in a text document by typing search syntax (such as field codes, parentheses, and Boolean operators) before copying and pasting search terms (keywords and free-text synonyms) that are found in the thesaurus. To help ensure term completeness, we developed a novel optimization technique that is mainly based on comparing the results retrieved by thesaurus terms with those retrieved by the free-text search words to identify potentially relevant candidate search terms. Macros in Microsoft Word have been developed to convert syntaxes between databases and interfaces almost automatically. This method helps information specialists in developing librarian-mediated searches for systematic reviews as well as medical and health care practitioners who are searching for evidence to answer clinical questions. The described method can be used to create complex and comprehensive search strategies for different databases and interfaces, such as those that are needed when searching for relevant references for systematic reviews, and will assist both information specialists and practitioners when they are searching the biomedical literature.

INTRODUCTION

Librarians and information specialists are often involved in the process of preparing and completing systematic reviews (SRs), where one of their main tasks is to identify relevant references to include in the review [ 1 ]. Although several recommendations for the process of searching have been published [ 2 – 6 ], none describe the development of a systematic search strategy from start to finish.

Traditional methods of SR search strategy development and execution are highly time consuming, reportedly requiring up to 100 hours or more [ 7 , 8 ]. The authors wanted to develop systematic and exhaustive search strategies more efficiently, while preserving the high sensitivity that SR search strategies necessitate. In this article, we describe the method developed at Erasmus University Medical Center (MC) and demonstrate its use through an example search. The efficiency of the search method and outcome of 73 searches that have resulted in published reviews are described in a separate article [ 9 ].

As we aimed to describe the creation of systematic searches in full detail, the method starts at a basic level with the analysis of the research question and the creation of search terms. Readers who are new to SR searching are advised to follow all steps described. More experienced searchers can consider the basic steps to be existing knowledge that will already be part of their normal workflow, although step 4 probably differs from general practice. Experienced searchers will gain the most from reading about the novelties in the method as described in steps 10–13 and comparing the examples given in the supplementary appendix to their own practice.

CREATING A SYSTEMATIC SEARCH STRATEGY

Our methodology for planning and creating a multi-database search strategy consists of the following steps:

• Determine a clear and focused question
• Describe the articles that can answer the question
• Decide which key concepts address the different elements of the question
• Decide which elements should be used for the best results
• Choose an appropriate database and interface to start with
• Document the search process in a text document
• Identify appropriate index terms in the thesaurus of the first database
• Identify synonyms in the thesaurus
• Add variations in search terms
• Use database-appropriate syntax, with parentheses, Boolean operators, and field codes
• Optimize the search
• Evaluate the initial results
• Check for errors
• Translate to other databases
• Test and reiterate

Each step in the process is reflected by an example search described in the supplementary appendix .

1. Determine a clear and focused question

A systematic search can best be applied to a well-defined and precise research or clinical question. Questions that are too broad or too vague cannot be answered easily in a systematic way and will generally result in an overwhelming number of search results. On the other hand, a question that is too specific will result into too few or even zero search results. Various papers describe this process in more detail [ 10 – 12 ].

2. Describe the articles that can answer the question

Although not all clinical or research questions can be answered in the literature, the next step is to presume that the answer can indeed be found in published studies. A good starting point for a search is hypothesizing what the research that can answer the question would look like. These hypothetical (when possible, combined with known) articles can be used as guidance for constructing the search strategy.

3. Decide which key concepts address the different elements of the question

Key concepts are the topics or components that the desired articles should address, such as diseases or conditions, actions, substances, settings, domains (e.g., therapy, diagnosis, etiology), or study types. Key concepts from the research question can be grouped to create elements in the search strategy.

Elements in a search strategy do not necessarily follow the patient, intervention, comparison, outcome (PICO) structure or any other related structure. Using the PICO or another similar framework as guidance can be helpful to consider, especially in the inclusion and exclusion review stage of the SR, but this is not necessary for good search strategy development [ 13 – 15 ]. Sometimes concepts from different parts of the PICO structure can be grouped together into one search element, such as when the desired outcome is frequently described in a certain study type.

4. Decide which elements should be used for the best results

Not all elements of a research question should necessarily be used in the search strategy. Some elements are less important than others or may unnecessarily complicate or restrict a search strategy. Adding an element to a search strategy increases the chance of missing relevant references. Therefore, the number of elements in a search strategy should remain as low as possible to optimize recall.

Using the schema in Figure 1 , elements can be ordered by their specificity and importance to determine the best search approach. Whether an element is more specific or more general can be measured objectively by the number of hits retrieved in a database when searching for a key term representing that element. Depending on the research question, certain elements are more important than others. If articles (hypothetically or known) exist that can answer the question but lack a certain element in their titles, abstracts, or keywords, that element is unimportant to the question. An element can also be unimportant because of expected bias or an overlap with another element.

Schema for determining the optimal order of elements

Bias in elements

The choice of elements in a search strategy can introduce bias through use of overly specific terminology or terms often associated with positive outcomes. For the question “does prolonged breastfeeding improve intelligence outcomes in children?,” searching specifically for the element of duration will introduce bias, as articles that find a positive effect of prolonged breastfeeding will be much more likely to mention time factors in their titles or abstracts.

Overlapping elements

Elements in a question sometimes overlap in their meaning. Sometimes certain therapies are interventions for one specific disease. The Lichtenstein technique, for example, is a repair method for inguinal hernias. There is no need to include an element of “inguinal hernias” to a search for the effectiveness of the Lichtenstein therapy. Likewise, sometimes certain diseases are only found in certain populations. Adding such an overlapping element could lead to missing relevant references.

The elements to use in a search strategy can be found in the plot of elements in Figure 1 , by following the top row from left to right. For this method, we recommend starting with the most important and specific elements. Then, continue with more general and important elements until the number of results is acceptable for screening. Determining how many results are acceptable for screening is often a matter of negotiation with the SR team.

5. Choose an appropriate database and interface to start with

Important factors for choosing databases to use are the coverage and the presence of a thesaurus. For medically oriented searches, the coverage and recall of Embase, which includes the MEDLINE database, are superior to those of MEDLINE [ 16 ]. Each of these two databases has its own thesaurus with its own unique definitions and structure. Because of the complexity of the Embase thesaurus, Emtree, which contains much more specific thesaurus terms than the MEDLINE Medical Subject Headings (MeSH) thesaurus, translation from Emtree to MeSH is easier than the other way around. Therefore, we recommend starting in Embase.

MEDLINE and Embase are available through many different vendors and interfaces. The choice of an interface and primary database is often determined by the searcher’s accessibility. For our method, an interface that allows searching with proximity operators is desirable, and full functionality of the thesaurus, including explosion of narrower terms, is crucial. We recommend developing a personal workflow that always starts with one specific database and interface.

6. Document the search process in a text document

We advise designing and creating the complete search strategies in a log document, instead of directly in the database itself, to register the steps taken and to make searches accountable and reproducible. The developed search strategies can be copied and pasted into the desired databases from the log document. This way, the searcher is in control of the whole process. Any change to the search strategy should be done in the log document, assuring that the search strategy in the log is always the most recent.

7. Identify appropriate index terms in the thesaurus of the first database

Searches should start by identifying appropriate thesaurus terms for the desired elements. The thesaurus of the database is searched for matching index terms for each key concept. We advise restricting the initial terms to the most important and most relevant terms. Later in the process, more general terms can be added in the optimization process, in which the effect on the number of hits, and thus the desirability of adding these terms, can be evaluated more easily.

Several factors can complicate the identification of thesaurus terms. Sometimes, one thesaurus term is found that exactly describes a specific element. In contrast, especially in more general elements, multiple thesaurus terms can be found to describe one element. If no relevant thesaurus terms have been found for an element, free-text terms can be used, and possible thesaurus terms found in the resulting references can be added later (step 11).

Sometimes, no distinct thesaurus term is available for a specific key concept that describes the concept in enough detail. In Emtree, one thesaurus term often combines two or more elements. The easiest solution for combining these terms for a sensitive search is to use such a thesaurus term in all elements where it is relevant. Examples are given in the supplementary appendix .

8. Identify synonyms in the thesaurus

Most thesauri offer a list of synonyms on their term details page (named Synonyms in Emtree and Entry Terms in MeSH). To create a sensitive search strategy for SRs, these terms need to be searched as free-text keywords in the title and abstract fields, in addition to searching their associated thesaurus terms.

The Emtree thesaurus contains more synonyms (300,000) than MeSH does (220,000) [ 17 ]. The difference in number of terms is even higher considering that many synonyms in MeSH are permuted terms (i.e., inversions of phrases using commas).

Thesaurus terms are ordered in a tree structure. When searching for a more general thesaurus term, the more specific (narrower) terms in the branches below that term will also be searched (this is frequently referred to as “exploding” a thesaurus term). However, to perform a sensitive search, all relevant variations of the narrower terms must be searched as free-text keywords in the title or abstract, in addition to relying on the exploded thesaurus term. Thus, all articles that describe a certain narrower topic in their titles and abstracts will already be retrieved before MeSH terms are added.

9. Add variations in search terms (e.g., truncation, spelling differences, abbreviations, opposites)

Truncation allows a searcher to search for words beginning with the same word stem. A search for therap* will, thus, retrieve therapy, therapies, therapeutic, and all other words starting with “therap.” Do not truncate a word stem that is too short. Also, limitations of interfaces should be taken into account, especially in PubMed, where the number of search term variations that can be found by truncation is limited to 600.

Databases contain references to articles using both standard British and American English spellings. Both need to be searched as free-text terms in the title and abstract. Alternatively, many interfaces offer a certain code to replace zero or one characters, allowing a search for “pediatric” or “paediatric” as “p?ediatric.” Table 1 provides a detailed description of the syntax for different interfaces.

Field codes in five most used interfaces for biomedical literature searching

Searching for abbreviations can identify extra, relevant references and retrieve more irrelevant ones. The search can be more focused by combining the abbreviation with an important word that is relevant to its meaning or by using the Boolean “NOT” to exclude frequently observed, clearly irrelevant results. We advise that searchers do not exclude all possible irrelevant meanings, as it is very time consuming to identify all the variations, it will result in unnecessarily complicated search strategies, and it may lead to erroneously narrowing the search and, thereby, reduce recall.

Searching partial abbreviations can be useful for retrieving relevant references. For example, it is very likely that an article would mention osteoarthritis (OA) early in the abstract, replacing all further occurrences of osteoarthritis with OA . Therefore, it may not contain the phrase “hip osteoarthritis” but only “hip oa.”

It is also important to search for the opposites of search terms to avoid bias. When searching for “disease recurrence,” articles about “disease free” may be relevant as well. When the desired outcome is survival , articles about mortality may be relevant.

10. Use database-appropriate syntax, with parentheses, Boolean operators, and field codes

Different interfaces require different syntaxes, the special set of rules and symbols unique to each database that define how a correctly constructed search operates. Common syntax components include the use of parentheses and Boolean operators such as “AND,” “OR,” and “NOT,” which are available in all major interfaces. An overview of different syntaxes for four major interfaces for bibliographic medical databases (PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest) is shown in Table 1 .

Creating the appropriate syntax for each database, in combination with the selected terms as described in steps 7–9, can be challenging. Following the method outlined below simplifies the process:

• Create single-line queries in a text document (not combining multiple record sets), which allows immediate checking of the relevance of retrieved references and efficient optimization.
• Type the syntax (Boolean operators, parentheses, and field codes) before adding terms, which reduces the chance that errors are made in the syntax, especially in the number of parentheses.
• Use predefined proximity structures including parentheses, such as (() ADJ3 ()) in Ovid, that can be reused in the query when necessary.
• Use thesaurus terms separately from free-text terms of each element. Start an element with all thesaurus terms (using “OR”) and follow with the free-text terms. This allows the unique optimization methods as described in step 11.
• When adding terms to an existing search strategy, pay close attention to the position of the cursor. Make sure to place it appropriately either in the thesaurus terms section, in the title/abstract section, or as an addition (broadening) to an existing proximity search.

The supplementary appendix explains the method of building a query in more detail, step by step for different interfaces: PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest. This method results in a basic search strategy designed to retrieve some relevant references upon which a more thorough search strategy can be built with optimization such as described in step 11.

11. Optimize the search

The most important question when performing a systematic search is whether all (or most) potentially relevant articles have been retrieved by the search strategy. This is also the most difficult question to answer, since it is unknown which and how many articles are relevant. It is, therefore, wise first to broaden the initial search strategy, making the search more sensitive, and then check if new relevant articles are found by comparing the set results (i.e., search for Strategy #2 NOT Strategy #1 to see the unique results).

A search strategy should be tested for completeness. Therefore, it is necessary to identify extra, possibly relevant search terms and add them to the test search in an OR relationship with the already used search terms. A good place to start, and a well-known strategy, is scanning the top retrieved articles when sorted by relevance, looking for additional relevant synonyms that could be added to the search strategy.

We have developed a unique optimization method that has not been described before in the literature. This method often adds valuable extra terms to our search strategy and, therefore, extra, relevant references to our search results. Extra synonyms can be found in articles that have been assigned a certain set of thesaurus terms but that lack synonyms in the title and/or abstract that are already present in the current search strategy. Searching for thesaurus terms NOT free-text terms will help identify missed free-text terms in the title or abstract. Searching for free-text terms NOT thesaurus terms will help identify missed thesaurus terms. If this is done repeatedly for each element, leaving the rest of the query unchanged, this method will help add numerous relevant terms to the query. These steps are explained in detail for five different search platforms in the supplementary appendix .

12. Evaluate the initial results

The results should now contain relevant references. If the interface allows relevance ranking, use that in the evaluation. If you know some relevant references that should be included in the research, search for those references specifically; for example, combine a specific (first) author name with a page number and the publication year. Check whether those references are retrieved by the search. If the known relevant references are not retrieved by the search, adapt the search so that they are. If it is unclear which element should be adapted to retrieve a certain article, combine that article with each element separately.

Different outcomes are desired for different types of research questions. For instance, in the case of clinical question answering, the researcher will not be satisfied with many references that contain a lot of irrelevant references. A clinical search should be rather specific and is allowed to miss a relevant reference. In the case of an SR, the researchers do not want to miss any relevant reference and are willing to handle many irrelevant references to do so. The search for references to include in an SR should be very sensitive: no included reference should be missed. A search that is too specific or too sensitive for the intended goal can be adapted to become more sensitive or specific. Steps to increase sensitivity or specificity of a search strategy can be found in the supplementary appendix .

13. Check for errors

Errors might not be easily detected. Sometimes clues can be found in the number of results, either when the number of results is much higher or lower than expected or when many retrieved references are not relevant. However, the number expected is often unknown, and very sensitive search strategies will always retrieve many irrelevant articles. Each query should, therefore, be checked for errors.

One of the most frequently occurring errors is missing the Boolean operator “OR.” When no “OR” is added between two search terms, many interfaces automatically add an “AND,” which unintentionally reduces the number of results and likely misses relevant references. One good strategy to identify missing “OR”s is to go to the web page containing the full search strategy, as translated by the database, and using Ctrl-F search for “AND.” Check whether the occurrences of the “AND” operator are deliberate.

Ideally, search strategies should be checked by other information specialists [ 18 ]. The Peer Review of Electronic Search Strategies (PRESS) checklist offers good guidance for this process [ 4 ]. Apart from the syntax (especially Boolean operators and field codes) of the search strategy, it is wise to have the search terms checked by the clinician or researcher familiar with the topic. At Erasmus MC, researchers and clinicians are involved during the complete process of structuring and optimizing the search strategy. Each word is added after the combined decision of the searcher and the researcher, with the possibility of directly comparing results with and without the new term.

14. Translate to other databases

To retrieve as many relevant references as possible, one has to search multiple databases. Translation of complex and exhaustive queries between different databases can be very time consuming and cumbersome. The single-line search strategy approach detailed above allows quick translations using the find and replace method in Microsoft Word (<Ctrl-H>).

At Erasmus MC, macros based on the find-and-replace method in Microsoft Word have been developed for easy and fast translation between the most used databases for biomedical and health sciences questions. The schema that is followed for the translation between databases is shown in Figure 2 . Most databases simply follow the structure set by the Embase.com search strategy. The translation from Emtree terms to MeSH terms for MEDLINE in Ovid often identifies new terms that need to be added to the Embase.com search strategy before the translation to other databases.

Schematic representation of translation between databases used at Erasmus University Medical Center

Dotted lines represent databases that are used in less than 80% of the searches.

Using five different macros, a thoroughly optimized query in Embase.com can be relatively quickly translated into eight major databases. Basic search strategies will be created to use in many, mostly smaller, databases, because such niche databases often do not have extensive thesauri or advanced syntax options. Also, there is not much need to use extensive syntax because the number of hits and, therefore, the amount of noise in these databases is generally low. In MEDLINE (Ovid), PsycINFO (Ovid), and CINAHL (EBSCOhost), the thesaurus terms must be adapted manually, as each database has its own custom thesaurus. These macros and instructions for their installation, use, and adaptation are available at bit.ly/databasemacros.

15. Test and reiterate

Ideally, exhaustive search strategies should retrieve all references that are covered in a specific database. For SR search strategies, checking searches for their recall is advised. This can be done after included references have been determined by the authors of the systematic review. If additional papers have been identified through other non-database methods (i.e., checking references in included studies), results that were not identified by the database searches should be examined. If these results were available in the databases but not located by the search strategy, the search strategy should be adapted to try to retrieve these results, as they may contain terms that were omitted in the original search strategies. This may enable the identification of additional relevant results.

A methodology for creating exhaustive search strategies has been created that describes all steps of the search process, starting with a question and resulting in thorough search strategies in multiple databases. Many of the steps described are not new, but together, they form a strong method creating high-quality, robust searches in a relatively short time frame.

Our methodology is intended to create thoroughness for literature searches. The optimization method, as described in step 11, will identify missed synonyms or thesaurus terms, unlike any other method that largely depends on predetermined keywords and synonyms. Using this method results in a much quicker search process, compared to traditional methods, especially because of the easier translation between databases and interfaces (step 13). The method is not a guarantee for speed, since speed depends on many factors, including experience. However, by following the steps and using the tools as described above, searchers can gain confidence first and increase speed through practice.

What is new?

This method encourages searchers to start their search development process using empty syntax first and later adding the thesaurus terms and free-text synonyms. We feel this helps the searcher to focus on the search terms, instead of on the structure of the search query. The optimization method in which new terms are found in the already retrieved articles is used in some other institutes as well but has to our knowledge not been described in the literature. The macros to translate search strategies between interfaces are unique in this method.

What is different compared to common practice?

Traditionally, librarians and information specialists have focused on creating complex, multi-line (also called line-by-line) search strategies, consisting of multiple record sets, and this method is frequently advised in the literature and handbooks [ 2 , 19 – 21 ]. Our method, instead, uses single-line searches, which is critical to its success. Single-line search strategies can be easily adapted by adding or dropping a term without having to recode numbers of record sets, which would be necessary in multi-line searches. They can easily be saved in a text document and repeated by copying and pasting for search updates. Single-line search strategies also allow easy translation to other syntaxes using find-and-replace technology to update field codes and other syntax elements or using macros (step 13).

When constructing a search strategy, the searcher might experience that certain parentheses in the syntax are unnecessary, such as parentheses around all search terms in the title/abstract portion, if there is only one such term, there are double parentheses in the proximity statement, or one of the word groups exists for only one word. One might be tempted to omit those parentheses for ease of reading and management. However, during the optimization process, the searcher is likely to find extra synonyms that might consist of one word. To add those terms to the first query (with reduced parentheses) requires adding extra parentheses (meticulously placing and counting them), whereas, in the latter search, it only requires proper placement of those terms.

Many search methods highly depend on the PICO framework. Research states that often PICO or PICOS is not suitable for every question [ 22 , 23 ]. There are other acronyms than PICO—such as sample, phenomenon of interest, design, evaluation, research type (SPIDER) [ 24 ]—but each is just a variant. In our method, the most important and specific elements of a question are being analyzed for building the best search strategy.

Though it is generally recommended that searchers search both MEDLINE and Embase, most use MEDLINE as the starting point. It is considered the gold standard for biomedical searching, partially due to historical reasons, since it was the first of its kind, and more so now that it is freely available via the PubMed interface. Our method can be used with any database as a starting point, but we use Embase instead of MEDLINE or another database for a number of reasons. First, Embase provides both unique content and the complete content of MEDLINE. Therefore, searching Embase will be, by definition, more complete than searching MEDLINE only. Second, the number of terms in Emtree (the Embase thesaurus) is three times as high as that of MeSH (the MEDLINE thesaurus). It is easier to find MeSH terms after all relevant Emtree terms have been identified than to start with MeSH and translate to Emtree.

At Erasmus MC, the researchers sit next to the information specialist during most of the search strategy design process. This way, the researchers can deliver immediate feedback on the relevance of proposed search terms and retrieved references. The search team then combines knowledge about databases with knowledge about the research topic, which is an important condition to create the highest quality searches.

Limitations of the method

One disadvantage of single-line searches compared to multi-line search strategies is that errors are harder to recognize. However, with the methods for optimization as described (step 11), errors are recognized easily because missed synonyms and spelling errors will be identified during the process. Also problematic is that more parentheses are needed, making it more difficult for the searcher and others to assess the logic of the search strategy. However, as parentheses and field codes are typed before the search terms are added (step 10), errors in parentheses can be prevented.

Our methodology works best if used in an interface that allows proximity searching. It is recommended that searchers with access to an interface with proximity searching capabilities select one of those as the initial database to develop and optimize the search strategy. Because the PubMed interface does not allow proximity searches, phrases or Boolean “AND” combinations are required. Phrase searching complicates the process and is more specific, with the higher risk of missing relevant articles, and using Boolean “AND” combinations increases sensitivity but at an often high loss of specificity. Due to some searchers’ lack of access to expensive databases or interfaces, the freely available PubMed interface may be necessary to use, though it should never be the sole database used for an SR [ 2 , 16 , 25 ]. A limitation of our method is that it works best with subscription-based and licensed resources.

Another limitation is the customization of the macros to a specific institution’s resources. The macros for the translation between different database interfaces only work between the interfaces as described. To mitigate this, we recommend using the find-and-replace functionality of text editors like Microsoft Word to ease the translation of syntaxes between other databases. Depending on one’s institutional resources, custom macros can be developed using similar methods.

Results of the method

Whether this method results in exhaustive searches where no important article is missed is difficult to determine, because the number of relevant articles is unknown for any topic. A comparison of several parameters of 73 published reviews that were based on a search developed with this method to 258 reviews that acknowledged information specialists from other Dutch academic hospitals shows that the performance of the searches following our method is comparable to those performed in other institutes but that the time needed to develop the search strategies was much shorter than the time reported for the other reviews [ 9 ].

CONCLUSIONS

With the described method, searchers can gain confidence in their search strategies by finding many relevant words and creating exhaustive search strategies quickly. The approach can be used when performing SR searches or for other purposes such as answering clinical questions, with different expectations of the search’s precision and recall. This method, with practice, provides a stepwise approach that facilitates the search strategy development process from question clarification to final iteration and beyond.

SUPPLEMENTAL FILE

Acknowledgments.

We highly appreciate the work that was done by our former colleague Louis Volkers, who in his twenty years as an information specialist in Erasmus MC laid the basis for our method. We thank Professor Oscar Franco for reviewing earlier drafts of this article.