systematic literature review tutorial

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard
• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Mar 4, 2024 12:09 PM
• URL: https://lib.guides.umd.edu/SR

• Online Degree Explore Bachelor’s & Master’s degrees
• MasterTrack™ Earn credit towards a Master’s degree
• University Certificates Advance your career with graduate-level learning
• Top Courses
• Join for Free

Introduction to Systematic Review and Meta-Analysis

Taught in English

Some content may not be translated

Financial aid available

163,923 already enrolled

Gain insight into a topic and learn the fundamentals

Instructors: Tianjing Li, MD, MHS, PHD +1 more

Instructors

Instructor ratings.

We asked all learners to give feedback on our instructors based on the quality of their teaching style.

Included with Coursera Plus

(3,256 reviews)

Details to know

Add to your LinkedIn profile

See how employees at top companies are mastering in-demand skills

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 6 modules in this course

We will introduce methods to perform systematic reviews and meta-analysis of clinical trials. We will cover how to formulate an answerable research question, define inclusion and exclusion criteria, search for the evidence, extract data, assess the risk of bias in clinical trials, and perform a meta-analysis.

Upon successfully completing this course, participants will be able to: - Describe the steps in conducting a systematic review - Develop an answerable question using the “Participants Interventions Comparisons Outcomes” (PICO) framework - Describe the process used to collect and extract data from reports of clinical trials - Describe methods to critically assess the risk of bias of clinical trials - Describe and interpret the results of meta-analyses

Introduction

To get the ball rolling, we'll take a broad overview of what to expect in this course and then introduce you to the high-level concepts of systematic review and meta-analysis and take a look at who produces and uses systematic reviews.

What's included

5 videos 3 readings

5 videos • Total 40 minutes

• Welcome to Systematic Reviews and Meta-Analysis • 2 minutes • Preview module
• Lecture 1A : Introduction to Systematic Reviews • 15 minutes
• Lecture 1B: Introduction to Meta-Analysis • 12 minutes
• Lecture 1C: Producers and Users of Systematic Reviews • 8 minutes
• Hannah Rothstein, PhD • 1 minute

3 readings • Total 30 minutes

• Syllabus • 10 minutes
• Pre Course Survey • 10 minutes
• From the Field Videos • 10 minutes

Framing the Question

In this module, we will discuss how to frame a question, as well as scope, elements, and refining the question.

7 videos 1 reading 1 quiz 1 peer review

7 videos • Total 72 minutes

• Lecture 2A: Resources for How to Frame Your Question • 7 minutes • Preview module
• Lecture 2B: Deciding the Type and Scope of Your Question • 17 minutes
• Lecture 2C: Elements of the Question • 16 minutes
• Lecture 2D: Refining the Question • 10 minutes
• Lecture 2E: Some Examples • 8 minutes
• Lecture 2F: Analytic Frameworks • 10 minutes
• Betsy Becker, PhD • 2 minutes

1 reading • Total 10 minutes

• Welcome to the first Peer Assignment! • 10 minutes

1 quiz • Total 30 minutes

• Module 2 • 30 minutes

1 peer review • Total 60 minutes

• Peer Assessment 1 • 60 minutes

Searching Principles and Bias Assessment

In this module we will look at finding the evidence, as well as key sources, search strategy, and assessing the risk of bias.

11 videos 2 quizzes

11 videos • Total 118 minutes

• Searching Principles and Assessing Bias • 2 minutes • Preview module
• Lecture 3A: Finding the Evidence: Searching Principles • 5 minutes
• Lecture 3B: Identifying Key Sources and Techniques for Searching • 17 minutes
• Lecture 3C: Building a High-Quality Search Strategy • 17 minutes
• Lecture 3D: Documenting Your Search and Conclusions • 8 minutes
• Lecture 4A: Why Bias in the Individual Study is Important to a Systematic Review and Meta-Analysis • 7 minutes
• Lecture 4B: Selection Bias • 18 minutes
• Lecture 4C: Information Bias • 8 minutes
• Lecture 4D: Bias in the Analysis • 15 minutes
• Lecture 4E: Displaying Study "Quality" in Your Systematic Review • 16 minutes
• Byron Wallace, Phd • 2 minutes

2 quizzes • Total 60 minutes

• Module 3 • 30 minutes
• Module 4 • 30 minutes

Minimizing Metabias, Qualitative Synthesis, and Interpreting Results

In this module, we will cover minimizing metabias, selection bias, information bias, how to report transparently, qualitative synthesis, and interpreting results.

9 videos 2 quizzes

9 videos • Total 87 minutes

• Lecture 5A: Standards for Systematic Reviews • 6 minutes • Preview module
• Lecture 5B: Selection Bias • 22 minutes
• Lecture 5C: Information Bias • 12 minutes
• Lecture 5D: Bias in the Analysis • 15 minutes
• Lecture 5E: Reporting Transparently • 6 minutes
• Lecture 6A: Qualitative Synthesis and Interpreting Results Section A • 7 minutes
• Lecture 6B: What is Qualitative Synthesis • 10 minutes
• Lecture 6C: Some Examples • 5 minutes
• Christopher Schmid, PhD • 1 minute
• Module 5 • 30 minutes
• Module 6 • 30 minutes

Planning the Meta-Analysis and Statistical Methods

This module will cover the planning of your meat-analysis and the statistical methods for meta-analysis.

9 videos • Total 109 minutes

• Planning Meta-Analysis and Statistical Methods • 1 minute • Preview module
• Lecture 7A: Planning Your Meta-Analysis Section A • 7 minutes
• Lecture 7B: Introduction to Meta-Analysis • 15 minutes
• Lecture 7C: Why Do a Meta-Analysis? • 10 minutes
• Lecture 7D: Types of Data and Effect Measures • 19 minutes
• Lecture 8A: Fixed Effect Model • 20 minutes
• Lecture 8B: Random Effects Model • 14 minutes
• Lecture 8C: Random Effects Model • 17 minutes
• Michael Borenstein, PhD • 2 minutes
• Module 7 • 30 minutes
• Module 8 • 30 minutes

Wrap Up and Final Peer Review Assignment

In this final module, we'll wrap up with a look back at the key concepts covered over the past few weeks. Afterwards, you will submit your final Peer Review Assignment and evaluate some of your classmates' submissions.

2 videos 1 reading 1 peer review

2 videos • Total 12 minutes

• Lecture 9A: Wrap Up • 11 minutes • Preview module
• Closing Remarks • 1 minute
• Welcome to Your Final Peer Review Assignment • 10 minutes
• Peer Review Assignment 2 • 60 minutes

The mission of The Johns Hopkins University is to educate its students and cultivate their capacity for life-long learning, to foster independent and original research, and to bring the benefits of discovery to the world.

Recommended if you're interested in Research

Yale University

Understanding Medical Research: Your Facebook Friend is Wrong

University of Cape Town

Understanding Clinical Research: Behind the Statistics

Johns Hopkins University

Design and Interpretation of Clinical Trials

University of London

Understanding Research Methods

Why people choose coursera for their career.

Learner reviews

Showing 3 of 3256

3,256 reviews

Reviewed on Jan 15, 2021

This course has added valuable insights to my existing understanding of systematic reviews. After completing this course, I feel more confident in my pursuit of conducting a systematic review.

Reviewed on Aug 5, 2020

thank you very much for the opportunity to learn this subject. at least now, i will not be "left out" when my colleagues are discussing these things during critical appraisals of journals.

Reviewed on Aug 10, 2016

The course is a good introduction to the topic, however, it is a pity that there is no follow up to the course. At the end of the course you will not be able to perform a SLR + meta-analysis.

New to Research? Start here.

Open new doors with Coursera Plus

Unlimited access to 7,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Frequently asked questions

When will i have access to the lectures and assignments.

Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:

The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.

The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

What will I get if I purchase the Certificate?

When you purchase a Certificate you get access to all course materials, including graded assignments. Upon completing the course, your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile. If you only want to read and view the course content, you can audit the course for free.

What is the refund policy?

You will be eligible for a full refund until two weeks after your payment date, or (for courses that have just launched) until two weeks after the first session of the course begins, whichever is later. You cannot receive a refund once you’ve earned a Course Certificate, even if you complete the course within the two-week refund period. See our full refund policy Opens in a new tab .

Is financial aid available?

Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.

More questions

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

• 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
• 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
• 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
• PMID: 30089228
• DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

• Guidelines as Topic
• Meta-Analysis as Topic*
• Publication Bias
• Review Literature as Topic
• Systematic Reviews as Topic*

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

Systematic Review Tutorial

• Other Types of Evidence Synthesis
• Special Types of Systematic Reviews
• Systematic Review Tools
• 1.1 Develop a Research Question
• 2.1 Select Databases
• 2.2 Develop Terms
• 2.3 Subject Headings vs. Keywords
• 2.5 Test the Searches

Get Systematic Review Help

Schedule A Consultation

When to do a Systematic Review?

Syst ematic reviews are most useful

• when there is a large body of published literature pertaining to a specific question
• when a transparent search methodology and replicability are needed
• when multiple published studies point to contradictory or unc ertain results or outcomes

Systematic Reviews: Transparent, Rigorous and Replicable

Systematic Review (from the Cochrane Glossary ) : a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research , and to collect and analyze data from the studies that are included in the review.

Systematic Review vs Literature Review

Adapted from - Kysh, Lynn (2013): Difference between a systematic review and a literature review. [ MLGSCA] . Available at:  http://dx.doi.org/10.6084/m9.figshare.766364

• Next: Other Types of Evidence Synthesis >>
• Last Updated: Jul 24, 2023 7:54 AM
• URL: https://guides.library.uwm.edu/SystematicReview

• © 2020

How to Perform a Systematic Literature Review

A Guide for Healthcare Researchers, Practitioners and Students

• Edward Purssell   ORCID: https://orcid.org/0000-0003-3748-0864 0 ,
• Niall McCrae   ORCID: https://orcid.org/0000-0001-9776-7694 1

School of Health Sciences, City, University of London, London, UK

You can also search for this author in PubMed   Google Scholar

Florence Nightingale Faculty of Nursing Midwifery & Palliative Care, King’s College London, London, UK

• Presents a logical approach to systematic literature reviewing
• offers a corrective to flawed guidance in existing books
• An accessible but intellectually stimulating guide with illuminating examples and analogies

75k Accesses

27 Citations

10 Altmetric

• Table of contents

About this book

Authors and affiliations, about the authors, bibliographic information.

• Publish with us

Buying options

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

This is a preview of subscription content, log in via an institution to check for access.

Table of contents (11 chapters)

Front matter, introduction.

• Edward Purssell, Niall McCrae

A Brief History of the Systematic Review

The aim and scope of a systematic review: a logical approach, searching the literature, screening search results: a 1-2-3 approach, critical appraisal: assessing the quality of studies, reviewing quantitative studies: meta-analysis and narrative approaches, reviewing qualitative studies and metasynthesis, reviewing qualitative and quantitative studies and mixed-method reviews, meaning and implications: the discussion, making an impact: dissemination of results, back matter.

The systematic review is a rigorous method of collating and synthesizing evidence from multiple studies, producing a whole greater than the sum of parts. This textbook is an authoritative and accessible guide to an activity that is often found overwhelming. The authors steer readers on a logical, sequential path through the process, taking account of the different needs of researchers, students and practitioners. Practical guidance is provided on the fundamentals of systematic reviewing and also on advanced techniques such as meta-analysis. Examples are given in each chapter, with a succinct glossary to support the text.  

This up-to-date, accessible textbook will satisfy the needs of students, practitioners and educators in the sphere of healthcare, and contribute to improving the quality of evidence-based practice. The authors will advise some freely available or inexpensive open source/access resources (such as PubMed, R and Zotero) to help students how to perform a systemic review, in particular those with limited resources.

• Methodology
• Evidence-based practice

Edward Purssell

Florence Nightingale Faculty of Nursing Midwifery & Palliative Care, King’s College London, London, UK

Niall McCrae

Dr. Niall McCrae teaches mental health nursing and research methods at the Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care at King’s College London. His research interests are dementia, depression, the impact of social media on younger people, and the history of mental health care. Niall has written two previous books: The Moon and Madness (Imprint Academic, 2011) and The Story of Nursing in British Mental Hospitals: Echoes from the Corridors (Routledge, 2016). He is a regular writer for Salisbury Review magazine. 

In partnershipPurssell and McCrae have written several papers on research methodology and literature reviewing for healthcare journals. Both have extensive experience of teaching literature reviewing at all academic levels, and explaining complex concepts in a way that is accessible to all

Book Title : How to Perform a Systematic Literature Review

Book Subtitle : A Guide for Healthcare Researchers, Practitioners and Students

Authors : Edward Purssell, Niall McCrae

DOI : https://doi.org/10.1007/978-3-030-49672-2

Publisher : Springer Cham

eBook Packages : Medicine , Medicine (R0)

Copyright Information : The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020

Softcover ISBN : 978-3-030-49671-5 Published: 05 August 2020

eBook ISBN : 978-3-030-49672-2 Published: 04 August 2020

Edition Number : 1

Number of Pages : VII, 188

Number of Illustrations : 7 b/w illustrations, 12 illustrations in colour

Topics : Nursing Research , Nursing Education , Research Skills

Policies and ethics

• Find a journal
• Track your research

Systematic Reviews and Meta Analysis

• Getting Started
• Guides and Standards
• Review Protocols
• Databases and Sources
• Randomized Controlled Trials
• Controlled Clinical Trials
• Observational Designs
• Tests of Diagnostic Accuracy
• Software and Tools
• Where do I get all those articles?
• Collaborations
• EPI 233/528
• Countway Mediated Search
• Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

• Next: Guides and Standards >>
• Last Updated: Feb 26, 2024 3:17 PM
• URL: https://guides.library.harvard.edu/meta-analysis

Literature Reviews

• Introduction
• Tutorials and resources
• Step 1: Literature search
• Step 2: Analysis, synthesis, critique
• Step 3: Writing the review

If you need any assistance, please contact the library staff at the Georgia Tech Library Help website . 

Literature review tutorials

There are many helpful Literature Review video tutorials online. Here is an excellent, succinct (10 min) introduction to how to succeed at a literature review:

Literature Reviews: An Overview for Graduate Students from NC State University Libraries on Vimeo .

For a longer, high quality in-depth look at how literature reviews come together, see this set of  literature review tutorials  from RMIT University.

Literature review resources

We recommend these resources for more information.

This literature review tutorial is from SAGE Research Methods, which has additional resources for learning about literature reviews.

• << Previous: Introduction
• Next: Step 1: Literature search >>
• Last Updated: Apr 2, 2024 11:21 AM
• URL: https://libguides.library.gatech.edu/litreview

• Subject guides
• Systematic Review

Systematic Review: Home

• Getting started
• Manuals, documentation & PRISMA
• Develop question & key concepts
• Look for existing reviews
• Scoping searches & gold set
• Identify search terms
• Select databases & grey literature sources
• Develop criteria & protocol
• Run your search
• Limits & filters
• Review & test your search
• Save & manage your search results
• Database search translation
• Screening process steps
• Assess quality of your included studies
• Request a consultation

About systematic reviews 

A systematic review is a type of literature review that demonstrates your awareness of existing primary research in your field. It is sometimes referred to as secondary research as it is research conducted on research.

Using this guide

Our guide is designed to help you:

• Understand the purpose of systematic reviews
• Follow a clear process to create your systematic review
• Adhere to relevant standards, guidelines or manuals
• Search for existing reviews
• Develop a research question and key concepts
• Select databases and grey literature sources

Get help 

Get help from a librarian  when you need support beyond what you can find in this guide. 15-minute bookings are available for students, educators, and researchers.

1 hour consultations are available to HDR students and Researchers only. For medical/biomedical systematic-style reviews, see Request a consultation for details on the review support service.

Other useful resources

• SR Toolbox -  an online catalogue of guidance and tools that support various processes of systematic reviews or evidence syntheses. The toolbox is developed and maintained by York Health Economics Consortium, University of York.
• Next: Getting started >>

To read this content please select one of the options below:

Please note you do not have access to teaching notes, get on task: a pragmatic tutorial on planning and conducting a systematic literature review.

Contemporary Research Methods in Hospitality and Tourism

ISBN : 978-1-80117-547-0 , eISBN : 978-1-80117-546-3

Publication date: 13 April 2022

This chapter provides a roadmap for a systematic literature review built around the guiding questions of basic research design. First, we highlight the relevance and development of systematic literature reviews in tourism research. Second, we put the systematic review into perspective by outlining its characteristics and by clarifying the methodological assumptions. Third, we bring together recommendations based on previous research and review guidelines and present a step-by-step tutorial for a systematic literature review. From this chapter, readers will understand the foundations of systematic literature reviews, will be able to apply the methodology to their review projects and are introduced to further readings and best practice examples.

• Literature review
• Review guidelines
• Hospitality
• Systematic review
• Research agenda

Bichler, B.F. , Petry, T. , Kallmuenzer, A. and Peters, M. (2022), "Get on Task: A Pragmatic Tutorial on Planning and Conducting a Systematic Literature Review", Okumus, F. , Rasoolimanesh, S.M. and Jahani, S. (Ed.) Contemporary Research Methods in Hospitality and Tourism , Emerald Publishing Limited, Leeds, pp. 39-53. https://doi.org/10.1108/978-1-80117-546-320221004

Emerald Publishing Limited

Copyright © 2022 Bernhard F. Bichler, Tanja Petry, Andreas Kallmuenzer and Mike Peters. Published under exclusive licence by Emerald Publishing Limited

We’re listening — tell us what you think

Something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here