literature synthesis cracker

Literature Synthesis Assistant

Ai-powered tool for synthesizing literature.

• Academic Research: Summarize and synthesize key points from multiple research papers or books for a literature review or research project.
• Content Creation: Combine insights from multiple articles or blog posts to create a comprehensive piece of content.
• Book Reviews: Summarize and synthesize the main themes and ideas from multiple books for a comparative book review.
• Business Analysis: Combine insights from different reports or case studies to create a comprehensive analysis or report.

New & Trending Tools

Ai text generator, webpage text extractor ai.

Literature Syntheis 101

How To Synthesise The Existing Research (With Examples)

By: Derek Jansen (MBA) | Expert Reviewer: Eunice Rautenbach (DTech) | August 2023

One of the most common mistakes that students make when writing a literature review is that they err on the side of describing the existing literature rather than providing a critical synthesis of it. In this post, we’ll unpack what exactly synthesis means and show you how to craft a strong literature synthesis using practical examples.

This post is based on our popular online course, Literature Review Bootcamp . In the course, we walk you through the full process of developing a literature review, step by step. If it’s your first time writing a literature review, you definitely want to use this link to get 50% off the course (limited-time offer).

Overview: Literature Synthesis

• What exactly does “synthesis” mean?
• Aspect 1: Agreement
• Aspect 2: Disagreement
• Aspect 3: Key theories
• Aspect 4: Contexts
• Aspect 5: Methodologies
• Bringing it all together

What does “synthesis” actually mean?

As a starting point, let’s quickly define what exactly we mean when we use the term “synthesis” within the context of a literature review.

Simply put, literature synthesis means going beyond just describing what everyone has said and found. Instead, synthesis is about bringing together all the information from various sources to present a cohesive assessment of the current state of knowledge in relation to your study’s research aims and questions .

Put another way, a good synthesis tells the reader exactly where the current research is “at” in terms of the topic you’re interested in – specifically, what’s known , what’s not , and where there’s a need for more research .

So, how do you go about doing this?

Well, there’s no “one right way” when it comes to literature synthesis, but we’ve found that it’s particularly useful to ask yourself five key questions when you’re working on your literature review. Having done so,  you can then address them more articulately within your actual write up. So, let’s take a look at each of these questions.

1. Points Of Agreement

The first question that you need to ask yourself is: “Overall, what things seem to be agreed upon by the vast majority of the literature?”

For example, if your research aim is to identify which factors contribute toward job satisfaction, you’ll need to identify which factors are broadly agreed upon and “settled” within the literature. Naturally, there may at times be some lone contrarian that has a radical viewpoint , but, provided that the vast majority of researchers are in agreement, you can put these random outliers to the side. That is, of course, unless your research aims to explore a contrarian viewpoint and there’s a clear justification for doing so. 

Identifying what’s broadly agreed upon is an essential starting point for synthesising the literature, because you generally don’t want (or need) to reinvent the wheel or run down a road investigating something that is already well established . So, addressing this question first lays a foundation of “settled” knowledge.

Need a helping hand?

2. Points Of Disagreement

Related to the previous point, but on the other end of the spectrum, is the equally important question: “Where do the disagreements lie?” .

In other words, which things are not well agreed upon by current researchers? It’s important to clarify here that by disagreement, we don’t mean that researchers are (necessarily) fighting over it – just that there are relatively mixed findings within the empirical research , with no firm consensus amongst researchers.

This is a really important question to address as these “disagreements” will often set the stage for the research gap(s). In other words, they provide clues regarding potential opportunities for further research, which your study can then (hopefully) contribute toward filling. If you’re not familiar with the concept of a research gap, be sure to check out our explainer video covering exactly that .

3. Key Theories

The next question you need to ask yourself is: “Which key theories seem to be coming up repeatedly?” .

Within most research spaces, you’ll find that you keep running into a handful of key theories that are referred to over and over again. Apart from identifying these theories, you’ll also need to think about how they’re connected to each other. Specifically, you need to ask yourself:

• Are they all covering the same ground or do they have different focal points  or underlying assumptions ?
• Do some of them feed into each other and if so, is there an opportunity to integrate them into a more cohesive theory?
• Do some of them pull in different directions ? If so, why might this be?
• Do all of the theories define the key concepts and variables in the same way, or is there some disconnect? If so, what’s the impact of this ?

Simply put, you’ll need to pay careful attention to the key theories in your research area, as they will need to feature within your theoretical framework , which will form a critical component within your final literature review. This will set the foundation for your entire study, so it’s essential that you be critical in this area of your literature synthesis.

If this sounds a bit fluffy, don’t worry. We deep dive into the theoretical framework (as well as the conceptual framework) and look at practical examples in Literature Review Bootcamp . If you’d like to learn more, take advantage of our limited-time offer to get 60% off the standard price.

4. Contexts

The next question that you need to address in your literature synthesis is an important one, and that is: “Which contexts have (and have not) been covered by the existing research?” .

For example, sticking with our earlier hypothetical topic (factors that impact job satisfaction), you may find that most of the research has focused on white-collar , management-level staff within a primarily Western context, but little has been done on blue-collar workers in an Eastern context. Given the significant socio-cultural differences between these two groups, this is an important observation, as it could present a contextual research gap .

In practical terms, this means that you’ll need to carefully assess the context of each piece of literature that you’re engaging with, especially the empirical research (i.e., studies that have collected and analysed real-world data). Ideally, you should keep notes regarding the context of each study in some sort of catalogue or sheet, so that you can easily make sense of this before you start the writing phase. If you’d like, our free literature catalogue worksheet is a great tool for this task.

5. Methodological Approaches

Last but certainly not least, you need to ask yourself the question: “What types of research methodologies have (and haven’t) been used?”

For example, you might find that most studies have approached the topic using qualitative methods such as interviews and thematic analysis. Alternatively, you might find that most studies have used quantitative methods such as online surveys and statistical analysis.

But why does this matter?

Well, it can run in one of two potential directions . If you find that the vast majority of studies use a specific methodological approach, this could provide you with a firm foundation on which to base your own study’s methodology . In other words, you can use the methodologies of similar studies to inform (and justify) your own study’s research design .

On the other hand, you might argue that the lack of diverse methodological approaches presents a research gap , and therefore your study could contribute toward filling that gap by taking a different approach. For example, taking a qualitative approach to a research area that is typically approached quantitatively. Of course, if you’re going to go against the methodological grain, you’ll need to provide a strong justification for why your proposed approach makes sense. Nevertheless, it is something worth at least considering.

Regardless of which route you opt for, you need to pay careful attention to the methodologies used in the relevant studies and provide at least some discussion about this in your write-up. Again, it’s useful to keep track of this on some sort of spreadsheet or catalogue as you digest each article, so consider grabbing a copy of our free literature catalogue if you don’t have anything in place.

Bringing It All Together

Alright, so we’ve looked at five important questions that you need to ask (and answer) to help you develop a strong synthesis within your literature review.  To recap, these are:

• Which things are broadly agreed upon within the current research?
• Which things are the subject of disagreement (or at least, present mixed findings)?
• Which theories seem to be central to your research topic and how do they relate or compare to each other?
• Which contexts have (and haven’t) been covered?
• Which methodological approaches are most common?

Importantly, you’re not just asking yourself these questions for the sake of asking them – they’re not just a reflection exercise. You need to weave your answers to them into your actual literature review when you write it up. How exactly you do this will vary from project to project depending on the structure you opt for, but you’ll still need to address them within your literature review, whichever route you go.

The best approach is to spend some time actually writing out your answers to these questions, as opposed to just thinking about them in your head. Putting your thoughts onto paper really helps you flesh out your thinking . As you do this, don’t just write down the answers – instead, think about what they mean in terms of the research gap you’ll present , as well as the methodological approach you’ll take . Your literature synthesis needs to lay the groundwork for these two things, so it’s essential that you link all of it together in your mind, and of course, on paper.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

excellent , thank you

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

The Sheridan Libraries

• Write a Literature Review
• Sheridan Libraries
• Find This link opens in a new window
• Evaluate This link opens in a new window

Get Organized

• Lit Review Prep Use this template to help you evaluate your sources, create article summaries for an annotated bibliography, and a synthesis matrix for your lit review outline.

Synthesize your Information

Synthesize: combine separate elements to form a whole.

Synthesis Matrix

A synthesis matrix helps you record the main points of each source and document how sources relate to each other.

After summarizing and evaluating your sources, arrange them in a matrix or use a citation manager to help you see how they relate to each other and apply to each of your themes or variables.  

By arranging your sources by theme or variable, you can see how your sources relate to each other, and can start thinking about how you weave them together to create a narrative.

• Step-by-Step Approach
• Example Matrix from NSCU
• Matrix Template
• << Previous: Summarize
• Next: Integrate >>
• Last Updated: Sep 26, 2023 10:25 AM
• URL: https://guides.library.jhu.edu/lit-review

• University of Oregon Libraries
• Research Guides

How to Write a Literature Review

• 6. Synthesize
• Literature Reviews: A Recap
• Reading Journal Articles
• Does it Describe a Literature Review?
• 1. Identify the Question
• 2. Review Discipline Styles
• Searching Article Databases
• Finding Full-Text of an Article
• Citation Chaining
• When to Stop Searching
• 4. Manage Your References
• 5. Critically Analyze and Evaluate

Synthesis Visualization

Synthesis matrix example.

• 7. Write a Literature Review

• Synthesis Worksheet

About Synthesis

Approaches to synthesis.

You can sort the literature in various ways, for example:

How to Begin?

Read your sources carefully and find the main idea(s) of each source

Look for similarities in your sources – which sources are talking about the same main ideas? (for example, sources that discuss the historical background on your topic)

Use the worksheet (above) or synthesis matrix (below) to get organized

This work can be messy. Don't worry if you have to go through a few iterations of the worksheet or matrix as you work on your lit review!

Four Examples of Student Writing

In the four examples below, only ONE shows a good example of synthesis: the fourth column, or  Student D . For a web accessible version, click the link below the image.

Long description of "Four Examples of Student Writing" for web accessibility

• Download a copy of the "Four Examples of Student Writing" chart

Click on the example to view the pdf.

From Jennifer Lim

• << Previous: 5. Critically Analyze and Evaluate
• Next: 7. Write a Literature Review >>
• Last Updated: Jan 10, 2024 4:46 PM
• URL: https://researchguides.uoregon.edu/litreview

Contact Us Library Accessibility UO Libraries Privacy Notices and Procedures

1501 Kincaid Street Eugene, OR 97403 P: 541-346-3053 F: 541-346-3485

• Visit us on Facebook
• Visit us on Twitter
• Visit us on Youtube
• Visit us on Instagram
• Report a Concern
• Nondiscrimination and Title IX
• Accessibility
• Privacy Policy
• Find People

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 7: Synthesizing Sources

Learning objectives.

At the conclusion of this chapter, you will be able to:

• synthesize key sources connecting them with the research question and topic area.

7.1 Overview of synthesizing

7.1.1 putting the pieces together.

Combining separate elements into a whole is the dictionary definition of synthesis.  It is a way to make connections among and between numerous and varied source materials.  A literature review is not an annotated bibliography, organized by title, author, or date of publication.  Rather, it is grouped by topic to create a whole view of the literature relevant to your research question.

Your synthesis must demonstrate a critical analysis of the papers you collected as well as your ability to integrate the results of your analysis into your own literature review.  Each paper collected should be critically evaluated and weighed for “adequacy, appropriateness, and thoroughness” ( Garrard, 2017 ) before inclusion in your own review.  Papers that do not meet this criteria likely should not be included in your literature review.

Begin the synthesis process by creating a grid, table, or an outline where you will summarize, using common themes you have identified and the sources you have found. The summary grid or outline will help you compare and contrast the themes so you can see the relationships among them as well as areas where you may need to do more searching. Whichever method you choose, this type of organization will help you to both understand the information you find and structure the writing of your review.  Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” ( Bennard et al., 2014 ).

As you read through the material you gather, look for common themes as they may provide the structure for your literature review.  And, remember, research is an iterative process: it is not unusual to go back and search information sources for more material.

At one extreme, if you are claiming, ‘There are no prior publications on this topic,’ it is more likely that you have not found them yet and may need to broaden your search.  At another extreme, writing a complete literature review can be difficult with a well-trod topic.  Do not cite it all; instead cite what is most relevant.  If that still leaves too much to include, be sure to reference influential sources…as well as high-quality work that clearly connects to the points you make. ( Klingner, Scanlon, & Pressley, 2005 ).

7.2 Creating a summary table

Literature reviews can be organized sequentially or by topic, theme, method, results, theory, or argument.  It’s important to develop categories that are meaningful and relevant to your research question.  Take detailed notes on each article and use a consistent format for capturing all the information each article provides.  These notes and the summary table can be done manually, using note cards.  However, given the amount of information you will be recording, an electronic file created in a word processing or spreadsheet is more manageable. Examples of fields you may want to capture in your notes include:

• Authors’ names
• Article title
• Publication year
• Main purpose of the article
• Methodology or research design
• Participants
• Measurement
• Conclusions

  Other fields that will be useful when you begin to synthesize the sum total of your research:

• Specific details of the article or research that are especially relevant to your study
• Key terms and definitions
• Strengths or weaknesses in research design
• Relationships to other studies
• Possible gaps in the research or literature (for example, many research articles conclude with the statement “more research is needed in this area”)
• Finally, note how closely each article relates to your topic.  You may want to rank these as high, medium, or low relevance.  For papers that you decide not to include, you may want to note your reasoning for exclusion, such as ‘small sample size’, ‘local case study,’ or ‘lacks evidence to support assertion.’

This short video demonstrates how a nursing researcher might create a summary table.

7.2.1 Creating a Summary Table

  Summary tables can be organized by author or by theme, for example:

For a summary table template, see http://blogs.monm.edu/writingatmc/files/2013/04/Synthesis-Matrix-Template.pdf

7.3 Creating a summary outline

An alternate way to organize your articles for synthesis it to create an outline. After you have collected the articles you intend to use (and have put aside the ones you won’t be using), it’s time to identify the conclusions that can be drawn from the articles as a group.

  Based on your review of the collected articles, group them by categories.  You may wish to further organize them by topic and then chronologically or alphabetically by author.  For each topic or subtopic you identified during your critical analysis of the paper, determine what those papers have in common.  Likewise, determine which ones in the group differ.  If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for the contradiction (for example, differences in population demographics).  Determine what general conclusions you can report about the topic or subtopic as the entire group of studies relate to it.  For example, you may have several studies that agree on outcome, such as ‘hands on learning is best for science in elementary school’ or that ‘continuing education is the best method for updating nursing certification.’ In that case, you may want to organize by methodology used in the studies rather than by outcome.

Organize your outline in a logical order and prepare to write the first draft of your literature review.  That order might be from broad to more specific, or it may be sequential or chronological, going from foundational literature to more current.  Remember, “an effective literature review need not denote the entire historical record, but rather establish the raison d’etre for the current study and in doing so cite that literature distinctly pertinent for theoretical, methodological, or empirical reasons.” ( Milardo, 2015, p. 22 ).

As you organize the summarized documents into a logical structure, you are also appraising and synthesizing complex information from multiple sources.  Your literature review is the result of your research that synthesizes new and old information and creates new knowledge.

7.4 Additional resources:

Literature Reviews: Using a Matrix to Organize Research / Saint Mary’s University of Minnesota

Literature Review: Synthesizing Multiple Sources / Indiana University

Writing a Literature Review and Using a Synthesis Matrix / Florida International University

 Sample Literature Reviews Grid / Complied by Lindsay Roberts

Select three or four articles on a single topic of interest to you. Then enter them into an outline or table in the categories you feel are important to a research question. Try both the grid and the outline if you can to see which suits you better. The attached grid contains the fields suggested in the video .

Literature Review Table  

Test yourself.

• Select two articles from your own summary table or outline and write a paragraph explaining how and why the sources relate to each other and your review of the literature.
• In your literature review, under what topic or subtopic will you place the paragraph you just wrote?

Image attribution

Literature Reviews for Education and Nursing Graduate Students Copyright © by Linda Frederiksen is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

Literature Reviews

• 5. Synthesize your findings
• Getting started
• Types of reviews
• 1. Define your research question
• 2. Plan your search
• 3. Search the literature
• 4. Organize your results

How to synthesize

Approaches to synthesis.

• 6. Write the review
• Artificial intelligence (AI) tools
• Thompson Writing Studio This link opens in a new window
• Need to write a systematic review? This link opens in a new window

Contact a Librarian

Ask a Librarian

In the synthesis step of a literature review, researchers analyze and integrate information from selected sources to identify patterns and themes. This involves critically evaluating findings, recognizing commonalities, and constructing a cohesive narrative that contributes to the understanding of the research topic.

Here are some examples of how to approach synthesizing the literature:

💡 By themes or concepts

🕘 Historically or chronologically

📊 By methodology

These organizational approaches can also be used when writing your review. It can be beneficial to begin organizing your references by these approaches in your citation manager by using folders, groups, or collections.

Create a synthesis matrix

A synthesis matrix allows you to visually organize your literature.

Topic: ______________________________________________

Topic: Chemical exposure to workers in nail salons

• << Previous: 4. Organize your results
• Next: 6. Write the review >>
• Last Updated: Apr 3, 2024 12:40 PM
• URL: https://guides.library.duke.edu/lit-reviews

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

Literature Review Basics

• What is a Literature Review?
• Synthesizing Research
• Using Research & Synthesis Tables
• Additional Resources

Synthesis: What is it?

First, let's be perfectly clear about what synthesizing your research isn't :

• - It isn't  just summarizing the material you read
• - It isn't  generating a collection of annotations or comments (like an annotated bibliography)
• - It isn't  compiling a report on every single thing ever written in relation to your topic

When you  synthesize  your research, your job is to help your reader understand the current state of the conversation on your topic, relative to your research question.  That may include doing the following:

• - Selecting and using representative work on the topic
• - Identifying and discussing trends in published data or results
• - Identifying and explaining the impact of common features (study populations, interventions, etc.) that appear frequently in the literature
• - Explaining controversies, disputes, or central issues in the literature that are relevant to your research question
• - Identifying gaps in the literature, where more research is needed
• - Establishing the discussion to which your own research contributes and demonstrating the value of your contribution

Essentially, you're telling your reader where they are (and where you are) in the scholarly conversation about your project.

Synthesis: How do I do it?

Synthesis, step by step.

This is what you need to do  before  you write your review.

• Identify and clearly describe your research question (you may find the Formulating PICOT Questions table at  the Additional Resources tab helpful).
• Collect sources relevant to your research question.
• Organize and describe the sources you've found -- your job is to identify what  types  of sources you've collected (reviews, clinical trials, etc.), identify their  purpose  (what are they measuring, testing, or trying to discover?), determine the  level of evidence  they represent (see the Levels of Evidence table at the Additional Resources tab ), and briefly explain their  major findings . Use a Research Table to document this step.
• Study the information you've put in your Research Table and examine your collected sources, looking for  similarities  and  differences . Pay particular attention to  populations ,   methods  (especially relative to levels of evidence), and  findings .
• Analyze what you learn in (4) using a tool like a Synthesis Table. Your goal is to identify relevant themes, trends, gaps, and issues in the research.  Your literature review will collect the results of this analysis and explain them in relation to your research question.

Analysis tips

• - Sometimes, what you  don't  find in the literature is as important as what you do find -- look for questions that the existing research hasn't answered yet.
• - If any of the sources you've collected refer to or respond to each other, keep an eye on how they're related -- it may provide a clue as to whether or not study results have been successfully replicated.
• - Sorting your collected sources by level of evidence can provide valuable insight into how a particular topic has been covered, and it may help you to identify gaps worth addressing in your own work.
• << Previous: What is a Literature Review?
• Next: Using Research & Synthesis Tables >>
• Last Updated: Sep 26, 2023 12:06 PM
• URL: https://usi.libguides.com/literature-review-basics

How to Synthesize Written Information from Multiple Sources

Shona McCombes

Content Manager

B.A., English Literature, University of Glasgow

Shona McCombes is the content manager at Scribbr, Netherlands.

Learn about our Editorial Process

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

On This Page:

When you write a literature review or essay, you have to go beyond just summarizing the articles you’ve read – you need to synthesize the literature to show how it all fits together (and how your own research fits in).

Synthesizing simply means combining. Instead of summarizing the main points of each source in turn, you put together the ideas and findings of multiple sources in order to make an overall point.

At the most basic level, this involves looking for similarities and differences between your sources. Your synthesis should show the reader where the sources overlap and where they diverge.

Unsynthesized Example

Franz (2008) studied undergraduate online students. He looked at 17 females and 18 males and found that none of them liked APA. According to Franz, the evidence suggested that all students are reluctant to learn citations style. Perez (2010) also studies undergraduate students. She looked at 42 females and 50 males and found that males were significantly more inclined to use citation software ( p < .05). Findings suggest that females might graduate sooner. Goldstein (2012) looked at British undergraduates. Among a sample of 50, all females, all confident in their abilities to cite and were eager to write their dissertations.

Synthesized Example

Studies of undergraduate students reveal conflicting conclusions regarding relationships between advanced scholarly study and citation efficacy. Although Franz (2008) found that no participants enjoyed learning citation style, Goldstein (2012) determined in a larger study that all participants watched felt comfortable citing sources, suggesting that variables among participant and control group populations must be examined more closely. Although Perez (2010) expanded on Franz’s original study with a larger, more diverse sample…

Step 1: Organize your sources

After collecting the relevant literature, you’ve got a lot of information to work through, and no clear idea of how it all fits together.

Before you can start writing, you need to organize your notes in a way that allows you to see the relationships between sources.

One way to begin synthesizing the literature is to put your notes into a table. Depending on your topic and the type of literature you’re dealing with, there are a couple of different ways you can organize this.

Summary table

A summary table collates the key points of each source under consistent headings. This is a good approach if your sources tend to have a similar structure – for instance, if they’re all empirical papers.

Each row in the table lists one source, and each column identifies a specific part of the source. You can decide which headings to include based on what’s most relevant to the literature you’re dealing with.

For example, you might include columns for things like aims, methods, variables, population, sample size, and conclusion.

For each study, you briefly summarize each of these aspects. You can also include columns for your own evaluation and analysis.

The summary table gives you a quick overview of the key points of each source. This allows you to group sources by relevant similarities, as well as noticing important differences or contradictions in their findings.

Synthesis matrix

A synthesis matrix is useful when your sources are more varied in their purpose and structure – for example, when you’re dealing with books and essays making various different arguments about a topic.

Each column in the table lists one source. Each row is labeled with a specific concept, topic or theme that recurs across all or most of the sources.

Then, for each source, you summarize the main points or arguments related to the theme.

The purposes of the table is to identify the common points that connect the sources, as well as identifying points where they diverge or disagree.

Step 2: Outline your structure

Now you should have a clear overview of the main connections and differences between the sources you’ve read. Next, you need to decide how you’ll group them together and the order in which you’ll discuss them.

For shorter papers, your outline can just identify the focus of each paragraph; for longer papers, you might want to divide it into sections with headings.

There are a few different approaches you can take to help you structure your synthesis.

If your sources cover a broad time period, and you found patterns in how researchers approached the topic over time, you can organize your discussion chronologically .

That doesn’t mean you just summarize each paper in chronological order; instead, you should group articles into time periods and identify what they have in common, as well as signalling important turning points or developments in the literature.

If the literature covers various different topics, you can organize it thematically .

That means that each paragraph or section focuses on a specific theme and explains how that theme is approached in the literature.

Source Used with Permission: The Chicago School

If you’re drawing on literature from various different fields or they use a wide variety of research methods, you can organize your sources methodologically .

That means grouping together studies based on the type of research they did and discussing the findings that emerged from each method.

If your topic involves a debate between different schools of thought, you can organize it theoretically .

That means comparing the different theories that have been developed and grouping together papers based on the position or perspective they take on the topic, as well as evaluating which arguments are most convincing.

Step 3: Write paragraphs with topic sentences

What sets a synthesis apart from a summary is that it combines various sources. The easiest way to think about this is that each paragraph should discuss a few different sources, and you should be able to condense the overall point of the paragraph into one sentence.

This is called a topic sentence , and it usually appears at the start of the paragraph. The topic sentence signals what the whole paragraph is about; every sentence in the paragraph should be clearly related to it.

A topic sentence can be a simple summary of the paragraph’s content:

“Early research on [x] focused heavily on [y].”

For an effective synthesis, you can use topic sentences to link back to the previous paragraph, highlighting a point of debate or critique:

“Several scholars have pointed out the flaws in this approach.” “While recent research has attempted to address the problem, many of these studies have methodological flaws that limit their validity.”

By using topic sentences, you can ensure that your paragraphs are coherent and clearly show the connections between the articles you are discussing.

As you write your paragraphs, avoid quoting directly from sources: use your own words to explain the commonalities and differences that you found in the literature.

Don’t try to cover every single point from every single source – the key to synthesizing is to extract the most important and relevant information and combine it to give your reader an overall picture of the state of knowledge on your topic.

Step 4: Revise, edit and proofread

Like any other piece of academic writing, synthesizing literature doesn’t happen all in one go – it involves redrafting, revising, editing and proofreading your work.

Checklist for Synthesis

•   Do I introduce the paragraph with a clear, focused topic sentence?
•   Do I discuss more than one source in the paragraph?
•   Do I mention only the most relevant findings, rather than describing every part of the studies?
•   Do I discuss the similarities or differences between the sources, rather than summarizing each source in turn?
•   Do I put the findings or arguments of the sources in my own words?
•   Is the paragraph organized around a single idea?
•   Is the paragraph directly relevant to my research question or topic?
•   Is there a logical transition from this paragraph to the next one?

Further Information

How to Synthesise: a Step-by-Step Approach

Help…I”ve Been Asked to Synthesize!

Learn how to Synthesise (combine information from sources)

How to write a Psychology Essay

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

3.2 Synthesizing literature

Learning objectives.

• Connect the sources you read with key concepts in your research question and proposal
• Systematize the information and facts from each source you read

Putting the pieces together

Combining separate elements into a whole is the dictionary definition of synthesis. It is a way to make connections among and between numerous and varied source materials. A literature review is not an annotated bibliography, organized by title, author, or date of publication. Rather, it is grouped by topic and argument to create a whole view of the literature relevant to your research question.

Your synthesis must demonstrate a critical analysis of the papers you collected, as well as your ability to integrate the results of your analysis into your own literature review. Each source you collect should be critically evaluated and weighed based on the criteria from Chapter 2 before you include it in your review.

Begin the synthesis process by creating a grid, table, or an outline where you will summarize your literature review findings, using common themes you have identified and the sources you have found. The summary, grid, or outline will help you compare and contrast the themes, so you can see the relationships among them as well as areas where you may need to do more searching. A basic summary table is provided in Figure 3.2. Whichever method you choose, this type of organization will help you to both understand the information you find and structure the writing of your review. Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” (Bennard et al., 2014, para. 10).

As you read through the material you gather, look for common themes as they may provide the structure for your literature review. And, remember, writing a literature review is an iterative process. It is not unusual to go back and search academic databases for more sources of information as you read the articles you’ve collected.

Literature reviews can be organized sequentially or by topic, theme, method, results, theory, or argument. It’s important to develop categories that are meaningful and relevant to your research question. Take detailed notes on each article and use a consistent format for capturing all the information each article provides. These notes and the summary table can be done manually using note cards. However, given the amount of information you will be recording, an electronic file created in a word processing or spreadsheet (like this example Literature Search Template ) is more manageable. Examples of fields you may want to capture in your notes include:

• Authors’ names
• Article title
• Publication year
• Main purpose of the article
• Methodology or research design
• Participants
• Measurement
• Conclusions

Other fields that will be useful when you begin to synthesize the sum total of your research:

• Specific details of the article or research that are especially relevant to your study
• Key terms and definitions
• Strengths or weaknesses in research design
• Relationships to other studies
• Possible gaps in the research or literature (for example, many research articles conclude with the statement “more research is needed in this area”)
• Finally, note how closely each article relates to your topic. You may want to rank these as high, medium, or low relevance. For papers that you decide not to include, you may want to note your reasoning for exclusion, such as small sample size, local case study, or lacks evidence to support conclusions.

An example of how to organize summary tables by author or theme is shown in Table 3.1.

Here is an example summary table template .

Creating a topical outline

An alternative way to organize your articles for synthesis it to create an outline. After you have collected the articles you intend to use (and have put aside the ones you won’t be using), it’s time to extract as much as possible from the facts provided in those articles. You are starting your research project without a lot of hard facts on the topics you want to study, and by using the literature reviews provided in academic journal articles, you can gain a lot of knowledge about a topic in a short period of time.

As you read an article in detail, try copying the information you find relevant to your research topic in a separate word processing document. Copying and pasting from PDF to Word can be a pain because PDFs are image files not documents. To make that easier, use the HTML version of the article, convert the PDF to Word in Adobe Acrobat or another PDF reader, or use “paste special” command to paste the content into Word without formatting. If it’s an old PDF, you may have to simply type out the information you need. It can be a messy job, but having all of your facts in one place is very helpful for drafting your literature review.  Of course, you will not be using other authors’ words in your own literature review, but this is a good way to start compiling your notes.

You should copy and paste any fact or argument you consider important. Some good examples include definitions of concepts, statistics about the size of the social problem, and empirical evidence about the key variables in the research question, among countless others. It’s a good idea to consult with your professor and the syllabus for the course about what they are looking for when they read your literature review. Facts for your literature review are principally found in the introduction, results, and discussion section of an empirical article or at any point in a non-empirical article. Copy and paste into your notes anything you may want to use in your literature review.

Importantly, you must make sure you note the original source of that information. Nothing is worse than searching your articles for hours only to realize you forgot to note where your facts came from. If you found a statistic that the author used in the introduction, it almost certainly came from another source that the author cited in a footnote or internal citation. You will want to check the original source to make sure the author represented the information correctly. Moreover, you may want to read the original study to learn more about your topic and discover other sources relevant to your inquiry.

Assuming you have pulled all of the facts out of multiple articles, it’s time to start thinking about how these pieces of information relate to each other. Start grouping each fact into categories and subcategories as shown in Figure 3.3. For example, a statistic stating that homeless single adults are more likely to be male may fit into a category of gender and homelessness. For each topic or subtopic you identified during your critical analysis of each paper, determine what those papers have in common. Likewise, determine which ones in the group differ. If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for the contradiction. For example, one study may sample only high-income earners or those in a rural area. Determine what general conclusions you can report about the topic or subtopic, based on all of the information you’ve found.

Create a separate document containing a topical outline that combines your facts from each source and organizes them by topic or category. As you include more facts and more sources into your topical outline, you will begin to see how each fact fits into a category and how categories are related to each other. Your category names may change over time, as may their definitions. This is a natural reflection of the learning you are doing.

A complete topical outline is a long list of facts, arranged by category about your topic. As you step back from the outline, you should understand the topic areas where you have enough information to make strong conclusions about what the literature says. You should also assess in what areas you need to do more research before you can write a robust literature review. The topical outline should serve as a transitional document between the notes you write on each source and the literature review you submit to your professor. It is important to note that they contain plagiarized information that is copied and pasted directly from the primary sources. That’s okay because these are just notes and are not meant to be turned in as your own ideas. For your final literature review, you must paraphrase these sources to avoid plagiarism. More importantly, you should keep your voice and ideas front-and-center in what you write as this is your analysis of the literature. Make strong claims and support them thoroughly using facts you found in the literature. We will pick up the task of writing your literature review in section 3.3.

Additional resources for synthesizing literature

There are many ways to approach synthesizing literature. We’ve reviewed two examples here: summary tables and topical outlines. Other examples you may encounter include annotated bibliographies and synthesis matrices. As you are learning research, find a method that works for you. Reviewing the literature is a core component of evidence-based practice in social work at any level. See the resources below if you need some additional help:

Literature Reviews: Using a Matrix to Organize Research  / Saint Mary’s University of Minnesota

Literature Review: Synthesizing Multiple Sources  / Indiana University

Writing a Literature Review and Using a Synthesis Matrix  / Florida International University

Sample Literature Reviews Grid  / Complied by Lindsay Roberts

Killam, Laura (2013) . Literature review preparation: Creating a summary table . Includes transcript.

Key Takeaways

• It is necessary to take notes on research articles as you read. Try to develop a system that works for you to keep your notes organized, such as a summary table.
• Summary tables and topical outlines help researchers synthesize sources for the purpose of writing a literature review.

Image attributions

Pieces of the puzzle by congerdesign cc-0, adult diary by pexels cc-0.

• Figure 3.2 copied from Frederiksen, L. & Phelps, S. F. (2018). Literature reviews for education and nursing graduate students. Shared under a CC-BY 4.0 license. ↵
• This table was adapted from the work of Amanda Parsons ↵

Guidebook for Social Work Literature Reviews and Research Questions Copyright © 2020 by Rebecca Mauldin and Matthew DeCarlo is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• AIMS Public Health
• v.3(1); 2016

What Synthesis Methodology Should I Use? A Review and Analysis of Approaches to Research Synthesis

Kara schick-makaroff.

1 Faculty of Nursing, University of Alberta, Edmonton, AB, Canada

Marjorie MacDonald

2 School of Nursing, University of Victoria, Victoria, BC, Canada

Marilyn Plummer

3 College of Nursing, Camosun College, Victoria, BC, Canada

Judy Burgess

4 Student Services, University Health Services, Victoria, BC, Canada

Wendy Neander

Associated data, additional file 1.

When we began this process, we were doctoral students and a faculty member in a research methods course. As students, we were facing a review of the literature for our dissertations. We encountered several different ways of conducting a review but were unable to locate any resources that synthesized all of the various synthesis methodologies. Our purpose is to present a comprehensive overview and assessment of the main approaches to research synthesis. We use ‘research synthesis’ as a broad overarching term to describe various approaches to combining, integrating, and synthesizing research findings.

We conducted an integrative review of the literature to explore the historical, contextual, and evolving nature of research synthesis. We searched five databases, reviewed websites of key organizations, hand-searched several journals, and examined relevant texts from the reference lists of the documents we had already obtained.

We identified four broad categories of research synthesis methodology including conventional, quantitative, qualitative, and emerging syntheses. Each of the broad categories was compared to the others on the following: key characteristics, purpose, method, product, context, underlying assumptions, unit of analysis, strengths and limitations, and when to use each approach.

Conclusions

The current state of research synthesis reflects significant advancements in emerging synthesis studies that integrate diverse data types and sources. New approaches to research synthesis provide a much broader range of review alternatives available to health and social science students and researchers.

1. Introduction

Since the turn of the century, public health emergencies have been identified worldwide, particularly related to infectious diseases. For example, the Severe Acute Respiratory Syndrome (SARS) epidemic in Canada in 2002-2003, the recent Ebola epidemic in Africa, and the ongoing HIV/AIDs pandemic are global health concerns. There have also been dramatic increases in the prevalence of chronic diseases around the world [1] – [3] . These epidemiological challenges have raised concerns about the ability of health systems worldwide to address these crises. As a result, public health systems reform has been initiated in a number of countries. In Canada, as in other countries, the role of evidence to support public health reform and improve population health has been given high priority. Yet, there continues to be a significant gap between the production of evidence through research and its application in practice [4] – [5] . One strategy to address this gap has been the development of new research synthesis methodologies to deal with the time-sensitive and wide ranging evidence needs of policy makers and practitioners in all areas of health care, including public health.

As doctoral nursing students facing a review of the literature for our dissertations, and as a faculty member teaching a research methods course, we encountered several ways of conducting a research synthesis but found no comprehensive resources that discussed, compared, and contrasted various synthesis methodologies on their purposes, processes, strengths and limitations. To complicate matters, writers use terms interchangeably or use different terms to mean the same thing, and the literature is often contradictory about various approaches. Some texts [6] , [7] – [9] did provide a preliminary understanding about how research synthesis had been taken up in nursing, but these did not meet our requirements. Thus, in this article we address the need for a comprehensive overview of research synthesis methodologies to guide public health, health care, and social science researchers and practitioners.

Research synthesis is relatively new in public health but has a long history in other fields dating back to the late 1800s. Research synthesis, a research process in its own right [10] , has become more prominent in the wake of the evidence-based movement of the 1990s. Research syntheses have found their advocates and detractors in all disciplines, with challenges to the processes of systematic review and meta-analysis, in particular, being raised by critics of evidence-based healthcare [11] – [13] .

Our purpose was to conduct an integrative review of the literature to explore the historical, contextual, and evolving nature of research synthesis [14] – [15] . We synthesize and critique the main approaches to research synthesis that are relevant for public health, health care, and social scientists. Research synthesis is the overarching term we use to describe approaches to combining, aggregating, integrating, and synthesizing primary research findings. Each synthesis methodology draws on different types of findings depending on the purpose and product of the chosen synthesis (see Additional File 1 ).

3. Method of Review

Based on our current knowledge of the literature, we identified these approaches to include in our review: systematic review, meta-analysis, qualitative meta-synthesis, meta-narrative synthesis, scoping review, rapid review, realist synthesis, concept analysis, literature review, and integrative review. Our first step was to divide the synthesis types among the research team. Each member did a preliminary search to identify key texts. The team then met to develop search terms and a framework to guide the review.

Over the period of 2008 to 2012 we extensively searched the literature, updating our search at several time points, not restricting our search by date. The dates of texts reviewed range from 1967 to 2015. We used the terms above combined with the term “method* (e.g., “realist synthesis” and “method*) in the database Health Source: Academic Edition (includes Medline and CINAHL). This search yielded very few texts on some methodologies and many on others. We realized that many documents on research synthesis had not been picked up in the search. Therefore, we also searched Google Scholar, PubMed, ERIC, and Social Science Index, as well as the websites of key organizations such as the Joanna Briggs Institute, the University of York Centre for Evidence-Based Nursing, and the Cochrane Collaboration database. We hand searched several nursing, social science, public health and health policy journals. Finally, we traced relevant documents from the references in obtained texts.

We included works that met the following inclusion criteria: (1) published in English; (2) discussed the history of research synthesis; (3) explicitly described the approach and specific methods; or (4) identified issues, challenges, strengths and limitations of the particular methodology. We excluded research reports that resulted from the use of particular synthesis methodologies unless they also included criteria 2, 3, or 4 above.

Based on our search, we identified additional types of research synthesis (e.g., meta-interpretation, best evidence synthesis, critical interpretive synthesis, meta-summary, grounded formal theory). Still, we missed some important developments in meta-analysis, for example, identified by the journal's reviewers that have now been discussed briefly in the paper. The final set of 197 texts included in our review comprised theoretical, empirical, and conceptual papers, books, editorials and commentaries, and policy documents.

In our preliminary review of key texts, the team inductively developed a framework of the important elements of each method for comparison. In the next phase, each text was read carefully, and data for these elements were extracted into a table for comparison on the points of: key characteristics, purpose, methods, and product; see Additional File 1 ). Once the data were grouped and extracted, we synthesized across categories based on the following additional points of comparison: complexity of the process, degree of systematization, consideration of context, underlying assumptions, unit of analysis, and when to use each approach. In our results, we discuss our comparison of the various synthesis approaches on the elements above. Drawing only on documents for the review, ethics approval was not required.

We identified four broad categories of research synthesis methodology: Conventional, quantitative, qualitative, and emerging syntheses. From our dataset of 197 texts, we had 14 texts on conventional synthesis, 64 on quantitative synthesis, 78 on qualitative synthesis, and 41 on emerging syntheses. Table 1 provides an overview of the four types of research synthesis, definitions, types of data used, products, and examples of the methodology.

Although we group these types of synthesis into four broad categories on the basis of similarities, each type within a category has unique characteristics, which may differ from the overall group similarities. Each could be explored in greater depth to tease out their unique characteristics, but detailed comparison is beyond the scope of this article.

Additional File 1 presents one or more selected types of synthesis that represent the broad category but is not an exhaustive presentation of all types within each category. It provides more depth for specific examples from each category of synthesis on the characteristics, purpose, methods, and products than is found in Table 1 .

4.1. Key Characteristics

4.1.1. what is it.

Here we draw on two types of categorization. First, we utilize Dixon Woods et al.'s [49] classification of research syntheses as being either integrative or interpretive . (Please note that integrative syntheses are not the same as an integrative review as defined in Additional File 1 .) Second, we use Popay's [80] enhancement and epistemological models .

The defining characteristics of integrative syntheses are that they involve summarizing the data achieved by pooling data [49] . Integrative syntheses include systematic reviews, meta-analyses, as well as scoping and rapid reviews because each of these focus on summarizing data. They also define concepts from the outset (although this may not always be true in scoping or rapid reviews) and deal with a well-specified phenomenon of interest.

Interpretive syntheses are primarily concerned with the development of concepts and theories that integrate concepts [49] . The analysis in interpretive synthesis is conceptual both in process and outcome, and “the product is not aggregations of data, but theory” [49] , [p.12]. Interpretive syntheses involve induction and interpretation, and are primarily conceptual in process and outcome. Examples include integrative reviews, some systematic reviews, all of the qualitative syntheses, meta-narrative, realist and critical interpretive syntheses. Of note, both quantitative and qualitative studies can be either integrative or interpretive

The second categorization, enhancement versus epistemological , applies to those approaches that use multiple data types and sources [80] . Popay's [80] classification reflects the ways that qualitative data are valued in relation to quantitative data.

In the enhancement model , qualitative data adds something to quantitative analysis. The enhancement model is reflected in systematic reviews and meta-analyses that use some qualitative data to enhance interpretation and explanation. It may also be reflected in some rapid reviews that draw on quantitative data but use some qualitative data.

The epistemological model assumes that quantitative and qualitative data are equal and each has something unique to contribute. All of the other review approaches, except pure quantitative or qualitative syntheses, reflect the epistemological model because they value all data types equally but see them as contributing different understandings.

4.1.2. Data type

By and large, the quantitative approaches (quantitative systematic review and meta-analysis) have typically used purely quantitative data (i.e., expressed in numeric form). More recently, both Cochrane [81] and Campbell [82] collaborations are grappling with the need to, and the process of, integrating qualitative research into a systematic review. The qualitative approaches use qualitative data (i.e., expressed in words). All of the emerging synthesis types, as well as the conventional integrative review, incorporate qualitative and quantitative study designs and data.

4.1.3. Research question

Four types of research questions direct inquiry across the different types of syntheses. The first is a well-developed research question that gives direction to the synthesis (e.g., meta-analysis, systematic review, meta-study, concept analysis, rapid review, realist synthesis). The second begins as a broad general question that evolves and becomes more refined over the course of the synthesis (e.g., meta-ethnography, scoping review, meta-narrative, critical interpretive synthesis). In the third type, the synthesis begins with a phenomenon of interest and the question emerges in the analytic process (e.g., grounded formal theory). Lastly, there is no clear question, but rather a general review purpose (e.g., integrative review). Thus, the requirement for a well-defined question cuts across at least three of the synthesis types (e.g., quantitative, qualitative, and emerging).

4.1.4. Quality appraisal

This is a contested issue within and between the four synthesis categories. There are strong proponents of quality appraisal in the quantitative traditions of systematic review and meta-analysis based on the need for strong studies that will not jeopardize validity of the overall findings. Nonetheless, there is no consensus on pre-defined criteria; many scales exist that vary dramatically in composition. This has methodological implications for the credibility of findings [83] .

Specific methodologies from the conventional, qualitative, and emerging categories support quality appraisal but do so with caveats. In conventional integrative reviews appraisal is recommended, but depends on the sampling frame used in the study [18] . In meta-study, appraisal criteria are explicit but quality criteria are used in different ways depending on the specific requirements of the inquiry [54] . Among the emerging syntheses, meta-narrative review developers support appraisal of a study based on criteria from the research tradition of the primary study [67] , [84] – [85] . Realist synthesis similarly supports the use of high quality evidence, but appraisal checklists are viewed with scepticism and evidence is judged based on relevance to the research question and whether a credible inference may be drawn [69] . Like realist, critical interpretive syntheses do not judge quality using standardized appraisal instruments. They will exclude fatally flawed studies, but there is no consensus on what ‘fatally flawed’ means [49] , [71] . Appraisal is based on relevance to the inquiry, not rigor of the study.

There is no agreement on quality appraisal among qualitative meta-ethnographers with some supporting and others refuting the need for appraisal. [60] , [62] . Opponents of quality appraisal are found among authors of qualitative (grounded formal theory and concept analysis) and emerging syntheses (scoping and rapid reviews) because quality is not deemed relevant to the intention of the synthesis; the studies being reviewed are not effectiveness studies where quality is extremely important. These qualitative synthesis are often reviews of theoretical developments where the concept itself is what is important, or reviews that provide quotations from the raw data so readers can make their own judgements about the relevance and utility of the data. For example, in formal grounded theory, the purpose of theory generation and authenticity of data used to generate the theory is not as important as the conceptual category. Inaccuracies may be corrected in other ways, such as using the constant comparative method, which facilitates development of theoretical concepts that are repeatedly found in the data [86] – [87] . For pragmatic reasons, evidence is not assessed in rapid and scoping reviews, in part to produce a timely product. The issue of quality appraisal is unresolved across the terrain of research synthesis and we consider this further in our discussion.

4.2. Purpose

All research syntheses share a common purpose -- to summarize, synthesize, or integrate research findings from diverse studies. This helps readers stay abreast of the burgeoning literature in a field. Our discussion here is at the level of the four categories of synthesis. Beginning with conventional literature syntheses, the overall purpose is to attend to mature topics for the purpose of re-conceptualization or to new topics requiring preliminary conceptualization [14] . Such syntheses may be helpful to consider contradictory evidence, map shifting trends in the study of a phenomenon, and describe the emergence of research in diverse fields [14] . The purpose here is to set the stage for a study by identifying what has been done, gaps in the literature, important research questions, or to develop a conceptual framework to guide data collection and analysis.

The purpose of quantitative systematic reviews is to combine, aggregate, or integrate empirical research to be able to generalize from a group of studies and determine the limits of generalization [27] . The focus of quantitative systematic reviews has been primarily on aggregating the results of studies evaluating the effectiveness of interventions using experimental, quasi-experimental, and more recently, observational designs. Systematic reviews can be done with or without quantitative meta-analysis but a meta-analysis always takes place within the context of a systematic review. Researchers must consider the review's purpose and the nature of their data in undertaking a quantitative synthesis; this will assist in determining the approach.

The purpose of qualitative syntheses is broadly to synthesize complex health experiences, practices, or concepts arising in healthcare environments. There may be various purposes depending on the qualitative methodology. For example, in hermeneutic studies the aim may be holistic explanation or understanding of a phenomenon [42] , which is deepened by integrating the findings from multiple studies. In grounded formal theory, the aim is to produce a conceptual framework or theory expected to be applicable beyond the original study. Although not able to generalize from qualitative research in the statistical sense [88] , qualitative researchers usually do want to say something about the applicability of their synthesis to other settings or phenomena. This notion of ‘theoretical generalization’ has been referred to as ‘transferability’ [89] – [90] and is an important criterion of rigour in qualitative research. It applies equally to the products of a qualitative synthesis in which the synthesis of multiple studies on the same phenomenon strengthens the ability to draw transferable conclusions.

The overarching purpose of emerging syntheses is challenging the more traditional types of syntheses, in part by using data from both quantitative and qualitative studies with diverse designs for analysis. Beyond this, however, each emerging synthesis methodology has a unique purpose. In meta-narrative review, the purpose is to identify different research traditions in the area, synthesize a complex and diverse body of research. Critical interpretive synthesis shares this characteristic. Although a distinctive approach, critical interpretive synthesis utilizes a modification of the analytic strategies of meta-ethnography [61] (e.g., reciprocal translational analysis, refutational synthesis, and lines of argument synthesis) but goes beyond the use of these to bring a critical perspective to bear in challenging the normative or epistemological assumptions in the primary literature [72] – [73] . The unique purpose of a realist synthesis is to amalgamate complex empirical evidence and theoretical understandings within a diverse body of literature to uncover the operative mechanisms and contexts that affect the outcomes of social interventions. In a scoping review, the intention is to find key concepts, examine the range of research in an area, and identify gaps in the literature. The purpose of a rapid review is comparable to that of a scoping review, but done quickly to meet the time-sensitive information needs of policy makers.

4.3. Method

4.3.1. degree of systematization.

There are varying degrees of systematization across the categories of research synthesis. The most systematized are quantitative systematic reviews and meta-analyses. There are clear processes in each with judgments to be made at each step, although there are no agreed upon guidelines for this. The process is inherently subjective despite attempts to develop objective and systematic processes [91] – [92] . Mullen and Ramirez [27] suggest that there is often a false sense of rigour implied by the terms ‘systematic review’ and ‘meta-analysis’ because of their clearly defined procedures.

In comparison with some types of qualitative synthesis, concept analysis is quite procedural. Qualitative meta-synthesis also has defined procedures and is systematic, yet perhaps less so than concept analysis. Qualitative meta-synthesis starts in an unsystematic way but becomes more systematic as it unfolds. Procedures and frameworks exist for some of the emerging types of synthesis [e.g., [50] , [63] , [71] , [93] ] but are not linear, have considerable flexibility, and are often messy with emergent processes [85] . Conventional literature reviews tend not to be as systematic as the other three types. In fact, the lack of systematization in conventional literature synthesis was the reason for the development of more systematic quantitative [17] , [20] and qualitative [45] – [46] , [61] approaches. Some authors in the field [18] have clarified processes for integrative reviews making them more systematic and rigorous, but most conventional syntheses remain relatively unsystematic in comparison with other types.

4.3.2. Complexity of the process

Some synthesis processes are considerably more complex than others. Methodologies with clearly defined steps are arguably less complex than the more flexible and emergent ones. We know that any study encounters challenges and it is rare that a pre-determined research protocol can be followed exactly as intended. Not even the rigorous methods associated with Cochrane [81] systematic reviews and meta-analyses are always implemented exactly as intended. Even when dealing with numbers rather than words, interpretation is always part of the process. Our collective experience suggests that new methodologies (e.g., meta-narrative synthesis and realist synthesis) that integrate different data types and methods are more complex than conventional reviews or the rapid and scoping reviews.

4.4. Product

The products of research syntheses usually take three distinct formats (see Table 1 and Additional File 1 for further details). The first representation is in tables, charts, graphical displays, diagrams and maps as seen in integrative, scoping and rapid reviews, meta-analyses, and critical interpretive syntheses. The second type of synthesis product is the use of mathematical scores. Summary statements of effectiveness are mathematically displayed in meta-analyses (as an effect size), systematic reviews, and rapid reviews (statistical significance).

The third synthesis product may be a theory or theoretical framework. A mid-range theory can be produced from formal grounded theory, meta-study, meta-ethnography, and realist synthesis. Theoretical/conceptual frameworks or conceptual maps may be created in meta-narrative and critical interpretive syntheses, and integrative reviews. Concepts for use within theories are produced in concept analysis. While these three product types span the categories of research synthesis, narrative description and summary is used to present the products resulting from all methodologies.

4.5. Consideration of context

There are diverse ways that context is considered in the four broad categories of synthesis. Context may be considered to the extent that it features within primary studies for the purpose of the review. Context may also be understood as an integral aspect of both the phenomenon under study and the synthesis methodology (e.g., realist synthesis). Quantitative systematic reviews and meta-analyses have typically been conducted on studies using experimental and quasi-experimental designs and more recently observational studies, which control for contextual features to allow for understanding of the ‘true’ effect of the intervention [94] .

More recently, systematic reviews have included covariates or mediating variables (i.e., contextual factors) to help explain variability in the results across studies [27] . Context, however, is usually handled in the narrative discussion of findings rather than in the synthesis itself. This lack of attention to context has been one criticism leveled against systematic reviews and meta-analyses, which restrict the types of research designs that are considered [e.g., [95] ].

When conventional literature reviews incorporate studies that deal with context, there is a place for considering contextual influences on the intervention or phenomenon. Reviews of quantitative experimental studies tend to be devoid of contextual considerations since the original studies are similarly devoid, but context might figure prominently in a literature review that incorporates both quantitative and qualitative studies.

Qualitative syntheses have been conducted on the contextual features of a particular phenomenon [33] . Paterson et al. [54] advise researchers to attend to how context may have influenced the findings of particular primary studies. In qualitative analysis, contextual features may form categories by which the data can be compared and contrasted to facilitate interpretation. Because qualitative research is often conducted to understand a phenomenon as a whole, context may be a focus, although this varies with the qualitative methodology. At the same time, the findings in a qualitative synthesis are abstracted from the original reports and taken to a higher level of conceptualization, thus removing them from the original context.

Meta-narrative synthesis [67] , [84] , because it draws on diverse research traditions and methodologies, may incorporate context into the analysis and findings. There is not, however, an explicit step in the process that directs the analyst to consider context. Generally, the research question guiding the synthesis is an important factor in whether context will be a focus.

More recent iterations of concept analysis [47] , [96] – [97] explicitly consider context reflecting the assumption that a concept's meaning is determined by its context. Morse [47] points out, however, that Wilson's [98] approach to concept analysis, and those based on Wilson [e.g., [45] ], identify attributes that are devoid of context, while Rodgers' [96] , [99] evolutionary method considers context (e.g., antecedents, consequences, and relationships to other concepts) in concept development.

Realist synthesis [69] considers context as integral to the study. It draws on a critical realist logic of inquiry grounded in the work of Bhaskar [100] , who argues that empirical co-occurrence of events is insufficient for inferring causation. One must identify generative mechanisms whose properties are causal and, depending on the situation, may nor may not be activated [94] . Context interacts with program/intervention elements and thus cannot be differentiated from the phenomenon [69] . This approach synthesizes evidence on generative mechanisms and analyzes contextual features that activate them; the result feeds back into the context. The focus is on what works, for whom, under what conditions, why and how [68] .

4.6. Underlying Philosophical and Theoretical Assumptions

When we began our review, we ‘assumed’ that the assumptions underlying synthesis methodologies would be a distinguishing characteristic of synthesis types, and that we could compare the various types on their assumptions, explicit or implicit. We found, however, that many authors did not explicate the underlying assumptions of their methodologies, and it was difficult to infer them. Kirkevold [101] has argued that integrative reviews need to be carried out from an explicit philosophical or theoretical perspective. We argue this should be true for all types of synthesis.

Authors of some emerging synthesis approaches have been very explicit about their assumptions and philosophical underpinnings. An implicit assumption of most emerging synthesis methodologies is that quantitative systematic reviews and meta-analyses have limited utility in some fields [e.g., in public health – [13] , [102] ] and for some kinds of review questions like those about feasibility and appropriateness versus effectiveness [103] – [104] . They also assume that ontologically and epistemologically, both kinds of data can be combined. This is a significant debate in the literature because it is about the commensurability of overarching paradigms [105] but this is beyond the scope of this review.

Realist synthesis is philosophically grounded in critical realism or, as noted above, a realist logic of inquiry [93] , [99] , [106] – [107] . Key assumptions regarding the nature of interventions that inform critical realism have been described above in the section on context. See Pawson et al. [106] for more information on critical realism, the philosophical basis of realist synthesis.

Meta-narrative synthesis is explicitly rooted in a constructivist philosophy of science [108] in which knowledge is socially constructed rather than discovered, and what we take to be ‘truth’ is a matter of perspective. Reality has a pluralistic and plastic character, and there is no pre-existing ‘real world’ independent of human construction and language [109] . See Greenhalgh et al. [67] , [85] and Greenhalgh & Wong [97] for more discussion of the constructivist basis of meta-narrative synthesis.

In the case of purely quantitative or qualitative syntheses, it may be an easier matter to uncover unstated assumptions because they are likely to be shared with those of the primary studies in the genre. For example, grounded formal theory shares the philosophical and theoretical underpinnings of grounded theory, rooted in the theoretical perspective of symbolic interactionism [110] – [111] and the philosophy of pragmatism [87] , [112] – [114] .

As with meta-narrative synthesis, meta-study developers identify constructivism as their interpretive philosophical foundation [54] , [88] . Epistemologically, constructivism focuses on how people construct and re-construct knowledge about a specific phenomenon, and has three main assumptions: (1) reality is seen as multiple, at times even incompatible with the phenomenon under consideration; (2) just as primary researchers construct interpretations from participants' data, meta-study researchers also construct understandings about the primary researchers' original findings. Thus, meta-synthesis is a construction of a construction, or a meta-construction; and (3) all constructions are shaped by the historical, social and ideological context in which they originated [54] . The key message here is that reports of any synthesis would benefit from an explicit identification of the underlying philosophical perspectives to facilitate a better understanding of the results, how they were derived, and how they are being interpreted.

4.7. Unit of Analysis

The unit of analysis for each category of review is generally distinct. For the emerging synthesis approaches, the unit of analysis is specific to the intention. In meta-narrative synthesis it is the storyline in diverse research traditions; in rapid review or scoping review, it depends on the focus but could be a concept; and in realist synthesis, it is the theories rather than programs that are the units of analysis. The elements of theory that are important in the analysis are mechanisms of action, the context, and the outcome [107] .

For qualitative synthesis, the units of analysis are generally themes, concepts or theories, although in meta-study, the units of analysis can be research findings (“meta-data-analysis”), research methods (“meta-method”) or philosophical/theoretical perspectives (“meta-theory”) [54] . In quantitative synthesis, the units of analysis range from specific statistics for systematic reviews to effect size of the intervention for meta-analysis. More recently, some systematic reviews focus on theories [115] – [116] , therefore it depends on the research question. Similarly, within conventional literature synthesis the units of analysis also depend on the research purpose, focus and question as well as on the type of research methods incorporated into the review. What is important in all research syntheses, however, is that the unit of analysis needs to be made explicit. Unfortunately, this is not always the case.

4.8. Strengths and Limitations

In this section, we discuss the overarching strengths and limitations of synthesis methodologies as a whole and then highlight strengths and weaknesses across each of our four categories of synthesis.

4.8.1. Strengths of Research Syntheses in General

With the vast proliferation of research reports and the increased ease of retrieval, research synthesis has become more accessible providing a way of looking broadly at the current state of research. The availability of syntheses helps researchers, practitioners, and policy makers keep up with the burgeoning literature in their fields without which evidence-informed policy or practice would be difficult. Syntheses explain variation and difference in the data helping us identify the relevance for our own situations; they identify gaps in the literature leading to new research questions and study designs. They help us to know when to replicate a study and when to avoid excessively duplicating research. Syntheses can inform policy and practice in a way that well-designed single studies cannot; they provide building blocks for theory that helps us to understand and explain our phenomena of interest.

4.8.2. Limitations of Research Syntheses in General

The process of selecting, combining, integrating, and synthesizing across diverse study designs and data types can be complex and potentially rife with bias, even with those methodologies that have clearly defined steps. Just because a rigorous and standardized approach has been used does not mean that implicit judgements will not influence the interpretations and choices made at different stages.

In all types of synthesis, the quantity of data can be considerable, requiring difficult decisions about scope, which may affect relevance. The quantity of available data also has implications for the size of the research team. Few reviews these days can be done independently, in particular because decisions about inclusion and exclusion may require the involvement of more than one person to ensure reliability.

For all types of synthesis, it is likely that in areas with large, amorphous, and diverse bodies of literature, even the most sophisticated search strategies will not turn up all the relevant and important texts. This may be more important in some synthesis methodologies than in others, but the omission of key documents can influence the results of all syntheses. This issue can be addressed, at least in part, by including a library scientist on the research team as required by some funding agencies. Even then, it is possible to miss key texts. In this review, for example, because none of us are trained in or conduct meta-analyses, we were not even aware that we had missed some new developments in this field such as meta-regression [117] – [118] , network meta-analysis [119] – [121] , and the use of individual patient data in meta-analyses [122] – [123] .

One limitation of systematic reviews and meta-analyses is that they rapidly go out of date. We thought this might be true for all types of synthesis, although we wondered if those that produce theory might not be somewhat more enduring. We have not answered this question but it is open for debate. For all types of synthesis, the analytic skills and the time required are considerable so it is clear that training is important before embarking on a review, and some types of review may not be appropriate for students or busy practitioners.

Finally, the quality of reporting in primary studies of all genres is variable so it is sometimes difficult to identify aspects of the study essential for the synthesis, or to determine whether the study meets quality criteria. There may be flaws in the original study, or journal page limitations may necessitate omitting important details. Reporting standards have been developed for some types of reviews (e.g., systematic review, meta-analysis, meta-narrative synthesis, realist synthesis); but there are no agreed upon standards for qualitative reviews. This is an important area for development in advancing the science of research synthesis.

4.8.3. Strengths and Limitations of the Four Synthesis Types

The conventional literature review and now the increasingly common integrative review remain important and accessible approaches for students, practitioners, and experienced researchers who want to summarize literature in an area but do not have the expertise to use one of the more complex methodologies. Carefully executed, such reviews are very useful for synthesizing literature in preparation for research grants and practice projects. They can determine the state of knowledge in an area and identify important gaps in the literature to provide a clear rationale or theoretical framework for a study [14] , [18] . There is a demand, however, for more rigour, with more attention to developing comprehensive search strategies and more systematic approaches to combining, integrating, and synthesizing the findings.

Generally, conventional reviews include diverse study designs and data types that facilitate comprehensiveness, which may be a strength on the one hand, but can also present challenges on the other. The complexity inherent in combining results from studies with diverse methodologies can result in bias and inaccuracies. The absence of clear guidelines about how to synthesize across diverse study types and data [18] has been a challenge for novice reviewers.

Quantitative systematic reviews and meta-analyses have been important in launching the field of evidence-based healthcare. They provide a systematic, orderly and auditable process for conducting a review and drawing conclusions [25] . They are arguably the most powerful approaches to understanding the effectiveness of healthcare interventions, especially when intervention studies on the same topic show very different results. When areas of research are dogged by controversy [25] or when study results go against strongly held beliefs, such approaches can reduce the uncertainty and bring strong evidence to bear on the controversy.

Despite their strengths, they also have limitations. Systematic reviews and meta-analyses do not provide a way of including complex literature comprising various types of evidence including qualitative studies, theoretical work, and epidemiological studies. Only certain types of design are considered and qualitative data are used in a limited way. This exclusion limits what can be learned in a topic area.

Meta-analyses are often not possible because of wide variability in study design, population, and interventions so they may have a narrow range of utility. New developments in meta-analysis, however, can be used to address some of these limitations. Network meta-analysis is used to explore relative efficacy of multiple interventions, even those that have never been compared in more conventional pairwise meta-analyses [121] , allowing for improved clinical decision making [120] . The limitation is that network meta-analysis has only been used in medical/clinical applications [119] and not in public health. It has not yet been widely accepted and many methodological challenges remain [120] – [121] . Meta-regression is another development that combines meta-analytic and linear regression principles to address the fact that heterogeneity of results may compromise a meta-analysis [117] – [118] . The disadvantage is that many clinicians are unfamiliar with it and may incorrectly interpret results [117] .

Some have accused meta-analysis of combining apples and oranges [124] raising questions in the field about their meaningfulness [25] , [28] . More recently, the use of individual rather than aggregate data has been useful in facilitating greater comparability among studies [122] . In fact, Tomas et al. [123] argue that meta-analysis using individual data is now the gold standard although access to the raw data from other studies may be a challenge to obtain.

The usefulness of systematic reviews in synthesizing complex health and social interventions has also been challenged [102] . It is often difficult to synthesize their findings because such studies are “epistemologically diverse and methodologically complex” [ [69] , p.21]. Rigid inclusion/exclusion criteria may allow only experimental or quasi-experimental designs into consideration resulting in lost information that may well be useful to policy makers for tailoring an intervention to the context or understanding its acceptance by recipients.

Qualitative syntheses may be the type of review most fraught with controversy and challenge, while also bringing distinct strengths to the enterprise. Although these methodologies provide a comprehensive and systematic review approach, they do not generally provide definitive statements about intervention effectiveness. They do, however, address important questions about the development of theoretical concepts, patient experiences, acceptability of interventions, and an understanding about why interventions might work.

Most qualitative syntheses aim to produce a theoretically generalizable mid-range theory that explains variation across studies. This makes them more useful than single primary studies, which may not be applicable beyond the immediate setting or population. All provide a contextual richness that enhances relevance and understanding. Another benefit of some types of qualitative synthesis (e.g., grounded formal theory) is that the concept of saturation provides a sound rationale for limiting the number of texts to be included thus making reviews potentially more manageable. This contrasts with the requirements of systematic reviews and meta-analyses that require an exhaustive search.

Qualitative researchers debate about whether the findings of ontologically and epistemological diverse qualitative studies can actually be combined or synthesized [125] because methodological diversity raises many challenges for synthesizing findings. The products of different types of qualitative syntheses range from theory and conceptual frameworks, to themes and rich descriptive narratives. Can one combine the findings from a phenomenological study with the theory produced in a grounded theory study? Many argue yes, but many also argue no.

Emerging synthesis methodologies were developed to address some limitations inherent in other types of synthesis but also have their own issues. Because each type is so unique, it is difficult to identify overarching strengths of the entire category. An important strength, however, is that these newer forms of synthesis provide a systematic and rigorous approach to synthesizing a diverse literature base in a topic area that includes a range of data types such as: both quantitative and qualitative studies, theoretical work, case studies, evaluations, epidemiological studies, trials, and policy documents. More than conventional literature reviews and systematic reviews, these approaches provide explicit guidance on analytic methods for integrating different types of data. The assumption is that all forms of data have something to contribute to knowledge and theory in a topic area. All have a defined but flexible process in recognition that the methods may need to shift as knowledge develops through the process.

Many emerging synthesis types are helpful to policy makers and practitioners because they are usually involved as team members in the process to define the research questions, and interpret and disseminate the findings. In fact, engagement of stakeholders is built into the procedures of the methods. This is true for rapid reviews, meta-narrative syntheses, and realist syntheses. It is less likely to be the case for critical interpretive syntheses.

Another strength of some approaches (realist and meta-narrative syntheses) is that quality and publication standards have been developed to guide researchers, reviewers, and funders in judging the quality of the products [108] , [126] – [127] . Training materials and online communities of practice have also been developed to guide users of realist and meta-narrative review methods [107] , [128] . A unique strength of critical interpretive synthesis is that it takes a critical perspective on the process that may help reconceptualize the data in a way not considered by the primary researchers [72] .

There are also challenges of these new approaches. The methods are new and there may be few published applications by researchers other than the developers of the methods, so new users often struggle with the application. The newness of the approaches means that there may not be mentors available to guide those unfamiliar with the methods. This is changing, however, and the number of applications in the literature is growing with publications by new users helping to develop the science of synthesis [e.g., [129] ]. However, the evolving nature of the approaches and their developmental stage present challenges for novice researchers.

4.9. When to Use Each Approach

Choosing an appropriate approach to synthesis will depend on the question you are asking, the purpose of the review, and the outcome or product you want to achieve. In Additional File 1 , we discuss each of these to provide guidance to readers on making a choice about review type. If researchers want to know whether a particular type of intervention is effective in achieving its intended outcomes, then they might choose a quantitative systemic review with or without meta-analysis, possibly buttressed with qualitative studies to provide depth and explanation of the results. Alternately, if the concern is about whether an intervention is effective with different populations under diverse conditions in varying contexts, then a realist synthesis might be the most appropriate.

If researchers' concern is to develop theory, they might consider qualitative syntheses or some of the emerging syntheses that produce theory (e.g., critical interpretive synthesis, realist review, grounded formal theory, qualitative meta-synthesis). If the aim is to track the development and evolution of concepts, theories or ideas, or to determine how an issue or question is addressed across diverse research traditions, then meta-narrative synthesis would be most appropriate.

When the purpose is to review the literature in advance of undertaking a new project, particularly by graduate students, then perhaps an integrative review would be appropriate. Such efforts contribute towards the expansion of theory, identify gaps in the research, establish the rationale for studying particular phenomena, and provide a framework for interpreting results in ways that might be useful for influencing policy and practice.

For researchers keen to bring new insights, interpretations, and critical re-conceptualizations to a body of research, then qualitative or critical interpretive syntheses will provide an inductive product that may offer new understandings or challenges to the status quo. These can inform future theory development, or provide guidance for policy and practice.

5. Discussion

What is the current state of science regarding research synthesis? Public health, health care, and social science researchers or clinicians have previously used all four categories of research synthesis, and all offer a suitable array of approaches for inquiries. New developments in systematic reviews and meta-analysis are providing ways of addressing methodological challenges [117] – [123] . There has also been significant advancement in emerging synthesis methodologies and they are quickly gaining popularity. Qualitative meta-synthesis is still evolving, particularly given how new it is within the terrain of research synthesis. In the midst of this evolution, outstanding issues persist such as grappling with: the quantity of data, quality appraisal, and integration with knowledge translation. These topics have not been thoroughly addressed and need further debate.

5.1. Quantity of Data

We raise the question of whether it is possible or desirable to find all available studies for a synthesis that has this requirement (e.g., meta-analysis, systematic review, scoping, meta-narrative synthesis [25] , [27] , [63] , [67] , [84] – [85] ). Is the synthesis of all available studies a realistic goal in light of the burgeoning literature? And how can this be sustained in the future, particularly as the emerging methodologies continue to develop and as the internet facilitates endless access? There has been surprisingly little discussion on this topic and the answers will have far-reaching implications for searching, sampling, and team formation.

Researchers and graduate students can no longer rely on their own independent literature search. They will likely need to ask librarians for assistance as they navigate multiple sources of literature and learn new search strategies. Although teams now collaborate with library scientists, syntheses are limited in that researchers must make decisions on the boundaries of the review, in turn influencing the study's significance. The size of a team may also be pragmatically determined to manage the search, extraction, and synthesis of the burgeoning data. There is no single answer to our question about the possibility or necessity of finding all available articles for a review. Multiple strategies that are situation specific are likely to be needed.

5.2. Quality Appraisal

While the issue of quality appraisal has received much attention in the synthesis literature, scholars are far from resolution. There may be no agreement about appraisal criteria in a given tradition. For example, the debate rages over the appropriateness of quality appraisal in qualitative synthesis where there are over 100 different sets of criteria and many do not overlap [49] . These differences may reflect disciplinary and methodological orientations, but diverse quality appraisal criteria may privilege particular types of research [49] . The decision to appraise is often grounded in ontological and epistemological assumptions. Nonetheless, diversity within and between categories of synthesis is likely to continue unless debate on the topic of quality appraisal continues and evolves toward consensus.

5.3. Integration with Knowledge Translation

If research syntheses are to make a difference to practice and ultimately to improve health outcomes, then we need to do a better job of knowledge translation. In the Canadian Institutes of Health Research (CIHR) definition of knowledge translation (KT), research or knowledge synthesis is an integral component [130] . Yet, with few exceptions [131] – [132] , very little of the research synthesis literature even mentions the relationship of synthesis to KT nor does it discuss strategies to facilitate the integration of synthesis findings into policy and practice. The exception is in the emerging synthesis methodologies, some of which (e.g., realist and meta-narrative syntheses, scoping reviews) explicitly involve stakeholders or knowledge users. The argument is that engaging them in this way increases the likelihood that the knowledge generated will be translated into policy and practice. We suggest that a more explicit engagement with knowledge users in all types of synthesis would benefit the uptake of the research findings.

Research synthesis neither makes research more applicable to practice nor ensures implementation. Focus must now turn seriously towards translation of synthesis findings into knowledge products that are useful for health care practitioners in multiple areas of practice and develop appropriate strategies to facilitate their use. The burgeoning field of knowledge translation has, to some extent, taken up this challenge; however, the research-practice gap continues to plague us [133] – [134] . It is a particular problem for qualitative syntheses [131] . Although such syntheses have an important place in evidence-informed practice, little effort has gone into the challenge of translating the findings into useful products to guide practice [131] .

5.4. Limitations

Our study took longer than would normally be expected for an integrative review. Each of us were primarily involved in our own dissertations or teaching/research positions, and so this study was conducted ‘off the sides of our desks.’ A limitation was that we searched the literature over the course of 4 years (from 2008–2012), necessitating multiple search updates. Further, we did not do a comprehensive search of the literature after 2012, thus the more recent synthesis literature was not systematically explored. We did, however, perform limited database searches from 2012–2015 to keep abreast of the latest methodological developments. Although we missed some new approaches to meta-analysis in our search, we did not find any new features of the synthesis methodologies covered in our review that would change the analysis or findings of this article. Lastly, we struggled with the labels used for the broad categories of research synthesis methodology because of our hesitancy to reinforce the divide between quantitative and qualitative approaches. However, it was very difficult to find alternative language that represented the types of data used in these methodologies. Despite our hesitancy in creating such an obvious divide, we were left with the challenge of trying to find a way of characterizing these broad types of syntheses.

6. Conclusion

Our findings offer methodological clarity for those wishing to learn about the broad terrain of research synthesis. We believe that our review makes transparent the issues and considerations in choosing from among the four broad categories of research synthesis. In summary, research synthesis has taken its place as a form of research in its own right. The methodological terrain has deep historical roots reaching back over the past 200 years, yet research synthesis remains relatively new to public health, health care, and social sciences in general. This is rapidly changing. New developments in systematic reviews and meta-analysis, and the emergence of new synthesis methodologies provide a vast array of options to review the literature for diverse purposes. New approaches to research synthesis and new analytic methods within existing approaches provide a much broader range of review alternatives for public health, health care, and social science students and researchers.

Acknowledgments

KSM is an assistant professor in the Faculty of Nursing at the University of Alberta. Her work on this article was largely conducted as a Postdoctoral Fellow, funded by KRESCENT (Kidney Research Scientist Core Education and National Training Program, reference #KRES110011R1) and the Faculty of Nursing at the University of Alberta.

MM's work on this study over the period of 2008-2014 was supported by a Canadian Institutes of Health Research Applied Public Health Research Chair Award (grant #92365).

We thank Rachel Spanier who provided support with reference formatting.

List of Abbreviations (in Additional File 1 )

Conflict of interest: The authors declare that they have no conflicts of interest in this article.

Authors' contributions: KSM co-designed the study, collected data, analyzed the data, drafted/revised the manuscript, and managed the project.

MP contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.

JB contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.

WN contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.

All authors read and approved the final manuscript.

Additional Files: Additional File 1 – Selected Types of Research Synthesis

This Additional File is our dataset created to organize, analyze and critique the literature that we synthesized in our integrative review. Our results were created based on analysis of this Additional File.

Academic Research: Literature Synthesis

• Research Information Guide
• Research Ethics
• Planning Research
• Study Design
• Data Analysis
• Self-Guided Learning
• Planning Search
• Literature Analysis
• Literature Synthesis
• Information Resources
• Citation Management
• Data Management Plan
• Writing Guidelines
• Academic Writing
• Medical Writing
• Literature Review
• Academic Publication
• Self-Guided learning

Literature synthesis is a process of analyzing various material (textual, statistical, graphical, programming, and other) to understand the studied phenomenon by defining the differences, similarities, relationships, and new ideas, which emerge after interpreting published information, statements, arguments.  Analysis of published literature is a process of asking questions about the information, meaning, goal, and implication of published research. Questions based on -  what, why, when, where, who - provide an insight for analysis of published literature, evaluation, and interpretation.   

Four synthesis types are defined from integrative literature reviews

• Research agenda - identified from the critical analysis through the provocative question or proposition that is used for the next studies
• Conceptual constructs - provides an analysis of the classification of constructs to classify previous research, which provided the direction for new theories
• Alternative models or conceptual frameworks - new thinking about the published topics. Alternative models and conceptions are proposed deriving from critical analysis and synthesis
• Metatheory - integration, and synthesis of literature review provides the foundation for further development of the metatheory across the theoretical domains

Adapted from  Torraco, R. J. (2005). Writing integrative literature reviews: Guidelines and examples.  Human resource development review ,  4 (3), 356-367.

Literature Annotation   

• Passive - light process (e.g., underlining and highlighting)
• Active - thought-process (e.g., questions, observation, summary, and other
• SIFT annotation - symbols, imagery, figurative language, tone, theme)

Questions 

• What are the question and answer?
• What are the main ideas and findings in the article?
• What are the unique methods and techniques?
• What are the patterns in the published literature?
• Is there a collaborative pattern in published research? 
• How this article relevant to the study?
• What are the important concepts?
• Individual interpretation 

Image Annotation 

• Ramisa A, Yan F, Moreno-Noguer F, Mikolajczyk K. BreakingNews: Article Annotation by Image and Text Processing. IEEE Trans Pattern Anal Mach Intell. 2018 May;40(5):1072-1085. Epub 2017 Jun 30. PMID: 28682246.
• Pelka O, Nensa F, Friedrich CM. Annotation of enhanced radiographs for medical image retrieval with deep convolutional neural networks. PLoS One. 2018 Nov 12;13(11):e0206229. PMID: 30419028; PMCID: PMC6231616.

Critically Appraised Topic

Critically Appraised Topic (CAT) - is a “summary of a search and critical appraisal of the literature related to a focused clinical question, which should be kept in an easily accessible place so that it can be used to help make clinical decisions” Dawes, M. (2005). Critically appraised topics and evidence-based medicine journals. Singapore Medical Journal, 46(9), 442.

"5S" Pyramid of Evidence Resources

Graph by Dr. Tmanova. 

Based on the "5S" Pyramid of Evidence Resources from the Indiana University School of Medicine Libraries, the Dartmouth /Yale EBM Page Generator and Haynes RB, ACP J Club 2006;145(3):A8.

What is a Scientific Knowledge?

• Description - observational analysis of the phenomenon by describing its characteristics 
• Analysis - evaluating the studies phenomenon to conclude by testing assumptions through the analytical processes by evaluating studies characteristics 
• Correlation - analysis of relationships among studied phenomena
• Comparison - analysis of the differences and similarities 
• Prediction - predictive forecast on emerging phenomenon based on information and knowledge
• Control/validation - correlation, cause-effect
• Categorization - typology, specification 
• Explanation - study hypothesis, research questions - what, when, why, how, where, who

Selected key concept on scientific information cycle and management at Information Management guide  !  Note: additional resources on the information management and knowledge management are at the designing phase. 

EBM Appraisal Tools

• Critical Appraisal Checklist, CASP
• Critical Appraisal of Qualitative Studies Sheet
• Diagnostics 
• Randomized Controlled Trials (RCT)
• Systematic review
• IDP Review Sheet 

Citation management, annotation, and synthesis 

Tools useful for information and citation management, annotation, and synthesis: 

• EndNote 
• Torraco, R. J. (2016). Writing integrative reviews of the literature: Methods and purposes.  International Journal of Adult Vocational Education and Technology (IJAVET) ,  7 (3), 62-70.
• Klopper, R., Lubbe, S., & Rugbeer, H. (2007). The matrix method of literature review.  Alternation ,  14 (1), 262-276. For more books and journals, browse  DML Book Catalog  and  DML A-Z Journal Catalog , and  DML Database Catalog . Schedule  Consultation  on finding associated sources. 

Services We Offer

Research Information Ask the Library Clinical Support CME / CE Links Copyright Overview Interlibrary Loan Library Instruction Suggest a Purchase

Resources Available

Information by specialty.

Academic Research Information Guide Current Events: COVID-19 Current Events: Diversity, Equity, & Inclusion Internal Medicine Portal Military Sites Pediatrics Portal Professional Reading (In Progress) All Other Portals

Information for YOU

Other questions.

Disclaimer: The appearance of hyperlinks does not constitute endorsement by the Department of Defense, Defense Health Agency of non-U.S. Government sites or the information, products, or services contained therein. Although the Darnall Medical Library, WRNMMC may or may not use these sites as additional distribution channels for Department of Defense information, it does not exercise editorial control over all of the information that you may find at these locations. Such hyperlinks are provided consistent with the stated purpose of this website.

• << Previous: Literature Analysis
• Next: Information Resources >>
• Last Updated: Feb 21, 2024 12:38 PM
• URL: https://wrnmmc.libguides.com/sci_initiative_ar

Darnall Medical Library | Walter Reed NMMC | Building 1, Room 3458 | 8955 Wood Road | Bethesda, MD 20889 | 301-295-1184/85 | Open Monday-Friday, 0700-1630

After-hours access to the library is available to WRNMMC Staff via the CDO at 301-295-4611.

Systematic Reviews & Literature Reviews

Evidence synthesis: part 1.

This blog post is the first in a series exploring Evidence Synthesis . We’re going to start by looking at two types of evidence synthesis: literature reviews and systemic reviews . To help me with this topic I looked at a number of research guides from other institutions, e.g., Cornell University Libraries.

The Key Differences Between a Literature Review and a Systematic Review

Overall, while both literature reviews and systematic reviews involve reviewing existing research literature, systematic reviews adhere to more rigorous and transparent methods to minimize bias and provide robust evidence to inform decision-making in education and other fields. If you are interested in learning about other evidence synthesis this decision tree created by Cornell Libraries (Robinson, n.d.) is a nice visual introduction.

Along with exploring evidence synthesis I am also interested in generative A.I.   I want to be transparent about how I used A.I. to create the table above. I fed this prompt into ChatGPT:

“ List the differences between a literature review and a systemic review for a graduate student of education “

I wanted to see what it would produce. I reformatted the list into a table so that it would be easier to compare and contrast these two reviews much like the one created by Cornell University Libraries (Kibbee, 2024). I think ChatGPT did a pretty good job. I did have to do quite a bit of editing, and make sure that what was created matched what I already knew. There are things ChatGPT left out, for example time frames, and how many people are needed for a systemic review, but we can revisit that in a later post.

Kibbee, M. (2024, April 10). Libguides: A guide to evidence synthesis: Cornell University Library Evidence Synthesis Service. Cornell University Library. https://guides.library.cornell.edu/evidence-synthesis/intro

• Blog Archive 2009-2018
• Library Hours
• Library Salons
• Library Spaces
• Library Workshops
• Reference Desk Questions

Subscribe to the Bank Street Library Blog

• Current Students
• Faculty / Staff
• Paying for College
• Alumni Services
• Partnerships
• Program Finder
• Affordable, Flexible, Accessible
• Distance Education
• All Online Courses & Degrees
• Baccalaureate Online
• Graduate Online
• Start Dates
• Admissions, Costs & Aid
• Faculty and Contacts
• Academic and Career Support
• Student Testimonials
• Distance Education Advantage
• About Hybrid Learning
• Hybrid Learning Degrees
• Admissions Requirements
• Top Destination Courses
• Student Life
• Academic Support
• Academic Calendar
• Faculty & Contacts
• Technical Institute for Environmental Professions
• Term Calendar
• Sustainable Ventures
• Careers & Outcomes
• About Unity
• Office of the President
• Announcing Our Evolution
• Sustainable Achievements & Initiatives
• Reinventing College
• Extended Reality (XR)
• Commencement
• Give to Unity Environmental University
• Institutional Communications
• Unity Environmental University News

Home  /  Unity Distance Education Communications Hub  /  Best Writing Practices  /  Synthesizing Cited Literature

• Synthesizing Cited Literature
• Peer Review Best Practices
• How to Write a Strong Introduction
• Mastering Academic Writing
• Writing for General Audiences
• Using Stories for More Effective Communication

So you found a really great peer-reviewed resource and you want to incorporate ideas from this source into a paper you are writing. Congrats! There are few ways to do this.

The aim of this guide is to help you move from the basics (paraphrasing and summarizing) to synthesizing ideas from multiple sources to write at the scholarly level!

Paraphrasing and Summarizing: Tell us what’s important and keep it brief!

When paraphrasing and summarizing we learned to identify key elements of a text and condense important information into our own words. We are not adding new ideas or arguments, just briefly stating what we read in our own language.

What’s the main objective of summarizing?

Distill a text down into the most critical take away ideas in a succinct fashion

When do we use summarization in our writing?

• Case study and literature reviews
• Popular science articles or blog posts
• Educational videos

In a summary, you succinctly share key points from an individual source in your own words, then move on and do the same for the next source.

Synthesis: Put many ideas together to see them in a new way!

In scholarly and academic writing we aim to synthesize or combine ideas from multiple sources to support our argument and create new ways of thinking around an issue based on past knowledge.

What’s the main objective of synthesizing?

Using past knowledge to support new argument or hypothesis to move thinking in a specific field of research forward

When do we use synthesis in our writing?

• Scientific and academic reports
• Government reports or white papers
• Grant writing

In synthesis, you need to combine information and ideas from multiple sources, adding your own analysis to generate a new, bigger idea.

Further Resources

• Reading strategies: Differences between summarizing and synthesizing
• University of Arizona Writing Center
• Summarizing, Paraphrasing, and Synthesizing Sources

Here are three ideas pulled from three separate but  topically related  peer-reviewed sources written in  summary  form:

• Relevant point from Source 1 : >80% of terrestrial plant species depend on animal-mediated pollination services for reproduction (Ollerton et al. 2011)
• Relevant point from Source 2 : The genetic diversity of pollen transported by pollinators is negatively affected by forest fragmentation (Breed et al. 2015)
• Relevant point from Source 3 : Climate change will negatively impact pollination services by altering flowering times in plant species (Memmott et al. 2007)

Now here are these three relevant points  synthesized  into one sentence, incorporating the writer’s analysis to  generate a new, bigger idea [4] :

Seeing that >80% of terrestrial plant species depend on pollinators for reproduction [1] , and that habitat fragmentation [2]  and climate change [3]  both negatively impact pollination services, animal-pollinated plants may be particularly vulnerable to anthropogenically-induced global change [4] (Memmott et al. 2007, Ollerton et al. 2011, Breed et al. 2015)

Start Your Journey

Looking for Answers

Get More Info

© Unity Environmental University 2024. “America’s Environmental University.™”

Privacy Overview

An Explainable Online Password Strength Estimator

• Conference paper
• First Online: 30 September 2021
• Cite this conference paper

• Liron David 11 &
• Avishai Wool 11  

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 12972))

Included in the following conference series:

• European Symposium on Research in Computer Security

3764 Accesses

4 Citations

Human-chosen passwords are the dominant form of authentication systems. Passwords strength estimators are used to help users avoid picking weak passwords by predicting how many attempts a password cracker would need until it finds a given password.

In this paper we propose a novel password strength estimator, called PESrank, which accurately models the behavior of a powerful password cracker. PESrank calculates the rank of a given password in an optimal descending order of likelihood. PESrank estimates a given password’s rank in fractions of a second—without actually enumerating the passwords—so it is practical for online use. It also has a training time that is drastically shorter than previous methods. Moreover, PESrank is efficiently tweakable to allow model personalization in fractions of a second, without the need to retrain the model; and it is explainable : it is able to provide information on why the password has its calculated rank, and gives the user insight on how to pick a better password.

We implemented PESrank in Python and conducted an extensive evaluation study of it. We also integrated it into the registration page of a course at our university. Even with a model based on 905 million passwords, the response time was well under 1 s, with up to a 1-bit accuracy margin between the upper bound and the lower bound on the rank.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Agrawal, D., Archambeault, B., Rao, J.R., Rohatgi, P.: The EM side-channel(s). In: Cryptographic Hardware and Embedded Systems-CHES 2002, 29–45 (2003)

Google Scholar  

David, L., Wool, A.: Online password guessability via multi-dimensional rank estimation. arXiv preprint arXiv:1912.02551 (2019)

Bogdanov, A., Kizhvatov, I., Manzoor, K., Tischhauser, E., Witteman, M.: Fast and memory-efficient key recovery in side-channel attacks. In: Selected Areas in Cryptography (SAC) (2015)

Burr, W., Dodson, D., Polk, W.: Electronic authentication guideline. Technical report, National Institute of Standards and Technology (2004)

Carnegie Mellon University Password Research Group. Password guessability service (pgs) (2019). https://pgs.ece.cmu.edu/

Castelluccia, C., Chaabane, A., Dürmuth, M., Perito, D.: When privacy meets security: Leveraging personal information for password cracking. arXiv preprint arXiv:1304.6584 (2013)

Castelluccia, C., Dürmuth, M., Perito, D.: Adaptive password-strength meters from markov models. In: NDSS (2012)

Das, A., Bonneau, J., Caesar, M., Borisov, N., Wang, X.: The tangled web of password reuse. In: NDSS 2014, pp. 23–26 (2014)

David, L., Wool, A.: A bounded-space near-optimal key enumeration algorithm for multi-subkey side-channel attacks. In: Handschuh, H. (ed.) CT-RSA 2017. LNCS, vol. 10159, pp. 311–327. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52153-4_18

David, L., Wool, A.: Poly-logarithmic side channel rank estimation via exponential sampling. In: Matsui, M. (ed.) CT-RSA 2019. LNCS, vol. 11405, pp. 330–349. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12612-4_17

David, L., Wool, A.: PESrank Python implementation (2020). https://github.com/lirondavid/PESrank

From very weak to very strong: de Carné de Carnavalet, X., Mannan, M. Analyzing password-strength meters. In: NDSS 14 , 23–26 (2014)

Dell’Amico, M., Filippone, M.: Monte carlo strength evaluation: Fast and reliable password checking. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 158–169. ACM (2015)

Dell’Amico, M., Michiardi, P., Roudier, Y.: Password strength: an empirical analysis. In: 2010 Proceedings IEEE INFOCOM, pp. 1–9. IEEE (2010)

Dürmuth, M., Angelstorf, F., Castelluccia, C., Perito, D., Chaabane, A.: OMEN: Faster Password Guessing Using an Ordered Markov Enumerator. In: Piessens, F., Caballero, J., Bielova, N. (eds.) ESSoS 2015. LNCS, vol. 8978, pp. 119–132. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15618-7_10

Chapter   Google Scholar  

Gandolfi, K., Mourtel, C., Olivier, F.: Electromagnetic Analysis: Concrete Results. In: Koç, Ç.K., Naccache, D., Paar, C. (eds.) CHES 2001. LNCS, vol. 2162, pp. 251–261. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44709-1_21

Glowacz, C., Grosso, V., Poussier, R., Schueth, J., Standaert, F.-X.: Simpler and more efficient rank estimation for side-channel security assessment. In: Fast Software Encryption, pp. 117–129 (2015)

Goodin, D.: Anatomy of a hack: How crackers ransack passwords like "qeadzcwrsfxv1331”. Ars Technica (2013)

Grassi, P.A., et al.: NIST special publication 800–63b: Digital identity guidelines (2017)

Guo, Y., Zhang, Z.: LPSE: lightweight password-strength estimation for password meters. Comput. Secur. 73 , 507–518 (2018)

Article   Google Scholar  

hashcat. Hashcat advanced password recovery (2019)

Hitaj, B., Gasti, P., Ateniese, G., Perez-Cruz, F.: PassGAN: a deep learning approach for password guessing. In: Deng, R.H., Gauthier-Umaña, V., Ochoa, M., Yung, M. (eds.) ACNS 2019. LNCS, vol. 11464, pp. 217–237. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21568-2_11

Houshmand, S., Aggarwal, S.: Using personal information for targeted attacks in grammar based probabilistic password cracking. In: IFIP Advances in Information and Communication Technology, vol. 511 (2017)

Jason. 1.4 billion leaked passwords in over 40GB of data (2019)

Kelley, P.G., et al.: Guess again (and again and again): measuring password strength by simulating password-cracking algorithms. In: 2012 IEEE Symposium on Security and Privacy, pp. 523–537. IEEE (2012)

Kocher, P., Jaffe, J., Jun, B.: Differential power analysis. In: Advances in Cryptology-CRYPTO 1999, pp. 388–397. Springer (1999)

Kocher, P.C.: Timing attacks on implementations of diffie-Hellman, RSA, DSS, and Other Systems. In: Koblitz, N. (ed.) CRYPTO 1996. LNCS, vol. 1109, pp. 104–113. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-68697-5_9

Komanduri, S.: Modeling the adversary to evaluate password strength with limited samples. Ph.D. thesis, Carnegie Mellon University (2016)

Li, Y., Wang, H., Sun, K.: Personal information in passwords and its security implications. IEEE Trans. Inf. Forensics Secur. 12 (10), 2320–2333 (2017)

Li, Z., Han, W., Xu, W.: A large-scale empirical analysis of Chinese web passwords. In: 23rd USENIX Security Symposium, pp. 559–574 (2014)

Ma, J., Yang, W., Luo, M., Li, N.: A study of probabilistic password models. In: 2014 IEEE Symposium on Security and Privacy, pp. 689–704. IEEE (2014)

Martin, D.P., Mather, L., Oswald, E.: Two sides of the same coin: counting and enumerating keys post side-channel attacks revisited. In: Smart, N.P. (ed.) CT-RSA 2018. LNCS, vol. 10808, pp. 394–412. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76953-0_21

Martin, D.P., O’Connell, J.F., Oswald, E., Stam, M.: Counting keys in parallel after a side channel attack. In: Iwata, T., Cheon, J.H. (eds.) ASIACRYPT 2015. LNCS, vol. 9453, pp. 313–337. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-48800-3_13

Melicher, W., et al.: Fast, lean, and accurate: modeling password guessability using neural networks. In: Proceedings of 25th USENIX Security Symposium, pp. 175–191 (2016)

Narayanan, A., Shmatikov, V.: Fast dictionary attacks on passwords using time-space tradeoff. In: Proceedings of the 12th ACM Conference on Computer and Communications Security, pages 364–372. ACM (2005)

OpenWall. John the ripper password cracker (2019)

Pal, B., Daniel, T., Chatterjee, R., Ristenpart, T.: Beyond credential stuffing: Password similarity models using neural networks. In: 2019 IEEE Symposium on Security and Privacy (SP), pp. 417–434. IEEE (2019)

Poussier, R., Standaert, F.-X., Grosso, V.: Simple key enumeration (and rank estimation) using histograms: an integrated approach. In: Gierlichs, B., Poschmann, A.Y. (eds.) CHES 2016. LNCS, vol. 9813, pp. 61–81. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-53140-2_4

Quisquater, J.-J., Samyde, D.: ElectroMagnetic analysis (EMA): measures and counter-measures for smart cards. In: Attali, I., Jensen, T. (eds.) E-smart 2001. LNCS, vol. 2140, pp. 200–210. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45418-7_17

Shay, R., et al.: Encountering stronger password requirements: user attitudes and behaviors. In: Proceedings of the Sixth Symposium on Usable Privacy and Security (SOUPS 2010), p. 2. ACM (2010)

Ur, B., et al.: Design and evaluation of a data-driven password meter. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3775–3786. ACM (2017)

Ur, B., et al.: How does your password measure up? the effect of strength meters on password creation. In 21st USENIX Security Symposium, pp. 65–80 (2012)

Veras, R., Collins, C., Thorpe, J.: On semantic patterns of passwords and their security impact. In NDSS (2014)

Veyrat-Charvillon, N., Gérard, B., Renauld, M., Standaert, F.-X.: An Optimal Key Enumeration Algorithm and Its Application to Side-Channel Attacks. In: Knudsen, L.R., Wu, H. (eds.) SAC 2012. LNCS, vol. 7707, pp. 390–406. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35999-6_25

Wang, D., Zhang, Z., Wang, P., Yan, J., Huang, X.: Targeted online password guessing: an underestimated threat. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1242–1254 (2016)

Weir, M., Aggarwal, S., Collins, M., Stern, H.: Testing metrics for password creation policies by attacking large sets of revealed passwords. In: Proceedings of the 17th ACM conference on Computer and Communications Security, pp. 162–175. ACM (2010)

Weir, M., Aggarwal, S., De Medeiros, B., Glodek, B.: Password cracking using probabilistic context-free grammars. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 391–405. IEEE (2009)

Wheeler, D.: zxcvbn: realistic password strength estimation. Dropbox TechBlog (2012)

Wheeler, D.L.: zxcvbn: low-budget password strength estimation. In: Proceedings of 25th USENIX Security Symposium, pp. 157–173 (2016)

Download references

Acknowledgments

We thank Lujo Bauer and Michael Stroucken for allowing us broad use of the PGS service and assisting us in obtaining the PGS and PGS++ sets.

Author information

Authors and affiliations.

School of Electrical Engineering, Tel Aviv University, Tel Aviv, Israel

Liron David & Avishai Wool

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Liron David or Avishai Wool .

Editor information

Editors and affiliations.

Purdue University, West Lafayette, IN, USA

Elisa Bertino

National Research Center for Applied Cybersecurity ATHENE, Fraunhofer Institute for Secure Information Technology SIT, Darmstadt, Germany

Haya Shulman

National Research Center for Applied Cybersecurity ATHENE , Technische Universität Darmstadt, Fraunhofer Institute for Secure Information Technology SIT, Darmstadt, Germany

Michael Waidner

A Additional Related Work

1.1 a.1 heuristic pure-estimator approaches.

The earliest and probably the most popular methods of password strength estimation are based on LUDS: counts of lower- and uppercase letters, digits and symbols. The de-facto standard for this type of method is the NIST 800-63 standard [ 4 , 19 ]. It proposes to measure password strength in entropy bits, on the basis of some simple rules such as the length of the password and the type of characters used (e.g., lower-case, upper-case, or digits). These methods are known to be quite inaccurate [ 12 ].

Wheeler proposed an advanced password strength estimator [ 48 ], that extends the LUDS approach by including dictionaries, considering l33t speak transformations, keyboard walks, and more. Due to its easy-to-integrate design, it is deployed on many websites. The meter’s accuracy was later backed up by scientific analysis [ 49 ].

Guo et al. [ 20 ] proposed a lightweight client-side meter. It is based on cosine-length and password-edit distance similarity. It transforms a password into a LUDS vector and compares it to a standardized strong-password vector using the aforementioned similarity measures.

Such pure-estimator approaches have the advantage of very fast estimation—typically in fractions of a second—which makes them suitable for online client-side implementation. However, they do not directly model adversarial guessing so their accuracy requires evaluation.

1.2 A.2 Tweakable extensions and variations

Several authors (cf. [ 23 , 29 , 45 ]) extended the PCFG approach to develop systems that also use personal information. The nature of the extensions was to add a new grammar variable for each type of personal information, (e.g., B for birthday, N for name and E for email) which makes the approach tweakable. However these extended methods are impractical for online use for the same reasons PCFG is impractical: they are all generative.

Personalized password strength meters (PPSMs) which rely on previous password knowledge have also been proposed [ 8 , 37 ]: PPSMs warns users when they pick passwords that are vulnerable based on previously compromised passwords. Similarly, PESrank can be personalized based on previous passwords, but also can be personalized based on any kind of user personal information (name, email, etc.).

Recently [ 22 ] introduced PassGAN, an approach that replaces human-generated password rules by machine learning algorithms. The PassGAN uses a Generative Adversarial Network (GAN) to learn the distribution of real passwords from actual password leaks, and to generate password guesses. The authors did not compare their results with previous rank estimators and did not report on the required training time.

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Cite this paper.

David, L., Wool, A. (2021). An Explainable Online Password Strength Estimator. In: Bertino, E., Shulman, H., Waidner, M. (eds) Computer Security – ESORICS 2021. ESORICS 2021. Lecture Notes in Computer Science(), vol 12972. Springer, Cham. https://doi.org/10.1007/978-3-030-88418-5_14

Download citation

DOI : https://doi.org/10.1007/978-3-030-88418-5_14

Published : 30 September 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-88417-8

Online ISBN : 978-3-030-88418-5

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research