what are the sources of literature review in research

Literature review sources

Sources for literature review can be divided into three categories as illustrated in table below. In your dissertation you will need to use all three categories of literature review sources:

Sources for literature review and examples

Generally, your literature review should integrate a wide range of sources such as:

• Books . Textbooks remain as the most important source to find models and theories related to the research area. Research the most respected authorities in your selected research area and find the latest editions of books authored by them. For example, in the area of marketing the most notable authors include Philip Kotler, Seth Godin, Malcolm Gladwell, Emanuel Rosen and others.
• Magazines . Industry-specific magazines are usually rich in scholarly articles and they can be effective source to learn about the latest trends and developments in the research area. Reading industry magazines can be the most enjoyable part of the literature review, assuming that your selected research area represents an area of your personal and professional interests, which should be the case anyways.
• Newspapers can be referred to as the main source of up-to-date news about the latest events related to the research area. However, the proportion of the use of newspapers in literature review is recommended to be less compared to alternative sources of secondary data such as books and magazines. This is due to the fact that newspaper articles mainly lack depth of analyses and discussions.
• Online articles . You can find online versions of all of the above sources. However, note that the levels of reliability of online articles can be highly compromised depending on the source due to the high levels of ease with which articles can be published online. Opinions offered in a wide range of online discussion blogs cannot be usually used in literature review. Similarly, dissertation assessors are not keen to appreciate references to a wide range of blogs, unless articles in these blogs are authored by respected authorities in the research area.

Your secondary data sources may comprise certain amount of grey literature as well. The term grey literature refers to type of literature produced by government, academics, business and industry in print and electronic formats, which is not controlled by commercial publishers. It is called ‘grey’ because the status of the information in grey literature is not certain. In other words, any publication that has not been peer reviewed for publication is grey literature.

The necessity to use grey literature arises when there is no enough peer reviewed publications are available for the subject of your study.

John Dudovskiy

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: May 22, 2024 12:03 PM
• URL: https://libguides.usc.edu/writingguide

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

Prevent plagiarism, run a free check.

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 21 May 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

Literature Review: Lit Review Sources

• Lit Review Types
• GRADE System
• Do a Lit Review
• Citation Justice
• Lit Review Sources

Where do I find information for a literature review?

Research is done by...

...by way of...

...communicated through...

...and organized in...

Types of sources for a review...

• Primary source: Usually a report by the original researchers of a study (unfiltered sources)
• Secondary source: Description or summary by somebody other than the original researcher, e.g. a review article (filtered sources)
• Conceptual/theoretical: Papers concerned with description or analysis of theories or concepts associated with the topic
• Anecdotal/opinion/clinical: Views or opinions about the subject that are not research, review or theoretical (case studies or reports from clinical settings)

A Heirarchy of research information:

Source: SUNY Downstate Medical Center. Medical Research Library of Brooklyn. Evidence Based Medicine Course. A Guide to Research Methods: The Evidence Pyramid: http://library.downstate.edu/EBM2/2100.htm

Life Cycle of Publication

Click image to enlarge

Scientific information has a ‘life cycle’ of its own… it is born as an idea, and then matures and becomes more available to the public. First it appears within the so-called ‘invisible college’ of experts in the field, discussed at conferences and symposia or posted as pre-prints for comments and corrections. Then it appears in the published literature (the primary literature), often as a journal article in a peer-reviewed journal.

Researchers can use the indexing and alerting services of the secondary literature to find out what has been published in a field. Depending on how much information is added by the indexer or abstracter, this may take a few months (though electronic publication has sped up this process). Finally, the information may appear in more popular or reference sources, sometimes called the tertiary literature.

The person beginning a literature search may take this process in reverse: using tertiary sources for general background, then going to the secondary literature to survey what has been published, following up by finding the original (primary) sources, and generating their own research Idea.

(Original content by Wade Lee-Smith)

• << Previous: Citation Justice
• Next: Readings >>
• Last Updated: Feb 16, 2024 8:43 AM
• URL: https://libguides.utoledo.edu/litreview

• Subject guides
• Researching for your literature review
• Literature sources

Researching for your literature review: Literature sources

• Literature reviews
• Before you start
• Develop a search strategy
• Keyword search activity
• Subject search activity
• Combined keyword and subject searching
• Online tutorials
• Apply search limits
• Run a search in different databases
• Supplementary searching
• Save your searches
• Manage results

Scholarly databases

It's important to make a considered decision as to where to search for your review of the literature. It's uncommon for a disciplinary area to be covered by a single publisher, so searching a single publisher platform or database is unlikely to give you sufficient coverage of studies for a review. A good quality literature review involves searching a number of databases individually.

The most common method is to search a combination of large inter-disciplinary databases such as Scopus & Web of Science Core Collection, and some subject-specific databases (such as PsycInfo or EconLit etc.). The Library databases are an excellent place to start for sources of peer-reviewed journal articles.

Depending on disciplinary expectations, or the topic of our review, you may also need to consider sources or search methods other than database searching. There is general information below on searching grey literature. However, due to the wide varieties of grey literature available, you may need to spend some time investigating sources relevant for your specific need.

Grey literature

Grey literature is information which has been published informally or non-commercially (where the main purpose of the producing body is not commercial publishing) or remains unpublished. One example may be Government publications.

Grey literature may be included in a literature review to minimise publication bias . The quality of grey literature can vary greatly - some may be peer reviewed whereas some may not have been through a traditional editorial process.

See the Grey Literature guide for further information on finding and evaluating grey sources.

See the Moodle book MNHS: Systematically searching the grey literature for a comprehensive module on grey literature for systematic reviews.

In certain disciplines (such as physics) there can be a culture of preprints being made available prior to submissions to journals. There has also been a noticeable rise in preprints in medical and health areas in the wake of Covid-19.

If preprints are relevant for you, you can search preprint servers directly. A workaround might be to utilise a search engine such as Google Scholar to search specifically for preprints, as Google Scholar has timely coverage of most preprint servers including ArXiv, RePec, SSRN, BioRxiv, and MedRxiv.

Articles in Press are not preprints, but are accepted manuscripts that are not yet formally published. Articles in Press have been made available as an early access online version of a paper that may not yet have received its final formatting or an allocation of a volume/issue number. As well as being available on a journal's website, Articles in Press are available in databases such as Scopus and Web of Science, and so (unlike preprints) don't necessarily require a separate search.

Conference papers

Conference papers are typically published in conference proceedings (the collection of papers presented at a conference), and may be found on an organisation or Society's website, as a journal, or as a special issue of journal.

In certain disciplines (such as computer science), conference papers may be highly regarded as a form of scholarly communication; the conferences are highly selective, the papers are generally peer reviewed, and papers are published in proceedings affiliated with high-quality publishing houses.

Conference papers may be indexed in a range of scholarly databases. If you only want to see conference papers, database limits can be used to filter results, or try a specific index such as the examples below:

• Conference proceedings citation index. Social science & humanities (CPCI-SSH)
• Conference proceedings citation index. Science (CPCI-S)
• ASME digital library conference proceedings

Honours students and postgraduates may request conference papers through Interlibrary Loans . However, conference paper requests may take longer than traditional article requests as they can be difficult to locate; they may have been only supplied to attendees or not formally published. Sometimes only the abstract is available.

If you are specifically looking for statistical data, try searching for the keyword statistics in a Google Advanced Search and limiting by a relevant site or domain. Below are some examples of sites, or you can try a domain such as .gov for government websites.

Statistical data can be found in the following selected sources:

• Australian Bureau of Statistics
• World Health Organization: Health Data and statistics
• Higher Education Statistics
• UNESCO Institute for Statistics
• Tourism Australia Statistics

For a list of databases that include statistics see: Databases by Subject: Statistics .

If you are specifically looking for information found in newspapers, the library has a large collection of Australian and overseas newspapers, both current and historical.

To search the full-text of newspapers in electronic format use a database such as  Newsbank.

Alternatively, see the Newspapers subject guide for comprehensive information on newspaper sources available via Monash University library and open source databases, as well as searching tips, online videos and more.

Dissertations and theses

The Monash University Library Theses subject guide provides resources and guidelines for locating and accessing theses (dissertations) produced by Monash University as well as other universities in Australia and internationally.  

International theses:

There are a number of theses databases and repositories.

A popular source is:

• ProQuest Dissertations & Theses Global  which predominantly, covers North American masters and doctoral theses. Full text is available for theses added since 1997. 

Australia and New Zealand theses:

Theses that are available in the library can be found using the  Search catalogue.

These include:

• Monash doctoral, masters and a small number of honours theses 
• other Australian and overseas theses that have been purchased for the collection.

Formats include print (not available for loan), microfiche and online (some may have access restrictions).

Trove includes doctoral, masters and some honours theses from all Australian and New Zealand universities, as well as theses awarded elsewhere but held by Australian institutions.

Tips:  

• Type in the title, author surname and/or keywords. Then on the results page refine your search to 'thesis'.
• Alternatively, use the Advanced search and include 'thesis' as a keyword or limi t your result to format = thesis
• << Previous: Literature reviews
• Next: Before you start >>

• Library Guides

The Literature Review

Primary and secondary sources, the literature review: primary and secondary sources.

• Searching the literature
• Grey literature
• Organising and analysing
• Systematic Reviews
• The Literature Review Toolbox

On this page

• Primary vs secondary sources: The differences explained 

Can something be both a primary and secondary source?

Research for your literature review can be categorised as either primary or secondary in nature. The simplest definition of primary sources is either original information (such as survey data) or a first person account of an event (such as an interview transcript). Whereas secondary sources are any publshed or unpublished works that describe, summarise, analyse, evaluate, interpret or review primary source materials. Secondary sources can incorporate primary sources to support their arguments.

Ideally, good research should use a combination of both primary and secondary sources. For example, if a researcher were to investigate the introduction of a law and the impacts it had on a community, he/she might look at the transcripts of the parliamentary debates as well as the parliamentary commentary and news reporting surrounding the laws at the time. 

Examples of primary and secondary sources

Primary vs secondary sources: The differences explained

Finding primary sources

• VU Special Collections  - The Special Collections at Victoria University Library are a valuable research resource. The Collections have strong threads of radical literature, particularly Australian Communist literature, much of which is rare or unique. Women and urban planning also feature across the Collections. There are collections that give you a picture of the people who donated them like Ray Verrills, John McLaren, Sir Zelman Cowen, and Ruth & Maurie Crow. Other collections focus on Australia's neighbours – PNG and Timor-Leste.
• POLICY - Sharing the latest in policy knowledge and evidence, this database supports enhanced learning, collaboration and contribution.
• Indigenous Australia  -  The Indigenous Australia database represents the collections of the Aboriginal and Torres Strait Islander Commission Library.
• Australian Heritage Bibliography - Aboriginal and Torres Strait Islander Subset (AHB-ATSIS)  - AHB is a bibliographic database that indexes and abstracts articles from published and unpublished material on Australia's natural and cultural environment. The AHB-ATSIS subset contains records that specifically relate to the Aboriginal and Torres Strait Islander peoples.include journal articles, unpublished reports, books, videos and conference proceedings from many different sources around Australia. Emphasis is placed on reports written or commissioned by government and non-government heritage agencies throughout the country.
• ATSIhealth  - The Aboriginal and Torres Strait Islander Health Bibliography (ATSIhealth), compiled by Neil Thomson and Natalie Weissofner at the School of Indigenous Australian Studies, Kurongkurl Katitjin, Edith Cowan University, is a bibliographic database that indexes published and unpublished material on Australian Indigenous health. Source documents include theses, unpublished articles, government reports, conference papers, abstracts, book chapters, books, discussion and working papers, and statistical documents. 
• National Archive of Australia  - The National Archives of Australia holds the memory of our nation and keeps vital Australian Government records safe. 
• National Library of Australia: Manuscripts  - Manuscripts collection that is wide ranging and provides rich evidence of the lives and activities of Australians who have shaped our society.
• National Library of Australia: Printed ephemera  - The National Library has been selectively collecting Australian printed ephemera since the early 1960s as a record of Australian life and social customs, popular culture, national events, and issues of national concern.
• National Library of Australia: Oral history and folklore - The Library’s Oral History and Folklore Collection dates back to the 1950’s and includes a rich and diverse collection of interviews and recordings with Australians from all walks of life.
• Historic Hansard - Commonwealth of Australia parliamentary debates presented in an easy-to-read format for historians and other lovers of political speech.
• The Old Bailey Online - A fully searchable edition of the largest body of texts detailing the lives of non-elite people ever published, containing 197,745 criminal trials held at London's central criminal court.

Whether or not a source can be considered both primary and  secondary, depends on the context. In some instances, material may act as a secondary source for one research area, and as a primary source for another. For example, Niccolò Machiavelli’s The Prince , published in 1513, is an important secondary source for any study of the various Renaissance princes in the Medici family; but the same book is also a primary source for the political thought that was characteristic of the sixteenth century because it reflects the attitudes of a person living in the 1500s.

Source: Craver, 1999, as cited in University of South Australia Library. (2021, Oct 6).  Can something be a primary and secondary source?.  University of South Australia Library. https://guides.library.unisa.edu.au/historycultural/sourcetypes

• << Previous: Overview
• Next: Searching the literature >>
• Last Updated: Mar 27, 2024 2:06 PM
• URL: https://libraryguides.vu.edu.au/the-literature-review

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Literature Reviews

What this handout is about.

This handout will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.

Introduction

OK. You’ve got to write a literature review. You dust off a novel and a book of poetry, settle down in your chair, and get ready to issue a “thumbs up” or “thumbs down” as you leaf through the pages. “Literature review” done. Right?

Wrong! The “literature” of a literature review refers to any collection of materials on a topic, not necessarily the great literary texts of the world. “Literature” could be anything from a set of government pamphlets on British colonial methods in Africa to scholarly articles on the treatment of a torn ACL. And a review does not necessarily mean that your reader wants you to give your personal opinion on whether or not you liked these sources.

What is a literature review, then?

A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period.

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

But how is a literature review different from an academic research paper?

The main focus of an academic research paper is to develop a new argument, and a research paper is likely to contain a literature review as one of its parts. In a research paper, you use the literature as a foundation and as support for a new insight that you contribute. The focus of a literature review, however, is to summarize and synthesize the arguments and ideas of others without adding new contributions.

Why do we write literature reviews?

Literature reviews provide you with a handy guide to a particular topic. If you have limited time to conduct research, literature reviews can give you an overview or act as a stepping stone. For professionals, they are useful reports that keep them up to date with what is current in the field. For scholars, the depth and breadth of the literature review emphasizes the credibility of the writer in his or her field. Literature reviews also provide a solid background for a research paper’s investigation. Comprehensive knowledge of the literature of the field is essential to most research papers.

Who writes these things, anyway?

Literature reviews are written occasionally in the humanities, but mostly in the sciences and social sciences; in experiment and lab reports, they constitute a section of the paper. Sometimes a literature review is written as a paper in itself.

Let’s get to it! What should I do before writing the literature review?

If your assignment is not very specific, seek clarification from your instructor:

• Roughly how many sources should you include?
• What types of sources (books, journal articles, websites)?
• Should you summarize, synthesize, or critique your sources by discussing a common theme or issue?
• Should you evaluate your sources?
• Should you provide subheadings and other background information, such as definitions and/or a history?

Find models

Look for other literature reviews in your area of interest or in the discipline and read them to get a sense of the types of themes you might want to look for in your own research or ways to organize your final review. You can simply put the word “review” in your search engine along with your other topic terms to find articles of this type on the Internet or in an electronic database. The bibliography or reference section of sources you’ve already read are also excellent entry points into your own research.

Narrow your topic

There are hundreds or even thousands of articles and books on most areas of study. The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the material. Your instructor will probably not expect you to read everything that’s out there on the topic, but you’ll make your job easier if you first limit your scope.

Keep in mind that UNC Libraries have research guides and to databases relevant to many fields of study. You can reach out to the subject librarian for a consultation: https://library.unc.edu/support/consultations/ .

And don’t forget to tap into your professor’s (or other professors’) knowledge in the field. Ask your professor questions such as: “If you had to read only one book from the 90’s on topic X, what would it be?” Questions such as this help you to find and determine quickly the most seminal pieces in the field.

Consider whether your sources are current

Some disciplines require that you use information that is as current as possible. In the sciences, for instance, treatments for medical problems are constantly changing according to the latest studies. Information even two years old could be obsolete. However, if you are writing a review in the humanities, history, or social sciences, a survey of the history of the literature may be what is needed, because what is important is how perspectives have changed through the years or within a certain time period. Try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to consider what is currently of interest to scholars in this field and what is not.

Strategies for writing the literature review

Find a focus.

A literature review, like a term paper, is usually organized around ideas, not the sources themselves as an annotated bibliography would be organized. This means that you will not just simply list your sources and go into detail about each one of them, one at a time. No. As you read widely but selectively in your topic area, consider instead what themes or issues connect your sources together. Do they present one or different solutions? Is there an aspect of the field that is missing? How well do they present the material and do they portray it according to an appropriate theory? Do they reveal a trend in the field? A raging debate? Pick one of these themes to focus the organization of your review.

Convey it to your reader

A literature review may not have a traditional thesis statement (one that makes an argument), but you do need to tell readers what to expect. Try writing a simple statement that lets the reader know what is your main organizing principle. Here are a couple of examples:

The current trend in treatment for congestive heart failure combines surgery and medicine. More and more cultural studies scholars are accepting popular media as a subject worthy of academic consideration.

Consider organization

You’ve got a focus, and you’ve stated it clearly and directly. Now what is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? And in what order should you present them? Develop an organization for your review at both a global and local level:

First, cover the basic categories

Just like most academic papers, literature reviews also must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper. The following provides a brief description of the content of each:

• Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern.
• Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).
• Conclusions/Recommendations: Discuss what you have drawn from reviewing literature so far. Where might the discussion proceed?

Organizing the body

Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further.

To help you come up with an overall organizational framework for your review, consider the following scenario:

You’ve decided to focus your literature review on materials dealing with sperm whales. This is because you’ve just finished reading Moby Dick, and you wonder if that whale’s portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 1980’s. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 1968 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick, so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel.

Now consider some typical ways of organizing the sources into a review:

• Chronological: If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (1968), and finally the biology articles (1980s) and the recent articles on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus.
• By publication: Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies.
• By trend: A better way to organize the above sources chronologically is to examine the sources under another trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Under this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote a century apart.
• Thematic: Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For instance, the sperm whale review could focus on the development of the harpoon for whale hunting. While the study focuses on one topic, harpoon technology, it will still be organized chronologically. The only difference here between a “chronological” and a “thematic” approach is what is emphasized the most: the development of the harpoon or the harpoon technology.But more authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as “evil” in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.
• Methodological: A methodological approach differs from the two above in that the focusing factor usually does not have to do with the content of the material. Instead, it focuses on the “methods” of the researcher or writer. For the sperm whale project, one methodological approach would be to look at cultural differences between the portrayal of whales in American, British, and French art work. Or the review might focus on the economic impact of whaling on a community. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed. Once you’ve decided on the organizational method for the body of the review, the sections you need to include in the paper should be easy to figure out. They should arise out of your organizational strategy. In other words, a chronological review would have subsections for each vital time period. A thematic review would have subtopics based upon factors that relate to the theme or issue.

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

• Current Situation: Information necessary to understand the topic or focus of the literature review.
• History: The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Methods and/or Standards: The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

Begin composing

Once you’ve settled on a general pattern of organization, you’re ready to write each section. There are a few guidelines you should follow during the writing stage as well. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as “writer,” “pedestrian,” and “persons.” The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine “generic” condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, “Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense,” Women and Language19:2).

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review’s focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though, if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton’s study, but then synthesize it by rephrasing the study’s significance and relating it to their own work.

Keep your own voice

While the literature review presents others’ ideas, your voice (the writer’s) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author’s information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil’s. For more information, please see our handout on plagiarism .

Revise, revise, revise

Draft in hand? Now you’re ready to revise. Spending a lot of time revising is a wise idea, because your main objective is to present the material, not the argument. So check over your review again to make sure it follows the assignment and/or your outline. Then, just as you would for most other academic forms of writing, rewrite or rework the language of your review so that you’ve presented your information in the most concise manner possible. Be sure to use terminology familiar to your audience; get rid of unnecessary jargon or slang. Finally, double check that you’ve documented your sources and formatted the review appropriately for your discipline. For tips on the revising and editing process, see our handout on revising drafts .

Works consulted

We consulted these works while writing this handout. This is not a comprehensive list of resources on the handout’s topic, and we encourage you to do your own research to find additional publications. Please do not use this list as a model for the format of your own reference list, as it may not match the citation style you are using. For guidance on formatting citations, please see the UNC Libraries citation tutorial . We revise these tips periodically and welcome feedback.

Anson, Chris M., and Robert A. Schwegler. 2010. The Longman Handbook for Writers and Readers , 6th ed. New York: Longman.

Jones, Robert, Patrick Bizzaro, and Cynthia Selfe. 1997. The Harcourt Brace Guide to Writing in the Disciplines . New York: Harcourt Brace.

Lamb, Sandra E. 1998. How to Write It: A Complete Guide to Everything You’ll Ever Write . Berkeley: Ten Speed Press.

Rosen, Leonard J., and Laurence Behrens. 2003. The Allyn & Bacon Handbook , 5th ed. New York: Longman.

Troyka, Lynn Quittman, and Doug Hesse. 2016. Simon and Schuster Handbook for Writers , 11th ed. London: Pearson.

You may reproduce it for non-commercial use if you use the entire handout and attribute the source: The Writing Center, University of North Carolina at Chapel Hill

Make a Gift

Literature Reviews

• 3. Search the literature
• Getting started
• Types of reviews
• 1. Define your research question
• 2. Plan your search

Creating a search strategy

Document your search, rinse and repeat, grey literature, grey literature sources.

• 4. Organize your results
• 5. Synthesize your findings
• 6. Write the review
• Artificial intelligence (AI) tools
• Thompson Writing Studio This link opens in a new window
• Need to write a systematic review? This link opens in a new window

Contact a Librarian

Ask a Librarian

• Thesauri / subject headings
• Ask a librarian!

When conducting a literature review, it is imperative to brainstorm a list of keywords related to your topic. Examining the titles, abstracts, and author-provided keywords of pertinent literature is a great starting point.

Things to keep in mind:

• Alternative spellings (e.g., behavior and behaviour)
• Variants and truncation (e.g., environ* = environment, environments, environmental, environmentally)
• Synonyms (e.g., alternative fuels >> electricity, ethanol, natural gas, hydrogen fuel cells)
• Phrases and double quotes (e.g., "food security" versus food OR security) 

One way to visually organize your thoughts is to create a table where each column represents one concept in your research question. For example, if your research question is...

Does social media play a role in the number of eating disorder diagnoses in college-aged women?

...then your table might look something like this:

Generative AI tools, such as chatbots, are actually quite helpful at this stage when it comes to brainstorming synonyms and other related terms. You can also look at author-provided keywords from benchmark articles (key papers related to your topic), databases' controlled vocabularies, or do a preliminary search and look through abstracts from relevant papers.

Generative AI tools :  ChatGPT ,  Google Gemini (formerly Bard) ,  Claude , Microsoft Copilot

For more information on how to incorporate AI tools into your research, check out the section on  AI Tools .

Boolean searching yields more effective and precise search results. Boolean operators include  AND , OR , and NOT . These are logic-based words that help search engines narrow down or broaden search results.

Using the Operators

The Boolean operator  AND  tells a search engine that you want to find information about two (or more) search terms. For example, sustainability AND plastics. This will narrow down your search results because the search engine will only bring back results that include both search terms.

The Boolean operator  OR  tells the search engine that you want to find information about either search term you've entered. For example, sustainability OR plastics. This will broaden your search results because the search engine will bring back any results that have either search term in them.

The Boolean operator  NOT  tells the search engine that you want to find information about the first search term, but nothing about the second. For example, sustainability NOT plastics. This will narrow down your research results because the search engine will bring back only resources about the first search term (sustainability), but exclude any resources that include the second search term (plastics).

Some databases offer a thesaurus , controlled vocabulary , or list of available subject headings that are assigned to each of its records, either by an indexer or by the original author. The use of controlled vocabularies is a highly effective, efficient, and deliberate way of comprehensively discovering the material within a field of study.

• APA Thesaurus of Psychological Index Terms  (via PsycInfo database)
• Medical Subject Headings (MeSH)  (via PubMed)
• List of ProQuest database thesauri

Web of Science's Core Collection offers a list of subject categories that are searchable by the  Web of Science  Categories field .

Reach out to a Duke University Libraries librarian at [email protected] or use the chat function.

Information animated icons created by Freepik - Flaticon

While not essential for traditional literature reviews, documenting your search can help you:

• Keep track of what you've done so that you don't repeat unproductive searches
• Reuse successful search strategies for future papers
• Help you describe your search process for manuscripts
• Justify your search process

Documenting your search will help you stay organized and save time when tweaking your search strategy. This is a critical step for rigorous review papers, such as  systematic reviews .

One of the easiest ways to document your search strategy is to use a table like this:

If you find that you're receiving too many results , try the following tips:

• Use more AND operators to connect keywords/concepts in order to narrow down your search.
• Use more specific keywords rather than an umbrella term (e.g., "formaldehyde" instead of "chemical").
• Use quotation marks (" ") to search an entire phrase.
• Use filters such as date, language, document type, etc.
• Examine your research question to see if it's still too broad.

On the other hand, if you're not receiving enough results :

• Use more OR operators to connect related terms and bring in additional results.
• Use more generic terms (e.g., "acetone" instead of "dimethyl ketone") or fewer keywords altogether.
• Use wildcard operators (*) to expand your results (e.g., toxi* searches toxic, toxin, toxins).
• Examine your research question to see if it's too narrow.

Grey (or gray) literature refers to research materials and publications that are not commercially published or widely distributed through traditional academic channels. If you are tasked with doing an intensive type of review or evidence synthesis, or you are involved in research related to policy-making, you will likely want to include searching for grey literature.   This type of literature includes:

• working papers
• government documents
• conference proceedings
• theses and dissertations
• white papers...etc.

For more information on grey literature, please see our Grey Literature guide .

• Public policy
• Health/medicine
• Statistics/data
• Thesis/dissertation
• ProQuest Central This link opens in a new window Search for articles from thousands of scholarly journals
• OpenDOAR OpenDOAR is the quality-assured, global Directory of Open Access Repositories. We host repositories that provide free, open access to academic outputs and resources.
• OAIster A catalog of millions of open-access resources harvested from WorldCat.
• GreySource An index of repository hyperlinks across all disciplines.
• Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. We conduct public opinion polling, demographic research, content analysis and other data-driven social science research.
• The World Bank The World Bank is a vital source of financial and technical assistance to developing countries around the world.
• World Health Organization (WHO): IRIS IRIS is the Institutional Repository for Information Sharing, a digital library of WHO's published material and technical information in full text produced since 1948.
• PolicyArchive PolicyArchive is a comprehensive digital library of public policy research containing over 30,000 documents.
• Kaiser Family Foundation KFF is the independent source for health policy research, polling, and journalism. Our mission is to serve as a nonpartisan source of information for policymakers, the media, the health policy community, and the public.
• MedNar Mednar is a free, medically-focused deep web search engine that uses Explorit Everywhere!, an advanced search technology by Deep Web Technologies. As an alternative to Google, Mednar accelerates your research with a search of authoritative public and deep web resources, returning the most relevant results to one easily navigable page.
• Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.

For more in-depth information related to grey literature searching in medicine, please visit Duke Medical Center Library's guide .

• Education Resources Information Center (ERIC) ERIC is a comprehensive, easy-to-use, searchable, Internet-based bibliographic and full-text database of education research and information. It is sponsored by the Institute of Education Sciences within the U.S. Department of Education.
• National Center for Occupational Safety and Health (NIOSHTIC-2) NIOSHTIC-2 is a searchable bibliographic database of occupational safety and health publications, documents, grant reports, and other communication products supported in whole or in part by NIOSH (CDC).
• National Technical Information Service (NTIS) The National Technical Information Service acquires, indexes, abstracts, and archives the largest collection of U.S. government-sponsored technical reports in existence. The NTRL offers online, free and open access to these authenticated government technical reports.
• Science.gov Science.gov provides access to millions of authoritative scientific research results from U.S. federal agencies.
• GovInfo GovInfo is a service of the United States Government Publishing Office (GPO), which is a Federal agency in the legislative branch. GovInfo provides free public access to official publications from all three branches of the Federal Government.
• CQ Press Library This link opens in a new window Search for analysis of Congressional actions and US political issues. Includes CQ Weekly and CQ Researcher.
• Congressional Research Service (CRS) This collection provides the public with access to research products produced by the Congressional Research Service (CRS) for the United States Congress.

Please see the Data Sets and Collections page from our Statistical Sciences guide.

• arXiv arXiv is a free distribution service and an open-access archive for nearly 2.4 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. Materials on this site are not peer-reviewed by arXiv.
• OSF Preprints OSF Preprints is an open access option for discovering multidisciplinary preprints as well as postprints and working papers.
• << Previous: 2. Plan your search
• Next: 4. Organize your results >>
• Last Updated: May 17, 2024 8:42 AM
• URL: https://guides.library.duke.edu/litreviews

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

• Library Hours
• Strategic Plan
• Giving to the Libraries
• Jobs at the Libraries
• Find Your Librarian
• View All →
• Google Scholar
• Research Guides
• Textbook/Reserves
• Government Documents
• Get It For Me
• Print/Copy/Scan
• Renew Materials
• Study Rooms
• Use a Computer
• Borrow Tech Gear
• Student Services
• Faculty Services
• Users with Disabilities
• Visitors & Alumni
• Special Collections
• Find Information

Basics of Systematic Reviews

• About Systematic Review

Types of Reviews

Literature review.

Collects key sources on a topic and discusses those sources in conversation with each other

• Standard for research articles in most disciplines
• Tells the reader what is known, or not known, about a particular issue, topic, or subject
• Demonstrates knowledge and understanding of a topic
• Establishes context or background for a case or argument
• Helps develop the author’s ideas and perspective

Rapid Review

Thorough methodology but with process limitations in place to expeditethe completion of a review.

• For questions that require timely answers
• 3-4 months vs. 12-24 months
• Limitations - scope, comprehensiveness bias, and quality of appraisal
• Discusses potential effects that the limited methods may have had on results

Scoping Review

Determine the scope or coverage of a body of literature on a given topic and give clear indication of the volume of literature and studies available as well as an overview of its focus.

• Identify types of available evidence in a given field
• Clarify key concepts/definitions in the literature
• Examine how research is conducted on a certain topic or field
• Identify key factors related to a concept
• Key difference is focus
• Identify and analyze knowledge gaps

Systematic Review

Attempts to identify, appraise, and summarize all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question.

• clearly defined question with inclusion/exclusion criteria
• rigorous and systematic search of the literature
• thorough screening of results
• data extraction and management
• analysis and interpretation of results
• risk of bias assessment of included studies

Meta-Analysis

Used to systematically synthesize or merge the findings of single, independent studies, using statistical methods to calculate an overall or ‘absolute’ effect.

• Combines results from multiple empirical studies
• Requires systematic review first
• Use well recognized, systematic methods to account for differences in sample size, variability (heterogeneity) in study approach and findings (treatment effects)
• Test how sensitive their results are to their own systematic review protocol

For additional types of reviews please see these articles:

• Sutton, A., Clowes, M., Preston, L. and Booth, A. (2019), Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J, 36: 202-222. https://doi.org/10.1111/hir.12276
• Grant, M.J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
• << Previous: About Systematic Review
• Next: Sources >>
• Last Updated: May 17, 2024 10:04 AM
• URL: https://libguides.utsa.edu/systematicreview
• Library Locations
• Staff Directory
• 508 Compliance
• Site Search
• © The University of Texas at San Antonio
• Information: 210-458-4011
• Campus Alerts
• Required Links
• UTSA Policies
• Report Fraud

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Mark Access Health Policy
• v.11(1); 2023
• PMC10392303

Rapid literature review: definition and methodology

Beata smela.

a Assignity, Cracow, Poland

Mondher Toumi

b Public Health Department, Aix-Marseille University, Marseille, France

Karolina Świerk

Clement francois, małgorzata biernikiewicz.

c Studio Slowa, Wroclaw, Poland

Emilie Clay

d Clever-Access, Paris, France

Laurent Boyer

Introduction: A rapid literature review (RLR) is an alternative to systematic literature review (SLR) that can speed up the analysis of newly published data. The objective was to identify and summarize available information regarding different approaches to defining RLR and the methodology applied to the conduct of such reviews.

Methods: The Medline and EMBASE databases, as well as the grey literature, were searched using the set of keywords and their combination related to the targeted and rapid review, as well as design, approach, and methodology. Of the 3,898 records retrieved, 12 articles were included.

Results: Specific definition of RLRs has only been developed in 2021. In terms of methodology, the RLR should be completed within shorter timeframes using simplified procedures in comparison to SLRs, while maintaining a similar level of transparency and minimizing bias. Inherent components of the RLR process should be a clear research question, search protocol, simplified process of study selection, data extraction, and quality assurance.

Conclusions: There is a lack of consensus on the formal definition of the RLR and the best approaches to perform it. The evidence-based supporting methods are evolving, and more work is needed to define the most robust approaches.

Introduction

A systematic literature review (SLR) summarizes the results of all available studies on a specific topic and provides a high level of evidence. Authors of the SLR have to follow an advanced plan that covers defining a priori information regarding the research question, sources they are going to search, inclusion criteria applied to choose studies answering the research question, and information regarding how they are going to summarize findings [ 1 ].

The rigor and transparency of SLRs make them the most reliable form of literature review [ 2 ], providing a comprehensive, objective summary of the evidence for a given topic [ 3 , 4 ]. On the other hand, the SLR process is usually very time-consuming and requires a lot of human resources. Taking into account a high increase of newly published data and a growing need to analyze information in the fastest possible way, rapid literature reviews (RLRs) often replace standard SLRs.

There are several guidelines on the methodology of RLRs [ 5–11 ]; however, only recently, one publication from 2021 attempted to construct a unified definition [ 11 ]. Generally, by RLRs, researchers understand evidence synthesis during which some of the components of the systematic approach are being used to facilitate answering a focused research question; however, scope restrictions and a narrower search strategy help to make the project manageable in a shorter time and to get the key conclusions faster [ 4 ].

The objective of this research was to collect and summarize available information on different approaches to the definition and methodology of RLRs. An RLR has been run to capture publications providing data that fit the project objective.

To find publications reporting information on the methodology of RLRs, searches were run in the Medline and EMBASE databases in November 2022. The following keywords were searched for in titles and abstracts: ‘targeted adj2 review’ OR ‘focused adj2 review’ OR ‘rapid adj2 review’, and ‘methodology’ OR ‘design’ OR ‘scheme’ OR ‘approach’. The grey literature was identified using Google Scholar with keywords including ‘targeted review methodology’ OR ‘focused review methodology’ OR ‘rapid review methodology’. Only publications in English were included, and the date of publication was restricted to year 2016 onward in order to identify the most up-to-date literature. The reference lists of each included article were searched manually to obtain the potentially eligible articles. Titles and abstracts of the retrieved records were first screened to exclude articles that were evidently irrelevant. The full texts of potentially relevant papers were further reviewed to examine their eligibility.

A pre-defined Excel grid was developed to extract the following information related to the methodology of RLR from guidelines:

• Definition,
• Research question and searches,
• Studies selection,
• Data extraction and quality assessment,
• Additional information.

There was no restriction on the study types to be analyzed; any study reporting on the methodology of RLRs could be included: reviews, practice guidelines, commentaries, and expert opinions on RLR relevant to healthcare policymakers or practitioners. The data extraction and evidence summary were conducted by one analyst and further examined by a senior analyst to ensure that relevant information was not omitted. Disagreements were resolved by discussion and consensus.

Studies selection

A total of 3,898 records (3,864 articles from a database search and 34 grey literature from Google Scholar) were retrieved. After removing duplicates, titles and abstracts of 3,813 articles were uploaded and screened. The full texts of 43 articles were analyzed resulting in 12 articles selected for this review, including 7 guidelines [ 5–11 ] on the methodology of RLRs, together with 2 papers summarizing the results of the Delphi consensus on the topic [ 12 , 13 ], and 3 publications analyzing and assessing different approaches to RLRs [ 4 , 14 , 15 ].

Overall, seven guidelines were identified: from the World Health Organization (WHO) [ 5 ], National Collaborating Centre for Methods and Tools (NCCMT) [ 7 ], the UK government [ 8 ], the Oxford Centre for Evidence Based Medicine [ 9 ], the Cochrane group [ 6 , 11 ], and one multi-national review [ 10 ]. Among the papers that did not describe the guidelines, Gordon et al. [ 4 ] proposed 12 tips for conducting a rapid review in the right settings and discussed why these reviews may be more beneficial in some circumstances. The objective of work conducted by Tricco et al. [ 13 ] and Pandor et al. [ 12 ] was to collect and compare perceptions of rapid reviews from stakeholders, including researchers, policymakers, industry, journal editors, and healthcare providers, and to reach a consensus outlining the domains to consider when deciding on approaches for RLRs. Haby et al. [ 14 ] run a rapid review of systematic reviews and primary studies to find out the best way to conduct an RLR in health policy and practice. In Tricco et al. (2022) [ 15 ], JBI position statement for RLRs is presented.

From all the seven identified guidelines information regarding definitions the authors used for RLRs, approach to the PICOS criteria and search strategy development, studies selection, data extractions, quality assessment, and reporting were extracted.

Cochrane Rapid Reviews Methods Group developed methods guidance based on scoping review of the underlying evidence, primary methods studies conducted, as well as surveys sent to Cochrane representative and discussion among those with expertise [ 11 ]. They analyzed over 300 RLRs or RLR method papers and based on the methodology of those studies, constructed a broad definition RLR, one that meets a minimum set of requirements identified in the thematic analysis: ‘ A rapid review is a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting a variety of methods to produce evidence in a resource-efficient manner .’ This interpretation aligns with more than 50% of RLRs identified in this study. The authors additionally provided several other definitions, depending on specific situations or requirements (e.g., when RLR is produced on stakeholder’s request). It was additionally underlined that RLRs should be driven by the need of timely evidence for decision-making purposes [ 11 ].

Rapid reviews vary in their objective, format, and methods used for evidence synthesis. This is a quite new area, and still no agreement on optimal methods can be found [ 5 ]. All of the definitions are highlighting that RLRs are completed within shorter timeframes than SLRs, and also lack of time is one of the main reasons they are conducted. It has been suggested that most rapid reviews are conducted within 12 weeks; however, some of the resources suggest time between a few weeks to no more than 6 months [ 5 , 6 ]. Some of the definitions are highlighting that RLRs follow the SLR process, but certain phases of the process are simplified or omitted to retrieve information in a time-saving way [ 6 , 7 ]. Different mechanisms are used to enhance the timeliness of reviews. They can be used independently or concurrently: increasing the intensity of work by intensifying the efforts of multiple analysts by parallelization of tasks, using review shortcuts whereby one or more systematic review steps may be reduced, automatizing review steps by using new technologies [ 5 ]. The UK government report [ 8 ] referred to two different RLRs: in the form of quick scoping reviews (QSR) or rapid evidence assessments (REA). While being less resource and time-consuming compared to standard SLRs, QSRs and REAs are designed to be similarly transparent and to minimize bias. QSRs can be applied to rather open-ended questions, e.g., ‘what do we know about something’ but both, QSRs and REAs, provide an understanding of the volume and characteristics of evidence on a specific topic, allowing answering questions by maximizing the use of existing data, and providing a clear picture of the adequacy of existing evidence [ 8 ].

Research questions and searches

The guidelines suggest creating a clear research question and search protocol at the beginning of the project. Additionally, to not duplicate RLRs, the Cochrane Rapid Reviews Methods Group encourages all people working on RLRs to consider registering their search protocol with PROSPERO, the international prospective register of reviews; however, so far they are not formally registered in most cases [ 5 , 6 ]. They also recommend involving key stakeholders (review users) to set and refine the review question, criteria, and outcomes, as well as consulting them through the entire process [ 11 ].

Regarding research questions, it is better to structure them in a neutral way rather than focus on a specific direction for the outcome. By doing so, the researcher is in a better position to identify all the relevant evidence [ 7 ]. Authors can add a second, supportive research question when needed [ 8 ]. It is encouraged to limit the number of interventions, comparators and outcomes, to focus on the ones that are most important for decision-making [ 11 ]. Useful could be also reviewing additional materials, e.g., SLRs on the topic, as well as conducting a quick literature search to better understand the topic before starting with RLRs [ 7 ]. In SLRs researchers usually do not need to care a lot about time spent on creating PICOS, they need to make sure that the scope is broad enough, and they cannot use many restrictions. When working on RLRs, a reviewer may spend more or less time defining each of the components of the study question, and the main step is making sure that PICOS addresses the needs of those who requested the rapid review, and at the same time, it is feasible within the required time frame [ 7 ]. Search protocol should contain an outline of how the following review steps are to be carried out, including selected search keywords and a full strategy, a list of data sources, precise inclusion and exclusion criteria, a strategy for data extraction and critical appraisal, and a plan of how the information will be synthesized [ 8 ].

In terms of searches running, in most cases, an exhaustive process will not be feasible. Researchers should make sure that the search is effective and efficient to produce results in a timely manner. Cochrane Rapid Reviews Methods Group recommends involving an information specialist and conducting peer review of at least one search strategy [ 11 ]. According to the rapid review guidebook by McMaster University [ 7 ], it is important that RLRs, especially those that support policy and program decisions, are being fed by the results of a body of literature, rather than single studies, when possible. It would result in more generalizable findings applied at the level of a population and serve more realistic findings for program decisions [ 7 ]. It is important to document the search strategy, together with a record of the date and any date limits of the search, so that it can easily be run again, modified, or updated. Furthermore, the information on the individual databases included in platform services should always be reported, as this depends on organizations’ subscriptions and must be included for transparency and repeatability [ 7 , 8 ]. Good solution for RLRs is narrowing the scope or searching a limited number of databases and other sources [ 7 ]. Often, the authors use the PubMed/MEDLINE, Cochrane Library, and Embase databases. In most reviews, two or more databases are searched, and common limits are language (usually restricted to English), date, study design, and geographical area. Some RLRs include searching of grey literature; however, contact with authors is rather uncommon [ 5 , 8 ]. According to the flexible framework for restricted systematic review published by the University of Oxford, the search should be run in at least one major scientific database such as PubMed, and one other source, e.g., Google Scholar [ 9 ]. Grey literature and unpublished evidence may be particularly needed and important for intervention questions. It is related to the fact that studies that do not report the effects of interventions are less likely to be published [ 8 ]. If there is any type of evidence that will not be considered by the RLRs, e.g., reviews or theoretical and conceptual studies, it should also be stated in the protocol together with justification [ 8 ]. Additionally, authors of a practical guide published by WHO suggest using a staged search to identify existing SLRs at the beginning, and then focusing on studies with other designs [ 5 ]. If a low number of citations have been retrieved, it is acceptable to expand searches, remove some of the limits, and add additional databases and sources [ 7 ].

Searching for RLRs is an iterative process, and revising the approach is usually needed [ 7 ]. Changes should be confirmed with stakeholders and should be tracked and reflected in the final report [ 5 ].

The next step in the rapid review is the selection of studies consisting of two phases: screening of titles and abstracts, and analysis of full texts. Prior to screening initiation, it is recommended to conduct a pilot exercise using the same 30–50 abstracts and 5–10 full-texts for the entire screening team in order to calibrate and test the review form [ 11 ]. In contrast to SLRs, it can be done by one reviewer with or without verification by a second one. If verification is performed, usually the second reviewer checks only a subset of records and compares them. Cochrane Group, in contrast, recommends a stricter approach: at least 20% of references should be double-screened at titles and abstracts stage, and while the rest of the references may be screened by one reviewer, the excluded items need to be re-examined by second reviewer; similar approach is used in full-text screening [ 11 ]. This helps to ensure that bias was reduced and that the PICOS criteria are applied in a relevant way [ 5 , 8 , 9 , 11 ]. During the analysis of titles and abstracts, there is no need to report reasons for exclusion; however, they should be tracked for all excluded full texts [ 7 ].

Data extraction and quality assessment

According to the WHO guide, the most common method for data extraction in RLRs is extraction done by a single reviewer with or without partial verification. The authors point out that a reasonable approach is to use a second reviewer to check a random sample of at least 10% of the extractions for accuracy. Dual performance is more necessary for the extraction of quantitative results than for descriptive study information. In contrast, Cochrane group recommends that second reviewer should check the correctness and completeness of all data [ 11 ]. When possible, extractions should be limited to key characteristics and outcomes of the study. The same approach to data extraction is also suggested for a quality assessment process within rapid reviews [ 5 , 9 , 11 ]. Authors of the guidebook from McMaster University highlight that data extraction should be done ideally by two reviewers independently and consensus on the discrepancies should always be reached [ 7 ]. The final decision on the approach to this important step of review should depend on the available time and should also reflect the complexity of the research question [ 9 ].

For screening, analysis of full texts, extractions, and quality assessments, researchers can use information technologies to support them by making these review steps more efficient [ 5 ].

Before data reporting, a reviewer should prepare a document with key message headings, executive summary, background related to the topic and status of the current knowledge, project question, synthesis of findings, conclusions, and recommendations. According to the McMaster University guidebook, a report should be structured in a 1:2:20 format, that is, one page for key messages, two pages for an executive summary, and a full report of up to 20 pages [ 7 ]. All the limitations of the RLRs should be analyzed, and conclusions should be drawn with caution [ 5 ]. The quality of the accumulated evidence and the strength of recommendations can be assessed using, e.g., the GRADE system [ 5 ]. When working on references quoting, researchers should remember to use a primary source, not secondary references [ 7 ]. It would be worth considering the support of some software tools to automate reporting steps. Additionally, any standardization of the process and the usage of templates can support report development and enhance the transparency of the review [ 5 ].

Ideally, all the review steps should be completed during RLRs; however, often some steps may need skipping or will not be completed as thoroughly as should because of time constraints. It is always crucial to decide which steps may be skipped, and which are the key ones, depending on the project [ 7 ]. Guidelines suggest that it may be helpful to invite researchers with experience in the operations of SLRs to participate in the rapid review development [ 5 , 9 ]. As some of the steps will be completed by one reviewer only, it is important to provide them with relevant training at the beginning of the process, as well as during the review, to minimize the risk of mistakes [ 5 ].

Additional information

Depending on the policy goal and available resources and deadlines, methodology of the RLRs may be modified. Wilson et al. [ 10 ] provided extensive guidelines for performing RLR within days (e.g., to inform urgent internal policy discussions and/or management decisions), weeks (e.g., to inform public debates), or months (e.g., to inform policy development cycles that have a longer timeline, but that cannot wait for a traditional full systematic review). These approaches vary in terms of data synthesis, types of considered evidence and project management considerations.

In shortest timeframes, focused questions and subquestions should be formulated, typically to conduct a policy analysis; the report should consist of tables along with a brief narrative summary. Evidence from SLRs is often considered, as well as key informant interviews may be conducted to identify additional literature and insights about the topic, while primary studies and other types of evidence are not typically feasible due to time restrictions. The review would be best conducted with 1–2 reviewers sharing the work, enabling rapid iterations of the review. As for RLRs with longer timeline (weeks), these may use a mix of policy, systems and political analysis. Structure of the review would be similar to shorter RLRs – tabular with short narrative summary, as the timeline does not allow for comprehensive synthesis of data. Besides SLRs, primary studies and other evidence may be feasible in this timeframe, if obtained using the targeted searches in the most relevant databases. The review team should be larger, and standardized procedures for reviewing of the results and data extraction should be applied. In contrast to previous timeframe, merit review process may be feasible. For both timeframes, brief consultations with small transdisciplinary team should be conducted at the beginning and in the final stage of the review to discuss important matters.

For RLRs spanning several months, more comprehensive methodology may be adapted in terms of data synthesis and types of evidence. However, authors advise that review may be best conducted with a small review team in order to allow for more in-depth interpretation and iteration.

Studies analyzing methodology

There have been two interesting publications summarizing the results of Delphi consensus on the RLR methodology identified and included in this review [ 12 , 13 ].

Tricco et al. [ 13 ] first conducted an international survey and scoping review to collect information on the possible approaches to the running of rapid reviews, based on which, they employed a modified Delphi method that included inputs from 113 stakeholders to explore the most optimized approach. Among the six most frequent rapid review approaches (not all detailed here) being evaluated, the approach that combines inclusion of published literature only, a search of more than one database and limitations by date and language, study selection by one analyst, data extraction, and quality assessment by one analyst and one verifier, was perceived as the most feasible approach (72%, 81/113 responses) with the potentially lowest risk of bias (12%, 12/103). The approach ranked as the first one when considering timelines assumes updating of the search from a previously published review, no additional limits on search, studies selection and data extraction done by one reviewer, and no quality assessment. Finally, based on the publication, the most comprehensive RLRs can be made by moving on with the following rules: searching more than one database and grey literature and using date restriction, and assigning one reviewer working on screening, data extraction, and risk of bias assessment ( Table 1 ). Pandor et al. [ 12 ] introduced a decision tool for SelecTing Approaches for Rapid Reviews (STARR) that were produced through the Delphi consensus of international experts through an iterative and rigorous process. Participants were asked to assess the importance of predefined items in four domains related to the rapid review process: interaction with commissioners, understanding the evidence base, data extraction and synthesis methods, and reporting of rapid review methods. All items assigned to four domains achieved > 70% of consensus, and in that way, the first consensus-driven tool has been created that supports authors of RLRs in planning and deciding on approaches.

Six most frequent approaches to RLRs (adapted from Tricco et al. [ 13 ]).

Haby et al. [ 14 ] run searches of 11 databases and two websites and developed a comprehensive overview of the methodology of RLRs. With five SLRs and one RCT being finally included, they identified the following approaches used in RLRs to make them faster than full SLRs: limiting the number and scope of questions, searching fewer databases, limited searching of grey literature, restrictions on language and date (e.g., English only, most recent publications), updating the existing SLRs, eliminating or limiting hand searches of reference lists, noniterative search strategies, eliminating consultation with experts, limiting dual study selection, data extraction and quality assessment, minimal data synthesis with short concise conclusions or recommendations. All the SLRs included in this review were consistent in stating that no agreed definition of rapid reviews is available, and there is still no final agreement on the best methodological rules to be followed.

Gordon et al. [ 4 ] explained the advantages of performing a focused review and provided 12 tips for its conduction. They define focused reviews as ‘a form of knowledge synthesis in which the components of the systematic process are applied to facilitate the analysis of a focused research question’. The first tip presented by the authors is related to deciding if a focused review is a right solution for the considered project. RLRs will suit emerging topics, approaches, or assessments where early synthesis can support doctors, policymakers, etc., but also can direct future research. The second, third, and fourth tips highlight the importance of running preliminary searches and considering narrowing the results by using reasonable constraints taking into account the local context, problems, efficiency perspectives, and available time. Further tips include creating a team of experienced reviewers working on the RLRs, thinking about the target journal from the beginning of work on the rapid review, registering the search protocol on the PROSPERO registry, and the need for contacting authors of papers when data available in publications are missing or incongruent. The last three tips are related to the choice of evidence synthesis method, using the visual presentation of data, and considering and describing all the limitations of the focused review.

Finally, a new publication by Tricco et al. from 2022, describing JBI position statement [ 15 ] underlined that for the time being, there is no specific tool for critical appraisal of the RLR’s methodological quality. Instead, reviewers may use available tools to assess the risk of bias or quality of SLRs, like ROBIS, the JBI critical appraisal tools, or the assessment of multiple systematic reviews (AMSTAR).

Inconsistency in the definitions and methodologies of RLR

Although RLR was broadly perceived as an approach to quicken the conduct of conventional SLR, there is a lack of consensus on the formal definition of the RLR, so as to the best approaches to perform it. Only in 2021, a study proposing unified definition was published; however, it is important to note that the most accurate definition was only matching slightly over 50% of papers analysed by the authors, which underlines the lack of homogeneity in the field [ 11 ]. The evidence-based supporting methods are evolving, and more evidence is needed to define the most robust approaches [ 5 ].

Diverse terms are used to describe the RLR, including ‘rapid review’, focused systematic review’, ‘quick scoping reviews’, and ‘rapid evidence assessments’. Although the general principles of conducting RLR are to accelerate the whole process, complexity was seen in the methodologies used for RLRs, as reflected in this study. Also, inconsistencies related to the scope of the questions, search strategies, inclusion criteria, study screening, full-text review, quality assessment, and evidence presentation were implied. All these factors may hamper decision-making about optimal methodologies for conducting rapid reviews, and as a result, the efficiency of RLR might be decreased. Additionally, researchers may tend to report the methodology of their reviews without a sufficient level of detail, making it difficult to appraise the quality and robustness of their work.

Advantages and weaknesses of RLR

Although RLR used simplified approaches for evidence synthesis compared with SLR, the methodologies for RLR should be replicable, rigorous, and transparent to the greatest extent [ 16 ]. When time and resources are limited, RLR could be a practical and efficient tool to provide the summary of evidence that is critical for making rapid clinical or policy-related decisions [ 5 ]. Focusing on specific questions that are of controversy or special interest could be powerful in reaffirming whether the existing recommendation statements are still appropriate [ 17 ].

The weakness of RLR should also be borne in mind, and the trade-off of using RLR should be carefully considered regarding the thoroughness of the search, breadth of a research question, and depth of analysis [ 18 ]. If allowed, SLR is preferred over RLR considering that some relevant studies might be omitted with narrowed search strategies and simplified screening process [ 14 ]. Additionally, omitting the quality assessment of included studies could result in an increased risk of bias, making the comprehensiveness of RLR compromised [ 13 ]. Furthermore, in situations that require high accuracy, for example, where a small relative difference in an intervention has great impacts, for the purpose of drafting clinical guidelines, or making licensing decisions, a comprehensive SLR may remain the priority [ 19 ]. Therefore, clear communications with policymakers are recommended to reach an agreement on whether an RLR is justified and whether the methodologies of RLR are acceptable to address the unanswered questions [ 18 ].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Individualism and collectivism’s impact on students’ academic helping interactions: an integrative review

• Review of the literature
• Published: 21 May 2024

Cite this article

• Amos Jeng   ORCID: orcid.org/0000-0003-1927-3124 1  

In academic settings, help-seeking and help-giving are two learning behaviors that have been shown to support student interaction and success. However, existing conceptualizations of these behaviors often overlook the influence of a student’s cultural context. Specifically, there remains a lack of clarity around how students’ attitudes and behaviors related to academic help-seeking and help-giving may differ in predominantly individualist versus collectivist cultural contexts. To address this issue, an integrative review of 18 sources from PsycINFO, ERIC, and Google Scholar was conducted to examine individualism and collectivism’s relationship to students’ academic help-seeking and help-giving behaviors. Results demonstrated that cultural orientation plays an important role in impacting students’ willingness to seek and provide academic help, their motivations for participating in or avoiding helping interactions, as well as their preferred avenues for seeking and providing academic help. Ultimately, this review highlights the intertwined nature of culture and students’ helping behaviors, as well as enhances existing understandings of how future research and educators can support students’ help-seeking and help-giving behaviors in a culturally sensitive manner.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

An asterisk (*) indicates references in the reference list that are part of the integrative literature review.

*Asterhan, C. S. C., & Bouton, E. (2017). Teenage peer-to-peer knowledge sharing through social network sites in secondary schools. Computers & Education , 110 , 16–34. https://doi.org/10.1016/j.compedu.2017.03.007

Balkaya, M., Cheah, C. S. L., Yu, J., Hart, C. H., & Sun, S. (2018). Maternal encouragement of modest behavior, temperamental shyness, and anxious withdrawal linkages to Chinese American children’s social adjustment: A moderated mediation analysis. Social Development, 27 (4), 876–890. https://doi.org/10.1111/sode.12295

Article   Google Scholar  

Black, S., & Allen, J. D. (2019). Part 8: Academic help seeking. The Reference Librarian, 60 (1), 62–76. https://doi.org/10.1080/02763877.2018.1533910

Bontempo, R., Lobel, S., & Triandis, H. (1990). Compliance and value internalization in Brazil and the U.S.: Effects of allocentrism and anonymity. Journal of Cross-Cultural Psychology, 21 (2), 200–213. https://doi.org/10.1177/0022022190212004

Bornschlegl, M., Meldrum, K., & Caltabiano, N. J. (2020). Variables related to academic help-seeking behaviour in higher education – findings from a multidisciplinary perspective. Review of Education, 8 (2), 486–522. https://doi.org/10.1002/rev3.3196

*Bouton, E., Tal, S. B., & Asterhan, C. S. C. (2021). Students, social network technology and learning in higher education: Visions of collaborative knowledge construction vs. the reality of knowledge sharing. The Internet and Higher Education , 49 , 100787. https://doi.org/10.1016/j.iheduc.2020.100787

Brewer, M. B., & Chen, Y.-R. (2007). Where (who) are collectives in collectivism? Toward conceptual clarification of individualism and collectivism. Psychological Review, 114 (1), 133–151. https://doi.org/10.1037/0033-295X.114.1.133

Bruner, J. S. (1996). The culture of education . Harvard University Press.

Book   Google Scholar  

*Cao, C., Zhu, C., & Meng, Q. (2021). Chinese international students’ coping strategies, social support resources in response to academic stressors: Does heritage culture or host context matter? Current Psychology , 40 (1), 242–252. https://doi.org/10.1007/s12144-018-9929-0

*Cerna, M. A., & Pavliushchenko, K. (2015). Influence of study habits on academic performance of international college students in Shanghai. Higher Education Studies , 5 (4), 42–55.

*Chang, J. (2015). The interplay between collectivism and social support processes among Asian and Latino American college students. Asian American Journal of Psychology , 6 (1), 4–14. https://doi.org/10.1037/a0035820

*Chang, J., Wang, S., Mancini, C., McGrath-Mahrer, B., & Orama de Jesus, S. (2020). The complexity of cultural mismatch in higher education: Norms affecting first-generation college students’ coping and help-seeking behaviors. Cultural Diversity and Ethnic Minority Psychology , 26 (3), 280–294. https://doi.org/10.1037/cdp0000311

Chen, J.J.-L. (2005). Relation of academic support from parents, teachers, and peers to Hong Kong adolescents’ academic achievement: The mediating role of academic engagement. Genetic, Social, and General Psychology Monographs, 131 (2), 77–127. https://doi.org/10.3200/MONO.131.2.77-127

Chu, H.-C., Chen, J.-M., & Tsai, C.-L. (2017). Effects of an online formative peer-tutoring approach on students’ learning behaviors, performance and cognitive load in mathematics. Interactive Learning Environments, 25 (2), 203–219. https://doi.org/10.1080/10494820.2016.1276085

*Covarrubias, R., Laiduc, G., & Valle, I. (2019). Growth messages increase help-seeking and performance for women in STEM. Group Processes & Intergroup Relations , 22 (3), 434–451. https://doi.org/10.1177/1368430218802958

Cross, S. E., Bacon, P. L., & Morris, M. L. (2000). The relational-interdependent self-construal and relationships. Journal of Personality and Social Psychology, 78 (4), 791–808. https://doi.org/10.1037/0022-3514.78.4.791

Cross, S. E., Hardin, E. E., & Gercek-Swing, B. (2011). The what, how, why, and where of self-construal. Personality and Social Psychology Review, 15 (2), 142–179. https://doi.org/10.1177/1088868310373752

*Crystal, D. S., Kakinuma, M., DeBell, M., Azuma, H., & Miyashita, T. (2008). Who helps you? Self and other sources of support among youth in Japan and the USA. International Journal of Behavioral Development , 32 (6), 496–508. https://doi.org/10.1177/0165025408095554

Dalton, J. E., Bolen, S. D., & Mascha, E. J. (2016). Publication bias: The elephant in the review. Anesthesia & Analgesia, 123 (4), 812–813. https://doi.org/10.1213/ANE.0000000000001596

Dorfman, P. W., & Howell, J. P. (1988). Dimensions of national culture and effective leadership patterns: Hofstede revisited. Advances in International Comparative Management, 3 (1), 127–150.

Google Scholar  

Dwyer, P. A. (2020). Analysis and synthesis. In C. E. Toronto & R. Remington (Eds.), A step-by-step guide to conducting an integrative review (pp. 57–70). Springer International Publishing. https://doi.org/10.1007/978-3-030-37504-1_5

Elsbach, K. D., & Knippenberg, D. (2020). Creating high-impact literature reviews: An argument for ‘integrative reviews.’ Journal of Management Studies, 57 (6), 1277–1289. https://doi.org/10.1111/joms.12581

Esparza-Reig, J., Martí-Vilar, M., González-Sala, F., Merino-Soto, C., & Toledano-Toledano, F. (2022). Social support and resilience as predictors of prosocial behaviors before and during COVID-19. Healthcare, 10 (9), 1669. https://doi.org/10.3390/healthcare10091669

Fong, C. J., Gonzales, C., Hill-Troglin Cox, C., & Shinn, H. B. (2023). Academic help-seeking and achievement of postsecondary students: A meta-analytic investigation. Journal of Educational Psychology, 115 (1), 1–21. https://doi.org/10.1037/edu0000725

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., Karns, K., & Dutka, S. (1997). Enhancing students’ helping behavior during peer-mediated instruction with conceptual mathematical explanations. The Elementary School Journal, 97 (3), 223–249. https://doi.org/10.1086/461863

Grayson, A., Miller, H., & Clarke, D. D. (1998). Identifying barriers to help-seeking: A qualitative analysis of students’ preparedness to seek help from tutors. British Journal of Guidance & Counselling, 26 (2), 237–253. https://doi.org/10.1080/03069889808259704

Green, E. G. T., Deschamps, J.-C., & Páez, D. (2005). Variation of individualism and collectivism within and between 20 countries: A typological analysis. Journal of Cross-Cultural Psychology, 36 (3), 321–339. https://doi.org/10.1177/0022022104273654

Greenfield, P. M. (1994). Independence and interdependence as developmental scripts. In P. M. Greenfield & R. R. Cocking (Eds.), Cross-Cultural Roots of Minority Child Development (1st ed., pp. 1–37). Lawrence Erlbaum Associates, Inc.

Grillo, M. C., & Leist, C. W. (2013). Academic support as a predictor of retention to graduation: New insights on the role of tutoring, learning assistance, and supplemental instruction. Journal of College Student Retention: Research, Theory & Practice, 15 (3), 387–408. https://doi.org/10.2190/CS.15.3.e

Han, M., & Pong, H. (2015). Mental health help-seeking behaviors among Asian American community college students: The effect of stigma, cultural barriers, and acculturation. Journal of College Student Development, 56 (1), 1–14. https://doi.org/10.1353/csd.2015.0001

Hao, Z., & Liang, B. (2007). Predictors of college students’ attitudes toward seeking professional psychological help. Chinese Journal of Clinical Psychology, 15 (3), 321–325.

Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34 (4), 285–291. https://doi.org/10.3233/EFI-180221

Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I. (2018). Mixed Methods Appraisal Tool (MMAT), version 2018 .

Huang, K., & Law, V. (2022). Help seeking from peers in an online class: Roles of students’ help-seeking profiles and epistemic beliefs. Journal of Educational Technology & Society, 25 (3), 1–14.

*Hwang, A., Francesco, A. M., & Kessler, E. (2003). The relationship between individualism-collectivism, face, and feedback and learning processes in Hong Kong, Singapore, and the United States. Journal of Cross-Cultural Psychology , 34 (1), 72–90. https://doi.org/10.1177/0022022102239156

*Hwang, Y., & Kim, D. J. (2007). Understanding affective commitment, collectivist culture, and social influence in relation to knowledge sharing in technology mediated learning. IEEE Transactions on Professional Communication , 50 (3), 232–248. https://doi.org/10.1109/TPC.2007.902664

*Janoff-Bulman, R., & Leggatt, H. K. (2002). Culture and social obligation: When “shoulds” are perceived as “wants.” Journal of Research in Personality, 36 (3), 260–270. https://doi.org/10.1006/jrpe.2001.2345

Kai, J. (2012). The origin and consequences of excess competition in education: A mainland Chinese perspective. Chinese Education & Society, 45 (2), 8–20. https://doi.org/10.2753/CED1061-1932450201

Karabenick, S. A. (2003). Seeking help in large college classes: A person-centered approach. Contemporary Educational Psychology, 28 (1), 37–58. https://doi.org/10.1016/S0361-476X(02)00012-7

Karabenick, S. A., & Knapp, J. R. (1991). Relationship of academic help seeking to the use of learning strategies and other instrumental achievement behavior in college students. Journal of Educational Psychology, 83 (2), 221–230. https://doi.org/10.1037/0022-0663.83.2.221

Kennedy, T. J. T., Regehr, G., Baker, G. R., & Lingard, L. A. (2009). It’s a cultural expectation…’ the pressure on medical trainees to work independently in clinical practice. Medical Education, 43 (7), 645–653. https://doi.org/10.1111/j.1365-2923.2009.03382.x

Kim, H. S., Sherman, D. K., Ko, D., & Taylor, S. E. (2006). Pursuit of comfort and pursuit of harmony: Culture, relationships, and social support seeking. Personality and Social Psychology Bulletin, 32 (12), 1595–1607. https://doi.org/10.1177/0146167206291991

*Komissarouk, S., & Nadler, A. (2014). “I” seek autonomy, “we” rely on each other: Self-construal and regulatory focus as determinants of autonomy- and dependency-oriented help-seeking behavior. Personality and Social Psychology Bulletin , 40 (6), 726–738. https://doi.org/10.1177/0146167214524444

Kumru, A., Carlo, G., & Edwards, C. P. (2004). Relational, cultural, cognitive, and affective predictors of prosocial behaviors [Olumlu sosyal davranişlarin ilişkisel, kültürel, bilişsel ve duyuşsal bazi değişkenlerle ilişkisi]. Türk Psikoloji Dergisi, 19 (54), 109–125.

Louie, J. Y., Wang, S., Fung, J., & Lau, A. (2015). Children’s emotional expressivity and teacher perceptions of social competence. International Journal of Behavioral Development, 39 (6), 497–507. https://doi.org/10.1177/0165025414548775

*Luo, Y., Sun, Y., & Strobel, J. (2013). The effects of collectivism-individualism on the cooperative learning of motor skill. Journal of International Students , 3 (1), 41–51.

Marchand, G., & Skinner, E. A. (2007). Motivational dynamics of children’s academic help-seeking and concealment. Journal of Educational Psychology, 99 (1), 65–82. https://doi.org/10.1037/0022-0663.99.1.65

Mare, L. L., & Sohbat, E. (2002). Canadian students’ perceptions of teacher characteristics that support or inhibit help seeking. The Elementary School Journal, 102 (3), 239–253.

Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98 (2), 224–253. https://doi.org/10.1037/0033-295X.98.2.224

Martin, J. (2014). Psychologism, individualism and the limiting of the social context in educational psychology. In T. Corcoran (Ed.), Psychology in Education (pp. 167–180). SensePublishers. https://doi.org/10.1007/978-94-6209-566-3_11

Martí-Vilar, M., Serrano-Pastor, L., & Sala, F. G. (2019). Emotional, cultural and cognitive variables of prosocial behaviour. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues, 38 (4), 912–919. https://doi.org/10.1007/s12144-019-0168-9

McCoy, S., Galletta, D. F., & King, W. R. (2005). Integrating national culture into IS research: The need for current individual level measures. Communications of the Association for Information Systems . https://doi.org/10.17705/1CAIS.01512

Miles, M. B., & Huberman, M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). SAGE Publications.

*Mok, M. C. M., Kennedy, K. J., Moore, P. J., Shan, P. W., & Leung, S. O. (2008). The use of help-seeking by Chinese secondary school students: Challenging the myth of ‘the Chinese learner.’ Evaluation & Research in Education , 21 (3), 188–213. https://doi.org/10.1080/09500790802485229

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151 (4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

Moscardino, U., Miconi, D., & Carraro, L. (2020). Implicit and explicit self-construals in Chinese-heritage and Italian nonimmigrant early adolescents: Associations with self-esteem and prosocial behavior. Developmental Psychology, 56 (7), 1397–1412. https://doi.org/10.1037/dev0000937

Nelson-Le Gall, S. (1981). Help-seeking: An understudied problem-solving skill in children. Developmental Review, 1 (3), 224–246. https://doi.org/10.1016/0273-2297(81)90019-8

Nelson-Le Gall, S. (1985). Help-seeking behavior in learning. Review of Research in Education, 12 (1), 55–90. https://doi.org/10.3102/0091732X012001055

Nelson-Le Gall, S., & Resnick, L. (1998). Help seeking, achievement motivation, and the social practice of intelligence in school. In S. A. Karabenick (Ed.), Strategic help seeking: Implications for learning and teaching (pp. 49–69). Routledge. https://doi.org/10.4324/9781410602701

Newman, R. S. (1990). Children’s help-seeking in the classroom: The role of motivational factors and attitudes. Journal of Educational Psychology, 82 (1), 71–80. https://doi.org/10.1037/0022-0663.82.1.71

Newman, R. S. (1994). Adaptive help-seeking: A strategy of self-regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Self-regulation of learning and performance: Issues and educational applications (pp. 283–301). Lawrence Erlbaum Associates Inc.

Newman, R. S. (2002). How self-regulated learners cope with academic difficulty: The role of adaptive help seeking. Theory into Practice, 41 (2), 132–138. https://doi.org/10.1207/s15430421tip4102_10

Oyserman, D. (2017). Culture three ways: Culture and subcultures within countries. Annual Review of Psychology, 68 (1), 435–463. https://doi.org/10.1146/annurev-psych-122414-033617

Oyserman, D., Coon, H. M., & Kemmelmeier, M. (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin, 128 (1), 3–72. https://doi.org/10.1037/0033-2909.128.1.3

*Petrella, M. V., & Gore, J. S. (2013). Relational self-construal and its relationship to academic citizenship behavior. Psychological Studies , 58 (2), 115–121. https://doi.org/10.1007/s12646-013-0182-1

Poortvliet, P. M., & Darnon, C. (2014). Understanding positive attitudes toward helping peers: The role of mastery goals and academic self-efficacy. Self and Identity, 13 (3), 345–363. https://doi.org/10.1080/15298868.2013.832363

*Popov, V., Noroozi, O., Barrett, J. B., Biemans, H. J. A., Teasley, S. D., Slof, B., & Mulder, M. (2014). Perceptions and experiences of, and outcomes for, university students in culturally diversified dyads in a computer-supported collaborative learning environment. Computers in Human Behavior , 32 , 186–200. https://doi.org/10.1016/j.chb.2013.12.008

Reeves, P. M., & Sperling, R. A. (2015). A comparison of technologically mediated and face-to-face help-seeking sources. British Journal of Educational Psychology, 85 (4), 570–584. https://doi.org/10.1111/bjep.12088

*Rothstein-Fisch, C., Trumbull, E., Isaac, A., Daley, C., & Pérez, A. I. (2003). When “helping someone else” is the right answer: Bridging cultures in assessment. Journal of Latinos and Education , 2 (3), 123–140. https://doi.org/10.1207/S1532771XJLE0203_01

Rovers, S. F. E., Clarebout, G., Savelberg, H. H. C. M., De Bruin, A. B. H., & Van Merriënboer, J. J. G. (2019). Granularity matters: Comparing different ways of measuring self-regulated learning. Metacognition and Learning, 14 (1), 1–19. https://doi.org/10.1007/s11409-019-09188-6

Ryan, A. M., Gheen, M. H., & Midgley, C. (1998). Why do some students avoid asking for help? An examination of the interplay among students’ academic efficacy, teachers’ social–emotional role, and the classroom goal structure. Journal of Educational Psychology, 90 (3), 528–535. https://doi.org/10.1037/0022-0663.90.3.528

Ryan, A. M., Shim, S. S., Lampkins-uThando, S. A., Kiefer, S. M., & Thompson, G. N. (2009). Do gender differences in help avoidance vary by ethnicity? An examination of African American and European American students during early adolescence. Developmental Psychology, 45 (4), 1152–1163. https://doi.org/10.1037/a0013916

Sandoval, B. A., & Lee, F. (2006). When is help seeking appropriate? How norms affect help seeking in organizations. In S. A. Karabenick & R. S. Newman (Eds.), Help-seeking in academic settings: Goals, groups, and contexts (pp. 151–173). Lawrence Erlbaum Associates Inc.

Shwalb, D. W., & Sukemune, S. (1998). Help seeking in the Japanese college classroom: Cultural, developmental and social-psychological influences. In S. A. Karabenick (Ed.), Strategic help seeking: Implications for learning and teaching (pp. 139–163). Routledge. https://doi.org/10.4324/9781410602701

Singelis, T. M. (1994). The measurement of independent and interdependent self-construals. Personality and Social Psychology Bulletin, 20 (5), 580–591. https://doi.org/10.1177/0146167294205014

Singelis, T. M., Triandis, H. C., Bhawuk, D. P. S., & Gelfand, M. J. (1995). Horizontal and vertical dimensions of individualism and collectivism: A theoretical and measurement refinement. Cross-Cultural Research, 29 (3), 240–275. https://doi.org/10.1177/106939719502900302

Torraco, R. J. (2016). Writing integrative reviews of the literature: Methods and purposes. International Journal of Adult Vocational Education and Technology, 7 (3), 62–70. https://doi.org/10.4018/IJAVET.2016070106

Triandis, H. C. (1995). Individualism and collectivism . Routledge.

Triandis, H. C., & Gelfand, M. J. (1998). Converging measurement of horizontal and vertical individualism and collectivism. Journal of Personality and Social Psychology, 74 (1), 118–128. https://doi.org/10.1037/0022-3514.74.1.118

Tripathi, R. C. (2019). Unity of the individual and the collective. In R. C. Tripathi, Psychology (Vol. 2, pp. 283–331). Oxford University Press. https://doi.org/10.1093/oso/9780199498857.003.0007

Volet, S., & Karabenick, S. A. (2006). Help-seeking in cultural context. In S. A. Karabenick & R. S. Newman (Eds.), Help-seeking in academic settings: Goals, groups, and contexts (pp. 117–150). Lawrence Erlbaum Associates Inc.

Wagner, J. A. (1995). Studies of individualism-collectivism: Effects of cooperation in groups. Academy of Management Journal, 38 (1), 152–173. https://doi.org/10.2307/256731

Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Educational Research, 13 (1), 21–39. https://doi.org/10.1016/0883-0355(89)90014-1

Webb, N. M., & Mastergeorge, A. (2003). Promoting effective helping behavior in peer-directed groups. International Journal of Educational Research, 39 (1–2), 73–97. https://doi.org/10.1016/S0883-0355(03)00074-0

Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52 (5), 546–553. https://doi.org/10.1111/j.1365-2648.2005.03621.x

Wong, C., & Cho, G. E. (2005). Two-headed coins or kandinskys: White racial identification. Political Psychology, 26 (5), 699–720. https://doi.org/10.1111/j.1467-9221.2005.00440.x

Yamauchi, L. A. (1998). Individualism, collectivism, and cultural compatibility: Implications for counselors and teachers. The Journal of Humanistic Education and Development, 36 (4), 189–198. https://doi.org/10.1002/j.2164-4683.1998.tb00391.x

Zimmerman, B. J. (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25 (1), 3–17. https://doi.org/10.1207/s15326985ep2501_2

*Zusho, A., & Barnett, P. A. (2011). Personal and contextual determinants of ethnically diverse female high school students’ patterns of academic help seeking and help avoidance in English and mathematics. Contemporary Educational Psychology , 36 (2), 152–164. https://doi.org/10.1016/j.cedpsych.2011.02.002

Download references

Acknowledgements

I would like to acknowledge Professor Michelle Perry (University of Illinois Urbana-Champaign) for her guidance, feedback, and support when preparing this manuscript.

Author information

Authors and affiliations.

Department of Educational Psychology, University of Illinois Urbana-Champaign, 1310 South 6th Street, Champaign, IL, 61820, USA

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Amos Jeng .

Ethics declarations

Conflict of interest.

The author has no relevant financial or non-financial interests to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table  3 .

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Jeng, A. Individualism and collectivism’s impact on students’ academic helping interactions: an integrative review. Soc Psychol Educ (2024). https://doi.org/10.1007/s11218-024-09920-4

Download citation

Received : 23 April 2023

Accepted : 17 April 2024

Published : 21 May 2024

DOI : https://doi.org/10.1007/s11218-024-09920-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Help-seeking
• Help-giving
• Individualism
• Collectivism
• Literature review
• Find a journal
• Publish with us
• Track your research

Advertisement

• Previous Article
• Next Article

PEER REVIEW

1. introduction, 2. literature review, 3. objectives, 6. discussion, 7. conclusions, author contributions, competing interests, funding information, data availability, completeness degree of publication metadata in eight free-access scholarly databases.

Handling Editor: Vincent Larivière

• Funder(s):  Agencia Estatal de Investigación
• Award Id(s): PID2019-106510GB-I00
• Cite Icon Cite
• Open the PDF for in another window
• Permissions
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Search Site

Lorena Delgado-Quirós , José Luis Ortega; Completeness degree of publication metadata in eight free-access scholarly databases. Quantitative Science Studies 2024; 5 (1): 31–49. doi: https://doi.org/10.1162/qss_a_00286

Download citation file:

• Ris (Zotero)
• Reference Manager

The main objective of this study is to compare the amount of metadata and the completeness degree of research publications in new academic databases. Using a quantitative approach, we selected a random Crossref sample of more than 115,000 records, which was then searched in seven databases (Dimensions, Google Scholar, Microsoft Academic, OpenAlex, Scilit, Semantic Scholar, and The Lens). Seven characteristics were analyzed (abstract, access, bibliographic info, document type, publication date, language, and identifiers), to observe fields that describe this information, the completeness rate of these fields, and the agreement among databases. The results show that academic search engines (Google Scholar, Microsoft Academic, and Semantic Scholar) gather less information and have a low degree of completeness. Conversely, third-party databases (Dimensions, OpenAlex, Scilit, and The Lens) have more metadata quality and a higher completeness rate. We conclude that academic search engines lack the ability to retrieve reliable descriptive data by crawling the web, and the main problem of third-party databases is the loss of information derived from integrating different sources.

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00286

The recent proliferation of bibliographic scholarly databases has stimulated interest in these new platforms, given their possibilities of finding scientific literature and providing different bibliometric indicators. This attention has focused on testing the performance of these new systems in relation to traditional products, such as citation indexes (e.g., Web of Science (WoS), Scopus) and academic search engines (e.g., Google Scholar, Microsoft Academic). These new products could be defined as hybrid databases because they share characteristics with the former. On the one hand, these platforms also extract and process citations for computing ad hoc bibliometric indicators as classical citation indexes. On the other hand, they are similar to search engines because they opt for a free-access model in which users require no subscription fee to search and retrieve documents. Moreover, in some cases, they provide open data through REST APIs or dump files.

However, these hybrid products have some particularities that make them different. Most importantly, they are fed by third-party sources. The appearance of Crossref as a repository of publishers’ metadata, the availability of APIs and dump files from academic search engines (e.g., Microsoft Academic, Semantic Scholar), and the possibility of reusing other bibliographic databases (e.g., PubMed, Directory of Open Access Journals (DOAJ), repositories) have facilitated the emergence of these bibliographic products that have quickly and inexpensively covered much of the scientific literature.

However, this multiple and varied availability of bibliographic data also presents a challenge for these new platforms because the integration of data from different sources requires intense data processing to prevent duplicate records, filter nonscholarly materials, and manage different versions of the same document. All of these actions are addressed to achieve optimal metadata quality, which could be defined as the quantity of information that describes an entity in an accurate and reliable form. The integration of internal and external descriptions determines this quality to a large extent.

For this reason, examining the quality of publication metadata in the new scholarly databases allows us to appreciate the extent to which these processing efforts are accomplished and to value the suitability and reliability of these search tools to provide rich information about scientific literature. This study aims to explore the metadata publication quality of these new databases to obtain a global picture about the richness of the information provided by each platform.

Many studies have focused on the evaluation of the performance of these new academic databases, comparing the coverage and overlap of records ( Gusenbauer, 2019 ; Martín-Martín, Thelwall et al., 2021 ; van Eck, Waltman et al., 2018 ; Visser, van Eck, & Waltman, 2021 ). This quantitative procedure is excessively centered on the size of each platform and overlooks the amount and quality of the content included in each database. In this sense, some articles have described the metadata quality of specific sources as a way of informing about the richness and limitations of those sources. Hendricks, Tkaczyk et al. (2020) described how the Crossref database functions and analyzed the completeness of its metadata. Similar papers were published describing Semantic Scholar ( Wade, 2022 ), The Lens ( Jefferson, Koellhofer et al., 2019 ), Dimensions ( Herzog, Hook, & Konkiel, 2020 ), and Microsoft Academic ( Wang, Shen et al., 2020 ). Many of these studies were descriptive reviews written by their employees with no critical discussion of the quality of the data.

In other cases, the coverage of certain elements or entities in different scholarly databases was studied to test their performance in processing specific information. Hug and Brändle (2017) analyzed in detail the coverage of Microsoft Academic, finding important problems in the assigning of author and publication data in comparison to WoS and Scopus. Ranjbar-Sahraei and van Eck (2018) also tested the problems of Microsoft linking papers with organizations. Guerrero-Bote, Chinchilla-Rodríguez et al. (2021) compared affiliation information between Scopus and Dimensions and found that close to half of all documents in Dimensions were not associated with any country. Purnell (2022) evaluated affiliation discrepancies in four scholarly databases, finding that the larger the database, the greater the disambiguation problems. Kramer and de Jonge (2022) analyzed the information about funders included by Crossref, The Lens, WoS, Scopus, and Dimensions, uncovering important differences when they came to extract and process that information. Lutai and Lyubushko (2022) also analyzed the coverage in six databases, detecting discrepancies and similarities in the identification and indexation of Russian authors.

As for publications, some studies have explored the amount of information and quality of this key entity in scholarly databases. Herrmannova and Knoth (2016) tested the reliability of the publication date in Microsoft Academic Graph, finding that 88% of cases showed a correct date. Liu, Hu, and Tang (2018) detected that approximately 20% of WoS publications have no information in the address field. Basson, Simard et al. (2022) showed that the proportion of open access documents in Dimensions is higher than in WoS because the former indexes more publications from Asian and Latin American countries. Other studies have examined errors and inconsistencies in different academic databases to test their suitability for bibliometric studies or for bibliographic searches only. Thus, some articles have analyzed duplicate records management in Scopus ( Valderrama-Zurián, Aguilar-Moya et al., 2015 ) and WoS ( Franceschini, Maisano, & Mastrogiacomo, 2016 ).

Metadata quality is a concept that has emerged recently as a result of the abundance of available data sets and sources. Bruce and Hillman (2004) defined seven characteristics that should be considered when assessing the quality of metadata: completeness, accuracy, provenance, conformance to expectations, logical consistency and coherence, timeliness, and accessibility. Subsequently, Ochoa and Duval (2009) proposed varied metrics to automatically assess these features. However, many of these studies have been applied to digital libraries or repositories ( Kubler, Robert et al., 2018 ; Tani, Candela, & Castelli, 2013 ), with an important gap in the analysis of scholarly databases.

Is it possible to quantitatively compare the metadata content of different databases using a third source (i.e., Crossref)? What advantages and limitations could this procedure report?

Do similarities or discrepancies among databases allow us to delimit different models of databases, with their advantages and limitations?

Which databases provide the most metadata and have a higher completeness rate?

4.1. Source Selection Criteria

They had to be freely accessible through the web: a free-subscription search interface.

They had to provide metrics for research evaluation.

4.2. Sample Selection and Extraction

Crossref was selected as a control sample for several reasons. The first was an operational question. Crossref is a publishers’ consortium that assigns the Digital Object Identifier (DOI), the most extended persistent identifier of research publications in the publishing system. Although coverage is limited to publisher members ( Visser et al., 2021 ), its use is justified because all these platforms allow publications to be queried by DOIs, thereby facilitating rapid and exact matching. The second reason is related to methodological issues: Crossref enables the extraction of random samples of documents ( https://api.crossref.org/works?sample=100 ). The representativeness of the sample is thus reinforced because it avoids the influence of ranking algorithms, filters, or matching procedures that could distort the quality of the sample. A third motive is that publishers can request a DOI for any published material, regardless of typology, discipline, or language. The Crossref database therefore has no inclusion criteria that could limit the coverage of certain types of documents (e.g., indexes, acknowledgements, or front covers). This nonselective criterion would lead us to clearly appreciate the inclusion policies of the different bibliographic platforms. Finally, Crossref is fed by publishers that deposit metadata about their publications and could be considered the most authoritative source of the reliability and accuracy of their own publications.

4.3. Sources Description

Dimensions : Created in 2018 by Digital Science, this database is supported by external products, Crossref being the main source ( Hook, Porter, & Herzog, 2018 ). It gathers more than 138 million publications, in addition to patents (154 million), data sets (12 million), and grants (7 million).

Google Scholar : One of the most important academic search engines due to its estimated size (389 million) and age (2004) ( Gusenbauer, 2019 ), it obtains data directly from the web. Special agreements with publishers allow it to extract citations and content information from paywalled journals. Its search interface can also access books (Google Books) and patents (Google Patents).

OpenAlex : This is the newest service (2022). It was created by OurResearch, a nonprofit organization that recovered the defunct Microsoft Academic Graph (203 million) to implement a new open product. The core of OpenAlex was then set up by Microsoft Academic Graph, with the addition of data from other open sources such as Crossref, PubMed, and ORCID. OpenAlex now indexes 240 million publications.

Microsoft Academic : The last version of this search engine functioned between 2016 and 2021. Like Google Scholar, it also crawled the web extracting metadata from scholarly publications, reaching 260 million publications. Part of its database was openly released (Microsoft Academic Graph), which has enabled reuse by other bibliographic products.

Scilit : Active since 2016, this database was created by the publisher MDPI as a way to compete in the scholarly database market. It indexes 159 million scholarly publications, taken mainly from Crossref and PubMed.

Semantic Scholar : This search engine was launched in 2015 by the Allen Institute for Artificial Intelligence. Although it uses crawlers to extract information from the web, in 2018, it agreed to include Microsoft Academic Graph as a primary source ( Boyle, 2018 ). It now encompasses almost 214 million research papers.

The Lens : This database was developed by Cambia, a nonprofit organization, in 2000. It initially started as a patent database, but in 2018 incorporated scholarly publications from Crossref, PubMed, and Microsoft Academic. It now provides more than 225 million scholarly works.

4.4. Data Retrieving

A sample of 116,648 DOIs was randomly extracted from Crossref in August 2020 and July 2021, the only limitation being that documents were to have been published between 2014 and 2018. This time window was selected so that publications could accrue a significant number of citations and other metrics. The sample was generated by sending 1,200 automatic requests to https://api.crossref.org/works?sample=100 . Duplicate records produced by this random process were removed to obtain the final list. The resulting distribution by document type largely coincides with the entire database ( Hendricks et al., 2020 ), thereby reinforcing the reliability of the sample. Hendricks et al. (2020) adapt some categories, in which Book presumably includes book and book-chapter, and Preprints might be posted-content ( Table 1 ).

Document type distribution in the Crossref random sample and the total coverage in June 2019 ( Hendricks et al., 2020 ). Only the first 10 most frequent categories are shown

Dimensions : This database was accessed through the API ( https://app.dimensions.ai/dsl/v2 ). An R package (dimensionsR 1 ) was used to extract the data. The results were downloaded in the JSON format because dimensionsR caused problems in the transformation of JSON outputs to CSV format.

Google Scholar : Web scraping was used to automatically query each DOI in the search box because Google Scholar does not facilitate access to its data. The RSelenium 2 R package was used to emulate a browser session and avoid antirobot actions (i.e., CAPTCHAs). Because some DOIs could not be indexed ( Martín-Martín, Orduna-Malea et al., 2018 ), a title search with the query “allintitle:title” was used to complete the results.

OpenAlex : This bibliographic repository was accessed through its public API ( https://api.openalex.org/ ). A Python routine was written to extract and process the data.

Microsoft Academic : Coverage of this service was obtained by several methods. Firstly, SPARQL ( https://makg.org/sparql ) and REST API ( https://api.labs.cognitive.microsoft.com/academic/v1.0/evaluate ) endpoints were used to extract publications using DOIs. Microdemic 3 , an R package, was used to query the API. However, the low indexation of DOIs (37.1%) and their case sensitivity forced us to download the entire table of publications available in Zenodo ( https://zenodo.org/record/2628216 ) and locally match them with the sample, using DOIs and titles.

Scilit : This platform was accessed via a public API ( https://app.scilit.net/api/v1/ ). Because access must be via a POST method, a Python script was designed to extract the data.

Semantic Scholar : This database provides a public API ( https://api.semanticscholar.org/v1 ). The semscholar 4 R package was used to extract the data. However, the API was directly queried subsequently to detect any problems in the retrieval process. A Python script was written.

The Lens : After a formal request, this service provided temporary access to its API ( https://api.lens.org/scholarly/search ). In this case, an R script was written to directly extract the data. However, some relevant fields (i.e., abstract, source_urls, funders) for this study were not properly retrieved, due to technical reasons, in July 2021. We then decided to extract a small sample of 5,000 records directly from the main search page ( https://www.lens.org/lens/ ) to make up for this limitation in January 2023. From that request, 4,996 records were successfully retrieved.

Web source with information about publication metadata in each database

Completeness: the quantity of fields and degree of filling for an entity.

Accuracy: proportion of data integrity with respect to the original data or sources.

Provenance: description of primary data sources.

Logical consistency and coherence: the degree of resemblance in the description and classification of objects (e.g., language, document type).

This study describes the amount and quality of metadata associated with the description of research publications indexed in these databases. Publications are central to the publishing ecosystem and are therefore the main asset of a bibliographic database. A clear and complete description of their elements and characteristics improves the identification and retrieval of these items, and their connection with other entities. As a result, publications are the entity with most fields, ranging from 38 fields in Crossref to 18 in Semantic Scholar. Next, we analyze the fields used by each database to describe the main characteristics of a publication.

5.1. Abstract

This is an important access point to publication content because it provides a summary of the research. All the databases analyzed index this element. For Microsoft Academic, the table with this information ( PaperAbstractsInvertedIndex ) is not yet available. However, early studies detected a coverage of 58% ( Färber & Ao, 2022 ). Google Scholar does not index the publication abstract, but it does extract parts of the document text ( Google Scholar, 2023 ).

Table 3 shows the number and proportion of publications with an abstract. Google Scholar indexes the most articles with a summary (91.66%), although some are merely an extract of the text. Aside from this, Dimensions indexes most articles with their abstract (69.6%), followed by OpenAlex (63.8%). Conversely, Crossref has fewer publications with an abstract (13.7%). This last percentage is lower than that reported by Waltman, Kramer et al. (2020) (21%), perhaps because our study also gathers other materials, such as book chapters and conference papers, which do not always include a formal abstract. This low percentage of abstracts in Crossref shows that this information is not usually provided by publishers. Indexation services therefore need to process documents to obtain this data, which would explain the overall low availability of abstracts in free-access databases, in particular in Scilit (50.5%) and Semantic Scholar (54.45%). Only OpenAlex and The Lens ( χ 2 = 2.2881, p -value = 0.1304) show no statistical differences, perhaps due to the small sample size of The Lens or because both databases use Microsoft Academic as a source.

Proportion of publications with abstract in each database. Margin of error at 99%

5.2. Access

Today, a positive feature of scholarly databases is that they provide some type of access to original publications. Widespread electronic publishing allows for the provision of links to different venues where the document is partially or fully hosted. All the databases include external links to the original publication. Microsoft Academic and Crossref have no specific field for open access publications. In March 2023, OpenAlex changed some fields about access: Landing _ page _ url is designated as an external link, but only includes DOI links (99.8%), and oa_url indexes external links to open access versions only. We therefore concluded that OpenAlex includes external links for open access publications only. This is also the case for Dimensions, which only indexes external links ( linkout ) for open_access ( open _ access ) articles.

Figure 1 and Table 4 depict the percentage of publications with external links to the original source and information about whether or not they are open access per database. Google Scholar (97.1%) includes the most external links (97.1%), followed by The Lens (82.9%), Microsoft Academic (80.8%), and Crossref (79%). It is evident that the academic search engines Google Scholar and Microsoft Academic stand out in this aspect because they only index documents that are accessible on the web. The remaining 19.2% of documents without links in Microsoft Academic is explained by the removal of handles. Färber and Ao (2022) detected more documents with links (94%), which would explain this difference, as well as the coverage of The Lens, because it also uses Microsoft Academic Graph as a source. In the case of Crossref, it could be because publishers deposit their landing pages to generate incoming traffic to their publications. Conversely, Semantic Scholar (39.1%) and Scilit (45.4%) provide fewer URLs, despite the fact that the former uses Crossref as a source. The reason is that Semantic Scholar only includes URLs of the venues, but not of the papers, and Scilit only indexes URLs with a pdf ( pdf _ url ). This also occurs in Dimensions, where the proportion of publications with external links is the same as for open access articles (44.5%).

Proportion of bibliographic records with information about open access and external links.

Fields, publications, and percentage of publications with external links and information about open access by database. Margin of error at 99%

According to open access information, Google Scholar again identifies more open versions (52.2%), followed by Dimensions (44.5%), The Lens (43.6%), and OpenAlex (42.6%). These differences between Google Scholar and the other databases could be because Google Scholar indexes any open copy accessible on the web, regardless of whether the publications were released as open access or not (green open access). Conversely, Semantic Scholar (35.4%) and Scilit (25.4%) capture the fewest open documents. Figure 2 depicts two Venn diagrams showing the overlap between databases according to open access records. Overall, the picture shows that, although the databases index a similar proportion of open access documents, the overlap is not significant. Figure 2(a) shows that OpenAlex and Dimensions share the largest proportion of records (81.1%), whereas OpenAlex and Scilit only have in common 46.1% of the records. This low overlap is surprising because both services use Unpaywall to detect open access publications. A possible explanation would be that Scilit only covers open access publications in pdf format ( unpaywall _ pdf _ url ). Semantic Scholar ( Figure 2(b) ) also shows a disparity with Scilit (49.8%) and OpenAlex (50%). In this case, Semantic Scholar uses its own criterion to detect open access publications.

Overlap among databases identifying open access publications.

5.3. Bibliographic Information

A critical element in a bibliographic database is the correct identification of indexed publications. Journal articles are identified using information that enables the document to be placed into the journal. Volume, issue, and pages are three fields that facilitate correct identification. All the databases include these fields, with the exception of Semantic Scholar, which has no field for issue.

Figure 3 and Table 5 depict the proportion of bibliographic data for journal articles in each database. Google Scholar is not included because it does not provide bibliographic information. In general, all the databases show high rates of completeness, including more information about volume than pages and issues. In this sense, Dimensions is again the platform with the highest completeness rates, with 100% of volume and 94.3% of pages, followed by The Lens with the highest number of pages (94.2%). The most noteworthy result is the low completeness degree of OpenAlex, with 50.2% of issue, 57.6% of pages, and 62.4% of volume. These figures are much lower than those reported by Microsoft Academic, its primary source. A manual inspection confirmed this lack of data, in which almost all the records ingested in December 2022 failed to include this information. This high completeness is also observable in the considerable overlap among confidence intervals, which shows that this information is essential in any scholarly database.

Proportion of bibliographic records with information about volume, pages, and issue.

Percentage and number of articles with bibliographic info (volume, issue, and pages) in each database. Margin of error at 99%

5.4. Document Type

Although journal articles make up more than 70% of the scientific literature, there is a large variety of scholarly documents (book, book chapters, conference papers, etc.) that also provide relevant scientific information and that are incorporated by many scholarly databases in their indexes. Scholarly databases categorize these typologies to inform about the academic nature of each item. However, the range of categories in each database varies significantly. For instance, although Crossref includes 33 document types, Dimensions summarizes its classification in only six classes ( Table 6 ).

Number of document typologies and completeness degree in each database. Margin of error at 99%

Table 6 displays the number of different document types and the number of records categorized in each database. Again, Google Scholar is excluded because the database has no document types. All the publications in Crossref (100%) and Dimensions (100%) are assigned to a typology, and OpenAlex (100%), The Lens (99.2%), and Scilit (99.8%) only find assignment problems in exceptional cases. This high completeness could be due to the fact that all these sources take document typologies from Crossref. Therefore, this lack of statistical differences could be more related to completeness than to data origin.

However, Microsoft Academic (80.3%) and Semantic Scholar (41.3%) present serious problems when classifying their records by typology. A possible explanation is that both search engines extract metadata from the web, and this information is not always available. Worth mentioning is the case of Semantic Scholar, which appears to use an automatic procedure to assign more than one typology based more on content criteria ( Review , Study , CaseReport , etc.) than on formal criteria.

Figure 4 shows different alluvial graphs illustrating document type transfers between the Crossref classification and the systems of each database. The aim is to elucidate how each database assigns document types to their records. To improve the clarity of the graph, only the 10 most common categories in Crossref are displayed. For instance, Dimensions significantly reduces the document categories, integrating book-chapter (99.9%), component (78.7%), reference-entry (100%), and other (100%) into the chapter category, and dataset (100%) and journal-article (99.1%) into article . Microsoft Academic shows important problems when classifying book chapters (46.8%) and proceeding-articles (65.3%). Up to July 2023, OpenAlex directly used the Crossref scheme without any variation. Today, monograph (100%) is entered under book , and some marginal typologies (i.e., others , component , journal , proceeding , journal-issue ) are grouped under the paratext category. Scilit also presents slight variations to the Crossref framework, creating Conference-Paper to group any document related to a conference, regardless of whether it is published as a journal-article , book-chapter , or proceeding-article . Semantic Scholar has serious problems when classifying most of the document typologies because only 46.2% of proceeding-articles are classified as Conference and 35.3% of journal-articles as JournalArticle . Finally, The Lens also shows similarities with the Crossref classification: proceeding-articles are split into conference proceedings (56.2%) and conference proceeding articles (35.7%), and posted-content is entered under other (94.4%).

Alluvial graph with the transfer of document types between Crossref and the other databases. The stratum on the left shows the original Crossref classification and on the right the classification system of each database. Some labels were omitted to avoid overlaps.

5.5. Publication Dates

Electronic publishing has led to the appearance of multiple dates associated with the same document, describing different lifespan stages. This variety of dates also causes problems in the management of these publications ( Ortega, 2022 ). Crossref includes more dates (up to eight), followed by Dimensions with five, and Microsoft Academic and The Lens with four. Crossref ( created ), Dimensions ( date _ inserted ), Microsoft Academic ( CreatedDate ), and The Lens ( created ) display the date when the record was created; and Crossref ( published-print , published-online ), Dimensions ( date _ print , date _ online ), and Scilit ( date _ print , publication _ year ) distinguish between the print and online date.

Publication date is common to all the databases and allows us to analyze the reliability of this information in each. A way to test the accuracy of these data is to compare the matching percentage with Crossref dates. Crossref is directly fed by publishers, which could make it the most authoritative source for exact and correct publication dates. Crossref fields that most match the publication date are published (88%), published-online (6%), and created (5%).

Figure 5 and Table 7 depict the proportion of publication dates that match some Crossref dates (i.e., published-print , published-online , created , deposited , indexed , and issued ) and the percentage of publications with no date. Google Scholar only includes publication year. The comparison is thus done by year, not by date, resulting in a much higher match. That said, Google Scholar matches only 79.6%. The bar graph shows that Scilit (94.9%) and Dimensions (91.9%) have the best match with Crossref, and Microsoft Academic (70.1%) and OpenAlex (72.6%) have lower matching rates. These results could be explained because Dimensions and Scilit take their data from Crossref, and OpenAlex is an adaptation of the Microsoft Academic database. The most interesting result is perhaps the high proportion of publications with no date in Semantic Scholar (57.9%) and The Lens (35.8%). In the case of Semantic Scholar, it could be due to parsing problems when information is extracted from websites. For The Lens, however, this absence of information could be due to technical problems. It has the lowest proportion of no matching publication dates (4.3%), which could indicate that The Lens is also extracting the publication date from Crossref.

Percentage of publication dates matching with Crossref dates.

Percentage and number of articles with and without publication date matching and not matching with Crossref. Margin of error at 99%

5.6. Language

A relevant factor to consider in a scholarly database is the language of the full text. The release of research documents in a language other than English is growing, and there is increasing demand for publications by local research communities. However, this information is only supplied by Crossref ( language ), Microsoft Academic ( LanguageCode ), The Lens ( languages ), and Scilit ( language ). In our study, we extracted this information from Crossref, Scilit, and Microsoft Academic. Technical problems made it impossible to retrieve this information from The Lens. The results show that Scilit is the platform that identifies the language of most publications (99.9%), followed by Microsoft Academic (77.6%) and Crossref (57.1%). A manual inspection of the language assignment seems to indicate that Crossref assigns language according to venue, Scilit according to titles and abstracts, and Microsoft takes the language from webpage metadata.

Figure 6 displays a Venn diagram plotting the overlap between Crossref, Microsoft Academic, and Scilit when identifying the same publication language. The results show that Scilit largely matches Crossref (93.3%) and Microsoft Academic (89.8%), and the coincidence between Crossref (38%) and Microsoft Academic (54.9%) is low. These differences show how the methodological differences between Crossref (venues) and Microsoft Academic (webpages) influence language assignation, and how the use of content elements (title and abstract) improves language detection in Scilit.

Overlap between databases assigning the same language.

5.7. Identifiers

Much of the current proliferation of scholarly databases is due to the consolidation of external identifiers that make it possible to individualize publications (duplicate management) and connect with other sources, thereby enriching the information about publications. Apart from DOIs, many databases index different external identifiers. Semantic Scholar ( externalIds ), The Lens ( external _ ids ), OpenAlex ( ids ), and Microsoft Academic ( AttributeType ) have a specific field for external identifiers. Crossref, Scilit, and Dimensions have different fields for each identifier. Google Scholar provides no identifier.

Figure 7 shows the proportion of the three most frequent identifiers (ISSN, MAG, and PMID) in each database. ISSN is the code that identifies journals and series, MAG is the Microsoft Academic identifier, and PMID is the PubMed ID. The aim is to discover the sources of these databases and to explore the identification of publication venues. The results show that all databases index or identify publications from PubMed to a similar extent, ranging from 21.5% for Crossref to 26.5% for Semantic Scholar ( Table 8 ). Only three databases—OpenAlex, Semantic Scholar, and The Lens—take data from Microsoft Academic. Semantic Scholar (97.8%) and OpenAlex (87.4%) index the most publications. Their differences suggest, on the one hand, that Semantic Scholar is highly dependent on Microsoft Academic and, on the other, that OpenAlex is already using other sources (Crossref) to expand its database. As for ISSN, all the services have a similar coverage of ISSNs (≈80%), although Microsoft Academic (22.9%) and Semantic Scholar (37.4%) have rather low proportions, suggesting deficient journal identification possibly deriving from the web extraction process.

Percentage of different identifiers in each database.

Percentage and number of articles with different identifiers. Margin of error at 99%

This comparative analysis between free-access bibliographic databases has mainly focused on the completeness degree and quality of their metadata. This quantitative approach has made it possible to detect coverage limitations, allowing us to speculate about the technical reasons for these results. The study reports important outcomes about data sources and how information is processed. Overall, the results allow us to distinguish between academic search engines (Google Scholar, Microsoft Academic, Semantic Scholar) and third-party databases (Dimensions, The Lens, Scilit, OpenAlex). The former reveal clear problems with the number of fields that describe publications and their completeness. One explanation could be that these databases mainly obtain their data by crawling the web, and the information that webpages make available could be insufficient to correctly describe a publication. Thus, Semantic Scholar only includes abstracts for 54.5% of its publications and is the service with the lowest proportion of open access documents (35.4%) and external links (39.1%). Almost 60% of its publications have no document type, and it is the database that has the most publications with no publication date (57.9%). These figures reveal that Semantic Scholar has serious problems processing bibliographic information, which could be the root of the low quality of its metadata. In addition, the high proportion of records with the Microsoft Academic id (97.8%) and the similar coverage of PMIDs (Microsoft Academic = 26.2% (±0.4); Semantic Scholar = 26.5% (±0.4)) and no matching dates with Crossref (Microsoft Academic = 29.8% (±0.4); Semantic Scholar = 29.1% (±0.4)) suggests that the core of Semantic Scholar relies on Microsoft Academic and less on crawling the web.

To a lesser extent, Microsoft Academic also shows a low document type classification (79.5%), lack of information about open access publications, and the lowest proportion of ISSNs per publication (22.9%). According to document type, some recent studies observed that more than half of publications include this information ( Färber & Ao, 2022 ; Visser et al., 2021 ). This disparity with our results could be because patents are not included in our study (approximately 20%). It does, however, highlight the coverage of external links (80%). These results are important for understanding how other products, based on their data (Semantic Scholar, The Lens, and OpenAlex), have inherited or solved these problems.

The main issue with Google Scholar is that it provides very little information about publications. Basic information, such as document type, bibliographic information, publication date, or identifiers, is missing from this database. With the exception of citations and versions, Google Scholar barely adds value to their records. However, as a search engine, it is the service that provides the most external links (97.1%) and detects the most open versions (52.2%), thereby confirming that it is the best gateway to scientific literature on the web ( Martín-Martín et al., 2021 ).

Conversely, third-party databases supported on Crossref (Dimensions, The Lens, and Scilit) have more metadata details, with a high completeness degree. Dimensions could be considered the database that most enriches and improves information provided by Crossref. It indexes the most publication abstracts (69.6%), identifies the most open access articles (44.5%) ( Basson et al., 2022 ), has the best coverage of bibliographic info (volume = 100%, issue = 81%, pages = 92%), and 100% of publications are listed according to their typology. These results illustrate that Dimensions endeavors to improve Crossref metadata by adding abstracts, document typology, open access status, and so forth, to their records, with a high completeness rate. However, other sources, such as Scilit and The Lens, show signs of low data processing efficiency. For instance, Scilit is the commercial product that indexes the smallest number of abstracts (50.5%) and the lowest proportion of open access documents (25.4%). The Lens has reported serious problems with publication dates (35.8%). These findings reveal that the main risk of third-party databases is that they require a considerable processing effort to improve the quality of their metadata. A similar case is OpenAlex, which is based on both Microsoft Academic and Crossref. This integration of different sources would cause a loss of information, with a high proportion of missing bibliographic data (volume = 62%, issue = 50%, pages = 51%). The results for OpenAlex indicate that this new database is similar to Microsoft Academic (same proportion of abstracts and dates, the second database with the highest number of Microsoft Academic identifiers), but with the addition of DOIs and document types from Crossref ( Scheidsteger & Haunschild, 2023 ). This active processing of different sources illustrates the importance of these tasks for offering a reliable scholarly bibliographic product ( Priem, Piwowar, & Orr, 2022 ).

Any comparative analysis of bibliographic databases is always limited by the reference database used in the study. In our case, Crossref presents some particularities that should be considered in the interpretation of the results. Crossref is merely a repository of unprocessed publishers’ data. Its coverage is determined by its partners, who decide on the data and how they are deposited, which could influence the representativeness of the sample and the quality of the metadata. A way of lessening this limitation is to make a random selection of the sample. This problem would also influence reliability because publishers may deposit incomplete records, lacking some fields and containing errors, including incorrect or null data about their own publications.

The results lead us to conclude that a random Crossref sample enables the comparison of a wide range of scholarly bibliographic databases, and the benchmarking of the amount of information and completeness degree of these databases, with regard to different facets. Accordingly, the numbers of publications with abstract, external links, open access status, document type, or publication date have been measured across databases. Extraction of the same data from each database has shed light on overlaps, which has led us to identify possible connections between databases.

The results show that databases based on external sources can generate more and improved metadata than academic search engines extracting information from the web. Search engines have the power to reach distant publications and detect more open copies, but they lack the ability to retrieve reliable descriptive data about publications from webpages. However, this integration of different sources also produces problems, such as the loss of information (The Lens and publication date, or OpenAlex and bibliographic info), or limitations inherited from the primary sources (OpenAlex and the publication dates of Microsoft Academic).

Finally, Dimensions provides the greatest number of fields about publications and the highest completeness degree. OpenAlex, The Lens, and Scilit also include a varied range of fields but display certain integration problems, with a lack of information and low completeness rates in specific fields. Conversely, search engines such as Semantic Scholar and Google Scholar lack important fields for identifying and searching publications (document types, certain bibliographic info). Microsoft Academic is the search engine that provides the most publication fields, and its completeness rate is high, though it lacks information on open access status and document type for some publications.

Lorena Delgado-Quirós: Data curation, Formal analysis, Resources, Software. José Luis Ortega: Conceptualization, Funding acquisition, Investigation, Methodology, Writing—original draft, Writing—review & editing.

José Luis Ortega is member of the Scilit advisory board, which facilitated access to API documentation.

This work was supported by the research project (NewSIS) “New scientific information sources: Analysis and evaluation for a national scientific information system” (Ref. PID2019-106510GB-I00) funded by the Spanish State Research Agency (AEI) PN2019.

For legal reasons, data from many of the databases (Dimensions, Google Scholar, Scilit, and The Lens) cannot be made openly available. Data from Crossref, OpenAlex, Microsoft Academic, and Semantic Scholar are openly available because they have been released under a CC-BY license. We have uploaded the instructions on how to retrieve the data in each database ( https://osf.io/yw6j4 ). In this form, readers with credentials can download the data and reproduce the study.

https://github.com/massimoaria/dimensionsR .

https://docs.ropensci.org/RSelenium/ .

https://docs.ropensci.org/microdemic/ .

https://github.com/njahn82/semscholar .

Author notes

Email alerts, related articles, affiliations.

• Online ISSN 2641-3337

A product of The MIT Press

Mit press direct.

• About MIT Press Direct

Information

• Accessibility
• For Authors
• For Customers
• For Librarians
• Direct to Open
• Open Access
• Media Inquiries
• Rights and Permissions
• For Advertisers
• About the MIT Press
• The MIT Press Reader
• MIT Press Blog
• Seasonal Catalogs
• MIT Press Home
• Give to the MIT Press
• Direct Service Desk
• Terms of Use
• Privacy Statement
• Crossref Member
• COUNTER Member  
• The MIT Press colophon is registered in the U.S. Patent and Trademark Office

This Feature Is Available To Subscribers Only

Sign In or Create an Account