how to select relevant literature in quantitative research

Archer Library

Quantitative research: literature review .

• Archer Library This link opens in a new window
• Locating Books
• Library eBook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Statistics
• Literature Review Resources
• Citations & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

Archer Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: scholarly & practitioner journals.

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Basic Searching with EBSCO
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Statistics
• Next: Literature Review Resources >>
• Last Updated: Nov 17, 2023 7:46 AM
• URL: https://libguides.ashland.edu/quantitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

• Chester Fritz Library
• Library of the Health Sciences
• Thormodsgard Law Library

Best Practices for Communicating Your Research

• Step 1: Your Manuscript
• Step 2: Authorship
• Step 3: Research Transparency

Search Strategies

Types of literature, search techniques, documenting & organizing, references & further reading.

• Library Books
• Dissertations & Theses This link opens in a new window
• Topics in Scholarly Communication This link opens in a new window
• Literature Reviews This link opens in a new window

On This Page:

Your search strategy, search strategy resources.

• Documenting & Organizing Your Search

Your search strategy for identifying appropriate research and literature for your literature review will change depending on your topic, the type of research you are conducting, the manuscript you plan to write, and the particular resource/database you are searching. It will also depend on how much time you have to conduct the search--is this a short- or long-term project or research paper? Does it require an initial scoping review? Read more about review types in Step 1 . You will need to identify appropriate (both discipline-specific and general) research databases, and a variety of search terms and limiters (e.g., population, geographic region, date) according to your research type, topic, and manuscript type.

Though the literature search is listed as Step 4 in this guide, often, you will have begun a brief, investigative literature search or scoping review before engaging in transparent research and pre-registering your research ( Step 3 ) as you need to be able to have some idea of how much literature there is out there on your topic before you solidify your hypothesis or systematic review question. This knowledge of pre-existing literature is of the utmost important when completing a thesis or dissertation. If you are conducting a short-term project with a looming deadline that does not need to be comprehensive as far as finding and synthesizing all topical existing/previous research, you can conduct a more economical but relevant search of the literature by targeting 2-4 specific databases and using carefully selected search terms.

Defining Your Research Question

It is integral to spend time honing and defining your research question before searching the literature. Highly recommended by faculty and librarians alike, are these two tools to help you define your research question:

• PICOT  / PICO  (quantitative evidence-based research/synthesis) and
• SPIDER  (qualitative evidence-based research/synthesis) 

PICO (Quantitative) and SPIDER (Qualitative)

Cooke, Smith, & Booth (2012).

*The "T" (PICOT) is left out of the above study. It represents Time, or the duration of data collection (Riva, Malik, Burnie, Endicott, & Busse, 2012)

Engineering PICO*

P = Population, Problem, Process

I = Intervention, Inquiry, Investigation, Improvement

C = Comparison (current practice or opposing viewpoints)

O = Outcomes (measuring what worked best)

*Read more about it on Arizona State University Library's "Engineering -- Formulating questions w/PICO" guide:  https://libguides.asu.edu/engineering/PICO

Some of the best sources for learning how to conduct a systematic review of the literature, or a review for the introduction of a research report, are in books. Check out the  Library Books  page for some examples. Relevant research articles are also included on this page. Your  Subject Librarian  is here to help - set up a research consultation with us today!

Strategies for searching the literature for your literature review change depending upon the type of research you are conducting:  primary research  or  research synthesis.   Whether you are collecting new data (primary research), or synthesizing results of previous studies (review articles, including systematic reviews and meta-analyses), the first, and most important step before searching the literature, is to: define the target population (or thing) you are investigating in the case of  primary research , or all studies that test a particular hypothesis or address a particular problem in the case of  research synthesis  (Cooper, 2010). 

Where Will You Look?

The literature you gather greatly depends upon the sources that you look in. Studies appearing in peer-reviewed journals are easy to locate but will likely over-represent significant and novel results, while certain types of grey literature (e.g., dissertations and theses; self-published manuscripts; unpublished studies; conference abstracts, presentations, and proceedings; regulatory data; unpublished trial data; government publications; reports such as white papers, working papers, and internal documentation; patents; and policies & procedures) may be more difficult to find and access in full text--for example, you may need to contact authors or organizations directly. It is good practice to use listserv and distribution lists for this type of material along with direct personal contacts, keeping in mind that the latter may bias the results towards those in support of a particular contact's central beliefs and research results (Cooper, 2010).

Obviously, this means that limiting your search to journals in databases may skew results towards statistically significant findings, biasing your pool of studies which would be lacking in null, or inconclusive, results. You can also search for grey literature in institutional repositories like UND Scholarly Commons , government/professional organizations and conference websites, Open Data Repositories , open preprint repositories, theses and dissertation databases, online Researcher Communities , and journals that publish Registered Reports  or null and inconclusive findings like PLOS ONE .

Author's Versions & Grey Literature Database Examples:

Dissertations from 1861 - present. Master's theses from 1988 - present. Includes full text from both UND and external dissertations and theses. Hosted by Proquest.

• Open Science Framework Search Search OSF projects and data files. OSF is a free and open source project management repository that supports researchers across their entire project lifecycle.

How Will You Look?

Conventional subject searching in databases.

A full, systematic search involves the development of a search strategy around subject terms, reflecting aspects of the research question. Searches are typically carried out in several or more multidisciplinary or discipline-specific databases. Searches are often restricted by language and date, and sometimes geographic region, through the use of database  limiters . Your subject search will likely identify the majority of references ultimately included in your review, however,  additional search methods, as described below, are necessary to locate additional unique and appropriate sources for your review.

Citation Chaining & Citation Searching (Backward & Forward Snowballing)

These techniques refer to checking reference lists and citing articles (articles that have cited the article that you are currently looking at). Citation chaining involves checking references on all included papers identified by various search methods so that relevant references not yet identified can be added to the pool of included studies. It also includes checking articles that cited an included paper. Many research databases link citing articles to each article record. Databases that are useful for citation searching include Google Scholar, Scopus, CINAHL Complete, Wiley Online, and others. Access Chester Fritz Library's most used databases by visiting our  home page   and clicking on QuickLinks or the complete list by visiting our  A-Z Databases page .

Traditional vs. Comprehensive Pearl Growing

Traditional Pearl Growing (TPG) begins with one or more target articles, judged to be such due to their relevancy to the research topic. The target article is called a pearl. It's a step beyond the citation chaining and searching methods. The researcher then identifies keywords to add to their search from aspects of the article (e.g., abstract, subject terms, author, etc.). Hawkins and Wagers (1982) coined this process as "growing more pearls" (as cited in  Schlosser,  Wendt ,  Bhavnani , & Nail‐Chiwetalu, 2006).

Comprehensive Pearl Growing (CPG) involves the following process: (1) Start with a compilation of studies from a relevant review or a topical bibliography; (2) determine relevant databases for these studies; (3) determine how these studies are indexed in database 1 in terms of keywords and quality filters; (4) find other relevant articles in database 1 (or as many are relevant) using the index terms in a Building Block query; and (5) end when articles retrieved provide diminishing relevance. Thus, rather than beginning with only one pearl, CPG requires of the searcher to begin with a compilation of studies from a relevant narrative review or a topical bibliography. Like TPG, CPG makes use of existing studies to determine the keywords and quality filters under which they are indexed in order to retrieve more articles of the same kind   (Schlosser, Wendt, Bhavnani, & Nail‐Chiwetalu, 2006).

Although pearl growing techniques are effective across disciplines, they may be particularly strategic for interdisciplinary research questions in which multiple controlled vocabularies (e.g., thesauri, database subject terms, discipline-specific terminology), are integral to pulling together  sources across research databases (Schlosser, Wendt, Bhavnani , & Nail‐Chiwetalu, 2006).

Text-Mining

In Software Engineering, various text-mining (TM) techniques are used more and more to implement systematic literature review processes, however further research is needed--read Feng, Chiam, and Lo (2017) linked below for more information.

Check out our   Library Books  on reviews and research synthesis to take your literature search to the next level by creating coding criteria for systematic reviews and meta-analyses.

Document Your Literature Search

Use paper and pen, the below excel file, or online tools or applications like Trello to set up a system for documenting your search strategy. This contributes to research transparency and gives you a mechanism to provide quick and accurate documentation of your search strategies when pre-registering systematic review protocol or being questioned about how you searched the literature (and what you may have missed) by supervisors, colleagues, or reviewers.

• Search Strategy Documentation Template Be systematic by documenting your search strategy (keywords, databases, etc.). This helps you to remember what you have done before and provides documentation for research transparency.

For systematic reviews or meta-analyses, use the PRISMA or MOOSE checklists to evaluate each included resource for inclusion.

• PRISMA Checklist Transparent Reporting of Systematic Reviews and Meta-analyses.
• MOOSE Guidelines for Meta-Analyses and Systematic Reviews of Observational Studies* *Modified from Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283:2008–12.

Organize Your Literature

• Citation Managers A research guide providing information and resources for a handful of the most popular citation managers.
• EndNote @ UND Page of the Citation Managers research guide discussing UND's EndNote subscription and use.
• Mendeley Mendeley (Elsevier) is a free reference manager and an academic social network. Manage your research, showcase your work, connect and collaborate.
• Zotero Zotero is a free, easy-to-use citation management tool to help you collect, organize, cite, and share research.
• Beyond PICO: the SPIDER tool for qualitative evidence synthesis Cooke, A., Smith, D., & Booth, A. (2012). Qualitative Health Research, 22(10), 1435-1443.
• Checking reference lists to find additional studies for systematic reviews Horsley, T., Dingwall, O., & Sampson, M. (2011). The Cochrane Library, 8.
• Domain Definition and Search Techniques in Meta-analyses of L2 Research (Or why 18 meta-analyses of feedback have different results) Plonsky, L., & Brown, D. (2015). Second Language Research, 31(2), 267–278.
• Effectiveness and Efficiency of Search Methods in Systematic Reviews of Complex Evidence: Audit of Primary Sources Greenhalgh, T., & Peacock, R. (2005). BMJ, 331(7524), 1064-1065. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by “snowballing” (such as pursuing references of references), and 24% by personal knowledge or personal contacts. Conclusion: Systematic reviews of complex evidence cannot rely solely on protocol-driven search strategies.
• An Empirical Assessment of A Systematic Search Process for Systematic Reviews Zhang, H., Babar, M. A., Bai, X., Li, J., & Huang, L. (2011, April). In Evaluation & Assessment in Software Engineering (EASE 2011), 15th Annual Conference on (pp. 56-65). IET.
• The Impact of Limited Search Procedures for Systematic Literature Reviews – A Participant-Observer Case Study Kitchenham, B., Brereton, P., Turner, M., Niazi, M., Linkman, S., Pretorius, R., & Budgen, D. (2009, October). In Empirical Software Engineering and Measurement, 2009. ESEM 2009. 3rd International Symposium on (pp. 336-345). IEEE.
• Information retrieval in systematic reviews: Challenges in the public health arena Beahler, C. C., Sundheim, J. J., & Trapp, N. I. (2000). American Journal of Preventive Medicine, (18)4, 6-10.
• Literature Searching for Social Science Systematic Reviews: Consideration of a range of search techniques. Papaioannou, D. , Sutton, A. , Carroll, C. , Booth, A. and Wong, R. (2010). Health Information & Libraries Journal, 27, 114-122.
• Literature search strategies for conducting knowledge‐building and theory‐generating qualitative systematic reviews Finfgeld‐Connett, D., & Johnson, E. D. (2013). Journal of Advanced Nursing, 69(1), 194-204.
• Performing a Literature Review. Reed, L. E. (1998, November). In fie (pp. 380-383). IEEE.
• Searching for qualitative research for inclusion in Systematic Reviews: A Structured Methodological Review Booth, A. (2016). Systematic Reviews, (5)74, 1-23.
• Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews Carroll, C., Booth, A., & Lloyd-Jones, M. (2012). Qualitative Health Research, 22(10), 1425-1434.
• Systematic Literature Studies: Database Searches vs. Backward Snowballing Jalali, S., & Wohlin, C. (2012, September). In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement (pp. 29-38). ACM.
• Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review Feng, L., Chiam, Y. K., & Lo, S. K. (2017, December). In Asia-Pacific Software Engineering Conference (APSEC), 2017 24th (pp. 41-50). IEEE. Also available open access: http://eprints.um.edu.my/18515/1/All.pdf
• Use of information-seeking strategies for developing systematic reviews and engaging in evidence-based practice: the application of traditional and comprehensive Pearl Growing. A review Schlosser, R. W., Wendt, O., Bhavnani, S., & Nail‐Chiwetalu, B. (2006). International Journal of Language & Communication Disorders, 41(5), 567-582.
• What is your research question? An introduction to the PICOT format for clinicians. Riva, J. J., Malik, K. M., Burnie, S. J., Endicott, A. R., & Busse, J. W. (2012). The Journal of the Canadian Chiropractic Association, 56(3), 167-71.
• << Previous: Step 3: Research Transparency
• Next: References & Resources >>
• Last Updated: Aug 17, 2023 12:20 PM
• URL: https://libguides.und.edu/communicating-research

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

How to Operate Literature Review Through Qualitative and Quantitative Analysis Integration?

• Conference paper
• First Online: 05 May 2022
• Cite this conference paper

• Eduardo Amadeu Dutra Moresi   ORCID: orcid.org/0000-0001-6058-3883 13 ,
• Isabel Pinho   ORCID: orcid.org/0000-0003-1714-8979 14 &
• António Pedro Costa   ORCID: orcid.org/0000-0002-4644-5879 14  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 466))

Included in the following conference series:

• World Conference on Qualitative Research

460 Accesses

2 Citations

Usually, a literature review takes time and becomes a demanding step in any research project. The proposal presented in this article intends to structure this work in an organised and transparent way for all project participants and the structured elaboration of its report. Integrating qualitative and quantitative analysis provides opportunities to carry out a solid, practical, and in-depth literature review. The purpose of this article is to present a guide that explores the potentials of qualitative and quantitative analysis integration to develop a solid and replicable literature review. The paper proposes an integrative approach comprising six steps: 1) research design; 2) Data Collection for bibliometric analysis; 3) Search string refinement; 4) Bibliometric analysis; 5) qualitative analysis; and 6) report and dissemination of research results. These guidelines can facilitate the bibliographic analysis process and relevant article sample selection. Once the sample of publications is defined, it is possible to conduct a deep analysis through Content Analysis. Software tools, such as R Bibliometrix, VOSviewer, Gephi, yEd and webQDA, can be used for practical work during all collection, analysis, and reporting processes. From a large amount of data, selecting a sample of relevant literature is facilitated by interpreting bibliometric results. The specification of the methodology allows the replication and updating of the literature review in an interactive, systematic, and collaborative way giving a more transparent and organised approach to improving the literature review.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Pritchard, A.: Statistical bibliography or bibliometrics? J. Doc. 25 (4), 348–349 (1969)

Google Scholar  

Nalimov, V., Mulcjenko, B.: Measurement of Science: Study of the Development of Science as an Information Process. Foreign Technology Division, Washington DC (1971)

Hugar, J.G., Bachlapur, M.M., Gavisiddappa, A.: Research contribution of bibliometric studies as reflected in web of science from 2013 to 2017. Libr. Philos. Pract. (e-journal), 1–13 (2019). https://digitalcommons.unl.edu/libphilprac/2319

Verma, M.K., Shukla, R.: Library herald-2008–2017: a bibliometric study. Libr. Philos. Pract. (e-journal), 2–12 (2018). https://digitalcommons.unl.edu/libphilprac/1762

Pandita, R.: Annals of library and information studies (ALIS) journal: a bibliometric study (2002–2012). DESIDOC J. Libr. Inf. Technol. 33 (6), 493–497 (2013)

Article   Google Scholar  

Kannan, P., Thanuskodi, S.: Bibliometric analysis of library philosophy and practice: a study based on scopus database. Libr. Philos. Pract. (e-journal), 1–13 (2019). https://digitalcommons.unl.edu/libphilprac/2300/

Marín-Marín, J.-A., Moreno-Guerrero, A.-J., Dúo-Terrón, P., López-Belmonte, J.: STEAM in education: a bibliometric analysis of performance and co-words in Web of Science. Int. J. STEM Educ. 8 (1) (2021). Article number 41

Khalife, M.A., Dunay, A., Illés, C.B.: Bibliometric analysis of articles on project management research. Periodica Polytechnica Soc. Manag. Sci. 29 (1), 70–83 (2021)

Pech, G., Delgado, C.: Screening the most highly cited papers in longitudinal bibliometric studies and systematic literature reviews of a research field or journal: widespread used metrics vs a percentile citation-based approach. J. Informet. 15 (3), 101161 (2021)

Das, D.: Journal of informetrics: a bibliometric study. Libr. Philos. Pract. (e-journal), 1–15 (2021). https://digitalcommons.unl.edu/libphilprac/5495/

Schmidt, F.: Meta-analysis: a constantly evolving research integration tool. Organ. Res. Methods 11 (1), 96–113 (2008)

Zupic, I., Cater, T.: Bibliometric methods in management organisation. Organ. Res. Methods 18 (3), 429–472 (2014)

Noyons, E., Moed, H., Luwel, M.: Combining mapping and citation analysis for evaluative bibliometric purposes: a bibliometric study. J. Am. Soc. Inf. Sci. 50 , 115–131 (1999)

van Rann, A.: Measuring science. Capita selecta of current main issues. In: Moed, H., Glänzel, W., Schmoch, U. (eds.) Handbook of Quantitative Science and Technology Research, pp. 19–50. Kluwer Academic, Dordrecht (2004)

Chapter   Google Scholar  

Garfield, E.: Citation analysis as a tool in journal evaluation. Science 178 , 417–479 (1972)

Hirsch, J.: An index to quantify an individuals scientific research output. In: Proceedings of the National Academy of Sciences, vol. 102, pp. 16569–1657. National Academy of Sciences, Washington DC (2005)

Cobo, M., López-Herrera, A., Herrera-Viedma, E., Herrera, F.: Science mapping software tools: review, analysis and cooperative study among tools. J. Am. Soc. Inform. Sci. Technol. 62 , 1382–1402 (2011)

Noyons, E., Moed, H., van Rann, A.: Integrating research perfomance analysis and science mapping. Scientometrics 46 , 591–604 (1999)

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., Lim, W.M.: How to conduct a bibliometric analysis: an overview and guidelines. J. Bus. Res. 133 , 285–296 (2021)

Aria, M., Cuccurullo, C.: Bibliometrix: an R-tool for comprehensive science mapping analysis. J. Informet. 11 (4), 959–975 (2017)

Aria, M., Cuccurullo, C.: Package ‘bibliometrix’. https://cran.r-project.org/web/packages/bibliometrix/bibliometrix.pdf . Accessed 10 July 2021

Börner, K., Chen, C., Boyack, K.: Visualisingg knowledge domains. Ann. Rev. Inf. Sci. Technol. 37 , 179–255 (2003)

Morris, S., van der Veer Martens, B.: Mapping research specialities. Ann. Rev. Inf. Sci. Technol. 42 , 213–295 (2008)

Zitt, M., Ramanana-Rahary, S., Bassecoulard, E.: Relativity of citation performance and excellence measures: from cross-field to cross-scale effects of field-normalisation. Scientometrics 63 (2), 373–401 (2005)

Li, L.L., Ding, G., Feng, N., Wang, M.-H., Ho, Y.-S.: Global stem cell research trend: bibliometric analysis as a tool for mapping trends from 1991 to 2006. Scientometrics 80 (1), 9–58 (2009)

Ebrahim, A.N., Salehi, H., Embi, M.A., Tanha, F.H., Gholizadeh, H., Motahar, S.M.: Visibility and citation impact. Int. Educ. Stud. 7 (4), 120–125 (2014)

Canas-Guerrero, I., Mazarrón, F.R., Calleja-Perucho, C., Pou-Merina, A.: Bibliometric analysis in the international context of the “construction & building technology” category from the web of science database. Constr. Build. Mater. 53 , 13–25 (2014)

Gaviria-Marin, M., Merigó, J.M., Baier-Fuentes, H.: Knowledge management: a global examination based on bibliometric analysis. Technol. Forecast. Soc. Chang. 140 , 194–220 (2019)

Heradio, R., Perez-Morago, H., Fernandez-Amoros, D., Javier Cabrerizo, F., Herrera-Viedma, E.: A bibliometric analysis of 20 years of research on software product lines. Inf. Softw. Technol. 72 , 1–15 (2016)

Furstenau, L.B., et al.: Link between sustainability and industry 4.0: trends, challenges and new perspectives. IEEE Access 8 , 140079–140096 (2020). Article 9151934

van Eck, N.J., Waltman, L.: VOSviewer manual. Universiteit Leiden, Leiden (2021)

Bastian, M., Heymann, S., Jacomy, M.: Gephi: an open source software for exploring and manipulating networks. In: Proceedings of the Third International ICWSM Conference, pp. 361–362. Association for the Advancement of Artificial Intelligence, San Jose CA (2009)

Chen, C.: How to use CiteSpace. Leanpub, Victoria, British Columbia, CA (2019)

yWorks.: yEd Graph Editor Manual. https://yed.yworks.com/support/manual/index.html . Accessed 13 July 2020

Moresi, E.A.D., Pierozzi Júnior, I.: Representação do conhecimento para ciência e tecnologia: construindo uma sistematização metodológica. In: 16th International Conference on Information Systems and Technology Management, TECSI, São Paulo SP (2019). Article 6275

Moresi, E.A.D., Pinho, I.: Proposta de abordagem para refinamento de pesquisa bibliográfica. New Trends Qual. Res. 9 , 11–20 (2021)

Moresi, E.A.D., Pinho, I.: Como identificar os tópicos emergentes de um tema de investigação? New Trends Qual. Res. 9 , 46–55 (2021)

Chen, Y.H., Chen, C.Y., Lee, S.C.: Technology forecasting of new clean energy: the example of hydrogen energy and fuel cell. Afr. J. Bus. Manag. 4 (7), 1372–1380 (2010)

Ernst, H.: The use of patent data for technological forecasting: the diffusion of CNC-technology in the machine tool industry. Small Bus. Econ. 9 (4), 361–381 (1997)

Chen, C.: Science mapping: a systematic review of the literature. J. Data Inf. Sci. 2 (2), 1–40 (2017)

Prabhakaran, T., Lathabai, H.H., Changat, M.: Detection of paradigm shifts and emerging fields using scientific network: a case study of information technology for engineering. Technol. Forecast. Soc. Change 91 , 124–145 (2015)

Klavans, R., Boyack, K.W.: Identifying a better measure of relatedness for mapping science. J. Am. Soc. Inf. Sci. 57 (2), 251–263 (2006)

Kauffman, J., Kittas, A., Bennett, L., Tsoka, S.: DyCoNet: a Gephi plugin for community detection in dynamic complex networks. PLoS ONE 9 (7), e101357 (2014)

Grant, M.J., Booth, A.: A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info. Libr. J. 26 (2), 91–108 (2009)

Costa, A.P., Soares, C.B., Fornari, L., Pinho, I.: Revisão da Literatura com Apoio de Software - Contribuição da Pesquisa Qualitativa. Ludomedia, Aveiro Portugal (2019)

Tranfield, D., Denyer, D., Smart, P.: Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manag. 14 (3), 207–222 (2003)

Costa, A.P., Amado, J.: Content Analysis Supported by Software. Ludomedia, Oliveira de Azeméis - Aveiro - Portugal (2018)

Pinho, I., Leite, D.: Doing a literature review using content analysis - research networks review. In: Atas CIAIQ 2014 - Investigação Qualitativa em Ciências Sociais, vol. 3, pp. 377–378. Ludomedia, Aveiro Portugal (2014)

White, M.D., Marsh, E.E.: Content analysis: a flexible methodology. Libr. Trends 55 (1), 22–45 (2006)

Souza, F.N., Neri, D., Costa, A.P.: Asking questions in the qualitative research context. Qual. Rep. 21 (13), 6–18 (2016)

Pinho, I., Pinho, C., Rosa, M.J.: Research evaluation: mapping the field structure. Avaliação: Revista da Avaliação da Educação Superior (Campinas) 25 , 546–574 (2020)

Costa, A., Moreira, A. de Souza, F.: webQDA - Qualitative Data Analysis (2019). www.webqda.net

Download references

Author information

Authors and affiliations.

Catholic University of Brasília, Brasília, DF, 71966-700, Brazil

Eduardo Amadeu Dutra Moresi

University of Aveiro, 3810-193, Aveiro, Portugal

Isabel Pinho & António Pedro Costa

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Eduardo Amadeu Dutra Moresi .

Editor information

Editors and affiliations.

Department of Education and Psychology, University of Aveiro, Aveiro, Portugal

António Pedro Costa

António Moreira

Department Didactics, Organization and Research Methods, University of Salamanca, Salamanca, Salamanca, Spain

Maria Cruz Sánchez‑Gómez

Adventist University of Africa, Nairobi, Kenya

Safary Wa-Mbaleka

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Moresi, E.A.D., Pinho, I., Costa, A.P. (2022). How to Operate Literature Review Through Qualitative and Quantitative Analysis Integration?. In: Costa, A.P., Moreira, A., Sánchez‑Gómez, M.C., Wa-Mbaleka, S. (eds) Computer Supported Qualitative Research. WCQR 2022. Lecture Notes in Networks and Systems, vol 466. Springer, Cham. https://doi.org/10.1007/978-3-031-04680-3_13

Download citation

DOI : https://doi.org/10.1007/978-3-031-04680-3_13

Published : 05 May 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-04679-7

Online ISBN : 978-3-031-04680-3

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Finding Relevant Scholarly Research for Literature Review: How can we be systematic?

On an average, it takes 15 clicks for a researcher to find an article (which may or may not be related to their research topic) online. This time is not productive because it does not help them gain any knowledge and it could potentially be spent doing something more vital in fostering research and development. Moreover, as most researchers rely on two to three databases to find information for their literature review , the time to find relevant scholarly research data also increases.

Table of Contents

Navigating Through Multiple Scholarly Databases—Is it even necessary?

The Internet has revolutionized the way we access information. Websites and online resources within and outside of academic bibliography are significant resources of literature. However, the challenge in searching and managing the results is undeniable.

Considering the exponential growth in scholarly research data and literature, finding relevant information and reporting your research sooner is imperative. While Open Science has been a positive reform of information access, not all data is available at a click, let alone the relevant one. Researchers fear the possibility of missing out on critical information related to their research topic or accidentally committing plagiarism. Hence, they spend time in toggling through multiple scholarly databases.

In this process of searching for literature on multiple databases, researchers tend to download irrelevant information too. Furthermore, the probability of finding similar resources on multiple databases is higher if the resource is on an Open Access platform. These downloaded papers not only occupy the space in reference managers but also make researchers spend a lot of time deciding whether the paper is worth reading or not.

5 Major Challenges Faced on Multiple Scholarly Databases—How to overcome them?

Finding scholarly research data involves navigating through institutional login pages, subscriptions, and paywalls. Apart from the time, effort, and money spent there are several other challenges that researchers encounter while searching literature on multiple scholarly databases.

Here we discuss 5 major challenges faced by researchers while using multiple scholarly databases:

1. Identifying and Deciding the Resources to Search

The Internet provides information in numerous formats, viz. journal articles, preprints, video recordings, podcasts, infographics, conference proceedings, etc. This wide pool of knowledge gets deeper with advances in scholarly research and literature. Hence, while finding research data on multiple scholarly databases in multiple formats, it becomes difficult to identify and decide the resources to download based on their relevance to the research topic. However, these resources can be easily traced if they all are on a single platform.

2. Search or Navigate Resources Correctly

Researchers use keywords and questions to find scholarly data related to their topic of interest. Databases search for the exact words and phrases. Hence, if researchers use a different word or a synonym that describes the concept, the search results are not relevant. If a single database with optimized keywords is used to access billions of scholarly resources, it not only avoids information overload but also allows navigation of relevant information.

3. Assessing Obtained Search Results

Information overload makes it difficult for researchers to assess every discovered resource. One cannot decide the relevancy of search results based on the research paper ’s title. And reading all sections of all papers—abstract, introduction, results, and/or conclusion—will be extremely time consuming. Furthermore, spending time reading these sections of papers to later find out that it’s not related to your research topic will not help anyone. So, what if there was a tool that could search results beyond keywords using research ideas, questions, etc., and also could summarize the key aspects of each downloaded resource? Definitely something to ponder about.

4. Deciding Which Literature to Select and Cite

Scientists are often overwhelmed with the scholarly research data they find online. It is a never-ending task to decide which literature to select and cite. Thus, it is essential to download only relevant data and assess them based on their relevance to the research topic. Furthermore, citing the literature accurately by following journal-specific guidelines and writing style guides will avoid accidental plagiarism. Such cumbersome tasks can be handled with accuracy using an AI-based tool particularly designed to make academic research and publishing easier.

5. Retrieving Relevant Literature in an Accessible and Editable Format

The inability of some software to save, process, and/or retrieve data in all formats is displeasing in this age of digitization. Hence, scientists prefer software that allows accessing, downloading, managing, and editing research data files in all formats.

How to Find Scholarly Research Data with a Systematic Approach? – 7 simple steps

Given the amount of intelligence on the internet, it is only wise to resort to a reliable system. One which is smart, efficient, precise, accessible, and affordable to integrate the scattered information, help researchers through every step of research reporting and publishing, and save time, effort, and money.

A simple 7-step systematic approach to find relevant scholarly research data

• Search literature based on research ideas, keywords, conference talks, author details, etc.
• Assess the found resources based on their key aspects and findings.
• Search, save, manage, read, and annotate relevant literature on a single platform.
• Use easily accessible and editable formats.
• Cite the literature to avoid plagiarism.
• Follow journal guidelines and format the research paper.
• Connect with co-authors and share your work with them for insights and edits.

An extensive and accurate literature search is the key to performing, reporting, and publishing authentic research. A systematic single-platform search database provides a much better comprehension of insights of the research topic. It helps draw comparisons faster as all results are saved and managed in one place! Moreover, it helps researchers to stimulate the interpretation of ideas, analyze shortcomings, and recognize opportunities of future research.

With advances in technology, this process can be simplified without compromising the quality of the final product. As Artificial Intelligence takes over other realms of society, it’s about time researchers leverage these advances to further streamline research publishing.

What are your ways of literature search? How many databases do you have to use simultaneously? Wouldn’t you want to have all your work on one platform without remembering several login IDs and passwords? This sounds like the future of publishing! What do you think?

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• AI in Academia
• Trending Now

Using AI for Journal Selection — Simplifying your academic publishing journey in the smart way

Strategic journal selection plays a pivotal role in maximizing the impact of one’s scholarly work.…

• Old Webinars
• Webinar Mobile App

Using AI for Plagiarism Prevention and AI-Content Detection in Academic Writing

Watch Now   In today’s digital age, where information is readily accessible and content creation…

• Thought Leadership

Authenticating AI-Generated Scholarly Outputs: Practical approaches to take

The rapid advancements in artificial intelligence (AI) have given rise to a new era of…

• Industry News

ICMJE Updates Guidelines for Medical Journal Publication, Emphasizes on Inclusivity and AI Transparency

The International Committee of Medical Journal Editors (ICMJE) recently updated its recommendations for best practices…

• Reporting Research

AI for Accuracy in Research and Academic Writing

Importance of Achieving Accuracy in Research Challenges Faced by Researchers Various Capabilities of AI Tools…

Upholding Academic Integrity in the Age of AI: Challenges and solutions

AI Assistance in Academia for Searching Credible Scholarly Sources

Call for Responsible Use of Generative AI in Academia: The battle between rising…

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

• JABSOM Library

Literature Review Basics and Searching Skills

• Selecting Criteria
• Getting Started
• Developing Research Questions
• Selecting Databases
• Searching Databases
• Documenting Searches
• Organizing Findings
• Managing Citations & References
• Submitting Your Manuscript
• Finding Help

Selection Criteria

You may want to think about criteria that will be used to select articles for your literature review based on your research question. These are commonly known as  inclusion criteria  and  exclusion criteria . Be aware that you may introduce bias into the final review if these are not used thoughtfully.

Inclusion Criteria

Inclusion criteria are the elements of an article that must be present in order for it to be eligible for inclusion in a literature review.  Some examples are:

• Included studies must have compared certain treatments
• Included studies must be experimental
• Included studies must have been published in the last 5 years

Exclusion Criteria

Exclusion criteria are the elements of an article that disqualify the study from inclusion in a literature review.  Some examples are:

• Study used an observational design
• Study used a qualitative methodology
• Study was published more than 5 years ago
• Study was published in a language other than English
• << Previous: Developing Research Questions
• Next: Selecting Databases >>
• Last Updated: Mar 20, 2024 11:38 AM
• URL: https://hslib.jabsom.hawaii.edu/lit_review

Health Sciences Library, John A. Burns School of Medicine, University of Hawai‘i at Mānoa, 651 Ilalo Street, MEB 101, Honolulu, HI 96813 - Phone: 808-692-0810, Fax: 808-692-1244

Copyright © 2004-2024. All rights reserved. Library Staff Page - Other UH Libraries

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 21, Issue 4
• How to appraise quantitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

This article has a correction. Please see:

• Correction: How to appraise quantitative research - April 01, 2019

• Xabi Cathala 1 ,
• Calvin Moorley 2
• 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

• View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

• 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
• Gerrish K ,
• Moorley C ,
• Tunariu A , et al
• Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

• Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF:

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.3 Critically Appraising the Literature

Now that you know the parts of a paper, we will discuss how to critically appraise a paper. Critical appraisal refers to the process of carefully and methodically reviewing research to determine its credibility, usefulness, and applicability in a certain context. 6 It is an essential element of evidence-based practice. As stated earlier, you want to ensure that what you read in the literature is trustworthy before considering applying the findings in practice. The key things to consider include the study’s results, if the results match the conclusion (validity) and if the findings will help you in practice (applicability). A stepwise approach to reading and analysing the paper is a good way to highlight important points in the paper. While there are numerous checklists for critical appraisal, we have provided a simple guide for critical appraisal of quantitative and qualitative studies. The guides were adapted from Epidemiology by Petra Buttner (2015) and How to Read a Paper [ the basics of evidence-based medicine and healthcare (2019);  papers that go beyond numbers- qualitative research (1997)] by Trisha Greenhalgh to aid your review of the papers. 5,7,8

A guide to reading scientific articles – Quantitative studies

What is the title of the study?

• Does the title clearly describe the study focus?
• Does it contain details about the population and the study design?

What was the purpose of the study (why was it performed)?

• Identify the research question
• Identify the exposure and outcome

What was the study design?

• Was the design appropriate for the study?

Describe the study population (sample).

• What was the sample size?
• How were participants recruited?
• Where did the research take place?
• Who was included, and who was excluded?
• Are there any potential sources of bias related to the choice of the sample?

What were data collection methods used?

• How were the exposure and outcome variables were measured
• How was data collected- instruments or equipment? Were the tools appropriate?
• Is there evidence of random selection as opposed to systematic or self-selection?
• How was bias minimised or avoided?

For experimental studies

•  How were subjects assigned to treatment or intervention: randomly or by some other method?
•  What control groups were included (placebo, untreated controls, both or neither)
•  How were the treatments compared?
•  Were there dropouts or loss to follow-up?
•  Were the outcomes or effects measured objectively?

For observational studies

• Was the data collection process adequate (including questionnaire design and pre-testing)?
• What techniques were used to handle non-response and/or incomplete data?
•  If a cohort study, was the follow-up rate sufficiently high?
•  If a case-control study, are the controls appropriate and adequately matched?

How was the data analysed?

• Is the statistical analysis appropriate, and is it presented in sufficient detail?

What are the findings?

• What are the main findings of the study? Pay specific attention to the presented text and tables in relation to the study’s main findings .
• Are the numbers consistent? Is the entire sample accounted for?

Experimental study

•  Do the authors find a difference between the treatment and control groups?
•  Are the results statistically significant? If there is a statistically significant difference, is it enough of a difference to be clinically significant?

Observational study

•  Did the authors find a difference between exposed and control groups or cases and controls?
•  Is there a statistically significant difference between groups?
•  Could the results be of public health significance, even though the difference is not statistically significant? (This may highlight the need for a larger study).
• Are the results likely to be affected by confounding? Why or why not?
• What (if any) variables are identified as potential confounders in the study?
• How is confounding dealt with in this study?
• Are there any potential confounders that the authors have not taken into account? What might the likely impact be on the results?

Summing it up

Activity 7.2a

Read the following article:

Chen X, Jiang X, Huang X, He H, Zheng J: Association between probiotic yogurt intake and gestational diabetes mellitus: a case-control study. Iran J Public Health. 2019, 48:1248-1256.

Let’s conduct a critical appraisal of this article.

A guide to reading scientific articles – Qualitative studies

What is the research question?

Was a qualitative approach appropriate?

• Identify the study design and if it was appropriate for the research question.

How were the setting and the subjects selected?

• What sampling strategy was used?
• Where was the study conducted?

Was the sampling strategy appropriate for the approach?

• Consider the qualitative approach used and decide if the sampling strategy or technique is appropriate

What was the researcher’s position, and has this been taken into account?

• Consider the researcher’s background, gender, knowledge, personal experience and relationship with participants

What were the data collection methods?

• How was data collected? What technique was used?

How were data analysed, and how were these checked?

• How did the authors analyse the data? Was this stated?
• Did two or more researchers conduct the analysis independently, and were the outcomes compared (double coding)?
• Did the researchers come to a consensus, and how were disagreements handled?

Are the results credible?

• Does the result answer the research question?
• Are themes presented with quotes and do they relate to the research question or aim?

Are the conclusions justified by the results?

• Have the findings been discussed in relation to existing theory and previous research?
• How well does the interpretation of the findings fit well with what is already known?

Are the findings transferable to other settings?

• Can the findings be applied to other settings? Consider the sample.

Activity 7.2b

Wallisch A, Little L, Pope E, Dunn W. Parent Perspectives of an Occupational Therapy Telehealth Intervention. Int J Telerehabil. 2019 Jun 12;11(1):15-22. doi: 10.5195/ijt.2019.6274. PMID: 31341543; PMCID: PMC6597151.

Let’s conduct a critical appraisal of this article

Now that you know how to critically appraise both quantitative and qualitative papers, it is also important to note that numerous critical appraisal tools exist. Some have different sub-tools for different study designs, while others are designed to be used for multiple study designs. These tools aid the critical appraisal process as they contain different questions to prompt the reader while assessing the study’s quality. 9 Examples of tools commonly used in health professions are listed below in Table 7.2. Please note that this list is not exhaustive, as numerous appraisal tools exist. You can use any of these tools to appraise the quality of an article before choosing to use their findings to inform your own research or to change practice.

Table 7.2 Critical appraisal tools

An Introduction to Research Methods for Undergraduate Health Profession Students Copyright © 2023 by Faith Alele and Bunmi Malau-Aduli is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Qualitative vs. Quantitative Research | Differences, Examples & Methods

Qualitative vs. Quantitative Research | Differences, Examples & Methods

Published on April 12, 2019 by Raimo Streefkerk . Revised on June 22, 2023.

When collecting and analyzing data, quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings. Both are important for gaining different kinds of knowledge.

Common quantitative methods include experiments, observations recorded as numbers, and surveys with closed-ended questions.

Quantitative research is at risk for research biases including information bias , omitted variable bias , sampling bias , or selection bias . Qualitative research Qualitative research is expressed in words . It is used to understand concepts, thoughts or experiences. This type of research enables you to gather in-depth insights on topics that are not well understood.

Common qualitative methods include interviews with open-ended questions, observations described in words, and literature reviews that explore concepts and theories.

Table of contents

The differences between quantitative and qualitative research, data collection methods, when to use qualitative vs. quantitative research, how to analyze qualitative and quantitative data, other interesting articles, frequently asked questions about qualitative and quantitative research.

Quantitative and qualitative research use different research methods to collect and analyze data, and they allow you to answer different kinds of research questions.

Quantitative and qualitative data can be collected using various methods. It is important to use a data collection method that will help answer your research question(s).

Many data collection methods can be either qualitative or quantitative. For example, in surveys, observational studies or case studies , your data can be represented as numbers (e.g., using rating scales or counting frequencies) or as words (e.g., with open-ended questions or descriptions of what you observe).

However, some methods are more commonly used in one type or the other.

Quantitative data collection methods

• Surveys :  List of closed or multiple choice questions that is distributed to a sample (online, in person, or over the phone).
• Experiments : Situation in which different types of variables are controlled and manipulated to establish cause-and-effect relationships.
• Observations : Observing subjects in a natural environment where variables can’t be controlled.

Qualitative data collection methods

• Interviews : Asking open-ended questions verbally to respondents.
• Focus groups : Discussion among a group of people about a topic to gather opinions that can be used for further research.
• Ethnography : Participating in a community or organization for an extended period of time to closely observe culture and behavior.
• Literature review : Survey of published works by other authors.

A rule of thumb for deciding whether to use qualitative or quantitative data is:

• Use quantitative research if you want to confirm or test something (a theory or hypothesis )
• Use qualitative research if you want to understand something (concepts, thoughts, experiences)

For most research topics you can choose a qualitative, quantitative or mixed methods approach . Which type you choose depends on, among other things, whether you’re taking an inductive vs. deductive research approach ; your research question(s) ; whether you’re doing experimental , correlational , or descriptive research ; and practical considerations such as time, money, availability of data, and access to respondents.

Quantitative research approach

You survey 300 students at your university and ask them questions such as: “on a scale from 1-5, how satisfied are your with your professors?”

You can perform statistical analysis on the data and draw conclusions such as: “on average students rated their professors 4.4”.

Qualitative research approach

You conduct in-depth interviews with 15 students and ask them open-ended questions such as: “How satisfied are you with your studies?”, “What is the most positive aspect of your study program?” and “What can be done to improve the study program?”

Based on the answers you get you can ask follow-up questions to clarify things. You transcribe all interviews using transcription software and try to find commonalities and patterns.

Mixed methods approach

You conduct interviews to find out how satisfied students are with their studies. Through open-ended questions you learn things you never thought about before and gain new insights. Later, you use a survey to test these insights on a larger scale.

It’s also possible to start with a survey to find out the overall trends, followed by interviews to better understand the reasons behind the trends.

Qualitative or quantitative data by itself can’t prove or demonstrate anything, but has to be analyzed to show its meaning in relation to the research questions. The method of analysis differs for each type of data.

Analyzing quantitative data

Quantitative data is based on numbers. Simple math or more advanced statistical analysis is used to discover commonalities or patterns in the data. The results are often reported in graphs and tables.

Applications such as Excel, SPSS, or R can be used to calculate things like:

• Average scores ( means )
• The number of times a particular answer was given
• The correlation or causation between two or more variables
• The reliability and validity of the results

Analyzing qualitative data

Qualitative data is more difficult to analyze than quantitative data. It consists of text, images or videos instead of numbers.

Some common approaches to analyzing qualitative data include:

• Qualitative content analysis : Tracking the occurrence, position and meaning of words or phrases
• Thematic analysis : Closely examining the data to identify the main themes and patterns
• Discourse analysis : Studying how communication works in social contexts

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

The research methods you use depend on the type of data you need to answer your research question .

• If you want to measure something or test a hypothesis , use quantitative methods . If you want to explore ideas, thoughts and meanings, use qualitative methods .
• If you want to analyze a large amount of readily-available data, use secondary data. If you want data specific to your purposes with control over how it is generated, collect primary data.
• If you want to establish cause-and-effect relationships between variables , use experimental methods. If you want to understand the characteristics of a research subject, use descriptive methods.

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

There are various approaches to qualitative data analysis , but they all share five steps in common:

• Prepare and organize your data.
• Review and explore your data.
• Develop a data coding system.
• Assign codes to the data.
• Identify recurring themes.

The specifics of each step depend on the focus of the analysis. Some common approaches include textual analysis , thematic analysis , and discourse analysis .

A research project is an academic, scientific, or professional undertaking to answer a research question . Research projects can take many forms, such as qualitative or quantitative , descriptive , longitudinal , experimental , or correlational . What kind of research approach you choose will depend on your topic.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Streefkerk, R. (2023, June 22). Qualitative vs. Quantitative Research | Differences, Examples & Methods. Scribbr. Retrieved April 17, 2024, from https://www.scribbr.com/methodology/qualitative-quantitative-research/

Is this article helpful?

Raimo Streefkerk

Other students also liked, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, mixed methods research | definition, guide & examples, what is your plagiarism score.

How to write a Literature Review: Quantitative vs qualitative method

• Literature review process
• Purpose of a literature review
• Evaluating sources
• Managing sources
• Request a literature search
• Selecting the approach to use

Quantitative vs qualitative method

• Summary of different research methodologies
• Research design vs research methodology
• Diagram: importance of research
• Attributes of a good research scholar

Source : Golafshani, 2003

• << Previous: Selecting the approach to use
• Next: Summary of different research methodologies >>
• Last Updated: Dec 7, 2023 1:32 PM
• URL: https://libguides.unisa.ac.za/literature_review

• Research Process

Choosing the Right Research Methodology: A Guide for Researchers

• 3 minute read
• 35.8K views

Table of Contents

Choosing an optimal research methodology is crucial for the success of any research project. The methodology you select will determine the type of data you collect, how you collect it, and how you analyse it. Understanding the different types of research methods available along with their strengths and weaknesses, is thus imperative to make an informed decision.

Understanding different research methods:

There are several research methods available depending on the type of study you are conducting, i.e., whether it is laboratory-based, clinical, epidemiological, or survey based . Some common methodologies include qualitative research, quantitative research, experimental research, survey-based research, and action research. Each method can be opted for and modified, depending on the type of research hypotheses and objectives.

Qualitative vs quantitative research:

When deciding on a research methodology, one of the key factors to consider is whether your research will be qualitative or quantitative. Qualitative research is used to understand people’s experiences, concepts, thoughts, or behaviours . Quantitative research, on the contrary, deals with numbers, graphs, and charts, and is used to test or confirm hypotheses, assumptions, and theories. 

Qualitative research methodology:

Qualitative research is often used to examine issues that are not well understood, and to gather additional insights on these topics. Qualitative research methods include open-ended survey questions, observations of behaviours described through words, and reviews of literature that has explored similar theories and ideas. These methods are used to understand how language is used in real-world situations, identify common themes or overarching ideas, and describe and interpret various texts. Data analysis for qualitative research typically includes discourse analysis, thematic analysis, and textual analysis. 

Quantitative research methodology:

The goal of quantitative research is to test hypotheses, confirm assumptions and theories, and determine cause-and-effect relationships. Quantitative research methods include experiments, close-ended survey questions, and countable and numbered observations. Data analysis for quantitative research relies heavily on statistical methods.

Analysing qualitative vs quantitative data:

The methods used for data analysis also differ for qualitative and quantitative research. As mentioned earlier, quantitative data is generally analysed using statistical methods and does not leave much room for speculation. It is more structured and follows a predetermined plan. In quantitative research, the researcher starts with a hypothesis and uses statistical methods to test it. Contrarily, methods used for qualitative data analysis can identify patterns and themes within the data, rather than provide statistical measures of the data. It is an iterative process, where the researcher goes back and forth trying to gauge the larger implications of the data through different perspectives and revising the analysis if required.

When to use qualitative vs quantitative research:

The choice between qualitative and quantitative research will depend on the gap that the research project aims to address, and specific objectives of the study. If the goal is to establish facts about a subject or topic, quantitative research is an appropriate choice. However, if the goal is to understand people’s experiences or perspectives, qualitative research may be more suitable. 

Conclusion:

In conclusion, an understanding of the different research methods available, their applicability, advantages, and disadvantages is essential for making an informed decision on the best methodology for your project. If you need any additional guidance on which research methodology to opt for, you can head over to Elsevier Author Services (EAS). EAS experts will guide you throughout the process and help you choose the perfect methodology for your research goals.

Why is data validation important in research?

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

Writing a good review article

Scholarly Sources: What are They and Where can You Find Them?

Input your search keywords and press Enter.

Market research

Research question: What is it and how do you do it correctly?

TRY OUT NOW

The Research question is the cornerstone of any study, whether it is market research or academic research in any field.

Although it is part of any research process, the aspects that allow it to be carried out correctly are not always known. That's why in this article we've put together everything you need to know when formulating the research question for your next project.

• 1 What is a research question?
• 2.1 Quantitative research question
• 2.2 Qualitative research questions
• 2.3 Mixed methods questions
• 3 What is the significance of the research question?
• 4.1 1. Start with a broad topic
• 4.2 2. carry out an initial review of the relevant literature
• 4.3 3. Narrow down your topic and identify possible research questions.
• 4.4 4. Evaluate the strength of your research question
• 5.1 PICOT model
• 5.2 PEO frame
• 6 Conclusion
• 7 1:1 Live Online Presentation: QUESTIONPRO MARKET RESEARCH SOFTWARE
• 8 Try software for market research and experience management now for 10 days free of charge!

What is a research question?

The research question is the central question that a study aims to answer. It is at the heart of systematic research and helps to clearly define the path for the research process.

The research question is usually the first step in the research methodology. It is the main point of the survey and determines the pace of the work.

This question usually refers to a problem or question that is answered by data analysis and interpretation in the conclusion of the study.

In most studies, the question is phrased in a way that highlights the various aspects of a study, including the problem the study is addressing, the population, and the variables being examined.

The research questions are often only determined during the course of the study. Therefore, these questions are dynamic, meaning researchers can change or refine the research question as they review the relevant literature and develop a framework for the study.

While many research projects focus on a single research question, larger studies may use more than one question.

Types of research questions

The type of research question depends on the focus and direction of the study being conducted and can therefore be different:

Quantitative research question

Quantitative research questions aim to understand processes that take place in specific contexts and locations. They generally fall into three types:

• Descriptive : They aim to obtain information about a variable or several variables in order to assign a size to the variable.
• Comparative : A comparison is made between two or more groups based on one or more reliable variables.
• Relational : These questions aim to understand the association, trends and causal relationship between two or more variables.

These questions are precise and typically include the population to be studied, the dependent and independent variables, and the research design to be used. They are usually formulated and defined at the beginning of the study. Check out these examples of quantitative questions for a survey

Qualitative research questions

These questions may cover broad areas of research or more specific areas of study. Like quantitative questions, these questions are also linked to the research design.

Unlike quantitative questions, qualitative research questions tend to be adaptive, non-directive, and more flexible, so studies with these types of questions generally aim to “discover,” “explain,” or “explore.”

Mixed methods questions

Mixed methods research requires a range of quantitative and qualitative questions. Separate research questions are appropriate when mixed studies focus on the importance and differences between quantitative and qualitative methods rather than the integrative factor of the study.

What is the significance of the research question?

The primary reason for asking a research question is that it narrows a broad topic of interest to a specific area of ​​inquiry.

The research questions, along with the hypotheses, serve as a framework that guides the research. These questions also reveal the limitations of a study by establishing its boundaries and ensuring that it is coherent.

Most importantly, the research question has implications for the rest of the study and influences factors such as research methods, sample size, data collection and analysis.

How do you formulate a research question?

Now that you know what a research question is and how important it is, here are the key steps to consider when formulating a research question in a project:

1. Start with a broad topic

A broad topic provides researchers with a number of options to explore in finding a viable research question.

Brainstorming and concept mapping are some techniques that help develop a topic into subtopics and potential research questions.

These techniques help you organize your thoughts so you can identify relevant connections and subtopics within a broad topic.

When researching a topic, it makes sense to choose a subject area that really interests you, because your interest in the topic will affect your motivation during the research.

It is also worth considering the interests that have recently been expressed by the research community, as this may influence the chances of your research being published.

   |   See also: How to formulate a Research problem ?

2. carry out an initial review of the relevant literature

Once you have decided on a topic, you can begin to create a preliminary review of the relevant literature. With this first phase of research you can achieve two goals:

• You can find out what topics are currently being discussed by scientists and other researchers. In this way, you gain current and relevant knowledge on your topic.
• You identify gaps or limitations in existing knowledge about your topic. With some degree of fine-tuning, you can later use these gaps as the focus of your research question.

3. Narrow down your topic and identify possible research questions.

Once you have accumulated enough knowledge about the topic you want to work on, you can begin to focus on a more specific area of ​​inquiry.

One option is to focus on gaps in existing knowledge or recent literature. This method involves developing research questions based on limitations identified in the literature that have been neglected in the past.

Similarly, researchers may choose research questions that extend or complement existing literature findings.

4. Evaluate the strength of your research question

The initial research and review of the relevant literature you have undertaken will have revealed some interesting questions that seem worth addressing.

However, are Not all interesting questions are also good research questions . Remember that research questions draw their answers or conclusions through an analysis of the evidence.

When formulating the research question, it is important to first consider relevance. A good strategy for this is to use the acronym FINER:

A good research question is feasible, meaning the question is within the researcher's capabilities. Researchers should be realistic about the scope of their research, as well as their ability to collect data and conduct the research within their skills and available resources.

Interesting

The ideal research question is interesting not only to the researcher, but also to his colleagues and the community. Novel The research question should be formulated in such a way that it provides new knowledge in the area under study.

The question and the subsequent study should be approved by the examination committees and the relevant authorities.

The research question should be relevant to the scientific community and the people working in your field of study. If possible, the question should also be relevant to the interest of the general public.

Recommendations for the appropriate formulation of a research question

Research questions should be appropriately structured to ensure clarity. There are several frameworks that can be used to correctly formulate a research question, the two most common are:

PICOT model

The PICOT framework allows research questions to be constructed to address important elements of the study, including the population to be studied, the expected results, and the time needed to achieve the result.

P : population, patient or problem. I: Intervention or indicator to be studied. C: Comparison group. O: Result of interest. T: Time frame of the study.

With these elements, the framework is commonly used in clinical research and evidence-based studies.

Example of a question constructed using the PICOT framework

Are children aged 5 to 20 of parents with addiction problems at a higher risk of developing depression or anxiety than children of parents without diagnosed mental health problems?

This question reflects the PICOT framework as follows:

P (population examined): Children I (indicator or intervention) : Parents with addiction problems C (comparison group) : Children of parents without addiction problems. O (result of interest) : Increased risk of depression T (interesting time frame) : Between the ages of 5 and 18

Like the PICOT framework, the PEO framework is also commonly used in clinical trials. However, this study is more suitable for conducting qualitative research questions.

This framework includes the following elements:

P: Population studied. E: Exposure to pre-existing conditions. O : result of interest.

Example of a question constructed using the PEO framework

How does exposure to classical music affect the development of emotional expression in infants and children under 8 years of age?

P (population examined): Infants and children under 8 years old E (exposure to given conditions) : exposure to classical music O (result of interest): Effects on the development of emotional expression

As you can see, the research question is the step you should pay the most attention to when conducting a study, as it will guide you for the rest of your work.

Once you have defined your research question, you can use QuestionPro's online surveys, which specialize in solving quantitative, qualitative and mixed methods research questions, to initiate the next phases of your project.

If you don't know them yet, we invite you to create a free account and explore all the options available to you by clicking the link below. We hope this article was of interest to you!

1:1 live online presentation: QUESTIONPRO MARKET RESEARCH SOFTWARE

Arrange an individual appointment and discover our market research software.

Try software for market research and experience management now for 10 days free of charge!

Do you have any questions about the content of this blog? Simply contact us via contact form . We look forward to a dialogue with you! You too can test QuestionPro for 10 days free of charge and without risk in depth!

Test the agile market research and experience management platform for qualitative and quantitative data collection and data analysis from QuestionPro for 10 days free of charge

FURTHER KEYWORDS

Types of research | Empirical research | Research topic

SHARE THIS ARTICLE

KEYWORDS OF THIS BLOG POST

Research question | Research  | Question

FURTHER INFORMATION

• Applied Research: Definition, Types and Examples
• Research methods: what are they and how to choose them?
• Types of research and their features
• What is exploratory research?
• Mixed methods research: what it is and what types there are
• Research topic: what is it and how to choose it correctly?
• Data collection tools: which are the best?
• Big Data and Artificial Intelligence: How do they work together?

PRESS RELEASES