Quantitative Research

  • Reference work entry
  • First Online: 13 January 2019
  • Cite this reference work entry

what is quantitative research peer reviewed articles

  • Leigh A. Wilson 2 , 3  

4802 Accesses

Quantitative research methods are concerned with the planning, design, and implementation of strategies to collect and analyze data. Descartes, the seventeenth-century philosopher, suggested that how the results are achieved is often more important than the results themselves, as the journey taken along the research path is a journey of discovery. High-quality quantitative research is characterized by the attention given to the methods and the reliability of the tools used to collect the data. The ability to critique research in a systematic way is an essential component of a health professional’s role in order to deliver high quality, evidence-based healthcare. This chapter is intended to provide a simple overview of the way new researchers and health practitioners can understand and employ quantitative methods. The chapter offers practical, realistic guidance in a learner-friendly way and uses a logical sequence to understand the process of hypothesis development, study design, data collection and handling, and finally data analysis and interpretation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

what is quantitative research peer reviewed articles

Writing Quantitative Research Studies

what is quantitative research peer reviewed articles

Qualitative Research Methods

Babbie ER. The practice of social research. 14th ed. Belmont: Wadsworth Cengage; 2016.

Google Scholar  

Descartes. Cited in Halverston, W. (1976). In: A concise introduction to philosophy, 3rd ed. New York: Random House; 1637.

Doll R, Hill AB. The mortality of doctors in relation to their smoking habits. BMJ. 1954;328(7455):1529–33. https://doi.org/10.1136/bmj.328.7455.1529 .

Article   Google Scholar  

Liamputtong P. Research methods in health: foundations for evidence-based practice. 3rd ed. Melbourne: Oxford University Press; 2017.

McNabb DE. Research methods in public administration and nonprofit management: quantitative and qualitative approaches. 2nd ed. New York: Armonk; 2007.

Merriam-Webster. Dictionary. http://www.merriam-webster.com . Accessed 20th December 2017.

Olesen Larsen P, von Ins M. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics. 2010;84(3):575–603.

Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg. 2010;126(2):619–25. https://doi.org/10.1097/PRS.0b013e3181de24bc .

Petrie A, Sabin C. Medical statistics at a glance. 2nd ed. London: Blackwell Publishing; 2005.

Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 3rd ed. New Jersey: Pearson Publishing; 2009.

Sheehan J. Aspects of research methodology. Nurse Educ Today. 1986;6:193–203.

Wilson LA, Black DA. Health, science research and research methods. Sydney: McGraw Hill; 2013.

Download references

Author information

Authors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Leigh A. Wilson

Faculty of Health Science, Discipline of Behavioural and Social Sciences in Health, University of Sydney, Lidcombe, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Leigh A. Wilson .

Editor information

Editors and affiliations.

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

Wilson, L.A. (2019). Quantitative Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_54

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_54

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research
  • Other Journals

Chapter 3. Introduction to Quantitative Research and Data

T he foundation of any e-book analysis framework rests on knowledge of the general e-book landscape and the existing information needs of a local user community. From this starting point, quantitative methods, such as cost analysis, can provide evidence for collection development initiatives and demonstrate how they align with patrons’ needs and the overarching goals of library administrators or funding agencies.

Essentially, “data stands in place of reality we wish to study. We cannot simply know a phenomenon, but we can attempt to capture it as data which represents the reality we have experienced . . . and are trying to explain.” 1 The data collected through quantitative investigations provides a baseline for future evaluation, evidence for when and how patrons make use of electronic collections, and promotes data-driven decisions throughout collection development departments. To get the most mileage out of the time and resources invested into quantitative investigations, it is essential to first understand what quantitative research is and what types of questions it can answer.

What Is Quantitative Research?

In the most basic terms, quantitative research methods are concerned with collecting and analyzing data that is structured and can be represented numerically. 2 One of the central goals is to build accurate and reliable measurements that allow for statistical analysis.

Because quantitative research focuses on data that can be measured, it is very effective at answering the “what” or “how” of a given situation. Questions are direct, quantifiable, and often contain phrases such as what percentage? what proportion? to what extent? how many? how much?

Quantitative research allows librarians to learn more about the demographics of a population, measure how many patrons use a service or product, examine attitudes and behaviors, document trends, or explain what is known anecdotally. Measurements like frequencies (i.e., counts), percentages, proportions, and relationships provide means to quantify and provide evidence for the variables listed above.

Findings generated from quantitative research uncover behaviors and trends. However, it is important to note that they do not provide insight into why people think, feel, or act in certain ways. In other words, quantitative research highlights trends across data sets or study groups, but not the motivation behind observed behaviors. To fill in these knowledge gaps, qualitative studies like focus groups, interviews, or open-ended survey questions are effective.

Whenever I sit down to a new quantitative research project and begin to think about my goals and objectives, I like to keep a small cheat sheet on my desk to remind me of the trends quantitative data can uncover and the stories that I can tell with study conclusions. This serves as one quick strategy that keeps my thoughts focused and prevents scope creep as I discuss project plans with various stakeholders.

Quantitative Research Cheat Sheet

Six key characteristics of quantitative research:

  • It deals with numbers to assess information.
  • Data can be measured and quantified.
  • It aims to be objective.
  • Findings can be evaluated using statistical analysis.
  • It represents complex problems through variables.
  • Results can be summarized, compared, or generalized.

Quantitative findings can provide evidence or answers in the following areas:

  • Demonstrate to what extent services and collection are used and accessed.
  • Back up claims about use and impact.
  • Provide evidence for how the budget is spent and whether adjustments should be made.
  • Demonstrate return on investment when presenting budget figures.
  • Inform decisions regarding packages and subscriptions that are or are not worth pursuing.
  • Demonstrate evidence for trends and prove or discount what is known anecdotally.
  • Provide a method to make information accessible to audiences.
  • Provide evidence of success and highlight areas where unmet information needs exist.

Main advantages of quantitative research:

  • Findings can be generalized to a specific population.
  • Data sets are large, and findings are representative of a population.
  • Documentation regarding the research framework and methods can be shared and replicated.
  • Standardized approaches permit the study to be replicated over time.

Main limitations of quantitative research:

  • Data does not provide evidence for why populations think, feel, or act in certain ways.
  • Specific demographic groups, particularly vulnerable or disadvantaged groups, may be difficult to reach.
  • Studies can be time consuming and require data collection over long periods of time. 3

Quantitative Research in Information Management Environments

In the current information landscape, a wealth of quantitative data sources is available to librarians. One of the challenges surrounding quantitative research in the information management profession is “how to make sense of all these data sources and use them in a way that supports effective decision-making.” 4

Most libraries pay for and receive materials through multiple routes. As a result, a quantitative research framework for e-book collections often consist of two central components: an examination of resource allocations and expenditures from funds, endowments, or gifts; and an examination of titles received through firm orders, subscriptions, packages, and large aggregated databases. 5 In many cases, examining funds and titles according to subject areas adds an extra layer of knowledge that can provide evidence for teaching, learning, or research activities in a specific field or justify requests for budget increases. 6

Many of the quantitative research projects that I have conducted over the past four years are in direct response to an inquiry from library administrators. In most cases, I have been asked to provide evidence for collection development activities that support expressed information needs, justify expenditures, or project annual increases in preparation for a new fiscal year. Study results are often expected to describe or weigh several courses of action in the short and long term. Essentially, my work is categorized into three basic concepts related to library management:

  • Distinguish between recurrent and capital expenditure and projects, and between past, present, and future states.
  • Accommodate priorities and determine how resources are spread across collections.
  • Indicate the ways of allocating resources at input, monitor performance, and assess performance at output. 7

To assist in my prep work for a quantitative research project, I put together a file of background information about my library system and local user community to ensure that the project supports institutional goals and aligns with the general direction of programs and services on campus. Below are seven categories of information that I have on file at all times:

  • the institutional identity of the library
  • the stakeholder groups to be served
  • collection resources
  • financial resources
  • library personnel
  • facilities and equipment
  • the various programs and services related to the quantitative investigation 8

Typically, I take a day or two at the beginning of each fiscal year to update this information and ensure that it accurately reflects the landscape of collections and services available at CUL. From this starting point, it is simple to look at new project descriptions and think about the data required to support high-level decisions regarding the allocation of resources, to assess the effectiveness of collections and services, or to measure the value and impact of collections.

A wealth of local and external data sources is available to librarians, and each one can be used to tell a story about collection size, value, and impact. All that is required is an understanding of what the data measures and how different sources can be combined to tell a story about a user community.

Definitions of Local and External Data Sources

The remaining sections of this issue of Library Technology Reports discuss how I use quantitative data, what evidence I have uncovered to support e-book collection decisions, and how I apply quantitative findings in practical library settings. For the purposes of these discussions, I will use the following terminology:

Bibliographic record: A library catalog record that represents a specific title or resource.

Catalog clickthroughs: Counts of patron use of the catalog to access electronic full texts.

Citation analysis: Measurement of the impact of an article based on the number of times it has been cited.

Consortia reports: Consolidated usage reports for consortia. Often used to view usage linked to each individual consortia member.

COUNTER (Counting Online Usage of Networked Electronic Resources): An international initiative to improve the reliability of online usage statistics by providing a Code of Practice that standardizes the collection of usage data. It works to ensure vendor usage data is credible and comparable.

Cost data: Factual information concerning the cost of library materials, annual budget allocations, and general acquisitions budget.

FTE (full-time equivalent): The number of full-time faculty and students working or studying at a specific institution.

IP (Internet Protocol) address: A numerical label usually assigned to a library router or firewall that provides access to a private network (e.g., school or library network).

Link resolver statistics: Information regarding the pathways users take to access electronic resources.

Overlap data: Measurement of the degree of duplication across a collection.

Publication analysis: Measurement of impact by counting the research output of an author. Metrics include the number of peer-reviewed articles, coauthor collaborations, publication patterns, and extent of interdisciplinary research.

Title lists: Lists of e-book titles available in subscriptions, databases, or packages. These lists are generated and maintained by vendors and publishers.

Turnaway statistics: The number of patrons denied access to a specific title.

Vendor use data: Electronic use statistics provided by vendors.

Indicators and Performance Measures That Support Quantitative Research

I regularly use several indicators and performance measures to analyze e-book collections. Local and external data sources (listed in the section above) inform these investigations and provide the necessary “ingredients” to conduct cost analysis, examine return on investment, or measure the value of e-book collections to the community at CUL. Below is a breakdown of how I classify data and relate it to different indicators. 9

Input Cost Measures

Data source: Cost data pulled from Voyager reports (or your institution’s ILS system).

In general, cost data demonstrates how funds are allocated across a budget. Analysis can identify areas where additional resources are required, monitor cost changes over time, and flag collection areas where funds can be pulled (e.g., overbudgeted funds, subject areas that no longer support the curriculum, etc.) and “reinvested” in the collection to support current information needs.

Each of the investigations described in the following chapter began with a review of cost data. I relied on a basic knowledge of how e-book acquisition budgets are distributed across subject areas or pooled to purchase interdisciplinary materials. Essentially, these investigations involved the identification of fund codes linked to subject areas, expenditures across set date ranges (e.g., calendar years, fiscal years, academic years), and bulk versus long-tail purchases.

Tip: When working with cost data and examining input cost measures, I have found it helpful to categorize data by fund type. E-book collections at CUL are often built with general income (GI) funds, endowments, and gifts. Policies and procedures regarding how funds can be transferred and what materials can be purchased impact how resources are allocated to build e-book collections. Before beginning a cost analysis project at your institution, it may be helpful to review the policies in place and determine how they relate to overarching institutional goals and collection priorities.

Collection Output Measures

Data sources: Cost data, title lists, overlap data, bibliographic records (particularly subject headings).

Collection output measures are related to the quantity and quality of output. Examples include the number of e-book titles included in a subscription or package deal acquired by a library, the number of e-book records acquired over a given period of time, the number of publishers and unique subject areas represented in an e-book collection, the currency of information (e.g., publication year), and the percentage of title overlap, or duplication, within a collection.

At this stage in my cost analysis projects, it is often necessary to combine data to create a snapshot of how funds flow in and out of subject areas to acquire research and teaching materials. For example, many of our large e-book packages are interdisciplinary. By pulling cost data, I can determine how the total cost was split across subject divisions based on fund code counts. Then, I break title lists apart by subject to determine what percentage of total content relates to each library division. By comparing the cost breakdown and title list breakdown, it is possible to determine what percentage of total content each library division receives and if it is on par with the division’s financial contribution.

Effectiveness Measures and Indicators

Data sources: Cost data, title lists, COUNTER reports, vendor reports, consortia reports, resolver statistics, turnaway statistics, Google Analytics.

Examining input and output measures is an effective way of determining how budgets are allocated and the quantity and quality of materials available to patrons. To develop a quantitative baseline for the general value of e-book collections, measures like rate of use, cost per use, and turnaway rates can be very effective.

Again, this form of analysis relies on data from multiple sources. The ability to combine cost data, title lists, and COUNTER data (or vendor data) has yielded actionable results at my library. For instance, I combine data from these three sources to measure the value of databases. By pulling cost data covering three fiscal years and matching title lists against COUNTER reports, I have been able to examine trends in annual increase rates, examine overlap between subscriptions in the same subject area, and calculate cost per use to determine what percentage of the user community makes use of subscriptions.

Finally, by looking at turnaway statistics (also found in COUNTER data), it is possible to determine if sufficient access is provided to users. For instance, I look at turnaway statistics to evaluate if e-books listed on course reading lists provide sufficient access to a class of students over a semester. In cases where access is limited to a single user, I may look at the budget to find areas where funds can be shifted to purchase simultaneous usage instead.

Together, the data sets mentioned above provide evidence for how funds are invested, if they are invested in materials that are heavily used by patrons, and if access models are suited to the needs of the local user community.

In some cases, particularly when dealing with foreign language materials, I have encountered challenges because COUNTER data is not provided, and in some cases, it is difficult to obtain vendor reports as well. In the absence of usage data, I have experimented with link resolver statistics to determine what information they provide about user activities and the value of e-book materials.

Link resolver statistics provide information about the pathways users take to access electronic resources. 10 Resolver statistics show that a patron made a “request” via the link resolver and started the process of trying to view a full text. If the patron successfully accesses the full text, this is counted as a “clickthrough.”

It is important to note that link resolver statistics and usage statistics (like COUNTER) are not comparable because they measure different activities. Link resolvers measure attempts to connect while usage data measures usage activity. However, comparing sets of link resolver statistics against each other may provide insight into which resources patrons attempt to access most frequently. This can provide a ballpark idea of resource value in cases where usage statistics are not available.

Domain Measures

Data sources: FTE (full-time equivalent), IP address, demographic information.

Domain measures relate to the user community served by a library. They include total population, demographic information, attributes (e.g., undergraduate level, graduate level), and information needs.

In my work, domain measures impact subscription or package costs because campus-wide access is often priced according to FTE. Due to the size of CUL’s student body, access to essential collections can become extremely expensive and fall outside of the budget range. When this occurs, examining patron access by IP address has opened the door to negotiation, particularly when dealing with content that is discipline-specific. For instance, when negotiating subscription prices for science materials, IP data provided evidence that usage is concentrated at the library router located in the Science and Engineering Library. This allowed science selectors to negotiate pricing models based around the FTE of natural science programs as opposed to the campus community as a whole.

Cost-Effectiveness Indicators

Data sources: COUNTER reports, vendor reports, turnaway statistics, citation analysis, publication analysis.

Cost-effectiveness indicators are related to measures like cost per use and ultimately examine the general return on investment. They evaluate the financial resources invested in a product and determine if the investment brings added value to the existing collection.

In my work, I often combine cost data with usage data to calculate cost per use and also capture usage trends spanning at least three calendar years. The results provide a benchmark regarding whether the financial investment in the product is equivalent to its general “demand” within the user community. A recent project with colleagues at the science and medical science libraries has examined how to use citation and publication data to determine general impact of electronic resources.

Challenges Presented by Quantitative Research

One of the challenges surrounding quantitative research in library environments is a lack of standardization across data sets, particularly vendor reports. The general situation has improved in recent years due to widespread compliance with the COUNTER Code of Practice, but there is still work to be done. It is difficult to interpret the meaning of vendor usage data that is still not COUNTER-compliant because clear definitions of use do not exist. This can create significant roadblocks when running quantitative projects that examine multiple e-book collections to get a sense of comparative value.

Also, usage data is generated outside of libraries by publishers or aggregators and vendors. Factors like turnover, company mergers, or password changes result in significant time lags between when usage statistics are generated and when libraries receive them. Also, some vendors pull down usage statistics after a period of months. In most cases, librarians need statistics captured over two or three years to meet reporting requirements, and data dating back this far can be difficult to obtain. Finally, annual usage statistics are provided according to calendar year. However, librarians look at usage by fiscal year and academic year as well. In many cases, this means that multiple usage reports have to be stitched together in order to capture the appropriate timeframe for reporting purposes. This process is labor intensive and takes a considerable amount of time to complete.

These challenges emphasize an ongoing need to build positive working relationships with publishers, aggregators, and vendors to discuss challenges and develop solutions that benefit all stakeholders. It is important to note that libraries have valuable information that is not available to content providers, namely how e-books are discovered and used. Strong relationships allow for the transparent exchange of information between all parties, which ultimately benefits patrons by providing a seamless e-book experience.

Designing a Quantitative Research Framework

As mentioned earlier in this chapter, data stands in place of a reality we wish to study, quantify, and explain. In order to prevent scope creep and pull together bodies of data that add value to local work environments, it is essential to begin any quantitative research project with a set of clearly defined objectives, a strong understanding of the stakeholder group or audience, and knowledge of local information needs. These bits of information serve as markers to measure progress and ensure the project stays on track.

It is tempting to dive straight into a project and investigate if anecdotal information or assumptions are correct, but time spent developing a project outline is never wasted. The development of a successful plan requires “a clear idea of what it is to be achieved among the stakeholders. Clearly articulated objectives are the engine that drives the assessment process. This is one of the most difficult but most rewarding stages of the assessment process.” 11 Creating a roadmap for research projects can save countless hours down the line and ensures the correct quantitative method is selected. The plan also provides focus when the analysis phase of a project begins. Keep in mind that the data set you end up working with will be large; approaching it with stated goals and objectives saves significant amounts of time, which is especially important when working under a tight deadline!

Below is a checklist that I use at the beginning of any research project. It is based on recommendations made by Bakkalbasi, Sundre, and Fulcher. 12

While goals and objectives are closely related, they are not the same. Project goals should state exactly what you hope to learn or demonstrate through your research. Objectives state what you will assess or measure in order to achieve your overarching project goal.

Example of a project goal:

Example of project objectives:

  • To learn what activities local patrons engage in when using library facilities.
  • Consider how results may support improvement of collection development initiatives or lead to evaluation of existing workflows, policies, and procedures.
  • What questions and/or evidence are required by stakeholders?
  • What information do stakeholders require to make decisions?
  • How will results support the improvement of collection development initiatives?
  • How will results be made accessible to stakeholders?
  • Are the results intended for internal use, or will they be shared with the professional community?
  • Will findings be used to support grant or funding applications?
  • Is there a stated project deadline? What methods or resources will allow you to collect data, conduct analysis, and provide findings within the stated timeframe?
  • Does the project coincide with other activities that may require your attention (e.g., fiscal year, subscription renewal period)?
  • Are there colleagues at the library who may be able to provide assistance given the timeline of the project?
  • What data collected through the study cannot be shared with external stakeholders (e.g., cost data, FOIP compliance, etc.)?
  • Are there any permissions required before study results can be disseminated to external stakeholders?
  • Is clearance required to collect data from a user community?
  • What data sources are most valued and meaningful to your library?
  • What data sources will allow results to be applied at your library?
  • What data collection methods will be most effective?
  • What data collection methods will provide valid and reliable results?
  • Are there parameters such as specific fiscal years, calendar years, or academic years that you are required to report on?
  • How will data be summarized and described?
  • What features of the data set are most relevant to project objectives and goals?
  • What are the relationships between different data sets?
  • How is data evaluated?
  • How is data interpreted into meaningful results and conclusions?
  • What are the recommendations for action or improvements?
  • How will findings be communicated to stakeholders?

The data sets collected through quantitative methods are large and can easily be examined from a variety of perspectives. As the project develops, mentally frame emerging trends into a story that can be shared with stakeholders. This process determines how results will ultimately be applied to collection development initiatives. Background knowledge of the local patron community and institutional goals serves as a compass; use it to shape results that bring value to your library or the greater professional community.

From my experience, each quantitative project that I work on allows me to expand my skill sets and understand how I can structure my daily activities to support overarching institutional goals. During many projects, I have encountered unexpected challenges or had to improvise when quantitative methods did not yield expected results (e.g., low survey response rates). However, each challenge equipped me to take on larger projects, better understand how our budget is structured, or build stronger relationships with patrons and colleagues.

One skill that has been invaluable to my work is the ability to develop a quantitative research plan. I hope that by sharing this structure, along with performance measures and data sources that I use, readers have a behind-the-scenes view of my process and all of the moving parts that I work with to conduct e-book collection analysis. And of course, now to the fun part! It is time to get down to the nitty-gritty and demonstrate how I conduct analysis to inform budget decisions and collection development activities at CUL.

  • Bob Matthews and Liz Ross, Research Methods: A Practical Guide for the Social Sciences (Harlow, UK: Pearson Education, 2010), 45.
  • Ibid., 465.
  • Based on information provided by Stephen A. Roberts, Financial and Cost Management for Libraries and Information Management Services (London: Bowker-Saur, 1998), 140–41.
  • Darby Orcutt, Library Data: Empowering Practice and Persuasion (Santa Barbara, CA: Libraries Unlimited, 2009), 106.
  • Northwestern University Libraries, “DataBank: How to Interpret Your Data: Financial Support,” LibGuide, last updated December 8 2015, http://libguides.northwestern.edu/c.php?g=115065&p=748741 .
  • Roberts, Financial and Cost Management , 132.
  • For further information regarding indicators and performance measures, please see Roberts, Financial and Cost Management , 140–41.
  • Orcutt, Library Data , 107.
  • Nisa Bakkalbasi, Donna Sundre, and Kenton Fulcher, “Assessing Assessment: A Framework to Evaluate Assessment Practices and Progress for Library Collections and Services,” in Proceedings of the 2012 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment, October 29–31, 2012, Charlottesville, VA , ed. Steve Hiller, Martha Kyrillidou, Angela Pappalardo, Jim Self, and Amy Yeager (Washington, DC: Association of Research Libraries, 2013), 538-545.
  • Based on information provided by Matthews and Ross, Research Methods , 345.
  • There are currently no refbacks.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Recent quantitative research on determinants of health in high income countries: A scoping review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Centre for Health Economics Research and Modelling Infectious Diseases, Vaccine and Infectious Disease Institute, University of Antwerp, Antwerp, Belgium

ORCID logo

Roles Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

  • Vladimira Varbanova, 
  • Philippe Beutels

PLOS

  • Published: September 17, 2020
  • https://doi.org/10.1371/journal.pone.0239031
  • Peer Review
  • Reader Comments

Fig 1

Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature.

We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that performed cross-national statistical analyses aiming to evaluate the impact of one or more aggregate level determinants on one or more general population health outcomes in high-income countries. To assess in which combinations and to what extent individual (or thematically linked) determinants had been studied together, we performed multidimensional scaling and cluster analysis.

Sixty studies were selected, out of an original yield of 3686. Life-expectancy and overall mortality were the most widely used population health indicators, while determinants came from the areas of healthcare, culture, politics, socio-economics, environment, labor, fertility, demographics, life-style, and psychology. The family of regression models was the predominant statistical approach. Results from our multidimensional scaling showed that a relatively tight core of determinants have received much attention, as main covariates of interest or controls, whereas the majority of other determinants were studied in very limited contexts. We consider findings from these studies regarding the importance of any given health determinant inconclusive at present. Across a multitude of model specifications, different country samples, and varying time periods, effects fluctuated between statistically significant and not significant, and between beneficial and detrimental to health.

Conclusions

We conclude that efforts to understand the underlying mechanisms of population health are far from settled, and the present state of research on the topic leaves much to be desired. It is essential that future research considers multiple factors simultaneously and takes advantage of more sophisticated methodology with regards to quantifying health as well as analyzing determinants’ influence.

Citation: Varbanova V, Beutels P (2020) Recent quantitative research on determinants of health in high income countries: A scoping review. PLoS ONE 15(9): e0239031. https://doi.org/10.1371/journal.pone.0239031

Editor: Amir Radfar, University of Central Florida, UNITED STATES

Received: November 14, 2019; Accepted: August 28, 2020; Published: September 17, 2020

Copyright: © 2020 Varbanova, Beutels. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This study (and VV) is funded by the Research Foundation Flanders ( https://www.fwo.be/en/ ), FWO project number G0D5917N, award obtained by PB. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Identifying the key drivers of population health is a core subject in public health and health economics research. Between-country comparative research on the topic is challenging. In order to be relevant for policy, it requires disentangling different interrelated drivers of “good health”, each having different degrees of importance in different contexts.

“Good health”–physical and psychological, subjective and objective–can be defined and measured using a variety of approaches, depending on which aspect of health is the focus. A major distinction can be made between health measurements at the individual level or some aggregate level, such as a neighborhood, a region or a country. In view of this, a great diversity of specific research topics exists on the drivers of what constitutes individual or aggregate “good health”, including those focusing on health inequalities, the gender gap in longevity, and regional mortality and longevity differences.

The current scoping review focuses on determinants of population health. Stated as such, this topic is quite broad. Indeed, we are interested in the very general question of what methods have been used to make the most of increasingly available region or country-specific databases to understand the drivers of population health through inter-country comparisons. Existing reviews indicate that researchers thus far tend to adopt a narrower focus. Usually, attention is given to only one health outcome at a time, with further geographical and/or population [ 1 , 2 ] restrictions. In some cases, the impact of one or more interventions is at the core of the review [ 3 – 7 ], while in others it is the relationship between health and just one particular predictor, e.g., income inequality, access to healthcare, government mechanisms [ 8 – 13 ]. Some relatively recent reviews on the subject of social determinants of health [ 4 – 6 , 14 – 17 ] have considered a number of indicators potentially influencing health as opposed to a single one. One review defines “social determinants” as “the social, economic, and political conditions that influence the health of individuals and populations” [ 17 ] while another refers even more broadly to “the factors apart from medical care” [ 15 ].

In the present work, we aimed to be more inclusive, setting no limitations on the nature of possible health correlates, as well as making use of a multitude of commonly accepted measures of general population health. The goal of this scoping review was to document the state of the art in the recent published literature on determinants of population health, with a particular focus on the types of determinants selected and the methodology used. In doing so, we also report the main characteristics of the results these studies found. The materials collected in this review are intended to inform our (and potentially other researchers’) future analyses on this topic. Since the production of health is subject to the law of diminishing marginal returns, we focused our review on those studies that included countries where a high standard of wealth has been achieved for some time, i.e., high-income countries belonging to the Organisation for Economic Co-operation and Development (OECD) or Europe. Adding similar reviews for other country income groups is of limited interest to the research we plan to do in this area.

In view of its focus on data and methods, rather than results, a formal protocol was not registered prior to undertaking this review, but the procedure followed the guidelines of the PRISMA statement for scoping reviews [ 18 ].

We focused on multi-country studies investigating the potential associations between any aggregate level (region/city/country) determinant and general measures of population health (e.g., life expectancy, mortality rate).

Within the query itself, we listed well-established population health indicators as well as the six world regions, as defined by the World Health Organization (WHO). We searched only in the publications’ titles in order to keep the number of hits manageable, and the ratio of broadly relevant abstracts over all abstracts in the order of magnitude of 10% (based on a series of time-focused trial runs). The search strategy was developed iteratively between the two authors and is presented in S1 Appendix . The search was performed by VV in PubMed and Web of Science on the 16 th of July, 2019, without any language restrictions, and with a start date set to the 1 st of January, 2013, as we were interested in the latest developments in this area of research.

Eligibility criteria

Records obtained via the search methods described above were screened independently by the two authors. Consistency between inclusion/exclusion decisions was approximately 90% and the 43 instances where uncertainty existed were judged through discussion. Articles were included subject to meeting the following requirements: (a) the paper was a full published report of an original empirical study investigating the impact of at least one aggregate level (city/region/country) factor on at least one health indicator (or self-reported health) of the general population (the only admissible “sub-populations” were those based on gender and/or age); (b) the study employed statistical techniques (calculating correlations, at the very least) and was not purely descriptive or theoretical in nature; (c) the analysis involved at least two countries or at least two regions or cities (or another aggregate level) in at least two different countries; (d) the health outcome was not differentiated according to some socio-economic factor and thus studied in terms of inequality (with the exception of gender and age differentiations); (e) mortality, in case it was one of the health indicators under investigation, was strictly “total” or “all-cause” (no cause-specific or determinant-attributable mortality).

Data extraction

The following pieces of information were extracted in an Excel table from the full text of each eligible study (primarily by VV, consulting with PB in case of doubt): health outcome(s), determinants, statistical methodology, level of analysis, results, type of data, data sources, time period, countries. The evidence is synthesized according to these extracted data (often directly reflected in the section headings), using a narrative form accompanied by a “summary-of-findings” table and a graph.

Search and selection

The initial yield contained 4583 records, reduced to 3686 after removal of duplicates ( Fig 1 ). Based on title and abstract screening, 3271 records were excluded because they focused on specific medical condition(s) or specific populations (based on morbidity or some other factor), dealt with intervention effectiveness, with theoretical or non-health related issues, or with animals or plants. Of the remaining 415 papers, roughly half were disqualified upon full-text consideration, mostly due to using an outcome not of interest to us (e.g., health inequality), measuring and analyzing determinants and outcomes exclusively at the individual level, performing analyses one country at a time, employing indices that are a mixture of both health indicators and health determinants, or not utilizing potential health determinants at all. After this second stage of the screening process, 202 papers were deemed eligible for inclusion. This group was further dichotomized according to level of economic development of the countries or regions under study, using membership of the OECD or Europe as a reference “cut-off” point. Sixty papers were judged to include high-income countries, and the remaining 142 included either low- or middle-income countries or a mix of both these levels of development. The rest of this report outlines findings in relation to high-income countries only, reflecting our own primary research interests. Nonetheless, we chose to report our search yield for the other income groups for two reasons. First, to gauge the relative interest in applied published research for these different income levels; and second, to enable other researchers with a focus on determinants of health in other countries to use the extraction we made here.

thumbnail

  • PPT PowerPoint slide
  • PNG larger image
  • TIFF original image

https://doi.org/10.1371/journal.pone.0239031.g001

Health outcomes

The most frequent population health indicator, life expectancy (LE), was present in 24 of the 60 studies. Apart from “life expectancy at birth” (representing the average life-span a newborn is expected to have if current mortality rates remain constant), also called “period LE” by some [ 19 , 20 ], we encountered as well LE at 40 years of age [ 21 ], at 60 [ 22 ], and at 65 [ 21 , 23 , 24 ]. In two papers, the age-specificity of life expectancy (be it at birth or another age) was not stated [ 25 , 26 ].

Some studies considered male and female LE separately [ 21 , 24 , 25 , 27 – 33 ]. This consideration was also often observed with the second most commonly used health index [ 28 – 30 , 34 – 38 ]–termed “total”, or “overall”, or “all-cause”, mortality rate (MR)–included in 22 of the 60 studies. In addition to gender, this index was also sometimes broken down according to age group [ 30 , 39 , 40 ], as well as gender-age group [ 38 ].

While the majority of studies under review here focused on a single health indicator, 23 out of the 60 studies made use of multiple outcomes, although these outcomes were always considered one at a time, and sometimes not all of them fell within the scope of our review. An easily discernable group of indices that typically went together [ 25 , 37 , 41 ] was that of neonatal (deaths occurring within 28 days postpartum), perinatal (fetal or early neonatal / first-7-days deaths), and post-neonatal (deaths between the 29 th day and completion of one year of life) mortality. More often than not, these indices were also accompanied by “stand-alone” indicators, such as infant mortality (deaths within the first year of life; our third most common index found in 16 of the 60 studies), maternal mortality (deaths during pregnancy or within 42 days of termination of pregnancy), and child mortality rates. Child mortality has conventionally been defined as mortality within the first 5 years of life, thus often also called “under-5 mortality”. Nonetheless, Pritchard & Wallace used the term “child mortality” to denote deaths of children younger than 14 years [ 42 ].

As previously stated, inclusion criteria did allow for self-reported health status to be used as a general measure of population health. Within our final selection of studies, seven utilized some form of subjective health as an outcome variable [ 25 , 43 – 48 ]. Additionally, the Health Human Development Index [ 49 ], healthy life expectancy [ 50 ], old-age survival [ 51 ], potential years of life lost [ 52 ], and disability-adjusted life expectancy [ 25 ] were also used.

We note that while in most cases the indicators mentioned above (and/or the covariates considered, see below) were taken in their absolute or logarithmic form, as a—typically annual—number, sometimes they were used in the form of differences, change rates, averages over a given time period, or even z-scores of rankings [ 19 , 22 , 40 , 42 , 44 , 53 – 57 ].

Regions, countries, and populations

Despite our decision to confine this review to high-income countries, some variation in the countries and regions studied was still present. Selection seemed to be most often conditioned on the European Union, or the European continent more generally, and the Organisation of Economic Co-operation and Development (OECD), though, typically, not all member nations–based on the instances where these were also explicitly listed—were included in a given study. Some of the stated reasons for omitting certain nations included data unavailability [ 30 , 45 , 54 ] or inconsistency [ 20 , 58 ], Gross Domestic Product (GDP) too low [ 40 ], differences in economic development and political stability with the rest of the sampled countries [ 59 ], and national population too small [ 24 , 40 ]. On the other hand, the rationales for selecting a group of countries included having similar above-average infant mortality [ 60 ], similar healthcare systems [ 23 ], and being randomly drawn from a social spending category [ 61 ]. Some researchers were interested explicitly in a specific geographical region, such as Eastern Europe [ 50 ], Central and Eastern Europe [ 48 , 60 ], the Visegrad (V4) group [ 62 ], or the Asia/Pacific area [ 32 ]. In certain instances, national regions or cities, rather than countries, constituted the units of investigation instead [ 31 , 51 , 56 , 62 – 66 ]. In two particular cases, a mix of countries and cities was used [ 35 , 57 ]. In another two [ 28 , 29 ], due to the long time periods under study, some of the included countries no longer exist. Finally, besides “European” and “OECD”, the terms “developed”, “Western”, and “industrialized” were also used to describe the group of selected nations [ 30 , 42 , 52 , 53 , 67 ].

As stated above, it was the health status of the general population that we were interested in, and during screening we made a concerted effort to exclude research using data based on a more narrowly defined group of individuals. All studies included in this review adhere to this general rule, albeit with two caveats. First, as cities (even neighborhoods) were the unit of analysis in three of the studies that made the selection [ 56 , 64 , 65 ], the populations under investigation there can be more accurately described as general urban , instead of just general. Second, oftentimes health indicators were stratified based on gender and/or age, therefore we also admitted one study that, due to its specific research question, focused on men and women of early retirement age [ 35 ] and another that considered adult males only [ 68 ].

Data types and sources

A great diversity of sources was utilized for data collection purposes. The accessible reference databases of the OECD ( https://www.oecd.org/ ), WHO ( https://www.who.int/ ), World Bank ( https://www.worldbank.org/ ), United Nations ( https://www.un.org/en/ ), and Eurostat ( https://ec.europa.eu/eurostat ) were among the top choices. The other international databases included Human Mortality [ 30 , 39 , 50 ], Transparency International [ 40 , 48 , 50 ], Quality of Government [ 28 , 69 ], World Income Inequality [ 30 ], International Labor Organization [ 41 ], International Monetary Fund [ 70 ]. A number of national databases were referred to as well, for example the US Bureau of Statistics [ 42 , 53 ], Korean Statistical Information Services [ 67 ], Statistics Canada [ 67 ], Australian Bureau of Statistics [ 67 ], and Health New Zealand Tobacco control and Health New Zealand Food and Nutrition [ 19 ]. Well-known surveys, such as the World Values Survey [ 25 , 55 ], the European Social Survey [ 25 , 39 , 44 ], the Eurobarometer [ 46 , 56 ], the European Value Survey [ 25 ], and the European Statistics of Income and Living Condition Survey [ 43 , 47 , 70 ] were used as data sources, too. Finally, in some cases [ 25 , 28 , 29 , 35 , 36 , 41 , 69 ], built-for-purpose datasets from previous studies were re-used.

In most of the studies, the level of the data (and analysis) was national. The exceptions were six papers that dealt with Nomenclature of Territorial Units of Statistics (NUTS2) regions [ 31 , 62 , 63 , 66 ], otherwise defined areas [ 51 ] or cities [ 56 ], and seven others that were multilevel designs and utilized both country- and region-level data [ 57 ], individual- and city- or country-level [ 35 ], individual- and country-level [ 44 , 45 , 48 ], individual- and neighborhood-level [ 64 ], and city-region- (NUTS3) and country-level data [ 65 ]. Parallel to that, the data type was predominantly longitudinal, with only a few studies using purely cross-sectional data [ 25 , 33 , 43 , 45 – 48 , 50 , 62 , 67 , 68 , 71 , 72 ], albeit in four of those [ 43 , 48 , 68 , 72 ] two separate points in time were taken (thus resulting in a kind of “double cross-section”), while in another the averages across survey waves were used [ 56 ].

In studies using longitudinal data, the length of the covered time periods varied greatly. Although this was almost always less than 40 years, in one study it covered the entire 20 th century [ 29 ]. Longitudinal data, typically in the form of annual records, was sometimes transformed before usage. For example, some researchers considered data points at 5- [ 34 , 36 , 49 ] or 10-year [ 27 , 29 , 35 ] intervals instead of the traditional 1, or took averages over 3-year periods [ 42 , 53 , 73 ]. In one study concerned with the effect of the Great Recession all data were in a “recession minus expansion change in trends”-form [ 57 ]. Furthermore, there were a few instances where two different time periods were compared to each other [ 42 , 53 ] or when data was divided into 2 to 4 (possibly overlapping) periods which were then analyzed separately [ 24 , 26 , 28 , 29 , 31 , 65 ]. Lastly, owing to data availability issues, discrepancies between the time points or periods of data on the different variables were occasionally observed [ 22 , 35 , 42 , 53 – 55 , 63 ].

Health determinants

Together with other essential details, Table 1 lists the health correlates considered in the selected studies. Several general categories for these correlates can be discerned, including health care, political stability, socio-economics, demographics, psychology, environment, fertility, life-style, culture, labor. All of these, directly or implicitly, have been recognized as holding importance for population health by existing theoretical models of (social) determinants of health [ 74 – 77 ].

thumbnail

https://doi.org/10.1371/journal.pone.0239031.t001

It is worth noting that in a few studies there was just a single aggregate-level covariate investigated in relation to a health outcome of interest to us. In one instance, this was life satisfaction [ 44 ], in another–welfare system typology [ 45 ], but also gender inequality [ 33 ], austerity level [ 70 , 78 ], and deprivation [ 51 ]. Most often though, attention went exclusively to GDP [ 27 , 29 , 46 , 57 , 65 , 71 ]. It was often the case that research had a more particular focus. Among others, minimum wages [ 79 ], hospital payment schemes [ 23 ], cigarette prices [ 63 ], social expenditure [ 20 ], residents’ dissatisfaction [ 56 ], income inequality [ 30 , 69 ], and work leave [ 41 , 58 ] took center stage. Whenever variables outside of these specific areas were also included, they were usually identified as confounders or controls, moderators or mediators.

We visualized the combinations in which the different determinants have been studied in Fig 2 , which was obtained via multidimensional scaling and a subsequent cluster analysis (details outlined in S2 Appendix ). It depicts the spatial positioning of each determinant relative to all others, based on the number of times the effects of each pair of determinants have been studied simultaneously. When interpreting Fig 2 , one should keep in mind that determinants marked with an asterisk represent, in fact, collectives of variables.

thumbnail

Groups of determinants are marked by asterisks (see S1 Table in S1 Appendix ). Diminishing color intensity reflects a decrease in the total number of “connections” for a given determinant. Noteworthy pairwise “connections” are emphasized via lines (solid-dashed-dotted indicates decreasing frequency). Grey contour lines encircle groups of variables that were identified via cluster analysis. Abbreviations: age = population age distribution, associations = membership in associations, AT-index = atherogenic-thrombogenic index, BR = birth rate, CAPB = Cyclically Adjusted Primary Balance, civilian-labor = civilian labor force, C-section = Cesarean delivery rate, credit-info = depth of credit information, dissatisf = residents’ dissatisfaction, distrib.orient = distributional orientation, EDU = education, eHealth = eHealth index at GP-level, exch.rate = exchange rate, fat = fat consumption, GDP = gross domestic product, GFCF = Gross Fixed Capital Formation/Creation, GH-gas = greenhouse gas, GII = gender inequality index, gov = governance index, gov.revenue = government revenues, HC-coverage = healthcare coverage, HE = health(care) expenditure, HHconsump = household consumption, hosp.beds = hospital beds, hosp.payment = hospital payment scheme, hosp.stay = length of hospital stay, IDI = ICT development index, inc.ineq = income inequality, industry-labor = industrial labor force, infant-sex = infant sex ratio, labor-product = labor production, LBW = low birth weight, leave = work leave, life-satisf = life satisfaction, M-age = maternal age, marginal-tax = marginal tax rate, MDs = physicians, mult.preg = multiple pregnancy, NHS = Nation Health System, NO = nitrous oxide emissions, PM10 = particulate matter (PM10) emissions, pop = population size, pop.density = population density, pre-term = pre-term birth rate, prison = prison population, researchE = research&development expenditure, school.ref = compulsory schooling reform, smoke-free = smoke-free places, SO = sulfur oxide emissions, soc.E = social expenditure, soc.workers = social workers, sugar = sugar consumption, terror = terrorism, union = union density, UR = unemployment rate, urban = urbanization, veg-fr = vegetable-and-fruit consumption, welfare = welfare regime, Wwater = wastewater treatment.

https://doi.org/10.1371/journal.pone.0239031.g002

Distances between determinants in Fig 2 are indicative of determinants’ “connectedness” with each other. While the statistical procedure called for higher dimensionality of the model, for demonstration purposes we show here a two-dimensional solution. This simplification unfortunately comes with a caveat. To use the factor smoking as an example, it would appear it stands at a much greater distance from GDP than it does from alcohol. In reality however, smoking was considered together with alcohol consumption [ 21 , 25 , 26 , 52 , 68 ] in just as many studies as it was with GDP [ 21 , 25 , 26 , 52 , 59 ], five. To aid with respect to this apparent shortcoming, we have emphasized the strongest pairwise links. Solid lines connect GDP with health expenditure (HE), unemployment rate (UR), and education (EDU), indicating that the effect of GDP on health, taking into account the effects of the other three determinants as well, was evaluated in between 12 to 16 studies of the 60 included in this review. Tracing the dashed lines, we can also tell that GDP appeared jointly with income inequality, and HE together with either EDU or UR, in anywhere between 8 to 10 of our selected studies. Finally, some weaker but still worth-mentioning “connections” between variables are displayed as well via the dotted lines.

The fact that all notable pairwise “connections” are concentrated within a relatively small region of the plot may be interpreted as low overall “connectedness” among the health indicators studied. GDP is the most widely investigated determinant in relation to general population health. Its total number of “connections” is disproportionately high (159) compared to its runner-up–HE (with 113 “connections”), and then subsequently EDU (with 90) and UR (with 86). In fact, all of these determinants could be thought of as outliers, given that none of the remaining factors have a total count of pairings above 52. This decrease in individual determinants’ overall “connectedness” can be tracked on the graph via the change of color intensity as we move outwards from the symbolic center of GDP and its closest “co-determinants”, to finally reach the other extreme of the ten indicators (welfare regime, household consumption, compulsory school reform, life satisfaction, government revenues, literacy, research expenditure, multiple pregnancy, Cyclically Adjusted Primary Balance, and residents’ dissatisfaction; in white) the effects on health of which were only studied in isolation.

Lastly, we point to the few small but stable clusters of covariates encircled by the grey bubbles on Fig 2 . These groups of determinants were identified as “close” by both statistical procedures used for the production of the graph (see details in S2 Appendix ).

Statistical methodology

There was great variation in the level of statistical detail reported. Some authors provided too vague a description of their analytical approach, necessitating some inference in this section.

The issue of missing data is a challenging reality in this field of research, but few of the studies under review (12/60) explain how they dealt with it. Among the ones that do, three general approaches to handling missingness can be identified, listed in increasing level of sophistication: case-wise deletion, i.e., removal of countries from the sample [ 20 , 45 , 48 , 58 , 59 ], (linear) interpolation [ 28 , 30 , 34 , 58 , 59 , 63 ], and multiple imputation [ 26 , 41 , 52 ].

Correlations, Pearson, Spearman, or unspecified, were the only technique applied with respect to the health outcomes of interest in eight analyses [ 33 , 42 – 44 , 46 , 53 , 57 , 61 ]. Among the more advanced statistical methods, the family of regression models proved to be, by and large, predominant. Before examining this closer, we note the techniques that were, in a way, “unique” within this selection of studies: meta-analyses were performed (random and fixed effects, respectively) on the reduced form and 2-sample two stage least squares (2SLS) estimations done within countries [ 39 ]; difference-in-difference (DiD) analysis was applied in one case [ 23 ]; dynamic time-series methods, among which co-integration, impulse-response function (IRF), and panel vector autoregressive (VAR) modeling, were utilized in one study [ 80 ]; longitudinal generalized estimating equation (GEE) models were developed on two occasions [ 70 , 78 ]; hierarchical Bayesian spatial models [ 51 ] and special autoregressive regression [ 62 ] were also implemented.

Purely cross-sectional data analyses were performed in eight studies [ 25 , 45 , 47 , 50 , 55 , 56 , 67 , 71 ]. These consisted of linear regression (assumed ordinary least squares (OLS)), generalized least squares (GLS) regression, and multilevel analyses. However, six other studies that used longitudinal data in fact had a cross-sectional design, through which they applied regression at multiple time-points separately [ 27 , 29 , 36 , 48 , 68 , 72 ].

Apart from these “multi-point cross-sectional studies”, some other simplistic approaches to longitudinal data analysis were found, involving calculating and regressing 3-year averages of both the response and the predictor variables [ 54 ], taking the average of a few data-points (i.e., survey waves) [ 56 ] or using difference scores over 10-year [ 19 , 29 ] or unspecified time intervals [ 40 , 55 ].

Moving further in the direction of more sensible longitudinal data usage, we turn to the methods widely known among (health) economists as “panel data analysis” or “panel regression”. Most often seen were models with fixed effects for country/region and sometimes also time-point (occasionally including a country-specific trend as well), with robust standard errors for the parameter estimates to take into account correlations among clustered observations [ 20 , 21 , 24 , 28 , 30 , 32 , 34 , 37 , 38 , 41 , 52 , 59 , 60 , 63 , 66 , 69 , 73 , 79 , 81 , 82 ]. The Hausman test [ 83 ] was sometimes mentioned as the tool used to decide between fixed and random effects [ 26 , 49 , 63 , 66 , 73 , 82 ]. A few studies considered the latter more appropriate for their particular analyses, with some further specifying that (feasible) GLS estimation was employed [ 26 , 34 , 49 , 58 , 60 , 73 ]. Apart from these two types of models, the first differences method was encountered once as well [ 31 ]. Across all, the error terms were sometimes assumed to come from a first-order autoregressive process (AR(1)), i.e., they were allowed to be serially correlated [ 20 , 30 , 38 , 58 – 60 , 73 ], and lags of (typically) predictor variables were included in the model specification, too [ 20 , 21 , 37 , 38 , 48 , 69 , 81 ]. Lastly, a somewhat different approach to longitudinal data analysis was undertaken in four studies [ 22 , 35 , 48 , 65 ] in which multilevel–linear or Poisson–models were developed.

Regardless of the exact techniques used, most studies included in this review presented multiple model applications within their main analysis. None attempted to formally compare models in order to identify the “best”, even if goodness-of-fit statistics were occasionally reported. As indicated above, many studies investigated women’s and men’s health separately [ 19 , 21 , 22 , 27 – 29 , 31 , 33 , 35 , 36 , 38 , 39 , 45 , 50 , 51 , 64 , 65 , 69 , 82 ], and covariates were often tested one at a time, including other covariates only incrementally [ 20 , 25 , 28 , 36 , 40 , 50 , 55 , 67 , 73 ]. Furthermore, there were a few instances where analyses within countries were performed as well [ 32 , 39 , 51 ] or where the full time period of interest was divided into a few sub-periods [ 24 , 26 , 28 , 31 ]. There were also cases where different statistical techniques were applied in parallel [ 29 , 55 , 60 , 66 , 69 , 73 , 82 ], sometimes as a form of sensitivity analysis [ 24 , 26 , 30 , 58 , 73 ]. However, the most common approach to sensitivity analysis was to re-run models with somewhat different samples [ 39 , 50 , 59 , 67 , 69 , 80 , 82 ]. Other strategies included different categorization of variables or adding (more/other) controls [ 21 , 23 , 25 , 28 , 37 , 50 , 63 , 69 ], using an alternative main covariate measure [ 59 , 82 ], including lags for predictors or outcomes [ 28 , 30 , 58 , 63 , 65 , 79 ], using weights [ 24 , 67 ] or alternative data sources [ 37 , 69 ], or using non-imputed data [ 41 ].

As the methods and not the findings are the main focus of the current review, and because generic checklists cannot discern the underlying quality in this application field (see also below), we opted to pool all reported findings together, regardless of individual study characteristics or particular outcome(s) used, and speak generally of positive and negative effects on health. For this summary we have adopted the 0.05-significance level and only considered results from multivariate analyses. Strictly birth-related factors are omitted since these potentially only relate to the group of infant mortality indicators and not to any of the other general population health measures.

Starting with the determinants most often studied, higher GDP levels [ 21 , 26 , 27 , 29 , 30 , 32 , 43 , 48 , 52 , 58 , 60 , 66 , 67 , 73 , 79 , 81 , 82 ], higher health [ 21 , 37 , 47 , 49 , 52 , 58 , 59 , 68 , 72 , 82 ] and social [ 20 , 21 , 26 , 38 , 79 ] expenditures, higher education [ 26 , 39 , 52 , 62 , 72 , 73 ], lower unemployment [ 60 , 61 , 66 ], and lower income inequality [ 30 , 42 , 53 , 55 , 73 ] were found to be significantly associated with better population health on a number of occasions. In addition to that, there was also some evidence that democracy [ 36 ] and freedom [ 50 ], higher work compensation [ 43 , 79 ], distributional orientation [ 54 ], cigarette prices [ 63 ], gross national income [ 22 , 72 ], labor productivity [ 26 ], exchange rates [ 32 ], marginal tax rates [ 79 ], vaccination rates [ 52 ], total fertility [ 59 , 66 ], fruit and vegetable [ 68 ], fat [ 52 ] and sugar consumption [ 52 ], as well as bigger depth of credit information [ 22 ] and percentage of civilian labor force [ 79 ], longer work leaves [ 41 , 58 ], more physicians [ 37 , 52 , 72 ], nurses [ 72 ], and hospital beds [ 79 , 82 ], and also membership in associations, perceived corruption and societal trust [ 48 ] were beneficial to health. Higher nitrous oxide (NO) levels [ 52 ], longer average hospital stay [ 48 ], deprivation [ 51 ], dissatisfaction with healthcare and the social environment [ 56 ], corruption [ 40 , 50 ], smoking [ 19 , 26 , 52 , 68 ], alcohol consumption [ 26 , 52 , 68 ] and illegal drug use [ 68 ], poverty [ 64 ], higher percentage of industrial workers [ 26 ], Gross Fixed Capital creation [ 66 ] and older population [ 38 , 66 , 79 ], gender inequality [ 22 ], and fertility [ 26 , 66 ] were detrimental.

It is important to point out that the above-mentioned effects could not be considered stable either across or within studies. Very often, statistical significance of a given covariate fluctuated between the different model specifications tried out within the same study [ 20 , 49 , 59 , 66 , 68 , 69 , 73 , 80 , 82 ], testifying to the importance of control variables and multivariate research (i.e., analyzing multiple independent variables simultaneously) in general. Furthermore, conflicting results were observed even with regards to the “core” determinants given special attention, so to speak, throughout this text. Thus, some studies reported negative effects of health expenditure [ 32 , 82 ], social expenditure [ 58 ], GDP [ 49 , 66 ], and education [ 82 ], and positive effects of income inequality [ 82 ] and unemployment [ 24 , 31 , 32 , 52 , 66 , 68 ]. Interestingly, one study [ 34 ] differentiated between temporary and long-term effects of GDP and unemployment, alluding to possibly much greater complexity of the association with health. It is also worth noting that some gender differences were found, with determinants being more influential for males than for females, or only having statistically significant effects for male health [ 19 , 21 , 28 , 34 , 36 , 37 , 39 , 64 , 65 , 69 ].

The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries.

Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art. What is more, these indicators are routinely considered one at a time, instead of, for example, employing existing statistical procedures to devise a more general, composite, index of population health, or using some of the established indices, such as disability-adjusted life expectancy (DALE) or quality-adjusted life expectancy (QALE). Although strong arguments for their wider use were already voiced decades ago [ 84 ], such summary measures surface only rarely in this research field.

On a related note, the greater data availability and accessibility that we enjoy today does not automatically equate to data quality. Nonetheless, this is routinely assumed in aggregate level studies. We almost never encountered a discussion on the topic. The non-mundane issue of data missingness, too, goes largely underappreciated. With all recent methodological advancements in this area [ 85 – 88 ], there is no excuse for ignorance; and still, too few of the reviewed studies tackled the matter in any adequate fashion.

Much optimism can be gained considering the abundance of different determinants that have attracted researchers’ attention in relation to population health. We took on a visual approach with regards to these determinants and presented a graph that links spatial distances between determinants with frequencies of being studies together. To facilitate interpretation, we grouped some variables, which resulted in some loss of finer detail. Nevertheless, the graph is helpful in exemplifying how many effects continue to be studied in a very limited context, if any. Since in reality no factor acts in isolation, this oversimplification practice threatens to render the whole exercise meaningless from the outset. The importance of multivariate analysis cannot be stressed enough. While there is no “best method” to be recommended and appropriate techniques vary according to the specifics of the research question and the characteristics of the data at hand [ 89 – 93 ], in the future, in addition to abandoning simplistic univariate approaches, we hope to see a shift from the currently dominating fixed effects to the more flexible random/mixed effects models [ 94 ], as well as wider application of more sophisticated methods, such as principle component regression, partial least squares, covariance structure models (e.g., structural equations), canonical correlations, time-series, and generalized estimating equations.

Finally, there are some limitations of the current scoping review. We searched the two main databases for published research in medical and non-medical sciences (PubMed and Web of Science) since 2013, thus potentially excluding publications and reports that are not indexed in these databases, as well as older indexed publications. These choices were guided by our interest in the most recent (i.e., the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore, despite holding a critical stance with regards to some aspects of how determinants-of-health research is currently conducted, we opted out of formally assessing the quality of the individual studies included. The reason for that is two-fold. On the one hand, we are unaware of the existence of a formal and standard tool for quality assessment of ecological designs. And on the other, we consider trying to score the quality of these diverse studies (in terms of regional setting, specific topic, outcome indices, and methodology) undesirable and misleading, particularly since we would sometimes have been rating the quality of only a (small) part of the original studies—the part that was relevant to our review’s goal.

Our aim was to investigate the current state of research on the very broad and general topic of population health, specifically, the way it has been examined in a multi-country context. We learned that data treatment and analytical approach were, in the majority of these recent studies, ill-equipped or insufficiently transparent to provide clarity regarding the underlying mechanisms of population health in high-income countries. Whether due to methodological shortcomings or the inherent complexity of the topic, research so far fails to provide any definitive answers. It is our sincere belief that with the application of more advanced analytical techniques this continuous quest could come to fruition sooner.

Supporting information

S1 checklist. preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (prisma-scr) checklist..

https://doi.org/10.1371/journal.pone.0239031.s001

S1 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s002

S2 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s003

  • View Article
  • Google Scholar
  • PubMed/NCBI
  • 75. Dahlgren G, Whitehead M. Policies and Strategies to Promote Equity in Health. Stockholm, Sweden: Institute for Future Studies; 1991.
  • 76. Brunner E, Marmot M. Social Organization, Stress, and Health. In: Marmot M, Wilkinson RG, editors. Social Determinants of Health. Oxford, England: Oxford University Press; 1999.
  • 77. Najman JM. A General Model of the Social Origins of Health and Well-being. In: Eckersley R, Dixon J, Douglas B, editors. The Social Origins of Health and Well-being. Cambridge, England: Cambridge University Press; 2001.
  • 85. Carpenter JR, Kenward MG. Multiple Imputation and its Application. New York: John Wiley & Sons; 2013.
  • 86. Molenberghs G, Fitzmaurice G, Kenward MG, Verbeke G, Tsiatis AA. Handbook of Missing Data Methodology. Boca Raton: Chapman & Hall/CRC; 2014.
  • 87. van Buuren S. Flexible Imputation of Missing Data. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2018.
  • 88. Enders CK. Applied Missing Data Analysis. New York: Guilford; 2010.
  • 89. Shayle R. Searle GC, Charles E. McCulloch. Variance Components: John Wiley & Sons, Inc.; 1992.
  • 90. Agresti A. Foundations of Linear and Generalized Linear Models. Hoboken, New Jersey: John Wiley & Sons Inc.; 2015.
  • 91. Leyland A. H. (Editor) HGE. Multilevel Modelling of Health Statistics: John Wiley & Sons Inc; 2001.
  • 92. Garrett Fitzmaurice MD, Geert Verbeke, Geert Molenberghs. Longitudinal Data Analysis. New York: Chapman and Hall/CRC; 2008.
  • 93. Wolfgang Karl Härdle LS. Applied Multivariate Statistical Analysis. Berlin, Heidelberg: Springer; 2015.

Quantitative vs. Qualitative Research Design: Understanding the Differences

what is quantitative research peer reviewed articles

As a future professional in the social and education landscape, research design is one of the most critical strategies that you will master to identify challenges, ask questions and form data-driven solutions to address problems specific to your industry. 

Many approaches to research design exist, and not all work in every circumstance. While all data-focused research methods are valid in their own right, certain research design methods are more appropriate for specific study objectives.

Unlock our resource to learn more about jump starting a career in research design — Research Design and Data Analysis for the Social Good .

We will discuss the differences between quantitative (numerical and statistics-focused) and qualitative (non-numerical and human-focused) research design methods so that you can determine which approach is most strategic given your specific area of graduate-level study. 

Understanding Social Phenomena: Qualitative Research Design

Qualitative research focuses on understanding a phenomenon based on human experience and individual perception. It is a non-numerical methodology relying on interpreting a process or result. Qualitative research also paves the way for uncovering other hypotheses related to social phenomena. 

In its most basic form, qualitative research is exploratory in nature and seeks to understand the subjective experience of individuals based on social reality.

Qualitative data is…

  • often used in fields related to education, sociology and anthropology; 
  • designed to arrive at conclusions regarding social phenomena; 
  • focused on data-gathering techniques like interviews, focus groups or case studies; 
  • dedicated to perpetuating a flexible, adaptive approach to data gathering;
  • known to lead professionals to deeper insights within the overall research study.

You want to use qualitative data research design if:

  • you work in a field concerned with enhancing humankind through the lens of social change;
  • your research focuses on understanding complex social trends and individual perceptions of those trends;
  • you have interests related to human development and interpersonal relationships.

Examples of Qualitative Research Design in Education

Here are just a few examples of how qualitative research design methods can impact education:

Example 1: Former educators participate in in-depth interviews to help determine why a specific school is experiencing a higher-than-average turnover rate compared to other schools in the region. These interviews help determine the types of resources that will make a difference in teacher retention. 

Example 2: Focus group discussions occur to understand the challenges that neurodivergent students experience in the classroom daily. These discussions prepare administrators, staff, teachers and parents to understand the kinds of support that will augment and improve student outcomes.

Example 3: Case studies examine the impacts of a new education policy that limits the number of teacher aids required in a special needs classroom. These findings help policymakers determine whether the new policy affects the learning outcomes of a particular class of students.

Interpreting the Numbers: Quantitative Research Design

Quantitative research tests hypotheses and measures connections between variables. It relies on insights derived from numbers — countable, measurable and statistically sound data. Quantitative research is a strategic research design used when basing critical decisions on statistical conclusions and quantifiable data.

Quantitative research provides numerical-backed quantifiable data that may approve or discount a theory or hypothesis.

Quantitative data is…

  • often used in fields related to education, data analysis and healthcare; 
  • designed to arrive at numerical, statistical conclusions based on objective facts;
  • focused on data-gathering techniques like experiments, surveys or observations;
  • dedicated to using mathematical principles to arrive at conclusions;
  • known to lead professionals to indisputable observations within the overall research study.

You want to use quantitative data research design if:

  • you work in a field concerned with analyzing data to inform decisions;
  • your research focuses on studying relationships between variables to form data-driven conclusions;
  • you have interests related to mathematics, statistical analysis and data science.

Examples of Quantitative Research Design in Education

Here are just a few examples of how quantitative research design methods may impact education:

Example 1: Researchers compile data to understand the connection between class sizes and standardized test scores. Researchers can determine if and what the relationship is between smaller, intimate class sizes and higher test scores for grade-school children using statistical and data analysis.

Example 2: Professionals conduct an experiment in which a group of high school students must complete a certain number of community service hours before graduation. Researchers compare those students to another group of students who did not complete service hours — using statistical analysis to determine if the requirement increased college acceptance rates.

Example 3: Teachers take a survey to examine an education policy that restricts the number of extracurricular activities offered at a particular academic institution. The findings help better understand the far-reaching impacts of extracurricular opportunities on academic performance.

Making the Most of Research Design Methods for Good: Vanderbilt University’s Peabody College

Vanderbilt University's Peabody College of Education and Human Development offers a variety of respected, nationally-recognized graduate programs designed with future agents of social change in mind. We foster a culture of excellence and compassion and guide you to become the best you can be — both in the classroom and beyond.

At Peabody College, you will experience

  • an inclusive, welcoming community of like-minded professionals;
  • the guidance of expert faculty with real-world industry experience;
  • opportunities for valuable, hands-on learning experiences,
  • the option of specializing depending on your specific area of interest.

Explore our monthly publication — Ideas in Action — for an inside look at how Peabody College translates discoveries into action.

Please click below to explore a few of the graduate degrees offered at Peabody College:

  • Child Studies M.Ed. — a rigorous Master of Education degree that prepares students to examine the developmental, learning and social issues concerning children and that allows students to choose from one of two tracks (the Clinical and Developmental Research Track or the Applied Professional Track).
  • Cognitive Psychology in Context M.S. — an impactful Master of Science program that emphasizes research design and statistical analysis to understand cognitive processes and real-world applications best, making it perfect for those interested in pursuing doctoral studies in cognitive science.
  • Education Policy M.P.P — an analysis-focused Master of Public Policy program designed for future leaders in education policy and practice, allowing students to specialize in either K-12 Education Policy, Higher Education Policy or Quantitative Methods in Education Policy. 
  • Quantitative Methods M.Ed. — a data-driven Master of Education degree that teaches the theory and application of quantitative analysis in behavioral, social and educational sciences.

Connect with the Community of Professionals Seeking to Enhance Humankind at Peabody College

At Peabody College, we equip you with the marketable, transferable skills needed to secure a valuable career in education and beyond. You will emerge from the graduate program of your choice ready to enhance humankind in more meaningful ways than you could have imagined.

If you want to develop the sought-after skills needed to be a force for change in the social and educational spaces, you are in the right place .

We invite you to request more information ; we will connect you with an admissions professional who can answer all your questions about choosing one of these transformative graduate degrees at Peabody College. You may also take this opportunity to review our admissions requirements and start your online application today. 

A young boy is sitting at a table with his parents.

Subscribe to the Blog

← All Posts

Media Inquiries

615-322-6397 Email

Latest Stories

7 quantitative data careers in education that can make a difference, the rewarding outcomes of being a special education teacher, should i earn an m.p.p. degree in education policy 4 questions to ask, keep reading.

what is quantitative research peer reviewed articles

Explore Stories by Topic

  • M.Ed. Degrees
  • Research Design

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • Write for Us
  • BMJ Journals

You are here

  • Volume 21, Issue 4
  • How to appraise quantitative research
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

This article has a correction. Please see:

  • Correction: How to appraise quantitative research - April 01, 2019

Download PDF

  • Xabi Cathala 1 ,
  • Calvin Moorley 2
  • 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
  • 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
  • Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

  • View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

  • 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
  • Gerrish K ,
  • Moorley C ,
  • Tunariu A , et al
  • Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

  • Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF:

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • What Is Peer Review? | Types & Examples

What Is Peer Review? | Types & Examples

Published on December 17, 2021 by Tegan George . Revised on June 22, 2023.

Peer review, sometimes referred to as refereeing , is the process of evaluating submissions to an academic journal. Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.

Peer-reviewed articles are considered a highly credible source due to the stringent process they go through before publication.

There are various types of peer review. The main difference between them is to what extent the authors, reviewers, and editors know each other’s identities. The most common types are:

  • Single-blind review
  • Double-blind review
  • Triple-blind review

Collaborative review

Open review.

Relatedly, peer assessment is a process where your peers provide you with feedback on something you’ve written, based on a set of criteria or benchmarks from an instructor. They then give constructive feedback, compliments, or guidance to help you improve your draft.

Table of contents

What is the purpose of peer review, types of peer review, the peer review process, providing feedback to your peers, peer review example, advantages of peer review, criticisms of peer review, other interesting articles, frequently asked questions about peer reviews.

Many academic fields use peer review, largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the manuscript. For this reason, academic journals are among the most credible sources you can refer to.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure.

Peer assessment is often used in the classroom as a pedagogical tool. Both receiving feedback and providing it are thought to enhance the learning process, helping students think critically and collaboratively.

Prevent plagiarism. Run a free check.

Depending on the journal, there are several types of peer review.

Single-blind peer review

The most common type of peer review is single-blind (or single anonymized) review . Here, the names of the reviewers are not known by the author.

While this gives the reviewers the ability to give feedback without the possibility of interference from the author, there has been substantial criticism of this method in the last few years. Many argue that single-blind reviewing can lead to poaching or intellectual theft or that anonymized comments cause reviewers to be too harsh.

Double-blind peer review

In double-blind (or double anonymized) review , both the author and the reviewers are anonymous.

Arguments for double-blind review highlight that this mitigates any risk of prejudice on the side of the reviewer, while protecting the nature of the process. In theory, it also leads to manuscripts being published on merit rather than on the reputation of the author.

Triple-blind peer review

While triple-blind (or triple anonymized) review —where the identities of the author, reviewers, and editors are all anonymized—does exist, it is difficult to carry out in practice.

Proponents of adopting triple-blind review for journal submissions argue that it minimizes potential conflicts of interest and biases. However, ensuring anonymity is logistically challenging, and current editing software is not always able to fully anonymize everyone involved in the process.

In collaborative review , authors and reviewers interact with each other directly throughout the process. However, the identity of the reviewer is not known to the author. This gives all parties the opportunity to resolve any inconsistencies or contradictions in real time, and provides them a rich forum for discussion. It can mitigate the need for multiple rounds of editing and minimize back-and-forth.

Collaborative review can be time- and resource-intensive for the journal, however. For these collaborations to occur, there has to be a set system in place, often a technological platform, with staff monitoring and fixing any bugs or glitches.

Lastly, in open review , all parties know each other’s identities throughout the process. Often, open review can also include feedback from a larger audience, such as an online forum, or reviewer feedback included as part of the final published product.

While many argue that greater transparency prevents plagiarism or unnecessary harshness, there is also concern about the quality of future scholarship if reviewers feel they have to censor their comments.

In general, the peer review process includes the following steps:

  • First, the author submits the manuscript to the editor.
  • Reject the manuscript and send it back to the author, or
  • Send it onward to the selected peer reviewer(s)
  • Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made.
  • Lastly, the edited manuscript is sent back to the author. They input the edits and resubmit it to the editor for publication.

The peer review process

In an effort to be transparent, many journals are now disclosing who reviewed each article in the published product. There are also increasing opportunities for collaboration and feedback, with some journals allowing open communication between reviewers and authors.

It can seem daunting at first to conduct a peer review or peer assessment. If you’re not sure where to start, there are several best practices you can use.

Summarize the argument in your own words

Summarizing the main argument helps the author see how their argument is interpreted by readers, and gives you a jumping-off point for providing feedback. If you’re having trouble doing this, it’s a sign that the argument needs to be clearer, more concise, or worded differently.

If the author sees that you’ve interpreted their argument differently than they intended, they have an opportunity to address any misunderstandings when they get the manuscript back.

Separate your feedback into major and minor issues

It can be challenging to keep feedback organized. One strategy is to start out with any major issues and then flow into the more minor points. It’s often helpful to keep your feedback in a numbered list, so the author has concrete points to refer back to.

Major issues typically consist of any problems with the style, flow, or key points of the manuscript. Minor issues include spelling errors, citation errors, or other smaller, easy-to-apply feedback.

Tip: Try not to focus too much on the minor issues. If the manuscript has a lot of typos, consider making a note that the author should address spelling and grammar issues, rather than going through and fixing each one.

The best feedback you can provide is anything that helps them strengthen their argument or resolve major stylistic issues.

Give the type of feedback that you would like to receive

No one likes being criticized, and it can be difficult to give honest feedback without sounding overly harsh or critical. One strategy you can use here is the “compliment sandwich,” where you “sandwich” your constructive criticism between two compliments.

Be sure you are giving concrete, actionable feedback that will help the author submit a successful final draft. While you shouldn’t tell them exactly what they should do, your feedback should help them resolve any issues they may have overlooked.

As a rule of thumb, your feedback should be:

  • Easy to understand
  • Constructive

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Below is a brief annotated research example. You can view examples of peer feedback by hovering over the highlighted sections.

Influence of phone use on sleep

Studies show that teens from the US are getting less sleep than they were a decade ago (Johnson, 2019) . On average, teens only slept for 6 hours a night in 2021, compared to 8 hours a night in 2011. Johnson mentions several potential causes, such as increased anxiety, changed diets, and increased phone use.

The current study focuses on the effect phone use before bedtime has on the number of hours of sleep teens are getting.

For this study, a sample of 300 teens was recruited using social media, such as Facebook, Instagram, and Snapchat. The first week, all teens were allowed to use their phone the way they normally would, in order to obtain a baseline.

The sample was then divided into 3 groups:

  • Group 1 was not allowed to use their phone before bedtime.
  • Group 2 used their phone for 1 hour before bedtime.
  • Group 3 used their phone for 3 hours before bedtime.

All participants were asked to go to sleep around 10 p.m. to control for variation in bedtime . In the morning, their Fitbit showed the number of hours they’d slept. They kept track of these numbers themselves for 1 week.

Two independent t tests were used in order to compare Group 1 and Group 2, and Group 1 and Group 3. The first t test showed no significant difference ( p > .05) between the number of hours for Group 1 ( M = 7.8, SD = 0.6) and Group 2 ( M = 7.0, SD = 0.8). The second t test showed a significant difference ( p < .01) between the average difference for Group 1 ( M = 7.8, SD = 0.6) and Group 3 ( M = 6.1, SD = 1.5).

This shows that teens sleep fewer hours a night if they use their phone for over an hour before bedtime, compared to teens who use their phone for 0 to 1 hours.

Peer review is an established and hallowed process in academia, dating back hundreds of years. It provides various fields of study with metrics, expectations, and guidance to ensure published work is consistent with predetermined standards.

  • Protects the quality of published research

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. Any content that raises red flags for reviewers can be closely examined in the review stage, preventing plagiarized or duplicated research from being published.

  • Gives you access to feedback from experts in your field

Peer review represents an excellent opportunity to get feedback from renowned experts in your field and to improve your writing through their feedback and guidance. Experts with knowledge about your subject matter can give you feedback on both style and content, and they may also suggest avenues for further research that you hadn’t yet considered.

  • Helps you identify any weaknesses in your argument

Peer review acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process. This way, you’ll end up with a more robust, more cohesive article.

While peer review is a widely accepted metric for credibility, it’s not without its drawbacks.

  • Reviewer bias

The more transparent double-blind system is not yet very common, which can lead to bias in reviewing. A common criticism is that an excellent paper by a new researcher may be declined, while an objectively lower-quality submission by an established researcher would be accepted.

  • Delays in publication

The thoroughness of the peer review process can lead to significant delays in publishing time. Research that was current at the time of submission may not be as current by the time it’s published. There is also high risk of publication bias , where journals are more likely to publish studies with positive findings than studies with negative findings.

  • Risk of human error

By its very nature, peer review carries a risk of human error. In particular, falsification often cannot be detected, given that reviewers would have to replicate entire experiments to ensure the validity of results.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Normal distribution
  • Measures of central tendency
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Thematic analysis
  • Discourse analysis
  • Cohort study
  • Ethnography

Research bias

  • Implicit bias
  • Cognitive bias
  • Conformity bias
  • Hawthorne effect
  • Availability heuristic
  • Attrition bias
  • Social desirability bias

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

In general, the peer review process follows the following steps: 

  • Reject the manuscript and send it back to author, or 
  • Send it onward to the selected peer reviewer(s) 
  • Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made. 
  • Lastly, the edited manuscript is sent back to the author. They input the edits, and resubmit it to the editor for publication.

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. It also represents an excellent opportunity to get feedback from renowned experts in your field. It acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process.

Peer-reviewed articles are considered a highly credible source due to this stringent process they go through before publication.

Many academic fields use peer review , largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the published manuscript.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure. 

A credible source should pass the CRAAP test  and follow these guidelines:

  • The information should be up to date and current.
  • The author and publication should be a trusted authority on the subject you are researching.
  • The sources the author cited should be easy to find, clear, and unbiased.
  • For a web source, the URL and layout should signify that it is trustworthy.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). What Is Peer Review? | Types & Examples. Scribbr. Retrieved July 10, 2024, from https://www.scribbr.com/methodology/peer-review/

Is this article helpful?

Tegan George

Tegan George

Other students also liked, what are credible sources & how to spot them | examples, ethical considerations in research | types & examples, applying the craap test & evaluating sources, what is your plagiarism score.

MCPHS Library Logo

  • Scholarly Literature Types
  • Peer Review This link opens in a new window
  • Examples of Academic Sources

Librarian Tip

Qualitative vs. quantitative research, mixed methods research, strategy one: keywords, strategy two: filters & limits, strategy three: subject terms, what does the abstract tell you, initial evaluation: what kind of research is this article, practice: identifying qualitative and quantitative research articles, practice: finding qualitative and quantitative research articles.

  • Anatomy of a Scholarly Article

Differentiating between types of sources can be challenging and is an important skill to develop. This page explains three different search strategies for finding articles of particular methodologies and how to differentiate between them. In reality, you're going to combine these strategies to be as efficient and effective as possible. 

Definitions |  Search Strategies |  Evaluation Techniques  | Hands-on Practice

  Qualitative Quantitative
Definition:

To understand and interpret social interactions.

Research that seeks to provide understanding of human experience, perceptions, motivations, intentions, and behaviours based on description and observation and utilizing a naturalistic interpretative approach to a subject and its contextual setting.

To test hypotheses, look at cause & effect, and make predictions.

Research based on traditional scientific methods, which generates numerical data and usually seeks to establish causal relationships between two or more variables, using statistical methods to test the strength and significance of the relationships.

Involves: Observations described in words of behavior in natural environment. Observations measured in numbers of behavior under controlled conditions; isolate causal effects.
Starts with: A situation the researcher can observe. A testable hypothesis.
Scientific Method: Exploratory or bottom up: the researcher can generate a new hypothesis and theory from the data collected. Confirmatory or top-down: the researcher tests the hypothesis and theory with the data.
Nature of Reality: Multiple realities; subjective. Human behavior is dynamic, situational, social and personal. Single reality; objective. Human behavior is regular and predictable.
Goals of study design: Participants are comfortable with the researcher. They are honest and forthcoming, so that the researcher can make robust observations.

Others can repeat the findings of the study.

Variables are defined and correlations between them are studied.

Drawbacks: If the researcher is biased, or is expecting to find certain results, it can be difficult to make completely objective observations. Researchers may be so careful about measurement methods that they do not make connections to a greater context.
Variables: Study of the whole, not variables. Specific variables studied.
Group Studied: Smaller and not as randomly selected. Larger and more randomly selected.
Some methods:

Open-ended interviews

Focus groups

Observation

Participant observation

Field notes

Close-ended interviews

Surveys and other instruments

Clinical Trials

Laboratory Experiments

Final Report Narrative report with contextual description and direct quotes from research participants. Statistical report with correlations, comparisons of means, statistical significance of findings.

Mixed-methods  is more than simply the ad hoc combination of qualitative and quantiative data in a single study. It involves the planned mixing of qualitative and quantitative methods at a predetermined stage of the research process, be it during the initial study planning, the process of data collection, data analysis or reporting, in order to better answer the research question. 

Dictionary of Nursing Theory and Research

You may remember from previous research instruction that keywords are words or phrases used to describe your research topic. Try including a type of research or research methodology as one of your keywords - for example, diabetes AND quantitative. This should help narrow down your search results to exactly the type of article you're looking for. Below are a couple of other examples:

CINAHL search showing use of keyword for content

Database Tools:

Databases are here to help! Be sure to examine the limiter or filter options that are available. Some databases will include ways you can narrow your results to specific types of scholarly articles or specific types of research. For example:

Limit results to Scholarly (Peer Reviewed) Journals

Subject Terms:

Subject terms are search terms that have already been assigned to each source within a database as one way to describe the content. Most databases include Subject Terms, Subject Headings, Keywords, or some other option amongst the limiters/filters to browse terms related to what you've already searched for. You can then select terms related to types of scholarly articles or types of research to help limit your results.  Below is am example of the keywords available after searching in the database Scopus. Potentially useful limiters have a red circle next to them.

what is quantitative research peer reviewed articles

Remember: If you're stuck,  consider contacting your librarian!  

Very often, the abstract of an article will make it clear what type of research has been done, as seen in the example below. This journal even lists the type of research as one of the article's keywords.

what is quantitative research peer reviewed articles

  • Start with the title and work your way down through publication type, abstract, and the other items in the database record.
  • The author(s), journal and database want you to know what the article is, they aren't trying to trick you!
  • Reviews are when the authors are summarizing, or reviewing the available literature on a topic. These are not considered original research articles.
  • Mixed methods research uses qualitative AND quantitative methods, so remember to check for both. 
  • Are there other assignment criteria for the article you're evaluating? Don't forget those too!

Think you found an article that works? Great!

Remember to read the whole article to be certain!

  • Identifying Qualitative and Quantitative Research Articles One of the first things you'll do when you find an interesting article is determine what type of research it encompasses. Use this interactive tutorial to practice determining if an article is likely qualitative or quantitative research.
  • Finding Qualitative and Quantitative Research Articles Now that you know what to look for in an article, practice how to find the type of articles you need.
  • << Previous: Examples of Academic Sources
  • Next: Anatomy of a Scholarly Article >>
  • Last Updated: Sep 13, 2023 5:23 PM
  • URL: https://mcphs.libguides.com/ScholarlySources

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

  • Advanced Search
  • Journal List
  • HHS Author Manuscripts

Logo of nihpa

Quantitative Approaches for the Evaluation of Implementation Research Studies

Justin d. smith.

1 Center for Prevention Implementation Methodology (Ce-PIM) for Drug Abuse and HIV, Department of Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of Medicine, 750 N Lake Shore Dr., Chicago, Illinois, USA.

Mohamed Hasan

2 Center for Healthcare Studies, Institute of Public Health and Medicine, Northwestern University Feinberg School of Medicine, 633 N St. Claire St., Chicago, Illinois, USA.

Authors’ contributions

Implementation research necessitates a shift from clinical trial methods in both the conduct of the study and in the way that it is evaluated given the focus on the impact of implementation strategies. That is, the methods or techniques to support the adoption and delivery of a clinical or preventive intervention, program, or policy. As strategies target one or more levels within the service delivery system, evaluating their impact needs to follow suit. This article discusses the methods and practices involved in quantitative evaluations of implementation research studies. We focus on evaluation methods that characterize and quantify the overall impacts of an implementation strategy on various outcomes. This article discusses available measurement methods for common quantitative implementation outcomes involved in such an evaluation—adoption, fidelity, implementation cost, reach, and sustainment—and the sources of such data for these metrics using established taxonomies and frameworks. Last, we present an example of a quantitative evaluation from an ongoing randomized rollout implementation trial of the Collaborative Care Model for depression management in a large primary healthcare system.

1. Background

As part of this special issue on implementation science, this article discusses quantitative methods for evaluating implementation research studies and presents an example of an ongoing implementation trial for illustrative purposes. We focus on what is called “summative evaluation,” which characterizes and quantifies the impacts of an implementation strategy on various outcomes ( Gaglio & Glasgow, 2017 ). This type of evaluation involves aggregation methods conducted at the end of a study to assess the success of an implementation strategy on the adoption, delivery, and sustainment of an evidence-based practice (EBP), and the cost associated with implementation ( Bauer, Damschroder, Hagedorn, Smith, & Kilbourne, 2015 ). These results help decision makers understand the overall worth of an implementation strategy and whether to upscale, modify, or discontinue ( Bauer et al., 2015 ). This topic complements others in this issue on formative evaluation (Elwy et al.) and qualitative methods (Hamilton et al.), which are also used in implementation research evaluation.

Implementation research, as defined by the United States National Institutes of Health (NIH), is “the scientific study of the use of strategies [italics added] to adopt and integrate evidence-based health interventions into clinical and community settings in order to improve patient outcomes and benefit population health. Implementation research seeks to understand the behavior of healthcare professionals and support staff, healthcare organizations, healthcare consumers and family members, and policymakers in context as key influences on the adoption, implementation and sustainability of evidence-based interventions and guidelines” ( Department of Health and Human Services, 2019 ). Implementation strategies are methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice ( Powell et al., 2015 ).

To grasp the evaluation methods used in implementation research, one must appreciate the nature of this research and how the study designs, aims, and measures differ in fundamental ways from those methods with which readers will be most familiar—that is, evaluations of clinical efficacy or effectiveness trials. First, whereas clinical intervention research focuses on how a given clinical intervention—meaning a pill, program, practice, principle, product, policy, or procedure ( Brown et al., 2017 )—affects a health outcome at the patient level, implementation research focuses on how systems can take that intervention to scale in order to improve health outcomes of the broader community ( Colditz & Emmons, 2017 ). Thus, when implementation strategies are the focus, the outcomes evaluated are at the system level. Figure 1 illustrates the emphasis (foreground box) of effectiveness versus implementation research and the corresponding outcomes that would be included in the evaluation. This difference can be illustrated by “hybrid trials” in which effectiveness and implementation are evaluated simultaneously but with different outcomes for each aim ( Curran, Bauer, Mittman, Pyne, & Stetler, 2012 ; also see Landes et al., this issue).

An external file that holds a picture, illustration, etc.
Object name is nihms-1539194-f0001.jpg

Emphasis and Outcomes Evaluated in Clinical Effectiveness versus Implementation Research

Note. Adapted from a slide developed by C. Hendricks Brown.

2. Design Considerations for Evaluating Implementation Research Studies

The stark contrast between the emphasis in implementation versus effectiveness trials occurs largely because implementation strategies most often, but not always, target one or more levels within the system that supports the adoption and implementation of the intervention, such as the provider, clinic, school, health department, or even state or national levels ( Powell et al., 2015 ). Implementation strategies are discussed in this issue by Kirchner and colleagues. With the focus on levels within which patients who receive the clinical or preventive intervention are embedded, research designs in implementation research follow suit. The choice of a study design to evaluate an implementation strategy influences the confidence in the association drawn between a strategy and an observed effect ( Grimshaw, Campbell, Eccles, & Steen, 2000 ). Strong designs and methodologically-robust studies support the validity of the evaluations and provide evidence likely to be used by policy makers. Study designs are generally classified into observational (descriptive) and experimental/quasi-experimental.

Brown et al. (2017) described three broad types of designs for implementation research. ( 1 ) Within-site designs involve evaluation of the effects of implementation strategies within a single service system unit (e.g., clinic, hospital). Common within-site designs include post, pre-post, and interrupted time series. While these designs are simple and can be useful for understanding the impact in a local context ( Cheung & Duan, 2014 ), they contribute limited generalizable knowledge due to the biases inherent small-sample studies with no direct comparison condition. Brown et al. describe two broad design types can be used to create generalizable knowledge as they inherently involve multiple units for aggregation and comparison using the evaluation methods described in this article. ( 2 ) Between-site designs involve comparison of outcomes between two or more service system units or clusters/groups of units. While they commonly involve the testing of a novel implementation strategy compared to routine practice (i.e., implementation as usual), they can also be head-to-head tests of two or more novel implementation strategies for the same intervention, which we refer to as a comparative implementation trial (e.g., Smith et al., 2019 ). ( 3 ) Within- and between-site designs add a time-based crossover for each unit in which they begin in one condition—usually routine practice—and then move to a second condition involving the introduction of the implementation strategy. We refer to this category as rollout trials, which includes the stepped-wedge and dynamic wait-list design ( Brown et al., 2017 ; Landsverk et al., 2017 ; Wyman, Henry, Knoblauch, & Brown, 2015 ). Designs for implementation research are discussed in this issue by Miller and colleagues.

3. Quantitative Methods for Evaluating Implementation Outcomes

While summative evaluation is distinguishable from formative evaluation (see Elwy et al. this issue ), proper understanding of the implementation strategy requires using both methods, perhaps at different stages of implementation research ( The Health Foundation, 2015 ). Formative evaluation is a rigorous assessment process designed to identify potential and actual influences on the effectiveness of implementation efforts ( Stetler et al., 2006 ). Earlier stages of implementation research might rely solely on formative evaluation and the use of qualitative and mixed methods approaches. In contrast, later stage implementation research involves powered tests of the effect of one or more implementation strategies and are thus likely to use a between-site or a within- and between-site research design with at least one quantitative outcome. Quantitative methods are especially important to explore the extent and variation of change (within and across units) induced by the implementation strategies.

Proctor and colleagues (2011) provide a taxonomy of available implementation outcomes, which include acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration/reach, and sustainability/sustainment. Table 1 in this article presents a modified version of Table 1 from Proctor et al. (2011) , focusing only on the quantitative measurement characteristics of these outcomes. Table 1 also includes the additional metrics of speed and quantity, which will be discussed in more detail in the case example. As noted in Table 1 , and by Proctor et al. (2011) , certain outcomes are more applicable at earlier versus later stages of implementation research. A recent review of implementation research in the field of HIV indicated that earlier stage implementation research was more likely to focus on acceptability and feasibility, whereas later stage testing of implementation strategies focused less on these and more on adoption, cost, penetration/reach, fidelity, and sustainability ( Smith et al., 2019 ). These sources of quantitative information are at multiple levels in the service delivery system, such as the intervention delivery agent, leadership, and key stakeholders in and outside of a particular delivery system ( Brown et al., 2013 ).

Quantitative Measurement Characteristics of Common Implementation Outcomes

Implementation outcomeLevel of analysisOther terms in the literatureSalience by implementation stageQuantitative measurement methodExample from the published literature
AcceptabilityIndividual provider Individual consumerSatisfaction with various aspects of the innovation (e.g. content, complexity, comfort, delivery, and credibility)Early for adoption Ongoing for penetration Late for sustainabilitySurvey Administrative data Refused/blankAuslander, W., McGinnis, H., Tlapek, S., Smith, P., Foster, A., Edmond, T., & Dunn, J. (2017). Adaptation and implementation of a trauma-focused cognitive behavioral intervention for girls in child welfare. 87(3), 206–215. doi :
AdoptionIndividual provider Organization or settingUptake; utilization; initial implementation; intention to tryEarly to midAdministrative data Observation SurveyKnudsen, H. K., & Roman, P. M. (2014). Dissemination, adoption, and implementation of acamprosate for treating alcohol use disorders. Journal of studies on alcohol and drugs, 75(3), 467–475. doi :
AppropriatenessIndividual provider Individual consumer Organization or settingPerceived fit; relevance; compatibility; suitability; usefulness; practicabilityEarly (prior to adoption)SurveyProctor, E., Ramsey, A. T., Brown, M. T., Malone, S., Hooley, C., & McKay, V. (2019). Training in Implementation Practice Leadership (TRIPLE): evaluation of a novel practice change strategy in behavioral health organizations. Implementation science: IS, 14(1), 66. doi:
FeasibilityIndividual providers Organization or settingActual fit or utility; suitability for everyday use; practicabilityEarly (during adoption)Survey Administrative dataLyon, A. R., Bruns, E. J., Ludwig, K., Stoep, A. V., Pullmann, M. D., Dorsey, S.,… McCauley, E. (2015). The Brief Intervention for School Clinicians (BRISC): A mixed-methods evaluation of feasibility, acceptability, and contextual appropriateness. School mental health, 7(4), 273–286. doi:
FidelityIndividual provider ProgramDelivered as intended; adherence; integrity; quality of program deliveryEarly to midObservation Checklists Self-reportSmith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013). Indirect effects of fidelity to the family check-up on changes in parenting and early childhood problem behaviors. Journal of consulting and clinical psychology, 81(6), 962–974. doi :
Implementation costProvider or providing Institution Payor Individual consumerMarginal cost; cost-effectiveness; cost-benefitEarly for adoption and feasibility Mid for penetration Late for sustainabilityAdministrative dataJordan N, Graham AK, Berkel C, Smith JD (2019). Budget impact analysis of preparing to implement the Family Check-Up 4 Health in primary care to reduce pediatric obesity. 20(5), 655–664. doi:
Penetration/ReachOrganization or settingLevel of institutionalization? Spread? Service access?Mid to lateCase audit ChecklistsEmily M. Woltmann, M. S. W., Rob Whitley, P. D., Gregory J. McHugo, P. D., Mary Brunette, M. D., William C. Torrey, M. D., Laura Coots, M. S.,… Robert E. Drake, M. D., Ph.D.,. (2008). The Role of Staff Turnover in the Implementation of Evidence-Based Practices in Mental Health Care. 59(7), 732737. doi :
SustainabilityAdministrators Organization or settingMaintenance; continuation; durability; incorporation; integration; sustained use; institutionalization; routinization;LateCase audit Checklists SurveyScudder, A. T., Taber-Thomas, S. M., Schaffner, K., Pemberton, J. R., Hunter, L., & Herschell, A. D. (2017). A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives. 15(1), 102. doi:
QuantityOrganization or settingProportion; quantityEarly through lateAdministrative data ObservationBrown, C. H., Chamberlain, P., Saldana, L., Padgett, C., Wang, W., & Cruden, G. (2014). Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial. 134. doi:
SpeedOrganization or settingDuration (speed)Early through lateAdministrative data ObservationSaldana, L., Chamberlain, P., Wang, W., & Hendricks Brown, C. (2012). Predicting program start-up using the stages of implementation measure. 39(6), 419–425. doi:

Note. This table is modeled after Table 1 in the Proctor et al. (2011) article.

Methods for quantitative data collection include structured surveys; use of administrative records, including payor and health expenditure records; extraction from the electronic health record (EHR); and direct observation. Structured surveys are commonly used to assess attitudes and perceptions of providers and patients concerning such factors as the ability to sustain the intervention and a host of potential facilitators and barriers to implementation (e.g., Bertrand, Holtgrave, & Gregowski, 2009 ; Luke, Calhoun, Robichaux, Elliott, & Moreland-Russell, 2014 ). Administrative databases and the EHR are used to assess aspects of intervention delivery that result from the implementation strategies ( Bauer et al., 2015 ). Although the EHR supports automatic and cumulative data acquisition, its utility for measuring implementation outcomes is limited depending on the type of implementation strategy and the intervention. For example, it is well suited for gathering data on EHR-based implementation strategies, such as clinical decision supports and symptom screening, but less useful for behaviors that would not otherwise be documented in the EHR (e.g., effects of a learning collaborative on adoption of a cognitive behavioral therapy protocol). Last, observational assessment of implementation is fairly common but resource intensive, which limits its use outside of funded research. This is particularly germane to assessing fidelity of implementation, which is commonly observational in funded research but is rarely done when the intervention is adopted under real-world circumstances ( Schoenwald et al., 2011 ). The costs associated with observational fidelity measurement has led to promising efforts to automate this process with machine learning methods (e.g., Imel et al., 2019 ).

Quantitative evaluation of implementation research studies most commonly involves assessment of multiple outcome metrics to garner a comprehensive appraisal of the effects of the implementation strategy. This is due in large part to the interrelatedness and interdependence of these metrics. A shortcoming of the Proctor et al. (2011) taxonomy is that it does not specify relations between outcomes, rather they are simply listed. The RE-AIM evaluation framework ( Gaglio, Shoup, & Glasgow, 2013 ; Glasgow, Vogt, & Boles, 1999 ) is commonly used and includes consideration of the interrelatedness between both the implementtion outcomes and the clinical effectiveness of the intervention being implemented. Thus, it is particularly well-suited for effectiveness-implementation hybrid trials ( Curran et al., 2012 ; also see Landes et al., this issue). RE-AIM stands for Reach, Effectiveness (of the clinical or preventive intervention), Adoption, Implementation, and Maintenance. Each metric is important for determining the overall public health impact of the implementation, but they are somewhat interdependent. As such, RE-AIM dimensions can be presented in some combination, such as the “public health impact” metric (reach rate multiplied by the effect size of the intervention) ( Glasgow, Klesges, Dzewaltowski, Estabrooks, & Vogt, 2006 ). RE-AIM is one in a class of evaluation frameworks. For a review, see Tabak, Khoong, Chambers, and Brownson (2012) .

4. Resources for Quantitative Evaluation in Implementation Research

There are a number of useful resources for the quantitative measures used to evaluate implementation research studies. First is the Instrument Review Project affiliated with the Society for Implementation Research Collaboration ( Lewis, Stanick, et al., 2015 ). The results of this systematic review of measures indicated significant variability in the coverage of measures across implementation outcomes and salient determinants of implementation (commonly referred to as barriers and facilitators). The authors reviewed each identified measure for the psychometric properties of internal consistency, structural validity, predictive validity, having norms, responsiveness, and usability (pragmatism). Few measures were deemed high-quality and psychometrically sound due in large part to not using gold-standard measure development methods. This review is ongoing and a website ( https://societyforimplementationresearchcollaboration.org/sirc-instrument-project/ ) is continuously updated to reflect completed work, as well as emerging measures in the field, and is available to members of the society. A number of articles and book chapters provide critical discussions of the state of measurement in implementation research, noting the need for validation of instruments, use across studies, and pragmatism ( Emmons, Weiner, Fernandez, & Tu, 2012 ; Lewis, Fischer, et al., 2015 ; Lewis, Proctor, & Brownson, 2017 ; Martinez, Lewis, & Weiner, 2014 ; Rabin et al., 2016 ).

The RE-AIM website also includes various means of operationalizing the components of this evaluation framework ( http://www.re-aim.org/resources-and-tools/measures-and-checklists/ ) and recent reviews of the use of RE-AIM are also helpful when planning a quantitative evaluation ( Gaglio et al., 2013 ; Glasgow et al., 2019 ). Additionally, the Grid-Enabled Measures Database (GEM), hosted by the National Cancer Institute, has an ever-growing list of implementation-related measures (130 as of July, 2019) with a general rating by users ( https://www.gem-measures.org/public/wsmeasures.aspx?cat=8&aid=1&wid=11 ). Last, Rabin et al. (2016) provide an environmental scan of resources for measures in implementation and dissemination science.

5. Pragmatism: Reducing Measurement Burden

An emphasis in the field has been on finding ways to reduce the measurement burden on implementers, and to a lesser extent on implementation researchers to reduce costs and increase the pace of dissemination ( Glasgow et al., 2019 ; Glasgow & Riley, 2013 ). Powell et al. (2017) established criteria for pragmatic measures that resulted in four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful; next steps are to develop consensus regarding the most important criteria and developing quantifiable rating criteria for assessing implementation measures on their pragmatism. Advancements have occurred using technology for the evaluation of implementation ( Brown et al., 2015 ). For example, automated and unobtrusive implementation measures can greatly reduce stakeholder burden and increase response rates. As an example, our group ( Wang et al., 2016 ) conducted a proof-of-concept demonstrating the use text analysis to automatically classify the completion of implementation activities using communication logs between implementer and implementing agency. As mentioned earlier in this article, researchers have begun to automate the assessment of implementation fidelity to such evidence-based interventions as motivational interviewing (e.g., Imel et al., 2019 ; Xiao, Imel, Georgiou, Atkins, & Narayanan, 2015 ), and this work is expanding to other intervention protocols to aid in implementation quality ( Smith et al., 2018 ).

6. Example of a Quantitative Evaluation of an Implementation Research Study

We now present the quantitative evaluation plan for an ongoing hybrid type II effectiveness-implementation trial (see Landes et al., this issue ) examining the effectiveness and implementation of the Collaborative Care Model (CCM; Unützer et al., 2002 ) for the management of depression in adult primary care clinics of Northwestern Medicine (Principal Investigator: Smith). CCM is a structure for population-based management of depression involving the primary care provider, a behavioral care manager, and a consulting psychiatrist. A meta-analysis of 79 randomized trials (n=24,308), concluded that CCM is more effective than standard care for short- and long-term treatment of depression ( Archer et al., 2012 ). CCM has also been shown to provide good economic value ( Jacob et al., 2012 ).

Our study involves 11 primary care practices in a rollout implementation design (see Figure 2 ). Randomization in roll-out designs occurs by start time of the implementation strategy, and ensures confidence in the results of the evaluation because known and unknown biases are equally distributed in the case and control groups ( Grimshaw et al., 2000 ). Rollout trials are both powerful and practical as many organizations feel it is unethical to withhold effective interventions, and roll-out designs reduce the logistic and resource demands of delivering the strategy to all units simultaneously. The co-primary aims of the study concern the effectiveness of CCM and its implementation, respectively: 1) Test the effectiveness of CCM to improve depression symptomatology and access to psychiatric services within the primary care environment; and 2) Evaluate the impact of our strategy package on the progressive improvement in speed and quantity of CCM implementation over successive clinics. We will use training and educational implementation strategies, provided to primary care providers, support staff (e.g., nurses, medical assistants), and to practice and system leadership, as well as monitoring and feedback to the practices. Figure 3 summarizes the quantitative evaluation being conducted in this trial using the RE-AIM framework.

An external file that holds a picture, illustration, etc.
Object name is nihms-1539194-f0002.jpg

Design and Timeline of Randomized Rollout Implementation Trial of CCM

Note. CCM = Collaborative Care Model. Clinics will have a staggered start every 3–4 months randomized using a matching scheme. Pre-implementation assessment period is 4 months. Evaluation of CCM implementation will be a minimum of 24 months at each clinic.

An external file that holds a picture, illustration, etc.
Object name is nihms-1539194-f0003.jpg

Summative Evaluation Metrics of CCM Implementation Using the RE-AIM Framework

Note. CCM = Collaborative Care Model. EHR = electronic health record.

7.1. EHR and other administrative data sources

As this is a Type 2effectiveness-implementation hybrid trial, Aim 1 encompasses both reach —an implementation outcome—of depression management by CCM within primary care—and the effectiveness of CCM at improving patient and service outcomes. Within RE-AIM, the Public Health Impact metric is effectiveness (effect size) multiplied by reach rate. EHR and administrative data are being used to evaluate the primary implementation outcomes of reach (i.e., the proportion of patients in the practice who are eligible for CCM and who are referred). The reach rates achieved after implementation of CCM can be compared to rates of mental health contact for patients with depression prior to implementation as well as to that achieved by other CCM implementation evaluations in the literature.

The primary effectiveness outcome of CCM is the reduction of patients’ depression symptom severity. De-identified longitudinal patient outcome data from the EHR—principally depression diagnosis and scores on the PHQ-9 ( Kroenke, Spitzer, & Williams, 2001 )—will be analyzed to evaluate the impact of CCM. Other indicators of the effectiveness of CCM will be evaluated as well but are not discussed here as they are likely to be familiar to most readers with knowledge of clinical trials. Service outcomes, from the Institute of Medicine’s Standards of Care ( Institute of Medicine Committee on Crossing the Quality Chasm, 2006 ), centered on providing care that is effective (providing services based on scientific knowledge to all who could benefit and refraining from providing services to those not likely to benefit), timely (reducing waits and sometimes harmful delays for both those who receive and those who give care), and equitable (providing care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status). We also sought to provide care that is safe, patient-centered, and efficient.

EHR data will also be used to determine adoption of CCM (i.e., the number of providers with eligible patients who refer to CCM). This can be accomplished by tracking patient screening results and intakes completed by the CCM behavioral care manager within the primary care clinician’s encounter record.

7.2. Speed and quantity of implementation

Achievement of Aim 2 requires an evaluation approach and an appropriate trial design to obtain results that can contribute to generalizable knowledge. A rigorous rollout implementation trial design, with matched-pair randomization to when the practice would change from usual care to CCM was devised. Figure 2 provides a schematic of the design with the timing of the crossover from standard practice to CCM implementation. The first thing one will notice about the design is that the sequential nature of the rollout in which implementation at earlier sites precedes the onset of implementation in later sites. This suggests the potential to learn from successes and challenges to improve implementation efficiency (speed) over time. We will use the Universal SIC® ( Saldana, Schaper, Campbell, & Chapman, 2015 ), a date-based, observational measure, to capture the speed of implementation of various activities needed to successfully implement CCM, such as “establishing a workflow”, “preparing for training”, and “behavioral care manager hired.” This measure is completed by practice staff and members of the implementation team based on their direct knowledge of precisely when the activity was completed. Using the completion date of each activity, we will analyze the time elapsed in each practice to complete each stage (Duration Score). Then, we will calculate the percentage of stages completed (Proportion Score). These scores can then be used in statistical analyses to understand the factors that contributed to timely stage completion, the number of stages that are important for successful program implementation by relating the SIC to other implementation outcomes, such as reach rate; and simply whether there was a degree of improvement in implementation efficiency and scale as the rollout took place. That is, were more stages completed more quickly by later sites compared to earlier ones in the rollout schedule. This analysis comprises the implementation domain of RE-AIM. It will be used in combination with other metrics from the EHR to determine the fidelity of implementation, which is consistent with RE-AIM.

7.3. Surveys

To understand the process and the determinants of implementation—the factors that impede or promote adoption and delivery with fidelity—a battery of surveys was administered at multiple time-points to key staff members in each practice. One challenge with large-scale implementation research is the need for measures to be both psychometrically sound as well as pragmatic. With this in mind, we adapted a set of questions for the current trial that were developed and validated in prior studies. This low-burden assessment is comprised of items from four validated implementation surveys concerning factors at the inner setting of the organization: the Implementation Leadership Scale ( Aarons, Ehrhart, & Farahnak, 2014 ), the Evidence-Based Practice Attitude Scale ( Aarons, 2004 ), the Clinical Effectiveness and Evidence-Based Practice Questionnaire ( Upton & Upton, 2006 ), and the Organizational Change Recipient’s Belief Scale ( Armenakis, Bernerth, Pitts, & Walker, 2007 ). In a prior study, we used confirmatory factor analysis to evaluate the four scales after shortening for pragmatism and tailoring the wording of the items (when appropriate) to the context under investigation in the study (Smith et al., under review). Further, different versions of the survey were created for administration to the various professional roles in the organization. Results showed that the scales were largely replicated after shortening and tailoring; internal consistencies were acceptable; and the factor structures were statistically invariant across professional role groups. The same process was undertaken for this study with versions of the battery developed for providers, practice leadership, support staff, and the behavioral care managers. The survey was administered immediately after initial training in the model and then again at 4, 12, and 24 months. Items were added after the baseline survey regarding the process of implementation thus far and the most prominent barriers and facilitators to implementation of CCM in the practice. Survey-based evaluation of maintenance in RE-AIM, also called sustainability, will occur via the Clinical Sustainability Assessment Tool ( Luke, Malone, Prewitt, Hackett, & Lin, 2018 ) to key decision makers at multiple levels in the healthcare system.

7.4. Cost of implementation

The costs incurred when adopting and delivering a new clinical intervention are a top reason attributed to lack of adoption of behavioral interventions ( Glasgow & Emmons, 2007 ). While cost-effectiveness and cost-benefit analyses demonstrate the long-term economic benefits associated with the effects of these interventions, they rarely consider the costs to the implementer associated with these endeavors as a unique component ( Ritzwoller, Sukhanova, Gaglio, & Glasgow, 2009 ). As such, decision makers value different kinds of economic evaluations, such as budget impact analysis, which assesses the expected short-term changes in expenditures for a health care organization or system in adopting a new intervention ( Jordan, Graham, Berkel, & Smith, 2019 ), and cost-effectiveness analysis from the perspective of the implementing system and not simply the individual recipient of the evidence-based intervention being implemented ( Raghavan, 2017 ). Eisman and colleagues ( this issue ) discuss economic evaluations in implementation research.

In our study, our economic approach focuses on the cost to Northwestern Medicine to deliver CCM and will incorporate reimbursement from payors to ensure that the costs to the system are recouped in such a way that it can be sustained over time under current models of compensated care. The cost-effectiveness of CCM has been established ( Jacob et al., 2012 ), but we will also quantify the cost of achieving salient health outcomes for the patients involved, such as cost to achieve remission as well as projected costs that would increase remission rates.

7. Conclusions

The field of implementation research has developed methods for conducting quantitative evaluation to summarize the overall, aggregate impact of implementation strategies on salient outcomes. Methods are still emerging to aid researchers in the specification and planning of evaluations for implementation studies (e.g., Smith, 2018 ). However, as noted in the case example, evaluations focused only on the aggregate results of a study should not be done in the absence of ongoing formative evaluations, such as in-protocol audit and feedback and other methods (see Elwy et al., this issue ),and mixed and/or qualitative methods (see Hamilton et al., this issue ). Both of which are critical for interpreting the results of evaluations that aggregate the results of a large trial and gaging the generalizability of the findings. In large part, the intent of quantitative evaluations of large trials in implementation research aligns with its clinical-level counterparts, but with the emphasis on the factors in the service delivery system associated with adoption and delivery of the clinical intervention rather than on the direct recipients of that intervention (see Figure 1 ). The case example shows how both can be accomplished in an effectiveness-implementation hybrid design (see Landes et al., this issue ). This article shows current thinking on quantitative outcome evaluation in the context of implementation research. Given the quickly-evolving nature of the field of implementation research, it is imperative for interested readers to consult the most up-to-date resources for guidance on quantitative evaluation.

  • Quantitative evaluation can be conducted in the context of implementation research to determine impact of various strategies on salient outcomes.
  • The defining characteristics of implementation research studies are discussed.
  • Quantitative evaluation frameworks and measures for key implementation research outcomes are presented.
  • Application is illustrated using a case example of implementing collaborative care for depression in primary care practices in a large healthcare system.

Acknowledgements

The authors wish to thank Hendricks Brown who provided input on the development of this article and to the members of the Collaborative Behavioral Health Program research team at Northwestern: Lisa J. Rosenthal, Jeffrey Rado, Grace Garcia, Jacob Atlas, Michael Malcolm, Emily Fu, Inger Burnett-Zeigler, C. Hendricks Brown, and John Csernansky. We also wish to thank the Woman’s Board of Northwestern Memorial Hospital, who generously provided a grant to support and evaluate the implementation and effectiveness of this model of care as it was introduced to the Northwestern Medicine system, and our clinical, operations, and quality partners in Northwestern Medicine’s Central Region.

This study was supported by a grant from the Woman’s Board of Northwestern Memorial Hospital and grant P30DA027828 from the National Institute on Drug Abuse, awarded to C. Hendricks Brown. The opinions expressed herein are the views of the authors and do not necessarily reflect the official policy or position of the Woman’s Board, Northwestern Medicine, the National Institute on Drug Abuse, or any other part of the US Department of Health and Human Services.

List of Abbreviations

CCMcollaborative care model
HERelectronic health record

Competing interests

None declared.

Declarations

Ethics approval and consent to participate

Not applicable. This study did not involve human subjects.

Availability of data and material

Not applicable.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

  • Aarons GA (2004). Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS) . Ment Health Serv Res , 6 ( 2 ), 61–74. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Aarons GA, Ehrhart MG, & Farahnak LR (2014). The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership . Implementation Science , 9 . doi: 10.1186/1748-5908-9-45 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Archer J, Bower P, Gilbody S, Lovell K, Richards D, Gask L,… Coventry P (2012). Collaborative care for depression and anxiety problems . Cochrane Database of Systematic Reviews(10) doi: 10.1002/14651858.CD006525.pub2 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Armenakis AA, Bernerth JB, Pitts JP, & Walker HJ (2007). Organizational Change Recipients Beliefs Scale: Development of an Assessmetn Instrument . The Journal of Applied Behavioral Science , 42 , 481–505. doi:DOI: 10.1177/0021886307303654 [ CrossRef ] [ Google Scholar ]
  • Bauer MS, Damschroder L, Hagedorn H, Smith J, & Kilbourne AM (2015). An introduction to implementation science for the non-specialist . BMC Psychology , 3 ( 1 ), 32. doi: 10.1186/s40359-015-0089-9 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Bertrand JT, Holtgrave DR, & Gregowski A (2009). Evaluating HIV/AIDS programs in the US and developing countries In Mayer KH & Pizer HF (Eds.), HIV Prevention (pp. 571–590). San Diego: Academic Press. [ Google Scholar ]
  • Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L,… Cruden G (2017). An overview of research and evaluation designs for dissemination and implementation . Annual Review of Public Health , 38 ( 1 ), null. doi:doi: 10.1146/annurev-publhealth-031816-044215 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Brown CH, Mohr DC, Gallo CG, Mader C, Palinkas L, Wingood G,… Poduska J (2013). A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs . J Acquir Immune Defic Syndr , 63 . doi: 10.1097/QAI.0b013e31829372bd [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Brown CH, PoVey C, Hjorth A, Gallo CG, Wilensky U, & Villamar J (2015). Computational and technical approaches to improve the implementation of prevention programs . Implementation Science , 10 ( Suppl 1 ), A28. doi: 10.1186/1748-5908-10-S1-A28 [ CrossRef ] [ Google Scholar ]
  • Cheung K, & Duan N (2014). Design of implementation studies for quality improvement programs: An effectiveness-cost-effectiveness framework . American Journal of Public Health , 104 ( 1 ), e23–e30. doi: 10.2105/ajph.2013.301579 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Colditz GA, & Emmons KM (2017). The promise and challenges of dissemination and implementation research In Brownson RC, Colditz GA, & Proctor EK (Eds.), Dissemination and implementation research in health: Translating science to practice . New York, NY: Oxford University Press. [ Google Scholar ]
  • Curran GM, Bauer M, Mittman B, Pyne JM, & Stetler C (2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact . Medical Care , 50 ( 3 ), 217–226. doi: 10.1097/MLR.0b013e3182408812 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Department of Health and Human Services. (2019). PAR-19–274: Dissemination and Implementation Research in Health (R01 Clinical Trial Optional) . Retrieved from https://grants.nih.gov/grants/guide/pa-files/PAR-19-274.html
  • Emmons KM, Weiner B, Fernandez ME, & Tu S (2012). Systems antecedents for dissemination and implementation: a review and analysis of measures . Health Educ Behav , 39 . doi: 10.1177/1090198111409748 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Gaglio B, & Glasgow RE (2017). Evaluation approaches for dissemination and implementation research In Brownson R, Colditz G, & Proctor E (Eds.), Dissemination and Implementation Research in Health: Translating Science into Practice (2nd ed., pp. 317–334). New York: Oxford University Press. [ Google Scholar ]
  • Gaglio B, Shoup JA, & Glasgow RE (2013). The RE-AIM Framework: A systematic review of use over time . American Journal of Public Health , 103 ( 6 ), e38–e46. doi: 10.2105/ajph.2013.301299 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Glasgow RE, & Emmons KM (2007). How can we increase translation of research into practice? Types of evidence needed . Annual Review of Public Health , 28 , 413–433. [ PubMed ] [ Google Scholar ]
  • Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC,… Estabrooks PA (2019). RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review . Frontiers in Public Health , 7 ( 64 ). doi: 10.3389/fpubh.2019.00064 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, & Vogt TM (2006). Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues . Health Education Research , 21 ( 5 ), 688–694. doi: 10.1093/her/cyl081 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Glasgow RE, & Riley WT (2013). Pragmatic measures: what they are and why we need them . Am J Prev Med , 45 . doi: 10.1016/j.amepre.2013.03.010 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Glasgow RE, Vogt TM, & Boles SM (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework . American Journal of Public Health , 89 ( 9 ), 1322–1327. doi: 10.2105/AJPH.89.9.1322 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Grimshaw J, Campbell M, Eccles M, & Steen N (2000). Experimental and quasi-experimental designs for evaluating guideline implementation strategies . Family practice , 17 Suppl 1 , S11–16. doi: 10.1093/fampra/17.suppl_1.s11 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Imel ZE, Pace BT, Soma CS, Tanana M, Hirsch T, Gibson J,… Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing . Psychotherapy , 56 ( 2 ), 318–328. doi: 10.1037/pst0000221 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Institute of Medicine Committee on Crossing the Quality Chasm. (2006). Adaption to mental health and addictive disorder: Improving the quality of health care for mental and substanceuse conditions . Retrieved from Washington, D.C.: [ Google Scholar ]
  • Jacob V, Chattopadhyay SK, Sipe TA, Thota AB, Byard GJ, & Chapman DP (2012). Economics of collaborative care for management of depressive disorders: A community guide systematic review . American Journal of Preventive Medicine , 42 ( 5 ), 539–549. doi: 10.1016/j.amepre.2012.01.011 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Jordan N, Graham AK, Berkel C, & Smith JD (2019). Budget impact analysis of preparing to implement the Family Check-Up 4 Health in primary care to reduce pediatric obesity . Prevention Science , 20 ( 5 ), 655–664. doi: 10.1007/s11121-018-0970-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Kroenke K, Spitzer R, & Williams JW (2001). The PHQ-9 . Journal of General Internal Medicine , 16 ( 9 ), 606–613. doi: 10.1046/j.1525-1497.2001.016009606.x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Landsverk J, Brown CH, Smith JD, Chamberlain P, Palinkas LA, Ogihara M,… Horwitz SM (2017). Design and analysis in dissemination and implementation research In Brownson RC, Colditz GA, & Proctor EK (Eds.), Dissemination and implementation research in health: Translating research to practice (2nd ed., pp. 201–227). New York: Oxford University Press. [ Google Scholar ]
  • Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, & Martinez RG (2015). Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria . Implementation Science , 10 ( 1 ), 155. doi: 10.1186/s13012-015-0342-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Lewis CC, Proctor EK, & Brownson RC (2017). Measurement issues in dissemination and implementation research In Brownson RC, Colditz GA, & Proctor EK (Eds.), Dissemination and implementation research in health: Translating research to practice (2nd ed., pp. 229–244). New York: Oxford University Press. [ Google Scholar ]
  • Lewis CC, Stanick CF, Martinez RG, Weiner BJ, Kim M, Barwick M, & Comtois KA (2015). The Society for Implementation Research Collaboration Instrument Review Project: A methodology to promote rigorous evaluation . Implementation Science , 10 ( 1 ), 2. doi: 10.1186/s13012-014-0193-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Luke DA, Calhoun A, Robichaux CB, Elliott MB, & Moreland-Russell S (2014). The Program Sustainability Assessment Tool: A new instrument for public health programs . Preventing Chronic Disease , 11 , E12. doi: 10.5888/pcd11.130184 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Luke DA, Malone S, Prewitt K, Hackett R, & Lin J (2018). The Clinical Sustainability Assessment Tool (CSAT): Assessing sustainability in clinical medicine settings . Paper presented at the Conference on the Science of Dissemination and Implementation in Health, Washington, DC. [ Google Scholar ]
  • Martinez RG, Lewis CC, & Weiner BJ (2014). Instrumentation issues in implementation science . Implement Sci , 9 . doi: 10.1186/s13012-014-0118-8 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA,… Lewis CC (2017). Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping . Implementation Science , 12 ( 1 ), 118. doi: 10.1186/s13012-017-0649-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM,… Kirchner JE (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project . Implement Sci , 10 . doi: 10.1186/s13012-015-0209-1 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A,… Hensley M (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda . Adm Policy Ment Health Ment Health Serv Res , 38 . doi: 10.1007/s10488-010-0319-7 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN,… Glasgow RE (2016). Measurement resources for dissemination and implementation research in health . Implementation Science , 11 ( 1 ), 42. doi: 10.1186/s13012-016-0401-y [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Raghavan R (2017). The role of economic evaluation in dissemination and implementation research In Brownson RC, Colditz GA, & Proctor EK (Eds.), Dissemination and implementation research in health: Translating science to practice (2nd ed.). New York: Oxford University Press. [ Google Scholar ]
  • Ritzwoller DP, Sukhanova A, Gaglio B, & Glasgow RE (2009). Costing behavioral interventions: a practical guide to enhance translation . Annals of Behavioral Medicine , 37 ( 2 ), 218–227. [ PubMed ] [ Google Scholar ]
  • Saldana L, Schaper H, Campbell M, & Chapman J (2015). Standardized Measurement of Implementation: The Universal SIC . Implementation Science , 10 ( 1 ), A73. doi: 10.1186/1748-5908-10-s1-a73 [ CrossRef ] [ Google Scholar ]
  • Schoenwald S, Garland A, Chapman J, Frazier S, Sheidow A, & Southam-Gerow M (2011). Toward the effective and efficient measurement of implementation fidelity . Admin Policy Mental Health Mental Health Serv Res , 38 . doi: 10.1007/s10488-010-0321-0 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Smith JD (2018). An implementation research logic model: A step toward improving scientific rigor, transparency, reproducibility, and specification . Implementation Science , 14 ( Supp 1 ), S39. [ Google Scholar ]
  • Smith JD, Berkel C, Jordan N, Atkins DC, Narayanan SS, Gallo C,… Bruening MM (2018). An individually tailored family-centered intervention for pediatric obesity in primary care: Study protocol of a randomized type II hybrid implementation-effectiveness trial (Raising Healthy Children study) . Implementation Science , 13 ( 11 ), 1–15. doi: 10.1186/s13012-017-0697-2 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Smith JD, Li DH, Hirschhorn LR, Gallo C, McNulty M, Phillips GI,… Benbow ND (2019). Landscape of HIV implementation research funded by the National Institutes of Health: A mapping review of project abstracts (submitted for publication) . [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Smith JD, Rafferty MR, Heinemann AW, Meachum MK, Villamar JA, Lieber RL, & Brown CH (under review). Evaluation of the factor structure of implementation research measures adapted for a novel context and multiple professional roles . [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H,… Smith JL (2006). The role of formative evaluation in implementation research and the QUERI experience . Journal of General Internal Medicine , 21 ( 2 ), S1. doi : 10.1007/s11606-006-0267-9 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Tabak RG, Khoong EC, Chambers DA, & Brownson RC (2012). Bridging research and practice: Models for dissemination and implementation research . American Journal of Preventive Medicine , 43 ( 3 ), 337–350. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • The Health Foundation. (2015). Evaluation: What to consider. Commonly asked questions about how to approach evaluation of quality improvement in health care . Retrieved from London, England: https://www.health.org.uk/sites/default/files/EvaluationWhatToConsider.pdf [ Google Scholar ]
  • Unützer J, Katon W, Callahan CM, Williams J, John W, Hunkeler E, Harpole L,… Investigators, f. t. I. (2002). Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial . JAMA , 288 ( 22 ), 2836–2845. doi: 10.1001/jama.288.22.2836 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Upton D, & Upton P (2006). Development of an evidence-based practice questionnaire for nurses . Journal of Advanced Nursing , 53 ( 4 ), 454–458. [ PubMed ] [ Google Scholar ]
  • Wang D, Ogihara M, Gallo C, Villamar JA, Smith JD, Vermeer W,… Brown CH (2016). Automatic classification of communication logs into implementation stages via text analysis . Implementation Science , 11 ( 1 ), 1–14. doi: 10.1186/s13012-016-0483-6 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Wyman PA, Henry D, Knoblauch S, & Brown CH (2015). Designs for testing group-based interventions with limited numbers of social units: The dynamic wait-listed and regression point displacement designs . Prevention Science , 16 ( 7 ), 956–966. doi : 10.1007/s11121-014-0535-6 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Xiao B, Imel ZE, Georgiou PG, Atkins DC, & Narayanan SS (2015). “Rate My Therapist”: Automated detection of empathy in drug and alcohol counseling via speech and language processing . PLoS ONE , 10 ( 12 ), e0143055. doi: 10.1371/journal.pone.0143055 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Key disparities between quantitative and qualitative research methodologies

Mohanasundari Kuppuswamy at All India Institute of Medical Sciences bibinagar

  • All India Institute of Medical Sciences bibinagar
  • This person is not on ResearchGate, or hasn't claimed this research yet.

Discover the world's research

  • 25+ million members
  • 160+ million publication pages
  • 2.3+ billion citations
  • Edward Barroga
  • Glafera Janet Matanguihan
  • Atsuko Furuta

Mohanasundari Kuppuswamy

  • Mayuri Mahendran

Alma Angélica Villa-Rueda

  • Rural Rem Health

Belinda O'Sullivan

  • Tiana Gurney
  • AM J PHARM EDUC

Jessica L Johnson

  • Donna Adkins

Sheila W Chauvin

  • REV SCI TECH OIE
  • K.B. STEVENS
  • Recruit researchers
  • Join for free
  • Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Columbia College

Quantitative research methods - psyc/soci 225 research guide.

  • Find Articles
  • PsycARTICLES Tips

What Is Peer Review?

Words of caution, how can i limit my searches to peer-reviewed journals, how can i know if a particular journal is peer reviewed, why is peer review important, to learn more about peer review, acknowledgement.

Peer review is the process by which an article is screened and evaluated by a panel of experts before it is published. 

The reviewers will evaluate the article for:

  • Credibility.
  • Clear writing.
  • New contributions to knowledge.
  • Sound methodology.
  • Conclusions supported by the data. 

Peer-reviewed journals are also known as refereed or juried journals. 

Peer-reviewed journals do contain some information that is itself not refereed, such as editorials, opinions, letters, and book reviews.  So, you need to consider whether the individual article or essay has been reviewed and approved by other scholars.

Many databases have an option for limiting to peer-reviewed journals.  Check the limit options on the advanced search screen of a database to see if you can limit your search to articles that have been published in peer-reviewed journals.

Look for options such as

  • Peer Reviewed.
  • Academic or scholarly journals.  (Be advised that journals can be academic or scholarly without being peer reviewed.)

The Psychology & Behavioral Sciences Collection and SocINDEX with Full Text d atabases each has a publications section in which you can look up a journal title.  The last field in the description of a journal tells you if it is peer reviewed or not. To do this:

  • Access the database.
  • Click on Publications in the blue banner near the top.
  • Type the journal title in the lower of the two boxes and click on Browse .
  • Look at the Peer Reviewed field, the last line of the publication record.

If a journal is not listed in either of those two databases, try these options:

  • Access the PsycARTICLES database's publications section.  If a journal title is included in it, the journal is peer reviewed.  All journals in PsycARTICLES are peer reviewed.
  • Look at the website of the journal for descriptive information about the publication or for instructions for authors interested in submitting a manuscript.  If the journal is peer reviewed, that will be stated.  Another website to check is that of the publisher of the journal.
  • Look for this same type of information in the first few pages of the journal itself.  The front or back cover may also indicate the publication is peer reviewed.

It helps ensures that published articles are of high quality and that new knowledge is being created.

As a student, you can be confident that you are using authoritative sources.

Watch these two videos produced by other libraries:

  • The Peer Review Process A video from the Western University Libraries. A written copy of the script's text is provided.
  • Peer Review in Five Minutes A video created by the North Carolina State University Libraries.

This page has been adapted from the "Understand Peer Review" guide created by the library staff of Red Deer College (RDC) in Alberta, Canada.

We appreciate receiving permission from the RDC librarians to use and adapt their guide.

  • << Previous: PsycARTICLES Tips
  • Last Updated: Jul 7, 2024 7:25 PM
  • URL: https://library.ccis.edu/pscy325
  • Open access
  • Published: 03 July 2024

Exploring medical students' experience of the learning environment: a mixed methods study in Saudi medical college

  • Mohammed Almansour 1 ,
  • Noura Abouammoh 2 ,
  • Reem Bin Idris 3 ,
  • Omar Abdullatif Alsuliman 3 ,
  • Renad Abdulrahman Alhomaidi 3 ,
  • Mohammed Hamad Alhumud 3 &
  • Hani A. Alghamdi 2  

BMC Medical Education volume  24 , Article number:  723 ( 2024 ) Cite this article

384 Accesses

5 Altmetric

Metrics details

In medical education, the learning environment (LE) significantly impacts students' professionalism and academic performance. Positive LE perceptions are linked to better academic outcomes. Our study, which was conducted 15 years after curriculum reform at King Saud University's College of Medicine, aimed to explore students' perspectives on their LE and identify areas for improvement. By understanding their experiences, we strive to enhance LE and promote academic success.

This mixed-method study employed an explanatory sequential approach in which a cross-sectional analytical survey phase was collected first using the Johns Hopkins Learning Environment Scale (JHLES), followed by qualitative focus groups. Findings from quantitative and qualitative methods were integrated using joint display.

A total of 653 medical students completed the JHLES. The total average score was 81 out of 140 (16.8), and the average subscale scores ranged from 2.27 (0.95) for inclusion and safety to 3.37 (0.91) for community of peers. The qualitative approach encompasses both inductive and deductive analyses, identifying overarching themes comprising proudness, high expectations and competition, and views about the curriculum. The integration of results emphasizes the need for continued efforts to create a supportive and inclusive LE that positively influences students' experiences and academic success.

This research offers valuable insights for educational institutions seeking to enhance medical education quality and support systems. Recommendations include faculty development, the cultivation of supportive environments, curriculum revision, improved mentorship programs, and initiatives to promote inclusivity and gender equity. Future research should explore longitudinal and comparative studies, innovative mixed methods approaches, and interventions to further optimize medical education experiences. Overall, this study contributes to the ongoing dialog on medical education, offering a nuanced understanding of the complex factors influencing students' perceptions and suggesting actionable strategies for improvement.

Peer Review reports

The learning environment of medical students plays a significant role in shaping qualified, well-rounded physicians. It can also impact students' professionalism, ethics, and morals. As these students graduate and begin their professional practice, their competency can be a direct reflection of the medical institutes from which they graduated. The learning environment (LE) is a term used to describe the physical, cultural, and psychosocial climate in which learning takes place [ 1 ]. Students' skills, knowledge, and attitudes are influenced by the teaching and learning environment of their educational institutes. The interactions they have with their peers, faculty members, and administrators play a role in their learning environment. The curriculum that is taught to students is part of this environment, and the curriculum's design is a vital component [ 2 ].

The impact of LE on the academic performance of medical students is significant. Therefore, it is crucial to provide a supportive environment that positively influences students' perceptions of their LE. Research has consistently shown that students who perceive their LE to be positive and supportive are more likely to perform well academically [ 3 ]. Conversely, students who perceive their LE to be negative may experience adverse effects on their academic performance [ 3 ].

A student-centered curriculum of outstanding standards must be provided, and evaluation of the educational setting at both academic and clinical sites is essential [ 4 ]. King Saud University's College of Medicine program is seven years long, starting with a preparatory year, followed by two basic sciences (preclinical) years, then three clinical-practice years, and a one-year internship. The program employs a combination of problem-based learning and interactive lecturing to teach medical and healthcare-related sciences, emphasizing critical thinking and self-directed learning. Clinical training programs provide hands-on experience, with the goal of producing skilled and compassionate healthcare professionals.

Two studies were conducted at the College of Medicine at King Saud University (COM-KSU). The first study was conducted in 2008, prior to the college's curriculum reform in 2009, which transitioned from a traditional to a system-oriented hybrid curriculum [ 5 ]. Researchers utilized the Dundee Ready Educational Environment Measure (DREEM) scale to evaluate the learning environment (LE), and the results indicated that first-year students had significantly higher scores than other students [ 5 ]. Additionally, preclinical students had significantly greater scores than did clinical students, and gender was not a statistically significant factor [ 5 ].

The second study was conducted in 2014, where fifth-year medical students were evaluated using the DREEM scale to assess their perception of the LE [ 6 ]. The study revealed that the students' perception of the educational environment was satisfactory [ 6 ].

The Johns Hopkins Learning Environment Scale (JHLES) was created by the Johns Hopkins University School of Medicine to evaluate the quality of the learning environment for residents and medical students [ 7 ]. The 28-item scale helps medical educators identify areas of improvement by assessing seven factors or subscales, comprising community of peers, faculty relationships, academic climate, meaningful engagement, mentoring, inclusion and safety, and physical space [ 7 ].

The aim of our study was to investigate the perceptions of medical students regarding their LE at the COM-KSU 15 years after the curriculum was reformed. We seek to understand the experiences of students in this particular LE and gain insights into the factors that influence their perceptions of the LE. By exploring the students' perspectives, we aim to identify areas where improvements can be made to enhance LE and ensure that it is conducive to learning and promotes academic success.

Aim, design, and setting

This mixed-method study aimed to investigate students’ perceptions of the LE at COM-KSU 15 year proceeding a curriculum change, followed by an exploration of their perspectives aiming to identify areas of improvement of the LE. This study employed an explanatory sequential approach in which a cross-sectional analytical survey phase collected first, followed by qualitative focus groups. The research was carried out between November 2022 and March 2023 within the College of Medicine at King Saud University (COM-KSU), which is the pioneering medical education institution in the Kingdom of Saudi Arabia and is located in the capital city of Riyadh.

Participants and sampling

All the COM-KSU undergraduate students and interns were invited to participate in the study, with a total of 1471 students and 268 interns. The total number of enumeration techniques over the period of the study was used. Convenient sampling was employed in this study. The decision to employ convenient sampling was based on practical considerations of the accessibility and availability of participants. Consequently, a total of 653 individuals voluntarily participated in the first phase of the study, and the research team initiated the participant recruitment process by extending invitations to all undergraduate students and interns enrolled in the COM-KSU. The invitations were disseminated via multiple channels, including email, WhatsApp groups, and personal visits to each classroom within the college.

The data collection process comprised two distinct online surveys, each serving a specific purpose. The first survey focused on the quantitative phase and included questions related to demographic information and the Johns Hopkins Learning Environment Scale (JHLES). The second survey, designed for registration in the qualitative phase, included demographic inquiries along with a means of contact and the provision of available time slots. Subsequently, the research team communicated with the registered participants and arranged for focused group discussions (FGDs) to be conducted. Two FGDs were needed (5 and 7 participants) based on the theory of data saturation. Each FGD lasted approximately 70 min and was held at the College of Medicine. The discussions were facilitated by one of the authors, who is a qualitative methodologist and a faculty member at the same college, and the participants were comfortable discussing negative views as they were discussing positive views.

In the quantitative study phase, an online survey encompassing various components was developed. This survey collected demographic data, including information on gender, age, academic year, GPA, employment status, marital status, and residence type. Additionally, the Johns Hopkins Learning Environment Scale (JHLES), a validated tool used for assessing undergraduate medical school learning environments, was used. The JHLES consists of 28 items distributed across seven domains, and its use for this study was conducted without the need for direct permission, as it is publicly available.

In the qualitative study phase, students and interns were actively engaged in Focus Group Discussions (FGDs), aimed at eliciting their perspectives on the learning environment (LE). The FGDs employed a topic guide comprising open-ended questions aligned with the LE domains delineated by the JHLES. These questions included inquiries such as "How would you characterize your relationships with your peers?" and "To what extent does the college environment support collaboration with fellow students from the same college?" Furthermore, participants were asked to share their opinions regarding the faculty and provide insights into their perceptions of the curriculum. The FGDs were complemented with probing questions and follow-up queries to delve deeper into participants' experiences and perspectives.

Statistical analysis

For the first phase in this study, sociodemographic data were presented using descriptive statistics. The mean and standard deviation (SD) for the total score and the seven domains of the JHLES were calculated. Cross-tabulation was used to explore the relationships between the JHLES scores and the sociodemographic variables, and tests of significance through chi-square tests and ANOVA were performed. All analyses were performed using R (version 4.2.2), [ 8 ].

Qualitative data collection

The questions in the topic guide included probing questions and encompassed domains and questions from the JHLES. As open-ended questions were used to collect data, themes included deductive and inductive analysis. Inductive analysis was based on a priori themes based on the JHLES domains.

Qualitative analysis

Thematic analysis was adopted for qualitative analysis. This approach was proposed by Ritchie and Spencer (1994) to be helpful in providing a sequential structure for data analysis. This was conducted using NVivo software version 11.4.2. Using software increases the efficiency of data organization and retrieval. Familiarization, descriptive coding, basic analysis, and interpretation are the steps followed in the data analysis, and quotes from the participants were used to support the themes. Analyzing the data and identifying common descriptive themes were tasks shared with the team. The team agreed on a coding frame. The analysis was conducted independently, and the results are presented in comparison to the quantitative findings in Table  5 .

Mixed methods integration

Findings from quantitative and qualitative methods were integrated using joint display. The outcomes of the JHLES and FGDs were compared side-by-side. Integrating findings can create a holistic understanding of the learning environment of the College of Medicine, leading to a conclusion where the whole is greater than the sum of its parts.

Joint display of the data provided visual means of presenting qualitative and quantitative findings granting the ability to associate reasoning with different item score. Qualitative findings complement the quantitative findings in providing meaning to the score and explored in students’ perspective reasons for these scores. The qualitative findings also explained how students’ pride and perception about their own status reflect on the JHLES score. Students’ needs and preferences were expressed explicitly during the FGDs.

Ethical considerations

This study was approved by King Saud University’s Institutional Review Board (KSU IRB) with the approval number E-22–7298. Electronic informed consent was obtained from all participants in the quantitative arm, and written informed consent was obtained from all participants in the qualitative arm prior to their participation in the study.

Quantitative results

Sociodemographic characteristics.

Table 1 presents the sociodemographic characteristics of all participants. The total number of medical students and interns included in this phase of the study was 653. Of those studied, there was an almost equal gender distribution, with males making up slightly more than half (59%). There were relatively varied numbers of academic years, with less than average representation coming from the intern level at a participation rate of only 4%, while the highest engagement occurred during fourth-year studies at approximately 26%. Most individuals boasted high academic records, achieving an above-average GPA of 4.50–5.00 (65.7%). Of those who participated, a small fraction had lower grades below a GPA of 4 (11.5%). The majority of the participants were unemployed (96.2%), while less than 4% were either employed (full- or part-time) or freelancers (1.5%). Regarding personal life traits, most of the participants were single (98.5%) and lived with their families residing in Riyadh (93%).

As shown in Table  2 , the overall mean score for student experience was 81 ( SD  = 16.76). Among the specific subscales, the highest mean score was observed for physical space (3.52; SD  = 0.95), and the lowest mean score was found for inclusion and safety (2.27; SD  = 0.95).

Sociodemographic variables and overall and domain scores of the JHLES (mean and SD)

Associations between sociodemographic variables and the different domains of the JHLES as well as the overall score are represented in Table  3 . Male students reported a higher mean overall score than females did (83.4 ± 17.1 and 77.5 ± 15.7, respectively). As the number of academic years increased, the first-year students reported a greater average score than did the senior-year students, with a mean overall score for first-year medical students of 87.6 ( SD  = 16.9), whereas the average score for senior-year students (fifth-year) was 74.8 ± 18.2. Students who possessed higher GPAs (4.50–5.00) achieved the highest mean score of 82.2 ± 16, while those with GPAs less than 4.00 reported the lowest average score of 73.3 ( SD  = 15). Employment status was another variable impacting students' individual perceptions of this survey total score, where employed students generally outperformed unemployed students, with higher scores (88.6 ± 18.6) than unemployed students (80.7 ± 16.7). A significant association was observed between the overall JHELS score and gender, academic year, and GPA at the 0.05 level.

There was a notable difference in scores between males and females, with males reporting higher scores for all domains except “inclusion and safety”. Intriguingly, both genders reflected a similar pattern for reporting the highest score for physical space and the lowest for inclusion and safety. However, four domains showed statistically significant associations at the 0.05 level: peer community, faculty relationships, academic climate, and meaningful engagement.

Among the different academic year levels, first-year students reported the highest score for community of peers (3.64 ± 0.88) and the lowest for inclusion and safety (1.82 ± 0.90). Similarly, second- to fourth-year students reported the highest scores for physical space and the lowest scores for inclusion and safety. As the academic year progressed, fifth-year students and interns obtained the lowest scores in meaningful engagement (2.26 ± 0.94 and 2.18 ± 0.95, respectively), but the highest score was given for physical space among 5th-year students (3.42 ± 0.93) and communities of peers for interns (3.43 ± 0.87). There were statistically significant associations with all subscales except physical space ( P value = 0.33).

Students with high GPA (4.50–5.0) recorded higher results across all domains than did their peers who earned a lower GPA (i.e., less than 4.00), with the exception of inclusion and safety. It is interesting to note that the physical space domain stood out as the highest scorer for all groups, while the scores for inclusion and safety fell short among all groups according to GPA. There was a statistically significant association with the first three domains only, community of peers, faculty relationships, and academic climate.

Employed students reported higher scores on measures related to community engagement reflected in the community of peers (3.70 ± 0.63), while unemployed and freelance students had the highest scores for physical space (3.52 ± 0.95 and 3.65 ± 1.13, respectively). The inclusion and safety subscale scores were the lowest for unemployed and employed students (2.25 ± 0.95 and 2.49 ± 1.20, respectively), while freelancers reported the lowest score for the academic climate subscale (2.36 ± 0.61). Employment status was significantly associated with only the mentoring subscale ( P value = 0.02).

Students who were single attained the highest average score of 3.37 ± 0.91 on the physical space domain, while inclusion and safety presented a challenging component (2.26 ± 0.95). Conversely, those who were married or engaged garnered the highest community of peer ratings, averaging 3.53 ± 0.78, and the lowest for faculty relationships, with a mean value of 2.72 ± 0.92. Students residing with family or in private accommodations, as well as those with families living in Riyadh or outside Riyadh, reported the highest scores in the physical space domain and the lowest scores in inclusion and safety. However, the association was not statistically significant between all groups or across all subscales ( P value > 0.05) .

Qualitative results

Participants of both genders, senior and junior years, represented the FGDs (Table  4 ). One participant was employed, and all were living with their families.

As open-ended questions were used to collect data, themes were derived from deductive and inductive analysis. Inductive analysis was based on a priori themes based on the JHLES domains. Table 5 shows the domains in which participants’ perceptions were compared with the quantitative findings. Some qualitative findings aligned with the quantitative findings, while others contradicted or explained them.

Evaluating the learning environment for medical students is essential for improving their professional standards, knowledge, and skills. This mixed methods study explored medical students' perceptions about the learning environment at the College of Medicine, a well-known university in Saudi Arabia, King Saud University. This study is two-pronged, first, to quantitatively assess students’ perceptions of the COM-KSU learning environment and, second, to qualitatively explore their experience in the same medical school.

Our study yielded an overall average score of 81 out of 140 on the JHLES. Notably, there was no predefined threshold for a passing or positive score on this tool. Compared to the original study where the scale was first used and validated, the average score in our study was lower (107 vs 81, respectively) [ 7 ]. This discrepancy might be related to the original study's single-institute design affects the generalizability of its results, and the differences in student characteristics due to the U.S. requiring a bachelor’s degree for medical school admission, unlike KSA, where students enter directly after high school, play a role. Additionally, the original study did not focus on the "hidden curriculum" influenced by organizational culture and structure, which may explain the discrepancy given the distinct social, organizational, and learning cultures between our context and the American one. However, our results were consistent with those of other studies that were conducted in other medical schools in different countries, including Malaysia, India and Pakistan, ranging from 81.1 to 86 [ 9 , 10 , 11 ].

Two previously published studies in the same setting, COM-KSU (2008 and 2017), utilized the DREEM survey and revealed that medical students reported different average scores (89.9 out of 200 and 171.57 out of 250, respectively) [ 5 , 6 ]. Compared to the current study utilizing the JHLES, we may compare the findings based on a significant correlation between the two measures that support the use of the JHLES in the assessment of the same construct [ 11 ]. This comparison yielded reassuring results that the perceptions of medical students are still positive, with variations in the domains of LE, as described below. The added value of the qualitative component of the current study elicits more depth in understanding LE in the COM-KSU.

Although there was no difference among male and female students in the DREEM overall average score in a previous study that was conducted at the same college in 2017, our study revealed a higher overall average score among males (83.4) than females (77.5). The lower recorded score among females might be explained by their tendency to have higher expectations of a learning environment that was not achieved as their counterpart expected [ 12 , 13 ]. For explanations, male students had higher scores in different domains related to their relationships with the faculty and peers, including mentorship, peer support, and the academic climate. Nevertheless, both genders perceived a negative view where they expressed potential gender discrimination in the focus group interviews. Male students felt that they were treated differently than females, while their counterparts believed that males had more opportunities to build relationships with the faculty and gain more experience accordingly.

In terms of academic years, the domains and overall average scores decreased as the students progressed from the first year to their internships, with an exceptional decrease in the third year followed by the recovery of scores afterward. Nevertheless, students in the first year had higher average scores than interns, possibly due to the new environment and the support provided during their first year. Qualitative group interviews elaborated more on this variation, where medical students in the first year felt a sense of pride and honor upon being accepted in the COM-KSU. They believe that this was a validation of their social status.

Although the relationship between medical students’ feelings of pride in belonging to their college and the learning environment is complex and multifaceted [ 14 ], a positive and supportive learning environment that fosters a sense of belonging can enhance medical students’ feelings of pride and affiliation with their college [ 15 , 16 ], which is evident among first-year medical students. In contrast, a negative learning environment that lacks support and inclusivity can detrimentally impact medical students’ feelings of pride and belonging [ 17 ]. Nevertheless, first-year students still experienced negative emotional effects that were not captured by the quantitative questionnaire due to the lack of professional identification they encountered when they moved from the preparatory year to medical school.

However, the decrease in the average score during the third year could be explained by engagement in clinical rotations and practical applications instead of merely learning basic science. This perception was explained during focus group interviews where students explained the third year as the most challenging due to the preparation for their actual medical practice. This included starting to see patients, taking medical history, and performing physical examinations. Interestingly, this result was consistent with other studies that were conducted in different medical schools, although different assessment tools were used, including MSLES, DREEM, and the same tool used in this study (i.e., JHLES) [ 3 , 7 , 10 ]. In contrast, other studies have shown that medical students feel more satisfied with clinical practice than with basic science during the first and second years [ 12 , 18 , 19 ].

This paradox might be explained by the difficulty students faced at the beginning of the clinical year, after which it decreased or diminished after they gained confidence in their practice under the supervision of well-trained faculty [ 20 , 21 , 22 ]. Hence, higher average scores in the following years could be explained by the maturity of the medical students and their ability to overcome early difficulties after they have more experience during clinical rotations. In the COM-KSU, medical students in their fifth year are prepared to experience life as physicians where they have pure clinical experience joining medical teams, attending rounds, clinics and doing procedures under the supervision of trained faculty and senior doctors. Hence, when mentoring was assessed among medical students, their perception reflected by the average score given to this domain increased as the number of academic years increased, with the highest score occurring during the internship. Mentorship plays an important role in the learning environment, as described in other studies [ 23 , 24 , 25 ]. The importance of the student‒faculty relationship and the enhancement of faculty influence on students are supported by the qualitative findings, which demonstrate that students' perceptions of faculty support vary, which is congruent with other studies [ 26 , 27 , 28 ]. However, a study revealed that the majority of faculty members are not prepared to provide the kind of support that has been shown to be most effective for students [ 29 ].

Furthermore, the meaningful engagement of students declines as the academic year progresses, as expressed by students’ responses to this domain in the JHLES. The qualitative approach elaborated more when students complained about the lack of support provided by the student council, which the COM-KSU perceived as the hub where medical students can engage and obtain the required support. From the students’ perspective, the student council was not able to provide effective support or bring about significant changes for students facing challenges related to their medical study needs. The qualitative study participants agreed with the findings of other local studies, highlighting the absence of a supportive environment for students in our local colleges [ 12 , 30 , 31 ]. On the other hand, the majority of students reflected positively on peer support, where they found it to have a positive impact on them. They identified college friends and colleagues as the main sources of support, which was congruent with other studies that explained the same attitude [ 32 , 33 , 34 ].

According to the students’ performance measured by their GPA, students with higher GPA had higher JHLES scores, both overall and domain average scores. High-achieving students tend to have more positive perceptions of the learning environment than do students with lower GPAs [ 10 , 11 , 21 , 22 , 35 ]. This could suggest a positive association between academic achievement and students’ perceptions of the educational setting [ 18 , 19 , 36 , 37 , 38 ]. However, students experienced positive consequences from high competition in the learning environment due to family and physician expectations that were captured during the focus group discussion. Similar results were found in another study that was conducted in the medical school of the University of Valladolid [ 39 ].

Inclusion and safety were negatively perceived in this study among medical students at all levels, regardless of their gender, academic year, or performance, which was reflected in their GPAs. This finding was consistent with other studies measuring the same domain average score of Cyberjaya University College of Medical Sciences (CUCMS), Nil Ratan Sircar Medical College (NRSMC), and College of Medicine and Sagore Dutta Hospital (CMSDH) [ 9 , 11 ]; however, this finding was in contrast to that of PUGSOM [ 40 ]. A possible explanation might be related to the aforementioned reasons, which were associated with students’ perceptions of gender discrimination, stress in the first year due to the new environment and in the third year due to engagement in clinical practice, and their achievements, which elevated stress when they had lower GPAs. Previous studies have shown that the prevalence of stress is greater during the first three years of medical education, which is consistent with our findings [ 35 ].

In contrast, the physical space domain in our study received the highest score, where we believe that physical space has improved as a result of the college's 2018 expansion [ 41 ].

Strengths and limitations

One key strength of this study is the employment of a comprehensive mixed methods approach to gain an understanding of how students perceive their learning environment. This approach collects numerical data, delves deeply into the students’ experiences and feelings, and provides valuable insights through the integration of findings from both approaches. Another strength of this study is the large number of participants from different academic years, which allows for a diverse range of perspectives from both new and experienced students.

Nevertheless, convenience sampling may not fully represent the student population and limits the generalizability of the findings. Additionally, focusing on one institution may not capture the experiences of students across different settings, cultures, or cities, potentially limiting the applicability of any recommendations to other medical colleges or regions. However, the large sample size, the diversity of data and the integration of results may enhance the transferability of the findings.

Recommendations for educational institutions

Enhance faculty development: Address the issues of perceived neutrality and reported negative interactions with faculty by investing in faculty development programs. These programs should focus on improving communication skills and mentoring abilities and cultivating more supportive and encouraging faculty‒student relationships. Creating opportunities for regular feedback from students can also aid in faculty improvement. This is important as students showed high tendency to be influenced by advice from faculty member.

Cultivate Supportive Environments: Foster a less stressful academic climate by promoting a culture of mutual respect and collaboration within the institution. Encourage open dialog between students and faculty, where questions and concerns can be raised without judgment. Stress management and well-being programs should be implemented to help students cope with academic pressures.

Revise Curriculum and Mentorship Programs: Address curriculum concerns by engaging students in the curriculum development process. Consider their suggestions for better organization, logical flow, and references. Additionally, structured mentorship programs that connect students with experienced doctors who can provide guidance, share experiences, and serve as positive role models should be established.

Evaluate and Improve Support Services: Reevaluate the effectiveness of support services such as the students' council and academic support departments. These services are responsive to students' needs and have the authority to enact meaningful changes. Regularly solicit feedback from students to gauge the impact of these services.

Promote Inclusivity and Gender Equity: as FGDs showed that both genders feel discriminated against, creating initiatives to address perceptions of discrimination and gender bias within the learning environment is important. This may involve raising awareness, offering training on gender sensitivity, and implementing policies that promote inclusivity and equal opportunities for all students, regardless of gender.

Recommendations for further research:

Longitudinal studies should be conducted to track the changes in students’ perceptions and experiences. This will help us identify emerging trends and understand the long-term effects of interventions and policy changes.

This research can be expanded by including studies with medical schools or institutions to validate our findings and assess how applicable they are in diverse educational settings.

The use of mixed methods research in the field of education should be further explored. Investigate approaches that combine qualitative and deductive methods to gain deeper insights into students’ educational experiences.

Dive deeper into specific areas highlighted in this research, such as mentoring programs and concerns related to the curriculum. Explore ways to enhance mentoring effectiveness and develop strategies for improving the curriculum to create a learning environment.

Interventions targeted at addressing identified areas should be implemented for improvement while thoroughly evaluating their impact. This will enable institutions to assess the effectiveness of these interventions based on data-driven decisions leading to the enhancement of education.

This study was the first to assess the learning environment of medical students at COM-KSU through quantitative and qualitative approaches. The overall average JHLES score indicated room for improvement, in line with global trends. Gender disparities, challenges in different academic years, and the critical role of mentorship were identified. Academic performance correlated positively with students' perceptions, while inclusion and safety were areas of concern. The physical space domain received the highest score, reflecting investments in infrastructure. These findings underscore the need for targeted interventions to address gender disparities, enhance mentorship, improve student engagement, and ensure inclusivity and safety, ultimately enhancing the educational experience of COM-KSU medical students.

Availability of data and materials

The datasets used and analyzed during the current study are available for request from the corresponding author.

Abbreviations

  • Learning environment

Johns Hopkins Learning Environment Scale

College of Medicine at King Saud University

Dundee Ready Educational Environment Measure

Focused group discussions

Medical School Learning Environment Survey

Cyberjaya University College of Medical Sciences

Nil Ratan Sircar Medical College

College of Medicine and Sagore Dutta Hospital

Rusticus SA, Wilson D, Casiro O, Lovato C. Evaluating the quality of health professions learning environments: development and validation of the Health Education Learning Environment Survey (HELES). Eval Health Prof. 2020;43:162–8.

Article   Google Scholar  

Dunham L, Dekhtyar M, Gruener G, CichoskiKelly E, Deitz J, Elliott D, Stuber ML, Skochelak SE. Medical student perceptions of the learning environment in medical school change as students transition to clinical training in undergraduate medical school. Teach Learn Med. 2017;29:383–91.

Wayne SJ, Fortner SA, Kitzes JA, Timm C, Kalishman S. Cause or effect? The relationship between student perception of the medical school learning environment and academic performance on USMLE Step 1. Med Teach. 2013;35:376–80.

Harden RM. The learning environment and the curriculum. Med Teach. 2001;23:335–6.

Al Ayed IH, Sheik SA. Assessment of the educational environment at the college of medicine of king Saud University, Riyadh. EMHJ - East Mediterr Health J. 2008;14(4):953–9.

Google Scholar  

Soliman MM, Sattar K, Alnassar S, Alsaif F, Alswat K, Alghonaim M, Alhaizan M, Al-Furaih N. Medical students’ perception of the learning environment at King Saud University Medical College, Saudi Arabia, using DREEM inventory. Adv Med Educ Pract. 2017;8:221–7.

Shochet RB, Colbert-Getz JM, Wright SM. The Johns Hopkins learning environment scale: measuring medical students’ perceptions of the processes supporting professional formation. Acad Med J Assoc Am Med Coll. 2015;90:810–8.

R Core Team (2022) R: The R Project for Statistical Computing. https://www.r-project.org/ . Accessed 13 Aug 2023

Sengupta P, Sharma A, Das N. Perception of Learning environment among undergraduate medical students in two different medical schools through DREEM and JHLES Questionnaire. J Clin Diagn Res JCDR. 2017;11:JC01–4.

Shahzad S, Wajid G. How different is the learning environment of public and private sector medical colleges in Lahore, Pakistan? Pak J Med Sci. 2023;39:757–63.

Tackett S, Shochet R, Shilkofski NA, Colbert-Getz J, Rampal K, Abu Bakar H, Wright S. Learning environment assessments of a single curriculum being taught at two medical schools 10,000 miles apart. BMC Med Educ. 2015;15:105.

Al-Hazimi A, Zaini R, Al-Hyiani A, Hassan N, Gunaid A, Ponnamperuma G, Karunathilake I, Roff S, McAleer S, Davis M. Educational environment in traditional and innovative medical schools: a study in four undergraduate medical schools. Educ Health Abingdon Engl. 2004;17:192–203.

Schultz-Jones B, Ledbetter C, Bishop J (2014) Investigating Differences Between Male and Female Students’ Perceptions of School Library Learning Environments. IASL Annu Conf Proc. https://doi.org/10.29173/iasl7867 .

Tanya Albert Henry (2016) Med students’ perceptions of learning environment matter. In: Am. Med. Assoc. https://www.ama-assn.org/education/accelerating-change-medical-education/med-students-perceptions-learning-environment . Accessed 13 Aug 2023.

Mahmood H, Chaudhry MA, Rashid H, Farani TM, Ali AR, Batool Z, Amin B. Medical student’s perception of educational environment. Proceedings. 2020;34:47–51.

Xie Y, Cao D, Sun T, Yang LB. The effects of academic adaptability on academic burnout, immersion in learning, and academic performance among Chinese medical students: a cross-sectional study. BMC Med Educ. 2019. https://doi.org/10.1186/s12909-019-1640-9 .

Verbree A-R, Isik U, Janssen J, Dilaver G. Inclusion and diversity within medical education: a focus group study of students’ experiences. BMC Med Educ. 2023;23:61.

Sarwar S, Tarique S. Perception of educational environment: does it impact academic performance of medical students? JPMA J Pak Med Assoc. 2016;66:1210–4.

Yoo D-M, Kim D-H. The relationship between students’ perception of the educational environment and their subjective happiness. BMC Med Educ. 2019;19:409.

Almansour M, AlMehmadi BA, Zeb NG, Matbouly G, Sami W, Badahdah A-M. Promising score for teaching and learning environment: an experience of a fledgling medical college in Saudi Arabia. BMC Med Educ. 2023;23:479.

Eyal L, Cohen R. Preparation for clinical practice: a survey of medical students’ and graduates’ perceptions of the effectiveness of their medical school curriculum. Med Teach. 2006;28:e162-170.

Fleit HB, Lu W-H, Olvet DM, Chandran L. Medical students’ perception of behaviors in the clinical learning environment evolve with increasing clinical exposure as measured with situational video vignettes. Med Teach. 2020;42:822–7.

Dobie S, Smith S, Robins L. How assigned faculty mentors view their mentoring relationships: an interview study of mentors in medical education. Mentor Tutoring Partnersh Learn. 2010;18:337–59.

Rose GL, Rukstalis MR, Schuckit MA. Informal mentoring between faculty and medical students. Acad Med J Assoc Am Med Coll. 2005;80:344–8.

Sonawane T, Meshram R, Jagia G, Gajbhiye R, Adhikari S. Effects of Mentoring in First Year Medical Undergraduate Students using DASS-21. J Clin Diagn Res. 2021. https://doi.org/10.7860/JCDR/2021/50102.15682 .

Cress CM. Creating inclusive learning communities: the role of student–faculty relationships in mitigating negative campus climate. Learn Inq. 2008;2:95–111.

Hendelman W, Byszewski A. Formation of medical student professional identity: categorizing lapses of professionalism, and the learning environment. BMC Med Educ. 2014;14:139.

Neufeld A, Malin G. How medical students’ perceptions of instructor autonomy-support mediate their motivation and psychological well-being. Med Teach. 2020;42:650–6.

Costino K. Equity-minded faculty development: an intersectional identity-conscious community of practice model for faculty learning. Metrop Univ. 2018;29:117–36.

Al-Mohaimeed A. Perceptions of the educational environment of a new medical school, Saudi Arabia. Int J Health Sci. 2013;7:150–9.

Altemani AH, Merghani TH. The quality of the educational environment in a medical college in Saudi Arabia. Int J Med Educ. 2017;8:128–32.

Lerchenfeldt S, Mi M, Eng M. The utilization of peer feedback during collaborative learning in undergraduate medical education: a systematic review. BMC Med Educ. 2019;19:321.

Nofziger AC, Naumburg EH, Davis BJ, Mooney CJ, Epstein RM. Impact of peer assessment on the professional development of medical students: a qualitative study. Acad Med J Assoc Am Med Coll. 2010;85:140–7.

Yamada Y, Klugar M, Ivanova K, Oborna I. Psychological distress and academic self-perception among international medical students: the role of peer social support. BMC Med Educ. 2014;14:256.

Abdulghani HM, AlKanhal AA, Mahmoud ES, Ponnamperuma GG, Alfaris EA. Stress and its effects on medical students: a cross-sectional study at a college of medicine in Saudi Arabia. J Health Popul Nutr. 2011;29:516–22.

Khursheed I, Baig L. Students’ perceptions of educational environment of a private medical school in Pakistan. JPMA J Pak Med Assoc. 2014;64:1244–9.

Mayya S, Roff S. Students’ perceptions of educational environment: a comparison of academic achievers and under-achievers at kasturba medical college, India. Educ Health Abingdon Engl. 2004;17:280–91.

Ahmed Y, Taha MH, Al-Neel S, Gaffar AM. Students’ perception of the learning environment and its relation to their study year and performance in Sudan. Int J Med Educ. 2018. https://doi.org/10.5116/ijme.5af0.1fee .

Corell A, Regueras LM, Verdú E, Verdú MJ, de Castro JP. Effects of competitive learning tools on medical students: a case study. PLoS ONE. 2018;13.

Tackett S, Bakar HA, Shilkofski NA, Coady N, Rampal K, Wright S. Profiling medical school learning environments in Malaysia: a validation study of the Johns Hopkins Learning Environment Scale. J Educ Eval Health Prof. 2015;12.

King Saud University Medical City. المدينة الطبية بجامعة الملك سعود | news. 2018. https://medicalcity.ksu.edu.sa/ar/news/details/26-10-2018 . Accessed 21 Oct 2023

Download references

Acknowledgements

Not applicable.

This research is non-funded.

Author information

Authors and affiliations.

Department of Medical Education, College of Medicine, King Saud University, Riyadh, Saudi Arabia

Mohammed Almansour

Department of Family and Community Medicine, College of Medicine, King Saud University, Riyadh, Saudi Arabia

Noura Abouammoh & Hani A. Alghamdi

College of Medicine, King Saud University, Riyadh, Saudi Arabia

Reem Bin Idris, Omar Abdullatif Alsuliman, Renad Abdulrahman Alhomaidi & Mohammed Hamad Alhumud

You can also search for this author in PubMed   Google Scholar

Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Corresponding author

Correspondence to Hani A. Alghamdi .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by King Saud University’s Institutional Review Board (KSU IRB) with the approval number E-22–7298.

Consent to participate

Electronic informed consent was obtained from all participants in the quantitative arm, and written informed consent was obtained from all participants in the qualitative arm before they participated in the study.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Almansour, M., Abouammoh, N., Idris, R.B. et al. Exploring medical students' experience of the learning environment: a mixed methods study in Saudi medical college. BMC Med Educ 24 , 723 (2024). https://doi.org/10.1186/s12909-024-05716-4

Download citation

Received : 14 March 2024

Accepted : 26 June 2024

Published : 03 July 2024

DOI : https://doi.org/10.1186/s12909-024-05716-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Medical education
  • Comparative studies
  • Mixed methods

BMC Medical Education

ISSN: 1472-6920

what is quantitative research peer reviewed articles

IMAGES

  1. Write Esse: Peer reviewed research articles

    what is quantitative research peer reviewed articles

  2. Peer-reviewed Quantitative Research

    what is quantitative research peer reviewed articles

  3. How to find if the journal is peer reviewed or not? How to tell if a paper is peer reviewed?

    what is quantitative research peer reviewed articles

  4. Sample Quantitative Nursing Research Article Critique : CHECK THESE

    what is quantitative research peer reviewed articles

  5. Quantitative Research

    what is quantitative research peer reviewed articles

  6. What is Peer Review?

    what is quantitative research peer reviewed articles

VIDEO

  1. Lecture 41: Quantitative Research

  2. Lecture 43: Quantitative Research

  3. Introduction to Quantitative Research Course 1 مقدمة مقرر البحث الكمي

  4. How to Develop Quantitative Research Titles: Means and Ends

  5. Quantitative Research Purposes: Updating the Previous Theories

  6. What Is Medical Writing?

COMMENTS

  1. A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles

    Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points. ... In quantitative research, hypotheses predict the expected relationships among variables.15 Relationships among variables that can be predicted include 1) ...

  2. Quantitative and Qualitative Approaches to Generalization and

    We conclude that quantitative research may benefit from a bottom-up generalization strategy as it is employed in most qualitative research programs. Inductive reasoning forces us to think about the boundary conditions of our theories and provides a framework for generalization beyond statistical testing. In this perspective, failed replications ...

  3. Synthesising quantitative and qualitative evidence to inform guidelines

    Qualitative and quantitative research is collected and analysed at the same time in a parallel or complementary manner. Integration can occur at three points: a. Data-based convergent synthesis design ... Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: No additional data are available.

  4. Quantitative Research

    The majority of journals that publish medical and health research have a peer review process that is thorough and robust. The peer review process includes a detailed review by experts in the field, who will assess the methods used, sample size, and limitations of the study. A study may be rejected for publication if it does not meet the ...

  5. Quantitative Research Excellence: Study Design and ...

    Quantitative Research Excellence: Study Design and Reliable and Valid Measurement of Variables. Laura J. Duckett, BSN, MS, PhD, ... Review of General Psychology. Dec 2000. Open Access. Validity Threats in Quantitative Data Collection With Games: A Narrative Survey. Show details Hide details.

  6. Review Article Synthesizing Quantitative Evidence for Evidence-based

    The purpose of this paper is to introduce an overview of the fundamental knowledge, principals and processes in SR. The focus of this paper is on SR especially for the synthesis of quantitative data from primary research studies that examines the effectiveness of healthcare interventions. To activate evidence-based nursing care in various ...

  7. Chapter 3. Introduction to Quantitative Research and Data

    Quantitative research allows librarians to learn more about the demographics of a population, measure how many patrons use a service or product, examine attitudes and behaviors, document trends, or explain what is known anecdotally. ... Metrics include the number of peer-reviewed articles, coauthor collaborations, publication patterns, and ...

  8. Recent quantitative research on determinants of health in high ...

    The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries. ... the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore ...

  9. What Is Quantitative Research?

    Revised on June 22, 2023. Quantitative research is the process of collecting and analyzing numerical data. It can be used to find patterns and averages, make predictions, test causal relationships, and generalize results to wider populations. Quantitative research is the opposite of qualitative research, which involves collecting and analyzing ...

  10. Quantitative vs. Qualitative Research Design: Understanding the Differences

    It relies on insights derived from numbers — countable, measurable and statistically sound data. Quantitative research is a strategic research design used when basing critical decisions on statistical conclusions and quantifiable data. Quantitative research provides numerical-backed quantifiable data that may approve or discount a theory or ...

  11. Quantitative Research

    Quantitative Research. Quantitative research is an objective, formal, systematic process in which numerical data are used to quantify or measure social phenomena and produce findings. ... We first examined the research methods used by the reviewed papers; most of the papers use a single analytical approach: quantitative (n = 77) or qualitative ...

  12. (PDF) Quantitative Research Methods : A Synopsis Approach

    Abstract. The aim of th is study i s to e xplicate the quanti tative methodology. The study established that. quantitative research de als with quantifying and analyzing variables in o rder to get ...

  13. How to appraise quantitative research

    Title, keywords and the authors. The title of a paper should be clear and give a good idea of the subject area. The title should not normally exceed 15 words 2 and should attract the attention of the reader. 3 The next step is to review the key words. These should provide information on both the ideas or concepts discussed in the paper and the ...

  14. The Methodological Underdog: A Review of Quantitative Research in the

    Differences in methodological strengths and weaknesses between quantitative and qualitative research are discussed, followed by a data mining exercise on 1,089 journal articles published in Adult Education Quarterly, Studies in Continuing Education, and International Journal of Lifelong Learning. A categorization of quantitative adult education ...

  15. What Is Peer Review?

    The most common types are: Single-blind review. Double-blind review. Triple-blind review. Collaborative review. Open review. Relatedly, peer assessment is a process where your peers provide you with feedback on something you've written, based on a set of criteria or benchmarks from an instructor.

  16. Critiquing Quantitative Research Reports: Key Points for the Beginner

    The first step in the critique process is for the reader to browse the abstract and article for an overview. During this initial review a great deal of information can be obtained. The abstract should provide a clear, concise overview of the study. During this review it should be noted if the title, problem statement, and research question (or ...

  17. A practical guide to data analysis in general literature reviews

    This article is a practical guide to conducting data analysis in general literature reviews. The general literature review is a synthesis and analysis of published research on a relevant clinical issue, and is a common format for academic theses at the bachelor's and master's levels in nursing, physiotherapy, occupational therapy, public health and other related fields.

  18. Qualitative vs Quantitative Research

    Qualitative Quantitative; Definition: To understand and interpret social interactions. Research that seeks to provide understanding of human experience, perceptions, motivations, intentions, and behaviours based on description and observation and utilizing a naturalistic interpretative approach to a subject and its contextual setting.

  19. Quantitative Approaches for the Evaluation of Implementation Research

    3. Quantitative Methods for Evaluating Implementation Outcomes. While summative evaluation is distinguishable from formative evaluation (see Elwy et al. this issue), proper understanding of the implementation strategy requires using both methods, perhaps at different stages of implementation research (The Health Foundation, 2015).Formative evaluation is a rigorous assessment process designed ...

  20. "Doing Research": Understanding the Different Types of Peer-Reviewed

    Notes. 1. A) Peer-reviewed articles are the most academic and scholarly in the journal; these articles create new ideas and drive the academic literature forward. These articles are often cross-disciplinary, using theories or research methodologies to explore an element of the voice field or voice pedagogy.

  21. (PDF) Key disparities between quantitative and qualitative research

    METHODS Adhering to PRISMA guidelines, we conducted a systematic review of peer-reviewed articles indexed within Scopus, Medline, ProQuest Political Science and Public Administration, and PsycINFO.

  22. Distinguishing Between Quantitative and Qualitative Research: A

    Living within blurry boundaries: The value of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research , 12(3), 268-279. Crossref

  23. Understand Peer Review

    Quantitative Research Methods - PSYC/SOCI 225 Research Guide. The study of applied research in the behavioral and social sciences, with an emphasis on design, methodology, results interpretation, and theory building ... Peer-reviewed journals do contain some information that is itself not refereed, such as editorials, opinions, letters, and ...

  24. Robust point and variance estimation for meta‐analyses with selective

    Meta-analysis produces a quantitative synthesis of evidence-based knowledge, shaping not only research trends but also policies and practices in biology. However, two statistical issues, selective reporting and statistical dependence, can severely distort meta-analytic parameter estimation and inference.

  25. Full article: Navigating the gig economy: exploring challenges and

    A quantitative approach involving 347 gig workers was employed to conduct this study. A structured interview was used to identify the motivations, challenges, stress, quality of life and well-being of gig workers involved in different gig work such as car driving, salon service or item delivery service.

  26. Quantitative Data Analysis—In the Graduate Curriculum

    Teaching quantitative data analysis is not teaching number crunching, but teaching a way of critical thinking for how to analyze the data. The goal of data analysis is to reveal the underlying patterns, trends, and relationships of a study's contextual situation. Learning data analysis is not learning how to use statistical tests to crunch ...

  27. Exploring medical students' experience of the learning environment: a

    Findings from quantitative and qualitative methods were integrated using joint display. A total of 653 medical students completed the JHLES. The total average score was 81 out of 140 (16.8), and the average subscale scores ranged from 2.27 (0.95) for inclusion and safety to 3.37 (0.91) for community of peers. ... This research offers valuable ...