reference in research format

• Walden University
• Faculty Portal

Reference List: Common Reference List Examples

Article (with doi).

Alvarez, E., & Tippins, S. (2019). Socialization agents that Puerto Rican college students use to make financial decisions. Journal of Social Change , 11 (1), 75–85. https://doi.org/10.5590/JOSC.2019.11.1.07

Laplante, J. P., & Nolin, C. (2014). Consultas and socially responsible investing in Guatemala: A case study examining Maya perspectives on the Indigenous right to free, prior, and informed consent. Society & Natural Resources , 27 , 231–248. https://doi.org/10.1080/08941920.2013.861554

Use the DOI number for the source whenever one is available. DOI stands for "digital object identifier," a number specific to the article that can help others locate the source. In APA 7, format the DOI as a web address. Active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. Also see our Quick Answer FAQ, "Can I use the DOI format provided by library databases?"

Jerrentrup, A., Mueller, T., Glowalla, U., Herder, M., Henrichs, N., Neubauer, A., & Schaefer, J. R. (2018). Teaching medicine with the help of “Dr. House.” PLoS ONE , 13 (3), Article e0193972. https://doi.org/10.1371/journal.pone.0193972

For journal articles that are assigned article numbers rather than page ranges, include the article number in place of the page range.

For more on citing electronic resources, see  Electronic Sources References .

Article (Without DOI)

Found in a common academic research database or in print.

Casler , T. (2020). Improving the graduate nursing experience through support on a social media platform. MEDSURG Nursing , 29 (2), 83–87.

If an article does not have a DOI and you retrieved it from a common academic research database through the university library, there is no need to include any additional electronic retrieval information. The reference list entry looks like the entry for a print copy of the article. (This format differs from APA 6 guidelines that recommended including the URL of a journal's homepage when the DOI was not available.) Note that APA 7 has additional guidance on reference list entries for articles found only in specific databases or archives such as Cochrane Database of Systematic Reviews, UpToDate, ProQuest Dissertations and Theses Global, and university archives. See APA 7, Section 9.30 for more information.

Found on an Open Access Website

Eaton, T. V., & Akers, M. D. (2007). Whistleblowing and good governance. CPA Journal , 77 (6), 66–71. http://archives.cpajournal.com/2007/607/essentials/p58.htm

Provide the direct web address/URL to a journal article found on the open web, often on an open access journal's website. In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Weinstein, J. A. (2010).  Social change  (3rd ed.). Rowman & Littlefield.

If the book has an edition number, include it in parentheses after the title of the book. If the book does not list any edition information, do not include an edition number. The edition number is not italicized.

American Nurses Association. (2015). Nursing: Scope and standards of practice (3rd ed.).

If the author and publisher are the same, only include the author in its regular place and omit the publisher.

Lencioni, P. (2012). The advantage: Why organizational health trumps everything else in business . Jossey-Bass. https://amzn.to/343XPSJ

As a change from APA 6 to APA 7, it is no longer necessary to include the ebook format in the title. However, if you listened to an audiobook and the content differs from the text version (e.g., abridged content) or your discussion highlights elements of the audiobook (e.g., narrator's performance), then note that it is an audiobook in the title element in brackets. For ebooks and online audiobooks, also include the DOI number (if available) or nondatabase URL but leave out the electronic retrieval element if the ebook was found in a common academic research database, as with journal articles. APA 7 allows for the shortening of long DOIs and URLs, as shown in this example. See APA 7, Section 9.36 for more information.

Chapter in an Edited Book

Poe, M. (2017). Reframing race in teaching writing across the curriculum. In F. Condon & V. A. Young (Eds.), Performing antiracist pedagogy in rhetoric, writing, and communication (pp. 87–105). University Press of Colorado.

Include the page numbers of the chapter in parentheses after the book title.

Christensen, L. (2001). For my people: Celebrating community through poetry. In B. Bigelow, B. Harvey, S. Karp, & L. Miller (Eds.), Rethinking our classrooms: Teaching for equity and justice (Vol. 2, pp. 16–17). Rethinking Schools.

Also include the volume number or edition number in the parenthetical information after the book title when relevant.

Freud, S. (1961). The ego and the id. In J. Strachey (Ed.),  The standard edition of the complete psychological works of Sigmund Freud  (Vol. 19, pp. 3-66). Hogarth Press. (Original work published 1923)

When a text has been republished as part of an anthology collection, after the author’s name include the date of the version that was read. At the end of the entry, place the date of the original publication inside parenthesis along with the note “original work published.” For in-text citations of republished work, use both dates in the parenthetical citation, original date first with a slash separating the years, as in this example: Freud (1923/1961). For more information on reprinted or republished works, see APA 7, Sections 9.40-9.41.

Classroom Resources

Citing classroom resources.

If you need to cite content found in your online classroom, use the author (if there is one listed), the year of publication (if available), the title of the document, and the main URL of Walden classrooms. For example, you are citing study notes titled "Health Effects of Exposure to Forest Fires," but you do not know the author's name, your reference entry will look like this:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Canvas. https://waldenu.instructure.com

If you do know the author of the document, your reference will look like this:

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Canvas. https://waldenu.instructure.com  

A few notes on citing course materials:

• [Lecture notes]
• [Course handout]
• [Study notes]
• It can be difficult to determine authorship of classroom documents. If an author is listed on the document, use that. If the resource is clearly a product of Walden (such as the course-based videos), use Walden University as the author. If you are unsure or if no author is indicated, place the title in the author spot, as above.
• If you cannot determine a date of publication, you can use n.d. (for "no date") in place of the year.

Note:  The web location for Walden course materials is not directly retrievable without a password, and therefore, following APA guidelines, use the main URL for the class sites: https://class.waldenu.edu.

Citing Tempo Classroom Resources

Clear author: 

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Brightspace. https://mytempo.waldenu.edu

Unclear author:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Brightspace. https://mytempo.waldenu.edu

Conference Sessions and Presentations

Feinman, Y. (2018, July 27). Alternative to proctoring in introductory statistics community college courses [Poster presentation]. Walden University Research Symposium, Minneapolis, MN, United States. https://scholarworks.waldenu.edu/symposium2018/23/

Torgerson, K., Parrill, J., & Haas, A. (2019, April 5-9). Tutoring strategies for online students [Conference session]. The Higher Learning Commission Annual Conference, Chicago, IL, United States. http://onlinewritingcenters.org/scholarship/torgerson-parrill-haas-2019/

Dictionary Entry

Merriam-Webster. (n.d.). Leadership. In Merriam-Webster.com dictionary . Retrieved May 28, 2020, from https://www.merriam-webster.com/dictionary/leadership

When constructing a reference for an entry in a dictionary or other reference work that has no byline (i.e., no named individual authors), use the name of the group—the institution, company, or organization—as author (e.g., Merriam Webster, American Psychological Association, etc.). The name of the entry goes in the title position, followed by "In" and the italicized name of the reference work (e.g., Merriam-Webster.com dictionary , APA dictionary of psychology ). In this instance, APA 7 recommends including a retrieval date as well for this online source since the contents of the page change over time. End the reference entry with the specific URL for the defined word.

Discussion Board Post

Osborne, C. S. (2010, June 29). Re: Environmental responsibility [Discussion post]. Walden University Canvas.  https://waldenu.instructure.com  

Dissertations or Theses

Retrieved From a Database

Nalumango, K. (2019). Perceptions about the asylum-seeking process in the United States after 9/11 (Publication No. 13879844) [Doctoral dissertation, Walden University]. ProQuest Dissertations and Theses.

Retrieved From an Institutional or Personal Website

Evener. J. (2018). Organizational learning in libraries at for-profit colleges and universities [Doctoral dissertation, Walden University]. ScholarWorks. https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6606&context=dissertations

Unpublished Dissertation or Thesis

Kirwan, J. G. (2005). An experimental study of the effects of small-group, face-to-face facilitated dialogues on the development of self-actualization levels: A movement towards fully functional persons [Unpublished doctoral dissertation]. Saybrook Graduate School and Research Center.

For further examples and information, see APA 7, Section 10.6.

Legal Material

For legal references, APA follows the recommendations of The Bluebook: A Uniform System of Citation , so if you have any questions beyond the examples provided in APA, seek out that resource as well.

Court Decisions

Reference format:

Name v. Name, Volume Reporter Page (Court Date). URL

Sample reference entry:

Brown v. Board of Education, 347 U.S. 483 (1954). https://www.oyez.org/cases/1940-1955/347us483

Sample citation:

In Brown v. Board of Education (1954), the Supreme Court ruled racial segregation in schools unconstitutional.

Note: Italicize the case name when it appears in the text of your paper.

Name of Act, Title Source § Section Number (Year). URL

Sample reference entry for a federal statute:

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 et seq. (2004). https://www.congress.gov/108/plaws/publ446/PLAW-108publ446.pdf

Sample reference entry for a state statute:

Minnesota Nurse Practice Act, Minn. Stat. §§ 148.171 et seq. (2019). https://www.revisor.mn.gov/statutes/cite/148.171

Sample citation: Minnesota nurses must maintain current registration in order to practice (Minnesota Nurse Practice Act, 2010).

Note: The § symbol stands for "section." Use §§ for sections (plural). To find this symbol in Microsoft Word, go to "Insert" and click on Symbol." Look in the Latin 1-Supplement subset. Note: U.S.C. stands for "United States Code." Note: The Latin abbreviation " et seq. " means "and what follows" and is used when the act includes the cited section and ones that follow. Note: List the chapter first followed by the section or range of sections.

Unenacted Bills and Resolutions

(Those that did not pass and become law)

Title [if there is one], bill or resolution number, xxx Cong. (year). URL

Sample reference entry for Senate bill:

Anti-Phishing Act, S. 472, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/senate-bill/472

Sample reference entry for House of Representatives resolution:

Anti-Phishing Act, H.R. 1099, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/house-bill/1099

The Anti-Phishing Act (2005) proposed up to 5 years prison time for people running Internet scams.

These are the three legal areas you may be most apt to cite in your scholarly work. For more examples and explanation, see APA 7, Chapter 11.

Magazine Article

Clay, R. (2008, June). Science vs. ideology: Psychologists fight back about the misuse of research. Monitor on Psychology , 39 (6). https://www.apa.org/monitor/2008/06/ideology

Note that for citations, include only the year: Clay (2008). For magazine articles retrieved from a common academic research database, leave out the URL. For magazine articles from an online news website that is not an online version of a print magazine, follow the format for a webpage reference list entry.

Newspaper Article (Retrieved Online)

Baker, A. (2014, May 7). Connecticut students show gains in national tests. New York Times . http://www.nytimes.com/2014/05/08/nyregion/national-assessment-of-educational-progress-results-in-Connecticut-and-New-Jersey.html

Include the full date in the format Year, Month Day. Do not include a retrieval date for periodical sources found on websites. Note that for citations, include only the year: Baker (2014). For newspaper articles retrieved from a common academic research database, leave out the URL. For newspaper articles from an online news website that is not an online version of a print newspaper, follow the format for a webpage reference list entry.

OASIS Resources

Oasis webpage.

OASIS. (n.d.). Common reference list examples . Walden University. https://academicguides.waldenu.edu/writingcenter/apa/references/examples

For all OASIS content, list OASIS as the author. Because OASIS webpages do not include publication dates, use “n.d.” for the year.

Interactive Guide

OASIS. (n.d.). Embrace iterative research and writing [Interactive guide]. Walden University. https://academics.waldenu.edu/oasis/iterative-research-writing-web

For OASIS multimedia resources, such as interactive guides, include a description of the resource in brackets after the title.

Online Video/Webcast

Walden University. (2013).  An overview of learning  [Video]. Walden University Canvas.  https://waldenu.instructure.com  

Use this format for online videos such as Walden videos in classrooms. Most of our classroom videos are produced by Walden University, which will be listed as the author in your reference and citation. Note: Some examples of audiovisual materials in the APA manual show the word “Producer” in parentheses after the producer/author area. In consultation with the editors of the APA manual, we have determined that parenthetical is not necessary for the videos in our courses. The manual itself is unclear on the matter, however, so either approach should be accepted. Note that the speaker in the video does not appear in the reference list entry, but you may want to mention that person in your text. For instance, if you are viewing a video where Tobias Ball is the speaker, you might write the following: Tobias Ball stated that APA guidelines ensure a consistent presentation of information in student papers (Walden University, 2013). For more information on citing the speaker in a video, see our page on Common Citation Errors .

Taylor, R. [taylorphd07]. (2014, February 27). Scales of measurement [Video]. YouTube. https://www.youtube.com/watch?v=PDsMUlexaMY

Walden University Academic Skills Center. (2020, April 15). One-way ANCOVA: Introduction [Video]. YouTube. https://youtu.be/_XnNDQ5CNW8

For videos from streaming sites, use the person or organization who uploaded the video in the author space to ensure retrievability, whether or not that person is the speaker in the video. A username can be provided in square brackets. As a change from APA 6 to APA 7, include the publisher after the title, and do not use "Retrieved from" before the URL. See APA 7, Section 10.12 for more information and examples.

See also reference list entry formats for TED Talks .

Technical and Research Reports

Edwards, C. (2015). Lighting levels for isolated intersections: Leading to safety improvements (Report No. MnDOT 2015-05). Center for Transportation Studies. http://www.cts.umn.edu/Publications/ResearchReports/reportdetail.html?id=2402

Technical and research reports by governmental agencies and other research institutions usually follow a different publication process than scholarly, peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature , and white papers are a type of this literature. See APA 7, Section 10.4 for more information.

Reference list entires for TED Talks follow the usual guidelines for multimedia content found online. There are two common places to find TED talks online, with slightly different reference list entry formats for each.

TED Talk on the TED website

If you find the TED Talk on the TED website, follow the format for an online video on an organizational website:

Owusu-Kesse, K. (2020, June). 5 needs that any COVID-19 response should meet [Video]. TED Conferences. https://www.ted.com/talks/kwame_owusu_kesse_5_needs_that_any_covid_19_response_should_meet

The speaker is the author in the reference list entry if the video is posted on the TED website. For citations, use the speaker's surname.

TED Talk on YouTube

If you find the TED Talk on YouTube or another streaming video website, follow the usual format for streaming video sites:

TED. (2021, February 5). The shadow pandemic of domestic violence during COVID-19 | Kemi DaSilvalbru [Video]. YouTube. https://www.youtube.com/watch?v=PGdID_ICFII

TED is the author in the reference list entry if the video is posted on YouTube since it is the channel on which the video is posted. For citations, use TED as the author.

Walden University Course Catalog

To include the Walden course catalog in your reference list, use this format:

Walden University. (2020). 2019-2020 Walden University catalog . https://catalog.waldenu.edu/index.php

If you cite from a specific portion of the catalog in your paper, indicate the appropriate section and paragraph number in your text:

...which reflects the commitment to social change expressed in Walden University's mission statement (Walden University, 2020, Vision, Mission, and Goals section, para. 2).

And in the reference list:

Walden University. (2020). Vision, mission, and goals. In 2019-2020 Walden University catalog. https://catalog.waldenu.edu/content.php?catoid=172&navoid=59420&hl=vision&returnto=search

Vartan, S. (2018, January 30). Why vacations matter for your health . CNN. https://www.cnn.com/travel/article/why-vacations-matter/index.html

For webpages on the open web, include the author, date, webpage title, organization/site name, and URL. (There is a slight variation for online versions of print newspapers or magazines. For those sources, follow the models in the previous sections of this page.)

American Federation of Teachers. (n.d.). Community schools . http://www.aft.org/issues/schoolreform/commschools/index.cfm

If there is no specified author, then use the organization’s name as the author. In such a case, there is no need to repeat the organization's name after the title.

In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Related Resources

Knowledge Check: Common Reference List Examples

Didn't find what you need? Email us at [email protected] .

• Previous Page: Reference List: Overview
• Next Page: Common Military Reference List Examples
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Referencing

A Quick Guide to Harvard Referencing | Citation Examples

Published on 14 February 2020 by Jack Caulfield . Revised on 15 September 2023.

Referencing is an important part of academic writing. It tells your readers what sources you’ve used and how to find them.

Harvard is the most common referencing style used in UK universities. In Harvard style, the author and year are cited in-text, and full details of the source are given in a reference list .

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Harvard in-text citation, creating a harvard reference list, harvard referencing examples, referencing sources with no author or date, frequently asked questions about harvard referencing.

A Harvard in-text citation appears in brackets beside any quotation or paraphrase of a source. It gives the last name of the author(s) and the year of publication, as well as a page number or range locating the passage referenced, if applicable:

Note that ‘p.’ is used for a single page, ‘pp.’ for multiple pages (e.g. ‘pp. 1–5’).

An in-text citation usually appears immediately after the quotation or paraphrase in question. It may also appear at the end of the relevant sentence, as long as it’s clear what it refers to.

When your sentence already mentions the name of the author, it should not be repeated in the citation:

Sources with multiple authors

When you cite a source with up to three authors, cite all authors’ names. For four or more authors, list only the first name, followed by ‘ et al. ’:

Sources with no page numbers

Some sources, such as websites , often don’t have page numbers. If the source is a short text, you can simply leave out the page number. With longer sources, you can use an alternate locator such as a subheading or paragraph number if you need to specify where to find the quote:

Multiple citations at the same point

When you need multiple citations to appear at the same point in your text – for example, when you refer to several sources with one phrase – you can present them in the same set of brackets, separated by semicolons. List them in order of publication date:

Multiple sources with the same author and date

If you cite multiple sources by the same author which were published in the same year, it’s important to distinguish between them in your citations. To do this, insert an ‘a’ after the year in the first one you reference, a ‘b’ in the second, and so on:

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

A bibliography or reference list appears at the end of your text. It lists all your sources in alphabetical order by the author’s last name, giving complete information so that the reader can look them up if necessary.

The reference entry starts with the author’s last name followed by initial(s). Only the first word of the title is capitalised (as well as any proper nouns).

Sources with multiple authors in the reference list

As with in-text citations, up to three authors should be listed; when there are four or more, list only the first author followed by ‘ et al. ’:

Reference list entries vary according to source type, since different information is relevant for different sources. Formats and examples for the most commonly used source types are given below.

• Entire book
• Book chapter
• Translated book
• Edition of a book

Journal articles

• Print journal
• Online-only journal with DOI
• Online-only journal with no DOI
• General web page
• Online article or blog
• Social media post

Sometimes you won’t have all the information you need for a reference. This section covers what to do when a source lacks a publication date or named author.

No publication date

When a source doesn’t have a clear publication date – for example, a constantly updated reference source like Wikipedia or an obscure historical document which can’t be accurately dated – you can replace it with the words ‘no date’:

Note that when you do this with an online source, you should still include an access date, as in the example.

When a source lacks a clearly identified author, there’s often an appropriate corporate source – the organisation responsible for the source – whom you can credit as author instead, as in the Google and Wikipedia examples above.

When that’s not the case, you can just replace it with the title of the source in both the in-text citation and the reference list:

Prevent plagiarism, run a free check.

Harvard referencing uses an author–date system. Sources are cited by the author’s last name and the publication year in brackets. Each Harvard in-text citation corresponds to an entry in the alphabetised reference list at the end of the paper.

Vancouver referencing uses a numerical system. Sources are cited by a number in parentheses or superscript. Each number corresponds to a full reference at the end of the paper.

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

Though the terms are sometimes used interchangeably, there is a difference in meaning:

• A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
• A bibliography also includes other sources which were consulted during the research but not cited.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2023, September 15). A Quick Guide to Harvard Referencing | Citation Examples. Scribbr. Retrieved 14 May 2024, from https://www.scribbr.co.uk/referencing/harvard-style/

Is this article helpful?

Jack Caulfield

Other students also liked, harvard in-text citation | a complete guide & examples, harvard style bibliography | format & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

University of Pittsburgh Library System

• Collections

Course & Subject Guides

Citation styles: apa, mla, chicago, turabian, ieee.

• APA 7th Edition
• Turabian 9th
• Writing & Citing Help
• Understanding Plagiarism

Quick Links

Listed below are a few quick links to resources that will aid you in citing sources.

• Sign up for a Mendeley, EndNote, or Zotero training class.
• APA 7th Edition Published in October 2019. Visit this page for links to resources and examples.
• MLA Need help with citing MLA style? Find information here along with links to books in PittCat and free online resources.
• Chicago/Turabian Need help with citing Chicago/Turabian style? Find examples here along with links to the online style manual and free online resources.

Getting Started: How to use this guide

This LibGuide was designed to provide you with assistance in citing your sources when writing an academic paper.

There are different styles which format the information differently. In each tab, you will find descriptions of each citation style featured in this guide along with links to online resources for citing and a few examples.

What is a citation and citation style?

A citation is a way of giving credit to individuals for their creative and intellectual works that you utilized to support your research. It can also be used to locate particular sources and combat plagiarism. Typically, a citation can include the author's name, date, location of the publishing company, journal title, or DOI (Digital Object Identifier).

A citation style dictates the information necessary for a citation and how the information is ordered, as well as punctuation and other formatting.

How to do I choose a citation style?

There are many different ways of citing resources from your research. The citation style sometimes depends on the academic discipline involved. For example:

• APA (American Psychological Association) is used by Education, Psychology, and Sciences
• MLA (Modern Language Association) style is used by the Humanities
• Chicago/Turabian style is generally used by Business, History, and the Fine Arts

*You will need to consult with your professor to determine what is required in your specific course.

Click the links below to find descriptions of each style along with a sample of major in-text and bibliographic citations, links to books in PittCat, online citation manuals, and other free online resources.

• APA Citation Style
• MLA Citation Style
• Chicago/Turabian Citation Style
• Tools for creating bibliographies (CItation Managers)

Writing Centers

Need someone to review your paper? Visit the Writing Center or Academic Success Center on your campus.

• Oakland Campus
• Greensburg Campus
• Johnstown Campus
• Titusville Campus
• Bradford Campus
• Next: APA 7th Edition >>
• Last Updated: Mar 22, 2024 9:53 AM
• URL: https://pitt.libguides.com/citationhelp

University Libraries

• Research Guides
• Blackboard Learn
• Interlibrary Loan
• Study Rooms
• University of Arkansas

Science Fair Resources

Citing your sources.

• The Scientific Method
• Books in Our Collection
• Online Encyclopedias
• General Science Journals
• Other Helpful Resources
• Find Articles
• Poster Techniques
• Community Borrower Privileges This link opens in a new window
• Fayetteville Public Library

When you use ideas that are not your own, it is important to credit or cite the author(s) or source, even if you do not quote their idea or words exactly as written. Citing your sources allows your reader to identify the works you have consulted and to understand the scope of your research. There are many different citation styles available. You may be required to use a particular style or you may choose one.

One of the commonly used styles is the APA (American Psychological Association) Style.

APA style stipulates that authors use brief references in the text of a work with full bibliographic details supplied in a Reference List (typically at the end of your document). In text, the reference is very brief and usually consists simply of the author's last name and a date.For example:

...Sheep milk has been proved to contain more nutrients than cow milk (Johnson, 2005).

In a Reference list, the reference contains full bibliographic details written in a format that depends on the type of reference. Examples of formats for some common types of references are listed below. For additional information, visit the University of Arkansas libraries webpage on citing your sources . Another useful web-site on this topic is here.

Author last name, Author First Initial. Author Second Initial. (Publication Year). Title of article. Title of Journal, volume(issue) (if issue numbered), pages.

Bass, M. A., Enochs, W. K., & DiBrezzo, R. (2002). Comparison of two exercise programs on general well-being of college students. Psychological Reports, 91(3), 1195-1201.

Author Last Name, Author First Initial. Author Second Initial. (if there is no author move entry title to first position) (Publication year). Title of article or entry. In Work title. (Vol. number, pp. pages). Place: Publisher.

"Ivory-billed woodpecker." (2002). In The new encyclopædia britannica. (Vol. 5, p. ). 15th ed. Chicago: Encyclopædia Britannica.

Author Last Name, Author First Initial. Author Second Initial. (if there is no author move entry title to first position) (Publication year). Title of article or entry. In Work title. Retrieved from (database name or URL).

Ivory-billed woodpecker. (2006). In Encyclopædia britannica online. Retrieved from http://search.eb.com/eb/article-9043081

Author last name, Author First Initial. Author Second Initial. (Publication Year, Month Day). Title of article. Title of Magazine,volume, pages.

Holloway, M. (2005, August). When extinct isn't. Scientific American, 293, 22-23.

Author last name, Author First Initial. Author Second Initial. (Publication Year, Month Day). Title of article. Title of Magazine. volume, pages. Retrieved from (database name or URL).

Holloway, M. (2005, August). When extinct isn't. Scientific American, 293, 22-23. Retrieved from Academic Search Premier database.

Page Author Last Name, Page Author First Initial. Page Author Second Initial. Page title [nature of work - web site, blog, forum posting, etc.]. (Publication Year). Retrieved from (URL)

Sabo, G., et al. Rock art in Arkansas [Web site]. (2001). Retrieved from http://arkarcheology.uark.edu/rockart/index.html

• << Previous: Find Articles
• Next: Poster Techniques >>
• Last Updated: May 9, 2024 12:12 PM
• URL: https://uark.libguides.com/sciencefair
• See us on Instagram
• Follow us on Twitter
• Phone: 479-575-4104

• Privacy Policy

Home » How to Cite Research Paper – All Formats and Examples

How to Cite Research Paper – All Formats and Examples

Table of Contents

Research Paper Citation

Research paper citation refers to the act of acknowledging and referencing a previously published work in a scholarly or academic paper . When citing sources, researchers provide information that allows readers to locate the original source, validate the claims or arguments made in the paper, and give credit to the original author(s) for their work.

The citation may include the author’s name, title of the publication, year of publication, publisher, and other relevant details that allow readers to trace the source of the information. Proper citation is a crucial component of academic writing, as it helps to ensure accuracy, credibility, and transparency in research.

How to Cite Research Paper

There are several formats that are used to cite a research paper. Follow the guide for the Citation of a Research Paper:

Last Name, First Name. Title of Book. Publisher, Year of Publication.

Example : Smith, John. The History of the World. Penguin Press, 2010.

Journal Article

Last Name, First Name. “Title of Article.” Title of Journal, vol. Volume Number, no. Issue Number, Year of Publication, pp. Page Numbers.

Example : Johnson, Emma. “The Effects of Climate Change on Agriculture.” Environmental Science Journal, vol. 10, no. 2, 2019, pp. 45-59.

Research Paper

Last Name, First Name. “Title of Paper.” Conference Name, Location, Date of Conference.

Example : Garcia, Maria. “The Importance of Early Childhood Education.” International Conference on Education, Paris, 5-7 June 2018.

Author’s Last Name, First Name. “Title of Webpage.” Website Title, Publisher, Date of Publication, URL.

Example : Smith, John. “The Benefits of Exercise.” Healthline, Healthline Media, 1 March 2022, https://www.healthline.com/health/benefits-of-exercise.

News Article

Last Name, First Name. “Title of Article.” Name of Newspaper, Date of Publication, URL.

Example : Robinson, Sarah. “Biden Announces New Climate Change Policies.” The New York Times, 22 Jan. 2021, https://www.nytimes.com/2021/01/22/climate/biden-climate-change-policies.html.

Author, A. A. (Year of publication). Title of book. Publisher.

Example: Smith, J. (2010). The History of the World. Penguin Press.

Author, A. A., Author, B. B., & Author, C. C. (Year of publication). Title of article. Title of Journal, volume number(issue number), page range.

Example: Johnson, E., Smith, K., & Lee, M. (2019). The Effects of Climate Change on Agriculture. Environmental Science Journal, 10(2), 45-59.

Author, A. A. (Year of publication). Title of paper. In Editor First Initial. Last Name (Ed.), Title of Conference Proceedings (page numbers). Publisher.

Example: Garcia, M. (2018). The Importance of Early Childhood Education. In J. Smith (Ed.), Proceedings from the International Conference on Education (pp. 60-75). Springer.

Author, A. A. (Year, Month Day of publication). Title of webpage. Website name. URL

Example: Smith, J. (2022, March 1). The Benefits of Exercise. Healthline. https://www.healthline.com/health/benefits-of-exercise

Author, A. A. (Year, Month Day of publication). Title of article. Newspaper name. URL.

Example: Robinson, S. (2021, January 22). Biden Announces New Climate Change Policies. The New York Times. https://www.nytimes.com/2021/01/22/climate/biden-climate-change-policies.html

Chicago/Turabian style

Please note that there are two main variations of the Chicago style: the author-date system and the notes and bibliography system. I will provide examples for both systems below.

Author-Date system:

• In-text citation: (Author Last Name Year, Page Number)
• Reference list: Author Last Name, First Name. Year. Title of Book. Place of publication: Publisher.
• In-text citation: (Smith 2005, 28)
• Reference list: Smith, John. 2005. The History of America. New York: Penguin Press.

Notes and Bibliography system:

• Footnote/Endnote citation: Author First Name Last Name, Title of Book (Place of publication: Publisher, Year), Page Number.
• Bibliography citation: Author Last Name, First Name. Title of Book. Place of publication: Publisher, Year.
• Footnote/Endnote citation: John Smith, The History of America (New York: Penguin Press, 2005), 28.
• Bibliography citation: Smith, John. The History of America. New York: Penguin Press, 2005.

JOURNAL ARTICLES:

• Reference list: Author Last Name, First Name. Year. “Article Title.” Journal Title Volume Number (Issue Number): Page Range.
• In-text citation: (Johnson 2010, 45)
• Reference list: Johnson, Mary. 2010. “The Impact of Social Media on Society.” Journal of Communication 60(2): 39-56.
• Footnote/Endnote citation: Author First Name Last Name, “Article Title,” Journal Title Volume Number, Issue Number (Year): Page Range.
• Bibliography citation: Author Last Name, First Name. “Article Title.” Journal Title Volume Number, Issue Number (Year): Page Range.
• Footnote/Endnote citation: Mary Johnson, “The Impact of Social Media on Society,” Journal of Communication 60, no. 2 (2010): 39-56.
• Bibliography citation: Johnson, Mary. “The Impact of Social Media on Society.” Journal of Communication 60, no. 2 (2010): 39-56.

RESEARCH PAPERS:

• Reference list: Author Last Name, First Name. Year. “Title of Paper.” Conference Proceedings Title, Location, Date. Publisher, Page Range.
• In-text citation: (Jones 2015, 12)
• Reference list: Jones, David. 2015. “The Effects of Climate Change on Agriculture.” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015. Springer, 10-20.
• Footnote/Endnote citation: Author First Name Last Name, “Title of Paper,” Conference Proceedings Title, Location, Date (Place of publication: Publisher, Year), Page Range.
• Bibliography citation: Author Last Name, First Name. “Title of Paper.” Conference Proceedings Title, Location, Date. Place of publication: Publisher, Year.
• Footnote/Endnote citation: David Jones, “The Effects of Climate Change on Agriculture,” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015 (New York: Springer, 10-20).
• Bibliography citation: Jones, David. “The Effects of Climate Change on Agriculture.” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015. New York: Springer, 10-20.
• In-text citation: (Author Last Name Year)
• Reference list: Author Last Name, First Name. Year. “Title of Webpage.” Website Name. URL.
• In-text citation: (Smith 2018)
• Reference list: Smith, John. 2018. “The Importance of Recycling.” Environmental News Network. https://www.enn.com/articles/54374-the-importance-of-recycling.
• Footnote/Endnote citation: Author First Name Last Name, “Title of Webpage,” Website Name, URL (accessed Date).
• Bibliography citation: Author Last Name, First Name. “Title of Webpage.” Website Name. URL (accessed Date).
• Footnote/Endnote citation: John Smith, “The Importance of Recycling,” Environmental News Network, https://www.enn.com/articles/54374-the-importance-of-recycling (accessed April 8, 2023).
• Bibliography citation: Smith, John. “The Importance of Recycling.” Environmental News Network. https://www.enn.com/articles/54374-the-importance-of-recycling (accessed April 8, 2023).

NEWS ARTICLES:

• Reference list: Author Last Name, First Name. Year. “Title of Article.” Name of Newspaper, Month Day.
• In-text citation: (Johnson 2022)
• Reference list: Johnson, Mary. 2022. “New Study Finds Link Between Coffee and Longevity.” The New York Times, January 15.
• Footnote/Endnote citation: Author First Name Last Name, “Title of Article,” Name of Newspaper (City), Month Day, Year.
• Bibliography citation: Author Last Name, First Name. “Title of Article.” Name of Newspaper (City), Month Day, Year.
• Footnote/Endnote citation: Mary Johnson, “New Study Finds Link Between Coffee and Longevity,” The New York Times (New York), January 15, 2022.
• Bibliography citation: Johnson, Mary. “New Study Finds Link Between Coffee and Longevity.” The New York Times (New York), January 15, 2022.

Harvard referencing style

Format: Author’s Last name, First initial. (Year of publication). Title of book. Publisher.

Example: Smith, J. (2008). The Art of War. Random House.

Journal article:

Format: Author’s Last name, First initial. (Year of publication). Title of article. Title of journal, volume number(issue number), page range.

Example: Brown, M. (2012). The impact of social media on business communication. Harvard Business Review, 90(12), 85-92.

Research paper:

Format: Author’s Last name, First initial. (Year of publication). Title of paper. In Editor’s First initial. Last name (Ed.), Title of book (page range). Publisher.

Example: Johnson, R. (2015). The effects of climate change on agriculture. In S. Lee (Ed.), Climate Change and Sustainable Development (pp. 45-62). Springer.

Format: Author’s Last name, First initial. (Year, Month Day of publication). Title of page. Website name. URL.

Example: Smith, J. (2017, May 23). The history of the internet. Encyclopedia Britannica. https://www.britannica.com/topic/history-of-the-internet

News article:

Format: Author’s Last name, First initial. (Year, Month Day of publication). Title of article. Title of newspaper, page number (if applicable).

Example: Thompson, E. (2022, January 5). New study finds coffee may lower risk of dementia. The New York Times, A1.

IEEE Format

Author(s). (Year of Publication). Title of Book. Publisher.

Smith, J. K. (2015). The Power of Habit: Why We Do What We Do in Life and Business. Random House.

Journal Article:

Author(s). (Year of Publication). Title of Article. Title of Journal, Volume Number (Issue Number), page numbers.

Johnson, T. J., & Kaye, B. K. (2016). Interactivity and the Future of Journalism. Journalism Studies, 17(2), 228-246.

Author(s). (Year of Publication). Title of Paper. Paper presented at Conference Name, Location.

Jones, L. K., & Brown, M. A. (2018). The Role of Social Media in Political Campaigns. Paper presented at the 2018 International Conference on Social Media and Society, Copenhagen, Denmark.

• Website: Author(s) or Organization Name. (Year of Publication or Last Update). Title of Webpage. Website Name. URL.

Example: National Aeronautics and Space Administration. (2019, August 29). NASA’s Mission to Mars. NASA. https://www.nasa.gov/topics/journeytomars/index.html

• News Article: Author(s). (Year of Publication). Title of Article. Name of News Source. URL.

Example: Johnson, M. (2022, February 16). Climate Change: Is it Too Late to Save the Planet? CNN. https://www.cnn.com/2022/02/16/world/climate-change-planet-scn/index.html

Vancouver Style

In-text citation: Use superscript numbers to cite sources in the text, e.g., “The study conducted by Smith and Johnson^1 found that…”.

Reference list citation: Format: Author(s). Title of book. Edition if any. Place of publication: Publisher; Year of publication.

Example: Smith J, Johnson L. Introduction to Molecular Biology. 2nd ed. New York: Wiley-Blackwell; 2015.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “Several studies have reported that^1,2,3…”.

Reference list citation: Format: Author(s). Title of article. Abbreviated name of journal. Year of publication; Volume number (Issue number): Page range.

Example: Jones S, Patel K, Smith J. The effects of exercise on cardiovascular health. J Cardiol. 2018; 25(2): 78-84.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “Previous research has shown that^1,2,3…”.

Reference list citation: Format: Author(s). Title of paper. In: Editor(s). Title of the conference proceedings. Place of publication: Publisher; Year of publication. Page range.

Example: Johnson L, Smith J. The role of stem cells in tissue regeneration. In: Patel S, ed. Proceedings of the 5th International Conference on Regenerative Medicine. London: Academic Press; 2016. p. 68-73.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “According to the World Health Organization^1…”.

Reference list citation: Format: Author(s). Title of webpage. Name of website. URL [Accessed Date].

Example: World Health Organization. Coronavirus disease (COVID-19) advice for the public. World Health Organization. https://www.who.int/emergencies/disease/novel-coronavirus-2019/advice-for-public [Accessed 3 March 2023].

In-text citation: Use superscript numbers to cite sources in the text, e.g., “According to the New York Times^1…”.

Reference list citation: Format: Author(s). Title of article. Name of newspaper. Year Month Day; Section (if any): Page number.

Example: Jones S. Study shows that sleep is essential for good health. The New York Times. 2022 Jan 12; Health: A8.

Author(s). Title of Book. Edition Number (if it is not the first edition). Publisher: Place of publication, Year of publication.

Example: Smith, J. Chemistry of Natural Products. 3rd ed.; CRC Press: Boca Raton, FL, 2015.

Journal articles:

Author(s). Article Title. Journal Name Year, Volume, Inclusive Pagination.

Example: Garcia, A. M.; Jones, B. A.; Smith, J. R. Selective Synthesis of Alkenes from Alkynes via Catalytic Hydrogenation. J. Am. Chem. Soc. 2019, 141, 10754-10759.

Research papers:

Author(s). Title of Paper. Journal Name Year, Volume, Inclusive Pagination.

Example: Brown, H. D.; Jackson, C. D.; Patel, S. D. A New Approach to Photovoltaic Solar Cells. J. Mater. Chem. 2018, 26, 134-142.

Author(s) (if available). Title of Webpage. Name of Website. URL (accessed Month Day, Year).

Example: National Institutes of Health. Heart Disease and Stroke. National Heart, Lung, and Blood Institute. https://www.nhlbi.nih.gov/health-topics/heart-disease-and-stroke (accessed April 7, 2023).

News articles:

Author(s). Title of Article. Name of News Publication. Date of Publication. URL (accessed Month Day, Year).

Example: Friedman, T. L. The World is Flat. New York Times. April 7, 2023. https://www.nytimes.com/2023/04/07/opinion/world-flat-globalization.html (accessed April 7, 2023).

In AMA Style Format, the citation for a book should include the following information, in this order:

• Title of book (in italics)
• Edition (if applicable)
• Place of publication
• Year of publication

Lodish H, Berk A, Zipursky SL, et al. Molecular Cell Biology. 4th ed. New York, NY: W. H. Freeman; 2000.

In AMA Style Format, the citation for a journal article should include the following information, in this order:

• Title of article
• Abbreviated title of journal (in italics)
• Year of publication; volume number(issue number):page numbers.

Chen H, Huang Y, Li Y, et al. Effects of mindfulness-based stress reduction on depression in adolescents and young adults: a systematic review and meta-analysis. JAMA Netw Open. 2020;3(6):e207081. doi:10.1001/jamanetworkopen.2020.7081

In AMA Style Format, the citation for a research paper should include the following information, in this order:

• Title of paper
• Name of journal or conference proceeding (in italics)
• Volume number(issue number):page numbers.

Bredenoord AL, Kroes HY, Cuppen E, Parker M, van Delden JJ. Disclosure of individual genetic data to research participants: the debate reconsidered. Trends Genet. 2011;27(2):41-47. doi:10.1016/j.tig.2010.11.004

In AMA Style Format, the citation for a website should include the following information, in this order:

• Title of web page or article
• Name of website (in italics)
• Date of publication or last update (if available)
• URL (website address)
• Date of access (month day, year)

Centers for Disease Control and Prevention. How to protect yourself and others. CDC. Published February 11, 2022. Accessed February 14, 2022. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html

In AMA Style Format, the citation for a news article should include the following information, in this order:

• Name of newspaper or news website (in italics)
• Date of publication

Gorman J. Scientists use stem cells from frogs to build first living robots. The New York Times. January 13, 2020. Accessed January 14, 2020. https://www.nytimes.com/2020/01/13/science/living-robots-xenobots.html

Bluebook Format

One author: Daniel J. Solove, The Future of Reputation: Gossip, Rumor, and Privacy on the Internet (Yale University Press 2007).

Two or more authors: Martha Nussbaum and Saul Levmore, eds., The Offensive Internet: Speech, Privacy, and Reputation (Harvard University Press 2010).

Journal article

One author: Daniel J. Solove, “A Taxonomy of Privacy,” University of Pennsylvania Law Review 154, no. 3 (January 2006): 477-560.

Two or more authors: Ethan Katsh and Andrea Schneider, “The Emergence of Online Dispute Resolution,” Journal of Dispute Resolution 2003, no. 1 (2003): 7-19.

One author: Daniel J. Solove, “A Taxonomy of Privacy,” GWU Law School Public Law Research Paper No. 113, 2005.

Two or more authors: Ethan Katsh and Andrea Schneider, “The Emergence of Online Dispute Resolution,” Cyberlaw Research Paper Series Paper No. 00-5, 2000.

WebsiteElectronic Frontier Foundation, “Surveillance Self-Defense,” accessed April 8, 2023, https://ssd.eff.org/.

News article

One author: Mark Sherman, “Court Deals Major Blow to Net Neutrality Rules,” ABC News, January 14, 2014, https://abcnews.go.com/Politics/wireStory/court-deals-major-blow-net-neutrality-rules-21586820.

Two or more authors: Siobhan Hughes and Brent Kendall, “AT&T Wins Approval to Buy Time Warner,” Wall Street Journal, June 12, 2018, https://www.wsj.com/articles/at-t-wins-approval-to-buy-time-warner-1528847249.

In-Text Citation: (Author’s last name Year of Publication: Page Number)

Example: (Smith 2010: 35)

Reference List Citation: Author’s last name First Initial. Title of Book. Edition. Place of publication: Publisher; Year of publication.

Example: Smith J. Biology: A Textbook. 2nd ed. New York: Oxford University Press; 2010.

Example: (Johnson 2014: 27)

Reference List Citation: Author’s last name First Initial. Title of Article. Abbreviated Title of Journal. Year of publication;Volume(Issue):Page Numbers.

Example: Johnson S. The role of dopamine in addiction. J Neurosci. 2014;34(8): 2262-2272.

Example: (Brown 2018: 10)

Reference List Citation: Author’s last name First Initial. Title of Paper. Paper presented at: Name of Conference; Date of Conference; Place of Conference.

Example: Brown R. The impact of social media on mental health. Paper presented at: Annual Meeting of the American Psychological Association; August 2018; San Francisco, CA.

Example: (World Health Organization 2020: para. 2)

Reference List Citation: Author’s last name First Initial. Title of Webpage. Name of Website. URL. Published date. Accessed date.

Example: World Health Organization. Coronavirus disease (COVID-19) pandemic. WHO website. https://www.who.int/emergencies/disease-coronavirus-2019. Updated August 17, 2020. Accessed September 5, 2021.

Example: (Smith 2019: para. 5)

Reference List Citation: Author’s last name First Initial. Title of Article. Title of Newspaper or Magazine. Year of publication; Month Day:Page Numbers.

Example: Smith K. New study finds link between exercise and mental health. The New York Times. 2019;May 20: A6.

Purpose of Research Paper Citation

The purpose of citing sources in a research paper is to give credit to the original authors and acknowledge their contribution to your work. By citing sources, you are also demonstrating the validity and reliability of your research by showing that you have consulted credible and authoritative sources. Citations help readers to locate the original sources that you have referenced and to verify the accuracy and credibility of your research. Additionally, citing sources is important for avoiding plagiarism, which is the act of presenting someone else’s work as your own. Proper citation also shows that you have conducted a thorough literature review and have used the existing research to inform your own work. Overall, citing sources is an essential aspect of academic writing and is necessary for building credibility, demonstrating research skills, and avoiding plagiarism.

Advantages of Research Paper Citation

There are several advantages of research paper citation, including:

• Giving credit: By citing the works of other researchers in your field, you are acknowledging their contribution and giving credit where it is due.
• Strengthening your argument: Citing relevant and reliable sources in your research paper can strengthen your argument and increase its credibility. It shows that you have done your due diligence and considered various perspectives before drawing your conclusions.
• Demonstrating familiarity with the literature : By citing various sources, you are demonstrating your familiarity with the existing literature in your field. This is important as it shows that you are well-informed about the topic and have done a thorough review of the available research.
• Providing a roadmap for further research: By citing relevant sources, you are providing a roadmap for further research on the topic. This can be helpful for future researchers who are interested in exploring the same or related issues.
• Building your own reputation: By citing the works of established researchers in your field, you can build your own reputation as a knowledgeable and informed scholar. This can be particularly helpful if you are early in your career and looking to establish yourself as an expert in your field.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Design – Types, Methods and Examples

Research Paper Title – Writing Guide and Example

Research Paper Introduction – Writing Guide and...

Research Paper Conclusion – Writing Guide and...

Trends in mathematics education and insights from a meta-review and bibliometric analysis of review studies

• Original Paper
• Open access
• Published: 15 May 2024

Cite this article

You have full access to this open access article

• Mustafa Cevikbas   ORCID: orcid.org/0000-0002-7844-4707 1 ,
• Gabriele Kaiser 2 , 3 &
• Stanislaw Schukajlow 4  

Review studies are vital for advancing knowledge in many scientific fields, including mathematics education, amid burgeoning publications. Based on an extensive consideration of existing review typologies, we conducted a meta-review and bibliometric analysis to provide a comprehensive overview of and deeper insights into review studies within mathematics education. After searching Web of Science, we identified 259 review studies, revealing a significant increase in such studies over the last five years. Systematic reviews were the most prevalent type, followed by meta-analyses, generic literature reviews, and scoping reviews. On average, the review studies had a sample size of 99, with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines commonly employed. Despite certain studies offering nuanced distinctions among review types, ambiguity persisted. Only about a quarter of the studies explicitly reported employing specific theoretical frameworks (particularly, technology, knowledge, and competence models). Co-authored publications were most common within American institutions and the leading countries are the United States, Germany, China, Australia, and England in publishing most review studies. Educational review journals, educational psychology journals, special education journals, educational technology journals, and mathematics education journals provided platforms for review studies, and prominent research topics included digital technologies, teacher education, mathematics achievement, and learning disabilities. In this study, we synthesised a range of reviews to facilitate readers’ comprehension of conceptual congruities and disparities across various review types, as well as to track current research trends. The results suggest that there is a need for discipline-specific standards and guidelines for different types of mathematics education reviews, which may lead to more high-quality review studies to enhance progress in mathematics education.

Avoid common mistakes on your manuscript.

1 Introduction

Comprehensive literature reviews serve as foundational pillars for advancing scholarly discourse, offering critical insights into existing research and shaping future inquiries across disciplines. In the realm of academic writing, spanning from journal articles to dissertations, literature reviews are highly regarded for their capacity to synthesize knowledge, identify gaps, and provide a cohesive framework for understanding complex topics (Boote & Beile, 2005 ). Moreover, reviews play a significant role in academia by setting new research agendas and informing decision-making processes in practice, policy, and society (Kunisch et al., 2023 ).

As empirical and theoretical research burgeons in diverse fields, the need for literature review studies has become even more pronounced, facilitating a deeper understanding of specific research areas or themes (Hart, 2018 ; Nane et al., 2023 ). Additional factors contributing to the popularity of review studies in recent years include the rise of specialized review journals (Kunisch et al., 2023 ), challenges associated with conducting various types of empirical studies during the prolonged COVID-19 crisis (Cevikbas & Kaiser, 2023 ), and a competitive research climate wherein factors such as impact factors and citations hold significant weight (Ketcham & Crawford, 2007 ). Review studies are particularly attractive as they often garner a substantial number of citations, thereby enhancing researchers’ visibility and scholarly impact (Grant & Booth, 2009 ; Taherdoost, 2023 ).

The importance of review studies has been duly acknowledged in mathematics education, as evidenced by the inclusion of review papers in thematically oriented special issues of journals such as ZDM– Mathematics Education (Kaiser & Schukajlow, 2024 ), which has been originally founded as review journal. Several upcoming or already published special issues of ZDM– Mathematics Education , which emphasise ‘reviews on important themes in mathematics education’, highlight the importance of review studies as valuable contributions to the field.

The proliferation of literature reviews has increased interest in developing typologies to categorise them and understand different literature review approaches (Grant & Booth, 2009 ; Paré et al., 2015 ; Schryen & Sperling, 2023 ). Despite its significance, there remains a notable lack of research aimed at comprehensively understanding review studies within the field of mathematics education from a meta-perspective. In response to this gap, we conducted a systematic meta-review with the aim of providing an overview of different types of review studies in mathematics education over the past few decades and consolidating insights from multiple high-level review studies (Becker & Oxman, 2008 ; Schryen & Sperling, 2023 ). Meta-reviews offer concise yet comprehensive synopses and curated lists of pertinent reviews, adeptly addressing the perennial challenge of balancing thorough coverage with focused specificity (Grant & Booth, 2009 ).

In addition, we applied bibliometric analysis as a valuable tool for identifying research trends, progress, reliable sources, and future directions within the field. The bibliometric analysis aids in identifying hot research topics and trends (Song et al., 2019 ), assessing progress, identifying reliable sources, recognising major contributors, and predicting future research success (Geng et al., 2017 ). Furthermore, it helps researchers to pinpoint potential topics, suitable institutions for cooperation, and potential scholars for scientific collaboration (Martínez et al., 2015 ). By combining a meta-review and bibliometric analysis, we aim to offer a comprehensive overview of and deeper insights into state-of-the-art review studies within mathematics education.

Specifically, we seek to understand how the distribution and development of literature review studies in mathematics education have evolved over the years, examining factors such as publication years, publishers, review types, sample sizes, and the use of theoretical or conceptual frameworks. Additionally, we aim to assess adherence to review study guidelines and protocols, providing insights into the rigor and quality of research methodologies employed, particularly in light of the lack of clear guidance on producing rigorous and impactful literature reviews (Kunisch et al., 2023 ).

Furthermore, we endeavour to identify authors who have made contribution to the field of mathematics education through review studies, as well as those whose work is most frequently cited. We also identify co-authorship network analysis as understanding research networks allows researchers to identify potential collaborators and build partnerships with other scholars in various countries. Collaborative research endeavours can lead to enhanced research outcomes, broader dissemination of findings, and increased opportunities for funding and professional development. It can also highlight interdisciplinary connections and collaborations within and across fields, leading to innovative approaches and solutions to complex research questions (RQs) that transcend disciplinary boundaries.

Moreover, we analysed the distribution of common keywords across review studies, identifying focal subjects and thematic areas prevalent in mathematics education research. This analysis can provide valuable insights into key topics and trends shaping the field, guiding future research directions and priorities.

Lastly, we identified the most cited review papers in mathematics education and the journals in which they have been published, recognizing seminal works and influential publications that have contributed to the advancement of the field.

Overall, in light of the preceding discourse, we addressed the following RQs to uncover the characteristics of review studies, identify research trends, and delineate future research directions in mathematics education:

How can the distribution and development of review studies in mathematics education over time be characterised according to the number of manuscripts, publishers, review types, sample sizes, the use of theoretical or conceptual frameworks, and adherence to review study guidelines and protocols?

Which authors have contributed the largest number of review studies in mathematics education, and which authors’ review papers are most frequently cited in the literature?

From which countries are the authors of the review studies in mathematics education?

Which author keywords can be identified in the review studies in mathematics education, how are these keywords distributed across the analysed review studies, and which focal topics do these keywords indicate?

What are the most cited review papers in mathematics education, and in which journals have they been published?

2 Literature review studies and review typologies– background information

In this chapter, we provide a thorough analysis of different typologies for review studies, as we seek to elucidate the primary characteristics of various review studies conducted within mathematics education (Sect.  2.1 ). This effort led to the identification of 28 review types presented in Table  1 , which were used in the current study’s literature search processes to access existing review studies and the analysis of identified studies in the field of mathematics education. Furthermore, we discuss the advancement of guidelines and protocols, highlighting their role in shaping the conduct of review studies (Sect. 2.2). Finally, we conclude the chapter by underscoring the importance and potential impact of meta-reviews and bibliometric analyses in the context of mathematics education (Sect. 2.3).

2.1 Literature review typologies

Researchers have defined and emphasized different review types with distinct features, objectives, and methodologies. To address the challenge of ambiguous review categorisations, we conducted an extensive search and analysis of the literature on Web of Science (WoS) using the search strings ‘typology of reviews’ and ‘taxonomy of reviews’ to search the titles of studies. We focused particularly on influential theoretical, conceptual, and review papers discussing the taxonomy and typology of review studies and recent advances driven by scholars across diverse fields.

2.1.1 Seminal work by Grant and Booth ( 2009 ) on the discourse of literature review typologies

The categorisation of literature reviews has been profoundly influenced by the seminal work of Grant and Booth ( 2009 ), on which typologies of literature reviews are often based. Their paper garnered significant attention, with over 10,304 citations as of 20 April 2024 according to Google Scholar. Originally in the field of health information theory and practice, these authors founded their work on earlier approaches, notably Cochrane’s ( 1979 ) approach. Grant and Booth ( 2009 ) claimed that the developed typology could standardise the diverse terminology used. They distinguished 14 review types, which we summarise below, highlighting the main scope and search methodologies (Grant & Booth, 2009 , pp. 94–95):

A critical review ‘goes beyond mere description of identified articles and includes a degree of analysis and conceptual innovation’; no formalised or systematic approach is required because the aim of such a review is ‘to identify conceptual contributions to embody existing or derive new theory’.

A generic literature review incorporates ‘published materials that provide examination of recent of current literature’; comprehensive searching may or may not be necessary.

A mapping review/systematic mapping is used to ‘categorize existing literature’ and identify gaps in the research literature. The completeness of a search is important, but no formal quality assessment is needed.

A meta-analysis is a ‘technique that statistically combines the results of quantitative studies to provide a more precise effect of the results’; a comprehensive search is conducted based on the inclusion and exclusion criteria.

A mixed-studies review/mixed-methods review incorporates ‘a combination of review approaches, for example combining quantitative with qualitative research… and requires a very sensitive search’.

An overview is a generic term describing a ‘summary of the… literature that attempts to survey the literature and describe its characteristics’; it may or may not include comprehensive searching and quality assessment.

A qualitative systematic review/qualitative evidence synthesis is a ‘method for integrating or comparing the findings from qualitative studies’, and it may involve selective sampling.

A rapid review comprises an ‘assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research’; a characteristic of such a review is that the ‘completeness of searching is determined by time constraints’.

A scoping review is a ‘preliminary assessment of the potential size and scope of available research literature’, with the ‘completeness of searching determined by time/scope constraints’.

A state-of-the-art review ‘tend[s] to address more current matters in contrast to other combined retrospective and current approaches’ and ‘aims for comprehensive searching of current literature’.

A systematic review ‘seeks to systematically search for, appraise and synthesise research evidence’ and should be comprehensive and based on inclusion/exclusion criteria.

A systematic search and review ‘combines [the] strengths of critical review with a comprehensive search process’, typically addressing broad questions to produce ‘best evidence synthesis’ based on ‘exhaustive, comprehensive searching’.

A systematised review ‘include[s] elements of systematic review process while stopping short of systematic review’, ‘typically conducted as postgraduate student assignment’; it ‘may or may not include comprehensive searching’.

An umbrella review ‘specifically refers to review compiling evidence from multiple reviews into one accessible and usable document’ via ‘identification of component reviews, but no search for primary studies’. ‘Primary studies’ refer to original research studies or individual studies conducted by researchers to gather data first-hand.

Booth with colleagues later expanded the typology by introducing the concept of a review family construct and amalgamating various types of reviews for further refinement, such as traditional reviews, systematic reviews, review of reviews, rapid reviews, mixed-methods reviews, and purpose-specific reviews (for details, see Sutton et al., 2019 ).

2.1.2 Further development of the review typologies

Many classifications for review studies have been developed, and in the following section, we present more recent approaches. Paré et al. ( 2015 ), in another highly cited study (2,059 Google Scholar citations as of 20 April 2024) considered seven recurrent dimensions: the goal of the review, the scope of the review questions, the search strategy, the nature of the primary sources, the explicitness of the study selection, quality appraisal, and the methods used to analyse/synthesise the findings. Based on these dimensions, they formulated nine different literature review types: narrative reviews, descriptive reviews, scoping/mapping reviews, meta-analyses, qualitative systematic reviews, umbrella reviews, critical reviews, theoretical reviews, and realist reviews.

In Paré et al.’s ( 2015 ) classification, the review categories that differ from Grant and Booth’s ( 2009 ) classification are theoretical reviews, realist reviews, narrative reviews, and descriptive reviews, which we therefore describe them briefly. A theoretical review draws on conceptual and empirical studies to develop a conceptual framework or model using structured approaches, such as taxonomies, to discover patterns or commonalities. The aim of a realist review (also called a meta-narrative review) is to formulate explanations; such reviews ‘are theory-driven interpretative reviews which were developed to inform, enhance, extend, or alternatively supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision making’ (Paré et al., 2015 , p. 188). The purpose of a narrative review is to survey the existing literature on a particular subject or topic without necessarily seeking generalisations or cumulative insights from the material reviewed (Davies, 2000 ). Typically, such reviews do not detail the underpinning review processes or involve systematic and exhaustive searches of all pertinent literature. This category resembles Grant and Booth’s ( 2009 ) description of ‘literature reviews’ and overlaps with Samnani et al.’s ( 2017 ) narrative reviews, literature reviews, and overviews, resulting in a somewhat ambiguous typology. The aim of a descriptive review is to identify patterns and trends across a set of empirical studies within a specific research field, encompassing pre-existing propositions, theories, methodological approaches, or findings. To accomplish this objective, descriptive reviews collect, structure, and analyse numerical data that reflect the frequency distribution of research elements.

MacEntee ( 2019 ), Samnani et al. ( 2017 ), Schryen et al. ( 2020 ), and Taherdoost ( 2023 ) corroborated Grant and Booth’s ( 2009 ) and Paré et al.’s ( 2015 ) classifications, identifying various common review categories (see Table  1 ). In Samnani et al.’s ( 2017 ) classification, a distinct review type based on the previously mentioned categories is meta-synthesis , the aim of which is to provide explanations for phenomena, in contrast to meta-analysis, which focuses on quantitative outcomes.

Later, Schryen and Sperling ( 2023 ) introduced a slightly revised typology of literature review studies, which they applied to a meta-review of operations research. Their study distinguished nine types of literature reviews, newly introduced categories included tutorial reviews, selective reviews, algorithmic reviews, computational reviews, and meta-reviews. The objective of a tutorial review is to offer a research-oriented summary of principles, mathematical fundamentals, and concepts, aiming to inspire and direct future research endeavours. The authors’ emphasis on foundational aspects has often provided a launching pad for research advances. A selective review typically has a limited scope because it is not based on a thorough search of all relevant literature. This type of review concentrates on specific segments of the literature, such as journals, time periods, methodologies, or issues, to delve deeper into specific questions and phenomena. An algorithmic review focuses on advances in algorithms and frameworks in the literature that address a spectrum of problems. It employs either selective or comprehensive search strategies, predominantly examining algorithm-related sources. A computational review investigates algorithms and/or parameterisations proposed in the literature, largely considering implementations and computational studies, measurement efficiency, effectiveness, and different forms of robustness. Finally, Schryen and Sperling ( 2023 ) defined a meta-review as an overview of systematic reviews or a systematic review of reviews and pointed out that a meta-review can also be called an umbrella review (which is the case by Grant and Booth), again confirming the fuzzy nature of the currently available typologies. According to Schryen and Sperling ( 2023 ), meta-reviews primarily aim to furnish descriptive overviews of literature reviews, serving as tertiary studies that integrate evidence from multiple (qualitative or quantitative) reviews into unified and user-friendly documents (Becker & Oxman, 2008 ; Paré et al., 2015 ). In contrast to the previously mentioned perspectives, Schryen and Sperling ( 2023 ) argued that meta-reviews are not limited to addressing specific research questions but can also address a wide range of enquiries.

Chigbu et al. ( 2023 , pp. 5–6) emphasised that there ‘is a continuum of literature types’ (p. 4) and distinguished twelve different types of literature reviews, six of which were not covered by the classifications provided by previously mentioned studies: integrated reviews, interpretative reviews, iterative reviews, semi-systematic reviews, and bibliometric reviews. According to their approach, an integrative review builds ‘new knowledge based on the existing body of literature following a rationalist perspective’, an interpretative review ‘interprets what other scholars have written to put into specific perspectives’, and an iterative review is an ‘algorithm-based approach performed to collate all studies in a specific field of research’. Moreover, a meta-synthesis review examines and analyses qualitative study findings and is often employed to clarify specific concepts. Additionally, a semi-systematic review analyses the data and findings of other studies to address specific research inquiries, using a partial systematic review methodology. Lastly, a bibliometric review systematically examines the literature on a specific subject or research discipline by quantitatively measuring indicators such as authors, citations, journals, countries, and years of publications.

As previously noted in this paper, this detailed description of review types is instrumental in facilitating our investigation of various review studies in the realm of mathematics education.

2.2 Advancements in guidelines and protocols for review studies

Various researchers have developed guidelines, protocols, and statements to assist authors in conducting, evaluating, and reporting their review studies. This academic endeavour has predominantly focused on enhancing the rigour and transparency of systematic reviews, meta-analyses, and, more recently, scoping reviews. For instance, the population, intervention, comparison, and outcomes (PICO) model, originally conceived to support evidence-based healthcare, serves as a cornerstone for establishing review criteria, crafting research questions and search strategies, and delineating the characteristics of included studies or meta-analyses (Richardson et al., 1995 ). In response to the observed deficiencies in reporting standards within meta-analyses, an international consortium introduced the Quality of Reporting of Meta-Analyses (QUOROM) statement in 1996, primarily to enhance the reporting quality of meta-analyses involving randomised controlled trials (Moher et al., 1999 ). Subsequently, Moher et al. ( 2009 ) updated these guidelines, which are now known as the PRISMA guidelines, and incorporated various conceptual and methodological advances in systematic reviews and meta-analyses. Additionally, Shea et al. ( 2007 ) introduced the Assessment of Multiple Systematic Reviews (AMSTAR) checklist to evaluate methodological quality and guide the conduct of systematic reviews, while Grant and Booth ( 2009 ) developed the search, appraisal, synthesis, and analysis (SALSA) framework to analyse and characterise review types. Most recently, Page et al. ( 2021 ) updated the PRISMA guidelines, providing updated reporting standards that reflect advances in methods for identifying, selecting, appraising, and synthesising studies, with the aim of promoting more transparent, complete, and accurate reporting of systematic reviews and meta-analyses. An extension of PRISMA guidelines for scoping reviews, known as PRISMA-ScR, aids readers in understanding relevant terminology, core concepts, and key items for reporting scoping reviews (Tricco et al., 2018 ). Despite the value of these efforts, further studies are warranted, particularly comprehensive guidelines for each type of review studies.

2.3 Literature reviews in mathematics education

The preceding section delineates various types of review studies, underscoring their key methodological attributes. Within the realm of mathematics education, akin to other disciplines, literature review studies, particularly systematic reviews, and meta-analyses, received considerable attention (Cevikbas et al., 2022 ; Cevikbas & Kaiser, 2023 ; Kaiser & Schukajlow, 2024 ). However, the understanding of the prevailing characteristics of review studies in mathematics education, including prevalent review types, trends, gaps, and avenues for future improvement, remains limited.

Meta-reviews can offer a promising avenue for pinpointing research gaps, evaluating evidence quality, and informing policy and intervention strategies and guiding evidence-based decision-making processes by synthesizing findings from multiple review studies (Schryen & Sperling, 2023 ). In addition to meta-reviews, the bibliometric analyses serve to ascertain the scope of prior research, discern contemporary review trends, identify literature gaps, and propose future research agendas (Chigbu et al., 2023 ). While meta-reviews provide a comprehensive assessment of the literature, bibliometric analyses aid in systematically screening literature on a specific subject, topic, or research discipline by quantitatively measuring various indicators such as authors, citations, journals, countries, and years of publication. These methodological approaches hold promise for instituting a systematic, transparent, and reproducible review process, thereby augmenting the overall quality of reviews in mathematics education. Bibliometric techniques serve as valuable tools in literature reviews, guiding researchers by pinpointing influential works and impartially mapping the research landscape prior to in-depth exploration (Zupic & Cater, 2015 ).

Despite their significance, meta-reviews and bibliometric analyses remain seldom within the domain of mathematics education, signifying a substantial gap in the literature. Our comprehensive literature review underscores an urgent need for meta-review studies encompassing literature review studies in the realm of mathematics education. Additionally, while no bibliometric analysis study specifically focusing on review studies in mathematics education was identified, several bibliometric studies in mathematics education on various topics were noted, such as mathematics anxiety (Radevic & Milovanovic, 2023 ), problem-solving (Suseelan et al., 2022 ), and teacher noticing (Wei et al., 2023 ).

Overall, there exists a compelling need for meta-reviews enriched by bibliometric analyses to explore the current state of literature review research in mathematics education, and the current study aims to address this gap in a timely manner.

3 Methodology

3.1 literature search and manuscript selection process.

In this study, following the latest PRISMA guidelines (Page et al., 2021 ), we aimed to conduct a systematic review of previous review studies in mathematics education. Specifically, we employed the meta-review (umbrella review) method supplemented by bibliometric analyses. We processed the manuscript selection under three stages: identification, screening, and included.

3.1.1 Identification

On 10 January 2024 (last access), we conducted an extensive literature search using the WoS electronic database, which includes publications in high-ranking peer-reviewed journals and is widely acknowledged as a primary source of review and bibliometric data that meet high quality standards (Korom, 2019 ). WoS facilitates effective literature searches, supports various information purposes, and aids research topic mapping, trend monitoring as well as scholarly activity analysis (Birkle et al., 2020 ).

To comprehensively identify potentially relevant review studies in mathematics education, we developed an inclusive search query targeting specific terms in the titles, abstracts, and keywords of papers. The query comprised terms that we extracted from the typologies of literature reviews described in Chap. 2, particularly the more general, commonly used types of reviews:

( TOPIC ) ((literature review*OR literature survey* OR systematic review* OR rapid review* OR scoping review* OR critical review* OR meta-analysis OR narrative review* OR umbrella review* OR meta review* OR meta-review OR bibliometric review OR bibliometric analysis OR mapping review OR mixed-methods review OR integrative review OR interpretative review OR iterative review OR meta-synthesis OR descriptive review OR theoretical review OR realist review OR selective review OR algorithmic review OR computational review)) AND ( TOPIC ) ((math* OR geometry OR algebra OR calculus OR probability OR statistics OR arithmetic).

Based on these search strings, we conducted an online search that initially yielded 63,462 records.

3.1.2 Screening

In this stage, we applied data cleaning filters based on the manuscript inclusion and exclusion criteria (see Table  2 ). First, we electronically filtered the identified records based on language, resulting in the retention of 61,787 papers published in English. Subsequently, we narrowed down the selection to 10,098 papers using the following five categories of research areas within the WoS: ‘education/educational research, psychology, social sciences other topics, mathematics, or science technology other topics’. Following this categorisation, we further refined the dataset by excluding non-review papers and accessing 3,344 records within the ‘review article’ and ‘early access’ categories of the WoS database. We categorised records lacking a final publication date that had undergone peer review and acceptance as ‘early access’. Notably, to comprehensively capture publication trends, we imposed no restrictions on the publication years of the studies. In the subsequent phase, a meticulous manual screening of the titles, abstracts, and keywords of 3,344 papers led to the identification of 357 studies in mathematics education.

3.1.3 Included

Ultimately, after an extensive review of the full-text versions of initially identified 357 papers, 259 eligible review articles remained for analysis as these papers fulfilled our criteria comprehensively (see the Appendix for the list of included studies; see Fig.  1 for the flow diagram of the entire manuscript selection process). Subsequently, as detailed below, the data analysis process commenced with the inclusion of these eligible review papers in mathematics education.

Flow diagram of the manuscript selection process

3.2 Data analysis

After incorporating 259 studies into this meta-review and bibliometric analysis, we compiled the identified records into a marked list on WoS. Subsequently, we exported the records into Excel, EndNote, and plain text file formats for analysis. The analysis consisted of content analysis and bibliometric analysis (see Fig.  2 , adapted from Wei et al., 2023 ).

For the content analysis, we meticulously organised the records using EndNote reference management software and Excel worksheets. We scrutinised the full-text versions of all included articles, coding them based on (1) publication year, (2) publisher, (3) review type, (4) number of included studies (sample size), (5) guidelines and protocols for the article selection process, and (6) the theoretical and conceptual framework of the studies.

Our coding manual, informed by prior studies (Cevikbas et al., 2022 , 2024 ), guided this process (see appendix for a sample of the coding manual). After completing the content analysis coding procedure, 20% of the papers ( n  = 52) were double-coded based on the initial coding protocol. The intercoder reliability, gauged at 0.92, signifies the presence of a coding system that exhibits satisfactory reliability (Creswell, 2013 ). Any discrepancies were addressed through discussions among the coders until consensus was reached.

For the bibliometric analysis, we employed VOSviewer software (version 1.6.20), which is widely recognised and extensively used in various fields, including the educational sciences (van Eck & Waltman, 2010 ). Chigbu et al. ( 2023 ) pointed out that the WoS database plays a pivotal role in facilitating bibliometric analyses across various disciplines. These analyses help establish trends in the development and application of knowledge within specific subjects and disciplines.

In our study, the bibliometric network presented in the results chapter consists of nodes and edges, with nodes representing entities such as publications, journals, researchers, or keywords. Edges denote relationships between pairs of nodes, indicating not only the presence or absence of connections but also conveying the intensity or strength of relationships (van Eck & Waltman, 2010 ). For distance-based approaches, the positioning of nodes in a bibliometric network reflects their approximate relatedness based on proximity.

Utilising VOSviewer software, we conducted (1) co-authorship analysis (authors and countries) to elucidate collaboration patterns and contributions, (2) co-occurrence analysis (focusing Author Keywords) to scrutinise knowledge structures and the distribution and development of key research topics in mathematics education, and (3) citation analysis to delve deeper into research influences and citation networks, drawing insights from the documents and sources.

This multifaceted approach allowed us to gain a comprehensive understanding of the bibliometric landscape and unravel collaborative structures, thematic foci, and the influence of key works on mathematics education.

Analytical process for this study

In this chapter, we present the key results of the meta-review and bibliometric analyses divided into two main categories: an overview of the review studies in mathematics education based on the content analysis, addressing RQ1, and the results of the bibliometric analysis, addressing RQ2 – RQ5.

4.1 Overview of review studies in mathematics education (RQ1)

To discern the research trends and essential attributes of review studies in mathematics education, we conducted a content analysis within our meta-review to examine the 259 included review studies. Our analysis encompassed publication years, publishers, review types, guidelines, protocols used, sample sizes, and the theoretical and conceptual frameworks employed in these review studies. A general overview of the included studies is presented in Table  3 .

Our literature search with no restriction on the publication years yielded review studies published between 1996 and 2023, with a notable increase within the last five years (2019–2023, see Fig.  3 ).

Distribution of publications from 1996 to 2023

The analysis showed that the Springer Group is the primary publisher of review articles in mathematics education, followed by Taylor & Francis, Elsevier, Sage, Frontiers, Wiley, MDPI, and the American Psychological Association (APA) (see Table  4 ). Other publishers published the remaining review articles ( n  = 43). This result may be attributed to the predominance of mathematics education journals published by Springer within the WoS database.

To explore the prevailing types of review studies in mathematics education, we scrutinised the review methodologies of the included studies, considering the review types presented earlier in Table  1 . The findings revealed that researchers conducted (according to their own classification) 10 different types of reviews in mathematics education as outlined in Fig.  4 .

Types of review studies Note: *systematic reviews and meta-analyses ( n  = 6), systematic reviews and bibliometric analyses ( n  = 3), meta-analyses and narrative reviews ( n  = 2), and meta-analysis and critical review ( n  = 1)

Our analysis did not yield further review types in mathematics education. Time-related analysis showed that recent studies were systematic reviews, meta-analyses, literature reviews, and scoping reviews, whereas early examples of review studies in mathematics education were primarily narrative or critical reviews or were not explicitly classified according to review type by their authors. Figure  4 shows that some researchers ( n  = 18) described their studies as literature reviews using Grant and Booth’s ( 2009 ) generic term, without providing further details about the type of review.

To comprehend the methodologies employed by researchers to conduct reviews and select eligible studies, we conducted an analysis of the guidelines and protocols the researchers used. The findings revealed that the PRISMA guidelines were the most frequently employed ( n  = 121), aligning with the distribution of review types—PRISMA guidelines are basically recommended for systematic reviews and meta-analyses (Page et al., 2021 ). For scoping reviews, the guidelines developed by Arksey and O’Malley ( 2005 ) were the most prevalent and were used in seven studies. In six instances, researchers applied various guidelines (e.g. PICO or SALSA guidelines) sourced from the literature. Almost half of the studies ( n  = 125) did not specify the use of guidelines for conducting literature searches and selecting eligible studies. Additionally, three studies aimed to provide protocols for conducting review studies. Furthermore, seven studies were preregistered as review studies, following the Open Science Framework (OSF) and/or the International Prospective Register of Systematic Reviews (PROSPERO) protocol.

A prevalent discourse among researchers in review studies revolved around determining the most suitable number of studies to include in reviews. Our results revealed that the sample sizes of the included studies (i.e. the number of primary studies) in the field of mathematics education ranged from 8 to 3,485. Unfortunately, this information was not reported in 19 review articles. In the remaining 240 review articles, the average was 99 included studies, with an overall total of 23,761. Most of the studies ( n  = 202) had sample sizes of less than 100, with an average of 34 (see Table  5 ). Although we harboured concerns that the review studies identified in this investigation might not have been aptly named and conceptualised by their authors, we deliberately refrained from addressing this issue because it fell outside the scope of our study. While including a substantial number of studies is common and potentially suitable for bibliometric analyses and meta-analyses, conducting a systematic review, scoping review, or narrative review that critically analyses exceptionally high volumes of studies may pose challenges. In this meta-review, for example, we observed that five articles included more than 1,000 studies in the review process. Two studies, enriched by bibliometric analysis, took this approach, while another study was identified by the authors as a scoping review with a sample size of 2,433. Additionally, two studies were labelled as systematic reviews with sample sizes of 1,968, and 3,485, respectively.

Finally, we conducted a content analysis to scrutinise the theoretical and conceptual frameworks underpinning the included review studies in mathematics education. The findings revealed that out of 259 review studies, only 61 incorporated any theoretical or conceptual framework. Notably, a subset of studies ( n  = 14) was based on technology-related conceptual frameworks, such as Technological Pedagogical Content Knowledge (TPACK), frameworks pertaining to augmented and virtual reality, embodied design, artificial intelligence, big data, and the European Framework for the Digital Competence for Educators (DigCompEdu). Another prevalent category ( n  = 10) relied on frameworks related to the knowledge and competence of individuals (e.g. teachers and/or students), encompassing models such as the competence as continuum framework, TPACK, the didactic-mathematical knowledge and competencies model, mathematical content knowledge, pedagogical content knowledge, mathematical knowledge for teaching, teacher noticing competence, and an integrative model for the study of developmental competencies in minority children. Bronfenbrenner’s ecological theories (e.g. ecological theory of human development, bioecological model of human development, ecological systems theory, and ecological dynamics—a blend of dynamic-systems theory and ecological psychology) were employed by researchers in five review studies in mathematics education. In a limited subset of the studies, social and cultural theories (e.g. sociocultural theory, social learning theory, and cultural activity theory ( n  = 3)), cognitive theories (e.g. cognitive developmental theory ( n  = 2)), affective theories (e.g. self-determination theory and expectancy-value theory ( n  = 2)), linguistic theories ( n  = 2), and constructivist theories ( n  = 2) were used as frameworks. Additionally, researchers used conceptual frameworks concerning computational thinking ( n  = 2) and engagement ( n  = 3) alongside a few less frequently reported frameworks.

4.2 Results of the bibliometric analysis (RQ2–RQ5)

To identify productive and most cited authors, important journals, and countries of origin of the authors, along with the underlying research collaborations between researchers and countries, as well as research trends and key topics of review studies in mathematics education, we conducted a bibliometric analysis based on co-authorship, co-occurrence, and citations.

4.2.1 Co-authorship analysis

We conducted a co-authorship analysis according to authors and countries within the units of analysis.

Co-authorship and author analysis

The bibliometric analysis, using VOSviewer, revealed that 761 authors contributed to mathematics education, each of whom conducted at least one review study. The review papers were predominantly authored through collaboration, with most being written by two authors (30,2%), followed by three authors (20,2%), four authors (19,4%), a single author (10,1%), five authors (8,9%), six authors (6,2%), seven authors (3,5%), eight authors (1,6%), and nine authors (0,4%). These results showed that researchers primarily collaborate with their colleagues in conducting review studies—a practice vital for reducing workload and enhancing the quality of analyses—with the advantage of incorporating the various perspectives of different authors.

Table  6 highlights the top 17 authors who published a minimum of three review papers each. Notably, Lieven Verschaffel is the only scholar present in both lists of prolific and highly cited authors. The researchers listed in Table  7 , except Lieven Verschaffel, contributed to the field with a single review study. Consequently, while these researchers rank among most cited authors, the low total link strength (TLS) values indicate their limited collaboration with other scholars. The TLS was automatically calculated by VOSviewer and represents the overall intensity of co-authorship connections between a particular researcher and others. According to the co-authorship analysis, it is also noteworthy that many of the highly cited authors’ review studies typically date back over ten years, which is expected as citations tend to accumulate gradually over time. The results from the detailed citation analyses provided in Sect. 4.2.3.

Upon examining the research domains of prolific and highly cited authors, we found a diverse range of topics spanning mathematics education, psychology, educational psychology, special education, and neuroscience. This diversity highlights the interdisciplinary nature of research in mathematics education, with contributions to the literature review studies from psychologists and special education and neuroscience scholars alongside mathematics educators.

Figure  5 shows a co-authorship network map for the authors of the included review studies based on the TLS. We set the minimum number of documents for an author as one, which encompassed 761 authors who contributed to review papers in mathematics education. This bibliometric co-authorship analysis yielded 51 clusters, each containing a minimum of five items (researchers). The prominent co-authorship clusters included a green cluster (led by Lieven Verschaffel), a blue cluster (led by Gabriele Kaiser and Mustafa Cevikbas), a red cluster (led by Nelson Gena), and a yellow cluster (led by Diane P. Bryant). Nelson Gena had the highest number of collaboration links, with a TLS of 26, followed by Lieven Verschaffel (TLS = 22), Gabriele Kaiser (TLS = 16), Soyoung Park (TLS = 16), Tassia Bradford (TLS = 13), Diane P. Bryant (TLS = 12), Johannes König (TLS = 12), Mikyung Shin (TLS = 12), Min Wook Ok (TLS = 12), Bert de Smedt (TLS = 10), Fred Spooner (TLS = 10), Jihyun Lee (TLS = 10), Mustafa Cevikbas (TLS = 10), Rosella Santagata (TLS = 10), Sarah R. Powell (TLS = 10), and Thorsten Scheiner (TLS = 10).

Co-authorship and author networks

Co-authorship and country analysis

We conducted a co-authorship–country analysis, setting the minimum number of documents for a country as one, and identified 50 countries. This selection resulted in five clusters, each containing a minimum of five items (countries).

The most prominent cluster was the green cluster, encompassing eight countries from various global regions: the United States (US; TLS = 30), Germany (TLS = 23), Australia (TLS = 21), China (TLS = 11), South Korea (TLS = 6), Sweden (TLS = 4), New Zealand (TLS = 2), and Jordan (TLS = 1). The US dominated research collaborations both within this cluster and overall.

The red cluster included nine countries, predominantly Nordic and European countries: Norway (TLS = 13), Finland (TLS = 7), Belgium (TLS = 6), the Netherlands (TLS = 6), Lithuania (TLS = 1), Portugal (TLS = 1), Luxembourg (TLS = 1), Scotland (TLS = 1), and Israel (TLS = 1).

The yellow cluster contained seven countries: Canada (TLS = 7), Malaysia (TLS = 7), Denmark (TLS = 3), Libya (TLS = 2), Singapore (TLS = 2), Indonesia (TLS = 1), and the United Arab Emirates (TLS = 1).

The blue cluster primarily highlighted European collaborations and included seven countries: England (TLS = 22), Switzerland (TLS = 4), Italy (TLS = 3), France (TLS = 3), Greece (TLS = 1), Chile (TLS = 1), and Saudi Arabia (TLS = 1).

Lastly, the purple cluster represented a network of predominantly South and North American countries featuring, among others, Brazil (TLS = 6), Ireland (TLS = 5), Mexico (TLS = 4), Ecuador (TLS = 2), and Cuba (TLS = 2)(See Fig. 6 ).

Co-authorship and country networks

4.2.2 Co-occurrence analysis

To explore the research hotspots within mathematics education, we ran a keyword co-occurrence analysis using Author Keywords.

Co-occurrence analysis based on author keywords

The author keyword co-occurrence analysis indicated that our repository contained 691 keywords (see Fig.  7 , left side), of which 23 met the minimum occurrence threshold of five occurrences ( n  = 5) (see Fig.  7 , right side). In the figure, the size of a node corresponds to the frequency of a keyword co-selected in review studies in mathematics education. The distance between any two keywords reflects their relative strength and topic similarity. Nodes within the same colour cluster indicate similar topics among these publications.

The red cluster comprises 11 closely related items, including ‘mathematics, meta-analysis, mathematics achievement, intervention, scoping review, bibliometric analysis, review, technology, learning disabilities, children, and math anxiety’. The green cluster emerges as the second prominent cluster, featuring 8 interrelated items such as ‘mathematics education, systematic review, systematic literature review, literature review, teacher education, education, teaching, and flipped classroom’. Lastly, the blue cluster consists of 4 items, namely ‘math, science, early childhood, and identity’.

Co-occurrence analysis of author keywords

Notably, the most frequently cited author keyword was ‘mathematics education’ ( n  = 55), followed by ‘systematic review’ ( n  = 44), ‘mathematics’ ( n  = 41), ‘meta-analysis’ ( n  = 34), ‘systematic literature review’ ( n  = 14), ‘literature review’ ( n  = 11), ‘teacher education’ ( n  = 9), ‘mathematics achievement’ ( n  = 8), ‘intervention’ ( n  = 6), ‘education’ ( n  = 6), ‘teaching’ ( n  = 6), ‘science’ ( n  = 6), ‘scoping review’ ( n  = 5), ‘bibliometric analysis’ ( n  = 5), ‘review’ ( n  = 5), ‘math’ ( n  = 5), ‘technology’ ( n  = 5), ‘flipped classroom’ ( n  = 5), ‘early childhood’ ( n  = 5), ‘children’ ( n  = 5), ‘identity’ ( n  = 5), ‘learning disabilities’ ( n  = 5), and ‘math anxiety’ ( n  = 6).

The keywords chosen by the authors highlighted the focus areas of reviews in mathematics education, emphasising themes such as mathematics achievement, teacher education, interventions, technology, and technology-enhanced approaches (e.g. flipped classrooms), special education, and early childhood education. Furthermore, the author keywords reflected the prevalent review types in mathematics education, specifically systematic reviews and meta-analyses. Additionally, they highlighted the interdisciplinary nature of reviews in mathematics education, encompassing both mathematics education and science education.

Furthermore, we conducted distinct author keyword co-occurrence analyses for review studies published within the periods of 2019 to 2023 and those preceding 2019, aiming to discern temporal trends in author keywords, particularly in recent years. The analysis yielded 606 keywords for the 2019–2023 period and 144 keywords for the period before 2019 (see Table  8 for the most popular 15 author keywords). A noteworthy disparity in prevalent keywords was observed between the two temporal segments. While predominant keyword regarding the review types prior to 2019 was meta-analysis, followed by literature review and systematic review, over the past five years, additional keywords such as scoping review and bibliometric analysis emerged, signalling an augmentation in the diversity of review types and methodologies. The findings indicated a notable increase in the popularity of systematic reviews over the past five years.

4.2.3 Citation analysis

To explore the most cited publications and journals in mathematics education, we conducted a citation analysis based on the units of analysis in documents and sources.

Citation and document analysis

The analysis of the 259 review papers in mathematics education included in this study indicated that they received a total of 7,050 citations between 1996 and 2023, averaging 251.79 citations per year and 27.22 citations per paper. Notably, 67% of these citations were received in the last five years (2019–2023).

The threshold for the minimum number of citations of documents was set at one, which 221 review studies out of 259 met. Figure  8 visualises the network between these review papers with the largest citation links and Table  9 shows the most cited documents. Not all the studies listed in Table  9 are among the top 10 studies with the highest TLS. Among them, only Gersten et al. ( 2009 ), Cheung and Slavin ( 2008 ), and Slavin and Lake ( 2008 ) are within the top 10 review studies in mathematics education with the highest TLS. While highly cited documents are influential in terms of direct references, the TLS metric provides additional insights into the collaborative relationships and connections between researchers and their work, which may not always correlate perfectly with citation counts as seen in our findings.

Our results showed that the largest number of citation links were for meta-analyses and systematic review studies. The most prominent review type among the most cited studies listed in Table  9 is meta-analysis ( n  = 6), followed by literature review ( n  = 2), systematic review ( n  = 1), and narrative review ( n  = 1). This result indicates the potential of meta-analysis studies in terms of citation performance. Most of these review studies were primarily published in high-ranking educational review journals ( n  = 6). Other review papers published in teacher education ( n  = 2), psychology ( n  = 1), and behavioural science and neuroscience journals ( n  = 1). These ten most cited review articles were all published in SSCI journals over a decade ago. Regarding research topics in the most cited papers, the dominant topics were mathematics achievement, content knowledge, working memory, learning disabilities, and educational technologies.

Specifically, we analysed the citation trends of the most cited 10 review papers over time and separately for the first five years after publication and the past five years (2019–2023). The results indicate a significant increase in the citations review studies have received in the last five years. We found that eight out of the ten most cited papers received more citations in the past five years (2019–2023) than in the first five years after their publication. The analysis revealed that the average annual citations for each paper ranged from 7 to 30. While the majority of these review studies ( n  = 8) received the least citations in the year of their publication, they received the most citations on average approximately 12 years after publication. This indicates that the peak citation period for review articles in mathematics education extends beyond the first decade following their publication.

Additionally, we investigated the ‘Enriched Cited References’ feature, which provides insight into why an author cited a particular reference; this beta enhancement is only available in selected journals (Clarivate, 2024 ). These references are presented to aid readers in quickly assessing sections of a review paper, allowing them to identify the most closely related or impactful references and infer their purpose. Articles containing enriched cited references are marked with the following labels (Clarivate, 2024 ):

Previously published research that contextualizes the current study within an academic domain.

References that supply the datasets, methodologies, concepts, and ideas directly utilized by the author or upon which the author’s work relies.

References introduced because the current study engages in a more thorough discussion.

References cited by the current study as yielding similar results. This may encompass methodological similarities or, in certain instances, replication of findings.

References noted by the current study as presenting contrasting results. This may also involve disparities in methodology or sample differences, influencing the outcomes.

The results, displayed in Table  10 , pertain to the classification of references based on the Enriched Cited References analysis conducted automatically by WoS. These results suggest that the most cited review studies in mathematics education were predominantly utilized by researchers to establish the background for their own research. Furthermore, these reviews also frequently employed to shape the discussion within the papers. In addition, some researchers utilize the mentioned most cited review studies to establish a conceptual, theoretical, or methodological basis. While the limited number of the studies cited these reviews to support their findings, they were not used to present opposing evidence. This suggests a reliance on existing literature review studies to inform, validate, or potentially challenge new research within the field.

Citation and source analysis

We conducted a citation source analysis and present the citation network map for the journals in Fig.  9 , listing the top 15 journals in Table  11 based on the citation and TLS metrics to represent the frequency of citations between articles in any two journals. The threshold for the minimum number of documents citing a source was one, and 103 records met the minimum number of citations of a source, also set at one. The network map shown in Fig.  9 indicates prominent clusters. The red cluster included 23 items (mostly special education, educational psychology, and educational review journals). The blue cluster included 16 items (predominantly educational psychology, educational technology, and educational review journals). The green cluster comprised 17 items (including mathematics and mathematics education journals, educational technology journals, and educational psychology journals).

The number of articles and the distribution of journals across various research fields were as follows: 25 educational sciences journals (43 papers), 20 psychology and educational psychology journals (41 papers), 15 special education journals (32 papers), 12 mathematics education journals (52 papers), 10 educational review journals (41 papers), 9 educational technology journals (28 papers), 3 mathematics journals (14 papers), and 9 other journals (8 articles).

Our findings indicate that ZDM– Mathematics Education ( n  = 16) has, so far, published the most review studies focusing on mathematics education, which is not unexpected due to the origin of the journal as a review journal publishing only special issues, for which a review article is compulsory in each issue. This was followed by Frontiers in Psychology ( n  = 14), Educational Research Review ( n  = 13), and Mathematics ( n  = 10) (see Table  11 for the top 15 journals).

The results highlighted that the most frequently cited papers were often published in specific educational review journals (e.g. Review of Educational Research , Educational Research Review , and Educational Psychology Review ), psychology and educational psychology journals (e.g. Frontiers in Psychology , Educational Psychology Review , European Journal of Cognitive Psychology , and Psychological Bulletin ), special education journals (e.g. Exceptional Children , Learning Disabilities Research & Practice , Learning Disability Quarterly , and Remedial and Special Educati on), educational technology journals (e.g. Computers & Education , Journal of Computer Assisted Learning , and Education and Information Technologies ), and mathematics and mathematics education journals (e.g. ZDM– Mathematics Education , Educational Studies in Mathematics , and Mathematics ).

Although the most visible mathematics education journals in citation network map were ZDM– Mathematics Education and Educational Studies in Mathematics (see Fig.  9 ), as mentioned earlier, twelve mathematics education journals provided platforms for review studies. These were ZDM– Mathematics Education ( n  = 16), Educational Studies in Mathematics ( n  = 5), International Journal of Science and Mathematics Education ( n  = 5), International Journal of Mathematical Education in Science and Technology ( n  = 5), International Electronic Journal of Mathematics Education ( n  = 3), Mathematics Education Research Journal ( n  = 3), International Journal for Technology in Mathematics Education ( n  = 3), International Journal of Education in Mathematics, Science and Technology ( n  = 3), Journal for Research in Mathematics Education ( n  = 2), Canadian Journal of Science, Mathematics and Technology Education ( n  = 1), Journal für Mathematik-Didaktik ( n  = 1), and Research in Mathematics Education ( n  = 1).

5 Discussion, conclusions, and limitations

In this study, we conducted a meta-review of literature review studies in mathematics education, enriched by a comprehensive bibliometric analysis. This paper significantly contributes to scholarly discourse by unravelling nuanced research trends, the most common review methodologies, and prevalent theoretical approaches in review studies in mathematics education. Based on content and bibliometric analysis, it delves into the research foci, providing an understanding of the relevant academic landscape. Additionally, it illuminates intricate connections among researchers, countries, and journals, elucidating collaborative networks in mathematics education research.

5.1 Insights from the meta-review and implications

The findings revealed a significant increase in the number of literature reviews in mathematics education, particularly in the past five years; 79% of the reviews we examined were published during this period. Multiple factors may have contributed to this surge, including researchers’ increased publication output during the pandemic (Cevikbas & Kaiser, 2023 ; Nane et al., 2023 ), challenges in collecting empirical data during the pandemic crisis (Uleanya & Yu, 2023 ), the relatively high citation rates associated with literature review studies, the growing prestige of educational review journals based on their increased impact factors (Miranda & Garcia-Carpintero, 2018 ), and the publication of review-oriented special issues in mathematics education journals.

Our findings revealed a prevalence of systematic reviews and meta-analyses; however, researchers also conducted diverse types of reviews, including scoping reviews, critical reviews, narrative reviews, theoretical reviews, and tutorial reviews. This methodological diversity is important as the advantages of one method can potentially overcome the disadvantages of another and combining different approaches can mitigate disadvantages (Taherdoost, 2023 ). Furthermore, our study revealed that rapid reviews, meta-reviews, umbrella reviews, mapping reviews, mixed-methods reviews, integrative reviews, interpretative reviews, iterative reviews, meta-syntheses, descriptive reviews, realist reviews, selective reviews, algorithmic reviews, and computational reviews indexed in WoS were not represented in mathematics education. The well-established PRISMA guidelines offer a defined framework for systematic reviews and meta-analyses to assist researchers in conducting reviews while adhering to quality and transparency criteria (Moher et al., 2009 ; Page et al., 2021 ). This adherence may have encouraged researchers to undertake such reviews, and future advancements in the development of specific guidelines and methodologies for each review type may further motivate researchers to conduct other types of reviews in mathematics education more frequently.

There were nuanced overlaps between the review types, leading to ambiguous distinctions. For instance, the structural similarity between systematic reviews and scoping reviews has led to misunderstandings. Munn et al. ( 2018 ) confirm inconsistency and confusion regarding the differentiation between scoping reviews and systematic reviews and offered guidelines for this decision-making process: a systematic review is preferable when addressing specific questions regarding the feasibility, appropriateness, significance, or efficacy of a specific treatment or practice. However, if the authors intend to demarcate the research field and explore its potential size and scope, a scoping review is more appropriate. Grant and Booth ( 2009 ) and Munn et al. ( 2018 ) clarified that a scoping review is preparation for a systematic literature review, not a deep study for a systematic literature review. The diverse taxonomies proposed by researchers have contributed to this complexity, with some employing various terms for similar review characteristics, and others applying the same terms to studies with distinct review attributes. Consequently, a consensus regarding the categorisation of review studies, both in a broad context and specifically in mathematics education, remains elusive. We also observed instances of researchers labelling their reviews inaccurately. However, we refrained from judging the appropriateness of these terminologies as they fall outside the scope of our study and may be difficult to justify due to the ambiguity of the current typologies. Borges Migliavaca et al. ( 2020 ) expressed a similar concern, highlighting substantial disparities in review studies concerning their conceptualisation, conduct, reporting, risk of bias assessment, and data synthesis. They called for the evidence synthesis community to promptly develop guidance and reporting standards for review studies. Future researchers could potentially examine inconsistencies in the conducting of review studies and their categorisation in mathematics education. In this study, we distilled the various existing types of review studies to provide clear explanations of the main review types and to help researchers and readers understand the key characteristics of various review studies (see Chap. 2).

An additional noteworthy consideration pertains to the sample sizes of review studies. A prevalent discourse considers the appropriate number of studies to be included in a review, but establishing such a minimum or maximum number may be challenging and not appropriate because this depends on various contextual factors, such as the research area, topic, inclusion/exclusion criteria, and applied protocols. For example, in technical terms, a systematic review can be conducted with as few as two studies or as many as a thousand. A review study with a small sample (e.g. two or three studies) may be due to the literature search methods used or insufficient number of existing studies in a particular field, suggesting a limited demand for such a review. As previously noted, the primary function of review studies is to inform readers in the relevant field about published studies to address the challenge posed by an increasing number of studies and to identify trends and research gaps (Fusar-Poli & Radua, 2018 ). Conversely, although it is technically feasible to include a substantial number of studies in a review (e.g. 1,000 or 2,000), conducting a comprehensive analysis (e.g. content analysis) of such a large dataset can present major time, cost, storage, memory, bias, and security challenges (Cohen et al., 2015 ). Nevertheless, the findings of our study provide insight into this issue. Notably, the sample size of the studies we analysed varied from 8 to 3,485, with an average of 99. Notably, most of these studies (78%) had sample sizes of less than 100, with an average of 34. Although this observation does not serve as a prescriptive recommendation, it offers valuable insights into the typical sample sizes with which mathematics education researchers have tended to work in the past.

Furthermore, as evidenced by our findings, literature reviews may serve various purposes, such as assessing the use of theoretical models or conceptual and methodological approaches, or advancing new theories, concepts, or research models through critical appraisal of previous research within a specific subject area (Cooper, 1988 ). However, our findings also indicate that it is not common in practice to use or develop a theoretical or conceptual framework in mathematics education review studies. Only 24% of the reviewed studies explicitly reported employing a specific framework, and very few sought to formulate a framework based on the literature under scrutiny. The results highlighted the researchers’ interest in frameworks related to technology, knowledge, and competence models. A few studies incorporated grand theories, such as constructivism, sociocultural theory, and cognitive development theory.

It is remarkable that despite focusing on mathematics education, there is a notable scarcity of review studies employing content-specific frameworks in mathematics education, such as those centred on problem-solving, reasoning, and mathematical thinking. Only a minority of the studies used frameworks related to mathematical modelling and mathematical content knowledge. This observation may reflect a gap in the literature, suggesting a need for greater integration of domain-specific frameworks into review studies in mathematics education to enhance the depth and specificity of the studies. Moreover, this trend prompts a critical examination of potential underlying factors. One plausible explanation lies in the interdisciplinary nature of review studies in mathematics education, which draws contributions from diverse fields including psychology, educational technology, special education, and neuroscience. The diverse disciplinary backgrounds of the researchers may influence their preferences for frameworks that are not necessarily specific to mathematics education but rather draw from broader fields.

5.2 Insights from the bibliometric analyses and implications

The bibliometric analysis revealed contributions to mathematics education, with 761 authors from 50 countries conducting review studies. In future studies, researchers may consider conducting detailed analyses of how these initiatives have influenced the landscape of mathematics education, examining their specific impacts on various subfields, and assessing their overall influence.

Our findings reveal a notable participation in literature review studies within mathematics education by scholars from diverse backgrounds, including educational psychologists, mathematics educators, and specialists in special education and neuroscience. This multidisciplinary engagement underscores the broader interest of researchers beyond the field of mathematics education. Notably, co-authorship connections within US institutions were the most extensive. The leading countries that published review studies included the US, Germany, China, Australia, and England. A robust network emerged among researchers in North America, Europe, Asia, and Australia, emphasising collaboration opportunities that warrant exploration by African and South American researchers.

Systematic reviews and meta-analyses stood out as the predominant review types in mathematics education, both in terms of the number of publications and citation counts. Systematic reviews offer rigorous and comprehensive syntheses of existing literature on specific research questions, providing valuable insights, identifying gaps in knowledge, and informing evidence-based decision-making in various fields. Moreover, meta-analyses enhance statistical power, resolve conflicting findings, and offer more precise estimates of effect sizes by combining data from various sources. However, there is a discernible need to diversify the types of reviews conducted in mathematics education.

The findings underscore a significant surge in both the quantity of review studies and their citation counts within mathematics education especially over the recent five-year period (2019–2023). This trend suggests a prevalent practice among authors to draw upon previously published reviews to contextualize their own studies, frequently engaging in discussions and citing references to corroborate or challenge existing findings. Such reliance on established literature highlights the discipline’s emphasis on leveraging prior knowledge to inform and substantiate new research endeavours.

The most cited review papers were associated with specific educational review journals, educational psychology journals, special education journals, educational technology journals, and mathematics education journals, further highlighting the interdisciplinary nature of impactful research in the field. The results revealed that ZDM– Mathematics Education , Educational Studies in Mathematics , International Journal of Science and Mathematics Education , and International Journal of Mathematical Education in Science and Technology were the key mathematics education journals committed to publishing review studies. The performance of these journals, particularly in recent years, reflects the escalating significance of review studies in mathematics education. Nevertheless, the limited visibility of some mathematics education journals in publishing review studies could be attributed, among other factors, to their restricted representation in the WoS database or to the overall small number of studies published yearly in particular mathematics education journals.

Prominent research topics in mathematics education review studies are digital technologies, technology-enhanced approaches (e.g. flipped classrooms), teacher education, mathematics achievement, early childhood education, and learning disabilities. Recent technological advances, including artificial intelligence and augmented/virtual reality, may soon attract mathematics education researchers’ attention to emerging technologies (Cevikbas, Bulut et al., 2023 ; Cevikbas, Greefrath et al., 2023 ). In addition to technology-enhanced mathematics education and special education, researchers have also explored the cognitive and affective aspects of learning and teaching mathematics.

In short, the absence of high-quality research syntheses may impede theoretical and conceptual advances within mathematics education (Webster & Watson, 2002 ). Therefore, future researchers may endeavour to develop discipline-specific standards and guidelines for conducting various types of review studies in mathematics education. Moreover, they could focus on expanding the content of mathematics education journals to accommodate a greater number of review studies. The scientific influence of review journals may also provide an opportunity to establish a dedicated review journal with a pronounced focus on mathematics education.

5.3 Limitations and conclusion

Finally, we want to point out that in this comprehensive meta-review, enriched by bibliometric analysis, we meticulously compiled and scrutinised the largest dataset of reviews in mathematics education available within the WoS database. Although this was a substantial sample ( n  = 259) that was reasonably representative of published review studies in mathematics education, it is important to acknowledge certain limitations. Our search was confined to WoS, and we specifically focused on review articles published in English. It is worth noting that the characteristics of review studies published in journals, international handbooks, or conference proceedings not indexed in WoS or published in a language other than English could potentially differ from those we examined. In addition, despite studies indexed in WoS theoretically being of high quality, we identified inconsistencies and variability in the review studies we examined, and it is possible that a more extensive search would have yielded different results.

In conclusion, we advocate producing high-quality review papers that adeptly synthesise available knowledge to improve professional practice (Templier & Paré, 2015 ). Such efforts may further advance mathematics education and contribute to the continuous improvement of teaching and learning activities, despite the demanding nature of comprehensive review studies.

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology , 8 (1), 19–32.

Article   Google Scholar  

Becker, L. A., & Oxman, A. D. (2008). Overviews of reviews. In J. P. T. Higgins, & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 607–631). Wiley.

Birkle, C., Pendlebury, D. A., Schnell, J., & Adams, J. (2020). Web of Science as a data source for research on scientific and scholarly activity. Quantitative Science Studies , 1 (1), 363–376.

Boote, D., & Beile, N. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher , 34 (6), 3–15.

Borges Migliavaca, C., Stein, C., Colpani, V., et al. (2020). How are systematic reviews of prevalence conducted? A methodological study. BMC Medical Research Methodology , 20 , 96.

Cevikbas, M., & Kaiser, G. (2023). Can flipped classroom pedagogy offer promising perspectives for mathematics education on pandemic-related issues? A systematic literature review. ZDM–Mathematics Education , 55 , 177–191.

Cevikbas, M., Kaiser, G., & Schukajlow, S. (2022). A systematic literature review of the current discussion on mathematical modelling competencies: State-of-the-art developments in conceptualizing, measuring, and fostering. Educational Studies in Mathematics , 109 (2), 205–236.

Cevikbas, M., Greefrath, G., & Siller, H. S. (2023). Advantages and challenges of using digital technologies in mathematical modelling education–a descriptive systematic literature review. Frontiers in Education , 8 , 1142556.

Cevikbas, M., Bulut, N., & Kaiser, G. (2023). Exploring the benefits and drawbacks of AR and VR technologies for learners of mathematics: Recent developments. Systems , 11 (5), 244.

Cevikbas, M., König, J., & Rothland, M. (2024). Empirical research on teacher competence in mathematics lesson planning: Recent developments. ZDM–Mathematics Education , 56 , 101–113.

Cheung, A. C., & Slavin, R. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review , 9 , 88–113.

Chigbu, U. E., Atiku, S. O., & Du Plessis, C. C. (2023). The science of literature reviews: Searching, identifying, selecting, and synthesising. Publications , 11 (1), 2.

Clarivate (2024). Citation context in Web of Science . URL.

Cochrane, A. L. (1979). 1931–1971: A critical review with particular reference to the medical profession. In G. Teeling Smith, & N. Wells (Eds.), Medicines for the year 2000 (pp. 2–12). Office of Health Economics.

Cohen, B., Vawdrey, D. K., Liu, J., Caplan, D., Furuya, E. Y., Mis, F. W., & Larson, E. (2015). Challenges associated with using large data sets for quality assessment and research in clinical settings. Policy Politics & Nursing Practice , 16 (3–4), 117–124.

Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society , 1 (1), 104–126.

Google Scholar  

Cooper, H., Nye, B., Charlton, K., Lindsay, J., & Greathouse, S. (1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research , 66 (3), 227–268.

Creswell, J. W. (2013). Qualitative inquiry and research design; choosing among five approaches . Sage.

Davies, P. (2000). The relevance of systematic reviews to educational policy and practice. Oxford Review of Education , 26 (3–4), 365–378.

Depaepe, F., Verschaffel, L., & Kelchtermans, G. (2013). Pedagogical content knowledge: A systematic review of the way in which the concept has pervaded mathematics educational research. Teaching and Teacher Education , 34 , 12–25.

DeStefano, D., & LeFevre, J. A. (2004). The role of working memory in mental arithmetic. European Journal of Cognitive Psychology , 16 (3), 353–386.

Friso-van den Bos, I., Van der Ven, S. H., Kroesbergen, E. H., & Van Luit, J. E. (2013). Working memory and mathematics in primary school children: A meta-analysis. Educational Research Review , 10 , 29–44.

Fusar-Poli, P., & Radua, J. (2018). Ten simple rules for conducting umbrella reviews. Evidence-Based Mental Health , 21 (3), 95.

Geary, D. C. (1996). Sexual selection and sex differences in mathematical abilities. Behavioral and Brain Sciences , 19 (2), 229–247.

Geng, Y., Chen, W., Liu, Z., Chiu, A. S., Han, W., Liu, Z., & Cui, X. (2017). A bibliometric review: Energy consumption and greenhouse gas emissions in the residential sector. Journal of Cleaner Production , 159 , 301–316.

Gersten, R., Chard, D. J., Jayanthi, M., Baker, S. K., Morphy, P., & Flojo, J. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research , 79 (3), 1202–1242.

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal , 26 (2), 91–108.

Hart, C. (2018). Doing a literature review: Releasing the research imagination (2nd edition). Sage.

Kaiser, G., & Schukajlow, S. (2024). Literature reviews in mathematics education and their significance to the field. ZDM–Mathematics Education , 56 , 1–3.

Ketcham, C. M., & Crawford, J. M. (2007). The impact of review articles. Laboratory Investigation , 87 (12), 1174–1185.

Korom, P. (2019). A bibliometric visualization of the economics and sociology of wealth inequality: A world apart? Scientometrics , 118 , 849–868.

Kunisch, S., Denyer, D., Bartunek, J. M., Menz, M., & Cardinal, L. B. (2023). Review research as scientific inquiry. Organizational Research Methods , 26 (1), 3–45.

Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review , 22 , 215–243.

MacEntee, M. I. (2019). A typology of systematic reviews for synthesising evidence on health care. Gerodontology , 36 (4), 303–312.

Martin, D. B. (2009). Researching race in mathematics education. Teachers College Record , 111 (2), 295–338.

Martínez, M. A., Cobo, M. J., Herrera, M., & Herrera-Viedma, E. (2015). Analyzing the scientific evolution of social work using science mapping. Research on Social Work Practice , 25 (2), 257–277.

Miranda, R., & Garcia-Carpintero, E. (2018). Overcitation and overrepresentation of review papers in the most cited papers. Journal of Informetrics , 12 (4), 1015–1030.

Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999). Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement. Lancet , 354 , 1896–1900.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Plos Medicine , 6(7), e1000097.

Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology , 18 , 1–7.

Nane, G. F., Robinson-Garcia, N., van Schalkwyk, F., & Torres-Salinas, D. (2023). COVID-19 and the scientific publishing system: Growth, open access and scientific fields. Scientometrics , 128 (1), 345–362.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., McDonald, S., & Moher, D. (2021). (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ , 372 (71), 1–9.

Paré, G., Trudel, M. C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management , 52 (2), 183–199.

Radevic, L., & Milovanovic, I. (2023). Current trends in math anxiety research: A bibliometric approach. International Journal of Science and Mathematics Education . https://doi.org/10.1007/s10763-023-10424-4 .

Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club , 123 (3), A12–A13.

Samnani, S. S., Vaska, M., Ahmed, S., & Turin, T. C. (2017). Review typology: The basic types of reviews for synthesizing evidence for the purpose of knowledge translation. Journal of the College of Physicians and Surgeons Pakistan , 27 (10), 635–641.

Schryen, G., & Sperling, M. (2023). Literature reviews in operations research: A new taxonomy and a meta review. Computers & Operations Research , 106269.

Schryen, G., Wagner, G., Benlian, A., & Paré, G. (2020). A knowledge development perspective on literature reviews: Validation of a new typology in the IS field. Communications of the AIS , 46 , 134–186.

Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., & Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology , 7 , 1–7.

Slavin, R. E., & Lake, C. (2008). Effective programs in elementary mathematics: A best-evidence synthesis. Review of Educational Research , 78 (3), 427–515.

Song, Y., Chen, X., Hao, T., Liu, Z., & Lan, Z. (2019). Exploring two decades of research on classroom dialogue by using bibliometric analysis. Computers & Education , 137 , 12–31.

Suseelan, M., Chew, C. M., & Chin, H. (2022). Research on Mathematics Problem solving in Elementary Education conducted from 1969 to 2021: A bibliometric review. IJEMST , 10 (4), 1003–1029.

Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information & Libraries Journal , 36 (3), 202–222.

Taherdoost, H. (2023). Towards nuts and bolts of conducting literature review: A typology of literature review. Electronics , 12 (4), 800.

Templier, M., & Paré, G. (2015). A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems , 37 (1), 6.

Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D., Horsley, T., Weeks, L., Hempel, S., et al. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine , 169 (7), 467–473.

Uleanya, C., & Yu, K. (2023). Data collection in times of pandemic: A self-study and revisit of research practices during a crisis. Sage Open , 13 (1), 21582440231160698.

van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics , 84 (2), 523–538.

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly , 26 (2), xiii–xxiii.

Wei, Y., Zhang, Q., Guo, J., & Chen, M. (2023). Learning to teach through noticing: A bibliometric review of teacher noticing research in mathematics education during 2006–2021. Humanities and Social Sciences Communications , 10 (1), 1–15.

Zupic, I., & Cater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods , 18 (3), 429–472.

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Humboldt-Universität zu Berlin Institute of Educational Sciences, Unter den Linden 6, 10099, Berlin, Germany

Mustafa Cevikbas

Faculty of Education, Universität Hamburg, Von-Melle-Park 8, 20146, Hamburg, Germany

Gabriele Kaiser

Nord University, Bodø, Norway

Institute of Mathematics Education and Computer Science Education, Universität Münster, Henriette-Son Str. 19, 48149, Münster, Germany

Stanislaw Schukajlow

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mustafa Cevikbas .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The electronic supplementary material includes the list of the reviewed studies and a sample of the coding manual.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cevikbas, M., Kaiser, G. & Schukajlow, S. Trends in mathematics education and insights from a meta-review and bibliometric analysis of review studies. ZDM Mathematics Education (2024). https://doi.org/10.1007/s11858-024-01587-7

Download citation

Accepted : 02 May 2024

Published : 15 May 2024

DOI : https://doi.org/10.1007/s11858-024-01587-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Bibliometric analysis
• Critical review
• Literature review
• Mapping review
• Mathematics education
• Meta-review
• Narrative review
• Systematic review
• Scoping review
• Theoretical review
• Tutorial review
• Umbrella review
• Find a journal
• Publish with us
• Track your research

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Research and Citation Resources

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

If you are having trouble locating a specific resource please visit the  search page  or the  Site Map . The Citation Chart  provides a detailed overview of MLA Style, APA Style, and Chicago Manual of Style source documentation by category.

Conducting Research

These OWL resources will help you conduct research using primary source methods, such as interviews and observations, and secondary source methods, such as books, journals, and the Internet. This area also includes materials on evaluating research sources.

Using Research

These OWL resources will help you use the research you have conducted in your documents. This area includes material on quoting and paraphrasing your research sources, as well as material on how to avoid plagiarism.

APA Style (7th Edition)

These OWL resources will help you learn how to use the American Psychological Association (APA) citation and format style. This section contains resources on in-text citation and the References page, as well as APA sample papers, slide presentations, and the APA classroom poster.

These OWL resources will help you learn how to use the Modern Language Association (MLA) citation and format style. This section contains resources on in-text citation and the Works Cited page, as well as MLA sample papers, slide presentations, and the MLA classroom poster

Chicago Manual of Style

This section contains information on the Chicago Manual of Style method of document formatting and citation. These resources follow the 17th edition of the Chicago Manual of Style, which was issued in 2017.

Institute of Electrical and Electronics Engineers (IEEE) Style

These resources describe how to structure papers, cite sources, format references, and handle the complexities of tables and figures according to the latest Institute of Electrical and Electronics Engineers (IEEE) guidelines.

American Medical Association (AMA) Style

These resources provide guidance on how to cite sources using American Medical Association (AMA) Style, 10th Ed., including examples for print and electronic sources.

Research Overview

We live in an age overflowing with sources of information. With so many information sources at our fingertips, knowing where to start, sorting through it all and finding what we want can be overwhelming! This handout provides answers to the following research-related questions: Where do I begin? Where should I look for information? What types of sources are available?

Conducting Primary Research

Primary research involves collecting data about a given subject directly from the real world. This section includes information on what primary research is, how to get started, ethics involved with primary research and different types of research you can do. It includes details about interviews, surveys, observations, and analysis.

Evaluating Sources of Information

Evaluating sources of information is an important step in any research activity. This section provides information on evaluating bibliographic citations, aspects of evaluation, reading evaluation, print vs. online sources, and evaluating Internet sources.

Searching Online

This section covers finding information online. It includes information about search engines, Boolean operators, Web directories, and the invisible Web. It also includes an extensive, annotated links section.

Internet References

This page contains links and short descriptions of writing resources including dictionaries, style manuals, grammar handbooks, and editing resources. It also contains a list of online reference sites, indexes for writers, online libraries, books and e-texts, as well as links to newspapers, news services, journals, and online magazines.

Archival Research

This resource discusses conducting research in a variety of archives. It also discusses a number of considerations and best practices for conducting archival research.

This resources was developed in consultation with Purdue University Virginia Kelly Karnes Archives and Special Collections staff.

A systematic analysis of the contribution of genetics to multimorbidity and comparisons with primary care data

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for Louise M Allan
• ORCID record for Frank Dudbridge
• ORCID record for Luke C Pilling
• ORCID record for João Delgado
• For correspondence: [email protected]
• Info/History
• Supplementary material
• Preview PDF

Background Multimorbidity, the presence of two or more conditions in one person, is increasingly prevalent. Yet shared biological mechanisms of specific pairs of conditions often remain poorly understood. We address this gap by integrating large-scale primary care and genetic data to elucidate potential causes of multimorbidity. Methods We defined chronic, common, and heritable conditions in individuals aged ≥65 years, using two large representative healthcare databases [CPRD (UK) N=2,425,014 and SIDIAP (Spain) N=1,053,640], and estimated heritability using the same definitions in UK Biobank (N=451,197). We used logistic regression models to estimate the co-occurrence of pairs of conditions in the primary care data. Linkage disequilibrium score regression was used to estimate genetic similarity between pairs of conditions. Meta-analyses were conducted across healthcare databases, and up to three sources of genetic data, for each condition pair. We classified pairs of conditions as across or within-domain based on the international classification of disease. Findings We identified N=72 chronic conditions, with 43.6% of 2546 pairs showing higher co-occurrence than expected and evidence of shared genetics. Notably, across-domain pairs like iron deficiency anaemia and peripheral arterial disease exhibited substantial shared genetics (genetic correlation Rg=0.45[95% Confidence Intervals 0.27:0.64]). N=33 pairs displayed negative genetic correlations, such as skin cancer and rheumatoid arthritis (Rg=-0.14[-0.21:-0.06]), indicating potential protective mechanisms. Discordance between genetic and primary care data was also observed, e.g., abdominal aortic aneurysm and bladder cancer co-occurred but were not genetically correlated (Odds-Ratio=2.23[2.09:2.37], Rg=0.04[-0.20:0.28]) and schizophrenia and fibromyalgia were less likely to co-occur but were positively genetically correlated (OR=0.84[0.75:0.94], Rg=0.20[0.11:0.29]). Interpretation Most pairs of chronic conditions show evidence of shared genetics and co-occurrence in primary care, suggesting shared mechanisms. The identified shared mechanisms, negative correlations and discordance between genetic and observational data provide a foundation for future research on prevention and treatment of multimorbidity. Funding UK Medical Research Council [MR/W014548/1].

Competing Interest Statement

ARL is now an employee of AstraZeneca and has interests in the company. The work undertaken here was prior to his appointment. SK's group has received funding support from Amgen BioPharma outside of this work. JB is a part time employee of Novo Nordisk Research Centre Oxford, limited, unrelated to this work. TF has consulted for several pharmaceutical companies. All other authors have no disclosures to declare.

Funding Statement

This work was supported by the UK Medical Research Council [grant number MR/W014548/1]. This study was supported by the National Institute for Health and Care Research (NIHR) Exeter Biomedical Research Centre (BRC), the NIHR Leicester BRC, the NIHR Oxford BRC, the NIHR Peninsula Applied Research Collaboration, and the NIHR HealthTech Research Centre. KB is partly funded by the NIHR Applied Research Collaboration South-West Peninsula. JM is funded by an NIHR Advanced Fellowship (NIHR302270). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. CV acknowledges research funding by a "Contratos para la intensificacion de la actividad investigadora en el Sistema Nacional de Salud" contract (INT23/00040) from the Spanish Ministry of Science and Innovation.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study was approved by the relevant ethics committees: SIDIAP Scientific and Ethical Committees (19/518-P) on 18/12/2019. The SIDIAP database is based on opt-out presumed consent. If a patient decides to opt out, their routine data would be excluded of the database. CPRD ISAC committee protocol number 23_003109. The Northwest Multi-Centre Research Ethics Committee approved the collection and use of UK Biobank data for health-related research (Research Ethics Committee reference 11/NW/0382). UKB was granted under Application Number 9072.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

We cannot make individual-level data available. Researchers can apply to UK Biobank (https://www.ukbiobank.ac.uk/enable-your-research/), CPRD (https://www.cprd.com/research-applications), and SIDIAP (https://www.sidiap.org/index.php/en/solicituds-en). We have made our diagnostic code lists, code and results available on our GitHub (https://github.com/GEMINI-multimorbidity/) site and Shiny website (https://gemini-multimorbidity.shinyapps.io/atlas/). GWAS summary statistics will be available following acceptance at the GWAS Catalog (https://www.ebi.ac.uk/gwas/home).

View the discussion thread.

Supplementary Material

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One
• Addiction Medicine (323)
• Allergy and Immunology (627)
• Anesthesia (163)
• Cardiovascular Medicine (2367)
• Dentistry and Oral Medicine (288)
• Dermatology (206)
• Emergency Medicine (379)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (834)
• Epidemiology (11765)
• Forensic Medicine (10)
• Gastroenterology (702)
• Genetic and Genomic Medicine (3729)
• Geriatric Medicine (348)
• Health Economics (632)
• Health Informatics (2392)
• Health Policy (929)
• Health Systems and Quality Improvement (896)
• Hematology (340)
• HIV/AIDS (780)
• Infectious Diseases (except HIV/AIDS) (13302)
• Intensive Care and Critical Care Medicine (767)
• Medical Education (365)
• Medical Ethics (104)
• Nephrology (398)
• Neurology (3493)
• Nursing (198)
• Nutrition (523)
• Obstetrics and Gynecology (673)
• Occupational and Environmental Health (661)
• Oncology (1819)
• Ophthalmology (535)
• Orthopedics (218)
• Otolaryngology (287)
• Pain Medicine (232)
• Palliative Medicine (66)
• Pathology (446)
• Pediatrics (1032)
• Pharmacology and Therapeutics (426)
• Primary Care Research (420)
• Psychiatry and Clinical Psychology (3172)
• Public and Global Health (6135)
• Radiology and Imaging (1278)
• Rehabilitation Medicine and Physical Therapy (745)
• Respiratory Medicine (825)
• Rheumatology (379)
• Sexual and Reproductive Health (372)
• Sports Medicine (322)
• Surgery (400)
• Toxicology (50)
• Transplantation (172)
• Urology (145)

• Open access
• Published: 08 May 2024

Does health voucher intervention increase antenatal consultations and skilled birth attendances in Cameroon? Results from an interrupted time series analysis

• Isidore Sieleunou   ORCID: orcid.org/0000-0001-7264-4540 1 , 2 &
• Roland Pascal Enok Bonong   ORCID: orcid.org/0000-0002-9552-5365 2  

BMC Health Services Research volume  24 , Article number:  602 ( 2024 ) Cite this article

261 Accesses

Metrics details

Limited access to health services during the antenatal period and during childbirth, due to financial barriers, is an obstacle to reducing maternal and child mortality. To improve the use of health services in the three regions of Cameroon, which have the worst reproductive, maternal, neonatal, child and adolescent health indicators, a health voucher project aiming to reduce financial barriers has been progressively implemented since 2015 in these three regions. Our research aimed to assess the impact of the voucher scheme on first antenatal consultation (ANC) and skilled birth attendance (SBA).

Routine aggregated data by month over the period January 2013 to May 2018 for each of the 33 and 37 health facilities included in the study sample were used to measure the effect of the voucher project on the first ANC and SBA, respectively. We estimated changes attributable to the intervention in terms of the levels of outcome indicators immediately after the start of the project and over time using an interrupted time series regression. A meta-analysis was used to obtain the overall estimates.

Overall, the voucher project contributed to an immediate and statistically significant increase, one month after the start of the project, in the monthly number of ANCs (by 26%) and the monthly number of SBAs (by 57%). Compared to the period before the start of the project, a statistically significant monthly increase was observed during the project implementation for SBAs but not for the first ANCs. The results at the level of health facilities (HFs) were mixed. Some HFs experienced an improvement, while others were faced with the status quo or a decrease.

Conclusions

Unlike SBAs, the voucher project in Cameroon had mixed results in improving first ANCs. These limited effects were likely the consequence of poor design and implementation challenges.

Peer Review reports

Reducing maternal, newborn, and child mortality is one of the world's top public health priorities. The third of the seventeen Sustainable Development Goals (SDGs) reflects the international commitment to improving maternal and child health. By 2030, the goals include reducing the global maternal mortality ratio to less than 70 per 100,000 live births, neonatal mortality to 12 per 1,000 live births at most, and under-five mortality to less than 25 per 1,000 live births [ 1 ].

However, despite considerable improvements in recent decades, maternal mortality has remained a major public health concern globally, with more than 295,000 maternal deaths in 2017 and sub-Saharan Africa (SSA) alone accounting for approximately 66% of this global picture [ 2 ]. On the other hand, despite dramatic reductions in child mortality over the last 30 years, the global burden of child deaths has remained immense, with a total of 5.2 million under-five deaths in 2019, representing an average of 14,000 deaths every day [ 3 ].

While from 2000 to 2017, the global maternal mortality ratio (MMR) decreased by 38% [ 2 ], Cameroon's MMR skyrocketed from 511 in 1998 to 782 in 2011 before declining to 467 in 2018 [ 4 ].

A priority toward ending preventable maternal and child deaths is to improve access to and use of quality health services and qualified nurses at birth [ 5 , 6 ]. One of the basic elements is the presence of pregnant women at antenatal consultations. Previous studies have shown that performing prenatal consultations reduces the risk of neonatal mortality [ 7 , 8 ].

However, women in developing countries encounter significant barriers to accessing conventional health services, including poor education, physical and financial barriers, and limited voice and decision-making power [ 9 , 10 ]. The poor quality of available health services offers a further disincentive [ 6 ]. This translates to only half of parturient women receiving skilled assistance at delivery and many fewer receiving postpartum cares [ 6 ].

In Cameroon, the country’s comparatively slow reduction in maternal and child mortalities is likely due to insufficient coverage of reproductive, maternal, neonatal, child and adolescent health (RMNCAH) services; for instance, in 2018, an estimated 65% of women in Cameroon attended at least four antenatal consultations (ANC) visits, 69% gave birth with the assistance of qualified personnel, and 59% received postnatal care (PNC) [ 11 ]. In addition, these general estimates hide enormous disparities. Overall, 65% of the pregnant women who attended the four ANCs included more than 79% of those in urban areas but only 52% of those in rural areas. Moreover, while this rate was 91% in the richest quintile, only one-third (37%) of the poorest pregnant women attended the four ANCs [ 11 ].

The complexity of barriers to accessing care in developing countries indicates that any solution to improving maternal health service utilization must be comprehensive and address both supply- and demand-side health system constraints. This is particularly important in a context such as Cameroon where household out-of-pocket (OOP) spending was the single largest source of financing for the health sector, at 71 percent of total health spending in 2017, well above the WHO benchmark of 15-20 percent, and exceeding the average for SSA (33 percent) and countries of similar income such as Kenya (24 percent) and Ghana (40 percent) [ 12 ].

As ability to pay remains an important determinant of women’s access to healthcare, many countries have sought to improve coverage of maternal services by reducing financial barriers to seeking services [ 13 , 14 ]. Strategies implemented at the country level include national health insurance and user fee removals/exemptions, and at the subnational level, community-based health insurance, health vouchers and conditional cash transfers [ 15 ].

Given that limited access to emergency obstetric and neonatal care (EmONC) is a major contributor to high maternal mortality [ 16 ], increasing pregnant women's use of health facilities for assisted delivery could help reduce maternal and new born morbidity and mortality, as previous studies have indicated [ 17 , 18 ].

In recent years, there has been growing interest in the use of vouchers and other innovative financing mechanisms to increase access to EmONCs for low-income women [ 19 , 20 , 21 , 22 , 23 , 24 , 25 ]. By providing a financial or in-kind reward conditioned on the achievement of agreed-upon performance goals, vouchers are described as a promising holistic approach to foster the use of cost-effective services by the poor and other disadvantaged populations [ 22 ].

Vouchers can act on the demand side, the supply side, or both sides. Demand-side incentives encourage service use not only by reducing the financial burden but also by offering women a choice of providers and informing them of the benefits of using maternal health services. Supply-side incentives aim to improve the quality and responsiveness of service delivery.

To date, findings from the few assessments of reproductive health voucher programs suggest that, if implemented well, they have the potential to improve both assisted and facility-based deliveries [ 19 , 20 , 22 , 24 , 26 ]. Yet, there is a paucity of evidence based on rigorous evaluation studies, making it challenging to draw consistent conclusions about the impact of voucher initiatives and to make subsequent policy recommendations.

The current study evaluated a pilot voucher program in Cameroon, a country where approximately 39% of all deliveries took place at home at the time of the program’s inception [ 27 ]. The research aimed to assess the impact of the voucher scheme on first antenatal consultation and skilled birth attendance (SBA). In the following, we present a brief description of the Cameroon voucher program. We then present our data and methods, followed by the results. We end with a discussion of the study’s results, as well as the implications of these findings.

Voucher program in Cameroon

Results from the 2014 Multiple Indicator Cluster Survey (MICS) indicate an enormous disparity in health outcomes among Cameroon's ten regions, with the three northern regions (Adamawa, North, Far North) bearing the brunt of the disease burden [ 27 ]. For example, while the Far North and North regions represented 27.5% of the total population of children under five years in 2014, both regions accounted for 63% of the total excess mortality during the same period [ 27 ]. In addition, while 65% of women nationwide gave birth with the help of qualified personnel, only 29%, 36% and 53% in the Far North, North and Adamawa regions, respectively, gave birth in the same conditions. Moreover, these three regions featuring the lowest frequencies of ANCs and assisted deliveries, were home to more than 60% of the country’s poorest population [ 28 ].

Initiated in 2015, the voucher programme is a government programme, supported with funding from German and French partners, that aims to reduce financial barriers to maternal and neonatal care in the three northern regions of Cameroon.

Under the project, (poor) women can purchase subsidized vouchers for 6000 FCFA (≈$11), a co-payment of 10% of the actual cost of the service package estimated at 60,000 FCFA (approximately USD109), that covered the cost of a benefit package including services for pregnant women and their new-borns up to 42 days after delivery. In addition, beneficiaries are provided with transportation from their house to the nearest health facility and transportation from health centers to referral hospitals. Health facilities offering services for the voucher scheme are compensated for extra costs incurred. All pregnant women living within the 3 northern regions of Cameroon were eligible for the programme. To be included in the programme, health facilities are required to meet minimum quality standards based on national guidelines for the provision of maternal care. Women can redeem vouchers at any participating facility, and the contracted facilities submit claims to be reimbursed at standard rates for each service provided.

At its inception, the programme implementation was outsourced to the ‘Centre International de Développement et de Recherche’ (CIDR), an international organization. Since November 2018, the management of the scheme has been transferred to a national entity: the Regional Funds for Health Promotion (RFHP). A transfer protocol signed between the ministry of public health (MPH) and CIDR made provisions for the training of the RFHP personnel to take over the implementation.

Study design, data source and study sample

To achieve the study objectives, we used a quasi-experimental study design. Specifically, for each health facility (HF) that was enrolled in the health voucher project, the potential effect of the project was measured using an analysis of interrupted time series [ 29 , 30 , 31 , 32 ]. This method compares changes in the indicators of interest before and after the start of the intervention. It is based on the fundamental assumption that, in the absence of intervention, the trend of the interest indicator remains unchanged over time [ 31 ]. It is desirable to have at least 12 observation points for the indicator or variable of interest before and after the start of the intervention, respectively [ 29 ].

We used secondary data from the monitoring and evaluation system database populated by the three regional implementing agencies of the health voucher project, let by the CIDR-CARE prior to the transfer of the project to the RFHP that began in 2018. These databases were updated quarterly by trained research assistants after monthly data collection from the registries of all health facilities enrolled in the project. Data quality control was carried out jointly by the team from the MPH in charge of monitoring project implementation and by the project team. The data used in this study are monthly aggregates of the variables of interest over the period from January 2013 to May 2018 (i.e., 65 months of observation).

The database contains information on 42 health facilities (HFs) enrolled in the health voucher project, spread across three regions: 12 HFs in the Adamawa region, 15 in the North region and 14 in the Far North region. These HFs were sequentially enrolled in the health voucher project and not at the same time. In the Adamawa region, activities started in 9 HFs in May 2015 and in 3 HFs in March 2016. In the North region, the implementation of activities started in May 2015 in one HF, in June 2015 in 5 HFs and in July 2016 in 9 HFs. For the Far North region, the intervention started in 4 HFs in June 2015, in 3 HFs in March 2016 and in 7 HFs in July 2016. For the analysis of each outcome, HFs included in the sample were those with at least 90% data completeness over the selected period. Thus, the sample sizes for analysis of the outcomes associated with the first antenatal consultation and assisted deliveries were 33 and 37, respectively.

Study variables

Two dependent variables were considered for this evaluation: (i) the monthly number of first ANC visits in each HF and (ii) the monthly number of SBA in each HF.

Covariables

X it : a time-dependent dichotomous variable that takes the value 0 for the months before the start of the health voucher project in HF i and 1 after the start of the project.

T t : time variable measured in months, with values ranging from 0 (January 2013) to 64 (May 2018).

X it *(T t -θ i ): interaction variable between the variables X it and T t centered on the value corresponding to the month of project start in HF i (θ i ).

Statistical analysis

Descriptive analysis.

To explore the outcomes, we used descriptive statistics (mean, median, standard deviation, interquartile range, absolute frequency, relative frequency) and trend curves.

Statistical modeling

For each HF and for each outcome, the estimation of the effects of the health voucher project was carried out using a negative binomial regression.

Since both outcomes are count variables, the choice of negative binomial regression instead of Poisson regression, which is the classic model for this type of variable, was considered to overcome the violation of the fundamental assumption underlying Poisson regression, which states that the mean is equal to the variance. Let Y it be the value of the considered outcome observed in HF i at time t. Y it follows a Poisson distribution with parameter μ it (Y it ~Poisson (μ it )). The general equation of the model used is shown below:

The other parameters of the model are described below.

β 0 = intercept (value of the dependent variable at month 1 of follow-up);

β 1 = slope of the outcome trajectory before the start of the health voucher project;

β 2 = change in the level of outcome at the end of the first month of implementation of the health voucher project;

β 3 = difference between the slope of the outcome trajectory after and before the start of the health voucher project;

variable γ it is the term that differentiates Poisson regression from negative binomial regression. In other words, e γit follows a gamma distribution with mean 1 and variance α (e γit ~ gamma (1/α, α)), with α being the overdispersion parameter.

The coefficient β 2 assesses the immediate effect of the project and β 3 assesses the effect of the project over time.

The graphs used to explore the evolution of outcomes over time highlighted the presence of seasonality. Thus, 11 dichotomous variables were considered in the different models. Equation ( 1 ) becomes log (μ it ) = β 0 + β 1 T t + β 2 X it + β 3 X it *(T t -θ i ) + ɸ 1 February + ɸ 2 March + ɸ 3 April + ɸ 4 May + ɸ 5 June + ɸ 6 July + ɸ 7 August + ɸ 8 September + ɸ 9 October + ɸ 10 November + ɸ 11 December + γ it .

The variables February, March … December take the value 1 if the observation relates to this month and 0 otherwise. The month of January was considered a reference.

Because the project did not start at the same time in all HFs, to obtain estimates representing the overall situation, a meta-analysis was used [ 33 ]. Thus, the pooled estimates and their confidence intervals were obtained by combining the regression coefficients of each HF using the inverse variance method. Random effects models were used to consider the strong heterogeneity highlighted by the statistics I 2 =100*(Q-df)/Q (with Q the statistics of Cochran's Q-test of heterogeneity and df the number of degrees of freedom corresponding here to the number of HFs minus one). The values 0%, 25%, 50% and 75% of the I 2 statistics represent the following levels of heterogeneity: absent, weak, moderate, and strong, respectively [ 33 , 34 ]. The incidence-rate ratio (IRR) for each HF per month as well as the aggregate estimates were graphically represented using a "forest plot". The analysis was stratified by region.

The statistical significance threshold used for interpreting the results was 5%. All the statistical analyses were performed with Stata/SE software version 14.2.

Descriptive statistics

The results in Table 1 show that the overall level of data completeness is 98.9% for the monthly number of first ANC visits and 99.3% for the monthly number of SBAs. In all regions, better data completeness was observed in the post-start period of the intervention. For the descriptive statistics of the two variables of interest, overall, the average (respectively the median) of the monthly number of first ANC visits was 58.6 (respectively 50.0). For the monthly number of SBAs, the mean and median were 52.3 and 31.0, respectively. The observed differences between the means and medians illustrate the asymmetry of the distributions of these variables. We also found that the means and medians of these two variables appeared to be greater during the implementation period of the project than during the period prior to the intervention.

Furthermore, Fig.  1 shows that there was an increasing trend over time for the monthly average of the first ANC and the monthly average of the SBA. It also emerged that the positive slope was more abrupt for SBA.

Evolution of the monthly averages of the number of first ANC visits and SBAs in the selected Health facilities between January 2013 and May 2018

Effects of the health voucher project

First antenatal consultation (anc).

Table 2 and Figure S 3 displays contrasting results. Overall, at the end of the first month of implementation of the project, controlling for other variables, a statistically significant increase of nearly 26% in the monthly number of first ANCs was observed in the 33 HFs considered in the study sample (IRR = 1.258 [95% CI: 1.075, 1.472]). A similar increase was recorded in the North region but was not statistically significant (IRR = 1.246 [95% CI: 0.976, 1.591]). In the Adamawa region, the increase was nearly 73% (IRR = 1.726 [95% CI: 1.117, 2.668]). Conversely, in the Far North region, a nonsignificant reduction of 0.2% was noted (IRR = 0.998 [95% CI: 0.882, 1.129]). These overall results hid disparities across facilities. In the Adamawa region, out of 10 HFs, there was a statistically significant increase in the monthly number of first ANCs at the end of the first month of project implementation in five HFs and a statistically significant decrease in one HF. In the Far North region, of the 10 HFs, a statistically significant increase was recorded in two HFs, and a statistically significant reduction was recorded in one HF. In the North region, of the 13 HFs, six exhibited a statistically significant increase in the aforementioned indicator and one exhibited a statistically significant decrease.

Moreover, regarding the difference between the slope of the trajectory of the first ANC after and before the start of the project, Table 2 and Figure S 4 does not show statistically significant results, either overall or by region. However, in one HF in the Adamawa region, a statistically significant increase in the slope of the trajectory of the first ANC was observed during the project implementation period compared to the situation prior to the intervention. Conversely, a statistically significant decrease was recorded in one HF. In the Far North region, no HF exhibited a statistically significant increase, but a statistically significant decrease was observed in two HFs. In the North region, two HFs exhibited a statistically significant increase, and five HFs exhibited a statistically significant decrease.

Skilled birth attendance (SBA)

Table 2 and Figure S 7 shows that by the end of the first month of implementation of the project, a statistically significant increase of nearly 57% in the monthly number of SBAs was recorded in the 37 HFs selected in the study sample, controlling for other variables (IRR = 1.566 [95% CI: 1.358, 1.806]).

A statistically significant increase in this indicator was also observed in each of the three regions. However, there were disparities between HFs. In the Adamawa region, out of 13 health facilities, there was a statistically significant increase in the monthly number of assisted deliveries at the end of the first month of project implementation in nine HFs and a statistically significant decrease in one HF. In the Far North region, of the 11 HFs, a statistically significant increase was recorded in eight HFs and a statistically significant decrease was recorded in two HFs. In the North region, of the 13 HFs, seven recorded a statistically significant increase and one a statistically significant decrease in the indicator of interest.

In addition, Table 2 and Figure S 8 indicates that, overall, the intervention had a positive effect on SBAs (IRR = 1.009 [95% CI: 1.002, 1.016]). A similar finding is observed in the three regions, with the Far North region being the only region that was statistically significant. When considering the analysis of HFs, the results are mixed. In the Adamawa region, a positive and statistically significant result was recorded for four HFs while a negative and statistically significant result was observed for three HFs. In the Far North region, statistically significant results were recorded for five HFs and all these results were positive. In the North region, two HFs recorded a positive result and three recorded a negative result.

The high values of the I 2 statistics reveal that a very large proportion of the total observed variance is due to a real difference in effect measures between HFs (Figure S 1 to S 8 ).

Our study explored the effect of the Health voucher Project on the use of health services. Overall, a statistically significant increase was observed in the number of first ANCs at the end of the first month of project implementation (success). However, this improvement was not sustained over time, with less than 10% of all HFs (3/33) experiencing an increase in ANCs.

For the SBAs, there was a statistically significant increase at the end of the first month of project implementation, with a sustained pattern over time. When looking at the individual HFs, 2/3 (65%) recorded success at the end of the first month of implementation, while 30% experienced overall improvement during the project implementation compared to the period before the start of the project.

These findings suggest that between the pre-intervention/roll-out and full implementation phases, the Cameroon voucher programme modestly increased the use of facility for ANC and SBA, consistent with previously reported results from evaluations of maternal health voucher programmes from other LMICs [ 21 , 35 , 36 , 37 , 38 ].

Our results therefore indicate that in a country such as Cameroon, where progress toward universal health coverage is still to be achieved [ 39 ], reducing financial risk by providing subsidies to offset the costs of receiving RMNCAH services may be a good cost-effective intervention to improve service utilization.

Pregnant women were more likely to use the voucher system for SBAs than for the first ANC visits. One explanation could be the late attendance of pregnant women at health facilities, as more than 70% of pregnant women in these three regions are reported to have their first contact with a health facility after the first trimester of pregnancy [ 27 ], or the late acquisition of vouchers. In-depth discussions with health care providers and direct beneficiaries are needed to better understand the realities underlying these trends.

The decrease in first ANC and SBA over time in some HFs could be explained by the increasing expansion of service coverage, with the opening of new health facilities that were not yet included in the project and that were used by some pregnant women. On the other hand, the context of growing insecurity linked to Boko Haram and other rebel groups in neighboring countries could also constitute a barrier to the use of health facilities in these regions.

It is also important to note that the voucher program is conceptually designed to target the poorest populations. In Cameroon, however, the project covers all women of reproductive age in the intervention areas, regardless of socioeconomic status. We suspect that the contribution of the 6,000 FCFA ($11 US) remains a major barrier to the use of health services for the poorest women, especially since the project covers mostly urban areas, raising the question of program equity as reported elsewhere [ 13 , 14 , 16 ]. This challenge was also highlighted in an unpublished qualitative study.

Focusing on strategies that prioritize the poorest women and strengthen community engagement can ensure equity and achieve sustainable results over time. For example, in Bangladesh and Cambodia, the voucher programme focused on those most in need and reimbursed care givers in facilities to motivate them [ 40 , 41 ]. Moreover, both countries have successfully partnered with recipient communities to improve the targeting of the poor [ 40 , 41 , 42 ].

In addition to stimulating demand, voucher schemes are often proposed as a way to improve the quality of care, as is the case in Cameroon, where health facility accreditation mechanisms are used, alongside the performance-based financing scheme implemented nationwide. However, experiences show that providers may find reimbursement rates to be unattractive and engage in practices such as providing inconsistent quality of care or ‘skimming’ programme users who require minimal intervention. Moreover, as reported in other voucher programs, the most significant problem faced by the voucher scheme in Cameroon was the delay in paying for health facilities, which led to staff demotivation and mistrust between the managers of the scheme and the beneficiaries [ 41 ] and suggested a need for greater attention to issues related to implementation in such a program [ 26 ].

This study helps to extend the body of knowledge generated by previous research on health voucher programmes in LMICs. However, in interpreting our findings, the strengths and limitations of the study design should be considered.

First, most studies on voucher programmes to date have examined the immediate or shorter-term impact of the intervention on service utilization [ 21 ]. Our study examined the immediate to longer-term effects of the intervention and used a quasi-experimental design, known as a reliable approach, to provide robust estimates of the effect of an intervention when a randomized controlled trial cannot be conducted or when a control group is lacking [ 29 , 31 ]. Unlike in cross-sectional observational studies, interrupted time series analysis allows us to estimate the dynamics of change driven by the intervention, controlling for secular changes that might have occurred in the absence of the intervention [ 29 , 43 ]. This approach thus makes it possible to observe whether the intervention has an immediate or delayed, sudden or gradual effect and whether this effect persists or is temporary. Furthermore, there is no real consensus on the number of observation points needed to use the interrupted time series method. However, the statistical power increases with the number of time points [ 30 ]. Some authors recommend 12 observation points before and after the start of the intervention [ 29 ]. In our study, only one HF had 10 observation points before the start of the project, and the others had observation points ranging from 14 and 42. During the project implementation period, the number of observations varied between 23 and 37.

At the time of the study, 81 facilities had already enrolled in the voucher project. We limited ourselves to 33 HFs for the first ANC and to 37 HFs for the SBA analysis because the data prior to the project were either unavailable or insufficient. Therefore, the results presented in this study may be a fragmented view of the project’s effect. In addition, analysis that could provide insight into the RMNCAH continuum of care was not possible due to the limited quality of data (high frequency of missing data) for some key indicators, such as the fourth ANC and postnatal consultation, as reported with other voucher programs [ 22 , 44 , 45 , 46 ].

In identifying the impact of an intervention, it is important that there are no exogenous factors influencing the results. During the implementation of the voucher program in Cameroon, there were no closures of health facilities that could have an impact on the two selected indicators. Population growth naturally leads to an increase in the number of pregnant women in absolute terms, and consequently to an increase in the number of SBAs. Because demographic data were only available for each health district and not for each health facility, estimates of expected populations or pregnant women were not included into the various negative binomial regression models as a control variable. As a result, the estimates obtained may be biased.

It is also important to point out that due to its fragility, the northern part of the country is a convergence zone of several programs and projects, including those of health. Therefore, other interventions may have also contributed to the achievement of these outcome levels. One of the most important programmes is the National Multi-sector Program to Combat Maternal, Newborn and Child Mortality, which was created in 2013.

Finally, we would like to underline that the fidelity of the program's implementation was hampered by deviations, leading for instance to extending the intervention to all women of childbearing age. At present, the program is more akin to an obstetric risk insurance system, as described for example in Mauritania [ 47 ].

This study provided important insight into the Cameroon voucher scheme. The intervention had a significant early effect on the first ANC and SBA but failed to effectively sustain these results over time for the first indicator. These mixed effects were likely the consequence of poor design and implementation challenges, including the fact that the programme did not include specific equity measures to facilitate uptake by the poorest people. This suggests that for a complex intervention such as a voucher, it is critical to properly implement practice strategies that can sustain the long-term impact of the programme.

Availability of data and materials

The data that support the findings of this study are available from the Ministry of Public Health (MPH) of Cameroon, but restrictions apply due to the terms of our contract with the MPH, and so, data are not publicly available. The corresponding author should be contacted for the process to request data access.

Abbreviations

• Antenatal consultation

Centre International de Développement et de Recherche

Emergency obstetric and neonatal care

Communauté financière africaine

Health facility

Low- and middle-income country

Maternal mortality ratio

Multiple Indicator Cluster Survey

Ministry of public health

Out-of-pocket

Post-natal care

Regional Funds for Health Promotion

Reproductive, maternal, neonatal, child and adolescent health

• Skilled birth attendance

Sustainable Development Goals

Sub Saharan Africa

United States dollar

World health organization

Nations Unies. Développement durable. 2021. Objectif de Développement Durable - Santé et Bien-Être pour tous. 2021. Available at: https://www.un.org/sustainabledevelopment/fr/health/ . Accessed 3 June 2023.

WHO, UNICEF, UNFPA, World Bank Group, United Nations Population Division. Trends in maternal mortality: 2000 to 2017. Geneva, Switzerland: World Health Organization; 2019. p. 119. Available at: http://www.who.int/reproductivehealth/publications/maternal-mortality-2000-2017/en/ . Accessed 10 May 2023.

UN Inter-agency Group for Child Mortality Estimation. Levels & Trends in Child Mortality. New York, NY 10017 États-Unis: United Nations Children’s Fund; 2020 p. 56. Available at: https://www.un.org/development/desa/pd/sites/www.un.org.development.desa.pd/files/unpd_2020_levels-and-trends-in-child-mortality-igme-.pdf . Accessed 2 Jul 2023.

Institut National de la Statistique (INS). Enquête Démographique et de Santé et à Indicateurs Multiples EDS-MICS 2011. 2011. Available at: https://dhsprogram.com/pubs/pdf/fr260/fr260.pdf . Accessed 2 Feb 2023.

Donnay F. Maternal survival in developing countries: what has been done, what can be achieved in the next decade. Int J Gynecol Obstet. 2000;70(1):89–97. https://doi.org/10.1016/S0020-7292(00)00236-8 .

Article   CAS   Google Scholar  

Singh S, Darroch JE, Ashford LS, Vlassoff M. Adding It Up: The costs and Benefits of Investing in family Planning and maternal and new born health. GUTTMACHER INSTITUTE; 2009. Available at: https://www.guttmacher.org/sites/default/files/pdfs/pubs/AddingItUp2009.pdf . Accessed 20 April 2023.

Tekelab T, Chojenta C, Smith R, Loxton D. The impact of antenatal care on neonatal mortality in sub-Saharan Africa: a systematic review and meta-analysis. PLOS One. 2019;14(9):e0222566. https://doi.org/10.1371/journal.pone.0222566 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wondemagegn AT, Alebel A, Tesema C, Abie W. The effect of antenatal care follow-up on neonatal health outcomes: a systematic review and meta-analysis. Public Health Rev. 2018;39(1):33. https://doi.org/10.1186/s40985-018-0110-y .

Article   PubMed   PubMed Central   Google Scholar  

Matsuoka S, Aiga H, Rasmey LC, Rathavy T, Okitsu A. Perceived barriers to utilization of maternal health services in rural Cambodia. Health Policy. 2010;95(2):255–63. https://doi.org/10.1016/j.healthpol.2009.12.011 .

Article   PubMed   Google Scholar  

Sharma S, Smith S, Sonneveldt E, Pine M, Dayaratna V, Sanders R. Formal and Informal Fees for Maternal Health Care Services in Five Countries. USAID; 2005. Available at: http://www.policyproject.com/pubs/workingpapers/WPS16.pdf . Accessed 30 Jan 2023.

Institut National de la Statistique (INS) and IFC. Enquête Démographique et de Santé EDS 2018. INS et IFC 2020. Available at: https://dhsprogram.com/what-we-do/survey/survey-display-511.cfm . Accessed 4 Jan 2023.

WHO. World Health Organization. 2020. Global Health Expenditure Database. Available at: https://apps.who.int/nha/database/Home/Index/en . Accessed 13 Feb 2023.

Gabrysch S, Campbell OM. Still too far to walk: Literature review of the determinants of delivery service use. BMC Pregnancy Childbirth. 2009;11(9):34. https://doi.org/10.1186/1471-2393-9-34 .

Article   Google Scholar  

Dzakpasu S, Powell-Jackson T, Campbell OMR. Impact of user fees on maternal health service utilization and related health outcomes: a systematic review. Health Policy Plan. 2014;29(2):137–50. https://doi.org/10.1093/heapol/czs142 .

Ensor T, Ronoh J. Effective financing of maternal health services: a review of the literature. Health Policy. 2005;75(1):49–58. https://doi.org/10.1016/j.healthpol.2005.02.002 .

Richard F, Witter S, de Brouwere V. Innovative approaches to reducing financial barriers to obstetric care in low-income countries. Am J Public Health. 2010;100(10):1845–52. https://doi.org/10.2105/AJPH.2009.179689 .

World Health Organization. Making pregnancy safer: the critical role of the skilled attendant: a joint statement by WHO, ICM and FIGO. Geneva: WHO; 2004. 24 p. Available at: https://apps.who.int/iris/bitstream/handle/10665/42955/9241591692.pdf?sequence=1&isAllowed=y . Accessed 15 April 2023.

Baral YR, Lyons K, Skinner J, van Teijlingen ER. Determinants of skilled birth attendants for delivery in Nepal. Kathmandu Univ Med J. 2010;8(3):325–32. https://doi.org/10.3126/kumj.v8i3.6223 .

Bellows NM, Bellows BW, Warren C. Systematic review: the use of vouchers for reproductive health services in developing countries: systematic review. Trop Med Int Health TM IH. 2011;16(1):84–96. https://doi.org/10.1111/j.1365-3156.2010.02667.x .

Brody CM, Bellows N, Campbell M, Potts M. The impact of vouchers on the use and quality of health care in developing countries: a systematic review. Glob Public Health. 2013;8(4):363–88. https://doi.org/10.1080/17441692.2012.759254 .

Hunter BM, Harrison S, Portela A, Bick D. The effects of cash transfers and vouchers on the use and quality of maternity care services: a systematic review. PLOS One. 2017;12(3):e0173068. https://doi.org/10.1371/journal.pone.0173068 .

Nguyen HTH, Hatt L, Islam M, Sloan NL, Chowdhury J, Schmidt JO, et al. Encouraging maternal health service utilization: an evaluation of the Bangladesh voucher program. Soc Sci Med. 2012;74(7):989–96. https://doi.org/10.1016/j.socscimed.2011.11.030 .

Azmat SK, Ali M, Rahman MdM. Assessing the sustainability of two independent voucher-based family planning programs in Pakistan: a 24-months post-intervention evaluation. Contracept Reprod Med. 2023;8(1):43. https://doi.org/10.1186/s40834-023-00244-w .

Nandi A, Charters TJ, Quamruzzaman A, Strumpf EC, Kaufman JS, Heymann J, et al. Health care services use, stillbirth, and neonatal and infant survival following implementation of the Maternal Health Voucher Scheme in Bangladesh: A difference-in-differences analysis of Bangladesh Demographic and Health Survey data, 2000 to 2016. PLOS Med. 2022;19(8):e1004022. https://doi.org/10.1371/journal.pmed.1004022 .

Sultana N, Hossain A, Das H, Pallikadavath S, Koeryaman M, Rahman M, et al. Is the maternal health voucher scheme associated with increasing routine immunization coverage? Experience from Bangladesh. Front Public Health. 2023;2(11):963162. https://doi.org/10.3389/fpubh.2023.963162 .

Hunter BM, Murray SF. Demand-side financing for maternal and newborn health: what do we know about factors that affect implementation of cash transfers and voucher programmes? BMC Pregnancy Childbirth. 2017;17(1):262. https://doi.org/10.1186/s12884-017-1445-y .

Institut National de la Statistique. Cameroon - Enquête par Grappes à Indicateurs Multiples 2014. Yaoundé, Cameroun: Institut National de la Statistique; 2015; p. 504. Available at: https://mics-surveys-prod.s3.amazonaws.com/MICS5/West%20and%20Central%20Africa/Cameroon/2014/Final/Cameroon%202014%20MICS_French.pdf . Accessed 22 Jan 2023

Institut National de la Statistique. Annuaire statistique 2017. 2018. Available at: http://www.statistics-cameroon.org/news.php?id=513 . Accessed May 10, 2023.

Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27(4):299–309. https://doi.org/10.1046/j.1365-2710.2002.00430.x .

Article   CAS   PubMed   Google Scholar  

Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348–55. https://doi.org/10.1093/ije/dyw098 .

Linden A. ITSA: Stata module to perform interrupted time series analysis for single and multiple groups. Statistical Software Components. Boston College Department of Economics; 2021. Available at: https://ideas.repec.org/c/boc/bocode/s457793.html . Accessed 21Oct 2023.

Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE. Interrupted time series designs in health technology assessment: Lessons from two systematic reviews of behavior change strategies. Int J Technol Assess Health Care. 2003;19(4):613–23. https://doi.org/10.1017/S0266462303000576 .

Gebski V, Ellingson K, Edwards J, Jernigan J, Kleinbaum D. Modelling interrupted time series to evaluate prevention and control of infection in healthcare. Epidemiol Infect. 2012;140(12):2131–41. https://doi.org/10.1017/S0950268812000179 .

Borenstein M, Hedges LV, Higgins J, Rothstein HR. Introduction to Meta-Analysis. 2nd ed. Hoboken: Wiley; 2021. p. 500.

Book   Google Scholar  

Obare F., Warren C., Abuya T., Askew I., Bellows B. Assessing the population-level impact of vouchers on access to health facility delivery for women in Kenya. Soc Sci Med. 2014;183–9. https://doi.org/10.1016/j.socscimed.2013.12.007 .

Obare F, Warren C, Kanya L, Abuya T, Bellows B. Community-level effect of the reproductive health vouchers program on out-of-pocket spending on family planning and safe motherhood services in Kenya. BMC Health Serv Res. 2015;15(1):343. https://doi.org/10.1186/s12913-015-1000-3 .

Bellows B, Kyobutungi C, Mutua MK, Warren C, Ezeh A. Increase in facility-based deliveries associated with a maternal health voucher programme in informal settlements in Nairobi Kenya. Health Policy Plan. 2013;28(2):134–42. https://doi.org/10.1093/heapol/czs030 .

Amendah DD, Mutua MK, Kyobutungi C, Buliva E, Bellows B. Reproductive health voucher program and facility based delivery in informal settlements in nairobi: a longitudinal analysis. PLOS One. 2013;8(11):e80582. https://doi.org/10.1371/journal.pone.0080582 .

Sieleunou I, Tamga DM, Maabo Tankwa J, Aseh Munteh P, Longang Tchatchouang EV. Strategic health purchasing progress mapping in cameroon: a scoping review. Health Syst Reform. 2021;10(7):1. https://doi.org/10.1080/23288604.2021.1909311 .

Ir P, Horemans D, Souk N, Van Damme W. Using targeted vouchers and health equity funds to improve access to skilled birth attendants for poor women: a case study in three rural health districts in Cambodia. BMC Pregnancy Childbirth. 2010;10(1):1. https://doi.org/10.1186/1471-2393-10-1 .

Ahmed S, Khan MM. A maternal health voucher scheme: what have we learned from the demand-side financing scheme in Bangladesh? Health Policy Plan. 2011;26(1):25–32. https://doi.org/10.1093/heapol/czq015 .

Ridde V, Yaogo M, Kafando Y, Sanfo O, Coulibaly N, Nitiema PA, et al. A community-based targeting approach to exempt the worst-off from user fees in Burkina Faso. J Epidemiol Community Health. 2010;64(01):10–5. https://doi.org/10.1136/jech.2008.086793 .

Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. BMJ Qual Saf. 2003;12(1):47–52. https://doi.org/10.1136/qhc.12.1.47 .

Ahmed S, Khan MM. Is demand-side financing equity enhancing? Lessons from a maternal health voucher scheme in Bangladesh. Soc Sci Med. 2011;72(10):1704–10. https://doi.org/10.1016/j.socscimed.2011.03.031 .

Agha S. Impact of a maternal health voucher scheme on institutional delivery among low income women in Pakistan. Reprod Health. 2011;8(1):10. https://doi.org/10.1186/1742-4755-8-10 .

Van de Poel E, Flores G, Ir P, O’Donnell O, Van Doorslaer E. Can vouchers deliver? An evaluation of subsidies for maternal health care in Cambodia. Bull World Health Organ. 2014;92(5):331–9. https://doi.org/10.2471/BLT.13.129122 .

Philibert A, Ravit M, Ridde V, Dossa I, Bonnet E, Bedecarrats F, Dumont A. Maternal and neonatal health impact of obstetrical risk insurance scheme in Mauritania: a quasi experimental before-and-after study. Health Policy Plan. 2017;32(3):405–17. https://doi.org/10.1093/heapol/czw142 .

Download references

Acknowledgements

We would like to thank Dr Bassirou Bouba and Dr Okala from the voucher project, Dr Yumo Habakkuk and Bashirou Ndindumouh from Research for Development International, Dr Denise Tamga from the Worlb Bank Office, and Dr Aubin Baleba from UNFPA. We are much indebted to the SPARC team for continuously reviewed our work and provided valuable comments. Finally, the authors would also like to acknowledge the work of the anonymous reviewers who provided us with extremely helpful comments and feedback.

This work was supported by the Bill & Melinda Gates Foundation [Grant number: OPP1179622].

Author information

Authors and affiliations.

The Global Financing Facility (GFF), Dakar, Senegal

Isidore Sieleunou

Research for Development International, 30883, Yaoundé, Cameroon

Isidore Sieleunou & Roland Pascal Enok Bonong

You can also search for this author in PubMed   Google Scholar

Contributions

IS and RPEB conceived and designed the study. RPEB managed the data, including quality control, provided statistical advice on study design and analyzed the data. IS drafted the manuscript, and all authors contributed substantially to its revision. All authors agreed to the final approval of the version to be published. All authors agreed to be accountable for all aspects of the work.

Corresponding author

Correspondence to Isidore Sieleunou .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval for the study was obtained from the Cameroon National Ethics Committee for Human Health Research (CNECHHR) (N0 2020/07/1274/CE/CNERSH/SP). Administrative authorization was granted by the Cameroonian Ministry of Health (D30-607/N/MINSANTE/SG/DROS/CRSPE/BBM, N0 631-32-20). All methods were performed in accordance with the relevant guidelines and regulations. The CNECHHR waived the need for participants’ informed consent in this retrospective study because the data used were fully anonymised and aggregated.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sieleunou, I., Enok Bonong, R.P. Does health voucher intervention increase antenatal consultations and skilled birth attendances in Cameroon? Results from an interrupted time series analysis. BMC Health Serv Res 24 , 602 (2024). https://doi.org/10.1186/s12913-024-10962-9

Download citation

Received : 27 December 2023

Accepted : 08 April 2024

Published : 08 May 2024

DOI : https://doi.org/10.1186/s12913-024-10962-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Results-based financing
• Health financing

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Help | Advanced Search

Astrophysics > Astrophysics of Galaxies

Title: active galactic nuclei and star formation in nearby galaxies (agnstrong). i. sample and strategy.

Abstract: We introduce our project, AGNSTRONG (Active Galactic Nuclei and STaR fOrmation in Nearby Galaxies). Our research goals encompass investigating the kinematic properties of ionized and molecular gas outflows, understanding the impact of AGN feedback, and exploring the coevolution dynamics between AGN strength activity and star formation activity. We aim to conduct a thorough analysis to determine whether there is an increase or suppression in SFRs among targets with and without powerful relativistic jets. Our sample consists of 35 nearby AGNs with and without powerful relativistic jet detections. Utilizing sub-millimeter (sub-mm) continuum observations at 450 {\mu}m and 850 {\mu}m from SCUBA-2 at the James Clerk Maxwell Telescope, we determine star-formation rates (SFRs) for our sources using spectral energy distribution (SED) fitting models. Additionally, we employ high-quality, spatially resolved spectra from UV-optical to near-infrared bands obtained with the Double Spectrograph and Triple Spectrograph mounted on the 200-inch Hale telescope at Palomar Observatory to study their multiphase gas outflow properties. This paper presents an overview of our sample selection methodology, research strategy, and initial results of our project. We find that the SFRs determined without including the sub-mm data in the SED fitting are overestimated by approximately 0.08 dex compared to those estimated with the inclusion of sub-mm data. Additionally, we compare the estimated SFRs in our work with those traced by the 4000Å break, as provided by the MPA-JHU catalog. We find that our determined SFRs are systematically higher than those traced by the 4000Å break. Finally, we outline our future research plans.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing

We introduce Area ReSTIR , extending ReSTIR reservoirs to also integrate each pixel’s 4D ray space, including 2D areas on the film and lens. We design novel subpixel-tracking temporal reuse and shift mappings that maximize resampling quality in such regions. This robustifies ReSTIR against high-frequency content, letting us importance sample subpixel and lens coordinates and efficiently render antialiasing and depth of field.

Markus Kettunen

Chris Wyman

Social Representativeness and Intervention Adherence—A Systematic Review of Clinical Physical Activity Trials in Breast Cancer Patients

• 1 Department of Circulation and Medical Imaging, Norwegian University of Technology and Science (NTNU), Trondheim, Norway
• 2 Department of Neuromedicine and Movement Science, Norwegian University of Technology and Science (NTNU), Trondheim, Norway

Objectives: Representativeness in physical activity randomised controlled trials (RCT) in breast cancer patients is essential to analyses of feasibility and validity considering privileged- social groups. A step-by-step exclusion of less privileged groups through the trial process could reinforce health inequality. This study aimed at examining representativeness in breast cancer (BC) physical activity trials, investigate associations between socio-economic status (SES) and intervention adherence, and explore associations between representativeness and the relationship between SES and intervention adherence.

Methods: Systematic, computerised searches were performed in PubMed, CINAHL, AMED, EMBASE and PsycINFO. Additional citation-based searches retrieved 37 articles. Distributions of education level, ethnicity, and marital status in study samples were compared to national populations data to estimate representativeness in less privileged groups.

Results: A preponderance of studies favoured educated, married and white patients. Only six studies reported SES-adherence associations, hampering conclusions on this relationship and possible associations between representativeness and an SES-adherence relationship.

Conclusion: Less educated, unmarried and non-white individuals may be underrepresented in BC physical activity RCTs, while SES-adherence associations in such trials are inconclusive. Unintentional social misrepresentations may indicate that disguised inequity warrants revived attention.

Introduction

Clinical trials in breast cancer (BC) research have contributed significantly to improved treatments, health-related quality of life, and survival probability for BC patients [ 1 ]. For example, numerous clinical physical activity (PA) trials have been conducted. Nevertheless, faced with the reality of social health inequalities [ 2 , 3 ], it is essential to ensure that the results of these trials benefit all, not exclusively the most privileged. Previous analyses have concluded that study samples only slightly represent real-world patient populations [ 4 ] due to participants having better health status, lower age, and a dominant ethnicity. However, research inequity on the part of socioeconomic status (SES)-groups has been scarcely investigated. As PA appears to improve physical fitness [ 5 – 8 ], fatigue [ 5 , 6 , 9 , 10 ], and physical functioning [ 5 , 9 , 11 ] in BC patients and survivors, knowledge about representativeness across SES and privilege in randomised controlled trials (RCT) with PA is warranted.

Privilege refers to (often unrecognised) advantages that benefit people belonging to certain groups based on factors like their SES, ethnicity, age, gender identity [ 12 ]. High-SES individuals, for example, are considered to have health benefits as a result of their belonging to a privileged social group [ 13 ]. SES, is ideally measured as the combination of an individual’s or groups’ education, income and occupation [ 14 ] and it is often used to examine social health inequalities [ 15 ]. In the case of BC, there is an excess risk associated with increased education, in both men [ 16 ] and women [ 17 ]. Unmarried women, who face a multitude of hardships and less privilege [ 18 ], may have a higher risk of developing BC [ 19 ]. Similarly, less privileged non-whites have higher odds of being diagnosed with aggressive BC [ 20 ]. However, BC in men and less privileged women is generally diagnosed later, making these cases more severe compared to BC diagnosed in privileged women [ 21 ]. Despite higher incidence rates of early BC, higher survival rates and better health-related quality of life after diagnosis [ 22 ] are associated with higher education [ 23 ]. In young men with invasive BC, overall survival is found to decrease in groups living in ZIP-code areas associated with low SES [ 24 ]. Presumably because <1% of all BC incidents occur in men [ 25 ], most PA research concern women. Female patients often report weight gain, which is associated with undesirable BC-outcomes [ 26 – 28 ]. Thus, it is often recommended that they pursue regular PA [ 29 ]. An inactive lifestyle is suggested to be a significant social determinant of decreased BC survival probability in low-SES groups [ 30 ], and according to Boer et al., there is strong evidence that BC recurrence and mortality are strongly associated with leisure-time PA (LTPA) due to the biological mechanisms affected [ 31 ].

PA is defined as any bodily movement produced by skeletal muscles that results in energy expenditure [ 32 ] and includes the active transport-, leisure time- (including exercise), job-related-, and household PA domains [ 33 ]. The relationship between PA and the risk of BC is well documented [ 34 – 44 ] and is evident for LTPA in postmenopausal BC [ 45 , 46 ]. However, it has been found that different SES-groups in general are inclined to different domains of PA, with high-SES groups being more physically active within the LTPA domain [ 47 ]. Comparable results are reported in studies of women with BC [ 48 – 51 ]. Hence, for RCTs with LTPA, there is a risk of selection bias, which is also acknowledged in the Exercise guidelines for cancer survivors: “ …the individuals enrolled in studies commonly meet prespecified eligibility criteria (…) and were willing to take part in research. This often results in a sample that is healthier or with higher physical function and exercise motivation that may not fully generalize to the broader population of cancer survivors. ” [ 52 ].

Moreover, if there are SES-differences in PA among women with BC in general, participation and adherence to PA trials may differ accordingly. Studies have shown different PA barriers across SES [ 53 , 54 ], partly explaining variances in activity levels [ 55 – 59 ]. If low-SES individuals perform less LTPA than high-SES individuals [ 47 ], the former may be prone to lower participation and adherence rates in exercise RCTs. In addition to findings of better exercise trial adherence in high-SES groups [ 60 , 61 ], less privileged individuals less frequently volunteer to participate in research [ 62 , 63 ], due to barriers to access [ 62 ], lower health literacy and negative attitudes towards research, additional costs, or disease status [ 64 ]. These findings may imply that RCTs with which a privileged group is more familiar and motivated for the provided intervention (i.e., LTPA/exercise) run higher risks of selection bias than RCTs for which compliance is less affected by social grouping.

Hence, the external validity of LTPA trials for BC patients may be impaired, albeit unintendedly, by a stepwise exclusion of less privileged patients; they appear to be recruited and participate less frequently, and may also have lower adherence rates [ 65 ]. The general view is that RCT samples must be homogeneous to gain internal validity and reliable results [ 66 , 67 ]. Simultaneously, the declarations of Helsinki [ 68 ], legitimately and necessarily, prevents researchers from obliging participation for the purpose of representativeness. Consequently, social biases may be fortified [ 64 , 65 ].

The credibility and success of medical progress depend on transparent reporting [ 69 ]. Thus, the CONSORT guidelines developed to improve RCT-articles [ 70 ]. Nevertheless, participant attributes, such as SES-indicators, are seldomly reported [ 71 , 72 ], impeding successful assessments of equitable research [ 72 ]. A reasonable alternative may be to employ the available variables that indicate, or strongly correlate with, privilege and SES. Previous reviews of PA-SES associations have found that education is the most reported SES-indicator [ 47 , 73 , 74 ]. Furthermore, frequently reported characteristics, such as ethnicity and marital status, both reflect social privilege and correlate with SES, thus they may be considered fair representations of privilege in the absence of precise SES-indicators.

The objective of this systematic review was to study previous PA RCTs on BC patients with the intention of examining a) SES-related information and representativeness, b) associations between SES-related indicators and PA intervention adherence, and c) associations between representativeness and reported relationships between SES and intervention adherences. There are, of course, real obstacles to recruting less privileged patients, so non-representative samples should not be seen as researchers’ unwillingness or lack of effort to include the less privileged. The overall aim of the study was not to evaluate single studies conducted for purposes other than representativeness per se , but rather to highlight disguised patterns across comparable studies [ 75 ]. Furthermore, the aim was not to provide meta-analyses and precise sample- vs real-world ratios; however, the study may render a departure point for improving representativeness on the part of less privileged groups.

Systematic, computerised searches were conducted in the PubMed, CINAHL, AMED, EMBASE and PsycINFO databases. The first round followed a traditional method [ 76 ]; we specified a search query by a set and combinations of words, and all publications indexed in the databases that contained those words were returned. “Compliance,” “persistence,” “fidelity,” “maintenance” and “concordance” were used as synonyms for the variable “adherence.” Similarly, “physical,” “exercise,” “fitness,” “sport” and “training” were used as synonyms, in addition to exercise-specific terms, such as “dance,” “swim,” “walk” and “yoga,” to cover any type of PA intervention. An asterisk (*) was attached to the roots of the words to broaden the search, and to retrieve variations on these terms. Because the terms are used interchangeably in the literature, both “ patients ” and “ survivors ” were used to cover relevant study samples. As recommended by Bramer et al. [ 77 ]. Boolean combinations were used to construct suitable search queries in combination with the basic term “breast cancer” (see Figure 1 ). There were no search limits as to publication year.

Figure 1 . Flow chart of inclusion and exclusion of articles through descriptive- and citation-based searches (Global, 2000–2020).

The following criteria guided the inclusion and exclusion process, defining the final dataset:

Inclusion Criteria

a) Empirical studies

b) …reporting RCTs including a PA intervention.

c) …including individuals undergoing BC treatment within one-year postdiagnosis.

d) …reporting adherence to the PA intervention.

e) …presenting a comprehensible calculation of adherence.

f) …written in English.

Exclusion Criteria

a) Reviews, theoretical and descriptive papers, books, theses, letters to editor or editorials.

b) …reporting on multiple cancer diagnoses (even if BC was included).

c) …not reporting any SES-related characteristics of the sample … written in any non-English language.

The descriptive searches retrieved 1,019 articles. After excluding immediate duplicates, 766 articles were manually examined and deemed potentially relevant based on a screening of titles and abstracts. This left, 166 potentially relevant articles that were examined to establish their eligibility according to the criteria. An algorithm for all variables was implemented using Python3 to ensure the eligibility of the articles selected. In total, 143 articles were excluded for a total of 23 remaining articles ( Figure 1 ).

To retrieve all possible relevant studies, thus reducing the risk of missing information [ 78 ], citation-based searches (described by Hu et al. [ 79 ], but implemented in Rysstad and Pedersen [ 80 ] and Darvik et al. [ 81 ]) were performed based on the 23 articles found in the descriptive searches. A forward citation-based (FCB) search of articles that cited the articles included in the descriptive searches was first reviewed for eligibility, and 34 potentially relevant articles were identified, 24 of which were excluded. A backwards citation-based (BCB) search, where the reference lists in the 33 articles already included in the material were scanned for eligible studies, retrieved 5 articles of which one replaced a previously included article, following the inclusion criteria.

In cases in which an article reported on two different parent trials, the article was included on the condition that at least one of the parent studies met inclusion criteria c. If two articles were reporting on the same parent RCT sample and overlapped in their data, the earliest article was included. If, however, only one of them reported adherence rates, the article omitting adherence rates was excluded. Nevertheless, the included article had to report on SES-related patient characteristics. In the subsequent step of the inclusion/exclusion process, only articles reporting the samples’ SES distributions were included for further analyses.

The second author performed the database searches but conferred with the first author to discuss any cases of doubt about the potentially relevant articles returned from the query.

Data Extraction and Analysis

The following variables were extracted from the articles: sample descriptions, intervention designs, aims of the studies, measures of SES or privilege, their distributions, and adherence calculations and -rates. For studies that investigated the associations between adherence and SES-related factors, the reported results were registered.

The SES-related distribution in the sample was compared with the corresponding distribution within the country in which the study was conducted, at the matching time of publication. Because adherence to intervention protocols was embedded in our study objectives, only intervention group characteristics were analysed. Due to third-party researchers’ limited access to patient population data, SES-related distribution in a country population, in the corresponding age groups, was used as a proxy for the patient population. For the education variable, the percentage of the study sample holding any tertiary education (no degree required) was compared with the percentage of the country population holding the same educational level (i.e., all formal postsecondary education, including public and private universities, colleges, technical training institutes, and vocational schools [ 82 ]), using public statistics for reference [ 83 – 86 ] (see Supplementary Files S2, S3 for details). Because the final data set included articles about women exclusively, all statistics retrieved and analysed, were relevant to women.

Similarly, the proportion of married women in the study samples and in the associated country populations was compared using United Nations World Marriage Data (age 45–49) [ 87 ] and a corresponding analysis for ethnicity based on national censuses [ 88 – 92 ]. Frequencies in the white alone and non-indigenous and non-visible minority categories from accessible censuses were used to estimate figures for intercensal years by interpolation. The relative differences between the study samples and the associated countries were calculated as follows:

Studies reporting on the association between SES-related factors and adherence to the PA intervention were further explored, and variations in adherence calculations were examined.

The preliminary screening excluded 20%–25% of all potentially relevant articles due to a lack of information about SES-related characteristics. The subsequent selection process, following descriptive, FBC- and BCB-searches, retrieved 37 eligible articles ( Figure 1 ).

A total of 28 (76%) of the included studies were published between 2010 and 2020, and the oldest article was published in 2002 [ 93 ]. Most studies (81%) were conducted in Western countries: 18 in the United States, 5 in Canada, two in the Netherlands, and one each in Spain, France, the United Kingdom, Denmark and Sweden. Six (16%) studies were Asian, including one Indian, two Chinese and three Taiwanese studies. In addition, one Brazilian study was included. The (full) sample sizes ranged from 14 [ 94 ] to 301 [ 95 ], including 11 studies with <50 participants, 15 studies with 50–96 participants, and 11 studies with 100–301 participants. The mean age ranged from 42.1 to 63.2 years (total mean = 51 years) ( Table 1 ). All included studies were of female BC patients.

Table 1 . Study aim, sample, SES and intervention descriptions, adherence formula, and adherence outcome in included studies. (Global, 2000–2020).

Education was the most frequently used indicator of privilege and was found in 30 (81%) of the articles [ 93 – 122 ] ( Table 1 ). Marital status was reported in 28 (76%) [ 93 – 95 , 97 – 99 , 101 – 109 , 111 – 115 , 122 ], ethnicity in 22 articles (59%) [ 93 – 95 , 102 , 105 , 108 , 110 , 112 , 114 – 116 , 119 , 123 – 128 ] employment status in 25 articles (68%) [ 93 – 95 , 98 , 101 , 102 , 104 , 106 – 111 , 113 – 115 , 118 , 119 ], and income in eight articles (22%) [ 95 , 96 , 103 , 104 , 114 , 116 , 122 , 123 ]. Two articles [ 95 , 114 ] reported five indicators, and four [ 124 , 126 , 127 , 129 ] reported one. Ethnicity was reported in all 18 USA studies, as well as in four of the five Canadian studies, but it was not reported in the remaining studies. For the rest of the indicators, no geographical patterns were identified.

There was no uniform definition of adherence although the most common calculation was attendance rates relative to total, or weekly, prescribed, or possible, PA sessions. The mean adherence rate for these studies was 78.9% (30.4%–100%). All studies reported on patients diagnosed with BC in stages 0–III.*. Most studies reported the feasibility or effect of a PA intervention on patients undergoing radiation-, chemotherapy, or both.

The PA-interventions were designed differently in terms of PA type (e.g., walking, exercise including endurance and strength, yoga, aerobics, martial arts, dancing, cycling, qigong, balance training, Pilates), intensity (e.g., according to Ainsworth et al. [ 130 ], yoga is performed at 2.5 METs, whereas bicycling could require 4–16 METs), duration (10–60 min/session, 6 weeks–12 months/intervention) and frequency (voluntarily–5 bouts/week).

Privileged Groups and Adherence to PA Interventions

Some articles were feasibility studies centred on adherence; others reported adherence as a sub-analysis. Six studies reported on the relationship between SES-related factors and adherence to or completion of the intervention [ 97 , 104 , 107 , 109 , 111 , 114 ]. However, two of these studies reported no significant difference in participant characteristics between withdrawers and completers [ 111 ] or between those with 75% attendance and other participants [ 114 ], without specified SES-indicators. Three of these six studies indicated a positive, however weak, association between adherence to an aerobic- and strength training protocol and educational level [ 97 , 104 , 109 ]. One study [ 107 ] noted that employment status was associated with adherence measured as intensity, but not when adherence was assessed in terms of exercise time. No differences in adherence across educational levels were reported in this study.

Representativeness in terms of Research Participants’ Education, Ethnicity and Marital Status

Figure 2 displays the sample-country differences in educational level. Eight studies had a lower proportion of individuals with higher education compared to the country population [ 97 – 99 , 101 , 106 , 107 , 114 , 121 ], whereas 21 studies showed the opposite trend. In 11 of these studies, the proportion of participants who had tertiary education in the population was >25% lower compared to the sample [ 94 , 96 , 100 , 102 – 105 , 109 , 111 , 120 , 122 ]. Four studies had a difference ≤10% [ 93 , 95 , 108 , 115 ] ( Figure 2 ; Table 1 in Supplementary Files S1–S3 ).

Figure 2 . Relative differences (%) between study samples and country populations considering either mean years of education (*) or % of women with >1 year of high education in study sample. Bars extending the right side indicate studies with high-education group overrepresentation (Global, 2000–2020).

The mean proportion of married women in the relevant samples was 71.9% (SD = 11.68), while the corresponding number was 73.9 (SD = 7.32) for the country populations ( Figure 3 ; Table 2 in Supplementary File S1 ). Sixteen of the studies using marital status reported their results on samples with a lower proportion of married women compared to the country population [ 93 , 95 , 97 , 98 , 103 , 104 , 107 , 108 , 111 – 115 , 121 , 122 , 125 ], while 11 examined samples with a higher proportion of married women compared to the country population [ 94 , 99 , 101 , 102 , 106 , 109 , 117 – 119 , 123 , 131 ]. One study [ 105 ] showed no difference (Tables 2, 3 in Supplementary File S1 ).

Figure 3 . (A) Study origin, publication year, and proportion (%) of married and non-married women in sample (left) and country (right). (B) Study origin, publication year, and proportion (%) of whites and other ethnicities in sample (left) and country (right) (Global, 2000–2020).

In the 22 studies that reported ethnicity , the mean proportion of whites was 81%: 77.6% Canadian studies, and 81.8% American studies ( Figure 3 ; Table 3 in Supplementary File S1 ). Three of these studies reported their results on samples with a lower proportion of whites compared with the country population, while the remaining 19 studies had samples with a higher proportion of whites. There was a difference between samples and countries of 10%–25% in 15 of these studies (Table 3 in Supplementary File S1 ).

Representativeness and Adherence

There was no clear relationship between the registered representativeness and the associations between indicators of SES and adherence. Three of the six mentioned studies reporting such associations [ 97 , 107 , 114 ] had samples with a lower proportion of high-SES BC patients compared to their country populations, while the other three [ 104 , 109 , 111 ] had samples with a higher proportion of high-SES patients.

To our knowledge, this is the first systematic review to report the social representativeness of RCTs examining the effects of PA on BC-related outcomes. In the included dataset, there was a preponderance of studies overrepresenting groups associated with higher privilege, supporting previous results indicating social bias [ 132 ]. Misrepresentations were found across different indicators of privilege, likely because different indicators tend to interrelate [ 133 ]. Our findings coincide with earlier observations that low-SES patients are less inclined to participate in research than high-SES patients [ 62 , 63 ], as well as with previous findings of obstacles to recruiting less privileged populations to cancer RCTs [ 64 ].

The most pronounced findings were based on studies that reported participants’ education. However, in six studies that did not record education but rather recorded ethnicity/race, as in the total dataset, there was a preponderance of white participants. In agreement with previous reports [ 62 , 132 , 134 ], this was interpreted as recruitment bias, although these analyses were based on less accurate data compared to the analyses on education. All 22 studies reporting ethnicity were from the USA and Canada, where the ethnic distribution varies over time and space, particularly in the United States [ 91 ]. Although education and income relate to ethnicity [ 135 ], our results on the representativeness of ethnic groups are unclear due to inaccurate data and increased confusion among the populations regarding ethnic categories. In general, it may appear inappropriate to use the category white as an exclusive indicator of social privilege in global comparative analyses such as the current study. Nonetheless, in the overall discussion of representativeness in RCTs, research favouring certain ethnicities over others should not be supported.

Representativeness was further assessed by marital status, and the mean difference between the study samples and the associated country populations was small. However, the differences ranged from −78% to 17.7%, and only 9 studies differed less than 5% from the country population. This means that although marital status is considered a measure of whether people have the social and financial benefits of being coupled, it could affect the possibility of being included in a PA trial.

What is considered an acceptable level of representativeness in RCT samples is difficult to decide. Recruitment issues may lead to sample biases, as patient populations often differ across social groups due to their diagnoses or hospital characteristics. However, the body of articles from the current dataset that employed education as the SES-indicator included eight studies with ( “negative” differences from the country population and 22 studies with “positive” differences (including 11 samples with >25% difference from the country population), which are significant findings. The fact that studies with a negative deviation from the country’s corresponding distribution in this subgroup were published after 2015 could be an effect of the CONSORT guidelines [ 70 ] and the Belmont report [ 136 ], which emphasised the implications of selection biases. However, our screening and analyses clearly show that there is still a potential for improvement among researchers in their attention to social distribution in samples.

Our results may have been more valid if baseline characteristics were compared with a real-world patient population and a patient sample (see [ 4 ]). However, such analyses require that third-party researchers also have access to exact data about the patient populations within each region, at the precise time of the studies, in addition to which patients are eligible according to each of the RCT criteria. The primary aim of the present study was to revive the question of external validity and feasibility from a perspective of social health inequalities in PA research. Therefore, status distribution in the respective age groups within the country populations served as an acceptable proxy for the distribution in the patient population.

Another objection may be that, for a patient to be considered eligible for a PA RCT, their cancer must be at a stage compatible with physical exertion and that more advanced BCs are associated with lower privilege [ 137 ]. Hence, RCTs with exercise often include patients with higher SES (education) because they have less advanced BC. However, this claim confirms the importance of putting focus on representativeness in RCTs targeting an already socially skewed patient population; it is important to be even more aware of sample representativeness to avoid excessive bias by favouring the already privileged. Likewise, employing already skewed patient populations as a reference could promote further social inequalities in health. Nonetheless, if the reference ratio for assessing representativeness was 1.22 [ 17 ], it would be possible to accept 22% more high-education patients before claiming misrepresentation of less privileged groups. However, according to our results, based on patients’ educational levels, approximately 45% of the included studies would still be considered biased in favour of the most privileged patients.

SES and PA Intervention Adherence

A proper body of articles with analyses of the association between SES-related characteristics and adherence was expected, and also that these articles would provide a basis for a synthesised examination of this relationship. However, the fact that only 6 studies investigated this association suggests that this question is perceived to be of little relevance in the field of BC and PA research. A previous review on the representativeness of RCTs reported no socioeconomic misrepresentation in oncology studies [ 4 ] however, our interpretation is supported by the fact that the review included only articles that provided a representativeness analysis, which the oncology articles had omitted in the case of SES.

No clear trend of associations between SES-related factors and adherence was seen across the six aforementioned studies, and the small number of studies and the differences between them hampered a clear conclusion. Previous studies have argued that poor representativeness may explain a lack of associations between SES and adherence rates [ 138 , 139 ]. In the current data set, there was no clear association between representativeness and SES-adherence associations. However, the small number of relevant articles formed an overly scarce base for reliable and conclusive analyses of this research question.

A tendency for studies that found a positive relationship between SES-related factors and adherence to report on longer interventions and to calculate adherence in terms of rates of attendance was observed. Three studies reported that the adherence rates decreased over time [ 97 , 104 , 107 ]. Previous results support the interpretation that time is a barrier to participation in PA interventions [ 53 , 140 ]. Strazdin et al. have substantiated how availability of time may have a larger impact on low-SES women in general [ 141 , 142 ]. Hence, the limited availability of time over a prolonged period may partly explain why less privileged patients have poorer adherence in such trials than their privileged counterparts.

Reasons for refusing to participate, or reasons for withdrawing, were not registered in our study. We should not ignore the fact that researchers experience real obstacles in recruiting of less privileged patients, so non-representative samples should not be seen as unwillingness or lack of effort to include the less privileged. There are reasons to believe that reasons for refusal to participate in the articles included coincide with the most common barriers previously reported [ 10 , 64 , 131 ] . However, although participation is voluntary and individual motivation or barriers are relevant, the adherence rates and the current social group distribution of patients in RCTs with PA may also be affected by the intervention design and the inclusion criteria defined by the researchers of each individual study [ 64 ].

All the included studies reported LTPA interventions. Considering that low-SES groups engage less in LTPA [ 47 ], it is reasonable to expect that low-SES patients were less inclined to participate in, and complete, such RCTs. Hence, a preponderance of RCTs employing exercise interventions could be perceived as being in favour of privileged patients. The searches of the current study did not, however, include search terms covering other PA domains and may thereby have strengthened the impression that LTPA is the prevailing protocol in RCTs of PA in BC patients. However, although many patients return to work during their first-year post diagnosis, RCTs designed for other PA-domains would not be verifiable as the patients’ level of PA would differ too much to control. In addition, the control group would have to be inactive within these PA-domains, which would be a requirement almost impossible to implement. The probability that many published RCTs of other types of PA exist, thus altering our results, is therefore small.

Consequences of Enhanced Selection Bias in Favour of Privileged Patients

Provided that our results reflect reality as they are based on synthesised data from a large set of systematically selected articles, social equity is a challenge to PA trials in BC patients. A stepwise, albeit unintended, social exclusion process causes attrition bias and a successive decrease in external validity, (i.e., the results are even less valid for individuals in other social groups than those who have completed the intervention when the intervention is finalised) [ 65 ]. Thus, PA-treatment is also less adaptable to patients in the same groups as those who withdraw. Drawing on the fundamental cause theory [ 143 ], these mechanisms show how affluent people could benefit more, in this case, from PA research, because studies serve the interests of these groups more than they care for all SES-groups. In addition, insignificant results from sub-analyses of social group-differences may be misinterpreted because the distribution of social groups in the samples does not mirror real life. This may disguise the possible fact that although all patients, by policy, should have equal access to health services, PA treatment interventions may not be suitable for all. An “inverse PA research law” analogous to the “inverse healthcare law” [ 144 ] describes how PA research interventions, initiated with the intention to treat a patient population, regardless of social status, nevertheless attend to privileged groups more than the less privileged who initially are prone to poorer health and thus more entitled to clinical research.

Strengths and Limitations

The major strength of this research is the thorough and systematic search of eligible studies, including traditional, and both BCB and FCB searches. To avoid our searches and not be included, an article would have to be published in a journal that is not indexed in any of the chosen databases, not be identified by the search terms, not be included in either the reference lists of the articles included from the descriptive searches or not have cited the articles included from the descriptive searches. It is possible that some articles were lost. However, it is less likely that any articles not included would be systematically more representative compared with the articles included.

The sample sizes in one-third of the studies were <50, seemingly decreasing the validity of our conclusions. However, rather than being a shortcoming of our study, the limitation is applicable to each RCT because the small samples, among other factors, hamper representativeness analyses such as ours. Hence, our analyses were based on the publication reality, whatever flawed original data materials.

SES and social privilege classifications vary across cultures, research fields, and public statistics, and they may cause imprecise standards for comparisons between studies of different origins. Nevertheless, our results present the big picture of misrepresentation across social groups with a precision sufficient to boost the debate within the scientific community.

Less educated, unmarried and non--white individuals may be underrepresented in BC PA RCTs, and SES-adherence associations in such trials are inconclusive. Unintentional social misrepresentation may create disguised inequity, warranting revived attention to this issue.

The current study provides a departure point for intensified attention to representativeness in RCTs. It should act as motivation both for seeking improved external validity, and for reconsidering whether LTPA in BC treatment is suitable for all.

Author Contributions

RS and MD made substantial contributions to the study: RS conceived the idea of the study, including its design, methodology and analysis. MD and RS performed descriptive- and FC searches, and BC searches, respectively, and the subsequent inclusion/exclusion of articles. RS and MD performed the analyses. RS drafted the manuscript. RS and MD reviewed and edited the manuscript and approved the final version. All authors contributed to the article and approved the submitted version.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of Interest

The authors declare that they do not have any conflicts of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.ssph-journal.org/articles/10.3389/ijph.2024.1607002/full#supplementary-material

1. Wickerham, DL, O'Connell, MJ, Costantino, JP, Cronin, WM, Paik, S, and Geyer, CE. The Half century of Clinical Trials of the National Surgical Adjuvant Breast and Bowel Project. In: Seminars in Oncology . Amsterdam, Netherlands: Elsevier (2008).

CrossRef Full Text | Google Scholar

2. Marmot, M, and Bell, R. Social Inequalities in Health: A Proper Concern of Epidemiology. Ann Epidemiol (2016) 26(4):238–40. doi:10.1016/j.annepidem.2016.02.003

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Mackenbach, JP. Persistence of Social Inequalities in Modern Welfare States: Explanation of a Paradox. Scand J Public Health (2017) 45(2):113–20. doi:10.1177/1403494816683878

4. Kennedy-Martin, T, Curtis, S, Faries, D, Robinson, S, and Johnston, J. A Literature Review on the Representativeness of Randomized Controlled Trial Samples and Implications for the External Validity of Trial Results. Trials (2015) 16(1):495–14. doi:10.1186/s13063-015-1023-4

5. Dantzer, R, Meagher, MW, and Cleeland, CS. Translational Approaches to Treatment-Induced Symptoms in Cancer Patients. Nat Rev Clin Oncol (2012) 9(7):414–26. doi:10.1038/nrclinonc.2012.88

6. Furmaniak, AC, Menig, M, and Markes, MH. Exercise for Women Receiving Adjuvant Therapy for Breast Cancer. Cochrane Database Syst Rev (2016) 9:CD005001. doi:10.1002/14651858.CD005001.pub3

7. Irwin, ML, Crumley, D, McTiernan, A, Bernstein, L, Baumgartner, R, Gilliland, FD, et al. Physical Activity Levels before and after a Diagnosis of Breast Carcinoma: The Health, Eating, Activity, and Lifestyle (HEAL) Study. Cancer Interdiscip Int J Am Cancer Soc (2003) 97(7):1746–57. doi:10.1002/cncr.11227

8. Protani, M, Coory, M, and Martin, JH. Effect of Obesity on Survival of Women with Breast Cancer: Systematic Review and Meta-Analysis. Breast Cancer Res Treat (2010) 123(3):627–35. doi:10.1007/s10549-010-0990-0

9. Husebo, AML, Karlsen, B, Allan, H, Soreide, JA, and Bru, E. Factors Perceived to Influence Exercise Adherence in Women with Breast Cancer Participating in an Exercise Programme during Adjuvant Chemotherapy: A Focus Group Study. J Clin Nurs (2015) 24(3-4):500–10. doi:10.1111/jocn.12633

10. Maddocks, M, Mockett, S, and Wilcock, A. Is Exercise an Acceptable and Practical Therapy for People with or Cured of Cancer? A Systematic Review. Cancer Treat Rev (2009) 35(4):383–90. doi:10.1016/j.ctrv.2008.11.008

11. Bhute, VJ, Ma, Y, Bao, X, and Palecek, SP. The Poly (ADP-Ribose) Polymerase Inhibitor Veliparib and Radiation Cause Significant Cell Line Dependent Metabolic Changes in Breast Cancer Cells. Sci Rep-uk (2016) 6(1):36061. doi:10.1038/srep36061

12. Black, LL, and Stone, D. Expanding the Definition of Privilege: The Concept of Social Privilege. J Multicultural Couns Dev (2005) 33(4):243–55. doi:10.1002/j.2161-1912.2005.tb00020.x

13. Reynolds, MM. Health Power Resources Theory: A Relational Approach to the Study of Health Inequalities. J Health Soc Behav (2021) 62(4):493–511. doi:10.1177/00221465211025963

14. Lampert, T, Hoebel, J, Kuntz, B, Müters, S, and Kroll, LE. Socioeconomic Status and Subjective Social Status Measurement in KiGGS Wave 2. J Health Monit (2018) 3(1):108–25. doi:10.17886/RKI-GBE-2018-033

15. Braveman, PA, Cubbin, C, Egerter, S, Chideya, S, Marchi, KS, Metzler, M, et al. Socioeconomic Status in Health Research: One Size Does Not Fit All. JAMA (2005) 294(22):2879–88. doi:10.1001/jama.294.22.2879

16. D'Avanzo, B, and La Vecchia, C. Risk Factors for Male Breast Cancer. Br J Cancer (1995) 71(6):1359–62. doi:10.1038/bjc.1995.264

17. Dong, J-Y, and Qin, L-Q. Education Level and Breast Cancer Incidence: A Meta-Analysis of Cohort Studies. Menopause (2020) 27(1):113–8. doi:10.1097/GME.0000000000001425

18. Kearney, MS. The Two-Parent Privilege: How the Decline in Marriage Has Increased Inequality and Lowered Social Mobility, and what We Can Do about it . Great Britain: Swift Press (2023).

Google Scholar

19. Li, M, Han, M, Chen, Z, Tang, Y, Ma, J, Zhang, Z, et al. Does Marital Status Correlate With the Female Breast Cancer Risk? A Systematic Review and Meta-Analysis of Observational Studies. PLoS One (2020) 15(3):e0229899.

20. Sineshaw, HM, Gaudet, M, Ward, EM, Flanders, WD, Desantis, C, Lin, CC, et al. Association of Race/ethnicity, Socioeconomic Status, and Breast Cancer Subtypes in the National Cancer Data Base (2010–2011). Breast Cancer Res Tr (2014) 145(3):753–63. doi:10.1007/s10549-014-2976-9

21. Krieger, N, Wright, E, Chen, JT, Waterman, PD, Huntley, ER, and Arcaya, M. Cancer Stage at Diagnosis, Historical Redlining, and Current Neighborhood Characteristics: Breast, Cervical, Lung, and Colorectal Cancers, Massachusetts, 2001–2015. Am J Epidemiol (2020) 189(10):1065–75. doi:10.1093/aje/kwaa045

22. Klassen, AC, and Smith, KC. The Enduring and Evolving Relationship between Social Class and Breast Cancer burden: A Review of the Literature. Cancer Epidemiol (2011) 35(3):217–34. doi:10.1016/j.canep.2011.02.009

23. Trewin, CB, Johansson, ALV, Hjerkind, KV, Strand, BH, Kiserud, CE, and Ursin, G. Stage-Specific Survival Has Improved for Young Breast Cancer Patients since 2000: But Not Equally. Breast Cancer Res Tr (2020) 182:477–89. doi:10.1007/s10549-020-05698-z

24. Flaherty, DC, Bawa, R, Burton, C, and Goldfarb, M. Breast Cancer in Male Adolescents and Young Adults. Ann Surg Oncol (2017) 24(1):84–90. doi:10.1245/s10434-016-5586-4

25. Siegel, RL, Miller, KD, and Jemal, A. Cancer Statistics, 2018. CA: a Cancer J clinicians (2018) 68(1):7–30. doi:10.3322/caac.21442

26. Patterson, RE, Cadmus, LA, Emond, JA, and Pierce, JP. Physical Activity, Diet, Adiposity and Female Breast Cancer Prognosis: A Review of the Epidemiologic Literature. Maturitas (2010) 66(1):5–15. doi:10.1016/j.maturitas.2010.01.004

27. Nichols, HB, Trentham-Dietz, A, Egan, KM, Titus-Ernstoff, L, Holmes, MD, Bersch, AJ, et al. Body Mass index before and after Breast Cancer Diagnosis: Associations with All-Cause, Breast Cancer, and Cardiovascular Disease Mortality. Cancer Epidemiol Biomarkers Prev (2009) 18(5):1403–9. doi:10.1158/1055-9965.EPI-08-1094

28. Healy, L, Ryan, A, Carroll, P, Ennis, D, Crowley, V, Boyle, T, et al. Metabolic Syndrome, central Obesity and Insulin Resistance Are Associated with Adverse Pathological Features in Postmenopausal Breast Cancer. Clin Oncol-uk (2010) 22(4):281–8. doi:10.1016/j.clon.2010.02.001

29. Rock, CL, Doyle, C, Demark-Wahnefried, W, Meyerhardt, J, Courneya, KS, Schwartz, AL, et al. Nutrition and Physical Activity Guidelines for Cancer Survivors. CA: A Cancer J Clinicians (2012) 62(4):243–74. doi:10.3322/caac.21142

30. Lundqvist, A, Andersson, E, Ahlberg, I, Nilbert, M, and Gerdtham, U Socioeconomic Inequalities in Breast Cancer Incidence and Mortality in Europe-A Systematic Review and Meta-Analysis. Eur J Public Health (2016) 26(5):804–13. doi:10.1093/eurpub/ckw070

31. de Boer, MC, Wörner, EA, Verlaan, D, and van Leeuwen, PA. The Mechanisms and Effects of Physical Activity on Breast Cancer. Clin Breast Cancer (2017) 17(4):272–8. doi:10.1016/j.clbc.2017.01.006

32. Caspersen, CJPK, and Christenson, GM. Physical Activity, Exercise, and Physical Fitness: Definitions and Distinctions for Health-Related Research. Public Health Rep (1985) 100(2):126–31.

PubMed Abstract | Google Scholar

33. Warren, JM, Ekelund, U, Besson, H, Mezzani, A, Geladas, N, Vanhees, L, et al. Assessment of Physical Activity – A Review of Methodologies with Reference to Epidemiological Research: A Report of the Exercise Physiology Section of the European Association of Cardiovascular Prevention and Rehabilitation. Eur J Cardiovasc Prev Rehabil (2010) 17(2):127–39. doi:10.1097/HJR.0b013e32832ed875

34. Dorgan, JF, Brown, C, Barrett, M, Splansky, GL, Kreger, BE, D'Agostino, RB, et al. Physical Activity and Risk of Breast Cancer in the Framingham Heart Study. Am J Epidemiol (1994) 139(7):662–9. doi:10.1093/oxfordjournals.aje.a117056

35. Fraser, GE, and Shavlik, D. Risk Factors, Lifetime Risk, and Age at Onset of Breast Cancer. Ann Epidemiol (1997) 7(6):375–82. doi:10.1016/s1047-2797(97)00042-2

36. Cerhan, JR, Chiu, BC, Wallace, RB, Lemke, JH, Lynch, CF, Torner, JC, et al. Physical Activity, Physical Function, and the Risk of Breast Cancer in a Prospective Study Among Elderly Women. J Gerontol Ser A: Biol Sci Med Sci (1998) 53(4):M251–6. doi:10.1093/gerona/53a.4.m251

37. Rockhill, B, Willett, WC, Hunter, DJ, Manson, JE, Hankinson, SE, and Colditz, GA. A Prospective Study of Recreational Physical Activity and Breast Cancer Risk. Arch Intern Med (1999) 159(19):2290–6. doi:10.1001/archinte.159.19.2290

38. Thune, I, Brenn, T, Lund, E, and Gaard, M. Physical Activity and the Risk of Breast Cancer. N Engl J Med (1997) 336(18):1269–75. doi:10.1056/NEJM199705013361801

39. Mittendorf, R. Strenuous Physical Activity in Young Adulthood and Risk of Breast Cancer (United States) (Vol 6, Pg 352, 1996). Cancer Cause Control (1996) 7(2):291.

40. Bernstein, L, Henderson, BE, Hanisch, R, Sullivan-Halley, J, and Ross, RK. Physical Exercise and Reduced Risk of Breast Cancer in Young Women. J Natl Cancer (1994) 86(18):1403–8. doi:10.1093/jnci/86.18.1403

41. Friedenreich, CM, and Rohan, TE. Physical Activity and Risk of Breast Cancer. Eur J Cancer Prev (1995) 4(2):145–51. doi:10.1097/00008469-199504000-00004

42. McTiernan, A, Stanford, JL, Weiss, NS, Daling, JR, and Voigt, LF. Occurrence of Breast Cancer in Relation to Recreational Exercise in Women Age 50-64 Years. Epidemiology (1996) 7(6):598–604. doi:10.1097/00001648-199611000-00006

43. Chen, CL, White, E, Malone, KE, and Daling, JR. Leisure-Time Physical Activity in Relation to Breast Cancer Among Young Women (Washington, United States). Cancer Causes Control (1997) 8(1):77–84. doi:10.1023/a:1018439306604

44. Gammon, MD, John, EM, and Britton, JA. Recreational and Occupational Physical Activities and Risk of Breast Cancer. J Natl Cancer Inst (1998) 90(2):100–17. doi:10.1093/jnci/90.2.100

45. Monninkhof, EM, Elias, SG, Vlems, FA, van der Tweel, I, Schuit, AJ, Voskuil, DW, et al. Physical Activity and Breast Cancer - A Systematic Review. Epidemiology (2007) 18(1):137–57. doi:10.1097/01.ede.0000251167.75581.98

46. Friedenreich, CM. Physical Activity and Breast Cancer: Review of the Epidemiologic Evidence and Biologic Mechanisms. Recent Results Cancer Res (2011) 188:125–39. doi:10.1007/978-3-642-10858-7_11

47. Stalsberg, R, and Pedersen, AV. Are Differences in Physical Activity across Socioeconomic Groups Associated with Choice of Physical Activity Variables to Report? Int J Environ Res Public Health (2018) 15(5):922. doi:10.3390/ijerph15050922

48. Hong, S, Bardwell, WA, Natarajan, L, Flatt, SW, Rock, CL, Newman, VA, et al. Correlates of Physical Activity Level in Breast Cancer Survivors Participating in the Women's Healthy Eating and Living (WHEL) Study. Breast Cancer Res Tr (2007) 101(2):225–32. doi:10.1007/s10549-006-9284-y

49. Bertram, LA, Stefanick, ML, Saquib, N, Natarajan, L, Patterson, RE, Bardwell, W, et al. Physical Activity, Additional Breast Cancer Events, and Mortality Among Early-Stage Breast Cancer Survivors: Findings from the WHEL Study. Cancer Causes Control (2011) 22(3):427–35. doi:10.1007/s10552-010-9714-3

50. Keegan, TH, Shariff-Marco, S, Sangaramoorthy, M, Koo, J, Hertz, A, Schupp, CW, et al. Neighborhood Influences on Recreational Physical Activity and Survival after Breast Cancer. Cancer Causes Control (2014) 25(10):1295–308. doi:10.1007/s10552-014-0431-1

51. Boyle, T, Vallance, JK, Ransom, EK, and Lynch, BM. How Sedentary and Physically Active Are Breast Cancer Survivors, and Which Population Subgroups Have Higher or Lower Levels of These Behaviors? Support Care Cancer (2016) 24(5):2181–90. doi:10.1007/s00520-015-3011-3

52. Campbell, KL, Winters-Stone, KM, Wiskemann, J, May, AM, Schwartz, AL, Courneya, KS, et al. Exercise Guidelines for Cancer Survivors: Consensus Statement from International Multidisciplinary Roundtable. Med Sci Sports Exerc (2019) 51(11):2375–90. doi:10.1249/MSS.0000000000002116

53. Oyekanmi, G, and Paxton, RJ. Barriers to Physical Activity Among African American Breast Cancer Survivors. Psycho-Oncology (2014) 23(11):1314–7. doi:10.1002/pon.3527

54. Owusu, C, Antognoli, E, Nock, N, Hergenroeder, P, Austin, K, Bennet, E, et al. Perspective of Older African-American and Non-Hispanic white Breast Cancer Survivors from Diverse Socioeconomic Backgrounds toward Physical Activity: A Qualitative Study. J Geriatr Oncol (2018) 9(3):235–42. doi:10.1016/j.jgo.2017.12.003

55. Grzywacz, JG, and Marks, NF. Social Inequalities and Exercise during Adulthood: Toward an Ecological Perspective. J Health Soc Behav (2001) 42(June):202–20. doi:10.2307/3090178

56. Lee, RE, and Cubbin, C. Neighborhood Context and Youth Cardiovascular Health Behaviors. Am J Public Health (2002) 92(3):428–36. doi:10.2105/ajph.92.3.428

57. Varo, JJ, Martinez-Gonzalez, MA, de Irala-Estevez, J, Kearney, J, Gibney, M, and Martinez, JA. Distribution and Determinants of Sedentary Lifestyles in the European Union. Int J Epidemiol (2003) 32(1):138–46. doi:10.1093/ije/dyg116

58. Jenum, AK, Lorentzen, CAN, and Ommundsen, Y. Targeting Physical Activity in a Low Socioeconomic Status Population: Observations from the Norwegian “Romsås in Motion” Study. Br J Sport Med (2009) 43(1):64–9. doi:10.1136/bjsm.2008.053637

59. Trost, SG, Owen, N, Bauman, AE, Sallis, JF, and Brown, W. Correlates of Adults’ Participation in Physical Activity: Review and Update. Med Sci Sports Exerc (2002) 34(12):1996–2001. doi:10.1097/00005768-200212000-00020

60. Austin, JL, Smith, JE, Gianini, L, and Campos-Melady, M. Attitudinal Familism Predicts Weight Management Adherence in Mexican–American Women. J Behav Med (2013) 36(3):259–69. doi:10.1007/s10865-012-9420-6

61. Lemstra, M, and Rogers, MR. The Importance of Community Consultation and Social Support in Adhering to an Obesity Reduction Program: Results from the Healthy Weights Initiative. Patient Preference and Adherence (2015) 9:1473–80. doi:10.2147/PPA.S91912

62. Sharrocks, K, Spicer, J, Camidge, DR, and Papa, S. The Impact of Socioeconomic Status on Access to Cancer Clinical Trials. Br J Cancer (2014) 111(9):1684–7. doi:10.1038/bjc.2014.108

63. Langhammer, A, Krokstad, S, Romundstad, P, Heggland, J, and Holmen, J. The HUNT Study: Participation Is Associated with Survival and Depends on Socioeconomic Status, Diseases and Symptoms. BMC Med Res Methodol (2012) 12(1):143. doi:10.1186/1471-2288-12-143

64. Ford, JG, Howerton, MW, Lai, GY, Gary, TL, Bolen, S, Gibbons, MC, et al. Barriers to Recruiting Underrepresented Populations to Cancer Clinical Trials: A Systematic Review. Cancer (2008) 112(2):228–42. doi:10.1002/cncr.23157

65. Stalsberg, R. Social Inequalities in Physical Activity Implications of Research Practices–The Case of Studying Breast Cancer Survivors [Doctoral Thesis] . Trondheim: Norwegian university of science and technology (2021).

66. Akobeng, AK. Assessing the Validity of Clinical Trials. J Pediatr Gastroenterol Nutr (2008) 47(3):277–82. doi:10.1097/MPG.0b013e31816c749f

67. Campbell, DT. Factors Relevant to the Validity of Experiments in Social Settings. Psychol Bull (1957) 54(4):297–312. doi:10.1037/h0040950

68. World Medical Association. World Medical Association Declaration of helsinki: Ethical Principles for Medical Research Involving Human Subjects. JAMA (2013) 310(20):2191–4. doi:10.1001/jama.2013.281053

69. Rennie, DJJ. CONSORT Revised—Improving the Reporting of Randomized Trials. JAMA (2001) 285(15):2006–7.doi:10.1001/jama.285.15.2006

70. Schulz, KF, Altman, DG, and Moher, D, CONSORT Group. CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials. BMC Med (2010) 8:18. doi:10.1186/1741-7015-8-18

71. Speck, RM, Courneya, KS, Masse, LC, Duval, S, and Schmitz, KH. An Update of Controlled Physical Activity Trials in Cancer Survivors: A Systematic Review and Meta-Analysis. J Cancer Survivorship (2010) 4(2):87–100. doi:10.1007/s11764-009-0110-5

72. Furler, J, Magin, P, Pirotta, M, and van Driel, M. Participant Demographics Reported in "Table 1" of Randomised Controlled Trials: A Case of "inverse Evidence. Int J Equity Health (2012) 11:14. doi:10.1186/1475-9276-11-14

73. Stalsberg, R, and Pedersen, AV. Effects of Socioeconomic Status on the Physical Activity in Adolescents: A Systematic Review of the Evidence. Scand J Med Sci Sports (2010) 20(3):368–83. doi:10.1111/j.1600-0838.2009.01047.x

74. Gidlow, C, Johnston, LH, Crone, D, Ellis, N, and James, D. A Systematic Review of the Relationship between Socio-Economic Position and Physical Activity. Health Edu J (2006) 65(4):338–67. doi:10.1177/0017896906069378

75. Evans, JA, and Foster, JG. Metaknowledge. Science (2011) 331(6018):721–5. doi:10.1126/science.1201765

76. Schatz, BR. Information Retrieval in Digital Libraries: Bringing Search to the Net. Science (1997) 275(5298):327–34. doi:10.1126/science.275.5298.327

77. Bramer, WM, De Jonge, GB, Rethlefsen, ML, Mast, F, and Kleijnen, J. A Systematic Approach to Searching: An Efficient and Complete Method to Develop Literature Searches. J Med Libr Assoc JMLA. (2018) 106(4):531–41. doi:10.5195/jmla.2018.283

78. Horsley, T, Dingwall, O, and Sampson, M. Checking Reference Lists to Find Additional Studies for Systematic Reviews. Cochrane Db Syst Rev (2011) 2011(8):MR000026. doi:10.1002/14651858.MR000026.pub2

79. Hu, X, Rousseau, R, and Chen, J. On the Definition of Forward and Backward Citation Generations. J Informetrics (2011) 5(1):27–36. doi:10.1016/j.joi.2010.07.004

80. Rysstad, AL, and Pedersen, AV. Brief Report: Non-right-Handedness within the Autism Spectrum Disorder. J autism Dev Disord (2016) 46(3):1110–7. doi:10.1007/s10803-015-2631-2

81. Darvik, M, Lorås, HW, and Pedersen, AV. The Prevalence of Left-Handedness Is Higher Among Individuals with Developmental Coordination Disorder Than in the General Population. Front Psychol (2018) 9:1948–11. doi:10.3389/fpsyg.2018.01948

82. The World Bank. Higher Education. THE WORLD BANK (2021). Available from: https://www.worldbank.org/en/topic/tertiaryeducation (Accessed December 1, 2023).

83. World Bank – processed by Our World in Data. Average Years of Schooling for Women [dataset]. World Bank, World Bank Education Statistics (EdStats) (2023). Available from: https://ourworldindata.org/grapher/mean-years-of-schooling-female?tab=table&time=2004.2005 (Accessed November 7, 2023).

84. World Bank – processed by Our World in Data. Average Years of Schooling for Women [dataset]. World Bank, World Bank Education Statistics (EdStats) (2023). Available from: https://ourworldindata.org/grapher/mean-years-of-schooling-female?tab=table&time=2006.2009 (Accessed November 7, 2023).

85. OECD. Educational Attainment and Labour-Force Status: Trends in Educational Attainment, by Educational Attainment and Age Group: OECD.Stat (2023). Available from: https://stats.oecd.org/Index.aspx?QueryId=93191 (Accessed November 7, 2023).

86. Ministry of Education, Republic of China (Taiwan). Enrollment Rates of School - Total Net Enrollment Rates: Ministry of Education (2022). Available from: https://stats.moe.gov.tw/files/ebook/indicators/13.pdf (Accessed May 04, 2022).

87. United Nations. Department of Economic and Social Affairs Population Division: Estimates and Projections of Women of Reproductive Age Who Are Married or in a Union (2022). Available from: https://population.un.org/dataportal/data/indicators/44/locations/840,124,156,158,250,528,826,724,208,752,76/start/2000/end/2020/table/pivotbyvariant (Accessed November 7, 2023).

88. Statistics Canada. Canada (Code 01) (Table). National Household Survey (NHS) Profile. 2011 National Household Survey. Statistics Canada Catalogue No. 99-004-XWE (2013). Available from: https://www12.statcan.gc.ca/nhs-enm/2011/dp-pd/prof/details/page.cfm?Lang=E&Geo1=PR&Code1=01&Data=Count&SearchText=Canada&SearchType=Begins&SearchPR=01&A1=All&B1=All&Custom=&TABID=1 (Accessed September 11, 2013).

89. Statistics Canada. Canada [Country] and Canada [Country] (Table). Census Profile. 2016 Census. Statistics Canada Catalogue No. 98-316-X2016001 (2017). Available from: https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?Lang=E&Geo1=PR&Code1=01&Geo2=PR&Code2=01&Data=Count&SearchText=canada&SearchType=Begins&SearchPR=01&B1=All&TABID=1 (Accessed November 29, 2017).

90. Statistics Canada. Visible Minority and Population Group by Generation Status: Canada, Provinces and Territories, Census Metropolitan Areas and Census Agglomerations with Parts. Table: 98-10-0324-01 (2023). Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810032401 (Accessed October 26, 2023).

91. United States Census Bureau. Race and Ethnicity in the United States: 2010 Census and 2020 Census 2021 (2021). Available from: https://www.census.gov/library/visualizations/interactive/race-and-ethnicity-in-the-united-state-2010-and-2020-census.html (Accessed November 29, 2023).

92. United States Census Bureau. "Profile of General Demographic Characteristics: 2000." Decennial Census, DEC State Legislative District Demographic Profile (100-Percent) (2000). Available from: https://data.census.gov/table?y=2000 (Accessed November 29, 2023).

93. Pickett, M, Mock, V, Ropka, ME, Cameron, L, Coleman, M, and Podewils, L. Adherence to Moderate-Intensity Exercise during Breast Cancer Therapy. Cancer Pract (2002) 10(6):284–92. doi:10.1046/j.1523-5394.2002.106006.x

94. Al-Majid, S, Wilson, LD, Rakovski, C, and Coburn, JW. Effects of Exercise on Biobehavioral Outcomes of Fatigue during Cancer Treatment: Results of a Feasibility Study. Biol Res Nurs (2015) 17(1):40–8. doi:10.1177/1099800414523489

95. Courneya, KS, McKenzie, DC, Mackey, JR, Gelmon, K, Friedenreich, CM, Yasui, Y, et al. Effects of Exercise Dose and Type during Breast Cancer Chemotherapy: Multicenter Randomized Trial. J Natl Cancer (2013) 105(23):1821–32. doi:10.1093/jnci/djt297

96. Anderson, RT, Kimmick, GG, McCoy, TP, Hopkins, J, Levine, E, Miller, G, et al. A Randomized Trial of Exercise on Well-Being and Function Following Breast Cancer Surgery: The RESTORE Trial. J Cancer Survivorship-Research Pract (2012) 6(2):172–81. doi:10.1007/s11764-011-0208-4

97. Arem, H, Sorkin, M, Cartmel, B, Fiellin, M, Capozza, S, Harrigan, M, et al. Exercise Adherence in a Randomized Trial of Exercise on Aromatase Inhibitor Arthralgias in Breast Cancer Survivors: The Hormones and Physical Exercise (HOPE) Study. J Cancer Survivorship (2016) 10(4):654–62. doi:10.1007/s11764-015-0511-6

98. Ariza-Garcia, A, Lozano-Lozano, M, Galiano-Castillo, N, Postigo-Martin, P, Arroyo-Morales, M, and Cantarero-Villanueva, I. A Web-Based Exercise System (E-CuidateChemo) to Counter the Side Effects of Chemotherapy in Patients with Breast Cancer: Randomized Controlled Trial. J Med Internet Res (2019) 21(7):e14418. doi:10.2196/14418

99. Bland, KA, Kirkham, AA, Bovard, J, Shenkier, T, Zucker, D, McKenzie, DC, et al. Effect of Exercise on Taxane Chemotherapy–Induced Peripheral Neuropathy in Women with Breast Cancer: A Randomized Controlled Trial. Clin Breast Cancer (2019) 19(6):411–22. doi:10.1016/j.clbc.2019.05.013

100. Cadmus, LA, Salovey, P, Yu, H, Chung, G, Kasl, S, and Irwin, ML. Exercise and Quality of Life during and after Treatment for Breast Cancer: Results of Two Randomized Controlled Trials. Psycho-Oncology (2009) 18(4):343–52. doi:10.1002/pon.1525

101. Carayol, M, Ninot, G, Senesse, P, Bleuse, J-P, Gourgou, S, Sancho-Garnier, H, et al. Short and Long-Term Impact of Adapted Physical Activity and Diet Counseling during Adjuvant Breast Cancer Therapy: The “APAD1” Randomized Controlled Trial. BMC cancer (2019) 19(1):737–20. doi:10.1186/s12885-019-5896-6

102. Chandwani, KD, Thornton, B, Perkins, GH, Arun, B, Raghuram, N, Nagendra, H, et al. Yoga Improves Quality of Life and Benefit Finding in Women Undergoing Radiotherapy for Breast Cancer. J Soc Integr Oncol (2010) 8(2):43–55.

103. Chen, Z, Meng, Z, Milbury, K, Bei, W, Zhang, Y, Thornton, B, et al. Qigong Improves Quality of Life in Women Undergoing Radiotherapy for Breast Cancer: Results of a Randomized Controlled Trial. Cancer (2013) 119(9):1690–8. doi:10.1002/cncr.27904

104. Courneya, KS, Segal, RJ, Gelmon, K, Reid, RD, Mackey, JR, Friedenreich, CM, et al. Predictors of Supervised Exercise Adherence during Breast Cancer Chemotherapy. Med Sci Sports Exerc (2008) 40:1180–7. doi:10.1249/MSS.0b013e318168da45

105. Demark-Wahnefried, W, Case, LD, Blackwell, K, Marcom, PK, Kraus, W, Aziz, N, et al. Results of a Diet/exercise Feasibility Trial to Prevent Adverse Body Composition Change in Breast Cancer Patients on Adjuvant Chemotherapy. Clin Breast Cancer (2008) 8(1):70–9. doi:10.3816/CBC.2008.n.005

106. Gokal, K, Wallis, D, Ahmed, S, Boiangiu, I, Kancherla, K, and Munir, F. Effects of a Self-Managed home-Based Walking Intervention on Psychosocial Health Outcomes for Breast Cancer Patients Receiving Chemotherapy: A Randomised Controlled Trial. Support Care Cancer (2016) 24(3):1139–66. doi:10.1007/s00520-015-2884-5

107. Huang, HP, Wen, FH, Tsai, JC, Lin, YC, Shun, SC, Chang, HK, et al. Adherence to Prescribed Exercise Time and Intensity Declines as the Exercise Program Proceeds: Findings from Women under Treatment for Breast Cancer. Support Care Cancer (2015) 23(7):2061–71. doi:10.1007/s00520-014-2567-7

108. Kirkham, AA, Van Patten, CL, Gelmon, KA, McKenzie, DC, Bonsignore, A, Bland, KA, et al. Effectiveness of Oncologist-Referred Exercise and Healthy Eating Programming as a Part of Supportive Adjuvant Care for Early Breast Cancer. Oncologist (2018) 23(1):105–15. doi:10.1634/theoncologist.2017-0141

109. Lund, LW, Ammitzbøll, G, Hansen, DG, Andersen, EAW, and Dalton, SO. Adherence to a Long-Term Progressive Resistance Training Program, Combining Supervised and home-Based Exercise for Breast Cancer Patients during Adjuvant Treatment. Acta Oncologica (2019) 58(5):650–7. doi:10.1080/0284186X.2018.1560497

110. Matthews, CE, Wilcox, S, Hanby, CL, Ananian, CD, Heiney, SP, Gebretsadik, T, et al. Evaluation of a 12-Week home-Based Walking Intervention for Breast Cancer Survivors. Support Care Cancer (2007) 15(2):203–11. doi:10.1007/s00520-006-0122-x

111. Mijwel, S, Backman, M, Bolam, KA, Jervaeus, A, Sundberg, CJ, Margolin, S, et al. Adding High-Intensity Interval Training to Conventional Training Modalities: Optimizing Health-Related Outcomes during Chemotherapy for Breast Cancer: The OptiTrain Randomized Controlled Trial. Breast Cancer Res Tr (2018) 168(1):79–93. doi:10.1007/s10549-017-4571-3

112. Mock, V, Frangakis, C, Davidson, NE, Ropka, ME, Pickett, M, Poniatowski, B, et al. Exercise Manages Fatigue during Breast Cancer Treatment: A Randomized Controlled Trial. Psycho-Oncology (2005) 14(6):464–77. doi:10.1002/pon.863

113. Paulo, TR, Rossi, FE, Viezel, J, Tosello, GT, Seidinger, SC, Simões, RR, et al. The Impact of an Exercise Program on Quality of Life in Older Breast Cancer Survivors Undergoing Aromatase Inhibitor Therapy: A Randomized Controlled Trial. Health Qual Life Out (2019) 17(1):17–2. doi:10.1186/s12955-019-1090-4

114. Ratcliff, CG, Milbury, K, Chandwani, KD, Chaoul, A, Perkins, G, Nagarathna, R, et al. Examining Mediators and Moderators of Yoga for Women with Breast Cancer Undergoing Radiotherapy. Integr Cancer Therapies (2016) 15(3):250–62. doi:10.1177/1534735415624141

115. D Reis, ME Walsh, S Young-McCaughan, and T Jones, Effects of Nia Exercise in Women Receiving Radiation Therapy for Breast Cancer . Oncol Nurs Forum (2013).

116. Rogers, LQ, Hopkins-Price, P, Vicari, S, Pamenter, R, Courneya, KS, Markwell, S, et al. A Randomized Trial to Increase Physical Activity in Breast Cancer Survivors. Med Sci Sports Exerc (2009) 41(4):935–46. doi:10.1249/MSS.0b013e31818e0e1b

117. Travier, N, Velthuis, MJ, Bisschop, CNS, van den Buijs, B, Monninkhof, EM, Backx, F, et al. Effects of an 18-Week Exercise Programme Started Early during Breast Cancer Treatment: A Randomised Controlled Trial. Bmc Med (2015) 13:121. doi:10.1186/s12916-015-0362-z

118. Vadiraja, SH, Rao, MR, Nagendra, RH, Nagarathna, R, Rekha, M, Vanitha, N, et al. Effects of Yoga on Symptom Management in Breast Cancer Patients: A Randomized Controlled Trial. Int J Yoga (2009) 2(2):73–9. doi:10.4103/0973-6131.60048

119. Vallance, JK, Friedenreich, CM, Lavallee, CM, Culos-Reed, N, Mackey, JR, Walley, B, et al. Exploring the Feasibility of a Broad-Reach Physical Activity Behavior Change Intervention for Women Receiving Chemotherapy for Breast Cancer: A Randomized Trial. Cancer Epidemiol Biomarkers Prev (2016) 25(2):391–8. doi:10.1158/1055-9965.EPI-15-0812

120. van Waart, H, Stuiver, MM, van Harten, WH, Geleijn, E, Kieffer, JM, Buffart, LM, et al. Effect of Low-Intensity Physical Activity and Moderate to High-Intensity Physical Exercise during Adjuvant Chemotherapy on Physical Fitness, Fatigue, and Chemotherapy Completion Rates: Results of the PACES Randomized Clinical Trial. J Clin Oncol (2015) 33(17):1918–27. doi:10.1200/JCO.2014.59.1081

121. Wang, YJ, Boehmke, M, Wu, YW, Dickerson, SS, and Fisher, N. Effects of a 6-Week Walking Program on Taiwanese Women Newly Diagnosed with Early-Stage Breast Cancer. Cancer Nurs (2011) 34(2):E1–13. doi:10.1097/NCC.0b013e3181e4588d

122. Zhou, K, Wang, W, An, J, Li, M, Li, J, and Li, X. Effects of Progressive Upper Limb Exercises and Muscle Relaxation Training on Upper Limb Function and Health-Related Quality of Life Following Surgery in Women with Breast Cancer: A Clinical Randomized Controlled Trial. Ann Surg Oncol (2019) 26(7):2156–65. doi:10.1245/s10434-019-07305-y

123. DeNysschen, CA, Brown, JK, Cho, MH, and Dodd, MJ. Nutritional Symptom and Body Composition Outcomes of Aerobic Exercise in Women with Breast Cancer. Clin Nurs Res (2011) 20(1):29–46. doi:10.1177/1054773810379402

124. Dieli-Conwright, CM, Parmentier, J-H, Sami, N, Lee, K, Spicer, D, Mack, WJ, et al. Adipose Tissue Inflammation in Breast Cancer Survivors: Effects of a 16-Week Combined Aerobic and Resistance Exercise Training Intervention. Breast Cancer Res Tr (2018) 168(1):147–57. doi:10.1007/s10549-017-4576-y

125. Kim, C-J, Kang, D-H, Smith, BA, and Landers, KA. Cardiopulmonary Responses and Adherence to Exercise in Women Newly Diagnosed with Breast Cancer Undergoing Adjuvant Therapy. Cancer Nurs (2006) 29(2):156–65. doi:10.1097/00002820-200603000-00013

126. Lee, K, Kang, I, Mack, WJ, Mortimer, J, Sattler, F, Salem, G, et al. Feasibility of High Intensity Interval Training in Patients with Breast Cancer Undergoing Anthracycline Chemotherapy: A Randomized Pilot Trial. BMC cancer (2019) 19(1):653–9. doi:10.1186/s12885-019-5887-7

127. Stan, DL, Kathleen Sundt, R, Cheville, AL, Youdas, JW, Krause, DA, Boughey, JC, et al. Pilates for Breast Cancer Survivors: Impact on Physical Parameters and Quality of Life after Mastectomy. Clin J Oncol Nurs (2012) 16(2):131–41. doi:10.1188/12.CJON.131-141

128. Swenson, KK, Nissen, MJ, Anderson, E, Shapiro, A, Schousboe, J, and Leach, J. Effects of Exercise vs Bisphosphonates on Bone mineral Density in Breast Cancer Patients Receiving Chemotherapy. J Support Oncol (2009) 7(3):101–7.

129. Yang, CY, Tsai, JC, Huang, YC, and Lin, CC. Effects of a home-Based Walking Program on Perceived Symptom and Mood Status in Postoperative Breast Cancer Women Receiving Adjuvant Chemotherapy. J Adv Nurs (2011) 67(1):158–68. doi:10.1111/j.1365-2648.2010.05492.x

130. Ainsworth, BE, Haskell, WL, Herrmann, SD, Meckes, N, Bassett, DR, Tudor-Locke, C, et al. 2011 Compendium of Physical Activities: A Second Update of Codes and MET Values. Med Sci Sports Exerc (2011) 43(8):1575–81. doi:10.1249/MSS.0b013e31821ece12

131. van Waart, H, van Harten, WH, Buffart, LM, Sonke, GS, Stuiver, MM, and Aaronson, NK. Why Do Patients Choose (Not) to Participate in an Exercise Trial during Adjuvant Chemotherapy for Breast Cancer? Psycho-Oncology (2016) 25(8):964–70. doi:10.1002/pon.3936

132. Susukida, R, Crum, RM, Stuart, EA, Ebnesajjad, C, and Mojtabai, R. Assessing Sample Representativeness in Randomized Controlled Trials: Application to the National Institute of Drug Abuse Clinical Trials Network. Addiction (2016) 111(7):1226–34. doi:10.1111/add.13327

133. Galobardes, B, Shaw, M, Lawlor, DA, Lynch, JW, and Smith, GD. Indicators of Socioeconomic Position (Part 1). J Epidemiol Community Health (2006) 60(1):7–12. doi:10.1136/jech.2004.023531

134. Murthy, VH, Krumholz, HM, and Gross, CP. Participation in Cancer Clinical Trials: Race, Sex, and Age-Based Disparities. Jama (2004) 291(22):2720–6. doi:10.1001/jama.291.22.2720

135. Semega, J, and Kollar, M. Income in the United States: 2021 . United States: U.S. CENSUS BUREAU (2022).

136. Sims, JM. A Brief Review of the Belmont Report. Dimensions Crit Care Nurs (2010) 29(4):173–4. doi:10.1097/DCC.0b013e3181de9ec5

137. Coughlin, SS. Social Determinants of Breast Cancer Risk, Stage, and Survival. Breast Cancer Res Treat (2019) 177:537–48. doi:10.1007/s10549-019-05340-7

138. Stalsberg, R, Bertheussen, GF, Børset, H, Thomsen, SN, Husøy, A, Flote, VG, et al. Do Breast Cancer Patients Manage to Participate in an Outdoor, Tailored, Physical Activity Program during Adjuvant Breast Cancer Treatment, Independent of Health and Socio-Demographic Characteristics? J Clin Med (2022) 11(3):843. doi:10.3390/jcm11030843

139. Daley, AJ, Crank, H, Mutrie, N, Saxton, JM, and Coleman, R. Determinants of Adherence to Exercise in Women Treated for Breast Cancer. Eur J Oncol Nurs (2007) 11(5):392–9. doi:10.1016/j.ejon.2007.03.001

140. Brunet, J, Taran, S, Burke, S, and Sabiston, CM. A Qualitative Exploration of Barriers and Motivators to Physical Activity Participation in Women Treated for Breast Cancer. Disabil Rehabil (2013) 35(24):2038–45. doi:10.3109/09638288.2013.802378

141. Strazdins, L, Griffin, AL, Broom, DH, Banwell, C, Korda, R, Dixon, J, et al. Time Scarcity: Another Health Inequality? Environ Plann A: Economy Space (2011) 43(3):545–59. doi:10.1068/a4360

142. Strazdins, L, Welsh, J, Korda, R, Broom, D, and Paolucci, F. Not All Hours Are Equal: Could Time Be a Social Determinant of Health? Sociol Health Ill (2016) 38(1):21–42. doi:10.1111/1467-9566.12300

143. Phelan, JC, Link, BG, Diez-Roux, A, Kawachi, I, and Levin, B. “Fundamental Causes” of Social Inequalities in Mortality: A Test of the Theory. J Health Soc Behav (2004) 45(3):265–85. doi:10.1177/002214650404500303

144. Hart, T. The Inverse Care Law. The Lancet (1971) 297(7696):405–12.

Keywords: sampling bias, social validity, research equity, exercise trials, breast cancer, mammae cancer, meta-analysis

Citation: Stalsberg R and Darvik MD (2024) Social Representativeness and Intervention Adherence—A Systematic Review of Clinical Physical Activity Trials in Breast Cancer Patients. Int J Public Health 69:1607002. doi: 10.3389/ijph.2024.1607002

Received: 21 December 2023; Accepted: 10 April 2024; Published: 09 May 2024.

Reviewed by:

Copyright © 2024 Stalsberg and Darvik. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Ragna Stalsberg, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.