meta analysis quantitative research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 08 March 2018

Meta-analysis and the science of research synthesis

• Jessica Gurevitch 1 ,
• Julia Koricheva 2 ,
• Shinichi Nakagawa 3 , 4 &
• Gavin Stewart 5  

Nature volume  555 ,  pages 175–182 ( 2018 ) Cite this article

55k Accesses

883 Citations

738 Altmetric

Metrics details

• Biodiversity
• Outcomes research

Meta-analysis is the quantitative, scientific synthesis of research results. Since the term and modern approaches to research synthesis were first introduced in the 1970s, meta-analysis has had a revolutionary effect in many scientific fields, helping to establish evidence-based practice and to resolve seemingly contradictory research outcomes. At the same time, its implementation has engendered criticism and controversy, in some cases general and others specific to particular disciplines. Here we take the opportunity provided by the recent fortieth anniversary of meta-analysis to reflect on the accomplishments, limitations, recent advances and directions for future developments in the field of research synthesis.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Eight problems with literature reviews and how to fix them

The past, present and future of Registered Reports

Raiders of the lost HARK: a reproducible inference framework for big data science

Jennions, M. D ., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 23 , 364–380 (Princeton Univ. Press, 2013)

Article   Google Scholar  

Roberts, P. D ., Stewart, G. B. & Pullin, A. S. Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132 , 409–423 (2006)

Bastian, H ., Glasziou, P . & Chalmers, I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7 , e1000326 (2010)

Article   PubMed   PubMed Central   Google Scholar  

Borman, G. D. & Grigg, J. A. in The Handbook of Research Synthesis and Meta-analysis 2nd edn (eds Cooper, H. M . et al.) 497–519 (Russell Sage Foundation, 2009)

Ioannidis, J. P. A. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94 , 485–514 (2016)

Koricheva, J . & Gurevitch, J. Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102 , 828–844 (2014)

Littell, J. H . & Shlonsky, A. Making sense of meta-analysis: a critique of “effectiveness of long-term psychodynamic psychotherapy”. Clin. Soc. Work J. 39 , 340–346 (2011)

Morrissey, M. B. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J. Evol. Biol. 29 , 1882–1904 (2016)

Article   CAS   PubMed   Google Scholar  

Whittaker, R. J. Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 91 , 2522–2533 (2010)

Article   PubMed   Google Scholar  

Begley, C. G . & Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature 483 , 531–533 (2012); clarification 485 , 41 (2012)

Article   CAS   ADS   PubMed   Google Scholar  

Hillebrand, H . & Cardinale, B. J. A critique for meta-analyses and the productivity-diversity relationship. Ecology 91 , 2545–2549 (2010)

Moher, D . et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6 , e1000097 (2009). This paper provides a consensus regarding the reporting requirements for medical meta-analysis and has been highly influential in ensuring good reporting practice and standardizing language in evidence-based medicine, with further guidance for protocols, individual patient data meta-analyses and animal studies.

Moher, D . et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4 , 1 (2015)

Nakagawa, S . & Santos, E. S. A. Methodological issues and advances in biological meta-analysis. Evol. Ecol. 26 , 1253–1274 (2012)

Nakagawa, S ., Noble, D. W. A ., Senior, A. M. & Lagisz, M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15 , 18 (2017)

Hedges, L. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 1985)

Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36 , 1–48 (2010)

Anzures-Cabrera, J . & Higgins, J. P. T. Graphical displays for meta-analysis: an overview with suggestions for practice. Res. Synth. Methods 1 , 66–80 (2010)

Egger, M ., Davey Smith, G ., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315 , 629–634 (1997)

Article   CAS   Google Scholar  

Duval, S . & Tweedie, R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56 , 455–463 (2000)

Article   CAS   MATH   PubMed   Google Scholar  

Leimu, R . & Koricheva, J. Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc. R. Soc. Lond. B 271 , 1961–1966 (2004)

Higgins, J. P. T . & Green, S. (eds) Cochrane Handbook for Systematic Reviews of Interventions : Version 5.1.0 (Wiley, 2011). This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.

Lau, J ., Rothstein, H. R . & Stewart, G. B. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 25 , 407–419 (Princeton Univ. Press, 2013)

Lortie, C. J ., Stewart, G ., Rothstein, H. & Lau, J. How to critically read ecological meta-analyses. Res. Synth. Methods 6 , 124–133 (2015)

Murad, M. H . & Montori, V. M. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. J. Am. Med. Assoc. 309 , 2217–2218 (2013)

Rasmussen, S. A ., Chu, S. Y ., Kim, S. Y ., Schmid, C. H . & Lau, J. Maternal obesity and risk of neural tube defects: a meta-analysis. Am. J. Obstet. Gynecol. 198 , 611–619 (2008)

Littell, J. H ., Campbell, M ., Green, S . & Toews, B. Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10–17. Cochrane Database Syst. Rev. https://doi.org/10.1002/14651858.CD004797.pub4 (2005)

Schmidt, F. L. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. Am. Psychol. 47 , 1173–1181 (1992)

Button, K. S . et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 365–376 (2013); erratum 14 , 451 (2013)

Parker, T. H . et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31 , 711–719 (2016)

Stewart, G. Meta-analysis in applied ecology. Biol. Lett. 6 , 78–81 (2010)

Sutherland, W. J ., Pullin, A. S ., Dolman, P. M . & Knight, T. M. The need for evidence-based conservation. Trends Ecol. Evol. 19 , 305–308 (2004)

Lowry, E . et al. Biological invasions: a field synopsis, systematic review, and database of the literature. Ecol. Evol. 3 , 182–196 (2013)

Article   PubMed Central   Google Scholar  

Parmesan, C . & Yohe, G. A globally coherent fingerprint of climate change impacts across natural systems. Nature 421 , 37–42 (2003)

Jennions, M. D ., Lortie, C. J . & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 24 , 381–403 (Princeton Univ. Press, 2013)

Balvanera, P . et al. Quantifying the evidence for biodiversity effects on ecosystem functioning and services. Ecol. Lett. 9 , 1146–1156 (2006)

Cardinale, B. J . et al. Effects of biodiversity on the functioning of trophic groups and ecosystems. Nature 443 , 989–992 (2006)

Rey Benayas, J. M ., Newton, A. C ., Diaz, A. & Bullock, J. M. Enhancement of biodiversity and ecosystem services by ecological restoration: a meta-analysis. Science 325 , 1121–1124 (2009)

Article   ADS   PubMed   CAS   Google Scholar  

Leimu, R ., Mutikainen, P. I. A ., Koricheva, J. & Fischer, M. How general are positive relationships between plant population size, fitness and genetic variation? J. Ecol. 94 , 942–952 (2006)

Hillebrand, H. On the generality of the latitudinal diversity gradient. Am. Nat. 163 , 192–211 (2004)

Gurevitch, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 19 , 313–320 (Princeton Univ. Press, 2013)

Rustad, L . et al. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia 126 , 543–562 (2001)

Adams, D. C. Phylogenetic meta-analysis. Evolution 62 , 567–572 (2008)

Hadfield, J. D . & Nakagawa, S. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J. Evol. Biol. 23 , 494–508 (2010)

Lajeunesse, M. J. Meta-analysis and the comparative phylogenetic method. Am. Nat. 174 , 369–381 (2009)

Rosenberg, M. S ., Adams, D. C . & Gurevitch, J. MetaWin: Statistical Software for Meta-Analysis with Resampling Tests Version 1 (Sinauer Associates, 1997)

Wallace, B. C . et al. OpenMEE: intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods Ecol. Evol. 8 , 941–947 (2016)

Gurevitch, J ., Morrison, J. A . & Hedges, L. V. The interaction between competition and predation: a meta-analysis of field experiments. Am. Nat. 155 , 435–453 (2000)

Adams, D. C ., Gurevitch, J . & Rosenberg, M. S. Resampling tests for meta-analysis of ecological data. Ecology 78 , 1277–1283 (1997)

Gurevitch, J . & Hedges, L. V. Statistical issues in ecological meta-analyses. Ecology 80 , 1142–1149 (1999)

Schmid, C. H . & Mengersen, K. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 11 , 145–173 (Princeton Univ. Press, 2013)

Eysenck, H. J. Exercise in mega-silliness. Am. Psychol. 33 , 517 (1978)

Simberloff, D. Rejoinder to: Don’t calculate effect sizes; study ecological effects. Ecol. Lett. 9 , 921–922 (2006)

Cadotte, M. W ., Mehrkens, L. R . & Menge, D. N. L. Gauging the impact of meta-analysis on ecology. Evol. Ecol. 26 , 1153–1167 (2012)

Koricheva, J ., Jennions, M. D. & Lau, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 15 , 237–254 (Princeton Univ. Press, 2013)

Lau, J ., Ioannidis, J. P. A ., Terrin, N ., Schmid, C. H . & Olkin, I. The case of the misleading funnel plot. Br. Med. J. 333 , 597–600 (2006)

Vetter, D ., Rucker, G. & Storch, I. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere 4 , 1–24 (2013)

Mengersen, K ., Jennions, M. D. & Schmid, C. H. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 16 , 255–283 (Princeton Univ. Press, 2013)

Patsopoulos, N. A ., Analatos, A. A. & Ioannidis, J. P. A. Relative citation impact of various study designs in the health sciences. J. Am. Med. Assoc. 293 , 2362–2366 (2005)

Kueffer, C . et al. Fame, glory and neglect in meta-analyses. Trends Ecol. Evol. 26 , 493–494 (2011)

Cohnstaedt, L. W. & Poland, J. Review Articles: The black-market of scientific currency. Ann. Entomol. Soc. Am. 110 , 90 (2017)

Longo, D. L. & Drazen, J. M. Data sharing. N. Engl. J. Med. 374 , 276–277 (2016)

Gauch, H. G. Scientific Method in Practice (Cambridge Univ. Press, 2003)

Science Staff. Dealing with data: introduction. Challenges and opportunities. Science 331 , 692–693 (2011)

Nosek, B. A . et al. Promoting an open research culture. Science 348 , 1422–1425 (2015)

Article   CAS   ADS   PubMed   PubMed Central   Google Scholar  

Stewart, L. A . et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. J. Am. Med. Assoc. 313 , 1657–1665 (2015)

Saldanha, I. J . et al. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Syst. Rev. 5 , 196 (2016)

Tipton, E. & Pustejovsky, J. E. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J. Educ. Behav. Stat. 40 , 604–634 (2015)

Mengersen, K ., MacNeil, M. A . & Caley, M. J. The potential for meta-analysis to support decision analysis in ecology. Res. Synth. Methods 6 , 111–121 (2015)

Ashby, D. Bayesian statistics in medicine: a 25 year review. Stat. Med. 25 , 3589–3631 (2006)

Article   MathSciNet   PubMed   Google Scholar  

Senior, A. M . et al. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology 97 , 3293–3299 (2016)

McAuley, L ., Pham, B ., Tugwell, P . & Moher, D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356 , 1228–1231 (2000)

Koricheva, J ., Gurevitch, J . & Mengersen, K. (eds) The Handbook of Meta-Analysis in Ecology and Evolution (Princeton Univ. Press, 2013) This book provides the first comprehensive guide to undertaking meta-analyses in ecology and evolution and is also relevant to other fields where heterogeneity is expected, incorporating explicit consideration of the different approaches used in different domains.

Lumley, T. Network meta-analysis for indirect treatment comparisons. Stat. Med. 21 , 2313–2324 (2002)

Zarin, W . et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 15 , 3 (2017)

Elliott, J. H . et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 11 , e1001603 (2014)

Vandvik, P. O ., Brignardello-Petersen, R . & Guyatt, G. H. Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? BMC Med. 14 , 59 (2016)

Jarvinen, A. A meta-analytic study of the effects of female age on laying date and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula hypoleuca . Ibis 133 , 62–67 (1991)

Arnqvist, G. & Wooster, D. Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol. Evol. 10 , 236–240 (1995)

Hedges, L. V ., Gurevitch, J . & Curtis, P. S. The meta-analysis of response ratios in experimental ecology. Ecology 80 , 1150–1156 (1999)

Gurevitch, J ., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res 32 , 199–247 (2001)

Lajeunesse, M. J. phyloMeta: a program for phylogenetic comparative analyses with meta-analysis. Bioinformatics 27 , 2603–2604 (2011)

CAS   PubMed   Google Scholar  

Pearson, K. Report on certain enteric fever inoculation statistics. Br. Med. J. 2 , 1243–1246 (1904)

Fisher, R. A. Statistical Methods for Research Workers (Oliver and Boyd, 1925)

Yates, F. & Cochran, W. G. The analysis of groups of experiments. J. Agric. Sci. 28 , 556–580 (1938)

Cochran, W. G. The combination of estimates from different experiments. Biometrics 10 , 101–129 (1954)

Smith, M. L . & Glass, G. V. Meta-analysis of psychotherapy outcome studies. Am. Psychol. 32 , 752–760 (1977)

Glass, G. V. Meta-analysis at middle age: a personal history. Res. Synth. Methods 6 , 221–231 (2015)

Cooper, H. M ., Hedges, L. V . & Valentine, J. C. (eds) The Handbook of Research Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009). This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields.

Rosenthal, R. Meta-analytic Procedures for Social Research (Sage, 1991)

Hunter, J. E ., Schmidt, F. L. & Jackson, G. B. Meta-analysis: Cumulating Research Findings Across Studies (Sage, 1982)

Gurevitch, J ., Morrow, L. L ., Wallace, A . & Walsh, J. S. A meta-analysis of competition in field experiments. Am. Nat. 140 , 539–572 (1992). This influential early ecological meta-analysis reports multiple experimental outcomes on a longstanding and controversial topic that introduced a wide range of ecologists to research synthesis methods.

O’Rourke, K. An historical perspective on meta-analysis: dealing quantitatively with varying study results. J. R. Soc. Med. 100 , 579–582 (2007)

Shadish, W. R . & Lecy, J. D. The meta-analytic big bang. Res. Synth. Methods 6 , 246–264 (2015)

Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5 , 3–8 (1976)

DerSimonian, R . & Laird, N. Meta-analysis in clinical trials. Control. Clin. Trials 7 , 177–188 (1986)

Lipsey, M. W . & Wilson, D. B. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. Am. Psychol. 48 , 1181–1209 (1993)

Chalmers, I. & Altman, D. G. Systematic Reviews (BMJ Publishing Group, 1995)

Moher, D . et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 354 , 1896–1900 (1999)

Higgins, J. P. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21 , 1539–1558 (2002)

Download references

Acknowledgements

We dedicate this Review to the memory of Ingram Olkin and William Shadish, founding members of the Society for Research Synthesis Methodology who made tremendous contributions to the development of meta-analysis and research synthesis and to the supervision of generations of students. We thank L. Lagisz for help in preparing the figures. We are grateful to the Center for Open Science and the Laura and John Arnold Foundation for hosting and funding a workshop, which was the origination of this article. S.N. is supported by Australian Research Council Future Fellowship (FT130100268). J.G. acknowledges funding from the US National Science Foundation (ABI 1262402).

Author information

Authors and affiliations.

Department of Ecology and Evolution, Stony Brook University, Stony Brook, 11794-5245, New York, USA

Jessica Gurevitch

School of Biological Sciences, Royal Holloway University of London, Egham, TW20 0EX, Surrey, UK

Julia Koricheva

Evolution and Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, 2052, New South Wales, Australia

Shinichi Nakagawa

Diabetes and Metabolism Division, Garvan Institute of Medical Research, 384 Victoria Street, Darlinghurst, Sydney, 2010, New South Wales, Australia

School of Natural and Environmental Sciences, Newcastle University, Newcastle upon Tyne, NE1 7RU, UK

Gavin Stewart

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed equally in designing the study and writing the manuscript, and so are listed alphabetically.

Corresponding authors

Correspondence to Jessica Gurevitch , Julia Koricheva , Shinichi Nakagawa or Gavin Stewart .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Additional information

Reviewer Information Nature thanks D. Altman, M. Lajeunesse, D. Moher and G. Romero for their contribution to the peer review of this work.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

PowerPoint slides

Powerpoint slide for fig. 1, rights and permissions.

Reprints and permissions

About this article

Cite this article.

Gurevitch, J., Koricheva, J., Nakagawa, S. et al. Meta-analysis and the science of research synthesis. Nature 555 , 175–182 (2018). https://doi.org/10.1038/nature25753

Download citation

Received : 04 March 2017

Accepted : 12 January 2018

Published : 08 March 2018

Issue Date : 08 March 2018

DOI : https://doi.org/10.1038/nature25753

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Investigate the relationship between the retraction reasons and the quality of methodology in non-cochrane retracted systematic reviews: a systematic review.

• Azita Shahraki-Mohammadi
• Leila Keikha
• Razieh Zahedi

Systematic Reviews (2024)

A meta-analysis on global change drivers and the risk of infectious disease

• Michael B. Mahon
• Alexandra Sack
• Jason R. Rohr

Nature (2024)

Systematic review of the uncertainty of coral reef futures under climate change

• Shannon G. Klein
• Cassandra Roch
• Carlos M. Duarte

Nature Communications (2024)

Meta-analysis reveals weak associations between reef fishes and corals

• Pooventhran Muruga
• Alexandre C. Siqueira
• David R. Bellwood

Nature Ecology & Evolution (2024)

Farming practices to enhance biodiversity across biomes: a systematic review

• Felipe Cozim-Melges
• Raimon Ripoll-Bosch
• Hannah H. E. van Zanten

npj Biodiversity (2024)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

30 Meta-Analysis and Quantitative Research Synthesis

Noel A. Card, Family Studies and Human Development, University of Arizona, Tucson, AZ

Deborah M. Casper, Family Studies and Human Development, University of Arizona, Tucson, AZ

• Published: 01 October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Meta-analysis is an increasingly common method of quantitatively synthesizing research results, with substantial advantages over traditional (i.e., qualitative or narrative) methods of literature review. This chapter is an overview of meta-analysis that provides the foundational knowledge necessary to understand the goals of meta-analysis and the process of conducting a meta-analysis, from the initial formulation of research questions through the interpretation of results. The chapter provides insights into the types of research questions that can and cannot be answered through meta-analysis as well as more practical information on the practices of meta-analysis. Finally, the chapter concludes with some advanced topics intended to alert readers to further possibilities available through meta-analysis.

Introduction to Meta-analysis

Meta-analysis, also referred to as quantitative research synthesis, is a systematic approach to quantitatively synthesizing empirical literature. By combining and comparing research results, metaanalysis is used to advance theory, resolve conflicts within a discipline, and identify directions for future research ( Cooper & Hedges, 2009 ). We begin by describing what meta-analysis is and what it is not.

Basic Terminology

It is important to provide a foundation of basic terminology on which to build a more technical and advanced understanding of meta-analysis. First, we draw the distinction between meta-analysis and primary and secondary analysis. The second distinction we draw is between quantitative research synthesis and qualitative literature review.

Glass (1976 ) defined primary-, secondary-, and meta-analysis as the analysis of data in an original study, the re-analysis of data previously explored in an effort to answer new questions or existing questions in a new way, and the quantitative analysis of results from multiple studies, respectively. A notable distinction between meta-analysis as compared to primary and secondary analysis involves the unit of analysis. In primary and secondary analyses, the units of analysis are most often the individual participants. In contrast, the units of analysis in a meta-analysis are the studies themselves or, more accurately, the effect sizes (defined below) of these studies.

A second foundational feature to consider is the distinction between quantitative research synthesis and qualitative literature review. Although both approaches are valuable to the advancement of knowledge, they differ with regard to focus and methodology. The focus of meta-analysis is on the integration of research outcomes, specifically in terms of effect sizes. In contrast, the focus of a qualitative literature review can be on research outcomes (although typically not focusing on effect sizes) but can also be on theoretical perspectives or typical practices in research. In terms of methods, scientists utilizing meta-analytic methodologies quantitatively synthesize findings to draw conclusions based on statistical principle. In contrast, scholars who conduct a qualitative literature review subjectively interpret and integrate research. Not considered in this chapter are other methodologies that fall between these two approaches on the taxonomy of literature review (for a more comprehensive review, see   Card, 2012 ; Cooper 1988 ).

As previously acknowledged, both quantitative research synthesis and qualitative literature review merit recognition for their respective contributions to the advancement of scientific knowledge. Quantitative literature reviews were developed to overcome many of the limitations of qualitative literature reviews, and we will highlight the advantages of quantitative literature reviews below. However, it is worth noting that quantitative research synthesis has also faced criticisms ( Chalmers, Hedges, & Cooper, 2002 ). Following are some highlights in the history of meta-analysis (for more thorough historical account, see   Chalmers, Hedges, & Cooper, 2002 ; Hedges, 1992 ; Hunt, 1997 ; Olkin, 1990 ).

A Brief History

Research synthesis methodology can be traced as far back as 1904 when Karl Pearson integrated five studies looking at the association between inoculation for typhoid fever and morality ( see   Olkin, 1990 ). By the 1970s, at least three independent groups had started to combine results from multiple studies ( Glass, 1976 ; Rosenthal & Rubin, 1978 ; Schmidt & Hunter, 1977 ), but the most influential work was Mary Smith and Gene Glass’ (1977) “meta-analysis” of psychotherapy, which was both ground-breaking and controversial. Smith and Glass’s (1977 ) meta-analysis sparked considerable controversy and debate as to the legitimacy of not only the findings but of the methodology itself ( Eysenck, 1978 ). It is worth noting, however, that some have suggested the controversy surrounding Smith and Glass’ (1977 ) meta-analysis had much more to do with the results than the methodology ( Card, 2012 ).

Following the somewhat turbulent introduction of meta-analysis into the social sciences, the 1980s offered significant contributions. These contributions came from both the advancement and dissemination of knowledge of meta-analytic techniques by way of published books describing the approach, as well as through the publication of research utilizing the methods ( Glass, McGaw, & Smith, 1981 ; Hedges & Olkin, 1985 ; Hunter, Schmidt, & Jackson, 1982 ; Rosenthal, 1984 ). Since its introduction into the social sciences in the 1970s, meta-analysis has become increasingly visible and has made considerable contributions to numerous bodies of scholarly research ( see   Cochran, 1937 ; Hunter, Schmidt, & Hunter, 1979 ; Pearsons, 1904 ; Rosenthal & Rubin, 1978 ; Glass & Smith, 1979 ; Smith & Glass, 1977 ).

Research Synthesis in the Social Sciences

Glass (1976 ) brought the need for meta-analysis to the forefront in a presidential address. It is not uncommon to observe conflicting findings across studies ( Cooper & Hedges, 2009 ). These inconsistencies lead to confusion and impede progress in social science (as well as in the so-called hard sciences; Hedges, 1987 ). Quantitative research synthesis is a powerful approach that addresses this problem through the systematic integration of results from multiple studies that often individually report conflicting results.

Chapter Overview

The following chapter is an overview of metaanalysis that provides the foundational knowledge necessary to understand the goals of meta-analysis and the process of conducting a meta-analysis, from the initial formulation of research questions through the interpretation of results. The chapter provides insights into the types of research questions that can and cannot be answered through meta-analysis as well as more practical information on the practices of meta-analysis. Finally, we conclude the chapter with some advanced topics intended to alert readers to further possibilities available through meta-analysis. To begin, we consider the types of questions that can and cannot be answered through meta-analysis.

Problem Formulation

Questions that can and cannot be answered through meta-analysis.

One of the first things to consider when conducting scientific research is the question for which you seek an answer; meta-analysis is no exception. A primary purpose for conducting a meta-analytic review is to integrate findings across multiple studies; however, not all questions are suitable for this type of synthesis. Hundreds, or sometimes thousands, of individual research reports potentially exist on any given topic; therefore, after an initial search of the literature, it is important to narrow the focus, identify goals, and articulate concise research questions that can be answered by conducting a tractable meta-analysis. A common misconception by those unfamiliar with meta-analysis is that an entire discipline or phenomenon can be “meta-analyzed” ( Card, 2012 ). Because of the infinite number of questions that could be asked—many of which could be answered using meta-analysis—this sort of goal is too as pecific. Rather, a more appropriate approach to quantitative research synthesis is to identify a narrowly focused goal or set of goals and corresponding research questions.

Identifying Goals and Research Questions

Cooper’s (1988 ) taxonomy of literature reviews identified multiple goals for meta-analysis. These include integration, theory development, and the identification of central issues within a discipline. We consider each of these goals in turn.

Integration . There are two general approaches to integrating research findings in meta-analysis: combining and comparing studies. The approach of combining studies is used to integrate effect sizes from multiple primary studies in an effort to estimate an overall, typical effect size. It would then be expectable to make inferences about this mean effect size by way of significance testing and/or confidence intervals. A second approach commonly used to integrate findings involves comparing studies. Also known as moderator analyses (addressed in more detail below), comparisons can be made across studies when a particular effect size is hypothesized to systematically vary on one or more of the coded study characteristics. Analyses to address each of these two approaches to integration will be described below.

Theory Development . A second goal of meta-analysis involves the development of theory. Meta-analysis can be used quite effectively and efficiently toward this end. If associations between variables that have been meta-analytically combined are weak, then this might indicate that a theory positing stronger relations of the constructs in question should be abandoned or modified ( Schmidt, 1992 ). If, on the other hand, associations are strong, then this may be an indication that the phenomenon under investigation is moving toward a more integrated theory. Ideally, meta-analyses can be used to evaluate competing theories that make different predictions about the associations studied. Either way, meta-analysis is a powerful tool that can be used toward the advancement of theory within the social sciences.

Integration of Central Issues . A final goal has to do with identifying central issues within a discipline or phenomenon. The exhaustive review of empirical findings can aid in the process of identifying key issues within a discipline, such as whether there is inadequate study of certain types of samples or methodologies. The statistical techniques of meta-analysis can address inconsistencies in the findings, attempting to predict these inconsistencies with coded study characteristics (i.e., moderator analyses). Both of these contributions are important to the process of identifying directions for future research and the advancement of knowledge.

Critiques of Meta-Analysis

Earlier, we described how the controversial nature of one of the earliest meta-analyses ( Smith & Glass, 1977 ) drew criticism not only of their findings but also of the technique of meta-analysis itself. Although these critiques have largely been rebuffed, they are still occasionally applied. Among the most common criticisms of meta-analysis are: (1) the “file drawer” problem; (2) the apples and oranges problem; (3) garbage in and garbage out; (4) the level of expertise required of the meta-analyst; and (5) the potential lack of qualitative finesse.

The “file drawer” problem . The “file drawer” problem, also known as the threat of publication bias, is based on the notion that significant results get published and nonsignificant findings get relegated to the “file drawer,” resulting in the potential for a publication bias in meta-analysis ( Rosenthal, 1979 ). To answer this criticism, however, meta-analysts typically employ both systematic and exhaustive search strategies to obtain published and unpublished reports in an effort to minimize this threat. In addition, there is an extensive collection of statistical procedures in meta-analysis that can be used to probe the existence, extent, and likely impact of publication bias ( Rothstein, Sutton, & Borenstein, 2005 ).

The apples and oranges problem . The apples and oranges problem describes the potential process of combing such a diverse range of studies that the aggregated results are meaningless. For example, if a meta-analyst attempted to investigate the predictors of childhood internalizing problems by including studies focusing on depression, anxiety, and social withdrawal, then it could be argued that the aggregation of results across this diverse range of problems is meaningless. This critique, in our opinion, is conceptual rather than methodological: Did the scientist using meta-analytic techniques define a sampling frame of studies within which it is useful to combine results? Fortunately, meta-analytic reviews can use both (1) combination to estimate mean results and (2) comparison to evaluate whether studies with certain features differ. Put differently, meta-analysis allows for both general and specific results. Returning to the example of a meta-analyst investigating the predictors of child psychopathology, it might be useful to present results of both (1) predictors of general internalizing problems, and (2) comparisons of the distinct predictors of depression, anxiety, and social withdrawal.

Garbage in and garbage out . Garbage in, garbage out describes the practice of including poor-quality research reports in a meta-analysis, which result in only poor-quality conclusions. Although this critique is valid in some situations, we believe a more nuanced consideration of “garbage” is needed before being used as a critique of a particular meta-analysis. In the next section , we will provide this consideration by discussing how the limits of primary research place limits on the conclusions that can be drawn from meta-analysis of that research.

The level of expertise required of the meta-analyst . A common misconception is that meta-analysis requires advanced statistical expertise. We would argue that with basic methodological and quantitative training, such as usually obtained in the first year of graduate school, many scientists could readily learn the basic techniques (through an introductory course or book on meta-analysis) to conduct a sound meta-analytic review.

The potential lack of qualitative finesse . A final criticism that has been raised is that meta-analysis lacks the “qualitative finesse” of a qualitative review. Perhaps tellingly, a definition of qualitative finesse is generally lacking when this critique is made, but it seems that this critique implies that a meta-analyst has not thought carefully and critically about the nuances of the studies and collection of studies. There certainly exist meta-analyses where this critique seems relevant—just as there exist primary quantitative studies in which careful thought seems lacking. The solution to this critique is not to abandon meta-analytic techniques, however, just as the solution to thoughtless primary studies is not to abandon statistical analyses of these data. Rather, this critique makes clear that meta-analysis—like any other methodological approach—is a tool to aid careful thinking, rather than a replacement for it.

Limits of Primary Research and Meta-Analysis

It is also important to recognize that the conclusions of a meta-analytic review must be tempered by the quality of the empirical research comprising this review. Many of the threats to drawing conclusions in primary research are likely to translate to meta-analysis as well. Perhaps the most salient threats involve flaws in the study design, sampling procedures, methodological artifacts, and statistical power.

Study design . The design of primary studies guides the types of conclusions that can be drawn from them; similarly, the design of studies included in a meta-analysis guides the types of conclusions that can be drawn. Experimental designs, although powerful in their ability to permit inferences of causality, often do not share the same ecological validity as correlational designs. Conversely, correlational designs cannot make inferences of causality. It would follow that any limitation existing within primary studies also exists within the meta-analyses that encompass these studies.

Sampling . Another limitation of primary studies is that it is difficult to support inferences generalizable beyond the sampling frame. When a sample is drawn from a homogeneous population, inferences can be made only for a limited set of individuals. Similarly, findings from a meta-analysis can only be generalized to populations within the sampling frame of the included studies; however, the collection of primary studies within a meta-analysis is likely to be more heterogeneous than one single primary study if it includes studies that are collectively diverse in their samples, even if each study sample is homogeneous.

Methodological artifacts . Both primary research and meta-analysis involve methodological shortcomings. Although it is difficult to describe all of the characteristics that make up a high-quality study, it is possible to identify those artifacts that likely lower the quality of the design. In primary studies, methodological issues need to be addressed prior to data collection. In contrast, meta-analysis can address these methodological artifacts in either one of two ways. The first way is to compare (through moderator analyses) whether studies with different methodological features actually yield different findings. Second, for some artifacts (e.g., measurement unreliability) described near the end of this chapter, corrections can be made that allow for the analysis of effect sizes free of these artifacts. Artifact correction is rarely performed in primary research (with the exception of latent variable modeling to correct for unreliability) but more commonly considered in meta-analyses.

Statistical power . Another limitation of much primary research is low statistical power ( Maxwell, 2004 ). Statistical power is the probability of detecting an effect that truly does exist but is often unacceptably low in many primary research studies. This low power results in incorrect conclusions in primary studies that an effect does not exist (despite cautions against “accepting” the null hypothesis). Fortunately, meta-analysis is usually less affected by inadequate power of primary studies because it combines a potentially large number of studies, thus resulting in greater statistical power.

Strengths of Meta-Analysis

As outlined above, there are limits to metaanalysis; however, meta-analysis should be recognized for its considerable strengths. We next briefly describe three of the most important of these: (1) a systematic and disciplined review process; (2) sophisticated reporting of findings; and (3) a way of combining and comparing large amounts of data ( Lipsey & Wilson, 2001 ).

Systematic and disciplined review process . First, systematic procedures must be followed to conduct a comprehensive literature search, consistently code comparable characteristics and effect sizes from studies, and to ensure the accuracy of combining results from multiple reports into one effect size. The processes of searching the literature, identifying studies, coding, and analyzing results have received tremendous attention in the literature on meta-analysis methodology, in contrast to most other forms of literature review. Although this work requires discipline, diligent attention to detail, and meticulous documentation on the part of the metaanalyst, when these procedures are followed, a large amount of data can be combined and compared and the outcome is likely to be a significant contribution to the field.

Combining and comparing large amounts of data . Perhaps one of the greatest strengths of meta-analytic techniques is the ability to combine and compare large amounts of data that would otherwise be impossible to integrate in a meaningful way. It would assuredly exceed the capacity of almost any scholar to combine the large amounts of data and draw meaningful conclusions without quantitative literature review techniques. Following the strength of combining and comparing large amounts of data is the strength in the way in which the findings are reported.

Sophisticated reporting of findings . Meta-analysis offers a level of sophistication in the way in which the findings are reported. Unlike qualitative literature reviews that derive and report conclusions and interpretations in a narrative format, meta-analysis uses statistical techniques to yield quantified conclusions. Meta-analysts commonly take advantage of visual tools such as stem-and-leaf plots, funnel plots, and tables of effect sizes to add a level of sophistication to the reporting of findings.

Searching the Literature

Defining a sampling frame.

Similarly to primary research, a sampling frame must be considered in meta-analysis. However, the unit of analysis in a meta-analysis is the study itself, as compared to the individuals in most primary studies. If we are to make inferences about the population of studies of interest, it is necessary to define the population a priori by articulating a set of criteria of the type of studies included versus excluded from this sampling frame.

Identifying Inclusion and Exclusion Criteria

As mentioned, the inclusion and exclusion criteria define the sampling frame of a meta-analysis. Establishing clear and explicit criteria will help guide the search process, a consideration particularly important if multiple individuals are working on the project. A second reason for identifying clear criteria is that it will help define the population of interest to which generalizations can be made. A final reason that clear criteria are necessary has to do with the ideas of transparency and replication. As with the sampling in well-conducted and well-reported primary studies, each decision and subsequent procedure utilized in the literature search of a meta-analysis must be transparent and replicable. Some of the more common search techniques and sources of information are described next.

Search Techniques and Identifying Resources

Many techniques have been used quite successfully toward the goal of searching the literature and identifying relevant resources. Two important concepts related to the literature search are recall and precision ( see   White, 2009 ). Recall is the percentage of studies retrieved that meet your inclusion criteria from all of those that actually exist. Precision is the percentage of studies retrieved that meet the inclusion criteria for the meta-analysis. The ideal literature search strategy provides both high recall and precision, although the reality is that decisions that affect efforts to improve recall often lower precision and vice versa.

By using multiple methods of searching for literature, meta-analysts strive to maximize recall without imposing impractical detriments on precision. The use of multiple search techniques helps this effort. The techniques most commonly used include searching: electronic databases using keywords, bibliographical reference volumes, unpublished works and other outlets (described below), conference presentations, funding agency lists, research registries, backward searches, forward searches, and personal communications with colleagues.

Electronic databases . Electronic databases are probably one of the most helpful tools for conducting literature searches developed in the past decades. Now, electronic database searches can identify as much of the relevant literature in a matter of hours or days, as would have taken weeks or months a few decades earlier (not to mention that these searches can be done from the comfort of one’s office rather than within the confines of a library). Most disciplines have electronic databases that serve primarily that particular discipline (e.g., PsychINFO for psychology, Medline for medicine, ERIC for education, etc.). With these and similar databases, the metaanalyst identifies the most relevant combination of keywords, wildcard marks (e.g., * ), and logical statements (e.g., and, or, not), and voluminous amounts of literature is quickly searched for matches. The electronic database is perhaps the most fruitful place to begin and is currently the primary tool used to search the literature.

Despite their advantages, it is worth mentioning a few cautions regarding electronic databases. First, an electronic search must not be used exclusively because of that which is not included in these databases. For example, many unpublished works might not be retrieved through electronic databases. Second, as mentioned previously, each discipline relies on one primary electronic database; therefore, multiple databases must be considered in your search. Third, electronic databases produce studies that match the keyword searches, but it is not possible to know what has been excluded. Using other search strategies and investigating why studies found by these strategies were not identified in the electronic database search is necessary to avoid unnecessary (and potentially embarrassing) omission of studies from a meta-analysis.

Bibliographical reference volumes . A method of locating relevant literature that was common as little as a decade ago is to search biographical reference volumes. These volumes are printed collections containing essentially the same information as electronic databases. Although these reference volumes are being phased out of circulation, you may find them useful if relevant literature was published some time ago (especially if the electronic databases have not yet incorporated this older literature).

Unpublished works . One of the challenges of meta-analysis has to do with publication bias ( see   Rothstein et al., 2005 ). If there is a tendency for significant findings to be more likely published than nonsignificant (presumably with smaller effect sizes) studies, then the exclusion of unpublished studies in a meta-analysis can be problematic. To balance this potential problem, the meta-analyst should make deliberate efforts to find and obtain unpublished studies. Some possible places to find such studies include conference program books, funding agency lists, and research registries.

Backward searches . Another technique commonly used in meta-analysis is the backward search. Once relevant reports are retrieved, it is recommended that the researcher thoroughly read each report and identify additional articles cited within these reports. This strategy is called a “backward” search because it proceeds backward in time from obtained studies toward previous studies.

Forward searches . A complimentary procedure, known as the forward search, involves searching for additional studies that have cited the relevant studies included in your meta-analysis (“forward” because the search proceeds from older studies to newer studies citing these previous works). To conduct this type of search, special databases (e.g., Social Science Citation Index) are used.

Personal communication with researchers in the field . A final search technique involves personal communication with the researchers in the field. It will be especially helpful to communicate with researchers in your field (those who will likely read your work) in an effort to locate resources that somehow escaped your comprehensive search efforts. An effective yet efficient way to do this is to simply email researchers in your field, let them know what type of meta-analysis you are conducting, and ask if they would be willing to peruse your reference list to see if there are any glaring oversights.

Coding Study Characteristics

In a meta-analysis, study characteristics are systematically coded for two reasons. First, this coded information is presented to describe the collective field being reviewed. For example, do studies primarily rely on White college students, or are the samples more diverse (either within or across studies)? Do studies rely on the same measures or types of measures, or has the phenomenon been studied using multiple measures?

A second reason for systematically coding study characteristics is for use as potential predictors of variation in effect sizes across studies (i.e., moderators, as described below in section titled Moderator Analyses). In other words, does variation across studies in the coded study characteristics co-occur with differences in results (i.e., effect sizes) from these studies? Ultimately, the decision of what study characteristics should be coded derives from the meta-analysts’ substantive understanding of the field. There are at least three general types of study features that are commonly considered: characteristics of the sample, the methodology, and the source.

Coding Sample Characteristics

Sample characteristics include any descriptions of the study samples that might systematically covary with study results (i.e., effect sizes). Some meta-analyses will include codes for the sampling procedures, such as whether the study used a representative sample or a convenience sample (e.g., college students), or whether the sample was selected from some specific setting, such as clinical treatment settings, schools, or prisons. Nearly all meta-analyses code various demographic features of the sample, such as the ethnic composition, proportion of the sample that is male or female, and the average age of participants in the sample.

Coding Methodological Characteristics

Potential methodological characteristics for coding include both design and measurement features. At a broad level, a meta-analyst might code broad types of designs, such as experimental, quasiexperimental, and single-subject ABAB studies. It might also be useful to code at more narrow levels, such as the type of control group used within experimental treatment studies (e.g., no contact, attention only, treatment as usual). Similarly, the types of measures used could be coded as either broad (e.g., parent vs. child reports) or narrow (CBCL vs. BASC parent reports). In practice, most meta-analysts will code methodological features at both broad and narrow levels, first considering broad-level features as predictors of variability in effect sizes, and then using more narrow-level feature if there exists unexplained variation in results within these broad features.

Coding Source Characteristics

Source characteristics include features of the report or author that might plausibly be related to study findings. The most commonly coded source characteristic is whether the study was published, which is often used to evaluate potential publication bias. The year of publication (or presentation, for unpublished works) is often used as a proxy for the historic time in which the study was conducted. If the year predicts differences in effect sizes, then this may be evidence for historic change in the phenomenon over time. Other source characteristics, such as characteristics of the researcher (e.g., gender, ethnicity, discipline), are less commonly coded but are possibilities. For example, some meta-analyses of gender differences have coded the gender of the first author to evaluate the possibility that the researchers’ presumed biases may somehow impact the results found (e.g., Card, Stucky, Sawalani, &Little, 2008 ).

Coding Effect Sizes

As mentioned, study results in meta-analysis are represented as effect sizes. To be useful in metaanalysis, a potential effect size needs to meet four criteria. First, it needs to quantify the direction and magnitude of a phenomenon of interest. Second, it needs to be comparable across studies that use different sample sizes and scales of measurement. Third, it needs to be either consistently reported in studies included in the meta-analysis or else it can be computed from commonly reported results. Fourth, it is necessary that the meta-analyst can compute its standard error, which is used for weighting of studies in subsequent meta-analytic combination and comparison.

The three effect sizes most commonly used in meta-analyses all index associations between two variables. The correlation coefficient (typically denoted as r ) quantifies associations between two continuous variables. The standardized mean differences are a family of effect sizes (we will focus on Hedges’ g ) that quantify associations between a dichotomous (group) variable and a continuous variable. The odds ratio (denoted as either o or OR) is a useful and commonly used index for associations between two dichotomous variables ( Fleiss, 1994 ). We next describe these three indexes of effect size, the correlation coefficient, the standardized mean difference, and the OR. After describing each of these effect sizes indexes, we will describe how these are computed from results commonly reported in empirical reports.

Correlation Coefficient

Correlation coefficients represent associations between two variables on a standardized scale from − 1 to +1. Correlations near 0 denote the absence of association between two variables, whereas positive values indicate that scores on one variable tend to be similar to scores on another (relatively high scores on one variable tend to occur with relatively high scores on the other, as do low scores tend to occur with low scores), whereas negative scores indicate the opposite (high scores with low scores). The correlation coefficient has the advantage of being widely recognized by scientists in diverse fields. A commonly applied suggestion is that r ≍ ±0.10 is considered small, r ≍ ±0.30 is considered medium, and r ≍ ±0.50 is considered large; however, disciplines and fields differ in their evaluations of what constitutes small or large correlations, and researchers should not be dogmatic in its application.

Although r has many advantages as an effect size, it has the undesirable property for meta-analysis of having sample estimates that are skewed around the population mean. For this reason, meta-analysts should transform r to Fisher’s Z r prior to analysis using the following equation:

Although Z r has desirable properties for meta-analytic combination and comparison, it is not very interpretable by most readers. Therefore, metaanalysts back-transform results in Z r metric (e.g., mean effect size) to r for reporting using the following equation:

As mentioned earlier, and will be described in greater detail below, it is necessary to compute the standard error of the estimation of the effect size ( Z r ) for use in weighting studies in meta-analysis. The equation for the standard error of Z r   ( S E Z r ) is a simple function of the study sample size:

Standardized Mean Differences

There exist several standardized mean differences, which index associations between a dichotomous “group” variable and a continuous variable. Each of these standardized mean differences indexes the direction and magnitude of differences between two groups in standard deviation units. We begin with one of the more common of these indices, Hedges’ g , which is defined as:

The numerator of this equation contains the difference between the means of two groups (groups 1 and 2) and will yield a positive value if group 1 has a higher mean than group 2 or a negative value if group 2 has a higher mean than group 1. Although it is arbitrary which group is designated 1 or 2, this designation must be consistent across all studies coded for a meta-analysis.

If all studies in a meta-analysis use the same measure, or else different measures with the same scale, then the numerator of this equation alone would suffice as an effect size for meta-analysis (this is the unstandardized mean difference). However, the more common situation is that different scales are used across different studies, and in this situation it would make no sense to attempt to combine these unstandardized mean differences across studies. To illustrate, if one study comparing treatment to control groups measured an outcome on a 1 to 100 scale and found a 10-point difference, whereas another study measured the outcome on a 0 to 5 scale and found a 2-point difference, then there would be no way of knowing which—if either—study had a larger effect. To make these differences comparable across studies, it is necessary to standardize them in some way, typically by dividing the mean difference by a standard deviation.

As seen in equation (4) above, this standard deviation in the divisor for g is the pooled (i.e., combined across the two groups) estimate of the population standard deviation. Other variants within the standardized mean difference family of effect sizes use different divisors. For example, the index d uses the pooled sample standard deviation and a less commonly used index, g Glass (also denoted as Glass’ Δ), uses the estimated population standard deviation for one group (the group that you believe is a more accurate estimate of population standard deviation, such as the control group if you believe that treatment impacts the standard deviation). The latter index ( g Glass ) is less preferred because it cannot be computed from some commonly reported statistics (e.g., t tests), and it is a poorer estimate if the standard deviations are, in fact, comparable across groups ( Hedges & Olkin, 1985 ).

In this chapter, we focus our attention primarily on g , and we will describe the computation of g from commonly reported results below. Like other standardized mean differences, g has a value of 0 when the groups do not differ (i.e., no association between the dichotomous group variable and the continuous variable), and positive or negative values depending on which group has a higher mean. Unlike r, g is not bounded at 1, but can have values greater than ±1.0 if the groups differ by more than one standard deviation.

Although g is a preferred index of standardized mean differences, it exhibits a slight bias when estimated from small samples (e.g., sample sizes less than 20). To correct for this bias, it is common to apply the following correction:

As with any effect size used in meta-analysis, it is necessary to compute the standard error of estimates of g for weighting during meta-analytic combination. The standard error of g is more precisely estimated using the sample sizes from both groups under consideration (i.e., n 1 and n 1 for groups 1 and 2, respectively) using the left portion of Equation 6 but can be reasonably estimated using overall sample size ( N Total ; right portion of Equation 6 ) when exact group sizes are unknown but approximately equal (no more than a 3-to-1 discrepancy in group sizes; Card, 2012 ; Rosenthal, 1991 ):

Odds Ratios

The odds ratio, denoted as either o or OR, is a useful index of associations between two dichotomous variables. Although readers might be familiar with other indices of two variable associations, such as the rate (also known as risk) ratio or the phi coefficient, the OR is advantageous because it is not affected by differences in the base rates of dichotomous variables across studies and is computed from a wider range of study designs ( see   Fleiss, 1994 ). The OR is estimated from 2 × 2 contingency tables by dividing the product of cell frequencies in the major diagonal (i.e., frequencies in the cells where values of the two variables are both 0 { n 00 }or both 1 { n 11 }) by the product of cell frequencies off the diagonal (i.e., frequencies in the cells where the two variables have different values, n 10 and n 01 ):

The OR has a rather different scale than either r or g . Values of 1.0 represent no association between the dichotomous variables, values from 1 down to 0 represent negative association, and values from 1 to infinity represent positive associations. Given this scale, o is obviously skewed; therefore, a log transformation is applied to o when included in a meta-analysis: ln ( o ). The standard error of this log-transformed odds ratio is a function of number of participants in each cell of the 2 × 2 contingency table:

Computing Effect Sizes From Commonly Reported Data

Ideally, all studies that you want to include in a meta-analysis will have effect sizes reported, and it is a fairly straightforward matter to simply record these. Unfortunately, many studies do not report effect sizes (despite many calls for this reporting; e.g., Wilkinson et al., 1999 ), and it is necessary to compute effect sizes from a wide variety of information reported in studies. Although it is not possible to consider all possibilities here, we next describe a few of the more common situations. Table 30.1 summarizes equations for computing r and g in these situations (note that it is typically necessary to reconstruct contingency tables from reported data to compute the odds ratio; see   Fleiss, 1994 ).

It is common for studies to report group comparisons in the form of either the (independent samples) t -test or as the results of a two-group (i.e., 1 df ) analysis of variance (ANOVA). This occurs either because the study focused on a truly dichotomous grouping variable (in which case, the desired effect size is a standardized mean difference such as g ) or because the study authors artificially dichotomized one of the continuous variables (in which case the desired effect size is r ). In these cases, either r or g can be computed from the t statistic of F ratio in Table 30.1 . For the F ratio, it is critical that the result is from a two-group (i.e., 1 df ) comparison (for discussion of computing effect sizes from > 1 df F ratios, see   Rosenthal, Rosnow, & Rubin, 2000 ). When computing g (but not r ), a more precise estimate can be made if the two group sizes are known; otherwise, it is necessary to use the approximations shown to the right of Table 30.1 (e.g., in the first row for g , the exact formula is on the left and the approximation is on the right).

An alternate situation is that the study has performed repeated-measures comparisons (e.g., pretreatment vs. posttreatment) and reported results of dependent, or repeated-measures, t -tests, or F ratios. The equations for computing r from these results are identical to those for computing from independent samples tests; however, for g , the equations differ for independent versus dependent sample statistics, as seen in Table 30.1 .

A third possibility is that the study authors represent both variables that constitute your effect size of interest as dichotomous variables. The study might report the 1 df χ 2 of this contingency or the data that can be used to construct the contingency table and the subsequent value. In this situation, r and g are computed from this χ 2 value and sample size ( N ). As with the F ratio, it is important to keep in mind that this equation only applied to 1 df χ 2 values (i.e., 2 × 2 contingency tables).

The last situation we will discuss is when the authors report none of the above statistics but do report a significance level (i.e., p ). Here, you can compute the one-tail standard normal deviate, Z , associated with this significance level (e.g., Z = 1.645 for p = 0.05) and then use the equations of Table 30.1 to compute r or g . These formulas are used when an exact significance level is reported (e.g., p = 0.027); if they are applied to ranges (e.g., p < 0.05), then they provide only a lower-bound estimate of the actual effect size.

Although we have certainly not covered all possible situations, these represent some of the most common situations you are likely to encounter when coding effect sizes for a meta-analysis. For details of these and other situations in which you might code effect sizes, see   Card (2012 ) or Lipsey and Wilson (2001 ).

Analysis of Mean Effect Sizes and Heterogeneity

After coding study characteristics and effect sizes from all studies included in a meta-analysis, it is possible to statistically combine and compare results across studies. In this section, we describe a method (fixed effects) of computing a mean effect size and making inferences about this mean. We then describe a test of heterogeneity that informs whether the between-study variability in effect sizes is greater than expectable by sampling fluctuation alone. Finally, we describe an alternative approach to computing mean effect sizes (random effects) that accounts for between-study variability.

Fixed-Effects Means

One of the primary goals of meta-analytic combination of effect sizes from multiple studies is to estimate an average effect size that exists in the literature and then to make inferences about this average effect size in the form of statistical significance and/or confidence intervals. Before describing how to estimate and make inferences about a mean effect size, we briefly describe the concept of weighting.

Weighting in Meta-Analysis . Nearly all (and all that we describe here) analyses of effect sizes in meta-analysis apply weights to studies. These weights are meant to in dex the degree of precision in each study’s estimate of the population effect size, such that studies with more precise estimates receive greater weight in the analyses than studies with less precise estimates. The most straightforward weight is the inverse of the variance of a study’s estimate of the population effect size. In other words, the weight of study i is the inverse of the squared standard error from that study:

As described above, the standard error of a study largely depends on the sample size (and for g , the effect size itself), such that studies with large samples have smaller standard errors than studies with small samples. Therefore, studies with large samples have larger weights than studies with smaller samples.

Fixed-Effects Mean Effect Sizes . After computing weights for each study using the equation above, estimating the mean effect size ( E ¯ S ¯ ) across studies is a relatively simple matter of computing the weighted mean of effect sizes across all studies:

This value represents the estimate of a single effect size in the population based on information combined from all studies included in the meta-analysis. Because it is often useful to draw inferential conclusions, the standard error of this estimate is computed using the equation:

This standard error can then be used to compute either statistical significance or confidence intervals. For determining statistical significance, the mean effect size is divided by the standard error, and the resulting ratio is evaluated as a standard normal deviate (i.e., Z -test, with, e.g., values larger than ±1.96 having p < 0.05). For computing confidence intervals, the standard error is multiplied by the standard normal deviate associated with the desired confidence interval (e.g., Z = 1.96 for a 95% confidence interval), and this product is then subtracted from and added to the mean effect size to identify the lower- and upper-bounds of the confidence interval.

If the effect size chosen for the meta-analysis (i.e., r, g , or o ) was transformed prior to analyses (e.g., r to Z r ), then the mean effect size and boundaries of its confidence interval will be in this transformed metric. It is usually more meaningful to back-transform these values to their original metrics for reporting.

Heterogeneity

In addition to estimating a mean effect size, meta-analysts evaluate the variability of effect sizes across studies. Some degree of variability in effect sizes across studies is always expectable; the fact that different studies relied on different samples results in somewhat different estimates of effect sizes because of sampling variability. In situations where effect sizes differ by an amount expectable due to sampling variability, the studies are considered homogeneous with respect to their population effect sizes. However, if effect sizes vary across studies more than expected by sampling fluctuation alone, then they are considered heterogeneous (or varying) with respect to their population effect sizes.

It is common to perform a statistical test to evaluate heterogeneity. In this test, the null hypothesis is of homogeneity, or no variability, in population effect sizes across studies (i.e., any variability in sample effect sizes is caused by sampling variability), whereas the alternative hypothesis is of heterogeneity, or variability, in population effect sizes across studies (i.e., variability in sample effect sizes that is not accounted for by sampling variability alone). The result of this test is denoted by Q :

The statistical significance of this Q is evaluated as a χ 2 distribution with df = number of studies – 1. You will note that this equation has two forms. The left portion of Equation 12 is the definitional equation, which makes clear that the squared deviation of the effect size from each study i from the overall mean effect size is being weighted and summed across studies. Therefore, small deviations from the mean will contribute to small values of Q (homogeneity), whereas large deviations from the mean will contribute to large values of Q (heterogeneity). The right portion of Equation 12 is an algebraic rearrangement that simplifies computation (i.e., a computational formula).

Results of this test have implications for subsequent analyses. Specifically, a conclusion of homogeneity (more properly, failure to conclude heterogeneity) suggests that the fixed-effects mean described above is an acceptable way to summarize effect sizes, and this conclusion may contraindicate moderator analyses (described below). In contrast, a conclusion of heterogeneity implies that the fixedeffects mean is not an appropriate way to summarize effect sizes, but, rather, a random-effects model (described in the next section ) should be used. Further, a conclusion of heterogeneity indicates that moderator analyses (described below) may help explain this between-study variance (i.e., heterogeneity). It is worth noting that the result of this heterogeneity test is not the sole basis of deciding to use random-effects models or to conduct moderator analyses, and meta-analysts often base these decisions on conceptual rather than empirical grounds ( see   Card, 2012 ; Hedges & Vevea, 1998 ).

Random-Effects Means

Estimation of means via a random-effects model relies on a different conceptual model and analytic approach than estimation via a fixed-effects model. We describe this conceptual model and estimation procedures next.

Conceptualization of Random-Effects Means . Previously, when we described estimation of a fixedeffects mean, we describe a single population effect size. In contrast, a random-effects model assumes that there is a normal distribution of population effect sizes. This distribution of population effect sizes has a mean, which we estimate as described next. However, it also has a degree of spread, which can be indexed by the standard deviation (or variance) of effect sizes at the population level. To explicate the assumptions in equation form, the fixed- and random-effects models assume that the effect sizes observed in study i ( ES i ) are a function of the following, respectively:

In both Equation 13 (fixed effects) and Equation 14 (random effects), effect sizes in a study partly result from the sampling fluctuation of that study (ε i ). In the fixed-effects model, this sampling fluctuation is around a single population effect size (θ). In contrast, the random-effects model specifies that the population effect size is a function of both a mean population effect size (μ) as well as the deviation of the population effect size of study i from this mean (ξ i ). Although it is impossible to know the sampling fluctuation and the population deviation from a single study, it is possible to estimate the respective variances of each across studies.

Estimating Between-Study Population Variance . We described above the heterogeneity test, indexed by Q , which is a statistical test of whether variability in observed effect sizes across studies could be accounted for by sampling variability alone (i.e., the null hypothesis of homogeneity) or was greater than expected by sampling variability (i.e., the alternate hypothesis of heterogeneity). To estimate betweenstudy population variance τ 2 in effect sizes, we evaluate how much greater Q is than that expected under the null hypothesis of homogeneity (i.e., sampling variance alone):

Note that this equation is used only if Q ≥ k − 1 to avoid negative variance estimates (if Q < k − 1, τ 2 = 0). Although this equation is not intuitively obvious, consideration of the numerator helps clarify. Recall that large values of Q result when studies have effect sizes with large deviations from the mean effect size and that under the null hypothesis of homogeneity, Q is expected to equal the number of studies ( k ) minus 1. To the extent that Q is much larger than this expected value, the numerator of this equation will be large, implying large population between-study variability. In contrast, if Q is not much higher than the expected value under homogeneity, then the population between-study variability will be near zero.

Estimating Random-Effects Means . If studies have a sizable amount of randomly distributed between-study variance in their population effect sizes, then this implies that each is a less precise estimate of mean population effect size. In other words, each contains more uncertainty as information for estimating this value. To capture this uncertainty, or lower precision, analyses under the random-effects model use a different weight than those of the fixed-effects model. Specifically, the random-effects weight, denoted as w ∗ (or sometimes w RE ), for study i is the inverse of the sum of this between-study variance (τ 2 ) and the sampling variance for that study (i.e., squared standard error, S E i 2 ):

This random-effects weight will be smaller than the comparable fixed-effects weight, with the discrepancy increasing with greater between-study variance. These random-effects weights are simply used in the equations above to estimate a random-effects mean effect size (Equation 10 ), as well as a standard error for this mean (Equation 9 ) for inferential tests.

Moderator Analyses

Moderator analyses are another approach to managing heterogeneity in effect sizes ( Hedges & Pigott, 2004 ), but here the focus is on explaining (versus simply modeling as random) this between-study variance. These analyses use coded study characteristics to predict effect sizes; the reason these analyses are called “moderator” analyses is because they evaluate whether the effect size—a two-variable association—differs depending on the level of the third, moderator, variable—the study characteristic. It is often of primary interest to understand whether the association between two variables differs based on the level of a third variable (the moderator). Therefore, moderator analyses identifying those characteristics of the study that lead to higher or lower effects sizes are very commonly performed in meta-analyses. In this section, we briefly consider two types of moderators (i.e., categorical and continuous) along with the procedures used to inves-tigate these two types of moderators in meta-analysis (i.e., an adapted ANOVA procedure and a multiple regression procedure, respectively).

Single Categorical Moderator

A categorical variable is any variable on which a participants, observations or, in the case of metaanalysis, studies can be distinctly classified. Testing categorical moderators in meta-analysis involves comparing the mean effects of groups of studies classified by their status on some categorical variable.

Evaluating the Significance of a Categorical Moderator . Categorical moderator analysis in meta-analysis is similar to ANOVA in primary research. In the context of primary research, ANOVA partitions variability between groups of individuals into variability between and within these groups. Similarly, in meta-analysis, the ANOVA procedure is used to partition between-study heterogeneity into heterogeneity that exists between and within groups of studies. Earlier (Equation 12 ), we provided equations for quantifying the heterogeneity as Q ; we now provide this equation again, but now specifying that this is the total heterogeneity among studies:

This Q Total refers to the heterogeneity that exists across all studies. It can be partitioned into between-group ( Q Between ) and within-group ( Q Within ) components by the fact that Q Total = Q Between + Q Within . It is simpler to compute Q Within than Q Between , so it is common to subtract this from the total heterogeneity to obtain the between group variance. Within each group of studies, g , the heterogeneity can be estimated among just the studies in this group:

Then, these estimates of heterogeneity within each group can be summed across groups to yield the within study heterogeneity:

As stated above, testing categorical moderators within an ANOVA framework is done by separating the total heterogeneity ( Q Total ) into between-group ( Q Between ) and within-group ( Q Within ) heterogeneity. Therefore, after computing the total heterogeneity ( Q Total ) and the within-group heterogeneity ( Q Within ), you simply subtract the within-group heterogeneity from the total heterogeneity to find Q Between . This value is evaluated as a χ 2 distribution with df = number of groups – 1. If this value is statistically significant, then this is evidence that the level of the categorical moderator predicts variability in effect sizes—moderation.

Single Continuous Moderator

A continuous study characteristic is one that is measured on a scale that can potentially take on an infinite, or at least large number of values. In metaanalysis, a continuous moderator is a coded study characteristic (e.g., sample age, SES) that varies along a continuum of values and is hypothesizes to predict effect sizes.

Similarly to the use of an adapted ANOVA procedure in the evaluation of categorical moderators in meta-analysis, we use an adapted multiple regression procedure for the evaluation of continuous moderators in meta-analysis ( Hedges & Pigott, 2004 ). This adaptation to the evaluation of a continuous moderator involves a weighted regression of the effect sizes (dependent variable) onto the continuous moderator.

To evaluate potential moderation of a continuous moderator within a multiple regression framework, we regress the effect sizes onto the hypothesized continuous moderator using a standard regression equation: Z ES = B O + B 1 (Study Characteistic) + e , using w as the (weighted least squares) weight. From the results, we are interested in the sum of squares of the regression model (which is the heterogeneity accounted for by the linear regression model, Q Regression , and evaluated on a chi-square distribution with df = number of predictors), and sometimes the residual sum of squares (which is Q Residual , or heterogeneity not explained by the study characteristic). The unstandardized regression coefficient indicates how the effect size changes per unit change in the continuous moderator. The standard error of this coefficient is inaccurate in the regression output and must be adjusted by dividing it by the square root of the MS Residual .

The statistical significance of the predictor can also be evaluated by dividing the regression coefficient ( B 1 ) by the adjusted standard error, evaluated on the standard two-tail Z distribution. Interpretation of moderation with continuous variables is not as straightforward as with categorical moderators; it is necessary to compute implied effect sizes at different levels of the continuous moderator.

Multiple Regression to Analyze Categorical Moderators

Thus far we have considered moderation by a single categorical variable within an ANOVA framework and by a continuous variable within a regression framework. Next, we address categorical moderators within a multiple regression framework. Before doing so, it is useful to consider how the analyses we have described to this point fit within this general multiple regression framework.

The Empty Model . By empty model, we are referring to a model that includes only an intercept (a constant value of 1 for all cases) as a predictor. A weighted regression of effect sizes predicted only by a constant is often useful for an initial analysis of the mean effect size and to evaluate heterogeneity of these effect sizes across studies. The following equation accomplishes this:

In this empty model, the intercept regression coefficient is the mean effect size, and the sum of squares of the residual is the heterogeneity.

Use of Dummy Variables to Analyze Categorical Moderators . To evaluate the categorical moderators in this meta-regression framework, dummy variables can be used to represent group membership. Here, we select a reference group to which we would assign the value 0 for all the dummy codes for studies using that particular reference group, and each dummy variable represents the difference of another group relative to this reference group. The effect size is regressed onto the dummy variables, weighted by the inverse variance weight w , with the following equation:

The results of this regression are interpreted as above. The Q Regression is equivalent to the Q Between of the ANOVA framework and is used to determine whether there is categorical moderation. To identify the particular groups that differ from the reference group, the regression coefficients for the dummy variables are considered. Again, the standard errors of these coefficients are inaccurate and need to be adjusted as described above.

Multiple Moderators . A multiple regression framework can also be used to evaluate multiple categorical and/or continuous predictors in meta-analysis. Here, it is likely of interest to consider both the overall regression model ( Q Regression ) as well as the results of particular predictors. The former is evaluated by interpreting the model sum of squares as a χ 2 distribution with df = number of predictors – 1. The latter is evaluated by dividing the regression coefficients by their adjusted standard errors.

Limitations to Interpretation of Moderators

Clearly, moderation analyses can enhance the conclusions drawn from meta-analysis, but there are some limitations that need also be considered. The first consideration is that of multicolinearity in meta-analytic moderator analyses. It is likely that some moderator variables will be correlated, but this can be assessed by regressing each moderator onto the set of other moderators using the weights you used in the moderator analyses. The second limitation is the possibility that uncoded variables are confounding the association and moderation of the variables that are coded. The best approach to avoid confounding variables is to code as many variables as possible. Finally, it will be important to feel confident that the literature included in your synthesis adequately covers the range of potential moderator values. This can best be analyzed by plotting the included studies at the various levels of the moderator.

Advanced Topics

Given the existence of meta-analytic techniques over several decades, and their widespread use during this time, it is not surprising that there exists a rich literature on meta-analytic techniques. Although space has precluded us from discussing all of these topics, we next briefly describe a few of the more advanced topics important in this field.

Alternative Effect Sizes

The three effect sizes described in this chapter (i.e., r, g , and o ) quantify two-variable associations and are the most commonly used effect sizes for meta-analysis. However, there exist many other possibilities that might be considered.

Single Variable Effect Sizes . In some cases, it may be valuable to meta-analytically combine and/or compare information about single variables rather than two-variable associations. Central tendency can be indexed by the mean for continuous data or by the proportion for dichotomous variables; both of these effect sizes can be used in meta-analyses ( see   Lipsey & Wilson, 2001 ). It is also possible to use standard deviations or variances as effect sizes for meta-analysis to draw conclusions about interindi-vidual differences. Meta-analytic combination of means and variances require that the same measure, or else different measures on the same scale, be used for all studies.

Meaningful Metric . The effect sizes we have described are all in some standardized metric. However, there may be instances when the scales of variables comprising the effect size are meaningful, and therefore it is useful to use unstandard-ized effect sizes. Meta-analysis of such effect sizes were described in a special section of the journal Psychological Methods ( see   Becker, 2003 ).

Multivariate Effect Sizes . Many research questions go beyond two-variable associations to consider multivariate effect sizes, such as whether X uniquely predicts Y above and beyond Z . It is statistically possible to meta-analytically combine and compare multivariate effects sizes, such as regression coefficients or partial/semipartial correlation to address the example associations among X, Y , and Z . However, it is typically not possible in practice to use multivariate effect sizes for meta-analyses. The primary reason is that their use would require that the same multivariate analyses are performed and reported across studies in the meta-analysis. For example, it would be necessary for all studies included to report the regression of Y on X controlling for Z ; studies that failed to control for Z , that instead controlled for W , or that controlled for both Z and W could not be included. A more tractable alternative to the meta-analysis of multivariate effect sizes is to perform multivari-ate meta-analysis of bivariate effect sizes, which we briefly describe below.

Artifact Corrections

Artifacts are study imperfections that lead to biases—typically underestimations—of effect sizes. For example, it is well known that unreliability of a measure attenuates (i.e., reduces) the magnitude of observed associations that this variable has with others relative to what would have been found with a perfectly measured variable. In addition to measurement unreliability, other artifacts include imperfect validity of measures, artificial dichotomization of continuous variables, and range restriction of the sample on a variable included in the effect size (direct range restriction) or another variable closely related to a variable in the effect size (indirect range restriction).

The general approach to correcting for artifacts is to compute a correction factor for each artifact using one of a variety of equations ( see   Hunter & Schmidt, 2004 ). For example, one of the more straightforward corrections is for unreliability of a measure of X (where r xx is the reliability of X ):

Each of the artifact corrections may yield a correction factor, which are then multiplied together to yield an overall artifact multiplier ( a ). This artifact multiplier is then used to estimate an adjusted effect size from the observed effect size to index what the effect size would likely have been if the artifacts (study imperfections) had not existed:

This estimation of artifact-free effect sizes from observed effect sizes is unbiased (i.e., it will not consistently over- or underestimate the true effect size), but it is also not entirely precise. In other words, the artifact correction introduces additional uncertainty in the effect size estimate that must be considered in the meta-analysis. Specifically, the standard error of the effect size, which can be thought of as representing imprecision in the estimate of the effect size, is also adjusted by this artifact multiplier to account for this additional uncertainty introduced by artifact correction:

Multivariate Meta-Analysis

Multivariate meta-analysis is a relatively new and underdeveloped approach, but one that has great potential for use. Because the approach is fairly complex, and there is not general agreement on what techniques are best in different situations, we describe this approach in fairly general terms, referring interested readers to Becker (2009 ) or Cheung and Chan (2005 ).

The key idea of multivariate meta-analysis is to meta-analytically combine multiple bivariate effect sizes, which are then used as sufficient statistics for multivariate analyses. For example, to fit a model in which variable X is regressed on variables Y and Z , you would perform three meta-analyses of the three correlations ( r XY , r XZ , and r YZ ), and this matrix of meta-analytically combined correlations would then be used to estimate the multiple regression parameters.

Although the logic of this approach is reasonably simple, the application is much more complex. Challenges include how one handles the likely possibility that different studies provide different effect sizes, what the effective sample size is for the multivariate model when different studies inform different correlations, how (or even whether) to test for and potentially model between-study heterogeneity, and how to perform moderator analyses. Answers to these challenges have not been entirely agreed upon even by quantitative experts, making it difficult for those wishing to apply these models to answer substantive research questions. However, these models offer an extremely valuable potential for extending meta-analytic techniques to answer richer research questions than two-variable associations that are the typical focus of meta-analyses.

Although we have been able to provide only a brief overview of meta-analysis in this chapter, we hope that the opportunities of this methodology are clear. Given the overwhelming and increasing quantity of empirical research in most fields, techniques for best synthesizing the existing research are a critical tool in advancing our understanding.

Becker, B. J. ( 2003 ). Introduction to the special section on metric in meta-analysis.   Psychological Methods, 8, 403–405.

Google Scholar

Becker, B. J. ( 2009 ). Model-based meta-analysis. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 377–395). New York: Russell Sage Foundation

Google Preview

Card, N. A. ( 2012 ). Meta-analysis: Quantitative synthesis of social science research . New York: Guilford.

Card, N. A. , Stucky, B. D. , Sawalani, G. M. , & Little, T. D. ( 2008 ). Direct and indirect aggression during childhood and adolescence: A meta-analytic review of gender differences, intercorrelations, and relations to maladjustment.   Child Development, 79, 1185–1229.

Chalmers, I. , Hedges, L.V. , & Cooper, H. ( 2002 ). A brief history of research synthesis.   Evaluation and Health Professions, 25, 12–37.

Cheung, M. W. L. , & Chan, W. ( 2005 ). Meta-analytic structural equation modeling: A two-stage approach.   Psychological Methods, 10, 40–64.

Cochran, W. G. ( 1937 ). Problems arising in the analysis of a series of similar experiments.   Journal of the Royal Statistical Societ y, 4(Suppl.), 102–118.

Cooper, H. M. ( 1988 ). Organizing knowledge syntheses: A taxonomy of literature reviews.   Knowledge in Societ y, 1, 104–126.

Cooper, H. M. , & Hedges, L. V. ( 2009 ). Research synthesis as a scientificprocess. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 3–16). New York: Russell Sage Foundation.

Eysenck, H. J. ( 1978 ). An exercise in mega-silliness.   American Psychologist, 33, 517.

Fleiss, J. H. ( 1994 ). Measures of effect size for categorical data. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 245–260). New York: Russell Sage Foundation.

Glass, G. V. ( 1976 ). The critical state of the critical review article.   Quarterly Review of Biology , 50th Anniversary Special Issue (1926–1976), 415–418.

Glass, G. V. , McGraw, B. , & Smith, M. L. ( 1981 ). Meta-analysis in social research . Thousand Oaks, CA: Sage.

Glass, G. V. & Smith, M. ( 1979 ). Meta-analysis of the relationship between class size and achievement.   Educational Evaluation and Policy Analysi s, 1, 2–16.

Hedges, L. V. ( 1987 ). How hard is hard science, how soft is soft science? The empirical cumulativeness of research.   American Psychologist, 42, 443–455.

Hedges, L. V. ( 1992 ). Meta-analysis.   Journal of Educational Statistics, 17, 279–296.

Hedges, L. V. , & Olkin, I. ( 1985 ). Statistical methods for meta-analysis . San Diego, CA: Academic Press.

Hedges, L. V. & Pigott, T. D. ( 2004 ). The power of statistical tests for moderators in meta analysis.   Psychological Method s, 9, 426–445.

Hedges, L. V. , & Vevea, J. L. ( 1998 ). Fixed- and random-effects models in meta-analysis.   Psychological Method s, 3, 486–504.

Hunt, M. ( 1997 ). How science takes stock: The story of meta-analysis . New York: Russell Sage Foundation.

Hunter, J. E. , & Schmidt, F. L. ( 2004 ). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.

Hunter, J. E. , Schmidt, F. L. , & Hunter, R. ( 1979 ). Differential validity of employment tests by race: A comprehensive review and analysis.   Psychological Bulletin, 86, 721–735.

Hunter, J. E. , Schmidt, F. L. , & Jackson, G. B. ( 1982 ). Meta-analysis: Cumulating research findings across studies . Beverly Hills, CA: Sage.

Lipsey, M. W. , & Wilson, D. B. ( 2001 ). Practical meta-analysis . Thousand Oaks, CA: Sage.

Maxwell, S. E. ( 2004 ). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies.   Psychological Method s, 9, 147–163.

Olkin , ( 1990 ). History and goals. In K. W. Wachter & M. L. Straf (Eds.), The future of meta analysis (pp. 3–10). New York: Russell Sage Foundation.

Pearsons, K. ( 1904 ). Report on certain enteric fever inoculation statistics.   British Medical Journa l, 3, 1243–1246.

Rosenthal, R. ( 1979 ). The “file drawer problem” and tolerance for null results.   Psychological Bulletin, 86, 638–641.

Rosenthal, R. ( 1984 ). Meta-analytic procedures for social research . Beverly Hills, CA: Sage.

Rosenthal, R. ( 1991 ). Meta-analytic procedures for social research (revised edition). Newbury Park, CA: Sage.

Rosenthal, R. , Rosnow, R. L. , & Rubin, D. B. ( 2000 ). Contrasts and effect sizes in behavioral research: A correlational approach . New York: Cambridge University Press.

Rosenthal, R. & Rubin, D. ( 1978 ). Interpersonal expectancy effects: The first 345 studies.   Behavioral and Brain Science s, 3, 377–415.

Rothstein, H. R. , Sutton, A. J. , & Borenstein, M. (Eds.) ( 2005 ). Publication bias in meta analysis: Prevention, assessment and adjustments . Hoboken, NJ: Wiley.

Schmidt, F. L. ( 1992 ). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.   American Psychologist, 47, 1173–1181.

Schmidt, F. L. , & Hunter, J. E. ( 1977 ). Development of a general solution to the problem of validity generalization.   Journal of Applied Psychology, 62, 529–540.

Smith, M. L. & Glass, G.V. ( 1977 ). Meta-analysis of psychotherapy outcome studies.   American Psychologist, 32, 752–760.

White, H. D. ( 2009 ). Scientific communication and literature retrieval. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 51–71). New York: Russell Sage Foundation.

Wilkinson, L. , & The Task Force on Statistical Significance ( 1999 ). Statistical methods in psychology journals: Guidelines and explanations.   American Psychologist, 54, 594–604.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• How it works

Meta-Analysis – Guide with Definition, Steps & Examples

Published by Owen Ingram at April 26th, 2023 , Revised On April 26, 2023

“A meta-analysis is a formal, epidemiological, quantitative study design that uses statistical methods to generalise the findings of the selected independent studies. “

Meta-analysis and systematic review are the two most authentic strategies in research. When researchers start looking for the best available evidence concerning their research work, they are advised to begin from the top of the evidence pyramid. The evidence available in the form of meta-analysis or systematic reviews addressing important questions is significant in academics because it informs decision-making.

What is Meta-Analysis  

Meta-analysis estimates the absolute effect of individual independent research studies by systematically synthesising or merging the results. Meta-analysis isn’t only about achieving a wider population by combining several smaller studies. It involves systematic methods to evaluate the inconsistencies in participants, variability (also known as heterogeneity), and findings to check how sensitive their findings are to the selected systematic review protocol.   

When Should you Conduct a Meta-Analysis?

Meta-analysis has become a widely-used research method in medical sciences and other fields of work for several reasons. The technique involves summarising the results of independent systematic review studies. 

The Cochrane Handbook explains that “an important step in a systematic review is the thoughtful consideration of whether it is appropriate to combine the numerical results of all, or perhaps some, of the studies. Such a meta-analysis yields an overall statistic (together with its confidence interval) that summarizes the effectiveness of an experimental intervention compared with a comparator intervention” (section 10.2).

A researcher or a practitioner should choose meta-analysis when the following outcomes are desirable. 

For generating new hypotheses or ending controversies resulting from different research studies. Quantifying and evaluating the variable results and identifying the extent of conflict in literature through meta-analysis is possible. 

To find research gaps left unfilled and address questions not posed by individual studies. Primary research studies involve specific types of participants and interventions. A review of these studies with variable characteristics and methodologies can allow the researcher to gauge the consistency of findings across a wider range of participants and interventions. With the help of meta-analysis, the reasons for differences in the effect can also be explored. 

To provide convincing evidence. Estimating the effects with a larger sample size and interventions can provide convincing evidence. Many academic studies are based on a very small dataset, so the estimated intervention effects in isolation are not fully reliable.

Elements of a Meta-Analysis

Deeks et al. (2019), Haidilch (2010), and Grant & Booth (2009) explored the characteristics, strengths, and weaknesses of conducting the meta-analysis. They are briefly explained below. 

Characteristics: 

• A systematic review must be completed before conducting the meta-analysis because it provides a summary of the findings of the individual studies synthesised. 
• You can only conduct a meta-analysis by synthesising studies in a systematic review. 
• The studies selected for statistical analysis for the purpose of meta-analysis should be similar in terms of comparison, intervention, and population. 

Strengths: 

• A meta-analysis takes place after the systematic review. The end product is a comprehensive quantitative analysis that is complicated but reliable. 
• It gives more value and weightage to existing studies that do not hold practical value on their own. 
• Policy-makers and academicians cannot base their decisions on individual research studies. Meta-analysis provides them with a complex and solid analysis of evidence to make informed decisions. 

Criticisms: 

• The meta-analysis uses studies exploring similar topics. Finding similar studies for the meta-analysis can be challenging.
• When and if biases in the individual studies or those related to reporting and specific research methodologies are involved, the meta-analysis results could be misleading.

Steps of Conducting the Meta-Analysis 

The process of conducting the meta-analysis has remained a topic of debate among researchers and scientists. However, the following 5-step process is widely accepted. 

Step 1: Research Question

The first step in conducting clinical research involves identifying a research question and proposing a hypothesis . The potential clinical significance of the research question is then explained, and the study design and analytical plan are justified.

Step 2: Systematic Review 

The purpose of a systematic review (SR) is to address a research question by identifying all relevant studies that meet the required quality standards for inclusion. While established journals typically serve as the primary source for identified studies, it is important to also consider unpublished data to avoid publication bias or the exclusion of studies with negative results.

While some meta-analyses may limit their focus to randomized controlled trials (RCTs) for the sake of obtaining the highest quality evidence, other experimental and quasi-experimental studies may be included if they meet the specific inclusion/exclusion criteria established for the review.

Step 3: Data Extraction

After selecting studies for the meta-analysis, researchers extract summary data or outcomes, as well as sample sizes and measures of data variability for both intervention and control groups. The choice of outcome measures depends on the research question and the type of study, and may include numerical or categorical measures.

For instance, numerical means may be used to report differences in scores on a questionnaire or changes in a measurement, such as blood pressure. In contrast, risk measures like odds ratios (OR) or relative risks (RR) are typically used to report differences in the probability of belonging to one category or another, such as vaginal birth versus cesarean birth.

Step 4: Standardisation and Weighting Studies

After gathering all the required data, the fourth step involves computing suitable summary measures from each study for further examination. These measures are typically referred to as Effect Sizes and indicate the difference in average scores between the control and intervention groups. For instance, it could be the variation in blood pressure changes between study participants who used drug X and those who used a placebo.

Since the units of measurement often differ across the included studies, standardization is necessary to create comparable effect size estimates. Standardization is accomplished by determining, for each study, the average score for the intervention group, subtracting the average score for the control group, and dividing the result by the relevant measure of variability in that dataset.

In some cases, the results of certain studies must carry more significance than others. Larger studies, as measured by their sample sizes, are deemed to produce more precise estimates of effect size than smaller studies. Additionally, studies with less variability in data, such as smaller standard deviation or narrower confidence intervals, are typically regarded as higher quality in study design. A weighting statistic that aims to incorporate both of these factors, known as inverse variance, is commonly employed.

Step 5: Absolute Effect Estimation

The ultimate step in conducting a meta-analysis is to choose and utilize an appropriate model for comparing Effect Sizes among diverse studies. Two popular models for this purpose are the Fixed Effects and Random Effects models. The Fixed Effects model relies on the premise that each study is evaluating a common treatment effect, implying that all studies would have estimated the same Effect Size if sample variability were equal across all studies.

Conversely, the Random Effects model posits that the true treatment effects in individual studies may vary from each other, and endeavors to consider this additional source of interstudy variation in Effect Sizes. The existence and magnitude of this latter variability is usually evaluated within the meta-analysis through a test for ‘heterogeneity.’

Forest Plot

The results of a meta-analysis are often visually presented using a “Forest Plot”. This type of plot displays, for each study, included in the analysis, a horizontal line that indicates the standardized Effect Size estimate and 95% confidence interval for the risk ratio used. Figure A provides an example of a hypothetical Forest Plot in which drug X reduces the risk of death in all three studies.

However, the first study was larger than the other two, and as a result, the estimates for the smaller studies were not statistically significant. This is indicated by the lines emanating from their boxes, including the value of 1. The size of the boxes represents the relative weights assigned to each study by the meta-analysis. The combined estimate of the drug’s effect, represented by the diamond, provides a more precise estimate of the drug’s effect, with the diamond indicating both the combined risk ratio estimate and the 95% confidence interval limits.

Figure-A: Hypothetical Forest Plot

Relevance to Practice and Research 

  Evidence Based Nursing commentaries often include recently published systematic reviews and meta-analyses, as they can provide new insights and strengthen recommendations for effective healthcare practices. Additionally, they can identify gaps or limitations in current evidence and guide future research directions.

The quality of the data available for synthesis is a critical factor in the strength of conclusions drawn from meta-analyses, and this is influenced by the quality of individual studies and the systematic review itself. However, meta-analysis cannot overcome issues related to underpowered or poorly designed studies.

Therefore, clinicians may still encounter situations where the evidence is weak or uncertain, and where higher-quality research is required to improve clinical decision-making. While such findings can be frustrating, they remain important for informing practice and highlighting the need for further research to fill gaps in the evidence base.

Methods and Assumptions in Meta-Analysis 

Ensuring the credibility of findings is imperative in all types of research, including meta-analyses. To validate the outcomes of a meta-analysis, the researcher must confirm that the research techniques used were accurate in measuring the intended variables. Typically, researchers establish the validity of a meta-analysis by testing the outcomes for homogeneity or the degree of similarity between the results of the combined studies.

Homogeneity is preferred in meta-analyses as it allows the data to be combined without needing adjustments to suit the study’s requirements. To determine homogeneity, researchers assess heterogeneity, the opposite of homogeneity. Two widely used statistical methods for evaluating heterogeneity in research results are Cochran’s-Q and I-Square, also known as I-2 Index.

Difference Between Meta-Analysis and Systematic Reviews

Meta-analysis and systematic reviews are both research methods used to synthesise evidence from multiple studies on a particular topic. However, there are some key differences between the two.

Systematic reviews involve a comprehensive and structured approach to identifying, selecting, and critically appraising all available evidence relevant to a specific research question. This process involves searching multiple databases, screening the identified studies for relevance and quality, and summarizing the findings in a narrative report.

Meta-analysis, on the other hand, involves using statistical methods to combine and analyze the data from multiple studies, with the aim of producing a quantitative summary of the overall effect size. Meta-analysis requires the studies to be similar enough in terms of their design, methodology, and outcome measures to allow for meaningful comparison and analysis.

Therefore, systematic reviews are broader in scope and summarize the findings of all studies on a topic, while meta-analyses are more focused on producing a quantitative estimate of the effect size of an intervention across multiple studies that meet certain criteria. In some cases, a systematic review may be conducted without a meta-analysis if the studies are too diverse or the quality of the data is not sufficient to allow for statistical pooling.

Software Packages For Meta-Analysis

Meta-analysis can be done through software packages, including free and paid options. One of the most commonly used software packages for meta-analysis is RevMan by the Cochrane Collaboration.

Assessing the Quality of Meta-Analysis 

Assessing the quality of a meta-analysis involves evaluating the methods used to conduct the analysis and the quality of the studies included. Here are some key factors to consider:

• Study selection: The studies included in the meta-analysis should be relevant to the research question and meet predetermined criteria for quality.
• Search strategy: The search strategy should be comprehensive and transparent, including databases and search terms used to identify relevant studies.
• Study quality assessment: The quality of included studies should be assessed using appropriate tools, and this assessment should be reported in the meta-analysis.
• Data extraction: The data extraction process should be systematic and clearly reported, including any discrepancies that arose.
• Analysis methods: The meta-analysis should use appropriate statistical methods to combine the results of the included studies, and these methods should be transparently reported.
• Publication bias: The potential for publication bias should be assessed and reported in the meta-analysis, including any efforts to identify and include unpublished studies.
• Interpretation of results: The results should be interpreted in the context of the study limitations and the overall quality of the evidence.
• Sensitivity analysis: Sensitivity analysis should be conducted to evaluate the impact of study quality, inclusion criteria, and other factors on the overall results.

Overall, a high-quality meta-analysis should be transparent in its methods and clearly report the included studies’ limitations and the evidence’s overall quality.

Hire an Expert Writer

Orders completed by our expert writers are

• Formally drafted in an academic style
• Free Amendments and 100% Plagiarism Free – or your money back!
• 100% Confidential and Timely Delivery!
• Free anti-plagiarism report
• Appreciated by thousands of clients. Check client reviews

Examples of Meta-Analysis

• STANLEY T.D. et JARRELL S.B. (1989), « Meta-regression analysis : a quantitative method of literature surveys », Journal of Economics Surveys, vol. 3, n°2, pp. 161-170.
• DATTA D.K., PINCHES G.E. et NARAYANAN V.K. (1992), « Factors influencing wealth creation from mergers and acquisitions : a meta-analysis », Strategic Management Journal, Vol. 13, pp. 67-84.
• GLASS G. (1983), « Synthesising empirical research : Meta-analysis » in S.A. Ward and L.J. Reed (Eds), Knowledge structure and use : Implications for synthesis and interpretation, Philadelphia : Temple University Press.
• WOLF F.M. (1986), Meta-analysis : Quantitative methods for research synthesis, Sage University Paper n°59.
• HUNTER J.E., SCHMIDT F.L. et JACKSON G.B. (1982), « Meta-analysis : cumulating research findings across studies », Beverly Hills, CA : Sage.

Frequently Asked Questions

What is a meta-analysis in research.

Meta-analysis is a statistical method used to combine results from multiple studies on a specific topic. By pooling data from various sources, meta-analysis can provide a more precise estimate of the effect size of a treatment or intervention and identify areas for future research.

Why is meta-analysis important?

Meta-analysis is important because it combines and summarizes results from multiple studies to provide a more precise and reliable estimate of the effect of a treatment or intervention. This helps clinicians and policymakers make evidence-based decisions and identify areas for further research.

What is an example of a meta-analysis?

A meta-analysis of studies evaluating physical exercise’s effect on depression in adults is an example. Researchers gathered data from 49 studies involving a total of 2669 participants. The studies used different types of exercise and measures of depression, which made it difficult to compare the results.

Through meta-analysis, the researchers calculated an overall effect size and determined that exercise was associated with a statistically significant reduction in depression symptoms. The study also identified that moderate-intensity aerobic exercise, performed three to five times per week, was the most effective. The meta-analysis provided a more comprehensive understanding of the impact of exercise on depression than any single study could provide.

What is the definition of meta-analysis in clinical research?

Meta-analysis in clinical research is a statistical technique that combines data from multiple independent studies on a particular topic to generate a summary or “meta” estimate of the effect of a particular intervention or exposure.

This type of analysis allows researchers to synthesise the results of multiple studies, potentially increasing the statistical power and providing more precise estimates of treatment effects. Meta-analyses are commonly used in clinical research to evaluate the effectiveness and safety of medical interventions and to inform clinical practice guidelines.

Is meta-analysis qualitative or quantitative?

Meta-analysis is a quantitative method used to combine and analyze data from multiple studies. It involves the statistical synthesis of results from individual studies to obtain a pooled estimate of the effect size of a particular intervention or treatment. Therefore, meta-analysis is considered a quantitative approach to research synthesis.

You May Also Like

Ethnography is a type of research where a researcher observes the people in their natural environment. Here is all you need to know about ethnography.

The authenticity of dissertation is largely influenced by the research method employed. Here we present the most notable research methods for dissertation.

Baffled by the concept of reliability and validity? Reliability refers to the consistency of measurement. Validity refers to the accuracy of measurement.

USEFUL LINKS

LEARNING RESOURCES

COMPANY DETAILS

• How It Works

What do meta-analysts need in primary studies? Guidelines and the SEMI checklist for facilitating cumulative knowledge

• Original Manuscript
• Open access
• Published: 16 April 2024

Cite this article

You have full access to this open access article

• Belén Fernández-Castilla   ORCID: orcid.org/0000-0002-3451-0637 1   na1 ,
• Sameh Said-Metwaly 2 , 3 , 4   na1 ,
• Rodrigo S. Kreitchmann 1 &
• Wim Van Den Noortgate 2 , 3  

482 Accesses

1 Altmetric

Explore all metrics

Meta-analysis is often recognized as the highest level of evidence due to its notable advantages. Therefore, ensuring the precision of its findings is of utmost importance. Insufficient reporting in primary studies poses challenges for meta-analysts, hindering study identification, effect size estimation, and meta-regression analyses. This manuscript provides concise guidelines for the comprehensive reporting of qualitative and quantitative aspects in primary studies. Adhering to these guidelines may help researchers enhance the quality of their studies and increase their eligibility for inclusion in future research syntheses, thereby enhancing research synthesis quality. Recommendations include incorporating relevant terms in titles and abstracts to facilitate study retrieval and reporting sufficient data for effect size calculation. Additionally, a new checklist is introduced to help applied researchers thoroughly report various aspects of their studies.

Similar content being viewed by others

What is Qualitative in Qualitative Research

Criteria for Good Qualitative Research: A Comprehensive Review

Literature reviews as independent studies: guidelines for academic practice

Avoid common mistakes on your manuscript.

Meta-analysis is a statistical technique that emerged in response to the need to combine results from studies addressing similar research questions to draw a general conclusion about the state-of-the-art of a given research topic (Glass, 1976 ). This methodology began to be implemented in the 1980s when it was uncommon for authors to make the datasets utilized in their studies freely available. The difficulty in accessing raw data led to the need to use the results reported in each study to obtain a quantitative measure of the strength of the effect of interest, namely the effect size measure (Glass et al., 1981 ; Ray & Shadish, 1996 ).

The introduction of meta-analysis as a research synthesis technique has led to several potential advantages. Meta-analyses rely on replicable, transparent, and inclusive methodology to identify relevant studies (encompassing not only peer-reviewed results but also pertinent gray literature; Rytwinski et al., 2021 ). By accumulating data from multiple studies, a meta-analysis allows for more accurate estimation of the overall effect size, maximizing the statistical power and generalizability of the effect size, assessing heterogeneity across studies and explaining it through moderator variables, answering questions not researched in individual studies, developing hypotheses for future consideration, and permitting a regular update of results using newly available data (Deeks et al., 2008 ; Egger & Smith,  1997 ; Haidich, 2010 ; Walker et al., 2008 ). For these reasons, meta-analysis is frequently considered the highest rank in the hierarchy of evidence (Cooper et al., 2019 ), implying greater trust in its results than in those of primary studies. This underscores the importance of ensuring that the results of meta-analyses are as reliable and valid as possible.

While meta-analysis is a valuable methodology, it poses a significant challenge due to the considerable time it demands. The process involves searching, screening, and extracting data from all relevant studies, calculating effect sizes and corresponding sampling variances, and carrying out statistical analyses (i.e., syntheses of effect sizes and meta-regression analyses, Cooper et al., 2019 ). Each step is time-consuming, and complications arise when relevant primary study information is not (clearly) reported. For instance, if the variables of interest are expressed in ambiguous terms in the title or abstract, meta-analysts might have difficulties retrieving that study. Also, insufficient reporting of key study characteristics (e.g., related to sample, design, or setting) hinders meta-analysts’ ability to extract relevant information and incorporate it into meta-regression analyses. Primary researchers might also fail to report quantitative information essential for meta-analysts to calculate effect sizes. Hence, proper reporting of various aspects of primary studies can facilitate more efficient work for meta-analysts, leading to thorough and rigorous research syntheses. Since primary researchers may not always be aware of the information required by a meta-analyst for integrating their study into a research synthesis, the goal of this manuscript is to offer concise instructions on reporting both qualitative and quantitative aspects of primary research. This will enable primary researchers to improve the eligibility of their studies for inclusion in future research syntheses, ultimately resulting in heightened visibility and impact within the academic community and society.

Numerous guidelines for conducting and reporting quantitative research are available and endorsed (e.g., Appelbaum et al., 2018 ). Adhering to these guidelines can enhance the overall quality of a study. However, it must be noted that improved quality does not necessarily guarantee eligibility for inclusion in a meta-analysis. Meta-analysis criteria often involve additional considerations beyond individual study quality, emphasizing factors such as data relevance and sufficiency. Therefore, meeting guidelines is a valuable step, but researchers should be mindful of the distinct requirements for meta-analytic eligibility. In addressing this issue, Chow et al. ( 2023 ) introduced guidelines, with a strong focus on open science. While our study incorporates several of their guidelines, we also introduce supplementary ones not covered by Chow et al. ( 2023 ). For instance, we emphasize the role of thorough reporting in aiding various steps of a meta-analysis, including study searching and screening, as well as effect size estimation. Additionally, while we also acknowledge the value of open data, there are instances where sharing data may not always be feasible or may not necessarily enable meta-analysts to retrieve the information needed for research synthesis. Therefore, unlike Chow et al. ( 2023 ), we place special emphasis on reporting readily available relevant statistics to streamline the meta-analyst's workflow and enhance clarity for all report users.

Through the remainder of this document, we outline the stages associated with conducting a meta-analysis, focusing particularly on those stages directly influenced by the quality of reporting in primary studies. At each stage, we highlight the essential components that need to be incorporated into primary studies to enable future meta-analyses. Furthermore, we discuss the significant role that open science practices play in incorporating a specific study into a research synthesis. Ultimately, we present the Study Eligibility for Meta-Analysis Inclusion (SEMI) checklist, offering concise and clear reporting guidelines for applied researchers to enhance the potential inclusion of their studies in a meta-analysis.

Searching and screening the literature

In general, a meta-analysis commences with a systematic literature search. Researchers select a set of keywords to search electronic databases for relevant studies. The omission of a crucial keyword may result in overlooking valuable studies in the meta-analysis (Alexander, 2020 ). The selection of these keywords thus holds substantial significance as it directly influences the number of studies retrieved and may induce bias in the meta-analytic dataset.

In an ideal scenario, meta-analysts would conduct an extensive search for the keywords throughout the full text of research papers. However, if the scope of the meta-analytic investigation or the keyword list is broad, an overwhelming quantity of potentially relevant studies may surface, and many of them may prove irrelevant. To streamline the search process, a commonly employed strategy involves restricting the keyword search to the study title and abstract, assuming that authors normally indicate the most pertinent information within these sections. In this regard, our first recommendation is that authors always clearly mention the most relevant variables under investigation and study characteristics in the study title and abstract so that their study can be easily located during the search phase (aligning with APA reporting standards, see Appelbaum et al., 2018 ).

Another approach to identifying pertinent studies involves a backward search, wherein references cited within studies are examined. Primary studies, which effectively provide a general overview of the most important literature on the topic and extend beyond the studies published or indexed in databases, serve as a valuable source of studies for meta-analysis. They contribute to the discovery of additional relevant studies that may not have been initially located through conventional database searches, preventing potential oversights in the search process.

Once the researcher has compiled a list of all potential studies, a subsequent step involves the initial screening phase. Based on predetermined inclusion criteria, the researcher (or a group of researchers) assesses the relevance of studies based on their titles and abstracts, excluding those that do not meet the criteria. To expedite the screening process, the title should be as informative as possible and an abstract should capture essential details about a study, offering an accurate record of its conduct and results within the space constraints of a journal (Appelbaum et al., 2018 ; Polanin et al., 2019 ). In cases where the title and abstract do not conclusively establish a study's relevance, a meta-analyst is compelled to delve into the full text. Thus, a clear presentation of research objectives or questions and research outcomes within the study is crucial for a swift determination of its relevance.

Coding the literature

Upon selecting the studies for inclusion in the research synthesis, the next step entails extracting the pertinent qualitative and quantitative information from each study. This information serves three primary purposes: (1) qualitatively summarizing the characteristics of the included studies, (2) quantitatively calculating the desired effect sizes, and (3) conducting moderator analyses, wherein study characteristics (referred to as moderator variables) are employed in a meta-regression model to examine their relationship with the observed effect sizes.

One common challenge in this phase is the incomplete reporting of study characteristics and/or insufficient data within studies to compute the effect size (Lee & Beretvas, 2022 ; Pigott, 2019 ; Tipton et al., 2019 ), which can lead to study exclusion from the meta-analysis (or from moderator analyses) and consequently impact statistical power. Hence, we urge researchers to follow the next guidelines and report study characteristics and outcomes in sufficient detail so that these aspects can be easily coded and used in future research synthesis.

To identify specific study characteristics relevant to future research synthesis, particularly for moderator analyses, the PICO framework (McGowan et al., 2016 ) can be employed. In this framework, P refers to participant characteristics (e.g., number, age, gender, or socioeconomic status), I refers to intervention or exposure details (e.g., experimental condition, modality, duration, or medication type), C refers to comparator characteristics (e.g., control condition such as a traditional treatment or waitlist), and O refers to outcome characteristics (e.g., a comprehensive description of dependent variables). Alternative frameworks, such as SPICE (setting, perspective, intervention, comparison, evaluation; Booth, 2006 ) and SPIDER (sample, phenomenon of interest, design, evaluation, research type; Cooke et al., 2012 ), can also be applied across various study designs.

Within all these frameworks, it is recommended to provide a comprehensive and accurate description of the sample, particularly highlighting characteristics that may impact the results. These include details such as the number of participants identifying as men and women, mean age (including standard deviation or range), region of origin, or socioeconomic status. For research synthesis purposes, authors are urged to present this information on the final analyzed sample, specifically after dropout removal, which may vary across analyses within the same study. Additionally, citing other studies utilizing the same sample or subset thereof is vital to preventing overrepresentation and ensure the unique contributions of samples in meta-analyses. Furthermore, avoiding duplication of samples in meta-analyses is essential for maintaining statistical independence among studies, which is crucial for accurate meta-analytic estimates.

These frameworks also emphasize the necessity of appropriately describing independent and dependent variable(s). For independent variables, such as interventions or experimental conditions (as seen in the PICO or SPICE framework), primary studies should include crucial information such as the modality and intensity of the intervention/experimental condition, its duration (number of sessions and session duration), and details on any administered drugs and their quantities. In correlational studies, where the independent variable is observed rather than experimentally manipulated, it is imperative to furnish information on how the independent variable is operationalized, measured (including reliability measures calculated on the observed data), and implemented. The same level of detail is essential for the dependent variables. Including these specifics not only increases the likelihood that a study can be included in a meta-analysis but also enables the assessment of its risk of bias.

Methodological characteristics are also crucial for meta-analysts to evaluate the methodological quality of primary studies (Pigott & Polalin, 2020 ). These include the specific research design (e.g., experimental, quasi-experimental, cross-sectional, or longitudinal), procedural details (e.g., where, how, and when data are collected, and the randomization of participants across groups), and specifics of the data-analytic methods (e.g., significance level, statistical tests, and whether the test is two-sided or one-sided).

In terms of reporting the methodological aspects of a study, applied researchers can utilize relevant risk-of-bias assessment tools for comprehensive reporting (refer to https://www.latitudes-network.org/ for an overview of pertinent risk-of-bias assessment tools, Whiting et al., 2023 ). For example, the widely used Risk of Bias Tool 2 (RoB2; Sterne et al., 2019 ) for assessing randomized controlled trials includes items such as evaluating the randomization process and assessing bias due to deviations from intended interventions. Authors of primary studies should accurately describe participant assignment and provide specific details on blinding and potential deviations from intended therapy. Systematically reviewing various items from diverse risk-of-bias assessment tools, available for different research designs, significantly assists applied researchers in providing necessary information for others to assess the quality of their studies.

As previously mentioned, proper reporting of numerical results is essential for calculating commonly used effect sizes. The following section provides a brief overview of the statistical outcomes required for effect size calculations.

Calculating and combining study outcomes

The next step in a meta-analysis involves calculating an index that summarizes the strength of the effect of interest targeted for meta-analysis. Commonly known as an effect size measure, it is defined as “ a quantitative reflection of the magnitude of some phenomenon that is used for the purpose of addressing a question of interest ” (Kelley & Preacher, 2012 , pp. 140). However, we do not recommend simply reporting effect sizes that address the research questions of the primary study. This is because the meta-analyst may be interested in an effect size associated with a different set of variables within that study. For example, consider a study with the aim of investigating the effectiveness of an intervention on two dependent variables: well-being and anxiety symptoms. The authors may report two Cohen’s d values summarizing the intervention's effectiveness, successfully addressing the intended effect size in that study. However, the meta-analyst might be interested in the correlation between well-being and anxiety symptoms. Since this correlation might not be the primary focus for the primary authors, it might be overlooked in their reporting, consequently leading to the exclusion of the study from meta-analysis. This exclusion can be avoided if authors are contacted to share the correlation value or if they make the dataset publicly available on an online repository, enabling meta-analysts to calculate any desired effect size related to the studied variables. Another reason why merely reporting effect sizes may not be sufficient for a study to qualify for inclusion in a meta-analysis is that for certain types of effect sizes, different formulas exist (e.g., Cohen’s d in correlated samples, more information is given in subsequent sections) that might represent different, incomparable parameters (Lakens, 2013 ). If the authors do not explicitly specify the formula they employed, the meta-analyst in question will be unable to determine whether the reported effect size is appropriate for the research synthesis.

As a result, primary studies should not only report the primary effect size relevant to their specific research question but also provide the necessary numerical information to facilitate its calculation, including its precision (i.e., sampling variance). Since primary researchers might not know which numerical information future meta-analysts will need for their studies, a significant section of this manuscript outlines guidelines regarding the specific quantitative data that should be reported. This aims to enable future meta-analysts to calculate their desired effect size, thereby facilitating the inclusion of the primary study in research synthesis.

The following sections are organized as follows: Firstly, we discuss the role of open science in research synthesis and associated barriers. Next, we attempt to unpack the information that primary investigators should provide in their papers (either in the main text or in supplementary material) to increase the likelihood that their study will be eligible for research synthesis. Although information on the calculation, reporting, and interpretation of effect sizes can be found elsewhere (e.g., Borenstein, et al., 2021 ; Cooper et al., 2019 ; Cumming, 2012 ; Durlak, 2009 ; Grissom & Kim, 2005 Lakens, 2013 ; Olejnik & Algina, 2000 ; Pek & Flora, 2018 ; Schmidt & Hunter, 2014 ; Trusty et al., 2004 ), in Table 1 we provide a summary of the formulas for calculating popular effect sizes to support the information stated below.

Open science

While comprehensive reporting is crucial for study eligibility in research synthesis as outlined in the following sections, the significance of this reporting may diminish if raw datasets are consistently accessible. If raw datasets are publicly available, meta-analysts could calculate any effect size of interest, whether the one reported in the study or any other beyond the primary study goal. Additionally, with raw data available in all studies, individual participant data meta-analyses (Riley et al., 2010 ) could be systematically performed. Hence, giving access to the datasets would undoubtedly assist meta-analysts in retrieving important data to conduct a research synthesis, namely the effect sizes and relevant information for the moderator analyses.

Despite the increasing number of journals and granting agencies mandating the sharing of collected data, the actual practice of data sharing remains relatively infrequent. Obstacles to data sharing extend beyond technical challenges. Issues such as the absence of recognition incentives for sharing research data, the absence of standardized formats for data and metadata (that offer the details necessary for other researchers to comprehend the data), privacy concerns, fear of misuse, and limited time and resources all pose potential hindrances to effective data sharing (Krumholz, 2012 ).

Even in cases of successful data sharing, it does not necessarily contribute to resolving reporting issues for meta-analysis. First, providing the dataset and the analytics code to reproduce the main results does not always ensure reproducibility (Hardwicke et al., 2018 ; Hardwicke et al., 2021 ; Obels et al., 2020 ). This is because authors may make errors in the dataset and/or code, or they may not provide the complete code necessary to reproduce all analyses. Additionally, authors may overlook the inclusion of metadata, hindering the comprehension of variables within the dataset. On top of this, the inadequate reporting of crucial study details, such as the research procedure, sample characteristics, instrument details, and research design, remains unresolved even with the availability of a publicly accessible dataset. In essence, having access to a dataset does not guarantee that meta-analysts will acquire comprehensive information from the study necessary for inclusion in meta-analysis or meta-regression analyses, especially details suitable for moderator analyses. Hence, our recommendation is not only to provide access to the dataset and code used but also to adhere to the guidelines outlined in this manuscript.

When providing public access to the dataset and analytics code, it is crucial to consider specific key factors for ensuring the success of the process (see also Obels et al., 2020 ; Wilkinson et al., 2016 ). First, ensure the public accessibility and proper functionality of the website link hosting the documents. Second, provide a comprehensive codebook that clearly explains the coding for each variable. Third, include explanatory comments in the analytical code to guide fellow researchers through its execution. Finally, to overcome interoperability challenges and to ensure compatibility across different statistical software packages and versions, store data in universally readable formats such as .ASCII, .CSV, and .TXT. For comprehensive guidance on the process of data sharing, please refer to the step-by-step guide provided by Logan et al. ( 2021 ). This resource offers detailed insights and instructions to help one effectively navigate the various stages of sharing data.

Univariate statistics of the whole sample

Descriptive summary statistics (e.g., sample sizes, means, standard deviations, frequencies, and proportions) are crucial for accurately describing the variables under study and for calculating the most relevant effect sizes, including standardized mean differences, risk ratios, and odds ratios (see Table 1 ). It is important to highlight the necessity of providing this information for the final sample of participants after excluding dropouts. In longitudinal studies, providing descriptive statistics for each time point is particularly vital, especially in instances where participants were absent, or data were missing.

When studying qualitative categorical variables, such as dichotomous, nominal, or ordinal variables (e.g., socioeconomic status, type of stimuli, or type of task), frequencies and proportions should be reported for each category of the qualitative variable, regardless of whether it is an independent or dependent variable. For instance, in studies on inattentional blindness—where individuals may fail to notice unexpected stimuli in their visual field due to focused attention on a different task or stimulus—the typical dependent variable is whether individuals notice an object unexpectedly introduced by the researcher in the task (e.g., Wiemer, et al., 2013 ), and authors should report the number and proportion of individuals who noticed the unexpected objects and those who did not.

Moving on to quantitative variables (e.g., age, income, or test scores), the descriptive statistics to be reported are means and standard deviations. Footnote 1 For instance, Harris ( 2004 ) examined the relationship between intelligence, achievement, openness to experience, and creativity. All these variables were quantitatively measured, and their means and standard deviations are appropriately presented in a table. Harris ( 2004 ) did not specify whether there was missing data, leading to the assumption that all variables are based on the complete sample. Ideally, it should be explicitly mentioned that no data were missing, or the sample size for each variable could have been indicated. Another instance is the study conducted by Goecke et al. ( 2020 ), where they investigated conflicting assertions regarding the overclaiming phenomenon (i.e., the inclination of individuals to overrate both their general cognitive abilities and their specific knowledge). The researchers measured various quantitative variables, including overclaiming, self-reported knowledge, and crystallized intelligence, and detailed their means, standard deviations, and corresponding sample sizes in a table. Notably, they provided precise information about the sample for each variable, with slight variations in sample sizes due to missing data. This meticulous reporting enables a future meta-analyst to discern the exact sample for each of these measures.

Descriptive statistics for the relationship between variables

When examining the relationship between variables, it is important to report the descriptive information associated with this relationship because this is the information commonly used by meta-analysts to calculate effect sizes. In the following subsections, we disaggregate this information by the types of variables involved in the relationship.

Relationship between categorical variables

The numerical information required for studying the relationship between categorical variables depends on the type of categorical variables under investigation. When studying the relationship between two dichotomous or nominal variables, it is imperative to present a cross-tabulation with disaggregated frequencies. Such cross-tabulation provides the necessary information to calculate effect sizes, such as odds ratios and risk ratios, which are commonly used in meta-analyses of categorical data. For instance, consider a study investigating the association between smoking status (smoker vs. non-smoker) and the presence of lung cancer (yes vs. no). A cross-tabulation of these variables would display the frequencies of individuals falling into each combination of categories, for instance, the number of smokers diagnosed with lung cancer, non-smokers diagnosed with lung cancer, smokers not diagnosed with lung cancer, and non-smokers not diagnosed with lung cancer (see, for example, Morabia & Wynder, 1991 ). This detailed breakdown is essential for meta-analysts aiming to synthesize the association between these two variables across studies.

When studying the relationship between a dichotomous or a nominal variable and an ordinal variable or between two ordinal variables, it is crucial for researchers to provide access to the dataset containing raw data. In other words, if researchers utilize ordinal variables and aim for their study to be eligible for future meta-analyses, adherence to open science practices is imperative. This is because most effect sizes applicable to ordinal variables cannot be computed solely from descriptive summary statistics. For instance, to assess the magnitude of the difference between two groups in an ordinal variable, one might calculate the delta Cliff (Cliff, 1993 ), but raw data are indispensable (see Macbeth et al., 2011 ). Similarly, the correlation between two ordinal variables can be determined using Spearman or Kendall’s tau-square correlation (Kendall, 1938 ), but once again, raw data are necessary for computation, as it involves examining concordant and discordant pairs of observations. Consequently, meta-analysts interested in effect sizes related to ordinal variables can include a particular study in their research synthesis only if the exact effect size of interest is reported or if authors have made their datasets publicly available.

Relationship between categorical and quantitative variables

When investigating the relationship between a categorical variable and a quantitative variable, means and standard deviations of the quantitative variable should be reported for each category of the categorical variable. Harris' ( 2004 ) study provides an example of how descriptive statistics for quantitative dependent variables are reported by pertinent groups. In this investigation, gender differences were examined, and a breakdown of means and standard deviations segregated by gender is provided in a table. This detailed presentation of descriptive statistics for relevant subgroups, such as based on gender, aids future meta-analysts in computing standardized mean differences between genders across all measured variables. Especially in studies where the primary analysis involves an analysis of variance (ANOVA), it is crucial to report means, standard deviations, and sample sizes for each combination of categories of the qualitative variables used as the independent variable in the analyses. For instance, consider a two-factor ANOVA with independent variables such as socioeconomic status (low, medium, and high) and educational level (primary, high school, and university). In this case, means, standard deviations, and sample sizes should be reported for each of the 3 × 3 = 9 subgroups resulting from the combination of categories. This detailed reporting is essential as it enables meta-analysts to calculate standardized mean differences for any of the resulting subgroups. It is important to note that this descriptive information should be reported regardless of the primary researcher's specific focus, which typically revolves around the interaction between the independent variables, and it does not necessarily have to be included in the main text; it can be relocated to the supplementary materials.

Relationship between quantitative variables

Pearson correlation coefficients summarize the (linear) relationship between two quantitative variables. These coefficients are incredibly useful in meta-analysis for several reasons. Firstly, correlation coefficients serve as effect sizes that can be readily integrated into meta-analytic datasets. Second, many partial effect sizes can be calculated from correlation coefficients, such as partial- and semi-partial correlations and standardized regression coefficients (Aloe & Becker, 2009 , 2012 ; Becker, 1992 ; Fernández-Castilla et al., 2019 ). However, if researchers only report the results of multiple regression models (i.e., unstandardized or standardized regression coefficients), correlation coefficients cannot be back-calculated, Footnote 2 and this is a reason why many primary studies are often discarded for meta-analysis. Although a procedure to convert regression coefficients to correlations has been proposed (Peterson & Brown, 2005 ), it does not work correctly in many scenarios (Aloe, 2015 ). Hence, simply reporting correlations among quantitative variables enables the calculation of many effect sizes that might be of interest to meta-analysts.

A third reason why correlation coefficients should always be reported is that, to implement multivariate meta-analytic models, the correlation between the raw scores of the variables of interest is needed. For instance, imagine that a meta-analyst is interested in synthesizing standardized mean differences that reflect the effectiveness of a given psychological intervention in reducing both anxiety and depressive symptoms, and that most studies report these two results. Since there are two correlated dependent variables within studies (anxiety and depression), a multivariate meta-analysis would have to be carried out to synthesize these effect sizes (Becker, 2000 ; Kalaian & Raudenbush 1996 ). To conduct this type of analysis, the covariance between the standardized mean differences reported in the same study (presumably one for depression and one for anxiety) needs to be estimated in advance (see Hedges & Olkin, 1985 ), and to calculate it, information on the correlation between the raw depression and anxiety scores is needed. By reporting the correlation coefficients between all quantitative variables, future meta-analysts will be able to retrieve this information to apply more sophisticated statistical methods, eventually leading to more precise meta-analytic estimates.

A final reason why reporting correlation coefficients is important is that new methods have been developed in the field of meta-analysis, such as meta-analytic structural equation modeling (also known as MASEM, Cheung, 2015 ; Jak, 2015 ; or one-stage MASEM [OSMASEM], Jak & Cheung, 2020 ). This methodology allows one to perform meta-analysis of more complex structural equation models, including mediation models (e.g., Ng et al., 2023 ), path analyses (Smith et al., 2022 ), or confirmatory factor analyses (Said-Metwaly et al., 2018 ). The input required for conducting MASEM is the correlations between the variables of interest organized in a correlation matrix. By reporting all the possible correlations of one’s dataset in a correlation matrix, meta-analysts performing MASEM could easily include all the correlations between their variables of interest.

Intraclass correlation coefficient and variance estimates in cluster-randomized studies

In primary research within the realms of psychology and educational sciences, it is commonplace to encounter hierarchical structures wherein observations are nested within higher-level clusters. Examples include students nested within classrooms or observations nested within participants in repeated measures designs. This hierarchical structure necessitates consideration not only during data analysis but also in the calculation of certain effect sizes, such as the standardized mean difference (Hedges, 2007 ; Snijders, 2005 ).

Consider, for instance, a scenario where two groups of participants from distinct experimental conditions are compared (level 1), and these participants are further grouped into different centers, forming the cluster at level 2. When calculating the standardized mean difference that compares means across these experimental conditions, it becomes imperative to acknowledge that participants are nested within different clusters (centers in this case). Consequently, participants belonging to the same center are expected to exhibit greater similarity than those from different centers.

There is no singular formula for calculating a standardized mean difference for clustered designs. The mean difference between groups may be standardized by the square root of the pooled within-cluster variance, the between-clusters variance, or the total variance, representing the sum of the two variances. Therefore, to facilitate the calculation of any of these versions of effect sizes, a meta-analyst must possess information on (1) the mean of the two compared groups, (2) the between-clusters variance, and (3) the within-cluster variance. These sources of variability can also be estimated from each other if the intraclass correlation coefficient is available. This coefficient signifies the correlation between observations within the same cluster, and the relevant formulas can be found in Borenstein and Hedges ( 2019 ). The intraclass correlation coefficient, coupled with the total sample size and average cluster size, is also essential for calculating the sampling variances of these effect sizes. Consequently, it is of utmost importance to thoroughly report all this information in studies employing such designs.

Pearson correlations between repeated measures

In meta-analysis, it is often of interest to include data from matched group experimental designs meant to test the effectiveness of an independent variable (e.g., intervention, program, or experimental condition). Typically, in each study, standardized mean differences for repeated measures (see \({g}_{rm(1)}\) in Table 1 ) or standardized mean changes (see \({g}_{igpp(1)}\) in Table 1 ) are calculated for posterior synthesis. Importantly, the formulas for these effect sizes incorporate the correlation between pre- and post-measures. Specifically, this correlation is essential for determining the standard deviation of the difference ( \({S}_{within})\) , which serves as the denominator in the formula for computing the standardized mean difference for repeated-measures designs ( \({g}_{rm(1)}\) in Table 1 ). Furthermore, this correlation between pre- and post-test scores is necessary for calculating the sampling variance of this effect size (see Morris and DeShon, 2002 ). Similarly, to calculate a standardized mean change (i.e., the standardized difference in the extent of change within one group relative to the change observed in another group, see \({g}_{igpp(1)}\) in Table 1 ), the correlation between pre- and post-measures within each involved group is also required.

Since authors seldom report this correlation, formulas have been proposed to circumvent its inclusion in the calculation of these effect sizes (see, for instance, Becker, 1988 ; see formula for \({g}_{rm(2)}\) and \({g}_{igpp(2)}\) in Table 1 ). However, this pre/post score correlation is still essential for computing the sampling variance of these effect sizes (see \({S}_{{g}_{rm(2)}}^{2}\) and \({S}_{{g}_{igpp(2)}}^{2}\) in Table 1 ). Therefore, when utilizing standardized mean differences or standardized mean changes in meta-analysis, the correlation between pre- and post-measures often needs to be estimated or imputed. Hence, we strongly encourage primary researchers to incorporate this correlation in their reports, along with any other pertinent descriptive information.

Reliability of the measurements

Reliability is commonly defined as the proportion of true score variance to total score variance (Novick, 1966 ). Reliability coefficients provide information on the precision of scores from psychological assessments. In psychological science, measurements frequently contain non-negligible degrees of error. For instance, self-reported outcomes may include nuisance related to the distortions in individuals’ self-perception or understanding of the response scale. These measurement errors are generally assumed to be random variations that cause scores to deviate from their true values.

Although often disregarded, the results of a primary study containing psychological assessments are largely influenced by measurement reliability. As an example, in the relationship between general intelligence and job performance, if both measures are precise (e.g., obtained using long questionnaires), the estimated regression/correlation coefficients are likely to approximate the true relationship between these constructs. On the other hand, if measurement reliability is low (e.g., using fewer or more imprecise questions), the coefficients between variables may be largely underestimated. To illustrate, a correlation of 0.51 between intelligence and job performance (e.g., Schmidt & Hunter, 2004 ) could be substantially reduced, to approximately 0.36, if both measurements have reliability coefficients of around 0.70.

Meta-analytic studies are often aimed at summarizing generalized coefficients for the relationships between constructs beyond one specific sample. Correcting these underestimated regression/correlation coefficients relies on the reliability indices reported in primary studies. Authors are encouraged to report reliability coefficients (e.g., Cronbach's ɑ or McDonald's ⍵) of their measurements. Finally, it is important to note that the reliability reported in an instrument’s manual or in the original validation studies may not precisely match reliability in empirical studies. Due to range restrictions of the scores and additional noise due to various random factors, the reliability in an empirical study can differ from the one in the original validation study. Hence, authors of primary studies are encouraged to report the reliability of measurements obtained in their datasets. This not only enhances their reporting but also makes their studies eligible for future reliability generalization meta-analysis.

Negative results

Researchers may conduct a study and find an effect that is either statistically nonsignificant or contradicts a hypothesis, referred to as a negative finding. Negative findings face a greater publication challenge than their positive counterparts (Fanelli, 2010 ; Franco et al., 2014 ). Researchers may fuel publication bias by selectively reporting positive findings or refraining from submitting studies with negative findings. This behavior is often driven by the anticipation of low acceptance rates, or the fear of professional consequences associated with publishing findings that challenge well-confirmed hypotheses or theories (Shields, 2000 ; Therrien & Cook, 2018 ). Journal editors and reviewers may also contribute to publication bias by rejecting submissions with null findings.

Publication bias has been observed in various fields, including medicine, social sciences, and psychology, indicating a widespread phenomenon (Therrien & Cook, 2018 ). Publication bias may inflate the estimates of relationships between variables and treatment effects in meta-analyses. The inclusion of even a few unpublished findings could substantially influence conclusions drawn from the literature (Howard et al., 2009 ; Polanin et al., 2016 ). Publication bias distorts scientific literature, leading to the formulation of hypotheses or taking decisions in practice based on inaccurate information, wasting research opportunities and funding and violating an implicit contract with funders (Shields, 2000 ). Moreover, when negative findings go unpublished, researchers may expend resources conducting studies that have already proven unsuccessful (Fanelli, 2012 ). The potential bias in the literature, however, is not the only problem with not reporting findings. We also have an ethical responsibility to our study participants who invest their valuable time and resources, trusting that their contributions benefit others. Failure to publish study findings violates this trust and may be deemed scientific misconduct (Chalmers & Moher, 1993 ; Shields, 2000 ). Additionally, we owe transparency to donors and taxpayers who support our research.

To encourage the publication of negative findings, it is crucial to recognize the value of negative results on par with positive ones. Acknowledging that understanding the absence of an effect holds equal importance to identifying its presence is essential (Fox & Kaufman, 2018 ). Instead of planning studies solely to determine "what works," a shift to planning studies to understand "how to make things work better" allows for useful insights from positive or negative findings (Jacob et al., 2019 ). By shifting our perspective and acknowledging the importance of negative findings, we contribute to a more balanced and comprehensive scientific literature, fostering a culture that appreciates the diverse outcomes of rigorous research efforts.

Recognizing their significance, initiatives have been undertaken to improve the visibility of negative findings in scientific literature through diverse approaches. For instance, certain journals have been initiated exclusively dedicated to publishing negative findings, such as the Journal of Negative Results , Journal of Negative Results in BioMedicine , Journal of Pharmaceutical Negative Results , Nature's Negative Results section, and Positively Negative (PLOS One) . In addition, mainstream journals have allocated special issues specifically for null findings (see, for instance, Landis et al., 2014 ; Therrien & Cook, 2018 ). However, this approach may inadvertently introduce bias favoring negative outcomes (Mlinarić et al., 2017 ). Publishing criteria should thus prioritize study quality and statistical power, irrespective of the direction and significance of the results.

Journal editors and reviewers can also play a pivotal role in shaping positive attitudes and behaviors regarding negative findings. For instance, editors can explicitly express in the author guidelines the openness to publish well-designed studies with null findings (Hubbard & Armstrong, 1992 ). Editors can also promote or mandate registered reports, where study plans are submitted for pre-publication review based on research design. If accepted, the study is published regardless of the reported findings, minimizing the likelihood of result-driven deviations or studies being overlooked in file drawers (Cook & Therrien, 2017 ). Moreover, during the revision process, editors and reviewers commonly ask for the removal of information that is deemed nonessential, frequently tied to negative findings. While brevity is important, we should not sacrifice information. Unless entirely unrelated to the primary research question, it is advisable to report findings regardless of their direction, even if placed in supplementary material—thus, providing more information is generally preferable (Landis et al., 2014 ). Such practices could address publication bias by directly publishing more studies with negative findings and indirectly affirming their value and publishability, encouraging researchers to submit rather than keep them in a file drawer (Cook & Therrien, 2017 ).

Study Eligibility for Meta-Analysis Inclusion (SEMI) checklist

Many reporting guidelines have been provided for studies of different fields: the STROMS checklist for research on human microbiome (Mirzayi, et al., 2021 ), the AGREE Reporting Checklist for clinical research (Brouwers et al., 2016 ), and the CROSS checklist for survey studies (Sharma et al., 2021 ). In this direction, some interesting initiatives have emerged, such as the EQUATOR Network (Altman et al. 2008 ), which brings together different resources and checklists that aim to improve the accuracy of the reporting and the quality of the research. There are also well-known reporting guidelines developed to properly report information in meta-analyses and systematic reviews (the PRISMA statement, Page et al., 2021 ; the REGEMA checklist, Sánchez-Meca et al., 2021 in reliability generalization meta‐analyses). However, there is currently a lack of reporting guidelines specifically aimed at enhancing the odds of a study being retrieved and being eligible for a meta-analysis, and that is the gap aimed to be filled with this manuscript.

In this section, we provide the SEMI checklist to supplement extant reporting guidelines in the hope of improving the completeness of information in primary empirical reports and thus optimizing for inclusion in future meta-analyses. The SEMI checklist may be used in conjunction with other checklists assessing basic reporting prerequisites (e.g., PRISMA, STROBE, and CONSORT), aiming to maximize the quality of reporting practices and facilitate accumulated meta-analytic knowledge.

The development of the checklist items was informed by existing reporting guidelines, our own experience in meta-analysis research, and consultation with expert researchers in the field. It was also guided by the PICO model, which is frequently used for planning literature search and study selection in research synthesis (McGowan et al., 2016 ).

Initially, The SEMI checklist involved 30 items, followed by a “yes/no/not applicable” judgment, covering five key parts of a paper: title and abstract, background, methods, results, and open science. We incorporated items related to study title and abstract to encourage researchers to consider reporting information that support meta-analysts to retrieve the study in database searching and to conduct title/abstract screening. We also incorporated items related to a study background to facilitate locating relevant studies via backward reference searching. In the Methods section, we present elements pertaining to the accurate reporting of study characteristics, crucial for the subsequent execution of moderator analyses in meta-analysis. Next, in the Results section, our focus is on elements related to the proper reporting of numerical information, essential for calculating effect sizes. We also incorporated items to prompt researchers to report results in a sequence that mirrors the description of analyses outlined in the Methods section and to ensure coherence between the textual results with those displayed in the tables and figures. This can help mitigate ambiguity and potential misinterpretation, offering meta-analysts a clear roadmap to navigate the study's design, methods, and results without unnecessary confusion. Finally, we also include some items related to open science practices. This initial version of the checklist underwent review by four external methodological experts in the field of meta-analysis, some with more than two decades of experience in meta-analysis, who provided valuable feedback to refine the tool. Incorporating expert opinions, we revised existing items and introduced new ones, resulting in a final set of 28 items.

We make the SEMI available in Table 2 for the research community and will register it on the EQUATOR website to enhance dissemination. We recommend journals and publishers endorse the use of the SEMI by referring to it in their instructions to authors and consider utilizing it in their review process.

Meta-analysis has emerged as a powerful tool for consolidating scientific knowledge and informing decision-making. However, the accurate execution of various stages of a meta-analysis may be hindered by the inaccurate reporting of information in primary research studies. If studies cannot be found or if effect sizes cannot be computed, they will be excluded from the research synthesis, ultimately impacting the statistical power to detect a significant overall effect or even inducing bias. Likewise, if the characteristics of the studies cannot be effectively encoded, there will be missing information in the moderator analysis, which in turn will affect the analytical power (Pigott, 2019 ). Although imputation techniques exist to prevent this problem (e.g., Lee & Beretvas, 2022 ), no technique will yield the same accurate estimates as having all the data available for the analyses.

For this reason, we have introduced the SEMI checklist, which can be utilized to assess the suitability of a study for inclusion in future meta-analyses. To the best of our knowledge, this checklist represents one of the first endeavors to improve the reporting quality of primary studies, with a specific focus on their potential inclusion in a meta-analysis. In a similar vein, Chow et al. ( 2023 ) have offered valuable recommendations for reporting specific elements of studies, such as procedures, results, and open access practices. Our checklist broadens the scope of Chow’s checklist to include additional critical elements. This encompasses aspects such as the study's title, abstract, background, sample characteristics, and other results essential for calculating various effect sizes in meta-analysis, thereby ensuring a more comprehensive reporting framework.

Hopefully, the use of the SEMI checklist and the Chow et al. ( 2023 ) guidelines can assist authors in describing the conducted research in sufficient detail, assist editors and reviewers in evaluating the comprehensiveness of reports submitted for publication, and ultimately maximize the use of research results in the quantitative synthesis. We believe that adhering to the suggested checklist can substantially enhance the reporting standard of primary studies. This, in turn, will ultimately contribute to conducting more precise and reliable meta-analyses.

Common analyses (e.g., regression analyses) assume that the residual scores on the outcome variable are normally distributed. A violation of this assumption or the existence of outliers may make means and standard deviations less informative. Therefore, information on the distribution of the scores is also required.

An exception would be the method presented by Fernández-Castilla et al. (2019), where correlation coefficients can be estimated from standardized regression coefficients under some scenarios.

Alexander, P. A. (2020). Methodological guidance paper: The art and science of quality systematic review. Review of Educational Research, 90 , 6–23. https://doi.org/10.3102/0034654319854352

Article   Google Scholar  

Aloe, A. M. (2015). Inaccuracy of regression results in replacing bivariate correlations. Research Synthesis Methods, 6 , 21–27. https://doi.org/10.1002/jrsm.1126

Article   PubMed   Google Scholar  

Aloe, A. M., & Becker, B. J. (2009). Teacher verbal ability and school outcomes: Where is the evidence? Educational Researcher, 38 (8), 612–624. https://doi.org/10.3102/0013189X09353939

Aloe, A. M., & Becker, B. J. (2012). An effect size for regression predictors in meta-analysis. Journal of Educational and Behavioral Statistics, 37 , 278–297. https://doi.org/10.3102/1076998610396901

Altman, D. G., Simera, I., Hoey, J., Moher, D., & Schulz, K. (2008). EQUATOR: Reporting guidelines for health research. The Lancet, 371 (9619), 1149–1150. https://doi.org/10.1016/S0140-6736(08)60505-X

Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73 , 3–25. https://doi.org/10.1037/amp0000191

Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41 , 257–278. https://doi.org/10.1111/j.2044-8317.1988.tb00901.x

Becker, B. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational and Behavioral Statistics, 17 , 341–362. https://doi.org/10.3102/10769986017004341

Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of Applied Multivariate Statistics and Mathematical Modeling (pp. 499–525). Academic Press.

Chapter   Google Scholar  

Booth, A. (2006). Clear and present questions: Formulating questions for evidence-based practice. Library Hi Tech, 24 , 355–368. https://doi.org/10.1108/07378830610692127

Borenstein, M., & Hedges, L. V. (2019). Effect sizes for meta-analysis. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), Handbook of Research Synthesis and Meta-Analysis (3rd ed., pp. 207–243). Wiley.

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2021). Introduction to meta-analysis . John Wiley & Sons. https://doi.org/10.1002/9780470743386

Book   Google Scholar  

Brouwers, M. C., Kerkvliet, K., Spithoff, K., & AGREE Next Steps Consortium. (2016). The AGREE Reporting checklist: A tool to improve reporting of clinical practice guidelines.  British Medical Journal , i1152. https://doi.org/10.1136/bmj.i1152

Chalmers, I., & Moher, D. (1993). Publication bias. Lancet, 342 , 1116. https://doi.org/10.1016/0140-6736(93)92099-F

Cheung, M.W.-L. (2015). Meta-Analysis: A structural equation modeling approach . John Wiley & Sons.

Chow, J. C., Sandbank, M., & Hampton, L. H. (2023). Guidance for increasing primary study inclusion and the usability of data in meta-analysis: A reporting tutorial. Journal of Speech, Language, and Hearing Research , 1-9. https://doi.org/10.1044/2023_JSLHR-22-00318

Cliff, N. (1993). Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114 , 494–509. https://doi.org/10.1037/0033-2909.114.3.494

Cook, B. G., & Therrien, W. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders, 42 , 149–158. https://doi.org/10.1177/0198742917709473

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22 , 1435–1443. https://doi.org/10.1177/1049732312452938

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The Handbook of Research Synthesis and Meta-Analysis . Russell Sage Foundation. https://doi.org/10.7758/9781610448864

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16 , 297–334. https://doi.org/10.1007/BF02310555

Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis . Routledge.

Google Scholar  

Deeks, J. J., Higgins, J. P. T., & Altman, D. G. (2008). Analyzing data and undertaking meta-analyses. In I. P. T. Higgins & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 243–296). John Wiley & Sons.

Durlak, J. A. (2009). How to select, calculate, and interpret effect sizes. Journal of Pediatric Psychology, 34 , 917–928. https://doi.org/10.1093/jpepsy/jsp004

Egger, M., & Smith, G. D. (1997). Meta-analysis: Potentials and promise. British Medical Journal, 315 , 1371–1374. https://doi.org/10.1136/bmj.315.7119.1371

Article   PubMed   PubMed Central   Google Scholar  

Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PloS One, 5 , e10068. https://doi.org/10.1371/journal.pone.0010068

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90 , 891–904. https://doi.org/10.1007/s11192-011-0494-7

Fernández-Castilla, B., Aloe, A. M., Declercq, L., Jamshidi, L., Onghena, P., Natasha Beretvas, S., & Van den Noortgate, W. (2019). Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients. Behavior Research Methods, 51 , 316–331. https://doi.org/10.3758/s13428-018-1123-7

Fox, M. P., & Kaufman, J. S. (2018). The WelTel Trial in context and the importance of null findings. The Lancet Public Health, 3 , e107–e108. https://doi.org/10.1016/S2468-2667(18)30004-5

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345 , 1502–1505. https://doi.org/10.1126/science.1255484

Glass, G. V. (1976). Primary, Secondary, and meta-analysis of research. Educational Researcher, 5 (10), 3–8. https://doi.org/10.3102/0013189X005010003

Glass, G. V., McGraw, B., & Smith, M. L. (1981). Meta-analysis for social research . Sage.

Goecke, B., Weiss, S., Steger, D., Schroeders, U., & Wilhelm, O. (2020). Testing competing claims about overclaiming. Intelligence, 81 , 101470. https://doi.org/10.1016/j.intell.2020.101470

Grissom, R. J., & Kim, J. J. (2005). Effect sizes for research: A broad practical approach . Lawrence Erlbaum Associates Publishers.

Haidich, A. B. (2010). Meta-analysis in medical research. Hippokratia, 14 , 29–37.

PubMed   PubMed Central   Google Scholar  

Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., & Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science , 5 (8), 180448. https://doi.org/10.1098/rsos.180448

Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., & Frank, M. C. (2021). Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study. Royal Society Open Science, 8 (1), 201494. https://doi.org/10.1098/rsos.201494

Harris, J. A. (2004). Measured intelligence, achievement, openness to experience, and creativity. Personality and Individual Differences, 36 (4), 913–929. https://doi.org/10.1016/S0191-8869(03)00161-2

Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32 , 341–370. https://doi.org/10.3102/1076998606298043

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis . Academic Press.

Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009). Do research literatures give correct answers? Review of General Psychology, 13 , 116–121. https://doi.org/10.1037/a0015468

Hubbard, R., & Armstrong, J. S. (1992). Are null results becoming an endangered species in marketing? Marketing Letters, 3 , 127–136. https://doi.org/10.1007/BF00993992

Hunter, J. E., Schmidt, F. L., & Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91 , 594–612. https://doi.org/10.1037/0021-9010.91.3.594

Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M. A. (2019). A framework for learning from null results. Educational Researcher, 48 , 580–589. https://doi.org/10.3102/0013189X19891955

Jak, S. (2015). Meta-analytic structural equation modelling . Springer. https://doi.org/10.1007/978-3-319-27174-3

Jak, S., & Cheung, M. W. L. (2020). Meta-analytic structural equation modeling with moderating effects on SEM parameters. Psychological Methods, 25 , 430. https://doi.org/10.1037/met0000245

Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1 , 227–235. https://doi.org/10.1037/1082-989X.1.3.227

Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17 (2), 137–151. https://doi.org/10.1037/a0028086

Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30 , 81–93. https://doi.org/10.1093/biomet/30.1-2.81

Krumholz, H. M. (2012). Open science and data sharing in clinical research: Basing informed decisions on the totality of the evidence. Circulation: Cardiovascular Quality and Outcomes, 5 , 141–142. https://doi.org/10.1161/circoutcomes.112.965848

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4 , 863. https://doi.org/10.3389/fpsyg.2013.00863

Landis, R. S., James, L. R., Lance, C. E., Pierce, C. A., & Rogelberg, S. G. (2014). When is nothing something? Editorial for the null results special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29 , 163–167. https://doi.org/10.1007/s10869-014-9347-8

Lee, J., & Beretvas, S. N. (2022). Comparing methods for handling missing covariates in meta-regression. Research Synthesis Methods, 14 , 117–136. https://doi.org/10.1002/jrsm.1585

Logan, J. A. R., Hart, S. A., & Schatschneider, C. (2021). Data sharing in education science. AERA Open, 7 , 233285842110064. https://doi.org/10.1177/23328584211006475

Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2011). Cliff´ s Delta Calculator: Un programa para calcular el tamaño del efecto no paramétrico al comparar dos grupos de observaciones. Universitas Psychologica, 10 , 545–555.

McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75 , 40–46. https://doi.org/10.1016/j.jclinepi.2016.01.021

Mirzayi, C., et al. (2021). Reporting guidelines for human microbiome research: The STORMS checklist. Nature Medicine, 27 , 1885–1892. https://doi.org/10.1038/s41591-021-01552-x

Mlinarić, A., Horvat, M., & ŠupakSmolčić, V. (2017). Dealing with the positive publication bias: Why you should really publish your negative results. Biochemia Medica, 27 , 447–452. https://doi.org/10.11613/BM.2017.030201

Morabia, A., & Wynder, E. L. (1991). Cigarette smoking and lung cancer cell types. Cancer, 68 , 2074–2078. https://doi.org/10.1002/1097-0142(19911101)68:9%3C2074::AID-CNCR2820680939%3E3.0.CO;2-X

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological methods, 7 , 105–125. https://doi.org/10.1037//1082-989x.7.1.105

Ng, M. Y., DiVasto, K. A., Cootner, S., Lipsey, M. W., & Weisz, J. R. (2023). How do cognitive behavioral therapy and interpersonal psychotherapy improve youth depression? Applying meta-analytic structural equation modeling to three decades of randomized trials. Psychological Bulletin, 149 , 507–548. https://doi.org/10.1037/bul0000395

Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3 , 1–18. https://doi.org/10.1016/0022-2496(66)90002-2

Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science, 3 , 229–237. https://doi.org/10.31234/osf.io/fk8vh

Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25 , 241–286. https://doi.org/10.1006/ceps.2000.1040

Özsoy, G., & Özsoy, S. (2013). Effect size reporting in educational research. Elementary Education Online, 12 , 334–346. http://earsiv.odu.edu.tr:8080/xmlui/handle/11489/1756 . Accessed 1 Jan 2024.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews.  International Journal of Surgery ,  88 , 105906. https://doi.org/10.1016/j.ijsu.2021.105906

Pek, J., & Flora, D. B. (2018). Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological Methods, 23 , 208–225. https://doi.org/10.1037/met0000126

Peterson, R. A., & Brown, S. P. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90 , 175–181. https://doi.org/10.1037/0021-9010.90.1.175

Pigott, T. D. (2019). Missing data in meta-analysis. In Russell Sage (Ed.), Handbook of Research Synthesis (pp. 367–381). Russell Sage Foundation.

Pigott, T. D., & Polanin, J. R. (2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90 , 24–46. https://doi.org/10.3102/0034654319877153

Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86 , 207–236. https://doi.org/10.3102/0034654315582067

Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. (2019). Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Research Synthesis Methods, 10 , 330–342. https://doi.org/10.1002/jrsm.1354

Article   PubMed Central   Google Scholar  

Ray, J. W., & Shadish, W. R. (1996). How interchangeable are different estimators of effect size? Journal of Consulting and Clinical Psychology, 64 , 1316–1325. https://doi.org/10.1037/0022-006X.64.6.1316

Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010). Meta-analysis of individual participant data: rationale, conduct, and reporting. British Medical Journal, 340 , c221. https://doi.org/10.1136/bmj.c221

Rytwinski, T., Cooke, S. J., Taylor, J. J., Roche, D. G., Smith, P. A., Mitchell, G. W., Smokorowski, K. E., Prior, K. A., & Bennett, J. R. (2021). Acting in the face of evidentiary ambiguity, bias, and absence arising from systematic reviews in applied environmental science. Science of the Total Environment, 775 , 145122. https://doi.org/10.1016/j.scitotenv.2021.145122

Said-Metwaly, S., Fernández-Castilla, B., Kyndt, E., & Van den Noortgate, W. (2018). The factor structure of the Figural Torrance Tests of Creative Thinking: A meta-confirmatory factor analysis. Creativity Research Journal, 30 , 352–360.

Sánchez‐Meca, J., Marín‐Martínez, F., López‐López, J. A., Núñez‐Núñez, R. M., Rubio‐Aparicio, M., López‐García, J. J., López‐Nicolás, R. (2021). Improving the reporting quality of reliability generalization meta‐analyses: The REGEMA checklist.  Research Synthesis Methods, 12 , 516-536. https://doi.org/10.1002/jrsm.1487

Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psychology, 86 , 162–173. https://doi.org/10.1037/0022-3514.86.1.162

Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis (3rd ed.). Sage.

Sharma, A., Minh Duc, N. T., Luu Lam Thang, T., Nam, N. H., Ng, S. J., Abbas, K. S., & Karamouzian, M. (2021). A consensus-based checklist for reporting of survey studies (CROSS).  Journal of General Internal Medicine, 36 , 3179-3187 https://doi.org/10.1007/s11606-021-06737-1

Shields, P. G. (2000). Publication bias is a scientific problem with adverse ethical outcomes: The case for a section for null results. Cancer Epidemiology Biomarkers & Prevention, 9 , 771–772.

Smith, M. M., Hewitt, P. L., Sherry, S. B., Flett, G. L., & Ray, C. (2022). Parenting behaviors and trait perfectionism: A meta-analytic test of the social expectations and social learning models. Journal of Research in Personality, 96 , 104180. https://doi.org/10.1016/j.jrp.2021.104180

Snijders, T. A. (2005). Power and sample size in multilevel modeling. Encyclopedia of Statistics in Behavioral Science, 3 , 1573. https://doi.org/10.1017/cbo9780511790942.026

Sterne, J. A., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., & Higgins, J. P. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials . BMJ , 366 . https://doi.org/10.1136/bmj.l4898

Therrien, W. J., & Cook, B. G. (2018). Introduction to special issue: Null effects and publication bias in learning disabilities research. Learning Disabilities Research & Practice, 33 , 5–10. https://doi.org/10.1111/ldrp.12163

Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10 , 161–179. https://doi.org/10.1002/jrsm.1338

Trusty, J., Thompson, B., & Petrocelli, J. V. (2004). Practical guide for reporting effect size in quantitative research in the Journal of Counseling & Development. Journal of Counseling & Development, 82 , 107–110. https://doi.org/10.1002/j.1556-6678.2004.tb00291.x

Walker, E., Hernandez, A. V., & Kattan, M. W. (2008). Meta-analysis: Its strengths and limitations. Cleveland Clinic Journal of Medicine, 75 (6), 431–439. https://doi.org/10.3949/ccjm.75.6.431

Whiting, P., Wolff, R., Savović, J., Mallett, S., & Devine, B. (2023). LATITUDES network. Retrieved from https://www.latitudes-network.org/ . Accessed 1 Jan 2024.

Wiemer, J., Gerdes, A. B. M., & Pauli, P. (2013). The effects of an unexpected spider stimulus on skin conductance responses and eye movements: An inattentional blindness study. Psychological Research, 77 , 155–166. https://doi.org/10.1007/s00426-011-0407-7

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., & Bouwman, J. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3 , 109. https://doi.org/10.1038/sdata.2016.18

Wood, K., & Simons, D. J. (2017). Selective attention in inattentional blindness: Selection is specific, but suppression is not. Collabra Psychology, 3 , 19. https://doi.org/10.1525/collabra.90

Download references

Acknowledgments

We would like to express our most sincere thanks to Julio Sánchez-Meca, S. Natasha Beretvas, Mariola Moeyaert, and Juan Botella for their feedback and suggestions for improving the SEMI.

Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.

Author information

Belén Fernández-Castilla and Sameh Said-Metwaly equally contributed to the work.

Authors and Affiliations

Faculty of Psychology, Universidad Nacional de Educación a Distancia, Juan del Rosal 10, 28040, Madrid, Spain

Belén Fernández-Castilla & Rodrigo S. Kreitchmann

Faculty of Psychology and Educational Sciences, KU, Leuven, Belgium

Sameh Said-Metwaly & Wim Van Den Noortgate

Imec-Itec, KU, Leuven, Belgium

Faculty of Education, Damanhour University, Damanhour, Egypt

Sameh Said-Metwaly

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Belén Fernández-Castilla .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Practices Statement

This manuscript has no associated data.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Fernández-Castilla, B., Said-Metwaly, S., Kreitchmann, R.S. et al. What do meta-analysts need in primary studies? Guidelines and the SEMI checklist for facilitating cumulative knowledge. Behav Res (2024). https://doi.org/10.3758/s13428-024-02373-9

Download citation

Accepted : 17 February 2024

Published : 16 April 2024

DOI : https://doi.org/10.3758/s13428-024-02373-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-analysis
• Cumulative knowledge
• Find a journal
• Publish with us
• Track your research

Study Design 101: Meta-Analysis

• Case Report
• Case Control Study
• Cohort Study
• Randomized Controlled Trial
• Practice Guideline
• Systematic Review

Meta-Analysis

• Helpful Formulas
• Finding Specific Study Types

A subset of systematic reviews; a method for systematically combining pertinent qualitative and quantitative study data from several selected studies to develop a single conclusion that has greater statistical power. This conclusion is statistically stronger than the analysis of any single study, due to increased numbers of subjects, greater diversity among subjects, or accumulated effects and results.

Meta-analysis would be used for the following purposes:

• To establish statistical significance with studies that have conflicting results
• To develop a more correct estimate of effect magnitude
• To provide a more complex analysis of harms, safety data, and benefits
• To examine subgroups with individual numbers that are not statistically significant

If the individual studies utilized randomized controlled trials (RCT), combining several selected RCT results would be the highest-level of evidence on the evidence hierarchy, followed by systematic reviews, which analyze all available studies on a topic.

• Greater statistical power
• Confirmatory data analysis
• Greater ability to extrapolate to general population affected
• Considered an evidence-based resource

Disadvantages

• Difficult and time consuming to identify appropriate studies
• Not all studies provide adequate data for inclusion and analysis
• Requires advanced statistical techniques
• Heterogeneity of study populations

Design pitfalls to look out for

The studies pooled for review should be similar in type (i.e. all randomized controlled trials).

Are the studies being reviewed all the same type of study or are they a mixture of different types?

The analysis should include published and unpublished results to avoid publication bias.

Does the meta-analysis include any appropriate relevant studies that may have had negative outcomes?

Fictitious Example

Do individuals who wear sunscreen have fewer cases of melanoma than those who do not wear sunscreen? A MEDLINE search was conducted using the terms melanoma, sunscreening agents, and zinc oxide, resulting in 8 randomized controlled studies, each with between 100 and 120 subjects. All of the studies showed a positive effect between wearing sunscreen and reducing the likelihood of melanoma. The subjects from all eight studies (total: 860 subjects) were pooled and statistically analyzed to determine the effect of the relationship between wearing sunscreen and melanoma. This meta-analysis showed a 50% reduction in melanoma diagnosis among sunscreen-wearers.

Real-life Examples

Goyal, A., Elminawy, M., Kerezoudis, P., Lu, V., Yolcu, Y., Alvi, M., & Bydon, M. (2019). Impact of obesity on outcomes following lumbar spine surgery: A systematic review and meta-analysis. Clinical Neurology and Neurosurgery, 177 , 27-36. https://doi.org/10.1016/j.clineuro.2018.12.012

This meta-analysis was interested in determining whether obesity affects the outcome of spinal surgery. Some previous studies have shown higher perioperative morbidity in patients with obesity while other studies have not shown this effect. This study looked at surgical outcomes including "blood loss, operative time, length of stay, complication and reoperation rates and functional outcomes" between patients with and without obesity. A meta-analysis of 32 studies (23,415 patients) was conducted. There were no significant differences for patients undergoing minimally invasive surgery, but patients with obesity who had open surgery had experienced higher blood loss and longer operative times (not clinically meaningful) as well as higher complication and reoperation rates. Further research is needed to explore this issue in patients with morbid obesity.

Nakamura, A., van Der Waerden, J., Melchior, M., Bolze, C., El-Khoury, F., & Pryor, L. (2019). Physical activity during pregnancy and postpartum depression: Systematic review and meta-analysis. Journal of Affective Disorders, 246 , 29-41. https://doi.org/10.1016/j.jad.2018.12.009

This meta-analysis explored whether physical activity during pregnancy prevents postpartum depression. Seventeen studies were included (93,676 women) and analysis showed a "significant reduction in postpartum depression scores in women who were physically active during their pregnancies when compared with inactive women." Possible limitations or moderators of this effect include intensity and frequency of physical activity, type of physical activity, and timepoint in pregnancy (e.g. trimester).

Related Terms

A document often written by a panel that provides a comprehensive review of all relevant studies on a particular clinical or health-related topic/question.

Publication Bias

A phenomenon in which studies with positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Therefore, conclusions based exclusively on published studies can be misleading.

Now test yourself!

1. A Meta-Analysis pools together the sample populations from different studies, such as Randomized Controlled Trials, into one statistical analysis and treats them as one large sample population with one conclusion.

a) True b) False

2. One potential design pitfall of Meta-Analyses that is important to pay attention to is:

a) Whether it is evidence-based. b) If the authors combined studies with conflicting results. c) If the authors appropriately combined studies so they did not compare apples and oranges. d) If the authors used only quantitative data.

Evidence Pyramid - Navigation

• Meta- Analysis
• Case Reports
• << Previous: Systematic Review
• Next: Helpful Formulas >>

• Last Updated: Sep 25, 2023 10:59 AM
• URL: https://guides.himmelfarb.gwu.edu/studydesign101

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

What is Meta-Analysis? Definition, Research & Examples

Appinio Research · 01.02.2024 · 39min read

Are you looking to harness the power of data and uncover meaningful insights from a multitude of research studies? In a world overflowing with information, meta-analysis emerges as a guiding light, offering a systematic and quantitative approach to distilling knowledge from a sea of research.

This guide will demystify the art and science of meta-analysis, walking you through the process, from defining your research question to interpreting the results. Whether you're an academic researcher, a policymaker, or a curious mind eager to explore the depths of data, this guide will equip you with the tools and understanding needed to undertake robust and impactful meta-analyses.

What is a Meta Analysis?

Meta-analysis is a quantitative research method that involves the systematic synthesis and statistical analysis of data from multiple individual studies on a particular topic or research question. It aims to provide a comprehensive and robust summary of existing evidence by pooling the results of these studies, often leading to more precise and generalizable conclusions.

The primary purpose of meta-analysis is to:

• Quantify Effect Sizes:  Determine the magnitude and direction of an effect or relationship across studies.
• Evaluate Consistency:  Assess the consistency of findings among studies and identify sources of heterogeneity.
• Enhance Statistical Power:  Increase the statistical power to detect significant effects by combining data from multiple studies.
• Generalize Results:  Provide more generalizable results by analyzing a more extensive and diverse sample of participants or contexts.
• Examine Subgroup Effects:  Explore whether the effect varies across different subgroups or study characteristics.

Importance of Meta-Analysis

Meta-analysis plays a crucial role in scientific research and evidence-based decision-making. Here are key reasons why meta-analysis is highly valuable:

• Enhanced Precision:  By pooling data from multiple studies, meta-analysis provides a more precise estimate of the effect size, reducing the impact of random variation.
• Increased Statistical Power:  The combination of numerous studies enhances statistical power, making it easier to detect small but meaningful effects.
• Resolution of Inconsistencies:  Meta-analysis can help resolve conflicting findings in the literature by systematically analyzing and synthesizing evidence.
• Identification of Moderators:  It allows for the identification of factors that may moderate the effect, helping to understand when and for whom interventions or treatments are most effective.
• Evidence-Based Decision-Making:  Policymakers, clinicians, and researchers use meta-analysis to inform evidence-based decision-making, leading to more informed choices in healthcare , education, and other fields.
• Efficient Use of Resources:  Meta-analysis can guide future research by identifying gaps in knowledge, reducing duplication of efforts, and directing resources to areas with the most significant potential impact.

Types of Research Questions Addressed

Meta-analysis can address a wide range of research questions across various disciplines. Some common types of research questions that meta-analysis can tackle include:

• Treatment Efficacy:  Does a specific medical treatment, therapy, or intervention have a significant impact on patient outcomes or symptoms?
• Intervention Effectiveness:  How effective are educational programs, training methods, or interventions in improving learning outcomes or skills?
• Risk Factors and Associations:  What are the associations between specific risk factors, such as smoking or diet, and the likelihood of developing certain diseases or conditions?
• Impact of Policies:  What is the effect of government policies, regulations, or interventions on social, economic, or environmental outcomes?
• Psychological Constructs:  How do psychological constructs, such as self-esteem, anxiety, or motivation, influence behavior or mental health outcomes?
• Comparative Effectiveness:  Which of two or more competing interventions or treatments is more effective for a particular condition or population?
• Dose-Response Relationships:  Is there a dose-response relationship between exposure to a substance or treatment and the likelihood or severity of an outcome?

Meta-analysis is a versatile tool that can provide valuable insights into a wide array of research questions, making it an indispensable method in evidence synthesis and knowledge advancement.

Meta-Analysis vs. Systematic Review

In evidence synthesis and research aggregation, meta-analysis and systematic reviews are two commonly used methods, each serving distinct purposes while sharing some similarities. Let's explore the differences and similarities between these two approaches.

Meta-Analysis

• Purpose:  Meta-analysis is a statistical technique used to combine and analyze quantitative data from multiple individual studies that address the same research question. The primary aim of meta-analysis is to provide a single summary effect size that quantifies the magnitude and direction of an effect or relationship across studies.
• Data Synthesis:  In meta-analysis, researchers extract and analyze numerical data, such as means, standard deviations, correlation coefficients, or odds ratios, from each study. These effect size estimates are then combined using statistical methods to generate an overall effect size and associated confidence interval.
• Quantitative:  Meta-analysis is inherently quantitative, focusing on numerical data and statistical analyses to derive a single effect size estimate.
• Main Outcome:  The main outcome of a meta-analysis is the summary effect size, which provides a quantitative estimate of the research question's answer.

Systematic Review

• Purpose:  A systematic review is a comprehensive and structured overview of the available evidence on a specific research question. While systematic reviews may include meta-analysis, their primary goal is to provide a thorough and unbiased summary of the existing literature.
• Data Synthesis:  Systematic reviews involve a meticulous process of literature search, study selection, data extraction, and quality assessment. Researchers may narratively synthesize the findings, providing a qualitative summary of the evidence.
• Qualitative:  Systematic reviews are often qualitative in nature, summarizing and synthesizing findings in a narrative format. They do not always involve statistical analysis .
• Main Outcome:  The primary outcome of a systematic review is a comprehensive narrative summary of the existing evidence. While some systematic reviews include meta-analyses, not all do so.

Key Differences

• Nature of Data:  Meta-analysis primarily deals with quantitative data and statistical analysis , while systematic reviews encompass both quantitative and qualitative data, often presenting findings in a narrative format.
• Focus on Effect Size:  Meta-analysis focuses on deriving a single, quantitative effect size estimate, whereas systematic reviews emphasize providing a comprehensive overview of the literature, including study characteristics, methodologies, and key findings.
• Synthesis Approach:  Meta-analysis is a quantitative synthesis method, while systematic reviews may use both quantitative and qualitative synthesis approaches.

Commonalities

• Structured Process:  Both meta-analyses and systematic reviews follow a structured and systematic process for literature search, study selection, data extraction, and quality assessment.
• Evidence-Based:  Both approaches aim to provide evidence-based answers to specific research questions, offering valuable insights for decision-making in various fields.
• Transparency:  Both meta-analyses and systematic reviews prioritize transparency and rigor in their methodologies to minimize bias and enhance the reliability of their findings.

While meta-analysis and systematic reviews share the overarching goal of synthesizing research evidence, they differ in their approach and main outcomes. Meta-analysis is quantitative, focusing on effect sizes, while systematic reviews provide comprehensive overviews, utilizing both quantitative and qualitative data to summarize the literature. Depending on the research question and available data, one or both of these methods may be employed to provide valuable insights for evidence-based decision-making.

How to Conduct a Meta-Analysis?

Planning a meta-analysis is a critical phase that lays the groundwork for a successful and meaningful study. We will explore each component of the planning process in more detail, ensuring you have a solid foundation before diving into data analysis.

How to Formulate Research Questions?

Your research questions are the guiding compass of your meta-analysis. They should be precise and tailored to the topic you're investigating. To craft effective research questions:

• Clearly Define the Problem:  Start by identifying the specific problem or topic you want to address through meta-analysis.
• Specify Key Variables:  Determine the essential variables or factors you'll examine in the included studies.
• Frame Hypotheses:  If applicable, create clear hypotheses that your meta-analysis will test.

For example, if you're studying the impact of a specific intervention on patient outcomes, your research question might be: "What is the effect of Intervention X on Patient Outcome Y in published clinical trials?"

Eligibility Criteria

Eligibility criteria define the boundaries of your meta-analysis. By establishing clear criteria, you ensure that the studies you include are relevant and contribute to your research objectives. Key considerations for eligibility criteria include:

• Study Types:  Decide which types of studies will be considered (e.g., randomized controlled trials, cohort studies, case-control studies).
• Publication Time Frame:  Specify the publication date range for included studies.
• Language:  Determine whether studies in languages other than your primary language will be included.
• Geographic Region:  If relevant, define any geographic restrictions.

Your eligibility criteria should strike a balance between inclusivity and relevance. Excluding certain studies based on valid criteria ensures the quality and relevance of the data you analyze.

Search Strategy

A robust search strategy is fundamental to identifying all relevant studies. To create an effective search strategy:

• Select Databases:  Choose appropriate databases that cover your research area (e.g., PubMed, Scopus, Web of Science).
• Keywords and Search Terms:  Develop a comprehensive list of relevant keywords and search terms related to your research questions.
• Search Filters:  Utilize search filters and Boolean operators (AND, OR) to refine your search queries.
• Manual Searches:  Consider conducting hand-searches of key journals and reviewing the reference lists of relevant studies for additional sources.

Remember that the goal is to cast a wide net while maintaining precision to capture all relevant studies.

Data Extraction

Data extraction is the process of systematically collecting information from each selected study. It involves retrieving key data points, including:

• Study Characteristics:  Author(s), publication year, study design, sample size, duration, and location.
• Outcome Data:  Effect sizes, standard errors, confidence intervals, p-values, and any other relevant statistics.
• Methodological Details:  Information on study quality, risk of bias, and potential sources of heterogeneity.

Creating a standardized data extraction form is essential to ensure consistency and accuracy throughout this phase. Spreadsheet software, such as Microsoft Excel, is commonly used for data extraction.

Quality Assessment

Assessing the quality of included studies is crucial to determine their reliability and potential impact on your meta-analysis. Various quality assessment tools and checklists are available, depending on the study design. Some commonly used tools include:

• Newcastle-Ottawa Scale:  Used for assessing the quality of non-randomized studies (e.g., cohort, case-control studies).
• Cochrane Risk of Bias Tool:  Designed for evaluating randomized controlled trials.

Quality assessment typically involves evaluating aspects such as study design, sample size, data collection methods , and potential biases. This step helps you weigh the contribution of each study to the overall analysis.

How to Conduct a Literature Review?

Conducting a thorough literature review is a critical step in the meta-analysis process. We will explore the essential components of a literature review, from designing a comprehensive search strategy to establishing clear inclusion and exclusion criteria and, finally, the study selection process.

Comprehensive Search

To ensure the success of your meta-analysis, it's imperative to cast a wide net when searching for relevant studies. A comprehensive search strategy involves:

• Selecting Relevant Databases:  Identify databases that cover your research area comprehensively, such as PubMed, Scopus, Web of Science, or specialized databases specific to your field.
• Creating a Keyword List:  Develop a list of relevant keywords and search terms related to your research questions. Think broadly and consider synonyms, acronyms, and variations.
• Using Boolean Operators:  Utilize Boolean operators (AND, OR) to combine keywords effectively and refine your search.
• Applying Filters:  Employ search filters (e.g., publication date range, study type) to narrow down results based on your eligibility criteria.

Remember that the goal is to leave no relevant stone unturned, as missing key studies can introduce bias into your meta-analysis.

Inclusion and Exclusion Criteria

Clearly defined inclusion and exclusion criteria are the gatekeepers of your meta-analysis. These criteria ensure that the studies you include meet your research objectives and maintain the quality of your analysis. Consider the following factors when establishing criteria:

• Study Types:  Determine which types of studies are eligible for inclusion (e.g., randomized controlled trials, observational studies, case reports).
• Publication Time Frame:  Specify the time frame within which studies must have been published.
• Language:  Decide whether studies in languages other than your primary language will be included or excluded.
• Geographic Region:  If applicable, define any geographic restrictions.
• Relevance to Research Questions:  Ensure that selected studies align with your research questions and objectives.

Your inclusion and exclusion criteria should strike a balance between inclusivity and relevance. Rigorous criteria help maintain the quality and applicability of the studies included in your meta-analysis.

Study Selection Process

The study selection process involves systematically screening and evaluating each potential study to determine whether it meets your predefined inclusion criteria. Here's a step-by-step guide:

• Screen Titles and Abstracts:  Begin by reviewing the titles and abstracts of the retrieved studies. Exclude studies that clearly do not meet your inclusion criteria.
• Full-Text Assessment:  Assess the full text of potentially relevant studies to confirm their eligibility. Pay attention to study design, sample size, and other specific criteria.
• Data Extraction:  For studies that meet your criteria, extract the necessary data, including study characteristics, effect sizes, and other relevant information.
• Record Exclusions:  Keep a record of the reasons for excluding studies. This transparency is crucial for the reproducibility of your meta-analysis.
• Resolve Discrepancies:  If multiple reviewers are involved, resolve any disagreements through discussion or a third-party arbitrator.

Maintaining a clear and organized record of your study selection process is essential for transparency and reproducibility. Software tools like EndNote or Covidence can facilitate the screening and data extraction process.

By following these systematic steps in conducting a literature review, you ensure that your meta-analysis is built on a solid foundation of relevant and high-quality studies.

Data Extraction and Management

As you progress in your meta-analysis journey, the data extraction and management phase becomes paramount. We will delve deeper into the critical aspects of this phase, including the data collection process, data coding and transformation, and how to handle missing data effectively.

Data Collection Process

The data collection process is the heart of your meta-analysis, where you systematically extract essential information from each selected study. To ensure accuracy and consistency:

• Create a Data Extraction Form:  Develop a standardized data extraction form that includes all the necessary fields for collecting relevant data. This form should align with your research questions and inclusion criteria.
• Data Extractors:  Assign one or more reviewers to extract data from the selected studies. Ensure they are familiar with the form and the specific data points to collect.
• Double-Check Accuracy:  Implement a verification process where a second reviewer cross-checks a random sample of data extractions to identify discrepancies or errors.
• Extract All Relevant Information:  Collect data on study characteristics, participant demographics, outcome measures, effect sizes, confidence intervals, and any additional information required for your analysis.
• Maintain Consistency:  Use clear guidelines and definitions for data extraction to ensure uniformity across studies.

To optimize your data collection process and streamline the extraction and management of crucial information, consider leveraging innovative solutions like Appinio . With Appinio, you can effortlessly collect real-time consumer insights, ensuring your meta-analysis benefits from the latest data trends and user perspectives.   Ready to learn more? Book a demo today and unlock a world of data-driven possibilities!

Get a free demo and see the Appinio platform in action!

Data Coding and Transformation

After data collection, you may need to code and transform the extracted data to ensure uniformity and compatibility across studies. This process involves:

• Coding Categorical Variables:  If studies report data differently, code categorical variables consistently . For example, ensure that categories like "male" and "female" are coded consistently across studies.
• Standardizing Units of Measurement:  Convert all measurements to a common unit if studies use different measurement units. For instance, if one study reports height in inches and another in centimeters, standardize to one unit for comparability.
• Calculating Effect Sizes:  Calculate effect sizes and their standard errors or variances if they are not directly reported in the studies. Common effect size measures include Cohen's d, odds ratio (OR), and hazard ratio (HR).
• Data Transformation:  Transform data if necessary to meet assumptions of statistical tests. Common transformations include log transformation for skewed data or arcsine transformation for proportions.
• Heterogeneity Adjustment:  Consider using transformation methods to address heterogeneity among studies, such as applying the Freeman-Tukey double arcsine transformation for proportions.

The goal of data coding and transformation is to make sure that data from different studies are compatible and can be effectively synthesized during the analysis phase. Spreadsheet software like Excel or statistical software like R can be used for these tasks.

Handling Missing Data

Missing data is a common challenge in meta-analysis, and how you handle it can impact the validity and precision of your results. Strategies for handling missing data include:

• Contact Authors:  If feasible, contact the authors of the original studies to request missing data or clarifications.
• Imputation:  Consider using appropriate imputation methods to estimate missing values, but exercise caution and report the imputation methods used.
• Sensitivity Analysis:  Conduct sensitivity analyses to assess the impact of missing data on your results by comparing the main analysis to alternative scenarios.

Remember that transparency in reporting how you handled missing data is crucial for the credibility of your meta-analysis.

By following these steps in data extraction and management, you will ensure the integrity and reliability of your meta-analysis dataset.

Meta-Analysis Example

Meta-analysis is a versatile research method that can be applied to various fields and disciplines, providing valuable insights by synthesizing existing evidence.

Example 1: Analyzing the Impact of Advertising Campaigns on Sales

Background:  A market research agency is tasked with assessing the effectiveness of advertising campaigns on sales outcomes for a range of consumer products. They have access to multiple studies and reports conducted by different companies, each analyzing the impact of advertising on sales revenue.

Meta-Analysis Approach:

• Study Selection:  Identify relevant studies that meet specific inclusion criteria, such as the type of advertising campaign (e.g., TV commercials, social media ads), the products examined, and the sales metrics assessed.
• Data Extraction:  Collect data from each study, including details about the advertising campaign (e.g., budget, duration), sales data (e.g., revenue, units sold), and any reported effect sizes or correlations.
• Effect Size Calculation:  Calculate effect sizes (e.g., correlation coefficients) based on the data provided in each study, quantifying the strength and direction of the relationship between advertising and sales.
• Data Synthesis:  Employ meta-analysis techniques to combine the effect sizes from the selected studies. Compute a summary effect size and its confidence interval to estimate the overall impact of advertising on sales.
• Publication Bias Assessment:  Use funnel plots and statistical tests to assess the potential presence of publication bias, ensuring that the meta-analysis results are not unduly influenced by selective reporting.

Findings:  Through meta-analysis, the market research agency discovers that advertising campaigns have a statistically significant and positive impact on sales across various product categories. The findings provide evidence for the effectiveness of advertising efforts and assist companies in making data-driven decisions regarding their marketing strategies.

These examples illustrate how meta-analysis can be applied in diverse domains, from tech startups seeking to optimize user engagement to market research agencies evaluating the impact of advertising campaigns. By systematically synthesizing existing evidence, meta-analysis empowers decision-makers with valuable insights for informed choices and evidence-based strategies.

How to Assess Study Quality and Bias?

Ensuring the quality and reliability of the studies included in your meta-analysis is essential for drawing accurate conclusions. We'll show you how you can assess study quality using specific tools, evaluate potential bias, and address publication bias.

Quality Assessment Tools

Quality assessment tools provide structured frameworks for evaluating the methodological rigor of each included study. The choice of tool depends on the study design. Here are some commonly used quality assessment tools:

For Randomized Controlled Trials (RCTs):

• Cochrane Risk of Bias Tool:  This tool assesses the risk of bias in RCTs based on six domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective reporting.
• Jadad Scale:  A simpler tool specifically for RCTs, the Jadad Scale focuses on randomization, blinding, and the handling of withdrawals and dropouts.

For Observational Studies:

• Newcastle-Ottawa Scale (NOS):  The NOS assesses the quality of cohort and case-control studies based on three categories: selection, comparability, and outcome.
• ROBINS-I:  Designed for non-randomized studies of interventions, the Risk of Bias in Non-randomized Studies of Interventions tool evaluates bias in domains such as confounding, selection bias, and measurement bias.
• MINORS:  The Methodological Index for Non-Randomized Studies (MINORS) assesses non-comparative studies and includes items related to study design, reporting, and statistical analysis.

Bias Assessment

Evaluating potential sources of bias is crucial to understanding the limitations of the included studies. Common sources of bias include:

• Selection Bias:  Occurs when the selection of participants is not random or representative of the target population.
• Performance Bias:  Arises when participants or researchers are aware of the treatment or intervention status, potentially influencing outcomes.
• Detection Bias:  Occurs when outcome assessors are not blinded to the treatment groups.
• Attrition Bias:  Results from incomplete data or differential loss to follow-up between treatment groups.
• Reporting Bias:  Involves selective reporting of outcomes, where only positive or statistically significant results are published.

To assess bias, reviewers often use the quality assessment tools mentioned earlier, which include domains related to bias, or they may specifically address bias concerns in the narrative synthesis.

We'll move on to the core of meta-analysis: data synthesis. We'll explore different effect size measures, fixed-effect versus random-effects models, and techniques for assessing and addressing heterogeneity among studies.

Data Synthesis

Now that you've gathered data from multiple studies and assessed their quality, it's time to synthesize this information effectively.

Effect Size Measures

Effect size measures quantify the magnitude of the relationship or difference you're investigating in your meta-analysis. The choice of effect size measure depends on your research question and the type of data provided by the included studies. Here are some commonly used effect size measures:

Continuous Outcome Data:

• Cohen's d:  Measures the standardized mean difference between two groups. It's suitable for continuous outcome variables.
• Hedges' g:  Similar to Cohen's d but incorporates a correction factor for small sample sizes.

Binary Outcome Data:

• Odds Ratio (OR):  Used for dichotomous outcomes, such as success/failure or presence/absence.
• Risk Ratio (RR):  Similar to OR but used when the outcome is relatively common.

Time-to-Event Data:

• Hazard Ratio (HR):  Used in survival analysis to assess the risk of an event occurring over time.
• Risk Difference (RD):  Measures the absolute difference in event rates between two groups.

Selecting the appropriate effect size measure depends on the nature of your data and the research question. When effect sizes are not directly reported in the studies, you may need to calculate them using available data, such as means, standard deviations, and sample sizes.

Formula for Cohen's d:

d = (Mean of Group A - Mean of Group B) / Pooled Standard Deviation

Fixed-Effect vs. Random-Effects Models

In meta-analysis, you can choose between fixed-effect and random-effects models to combine the results of individual studies:

Fixed-Effect Model:

• Assumes that all included studies share a common true effect size.
• Accounts for only within-study variability ( sampling error ).
• Appropriate when studies are very similar or when there's minimal heterogeneity.

Random-Effects Model:

• Acknowledges that there may be variability in effect sizes across studies.
• Accounts for both within-study variability (sampling error) and between-study variability (real differences between studies).
• More conservative and applicable when there's substantial heterogeneity.

The choice between these models should be guided by the degree of heterogeneity observed among the included studies. If heterogeneity is significant, the random-effects model is often preferred, as it provides a more robust estimate of the overall effect.

Forest Plots

Forest plots are graphical representations commonly used in meta-analysis to display the results of individual studies along with the combined summary estimate. Key components of a forest plot include:

• Vertical Line:  Represents the null effect (e.g., no difference or no effect).
• Horizontal Lines:  Represent the confidence intervals for each study's effect size estimate.
• Diamond or Square:  Represents the summary effect size estimate, with its width indicating the confidence interval around the summary estimate.
• Study Names:  Listed on the left side of the plot, identifying each study.

Forest plots help visualize the distribution of effect sizes across studies and provide insights into the consistency and direction of the findings.

Heterogeneity Assessment

Heterogeneity refers to the variability in effect sizes among the included studies. It's important to assess and understand heterogeneity as it can impact the interpretation of your meta-analysis results. Standard methods for assessing heterogeneity include:

• Cochran's Q Test:  A statistical test that assesses whether there is significant heterogeneity among the effect sizes of the included studies.
• I² Statistic:  A measure that quantifies the proportion of total variation in effect sizes that is due to heterogeneity. I² values range from 0% to 100%, with higher values indicating greater heterogeneity.

Assessing heterogeneity is crucial because it informs your choice of meta-analysis model (fixed-effect vs. random-effects) and whether subgroup analyses or sensitivity analyses are warranted to explore potential sources of heterogeneity.

How to Interpret Meta-Analysis Results?

With the data synthesis complete, it's time to make sense of the results of your meta-analysis.

Meta-Analytic Summary

The meta-analytic summary is the culmination of your efforts in data synthesis. It provides a consolidated estimate of the effect size and its confidence interval, combining the results of all included studies. To interpret the meta-analytic summary effectively:

• Effect Size Estimate:  Understand the primary effect size estimate, such as Cohen's d, odds ratio, or hazard ratio, and its associated confidence interval.
• Significance:  Determine whether the summary effect size is statistically significant. This is indicated when the confidence interval does not include the null value (e.g., 0 for Cohen's d or 1 for odds ratio).
• Magnitude:  Assess the magnitude of the effect size. Is it large, moderate, or small, and what are the practical implications of this magnitude?
• Direction:  Consider the direction of the effect. Is it in the hypothesized direction, or does it contradict the expected outcome?
• Clinical or Practical Significance:  Reflect on the clinical or practical significance of the findings. Does the effect size have real-world implications?
• Consistency:  Evaluate the consistency of the findings across studies. Are most studies in agreement with the summary effect size estimate, or are there outliers?

Subgroup Analyses

Subgroup analyses allow you to explore whether the effect size varies across different subgroups of studies or participants. This can help identify potential sources of heterogeneity or assess whether the intervention's effect differs based on specific characteristics. Steps for conducting subgroup analyses:

• Define Subgroups:  Clearly define the subgroups you want to investigate based on relevant study characteristics (e.g., age groups, study design , intervention type).
• Analyze Subgroups:  Calculate separate summary effect sizes for each subgroup and compare them to the overall summary effect.
• Assess Heterogeneity:  Evaluate whether subgroup differences are statistically significant. If so, this suggests that the effect size varies significantly among subgroups.
• Interpretation:  Interpret the subgroup findings in the context of your research question. Are there meaningful differences in the effect across subgroups? What might explain these differences?

Subgroup analyses can provide valuable insights into the factors influencing the overall effect size and help tailor recommendations for specific populations or conditions.

Sensitivity Analyses

Sensitivity analyses are conducted to assess the robustness of your meta-analysis results by exploring how different choices or assumptions might affect the findings. Common sensitivity analyses include:

• Exclusion of Low-Quality Studies:  Repeating the meta-analysis after excluding studies with low quality or a high risk of bias.
• Changing Effect Size Measure:  Re-running the analysis using a different effect size measure to assess whether the choice of measure significantly impacts the results.
• Publication Bias Adjustment:  Applying methods like the trim-and-fill procedure to adjust for potential publication bias.
• Subsample Analysis:  Analyzing a subset of studies based on specific criteria or characteristics to investigate their impact on the summary effect.

Sensitivity analyses help assess the robustness and reliability of your meta-analysis results, providing a more comprehensive understanding of the potential influence of various factors.

Reporting and Publication

The final stages of your meta-analysis involve preparing your findings for publication.

Manuscript Preparation

When preparing your meta-analysis manuscript, consider the following:

• Structured Format:  Organize your manuscript following a structured format, including sections such as introduction, methods, results, discussion, and conclusions.
• Clarity and Conciseness:  Write your findings clearly and concisely, avoiding jargon or overly technical language. Use tables and figures to enhance clarity.
• Transparent Methods:  Provide detailed descriptions of your methods, including eligibility criteria, search strategy, data extraction, and statistical analysis.
• Incorporate Tables and Figures:  Present your meta-analysis results using tables and forest plots to visually convey key findings.
• Interpretation:  Interpret the implications of your findings, discussing the clinical or practical significance and limitations.

Transparent Reporting Guidelines

Adhering to transparent reporting guidelines ensures that your meta-analysis is transparent, reproducible, and credible. Some widely recognized guidelines include:

• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses):  PRISMA provides a checklist and flow diagram for reporting systematic reviews and meta-analyses, enhancing transparency and rigor.
• MOOSE (Meta-analysis of Observational Studies in Epidemiology):  MOOSE guidelines are designed for meta-analyses of observational studies and provide a framework for transparent reporting.
• ROBINS-I:  If your meta-analysis involves non-randomized studies, follow the Risk Of Bias In Non-randomized Studies of Interventions guidelines for reporting.

Adhering to these guidelines ensures that your meta-analysis is transparent, reproducible, and credible. It enhances the quality of your research and aids readers and reviewers in assessing the rigor of your study.

PRISMA Statement

The PRISMA statement is a valuable resource for conducting and reporting systematic reviews and meta-analyses. Key elements of PRISMA include:

• Title:  Clearly indicate that your paper is a systematic review or meta-analysis.
• Structured Abstract:  Provide a structured summary of your study, including objectives, methods, results, and conclusions.
• Transparent Reporting:  Follow the PRISMA checklist, which covers items such as the rationale, eligibility criteria, search strategy, data extraction, and risk of bias assessment.
• Flow Diagram:  Include a flow diagram illustrating the study selection process.

By adhering to the PRISMA statement, you enhance the transparency and credibility of your meta-analysis, facilitating its acceptance for publication and aiding readers in evaluating the quality of your research.

Conclusion for Meta-Analysis

Meta-analysis is a powerful tool that allows you to combine and analyze data from multiple studies to find meaningful patterns and make informed decisions. It helps you see the bigger picture and draw more accurate conclusions than individual studies alone. Whether you're in healthcare, education, business, or any other field, the principles of meta-analysis can be applied to enhance your research and decision-making processes. Remember that conducting a successful meta-analysis requires careful planning, attention to detail, and transparency in reporting. By following the steps outlined in this guide, you can embark on your own meta-analysis journey with confidence, contributing to the advancement of knowledge and evidence-based practices in your area of interest.

How to Elevate Your Meta-Analysis With Real-Time Insights?

Introducing Appinio , the real-time market research platform that brings a new level of excitement to your meta-analysis journey. With Appinio, you can seamlessly collect your own market research data in minutes, empowering your meta-analysis with fresh, real-time consumer insights.

Here's why Appinio is your ideal partner for efficient data collection:

• From Questions to Insights in Minutes:  Appinio's lightning-fast platform ensures you get the answers you need when you need them, accelerating your meta-analysis process.
• No Research PhD Required:  Our intuitive platform is designed for everyone, eliminating the need for specialized research skills and putting the power of data collection in your hands.
• Global Reach, Minimal Time:  With an average field time of less than 23 minutes for 1,000 respondents and access to over 90 countries, you can define precise target groups and gather data swiftly.

Get free access to the platform!

Join the loop 💌

Be the first to hear about new updates, product news, and data insights. We'll send it all straight to your inbox.

Get the latest market research news straight to your inbox! 💌

Wait, there's more

07.05.2024 | 29min read

Interval Scale: Definition, Characteristics, Examples

03.05.2024 | 29min read

What is Qualitative Observation? Definition, Types, Examples

02.05.2024 | 32min read

What is a Perceptual Map and How to Make One? (+ Template)

Meta-analysis in medical research

Affiliation.

• 1 Department of Hygiene and Epidemiology, Aristotle University of Thessaloniki School of Medicine, Thessaloniki, Greece.
• PMID: 21487488
• PMCID: PMC3049418

The objectives of this paper are to provide an introduction to meta-analysis and to discuss the rationale for this type of research and other general considerations. Methods used to produce a rigorous meta-analysis are highlighted and some aspects of presentation and interpretation of meta-analysis are discussed.Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess previous research studies to derive conclusions about that body of research. Outcomes from a meta-analysis may include a more precise estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to the pooled analysis. The examination of variability or heterogeneity in study results is also a critical outcome. The benefits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently conflicting, body of literature. The specification of the outcome and hypotheses that are tested is critical to the conduct of meta-analyses, as is a sensitive literature search. A failure to identify the majority of existing studies can lead to erroneous conclusions; however, there are methods of examining data to identify the potential for studies to be missing; for example, by the use of funnel plots. Rigorously conducted meta-analyses are useful tools in evidence-based medicine. The need to integrate findings from many studies ensures that meta-analytic research is desirable and the large body of research now generated makes the conduct of this research feasible.

Keywords: bias; evidence-based medicine; meta-analysis; quality; randomized clinical trial; systematic review.

Quantitative Research Methods

• Introduction
• Descriptive and Inferential Statistics
• Hypothesis Testing
• Regression and Correlation
• Time Series

Meta-Analysis

• Mixed Methods
• Additional Resources
• Get Research Help

A meta-analysis uses statistical methods to synthesize the results of multiple studies, often by calculating a weighted average of effect sizes.  Before embarking on a meta-analysis, make sure you are familiar with reviews, in particular systematic reviews.  

• Meta-analysis in medical research Hippokratia article by A. B. Haidich.
• Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews Annual Review of Psychology article by R. Rosenthal and M. R. DiMatteo.
• Analyzing Data for Meta-analysis Chapter from the Cochrane Review Handbook.
• A typology of reviews Health Information & Libraries Journal article by M. J. Grant and A. Booth.

Forest Plots

A forest plot is a type of graph used in meta-analyses that displays the results of multiple studies next to each other.

Books and eBooks

• << Previous: Time Series
• Next: Mixed Methods >>
• Last Updated: Aug 18, 2023 11:55 AM
• URL: https://guides.library.duq.edu/quant-methods

Shopping Cart

No products in the cart.

Introduction to Quantitative Meta-Analysis

Length: Four Days Instructor: Tasha Beretvas Software Demonstrations: R Lifetime Access: No expirations

Student: $792 Professional: $1032

Introduction to Quantitative Meta-Analysis is a four-day workshop focused on the statistical techniques used to conduct quantitative meta-analyses. Quantitative meta-analysis allows synthesis of results from primary studies investigating relations among common variables. The procedure entails first capturing effect sizes that numerically describe the relationship among relevant variables in each primary study. Primary study characteristics can then be investigated as sources of variability in effect size estimates through the use of meta-analytic moderator analyses.

In this workshop, we will learn how to calculate the most common types of effect sizes (the standardized mean difference, correlation coefficient and log-odds ratio) given the different kinds of descriptive and inferential statistics that are reported. We will also learn how to average the effect size estimates across primary studies and how to conduct moderator analyses. Meta-analytic data are complicated and we will cover how best to handle some of the methodological complexities that are encountered. We will also learn how to assess and correct for potential publication bias.

Tasha Beretvas, Ph.D.

Tasha Beretvas is the senior vice provost of faculty affairs and the John L. and Elizabeth G. Hill Centennial professor of quantitative methods in the Educational Psychology department at the University of Texas at Austin. Tasha’s research focuses on the application and evaluation of statistical models in social, behavioral and health sciences research. Read More

• Workshop Details

The goal of the workshop is to cover the core skills needed to conduct a quantitative meta-analysis. While a good portion of the content has to include statistical formulas and models, the teaching will include demonstrations, explanations and interpretations to connect real data and methodological dilemmas with concepts, formulas and models. Because it is not possible for all methodological dilemmas as well as data and research question challenges to be covered, the understanding and skills learned in this class are intended to be generalized to new scenarios encountered by the learners in their future applied meta-analyses.

This workshop is designed for graduate students, post-doctoral fellows, faculty, and research scientists from the behavioral, social, and health sciences. It is recommended that participants have a working knowledge of the general multiple regression model. Relevant core statistical concepts will be reviewed at the beginning of the workshop. Participants who would benefit from a more in-depth refresher on linear regression prior to attending may wish to watch our (no cost) Linear Regression Playlist on YouTube.

Chapter 1. Introduction and Review 1.1 Introduction to Quantitative Meta-Analysis 1.2 Core Statistics Review

Chapter 2. Calculating the Standardized Mean Difference 2.1 Calculating the Standardized Mean Difference and its Variance 2.2 Transforming Inferential Statistics to Obtain the SMD 2.3 Calculating the SMD from repeated measures design data

Chapter 3. Pooling Standardized Mean Differences 3.1 Fixed-Effects Pooling of Effect Size Estimates 3.2 Random-Effects Pooling of Effect Size Estimates 3.3 Demonstration of Pooling SMDs by hand and with escalc

Chapter 4. Meta-Regression Models 4.1 Regression and Meta-Regression 4.2 Using rma to estimate fixed- & mixed-effects meta-regression models 4.3 Meta-Regression Models: An Example

Chapter 5. Handling Within-Study Dependence in SMDs 5.1 Dependent Effects Introduction 5.2 Using GLS to handle within-study dependence & moderation 5.3 Robust Variance Estimation of Meta-Regression Models

Chapter 6. Missing Data and Publication Bias 6.1 Assessing Publication Bias 6.2 Correcting for Publication Bias

Chapter 7. Meta-Analysis of Correlations 7.1 Pooling Correlation Estimates 7.2 Handling Within-Study Dependence in Correlation Estimates 7.3 Testing publication bias in meta-analysis of Correlations

Chapter 8. Meta-Analysis of Treatment Effects on Dichotomous Outcomes 8.1 Categorical Effect Size Measures 8.2 Meta-Regression of Categorical Effect Sizes 8.3 Final Hurrah of Equations and Stuff

Demonstrations of analyses are presented using the R software program because it is the software that is most rapidly and frequently updated with the latest methodological innovations in meta-analysis. Where possible, supplemental materials demonstrating use of SPSS statistical software are provided. Note that R can be downloaded for free .  While it is helpful to have some familiarity with R, it is not necessary.  The lectures which constitute the majority of the workshop are software-independent.  Note that while code will be shared for all analyses demonstrated in the workshop, the majority of the pedagogy will focus on the concepts and content rather than use of software.

Introduction to Quantitative Meta-Analysis is a four-day workshop originally taught live via Zoom by Tasha Beretvas. Daily lectures were held from 9:00 to 5:00 with morning, lunch, and afternoon breaks. Sessions consisted of comprehensive lectures, detailed presentation of real-data examples with live demonstrations in R, response to participant questions, and general discussion.

Self-paced participants receive  lifetime   access to all course materials, including complete course notes, lecture recordings, software demonstration notes, and data and code for all examples. You can revisit these materials any time you like, without worrying about expiration dates. Pretty awesome, huh?

Full recordings of all lectures and software demonstrations are provided.  You can log in to your account to access these recordings (Select My Workshops, then click on the corresponding workshop tile).

Please see the  sample videos and materials  we have posted for a subset of classes. Each class provides unique content but the format and style is similar across classes.

We offer reduced-price registrations for undergraduate and graduate students who are actively enrolled in a recognized bachelor's, master's or doctoral training program. No application is necessary to qualify for the student tuition rates; simply choose the student rate when beginning the registration process at the top of the page. Confirmation of student status may be requested at a later time.

Dr. Beretvas is the epitome of teaching excellence – she is extremely knowledgeable in her area and is able to explain complex concepts to learners with a variety of backgrounds and clear cares about teaching.

I was very, very impressed with Dr. Beretvas' dedication to teaching us. This class really covered an enormous amount, and she provided incredibly detailed and accessible material with the online format.

I cannot say enough good things about Tasha's teaching and this course. Everything was well organized and clearly presented.

Tasha goes to great lengths to make information accessible to quantitative methods and non-quantitative methods students. Her notes are detailed and organized and her preparation for each class is extensive. It is a joy to learn from someone as enthusiastic and skilled as Tasha!

Tasha was a super fun and entertaining instructor. I loved content and the way she broke down concepts in ways that were easy to understand.

Dr. Beretvas continues to be one of the best professors I have ever had. She was always very engaging during her lectures.

Dr. Beretvas was very knowledgeable about course content. She was very thorough and made learning difficult concepts manageable and fun. 

Quick Navigation

• Instructors

Self-Paced Workshops

View Upcoming Livestream Workshops

Free Introduction to Structural Equation Modeling

Length: Three Days Instructors: Dan Bauer & Patrick Curran Lecture Recordings: Lifetime Access Evergreen Content: Materials Continually Updated

Mixture Modeling and Latent Class Analysis

Length: Five Days Instructors: Dan Bauer & Doug Steinley Lifetime Access: No expirations

Applied Measurement Modeling

Length: Four Days Instructors: Patrick Curran & Greg Hancock Lifetime Access: No expirations

Causal Inference

Length: Five Days Instructor: Doug Steinley Lifetime Access: No expirations

Multilevel Modeling

Length: Five Days Instructors: Dan Bauer & Patrick Curran Lifetime Access: No expirations

Applied Qualitative Research

Length: Five Days Instructors: Greg Guest & Emily Namey Lifetime Access: No expirations

Modern Missing Data Analysis

Length: Three Days Instructor: Craig Enders Lifetime Access: No expirations

Machine Learning: Theory and Applications

Sample size planning for power and accuracy.

Length: Three Days Instructor: Samantha Anderson Lifetime Access: No expirations

Network Analysis

Applied research design using mixed methods.

Length: Two Days Instructor: Greg Guest Lifetime Access: No expirations

Introduction to Data Visualization in R

Length: Four Days Instructor: Michael Hallquist Lifetime Access: No expirations

Longitudinal Structural Equation Modeling

Analyzing intensive longitudinal data.

Length: Five Days Instructors: J-P Laurenceau & Niall Bolger Lifetime Access: No expirations