research article 2018

Meta-research: Why research on research matters

* E-mail: [email protected]

Affiliations Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, United States of America, Department of Medicine, Department of Health Research and Policy, and Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, United States of America, Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, United States of America

• John P. A. Ioannidis

Published: March 13, 2018

• https://doi.org/10.1371/journal.pbio.2005468
• Reader Comments

Meta-research is the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives. Given that science is the key driver of human progress, improving the efficiency of scientific investigation and yielding more credible and more useful research results can translate to major benefits. The research enterprise grows very fast. Both new opportunities for knowledge and innovation and new threats to validity and scientific integrity emerge. Old biases abound, and new ones continuously appear as novel disciplines emerge with different standards and challenges. Meta-research uses an interdisciplinary approach to study, promote, and defend robust science. Major disruptions are likely to happen in the way we pursue scientific investigation, and it is important to ensure that these disruptions are evidence based.

Citation: Ioannidis JPA (2018) Meta-research: Why research on research matters. PLoS Biol 16(3): e2005468. https://doi.org/10.1371/journal.pbio.2005468

Copyright: © 2018 Ioannidis John P. A.. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: Laura and John Arnold Foundation. The Meta-Research Innovation Center at Stanford (METRICS) has been funded by the Laura and John Arnold Foundation. The work of John Ioannidis is funded by an unrestricted gift from Sue and Bob O’Donnell. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: NIH, National Institutes of Health; R&D, Research and Development; STEM, Science, Technology, Engineering, and Math

Provenance: Commissioned; not externally peer reviewed

Science, like all human endeavors, is prone to biases. Yet science can assess its own methods, reporting, reproducibility, evaluation, and incentives [ 1 ]. A relatively new discipline, called meta-research, covers a wide range of theoretical, observational, and experimental investigations designed to study research itself and its practices. The objective is to understand and improve how we perform, communicate, verify, evaluate, and reward research [ 1 ].

Before elaborating on a discipline that studies biases, I should disclose some of my own. First, all scientists are meta-researchers to some extent, though most usually work on focused subject matter disciplines. And though the advice of my early lab mentors—“focus, focus, focus”—still rings in my ears, the piles on my desk and the files in my computers can be notoriously unfocused. I don’t have attention-deficit disorder, but plain unconstrained curiosity. What attracted me to science was its vastness and diversity. In my early training years, I enjoyed roaming in libraries in Athens and Boston, discovering scientific journals with fancy names, encountering intriguing articles, drifting from my initial search. Without yet realizing it, I was interested primarily in research itself apparently, much as others were interested primarily in Caenorhabditis elegans , volcanic eruptions, or automata.

Science and its literature is a marvelous maze of data, arguments, biases, errors, and the greatest achievements of humans. What can be more rewarding to study scientifically? Thirty years later, I still feel like a researcher-in-training—actually, in early training—barely scratching the surface. However, much has changed. Thirty years ago, articles had to be handpicked like flowers one by one from their journal shelves and photocopied one page at a time. Now, one can text mine a million articles overnight. Good research, however, still takes time and focus. Take, for example, a recent project I worked on with my friend David Chavalarias. We text mined 12,821,790 abstracts and 843,884 full-text articles. We initially joked that it would take two days max. Eventually, it took four years of work with innumerable iterations, meticulous corrections, and repeated downloads.

My other personal bias is a heightened interest in methods rather than results. Result narratives are supposedly always exciting. I find them unbearably boring. Conversely, methods typically are missing in action, left unsung, or hidden in small print. Many researchers hope to clarify how to do experiments chatting in corridors or conferences. Study design and analysis are still mostly taught (if at all) in statistics-lite courses. Most of us have mastered how to write papers through reading other (mostly poorly reported) papers. We freely volunteer peer review but lack formal training on how to do it. In many fields, issues surrounding reproducibility were dormant until recently.

Science remains the key driver of human progress, yet we have little evidence on how to best fund science and incentivize high-quality work. We do know that leaving research practices to serendipity, biasing influences, methodological illiteracy, and statistical innumeracy is inefficient. Science needs science to avoid wasted effort and optimize resources. Amateur approaches face the current gigantic magnitudes of the research endeavor. Google Scholar currently includes about 180,000,000 documents, accruing approximately 4,000,000 new papers annually [ 2 ]. Along this universe of visible (published) matter, dark matter abounds; probably most observations and data analyses remain unpublished. Ulrich’s directory includes more than 40,000 refereed academic journals, and this is probably an underestimate [ 3 ]. Thousands of journals follow predatory practices or have uncertain value. The Science, Technology, Engineering, and Math (STEM) publishing business market size ($28 billion) roughly equals the National Institutes of Health (NIH) budget. Webometrics lists 26,368 research-producing universities [ 4 ], and many other entities generate research. Probably 100,000 biomedical conferences happen annually [ 5 ]. Global Research and Development (R&D) investment recently exceeded $2 trillion per year. Industry has the lion’s share, while public funding is limited for basic research and it is even more sparse for evidence-based evaluation research. Financial conflicts may shape research agendas, results, and interpretations [ 6 ]. Consider that the $1 trillion tobacco industry still runs “research” on its products despite killing millions of people who use them as directed. Big Pharma, another behemoth of similar financial magnitude, but which probably saves lives (albeit often at high cost), has to sponsor most research on its own products. Understanding who should do what and how in research needs better study.

Science is no longer the occupation of few intellectual dilettanti. Millions (co)author scientific papers. Even more people participate in research. Currently, health record databases engulf hundreds of millions of individuals. Social media databases generate the possibility of using data on billions—active monthly Facebook users, for example, exceeded 2 billion by July 2017.

Currently, generated research data are massive but also fragmented and often nontransparent. Full data sharing and preregistration of protocols are still uncommon in most fields [ 7 ]. We need to understand whether results and inferences are correct, modestly biased, or plain wrong. Comparing patterns of data and biases across the vast number of available studies, one can help answer this important question [ 8 ]. We have mapped 235 biases in biomedical research alone [ 9 ]. With increasing research complexity, multifarious choices emerge on how to design studies and analyze data. With 20 binary choices, 2 20 = 1,048,576 different ways exist to analyze the same data. Therefore, almost any result is possible, unless we safeguard methods and analysis standards. Surveys show that questionable research practices are used by most scientists: not fraud (which is rare) but “cutting corners” to achieve more interesting-looking results [ 10 ]. Understanding the boundaries between bias and creative exploration is important. Efforts to reproduce high-profile studies have shown high rates of nonreproducibility [ 11 ] and most scientists agree that a reproducibility crisis exists [ 12 ]. Meta-analyses—efforts to combine all data on a given question—become increasingly popular but face their own problems and biases [ 13 ].

How should a scientist best train, work, collaborate, and contribute to scientific and broader communities? Researchers spend most of their time on grants [ 14 ] and administrative chores of unclear utility. Journal peer review takes another 64 million hours annually for biomedical papers alone [ 15 ]. Justifiably, we all despise bureaucracy and obstructions. Poor research practices make things worse.

Thousands of new scientific fields emerge, merge, split, and evolve [ 16 ]. Different disciplines may differ in research standards and challenges ( Box 1 ). Meta-research can help us disseminate efficient research practices and abandon wasteful ones. Publication and peer review models, scientific education, funding, and academic reward systems need to adapt successfully to a rapidly changing world. Some predict [ 17 ] that even researchers may disappear within decades, replaced by artificial intelligence. While this sounds extreme, several aspects of current “business as usual” in research will face disruption. Even 1% improvement in the yield and translation of useful discoveries effected through better research practices reflects value equivalent of many Nobel or Breakthrough prizes.

Box 1. Features of research practices, opportunities, and threats that vary across fields.

• ◦ Type of mix of research (basic, applied translational, evaluation, implementation)
• ◦ Types of study designs commonly used or misused
• ◦ Types of experimental/measurement tools commonly used or misused
• ◦ Types of statistical methods commonly used or misused
• ◦ Types of common biases encountered and whether they are easy to fix or not
• ◦ Extent of use of methods to prevent or correct for biases
• ◦ Prevalence of different types of questionable/detrimental research practices
• ◦ Distribution of effect sizes observed
• ◦ Typical heterogeneity of results across studies
• ◦ Proportion of results that are true, exaggerated, or entirely false
• ◦ Reputational impact for bias or wrong, refuted results
• ◦ Proportion of studies and analyses that are published
• ◦ Number and types of available publication venues
• ◦ Implementation of prepublication peer review (e.g., preprints)
• ◦ Implementation of postpublication peer review
• ◦ Extent from adoption of various research reporting standards
• ◦ Commonly accepted authorship and contributorship norms
• ◦ Extent of adoption of team science and consortia
• ◦ Type of training for scientists in the field
• ◦ Extent of methodological and statistical literacy/numeracy
• ◦ Extent and enforcement of preregistration of protocols
• ◦ Extent of use of replication studies
• ◦ Extent of use of exact replication versus corroboration or triangulation
• ◦ Extent of sharing of primary raw data and/or processed data
• ◦ Extent of sharing of software and code
• ◦ Extent and types of evidence synthesis used
• ◦ Main funders (government, industry, other) and types of studies that they fund
• ◦ Project-based versus person-based funding
• ◦ Mix and interplay of institutions performing research (university, industry, other)
• ◦ Types of metrics and criteria used for assessing researchers and institutions
• ◦ Typical conflicts of interest operating in the field
• ◦ Completeness of disclosure of conflicts of interest
• ◦ Extent and fidelity of dissemination of research findings to the general public
• ◦ Extent of public misperceptions about the field
• ◦ Threats from antiscience advocates attacking the field

Meta-research is interdisciplinary. For example, it benefits from better tools and methods in statistics and informatics. Complex issues of behavior change converge on modeling, psychology, sociology, and behavioral economics. Newly introduced, sophisticated measurement tools and techniques in various disciplines introduce new, peculiar errors and biases; their understanding requires combining expertise in biology, bioengineering, and data sciences. Properly communicating science and its value requires combining expertise in multiple fields and has become increasingly critical nowadays, when mistrust of science runs high and multiple interests hold a stake in influencing research results. Some interests set out to manipulate science and cause damage when their intentional bias pollutes the scientific record (e.g., tobacco companies or climate change deniers). Meta-research may be our best chance to defend science, gain public support for research, and counter antiscience movements. It may help provide a correcting mechanism closer to real time than the self-correcting scientific process that otherwise may take much longer.

Moreover, bird’s-eye metaviews of science are not separate and detached from focused field-specific research. In my experience, inspiration for new projects has often come from mistakes, shortcomings, or difficulties that I encountered while doing field-specific research. It is sometimes difficult to convey a message that something is wrong. However, it is paradoxically easier when the message says that thousands or millions of papers are doing something wrong rather than arousing personal animosity for a single failed paper. It is also easier when the constructive critique comes from within a field, recognized as necessary improvement rather than intrusion. Learning by collaborating with researchers in diverse disciplines and trying to understand the daily challenges in a specific field can be a highly rewarding experience for a meta-researcher. We need scientific curiosity but also intellectual humility and commitment to improve our efforts.

• View Article
• PubMed/NCBI
• Google Scholar
• 4. Webometrics. List of universities (as of January 2017). [Cited 21 January 2018]. Available from: http://www.webometrics.info/en/node/54 .

• Open supplemental data
• Reference Manager
• Simple TEXT file

People also looked at

Original research article, changing students minds and achievement in mathematics: the impact of a free online student course.

• 1 Graduate School of Education, Stanford University, Stanford, CA, United States
• 2 School of Education and Counseling Psychology, Santa Clara University, Santa Clara, CA, United States

This study reports on the impact of a “massive, open, online course” (MOOC) designed to change students' ideas about mathematics and their own potential and improve their mathematics achievement. Many students hold damaging fixed mindsets, believing that their intelligence is unchangeable. When students shift to a growth mindset (believing that their intelligence is malleable), their achievement increases. This study of a MOOC intervention differs from previous mindset research in three ways (1) the intervention was delivered through a free online course with the advantage of being scalable nationwide (2) the intervention infused mindset messages into mathematics, specifically targeting students' beliefs about mathematics (3) the research was conducted with a teacher randomized controlled design to estimate its effects. Results show that the treatment group who took the MOOC reported more positive beliefs about math, engaged more deeply in math in class, and achieved at significantly higher levels on standardized mathematics assessments.

Introduction

There are a number of damaging and pervasive myths about mathematics learning in the US that are believed by millions of school children, their parents and their teachers. These different myths hold students back on a daily basis and reduce their learning and achievement significantly ( Boaler, 2016 ). One of the most damaging is the idea that some people are born with a “math brain” and some are not, and that high achievement is only available to some students. Two areas of research are important in challenging this myth, and improving student learning. First, recent neuroscience showing the plasticity of the brain, revealing that brains can grow and change ( Maguire et al., 2000 ). Second, research on mindset showing that when people change their ideas about the malleability of their potential, from “fixed” (my ability is not changeable) to “growth” (my ability changes as I learn) their learning and achievement improves ( Dweck, 2006 ). Different studies, pioneered by Carol Dweck, have shown that students with a growth mindset achieve at higher levels than those with a fixed mindset ( Blackwell et al., 2007 ; Claro et al., 2016 ) and that when students change their mindset their achievement changes ( Aronson et al., 2002 ; Good et al., 2003 ). A second damaging myth is the idea that mathematics learning is all about procedures and memorization, rather than ideas, concepts, and creativity. Research shows that students who approach mathematics as a subject of memorization are lower achieving than those who approach it as a subject of ideas that they can think deeply about ( Boaler and Zoido, 2016 ). A third myth that students believe is that good mathematics students have to be fast when some of the world's leading mathematicians are slow thinkers ( Boaler, 2016 ). This study examines the impact of a “massive open online course” (MOOC) for students centered on changing these ideas and teaching students how to learn mathematics well.

The MOOC includes six modules, each of which takes 15–20 min to complete. The teacher of the course is the lead author, Jo Boaler, professor of mathematics education at Stanford, accompanied by some of her undergraduate students. Some of the key ideas in the course are:

• Everyone can learn mathematics to high levels

• Mistakes, challenge and struggle are the best times for brain growth

• Depth of thinking is more important than speed

• Mathematics is a creative and beautiful subject

• Good strategies for learning mathematics including talking and drawing

• Mathematics is all around us in life and is important—this was shown by different undergraduates showing mathematics in soccer, nature, juggling, and dance.

The course includes a series of short videos interspersed with opportunities for students to reflect on the ideas, connect with other students in the course, and work on open-ended mathematics tasks designed to shape students' perceptions related to these core ideas. (see Supplemental Note for more information about the online course.)

This paper describes the results of a randomized controlled trial (RCT) which examined the impact of the course on middle school students' engagement in mathematics class, their beliefs and mindset, and their academic achievement on state tests—the Smarter Balanced Assessment Consortium (SBAC) Summative Assessment. The SBAC assessments determine students' progress toward college and career readiness in English language arts/literacy and mathematics. These are given at the end of the school year and consist of two parts: a computer adaptive test and a performance task.

Research into the impact of free online classes—or MOOCs—has shown disappointing results with the early promise of equitable access to education being replaced with a harsh reality of low finishing rates and a predominance of privileged learners ( Hansen and Reich, 2015 ). This study gives a very different result, showing that a strategically designed course, with careful considerations to access, significantly impacted students' mathematics learning pathways and subsequent achievement, regardless of students' gender, ethnicity, language learning level, or wealth.

California school districts were recruited through a variety of announcements at conferences and workshops. School districts that were willing to provide data on the impact of the course were admitted. This study was carried out in accordance with the recommendations of Stanford University Research Compliance Office. The protocol was approved by the Stanford Graduate School of Education Institutional Review Board. All subjects gave written informed consent in accordance with the Declaration of Helsinki. Our analysis shows results from four school districts in California, with 1,090 students enrolled in 10 different middle schools across four districts. There were 439 students who took the online class, and 651 students who were control students. There were 14 teachers in this sample. Tables 1 , 2 provide additional descriptive statistics about the sample.

Table 1 . Observations used in the study.

Table 2 . Baseline traits description.

Using a delayed-treatment research design that enabled randomization of students without the constraint of certain students getting access to a helpful course while their classmates did not, we recruited middle school teachers who taught at least 2 classes of 6th, 7th or 8th grade mathematics. For each teacher half of their classes were randomly assigned to the treatment group and half to the control group. Students in the treatment and control conditions were taught by the same teachers, thus controlling for teacher characteristics. Classes assigned to the treatment group took the online class in the first few months of the school year. Students who completed at least 4 of the modules were considered as having received the treatment.

Table 3 provides a project timeline of key activities. The students in the control group were given access to the course at the conclusion of the study.

Table 3 . Project timeline.

Using Ordinary Least Squares (OLS) regression controlling for baseline differences (gender, ethnicity, free and reduced lunch status, English language learner (ELL) status, and special education status) we tested for the effects of the intervention. Summary descriptive statistics for SBAC measures are provided in Table S1.

We found a treatment effect indicating that MOOC participants obtained higher scores in their SBAC math overall scale score, overall proficiency levels, and concepts and procedures ( Smarter Balanced Assessment Consortium, 2013 ). In fact, students who receive the treatment obtained 0.33 standard deviation gains in SBAC math overall scale score; i.e., the average student in the treatment group would score higher than 63% of the control group that was initially equivalent (see Tables 4 , 5 ). The subscales of the SBAC test were also significantly higher for the treatment group. More details on the model specifications is given in Table S2. In addition, further analyses show a positive and significant treatment effect for student subgroups defined by ethnicity, gender, economic disadvantage status, ELL status, special education status, and school grade (see Table S3).

Table 4 . Regression estimates, MOOC effect on SBAC math scores.

Table 5 . Standard deviations gains on SBAC math scores.

The strong design offered by the RCT performed in this study was partially offset by data access constraints. At different stages of the data collection, some schools that were part of the original study design provided incomplete data on their students. The first call for participants yielded 193 teachers who expressed interest and participated in the initial orientation to the project. On the first survey measure in August 2014, 73 teachers and 6,727 students responded in 27 school districts. By December 2014, the sample size had reduced to 31 teachers and 1,645 students in 10 school districts. The attrition in the study was largely due to technical difficulties at the school level–students needed an email address and password to take the online class, which many districts could not provide. Some schools also reported being unable to access the online course from their classrooms because of district firewall security settings that could not be resolved in the timeframe of the study. Further attrition occurred when some districts did not provide full data from state tests, usually because of staff capacity.

Importantly, the attrition was not systematic and was not linked to the outcome variables. In fact, attrition can introduce selection bias in randomized trials so this was investigated fully, as explained below. The most crucial internal validity concern when estimating causal effects is the assumption that students' assignment to treatment and control condition is random. Under this assumption, the estimates are valid if students' baseline traits are statistically similar for treatment and control students. Table 6 validates this assumption by examining whether students' traits vary with treatment/control condition (Table S4 includes complete regression information). Each point estimate is from a separate regression where each baseline student's covariate (i.e., gender, ethnicity, economic disadvantage status, limited English proficiency, and special education) is the dependent variable. The estimated effect of treatment status on these covariates is small and statistically insignificant, suggesting that students' baseline traits are statistically similar for both treated and control students. This validity check shows that the treatment and control groups are comparable and equivalent at baseline. In other words, treatment and comparison groups are statistically equivalent in every observable aspect except for the intervention, ruling out the threat of selection bias.

Table 6 . Auxiliary regressions of baseline covariate balance.

Changes in Student Classroom Engagement

To understand possible mechanisms for the improved academic achievement treatment effect, the study also examined student engagement and beliefs related to mathematics teaching and learning. Participating teachers were asked to evaluate students' engagement, before and after students took the online class, in both their treatment and control classrooms. Teachers observed students along four dimensions of engagement: (a) student participates in class discussions, (b) student works as hard s/he can, (c) student appears to be involved in classwork, and (d) student gives up quickly. Table 7 indicates the two practices which showed significant differences in how students participated in class between control and treatment classes.

Table 7 . Outcomes and baseline traits of students in the randomized controlled trial MOOC.

The study of student engagement demanded a lot of teacher time, and only 4 of 14 teachers returned full data on the students' engagement in math class. The data from this subset show significant effects for students who took the online course. The effect size of the treatment on student engagement was 0.47 SD (see last row of Table 5 ), meaning that the average student in the treatment group would score higher on the engagement scale than 68% of the control group accounting for baseline differences (see Pre/post-gains in student engagement, the last column of Table 8 for the regression estimates). Students in the treatment group participated more in class discussions and did not give up on work as quickly as their counterparts in the control classes. These findings provide insight into the reasons that students in the treatment group achieved at significantly higher levels on state mathematics tests. One compromise in our design is that because we used a “within-teacher” design, teachers were aware of which of their classes were designated as control and treatment, thus posing a potential threat to the validity of this measure. We include the engagement gain measure as a mediator variable for the positive treatment effect on standardized test introduced earlier. Our aim in using this measure was to explore possible factors through which students' beliefs about math may have resulted in deeper forms of classroom engagement, helping to explain increases in student achievement. Future studies will include student engagement surveys that do not rely on teacher reports, which will strengthen our design.

Table 8 . Regression estimates, MOOC effect on student mindset surveys.

Changes in Student Mindset

A pre- and post-survey, measuring shifts in students' beliefs, was completed by 156 students and provides further insight into students' increased academic achievement. (These numbers are low as although 1,090 students took surveys, only 156 students took both the pre and the post-survey). Despite the response rate of 14%, Table S5 shows that the subset of students who completed the pre and post-survey were, in fact, representative of the larger sample group of 1,090 students.

There was a significant treatment effect on three student beliefs (see Table 8 for regression estimates and Figure 1 which compares treatment and control group survey responses). Students in the treatment had significantly higher reports of growth mindset (Mindset) and their perceptions of mathematics being an interesting and creative subject (Math Creative). They also reported feeling less fearful or easily deterred in math (Fear of Math). The specific survey items for each cluster and alpha levels are given in Table 9 .

Figure 1 . Student mindset survey scores and effect size by group.

Table 9 . Measures of student mindset survey (cluster items with alpha levels) 1 = Strongly disagree, 2 = Disagree, 3 = Somewhat disagree, 4 = Somewhat agree, 5 = Agree, 6 = Strongly agree.

The significant changes in engagement and beliefs that students showed is likely to explain, at least partly, the increase in students' mathematics achievement. This finding supports a growing body of work that shows a link between students' mindsets about their potential and their ideas about mathematics. It is difficult to maintain a growth mindset and the idea you can learn any mathematics when the subject is presented as a series of short, closed questions—with no space for growth or learning within them. Sun (2015) showed that students developed more growth mindsets when teachers presented mathematics as subject with more opportunities for growth and learning, as opposed to performing and answering questions. The finding that students in this study shifted in seeing mathematics as a more creative subject, as well as developing more growth mindsets, supports this important link (see also Boaler, 2016 ).

Consistent with previous research, this study finds a significant connection between students' mindset and their learning outcomes ( Mueller and Dweck, 1998 ). Students in the treatment group reported more growth mindset beliefs and more challenge-seeking behaviors than those in the control group. What is distinctive about this study is the impact of an online class in changing students' mindsets toward mathematics, with subsequent changes in student achievement. Much of the research on mindset has focused on changing students' mindsets outside of any content teaching and learning; by contrast this study examines an intervention that combines mindset with changed views of mathematics and mathematical engagement. This study shows that an intervention addressing the intersection of mindset and mathematics can improve students' academic achievement, as well as students' behavior and beliefs about mathematics.

These findings are particularly important in light of continued concerns with US mathematics achievement. In the most recent international comparisons students in the US ranked 40th out of 72 countries ( OECD, 2016 ). This is an issue that has prevailed for decades despite a vast body of research that has shown the ways to teach mathematics well ( Schoenfeld, 2002 ; Boaler, 2015 , 2016 ). Low mathematics achievement is not the only problem that faces the US—math anxiety is widespread among school children and the general population ( Ashcraft and Krause, 2007 ; Foley et al., 2017 ). Most of the attention that is given to this issue considers the curriculum standards and textbooks used in classrooms. While these issues are important they may not be more important than a completely neglected issue—the fact that most students sit in mathematics classrooms, from kindergarten to University, thinking “I am not a math person.” In addition to this damaging belief, few students have learned to approach mathematics as a conceptual domain, rather than a set of procedures. The evidence from this randomized control trial shows the academic impact of changing these beliefs and approaches for students.

We acknowledge the limitations of our study. Our sample was drawn from middle school students in one state, and so additional studies with wider grade spans and in more varied geographical areas would be needed to generalize more broadly. In addition, for the student engagement measures, we relied on teachers' classroom observations of students. Our study would have been strengthened if we had also included student engagement surveys.

Most of the research on MOOCS has portrayed disappointing results—with online classes having low retention and perpetuating the inequities of open access that MOOCs were originally aimed to challenge ( 9 ). The online class that was the subject of this study had a different outcome of students continuing the course and significantly improving their beliefs and achievement, regardless of students' gender, ethnicity, language learning level, or wealth. The fact that this MOOC was used as part of an educational intervention and administered by teachers is part of the reason for the students continued participation. Another, we contend is the pedagogy of active engagement inside the course. Most MOOC's are lecture based, which would likely have been ineffective, even inside a classroom setting. In the “How to Learn Math” course students were invited to engage every few minutes, through answering questions, commenting on videos, and interacting with others. As MOOCs are developed and refined over the next few decades, it seems that an important advancement will be the inclusion of opportunities for more active engagement.

Many school students in the US and world are held back by damaging ideas about learning and their potential—particularly in mathematics. There is a widespread myth that students are either born with a math brain or they are not, and when students struggle they often decide they are just not a math person. The RCT that is the focus of this study has shown that students can be liberated from these damaging ideas and when they are it improves their participation and achievement. Online courses for teachers that also focus on mindset messages, and ideas for teaching mathematics actively, have also been shown to change students' achievement and beliefs (Anderson et al., under review). Together these studies reveal the importance of changing the mindsets of teachers and students, in order that students can learn mathematics without being held back by damaging beliefs. They also show the potential of online courses - which have great scalability and wide-scale access—as effective teaching opportunities, bringing some of the best teachers and the most cutting-edge research to the students who most need it.

Author Contributions

JB designed and directed the study, CW led recruitment and implementation, GP-N and KS ran analyses, the full team interpreted results, JD and GP-N contributed to and managed the writing and revising process among all team members.

This study was funded by National Science Foundation (NSF), Research on Education and Learning (REAL), Award Number 1443790. JB, Principal Investigator, Stanford University.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2018.00026/full#supplementary-material

Supplementary Note on Online Student Course

The online class that was the focus of this study has now been taken by over 160,000 participants—students of mathematics of all levels from elementary school to college. It has also been taken by tens of thousands of teachers and parents as both sets of adults are helped by knowledge of the latest research on ways to learn mathematics. In addition to individuals taking the course teachers of students as young as 5 have shared the videos with their students. The class is free and can be taken at any time and at any pace. Students can take the class in their school class, as students in the study did, or at home. The modular nature of the course has enabled teachers to use the course in a variety of ways: using the course in summer school, as a way to launch the school year, or infused throughout the year.

The class, which is also available with Spanish sub-titles, is open to anyone with an internet connection. The ideas from the class are also disseminated in different forms including papers, videos and mathematics curriculum materials on youcubed.org, a Stanford center and accompanying website of almost entirely free resources. Accompanying teacher courses on ways to teach mathematics well are also available.

Aronson, J., Fried, C. B., and Good, C. (2002). Reducing the effects of stereotype threat on African American college students by shaping theories of intelligence. J. Appl. Dev. Psychol. 38, 113–125. doi: 10.1006/jesp.2001.1491

CrossRef Full Text | Google Scholar

Ashcraft, M. H., and Krause, J. A. (2007). Working memory, math performance, and math anxiety. Psychon. Bull. Rev. 14, 243–248. doi: 10.3758/BF03194059

PubMed Abstract | CrossRef Full Text | Google Scholar

Blackwell, L. S., Trzesniewski, K. H., and Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and an intervention. Child Dev. 78, 246–263. doi: 10.1111/j.1467-8624.2007.00995.x

Boaler, J. (2015). What's Math Got to Do with It? How Teachers and Parents Can Transform Mathematics Learning and Inspire Success. New York, NY: Penguin.

Boaler, J. (2016). Mathematical Mindsets: Unleashing Students' Potential Through Creative Math, Inspiring Messages and Innovative Teaching . San Francisco, CA: John Wiley & Sons.

Google Scholar

Boaler, J., and Zoido, P. (2016). Why math education in the U.S. doesn't add up. Sci . Am. Mind. 27, 18–19. doi: 10.1038/scientificamericanmind1116-18

Claro, S., Paunesku, D., and Dweck, C. S. (2016). Growth mindset tempers the effects of poverty on academic achievement. Proc. Natl. Acad. Sci. U.S.A. 113, 8664–8668. doi: 10.1073/pnas.1608207113

Dweck, C. S. (2006). Mindset: The New Psychology of Success . New York, NY: Random House Incorporated.

Foley, A. E., Herts, J. B., Borgonovi, F., Guerriero, S., Levine, S. C., and Beilock, S. L. (2017). The math anxiety-performance link: a global phenomenon. Curr. Dir. Psychol. Sci. 26, 52–58. doi: 10.1177/0963721416672463

Good, C., Aronson, J., and Inzlicht, M. (2003). Improving adolescents' standardized test performance: an intervention to reduce the effects of stereotype threat. J. Appl. Dev. Psychol. 24, 645–662. doi: 10.1016/j.appdev.2003.09.002

Hansen, J. D., and Reich, J. (2015). Democratizing education? Examining access and usage patterns in massive open online courses. Science 350, 1245–1248. doi: 10.1126/science.aab3782

Maguire, E. A., Gadian, D. G., Johnsrude, I. S., Good, C. D., Ashburner, J., Frackowiak, R. S., et al. (2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc. Natl. Acad. Sci. U.S.A. 97, 4398–4403. doi: 10.1073/pnas.070039597

Mueller, C. M., and Dweck, C. S. (1998). Praise for intelligence can undermine children's motivation and performance. J. Pers. Soc. Psychol. 75:33. doi: 10.1037/0022-3514.75.1.33

OECD. (2016). PISA 2015 Results (Volume I): Excellence and Equity in Education. Paris: OECD Publishing.

Schoenfeld, A. H. (2002). Making mathematics work for all children: issues of standards, testing, and equity. Educ. Res. 31, 13–25. doi: 10.3102/0013189X031001013

Smarter Balanced Assessment Consortium (2013). Initial Achievement Level Descriptors and College Content-readiness Policy . Smarter Balanced Assessment Consortium. Available online at: https://portal.smarterbalanced.org/library/en/mathematics-alds-and-college-content-readiness-policy.pdf

Sun, K. L. (2015). There's No Limit: Mathematics Teaching for a Growth Mindset. Doctoral dissertation, Stanford University, Stanford, CA.

Keywords: growth mindset, mathematical mindset, MOOC, math achievement, student beliefs, student engagement, randomized control trial

Citation: Boaler J, Dieckmann JA, Pérez-Núñez G, Sun KL and Williams C (2018) Changing Students Minds and Achievement in Mathematics: The Impact of a Free Online Student Course. Front. Educ . 3:26. doi: 10.3389/feduc.2018.00026

Received: 26 February 2018; Accepted: 11 April 2018; Published: 25 April 2018.

Reviewed by:

Copyright © 2018 Boaler, Dieckmann, Pérez-Núñez, Sun and Williams. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Jo Boaler, [email protected] Jack A. Dieckmann, [email protected]

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

• Social Media Use in 2018

A majority of Americans use Facebook and YouTube, but young adults are especially heavy users of Snapchat and Instagram

Table of contents.

• Acknowledgments
• Methodology
• Appendix A: Detailed table

For the latest survey data on social media and messaging app, see “ Social Media Use in 2021 . ”

A new Pew Research Center survey of U.S. adults finds that the social media landscape in early 2018 is defined by a mix of long-standing trends and newly emerging narratives.

Facebook and YouTube dominate this landscape, as notable majorities of U.S. adults use each of these sites. At the same time, younger Americans (especially those ages 18 to 24) stand out for embracing a variety of platforms and using them frequently. Some 78% of 18- to 24-year-olds use Snapchat, and a sizeable majority of these users (71%) visit the platform multiple times per day. Similarly, 71% of Americans in this age group now use Instagram and close to half (45%) are Twitter users.

As has been the case since the Center began surveying about the use of different social media in 2012, Facebook remains the primary platform for most Americans. Roughly two-thirds of U.S. adults (68%) now report that they are Facebook users, and roughly three-quarters of those users access Facebook on a daily basis. With the exception of those 65 and older, a majority of Americans across a wide range of demographic groups now use Facebook.

But the social media story extends well beyond Facebook. The video-sharing site YouTube – which contains many social elements, even if it is not a traditional social media platform – is now used by nearly three-quarters of U.S. adults and 94% of 18- to 24-year-olds. And the typical (median) American reports that they use three of the eight major platforms that the Center measured in this survey.

These findings also highlight the public’s sometimes conflicting attitudes toward social media. For example, the share of social media users who say these platforms would be hard to give up has increased by 12 percentage points compared with a survey conducted in early 2014. But by the same token, a majority of users (59%) say it would not be hard to stop using these sites, including 29% who say it would not be hard at all to give up social media.

Different social media platforms show varied growth

Facebook remains the most widely used social media platform by a relatively healthy margin: some 68% of U.S. adults are now Facebook users. Other than the video-sharing platform YouTube, none of the other sites or apps measured in this survey are used by more than 40% of Americans.

The Center has asked about the use of five of these platforms (Facebook, Twitter, Instagram, LinkedIn and Pinterest) in several previous surveys of technology use. And for the most part, the share of Americans who use each of these services is similar to what the Center found in its previous survey of social media use conducted in April 2016. The most notable exception is Instagram: 35% of U.S. adults now say they use this platform, an increase of seven percentage points from the 28% who said they did in 2016.

The youngest adults stand out in their social media consumption

As was true in previous Pew Research Center surveys of social media use, there are substantial differences in social media use by age. Some 88% of 18- to 29-year-olds indicate that they use any form of social media. That share falls to 78% among those ages 30 to 49, to 64% among those ages 50 to 64 and to 37% among Americans 65 and older.

At the same time, there are pronounced differences in the use of various social media platforms within the young adult population as well. Americans ages 18 to 24 are substantially more likely to use platforms such as Snapchat, Instagram and Twitter even when compared with those in their mid- to late-20s. These differences are especially notable when it comes to Snapchat: 78% of 18- to 24-year-olds are Snapchat users, but that share falls to 54% among those ages 25 to 29.

With the exception of those 65 and older, Facebook is used by a majority of Americans across a wide range of demographic groups. But other platforms appeal more strongly to certain subsets of the population. In addition to the age-related differences in the use of sites such as Instagram and Snapchat noted above, these are some of the more prominent examples:

• Pinterest remains substantially more popular with women (41% of whom say they use the site) than with men (16%).
• LinkedIn remains especially popular among college graduates and those in high-income households. Some 50% of Americans with a college degree use LinkedIn, compared with just 9% of those with a high school diploma or less.
• The messaging service WhatsApp is popular in Latin America , and this popularity also extends to Latinos in the United States – 49% of Hispanics report that they are WhatsApp users, compared with 14% of whites and 21% of blacks.

For more details on social media platform use by different demographic groups, see Appendix A .

Roughly three-quarters of Facebook users ­– and around six-in-ten Snapchat and Instagram users – visit each site daily

Along with being the most popular social media site, Facebook users also visit the site with high levels of frequency. Fully 74% of Facebook users say they visit the site daily, with around half (51%) saying they do several times a day. The share of Facebook users who visit the site on a daily basis is statistically unchanged compared with 2016, when 76% of Facebook users reported they visited the site daily.

While the overall share of Americans who use Snapchat is smaller than that of Facebook, a similar share of Snapchat users (49%) say they use the platform multiple times per day. All told, a majority of Snapchat (63%) and Instagram (60%) users indicate that they visit these platforms on a daily basis. The share of Instagram users who visit the platform daily has increased slightly since 2016 when 51% of Instagram users were daily visitors. (Note: this is the first year the Center has specifically asked about the frequency of Snapchat use in a telephone poll.)

In addition to adopting Snapchat and Instagram at high rates, the youngest adults also stand out in the frequency with which they use these two platforms. Some 82% of Snapchat users ages 18 to 24 say they use the platform daily, with 71% indicating that they use it multiple times per day. Similarly, 81% of Instagram users in this age group visit the platform on daily basis, with 55% reporting that they do so several times per day.

The median American uses three of these eight social platforms

As was true in previous surveys of social media use, there is a substantial amount of overlap between users of the various sites measured in this survey. Most notably, a significant majority of users of each of these social platforms also indicate that they use Facebook and YouTube. But this “reciprocity” extends to other sites as well. For instance, roughly three-quarters of both Twitter (73%) and Snapchat (77%) users also indicate that they use Instagram.

This overlap is broadly indicative of the fact that many Americans use multiple social platforms. Roughly three-quarters of the public (73%) uses more than one of the eight platforms measured in this survey, and the typical (median) American uses three of these sites. As might be expected, younger adults tend to use a greater variety of social media platforms. The median 18- to 29-year-old uses four of these platforms, but that figure drops to three among 30- to 49-year-olds, to two among 50- to 64-year-olds and to one among those 65 and older.

A majority of social media users say it would not be difficult to give up these sites

Even as a majority of Americans now use social platforms of various kinds, a relatively large share of these users feel that they could give up social media without much difficulty.

Some 59% of social media users think it would not be hard to give up social media, with 29% indicating it would not be hard at all. By contrast, 40% say they would indeed find it hard to give up social media – although just 14% think it would be “very hard” to do this. At the same time, the share of social media users who would find it hard to give up these services has grown somewhat in recent years. The Center asked an identical question in a survey conducted in January 2014, and at that time, 28% of social media users indicated they would have a hard time giving up social media, including 11% who said it would be “very hard.”

These findings vary by age. Roughly half of social media users ages 18 to 24 (51%) say it would be hard to give up social media, but just one-third of users ages 50 and older feel similarly. The data also fit broadly with other findings the Center has collected about Americans’ attitudes toward social media. Despite using them for a wide range of reasons, just 3% of social media users indicate that they have a lot of trust in the information they find on these sites. And relatively few have confidence in these platforms to keep their personal information safe from bad actors.

Sign up for our weekly newsletter

Fresh data delivery Saturday mornings

Sign up for The Briefing

Weekly updates on the world of news & information

• Digital News Landscape
• Emerging Technology
• Platforms & Services
• Social Media
• Technology Adoption

Introducing the Pew-Knight Initiative

Many americans find value in getting news on social media, but concerns about inaccuracy have risen, news platform fact sheet, a profile of the top-ranked podcasts in the u.s., most u.s. journalists are concerned about press freedoms, most popular, report materials.

• Jan. 3-10, 2018 – Core Trends Survey

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 18 December 2018

2018 in news: The science events that shaped the year

• Alison Abbott ,
• Ewen Callaway ,
• Davide Castelvecchi ,
• Holly Else ,
• Elizabeth Gibney ,
• Heidi Ledford ,
• Jane J. Lee ,
• Lauren Morello ,
• Jeff Tollefson &
• Alexandra Witze

You can also search for this author in PubMed   Google Scholar

Climate change is intensifying droughts in Australia. Credit: Brook Mitchell/Getty

A turbulent year marked by raging wildfires and allegations of bullying in the sciences comes to a close. But researchers can celebrate some milestones, including the most accurate map yet of the Milky Way’s stars, and the discovery of bones showing that a woman who lived 90,000 years ago had a Neanderthal mother and a Denisovan dad.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Nature 564 , 314-317 (2018)

doi: https://doi.org/10.1038/d41586-018-07685-3

Reprints and permissions

Related Articles

• Climate change

CRISPR therapy restores some vision to people with blindness

Research Highlight 09 MAY 24

How the cauliflower got its curlicues

Research Highlight 08 MAY 24

Powerful ‘nanopore’ DNA sequencing method tackles proteins too

Technology Feature 08 MAY 24

The US Congress is taking on AI —this computer scientist is helping

News Q&A 09 MAY 24

How I fled bombed Aleppo to continue my career in science

Career Feature 08 MAY 24

Who’s making chips for AI? Chinese manufacturers lag behind US tech giants

News 03 MAY 24

Finding millennia-old ‘monumental’ corals could unlock secrets of climate resilience

Correspondence 07 MAY 24

Countering extreme wildfires with prescribed burning can be counterproductive

Support communities that will lose out in the energy transition

Editorial 01 MAY 24

Recruitment of Principal Investigators by the School of Life Sciences, Peking University

The School of Life Sciences at Peking University is actively seeking talents, with an emphasis on bioinformatics/computational biology/AI/RNA biology.

Beijing (CN)

School of Life Sciences, Peking University

2024 Recruitment notice Shenzhen Institute of Synthetic Biology: Shenzhen, China

The wide-ranging expertise drawing from technical, engineering or science professions...

Shenzhen,China

Shenzhen Institute of Synthetic Biology

Head of Operational Excellence

In this key position, you’ll be responsible for ensuring efficiency and quality in journal workflows through continuous improvement and innovation.

United States (US) - Remote

American Physical Society

Rowland Fellowship

The Rowland Institute at Harvard seeks outstanding early-career experimentalists in all fields of science and engineering.

Cambridge, Massachusetts

Rowland Institute at Harvard

Postdoctoral Fellowship: Chemical and Cell Biology

The 2-year fellowship within a project that will combine biochemical, cell biological and chemical genetic approaches to elucidate migrasome biology

Umeå, Sweden

Umeå University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

UK Edition Change

• UK Politics
• News Videos
• Paris 2024 Olympics
• Rugby Union
• Sport Videos
• John Rentoul
• Mary Dejevsky
• Andrew Grice
• Sean O’Grady
• Photography
• Theatre & Dance
• Culture Videos
• Fitness & Wellbeing
• Food & Drink
• Health & Families
• Royal Family
• Electric Vehicles
• Car Insurance Deals
• Lifestyle Videos
• UK Hotel Reviews
• News & Advice
• Simon Calder
• Australia & New Zealand
• South America
• C. America & Caribbean
• Middle East
• Politics Explained
• News Analysis
• Today’s Edition
• Home & Garden
• Broadband deals
• Fashion & Beauty
• Travel & Outdoors
• Sports & Fitness
• Sustainable Living
• Climate Videos
• Solar Panels
• Behind The Headlines
• On The Ground
• Decomplicated
• You Ask The Questions
• Binge Watch
• Travel Smart
• Watch on your TV
• Crosswords & Puzzles
• Most Commented
• Newsletters
• Ask Me Anything
• Virtual Events
• Betting Sites
• Online Casinos
• Wine Offers

Thank you for registering

Please refresh the page or navigate to another page on the site to be automatically logged in Please refresh your browser to be logged in

Government to announce £40m package for brain tumour research

The funding was first committed to by the government in 2018., article bookmarked.

Find your bookmarks in your Independent Premium section, under my profile

Sign up for the View from Westminster email for expert analysis straight to your inbox

Get our free view from westminster email, thanks for signing up to the view from westminster email.

A £40 million Government package to help develop new life-saving brain tumour research is due to be announced.

The funding, which was first committed to by the Government in 2018, will be announced at a Parliamentary roundtable co-chaired by Health Minister Andrew Stephenson and chief executive of the National Institute for Health and Care Research (NIHR) Lucy Chappell.

The investment will be used to help develop new treatments for brain tumours, as well as improve patient care, support and rehabilitation.

It comes amid a collaboration between the Government, NIHR, the Tessa Jowell Brain Cancer Mission (TJBCM), research funders and other charities.

Brain tumours remain one of the hardest to treat cancers, with just 12% adults surviving beyond five years after a diagnosis of a brain tumour.

In 2018, the late Dame Tessa Jowell led calls on behalf of all patients to tackle brain tumours.

A new national strategy was designed in response and the Government committed £40 million for new research.

The roundtable comes during the week of the sixth anniversary of Dame Tessa’s death in May 2018.

Jess Mills, daughter of Dame Tessa and chief executive of the Tessa Jowell Foundation, welcomed the announcement but stressed the need for “no more delays”.

She said: “We are meeting today almost six years to the day that my mum Tessa Jowell died from Glioblastoma.

“Six years on, brain cancer is still the biggest cancer killer of children and under 40s, the need for patients to gain access to new and better treatments and care is as acute as ever.

“However, what is different now compared to then, is we at the TJBCM have spent six years building a thriving brain tumour community which is ready to transform brain cancer outcomes together.

“Almost six years to the day that the £40 million was first committed, it is imperative now that the money is made available with the urgency that this community deserves and there are no more delays.

“With this transformational investment we could unlock the potential of the cutting edge of treatment of care, which is precision medicine in every corner of the UK.”

Professor Lucy Chappell, chief executive at the NIHR, said: “This transformative brain tumour research funding we are announcing is a key moment in our search for novel therapies and better treatments to save lives and improve the quality of life for patients with this condition.

“We are pressing ahead in this innovative new step, made possible due to our strong and collaborative partnership with charities, patients, the life sciences industry and the brain tumour community.

“As we continue this journey together, it shows the crucial value of world-leading research shaped and funded by the public, integrated across the health and care system.”

Health and Secondary Care Minister Andrew Stephenson added: “Six years after Tessa Jowell’s death, I continue to be inspired by her campaign.

“Brain cancer is a dreadful disease, but this latest package of research and funding, developed in partnership with the brain tumour community will help accelerate improvements in treatment and care, so that we can beat this condition and save lives.”

Subscribe to Independent Premium to bookmark this article

Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Premium subscription today.

New to The Independent?

Or if you would prefer:

Want an ad-free experience?

Hi {{indy.fullName}}

• My Independent Premium
• Account details
• Help centre