research excellence framework gender

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Perspective
• Published: 24 September 2018

Making gender diversity work for scientific discovery and innovation

• Mathias Wullum Nielsen 1 ,
• Carter Walter Bloch   ORCID: orcid.org/0000-0003-4718-003X 1 &
• Londa Schiebinger 2  

Nature Human Behaviour volume  2 ,  pages 726–734 ( 2018 ) Cite this article

5869 Accesses

132 Citations

175 Altmetric

Metrics details

• Decision making
• Interdisciplinary studies

Gender diversity has the potential to drive scientific discovery and innovation. Here, we distinguish three approaches to gender diversity: diversity in research teams, diversity in research methods and diversity in research questions. While gender diversity is commonly understood to refer only to the gender composition of research teams, fully realizing the potential of diversity for science and innovation also requires attention to the methods employed and questions raised in scientific knowledge-making. We provide a framework for understanding the best ways to support the three approaches to gender diversity across four interdependent domains — from research teams to the broader disciplines in which they are embedded to research organizations and ultimately to the different societies that shape them through specific gender norms and policies. Our analysis demonstrates that realizing the benefits of diversity for science requires careful management of these four interdependent domains.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

111,21 € per year

only 9,27 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

The impact of gender diversity on scientific research teams: a need to broaden and accelerate future research

Hannah B. Love, Alyssa Stephens, … Ellen R. Fisher

Gender and Arctic climate change science in Canada

David Natcher, Ana Maria Bogdan, … Kent Spiers

Collaboration between women helps close the gender gap in ice core science

Bess G. Koffman, Matthew B. Osman, … Sofia Guest

Communication from the Commission to the European Parliament, the Council and the European Economic and Social Committee and the Committee of the Regions: A Reinforced European Research Area Partnership for Excellence and Growth (European Commission, 2012).

S tatement of Principles and Actions Promoting the Equality and Status of Women in Research (Global Research Council, 2016); https://www.globalresearchcouncil.org/fileadmin//documents/GRC_Publications/Statement_of_Principles_and_Actions_Promoting_the_Equality_and_Status_of_Women_in_Research.pdf

Huyer, S. in UNESCO Science Report: Towards 203 0 (ed. Schneegans, S.) 84–103 (UNESCO Publishing, Paris, 2015); http://unesdoc.unesco.org/images/0023/002354/235406e.pdf

Maes, K., Gvozdanovic, J., Buitendijk, S., Hallberg, I. R. & Mantilleri, B. Women, Research and Universities: Excellence Without Gender Bias (League of European Research Universities, Leuven, 2012).

Diversity in science. The Royal Society https://royalsociety.org/topics-policy/diversity-in-science/topic/ (2017).

Valantine, H. A. & Collins, F. S. National Institutes of Health addresses the science of diversity. Proc. Natl Acad. Sci. USA 112 , 12240–12242 (2015).

Article   CAS   Google Scholar  

Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl Acad. Sci. USA 101 , 16385–16389 (2004).

Page, S. E. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies (Princeton Univ. Press, Princeton, 2008).

Phillips, K. W. How diversity works. Sci. Am. 311 , 42–47 (2014).

Article   Google Scholar  

Nishii, L. H. The benefits of climate for inclusion for gender-diverse groups. Acad. Manage. J. 56 , 1754–1774 (2013).

Bear, J. B. & Woolley, A. W. The role of gender in team collaboration and performance. Interdiscip. Sci. Rev. 36 , 146–153 (2011).

Joshi, A., Liao, H. & Roh, H. Bridging domains in workplace demography research: a review and reconceptualization. J. Manage. 37 , 521–552 (2011).

Google Scholar  

Van Dijk, H., van Engen, M. L. & van Knippenberg, D. Defying conventional wisdom: a meta-analytical examination of the differences between demographic and job related diversity relationships with performance. Organ. Behav. Hum. Decis. Process 119 , 38–53 (2012).

Díaz-García, C., González-Moreno, A. & Jose Sáez-Martínez, F. Gender diversity within R&D teams: its impact on radicalness of innovation. Innovation 15 , 149–160 (2013).

Faems, D. & Subramanian, A. M. R&D manpower and technological performance: the impact of demographic and task-related diversity. Res. Policy 42 , 1624–1633 (2013).

Fernández, J. The impact of gender diversity in foreign subsidiaries’ innovation outputs. Int. J. Gend Entrep. 7 , 148–167 (2015).

Østergaard, C. R., Timmermans, B. & Kristinsson, K. Does a different view create something new? The effect of employee diversity on innovation. Res. Policy 40 , 500–509 (2011).

Sastre, J. F. The impact of R&D teams’ gender diversity on innovation outputs. Int. J. Entrep. Small Bus. 24 , 142–162 (2015).

Turner, L. Gender diversity and innovative performance. Int. J. Innov. Sustain. Dev. 4 , 123–134 (2009).

Campbell, L. G., Mehtani, S., Dozier, M. E. & Rinehart, J. Gender-heterogeneous working groups produce higher quality science. PLoS ONE 8 , e79147 (2013).

Joshi, A. By whom and when is women’s expertise recognized? The interactive effects of gender and education in science and engineering teams. Adm. Sci. Q. 25 , 202–239 (2014).

Lungeanu, A. & Contractor, N. S. The effects of diversity and network ties on innovations: the emergence of a new scientific field. Am. Behav. Sci. 59 , 548–564 (2015).

Saá-Pérez, D., Díaz-Díaz, N. L., Aguiar-Díaz, I. & Ballesteros-Rodríguez, J. L. How diversity contributes to academic research teams performance. R&D Manage. 47 , 165–179 (2015).

Stvilia, B. et al. Composition of scientific teams and publication productivity at a national science lab. J. Assoc. Inf. Sci. Technol. 62 , 270–283 (2011).

For a Better Integration of the Gender Dimension in Horizon 2020 Work Programme 2016–2017 (European Commission, 2015); http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetailDoc&id=18892&no=1

Buitendijk, S. & Maes, K. Gendered Research and Innovation: Integrating Sex and Gender Analysis into the Research Process (League of European Research Universities, Leuven, 2015).

Schiebinger, L. & Klinge, I. Gendered Innovations: How Gender Analysis Contributes to Research (Publications Office of the European Union, Luxembourg, 2013).

Sánchez de Madariaga, I., de Gregorio Hurtado, S. (eds). Advancing Gender in Research, Innovation and Sustainable Development (Fundación General de la Universidad Politécnica de Madrid, Madrid, 2016).

Bührer, S. & Schraudner, M. Gender-Aspekte in der Forschung: Wie können Gender-Aspekte in Forschungsvorhaben erkannt und bewertet werden? (Frauenhofer IRB Verlag, Stuttgart, 2006).

Adler, R. A. Osteoporosis in men: a review. Bone Res. 2 , 14001 (2014).

Schiebinger, L. et al. Sex and gender analysis policies of major granting agencies. Gendered Innovations http://genderedinnovations.stanford.edu/sex-and-gender-analysis-policies-major-granting-agencies.html (2018).

Johnson, J., Sharman, Z., Vissandjee, B. & Stewart, D. E. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS ONE 9 , e99900 (2014).

De Cheveigné, S. & Knoll, B. Interim Evaluation: Gender Equality as a Crosscutting Issue in Horizon 2020 (Publications Office of the European Union, Luxembourg, 2017).

European Commission She Figures 2015 (Publications Office of the European Union, Luxembourg, 2016).

US General Accounting Office. Drug Safety: Most Drugs Withdrawn in Recent Years had Greater Health Risks for Women (Government Publishing Office, Washington DC, 2001).

Roth, J. et al. Economic return from the women’s health initiative estrogen plus progestin clinical trial: a modeling study. Ann. Intern. Med. 160 , 594–602 (2014).

Ovseiko, P. V. et al. A global call for action to include gender in research impact assessment. Health Res. Policy Syst. 14 , 50 (2016).

Nielsen, M. W. et al. Opinion: gender diversity leads to better science. Proc. Natl Acad. Sci. USA 114 , 1740–1742 (2017).

Dolado, J. J., Felgueroso, F. & Almunia, M. Are men and women-economists evenly distributed across research fields? Some new empirical evidence. SERIEs 3 , 367–393 (2012).

Light, R. in Networks, Work, and Inequality (ed. Mcdonald, S.) 239–268 (Research in the Sociology of Work Vol. 24, Emerald Group Publishing, Bingley, 2013).

Mapping Gender in the German Research Area (Elsevier, 2015); https://www.elsevier.com/__data/assets/pdf_file/0004/126715/ELS_Germany_Gender_Research-SinglePages.pdf

Maliniak, D., Powers, R. & Walter, B. F. The gender citation gap in international relations. Int. Organ. 67 , 889–922 (2013).

West, J. D., Jacquet, J., King, M. M., Correll, S. J. & Bergstrom, C. T. The role of gender in scholarly authorship. PLoS ONE 8 , e66212 (2013).

Nonnemaker, L. Women physicians in academic medicine — new insights from cohort studies. N. Engl. J. Med. 342 , 399–405 (2000).

Rosser, S. V. An overview of women’s health in the US since the mid-1960s. Hist. Technol . 18 , 355–369 (2002).

Schiebinger, L. Has Feminism Changed Science? (Harvard Univ. Press, Cambridge, 1999).

Fedigan, L. M. Primate Paradigms: Sex Roles and Social Bonds (Univ. Chicago Press, Chicago, 1992).

Schiebinger, L. (ed.) Gendered Innovations in Science and Engineering (Stanford Univ. Press, Stanford, 2008).

Nielsen, M. W., Andersen, J. P., Schiebinger, L. & Schneider, J. W. One and a half million medical papers reveal a link between author gender and attention to gender and sex analysis. Nat. Hum. Behav. 1 , 791–796 (2017).

Herring, C. Does diversity pay? Race, gender, and the business case for diversity. Am. Sociol. Rev. 74 , 208–224 (2009).

Mannix, E. & Neale, M. A. What differences make a difference? The promise and reality of diverse teams in organizations. Psychol. Sci. Public Interest 6 , 31–55 (2005).

Shore, L. M. et al. Diversity in organizations: Where are we now and where are we going? Hum. Resour . Manage. Rev. 19 , 117–133 (2009).

Williams, K. Y. & O’Reilly, C. A. in Research in Organizational Behavior (eds Staw, B. M. & Cummings, L. L.) 77–140 (JAI Press, Greenwich, 1998).

Homan, A. C., Van Knippenberg, D., Van Kleef, G. A. & De Dreu, C. K. Bridging faultlines by valuing diversity: diversity beliefs, information elaboration, and performance in diverse work groups. J. Appl. Psychol. 92 , 1189–1199 (2007).

Lauring, J. & Villesèche, F. The performance of gender diverse teams: What is the relation between diversity attitudes and degree of diversity? Eur. Manage. Rev. https://doi.org/10.1111/emre.12164 (2017).

Van Knippenberg, D., Haslam, S. A. & Platow, M. J. Unity through diversity: value-in-diversity beliefs, work group diversity, and group identification. Group Dyn. 11 , 207–222 (2007).

Hobman, E. V., Bordia, P. & Gallois, C. Perceived dissimilarity and work group involvement: the moderating effects of group openness to diversity. Group Organ. Manage. 29 , 560–587 (2004).

Joshi, A. & Knight, A. P. Who defers to whom and why? Dual pathways linking demographic differences and dyadic deference to team effectiveness. Acad. Manag. J. 58 , 59–84 (2015).

Hobman, E. V. & Bordia, P. The role of team identification in the dissimilarity-conflict relationship. Group Process. Intergroup Relat. 9 , 483–507 (2006).

Mohammed, S. & Angell, L. C. Surface- and deep-level diversity in workgroups: examining the moderating effects of team orientation and team process on relationship conflict. J. Organ. Behav. 25 , 1015–1039 (2004).

Homan, A. C. et al. Facing differences with an open mind: openness to experience, salience of intragroup differences, and performance of diverse work groups. Acad. Manage. J. 51 , 1204–1222 (2008).

Roos, P. A. & Reskin, B. F. Occupational desegregation in the 1970s: Integration and economic equity? Sociol. Perspect. 35 , 69–91 (1992).

Lautenberger, D. M., Dandar, V. M. & Raezer, C. L. The State of Women in Academic Medicine: The Pipeline and Pathways to Leadership 2013–2014 (Association of American Medical Colleges, Washington DC, 2014).

Becher, T. The significance of disciplinary differences. Stud. Higher Educ. 19 , 151–161 (1994).

Zou, J. & Schiebinger, L. AI can be sexist and racist — it’s time to make it fair. Nature 559 , 324–326 (2018).

Felt, U. & Stöckelová, T. in Knowing and Living in Academic Research (ed. Felt, U.) 41–124 (Institute of Sociology of the Academy of Sciences of the Czech Republic, Prague, 2009).

Lamont, M. How Professors Think (Harvard Univ. Press, Cambridge, 2009).

Whitley, R. The Intellectual and Social Organization of the Sciences (Oxford Univ. Press, Oxford, 2000).

Stewart, A. J., Malley, J. E. & LaVaque-Manty, D. Transforming Science and Engineering: Advancing Academic Women (Univ. Michigan Press, Ann Arbor, 2007).

Courses. Course 1: the science of sex and gender in human health. National Institutes of Health https://sexandgendercourse.od.nih.gov/content/courses (2018).

Sex and Gender in Biomedical Research (Canadian Institutes of Health, 2018); http://www.cihr-irsc-igh-isfh.ca/

Schiebinger, L. et al. Sex and gender analysis policies of peer-reviewed journals. Gendered Innovations https://genderedinnovations.stanford.edu/sex-and-gender-analysis-policies-peer-reviewed-journals.html (2018).

Author instructions: manuscript preparation Circulation Research https://www.ahajournals.org/res/manuscript-preparation (2018).

Miller, V. M. In pursuit of scientific excellence: sex matters. Am. J. Physiol. Cell Physiol. 302 , C1269–C1270 (2012).

Schiebinger, L., Leopold, S. S. & Miller, V. M. Editorial policies for sex and gender analysis. Lancet 388 , 2841–2842 (2016).

Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (International Committee of Medical Journal Editors, 2017); http://www.icmje.org/recommendations/archives/

Ludwig, S. et al. A successful strategy to integrate sex and gender medicine into a newly developed medical curriculum. J. Womens Health 24 , 996–1005 (2015).

Brooks, C., Fenton, E. M. & Walker, T. J. Gender and the evaluation of research. Res. Policy 43 , 990–1001 (2014).

Hunter, L. A. & Leahey, E. Parenting and research productivity: new evidence and methods. Soc. Stud. Sci. 40 , 433–451 (2010).

Nielsen, M. W. Gender consequences of a national performance-based funding model: new pieces in an old puzzle. Stud. Higher Educ. 42 , 1033–1055 (2017).

Schneid, M., Isidor, R., Li, C. & Kabst, R. The influence of cultural context on the relationship between gender diversity and team performance: a meta-analysis. Int. J. Hum. Resour. Manage. 26 , 733–756 (2015).

European Commission Seventh FP7 Monitoring Report 2013 (Publications Office of the European Union, 2015).

C ommission Staff Working Document: Horizon 2020 Annual Monitoring Report 2015 (European Commission, 2016).

Fact Sheet: Gender Equality in Horizon 2020 (European Commission, 2013); https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/FactSheet_Gender_2.pdf

Topics with a gender dimension. European Commission http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/ftags/gender.html#c,topics=flags/s/Gender/1/1&+callDeadline/desc (2015).

Clayton, J. & Collins, F. NIH to balance sex in cell and animal studies. Nature 509 , 282–283 (2014).

Aksnes, D. et al. Centres of Excellence in the Nordic Countries (NIFU, Oslo, 2012).

Sandström, U., Wold, A., Jordansson, B., Ohlsson, B. & Smedberg, Å. Hans Excellens: om miljardsatsningarna på starka forskningsmiljöer (Delegationen för Jämställdhet i Högskolan, Stockholm, 2010).

Download references

Acknowledgements

We thank E. Steiner, Co-Director, Spatial History Project, Center for Spatial and Textual Analysis, Stanford University, for executing our graphics.

Author information

Authors and affiliations.

Danish Centre for Studies in Research and Research Policy, Department of Political Science, Aarhus University, Aarhus, Denmark

Mathias Wullum Nielsen & Carter Walter Bloch

History of Science, Stanford University, Stanford, CA, USA

Londa Schiebinger

You can also search for this author in PubMed   Google Scholar

Contributions

M.W.N., L.S. and C.W.B. conceptualized and wrote the paper. M.W.N. and L.S. carried out literature searches, and M.W.N. and C.W.B. prepared tables. L.S. and M.W.N. conceptualized Figs. 1 and 2.

Corresponding author

Correspondence to Mathias Wullum Nielsen .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

: Supplementary Methods; Supplementary Table 1 -4

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Nielsen, M.W., Bloch, C.W. & Schiebinger, L. Making gender diversity work for scientific discovery and innovation. Nat Hum Behav 2 , 726–734 (2018). https://doi.org/10.1038/s41562-018-0433-1

Download citation

Received : 09 February 2018

Accepted : 18 July 2018

Published : 24 September 2018

Issue Date : October 2018

DOI : https://doi.org/10.1038/s41562-018-0433-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Feminization of the health workforce in china: exploring gendered composition from 2002 to 2020.

• Joanna Raven
• Xiaoyun Liu

Human Resources for Health (2024)

How to diversify the dwindling physician–scientist workforce after the US affirmative action ban

• Jessica L. Ding
• Briana Christophers
• Alex D. Waldman

Nature Medicine (2024)

Artificial intelligence and illusions of understanding in scientific research

• Lisa Messeri
• M. J. Crockett

Nature (2024)

The cyclical ethical effects of using artificial intelligence in education

• Edward Dieterle
• Michael Walker

AI & SOCIETY (2024)

Gender inequality in genitourinary malignancies clinical trials leadership

• Abdulrahman Alhajahjeh
• Ahmed A. Abdulelah
• Mohammed Shahait

World Journal of Urology (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Help & FAQ

Gender and the Research Excellence Framework

Research output : Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

Access to Document

• 10.1163/9789463511438

T1 - Gender and the Research Excellence Framework

AU - Yarrow, Emily

PY - 2018/1/1

Y1 - 2018/1/1

U2 - 10.1163/9789463511438

DO - 10.1163/9789463511438

M3 - Chapter (peer-reviewed)

SN - 9463511423

SN - 978-94-6351-142-1

SN - 978-94-6351-141-4

BT - EqualBITE

A2 - Robertson, Judy

A2 - Williams, Alison

A2 - Jones, Derek

A2 - Loads, Daphne

PB - Brill Sense

CY - Rotterdam, Netherlands

Gender Plays a Role in Research Impact and Assessment

You may not have heard about the impact a-gender, but the chances are you have encountered it in water-cooler conversations, university corridors, or the formal settings of meetings and review panels…  By this, simply put, we mean the implicit and explicit gendered associations often drawn when academics refer to types of impact activities and how this can creep into the evaluation of impact. Previous work by  Yarrow and Davies  has shown that women were significantly less likely to be the authors of impact case studies in the 2014 round of the UK’s Research Excellence Framework (REF). Our own research reinforces this observation, revealing how gendered assumptions toward impact and its assessment play a significant role in all aspects of research impact; from deciding what kinds of research impact receive attention, to the way in which REF panels codify and assess the quality of research impacts.  

To explore how gender might underscore thinking about research impact, we re-evaluated two independent datasets to examine the ways in which impact was described in both the generation and evaluation of research impact: the first containing semi-structured interviews with mid-senior academics at two research-intensive universities in Australia and the UK, collected between 2011 and 2013 (n=51); and the second including pre- (n=62), and post-evaluation (n=57) interviews with REF2014 Main Panel A (medicine, health and life sciences) evaluators.

We found that gendered perceptions of research impact to run deep and that regardless of a participant’s gender, unconsciously assigned gendered orientations to impact activities were attributed to research impact that were then repeated during evaluation. We call this process, by which gendered notions of non-academic, societal impact and how it is generated feed into its evaluation, the ‘impact a-gender.’ Left unchecked, this risks compounding gender inequality in academia; creating an additional ‘ Matilda effect ’ for research impact, whereby the achievements of female researchers are further marginalized. 

Gendered definitions of impact

The gendered nature of impact begins with defining impact itself. For example, during our interviews, researchers in the UK and Australia used terms such as ‘hard’ (masculine) or ‘soft’ (feminine) when talking about the types of impact created and these were prevalent alongside gendered notions of excellence related to competitiveness. These binary associations are integral to the way gender bias infiltrates academia generally, but here we see this association towards non-academic impact.  When referring to impact more generally, it was not necessarily the masculinity or femininity of the researcher that was emphasized, but rather the activity used to generate it. ‘Soft’ impact activities were described as:

 “Stuff that’s on a flaky edge – it’s very much about social engagement”  (Languages, Australia, Professor, Male).

Researchers placed value on hard over soft impacts, referring to how soft impacts were of less value to institutions and amongst their colleagues. Even the word impact itself, is to some extent burdened with gendered connotations of ‘force’ and ‘weight.’

Gendered sorting of impact activities

When asked about engaging in Impact, both men and women were dissuaded (by inner narratives, institutions, or by colleagues) from engaging in impacts that were not conducive to ‘hard’ impacts. Researchers reported being pressured away from non-gender-acceptable forms of impact, either in preference to other more masculine notions of academic productivity (for men), or towards soft notions of academic productivity, i.e. public outreach (for women). 

In practice, some men, in particular, implicitly referred to women as better with the  ‘soft and woolly’  impacts and justified this based on a sense of  ‘duty’  or  ‘public service’  associated with the nurturing role commonly assigned to women. This immediately and implicitly assigned certain types of impact activities to women, such as public engagement, while the more dominant, activities that lead to economic impacts, such as technology transfer, relating less to people and more to things i.e. licensing or spin-outs, were more likely to be assigned to men. When the reason behind this association was explored, gendered notions of competitiveness emerged: (talking about public engagement)  “women are better at this! They are less competitive!”  (Environment, UK, Professor, Male).  W omen were perceived as not sufficiently competitive for ‘harder’ impacts, despite having already been steered away from engaging in these activities.

Gendered evaluations of impact

In a further extension of the impact a-gender, these gendered associations of Impact were echoed by Impact evaluators. Here, despite a shift away from impact and measurement, the types of impact that are not as easily measurable and are, by nature, more amenable to qualitative narratives around societal and cultural value, were again characterized as ‘soft.’ In contrast, quantifiable impact was often masculinized and more regularly depicted as ‘hard.’  For instance, when articulating impact, male evaluators were more likely to express impact as causal and linear, involving a single “star” or “champion,” who delivered it from start (underpinning research) to finish (impact). So too, when these binary interpretations were present, they were more likely to value impact in a way in which encoded perceptions of hard/ quantifiable/ masculine research (as opposed to soft/ qualitative/ feminine) were considered as being higher quality and of more esteem, regardless of the actual quality of the research or its impact. 

Male and female evaluators also conceptualized impact differently for the sake of its evaluation. Female evaluators were more open to and expressed sensitivity towards the complexity of impact and this included taking into account issues of time-lags, reasons for the lack of causality (especially for policy impacts) and reasonable attribution claims.  Male evaluators, on the other hand, strived to conceptualize impact as excellent if it was a  “straight -forward”  and  “measurable change.”

Gendered evaluators of impact

Finally, evaluators also suggested that the workings of the panel during the evaluation were also drawn down gendered lines. If accurate, this risks silencing conflicting opinions that may call for a more complex approach to Impact evaluation. In this way, women were valued as panel members if they played a supportive, supplementary role and were a good “team player”: 

“A good panel member was — someone who is happy to listen to discussions; to not be too dogmatic about their opinion, but can listen and learn, because impact is something we are all learning from scratch. Somebody who wasn’t too outspoken, was a team player.”  (Panel 3, Outputs and Impact, Female)

Calling out the impact a-gender

Our research suggests that notions of soft and hard impact are not simply benign, but can serve to marginalize individuals, groups and particular kinds of research. What we have uncovered, we fear, only scratches the surface of some of the ways in which an impact a-gender emerges within the academic community, and evaluation panels. However, its implications will be familiar to many researchers preparing REF case studies, and to policymakers that are working to ensure that the REF evaluation process is free from these implicit biases.

What we must do, as a community, is reflect on our scholarly norms of intellectual practice; to be skeptical about our own preconceptions and attitudes which come as second nature and to question our assumptions and prejudices. Individually, we must continually and explicitly check our own behavior to minimize any further follow on effects. This is regardless of whether we work as researchers generating impact, or as evaluators assessing it.  

The impact a-gender is known and it is not acceptable. Let’s not leave it unchecked.

Jennifer Chubb and Gemma Derrick

Jennifer Chubb (pictured) is a postdoctoral researcher at the University of York. Chubb's research career to date has focused on the philosophy and politics of research, identities and research cultures, research impact and science policy and advice. She has a particular interest in the impact and ethical implications of emerging technologies. Her current research focuses on the impact of artificial intelligence and related digital technologies. Gemma Derrick is a senior lecturer and director of research at the Department of Educational Research at Lancaster University. Derrick's research focuses on the dynamics of knowledge production and how researchers create, conform and participate in evaluative cultures within academia. She has a particular specialty in the evaluation Impact using peer review and its role in the academic governance system.

Related Articles

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Using Translational Research as a Model for Long-Term Impact

Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

New Podcast Series Applies Social Science to Social Justice Issues

Sage (the parent of Social Science Space) and the Surviving Society podcast have launched a collaborative podcast series, Social Science for Social […]

The Importance of Using Proper Research Citations to Encourage Trustworthy News Reporting

Based on a study of how research is cited in national and local media sources, Andy Tattersall shows how research is often poorly represented in the media and suggests better community standards around linking to original research could improve trust in mainstream media.

Connecting Legislators and Researchers, Leads to Policies Based on Scientific Evidence

The author’s team is developing ways to connect policymakers with university-based researchers – and studying what happens when these academics become the trusted sources, rather than those with special interests who stand to gain financially from various initiatives.

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Quality in Postgraduate Research Conference 2024

Scecsal 2024: transforming libraries, empowering communities, webinar: how to do research in a digital world, webinar: how to write and structure an article’s front matter, webinar: how to get more involved with a journal and develop your career, science of team science 2024 virtual conference, webinar: how to collaborate across paradigms – embedding culture in mixed methods designs.

Customize your experience

Select your preferred categories.

• Announcements
• Business and Management INK

Communication

Higher education reform, open access, recent appointments, research ethics, interdisciplinarity, international debate.

• Academic Funding

Public Engagement

• Recognition

Presentations

Science & social science, social science bites, the data bulletin.

Social, Behavioral Scientists Eligible to Apply for NSF S-STEM Grants

Solicitations are now being sought for the National Science Foundation’s Scholarships in Science, Technology, Engineering, and Mathematics program, and in an unheralded […]

With COVID and Climate Change Showing Social Science’s Value, Why Cut it Now?

What are the three biggest challenges Australia faces in the next five to ten years? What role will the social sciences play in resolving these challenges? The Academy of the Social Sciences in Australia asked these questions in a discussion paper earlier this year. The backdrop to this review is cuts to social science disciplines around the country, with teaching taking priority over research.

Testing-the-Waters Policy With Hypothetical Investment: Evidence From Equity Crowdfunding

While fundraising is time-consuming and entails costs, entrepreneurs might be tempted to “test the water” by simply soliciting investors’ interest before going through the lengthy process. Digitalization of finance has made it possible for small business to run equity crowdfunding campaigns, but also to initiate a TTW process online and quite easily.

AAPSS Names Eight as 2024 Fellows

The American Academy of Political and Social Science today named seven scholars and one journalist as its 2024 fellows class.

Apply for Sage’s 2024 Concept Grants

Three awards are available through Sage’s Concept Grant program, which is designed to support innovative products and tools aimed at enhancing social science education and research.

Economist Kaye Husbands Fealing to Lead NSF’s Social Science Directorate

Kaye Husbands Fealing, an economist who has done pioneering work in the “science of broadening participation,” has been named the new leader of the U.S. National Science Foundation’s Directorate for Social, Behavioral and Economic Sciences.

Big Think Podcast Series Launched by Canadian Federation of Humanities and Social Sciences

The Canadian Federation of Humanities and Social Sciences has launched the Big Thinking Podcast, a show series that features leading researchers in the humanities and social sciences in conversation about the most important and interesting issues of our time.

The We Society Explores Intersectionality and Single Motherhood

In a recently released episode of The We Society podcast, Ann Phoenix, a psychologist at University College London’s Institute of Education, spoke […]

Second Edition of ‘The Evidence’ Examines Women and Climate Change

The second issue of The Evidence explores the intersection of gender inequality and the global climate crisis. Author Josephine Lethbridge recounts the […]

New Report Finds Social Science Key Ingredient in Innovation Recipe

A new report from Britain’s Academy of Social Sciences argues that the key to success for physical science and technology research is a healthy helping of relevant social science.

Too Many ‘Gray Areas’ In Workplace Culture Fosters Racism And Discrimination

The new president of the American Sociological Association spent more than 10 years interviewing over 200 Black workers in a variety of roles – from the gig economy to the C-suite. I found that many of the problems they face come down to organizational culture. Too often, companies elevate diversity as a concept but overlook the internal processes that disadvantage Black workers.

A Social Scientist Looks at the Irish Border and Its Future

‘What Do We Know and What Should We Do About the Irish Border?’ is a new book from Katy Hayward that applies social science to the existing issues and what they portend.

Brexit and the Decline of Academic Internationalism in the UK

Brexit seems likely to extend the hostility of the UK immigration system to scholars from European Union countries — unless a significant change of migration politics and prevalent public attitudes towards immigration politics took place in the UK. There are no indications that the latter will happen anytime soon.

Brexit and the Crisis of Academic Cosmopolitanism

A new report from the Royal Society about the effects on Brexit on science in the United Kingdom has our peripatetic Daniel Nehring mulling the changes that will occur in higher education and academic productivity.

Challenging, But Worth It: Overcoming Paradoxical Tensions of Identity to Embrace Transformative Technologies in Teaching and Learning

In this article, Isabel Fischer and Kerry Dobbins reflect on their work, “Is it worth it? How paradoxical tensions of identity shape the readiness of management educators to embrace transformative technologies in their teaching,” which was recently published in the Journal of Management Education.

Data Analytics and Artificial Intelligence in the Complex Environment of Megaprojects: Implications for Practitioners and Project Organizing Theory

The authors review the ways in which data analytics and artificial intelligence can engender more stability and efficiency in megaprojects. They evaluate the present and likely future use of digital technology—particularly with regard to construction projects — discuss the likely benefits, and also consider some of the challenges around digitization.

Putting People at the Heart of the Research Process

In this article, Jessica Weaver, Philippa Hunter-Jones, and Rory Donnelly reflect on “Unlocking the Full Potential of Transformative Service Research by Embedding Collaboration Throughout the Research Process,” which can be found in the Journal of Service Research.

2024 Holberg Prize Goes to Political Theorist Achille Mbembe

Political theorist and public intellectual Achille Mbembe, among the most read and cited scholars from the African continent, has been awarded the 2024 Holberg Prize.

Edward Webster, 1942-2024: South Africa’s Pioneering Industrial Sociologist

Eddie Webster, sociologist and emeritus professor at the Southern Centre for Inequality Studies at the University of the Witwatersrand in South Africa, died on March 5, 2024, at age 82.

Charles V. Hamilton, 1929-2023: The Philosopher Behind ‘Black Power’

Political scientist Charles V. Hamilton, the tokenizer of the term ‘institutional racism,’ an apostle of the Black Power movement, and at times deemed both too radical and too deferential in how to fight for racial equity, died on November 18, 2023. He was 94.

National Academies Seeks Experts to Assess 2020 U.S. Census

The National Academies’ Committee on National Statistics seeks nominations for members of an ad hoc consensus study panel — sponsored by the U.S. Census Bureau — to review and evaluate the quality of the 2020 Census.

Will the 2020 Census Be the Last of Its Kind?

Could the 2020 iteration of the United States Census, the constitutionally mandated count of everyone present in the nation, be the last of its kind?

Will We See A More Private, But Less Useful, Census?

Census data can be pretty sensitive – it’s not just how many people live in a neighborhood, a town, a state or […]

Did the Mainstream Make the Far-Right Mainstream?

The processes of mainstreaming and normalization of far-right politics have much to do with the mainstream itself, if not more than with the far right.

The Use of Bad Data Reveals a Need for Retraction in Governmental Data Bases

Retractions are generally framed as a negative: as science not working properly, as an embarrassment for the institutions involved, or as a flaw in the peer review process. They can be all those things. But they can also be part of a story of science working the right way: finding and correcting errors, and publicly acknowledging when information turns out to be incorrect.

Free Online Course Reveals The Art of ChatGPT Interactions

You’ve likely heard the hype around artificial intelligence, or AI, but do you find ChatGPT genuinely useful in your professional life? A free course offered by Sage Campus could change all th

Research Integrity Should Not Mean Its Weaponization

Commenting on the trend for the politically motivated forensic scrutiny of the research records of academics, Till Bruckner argues that singling out individuals in this way has a chilling effect on academic freedom and distracts from efforts to address more important systemic issues in research integrity.

What Do We Know about Plagiarism These Days?

In the following Q&A, Roger J. Kreuz, a psychology professor who is working on a manuscript about the history and psychology of plagiarism, explains the nature and prevalence of plagiarism and the challenges associated with detecting it in the age of AI.

Webinar: iGen: Decoding the Learning Code of Generation Z

As Generation Z students continue to enter the classroom, they bring with them a host of new challenges. This generation of students […]

Year of Open Science Conference

The Center for Open Science (COS), in collaboration with NASA, is hosting a no-cost, online culminating conference on March 21 and 22 […]

“How to Collaborate Across Paradigms: Embedding Culture in Mixed Methods Designs” is another piece of Sage’s webinar series, How to Do Research […]

Returning Absentee Ballots during the 2020 Election – A Surprise Ending?

One of the most heavily contested voting-policy issues in the 2020 election, in both the courts and the political arena, was the deadline […]

Overconsumption or a Move Towards Minimalism?

(Over)consumption, climate change and working from home. These are a few of the concerns at the forefront of consumers’ minds and three […]

To Better Serve Students and Future Workforces, We Must Diversify the Syllabi

Ellen Hutti and Jenine Harris have quantified the extent to which female authors are represented in assigned course readings. In this blog post, they emphasize that more equal exposure to experts with whom they can identify will better serve our students and foster the growth, diversity and potential of this future workforce. They also present one repository currently being built for readings by underrepresented authors that are Black, Indigenous or people of color.

Drawing on the findings of a workshop on making translational research design principles the norm for European research, Gabi Lombardo, Jonathan Deer, Anne-Charlotte Fauvel, Vicky Gardner and Lan Murdock discuss the characteristics of translational research, ways of supporting cross disciplinary collaboration, and the challenges and opportunities of adopting translational principles in the social sciences and humanities.

Addressing the United Kingdom’s Lack of Black Scholars

In the UK, out of 164 university vice-chancellors, only two are Black. Professor David Mba was recently appointed as the first Black vice-chancellor […]

A longitudinal research project project covering 31 villages in rural South Africa has led to groundbreaking research in many fields, including genomics, HIV/Aids, cardiovascular conditions and stroke, cognition and aging.

Norman B. Anderson, a clinical psychologist whose work as both a researcher and an administrator saw him serve as the inaugural director of the U.S. National Institute of Health’s Office of Behavioral and Social Sciences Research and as chief executive officer of the American Psychological Association, died on March 1.

Why Social Science? Because It Makes an Outsized Impact on Policy

Euan Adie, founder of Altmetric and Overton and currently Overton’s managing director, answers questions about the outsized impact that SBS makes on policy and his work creating tools to connect the scholarly and policy worlds.

A Behavioral Scientist’s Take on the Dangers of Self-Censorship in Science

The word censorship might bring to mind authoritarian regimes, book-banning, and restrictions on a free press, but Cory Clark, a behavioral scientist at […]

Infrastructure

New Funding Opportunity for Criminal and Juvenile Justice Doctoral Researchers

A new collaboration between the National Institute of Justice (NIJ) and the U.S. National Science Foundation has founded the Graduate Research Fellowship […]

To Better Forecast AI, We Need to Learn Where Its Money Is Pointing

By carefully interrogating the system of economic incentives underlying innovations and how technologies are monetized in practice, we can generate a better understanding of the risks, both economic and technological, nurtured by a market’s structure.

There’s Something in the Air, Part 2 – But It’s Not a Miasma

Robert Dingwall looks at the once dominant role that miasmatic theory had in public health interventions and public policy.

The Fog of War

David Canter considers the psychological and organizational challenges to making military decisions in a war.

A Community Call: Spotlight on Women’s Safety in the Music Industry 

Women’s History Month is, when we “honor women’s contributions to American history…” as a nation. Author Andrae Alexander aims to spark a conversation about honor that expands the actions of this month from performative to critical

Philip Rubin: FABBS’ Accidental Essential Man Linking Research and Policy

As he stands down from a two-year stint as the president of the Federation of Associations in Behavioral & Brain Sciences, or FABBS, Social Science Space took the opportunity to download a fraction of the experiences of cognitive psychologist Philip Rubin, especially his experiences connecting science and policy.

How Intelligent is Artificial Intelligence?

Cryptocurrencies are so last year. Today’s moral panic is about AI and machine learning. Governments around the world are hastening to adopt […]

National Academies’s Committee On Law And Justice Seeks Experts

The National Academies of Sciences, Engineering and Medicine is seeking suggestions for experts interested in its Committee on Law and Justice (CLAJ) […]

Why Don’t Algorithms Agree With Each Other?

David Canter reviews his experience of filling in automated forms online for the same thing but getting very different answers, revealing the value systems built into these supposedly neutral processes.

A Black History Addendum to the American Music Industry

The new editor of the case study series on the music industry discusses the history of Black Americans in the recording industry.

When University Decolonization in Canada Mends Relationships with Indigenous Nations and Lands

Community-based work and building and maintaining relationships with nations whose land we live upon is at the heart of what Indigenizing is. It is not simply hiring more faculty, or putting the titles “decolonizing” and “Indigenizing” on anything that might connect to Indigenous peoples.

Jonathan Breckon On Knowledge Brokerage and Influencing Policy

Overton spoke with Jonathan Breckon to learn about knowledge brokerage, influencing policy and the potential for technology and data to streamline the research-policy interface.

Research for Social Good Means Addressing Scientific Misconduct

Social Science Space’s sister site, Methods Space, explored the broad topic of Social Good this past October, with guest Interviewee Dr. Benson Hong. Here Janet Salmons and him talk about the Academy of Management Perspectives journal article.

NSF Looks Headed for a Half-Billion Dollar Haircut

Funding for the U.S. National Science Foundation would fall by a half billion dollars in this fiscal year if a proposed budget the House of Representatives’ Appropriations Committee takes effect – the first cut to the agency’s budget in several years.

NSF Responsible Tech Initiative Looking at AI, Biotech and Climate

The U.S. National Science Foundation’s new Responsible Design, Development, and Deployment of Technologies (ReDDDoT) program supports research, implementation, and educational projects for multidisciplinary, multi-sector teams

Digital Transformation Needs Organizational Talent and Leadership Skills to Be Successful

Who drives digital change – the people of the technology? Katharina Gilli explains how her co-authors worked to address that question.

Six Principles for Scientists Seeking Hiring, Promotion, and Tenure

The negative consequences of relying too heavily on metrics to assess research quality are well known, potentially fostering practices harmful to scientific research such as p-hacking, salami science, or selective reporting. To address this systemic problem, Florian Naudet, and collegues present six principles for assessing scientists for hiring, promotion, and tenure.

Book Review: The Oxford Handbook of Creative Industries

Candace Jones, Mark Lorenzen, Jonathan Sapsed , eds.: The Oxford Handbook of Creative Industries. Oxford: Oxford University Press, 2015. 576 pp. $170.00, […]

Daniel Kahneman, 1934-2024: The Grandfather of Behavioral Economics

Nobel laureate Daniel Kahneman, whose psychological insights in both the academic and the public spheres revolutionized how we approach economics, has died […]

Gloria Media, with support from Sage, has launched The Evidence, a feminist newsletter that covers what you need to know about gender […]

Canadian Librarians Suggest Secondary Publishing Rights to Improve Public Access to Research

The Canadian Federation of Library Associations recently proposed providing secondary publishing rights to academic authors in Canada.

Webinar: How Can Public Access Advance Equity and Learning?

The U.S. National Science Foundation and the American Association for the Advancement of Science have teamed up present a 90-minute online session examining how to balance public access to federally funded research results with an equitable publishing environment.

• Open Access in the Humanities and Social Sciences in Canada: A Conversation

Five organizations representing knowledge networks, research libraries, and publishing platforms joined the Federation of Humanities and Social Sciences to review the present and the future of open access — in policy and in practice – in Canada

A Former Student Reflects on How Daniel Kahneman Changed Our Understanding of Human Nature

Daniel Read argues that one way the late Daniel Kahneman stood apart from other researchers is that his work was driven by a desire not merely to contribute to a research field, but to create new fields.

Four Reasons to Stop Using the Word ‘Populism’

Beyond poor academic practice, the careless use of the word ‘populism’ has also had a deleterious impact on wider public discourse, the authors argue.

The Added Value of Latinx and Black Teachers

As the U.S. Congress debates the reauthorization of the Higher Education Act, a new paper in Policy Insights from the Behavioral and Brain Sciences urges lawmakers to focus on provisions aimed at increasing the numbers of black and Latinx teachers.

A Collection: Behavioral Science Insights on Addressing COVID’s Collateral Effects

To help in decisions surrounding the effects and aftermath of the COVID-19 pandemic, the the journal ‘Policy Insights from the Behavioral and Brain Sciences’ offers this collection of articles as a free resource.

Susan Fiske Connects Policy and Research in Print

Psychologist Susan Fiske was the founding editor of the journal Policy Insights from the Behavioral and Brain Sciences. In trying to reach a lay audience with research findings that matter, she counsels stepping a bit outside your academic comfort zone.

Mixed Methods As A Tool To Research Self-Reported Outcomes From Diverse Treatments Among People With Multiple Sclerosis

What does heritage mean to you?

Personal Information Management Strategies in Higher Education

Working Alongside Artificial Intelligence Key Focus at Critical Thinking Bootcamp 2022

SAGE Publishing — the parent of Social Science Space – will hold its Third Annual Critical Thinking Bootcamp on August 9. Leaning more and register here

Watch the Forum: A Turning Point for International Climate Policy

On May 13, the American Academy of Political and Social Science hosted an online seminar, co-sponsored by SAGE Publishing, that featured presentations […]

Event: Living, Working, Dying: Demographic Insights into COVID-19

On Friday, April 23rd, join the Population Association of America and the Association of Population Centers for a virtual congressional briefing. The […]

Involving patients – or abandoning them?

The Covid-19 pandemic seems to be subsiding into a low-level endemic respiratory infection – although the associated pandemics of fear and action […]

Public Policy

Jane M. Simoni Named New Head of OBSSR

Clinical psychologist Jane M. Simoni has been named to head the U.S. National Institutes of Health’s Office of Behavioral and Social Sciences Research

Canada’s Federation For Humanities and Social Sciences Welcomes New Board Members

Annie Pilote, dean of the faculty of graduate and postdoctoral studies at the Université Laval, was named chair of the Federation for the Humanities and Social Sciences at its 2023 virtual annual meeting last month. Members also elected Debra Thompson as a new director on the board.

Britain’s Academy of Social Sciences Names Spring 2024 Fellows

Forty-one leading social scientists have been named to the Spring 2024 cohort of fellows for Britain’s Academy of Social Sciences.

National Academies Looks at How to Reduce Racial Inequality In Criminal Justice System

To address racial and ethnic inequalities in the U.S. criminal justice system, the National Academies of Sciences, Engineering and Medicine just released “Reducing Racial Inequality in Crime and Justice: Science, Practice and Policy.”

Survey Examines Global Status Of Political Science Profession

The ECPR-IPSA World of Political Science Survey 2023 assesses political science scholar’s viewpoints on the global status of the discipline and the challenges it faces, specifically targeting the phenomena of cancel culture, self-censorship and threats to academic freedom of expression.

Report: Latest Academic Freedom Index Sees Global Declines

The latest update of the global Academic Freedom Index finds improvements in only five countries

The Risks Of Using Research-Based Evidence In Policymaking

With research-based evidence increasingly being seen in policy, we should acknowledge that there are risks that the research or ‘evidence’ used isn’t suitable or can be accidentally misused for a variety of reasons. 

Surveys Provide Insight Into Three Factors That Encourage Open Data and Science

Over a 10-year period Carol Tenopir of DataONE and her team conducted a global survey of scientists, managers and government workers involved in broad environmental science activities about their willingness to share data and their opinion of the resources available to do so (Tenopir et al., 2011, 2015, 2018, 2020). Comparing the responses over that time shows a general increase in the willingness to share data (and thus engage in Open Science).

Unskilled But Aware: Rethinking The Dunning-Kruger Effect

As a math professor who teaches students to use data to make informed decisions, I am familiar with common mistakes people make when dealing with numbers. The Dunning-Kruger effect is the idea that the least skilled people overestimate their abilities more than anyone else. This sounds convincing on the surface and makes for excellent comedy. But in a recent paper, my colleagues and I suggest that the mathematical approach used to show this effect may be incorrect.

Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Institutional complexity occurs when the structures, interests, and activities of separate but collaborating organizations—often across national and cultural boundaries—are not well aligned. Institutional voids in this context are gaps in function or capability, including skills gaps, lack of an effective regulatory regime, and weak contract-enforcing mechanisms.

Maintaining Anonymity In Double-Blind Peer Review During The Age of Artificial Intelligence

The double-blind review process, adopted by many publishers and funding agencies, plays a vital role in maintaining fairness and unbiasedness by concealing the identities of authors and reviewers. However, in the era of artificial intelligence (AI) and big data, a pressing question arises: can an author’s identity be deduced even from an anonymized paper (in cases where the authors do not advertise their submitted article on social media)?

Hype Terms In Research: Words Exaggerating Results Undermine Findings

The claim that academics hype their research is not news. The use of subjective or emotive words that glamorize, publicize, embellish or exaggerate results and promote the merits of studies has been noted for some time and has drawn criticism from researchers themselves. Some argue hyping practices have reached a level where objectivity has been replaced by sensationalism and manufactured excitement. By exaggerating the importance of findings, writers are seen to undermine the impartiality of science, fuel skepticism and alienate readers.

Five Steps to Protect – and to Hear – Research Participants

Jasper Knight identifies five key issues that underlie working with human subjects in research and which transcend institutional or disciplinary differences.

New Tool Promotes Responsible Hiring, Promotion, and Tenure in Research Institutions

Modern-day approaches to understanding the quality of research and the careers of researchers are often outdated and filled with inequalities. These approaches […]

There’s Something In the Air…But Is It a Virus? Part 1

The historic Hippocrates has become an iconic figure in the creation myths of medicine. What can the body of thought attributed to him tell us about modern responses to COVID?

Alex Edmans on Confirmation Bias 

n this Social Science Bites podcast, Edmans, a professor of finance at London Business School and author of the just-released “May Contain Lies: How Stories, Statistics, and Studies Exploit Our Biases – And What We Can Do About It,” reviews the persistence of confirmation bias even among professors of finance.

Alison Gopnik on Care

Caring makes us human.  This is one of the strongest ideas one could infer from the work that developmental psychologist Alison Gopnik is discovering in her work on child development, cognitive economics and caregiving.

Tejendra Pherali on Education and Conflict

Tejendra Pherali, a professor of education, conflict and peace at University College London, researches the intersection of education and conflict around the world.

Gamification as an Effective Instructional Strategy

Gamification—the use of video game elements such as achievements, badges, ranking boards, avatars, adventures, and customized goals in non-game contexts—is certainly not a new thing.

Harnessing the Tide, Not Stemming It: AI, HE and Academic Publishing

Who will use AI-assisted writing tools — and what will they use them for? The short answer, says Katie Metzler, is everyone and for almost every task that involves typing.

Immigration Court’s Active Backlog Surpasses One Million

In the first post from a series of bulletins on public data that social and behavioral scientists might be interested in, Gary Price links to an analysis from the Transactional Records Access Clearinghouse.

Webinar Discusses Promoting Your Article

The next in SAGE Publishing’s How to Get Published webinar series focuses on promoting your writing after publication. The free webinar is set for November 16 at 4 p.m. BT/11 a.m. ET/8 a.m. PT.

Webinar Examines Open Access and Author Rights

The next in SAGE Publishing’s How to Get Published webinar series honors International Open Access Week (October 24-30). The free webinar is […]

Ping, Read, Reply, Repeat: Research-Based Tips About Breaking Bad Email Habits

At a time when there are so many concerns being raised about always-on work cultures and our right to disconnect, email is the bane of many of our working lives.

New Dataset Collects Instances of ‘Contentious Politics’ Around the World

The European Research Center is funding the Global Contentious Politics Dataset, or GLOCON, a state-of-the-art automated database curating information on political events — including confrontations, political turbulence, strikes, rallies, and protests

Matchmaking Research to Policy: Introducing Britain’s Areas of Research Interest Database

Kathryn Oliver discusses the recent launch of the United Kingdom’s Areas of Research Interest Database. A new tool that promises to provide a mechanism to link researchers, funders and policymakers more effectively collaboratively and transparently.

Watch The Lecture: The ‘E’ In Science Stands For Equity

According to the National Science Foundation, the percentage of American adults with a great deal of trust in the scientific community dropped […]

Watch a Social Scientist Reflect on the Russian Invasion of Ukraine

“It’s very hard,” explains Sir Lawrence Freedman, “to motivate people when they’re going backwards.”

Dispatches from Social and Behavioral Scientists on COVID

Has the ongoing COVID-19 pandemic impacted how social and behavioral scientists view and conduct research? If so, how exactly? And what are […]

Contemporary Politics Focus of March Webinar Series

This March, the Sage Politics team launches its first Politics Webinar Week. These webinars are free to access and will be delivered by contemporary politics experts —drawn from Sage’s team of authors and editors— who range from practitioners to instructors.

New Thought Leadership Webinar Series Opens with Regional Looks at Research Impact

Research impact will be the focus of a new webinar series from Epigeum, which provides online courses for universities and colleges. The […]

• Impact metrics
• Early Career
• In Memorium
• Curated-Collection Page Links
• Science communication
• Can We Trust the World Health Organization with So Much Power?
• Melissa Kearney on Marriage and Children
• True Crime: Insight Into The Human Fascination With The Who-Done-It

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Gender gaps in research productivity and recognition among elite scientists in the U.S., Canada, and South Africa

Creso sá.

Department of Leadership, Higher, and Adult Education, Ontario Institute for Studies in Education, University of Toronto, Toronto, Ontario, Canada

Summer Cowley

Magdalena martinez, nadiia kachynska, emma sabzalieva, associated data.

Data sharing is restricted by the University of Toronto Institutional Review Board for privacy and confidentiality reasons. Data can be shared upon request directed to [email protected] .

This study builds upon the literature documenting gender disparities in science by investigating research productivity and recognition among elite scientists in three countries. This analysis departs from both the general comparison of researchers across organizational settings and academic appointments on one hand, and the definition of “elite” by the research outcome variables on the other, which are common in previous studies. Instead, this paper’s approach considers the stratification of scientific careers by carefully constructing matched samples of men and women holding research chairs in Canada, the United States and South Africa, along with a control group of departmental peers. The analysis is based on a unique, hand-curated dataset including 943 researchers, which allows for a systematic comparison of successful scientists vetted through similar selection mechanisms. Our results show that even among elite scientists a pattern of stratified productivity and recognition by gender remains, with more prominent gaps in recognition. Our results point to the need for gender equity initiatives in science policy to critically examine assessment criteria and evaluation mechanisms to emphasize multiple expressions of research excellence.

Introduction

There is growing recognition in science policy debates of the interplay between gender, research productivity, and recognition in academic science [ 1 , 2 ]. Gender gaps are well documented in the participation of women in the scientific workforce, in their progression through senior and leadership positions, in earning grants and awards, in publication and citation rates, and in the length of their research careers [ 3 – 7 ]. A recent meta-analysis shows persistent gaps in research productivity and impact between man and women, and some evidence of gender bias in the assessment of research records [ 8 ]. Studies show that women in science are under-cited [ 4 , 9 , 10 ], under-paid [ 9 ], under-promoted [ 3 ] and professionally under-recognized [ 6 , 11 ] relative to their male counterparts. Moreover, relatively few women reach senior positions despite the growing number of women moving into doctoral studies and academic careers [ 3 , 7 , 12 ]. Gender inequalities continue to persist despite a number of policy initiatives and instruments at national levels aimed at redressing them [ 12 – 14 ].

In light of these disparities, scholars have questioned the meritocratic assumptions undergirding policy initiatives aimed at promoting research excellence [ 15 – 19 ]. Some highlight the role of explicit and implicit biases in the assessment of otherwise similar careers, indicating that women tend to get less recognition than men for similar research records [ 20 , 21 ]. Others contend that the metrics used to gauge research performance are unfair towards women [ 22 – 24 ]. Awareness of these issues has prompted science policy initiatives in multiple countries. In Canada, the low representation of women among government-funded research chair holders has motivated reviews and policy measures to address gender equity [ 2 , 25 ]. The promotion of gender initiatives in science has been a vital part of the European Union’s research policy for the last two decades, with a number of special projects funded by the EU to address the gender gap in science at different levels [ 1 ]. Despite many policy intentions and initiatives, the unequal recognition of scientific performance among male and female scientists is still prevalent in science [ 26 ].

Speaking to this problem, a few studies find gender gaps among the most productive scientists in their fields [ 27 ]—gaps which in some cases can be higher than for the general population of researchers [ 28 , 29 ]. Some studies showing that men are disproportionally represented among the most prolific researchers in STEM disciplines suggest that women may have to accumulate more knowledge, resources, and social capital to overcome biases and achieve similar publication rates as their male counterparts [ 29 ]. These studies tend to define “star scientists” by high publication and citation rates, invariably finding men to be overrepresented in this rarified segment of the population of researchers. Hence, by definition, elite scientists in these studies are those identified by the outcome variables used to measure productivity and recognition.

This study extends these efforts by considering the meaningful role of career stratification in academia. Prevalent approaches in the literature on gender and research productivity have in some ways ignored important markers of stratification in scientific careers. Generally, studies investigate large samples of researchers by drawing on bibliometric datasets and comparing all authors with publication records (see [ 8 ] for a comprehensive meta-analysis). These studies largely ignore the material and symbolic resources researchers draw from in their research careers, which accrue from being affiliated to high-status institutional settings and holding prestigious academic appointments [ 30 – 32 ]. As these appointments are less accessible to women on average, these need to be accounted for in explanations of gender gaps among scientists who are considered “elite”.

To consider the stratification of academic careers, this study departs from the usual approach of using bibliometric databases to define the sample of researchers to be investigated and identifying elites by high publication and citation rates. Instead, we chose a type of academic distinction that could be used to identify elite scientists as judged by peers across national settings: research chairs. Previous studies identified productivity gains among scientists selected and funded as chairs in comparison to non-chairs [ 33 , 34 ]. Sampling research chair holders allows us to isolate men and women who have been recognized as productive and meritorious scientists. As part of the general gender gap in science concerns the lower rate at which women achieve senior research positions, studying chairs minimizes that source of difference between men and women and allow us to investigate potential disparities in productivity and recognition among elite scientists.

In this paper we explore how the advantages of holding research chairs intersect with gender–do differences in productivity and recognition between men and women that have been described in previous studies hold among elite scientists who enjoy the resources and status of chairholders? To form our sample, we selected government-funded research chair programs in operation for at least five years that aim to recruit and retain senior scientists. We identified suitable programs in Canada, South Africa and in the US states of Georgia, Florida, and Kentucky. Subsequently, we created a unique, hand-curated dataset including 237 chairs and a control group of 706 non-chair peers identified from the same academic departments.

As the length of research careers is an important confounding variable in research productivity and recognition [ 7 ], our study focuses on research output over the five-year period following the appointment of research chairs. With this approach we sought to determine whether gender remains a factor in productivity and recognition during periods in the academic careers of elite scientists when they are expected, as a function of their appointments as research chairs, to be at their peak. Furthermore, we use the control group of peers to verify whether any similarities or differences between genders are unique to elite scientists. Our results show a persistent pattern of stratified productivity and recognition, which is consistent with the literature and yet intriguing considering the expected effects of prestigious academic appointments and resources on scientific careers.

Conceptual framework

Our study is grounded in the sociology of science that has examined the relationships between social structures and research activity. One of the central contributions of sociologists of science is the investigation of how the reward system in science determines research productivity. In the idealized ‘Mertonian’ world of scientific research [ 35 , 36 ], scientists are motivated by being the first to communicate an advance in knowledge and by getting the recognition awarded by the scientific community in the form of publications, citations and prizes. Peer recognition is the basic form of social reward in science from which other extrinsic rewards may be consequential, such as salary increases, career promotion and research funds, all of which usually progress in accordance with the degree of recognition achieved [ 37 ]. Thus, sociologists argue that the recognition and validation of researchers’ contributions to their field are crucial determinants of research productivity [ 36 , 38 , 39 ]. We frame our focus on research chairs as elite scientist through the concept of cumulative advantage.

Cumulative advantage and stratification in science

The phenomenon when more productive scientists get more recognition that supports their further productivity has been introduced by Merton as the principle of cumulative advantage [ 40 ]. His discussion of the ‘Matthew effect’ explains how the stratification in science unfolds when, for a variety of reasons, researchers tend to choose their readings on the basis of an author’s reputation and, as a result, two publications of equal merit will be unequally recognised. Overtime, the growing prevalence of ‘Big Science’ has had an impact on the dynamics of recognition in many disciplines. Scientists connected to large scale research consortia tend to reap the benefits of higher citation rates, although authorship contributions become increasingly difficult to assess [ 41 , 42 ]. Nonetheless, the principle of cumulative advantage has been a dominant theme in the studies of stratification in science. The widespread acceptance of the cumulative advantage hypothesis has been explained by its applicability in examining inequality of productivity and recognition in science [ 43 – 45 ].

The ‘Matthew effect’ has been confirmed through numerous empirical studies on scientific careers [ 46 – 49 ]. A recent study analyzed why scientists with similar backgrounds and abilities often end up achieving very different degrees of success, using data from a large academic funding program [ 50 ]. The results show that “winners just above the funding threshold accumulate more than twice as much funding during the subsequent eight years as nonwinners with near-identical review scores that fall just below the threshold” [ 50 ].

The gender gap in scientific productivity and recognition

As noted in the introduction, the literature has long documented the ‘productivity puzzle’ [ 51 ] whereby women publish less and are less cited than men [ 44 , 52 – 54 ]. The lower recognition and misattribution of work by female scientists, called the “Matilda Effect”, a phenomenon documented throughout history [ 55 ]. A recent meta-analysis suggests that the research productivity gap has remained consistent over generations since the mid-twentieth century [ 8 ]. Others have recently found that the growing participation of women in science over the past 60 years was accompanied by an increase of gender differences in research performance [ 7 ]. By reconstructing the publication history of over 1.5 million authors from 83 countries and 13 disciplines whose publishing career ended between 1955 and 2010, the study found that 35% of all active authors in 2005 were women comparing to only 12% of those in 1955. At the same time, the gender gap in total productivity rose from nearly 10% in the 1950s to around 35% gap in the 2000s [ 7 ]. Research also shows that the scientific awards won by women tend to be lower status [ 6 ].

The persistent evidence that men publish more than women throughout their careers has stimulated research looking for possible explanations. Thus, sociological research on academia suggests various factors which may explain gendered productivity and recognition: differences in family responsibilities [ 56 , 57 ]; different time use patterns as women dedicate more time to serve on committees, teaching and mentoring students [ 58 – 60 ]; unequal resource allocation [ 61 ]; different patterns in academic collaboration and networking [ 11 , 48 , 62 ]; and gender bias in peer-review [ 63 ]. The literature documents various forms of gender stratification in academic careers [ 6 ]. Previous descriptions of changes in the representation of women in science over time point to their increased presence at lower-ranking positions, holding less-prestigious awards, and working in marginalized subdisciplines that receive less funding and lower recognition [ 3 , 4 , 6 ]. So, despite an increase of women in the “pipeline” of scientific disciplines, stratification manifests in the niches and career levels they reach [ 5 ]. These social differences reflect the unequal accumulation of advantage among men and women in academia, which help explain gender differences in scientific careers [ 51 ].

This study frames research chairs as a source of advantage, as it provides material and symbolic resources to their holders who are already recognized and productive researchers. As such, they reinforce their reputation and support further scientific achievement. A focus on research chairs allows us to identify scientists of both genders who have undergone peer selection processes that designate them as part of an academic elite. These processes are arguably qualitatively and expert-based, as research chairs are usually vetted by search committees and their appointment is regulated by norms emphasizing research excellence.

Thus, we sought to identify research chair programs in different contexts to establish a sampling frame of chairholders. We focused on programs aimed at recruiting mid- to senior-level researchers in the sciences for long-term or permanent positions at the host university, employing “excellence”-related criteria. We selected two national policy initiatives—the Canada Research Chairs programs and the South Africa Research Chairs Initiative, and three state-level programs-the Georgia Research Alliance, the Kentucky Endowment Match program, and the endowment match program in the state of Florida.

The Canadian Research Chair (CRC) program was introduced in 2000 to attract and retain two thousand researchers with approximately $900 million investment from the federal government [ 34 , 64 ]. As of June 2019, 1,836 CRCs have been awarded to researchers at 70 universities and affiliated institutes and hospitals ( www.chairs.gc.ca ). Women represent approximately 34% of chairholders in 2019. Similarly, the South African Research Chairs Initiative (SARChI) was established by the Department of Science and Technology of South Africa in 2006 to attract and retain researchers in public universities to support excellence in research [ 33 , 65 ]. The program was designed based on the CRC program experience [ 33 ]. Since its implementation, the initiative has awarded 150 chairs in 21 public universities.

The three US state programs selected were part of a wave of “eminent scholars” programs introduced in the United States since the early 1990s to fund research-oriented professorships [ 66 ]. With the aim of attracting leader scholars to individual states, American research chair programs began in Georgia and spread across the United States through the 1990s and early 2000s, usually emphasizing fields of science and technology with economic potential [ 67 ].

Our unique dataset includes 943 researchers (237 chairs and 706 non-chair peers) along with data about their scholarly output during a five-year period. We included chairs appointed since 2000 (when the CRC program was created) in science and engineering disciplines. Through this approach we drew a sample of 237 research chairs: 264 in the US, 497 in Canada, and 182 in South Africa. Employing a matched-peers research design, we then identified a control group of 3 non-chairholders drawn from the same department as each chair, at the same academic rank, same gender (where possible) and with similar seniority (as determined by time since obtaining PhD). We gathered data from the open access self-reported resources as personal pages/CVs at university websites, LinkedIn profiles, and personal websites. The research team met during data collection to monitor the construction of the dataset and ensure consistency in the application of the selection criteria.

To measure scientific productivity, we retrieved the total number of papers published per year over a 5 year-period from Thomson Reuters Web of Science, starting one year after their appointment as chair, to capture publications more likely to have resulted from research conducted as chairholders. As explained above, we would expect to see high levels of productivity during this period as it arguably represents a high point in the chairs careers; not only have they been recognized as productive and holding potential for continued productivity and impact in their fields, but they also count on the material and symbolic resources associated with their chairs. Data from peers in each chairs’ department were gathered from the same time period to allow for direct comparison of chairs and their peers. We measure recognition through the total number of citations for the publications in this period, also retrieved from the Web of Science.

Data analysis

To analyze the data, we performed Poisson regressions to determine the relationship between the independent variables of status as chair, discipline, country, and gender and the dependent variables of the number of articles published and the number of citations on those articles. For ease of interpretation, we calculated Incidence Rate Ratios for each relationship of interest ( Table 1 ).

* p < .05

** p < .01

*** p < .001.

In order to verify whether differing self-citation rates between male and female authors [ 8 , 68 ] affect the citation counts recorded in our dataset, we conducted an additional analysis of articles from a sub-sample of the researchers in our dataset. We randomly selected 20% of all authors (n = 188) and compiled all their publications in the corresponding five-year period described above using the Web of Science search function (N = 3918). To subject these authors to a stringent test of self-citation practices by gender, we focused on papers with 1 or 2 authors, which a large-scale study of 1.5 million publications has found to tend to have the most self-citations [ 68 ]. We then categorized these papers by gender, including female solo and duo authorship (N = 34), male solo and duo authorship (N = 212), and mixed-gender authorship (N = 116). Next, we ran two simple regressions of weighted average self-citation counts on three categories of authorship gender, including male and female self-citation in the case of mixed-gender authorship.

The outcome variables are productivity ( articles ) and recognition ( citations ). The sample had a mean number of publications of 27.43 and a standard deviation of 29.44. Mean number of citations in the sample was 256.62, with a standard deviation of 542.38. The predictor variables in this study are chair (chairship status) and male (gender), and the control variables are discid (discipline), countryid (country), and yeardeg (year of degree). The dummy variable chair describes whether a researcher is a chair or a peer and has values “1” as chair and “0” as peer. The dummy variable male describes the gender of the researcher and has values “1” for men and “0” for women. In the sample, there are 778 men and 165 women. The categorical variable discid describes the discipline of the researcher and has values “0” for life sciences, “1” for engineering and computer sciences, and “2” for physical sciences. In the sample, there are 638 researchers in the life sciences, 178 researchers in engineering and computer sciences, and 127 researchers in the physical sciences. The categorical variable countryid describes where the country of the researcher and has values “0” for US, “1” for Canada, and “2” for SA. The continuous variable yeardeg represents the year when the researcher received his or her final degree.

In our subsample of articles examing self-citation, the outcome variable is the weighted average of a self-citation dummy variable for each article ( wtselfct ). The variable wtselfct describes whether self-citation occurred by an author, weighted by the number of authors of that gender on that paper and can have values of “0” if no self-citation occurred, “1” if self-citation occurred once for each author, and “0.5” if self-citation occurred once for one out of two authors. The predictor variable was authorship gender category ( gencatgrp ), which had values of “0” for articles authored by one or two male authors, “1” for articles authored by one or two female authors, “2” for articles authored by a female-male group counting male self-citation in subsample regression 1 ( Table 9 ) and counting female self-citation in subsample regression 2 ( Table 2 ).

Before going into the main analysis, we verified whether research chairs in our sample are indeed more productive and recognized than their departmental peers. As expected, research chairs outperformed their peers comfortably. Accounting for all control variables, research chairs published 81% more articles than their non-chair peers (Tables ​ (Tables1 1 and ​ and3) 3 ) in the five-year period of interest (p<0.001), and were cited 90% more than their peers (Tables ​ (Tables1 1 and ​ and4) 4 ) in the period (p<0.001). They can thus be safely considered as elite scientists, as their productivity and recognition is well beyond those of their departmental colleagues of similar seniority and academic rank.

Turning now to gender, we identify a pattern of stratified productivity and recognition involving male and female chairs and peers, with gendered differences in productivity and recognition differing between peers and chairs. Consistent with the literature, when considering both chairs and peers together, men in our sample generally produced more articles and were cited more times than women. All other things being equal, men published 16% more articles than women (p<0.001) (Tables ​ (Tables1 1 and ​ and3). 3 ). Furthermore, men were cited 68% more than women (p<0.001) (Tables ​ (Tables1 1 and ​ and4 4 ).

So, do things look different among elite scientists? Among the chairs, men published 30% more articles and were cited 64% more than women (p<0.001) (Tables ​ (Tables1, 1 , ​ ,5 5 and ​ and6). 6 ). However, the difference in publication activity between men and women in the peer group was not statistically significant (p = 0.081) (Tables ​ (Tables1 1 and ​ and7), 7 ), but the former were cited 62% more than the latter (p<0.001) (Tables ​ (Tables1 1 and ​ and8). 8 ). This finding suggests that, while there is no productivity gap between male and female peers, male scientists are nonetheless more frequently cited than their female peers. However, differences in productivity are marked between genders among elite scientists, and the recognition gap remains notable.

Finally, our testing for self-citation patterns in a sub-sample of articles showed that differences between groups (with male solo and duo authored papers as the references category) were not statistically significant ( Table 9 ). We do not claim these represent definitive proof, as it was not the purpose of this study to investigate citation practices of research chairs; that would entail an entire study on its own. But this analysis suggests that self-citation does not seem to represent a major threat to our model.

Our results show a pattern of stratified productivity and recognition among elite scientists: men outperform women in the number of publications, and receive substantially more citations. While this pattern is consistent with general findings in the literature, our study provides important qualifications to previous studies, and points to implications for gender equity initiatives in science policy.

First, our research design employed a strategy that defines elite scientists not by the outcome variable of interest as in other studies [ 28 , 29 ], but by their recognized standing in their fields as evidenced by their appointments. Their superior research performance relative to departmental peers confirms that their status is justified. Hence, our gender comparison entails an objective selection of scientists that belong in a research elite within their national, institutional, and disciplinary contexts. Among those scientists who have been regarded as sufficiently successful and productive to deserve an appointment as research chair, the gender gap remains. Therefore, our results adds to previous studies that identify an overrepresentation of men among the most productive and cited scientists [ 23 , 28 , 29 , 69 , 70 ].

Second, research shows that one explanatory factor for different productivity and citation rates is the lengh of research careers. A study examining 1.5 million researchers from 83 countries and 13 disciplines found the career length of women to be 1.7 years shorter than men’s [ 7 ]. After accounting for how long researchers actively published, annual differences in productivity became negligible (0.01 paper/year). Disparities in citation rates become about three times smaller when the researchers compared a matched sample based on career length. In contrast, our results show marked differences in productivity and recognition that cannot be attributed to the span of scientists’ careers. By looking at a delimited period of time when high productivity and recognition would be expected, as a function of research chair appointments, clear gender stratification remains.

Third, while earlier studies suggested that women's lack of recognition through citations could result from differences in publication rates [ 71 ], our study adds evidence that the gender recognition gap for elite scientists is more significant than the gender productivity gap. That is, despite sometimes relatively small differences in publication rates, men and women in this study differed greatly in the number of times their publications were cited. Among research chairs, which arguably includes well recognized scientists, the gender disparity in citation rates remain as large as that in the peer group. Considering previous studies showing that men tend to self-cite more often than women [ 4 , 68 ], we analyzed the citation patterns in sub-sample of articles and did not find significant differences between genders. While admittedly limited, this analysis suggests that the large citation advantage of men likely results from other the other factors that the literature has explored that contribute to the “Matilda effect” [ 4 , 11 ].

One limitation of this study is our inability to capture other indicators or research productivity, such as research funding obtained externally, PhD students supervised and graduated, and post-doctoral supervision, which go beyond the bibliometric data included in our analysis. Still, publication remains the prime currency in academic evaluation systems, providing a generally accepted measure of scientific productivity. Moreover, we are unable to explain why disparities remain among research chairs, although the literature reviewed above provides a number of possible explanations. Future research might combine the results of quantitative studies with qualitative approaches to examine in detail whether potential gender differences in how elite scientists form their research preferences, priorities, collaboration practices, and publication strategies might explain the disparities in productivity and recognition.

In conclusion, this paper adds evidence on the gendered nature of bibliographic indicators of merit among elite scientists. The use of such indicators pervades policy initiatives to promote research excellence and contribute to decisions that further reinforce gender disparities. A telling example comes from one of Canada’s federally funded programs that appointed only men to 19 highly prestigious research chairs, prompting a policy review and subsequent initiatives emphasizing gender equity [ 72 ]. Besides introducing a policy emphasis on gender representation in awarding chairs as in Canada’s case, science policy makers might critically examine the implications of relying on bibliometric indicators commonly used to establish quality or excellence when evaluating researchers. These concepts are recognized as “essentially contested” by science policy makers, and yet they often involve “inescapable simplifications” through the use of readily available quantitative indicators [ 73 ]. Acknowledging the gendered nature of bibliographic indicators of research achievement and impact allows for decision makers to thoughtfully design assessment mechanisms and evaluation criteria that emphasize multiple expressions of research excellence, going beyond the volume of publications and citations.

Funding Statement

C.S. is supported by the Social Sciences and Humanities Research Council of Canada through grant no.435170605. https://www.sshrc-crsh.gc.ca/home-accueil-eng.aspx The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability

• Help & FAQ

Gender and the research excellence framework

• Business School

Research output : Chapter in Book/Report/Conference proceeding › Chapter (peer-reviewed) › peer-review

T1 - Gender and the research excellence framework

AU - Yarrow, Emily

M3 - Chapter (peer-reviewed)

SN - 9789463511414

SN - 9789463511421

BT - EqualBITE Gender equality in higher education

A2 - Robertson, Judy

A2 - Williams, Alison

A2 - Jones, Derek

A2 - Isbel, Lara

A2 - Loads, Daphne

PB - Sense Publishing

Becoming a World-Class University pp 77–90 Cite as

Excellence in Research

• Michael Arthur 4  
• Open Access
• First Online: 20 December 2015

10k Accesses

2 Citations

World-class universities are defined as providing an excellent undergraduate and postgraduate education, informed by both the quality and extent of their research activity and profile. They conduct research of the very highest international quality and their work informs some of the most important scientific, technical, arts, humanities and social developments in global human society.

You have full access to this open access chapter,  Download chapter PDF

1 Introduction

Although such world-leading research features prominently, most world-class universities pride themselves in the quality of their graduates and their ability to contribute to global society. To be recognised as being among the very best universities in the world, both education and research must be at the very highest international level of quality, and there is evidence that both are having a significant impact on global society, through translation, innovation and policy formation.

The relentless pursuit of new knowledge is a characteristic of global human endeavour that is of fundamental importance to the future of our planet. Research excellence may be provided by universities, national institutes, government agencies or by industry, but it is critically important that it develops widely across the entire spectrum of global cultures. This diversity of engaged cultures is a critically important element of the most creative research environments and it is an essential element of solving the world’s most complex problems.

2 Definition of Research

There are many different definitions of research. The OECD defines research as “creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications” [ 1 ]. This definition works well across the panoply of disciplines, particularly if the phrase ‘new applications’ is interpreted broadly to include learned contributions to the arts and humanities and to social and public policy.

3 What Does Research Excellence Look Like

Research excellence is clearly discipline specific, but a common characteristic of world-class researchers is that they are involved in shifting paradigms and creating new models of human understanding. They are actively involved in constructively criticising perceived wisdom in a specific field. Their views are usually based on new observations, including qualitative observations or quantitative scientific evidence, or new interpretations that ultimately support logical conclusions. Not infrequently, such observations are stimulated by the application of new technologies and methodological advances.

The world’s greatest universities have a critical mass of individual researchers, or research teams, capable of performing at this level and a concentration of research excellence that allows or facilitates highly creative approaches to tackling some of the world’s most complex and difficult problems across disciplines [ 2 ]. Such ‘concentration’ of research excellence is a fundamental concept that every institutional leader must foster and support. At the very highest levels of world-class performance universities demonstrate this level of excellence, not only at the level of specific individuals, but also within and across multiple disciplines.

World-class research universities are typically engaged in trying to solve the most complex and intractable problems facing their own societies, or global society, in a coordinated and systemic manner across multiple departments, and in confident partnerships with other world-class institutions, local or global industry, governments and ministries and their health care, education, social care and public policy systems. Being an effective and seriously engaged partner is now an essential aspect of being a world-class university.

No world-class university can be universally excellent at everything and thus each institution must decide where to focus institutional efforts and resources. Institutional leaders must be bold enough to focus resources where they might have the greatest impact on research performance and profile. Deciding where an individual university can be truly distinctive and have the greatest international impact in its research activity is of fundamental importance to achieving world-class status.

4 The Integration of Research and Teaching in World Leading Universities

There are somewhere in the order of 17,000 higher education institutions across the globe and a conservative estimate suggests that at least 1,000 of these are actively and effectively engaged in research at scale. It should also be acknowledged that this research is equalled in volume by research conducted in non-university public institutions and by industry. This is an incredible force for the advancement of global human society and it is clear that high quality research is now widely distributed across the world.

Universities that are truly research-intensive must focus on ensuring that their students benefit from an education that is carefully and fully integrated with the research profile of the institution and its research partnerships. My personal philosophy is that all students (UG or PGT or PGR) should benefit from becoming involved in the process of ‘research’, irrespective of discipline. This is fundamentally important in developing students’ life skills and equipping them for their future. By engaging in the research process, students will learn to become critical independent thinkers and problem solvers. They will understand how knowledge is created, how it inevitably shifts with time, and they will be exposed to, and become personally experienced in, dealing with uncertainties at the ‘edge of knowledge’. They will also become more experienced in working in a team and actively engaged in improving their ability to communicate effectively. Graduates with this background experience are highly creative and of great value to the organisations that they join for employment. Many will become ‘leaders of the future’ because of their ability to research background information and analyse problems from first principles and subsequently exhibit confidence in employing innovative approaches and methods of problem solution.

5 The Role of High Quality Doctoral Research Training

There is a wide variety in the approach to doctoral level study across the globe. Most systems select the highest performing students at undergraduate and master’s level study and then engage such individuals in developing their research skills through a combination of more detailed taught programmes on research methodologies and techniques as well as their own individual research projects. High quality and dedicated research supervision by experienced supervisors is essential and dual supervision (by primary and secondary supervisors, usually bringing different expertise) is now commonplace. In the UK, research funders, together with universities, have been involved in creating the concept of high quality doctoral training centres that provide world-class doctoral training and experienced supervision of the highest quality. Doctoral students must be exposed to training in ethics, research integrity and the importance of engaging the public in their research findings as well as being intensely focused in their own independent research projects. A critical mass of PhD students is essential to creating a sense of community and collegiality in a research-intensive university. This relates not only to the absolute number of PhD students in the university, but also to the manner in which they are organised. The concept of high quality experience as PhD students is a central feature of many graduate schools and doctoral training centres. At UCL we currently have in the order of 4900 PhD students in our graduate school and doctoral training centres, in a total student population of just over 30,000 students.

The duration of doctoral level training and research varies across the globe from 3 to 7 years and is also quite discipline specific. A typical such duration in the UK is four years. The centre piece of PhD student training is the creation of a doctoral thesis that is ultimately subject to detailed external scrutiny by the world’s preeminent expert reviewers. The creation of new knowledge and an ability to demonstrate a logical and integrated set of arguments and research findings lies at the heart of every successful doctoral thesis.

6 The Postdoctoral Period and Breaking Through as an Individual Researcher and Principal Investigator

Successful completion of a PhD thesis is, however, only the beginning of a research career in most disciplines. A period of postdoctoral work and further study in the order of 3–9 years is now commonplace in most world-class universities, prior to individuals taking up their first academic appointment. This period usually involves moving to a different institution, or country, exposing postdoctoral students to a different research culture and, most importantly, to fresh ideas and techniques. In my opinion, this is the most formative period of any research career.

A world-class university will create a supportive and superb ongoing training environment for its postdoctoral staff, with continued expert oversight, support and advice, coupled with constructive challenge. During this period, postdoctoral staff will usually begin to develop and pursue their own independent ideas and acquire an increased level of expertise in their specific field. Wherever possible, highly talented postdoctoral staff should be encouraged to write grants for their own fellowship funding, such that they can formally, and more readily, branch out into an independent research career. Postdoctoral staff intending to pursue an academic career should also start to learn how to teach and ideally, they should gain experience and professional qualifications specific to teaching in a higher education institution during this period.

Inevitably, this period of development of the early stages of an academic career is highly variable around the globe. In some countries, it is possible to progress at an earlier stage to the junior phases of an academic career. There are also discipline specific variations. If however, you want to create a world-class research environment and profile for a university, then giving specific attention to this period of development of an independent research career is essential, as is the provision of ongoing mentorship, support and guidance. In the Western world, this is a phase of development through which only the very best successfully progress to develop their independent careers as principal investigators and this selectivity contributes to creating long-term world-class research performance in an institution.

7 The Importance of Cross-Disciplinary Research

While a great deal of research of world-class quality will be confined to individual disciplines, there is an increasing recognition that many of the world’s greatest problems need to be tackled by teams of researchers pulled together from different disciplines. At UCL we prefer the notion that each individual in a cross-disciplinary team brings their own disciplinary excellence to tackling a major problem and we therefore prefer the term ‘cross-disciplinary’ rather than ‘inter-disciplinary’ or ‘trans-disciplinary’. These terms are often used interchangeably, but the latter two are perhaps more indicative of individuals from one discipline beginning to work in another disciplinary area. Such discipline hopping may be productive, but it is not always successful, and in our view it is important to bring disciplinary excellence to the table as research teams are formed. When tackling complex problems a cross-disciplinary approach is highly creative as individuals bring their discipline specific approaches to the table and share ideas and concepts across the team.

There are numerous perceived barriers to the creation of effective cross-disciplinary research teams and it is important that they are recognized and dealt with by institutional leaders. The most common problems relate to the discipline specific nature of financial and other organisational structures within our institutions. Moreover most external organisations, for example many research funders, journals and learned societies are organised by discipline and this creates the perception that it is easier to obtain funding, to publish and to achieve personal recognition and promotion by remaining safely within a discipline’s confines.

It is important for institutional leaders to set a clear tone concerning the importance of cross-disciplinary research and to create opportunities for these activities to flourish. The rewards from getting this right are significant in terms of an institution’s global research profile. How best to achieve this will vary from one institution to another. At UCL we have developed the concept of a ‘grand challenge’ approach, and have actively promoted these via a series of ‘town meetings’ with expert panels, informed debate and open discussion. This has been sufficient to initiate collaborative cross-disciplinary discussion that has led on to new research and educational activities.

Each grand challenge is guided forward by a senior academic executive team. Small pump-priming grants are available to help new projects get off the ground, but as new activities develop, external grants are sought. Our current grand challenges are in Global Health, Human Well-Being, Intercultural Interaction and Sustainable Cities [ 3 ]. This approach is now sufficiently embedded in our research culture that it is also informing our Global Engagement (internationalisation) strategy and our approach to massive open on-line courses (MOOCs) and online distance learning.

Other approaches to promoting cross-disciplinary activity in research and education can also be taken. In higher education environments where philanthropy is commonplace and generous, then it is not uncommon to see very large donations given specifically to promote such new ‘centres’.

Another model is one that I learnt about from colleagues at Penn State University, USA, and then deployed at the University of Leeds, during my time there as Vice-Chancellor. This can be described as a ‘co-funding model’ and in my experience, it is highly successful. In essence monies are gathered into a central strategic fund on an annual basis (for example by applying a 1 % strategic surcharge), such that there is a central pot of recurrent funds to support cross-disciplinary activities of the highest quality. The highest quality projects are selected competitively via a bottom-up process, led by teams of faculty members. Such projects are then co-funded 50:50 by the centre jointly with the relevant departments, schools or faculties involved. Milestones of performance are set at the beginning of the co-funding award and these must include a significant return on investment in terms of raising external funding, industrial interaction, etc. Each cross-disciplinary activity funded in this manner is reviewed on a quinquennial basis and if successful, funding is continued for a second five-year period. One striking example of great success using this co-funding model was the creation of a new cross-disciplinary group in Leeds, who were interested in Water and its management in flood and drought, called Water@Leeds [ 4 ]. This is now a highly successful and internationally renowned group, achieving a return on investment (research £ raised, compared to £ invested), of greater than 10:1.

8 The Importance of Competitive Research Funding

The principal routes for funding of university research may vary considerably from country to country. There may be direct government funding of university faculty, research staff, projects and programmes, or there may be competitive government grant funding systems, charitable and generous funding, endowment support or funding directly from industry. Faculty members may have some or all of their salary funded directly by government, but in other jurisdictions, particularly the USA, many faculty members raise some or all of their salary via grant funding. There are relatively few sources of funding for international collaborative funding, with the most important being the European Union framework funding (the current programme is called Horizon 2020). More recently, the European Research Council has become a very important international source of funds for high quality response mode funding.

In the UK there is a dual funding system for the support of research. Each university receives a direct government grant, the size of which depends on an intermittent assessment of research performance, which is undertaken every 6–7 years. This used to be called the Research Assessment Exercise (RAE), or as it is known in its most recent form, the Research Excellence Framework. A detailed description of this process is available at [ 5 ].

In brief, it involves each member of academic staff submitting their best four research outputs over the preceding period, together with a description of the research environment, research strategy and plan, and evidence of the impact of that research activity in the form of ‘impact case studies’. For UCL this system currently informs the award of approximately £115 M per annum, as one limb of the dual support system.

The other element of dual support comes through government funding of the UK research councils, which is a typical peer-reviewed grant awarding system, largely organised by cognate groups of disciplines. This system drives the funding of individual research projects and programmes through a system of peer-reviewed grant awards, made to research teams or individual researchers via their institutions. From such government and other sources (charities, EU, Industry, other government departments and the UK National Institutes of Health Research) UCL achieved a total research income of £375 M last year.

Competition for research funding based on peer-reviewed grant submissions is commonplace across many countries, as it is held up as the best way of distributing scarce research funds (and in many cases, public money) to the highest quality research. In my experience, this mode of funding is a critically important element in driving forward personal research excellence. It is very demanding and exacting to write a grant that competes at the highest level, but in doing so, the quality of thought, preparation, methodology and preliminary data must all be in place, together with a carefully laid out research plan, prior to funding being ultimately awarded.

In many fields, this drives individual researchers to either acquire or develop the most advanced research methodologies that then allow them to tackle the more interesting and complex problems. This is particularly, but perhaps not exclusively, evident in Biomedicine and Biomedical Engineering in recent years, with the explosion of, for example, genetic manipulation techniques.

This competitive nature of research funding inexorably drives quality upwards across the entire research endeavour. It is clearly not a prerequisite for research excellence, but it certainly helps to drive it forward on a system-wide basis. In my opinion, the highly competitive nature of both limbs of our dual funding system is the key factor underlying the excellent, internationally competitive, research performance of the UK.

9 Writing Successful Research Grants

One of the most important transitions that every successful researcher has to undergo is to develop from being a good researcher at the bench, or in the library, into an independent principal investigator (PI). This is a difficult transition, but it is often the time when new ideas are born and exciting new paradigms develop. Critical to becoming a PI is the ability to attract competitive peer-reviewed grant funding, to allow you to explore your own ideas.

Grants that are funded have several key elements.

First and foremost, they must tackle an interesting problem or concept that excites the funding agencies and their boards and peer-reviewers to the extent that they envisage the results of the research having a significant impact on the field.

Many will have substantial arguments and/or preliminary data that support the case being made for funding. In my view, time spent on acquiring additional preliminary data to strengthen the case is extremely valuable.

They must have a clear and testable hypothesis and seek to deploy advanced technologies to help address some of the most complex of problems. For example, within the sciences, research that determines new mechanisms is of greater importance than that which is merely descriptive.

They must be very clearly written, such that peers can quickly assimilate the key points and are thus guided through the grant in the direction of a supportive decision.

In many years of active research, I gave the utmost care and attention to my grant writing skills and fortunately enjoyed considerable success. I was never complacent and I always asked my close colleagues and fellow professors (at least two independent and knowledgeable scientists) to read and critique my grants, well in advance of their submission. If that meant that I had to miss a grant deadline and submit later, then I always accepted that outcome. There is nothing more important than ensuring that grants are of the very highest calibre at the time of submission.

10 Research Leadership

World-class universities will typically be home to a significant number of outstanding research leaders and they will support and develop others to take up research leadership roles. Great research leaders have a number of important characteristics. First and foremost they are prominent figures in their respective fields through their own proven research excellence, which has usually been sustained over a prolonged period of time. They will be publishing at the highest international level, giving plenary session keynote speeches at international meetings, and perhaps winning major awards such as the Nobel Prize, the Kavli Prize, the Lasker Award or the Fields Medal.

They typically have a great awareness of the history of and a good feel for the future of their discipline. They are influential, set the pace for others to follow and tend to shape the future direction of the field. This is typically associated with an ability to constantly evolve their own research strategy, a significant level of personal persistence in seeking answers to complex research questions and a willingness to take risks in their approach.

Research leaders also support and nurture younger researchers and act as their mentors. The very best are generous with their time and create a research environment where others can flourish and follow in their footsteps. They are usually exceptionally talented, highly creative and effective at pulling together significant levels of research funding and they act as a centripetal force in terms of attracting the very best of the next generation (PhD students, postdoctoral staff and new faculty members) to come and work with them. They generate excitement and great research momentum in their home institution and act as role models for future generations of researchers.

Many of the skills required to become an excellent leader can be identified and taught through research leadership programmes that are aimed at up-skilling the younger generation of researchers. We have one such course at UCL, called ‘Leadership in Action’ [ 6 ].

This course is a three-day intensive programme designed to prepare researchers for leadership in their chosen research field or within the wider university community. The course is aimed at helping early career researchers to explore the concept of leadership and to develop confidence in their leadership style. It is intended to expose individuals to a variety of choices in how they lead and to provide them with understanding of the impact that may have on those they are leading. It is intended that course participants will learn how to influence people effectively towards a shared goal and that they will understand how all these skills will benefit them in their current research roles and beyond. The course is based on practical, experiential learning, rather than lectures, and it includes opportunities for all participants to lead a group project. Support and feedback are provided throughout the course on leadership style by expert coaches. It is our intention that this course will generate a cadre of early career researchers, equipped and skilled to become the distinguished research leaders of the future.

11 Research Integrity, Handling Research Data and Research Misconduct

It is incumbent on every world-class university to adhere to and promote the very highest standards of research integrity. This means creating an environment, which ensures that all research is conducted against a background of awareness of the key issues of transparency, honesty, collegiality, fairness and personal responsibility. Individual researchers (staff and students alike) must be trained in, and aware of, the institution’s, research funding agency’s and publisher’s research policies and practices, and must adhere to them and to the highest possible standards of good research conduct in their daily activity. This must include excellence in the design of research projects and frameworks, their subsequent operation and adherence to the principle that it is the personal responsibility of all involved to ensure that research data is robust and reproducible. Researchers must therefore demonstrate intellectual honesty at all times.

The very highest standards of research integrity encompass many different elements, all of which require compliance. In addition to those outlined above, these include:

Research ethics—all research should be conducted within the overarching framework of an ethical code. This is particularly important (and in most countries mandatory) if the research involves human subjects or animals.

The requirement to declare any actual, potential or perceived conflict of interest, whether pecuniary or non-pecuniary.

A responsibility to ask questions, to be aware and to report any reasonable suspicion of research or ethical misconduct.

A responsibility to record and store original research data in a format that substantiates the reporting and publication of research findings and that is accessible to others, if required. Many funding agencies and publishers have specific requirements for the duration of data storage of original data and it is critically important that such requirements are strictly adhered to. Research data must be stored in a secure manner, such that it remains authentic and complete.

Fairness and collegiality when dealing with research colleagues and collaborators, including the need to demonstrate fairness in peer review.

A requirement to avoid misrepresentation of personal contribution to a specific research project or publication.

It is important that institutions also have systems to identify, investigate and deal with research misconduct when and if it occurs. The latter takes many forms and this has been outlined by Research Councils UK as follows [ 7 ]:

Fabrication, or creation, of false data, or research documentation, such as consent to participate.

Falsification, which comprises the inappropriate manipulation and/or selection of data, imagery or consent.

Plagiarism, which comprises the misappropriation or use of others’ ideas, intellectual property or work (written or otherwise) without acknowledgement or permission.

Misrepresentation of data, duplication of publication, material interests, qualifications or experience, and of involvement in research.

Breach of duty of care, whether deliberately, recklessly or by gross negligence to include breach of confidentiality, placing individual research subjects in harm’s way, or not observing legal, ethical or statutory requirements and improper conduct in peer review, including failure to disclose conflicts of interest.

Improper dealing with allegations of misconduct, including attempts at ‘cover-up’ or reprisals against whistle-blowers, and failing to deal appropriately with malicious allegations.

The websites [ 8 – 10 ] provide a more detailed background to this subject for those looking for more in-depth information and for further guidance on how to promote good research conduct or how to further handle research misconduct.

12 From Research to Innovation and Enterprise

World-class universities drive innovation, entrepreneurship and enterprise through the generation of intellectual property and new ideas. They contribute to supporting local, national and international businesses and entrepreneurs and they help to generate economic growth. This relates, of course, to the commercial exploitation of their research activities, but is also related to the supply of graduates as highly talented individuals that contribute to the ongoing development of an economy. There are many different ways in which universities can contribute to economic growth through their own enterprise activities. These include:

The identification, protection and exploitation of intellectual property. All staff should be trained and supported by the University to recognise intellectual property and to learn how to protect it and where relevant, to exploit it. A dedicated professional team should be created to provide support for identifying novel concepts and ideas, which will not only develop strategies for their commercialisation, but also find investors to support them financially. The intention is that such a combined academic and professional endeavour, not only creates intellectual property, but also converts it into products of commercial value that can be licensed, and in some cases facilitated, to generate new spin-out companies. These, in turn, will create new products and businesses of long-term enduring commercial value. The complexity and risks involved in generating this level of success should not be underestimated, but the rewards for getting it right can also be significant. Guidance and advice should be sought from individuals that have successfully negotiated this space and that have first-hand experience of creating such high tech companies from their own intellectual property. Universities increasingly need to demonstrate to their own governments, and other funding agencies, that both their basic and applied research can bear such fruit.

The recognition that world-class universities must also support their students and staff to be entrepreneurial at a personal level is an increasingly important paradigm. This should include the recognition that new student and staff entrepreneurs will need advice to support and develop their own independent commercial ideas and business practice with potential for leading to new and successful businesses. Where possible, universities should provide the facilities within which student and staff entrepreneurs can conduct research of the highest calibre and further develop their own ideas. The facilities and the expertise contained therein should be readily accessible to businesses external to the University so that research and development ideas can be positively exploited.

World-class universities also need dedicated individuals and systems to interact with the external business world to provide expert consultancy and to support corporate partnerships. These are specialist areas, and individuals or teams dedicated to this sphere of external partnerships are important in presenting a professional interface for interaction with the business and corporate world. Such teams can also be extraordinarily important in improving the profile and reputation of a university with the corporate world on a global scale. Dedicated teams in this area are also extremely helpful in guiding academic staff’s navigation around this (perhaps previously) unfamiliar world and this helps enormously in managing risk and reputation.

13 Five Key Actions for KAU to Enhance Research Performance

King Abdulaziz University (KAU) is already a highly successful research-intensive university and a leading institution in the Middle East, with an established international profile. It is well led and has chosen to pull together a highly experienced, high profile International Advisory Board, of which I have the pleasure of being a member. I am still learning about KAU, and the following advice on five key actions is offered against that background.

It is essential to build both a critical mass of researchers and a concentration of research excellence in a defined and clearly identified number of high quality fields of research. Choices must therefore be made and resources focused on ensuring that these ‘peaks of excellence’ can perform at the highest level. Human capital in the form of world-class academic and research staff is the most important element of building this level of success.

As research performance improves, it is essential to think of the research pipeline and to recognise the importance of a clear ‘youth policy’ with respect to training the next generation. High quality doctoral training is essential, but this must also be aligned to very clear ongoing support and training during the postdoctoral period and into the early phases of a junior academic career.

Seeking external research funding from high quality research funding agencies that use international quality peer review is an excellent way of sharpening research protocols and for enhancing research programmes. It also drives the need to stay abreast of methodological advances, which ultimately feed into promoting the international quality of research activity.

Systems should be put in place to support and enhance cross-disciplinary research, allowing teams of researchers from many different disciplines to come together in order to tackle complex global and societal problems. Contributions to teamwork and working across disciplines must be recognised within the university’s promotion and reward systems.

Research findings are at the forefront of the process of innovation, but research alone does not drive economic growth and prosperity. Systems must be put in place to drive and support innovation such that new products can be licensed and/or new companies spun out of the university. This should include the concept of expert teams to encourage and support entrepreneurship, consultancy and greater interaction with industry.

14 Summary and Conclusions

The advancement of human society is critically dependent on the creation of new knowledge through the conduct of research in universities, institutes, government agencies and industry. In our global future, the world’s greatest problems will be tackled by teams of researchers from diverse cultures working across disciplinary, sectoral and national boundaries. World-class universities will be at the heart of this endeavour in many different ways. Our academic staff will not only be leading the immediate and long-term research effort, but they will also be closely involved in the training of the next generation of researchers through undergraduate, master’s, and doctoral level education and research supervision. The close integration of education of students (at every level) with research activity will continue to define the world’s leading universities.

Building a successful research-intensive university requires excellent institutional leadership that focuses on building world-leading quality in a defined number of research areas, as well as providing world-class research facilities. The ability to attract the very best research-active academic and postdoctoral staff and PhD students is of fundamental importance. Bidirectional student and staff mobility are of crucial importance for the creation of a positive research environment. Supporting these staff in their competitive grant writing is also critically important. Instruction on the key elements of ethics, research integrity and the manner in which primary research data should be handled and stored are now fundamentally important aspects of research training.

The future of the world’s leading universities is very exciting as we continue our current journey along the road of the ever-increasing globalisation of higher education. Those universities that rise to the challenges outlined in this chapter will be best placed to make significant impact and thereby achieve world-class university status in recognition of their contribution to the advancement of knowledge and the future of humankind.

OECD. (2002). Frascati manual: Proposed standard practice for surveys on research and experimental development (6th ed). Retrieved 27 May 2012 from www.oecd.org/sti/frascatimanual

Jewels in the Crown: The importance and characteristics of the UK’s world-class universities. http://russellgroup.org/JewelsInTheCrown.pdf

http://www.ucl.ac.uk/grand-challenges

http://www.wateratleeds.org

http://www.ref.ac.uk

http://www.ucl.ac.uk/hr/od/pdp/sdlia/index.php

http://www.rcuk.ac.uk/Publications/researchers/grc

http://www.grad.ucl.ac.uk/research-integrity

http://www.admin.ox.ac.uk/researchsupport/integrity

http://www.universitiesuk.ac.uk/aboutus/AssociatedOrganisations/Partnerships/Pages/ResearchIntegrity.aspx

Download references

Author information

Authors and affiliations.

University College London, London, UK

Michael Arthur

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Michael Arthur .

Editor information

Editors and affiliations.

King Abdulaziz University, Jeddah, Saudi Arabia

Osama Tayeb

Adnan Zahed

Maastricht University , Maastricht, The Netherlands

Jozef Ritzen

Rights and permissions

Open Access This chapter is distributed under the terms of the Creative Commons Attribution Noncommercial License, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and permissions

Copyright information

© 2016 King Abdulaziz University

About this chapter

Cite this chapter.

Arthur, M. (2016). Excellence in Research. In: Tayeb, O., Zahed, A., Ritzen, J. (eds) Becoming a World-Class University. Springer, Cham. https://doi.org/10.1007/978-3-319-26380-9_5

Download citation

DOI : https://doi.org/10.1007/978-3-319-26380-9_5

Published : 20 December 2015

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-26379-3

Online ISBN : 978-3-319-26380-9

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Research Excellence Framework

Securing a world-class, dynamic and responsive research base across the full academic spectrum within UK higher education

The REF is the UK’s system for assessing the quality of research in UK higher education institutions. It first took place in 2014 and 2021 . The next exercise is planned for 2029.

What is the REF’s purpose?

• To provide accountability for public investment in research and produce evidence of the benefits of this investment.
• To provide benchmarking information and establish reputational yardsticks, for use within the Higher Education sector and for public information.
• To inform the selective allocation of funding for research.

The Research Excellence Framework (REF) is the UK’s system for assessing the excellence of research in UK higher education institutions (HEIs). The REF outcomes are used to inform the allocation of around £2 billion per year of public funding for universities’ research. The REF is a process of expert review, carried out by sub-panels focused on subject-based units of assessment (UOAs), under the guidance of overarching main panels and advisory panels. Panels are made up of senior academics, international members, and research users.

Latest news

View all news

People, Culture and Environment Update March 2024

We are pleased to announce updates on the development of PCE for REF 2029, and associated community engagement opportunities. These are through:  We are committed to gathering input on PCE with a broad and inclusive approach and encourage all stakeholders to make the most of these opportunities to engage on this important issue for the…

The REF 2029 Open Access Policy consultation opens

The four UK higher education funding bodies have launched a consultation on the Research Excellence Framework (REF) 2029 Open Access Policy. The consultation will gather a deeper understanding of sector perspectives on key issues and impacts in relation to our policy proposals. The purpose of the REF 2029 Open Access Policy is to outline open…

People, Culture and Environment update January 2024

The REF Steering Group is pleased to announce an update on the development of approaches to the assessment of People, Culture and Environment (PCE) in the next REF exercise. A project has now been commissioned with Technopolis and CRAC-Vitae in collaboration with a number of sector organisations, which will develop indicators to be used for…

Quick links

About the REF

OA consultation

Publications

Contact REF

Logo guidelines

• Search Menu
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

The research excellence framework, topic modelling the research excellence framework, predicting environment scores, general discussion, acknowledgements, data availability, what is a high-quality research environment evidence from the uk’s research excellence framework.

• Article contents
• Figures & tables
• Supplementary Data

Matthew Inglis, Elizabeth Gadd, Elizabeth Stokoe, What is a high-quality research environment? Evidence from the UK’s research excellence framework, Research Evaluation , 2024;, rvae010, https://doi.org/10.1093/reseval/rvae010

• Permissions Icon Permissions

As part of the UK university sector’s performance-related research funding model, the ‘REF’ (Research Excellence Framework), each discipline-derived ‘Unit of Assessment’ must submit a statement to provide information about their environment, culture, and strategy for enabling research and impact. Our aim in this paper is to identify the topics on which these statements focus, and how topic variation predicts funding-relevant research environment quality profiles. Using latent Dirichlet allocation topic modelling, we analysed all 1888 disciplinary ‘unit-level’ environment statements from REF2021. Our model identified eight topics which collectively predicted a surprisingly large proportion—58.9%—of the variance in units’ environment scores, indicating that the way in which statements were written contributed substantially to the perceived quality of a unit’s research environment. Assessing research environments will increase in importance in the next REF exercise and the insights found through our analysis may support reflection and discussion about what it means to have a high-quality research environment.

For the past four decades, higher education institutions in the UK have been subject to evaluations of their research by the higher education funding councils. The first evaluation, the ‘Research Selectivity Exercise (RSE)’ took place in 1986 (for a history, see Bence and Oppenheim 2005 ). Over the years, and with six assessments between 1986–08, the RSE evolved into the Research Assessment Exercise (RAE) and then, in 2014, into the ‘Research Excellence Framework’ (REF). For each evaluation, university disciplines and fields of study were divided into ‘Units of Assessment’ (UoAs). The most recent assessment took place in 2021, with results published in 2022.

As well as name changes, the requirements for submissions have evolved (see Marques et al. 2017 ), from the “quick and dirty” ( Jones and Sizer 1990 ) approach taken in 1986 through to including/excluding particular categories of staff; changing the minimum/maximum numbers of publications per individual; the introduction of research environment statements (RAE 1996), and the introduction of impact case studies (REF 2014). Two constants about RSE/RAE/REF remain: the original principle of peer assessment, despite the rise of publication metrics in other domains, and the use of the results to distribute government funding.

The RSE represented the creation of “the first and most highly institutionalised research evaluation system worldwide” ( Marques et al., 2017 : 822). Since then, the RAE/REF has been widely discussed and used as a model for other countries (e.g. Geuna and Martin 2003 ) or resisted and rejected (e.g. Swedish Government 2016 ), but rarely adopted wholesale in countries internationally ( French, Massy and Young 2001 ; for overviews, see Sivertsen 2017 ; Thomas et al. 2020 ; Pinar and Horne 2022 ). Either way, the discourse of the RAE/REF reaches far beyond the UK.

Analysing the research excellence framework

Unsurprisingly, the RAE/REF has been scrutinized in terms of (i) critiques of the politics and methodologies that underpin the process and, (ii) quantitative and qualitative analyses of submissions, assessment processes, and results themselves. The former comprises a literature too vast to cover substantially here, but includes criticisms of the trend towards a competitive, neoliberal, and commodified higher education system (e.g. Fairclough, 1995 ; Brown and Carasso, 2013 ), of the impact of assessment on individual disciplines and interdisciplinarity (e.g. Pardo-Guerra 2022 ), and of unintended consequences (for overviews, see Gillies 2008 ; Brassington 2022 ; Pinar and Horne 2022 ; Watermeyer and Derrick 2022 ). The RAE/REF has driven both policy and debate in UK higher education, with a series of consultations, evaluations, recommendations, and iterated processes ( Manville et al. 2015 ; Curry, Gadd and Wilsdon 2022 ).

Another approach to evaluating and critiquing the RAE/REF focuses on the actual content of HEIs’ submissions to RAE/REF, using both quantitative and qualitative methods. Several of these studies have used similar text-mining or topic modelling and related methods to those used in this paper. Perhaps because of the increasing significance of research impact over the past decade ( Derrick and Samuel 2016 ; Kellard and Śliwa 2016 ; Sutton 2020 ; Jensen, Wong and Reed 2022 ), several studies have scrutinized the content and composition of case studies. For instance, a report commissioned by the UK research funding councils ( King’s College London and Digital Science 2015 ) used text-mining and qualitative analysis to provide an initial assessment of all REF2014 impact case studies, making observations about the diverse range of impacts, their underpinning research, and their global reach (see also Terämä et al. 2016 ). Reichard et al. (2020) conducted two studies using qualitative thematic and quantitative linguistic analysis of REF2014 impact case studies to identify what individual words and phrases were associated with high and low scores. They identified numerous “lexical bundles” associated with lower (e.g. “involved in”, “has been disseminated”, “the event”) and higher (e.g. “the government’s”, “in the UK”) scores, the former associated with describing activities and pathways to impact and the latter evidence of significant and far-reaching impact itself.

Within the wider literature on REF methodology and submissions, the least scrutinized aspect is the environment statement ( Thorpe et al. 2018a ). We now turn to discuss what we know and do not know about REF environment statements, starting by describing the submission requirements for the 2021 assessment.

The REF environment statement

The environment statement(s) and their submission requirements.

As noted above, an environment statement was introduced relatively early into the UK RAE/REF cycle. In 2021, two main changes occurred: the introduction of a pilot “Institutional-level environment statement”, and the removal of an ‘impact template’ from REF2014 and the incorporation of ‘research and impact’ as one environment statement in 2021.

The aim of REF2021 unit-level environment statements was to provide assessable information about each UoA’s “environment for research and enabling impact” (Guidance on Submissions, p. 82) and especially its “vitality” (“the extent to which a unit supports a thriving and inclusive research culture for all staff and research students, that is based on a clearly articulated strategy for research and enabling its impact, is engaged with the national and international research and user communities and is able to attract excellent postgraduate and postdoctoral researchers”) and “sustainability” (“the extent to which the research environment ensures the future health, diversity, wellbeing and wider contribution of the unit and the discipline(s), including investment in people and in infrastructure”) (Panel Criteria and Working Methods, p. 58). For REF 2021, detailed guidance notes and a template were provided to structure the information in four sections: (1) “unit context, research and impact strategy”; (2) “people, including: staffing strategy and staff development, research students, equality and diversity”; (3) “income, infrastructure and facilities”, and (4) “collaboration and contribution to the research base, economy and society.” The permitted length of the statement varied according to the number of staff in a UoA, from 8,000 (for a submission comprising 1–19.99 FTE) to 12,000 words (plus 800 further words per additional 20 FTE). The environment statement was worth 15% of the funding allocation. For Main Panels A, B, and C, each of the four subsections attracted equal weighting; in Main Panel D, sections (1) and (4) attracted 25%, the ‘People’ section 30%, and ‘income, infrastructure and facilities’ 20%.

Analyses of environment statements

To the best of our knowledge, no close analyses of REF2021 environment statements have yet been published, apart from Manville et al.’s (2021) real-time study of REF as it happened, including focus group research on academic and professional service staff experiences of completing them. However, research has been conducted on REF2014 environment statements. For example, Matthews and Kotzee (2022) analysed both REF (2014 submission) and TEF (Teaching Excellence Framework, 2017 submission) documentation with the aim of investigating links between research and teaching. They found that the term “research-led”, analysed in the context of its collocates, was often used in connection with teaching, and argued that “according to what universities themselves write in institutional texts, teaching and research are not always in a mutually beneficial entanglement, but often rather a one-way relationship in which research expertise and institutional prestige are used to bolster claims of teaching excellence” (p. 578).

Mellors-Bourne, Metcalfe and Gill (2017) also used text-mining to assess the level of engagement with equality and diversity in the ‘People’ section of REF2014 environment statements. This included evidence of participation in schemes such as ‘Athena Swan’ and Stonewall’s, and the relative frequencies of words pertaining to the topic: ‘equality’, ‘diversity’, ‘Athena’, ‘gender’, and ‘ethnicity’. Mellors-Bourne et al. found that statements focused predominantly on gender; that “the word ‘equality’ was used on average between once and twice within each environment statement” (p. 2), and that the Main Panels A and B (the STEM disciplines) mentioned ‘Athena’ more than twice as much as Main Panels C and D. The researchers also found “[e]vidence suggesting a positive relationship between REF research environment subprofiles (scores) and reference to key E&D terms within submissions, overall and at the level of the main panels” (p. 2). In REF2021, the focus on equality, diversity and inclusion in the guidance notes extended well beyond the ‘People’ section, presumably to encourage HEIs to demonstrate how EDI strategies and outcomes were embedded in all areas of research and impact.

Thorpe et al. (2018a , 2018b ) used computer-assisted text analysis to scrutinize the language content and ‘tone’ of REF2014 environment statements in business and management schools. They sought to understand whether the way environment statements were written differed between high and low scoring universities. They found that higher-ranked universities used less passive voice, were more coherent, adopted “a ‘finished article’ discourse rather than a ‘we are developing’ discourse”, cited “specifics rather than generalities”, and were more self-referential ( Thorpe et al., 2018a , p. 582). In terms of tone, the authors found that, perhaps counterintuitively, higher-scoring statements scored lower in terms of ‘activity’ tone “that evokes a ‘safe’, ‘staid’, ‘orthodox’, ‘conservative’, and ‘settled’ environment that is not disturbed (unduly at least) by reform, disruption, or major staff turnover” ( Thorpe et al. 2018b : 60). The authors concluded that “low-ranked universities could have achieved higher scores by reflecting on particular areas of word choice and the potential effects of those choices on assessors” ( Thorpe et al. 2018b : 53).

Like Reichard et al.’s (2020) analysis of impact case studies, Thorpe et al. (2018a , 2018b ) revealed the importance of language as well as content in the production of environment statements. They concluded that “the accompanying narrative played an important role in determining REF2014 environment scores” (2018a: 572). Furthermore, in contrast to impact case studies, which included corroborating evidence as part of submission, “little supporting evidence was required in environment submissions”, meaning that “there is potential for writing quality to have an even larger effect in environment submissions, and for HEIs to use language-related techniques to manage their image” ( Thorpe et al. 2018a : 574).

In June 2023 the ‘Future Research Assessment Programme’ (FRAP) published their Initial Decisions on REF 2028 ( Joint UK HE Funding Bodies 2023 ), although it has since been announced that the exercise will be delayed until 2029. Despite the Institution-Level Environment Panel Pilot Panel (Research England 2022) recommending the removal of unit-level environment statements and focusing instead on institution-level statements, the Initial Decisions propose the retention and assessment of both. In the 2029 exercise, ‘Environment Statements’ will be broadened to become ‘People, Culture & Environment (PCE) Statements’ with a concomitant increase in weighting from 15% to 25%. The Decisions state that “the collection of evidence for the people, culture and environment element will move towards a more tightly defined, questionnaire-style template that will create greater consistency across submissions and focus on demonstrable outcomes ( Joint UK HE Funding Bodies 2023 )”. At the time of writing this is yet to be developed and it is unclear what proportion of the submission will take a narrative format and what proportion will comprise data and evidence. However, given the qualitative nature of the dimensions being assessed, it is likely that there will be a significant narrative element. Furthermore, given the increased weighting of PCE, any such element should have an even greater bearing on overall results.

In sum, the focus on environment statements as a subset of research on the UK REF has, to date, been small. It has either been partial (e.g. has analysed just one UoA ( Thorpe et al. 2018a , 2018b )) or not focused on the most recent exercise. The aim of this paper is to investigate whether thematic patterns may be identified in the 2021 submissions, whether such patterns may be correlated with scores, and what this might teach us about crafting future environment narratives. We begin by introducing the method we adopted, latent Dirichlet allocation topic modelling.

Topic modelling is a computation method that seeks to analyse the content of many texts by identifying a small number of semantically connected themes or topics ( Blei, Ng and Jordan 2003 ). The aim is to take a collection of unstructured texts and identify the topics they cover by studying their words. For example, if a document uses the words ‘water’, ‘sand’, and ‘swimming’ with an unusually high frequency, this may constitute evidence that the document is about beaches. One way to understand the topic modelling approach is to think about how to create documents from a predefined set of topics, defined to be probability distributions over words. For instance, our beach topic might assign very high probabilities to ‘water’, ‘sand’ and ‘swimming’, medium probabilities to ‘inflatable’, ‘spade’ and ‘picnic’, and low probabilities to ‘dioxide’, ‘carpentry’ and ‘veneer’. Similarly, we might define a topic about Greece (which might perhaps have a high probability associated with the words ‘Greek’, ‘Athens’, ‘souvlaki’ and so on). If we wanted to create a new document about beach holidays in Greece, we might choose 30% of the new documents words to be from the beach topic, 30% from the Greece topic, perhaps 30% from a travel topic, and 10% from other topics. A document about Greek beach holidays can then be created by simply sampling words from each topic using the appropriate probabilities. This method makes two simplifying assumptions. First, the ‘bag of words’ model of text is adopted by ignoring word order; and second, so-called ‘stop words’ (words such as ‘the’, which is topic independent) are ignored.

The topic modelling approach can be thought of as carrying out this document construction process in reverse. We start with a large collection of texts, assume that they were created in this way, and then computationally identify the topics that would have been most likely to lead to these documents using a latent Dirichlet allocation (LDA) algorithm ( Blei, Ng and Jordan 2003 ; Grimmer and Stewart 2013 ). Topic modelling adopts a grounded theory mentality: the analyst has no preconceived ideas about what topics will be identified, instead topics/themes emerge from the analysis process. Once topics have been identified, the semantic content of each document can be analysed by studying the topic composition of each document. For instance, we may find that Document 1 contains 6% of words from Topic 1, 30% from Topic 2, 0% from Topic 3, and so on.

We downloaded all 1888 unit-level environment statements (a total of 18.0 m words) from the REF website. These were converted from pdf into plain text using the UNIX pdftotext command ( Poppler 2022 ). We used MALLET (version 2.0.8RC2), a UNIX topic modelling tool ( McCallum 2002 ), to calculate possible models, using MALLET’s default list of stop words.

Topic modelling requires that one specifies how many topics the LDA algorithm should identify. By making different choices researchers can specify the granularity of their analysis. We adopted the perplexity approach to decide on the number of topics. Each model can be assigned a perplexity, which is analogous to a model fit ( Blei, Ng and Jordan 2003 ). Perplexity can be calculated by fitting a model with a specified number of topics to a subset of the documents, and then assessing its fit to the remaining documents. One can always reduce the perplexity (or increase the fit) by fitting a model with a larger number of topics, although at some point the benefit of doing so will be offset by the increased difficulty of interpretation. Jacobi, van Atteveldt and Welbers (2016) suggested using a method similar to the scree test often used in factor analyses: by calculating the perplexity of models with a range of different topic numbers, it is possible to determine if there is a point at which the benefit, in terms of reduced perplexity, of increasing the number of topics appears to level off.

We split the environment statements into a training corpus (80% of statements) and a testing corpus (20% of statements), fitted models with 10, 20, 30, …, 100 topics to the training corpus, and calculated the associated perplexities using the testing corpus. These perplexity figures are shown in Figure 1 . We then fitted a piecewise linear regression to these points, which suggested that the ‘elbow’ of the graph appeared at 41.99 topics. We therefore selected 42 topics for our main analysis.

Perplexities associated with models with 10, 20, 30, …, 100 topics. The dotted lines show a piecewise linear regression line of best fit.

Topic modelling has an important advantage over more traditional qualitative analytical techniques in that it is extremely inclusive. Given that 1888 unit-level environment statements were returned to REF2021, containing ∼18 m words, it would have been impractical for a human analyst to read and analyse each statement. However, topic modelling is not purely quantitative: the LDA algorithm identifies topics which must then be interpreted. One common approach to this task is to conduct careful qualitative analyses of documents that contain a high proportion of words from each topic. We return to this issue later in the paper.

The 42 topics identified are shown in Table 1 . The table shows the characteristic words associated with each topic, the statement with the highest proportion of words from that topic, and the label we gave the topic. These labels were based on our interpretations of studying the characteristic words, the statements with particularly high proportions of words from the topic, and the statements with particularly low proportions of words from the topic.

Descriptive statistics, averaged across each of the four quarters of the experiment, for each of the four indices under consideration

Of the 42 topics, 28 were disciplinary specific. For example, Topic 42 was characterised by words including ‘clinical’, ‘cancer’, ‘medicine’, ‘disease’, ‘MRC’ and ‘NHS’, and the top 10 environment statements in terms of the proportion of words with this topic were all returned to the Clinical Medicine panel. We therefore labelled this topic ‘Clinical Medicine’. In some cases our model combined two or more disciplines. For instance, Topic 2 was characterised by the words ‘philosophy’, ‘religion’, ‘theology’, ‘religious’ and ‘ethics’. Of the top 20 statements with high proportions of words from this topic, 9 were returned to the Philosophy panel and 11 to the Theology and Religious Studies panel. We used the label ‘Philosophy and Religion’.

There were five geographical topics. For instance, Topic 8 was characterised by the words ‘Liverpool’, ‘Leeds’, ‘Manchester’, ‘York’, ‘Sheffield’ and ‘Yorkshire’, all cities/regions in the north of England, and the statements with the highest proportion of words from this topic were from northern universities. There were geographical topics associated with the North, the West Country, Scotland, London and Wales.

We also found a topic, Topic 25, that was used by institutions that organise their academic work through constituent colleges. The topic was characterised by words including ‘university’, ‘faculty’, ‘centre’, ‘college’ and ‘institute’, as well as geographical terms that referenced multi-college universities (‘London’, ‘Oxford’, and ‘Cambridge’). Of the 25 statements with the highest proportions from this topic, 23 were returned by constitute faculties or colleges of the University of Oxford, the University of Cambridge or the University of London. The exceptions were statements from Kingston University (an institution based in London) and Oxford Brookes University (an institution based in Oxford).

Of most interest for our purposes are the remaining eight topics. To identify appropriate labels we followed a similar process, separately for each topic. First, we studied the topic’s characterising words. Second, we read the five environment statements with the highest proportion of words from the topic, and the five environment statements with the lowest proportion of words from the topic. Finally, we conducted concordance analyses to identify how characterising words were used in statements with high and low proportions of words from the topic. This involved using a keyword in context (KWIC) tool from a traditional corpus linguistics package ( Anthony 2022 ). For instance, if ‘faculty’ was a characteristic word for a topic, we would find every occurrence of ‘faculty’ in the five statements with the highest proportion of words from the topic and read the surrounding context to identify how the word was typically being used. We would then do the same for the five statements with the lowest proportion of words from the topic. To illustrate our approach, we first discuss the reasoning behind our naming of Topic 16 in some detail, and then discuss each of the remaining seven topics in turn. Note that the online data associated with this article includes the topic weightings derived from our model for each topic and each environment statement submitted to REF2021, so interested readers can independently verify our analyses and assess for themselves the topic names’ appropriateness.

Although all REF environment statements are publicly available from the Research England website, we opted to redact individuals’ names in the quotes reported below as they are not relevant to the research questions we asked. We have, however, not anonymised at the institution or department level, as these may assist readers interpret the topics.

Topic 16—Immature Research Environment

The proportion of words from Topic 16 used by statements varied from 0.0001 (Imperial College’s Mathematical Sciences statement) to 0.362 (Bedfordshire’s Business and Management Studies statement). The topic was characterised by words such as ‘research’, ‘university’, ‘staff’, ‘international’, ‘REF’, ‘member’, and ‘members’ (see Table 1 ). Unlike with some of the other topics discussed below, we did not find these words very insightful for determining the semantic content of the topic.

Our next step was to carefully read the five statements with the highest proportion of words from the topic and the five statements with the lowest proportion of words from the topic. This revealed that the topic-defining words seemed to be being used in characteristic ways by those statements with a high proportion of words from the topic. Specifically, Topic 16 was characterised by descriptions of how the units were trying to encourage staff to engage in research. For example, the Wrexham Glyndŵr University Computer Science and Informatics statement (Topic 16 proportion 0.327) noted that “Data from October 2020 indicates that 38% of the 13 members of academic staff associated with UoA11 [the Computer Science and Informatics unit] have a doctoral qualification” and that “An encouraging sign is that 38% of UoA11 staff are studying towards a doctorate.” Similarly, the Liverpool John Moores University Business and Management Studies statement (T16 proportion 0.339) noted that 31 members of staff “are being supported in their research-related activities, with a view to them being research active in the next assessment period” and that staff were supported by holding seminars, where the invited external speakers were “editors of peer-reviewed journals with high impact factors” in order to assist staff “target publications in highly respected journals”. The Bedfordshire Business and Management Studies statement (T16 proportion 0.362) emphasised that “Staff members are strongly encouraged to attend international conferences and present their research results” and that their staff “are allocated dedicated research time as part of their workload”. In contrast, the statements with low frequencies of words from Topic 16 seemed to take for granted that their academic staff had doctorates and routinely conducted research.

Next, we conducted concordance analyses comparing how the topic’s defining words were used in the five statements with a high proportion of words from Topic 16 with how they were used in the five statements with a low proportion of words from the topic. For instance, we compared how these statements used the word “research”. In the five high statements there were 19 uses of “research active” (e.g. “continued to be research active”, “support to be research active”, “increased the number of research active staff”, “sought to retain research active staff”, “staff on the cusp of being research-active”) compared to just one in the five low statements, which appeared in a subheading in the University College London’s Law statement (“2.1 Research-active staff and output selection profile”).

As another example, across the five high Topic 16 statements there were 82 instances of “conference”, including numerous examples of conferences that had been attended by staff from these units. In contrast, this word appeared only 29 times in the five low Topic 16 statements and tended to be used as an illustration of a wider point. For example, in the Cambridge philosophy statement (T16 topic proportion 0.00003), the organisation of the Cambridge Platonism conference was given as an example of the unit’s interdisciplinary research (the conference was jointly organised with the Cambridge Faculty of Divinity).

To give one final example, there was also a difference in the way the high- and low-T16-proportion statements used the word “journal”. The five high-T16-proportion statements contained 37 instances of this word. In some cases, these were examples of how members of the unit had written research articles, e.g. Newman University’s Sport and Exercise Sciences, Leisure and Tourism statement (T16 proportion 0.360) noted that “Visiting Professor [anonymised] has produced a manuscript currently in review in the European Respiratory Journal Open”. In other cases these were lists of interactions with journals: Wrexham Glyndŵr University’s Computer Science and Informatics statement explained how one colleague was “on the review panel for a further 6 journals”. In contrast, the five low-T16-proportion statements had only 13 instances of the word ‘journal’. These tended to be examples of how the unit was contributing to the wider academic community. For instance, the University College London law environment statement (T16 proportion 0.0003) discussed how they were progressing towards an open research environment and exemplified this by noting how one member of the unit had “founded Europe and the World: A Law Review as a fully peer-reviewed OA [open access] journal”.

In sum, we concluded that those statements which had a high proportion of words from Topic 16 tended to spend a large proportion of their statement discussing how they were attempting to encourage or support routine research activities. These kinds of discussions were absent from those statements with a low proportion of words from this topic. We therefore named this topic “Immature Research Environment”.

Topic 4—Internal Structure of Research Units

Topic 4 was characterised by the high use of words such as ‘unit’, ‘unit’s’, ‘faculty’, ‘section’, ‘themes’, ‘theme’ and ‘institutional’. The proportion of words from this topic ranged from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.150 (the University of Cumbria’s Business and Management Studies statement). The topic tended to be characterised by detailed descriptions of the internal structure of the units. For instance, the University of Cumbria’s Business and Management Studies statement (T4 proportion 0.150) devoted 1.5 pages of their statement to the “unit context and structure” which noted how, during the assessment period, they had created a new institute and developed three new research themes. Similarly, the University of Winchester’s English Language and Literature statement (T4 proportion 0.137) spent just over a page discussing their unit context and structure, noting how the unit was situated within the University’s department and faculty structure, how it contained a research centre, and how the Centre interacted with other centres across the University. We named this topic “Internal Structure of Research Units”.

Topic 7—Career Development and EDI

Topic 7 was characterised by words such as ‘staff’, ‘support’, ‘training’, ‘including’, ‘access’, ‘career’ and ‘diversity’. The proportion of words from this topic ranged from 0.004 (The Royal Agricultural University’s Agriculture, Food and Veterinary Sciences statement) to 0.399 (Leeds Arts University’s Music, Drama, Dance, Performing Arts, Film and Screen Studies statement). Statements with a high proportion of words from Topic 7 tended to have long sections that discussed how the unit supported staff and student development, and about their equality and diversity processes. For instance, the University of Nottingham’s Politics and International Studies statement (T7 proportion 0.290) included careful statistical analyses of their gender balance at different career stages, as well as analyses of their staff by ethnicity, disability and age profiles. The statement then went on to discuss how these analyses informed “EDI-focused improvements”. For instance, in response to “too few members from underrepresented groups in leadership roles” the statement noted how the unit had reconstituted its EDI committee and “increased leadership by women in major committees”. In contrast, the Royal Agricultural College’s Agriculture, Food and Veterinary Sciences statement (T7 proportion 0.004) devoted just 50 words to EDI issues, and only used the word ‘diversity’ in the context of their research on “the global distribution of earthworm diversity”. We named this topic “Career Development and EDI”.

Topic 18—Staff Ways of Working

Topic 18 was characterised by words such as ‘work’, ‘school’, ‘colleagues’, ‘teaching’, ‘group’, ‘members’, ‘part’ and ‘years’. Some statements contained a very low proportion of words from this topic, e.g. Heriot-Watt University’s Architecture, Built Environment and Planning statement (T18 proportion 0.000), whereas others contained a substantial proportion from it, e.g. the University of East Anglia’s Law statement (T18 proportion 0.262). The statements with a high proportion of words from the topic were characterised by many concrete descriptions of staff working practices. For example, the University of Newcastle upon Tyne’s Classics statement (T18 proportion 0.244) described how “Members who have held a substantial administrative role are entitled to an extra semester of research leave”. Similarly, the University of St Andrews’s Economics and Econometrics statement (T18 proportion 0.255) discussed the process by which academic staff can apply for sabbatical leave: “The HoS considers applications in relation to the general workload allocation process and, if there are doubts about the feasibility of accommodating all applications, the HoS consults a panel of senior colleagues.”

In our concordance analysis we noted that ‘work’ was commonly used as a verb in high-T18-proportion statements to describe concrete examples of how the unit operated (“the University granted [anonymised] two years of unpaid leave to work and develop Impact at the European Central Bank”), whereas in low-T18-proportion statements it was often used as a noun (“our work on public health engineering”). We also found that ‘members’ was more often used in high-T18-proportion statements to describe concrete internal activities (“Individual staff members can request travel funding and leave to attend masterclasses and short courses”), whereas in low-T18-proportion statements it was typically used to describe esteem activities (“Our researchers also chaired or have served as members of important grant panels”). We named this topic “Staff Ways of Working”.

Topic 28—REF-Focused Research Strategy

Topic 28 was characterised by words such as ‘UoA’, ‘REF’, ‘UoA’s’, ‘UoAs’, ‘section’, ‘cycle’ and ‘submitted’. The proportion of words from this topic varied from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.126 (the University of Winchester’s History statement). Statements with a high proportion of words from Topic 28 tended to use REF terminology to describe their research environment. For example, they might characterise their internal structure in terms of UoAs or ‘units’ rather than departments, centres or institutes. For instance, the University of Worcester’s English Language and Literature statement (T28 proportion 0.126) described “The Unit’s strategic research objectives”, “the unit’s impact strategy” and “the unit team”; and the University of Winchester’s History statement (T28 proportion 0.126) described how “the UoA had a devolved budget”, the existence of a “UoA working group” and “the strategic aims of the UoA over the cycle”. In contrast, the joint engineering statement from the University of Edinburgh and Heriot-Watt University (T28 proportion 0.000) contained no instances of ‘UoA’, and only used “unit” in the generic text used in the page header (“unit-level environment template (REF5b)”). Instead, they described how their research was organised into “cross-cutting organisational themes” and “interdisciplinary global research challenge areas”. Similarly, the University College London education statement (T28 proportion 0.000) discussed how their research was organised into departments and research centres, and only used the word ‘UoA’ in a table reporting the submission’s demographic data. We named this topic “REF-Focused Research Strategy”.

Topic 30—Exemplification of Strategy and Processes

Words that characterised Topic 30 included “e.g”, “including”, “funding”, “supported”, “grant” “PGRs”, “impact” and “awards”. Like Topic 16, it was not immediately obvious to us from studying these words what the topic referred to. However, when we compared the high-T30-proportion statements and the low-T30-proportion statements, we concluded that the topic was capturing an increased use of concrete examples to illustrate strategies and processes. To illustrate, the five statements with the highest proportion of words from Topic 30 made liberal use of “e.g.” to give explicit examples of the research strategies being described. For example, the University of Nottingham’s Geography and Environmental Studies statement (T30 proportion 0.278) described how they enable and facilitate impact: “the new Institutional Institute of Policy and Engagement has helped fund pump-priming engagement work (e.g. [anonymised]); fund high-level policy relevant talks (e.g. [anonymised] at Asia House and Chatham House) and aid development of policy briefs (e.g. [anonymised] on water management in the Red River, French on indebtedness and financial exclusion).” In their section on open research, they wrote that “The School developed and hosts online, openly accessible maps, including the Blue-Green Cities multiple benefits toolbox ([anonymised]) and the ‘black presences and the legacies of slavery and colonialism’ online map ([anonymised])”. Similarly, the University of Leicester’s Communication, Cultural and Media Studies, Library and Information Management statement (T30 proportion 0.266) noted that their “strategy of enabling researcher development in Media focuses on supporting ECRs and mid-career academics, to achieve external funding success. For example, 23 of the 40 awards secured in the REF period were to Assistant Professors.”

While low-T30-proportion statements also used exemplification, these tended to be less related to the research strategies and processes described in the statements. For instance, there were 22 instances of “e.g.” in Imperial College London’s Clinical Medicine statement (T30 proportion 0.000), of which 12 were used in front of lists of journals (“Our reach is demonstrated by publishing in specialist journals (e.g. Lancet Infect Dis [11], Nature Immunology [7]”) and a further 4 were used in front of scientific concepts (“…to reveal mechanisms of cardiovascular disease. e.g. identification of titin variants in health and disease”).

We named Topic 30 “Exemplification of Strategy and Processes”.

Topic 34—Industry Partners and Funding

Topic 34 was characterised by words such as ‘award’, ‘awards’, ‘industry’, ‘society’, ‘data’, and ‘international’. The proportion of words from this topic in statements varied from 0.000 (the University of East Anglia’s Area Studies statement) to 0.259 (the University of Bristol’s chemistry statement). Those statements which had a high proportion of words from Topic 34 devoted considerable space to discussing their industrial partnerships and research funding. For example, the Imperial College London Chemistry statement (T34 proportion 0.252) noted that “Collaborations with industry include GSK and Pfizer”, and that “members are involved in industry collaborations e.g. a £3.2M EPSRC BP Prosperity Partnership”. The University of Surrey Physics statement (T34 proportion 0.249) argued that their “world-class research is evidenced by grant awards over the REF period that total more than £19.6 million” and noted that they “work closely with industrial partners”. Given this focus, it was unsurprising that the correlation between the proportion of an environment statement made up of words from Topic 34 was strongly correlated with a submission’s research funding per FTE, r = 0.642, P < 0.001. Notably, however, this correlation was much reduced if research income was standardised within each UoA (to r = 0.275, P < 0.001). In other words, Topic 34 related to overall unstandardised research funding, meaning that statements with a particularly high proportion of Topic 34 words tended to come from highly funded scientific disciplines. Indeed, the mean proportion of words from Topic 34 for statements to Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) were 0.109, 0.139, 0.044 and 0.020 respectively. In other words, statements from scientific disciplines tended to use more words from Topic 34 than statements from non-scientific disciplines, an observation consistent with our conclusion that the topic concerned industrial partnerships and funding. We named this topic “Industry Partners and Funding”.

Topic 40—Early Career Researcher (ECR) Development

The last of our eight general topics was Topic 40. This topic was characterised by words such as ‘development’, ‘develop’, ‘support’, ‘research’, ‘researchers’, ‘strategy’, ‘strategic’, ‘work’, ‘working’ and ‘funding’. Of the eight general topics, Topic 40 was the most common: on average environment statements devoted 18.6% of their content to it, although the proportions of words from Topic 40 ranged from 0.038 (Kingston University’s Philosophy statement) to 0.403 (Canterbury Christ Church University’s Sport and Exercise Sciences, Leisure and Tourism statement). Those statements with a high proportion of words from Topic 40 spoke at length about researcher development, with a particular focus on early career researchers. For instance, the Queen Margaret University Edinburgh Sociology statement (T40 proportion 0.360) discussed how they “support researchers in exploring and preparing for a diversity of careers, for example, through the use of mentors and careers professionals, training, and secondment” and the Solent University Southampton Sport and Exercise Sciences, Leisure and Tourism statement mentioned that a “research mentoring programme organised through [the School Advisory Group for Research] has been implemented to support researchers”. The five statements with the highest proportion of words from Topic 40 made 23 references to the Concordat to Support the Career Development of Researchers compared to 4 references in the five statements with the lowest proportion of words from this topic. We named the topic “Early Career Research (ECR) Development”.

For our main analysis, we asked whether the eight general topics that environment statements focused on were related to the quality profiles they received. Recall that panels assessed each submission’s environment using a five-point scale from ‘unclassified’ (“an environment that is not conducive to producing research of nationally recognised quality or enabling impact of reach and significance”) through to ‘4*’ (“an environment that is conducive to producing research of world-leading quality and enabling outstanding impact, in terms of its vitality and sustainability”). Each submission was awarded a ‘quality profile’ based on its environment statement and associated data (discussed below). For instance, the Open University’s submission to the Classics UoA was rated as having an environment where 25% of activity was 4* (world-leading), 50% was 3* (‘internationally excellent’), 25% was 2* (‘recognised internationally’) and 0% was 1* (‘recognised nationally’) or unclassified. For each submission we calculated a grade point average (GPA), which was a simple linear combination of the percentage of each quality level. So the Open University’s Classics submission obtained an environment GPA of 3.0 (0.25 × 4 + 0.5 × 3 + 0.25 × 2 + 0 × 1 + 0 × 0).

Alongside environment statements, the assessment panels were also provided with additional metrics associated with each submission. These included the full-time equivalent number of staff (FTE) being returned in the submission, the grant income that the unit had received during the assessment period (which could be broken down by source and date), and the number of doctoral degrees that the unit had awarded during the assessment period.

We ran a hierarchical regression predicting each unit’s environment GPA. In the first block we entered each unit’s FTE, their research income per FTE, and the number of doctoral degrees awarded per FTE. Each of these metrics was standardised (using z scores) within each UoA to take account of disciplinary norms (for instance, the mean grant income per FTE in the Clinical Medicine unit was £3.7 m compared to £74k in the English Language and Literature unit). In the second block we entered the proportion of each environment statement from the eight general topics discussed above.

The results of this regression are shown in Table 2 . Together the environment metrics could explain 47.3% of the variance in environment GPAs. When the topic weightings were added, an additional 21.9% of the variance could be explained, bringing the overall R 2 to 69.1%. Thus the weightings of these eight topics explained significant extra variance in environment GPAs, F (8, 1870) = 166, P < 0.001. When the eight topic weightings were used as predictors in the first block (ie before the metrics were entered) they explained 58.9% of the variance in environment GPAs, F (8, 1873) = 336, P < 0.001. In sum, the weightings associated with the eight general topics in our topic model predicted a surprisingly large proportion of the variance in submissions’ environment GPAs, indicating that the topics that environment statements focused upon made a substantial contribution to the perceived quality of each submission’s research environment.

A hierarchical regression analysis predicting environment GPA with various metrics (entered in Block 1) and topic weightings from the eight general topics (entered in Block 2)

P < 0.05.

P < 0.01.

P < 0.001.

As shown in Table 2 , of the eight topics, four were significant negative predictors of environment GPA, two were significant positive predictors and two were not significant predictors. Statements that had higher weightings from the Immature Research Environment, Staff Ways of Working, REF-Focused Research Strategy, and ECR Development topics were associated with lower environment GPAs. Statements that had higher weightings from the Exemplification of Strategy and Processes, and Industry Partners and Funding topics were associated with higher environment GPAs.

Because this regression analysis only assessed whether topic weightings were linearly associated with environment GPAs, we also investigated whether there were nonlinear relationships by inspecting scatterplots of topic weightings against environment GPAs separately for each topic. These are shown in Figure 2 , together with cubics of best fit. There appeared to be a clearly nonlinear relationship between topic weighting and environment GPA for Topic 7 Career Development & EDI. Placing little emphasis on this topic was associated with receiving a low environment GPA, but so was placing too much emphasis on it. The cubic of best fit obtained its maximum when 13.4% of the statement was made up of words from Topic 7 (recall that this figure is a percentage of all words in the statement, after stop words have been removed). Environment statements varied substantially in the extent that they discussed Career Development and EDI—the statement with the lowest emphasis on this issue had just 0.4% of its words from the topic, the statement with the highest had 39.9%. But the highest environment GPAs, on average, were obtained by statements where 13–14% of the statement focused on Career Development and EDI.

Scatterplots showing topic weightings (proportion of each statement made up of words from the given topic) against environment GPA, separately for the eight general topics. Bold lines are cubics of best fit.

Topic 40 ECR Development also showed a possibly nonlinear relationship between topic weighting and GPA, although this was less clearly the case than for Topic 7. For statements where between 0% and 20% of their words came from Topic 40 there was a reasonably flat relationship with GPA. But those statements with higher proportions from this topic showed a negative relationship between topic weighting and GPA.

Next, we explored whether environment statements that discussed more discipline-specific issues scored more highly than those which did not. In other words, we asked whether environment statements returned to, say, the Clinical Medicine UoA scored more highly when if they used more words from the Clinical Medicine topic. To investigate this we calculated the correlation between environment GPA and topic weighting for the disciplinary topic, separately for each UoA. These results are shown in Table 3 . While a large majority of these correlations were positive, indicating that environment statements that contained more discipline-specific language tended to score higher, there were systematic differences between broad subject areas. The mean correlations between the percentage of discipline-specific language and GPA for Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) respectively were 0.472, 0.217, 0.172 and 0.093 respectively. For all main panels other than D, these means were significantly greater than zero.

Correlations between environment GPA and disciplinary topic weightings, per UoA (e.g. within the Clinical Medicine UoA, the correlation between environment GPA and weightings on Topic 42 was r = 0.560)

To compare the strength of the association between the extent to which submissions discussed disciplinary issues and their GPAs with the strength of the associations between the eight general topics discussed above and GPAs, we ran a regression analysis on submissions to the Business and Management Studies panel. We chose Business and Management as it was the panel which received the largest number of submissions (108), and so offered the greatest statistical power for an analysis of this kind. In this regression we used the eight general topics, plus the Business and Management topic (Topic 20) to predict environment GPAs. This model is shown in Table 4 . Crucially, the Business and Management topic weighting variable was a significant predictor in this model and had a standardised regression coefficient of β = 0.120, larger than that associated with Exemplification of Strategy and Process (Topic 30, β = 0.102), and roughly a third the size of Industry Partners and Funding (Topic 40, β = 0.336). In other words, using language associated with business and management was a stronger predictor of environment GPAs than giving examples of the unit’s strategy, and around a third as strong a predictor as discussing external funding and industrial partnerships.

A regression analysis predicting environment GPA of submissions to the Business and Management Studies panel, with the topic weightings from the eight general topics and the Business and Management topic

In sum, the more an environment statement included content from the relevant discipline, the higher the environment GPA it received, although this effect was more pronounced for medicine, health and life sciences, and less pronounced for the arts and humanities. To illustrate this, we analysed statements with the most and the least discipline-specific language from the ‘Biological Sciences’ and ‘Economics and Econometrics’ panels (the two panels where the relationship between discipline-specific language use and environment GPA was strongest). The differing content and emphasis were clear. For example, University College London’s Economics and Econometrics statement began with a sentence that listed its research strengths in “microeconomics, macroeconomics and econometrics” and went on to link their research focus to “the most pressing national and international socio-economic challenges of our time, such as inequality, migration, globalization, and sustainable growth” as well as international economic policy. By contrast, the opening remarks in the University of Northampton’s Economics statement focused on being a first-time submission, and noted that this new development “has been led in part by structural changes at the University level, but more significantly by the appointment of a new Dean.” Their first paragraph continued to list internal structure rather than discipline-relevant topics of strength and expertise.

Similarly, in the opening paragraph of Birkbeck’s submission to the Biological Sciences UoA, discipline-specific language was used to articulate the “fundamental biological questions” conducted by staff “who are using microbial, plant and animal systems to advance our understanding of the fundamental principles underlying molecular and cellular function, physiology and behaviour” In contrast, the opening paragraphs of the University of Worcester’s Biological Sciences statement described their first-time submission to the panel and articulated their internal structures and strategies (e.g. “the University went through an academic restructure introducing Colleges and Schools”; “The University Research Strategy 2014–19 outlined the key role played by Research Groups in operationalising plans and ambitions for excellent research”). Thus, right from the start of these statements, a focus on disciplinary contribution was much clearer in the higher-scoring submissions than those with a lower GPA.

Summary of main findings

We asked whether the perceived quality of a research environment, as measured in the UK’s Research Excellence Framework, could be predicted by the text used by that unit to describe their environment. By topic modelling the full text of all 1888 unit-level environment statements submitted to REF2021, we settled on a model that included eight specific topics that were distinct from disciplinary or geographical topics. These were related to the Internal Structure of Research Units, Career Development and EDI, Immature Research Environments, Staff Ways of Working, REF-Focused Research Strategies, Exemplifications of Staff Ways of Working, Industry Partners and Funding, and ECR Development. The proportion of words each statement included from these eight topics was surprisingly predictive of the environment score that the unit received in REF2021. Specifically, these topic proportions collectively explained 58.9% of the variance in environment GPAs, and 21.9% of the variance over and above the variance explained by the unit’s (standardised) FTE staff number, the (standardised) number of doctoral degrees it awarded, and its (standardised) grant income. In total, these metrics and the topic proportions from these eight topics collectively explained 69.1% of the variance in environment GPAs. Alongside these main findings, we also identified that environment statements that contained a lot of disciplinary-specific language tended to score higher than those which did not, although this effect was stronger for medical and biological disciplines, and weaker for the arts and humanities.

All the analyses we have reported in this paper are correlational in nature. Clearly, we were not able to experimentally manipulate the environment statements submitted to the REF and then assess the effect that these manipulations had on GPAs. Given this, care must be taken before assuming that the relationships we have reported are causal . Of particular concern is that some of our findings might be attributable to confounding factors. Indeed, in at least some cases this seems quite plausible. For instance, perhaps the reason why a research strategy focused on the REF seems to be negatively correlated with GPAs is that departments which have (relatively) low levels of research activity tend to both have a less mature research environment and also choose to write their environment statements using a higher proportion of REF terminology, as they have fewer pre-existing structures with pre-existing terminology to draw upon. We discuss this issue further below. Given this possibility of confounding factors, caution is required when interpreting our findings. Clearly, we cannot confidently draw causal conclusions in the absence of an experimental study (which would inevitably be of questionable external validity). Nevertheless, we can speculate.

REF environment scores are awarded through a process of human judgement. These are, by necessity given the volume of reading required of REF panellists, produced relatively rapidly. Many theories of human judgement emphasise how judgements are formed by comparing to-be-judged objects against prototypical instances sampled from memory (e.g. Fiedler 2000 , 2008 ; Stewart, Chater and Brown, 2006 ; Unkelbach, Fiedler and Freytag, 2007 ). Such theories would likely conceptualise reaching judgements about REF environment quality as a process which involves storing multiple exemplars of high- and low-quality statements in memory and, when encountering a new statement, generating a quality estimate by matching the features of the to-be-judged statements against those exemplars or prototypes ( Glöckner and Witteman 2010 ). This process need not be conscious, meaning that panellists are unlikely to be fully aware of the features they use to decide upon environment scores. Given this, it perhaps reasonable to suspect that if low-scoring environment statements typically have a given feature, then when panellists encounter a new statement with that feature, there may be a bias towards it receiving a lower score. In other words, if the majority of high-scoring environment statements that a panellist sees avoid using REF-heavy terminology, then in light of the decision-making literature on reasoning from prototypes and exemplars, it seems plausible that their judgements of future environment statements will be influenced by the presence or absence of this terminology, perhaps only unconsciously. If this account is correct, then the correlations between topic weightings and environment GPAs that we have reported above may well be, in part at least, causal.

How to write a ‘good’ environment statement

Assuming there remains a significant narrative element to REF people, culture and environment statements in 2029, what lessons might we learn from this analysis to support the crafting of written submissions that include all those features that are associated with high-scoring statements, and none of those features associated with low-scoring statements? We make eight recommendations.

First, we would avoid stating things that high-quality research environments would consider trivial. For example, we would not mention that most staff in our unit have doctorates, or that our staff attend academic conferences and write articles in academic journals. We would avoid using the phrase “research-active”, especially in an aspirational way, and we would not mention that our staff review articles for academic journals unless they had substantial editorial roles. In short, routine research activities should not be discussed in REF environment statements. Doing this is likely to give readers the impression that research is not a central feature of the unit’s work and reduce perceptions of the vitality of the unit’s research environment.

Second, when discussing research strategy, we would not give the impression that our research strategy is solely driven by the REF. Instead, our strategy would be organised around research centres, research groups, and departments. It would be focused on an academic discipline, not a “UoA”. The staff who led our submission would not be characterised as a “UoA Working Group”, and if we appointed other academic staff to REF leadership roles we would not mention it in our submission. Although we have robustly demonstrated that there is a relationship between conceptualising strategy using REF-centred terminology and receiving lower environment scores, it is less clear why this might be. One possibility is that the most research-intensive universities are sufficiently self-confident, and have a sufficiently long history of conducting research, to define their research activity in their own terms. In contrast, less research-intensive universities may need to create research infrastructure and strategies primarily in order to produce a respectable REF statement. If this were the case, we might expect the whole research enterprise in less research-intensive institutions to be more likely to be conceptualised in REF terms.

Third, we would not go into too much detail about the specific ways in which staff-related processes operate. For instance, we would not explain how decisions about sabbatical leave are informed by input from both the research committee and the teaching allocation committee. Similarly, details about which staff are involved at which stages in approving requests for conference travel funding would be omitted. Why might including detail of this sort be associated with lower GPAs? One plausible explanation is simply that including such details is a waste of space. As noted in the Introduction, REF environment statements are word limited, so including superfluous details might simply prevent the inclusion of content that would be causally associated with higher GPAs. This might be sufficient to generate a small negative relationship with GPAs (even though none would exist if there were no length restriction on submissions).

Fourth, we would not focus too much attention on how we support the career development of our ECRs. This finding is particularly surprising in light of the REF submission guidance’s statement that submissions should include “evidence of how individuals at the beginning of their research careers are being supported and integrated into the research culture of the submitting unit” (REF Panel Criteria and Working Methods 2019: 63). What might explain this apparent contradiction? An inspection of Figure 2 reveals that the negative relationship between discussing ECR development and GPA was driven by statements which included a relatively high proportion of this topic. Perhaps too much discussion of ECR development had the effect of crowding out space which could have been used for content that was more strongly associated with positive GPAs. Another possibility is that an excessive focus on ECR development might indicate to panellists that a unit feels that they have an unusually low proportion of senior established researchers in post.

Fifth, we would take care to make sure that we discussed career development and EDI, but not too much. The REF guidance emphasised that EDI should be discussed throughout submissions but, as shown in Figure 2 , some submissions clearly failed to follow this guidance, and these tended to receive low GPAs. However, some submissions seemed to discuss career development and EDI too much. Our analysis suggested that devoting ∼13% of the statement to this topic was optimal, with scores falling off for submissions with substantially higher or lower figures than this. Explaining why submissions which did not spend much time discussing career development and EDI tended to score poorly is straightforward: they failed to follow the clear instructions provided, and panellists may have concluded that they were poor places for minoritized colleagues to work. But why might have submissions which discussed these topics at length received lower scores? Again, one plausible account involves appealing to the length limitations of the environment statement template. Perhaps discussing career development and EDI was a qualifying criterion: not taking this issue sufficiently seriously would harm a submission, but once a submission successfully demonstrated that career development and EDI was a matter of concern, then further discussions on the topic became unnecessary. Instead, extra details on these matters had the effect of crowding out space that could have been productively used to discuss other issues associated with higher GPAs. A second possibility is that some statements mentioned EDI so often that it conveyed a ‘tick box’ approach rather than an authentic embedding. One final possibility is that an unusually high level of discussion of EDI issues might give the impression to reviewers that the unit felt that they had an unusually high number of issues in this area which required particular attention. Again, this might give the impression of a poor environment for minoritized colleagues.

Sixth, we would illustrate our research strategy by giving as many concrete examples as possible of how it has been implemented in practice. For example, if we provided pump-priming research funding to our staff, we would give an example of someone who had received funding, what they did with it, and what this led to. If we had particular strategies in place to facilitate interdisciplinary work, we would give an example of how this had led to a successful interdisciplinary workshop or funding application. If we had a policy on sharing research data, we might state the proportion of empirical papers in our submission where data had been shared online and give an example of how these datasets had been used by external colleagues in their own work. If we had a particularly generous study leave allowance for colleagues returning from parental leave, we might give an example of outputs or successful grant applications that had been produced as a result of this policy, and so on.

Seventh, we would mention our research funding and industrial partnerships as much as possible. This might seem superfluous, as panellists were provided with each unit’s grant expenditure alongside the written environment statement. However, our data suggest that mentioning funding and partnerships explained significant variance in GPAs over and above standardised grant income per FTE. 1 In sum, having high levels of grant income is insufficient: one must use it to provide evidence of a successful research strategy and environment as well as what the income enabled.

Eighth and finally, we would discuss our discipline as much as possible, particularly if we were writing a statement as part of a submission to a STEM UoA. For example, we would illustrate the success of our research strategy by discussing some of the important research findings it facilitated, we would name our research groups using well-understood disciplinary terms, and we would describe the work that our research funding allowed us to do, rather than merely state grant funding amounts. This finding that the use of discipline-specific language tends to be associated with higher environment GPAs is particularly interesting given the original suggestion of the Institution-Level Environment Panel Pilot that future REF exercises should abandon unit-level environment statements altogether ( REF 2022 ). Our finding that the scores produced by discipline-specific panellists were correlated with the amount of discipline-specific language submissions used, suggests that they may have used their domain expertise to come to decisions about submission quality. Clearly this would not be possible to the same extent, if at all, if environment were assessed at the institutional level by a panel made up of experts from a variety of disciplines—even if, as the REF (2022) pilot panel proposed, brief unit-level narratives were incorporated into the institutional-level statement. In sum, our results indicate that research environment assessed at the institutional level would likely be a different construct from research environment assessed at the unit level.

It has now been communicated that the next Research Excellence Framework will continue to assess an institution’s people, culture and environment at both institution- and discipline level, with the greater weight being given to the discipline-level assessment ( Joint UK HE Funding Bodies 2023 ). It is not yet clear the extent to which this element will look beyond the domains assessed in REF2021 (context, people, income & infrastructure, and collaboration & contribution), nor the extent to which the assessment will consist of quantitative indicators relative to narrative description. Regardless, our analysis may help institutions reflect upon what it means to have a high-quality research environment.

We are grateful to Hugues Lortie-Forgues, Victoria Simms, Steve Rothberg, and two anonymous reviewers for insightful comments on earlier drafts of this manuscript.

This work was partially supported by Research England, via an Expanding Excellence in England grant to the Centre for Mathematical Cognition, and the Economic and Social Research Council [grant number ES/W002914/1].

Conflict of interest statement. None declared.

Data associated with this manuscript are available at https://doi.org/10.17028/rd.lboro.23912499.v1 .

A regression predicting environment GPAs using only two independent variables, standardised grant income per head and the weightings associated with Topic 34 Industry Partners and Funding, could explain 36% of the variance in environment GPA. In this regression the standardised coefficients associated with the grant income metric and Topic 34 were β = 0.476 and β = 0.255 respectively, indicating that the amount of grant income a unit received was only slightly less than twice as important as how much it discussed it in its environment return.

Anthony L. ( 2022 ) AntConc (Version 4.2.0) [Computer Software] . Tokyo, Japan : Waseda University .

Google Scholar

Google Preview

Bence V. , Oppenheim C. ( 2005 ) ‘ The Evolution of the UK’s Research Assessment Exercise: publications, Performance and Perceptions ’, Journal of Educational Administration and History , 37 : 137 – 55 .

Blei D. M. , Ng A. Y. , Jordan M. I. ( 2003 ) ‘ Latent Dirichlet Allocation ’, The Journal of Machine Learning Research , 3 : 993 – 1022 .

Brassington L. ( 2022 ) Research Evaluation: Past, Present and Future . HEPI Report 152. Higher Education Policy Institute.

Brown R. , Carasso H. ( 2013 ) Everything for Sale? The Marketisation of UK Higher Education . Routledge .

Curry S. , Gadd E. , Wilsdon J. ( 2022 ) Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment . < https://rori.figshare.com/articles/report/Harnessing_the_Metric_Tide/21701624/2/files/38515103.pdf > accessed 5 Feb 2024.

Derrick G. E. , Samuel G. N. ( 2016 ) ‘ The Evaluation Scale: Exploring Decisions about Societal Impact in Peer Review Panels ’, Minerva , 54 : 75 – 97 .

Fairclough N. ( 1995 ) Critical Discourse Analysis . London : Longman .

Fiedler K. ( 2000 ) ‘ Beware of Samples! A Cognitive-Ecological Sampling Approach to Judgement Biases ’, Psychological Review , 107 : 659 – 76 .

Fiedler K. ( 2008 ) ‘ The Ultimate Sampling Dilemma in Experience-Based Decision Making ’, Journal of Experimental Psychology: Learning, Memory, and Cognition , 34 : 186 – 203 .

French N. J. , Massy W. F. , Young K. ( 2001 ) ‘ Research Assessment in Hong Kong ’, Higher Education , 42 : 35 – 46 .

Geuna A. , Martin B. R. ( 2003 ) ‘ University Research Evaluation and Funding: An International Comparison ’, Minerva , 41 : 277 – 304 .

Gillies D. ( 2008 ) How Should Research Be Organised? London, UK : College Publications .

Glöckner A. , Witteman C. ( 2010 ) ‘ Beyond Dual-Process Models: A Categorisation of Processes Underlying Intuitive Judgement and Decision Making ’, Thinking & Reasoning , 16 : 1 – 25 .

Grimmer J. , Stewart B. M. ( 2013 ) ‘ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts ’, Political Analysis , 21 : 267 – 97 .

Jacobi C. , van Atteveldt W. , Welbers K. ( 2016 ) ‘ Quantitative analysis of large amounts of journalistic texts using topic modelling. ’, Digital Journalism , 4 : 89 – 106 .

Jensen E. A. , Wong P. , Reed M. S. ( 2022 ) ‘ How Research Data Deliver Non-Academic Impacts: A Secondary Analysis of UK Research Excellence Framework Impact Case Studies ’, Plos One , 17 : e0264914 .

Joint UK HE Funding Bodies . ( 2023 ) Research Excellence Framework 2028 Initial Decisions and issues for further consultation (REF/2028/23/01). < https://repository.jisc.ac.uk/9148/1/research-excellence-framework-2028-initial-decisions-report.pdf > accessed 5 Feb 2024.

Jones P. , Sizer J. ( 1990 ) ‘ The Universities Funding Council’s 1989 Research Selectivity Exercise ’, Beiträge Zur Hochschulforschung , 4 : 309 – 48 .

Kellard N. M. , Śliwa M. ( 2016 ) ‘ Business and Management Impact Assessment in Research Excellence Framework 2014: analysis and Reflection ’, British Journal of Management , 27 : 693 – 711 .

King’s College London and Digital Science ( 2015 ) The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies . Bristol, United Kingdom : HEFCE .

Manville C. , d’Angelo C. , Culora A. , Gloinson E.R. , Stevenson C. , Weinstein N. , Wilsdon J. , Haddock G. , Guthrie S. ( 2021 ) Understanding Perceptions of the Research Excellence Framework among UK Researchers . Cambridge, UK : RAND Corporation .

Manville C. , Guthrie S. , Henham M.-L. , Garrod B. , Sousa S. , Kirtley A. , Castle-Clarke S. , Ling T. ( 2015 ) Assessing Impact Submissions for REF 2014: An Evaluation . Cambridge, UK : RAND Corporation .

Marques M. , Powell J. J. , Zapp M. , Biesta G. ( 2017 ) ‘ How Does Research Evaluation Impact Educational Research? Exploring Intended and Unintended Consequences of Research Assessment in the United Kingdom, 1986–2014 ’, European Educational Research Journal , 16 : 820 – 42 .

Matthews A. , Kotzee B. ( 2022 ) ‘ Bundled or Unbundled? A Multi‐Text Corpus‐Assisted Discourse Analysis of the Relationship between Teaching and Research in UK Universities ’, British Educational Research Journal , 48 : 578 – 97 .

McCallum A. K. ( 2002 ) MALLET: MAchine Learning for LanguagE Toolkit . < http://mallet.cs.umass.edu > accessed 5 Feb 2024.

Mellors-Bourne R. , Metcalfe J. , Gill A. ( 2017 ) Exploring Equality and Diversity using REF2014 Environment Statements , CRAC LTD Report to HEFCE.

Pardo-Guerra J. P. ( 2022 ) The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences . Columbia University Press .

Pinar M. , Horne T. J. ( 2022 ) ‘ Assessing Research Excellence: evaluating the Research Excellence Framework ’, Research Evaluation , 31 : 173 – 87 .

Poppler ( 2022 ) [Computer Software] < https://poppler.freedesktop.org > accessed 5 Feb 2024.

REF ( 2019 ) REF Panel Criteria and Working Methods . < https://archive.ref.ac.uk/media/1450/ref-2019_02-panel-criteria-and-working-methods.pdf > accessed 5 Feb 2024.

REF ( 2022 ) REF Institutional-level Environment Pilot: Report of the Pilot Panel . < https://archive.ref.ac.uk/media/1908/ref-2021-ilepp-final-report.pdf > accessed 5 Feb 2024.

Reichard B. , Reed M. S. , Chubb J. , Hall G. , Jowett L. , Peart A. , Whittle A. ( 2020 ) ‘ Writing Impact Case Studies: A Comparative Study of High-Scoring and Low-Scoring Case Studies from REF2014 ’, Palgrave Communications , 6 : 1 – 17 .

Sivertsen G. ( 2017 ) ‘ Unique, but Still Best Practice? The Research Excellence Framework (REF) from an International Perspective ’, Palgrave Communications , 3 : 17078 .

Stewart N. , Chater N. , Brown G. ( 2006 ) ‘ Decision by Sampling ’, Cognitive Psychology , 53 : 1 – 26 .

Sutton E. ( 2020 ) ‘ The Increasing Significance of Impact within the Research Excellence Framework (REF) ’, Radiography , 26 : S17 – S19 .

Swedish Government . ( 2016 ) Regeringens Proposition 2016/17:50 . Kunskap i samverkan—för samhällets utmaningar och stärkt konkurrenskraft .

Terämä E. , Smallman M. , Lock S. J. , Johnson C. , Austwick M. Z. ( 2016 ) ‘ Beyond Academia–Interrogating Research Impact in the Research Excellence Framework ’, Plos One , 11 : e0168533 .

Thomas D. A. , Nedeva M. , Tirado M. M. , Jacob M. ( 2020 ) ‘ Changing Research on Research Evaluation: A Critical Literature Review to Revisit the Agenda ’, Research Evaluation , 29 : 275 – 88 .

Thorpe A. , Craig R. , Tourish D. , Hadikin G. , Batistic S. ( 2018a ) ‘ Environment’ Submissions in the Uk's Research Excellence Framework 2014 ’, British Journal of Management , 29 : 571 – 87 .

Thorpe A. , Craig R. , Hadikin G. , Batistic S. ( 2018b ) ‘ Semantic Tone of Research ‘Environment’submissions in the UK’s Research Evaluation Framework 2014 ’, Research Evaluation , 27 : 53 – 62 .

Unkelbach C. , Fiedler K. , Freytag P. ( 2007 ) ‘ Information Repetition in Evaluative Judgements: Easy to Monitor, Hard to Control ’, Organizational Behavior and Human Decision Processes , 103 : 37 – 52 .

Watermeyer R. , Derrick G. E. ( 2022 ) ‘ Affective Auditing: The Emotional Weight of the Research Excellence Framework on Middle Management ’, Research Evaluation , 31 : [rvac041]. 498 – 506 . https://doi.org/10.1093/reseval/rvac041

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

How Research England supports research excellence

Research excellence framework.

The Research Excellence Framework (REF) is the UK’s system for assessing the excellence of research in UK higher education providers (HEPs).

The REF outcomes are used to inform the allocation of around £2 billion per year of public funding for universities’ research.

The REF was first carried out in 2014, replacing the previous Research Assessment Exercise. Research England manages the REF on behalf of all the four UK higher education funding bodies:

• Research England
• Scottish Funding Council
• Higher Education Funding Council for Wales
• Department for the Economy, Northern Ireland.

The funding bodies’ shared policy aim for research assessment is to secure the continuation of a world-class, dynamic and responsive research base across the full academic spectrum within UK higher education.

REF objectives

The REF objectives are to:

• provide accountability for public investment in research and produce evidence of the benefits of this investment
• provide benchmarking information and establish reputational yardsticks, for use in the higher education sector and for public information
• inform the selective allocation of funding for research.

Find out more on the REF website.

REF progress update: November 2023

REF progress update: December 2023

Last updated: 8 April 2024

This is the website for UKRI: our seven research councils, Research England and Innovate UK. Let us know if you have feedback or would like to help improve our online products and services .

Putting science to work for the health of women

ORWH Accepting Applications for an Advancing Gender Inclusive Excellence Coordinating Center

ORWH and the National Institute of Diabetes and Digestive and Kidney Diseases recently re-issued a request for applications for an Advancing Gender Inclusive Excellence (AGIE) Coordinating Center. The purpose of the AGIE Coordinating Center is to provide the organizational framework for the management, direction, and overall coordination of all common activities aimed at investigating strategies, approaches, and interventions. This includes promoting gender equity or addressing barriers to gender equity for women, at the faculty and leadership levels in many areas of science, technology, engineering, mathematics, and medicine (STEMM). 

The research conducted under the AGIE Coordinating Center will provide a more inclusive environment to enhance gender equity in retention and advancement at the faculty and leadership levels in the STEMM academic and research workforce. The primary vision for the AGIE Coordinating Center is to ensure collaborative coordination of data collection. The AGIE Coordinating Center will serve as a means for developing and operating as a centralized resource hub to collect, store, and disseminate resources for and results of current and future programs. The research is anticipated to include diverse designs, approaches, and multidisciplinary and interdisciplinary academic disciplines. It will also target different or multiple organizational levels (from interpersonal to departmental to institutional). 

Visit the funding opportunity page to learn more information, key dates, award and eligibility information, application and submission instructions, and contact information. 

Director’s Messages

March 27, 2024

February 27, 2024

January 25, 2024

November 30, 2023