research paper of postgraduate

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• For authors
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 103, Issue 9
• How to do a postgraduate research project and write a minor thesis
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-4637-1416 Trevor Duke 1 , 2 , 3
• 1 Centre for International Child Health , University of Melbourne , Melbourne , Victoria , Australia
• 2 School of Medicine and Health Sciences , University of Papua New Guinea , Port Moresby , Papua New Guinea
• 3 Paediatric Intensive Care Unit , Royal Children’s Hospital , Melbourne , Victoria , Australia
• Correspondence to Prof Trevor Duke, University of Melbourne Department of Paediatrics,Royal Children’s Hospital, Parkville, VIC 3052 ; trevor.duke{at}rch.org.au

Many universities and colleges in low-income and middle-income countries require a masters dissertation or thesis for as part of postgraduate training, and some colleges offer a 1-year to 2-year diploma of child health as a clinical qualification to enable skills in child health for generalists, or as part of the early phase of paediatric training. This paper describes the stages of doing a research project for such a masters or diploma, and describes in detail how to write a minor thesis. The paper is designed to provide a practical approach for junior researchers, and their supervisors. Colleges differ in their formal requirements of a minor thesis (word count, line spacing, referencing style), but this paper outlines the principles and practical issues rarely covered elsewhere.

• medical education
• paediatric training
• research training
• low and middle income countries

https://doi.org/10.1136/archdischild-2018-315340

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

What is a minor thesis and why do it.

A minor thesis is a written, systematic description of your project. It has a structure, and tells the story of your research: why you did it, how you did it, what you found and what it means.

The reasons to do a research project during your postgraduate training include to:

learn the basics of research theory and techniques, including epidemiology and biostatistics;

understand how to do a literature review, and how to appraise the literature to address questions;

explore an area of interest for you (develop some expertise and a deeper understanding of a topic);

give you tools to critically and thoughtfully appraise problems you are faced with every day;

learn to communicate scientific research in verbal presentations and written form.

Making a start on your project

Figure 1 shows many of the stages in conducting a research project. The order is not necessarily fixed, some steps, such as a literature search can be done early when considering a research question, or done later in the writing phase (often it is done in both phases). The literature review summarised in the thesis is usually focused on the research question, that is the question leads to the literature review (which is done in a systematic way). Learning about literature searching (through PubMed or Medline), learning about a reference manager database like EndNote and learning about databases and statistical analysis programs like Excel or SPSS can be done at any time, but best to start these early in the process. Not all steps are needed for some projects, such as a relatively simple clinical audit, but it is good to know what steps to consider.

• Download figure
• Open in new tab
• Download powerpoint

Stages in a research project and writing a minor thesis.

Choosing a research topic

A diploma of child health (DCH) thesis should address a simple clinical or public health question. It could be an audit of a condition or practice which is of interest to you. A research project often starts with an idea that interests you, or a problem you have noticed. You may have 12 months for your DCH (realistically even less, about 10 months prior to exams). The project should ideally be able to be completed in 4–6 months. This is because the main part of DCH training is developing clinical competence in paediatrics, and being a good registrar.

A masters project is expected to be more involved, and should be planned early, ideally in the year before you begin collecting data. You usually have 18 months to 2 years. You should choose a topic of interest, explore what has already been written on the subject, what local research exists, in what context this has been done, talk to your supervisors and take time to identify clear research questions and choose a feasible and practical method for your study. But a masters project and thesis should not dominate your time, as the main task of postgraduate training is to develop the all-round skills to be a good paediatrician. This includes an understanding of child health epidemiology and research methods, but too much of a focus on research can detract from clinical responsibilities and clinical learning.

A clear research question

While many students find it easy to identify a theme or topic, it is often more difficult to define a research question or questions. A research question should be objective and answerable using a research methodology. Research questions can be quantitative, qualitative or a combination of both. Quantitative research questions generate data that are measures or values, which can be used for descriptive and inferential statistics (such as ‘what are the causes of anaemia in children presenting to Kimbe Hospital?’ ‘what is the prevalence of disability among children admitted to hospital in Vientiane?’ or what ‘what is the long-term outcome for very low birth weight neonates in Honiara?’ The questions may be general (‘what are the patient  characteristics of children with MDR TB?’) or specific (‘what is the success rate of outpatient treatment of moderate gastroenteritis?’ and ‘what proportion of newborns with PROM will need antibiotic treatment?’). Many questions will be objective/numerical (‘what is the case fatality rate for birth asphyxia at Goroka Hospital between 2011 and 2016?’); some will be categorical (‘what are the common types of seizures that children with epilepsy in Yogyakarta have?’).

Qualitative research generates broader understanding of opinions, or reasons, providing insight. It can help explain the reasons for quantitative results. Qualitative research questions may include perceptions of patients, parents or healthcare workers (‘what do adolescents with rheumatic heart disease understand about their condition?’ or ‘what are the greatest concerns of the parents of children with epilepsy?’ or ‘what are the perceptions and experiences of families of children with cancer?’). Such qualitative research may help explain, for example, why adolescents have low (or high) adherence rates with secondary prophylaxis with benzathine penicillin for rheumatic heart disease. But qualitative research helps us understand more deeply the reasons why things are as they are. 1 2

Many projects will have a mix of both quantitative and qualitative questions, and it is important to identify the differences early, so the right methodologies can be applied. Most research projects have more than one research question, and some research questions are not fully defined when you start a study, they develop along the way. However, it is important to have at least one clearly defined research question to begin.

Definition of terms and metrics of measurement

From an objective research question, the clear definition of terms and metrics of measurement are derived. For example, the population should be described in detail. A population descriptive could be as broad as ‘all children admitted to "X" hospital between March and July 2018’, or as focused as ‘infants with HIV as diagnosed by PCR testing’  box 1 .

Case example

Definition of terms: ‘what is the long-term outcome for very low birth weight (VLBW) neonates in Honiara ( the population )?’ The definition of terms might include:

Definition of population: VLBW 1000–1499 g, born in Honiara national referral hospital or born outside and referred to the special care nursery between January and December 2018.

Definition of outcomes and standardised methods of measurement:

Nutrition: z-scores weight for length and weight for age at 12 months. The proportion of children with moderate and severe malnutrition.

Motor outcomes: the proportion of children at 12 months with moderate or severe neurological deficit identified by a standardised developmental examination (such as the Bayley’s test).

Sensory outcomes: the proportion of children with impaired hearing based on autoacoustic screening.

Anaemia: the proportion of infants at 12 months with a haemoglobin <10 g/dL (WHO standardised definition).

Infectious disease morbidity: the number of hospital admissions from infections in the first 12 months, after initial hospital discharge: these could be subcategorised into acute respiratory infections, gastrointestinal infections, febrile illnesses, etc.

P—among children under 2 years of age with moderate-to-severe pneumonia or bronchiolitis presenting to an emergency department.

I—does nebulised hypertonic saline given in up to three doses over 2 hours?

C—compared with standard care including antibiotics, oxygen.

O—result in a lower respiratory distress score and fewer children requiring inpatient care.

T—over the first 12 hours.

Such a description of a study is not suitable to all studies that are not controlled intervention studies, and PICOT can rarely be used to describe all aspects of a study. However, if you read a journal article abstract of a randomised trial and cannot summarise it in terms of PICOT, then it is either a poorly written article, or poorly conducted study.

Choosing a valid methodology

Broadly, there are two major types of studies—descriptive and analytical. Descriptive studies describe a problem or outcome; analytical studies describe an association between exposures and outcomes. Descriptive study designs include case-reports, case series and descriptive epidemiology. Analytical study designs include randomised controlled trials, cohort studies, case-control studies, case-crossover studies, cross-sectional studies, pre-post studies and some observational studies.

What study design you use depends on many factors, including the research question being addressed, how common the condition is, whether the question is asking about short-term or long-term outcomes, how long you have to do the study, etc. You should discuss with your supervisor exactly what type of methodology is appropriate to answer the question(s) being addressed.

Basic steps to collecting, analysing and presenting qualitative descriptive information 2

Verbatim recording of responses from interviews, focus groups or observations.

Coding of this information, deidentifying where appropriate to ensure confidentiality.

Recording insights and reflections on the data.

Sorting through the data to identify similar phrases, themes, patterns, opinions and also identifying where these differ or diverge.

Gradually deciding on a small group of common themes that address the question, or generalisations (such as opinions or commonly reported practices) that hold true for the population studied.

Examine these generalisations in the light of existing knowledge and/or quantitative data.

Presenting the data according to themes, described in an objective way, with the verbatim responses from subjects as evidence for that generalisation. This helps to ensure that the analysis stays close to the informants’ points of view.

Developing your database: be clear about the metrics

An important element in successful research is the development of a data analysis plan. An analysis plan informs how you organise your data. It ensures your data collection form will have questions designed specifically to address your research questions, and that resultant databases are set up to answer your research questions. It is useful to develop a data analysis plan with your supervisor prior to designing any data collection tools, in order to ensure necessary questions are included in the most appropriate format, and that you understand the statistical tests you will be using.

With guidance from your data analysis plan, the data collection forms should will be designed to answer your research questions. The database should be developed early and checked by you and your supervisor to ensure it reflects the data collection forms, and that it is easy to enter data and analysable in order to answer these questions.

Some quantitative variables will have a numerator and denominator, and you should identify these. Identify the metrics clearly and early, and ensure you record them consistently (do not mix up metrics in your spreadsheet, such as recording some ages as months and some as years, or some weights in grams and others in kilograms). If there is more than one metric for an outcome—such as the definition of severe malnutrition, which may differ depending on age and tests done (weight-for-age / weight-for-height / mid-upper arm circumference / body mass index) then have four columns each labelled as the separate metric, and a ‘summary column’, which is binary 1/0 if severe malnutrition, based on whether the appropriate metric is present.

Ethics and funding

All studies should have ethics review. This is mandatory for publication in a peer-reviewed journal, and needed to fulfil the international standards of ethical research (Declaration of Helsinki 1964). Ethics approval is usually not required for a systematic review of previously published research, but most masters theses require an original research project with primary data collection. Research ethics is not meant to be a barrier, but an enabler of good research that is in the patients’ interests. There are a number of ethical principles that all researchers should adhere to, including consent, autonomy and human dignity, confidentiality, non-maleficence (doing no harm), beneficence (maximising good outcomes), justice and scientific integrity. 3 4 It is good to learn about ethics along the way, as this can help you be a good doctor and paediatrician, as well as a good researcher.

Set aside some time every week to do some work on your study and thesis

Begin to write your thesis early, just making a start is a big hurdle. You will have done a lot of work in the proposal stage, so use that document and expand on it. Organise your computer, with a separate folder for your study and thesis documents. Allocate time each week to work on your study or writing.

A masters study and thesis are best done over an extended period of time, and it is good to keep making some progress each week . Trainees sometimes have a writer’s block, or their study falters, or they become discouraged if they leave it for weeks or months. If you do something each week you maintain the threads of thought and keep others around you interested in your study, and it is good for your morale! Registrars who leave their project or writing to the end do not get as much out of their research experience as those who work steadily on it over 2 years, keeping clinical and research training requirements balanced.

Refer regularly back to the study questions

Each research project will have a number of questions, some major or primary outcomes and some supplementary or additional questions. They should be described clearly in the methods of the thesis and you should be able to identify them in your data collection form, your database and your thesis.

Keep your supervisor informed and interested in your study and thesis progress

Your supervisor is a busy person, with clinical, teaching and other responsibilities. But your supervisor should feel involved in your study, so keep them up to date on your progress, discuss problems with them early and listen to their feedback. Sometimes small research obstacles can be overcome easily if you discuss them, but if you keep them to yourself they become magnified in your mind. A problem shared is a problem halved! Communicate with your supervisor in person and by email, share your thesis writing with them. You should plan to meet your supervisor regularly; while some of these meetings will be to discuss clinical aspects of your training, set aside some time to focus on your project specifically. Sometimes your clinical supervisor will be different from the person who is supervising your research project. If you feel you are not getting the support you need from your supervisor, contact the masters coordinator or the professors to seek more assistance. Often this will be because your supervisor is busy, or may not be sure of how to help you—it is not because they do not want to! Do not let problems go unaddressed—always ask.

Documents and backup

Write one master document, which can be used for the initial technical and ethics proposal, then expanded during your study to form your thesis, or sometimes adapted as a report to a funding agency, and concisely revised as a journal article you may write after completion of the study. Some people use multiple documents for each of these purposes and have multiple thesis versions on their computer. That is OK if it works for you, but it can be inefficient and confusing. The advantage of using one master document is that when you update or revise it, add a table, figure or reference or correct an error, everything gets updated and you do not have ‘version confusion’. You will inevitably end up with a couple of versions: first the proposal, then use that document to expand to a thesis document, then a paper you might write that is based on your study. Clearly name (and date) each type of document and each version, for example, Audit of antibiotic use Masters project proposal Your Name February 10  2018.doc and Audit of antibiotic use Masters thesis Your Name May 6  2018.doc , or a file name that clearly indicates what the purpose of the document is and when it was written or revised. Discuss version control with your supervisor(s) early in your research project—ensure you all agree on a system. A good method might be for you to date each version, and when your supervisor(s) review or edit your work, they add their initials to the file name. This ensures it is clear who has worked on the document. Ask your supervisor to ‘track changes’ so any suggested edits are found easily.

Set up a folder that contains all the separate documents for your thesis, do not just have them randomly scattered in your computer.

Back up the latest version of all thesis and study documents regularly, and keep the backup flash-drive or external hard drive in a safe and secure place. On your backup drive, only have your project documents, and do not use it in other people’s computers to minimise the risk of viruses. Backing up your data and study documents on cloud storage can be useful if you have reliable internet access: Dropbox is an open-access program, which provides free (up to 2 GB) cloud storage ( https://www.dropbox.com ), Google Drive is another method for cloud storage of documents for security in case your computer malfunctions. Cloud storage also enables sharing of documents with your supervisor.

Writing style

In a thesis write in the past tense. A study proposal (ie, for a study not yet underway) is usually written in the future tense: " In this study we plan to recruit 30 children with asthma. Children with acute intermittent asthma and chronic persistent asthma will be eligible. Children with a past history of Tb or bronchiectasis will be excluded" . In a thesis the same information in the past tense reads: " In this study we recruited 30 children with asthma. Children with acute intermittent asthma and chronic persistent asthma were eligible. Children with a past history of Tb or bronchiectasis were excluded" .

Writing should be concise and clear. I find it useful to read sentences out aloud to myself and ask, is the meaning clear and unambiguous? It is useful to ask someone else to read over your thesis at the end; not necessarily a medical person who is familiar with the topic, but an educated person who writes well (such as a teacher or scientist in another field). A thesis should be written in language that people without specific knowledge of the subject can understand.

Thesis structure

There is no one right way to write a thesis, but there is a generally accepted structure which has a number of major parts. A thesis can be written with these major parts as chapters, or as one document like a journal article with major subheadings. The major parts are mentioned here in this article.

A very brief (total  250–300 words) overview of the whole minor thesis. It usually has a structure (introduction, method, results, conclusions), each part with just one or two sentences. The abstract should fit on one page. It should state clearly the research question, the study type, the main method, the primary results and a conclusion (one to two sentences) based on the primary result. The abstract does not need references.

Introduction should state why you did the study, why it is an important topic to research. This may only be three to four paragraphs, especially if you have a literature review which follows. The introduction provides contextual information as to why you did the study. The introduction can highlight the burden of the disease or condition, why it is an important one in your country or in the region or globally, why it is important for your specialty. Your introduction may outline whether the condition or topic has been well studied or neglected in terms of research or study, and identify the gap that will be addressed by your study. In a DCH thesis, you might just write about why you became interested in the topic because you cared for a patient with the condition, or you are seeing a lot of such problems in your ward or hospital, etc. In the introduction, you should state clearly the objectives of your study, and the research question(s) being addressed.

Literature review

Literature review is a summary of the existing knowledge of the research topic. It should be focused, on the general topic and specific research question being addressed. In doing a literature review, keep notes on the search terms you use and on the search engines used (eg, PubMed, Google Scholar). For accessing full-text articles, papers are increasingly published as open access and you can access these through these search engines, however if your university or hospital has registered for Hinari program set up by WHO ( http://www.who.int/hinari/en/ ), and you have good internet band speed you will have access to thousands of other journals and ebooks.

In the opening paragraph of your literature review state how you did the search, for example, " I searched the published literature using PubMed and Google Scholar using the terms ‘asthma’ ‘spacer device’ ‘salbutamol’ ‘treatment’ ‘metered dose inhaler’ ‘home-made’ and ‘developing country’ or ‘low and middle income country’" . While there are formal methods of searching for information using standardised key words, phrases and MESH terms, when you are starting out with literature searching, try to make the search reflect closely your research question, and record exactly the words you use in the search.

Usually a literature review is written from ‘general to very specific’; the opening paragraph outlines the previously published information on the burden of the condition or the magnitude and significance of the problem, then subsequent paragraphs focus more narrowly on the previous literature on your specific research question(s). A good literature review integrates existing research under themes or subheadings, rather than having a series of paragraphs which just state what previous studies have found. A literature review may be structured thematically and have subheadings to identify themes. For example, a literature review for a study of the role of home-made spacers for asthma could start with the literature on the burden/prevalence of asthma in the community, and a paragraph about overall treatments (beta-2 agonists, corticosteroids), another about mechanisms of drug delivery (oral, metered dose inhalers, nebulisation), one about the effectiveness of spacers and their availability in certain areas and then focus specifically on the existing literature about home-made spacer devices. It is not necessary to write everything about asthma, but focus the review on the question you are studying; start general and move to very specific. Sometimes there may not be much literature on the subject you are studying, that is OK, just summarise what exists and highlight the gaps. There is no need to ‘pad out’ the literature review with peripheral information, make it relevant, concise and informative.

See references and reference manager program (below) to plan how to organise and cite your references.

In most journal articles, the literature review is incorporated into the discussion (and a little by way of background in the introduction). This can be the case in a minor thesis, but it is up to you. Most people find it easier to have a separate section, which is the literature review at the beginning, but there are no fixed rules.

Methodology

Methodology is a detailed description of how you actually did the study. It should be reproducible, that is, be written in a way that would enable another investigator to reproduce the same methodology and arrive at the same result (or a different result in a different population). Write the methodology of your study early, while you are doing it. Details of the exact method can be missed if it is not written until the end of your study. You will have written a methodology for your study proposal, you should start with this, change the future to past tense and add any details that have changed. Often at the end of a study, the methodology is a little refined or different to that which was submitted as a proposal. In the methodology there are no results, or other literature, and the only references are for methodology, or statistical or analytical methods described by another researcher in the literature. The methodology should include:

the study population;

definitions of all terms, cases, conditions;

how the study was planned and carried out;

the exact data you collected;

how, when, how often and where it was collected;

how the data were managed (entered into a database);

what the database is;

the analytical tests you did;

ethics approval;

explain any methodological problems and how they were addressed.

Results are presented as narrative, tables, graphs and figures. There are no fixed rules, except the results should be organised and logical. Unless it is very important and central to your thesis, it is best not to duplicate the exact result in each medium of presentation (eg, the same information—such as the baseline characteristics of the study population presented in a figure, table and narrative). Tables can provide a lot of detail and be informative; if presented as a written narrative, this can make the results too long and be tedious. The narrative should highlight the main results, describing in detail the outcomes for the primary research question, important secondary outcomes and orientate the reader to the tables, graphs and figures: explain what they describe, their main findings (without describing every detail). Each table, figure or graph must be referenced in the narrative (eg, ‘see figure 2’) at the appropriate point in the narrative, or this can be written as ‘table 1 describes the baseline characteristics of the children with asthma’, then describe in narrative the key baseline characteristics, but other details can be left for the reader to discover from examining the tables.

It is often thought that the results cannot be written until the study is complete. While this is partly true, it is important to plan and organise the results section early. Put subheadings into your thesis document that will help guide your subsequent analysis and writing. Draw up ‘dummy tables’ that contain the variables you want to report even before you have analysed the data, but leave the cells which will contain the numerical result blank. You may add other variables later, but drawing up such tables as you do the study will help you organise the presentation of results.

There is no fixed limit to the number of tables a minor thesis can have, but avoid duplicating data in the tables and the figures, and avoid a figure or graph that describes an obscure or unimportant aspect of the study. You do not have to include every result or data point that you gathered in the results, this can make a thesis seem to be lacking perspective, you may not ‘see the wood for the trees’, that is, you may miss the really important points because of too much information being reported, and the reader will find it tedious to read.

In tables and the narrative, include numbers (%), not just percentages. Percentages give no indication of the spread or precision of the data. Saying that 33% of children with asthma have chronic symptoms tells us limited information, as it does not indicate precision. The CIs may be extremely wide depending on the number of subjects investigated: 3/10=30% (0.3; 95% CIs 0.06 to 0.65), which means the true proportion could be anything from 6% to 65% in a small sample of 10 children with asthma. If the sample that yielded 30% of children with asthma as having chronic symptoms was 100 children, that is, 30/100=30% (95% CIs 0.21 to 0.40), that is, the true proportion is somewhere between 21% and 40%, a much more precise result. So never just quote a % in the results without clarifying what ‘n’ is (the number with the characteristic/number studied), and ideally include a CI around such proportions. There are many programs that can calculate CIs around a simple proportion, one is: http://www.sample-size.net/confidence-interval-proportion/

The analyses of most diploma and masters studies involves primarily basic descriptive epidemiology , rather than the need for any complex analytical tests, like regression or p values. It is important to get the descriptive epidemiology right, describe the population or subjects in the study in a logical way, numbers, percentages or proportions, median or means, ranges, IQRs or CIs. Be clear about numerators and denominators. Denominators will sometimes be different depending on the variable being described, and the number of observations made. It is not useful to have p values for every statistic, and many good thesis can be written without p values at all! For some analytical tests, such as the comparison of two proportions to determine if they are different (5 of 83 children in grade 1–4 have asthma, whereas 9 of 206 children in grades 5–7 had asthma) a p value can be useful in indicating whether an observed difference is likely to have arisen by chance.

For the presentation of most analyses of a comparison between groups, CIs are much more informative than p values. Avoid the trap of reporting multiple p values and the mistaken view that finding a p value of <0.05 is ‘significant’. If you do enough statistical tests a p value of <0.05 will arise merely by chance, not because a real difference exists. A p value alone cannot determine that a hypothesis is ‘true’, or if results are important. Only report p values and other analytical tests on your primary and major secondary results if indicated. Avoid using p values to report minor post hoc results (findings from your study that you did not plan), these are often just chance findings that reflect unanticipated bias or confounding. Unanticipated post hoc results may be interesting and generate further hypotheses for another study, but they should not be interpreted as definitive or conclusive. Often inexperienced researchers or readers assume that a ‘significant p value’ is conclusive when it provides preliminary information only. So do not overinterpret your results or fill them with p values and obscure statistical tests, but describe the findings clearly and concisely in plain language. The results section of a minor thesis does not have to be long, it can be one to two pages of narrative, and can have subheadings for each of the relevant results.

Discussion is an interpretation of the results, what they mean. The discussion should return to the objectives and the primary research questions of the study and indicate whether they have been achieved. The discussion should be organised into thematic paragraphs or sections, and subheadings can be a good way to guide the reader. The first paragraph of the discussion should restate the main result(s) of the study in plain language (without statistical tests). The discussion should compare the results with previous studies or pre-existing knowledge of the subjects. It should not duplicate the literature review (but it can link to this), but the discussion should cite other research where the results can be compared or contrasted with your own. Other research may be in different contexts (different populations, different level of resources, different methodologies, different outcomes measured). It is important to understand why two different studies addressing the one problem may yield different results, it does not mean that one study is inferior or inadequate.

The primary outcome or question is usually addressed in the first few paragraphs (with comparisons with other research if appropriate) and then secondary outcomes discussed (with similar comparisons).

Write in ways that indicate you are scientifically open to unexpected outcomes, and understand the uncertainty of your study. Do not write, for example: "I did this study to prove that probiotics reduce the duration of watery diarrhoea", but rather ‘this study was done to evaluate whether probiotics would reduce the duration of watery diarrhoea’. Similarly, do not be too categorical or certain: ‘this study proves that steroids should be used to treat chronic otitis media’, rather ‘in this small study of school aged children with chronic serous otitis media, topical corticosteroids reduced the duration of purulent ear discharge’.

A discussion of the shortcomings or limitations of the study is appropriate. Every study ever done has some limitations, so it does not mean you have done a poor study, just because there are some limitations. You should be honest about the limitations, but do not overstate them either (ie, it is not necessary to make excuses for a study). This is especially the case for so-called ‘negative studies’, that is, studies that did not find a difference or a significant result in the primary outcome; such studies can be just as important as ‘positive studies’, so you should not feel such a study is unworthy. Furthermore, you should not try to highlight an obscure post hoc p value that is <0.05 in order to give your study ‘significance’. In the limitations section, try to identify if there are any systematic biases or confounding in your study; this will help you avoid overstating the results. In reading other papers for your literature review, and for your own research, beware of the risk of type II error; the finding that there is ‘no difference’ when this could be due to a small sample size (a study that is underpowered to find a statistical difference, even when one exists). Again, for a DCH or masters thesis, do not worry if your study seems ‘too small’, it is the quality of the work that is important, and this will not be judged solely on the number of patients recruited into a study.

No new results can be introduced in the discussion section; if in writing the discussion you find a new result that you want to include or comment on, you need to incorporate it in the results section first, and then you can make comment on it in the discussion.

Conclusions/recommendations

The conclusions should be brief. Return to the research questions and suggest whether they have been answered or solved, and what the main finding was. Return to the objectives of the study and suggest if they have been achieved. Indicate what has been learnt from the study and how it might be applied. And indicate any clear recommendations that you think can come out of your study, which may be a change in clinical practice, increased awareness of the topic or problem, the adoption of your research method into everyday practice or the need for further research to be conclusive about a finding.

Any important statement of fact in the thesis should have a reference; this is obvious in the literature review, but applies equally to the introduction and discussion. References are not usually included in the methods, unless you are referencing another investigator’s method that you have used, or a statistical test or program that you used. Appropriate references may be journal articles identified during your literature review, or books or websites. When referencing a fact or finding from a journal ideally go to the original research paper which identified the finding, not an opinion paper which just repeated the finding. Ideally, you should get a copy of the full-text (not just the abstract) of the article that contains the finding that you are reporting as a fact, to verify that the paper did indeed say that. This is a type of fact checking that is good to do.

There are many ways to reference a thesis, references can be cited in the text as numbers (eg, 3, 5 , or [3, 5]) or (authors name, year). It does not matter as long as it is consistent throughout the thesis. I find the (authors name, year) format can make it hard to read the text if there are many references interspersed throughout the paragraphs. The numerical citations are more acceptable for the vast majority of journal styles, so it is best to do it this way, so it is easier to revise your thesis into a journal article.

There are various standards for referencing, including the Harvard Referencing Standard. There is no source document for Harvard style, but many universities have their own guidelines based on the Harvard style, such as Adelaide University: http://www.adelaide.edu.au/writingcentre/referencing_guides/harvardStyleGuide.pdf.

It is best to have a reference manager program , which can make reference lists in a particular journal style for you after you input the references from your literature search. There are many such programs now, including EndNote ( www.endnote.org ), which is used by a lot of scientists and clinical researchers. Using EndNote you can import citations directly from Google Scholar and PubMed searches, and you can download the full-text article if it is Open Access. EndNote is a commercial product, but other reference manager programs are freely available. Zotero is a free, open-source reference manager with cloud backup into which you can manually insert citations or extract citations from internet search engines ( https://www.zotero.org/download/ ). Other programs include Mendelay ( https://www.mendeley.com/ ) and ‘Cite This For Me’ ( http://www.citethisforme.com/harvard-referencing ).

It takes a bit of work learning how to use EndNote or Zotera and how it links to Pubmed and Google Scholar and to learn how to insert references into such a program, but it is well worth it.

When referencing a website, you should state the title of the website, the title of the web page, the exact URL and the date you accessed it.

If in your thesis you include or adapt tables or figures from another source (such as a website), you need to reference these.

Journals have different styles as to the number of authors listed in a reference. Some, if a reference has six authors or less include all their names. If a reference has seven or more authors just include six names, then ‘ et al ’.

Thesis order and length

The order of the pages of a thesis.

The full title of the thesis.

The full name of the candidate.

The name of the school associated with the research.

The year the thesis was submitted.

Declaration

Except where acknowledgement has been made, the work is your own.

The work has not been submitted previously, in whole or in part, to qualify for another degree or assessment.

Acknowledgements

Table of contents

Lists of tables, figures and diagrams

Main body of the thesis (as described above)

Introduction—including objectives and specific research question(s);

Literature review;

Discussions;

Conclusions and recommendations;

Reference list.

Appendices—these are not mandatory in every thesis, but appendices can be helpful in enabling the reader to see what research tools you used or exactly how you gathered the data. Appendices may include:

Tables of raw data if appropriate, they have to be well formatted and not identify any patient by name or place of origin (confidentiality);

Ethics approval document;

Consent form used in the study;

Data collection form used in the study;

Any educational material used in the study.

Anything included in the appendices should be cited in the text of the main thesis, so the reader understands why they are included, for example, "We used a survey form that included basic demographic information, questions on frequency of asthma, asthma severity, previous hospitalisations, medication use, whether the child has an asthma action plan and what method the child used for taking Ventolin (see appendix 2)”. The reader can then refer to appendix 2 to see exactly what questions were asked.

Avoid plagiarism

It is really important that a postgraduate diploma or masters thesis be all your own work. It is too tempting these days to cut-and-paste information from Wikipedia, journal articles found online or other internet sites, but this is plagiarism and it is unacceptable and a form of cheating. Sometimes plagiarism occurs inadvertently if you are unsure of what the university standard is, or sometimes it occurs when people are under stress of deadlines. Regardless, it is always unacceptable. If you start only writing in your own words you will not fall into the trap of plagiarism. Do not directly copy anything from another source, but write it completely in your own words and add the original source as a reference. Many universities use programs like Turnitin to check for plagiarism. Another good program is Grammarly, which is both a Grammar and Plagiarism check. It is available at https://www.grammarly.com/1.

Writing up a paper for publication

It often takes several rounds of revisions to get a paper in shape for submission, but you should aim to write your thesis in a format where the most important or original data are publishable. You may not be able to include all the results or text in a journal publication. Brevity is important, and often ‘less is more’. Ask your supervisor about writing up your thesis for submission to a journal.

Acknowledgments

The author would like to thank Professor Mike English (Kenya Medical Research Institute), Eleanor Neal, Dr Jocelyn Chan, Dr Hamish Graham (University of Melbourne) and Dr Bob Phillips for very helpful comments, and to the postgraduate students for their inspiration and examples. This paper drew on a number of sources, including ‘Minor thesis and research report structure’, from the Study and Learning Centre, RMIT University: http://mams.rmit.edu.au/8zbeww6e1n3i.pdf (accessed 11 February 2017). There is a book by Paul Gruber and Justin Zobel from University of Melbourne, which the author could not access, but this may be a useful reference. ‘How to write a better minor thesis’ Melbourne University Press, 2014 ( https://www.mup.com.au/items/135100).

• Taylor SJ ,
• Neergaard MA ,
• Andersen RS , et al
• Munyaradzi M ,

Funding The author gratefully acknowledges the RE Ross Trust for support to child health research training in Papua New Guinea and the Pacific, which is the basis of this work.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

• The Open University
• Guest user / Sign out
• Study with The Open University

My OpenLearn Profile

Personalise your OpenLearn profile, save your favourite content and get recognition for your learning

About this free course

Become an ou student, download this course, share this free course.

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

1 Important points to consider when critically evaluating published research papers

Simple review articles (also referred to as ‘narrative’ or ‘selective’ reviews), systematic reviews and meta-analyses provide rapid overviews and ‘snapshots’ of progress made within a field, summarising a given topic or research area. They can serve as useful guides, or as current and comprehensive ‘sources’ of information, and can act as a point of reference to relevant primary research studies within a given scientific area. Narrative or systematic reviews are often used as a first step towards a more detailed investigation of a topic or a specific enquiry (a hypothesis or research question), or to establish critical awareness of a rapidly-moving field (you will be required to demonstrate this as part of an assignment, an essay or a dissertation at postgraduate level).

The majority of primary ‘empirical’ research papers essentially follow the same structure (abbreviated here as IMRAD). There is a section on Introduction, followed by the Methods, then the Results, which includes figures and tables showing data described in the paper, and a Discussion. The paper typically ends with a Conclusion, and References and Acknowledgements sections.

The Title of the paper provides a concise first impression. The Abstract follows the basic structure of the extended article. It provides an ‘accessible’ and concise summary of the aims, methods, results and conclusions. The Introduction provides useful background information and context, and typically outlines the aims and objectives of the study. The Abstract can serve as a useful summary of the paper, presenting the purpose, scope and major findings. However, simply reading the abstract alone is not a substitute for critically reading the whole article. To really get a good understanding and to be able to critically evaluate a research study, it is necessary to read on.

While most research papers follow the above format, variations do exist. For example, the results and discussion sections may be combined. In some journals the materials and methods may follow the discussion, and in two of the most widely read journals, Science and Nature, the format does vary from the above due to restrictions on the length of articles. In addition, there may be supporting documents that accompany a paper, including supplementary materials such as supporting data, tables, figures, videos and so on. There may also be commentaries or editorials associated with a topical research paper, which provide an overview or critique of the study being presented.

Box 1 Key questions to ask when appraising a research paper

• Is the study’s research question relevant?
• Does the study add anything new to current knowledge and understanding?
• Does the study test a stated hypothesis?
• Is the design of the study appropriate to the research question?
• Do the study methods address key potential sources of bias?
• Were suitable ‘controls’ included in the study?
• Were the statistical analyses appropriate and applied correctly?
• Is there a clear statement of findings?
• Does the data support the authors’ conclusions?
• Are there any conflicts of interest or ethical concerns?

There are various strategies used in reading a scientific research paper, and one of these is to start with the title and the abstract, then look at the figures and tables, and move on to the introduction, before turning to the results and discussion, and finally, interrogating the methods.

Another strategy (outlined below) is to begin with the abstract and then the discussion, take a look at the methods, and then the results section (including any relevant tables and figures), before moving on to look more closely at the discussion and, finally, the conclusion. You should choose a strategy that works best for you. However, asking the ‘right’ questions is a central feature of critical appraisal, as with any enquiry, so where should you begin? Here are some critical questions to consider when evaluating a research paper.

Look at the Abstract and then the Discussion : Are these accessible and of general relevance or are they detailed, with far-reaching conclusions? Is it clear why the study was undertaken? Why are the conclusions important? Does the study add anything new to current knowledge and understanding? The reasons why a particular study design or statistical method were chosen should also be clear from reading a research paper. What is the research question being asked? Does the study test a stated hypothesis? Is the design of the study appropriate to the research question? Have the authors considered the limitations of their study and have they discussed these in context?

Take a look at the Methods : Were there any practical difficulties that could have compromised the study or its implementation? Were these considered in the protocol? Were there any missing values and, if so, was the number of missing values too large to permit meaningful analysis? Was the number of samples (cases or participants) too small to establish meaningful significance? Do the study methods address key potential sources of bias? Were suitable ‘controls’ included in the study? If controls are missing or not appropriate to the study design, we cannot be confident that the results really show what is happening in an experiment. Were the statistical analyses appropriate and applied correctly? Do the authors point out the limitations of methods or tests used? Were the methods referenced and described in sufficient detail for others to repeat or extend the study?

Take a look at the Results section and relevant tables and figures : Is there a clear statement of findings? Were the results expected? Do they make sense? What data supports them? Do the tables and figures clearly describe the data (highlighting trends etc.)? Try to distinguish between what the data show and what the authors say they show (i.e. their interpretation).

Moving on to look in greater depth at the Discussion and Conclusion : Are the results discussed in relation to similar (previous) studies? Do the authors indulge in excessive speculation? Are limitations of the study adequately addressed? Were the objectives of the study met and the hypothesis supported or refuted (and is a clear explanation provided)? Does the data support the authors’ conclusions? Maybe there is only one experiment to support a point. More often, several different experiments or approaches combine to support a particular conclusion. A rule of thumb here is that if multiple approaches and multiple lines of evidence from different directions are presented, and all point to the same conclusion, then the conclusions are more credible. But do question all assumptions. Identify any implicit or hidden assumptions that the authors may have used when interpreting their data. Be wary of data that is mixed up with interpretation and speculation! Remember, just because it is published, does not mean that it is right.

O ther points you should consider when evaluating a research paper : Are there any financial, ethical or other conflicts of interest associated with the study, its authors and sponsors? Are there ethical concerns with the study itself? Looking at the references, consider if the authors have preferentially cited their own previous publications (i.e. needlessly), and whether the list of references are recent (ensuring that the analysis is up-to-date). Finally, from a practical perspective, you should move beyond the text of a research paper, talk to your peers about it, consult available commentaries, online links to references and other external sources to help clarify any aspects you don’t understand.

The above can be taken as a general guide to help you begin to critically evaluate a scientific research paper, but only in the broadest sense. Do bear in mind that the way that research evidence is critiqued will also differ slightly according to the type of study being appraised, whether observational or experimental, and each study will have additional aspects that would need to be evaluated separately. For criteria recommended for the evaluation of qualitative research papers, see the article by Mildred Blaxter (1996), available online. Details are in the References.

Activity 1 Critical appraisal of a scientific research paper

A critical appraisal checklist, which you can download via the link below, can act as a useful tool to help you to interrogate research papers. The checklist is divided into four sections, broadly covering:

• some general aspects
• research design and methodology
• the results
• discussion, conclusion and references.

Science perspective – critical appraisal checklist [ Tip: hold Ctrl and click a link to open it in a new tab. ( Hide tip ) ]

• Identify and obtain a research article based on a topic of your own choosing, using a search engine such as Google Scholar or PubMed (for example).
• The selection criteria for your target paper are as follows: the article must be an open access primary research paper (not a review) containing empirical data, published in the last 2–3 years, and preferably no more than 5–6 pages in length.
• Critically evaluate the research paper using the checklist provided, making notes on the key points and your overall impression.

Critical appraisal checklists are useful tools to help assess the quality of a study. Assessment of various factors, including the importance of the research question, the design and methodology of a study, the validity of the results and their usefulness (application or relevance), the legitimacy of the conclusions, and any potential conflicts of interest, are an important part of the critical appraisal process. Limitations and further improvements can then be considered.

Enhancing the recruitment of postgraduate researchers from diverse countries: managing the application process

• Published: 21 April 2021
• Volume 82 , pages 917–935, ( 2021 )

Cite this article

• Kyung Hye Kim   ORCID: orcid.org/0000-0001-6529-7050 1 &
• Helen Spencer-Oatey   ORCID: orcid.org/0000-0002-7676-9495 2  

3482 Accesses

6 Citations

7 Altmetric

Explore all metrics

International students form an important element of most universities’ internationalisation strategies, especially for research and the recruitment of high calibre PhD students (PGRs). Despite the numerous studies of PGRs’ post-arrival experiences, there is a major dearth of research into their pre-arrival, application experiences. Given the worldwide competition for high calibre PGRs, along with impact posed by the Covid-19 pandemic and by Brexit for the UK, it is vital for universities to ensure that factors clearly under their control, such as the information on their websites and the way they communicate, are as informative and helpful as possible. In this article, we draw on social media data to examine the challenges and uncertainties that Korean PGR applicants experienced in navigating the process of applying to UK universities. The paper compares their confusions with information available on university websites and recommends a series of points that higher education institutions should check for. It also reveals and discusses issues associated with communication. While the data has been collected from Korean social media websites, we argue that our paper has broader relevance for the following reasons. First, the same fundamental intercultural issues—different educational systems and different background knowledge—apply to PGR applicants from other countries and so their queries are likely to be similar or comparable. Second, the insights gained from social media websites to facilitate the application process and thereby enhance recruitment can usefully be applied to other countries and levels of study, in a way that has rarely been done to date.

Similar content being viewed by others

Barriers to Online Learning in the Time of COVID-19: A National Survey of Medical Students in the Philippines

International students challenges during the COVID-19 pandemic in a university in the United States: A focus group study

Student voice in higher education: the importance of distinguishing student representation and student partnership

Avoid common mistakes on your manuscript.

Introduction

The number of international students around the world has expanded rapidly in recent years. The OECD B6 indicator (OECD,  2018 ) shows that over the past two decades, the number of foreign enrolments in tertiary education worldwide has exploded from 2 million in 1998 to 5 million in 2016 for students from non-OECD countries, and from 1.5 million in 1998 to 3.5 million in 2016 for those from OECD countries. Moreover, it states that the more advanced the level of education, the higher the proportion of international students. The report points out that while international students account for only 6% of total enrolment in tertiary education, the proportion rises to 26% for doctoral programmes. On the other hand, the OECD B6 Indicator (OECD,  2018 ) also reveals that there are noticeable cross-national differences in (a) the inflows of international students from master’s to doctoral programmes and (b) the proportion of international students studying for a PhD. This suggests that some countries and/or universities may be better at attracting high calibre international PGRs than others.

In the UK there is a national strategic imperative to recruit more PGRs. The UK Council for Graduate Education (UKCGE,  2020 ) reports that there needs to be a 10.16% rise in postgraduate research enrolments, if the 2027 target of the government’s R&D Industrial Strategy is to be met. However, as can be seen from Fig.  1 , the trend over recent years is not encouraging. In fact, the UKCGE policy paper argues as follows:

PGR Enrolments in the UK (derived from HESA data, 2020 , https://www.hesa.ac.uk/data-and-analysis/students/where-from )

These trends in postgraduate research enrolments show that a 10.16% growth by 2027 will not happen by itself. The Industrial Strategy will depend on the strategic development of the postgraduate researcher pipeline.

Mellors-Bourne et al. ( 2014 ) report that although nearly all the UK universities in their study were seeking to increase their PGR numbers, they had no clear strategies for achieving this.

Almost all institutions are currently seeking at least some growth in their PGR numbers, although this is frequently not translated into formal quantitative targets. Growth is especially sought in international PGRs, although strategies for achieving it were not always clear, other than expressing an aspiration for the best possible talent. (p. 2)

This is concerning. Several institutions mentioned that there was strong competition for the most highly-qualified PGRs and that they often lost out to quicker and more competitive offers from the USA and Canada. Almost one third expected a decrease in demand from international PGRs, partly because of the increase in the number of universities in other countries offering attractive provision in English, and partly due to the reduction in attractiveness of the UK as a study location because of immigration policies. This is clearly in tension with their desire for higher numbers of PGRs.

Attracting more PGRs is likely to be exacerbated by both Brexit and Covid-19. It is extremely important, therefore, that the higher education sector considers ways of attracting a higher proportion of the international PhD student market, while retaining quality. One facet of this is the application experiences of potential applicants, and this is the focus of our article. If universities in (some) other countries are able to make faster offers, as Mellors-Bourne et al. ( 2014 ) report, then it is important to gain more insights into the application process and the challenges that international students may experience with the British system.

In this paper, we take a country case study approach and use social media data to examine the challenges that Korean PGR applicants experienced in navigating the process of applying to UK higher education institutions (HEIs).

PhD places in Korean universities are not particularly competitive, and the number of PhD graduates from Korean universities has increased steadily from 12,625 in 2013 to 15,308 in 2019 (Statistics Korea, 2020 ). However, both universities and companies in Korea prefer to recruit doctoral candidates who have graduated from overseas universities (Cheong & Song,  2010 ; Kim,  2020 ). As a result, those students who want a career in academia or a specialised job in industry are particularly keen to study abroad. These students have clear aims, are committed, and are likely to come back to Korea. They thus offer good opportunities for building research collaborations between their Korean employer and their UK university of study, and thereby help support the UK government’s R&D Industrial Strategy.

Despite our focus on Korea, we argue (as we explain further below) that the relevance of our study is not limited to Korean applicants, nor necessarily to UK HEIs, since many of the uncertainties that the applicants asked about are likely to be similar for other international students, especially those from Asia. We start by reviewing the literature on the (PGR) application process, looking first at HEI perspectives and then at student experiences. As Mellors-Bourne et al. ( 2014 ) point out, ‘the published evidence base relating specifically to the attraction, recruitment and selection, and deployment of PGRs is limited’.

HEI accounts of the PGR application process

The only substantial research into the PGR application process from the perspective of HEIs is that of Mellors-Bourne et al. ( 2014 ). One of the first things these authors point out (p. 3) is that the recruitment and selection process are complex and varied. This variation extends to the decision-making process and the individuals involved, including the roles played by the potential supervisor, departmental academic panel, senior research academic, research degrees committee, and central admissions. Often multiple people/units are involved, but there is variation both across and within universities, including by department and discipline, in their exact roles, power, and influence. In terms of the selection steps, interviews may or may not take place. They more often involve the potential supervisor than an academic panel, but they tend to vary in degree of formality as well as frequency. Mellors-Bourne et al. ( 2014 ) report that the final decision is rarely with the potential supervisor, but that there is often a well-known hierarchical chain of approval following a departmental recommendation.

Almost all institutions that participated in Mellors-Bourne et al. ( 2014 ) study said they would need to have a certain amount of interaction with potential applicants prior to any decision. Most said they would need to spend time explaining the application process, and many (but not all) reported having prior discussions about the research topic. How much feedback was given on a draft research proposal was much more varied, though. The authors argue that given the complex and individual nature of the PGR application process, building a positive initial relationship with a potential supervisor is highly advantageous. However, this in itself can risk issues of fairness. There was an indication that internal candidates would be more likely to receive feedback and support over the whole application process than external candidates, raising potentially serious questions of equity of access, especially for international students. The authors suggest that one way of addressing this problem could be to have pre-application workshops, and they describe one particular university where this had been introduced and seemed to be working well.

Mellors-Bourne et al. ( 2014 ) collected a small amount of data from PGR students, and these also pointed to significant differences between the experiences of internal and external applicants, including problems of transparency faced by the latter. As a result, two of their recommendations to HEIs and policy makers are on gathering information from PGR applicants and students themselves. They recommend collecting data from current or recent PGRs on their experiences, ‘especially application and recruitment processes’ (p. 60) and adding some additional questions to the Postgraduate Research Experience Survey (PRES) on application experiences.

We turn now to the experiences and viewpoints of international PGR students.

The application experiences of international PGR students

There have been numerous studies of the experiences of international students in higher education, including issues of integration (e.g. Spencer-Oatey & Dauber 2019a , b ), effective group work (e.g. Montgomery,  2009 ; Spencer-Oatey & Dauber,  2017 ), expectations around teaching and learning (e.g. Cortazzi & Jin,  2002 ), and classroom participation (e.g. Zhou et al.,  2005 ). However, none of these studies focus on PGR students and they all address issues that students face after the commencement of their studies. Yet, as reported in the previous section, it is quite possible or even likely that prospective students come up against issues during the application stage, and when this happens, both applicants and the UK HEIs may potentially lose out. The applicant may decide to go elsewhere (to a country or institution where the application process is clearer), and staff involved in handling applications (both academic and professional services staff) may spend time responding to queries that could otherwise be reduced. Such concerns were found by Mellors-Bourne et al. ( 2014 ), as reported above.

Despite the recommendation by Mellors-Bourne and his colleagues for some questions to be added to the PRES survey, the 2019 version had no additional questions on application experiences. On the other hand, a report by Archer ( 2016 ), using PGR data from i-graduate’s International Student Barometer, Footnote 1 identifies a section with questions on ‘application and decision-making’. Unfortunately, however, he does not include any account of them in the report. Nevertheless, he makes two relevant recommendations:

Recommendation 9 Given the intensely competitive environment, universities should establish strategies to ensure applications are processed expeditiously and that offers made to students are tracked in order to maximise conversation rates through to enrolment. Recommendation 10 Universities, sector bodies and Government should consider a more substantive process for collecting and collating evidence of the decision making, influences and routes to postgraduate research study in the UK.

(Archer,  2016 , p. 8)

In other words, he draws attention to the importance of monitoring and expediting the PGR application process.

Quacquarelli Symonds (QS) also include some relevant questions in their ‘International Student Survey’ (Berka & Strods,  2020 ). Some questions probed viewpoints on communication during the PGR application process and especially in relation to timing and frequency:

When communicating with a university about an enquiry or application, the importance of not having to wait very long to hear back

How often would you expect a university to contact you after you apply?

How often would you expect a university to contact you after you receive an offer?

Unsurprisingly, most Asian PGR respondents (as well as the sample as a whole) felt it was very important (33%) or fairly important (60%) to hear back soon. Much more surprising, though, was that about a quarter of them felt it was important for the university to contact them daily, both after application and after receiving an offer, and more than half of them felt it needed to be weekly, making an amazing 75% or more wanting frequent or extremely frequent communication.

Other questions in the survey concerned sources of information and pre-arrival contact with the university. Two of these questions were as follows:

Who do you think is best equipped to provide application information?

Who do you think is best equipped to provide information on fees and scholarships?

Respondents had three choices: a current student, university staff, someone else. Not surprisingly 76% of their Asian sample chose university staff. Another question listed a large number of potentially useful information sources for researching a university and asked respondents to select the five most useful to them. The results indicate that university websites, general online search, and university rankings are the most important sources to PGR respondents irrespective of region/nationality. After that, preferred sources that were selected by around one quarter or more of Asian respondents include course finder websites, institution information sessions, social media channels, printed university prospectus, and online forums/chat rooms.

In line with this, it would seem completely understandable that when digitally savvy and academically competent students have not been able to find the information they desire from their top preferred sources, they will turn to online channels and social networking sites. Previous studies (e.g. Royo-Vela & Hünermund,  2016 ; Galan et al.,  2015 ) have shown that for those who do not apply through agents, online channels have a significant impact on postgraduate students’ decision-making processes. However, as Gai et al. ( 2016 , 182) rightly note, previous studies on student recruitment at HEIs almost exclusively use interviews and surveys, overlooking the fact that the whole application procedure involves students carrying out extensive information searches online and a complicated decision making process (Mowen & Minor,  2006 ; Gai et al.,  2016 , 189). The HE context has also been ‘slower in recognising the value of student-generated social media data and content for HE marketing, branding and recruitment purposes’ (Bolat & O’Sullivan,  2017 , 745).

An exception to this is Gai et al. ( 2016 ). They examined the stages of the international student consumers’ decision-making process by looking at extensive authentic data generated and posted by prospective students on ChaseDream.com (an interactive online virtual community for Chinese-speaking students), focusing on the application process for a Business MA degree in the USA. Building on previous work by Galan et al. ( 2015 , 292), who identified four stages of a decision-making journey involving social media (problem recognition, information search, evaluation of alternatives, and purchase decision), Gai et al. ( 2016 , 190) propose a set of five stages: predisposition, information seeking for targeting schools, application, evaluating admission offers, and final decision. They report that during the application stage, ‘users read previous CDer’s [ChineseDeam user] testimonials while preparing for school/program interviews’ (p. 190). This clearly attests to the crucial role that students’ online comments play in the application decision-making process.

However, these authors did not analyse the nature and the type of the queries posted in their own right; nor did they focus on PGR students. So, in order to help complement the limited research data available on the PGR application process, we decided to explore the following research questions:

RQ1: What information is unclear to applicants?

RQ2: What challenges do PGR candidates experience during the application process?

RQ3: What PGR application information is available on UK national and university websites?

RQ3 was included so that we could make a contextual interpretation of the findings from RQ1 and RQ2, allowing us to compare what students were asking about with what information was or was not already available or clear.

Methodology

For RQ1 and RQ2 we collected data from two interactive online forums. South Korea has a particularly vibrant web culture—according to Statistica ( 2020 ), the percentage of the population using the internet has continuously grown since 2000 (44.7%), and it reached 95.9% by 2018. Korea topped (in 2008, 2010, 2011, 2012, 2015, 2016) or ranked second (2013, 2014, 2017) on the list of top thirty countries ranked by the ICT Development Index (IDI), an index published by the United Nations International Telecommunication Union. Footnote 2 Thus, it is not surprising that there are numerous online communities, forums, as well as personal blogs, including those for Korean students seeking help and advice for studying abroad and which allow dynamic interaction. Nevertheless, these online data have rarely been used previously for analysis, although they reveal fascinating insights into the genuine issues and uncertainties that students are facing. These online communities and forums allow users to pro-actively gather relevant background information about the application process, while also retaining anonymity in the cyberspace, thus allowing more open communication.

Questions and comments shared by prospective Korean students in two Korean online forums—‘영국사랑’ ( www.04uk.com , hereafter 04UK) and ‘고우해커스’ ( www.gohackers.com , hereafter GOH)—were retrieved. These online communities and forums were chosen mainly because they are the platforms that allow users to interact and exchange questions and comments relating to the UK, including issues of living and studying in the UK. At the time of writing, 04UK features a section ‘대학(원)’ [u/g and p/g] under the ‘Q&A’ section. GOH is a platform provided and operated by Hackers Language Institute Co., Ltd. As one of the biggest language institutes providing foreign language education in South Korea, it features more specific categories than 04UK, including a ‘영국유학 Q&A’ [Q&A about studying in the UK] section. These two sites contain rich archives of authentic communication and interactions, enabling us to investigate the real struggles that prospective Korean students have faced in terms of their UK application process.

At the time of data collection (16 December 2018), 04UK (which started from October 2010) featured a total of 2797 postings, while GOH (which started January 2006) featured 23,481. In order to obtain only the postings concerning the PhD application process that require some understanding of the UKHE system, postings were narrowed down first by using the search words ‘교수 [professor/lecturer/senior lecturer/reader]’, ‘지도교수 [supervisor]’, ‘튜터 [tutor]’, and ‘렉처러 [lecturer]’. These key words separated the postings that require communication with potential academic staff on the one hand, and on the other, they also allowed us to exclude the questions that could be easily answered by information available on institutions’ websites. Using ‘application’ as the search word would have introduced an unnecessary level of noise. For example, it will also return questions such as ‘what does ‘academic references’ on the application mean?’, ‘somehow I failed to upload the files onto the system’, and ‘May I submit the application now but references later?’. The selected postings were then further refined by excluding those that did not relate to application issues in the UK, e.g., questions about ways to get a reference letter from a Korean academic, as part of the application process required by a UK institution. All these processes resulted in 431 postings for 04UK and 1702 for GOH. No log-in process is required for these two websites for reading, and users can use alias.

The questions were then manually imported into MAXQDA, a software package widely used for qualitative data analysis, so that the comments could be coded and analysed systematically. The comments were translated into English by the first author, before they were imported into the analysis software programme, but both Korean and English wordings were retained in the imported documents to help ensure that no meanings had been lost or slightly altered during the translation process. Both authors then participated in the analysis, first reading through each of the entries several times to familiarise themselves with the data. After this, multiple iterations of conventional content analysis (Hsieh & Shannon,  2005 ) were carried out. In other words, the coding was done inductively, with the coding categories derived directly from the data rather than through the use of a pre-determined set of categories. Each full question was treated as the unit of analysis, and since each question often included multiple different elements, each entry (as is normal in qualitative data analysis) could be coded to more than one code. MAXQDA allows the code system to be built with a hierarchical structure, so that there are parent codes, children codes, grandchildren codes, and so on.

For RQ3 we examined the information on the application process available from the five UK universities who have the largest number of Korean PhD students (see Fig.  2 ). The institutions mentioned in the student queries could also in principle have been examined, but the names of institutions were not usually mentioned. Moreover, the aim of this study is not to offer tailor made suggestions to specific institutions but rather to (a) identify types of challenges Korean applicants experience in navigating the process of applying to UK HEIs and (b) make recommendations that can be applied across different institutions.

The five universities in the UK with the largest number of Korean PhD students (The Korean Education Centre UK,  2019 )

At each institution’s website, ‘Application’ or ‘postgraduate research programme’ pages (not a specific study or department) which offer current general application practices and information were examined. This is because each study/department may offer subtly different level of specific application information, and it is impractical to review it within the scope of this article. Each university has its respective YouTube channel, so their channels were checked as well. Although they present some videos about application processes, none were found to focus specifically on PhD student applications. Our analysis will thus be on the website information. In analysing these various data sources, we focus on the application challenges of prospective PGRs, which had a particularly high number of codings (for a different focus—challenges in relating to UK academics—please see Kim and Spencer-Oatey, 2020 ). This main theme had four main sub-themes: application steps, communication uncertainties, financial queries, and English language proficiency issues (see Fig.  3 ). For space reasons, we focus here on the first three sub-themes.

Screenshot of the main code categories of ‘Application Challenges’ and their coding frequencies (collapsed view)

This section reports on the three most frequently queried sub-themes and compares them with the information provided on key university websites.

Issue 1: Application steps—Social media findings

By far the largest number of questions concerned the steps involved in the application process. There were 114 such questions, covering a range of issues, as shown in Fig.  4 .

Screenshot of the main code ‘Application Steps’, with sub-themes and coding frequencies (uncollapsed view)

Application paperwork (23 codings) refers to questions about the content of the necessary application paperwork, particularly about the nature of the research proposal and queries about references. In other words, applicants were uncertain about how to write or prepare the documentation required by the university.

Another issue of concern was sequencing and timing. Applicants were unsure whether they should contact a potential supervisor prior to application or just apply directly. There were 23 codings on this issue, indicating a fair amount of concern among the potential applicants. Students became particularly concerned when there was a time delay in hearing back, leading them to wonder whether they had followed the right procedure or not. Others had a variety of questions about timing—how long it takes to get an offer/decision or even a response. For instance, one person had been waiting 6 weeks for a formal offer, when their future supervisor had said it would take about 3 weeks from his unofficial offer. This length of time contrasts strongly with the expectations revealed in the QS survey data reported above.

A third main set of codes related to uncertainties over the decision-making process. This received the largest number of questions relating to application steps. There were 32 codings to this category, indicating that there is quite widespread confusion over this issue. The uncertainty mentioned most frequently (16 codings) was responsibilities for decision-making, especially the relative power of the supervisory professor and the admissions office in the process (e.g. Q1 and Q2).

Q.1 I wonder if the potential supervisor agrees, the admission office would offer me an acceptance letter. Or do you think the admission office would reject my application even when the potential supervisor agreed? [GOH56, Code AS-DM-DMR] Q.2 Is there any case where a PhD application is rejected even when a professor showed a positive response? [GOH56, Code AS-DM-DMR]

The second most frequent uncertainty over decision-making (10 codings) concerned ways of handling a weak academic record. Sometimes this was interlinked with uncertainty over decision-making responsibilities, as indicated in the next quote (e.g. Q3).

Q.3 I had an interview with my professor in person 10 days ago. Everything went fine, and the professor said there would be no problem in getting the acceptance. However, I’m worried because my GPA is low. The professor said it would be covered by my three years’ experience in R&D, and some projects I was involved in, and because my research plan was impressive. I wonder if the supervisor has more power over the admission decision. Or, do other professors in the department also have to agree to the PhD candidate? [GOH224, Code AS-DM-HWR].

A few students had other decision-making queries, such as what key criteria an admission committee uses and whether a rejection decision can be reconsidered.

A fourth important group of questions relating to application steps concerned identifying a suitable supervisor (26 codings). Some of this stemmed from confusion over role labels and who was eligible to supervise PhD students (e.g. Q4).

Q.4 The university website shows Lecturer, Reader, and Professor under the Academic Staff. I wonder if they can all be PhD supervisors, or if only Professors can be the supervisors. [GOH66, Code AS-SS-RL]

This confusion, of course, affects students’ ease of identifying potential supervisors. Sometimes students asked about other aspects of finding a supervisor, such as whether a Korean academic would be willing to take a Korean PhD student and whether to trust a professor who is not willing/able to provide any funding.

The fifth group of questions relating to application steps concerned interviews (10 codings). Some had been invited to an interview and wondered who would conduct the interview, what questions would be asked, and so on. Others, who had not been invited for interview, wondered whether not being invited was problematic as they had heard of others being interviewed.

Issue 1: Application steps—Website information

Among the five universities in Fig. 2 , three provide a decision timeline or flow chart describing the application process, while the other two only provide a list of required documents and a general description of the procedure and right of appeal. In addition to a decision timeline, the University of Oxford describes six clear steps in their application guideline.

Few of the universities explicitly mention who is in charge of the application process. The University of Cambridge is one of the few who does, stating that the departmental faculty will consider the application first, before the application is approved by a degree committee and Graduate Admissions Office on behalf of the Board of Graduate Studies. This statement at least delivers a message to potential applicants that the admission process is beyond the control of individual academic staff. Other universities mention ‘our admissions selectors’ or ‘Graduate Admissions office’ as the body who will process the applications, but applicants may assume faculty members are part of the ‘selectors’, and none of the other four provide such explicit information as that of The University of Cambridge.

Some universities mention the possibility of re-evaluation of the submitted application, a potential waiting list, and a process for complaints. All universities in Fig.  3 quite clearly encourage applicants to identify an appropriate supervisor and to send an informal research enquiry either directly to the academic or the departmental contact prior to submitting the application. However, none of the five specifically state whether all academics are eligible to supervise PhD students. For example, University College London (UCL) offers a very detailed way to search for a potential supervisor, by stating ‘[s]earch our online research repository (UCL Discovery) where all UCL’s research papers are published, subject to approvals. If you identify a research paper that particularly interests you, it is likely that one of the authors would be a suitable research supervisor’. However, often an online research repository presents teaching fellows, too, who are in the teaching category. Although this can be checked through personal contact between the applicant and the academic, providing no indication in the general guide page on the supervision eligibility of the academics may cause confusion for international students who are not familiar with the UK HE system.

Issue 2: Communication uncertainties—Social media findings

The second broad area of concern to students was communication. There were 87 codings to this category, covering three main areas, as shown in Fig.  5 : making contact, interpreting responses/guidelines, and wording the content of a message.

Screenshot of the main code ‘Communication uncertainties’, with sub-themes and coding frequencies (uncollapsed view)

There were 38 queries about making contact and, as can be seen from Fig.  5 , the two most frequent issues were whether and how often to make contact, and what to do when there was no response or a slow response. In terms of making contact, queries with this coding were not simply about getting accepted (as with Application Steps, Sequencing and Timing). They were asked either before or after acceptance and indicated additional uncertainty over managing the supervisory relationship and research responsibilities (e.g. Q5 and Q6).

Q.5 I wonder if contacting the professor only twice is enough. It seems others have more frequent communication with a potential supervisor. I wonder what other things you discuss with the professor. [GOH257, Code CU-MC-S] Q.6 How often do you usually get in touch with your supervisor after receiving an offer, before you leave the country to study in the UK? [GOH138, Code CU-MC-S]

Students became particularly concerned when there was a delay in response and wondering whether they should send a follow-up email. When they did receive a reply, either from the supervisor or from admissions, they sometimes had difficulty interpreting the meaning (e.g. Q7 and Q8). There were 32 queries of this kind.

Q.7 I had an interview with a professor, which went fine. The professor said he was satisfied with the interview, but he thought my academic record might not be recognised and added that I should wait as he'd write a letter to the senior tutors. I do not understand 'your academic record may not be recognised'. What does this mean? [04UK3, Code CU-IR] Q.8 I sent an email with my CV and SOP attached to a professor to see if s/he could be my PhD supervisor. Only this strange answer came back. What does this mean? “Thank you for your enquiry. All requests for supervision are via the Doctoral School. Please contact them and they will give you details of the process.” [GOH214, Code CU-IR]

In all these cases, the student needed to draw inferences and/or rely on relevant background contextual knowledge (e.g. about the application system and procedures) in order to be able to make sense of the supervisor/university response.

A third but smaller group of queries over communication concerned how to handle awkward situations and what to say. Examples of issues here were how to provisionally accept an offer while expecting to turn it down later, how to politely decline an offer after the potential supervisor has been very helpful, and whether it is impolite to ask a professor about funding opportunities. The latter then brings us to the third main issue: financial queries.

Issue 2: Communication uncertainties—Website information

Some institutions provide detailed information about ways to contact supervisors prior to submitting applications. The University of Manchester (UoM) and UCL, for example, provide very specific guidelines. UoM even details how to compose an email to a prospective supervisor, and UCL offers detailed guidelines for ways to make a research enquiry, including the amount background needed, as follows:

Academic members of staff are extremely busy people and receive a lot of research enquiries. In a recent survey 67% of staff said they receive research enquiries that do not relate to their interests. It is extremely important to research supervisor’s interests thoroughly before you contact them. If they do not think your enquiry is related to their research they may not have time to respond to you (UCL, Guidance on how to contact potential supervisors).

Such information will certainly help applicants structure their emails to prospective supervisors, and further reduce unnecessary additional email exchange caused by inappropriate email or an email from applicants with insufficient information. It also clearly tells them they may not receive a reply if they have not contacted an appropriate academic.

Issue 3: Financial queries—Social media findings

A third broad area of concern to students was finance. There were 30 codings on this, covering three main areas, as shown in Figure 6 : finding financial support (13 questions), finance and supervisor relations (13 questions), and procedural queries (4 questions).

Screenshot of the main code ‘Financial queries’, with sub-themes and coding frequencies (partially collapsed view)

13 questions focused on obtaining financial support, including the opportunities available (e.g. different scholarship schemes, work opportunities, supervisor’s grant), criteria for success, and timings associate with acquiring funding. Another 13 questions asked about relational matters, particularly whether it was appropriate to ask the supervisor about financial issues or whether this would cause offence.

Q.9 I wonder if I also can ask the questions about funding. … I'm afraid I'll make a bad impression even before I apply. [GOH58, Code FQ-SR-A]

There were four further questions about procedural matters, such as how to complete a funding application form.

Issue 3: Financial queries—Website information

As mentioned in Gai et al. ( 2016 , 189), cost is a very important factor for prospective students. All universities provide a significant amount of information about access to financial support. The University of Oxford presents a separate page for their list of studentships by clearly stating the following: ‘[c]heck your academic department’s website for details of any current studentship opportunities and how to apply. There is a section in the online graduate application form where you will be able to enter the code(s), if required, for any studentships for which you wish to be considered’. Footnote 3 Imperial College London provides a search tool for all available scholarships and studentships. The University of Manchester, among others, goes further and specifies that candidates can discuss feasible funding opportunities with staff: ‘[a] good first step in exploring funding options is to speak to staff within your subject area to find out how previous students have financed their studies’. This should make prospective applicants feel comfortable over contacting staff about financial issues before they apply. Nevertheless, information such as this is rarely spelled out in other universities’ admission pages.

Discussion and recommendations

Information provision.

As shown by the findings reported above, the Korean students had a very large number of questions about the various steps involved in applying for a PhD at UK HEIs. Since educational systems are very different, this is completely understandable. The greatest number of questions was about the decision-making process, and especially who has responsibility for what. Of the five UK universities sites examined, only one of them included an explanation of this. Most universities provide information on funding opportunities, but students seem to have questions about more subtle aspects.

As a follow-up study, we looked at four UK national-level websites that provide information on studying in the UK, to check how much information they offered for PGR applicants: UKCISA, Study UK, Steps to Postgraduate Study, and Prospects. Only Prospects provides any information on the PGR application process; others only cover more generic information about studying in the UK, e.g. visa application, and the structure of lectures and seminars. Prospects provide a wealth of information for PGRs on many aspects of the application process, including detailed pages on how to write a research proposal and find a suitable supervisor. One possibility, therefore, would be for universities to include a link to relevant sections on the Prospects webpages. However, this could usefully be supplemented by more specific and local information on universities’ own websites. Similarly, each universities’ respective official YouTube channels can be linked to the university admission pages (strangely they are not currently), and a series of short videos exclusively focusing on a specific PhD application issue can be produced. Since the YouTube platform allows viewers to freely leave comments, just like the forums examined in this study, prospective students can ask questions in a more casual environment and receive prompt responses from the university as well as current registered students, while the university can identify issues international students may have with less effort.

We recommend that HEIs review the extent to which they provide information on the following:

The expected length of a research proposal.

A description of the parties involved (e.g. departmental academic staff and central admissions) in decision-making and what their respective roles are.

Criteria for decision-making and the extent to which they are/are not negotiable (e.g. whether a decision made by central admissions can(not) be overridden by other parties, including by the potential supervisor, or vice versa).

A timeline or a flow chart that details the whole process, including (a) estimated average time for a decision to be made and (b) sequencing of the various elements (e.g. whether prior contact with a potential supervisor is essential, recommended, or unimportant).

The role that interviews play.

Explanation of who has the right to supervise PhD candidates, including how this corresponds with academic titles.

All sources of funding—university-level, national-level, project-linked funding, other funding opportunities—with full details and relevant links.

Length of requirement for full fees.

Opportunities for teaching assistant work.

Given that there are noticeable differences between departments or faculties (such as is often the case for English language requirements), making this clear could significantly reduce confusion.

Too much information in one place can be overwhelming, so careful consideration also needs to be given as to how and when this is provided. Another way of helping to reduce students’ uncertainties would be for institutions to offer more ‘online enquiry sessions’ for PGR students. Students could be encouraged to submit queries in advance if they wish, and the event in itself could help legitimise for students their right to raise questions while simultaneously lowering their concerns that questions could in themselves reflect badly on them. In addition, such events could help address the unequal playing field for internal and external applicants (i.e. those already at the university and those applying from outside) that Mellors-Bourne et al. ( 2014 ) have identified.

Communication management

The two main areas of concern around communication were the timing and frequency of contact with the potential or future supervisor and difficulties in interpreting messages. It is useful here to consider how communication operates. Although human communication to a large extent exploits a language code (such as English, Chinese, or Korean), it is not feasible for everything to be conveyed explicitly in the code. As a result, people use two main sources of knowledge to construct meaning in interaction: linguistic knowledge (i.e. knowledge of the language code) and ‘world’ knowledge (i.e. experiential and theoretical knowledge of social processes, facts, concepts, and so on) (Spencer-Oatey & Franklin, 2009 ; Kecskés, 2014 ). In communication between international applicants and university staff, both sources of knowledge can give rise to potential difficulties, for the following reasons:

There is less shared background knowledge, so the interpretation of meaning becomes more dependent on explicit explanations/descriptions.

Staff may be unaware of the extent to which they are assuming shared background with applicants when wording their communications with them.

Staff may be unaware of the extent to which the wording of their communications with applicants is implicit rather than explicit.

When these situations prevail, such ‘meaning construction’ as explained above can be particularly problematic. As a result, mismatches may occur in the messages that people think have been communicated, or alternatively, one or both parties may be confused and may request clarification. We therefore recommend that universities review their information and messaging from the following angles:

How explicitly is the message conveyed?

What background information is needed to understand the message?

What guidelines are provided regarding contacting a supervisor?

In fact, the clearer and the more substantial the information provided for the application steps, the greater the background knowledge prospective applicants will have and the easier it will be for them to interpret the underlying meaning of messages. It would also be useful for staff to receive some training in intercultural communication, using examples such as Q7 and Q8 to help them understand the importance of considering the recipient’s level of background knowledge of application processes when structuring their messages. A specially designed e-course could be particularly helpful for this.

There were also concerns about maintaining a positive impression and how asking certain questions might affect the (potential) supervisor’s evaluation of them (e.g. Q9) (see Kim and Spencer-Oatey, 2020 , for further details on this aspect).  

Limitations and concluding comments

Our study has only focused on Korean students interested in studying for a PhD in the UK. As such, we cannot be sure whether PGR students of other nationalities are likely to face similar challenges. Nevertheless, since the same fundamental intercultural issues apply—different educational systems and different background knowledge—we believe that PhD applicants from other countries are likely to have similar or comparable queries. We maintain, therefore, that our findings not only relevant for Korean applicants but also relevant to PGR students from other countries, at the very least for reflection and checking.

The principle of gaining insights from social media websites to facilitate the application process may also be relevant for other levels of study. While the application process is more nationally controlled and less negotiable for undergraduate students (whatever their background), with fewer associated ‘application steps’, such students may still have uncertainties that emerge from such data and the universities are unaware of. Master’s students are likely to have even more such uncertainties than undergraduate applicants, and in fact within our dataset (not reported here), they raised quite a wide range of questions, several of which were similar to those reported here for PhD students.

So, we strongly recommend that universities establish projects to monitor online virtual communities in different countries, similar in principle to the two forums examined in this study. This will enable them to identify the uncertainties that prospective students are experiencing and to gain more insights into the information that is missing from their websites or application handling procedures. A very useful outcome could be set of FAQs that are based on the questions raised in such forums. One option could be to recruit alumni to help with this. Alternatively, university networks or national bodies could support such initiatives.

As reported above, Mellors-Bourne et al. ( 2014 ) recommend gathering more data on PGR applications from PGR students, including adding relevant questions to key national surveys. Social media data such as reported in this study could help inform the design of the questions in such surveys, in terms of both focus and detailed wording. For example, the QS survey International Student Survey (Berka & Strods,  2020 ) refers only to ‘university staff’ and does not distinguish between academic and admissions staff. Yet this distinction is extremely important for PGR applicants as it relates closely to uncertainties around who has the right to make application decisions. Such more specific options, therefore, need to be incorporated in future survey designs.

Consideration also needs to be given to national-level websites offering information on recruitment and study in the UK (e.g. UKCISA’s webpages). At present only one such site (Prospects) provides any meaningful detailed information on PGR applications, and others could usefully consider their own role in relation to PGRs. Since HEIs almost certainly have different postgraduate application procedures at a detailed level, it could be helpful for national-level webpages to make that clear, to help reduce the kind of confusion that can occur when some universities encourage prior contact with a potential supervisor and others do not.

Given the UK government’s R&D targets for 2027, the recruitment challenges resulting from Brexit and the Covid-19 pandemic, as well as the shifting power dynamics internationally (e.g. the trend for China to become a major regional hub for international students, especially for those from other Asian countries, as Bothwell,  2020 argues), it is clearly important for universities in the UK and beyond to maximise their PGR recruitment potential. Now could therefore be a crucial time to draw on the insights revealed by our study, to review current practice, and to plan for the future by exploring comparable forums in other countries.

Postgraduate Research Experience Survey (PRES): https://www.advance-he.ac.uk/reports-publications-and-resources/postgraduate-research-experience-survey-pres Prospects: prospects.ac.uk/postgraduate-study/phd-study. Steps to Postgraduate Study: postgradsteps.officeforstudents.org.uk. Study UK: study-uk.britishcouncil.org. UK Council for International Student Affairs (UKCISA): https://www.ukcisa.org.uk/ University of Cambridge Postgraduate Application Process page: https://www.graduate.study.cam.ac.uk/application-process/what-happens-next (last accessed 6 July 2020). University College London ‘Guidance on how to contact potential supervisors’ page: https://www.ucl.ac.uk/prospective-students/graduate/sites/prospective-students_graduate/files/potential-supervisor.pdf (last accessed 6 July 2020). University of Manchester ‘Choosing a supervisor’ page: https://www.manchester.ac.uk/study/postgraduate-research/admissions/how-to-apply/choose-supervisor/ (last accessed 6 July 2020). University of Manchester ‘Funding’ page: https://www.manchester.ac.uk/study/postgraduate-research/funding/ (last accessed 6 July 2020). University of Oxford Application Guide page: http://www.ox.ac.uk/admissions/graduate/applying-to-oxford/application-guide (last accessed 6 July 2020). University of Oxford Decision timeline page: https://www.ox.ac.uk/admissions/graduate/after-you-apply/decision-timeline?wssl=1 (last accessed 6 July 2020).

https://www.i-graduate.org/international-student-barometer

https://www.itu.int/en/Pages/default.aspx (last accessed 21 August 2020)

https://www.ox.ac.uk/admissions/graduate/courses/courses-open-for-studentships?wssl=1 (last accessed 6 July 2020)

Archer, W. (2016). International postgraduate research students: the UK’s competitive advantage. London: The UK HE International Unit. Retrieved from https://www.universitiesuk.ac.uk/International/news/Pages/uk-competitive-advantage.aspx [30 September 2020].

Berka, A & Strods, C. (2020) International Student Survey, electronic dataset, Quacquarelli Symonds, viewed 5 October 2020.  http://qs.com .

Bolat, E., & O’Sullivan, H. (2017). Radicalising the marketing of higher education: learning from student-generated social media data. Journal of Marketing Management, 33 (9–10), 742–763. https://doi.org/10.1080/0267257X.2017.1328458 .

Article   Google Scholar  

Bothwell, E. (2020). China told to reform education model to become global student hub. https://www.timeshighereducation.com/news/china-told-reform-education-model-become-global-student-hub . Accessed 19 August 2020.

Cheong, S., & Song, A. (2010). 대학·기업, ‘해외파 박사’ 선호가 문제 [Universities and Companies’ preference for recruiting doctoral candidates who have graduated from overseas universities is problematic], University News Network, http://news.unn.net/news/articleView.html?idxno=62668 . Accessed 3 November 2020.

Cortazzi, M., & Jin, L. (2002). Cultures of learning, the social construction of educational identities. In D. C. S. Li (Ed.), Discourses in Search of Members: In Honor of Ron Scollon (pp. 49–78). New York: University Press of America.

Gai, L., Xu, C., & Pelton, L. E. (2016). A netnographic analysis of prospective international students’ decision-making process: implications for institutional branding of American universities in the emerging markets. Journal of Marketing for Higher Education, 26 (2), 181–198.

Galan, M., Lawley, M., & Clements, M. (2015). Social media’s use in postgraduate students’ decision-making journey: an exploratory study. Marketing for Higher Education, 25 (2), 287–312. https://doi.org/10.1080/08841241.2015.1083512 .

HESA. (2020). Higher education student statistics: UK, 2018/19. https://www.hesa.ac.uk/news/16-01-2020/sb255-higher-education-student-statistics/location . Accessed 19 August 2020.

Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research , 15 (9), 1277–1288.  https://doi.org/10.1177/1049732305276687

Kecskés, I. (2014). Intercultural Pragmatics . Oxford: Oxford University Press.

Google Scholar  

Kim, K. H., & Spencer-Oatey, H. (2020). Goals, uncertainties and contextual assessments: Korean students’ concerns about relating to UK academics. Pragmatics . https://doi.org/10.1075/prag.20004.kim .

Kim, S.-C. (2020). [아침을 열며] 국내 대학원이 외면받는 이유 [[Opening Morning] Reasons why Korean local graduate schools are avoided], https://www.hankookilbo.com/News/Read/202001231101774318 Accessed 3 November 2020.

The Korean Education Centre UK. (2019). 2019년 주재국 내 한국인 유학생 현황 [Status of Korean international students in the host country in 2019] http://www.koreaneducentreinuk.org/wp-content/uploads/2019/05/2019%EB%85%84-%EC%98%81%EA%B5%AD-%EB%82%B4-%ED%95%9C%EA%B5%AD%EC%9D%B8-%EC%9C%A0%ED%95%99%EC%83%9D-%ED%98%84%ED%99%A9.pdf Accessed 3 November 2020.

Mellors-Bourne, R., Metcalfe, J., Pearce, E., & Hooley, T. (2014). Understanding the recruitment and selection of postgraduate researchers by English higher education institutions. Report to HEFCE by CRAC/Vitae and iCeGS. Available at http://www.ukcge.ac.uk/Search/Default.aspx?q=Understanding+the+recruitment+and+selection+of+postgraduate+researchers+by+English+higher+education+institutions [Retrieved 30 September 2020].

Montgomery, C. (2009). A decade of internationalisation: has it influenced students’ views of cross-cultural group work at university? Journal of Studies in International Education , 13 (2), 256–270.  https://doi.org/10.1177/1028315308329790

Mowen, J. C., & Minor, M. (2006). Consumer behaviour: a framework . Mason, OH: Thomson.

OECD. (2018). Education at a Glance 2018: OECD Indicators, https://www.oecd-ilibrary.org/education/education-at-a-glance-2018_eag-2018-en Accessed 19 August 2020.

Royo-Vela, M., & Hünermund, U. (2016). Effects of inbound marketing communications on HEIs’ brand equity: the mediating role of the student’s decision-making process. An exploratory research. Journal of Marketing for Higher Education , 26 (2), 143–167. https://www.tandfonline.com/doi/abs/10.1080/08841241.2016.1233165?journalCode=wmhe20

Spencer-Oatey, H., & Dauber, D. (2017). The gains and pains of mixed national group work at university. Journal of Multilingual and Multicultural Development, 38 (3), 219–236.

Spencer-Oatey, H., & Dauber, D. (2019a). What is integration and why is it important for internationalisation? A multidisciplinary review. Journal of Studies in International Education, 29 (5), 515–534.

Spencer-Oatey, H., & Dauber, D. (2019b). Internationalisation and student diversity: opportunities for personal growth or numbers-only targets? Higher Education, 78, 1035–1058.

Spencer-Oatey, H., & Franklin, P. (2009). Intercultural interaction: a multidisciplinary approach to intercultural communication . Basingstoke: Palgrave Macmillan.

Book   Google Scholar  

Statistica. (2020). Internet usage in South Korea - Statistics & Facts https://www.statista.com/topics/2230/internet-usage-in-south-korea/ [Accessed 4 November 2020].

Statistics Korea. (2020). 국내/외 박사학위 취득자 현황 [Current status of PhD graduates inside/ouside Korea] Retrieved from http://www.index.go.kr/potal/main/EachDtlPageDetail.do?idx_cd=1550 [Accessed 4 November 2020].

UK Council for Graduate Education (UKCGE). (2020). Industrial strategy: Postgraduate education's role in delivering the research and development target. UKCGE Resource: Policy Briefings. Retrieved from http://www.ukcge.ac.uk/article/policy-briefing-doctoral-researchers-rd-404.aspx#_edn2 [Accessed 2 October 2020].

Zhou, Y. R., Knoke, D., & Sakamoto, I. (2005). Rethinking silence in the classroom: Chinese students’ experiences of sharing indigenous knowledge. International Journal of Inclusive Education, 9 (3), 287–311.

Download references

Acknowledgement

We are grateful to Universities UK International (UUKi) and the UK Department for Business, Energy and Industrial Strategy (BIES) for a Rutherford Strategic International Fellowship in 2018 which allowed Kyung Hye Kim to carry out collaborative research in Warwick, UK, with Helen Spencer-Oatey, and to Shanghai Jiao Tong University and University of Warwick for their support. We also thank the anonymous reviewers for their helpful feedback on an earlier version of the article.

Author information

Authors and affiliations.

Shanghai Jiao Tong University, Shanghai, China

Kyung Hye Kim

University of Warwick, Coventry, UK

Helen Spencer-Oatey

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kyung Hye Kim .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Kim, K.H., Spencer-Oatey, H. Enhancing the recruitment of postgraduate researchers from diverse countries: managing the application process. High Educ 82 , 917–935 (2021). https://doi.org/10.1007/s10734-021-00681-z

Download citation

Accepted : 08 January 2021

Published : 21 April 2021

Issue Date : November 2021

DOI : https://doi.org/10.1007/s10734-021-00681-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Application process
• Recruitment
• Higher education
• International research students
• Uncertainties
• Find a journal
• Publish with us
• Track your research

POSTGRADUATE STUDENTS, MONA: ESSENTIALS: OFFICIAL UWI GUIDE FOR THESES / RESEARCH PAPERS

• THE LIBRARY
• THE POSTGRADUATE LEARNING COMMONS -- PGLC
• OFFICIAL UWI GUIDE FOR THESES / RESEARCH PAPERS
• ASSIGNMENT: IF YOU MUST PRODUCE ONE OF THESE
• MONA CAMPUS SERVICES

THE UWI THESIS GUIDE 2015

UWI THESIS GUIDE 2015

It's available ONLINE

Its new, just released  so check back for important pointers

Postgraduate students may request a one-on-one session with a Librarian. During the consultation the librarian will scan through your Research Paper/Thesis and advise about correct formatting and whether you are adhering to the reference style. 

Please complete the Thesis Consultation Request Form which is also available at the library. Please bring a printed copy of your thesis.

- Its a  free  service.

Thesis Consultation Request Form

• << Previous: THE POSTGRADUATE LEARNING COMMONS -- PGLC
• Next: RESOURCES >>
• Last Updated: Aug 10, 2019 4:19 PM
• URL: https://libguides.uwi.edu/c.php?g=401757

Dudley Knox Library

Nps theses - dudley knox library.

  Ask a Librarian       My Accounts

NPS Theses & Dissertations

Find NPS-authored scholarly works (NPS Theses, Dissertations, Capstone Project Reports, MBA Professional Reports, Joint Applied Project Reports, and other NPS degree-earning written works)—both publicly releasable and restricted.

Public Collection

NPS Archive: Calhoun

Publicly releasable information  1923-present

Search the Public Collection

Restricted Collection

NPS Restricted Collection

Controlled Unclassified Information (CUI) 2001-present

  Who can access?

To request an account, you must be both of these:

• Current NPS member
• Federal employee/contractor or U.S. military

Already have an account?              Search the Restricted Collection

Are you a new user?                       Request an Account

You might also be interested in...

• Restricted Resources
• Thesis Processing Office

Main Navigation

• Contact NeurIPS
• Code of Ethics
• Code of Conduct
• Create Profile
• Journal To Conference Track
• Diversity & Inclusion
• Proceedings
• Future Meetings
• Exhibitor Information
• Privacy Policy

NeurIPS 2024

Conference Dates: (In person) 9 December - 15 December, 2024

Homepage: https://neurips.cc/Conferences/2024/

Call For Papers 

Author notification: Sep 25, 2024

Camera-ready, poster, and video submission: Oct 30, 2024 AOE

Submit at: https://openreview.net/group?id=NeurIPS.cc/2024/Conference  

The site will start accepting submissions on Apr 22, 2024 

Subscribe to these and other dates on the 2024 dates page .

The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024) is an interdisciplinary conference that brings together researchers in machine learning, neuroscience, statistics, optimization, computer vision, natural language processing, life sciences, natural sciences, social sciences, and other adjacent fields. We invite submissions presenting new and original research on topics including but not limited to the following:

• Applications (e.g., vision, language, speech and audio, Creative AI)
• Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)
• Evaluation (e.g., methodology, meta studies, replicability and validity, human-in-the-loop)
• General machine learning (supervised, unsupervised, online, active, etc.)
• Infrastructure (e.g., libraries, improved implementation and scalability, distributed solutions)
• Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)
• Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces)
• Optimization (e.g., convex and non-convex, stochastic, robust)
• Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes)
• Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)
• Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior)
• Theory (e.g., control theory, learning theory, algorithmic game theory)

Machine learning is a rapidly evolving field, and so we welcome interdisciplinary submissions that do not fit neatly into existing categories.

Authors are asked to confirm that their submissions accord with the NeurIPS code of conduct .

Formatting instructions:   All submissions must be in PDF format, and in a single PDF file include, in this order:

• The submitted paper
• Technical appendices that support the paper with additional proofs, derivations, or results 
• The NeurIPS paper checklist  

Other supplementary materials such as data and code can be uploaded as a ZIP file

The main text of a submitted paper is limited to nine content pages , including all figures and tables. Additional pages containing references don’t count as content pages. If your submission is accepted, you will be allowed an additional content page for the camera-ready version.

The main text and references may be followed by technical appendices, for which there is no page limit.

The maximum file size for a full submission, which includes technical appendices, is 50MB.

Authors are encouraged to submit a separate ZIP file that contains further supplementary material like data or source code, when applicable.

You must format your submission using the NeurIPS 2024 LaTeX style file which includes a “preprint” option for non-anonymous preprints posted online. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review. Papers may be rejected without consideration of their merits if they fail to meet the submission requiremhttps://www.overleaf.com/read/kcffhyrygkqc#85f742ents, as described in this document. 

Paper checklist: In order to improve the rigor and transparency of research submitted to and published at NeurIPS, authors are required to complete a paper checklist . The paper checklist is intended to help authors reflect on a wide variety of issues relating to responsible machine learning research, including reproducibility, transparency, research ethics, and societal impact. The checklist forms part of the paper submission, but does not count towards the page limit.

Supplementary material: While all technical appendices should be included as part of the main paper submission PDF, authors may submit up to 100MB of supplementary material, such as data, or source code in a ZIP format. Supplementary material should be material created by the authors that directly supports the submission content. Like submissions, supplementary material must be anonymized. Looking at supplementary material is at the discretion of the reviewers.

We encourage authors to upload their code and data as part of their supplementary material in order to help reviewers assess the quality of the work. Check the policy as well as code submission guidelines and templates for further details.

Use of Large Language Models (LLMs): We welcome authors to use any tool that is suitable for preparing high-quality papers and research. However, we ask authors to keep in mind two important criteria. First, we expect papers to fully describe their methodology, and any tool that is important to that methodology, including the use of LLMs, should be described also. For example, authors should mention tools (including LLMs) that were used for data processing or filtering, visualization, facilitating or running experiments, and proving theorems. It may also be advisable to describe the use of LLMs in implementing the method (if this corresponds to an important, original, or non-standard component of the approach). Second, authors are responsible for the entire content of the paper, including all text and figures, so while authors are welcome to use any tool they wish for writing the paper, they must ensure that all text is correct and original.

Double-blind reviewing:   All submissions must be anonymized and may not contain any identifying information that may violate the double-blind reviewing policy.  This policy applies to any supplementary or linked material as well, including code.  If you are including links to any external material, it is your responsibility to guarantee anonymous browsing.  Please do not include acknowledgements at submission time. If you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”). If you need to cite one of your own papers that is in submission to NeurIPS and not available as a non-anonymous preprint, then include a copy of the cited anonymized submission in the supplementary material and write “Anonymous et al. [1] concurrently show...”). Any papers found to be violating this policy will be rejected.

OpenReview: We are using OpenReview to manage submissions. The reviews and author responses will not be public initially (but may be made public later, see below). As in previous years, submissions under review will be visible only to their assigned program committee. We will not be soliciting comments from the general public during the reviewing process. Anyone who plans to submit a paper as an author or a co-author will need to create (or update) their OpenReview profile by the full paper submission deadline. Your OpenReview profile can be edited by logging in and clicking on your name in https://openreview.net/ . This takes you to a URL "https://openreview.net/profile?id=~[Firstname]_[Lastname][n]" where the last part is your profile name, e.g., ~Wei_Zhang1. The OpenReview profiles must be up to date, with all publications by the authors, and their current affiliations. The easiest way to import publications is through DBLP but it is not required, see FAQ . Submissions without updated OpenReview profiles will be desk rejected. The information entered in the profile is critical for ensuring that conflicts of interest and reviewer matching are handled properly. Because of the rapid growth of NeurIPS, we request that all authors help with reviewing papers, if asked to do so. We need everyone’s help in maintaining the high scientific quality of NeurIPS.  

Please be aware that OpenReview has a moderation policy for newly created profiles: New profiles created without an institutional email will go through a moderation process that can take up to two weeks. New profiles created with an institutional email will be activated automatically.

Venue home page: https://openreview.net/group?id=NeurIPS.cc/2024/Conference

If you have any questions, please refer to the FAQ: https://openreview.net/faq

Ethics review: Reviewers and ACs may flag submissions for ethics review . Flagged submissions will be sent to an ethics review committee for comments. Comments from ethics reviewers will be considered by the primary reviewers and AC as part of their deliberation. They will also be visible to authors, who will have an opportunity to respond.  Ethics reviewers do not have the authority to reject papers, but in extreme cases papers may be rejected by the program chairs on ethical grounds, regardless of scientific quality or contribution.  

Preprints: The existence of non-anonymous preprints (on arXiv or other online repositories, personal websites, social media) will not result in rejection. If you choose to use the NeurIPS style for the preprint version, you must use the “preprint” option rather than the “final” option. Reviewers will be instructed not to actively look for such preprints, but encountering them will not constitute a conflict of interest. Authors may submit anonymized work to NeurIPS that is already available as a preprint (e.g., on arXiv) without citing it. Note that public versions of the submission should not say "Under review at NeurIPS" or similar.

Dual submissions: Submissions that are substantially similar to papers that the authors have previously published or submitted in parallel to other peer-reviewed venues with proceedings or journals may not be submitted to NeurIPS. Papers previously presented at workshops are permitted, so long as they did not appear in a conference proceedings (e.g., CVPRW proceedings), a journal or a book.  NeurIPS coordinates with other conferences to identify dual submissions.  The NeurIPS policy on dual submissions applies for the entire duration of the reviewing process.  Slicing contributions too thinly is discouraged.  The reviewing process will treat any other submission by an overlapping set of authors as prior work. If publishing one would render the other too incremental, both may be rejected.

Anti-collusion: NeurIPS does not tolerate any collusion whereby authors secretly cooperate with reviewers, ACs or SACs to obtain favorable reviews. 

Author responses:   Authors will have one week to view and respond to initial reviews. Author responses may not contain any identifying information that may violate the double-blind reviewing policy. Authors may not submit revisions of their paper or supplemental material, but may post their responses as a discussion in OpenReview. This is to reduce the burden on authors to have to revise their paper in a rush during the short rebuttal period.

After the initial response period, authors will be able to respond to any further reviewer/AC questions and comments by posting on the submission’s forum page. The program chairs reserve the right to solicit additional reviews after the initial author response period.  These reviews will become visible to the authors as they are added to OpenReview, and authors will have a chance to respond to them.

After the notification deadline, accepted and opted-in rejected papers will be made public and open for non-anonymous public commenting. Their anonymous reviews, meta-reviews, author responses and reviewer responses will also be made public. Authors of rejected papers will have two weeks after the notification deadline to opt in to make their deanonymized rejected papers public in OpenReview.  These papers are not counted as NeurIPS publications and will be shown as rejected in OpenReview.

Publication of accepted submissions:   Reviews, meta-reviews, and any discussion with the authors will be made public for accepted papers (but reviewer, area chair, and senior area chair identities will remain anonymous). Camera-ready papers will be due in advance of the conference. All camera-ready papers must include a funding disclosure . We strongly encourage accompanying code and data to be submitted with accepted papers when appropriate, as per the code submission policy . Authors will be allowed to make minor changes for a short period of time after the conference.

Contemporaneous Work: For the purpose of the reviewing process, papers that appeared online within two months of a submission will generally be considered "contemporaneous" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work. Authors are still expected to cite and discuss contemporaneous work and perform empirical comparisons to the degree feasible. Any paper that influenced the submission is considered prior work and must be cited and discussed as such. Submissions that are very similar to contemporaneous work will undergo additional scrutiny to prevent cases of plagiarism and missing credit to prior work.

Plagiarism is prohibited by the NeurIPS Code of Conduct .

Other Tracks: Similarly to earlier years, we will host multiple tracks, such as datasets, competitions, tutorials as well as workshops, in addition to the main track for which this call for papers is intended. See the conference homepage for updates and calls for participation in these tracks. 

Experiments: As in past years, the program chairs will be measuring the quality and effectiveness of the review process via randomized controlled experiments. All experiments are independently reviewed and approved by an Institutional Review Board (IRB).

Financial Aid: Each paper may designate up to one (1) NeurIPS.cc account email address of a corresponding student author who confirms that they would need the support to attend the conference, and agrees to volunteer if they get selected. To be considered for Financial the student will also need to fill out the Financial Aid application when it becomes available.

Science has an AI problem: Research group says they can fix it

A I holds the potential to help doctors find early markers of disease and policymakers to avoid decisions that lead to war. But a growing body of evidence has revealed deep flaws in how machine learning is used in science, a problem that has swept through dozens of fields and implicated thousands of erroneous papers.

Now an interdisciplinary team of 19 researchers, led by Princeton University computer scientists Arvind Narayanan and Sayash Kapoor, has published guidelines for the responsible use of machine learning in science.

"When we graduate from traditional statistical methods to machine learning methods, there are a vastly greater number of ways to shoot oneself in the foot," said Narayanan, director of Princeton's Center for Information Technology Policy and a professor of computer science.

"If we don't have an intervention to improve our scientific standards and reporting standards when it comes to machine learning-based science, we risk not just one discipline but many different scientific disciplines rediscovering these crises one after another."

The authors say their work is an effort to stamp out this smoldering crisis of credibility that threatens to engulf nearly every corner of the research enterprise. A paper detailing their guidelines appears May 1 in the journal Science Advances .

Because machine learning has been adopted across virtually every scientific discipline, with no universal standards safeguarding the integrity of those methods, Narayanan said the current crisis, which he calls the reproducibility crisis, could become far more serious than the replication crisis that emerged in social psychology more than a decade ago.

The good news is that a simple set of best practices can help resolve this newer crisis before it gets out of hand, according to the authors, who come from computer science, mathematics, social science and health research.

"This is a systematic problem with systematic solutions," said Kapoor, a graduate student who works with Narayanan and who organized the effort to produce the new consensus-based checklist.

The checklist focuses on ensuring the integrity of research that uses machine learning. Science depends on the ability to independently reproduce results and validate claims. Otherwise, new work cannot be reliably built atop old work, and the entire enterprise collapses.

While other researchers have developed checklists that apply to discipline-specific problems, notably in medicine, the new guidelines start with the underlying methods and apply them to any quantitative discipline.

One of the main takeaways is transparency. The checklist calls on researchers to provide detailed descriptions of each machine learning model, including the code, the data used to train and test the model, the hardware specifications used to produce the results, the experimental design, the project's goals and any limitations of the study's findings.

The standards are flexible enough to accommodate a wide range of nuance, including private datasets and complex hardware configurations, according to the authors.

While the increased rigor of these new standards might slow the publication of any given study, the authors believe wide adoption of these standards would increase the overall rate of discovery and innovation, potentially by a lot.

"What we ultimately care about is the pace of scientific progress," said sociologist Emily Cantrell, one of the lead authors, who is pursuing her Ph.D. at Princeton.

"By making sure the papers that get published are of high quality and that they're a solid base for future papers to build on, that potentially then speeds up the pace of scientific progress. Focusing on scientific progress itself and not just getting papers out the door is really where our emphasis should be."

Kapoor concurred. The errors hurt. "At the collective level, it's just a major time sink," he said. That time costs money. And that money, once wasted, could have catastrophic downstream effects, limiting the kinds of science that attract funding and investment, tanking ventures that are inadvertently built on faulty science, and discouraging countless numbers of young researchers.

In working toward a consensus about what should be included in the guidelines, the authors said they aimed to strike a balance: simple enough to be widely adopted, comprehensive enough to catch as many common mistakes as possible.

They say researchers could adopt the standards to improve their own work; peer reviewers could use the checklist to assess papers; and journals could adopt the standards as a requirement for publication.

"The scientific literature, especially in applied machine learning research, is full of avoidable errors," Narayanan said. "And we want to help people. We want to keep honest people honest."

More information: Sayash Kapoor et al, REFORMS: Consensus-based Recommendations for Machine-learning-based Science, Science Advances (2024). DOI: 10.1126/sciadv.adk3452 . www.science.org/doi/10.1126/sciadv.adk3452

Provided by Princeton University

An official website of the United States government

• Research @ BEA

Studies on the Value of Data

The U.S. Bureau of Economic Analysis has undertaken a series of studies that present methods for quantifying the value of simple data that can be differentiated from the complex data created by highly skilled workers that was studied in Calderón and Rassier 2022 . Preliminary studies in this series focus on tax data, individual credit data, and driving data. Additional examples include medical records, educational transcripts, business financial records, customer data, equipment maintenance histories, social media profiles, tourist maps, and many more. If new case studies under this topic are released, they will be added to the listing below.

• Capitalizing Data: Case Studies of Driving Records and Vehicle Insurance Claims | April 2024
• Private Funding of “Free” Data: A Theoretical Framework | April 2024
• Capitalizing Data: Case Studies of Tax Forms and Individual Credit Reports | June 2023

Rachel Soloveichik

JEL Code(s) E01 Published April 2024