methodology of research summary

• Privacy Policy

Home » Research Summary – Structure, Examples and Writing Guide

Research Summary – Structure, Examples and Writing Guide

Table of Contents

Research Summary

Definition:

A research summary is a brief and concise overview of a research project or study that highlights its key findings, main points, and conclusions. It typically includes a description of the research problem, the research methods used, the results obtained, and the implications or significance of the findings. It is often used as a tool to quickly communicate the main findings of a study to other researchers, stakeholders, or decision-makers.

Structure of Research Summary

The Structure of a Research Summary typically include:

• Introduction : This section provides a brief background of the research problem or question, explains the purpose of the study, and outlines the research objectives.
• Methodology : This section explains the research design, methods, and procedures used to conduct the study. It describes the sample size, data collection methods, and data analysis techniques.
• Results : This section presents the main findings of the study, including statistical analysis if applicable. It may include tables, charts, or graphs to visually represent the data.
• Discussion : This section interprets the results and explains their implications. It discusses the significance of the findings, compares them to previous research, and identifies any limitations or future directions for research.
• Conclusion : This section summarizes the main points of the research and provides a conclusion based on the findings. It may also suggest implications for future research or practical applications of the results.
• References : This section lists the sources cited in the research summary, following the appropriate citation style.

How to Write Research Summary

Here are the steps you can follow to write a research summary:

• Read the research article or study thoroughly: To write a summary, you must understand the research article or study you are summarizing. Therefore, read the article or study carefully to understand its purpose, research design, methodology, results, and conclusions.
• Identify the main points : Once you have read the research article or study, identify the main points, key findings, and research question. You can highlight or take notes of the essential points and findings to use as a reference when writing your summary.
• Write the introduction: Start your summary by introducing the research problem, research question, and purpose of the study. Briefly explain why the research is important and its significance.
• Summarize the methodology : In this section, summarize the research design, methods, and procedures used to conduct the study. Explain the sample size, data collection methods, and data analysis techniques.
• Present the results: Summarize the main findings of the study. Use tables, charts, or graphs to visually represent the data if necessary.
• Interpret the results: In this section, interpret the results and explain their implications. Discuss the significance of the findings, compare them to previous research, and identify any limitations or future directions for research.
• Conclude the summary : Summarize the main points of the research and provide a conclusion based on the findings. Suggest implications for future research or practical applications of the results.
• Revise and edit : Once you have written the summary, revise and edit it to ensure that it is clear, concise, and free of errors. Make sure that your summary accurately represents the research article or study.
• Add references: Include a list of references cited in the research summary, following the appropriate citation style.

Example of Research Summary

Here is an example of a research summary:

Title: The Effects of Yoga on Mental Health: A Meta-Analysis

Introduction: This meta-analysis examines the effects of yoga on mental health. The study aimed to investigate whether yoga practice can improve mental health outcomes such as anxiety, depression, stress, and quality of life.

Methodology : The study analyzed data from 14 randomized controlled trials that investigated the effects of yoga on mental health outcomes. The sample included a total of 862 participants. The yoga interventions varied in length and frequency, ranging from four to twelve weeks, with sessions lasting from 45 to 90 minutes.

Results : The meta-analysis found that yoga practice significantly improved mental health outcomes. Participants who practiced yoga showed a significant reduction in anxiety and depression symptoms, as well as stress levels. Quality of life also improved in those who practiced yoga.

Discussion : The findings of this study suggest that yoga can be an effective intervention for improving mental health outcomes. The study supports the growing body of evidence that suggests that yoga can have a positive impact on mental health. Limitations of the study include the variability of the yoga interventions, which may affect the generalizability of the findings.

Conclusion : Overall, the findings of this meta-analysis support the use of yoga as an effective intervention for improving mental health outcomes. Further research is needed to determine the optimal length and frequency of yoga interventions for different populations.

References :

• Cramer, H., Lauche, R., Langhorst, J., Dobos, G., & Berger, B. (2013). Yoga for depression: a systematic review and meta-analysis. Depression and anxiety, 30(11), 1068-1083.
• Khalsa, S. B. (2004). Yoga as a therapeutic intervention: a bibliometric analysis of published research studies. Indian journal of physiology and pharmacology, 48(3), 269-285.
• Ross, A., & Thomas, S. (2010). The health benefits of yoga and exercise: a review of comparison studies. The Journal of Alternative and Complementary Medicine, 16(1), 3-12.

Purpose of Research Summary

The purpose of a research summary is to provide a brief overview of a research project or study, including its main points, findings, and conclusions. The summary allows readers to quickly understand the essential aspects of the research without having to read the entire article or study.

Research summaries serve several purposes, including:

• Facilitating comprehension: A research summary allows readers to quickly understand the main points and findings of a research project or study without having to read the entire article or study. This makes it easier for readers to comprehend the research and its significance.
• Communicating research findings: Research summaries are often used to communicate research findings to a wider audience, such as policymakers, practitioners, or the general public. The summary presents the essential aspects of the research in a clear and concise manner, making it easier for non-experts to understand.
• Supporting decision-making: Research summaries can be used to support decision-making processes by providing a summary of the research evidence on a particular topic. This information can be used by policymakers or practitioners to make informed decisions about interventions, programs, or policies.
• Saving time: Research summaries save time for researchers, practitioners, policymakers, and other stakeholders who need to review multiple research studies. Rather than having to read the entire article or study, they can quickly review the summary to determine whether the research is relevant to their needs.

Characteristics of Research Summary

The following are some of the key characteristics of a research summary:

• Concise : A research summary should be brief and to the point, providing a clear and concise overview of the main points of the research.
• Objective : A research summary should be written in an objective tone, presenting the research findings without bias or personal opinion.
• Comprehensive : A research summary should cover all the essential aspects of the research, including the research question, methodology, results, and conclusions.
• Accurate : A research summary should accurately reflect the key findings and conclusions of the research.
• Clear and well-organized: A research summary should be easy to read and understand, with a clear structure and logical flow.
• Relevant : A research summary should focus on the most important and relevant aspects of the research, highlighting the key findings and their implications.
• Audience-specific: A research summary should be tailored to the intended audience, using language and terminology that is appropriate and accessible to the reader.
• Citations : A research summary should include citations to the original research articles or studies, allowing readers to access the full text of the research if desired.

When to write Research Summary

Here are some situations when it may be appropriate to write a research summary:

• Proposal stage: A research summary can be included in a research proposal to provide a brief overview of the research aims, objectives, methodology, and expected outcomes.
• Conference presentation: A research summary can be prepared for a conference presentation to summarize the main findings of a study or research project.
• Journal submission: Many academic journals require authors to submit a research summary along with their research article or study. The summary provides a brief overview of the study’s main points, findings, and conclusions and helps readers quickly understand the research.
• Funding application: A research summary can be included in a funding application to provide a brief summary of the research aims, objectives, and expected outcomes.
• Policy brief: A research summary can be prepared as a policy brief to communicate research findings to policymakers or stakeholders in a concise and accessible manner.

Advantages of Research Summary

Research summaries offer several advantages, including:

• Time-saving: A research summary saves time for readers who need to understand the key findings and conclusions of a research project quickly. Rather than reading the entire research article or study, readers can quickly review the summary to determine whether the research is relevant to their needs.
• Clarity and accessibility: A research summary provides a clear and accessible overview of the research project’s main points, making it easier for readers to understand the research without having to be experts in the field.
• Improved comprehension: A research summary helps readers comprehend the research by providing a brief and focused overview of the key findings and conclusions, making it easier to understand the research and its significance.
• Enhanced communication: Research summaries can be used to communicate research findings to a wider audience, such as policymakers, practitioners, or the general public, in a concise and accessible manner.
• Facilitated decision-making: Research summaries can support decision-making processes by providing a summary of the research evidence on a particular topic. Policymakers or practitioners can use this information to make informed decisions about interventions, programs, or policies.
• Increased dissemination: Research summaries can be easily shared and disseminated, allowing research findings to reach a wider audience.

Limitations of Research Summary

Limitations of the Research Summary are as follows:

• Limited scope: Research summaries provide a brief overview of the research project’s main points, findings, and conclusions, which can be limiting. They may not include all the details, nuances, and complexities of the research that readers may need to fully understand the study’s implications.
• Risk of oversimplification: Research summaries can be oversimplified, reducing the complexity of the research and potentially distorting the findings or conclusions.
• Lack of context: Research summaries may not provide sufficient context to fully understand the research findings, such as the research background, methodology, or limitations. This may lead to misunderstandings or misinterpretations of the research.
• Possible bias: Research summaries may be biased if they selectively emphasize certain findings or conclusions over others, potentially distorting the overall picture of the research.
• Format limitations: Research summaries may be constrained by the format or length requirements, making it challenging to fully convey the research’s main points, findings, and conclusions.
• Accessibility: Research summaries may not be accessible to all readers, particularly those with limited literacy skills, visual impairments, or language barriers.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

How To Write A Research Summary

It’s a common perception that writing a research summary is a quick and easy task. After all, how hard can jotting down 300 words be? But when you consider the weight those 300 words carry, writing a research summary as a part of your dissertation, essay or compelling draft for your paper instantly becomes daunting task.

A research summary requires you to synthesize a complex research paper into an informative, self-explanatory snapshot. It needs to portray what your article contains. Thus, writing it often comes at the end of the task list.

Regardless of when you’re planning to write, it is no less of a challenge, particularly if you’re doing it for the first time. This blog will take you through everything you need to know about research summary so that you have an easier time with it.

What is a Research Summary?

A research summary is the part of your research paper that describes its findings to the audience in a brief yet concise manner. A well-curated research summary represents you and your knowledge about the information written in the research paper.

While writing a quality research summary, you need to discover and identify the significant points in the research and condense it in a more straightforward form. A research summary is like a doorway that provides access to the structure of a research paper's sections.

Since the purpose of a summary is to give an overview of the topic, methodology, and conclusions employed in a paper, it requires an objective approach. No analysis or criticism.

Research summary or Abstract. What’s the Difference?

They’re both brief, concise, and give an overview of an aspect of the research paper. So, it’s easy to understand why many new researchers get the two confused. However, a research summary and abstract are two very different things with individual purpose. To start with, a research summary is written at the end while the abstract comes at the beginning of a research paper.

A research summary captures the essence of the paper at the end of your document. It focuses on your topic, methods, and findings. More like a TL;DR, if you will. An abstract, on the other hand, is a description of what your research paper is about. It tells your reader what your topic or hypothesis is, and sets a context around why you have embarked on your research.

Getting Started with a Research Summary

Before you start writing, you need to get insights into your research’s content, style, and organization. There are three fundamental areas of a research summary that you should focus on.

• While deciding the contents of your research summary, you must include a section on its importance as a whole, the techniques, and the tools that were used to formulate the conclusion. Additionally, there needs to be a short but thorough explanation of how the findings of the research paper have a significance.
• To keep the summary well-organized, try to cover the various sections of the research paper in separate paragraphs. Besides, how the idea of particular factual research came up first must be explained in a separate paragraph.
• As a general practice worldwide, research summaries are restricted to 300-400 words. However, if you have chosen a lengthy research paper, try not to exceed the word limit of 10% of the entire research paper.

How to Structure Your Research Summary

The research summary is nothing but a concise form of the entire research paper. Therefore, the structure of a summary stays the same as the paper. So, include all the section titles and write a little about them. The structural elements that a research summary must consist of are:

It represents the topic of the research. Try to phrase it so that it includes the key findings or conclusion of the task.

The abstract gives a context of the research paper. Unlike the abstract at the beginning of a paper, the abstract here, should be very short since you’ll be working with a limited word count.

Introduction

This is the most crucial section of a research summary as it helps readers get familiarized with the topic. You should include the definition of your topic, the current state of the investigation, and practical relevance in this part. Additionally, you should present the problem statement, investigative measures, and any hypothesis in this section.

Methodology

This section provides details about the methodology and the methods adopted to conduct the study. You should write a brief description of the surveys, sampling, type of experiments, statistical analysis, and the rationality behind choosing those particular methods.

Create a list of evidence obtained from the various experiments with a primary analysis, conclusions, and interpretations made upon that. In the paper research paper, you will find the results section as the most detailed and lengthy part. Therefore, you must pick up the key elements and wisely decide which elements are worth including and which are worth skipping.

This is where you present the interpretation of results in the context of their application. Discussion usually covers results, inferences, and theoretical models explaining the obtained values, key strengths, and limitations. All of these are vital elements that you must include in the summary.

Most research papers merge conclusion with discussions. However, depending upon the instructions, you may have to prepare this as a separate section in your research summary. Usually, conclusion revisits the hypothesis and provides the details about the validation or denial about the arguments made in the research paper, based upon how convincing the results were obtained.

The structure of a research summary closely resembles the anatomy of a scholarly article . Additionally, you should keep your research and references limited to authentic and  scholarly sources only.

Tips for Writing a Research Summary

The core concept behind undertaking a research summary is to present a simple and clear understanding of your research paper to the reader. The biggest hurdle while doing that is the number of words you have at your disposal. So, follow the steps below to write a research summary that sticks.

1. Read the parent paper thoroughly

You should go through the research paper thoroughly multiple times to ensure that you have a complete understanding of its contents. A 3-stage reading process helps.

a. Scan: In the first read, go through it to get an understanding of its basic concept and methodologies.

b. Read: For the second step, read the article attentively by going through each section, highlighting the key elements, and subsequently listing the topics that you will include in your research summary.

c. Skim: Flip through the article a few more times to study the interpretation of various experimental results, statistical analysis, and application in different contexts.

Sincerely go through different headings and subheadings as it will allow you to understand the underlying concept of each section. You can try reading the introduction and conclusion simultaneously to understand the motive of the task and how obtained results stay fit to the expected outcome.

2. Identify the key elements in different sections

While exploring different sections of an article, you can try finding answers to simple what, why, and how. Below are a few pointers to give you an idea:

• What is the research question and how is it addressed?
• Is there a hypothesis in the introductory part?
• What type of methods are being adopted?
• What is the sample size for data collection and how is it being analyzed?
• What are the most vital findings?
• Do the results support the hypothesis?

Discussion/Conclusion

• What is the final solution to the problem statement?
• What is the explanation for the obtained results?
• What is the drawn inference?
• What are the various limitations of the study?

3. Prepare the first draft

Now that you’ve listed the key points that the paper tries to demonstrate, you can start writing the summary following the standard structure of a research summary. Just make sure you’re not writing statements from the parent research paper verbatim.

Instead, try writing down each section in your own words. This will not only help in avoiding plagiarism but will also show your complete understanding of the subject. Alternatively, you can use a summarizing tool (AI-based summary generators) to shorten the content or summarize the content without disrupting the actual meaning of the article.

SciSpace Copilot is one such helpful feature! You can easily upload your research paper and ask Copilot to summarize it. You will get an AI-generated, condensed research summary. SciSpace Copilot also enables you to highlight text, clip math and tables, and ask any question relevant to the research paper; it will give you instant answers with deeper context of the article..

4. Include visuals

One of the best ways to summarize and consolidate a research paper is to provide visuals like graphs, charts, pie diagrams, etc.. Visuals make getting across the facts, the past trends, and the probabilistic figures around a concept much more engaging.

5. Double check for plagiarism

It can be very tempting to copy-paste a few statements or the entire paragraphs depending upon the clarity of those sections. But it’s best to stay away from the practice. Even paraphrasing should be done with utmost care and attention.

Also: QuillBot vs SciSpace: Choose the best AI-paraphrasing tool

6. Religiously follow the word count limit

You need to have strict control while writing different sections of a research summary. In many cases, it has been observed that the research summary and the parent research paper become the same length. If that happens, it can lead to discrediting of your efforts and research summary itself. Whatever the standard word limit has been imposed, you must observe that carefully.

7. Proofread your research summary multiple times

The process of writing the research summary can be exhausting and tiring. However, you shouldn’t allow this to become a reason to skip checking your academic writing several times for mistakes like misspellings, grammar, wordiness, and formatting issues. Proofread and edit until you think your research summary can stand out from the others, provided it is drafted perfectly on both technicality and comprehension parameters. You can also seek assistance from editing and proofreading services , and other free tools that help you keep these annoying grammatical errors at bay.

8. Watch while you write

Keep a keen observation of your writing style. You should use the words very precisely, and in any situation, it should not represent your personal opinions on the topic. You should write the entire research summary in utmost impersonal, precise, factually correct, and evidence-based writing.

9. Ask a friend/colleague to help

Once you are done with the final copy of your research summary, you must ask a friend or colleague to read it. You must test whether your friend or colleague could grasp everything without referring to the parent paper. This will help you in ensuring the clarity of the article.

Once you become familiar with the research paper summary concept and understand how to apply the tips discussed above in your current task, summarizing a research summary won’t be that challenging. While traversing the different stages of your academic career, you will face different scenarios where you may have to create several research summaries.

In such cases, you just need to look for answers to simple questions like “Why this study is necessary,” “what were the methods,” “who were the participants,” “what conclusions were drawn from the research,” and “how it is relevant to the wider world.” Once you find out the answers to these questions, you can easily create a good research summary following the standard structure and a precise writing style.

You might also like

Consensus GPT vs. SciSpace GPT: Choose the Best GPT for Research

Literature Review and Theoretical Framework: Understanding the Differences

Types of Essays in Academic Writing - Quick Guide (2024)

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Surveys Academic Research

Research Summary: What is it & how to write one

The Research Summary is used to report facts about a study clearly. You will almost certainly be required to prepare a research summary during your academic research or while on a research project for your organization.

If it is the first time you have to write one, the writing requirements may confuse you. The instructors generally assign someone to write a summary of the research work. Research summaries require the writer to have a thorough understanding of the issue.

This article will discuss the definition of a research summary and how to write one.

What is a research summary?

A research summary is a piece of writing that summarizes your research on a specific topic. Its primary goal is to offer the reader a detailed overview of the study with the key findings. A research summary generally contains the article’s structure in which it is written.

You must know the goal of your analysis before you launch a project. A research overview summarizes the detailed response and highlights particular issues raised in it. Writing it might be somewhat troublesome. To write a good overview, you want to start with a structure in mind. Read on for our guide.

Why is an analysis recap so important?

Your summary or analysis is going to tell readers everything about your research project. This is the critical piece that your stakeholders will read to identify your findings and valuable insights. Having a good and concise research summary that presents facts and comes with no research biases is the critical deliverable of any research project.

We’ve put together a cheat sheet to help you write a good research summary below.

Research Summary Guide

• Why was this research done?  – You want to give a clear description of why this research study was done. What hypothesis was being tested?
• Who was surveyed? – The what and why or your research decides who you’re going to interview/survey. Your research summary has a detailed note on who participated in the study and why they were selected. 
• What was the methodology? – Talk about the methodology. Did you do face-to-face interviews? Was it a short or long survey or a focus group setting? Your research methodology is key to the results you’re going to get. 
• What were the key findings? – This can be the most critical part of the process. What did we find out after testing the hypothesis? This section, like all others, should be just facts, facts facts. You’re not sharing how you feel about the findings. Keep it bias-free.
• Conclusion – What are the conclusions that were drawn from the findings. A good example of a conclusion. Surprisingly, most people interviewed did not watch the lunar eclipse in 2022, which is unexpected given that 100% of those interviewed knew about it before it happened.
• Takeaways and action points – This is where you bring in your suggestion. Given the data you now have from the research, what are the takeaways and action points? If you’re a researcher running this research project for your company, you’ll use this part to shed light on your recommended action plans for the business.

LEARN ABOUT:   Action Research

If you’re doing any research, you will write a summary, which will be the most viewed and more important part of the project. So keep a guideline in mind before you start. Focus on the content first and then worry about the length. Use the cheat sheet/checklist in this article to organize your summary, and that’s all you need to write a great research summary!

But once your summary is ready, where is it stored? Most teams have multiple documents in their google drives, and it’s a nightmare to find projects that were done in the past. Your research data should be democratized and easy to use.

We at QuestionPro launched a research repository for research teams, and our clients love it. All your data is in one place, and everything is searchable, including your research summaries! 

Authors: Prachi, Anas

MORE LIKE THIS

Top 20 Employee Engagement Software Solutions

May 3, 2024

15 Best Customer Experience Software of 2024

May 2, 2024

Journey Orchestration Platforms: Top 11 Platforms in 2024

Top 12 Employee Pulse Survey Tools Unlocking Insights in 2024

May 1, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Uncategorized
• Video Learning Series
• What’s Coming Up
• Workforce Intelligence

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 6. The Methodology
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The methods section describes actions taken to investigate a research problem and the rationale for the application of specific procedures or techniques used to identify, select, process, and analyze information applied to understanding the problem, thereby, allowing the reader to critically evaluate a study’s overall validity and reliability. The methodology section of a research paper answers two main questions: How was the data collected or generated? And, how was it analyzed? The writing should be direct and precise and always written in the past tense.

Kallet, Richard H. "How to Write the Methods Section of a Research Paper." Respiratory Care 49 (October 2004): 1229-1232.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

• Readers need to know how the data was obtained because the method you chose affects the results and, by extension, how you interpreted their significance in the discussion section of your paper.
• Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and, as a consequence, undermines the value of your analysis of the findings.
• In most cases, there are a variety of different methods you can choose to investigate a research problem. The methodology section of your paper should clearly articulate the reasons why you have chosen a particular procedure or technique.
• The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a multiple choice questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
• The method must be appropriate to fulfilling the overall aims of the study. For example, you need to ensure that you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
• The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring. For any problems that do arise, you must describe the ways in which they were minimized or why these problems do not impact in any meaningful way your interpretation of the findings.
• In the social and behavioral sciences, it is important to always provide sufficient information to allow other researchers to adopt or replicate your methodology. This information is particularly important when a new method has been developed or an innovative use of an existing method is utilized.

Bem, Daryl J. Writing the Empirical Journal Article. Psychology Writing Center. University of Washington; Denscombe, Martyn. The Good Research Guide: For Small-Scale Social Research Projects . 5th edition. Buckingham, UK: Open University Press, 2014; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I.  Groups of Research Methods

There are two main groups of research methods in the social sciences:

• The e mpirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences . This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for formulating hypotheses that need to be tested. This approach is focused on explanation.
• The i nterpretative group of methods is focused on understanding phenomenon in a comprehensive, holistic way . Interpretive methods focus on analytically disclosing the meaning-making practices of human subjects [the why, how, or by what means people do what they do], while showing how those practices arrange so that it can be used to generate observable outcomes. Interpretive methods allow you to recognize your connection to the phenomena under investigation. However, the interpretative group requires careful examination of variables because it focuses more on subjective knowledge.

II.  Content

The introduction to your methodology section should begin by restating the research problem and underlying assumptions underpinning your study. This is followed by situating the methods you used to gather, analyze, and process information within the overall “tradition” of your field of study and within the particular research design you have chosen to study the problem. If the method you choose lies outside of the tradition of your field [i.e., your review of the literature demonstrates that the method is not commonly used], provide a justification for how your choice of methods specifically addresses the research problem in ways that have not been utilized in prior studies.

The remainder of your methodology section should describe the following:

• Decisions made in selecting the data you have analyzed or, in the case of qualitative research, the subjects and research setting you have examined,
• Tools and methods used to identify and collect information, and how you identified relevant variables,
• The ways in which you processed the data and the procedures you used to analyze that data, and
• The specific research tools or strategies that you utilized to study the underlying hypothesis and research questions.

In addition, an effectively written methodology section should:

• Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
• Indicate how the approach fits the overall research design . Your methods for gathering data should have a clear connection to your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is not suitable to achieving the stated objective of your paper.
• Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom. Also be sure to explain how older data is still relevant to investigating the current research problem.
• Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors? Describe how you plan to obtain an accurate assessment of relationships, patterns, trends, distributions, and possible contradictions found in the data.
• Provide background and a rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
• Provide a justification for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of data being used? If other data sources exist, explain why the data you chose is most appropriate to addressing the research problem.
• Provide a justification for case study selection . A common method of analyzing research problems in the social sciences is to analyze specific cases. These can be a person, place, event, phenomenon, or other type of subject of analysis that are either examined as a singular topic of in-depth investigation or multiple topics of investigation studied for the purpose of comparing or contrasting findings. In either method, you should explain why a case or cases were chosen and how they specifically relate to the research problem.
• Describe potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE :   Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic. If necessary, consider using appendices for raw data.

ANOTHER NOTE : If you are conducting a qualitative analysis of a research problem , the methodology section generally requires a more elaborate description of the methods used as well as an explanation of the processes applied to gathering and analyzing of data than is generally required for studies using quantitative methods. Because you are the primary instrument for generating the data [e.g., through interviews or observations], the process for collecting that data has a significantly greater impact on producing the findings. Therefore, qualitative research requires a more detailed description of the methods used.

YET ANOTHER NOTE :   If your study involves interviews, observations, or other qualitative techniques involving human subjects , you may be required to obtain approval from the university's Office for the Protection of Research Subjects before beginning your research. This is not a common procedure for most undergraduate level student research assignments. However, i f your professor states you need approval, you must include a statement in your methods section that you received official endorsement and adequate informed consent from the office and that there was a clear assessment and minimization of risks to participants and to the university. This statement informs the reader that your study was conducted in an ethical and responsible manner. In some cases, the approval notice is included as an appendix to your paper.

III.  Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but concise. Do not provide any background information that does not directly help the reader understand why a particular method was chosen, how the data was gathered or obtained, and how the data was analyzed in relation to the research problem [note: analyzed, not interpreted! Save how you interpreted the findings for the discussion section]. With this in mind, the page length of your methods section will generally be less than any other section of your paper except the conclusion.

Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. An exception to this rule is if you select an unconventional methodological approach; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall process of discovery.

Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data, or, gaps will exist in existing data or archival materials. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose.

Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in and of itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. "How to Write a Scientific Paper: Writing the Methods Section." Revista Portuguesa de Pneumologia 17 (2011): 232-238; Blair Lorrie. “Choosing a Methodology.” In Writing a Graduate Thesis or Dissertation , Teaching Writing Series. (Rotterdam: Sense Publishers 2016), pp. 49-72; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Kallet, Richard H. “How to Write the Methods Section of a Research Paper.” Respiratory Care 49 (October 2004):1229-1232; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Rudestam, Kjell Erik and Rae R. Newton. “The Method Chapter: Describing Your Research Plan.” In Surviving Your Dissertation: A Comprehensive Guide to Content and Process . (Thousand Oaks, Sage Publications, 2015), pp. 87-115; What is Interpretive Research. Institute of Public and International Affairs, University of Utah; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University; Methods and Materials. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

To locate data and statistics, GO HERE .

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between the application of theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics . Part 1, Chapter 3. Boise State University; The Theory-Method Relationship. S-Cool Revision. United Kingdom.

Yet Another Writing Tip

Methods and the Methodology

Do not confuse the terms "methods" and "methodology." As Schneider notes, a method refers to the technical steps taken to do research . Descriptions of methods usually include defining and stating why you have chosen specific techniques to investigate a research problem, followed by an outline of the procedures you used to systematically select, gather, and process the data [remember to always save the interpretation of data for the discussion section of your paper].

The methodology refers to a discussion of the underlying reasoning why particular methods were used . This discussion includes describing the theoretical concepts that inform the choice of methods to be applied, placing the choice of methods within the more general nature of academic work, and reviewing its relevance to examining the research problem. The methodology section also includes a thorough review of the methods other scholars have used to study the topic.

Bryman, Alan. "Of Methods and Methodology." Qualitative Research in Organizations and Management: An International Journal 3 (2008): 159-168; Schneider, Florian. “What's in a Methodology: The Difference between Method, Methodology, and Theory…and How to Get the Balance Right?” PoliticsEastAsia.com. Chinese Department, University of Leiden, Netherlands.

• << Previous: Scholarly vs. Popular Publications
• Next: Qualitative Methods >>
• Last Updated: May 2, 2024 4:39 PM
• URL: https://libguides.usc.edu/writingguide

Please enable JavaScript in your browser to enjoy a better experience.

A Complete Guide to Writing a Research Summary

A summary is a key part of any research. So, how should you go about writing one?

You will find many guides on the Internet about writing research. But, any article seldom covers the prospect of writing a research summary. While many things are shortened versions of the original article, there’s much more to research summaries.

From descriptive statistics to writing scientific research, a summary plays a vital role in describing the key ideas within. So, it begs a few questions, such as:

• What exactly is a research summary?
• How do you write one?
• What are some of the tips for writing a good research summary ?

In this guide, we’ll answer all of these questions and explore a few essential factors about research writing. So, let’s jump right into it.

What is a Research Summary?

A research summary is a short, concise summary of an academic research paper. It is often used to summarize the results of an experiment, summarize the major findings and conclusions, and provide a brief overview of the methods and procedures used in the study.

The purpose of a research summary is to provide readers with enough information about an article to decide whether they want to read it in its entirety. It should be no more than two paragraphs long and should include:

• A brief introduction summarizing why the article was written
• The main idea of the article
• The major findings and conclusions
• An overview of how the study was conducted

In order to write effective research summaries, it is important that you can capture the essential points of the research and provide a concise overview. The key step in writing a good summary is to read through the article and make notes of the key points.

This can be done by underlining or highlighting key phrases in the article. One essential thing is to organize these points into an outline format, which includes an introduction and conclusion paragraph.

Another best and quick way to generate a precise summary of your research paper is to take assistance from the online text summarizer, like Summarizer.org .

The online summarizing tool gets the research paper and creates a precise summary of it by taking the important points.

Finally, you must edit your work for grammar and spelling errors before submitting it for grading.

The purpose of the research summary is to provide a comprehensive sum of everything that’s in the research. This includes a summarization of scientific/literal research, as well as of the writer’s aim and personal thoughts.

As for the summary length, it shouldn’t be more than 10% of the entire content. So, if your research is around 1000-words or so, then your summary should be 100-words. But, considering how most research papers are around 3000-4000 words, it should be 300-400 words.

Key pillars of a Research Summary

The summary of any research doesn’t just include the summarized text of the entire research paper. It includes a few other key things, which we’ll explore later on in this article. But, the purpose of a summary is to give proper insights to the reader, such as:

• The writer’s intention
• sources and bases of research
• the purpose & result.

That’s why it’s important to understand that the summary should tell your reader all these elements. So, the fundamentals of any summary include:

• Write a section and state the importance of the research paper from your perspective. In this section, you will have to describe the techniques, tools, and sources you employed to get the conclusion.
• Besides that, it’s also meant to provide a brief and descriptive explanation of the actionable aspect of your research. In other words, how it can be implemented in real life.
• Treat your research summary like a smaller article or blog. So, each important section of your research should be written within a subheading. However, this is highly optional to keep things organized.
• As mentioned before, the research summary shouldn’t exceed 300-400 words. But, some research summaries are known to surpass 10000-words. So, try to employ the 10% formula and write one-tenth of the entire length of your research paper.

These four main points allow you to understand how a research summary is different from the research itself. So, it’s like a documentary where research and other key factors are left to the science (research paper), while the narration explains the key points (research summary)

How do you write a Research Summary?

Writing a research summary is a straightforward affair. Yet, it requires some understanding, as it’s not a lengthy process but rather a tricky and technical one. In a research summary, a few boxes must be checked. To help you do just that, here are 6 things you should tend to separately:

A summary’s title can be the same as the title of your primary research. However, putting separate titles in both has a few benefits. Such as:

• A separate title shifts attention towards the conclusion.
• A different title can focus on the main point of your research.
• Using two different titles can provide a better abstract.

Speaking of an abstract, a summary is the abstract of your research. Therefore, a title representing that very thought is going to do a lot of good too. That’s why it’s better if the title of your summary differs from the title of your research paper.

2. Abstract

The abstract is the summarization of scientific or research methods used in your primary paper. This allows the reader to understand the pillars of the study conducted. For instance, there has been an array of astrological research since James Webb Space Telescope started sending images and data.

So, many research papers explain this Telescope’s technological evolution in their abstracts. This allows the reader to differentiate from the astrological research made by previous space crafts, such as Hubble or Chandra .

The point of providing this abstract is to ensure that the reader grasps the standards or boundaries within which the research was held.

3. Introduction

This is the part where you introduce your topic. In your main research, you’d dive right into the technicalities in this part. However, you’ll try to keep things mild in a research summary. Simply because it needs to summarize the key points in your main introduction.

So, a lot of introductions you’ll find as an example will be extensive in length. But, a research summary needs to be as concise as possible. Usually, in this part, a writer includes the basics and standards of investigation.

For instance, if your research is about James Webb’s latest findings , then you’ll identify how the studies conducted by this Telescope’s infrared and other technology made this study possible. That’s when your introduction will hook the reader into the main premise of your research.

4. Methodology / Study

This section needs to describe the methodology used by you in your research. Or the methodology you relied on when conducting this particular research or study. This allows the reader to grasp the fundamentals of your research, and it’s extremely important.

Because if the reader doesn’t understand your methods, then they will have no response to your studies. How should you tend to this? Include things such as:

• The surveys or reviews you used;
• include the samplings and experiment types you researched;
• provide a brief statistical analysis;
• give a primary reason to pick these particular methods.

Once again, leave the scientific intricacies for your primary research. But, describe the key methods that you employed. So, when the reader is perusing your final research, they’ll have your methods and study techniques in mind.

5. Results / Discussion

This section of your research needs to describe the results that you’ve achieved. Granted, some researchers will rely on results achieved by others. So, this part needs to explain how that happened – but not in detail.

The other section in this part will be a discussion. This is your interpretation of the results you’ve found. Thus, in the context of the results’ application, this section needs to dive into the theoretical understanding of your research. What will this section entail exactly? Here’s what:

• Things that you covered, including results;
• inferences you provided, given the context of your research;
• the theory archetype that you’ve tried to explain in the light of the methodology you employed;
• essential points or any limitations of the research.

These factors will help the reader grasp the final idea of your research. But, it’s not full circle yet, as the pulp will still be left for the actual research.

6. Conclusion

The final section of your summary is the conclusion. The key thing about the conclusion in your research summary, compared to your actual research, is that they could be different. For instance, the actual conclusion in your research should bring around the study.

However, the research in this summary should bring your own ideas and affirmations to full circle. Thus, this conclusion could and should be different from the ending of your research.

5 Tips for writing a Research Summary

Writing a research summary is easy once you tend to the technicalities. But, there are some tips and tricks that could make it easier. Remember, a research summary is the sum of your entire research. So, it doesn’t need to be as technical or in-depth as your primary work.

Thus, to make it easier for you, here are four tips you can follow:

1. Read & read again

Reading your own work repeatedly has many benefits. First, it’ll help you understand any mistakes or problems your research might have. After that, you’ll find a few key points that stand out from the others – that’s what you need to use in your summary.

So, the best advice anyone can give you is to read your research again and again. This will etch the idea in your mind and allow you to summarize it better.

2. Focus on key essentials in each section

As we discussed earlier, each section of your research has a key part. To write a thoroughly encapsulating summary, you need to focus on and find each such element in your research.

Doing so will give you enough leverage to write a summary that thoroughly condenses your research idea and gives you enough to write a summary out of it.

3. Write the research using a summarizing tool

The best advice you can get is to write a summary using a tool. Condensing each section might be a troublesome experience for some – as it can be time-consuming.

To avoid all that, you can simply take help from an online summarizer. It gets the lengthy content and creates a precise summary of it by using advanced AI technology.

As you can see, the tool condenses this particular section perfectly while the details are light.

Bringing that down to 10% or 20% will help you write each section accordingly. Thus, saving precious time and effort.

4. Word count limit

As mentioned earlier, word count is something you need to follow thoroughly. So, if your section is around 200-word, then read it again. And describe it to yourself in 20-words or so. Doing this to every section will help you write exactly a 10% summary of your research.

5. Get a second opinion

If you’re unsure about quality or quantity, get a second opinion. At times, ideas are in our minds, but we cannot find words to explain them. In research or any sort of creative process, getting a second opinion can save a lot of trouble.

There’s your guide to writing a research summary, folks. While it’s not different from condensing the entire premise of your research, writing it in simpler words will do wonders. So, try to follow the tips, tools, and ideas provided in this article, and write outstanding summaries for your research.

How To Write The Methodology Chapter

The what, why & how explained simply (with examples).

By: Jenna Crossley (PhD) | Reviewed By: Dr. Eunice Rautenbach | September 2021 (Updated April 2023)

So, you’ve pinned down your research topic and undertaken a review of the literature – now it’s time to write up the methodology section of your dissertation, thesis or research paper . But what exactly is the methodology chapter all about – and how do you go about writing one? In this post, we’ll unpack the topic, step by step .

Overview: The Methodology Chapter

• The purpose  of the methodology chapter
• Why you need to craft this chapter (really) well
• How to write and structure the chapter
• Methodology chapter example
• Essential takeaways

What (exactly) is the methodology chapter?

The methodology chapter is where you outline the philosophical underpinnings of your research and outline the specific methodological choices you’ve made. The point of the methodology chapter is to tell the reader exactly how you designed your study and, just as importantly, why you did it this way.

Importantly, this chapter should comprehensively describe and justify all the methodological choices you made in your study. For example, the approach you took to your research (i.e., qualitative, quantitative or mixed), who  you collected data from (i.e., your sampling strategy), how you collected your data and, of course, how you analysed it. If that sounds a little intimidating, don’t worry – we’ll explain all these methodological choices in this post .

Why is the methodology chapter important?

The methodology chapter plays two important roles in your dissertation or thesis:

Firstly, it demonstrates your understanding of research theory, which is what earns you marks. A flawed research design or methodology would mean flawed results. So, this chapter is vital as it allows you to show the marker that you know what you’re doing and that your results are credible .

Secondly, the methodology chapter is what helps to make your study replicable. In other words, it allows other researchers to undertake your study using the same methodological approach, and compare their findings to yours. This is very important within academic research, as each study builds on previous studies.

The methodology chapter is also important in that it allows you to identify and discuss any methodological issues or problems you encountered (i.e., research limitations ), and to explain how you mitigated the impacts of these. Every research project has its limitations , so it’s important to acknowledge these openly and highlight your study’s value despite its limitations . Doing so demonstrates your understanding of research design, which will earn you marks. We’ll discuss limitations in a bit more detail later in this post, so stay tuned!

Need a helping hand?

How to write up the methodology chapter

First off, it’s worth noting that the exact structure and contents of the methodology chapter will vary depending on the field of research (e.g., humanities, chemistry or engineering) as well as the university . So, be sure to always check the guidelines provided by your institution for clarity and, if possible, review past dissertations from your university. Here we’re going to discuss a generic structure for a methodology chapter typically found in the sciences.

Before you start writing, it’s always a good idea to draw up a rough outline to guide your writing. Don’t just start writing without knowing what you’ll discuss where. If you do, you’ll likely end up with a disjointed, ill-flowing narrative . You’ll then waste a lot of time rewriting in an attempt to try to stitch all the pieces together. Do yourself a favour and start with the end in mind .

Section 1 – Introduction

As with all chapters in your dissertation or thesis, the methodology chapter should have a brief introduction. In this section, you should remind your readers what the focus of your study is, especially the research aims . As we’ve discussed many times on the blog, your methodology needs to align with your research aims, objectives and research questions. Therefore, it’s useful to frontload this component to remind the reader (and yourself!) what you’re trying to achieve.

In this section, you can also briefly mention how you’ll structure the chapter. This will help orient the reader and provide a bit of a roadmap so that they know what to expect. You don’t need a lot of detail here – just a brief outline will do.

Section 2 – The Methodology

The next section of your chapter is where you’ll present the actual methodology. In this section, you need to detail and justify the key methodological choices you’ve made in a logical, intuitive fashion. Importantly, this is the heart of your methodology chapter, so you need to get specific – don’t hold back on the details here. This is not one of those “less is more” situations.

Let’s take a look at the most common components you’ll likely need to cover. 

Methodological Choice #1 – Research Philosophy

Research philosophy refers to the underlying beliefs (i.e., the worldview) regarding how data about a phenomenon should be gathered , analysed and used . The research philosophy will serve as the core of your study and underpin all of the other research design choices, so it’s critically important that you understand which philosophy you’ll adopt and why you made that choice. If you’re not clear on this, take the time to get clarity before you make any further methodological choices.

While several research philosophies exist, two commonly adopted ones are positivism and interpretivism . These two sit roughly on opposite sides of the research philosophy spectrum.

Positivism states that the researcher can observe reality objectively and that there is only one reality, which exists independently of the observer. As a consequence, it is quite commonly the underlying research philosophy in quantitative studies and is oftentimes the assumed philosophy in the physical sciences.

Contrasted with this, interpretivism , which is often the underlying research philosophy in qualitative studies, assumes that the researcher performs a role in observing the world around them and that reality is unique to each observer . In other words, reality is observed subjectively .

These are just two philosophies (there are many more), but they demonstrate significantly different approaches to research and have a significant impact on all the methodological choices. Therefore, it’s vital that you clearly outline and justify your research philosophy at the beginning of your methodology chapter, as it sets the scene for everything that follows.

Methodological Choice #2 – Research Type

The next thing you would typically discuss in your methodology section is the research type. The starting point for this is to indicate whether the research you conducted is inductive or deductive .

Inductive research takes a bottom-up approach , where the researcher begins with specific observations or data and then draws general conclusions or theories from those observations. Therefore these studies tend to be exploratory in terms of approach.

Conversely , d eductive research takes a top-down approach , where the researcher starts with a theory or hypothesis and then tests it using specific observations or data. Therefore these studies tend to be confirmatory in approach.

Related to this, you’ll need to indicate whether your study adopts a qualitative, quantitative or mixed  approach. As we’ve mentioned, there’s a strong link between this choice and your research philosophy, so make sure that your choices are tightly aligned . When you write this section up, remember to clearly justify your choices, as they form the foundation of your study.

Methodological Choice #3 – Research Strategy

Next, you’ll need to discuss your research strategy (also referred to as a research design ). This methodological choice refers to the broader strategy in terms of how you’ll conduct your research, based on the aims of your study.

Several research strategies exist, including experimental , case studies , ethnography , grounded theory, action research , and phenomenology . Let’s take a look at two of these, experimental and ethnographic, to see how they contrast.

Experimental research makes use of the scientific method , where one group is the control group (in which no variables are manipulated ) and another is the experimental group (in which a specific variable is manipulated). This type of research is undertaken under strict conditions in a controlled, artificial environment (e.g., a laboratory). By having firm control over the environment, experimental research typically allows the researcher to establish causation between variables. Therefore, it can be a good choice if you have research aims that involve identifying causal relationships.

Ethnographic research , on the other hand, involves observing and capturing the experiences and perceptions of participants in their natural environment (for example, at home or in the office). In other words, in an uncontrolled environment.  Naturally, this means that this research strategy would be far less suitable if your research aims involve identifying causation, but it would be very valuable if you’re looking to explore and examine a group culture, for example.

As you can see, the right research strategy will depend largely on your research aims and research questions – in other words, what you’re trying to figure out. Therefore, as with every other methodological choice, it’s essential to justify why you chose the research strategy you did.

Methodological Choice #4 – Time Horizon

The next thing you’ll need to detail in your methodology chapter is the time horizon. There are two options here: cross-sectional and longitudinal . In other words, whether the data for your study were all collected at one point in time (cross-sectional) or at multiple points in time (longitudinal).

The choice you make here depends again on your research aims, objectives and research questions. If, for example, you aim to assess how a specific group of people’s perspectives regarding a topic change over time , you’d likely adopt a longitudinal time horizon.

Another important factor to consider is simply whether you have the time necessary to adopt a longitudinal approach (which could involve collecting data over multiple months or even years). Oftentimes, the time pressures of your degree program will force your hand into adopting a cross-sectional time horizon, so keep this in mind.

Methodological Choice #5 – Sampling Strategy

Next, you’ll need to discuss your sampling strategy . There are two main categories of sampling, probability and non-probability sampling.

Probability sampling involves a random (and therefore representative) selection of participants from a population, whereas non-probability sampling entails selecting participants in a non-random  (and therefore non-representative) manner. For example, selecting participants based on ease of access (this is called a convenience sample).

The right sampling approach depends largely on what you’re trying to achieve in your study. Specifically, whether you trying to develop findings that are generalisable to a population or not. Practicalities and resource constraints also play a large role here, as it can oftentimes be challenging to gain access to a truly random sample. In the video below, we explore some of the most common sampling strategies.

Methodological Choice #6 – Data Collection Method

Next up, you’ll need to explain how you’ll go about collecting the necessary data for your study. Your data collection method (or methods) will depend on the type of data that you plan to collect – in other words, qualitative or quantitative data.

Typically, quantitative research relies on surveys , data generated by lab equipment, analytics software or existing datasets. Qualitative research, on the other hand, often makes use of collection methods such as interviews , focus groups , participant observations, and ethnography.

So, as you can see, there is a tight link between this section and the design choices you outlined in earlier sections. Strong alignment between these sections, as well as your research aims and questions is therefore very important.

Methodological Choice #7 – Data Analysis Methods/Techniques

The final major methodological choice that you need to address is that of analysis techniques . In other words, how you’ll go about analysing your date once you’ve collected it. Here it’s important to be very specific about your analysis methods and/or techniques – don’t leave any room for interpretation. Also, as with all choices in this chapter, you need to justify each choice you make.

What exactly you discuss here will depend largely on the type of study you’re conducting (i.e., qualitative, quantitative, or mixed methods). For qualitative studies, common analysis methods include content analysis , thematic analysis and discourse analysis . In the video below, we explain each of these in plain language.

For quantitative studies, you’ll almost always make use of descriptive statistics , and in many cases, you’ll also use inferential statistical techniques (e.g., correlation and regression analysis). In the video below, we unpack some of the core concepts involved in descriptive and inferential statistics.

In this section of your methodology chapter, it’s also important to discuss how you prepared your data for analysis, and what software you used (if any). For example, quantitative data will often require some initial preparation such as removing duplicates or incomplete responses . Similarly, qualitative data will often require transcription and perhaps even translation. As always, remember to state both what you did and why you did it.

Section 3 – The Methodological Limitations

With the key methodological choices outlined and justified, the next step is to discuss the limitations of your design. No research methodology is perfect – there will always be trade-offs between the “ideal” methodology and what’s practical and viable, given your constraints. Therefore, this section of your methodology chapter is where you’ll discuss the trade-offs you had to make, and why these were justified given the context.

Methodological limitations can vary greatly from study to study, ranging from common issues such as time and budget constraints to issues of sample or selection bias . For example, you may find that you didn’t manage to draw in enough respondents to achieve the desired sample size (and therefore, statistically significant results), or your sample may be skewed heavily towards a certain demographic, thereby negatively impacting representativeness .

In this section, it’s important to be critical of the shortcomings of your study. There’s no use trying to hide them (your marker will be aware of them regardless). By being critical, you’ll demonstrate to your marker that you have a strong understanding of research theory, so don’t be shy here. At the same time, don’t beat your study to death . State the limitations, why these were justified, how you mitigated their impacts to the best degree possible, and how your study still provides value despite these limitations .

Section 4 – Concluding Summary

Finally, it’s time to wrap up the methodology chapter with a brief concluding summary. In this section, you’ll want to concisely summarise what you’ve presented in the chapter. Here, it can be a good idea to use a figure to summarise the key decisions, especially if your university recommends using a specific model (for example, Saunders’ Research Onion ).

Importantly, this section needs to be brief – a paragraph or two maximum (it’s a summary, after all). Also, make sure that when you write up your concluding summary, you include only what you’ve already discussed in your chapter; don’t add any new information.

Methodology Chapter Example

In the video below, we walk you through an example of a high-quality research methodology chapter from a dissertation. We also unpack our free methodology chapter template so that you can see how best to structure your chapter.

Wrapping Up

And there you have it – the methodology chapter in a nutshell. As we’ve mentioned, the exact contents and structure of this chapter can vary between universities , so be sure to check in with your institution before you start writing. If possible, try to find dissertations or theses from former students of your specific degree program – this will give you a strong indication of the expectations and norms when it comes to the methodology chapter (and all the other chapters!).

Also, remember the golden rule of the methodology chapter – justify every choice ! Make sure that you clearly explain the “why” for every “what”, and reference credible methodology textbooks or academic sources to back up your justifications.

If you need a helping hand with your research methodology (or any other component of your research), be sure to check out our private coaching service , where we hold your hand through every step of the research journey. Until next time, good luck!

Psst... there’s more!

This post was based on one of our popular Research Bootcamps . If you're working on a research project, you'll definitely want to check this out ...

You Might Also Like:

51 Comments

highly appreciated.

This was very helpful!

This was helpful

Thanks ,it is a very useful idea.

Thanks ,it is very useful idea.

Thank you so much, this information is very useful.

Thank you very much. I must say the information presented was succinct, coherent and invaluable. It is well put together and easy to comprehend. I have a great guide to create the research methodology for my dissertation.

Highly clear and useful.

I understand a bit on the explanation above. I want to have some coach but I’m still student and don’t have any budget to hire one. A lot of question I want to ask.

Thank you so much. This concluded my day plan. Thank you so much.

Thanks it was helpful

Great information. It would be great though if you could show us practical examples.

Thanks so much for this information. God bless and be with you

Thank you so so much. Indeed it was helpful

This is EXCELLENT!

I was totally confused by other explanations. Thank you so much!.

justdoing my research now , thanks for the guidance.

Thank uuuu! These contents are really valued for me!

This is powerful …I really like it

Highly useful and clear, thank you so much.

Highly appreciated. Good guide

That was helpful. Thanks

This is very useful.Thank you

Very helpful information. Thank you

This is exactly what I was looking for. The explanation is so detailed and easy to comprehend. Well done and thank you.

Great job. You just summarised everything in the easiest and most comprehensible way possible. Thanks a lot.

Thank you very much for the ideas you have given this will really help me a lot. Thank you and God Bless.

Such great effort …….very grateful thank you

Please accept my sincere gratitude. I have to say that the information that was delivered was congruent, concise, and quite helpful. It is clear and straightforward, making it simple to understand. I am in possession of an excellent manual that will assist me in developing the research methods for my dissertation.

Thank you for your great explanation. It really helped me construct my methodology paper.

thank you for simplifieng the methodoly, It was realy helpful

Very helpful!

Thank you for your great explanation.

The explanation I have been looking for. So clear Thank you

Thank you very much .this was more enlightening.

helped me create the in depth and thorough methodology for my dissertation

Thank you for the great explaination.please construct one methodology for me

I appreciate you for the explanation of methodology. Please construct one methodology on the topic: The effects influencing students dropout among schools for my thesis

This helped me complete my methods section of my dissertation with ease. I have managed to write a thorough and concise methodology!

its so good in deed

wow …what an easy to follow presentation. very invaluable content shared. utmost important.

Peace be upon you, I am Dr. Ahmed Khedr, a former part-time professor at Al-Azhar University in Cairo, Egypt. I am currently teaching research methods, and I have been dealing with your esteemed site for several years, and I found that despite my long experience with research methods sites, it is one of the smoothest sites for evaluating the material for students, For this reason, I relied on it a lot in teaching and translated most of what was written into Arabic and published it on my own page on Facebook. Thank you all… Everything I posted on my page is provided with the names of the writers of Grad coach, the title of the article, and the site. My best regards.

A remarkably simple and useful guide, thank you kindly.

I real appriciate your short and remarkable chapter summary

Bravo! Very helpful guide.

Only true experts could provide such helpful, fantastic, and inspiring knowledge about Methodology. Thank you very much! God be with you and us all!

highly appreciate your effort.

This is a very well thought out post. Very informative and a great read.

THANKS SO MUCH FOR SHARING YOUR NICE IDEA

I love you Emma, you are simply amazing with clear explanations with complete information. GradCoach really helped me to do my assignment here in Auckland. Mostly, Emma make it so simple and enjoyable

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Get science-backed answers as you write with Paperpal's Research feature

What is Research Methodology? Definition, Types, and Examples

Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research. Several aspects must be considered before selecting an appropriate research methodology, such as research limitations and ethical concerns that may affect your research.

The research methodology section in a scientific paper describes the different methodological choices made, such as the data collection and analysis methods, and why these choices were selected. The reasons should explain why the methods chosen are the most appropriate to answer the research question. A good research methodology also helps ensure the reliability and validity of the research findings. There are three types of research methodology—quantitative, qualitative, and mixed-method, which can be chosen based on the research objectives.

What is research methodology ?

A research methodology describes the techniques and procedures used to identify and analyze information regarding a specific research topic. It is a process by which researchers design their study so that they can achieve their objectives using the selected research instruments. It includes all the important aspects of research, including research design, data collection methods, data analysis methods, and the overall framework within which the research is conducted. While these points can help you understand what is research methodology, you also need to know why it is important to pick the right methodology.

Why is research methodology important?

Having a good research methodology in place has the following advantages: 3

• Helps other researchers who may want to replicate your research; the explanations will be of benefit to them.
• You can easily answer any questions about your research if they arise at a later stage.
• A research methodology provides a framework and guidelines for researchers to clearly define research questions, hypotheses, and objectives.
• It helps researchers identify the most appropriate research design, sampling technique, and data collection and analysis methods.
• A sound research methodology helps researchers ensure that their findings are valid and reliable and free from biases and errors.
• It also helps ensure that ethical guidelines are followed while conducting research.
• A good research methodology helps researchers in planning their research efficiently, by ensuring optimum usage of their time and resources.

Writing the methods section of a research paper? Let Paperpal help you achieve perfection

Types of research methodology.

There are three types of research methodology based on the type of research and the data required. 1

• Quantitative research methodology focuses on measuring and testing numerical data. This approach is good for reaching a large number of people in a short amount of time. This type of research helps in testing the causal relationships between variables, making predictions, and generalizing results to wider populations.
• Qualitative research methodology examines the opinions, behaviors, and experiences of people. It collects and analyzes words and textual data. This research methodology requires fewer participants but is still more time consuming because the time spent per participant is quite large. This method is used in exploratory research where the research problem being investigated is not clearly defined.
• Mixed-method research methodology uses the characteristics of both quantitative and qualitative research methodologies in the same study. This method allows researchers to validate their findings, verify if the results observed using both methods are complementary, and explain any unexpected results obtained from one method by using the other method.

What are the types of sampling designs in research methodology?

Sampling 4 is an important part of a research methodology and involves selecting a representative sample of the population to conduct the study, making statistical inferences about them, and estimating the characteristics of the whole population based on these inferences. There are two types of sampling designs in research methodology—probability and nonprobability.

• Probability sampling

In this type of sampling design, a sample is chosen from a larger population using some form of random selection, that is, every member of the population has an equal chance of being selected. The different types of probability sampling are:

• Systematic —sample members are chosen at regular intervals. It requires selecting a starting point for the sample and sample size determination that can be repeated at regular intervals. This type of sampling method has a predefined range; hence, it is the least time consuming.
• Stratified —researchers divide the population into smaller groups that don’t overlap but represent the entire population. While sampling, these groups can be organized, and then a sample can be drawn from each group separately.
• Cluster —the population is divided into clusters based on demographic parameters like age, sex, location, etc.
• Convenience —selects participants who are most easily accessible to researchers due to geographical proximity, availability at a particular time, etc.
• Purposive —participants are selected at the researcher’s discretion. Researchers consider the purpose of the study and the understanding of the target audience.
• Snowball —already selected participants use their social networks to refer the researcher to other potential participants.
• Quota —while designing the study, the researchers decide how many people with which characteristics to include as participants. The characteristics help in choosing people most likely to provide insights into the subject.

What are data collection methods?

During research, data are collected using various methods depending on the research methodology being followed and the research methods being undertaken. Both qualitative and quantitative research have different data collection methods, as listed below.

Qualitative research 5

• One-on-one interviews: Helps the interviewers understand a respondent’s subjective opinion and experience pertaining to a specific topic or event
• Document study/literature review/record keeping: Researchers’ review of already existing written materials such as archives, annual reports, research articles, guidelines, policy documents, etc.
• Focus groups: Constructive discussions that usually include a small sample of about 6-10 people and a moderator, to understand the participants’ opinion on a given topic.
• Qualitative observation : Researchers collect data using their five senses (sight, smell, touch, taste, and hearing).

Quantitative research 6

• Sampling: The most common type is probability sampling.
• Interviews: Commonly telephonic or done in-person.
• Observations: Structured observations are most commonly used in quantitative research. In this method, researchers make observations about specific behaviors of individuals in a structured setting.
• Document review: Reviewing existing research or documents to collect evidence for supporting the research.
• Surveys and questionnaires. Surveys can be administered both online and offline depending on the requirement and sample size.

Let Paperpal help you write the perfect research methods section. Start now!

What are data analysis methods.

The data collected using the various methods for qualitative and quantitative research need to be analyzed to generate meaningful conclusions. These data analysis methods 7 also differ between quantitative and qualitative research.

Quantitative research involves a deductive method for data analysis where hypotheses are developed at the beginning of the research and precise measurement is required. The methods include statistical analysis applications to analyze numerical data and are grouped into two categories—descriptive and inferential.

Descriptive analysis is used to describe the basic features of different types of data to present it in a way that ensures the patterns become meaningful. The different types of descriptive analysis methods are:

• Measures of frequency (count, percent, frequency)
• Measures of central tendency (mean, median, mode)
• Measures of dispersion or variation (range, variance, standard deviation)
• Measure of position (percentile ranks, quartile ranks)

Inferential analysis is used to make predictions about a larger population based on the analysis of the data collected from a smaller population. This analysis is used to study the relationships between different variables. Some commonly used inferential data analysis methods are:

• Correlation: To understand the relationship between two or more variables.
• Cross-tabulation: Analyze the relationship between multiple variables.
• Regression analysis: Study the impact of independent variables on the dependent variable.
• Frequency tables: To understand the frequency of data.
• Analysis of variance: To test the degree to which two or more variables differ in an experiment.

Qualitative research involves an inductive method for data analysis where hypotheses are developed after data collection. The methods include:

• Content analysis: For analyzing documented information from text and images by determining the presence of certain words or concepts in texts.
• Narrative analysis: For analyzing content obtained from sources such as interviews, field observations, and surveys. The stories and opinions shared by people are used to answer research questions.
• Discourse analysis: For analyzing interactions with people considering the social context, that is, the lifestyle and environment, under which the interaction occurs.
• Grounded theory: Involves hypothesis creation by data collection and analysis to explain why a phenomenon occurred.
• Thematic analysis: To identify important themes or patterns in data and use these to address an issue.

How to choose a research methodology?

Here are some important factors to consider when choosing a research methodology: 8

• Research objectives, aims, and questions —these would help structure the research design.
• Review existing literature to identify any gaps in knowledge.
• Check the statistical requirements —if data-driven or statistical results are needed then quantitative research is the best. If the research questions can be answered based on people’s opinions and perceptions, then qualitative research is most suitable.
• Sample size —sample size can often determine the feasibility of a research methodology. For a large sample, less effort- and time-intensive methods are appropriate.
• Constraints —constraints of time, geography, and resources can help define the appropriate methodology.

Got writer’s block? Kickstart your research paper writing with Paperpal now!

How to write a research methodology .

A research methodology should include the following components: 3,9

• Research design —should be selected based on the research question and the data required. Common research designs include experimental, quasi-experimental, correlational, descriptive, and exploratory.
• Research method —this can be quantitative, qualitative, or mixed-method.
• Reason for selecting a specific methodology —explain why this methodology is the most suitable to answer your research problem.
• Research instruments —explain the research instruments you plan to use, mainly referring to the data collection methods such as interviews, surveys, etc. Here as well, a reason should be mentioned for selecting the particular instrument.
• Sampling —this involves selecting a representative subset of the population being studied.
• Data collection —involves gathering data using several data collection methods, such as surveys, interviews, etc.
• Data analysis —describe the data analysis methods you will use once you’ve collected the data.
• Research limitations —mention any limitations you foresee while conducting your research.
• Validity and reliability —validity helps identify the accuracy and truthfulness of the findings; reliability refers to the consistency and stability of the results over time and across different conditions.
• Ethical considerations —research should be conducted ethically. The considerations include obtaining consent from participants, maintaining confidentiality, and addressing conflicts of interest.

Streamline Your Research Paper Writing Process with Paperpal

The methods section is a critical part of the research papers, allowing researchers to use this to understand your findings and replicate your work when pursuing their own research. However, it is usually also the most difficult section to write. This is where Paperpal can help you overcome the writer’s block and create the first draft in minutes with Paperpal Copilot, its secure generative AI feature suite.  

With Paperpal you can get research advice, write and refine your work, rephrase and verify the writing, and ensure submission readiness, all in one place. Here’s how you can use Paperpal to develop the first draft of your methods section.  

• Generate an outline: Input some details about your research to instantly generate an outline for your methods section 
• Develop the section: Use the outline and suggested sentence templates to expand your ideas and develop the first draft.  
• P araph ras e and trim : Get clear, concise academic text with paraphrasing that conveys your work effectively and word reduction to fix redundancies. 
• Choose the right words: Enhance text by choosing contextual synonyms based on how the words have been used in previously published work.  
• Check and verify text : Make sure the generated text showcases your methods correctly, has all the right citations, and is original and authentic. .   

You can repeat this process to develop each section of your research manuscript, including the title, abstract and keywords. Ready to write your research papers faster, better, and without the stress? Sign up for Paperpal and start writing today!

Frequently Asked Questions

Q1. What are the key components of research methodology?

A1. A good research methodology has the following key components:

• Research design
• Data collection procedures
• Data analysis methods
• Ethical considerations

Q2. Why is ethical consideration important in research methodology?

A2. Ethical consideration is important in research methodology to ensure the readers of the reliability and validity of the study. Researchers must clearly mention the ethical norms and standards followed during the conduct of the research and also mention if the research has been cleared by any institutional board. The following 10 points are the important principles related to ethical considerations: 10

• Participants should not be subjected to harm.
• Respect for the dignity of participants should be prioritized.
• Full consent should be obtained from participants before the study.
• Participants’ privacy should be ensured.
• Confidentiality of the research data should be ensured.
• Anonymity of individuals and organizations participating in the research should be maintained.
• The aims and objectives of the research should not be exaggerated.
• Affiliations, sources of funding, and any possible conflicts of interest should be declared.
• Communication in relation to the research should be honest and transparent.
• Misleading information and biased representation of primary data findings should be avoided.

Q3. What is the difference between methodology and method?

A3. Research methodology is different from a research method, although both terms are often confused. Research methods are the tools used to gather data, while the research methodology provides a framework for how research is planned, conducted, and analyzed. The latter guides researchers in making decisions about the most appropriate methods for their research. Research methods refer to the specific techniques, procedures, and tools used by researchers to collect, analyze, and interpret data, for instance surveys, questionnaires, interviews, etc.

Research methodology is, thus, an integral part of a research study. It helps ensure that you stay on track to meet your research objectives and answer your research questions using the most appropriate data collection and analysis tools based on your research design.

Accelerate your research paper writing with Paperpal. Try for free now!

• Research methodologies. Pfeiffer Library website. Accessed August 15, 2023. https://library.tiffin.edu/researchmethodologies/whatareresearchmethodologies
• Types of research methodology. Eduvoice website. Accessed August 16, 2023. https://eduvoice.in/types-research-methodology/
• The basics of research methodology: A key to quality research. Voxco. Accessed August 16, 2023. https://www.voxco.com/blog/what-is-research-methodology/
• Sampling methods: Types with examples. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/types-of-sampling-for-social-research/
• What is qualitative research? Methods, types, approaches, examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-qualitative-research-methods-types-examples/
• What is quantitative research? Definition, methods, types, and examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-quantitative-research-types-and-examples/
• Data analysis in research: Types & methods. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/data-analysis-in-research/#Data_analysis_in_qualitative_research
• Factors to consider while choosing the right research methodology. PhD Monster website. Accessed August 17, 2023. https://www.phdmonster.com/factors-to-consider-while-choosing-the-right-research-methodology/
• What is research methodology? Research and writing guides. Accessed August 14, 2023. https://paperpile.com/g/what-is-research-methodology/
• Ethical considerations. Business research methodology website. Accessed August 17, 2023. https://research-methodology.net/research-methodology/ethical-considerations/

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Dangling Modifiers and How to Avoid Them in Your Writing 
• Webinar: How to Use Generative AI Tools Ethically in Your Academic Writing
• Research Outlines: How to Write An Introduction Section in Minutes with Paperpal Copilot
• How to Paraphrase Research Papers Effectively

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

38k Accesses

52 Citations

58 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
• EDIT Edit this Article
• EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
• Browse Articles
• Learn Something New
• Quizzes Hot
• This Or That Game
• Train Your Brain
• Explore More
• Support wikiHow
• About wikiHow
• Log in / Sign up
• Education and Communications
• College University and Postgraduate
• Academic Writing

How to Write Research Methodology

Last Updated: May 21, 2023 Approved

This article was co-authored by Alexander Ruiz, M.Ed. and by wikiHow staff writer, Jennifer Mueller, JD . Alexander Ruiz is an Educational Consultant and the Educational Director of Link Educational Institute, a tutoring business based in Claremont, California that provides customizable educational plans, subject and test prep tutoring, and college application consulting. With over a decade and a half of experience in the education industry, Alexander coaches students to increase their self-awareness and emotional intelligence while achieving skills and the goal of achieving skills and higher education. He holds a BA in Psychology from Florida International University and an MA in Education from Georgia Southern University. wikiHow marks an article as reader-approved once it receives enough positive feedback. In this case, several readers have written to tell us that this article was helpful to them, earning it our reader-approved status. This article has been viewed 519,881 times.

The research methodology section of any academic research paper gives you the opportunity to convince your readers that your research is useful and will contribute to your field of study. An effective research methodology is grounded in your overall approach – whether qualitative or quantitative – and adequately describes the methods you used. Justify why you chose those methods over others, then explain how those methods will provide answers to your research questions. [1] X Research source

Describing Your Methods

• In your restatement, include any underlying assumptions that you're making or conditions that you're taking for granted. These assumptions will also inform the research methods you've chosen.
• Generally, state the variables you'll test and the other conditions you're controlling or assuming are equal.

• If you want to research and document measurable social trends, or evaluate the impact of a particular policy on various variables, use a quantitative approach focused on data collection and statistical analysis.
• If you want to evaluate people's views or understanding of a particular issue, choose a more qualitative approach.
• You can also combine the two. For example, you might look primarily at a measurable social trend, but also interview people and get their opinions on how that trend is affecting their lives.

• For example, if you conducted a survey, you would describe the questions included in the survey, where and how the survey was conducted (such as in person, online, over the phone), how many surveys were distributed, and how long your respondents had to complete the survey.
• Include enough detail that your study can be replicated by others in your field, even if they may not get the same results you did. [4] X Research source

• Qualitative research methods typically require more detailed explanation than quantitative methods.
• Basic investigative procedures don't need to be explained in detail. Generally, you can assume that your readers have a general understanding of common research methods that social scientists use, such as surveys or focus groups.

• For example, suppose you conducted a survey and used a couple of other research papers to help construct the questions on your survey. You would mention those as contributing sources.

Justifying Your Choice of Methods

• Describe study participants specifically, and list any inclusion or exclusion criteria you used when forming your group of participants.
• Justify the size of your sample, if applicable, and describe how this affects whether your study can be generalized to larger populations. For example, if you conducted a survey of 30 percent of the student population of a university, you could potentially apply those results to the student body as a whole, but maybe not to students at other universities.

• Reading other research papers is a good way to identify potential problems that commonly arise with various methods. State whether you actually encountered any of these common problems during your research.

• If you encountered any problems as you collected data, explain clearly the steps you took to minimize the effect that problem would have on your results.

• In some cases, this may be as simple as stating that while there were numerous studies using one method, there weren't any using your method, which caused a gap in understanding of the issue.
• For example, there may be multiple papers providing quantitative analysis of a particular social trend. However, none of these papers looked closely at how this trend was affecting the lives of people.

Connecting Your Methods to Your Research Goals

• Depending on your research questions, you may be mixing quantitative and qualitative analysis – just as you could potentially use both approaches. For example, you might do a statistical analysis, and then interpret those statistics through a particular theoretical lens.

• For example, suppose you're researching the effect of college education on family farms in rural America. While you could do interviews of college-educated people who grew up on a family farm, that would not give you a picture of the overall effect. A quantitative approach and statistical analysis would give you a bigger picture.

• If in answering your research questions, your findings have raised other questions that may require further research, state these briefly.
• You can also include here any limitations to your methods, or questions that weren't answered through your research.

• Generalization is more typically used in quantitative research. If you have a well-designed sample, you can statistically apply your results to the larger population your sample belongs to.

Template to Write Research Methodology

Community Q&A

• Organize your methodology section chronologically, starting with how you prepared to conduct your research methods, how you gathered data, and how you analyzed that data. [13] X Research source Thanks Helpful 0 Not Helpful 0
• Write your research methodology section in past tense, unless you're submitting the methodology section before the research described has been carried out. [14] X Research source Thanks Helpful 2 Not Helpful 0
• Discuss your plans in detail with your advisor or supervisor before committing to a particular methodology. They can help identify possible flaws in your study. [15] X Research source Thanks Helpful 0 Not Helpful 0

You Might Also Like

• ↑ http://expertjournals.com/how-to-write-a-research-methodology-for-your-academic-article/
• ↑ http://libguides.usc.edu/writingguide/methodology
• ↑ https://www.skillsyouneed.com/learn/dissertation-methodology.html
• ↑ https://uir.unisa.ac.za/bitstream/handle/10500/4245/05Chap%204_Research%20methodology%20and%20design.pdf
• ↑ https://elc.polyu.edu.hk/FYP/html/method.htm

About This Article

To write a research methodology, start with a section that outlines the problems or questions you'll be studying, including your hypotheses or whatever it is you're setting out to prove. Then, briefly explain why you chose to use either a qualitative or quantitative approach for your study. Next, go over when and where you conducted your research and what parameters you used to ensure you were objective. Finally, cite any sources you used to decide on the methodology for your research. To learn how to justify your choice of methods in your research methodology, scroll down! Did this summary help you? Yes No

• Send fan mail to authors

Reader Success Stories

Prof. Dr. Ahmed Askar

Apr 18, 2020

Did this article help you?

M. Mahmood Shah Khan

Mar 17, 2020

Shimola Makondo

Jul 20, 2019

Zain Sharif Mohammed Alnadhery

Jan 7, 2019

Lundi Dukashe

Feb 17, 2020

Featured Articles

Trending Articles

Watch Articles

• Terms of Use
• Privacy Policy
• Do Not Sell or Share My Info
• Not Selling Info

Don’t miss out! Sign up for

wikiHow’s newsletter

About GreatSchools.org’s ratings

As an independent nonprofit, our mission at GreatSchools.org is to help all parents get a great education for their children and for communities to ensure that all students get the support they need to succeed. The definition of “success” is as diverse as the millions of parents who visit our site each year, and we strive to display a variety of indicators of school quality to provide a well-rounded picture of how effectively each school serves its students. We understand that the available information doesn’t always reflect parents’ priorities. That’s why we are constantly working to acquire a broader range of information in order to paint a richer picture of school quality and what it means to provide equitable opportunity for all students.

In 2017, we launched our Summary Rating, which is comprised of themed ratings that each reflect important factors in how students experience school, including how well schools serve students from different racial, ethnic, and socioeconomic backgrounds, how much students are improving within a school year, performance on state tests, and how well schools prepare students for college.

In August 2020, we piloted new methodologies in California and Michigan for both the Summary Rating and the Equity Rating to provide a clearer lens to see how schools serve all children. This new methodology was available nationwide as of September, 2020.

Quick links

Summary Rating Other ratings & flags Inputs & weights Methodology Data transparency Support & FAQs

Our approach to ratings

We believe that every parent — regardless of their background — needs reliable information in order to understand whether their child is being served by their school. We recognize that how well a school serves students from historically marginalized populations goes beyond test scores. A mounting body of evidence suggests that in comparison to just end-of-year test scores, student progress — or growth — is a more accurate way to measure how much value a school is adding for its students. (Student progress, or growth, measures how much each child learns in a given year, based on the change in their state test scores from one year to the next.) This information, when used in combination with other information, such as college readiness measures and test score data, can give parents a better understanding of a school’s quality.

With this in mind, we have added student progress and college readiness metrics to our Equity Rating, and we have updated our overall Summary Rating methodology to emphasize student growth relative to test scores. These changes will help uncover the strengths of schools successfully serving Black, Latinx, Native American and low-income students. We believe this is an important step toward providing richer, multifaceted information that is not only based on research, but also reflective of what parents find meaningful.

How do our ratings work?

Summary rating.

The GreatSchools Summary Rating appears at the top of a school’s profile. Our ratings follow a 1-10 scale, where 10 is the highest and 1 is the lowest. Ratings at the lower end of the scale (1-4) signal that the school is “below average,” 5-6 indicate “average,” and 7-10 are “above average.”

The Summary Rating calculation is based on four ratings, each of which is designed to show different facets of school success: the Student Progress Rating or Academic Progress Rating, College Readiness Rating, Equity Rating, and Test Score Rating. The ratings for each school vary based on data availability or relevance to a school level (for example, high schools have a College Readiness Rating, but elementary schools do not). We do not produce Summary Ratings for schools if we lack sufficient data. For more about how this rating is calculated, see the Summary Rating inputs and weights section below.

Note: Some states do not have sufficient information to generate a Summary Rating. In these states, we default to the school’s Test Score Rating as the overall rating displayed at the top of the profile.

Student Progress Rating

The Student Progress Rating (also known as “growth”) measures whether students at a school are making academic progress over time. Specifically, the Student Progress Rating measures how much progress individual students have made on state standardized assessments during the past year or more and how this performance aligns with expected progress for that student. Each state department of education establishes a student growth model. The Student Progress Rating shows how this school’s average growth data compares to growth data of other schools in the state. The goal of the Student Progress Rating is to provide transparency into schools that are improving student outcomes regardless of the student’s starting point in terms of academic achievement.

Student Progress data is less common for high schools, because in many states high school students only take state standardized tests in one year. It is also important to note that it is possible for schools with already-high-performing students to receive a high Student Progress Rating, or for schools with high test scores to receive a low Student Progress Rating. Research indicates that growth metrics are a more accurate measure of the value that a school adds for its students because they are less correlated with the socioeconomic background of the student population than test score performance. In the Summary Rating, we give more weight to the Student Progress Rating than the Test Score Rating. See the Summary Rating inputs & weights section below for more information about how this rating is calculated.

For more information about how we calculate this rating, see the GreatSchools Ratings methodology report.

Academic Progress Rating

For states that do not provide publicly available growth data which meets our requirements for a growth rating, we calculate an Academic Progress Rating , a proxy rating based on a model using school-level data instead of student-level data. This data is less common for high schools, since in many states high school students only take state standardized tests in one year, making it difficult to look at grade-to-grade improvement. When usable student growth data become publicly available in these states, we will replace the Academic Progress Rating with a Student Progress Rating. In response to low testing rates in the 2019-20 and 2020-21 school years, we temporarily adjusted our Academic Progress Rating Methodology based on recommendations from education research groups and policymakers.

College Readiness Rating

The College Readiness Rating is designed to measure how well high schools prepare their students for success in college and career, compared to other schools in the state. The rating is based on the high school’s graduation rate, data about college entrance exams (SAT/ACT performance) and/or advanced course information, including enrollment in International Baccalaureate courses, dual-enrollment courses, and Advanced Placement (AP) courses and AP exam passing rates. This rating only applies to high schools.

Equity Rating

The Equity Rating is designed to measure how well a school serves the academic development of disadvantaged student groups. The rating includes data on 1) how disadvantaged students are performing based on student progress (growth) or academic progress (growth proxy), college readiness (e.g., college entrance exam performance and high school graduation rates), and state proficiency tests, in comparison to the outcomes for all students in the state, and 2) performance gaps between disadvantaged students and non-disadvantaged students in a school. We define disadvantaged students as students from racial, ethnic, and/or socioeconomic groups that show persistent gaps across all schools and outcomes within the state. This allows us to understand how well the school is educating disadvantaged student groups compared to all students, and how these students are performing compared to non-disadvantaged students in each school. A low rating (1–4) may indicate some student groups are not getting the support they need at the school, while a high rating (7–10) may indicate that a school’s disadvantaged groups are doing well.

When the population of a student group is too small to provide reliable data (less than 5% of the total student body), that student group is omitted from the calculation. As a result, some groups may not be included in some charts within the equity sections of the school’s profile.

Some schools do not have a large enough population of disadvantaged students to calculate an Equity Rating (homogeneous schools). These schools do not receive an Equity Rating. Their Summary Rating is therefore composed of the other available data.

Test Score Rating

The Test Score Rating measures schools on academic proficiency, using performance on state tests (the percentage of students scoring at or above proficiency) across grades and subjects, compared to other schools in the state, to produce a 1-10 rating for each school. This school’s overall Test Score Ratings are displayed in the Academics section on school profiles and are broken out by student group (race/ethnicity and family income) in the Equity section. This rating is an important factor in understanding school quality because it helps reflect how well students are meeting academic proficiency standards set by each state.

Note: we also display discipline and attendance flags to identify schools with worrisome patterns of out-of-school suspensions and chronic student absenteeism. Creating these flags involves two primary steps: identifying schools with high rates of suspension or absenteeism, and identifying schools with significant differences in suspension or chronic absenteeism rates between students from different race/ethnicity groups. If a school has a flag, it appears in a school’s “Equity” section, under “Race/ethnicity” then toggle to view the “Discipline & attendance” information. These flags do not influence any of the themed ratings or the Summary Rating.

Summary Rating inputs and weights

The Summary Rating calculation is based on up to four of the school’s themed ratings (described above), based on available data. For example, college readiness measures like Advanced Placement classes and college entrance exams are available in most high schools but not elementary or middle schools.

To calculate the Summary Rating, we use weights for each themed rating based on the available data, the amount of information available about the school relative to other schools in the state, the amount of variability in the data, and the extent to which each data point has been proven to be related to student success in college and for long-term life outcomes. The maximum weight for each themed rating is capped to be no more than the Student Progress/Academic Progress Rating weight, regardless of its data availability and quality.

Below are representative examples of how a high school or elementary school’s ratings are weighted within the overall Summary Rating calculation for the school:

Inputs to the Summary Rating are school- and state-specific, depending on data availability. Each of the themed ratings that comprise the Summary Rating may be refreshed as new data becomes available, which in turn may cause the school’s Summary Rating to change. These changes may happen at different times throughout the course of a year. To see when underlying data was updated, click on the Sources information for each rating and flag. Note: rounding of percentages may cause some Summary Rating weights to exceed 100%.

Greater data transparency

At GreatSchools.org, we believe that transparency builds trust. We believe that government education agencies have an obligation to make data on school quality available to parents and the public. Every parent should feel informed and empowered to unlock educational opportunities for their child regardless of their family background. That’s why in recent years, GreatSchools.org has expanded data collection efforts in every state to include various types of school quality data broken down by student groups, including students from low-income families, diverse racial and ethnic backgrounds, and students with disabilities. These additional data (where available) are now part of GreatSchools.org school profiles and provide a more complete picture of how effectively a school serves all of its students.

Data transparency helps parents know how schools in their community are doing, where there is room for improvement, and what the best options are for their children. Sharing school information — good and bad — also cultivates parent engagement and trust. Additionally, it’s important that school data be made available in accessible, easy-to-use formats so that non-governmental organizations can use the information to inform parents and students about the quality of their local schools.

Our ongoing commitment to offering a broader view of school quality

We are committed to an ongoing evolution of how we can paint a broader picture of school quality that better captures the factors that matter most to parents and that research shows can make a difference in student success. We think the changes we are making to our methodology are a step forward in our ongoing mission to illuminate issues around equity in education. Going forward, we will continue to work in collaboration with research partners and to pursue an expanded definition of school quality that includes information about the resources, opportunities, and practices used in schools, and meaningful outcomes for all students. We will continue to advocate for the quality and types of data needed to help parents understand how schools are doing in their state. Where data is not available, we’ll continue to highlight the need for, and value of, transparency, as we have done for years.

Support and frequently asked questions

• Was this information helpful in better understanding our ratings? Share your feedback.
• For additional help, please visit our FAQ page.
• For more information on GreatSchools’ commitment to working with government education agencies and community partners to provide better information on student outcomes, please contact us.

About GreatSchools

• Vision and values
• School information
• Parenting guidance
• Research reports
• Parent insights
• For schools
• Family engagement
• Supporters and partners
• License our data
• Policies & financials

Yes! Sign me up for updates relevant to my child's grade.

Please enter a valid email address

Thank you for signing up!

Server Issue: Please try again later. Sorry for the inconvenience

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

U.S. Surveys

Pew Research Center has deep roots in U.S. public opinion research.  Launched initially  as a project focused primarily on U.S. policy and politics in the early 1990s, the Center has grown over time to study a wide range of topics vital to explaining America to itself and to the world. Our hallmarks: a rigorous approach to methodological quality, complete transparency as to our methods, and a commitment to exploring and evaluating ongoing developments in data collection. Learn more about how we conduct our domestic surveys  here .

The American Trends Panel

Try our email course on polling

Want to know more about polling? Take your knowledge to the next level with a short email mini-course from Pew Research Center. Sign up now .

From the 1980s until relatively recently, most national polling organizations conducted surveys by telephone, relying on live interviewers to call randomly selected Americans across the country. Then came the internet. While it took survey researchers some time to adapt to the idea of online surveys, a quick look at the public polls on an issue like presidential approval reveals a landscape now dominated by online polls rather than phone polls.

Most of our U.S. surveys are conducted on the American Trends Panel (ATP), Pew Research Center’s national survey panel of over 10,000 randomly selected U.S. adults. ATP participants are recruited offline using random sampling from the U.S. Postal Service’s residential address file. Survey length is capped at 15 minutes, and respondents are reimbursed for their time. Respondents complete the surveys online using smartphones, tablets or desktop devices. We provide tablets and data plans to adults without home internet. Learn more  about how people in the U.S. take Pew Research Center surveys.

Methods 101

Our video series helps explain the fundamental concepts of survey research including random sampling , question wording , mode effects , non probability surveys and how polling is done around. the world.

The Center also conducts custom surveys of special populations (e.g., Muslim Americans , Jewish Americans , Black Americans , Hispanic Americans , teenagers ) that are not readily studied using national, general population sampling. The Center’s survey research is sometimes paired with demographic or organic data to provide new insights. In addition to our U.S. survey research, you can also read more details on our  international survey research , our demographic research and our data science methods.

Our survey researchers are committed to contributing to the larger community of survey research professionals, and are active in AAPOR and is a charter member of the American Association of Public Opinion Research (AAPOR)  Transparency Initiative .

Frequently asked questions about surveys

• Why am I never asked to take a poll?
• Can I volunteer to be polled?
• Why should I participate in surveys?
• What good are polls?
• Do pollsters have a code of ethics? If so, what is in the code?
• How are your surveys different from market research?
• Do you survey Asian Americans?
• How are people selected for your polls?
• Do people lie to pollsters?
• Do people really have opinions on all of those questions?
• How can I tell a high-quality poll from a lower-quality one?

Reports on the state of polling

• Key Things to Know about Election Polling in the United States
• A Field Guide to Polling: 2020 Edition
• Confronting 2016 and 2020 Polling Limitations
• What 2020’s Election Poll Errors Tell Us About the Accuracy of Issue Polling
• Q&A: After misses in 2016 and 2020, does polling need to be fixed again? What our survey experts say
• Understanding how 2020 election polls performed and what it might mean for other kinds of survey work
• Can We Still Trust Polls?
• Political Polls and the 2016 Election
• Flashpoints in Polling: 2016

Sign up for our Methods newsletter

The latest on survey methods, data science and more, delivered quarterly.

OTHER RESEARCH METHODS

Sign up for our weekly newsletter.

Fresh data delivered Saturday mornings

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy

Digital media in informal learning activities

• Published: 03 May 2024

Cite this article

• Gulsara Tazhenova 1 ,
• Natalia Mikhaylova 2 &
• Botagul Turgunbayeva 1  

This study aims to investigate digital media in the context of informal educational activities to determine their impact on learning effectiveness and the level of participant engagement. It was assumed that the integration of digital media into informal education positively influences the effectiveness of learning, the level of student engagement, and their motivation toward educational activities. An experimental approach was employed in this research to assess the influence of digital media on participant engagement in informal educational activities, utilizing survey methods. The study was conducted with the participation of representatives from the Ural Federal University named after the first President of the Russian Federation, B. N. Yeltsin, and Al-Farabi Kazakh National University. The study compared the effectiveness of informal educational activities with and without the use of media technologies in terms of participant engagement. In summary, the use of digital media in informal educational activities has a positive impact on the behavioural and emotional completeness of participants, while cognitive completeness did not show statistically significant differences between the groups. The results of this study have practical significance for educational institutions and educators involved in organizing informal educational activities. It provides insights into how the integration of digital media into the learning process can influence participant engagement. It’s important to remember that the research sample had a specific geographic coverage, so generalizing the obtained results should be exercised with prudence. This study has significant practical implications for educators and institutions involved in organizing non-formal educational activities. Educators can use the acquired information to create more diverse learning environments that promote student engagement. Additionally, they can develop and implement new teaching methods that maximize the use of digital technologies and create stimulating and captivating lessons, fostering a more active student involvement in learning. Teachers can optimize their pedagogical approach by adapting and integrating new means of interaction with students, contributing to their deeper understanding and active participation in the learning process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data availability

Data will be available on request.

Abdallah, A., & Alkhrabsheh, A. (2019). The best leadership styles for preventing the educational crisis. Opción: Revista De Ciencias Humanas Y Sociales , 20 , 90–105.

Google Scholar  

Ali, P. A., & Naylor, P. B. (2010). Association between academic and non-academic variables and academic success of diploma nursing students in Pakistan. Nurse Education Today , 30 (2), 157–162. https://doi.org/10.1016/j.nedt.2009.07.006

Article   Google Scholar  

Ang, S. S., Orozco, M., Gijbels, D., & Van den Bossche, P. (2018). Learning in the context of work in a digital age: The use of digital media in informal and formal learning contexts. In C. Harteis (Ed.), The Impact of Digitalization in the Workplace (pp. 87–101). Springer. https://doi.org/10.1007/978-3-319-63257-5_7

Bakar, M. A., Mukhtar, M., & Khalid, F. (2019). The development of a visual output approach for programming via the application of cognitive load theory and constructivism. International Journal of Advanced Computer Science and Applications , 10 (11), 1. https://doi.org/10.14569/ijacsa.2019.0101142

Baker, F. (2014). Reflections on an informal learning environment with invocations for classroom learning in Dubai, the United Arab Emirates. International Journal of Adolescence and Youth , 19 (1), 50–66. https://doi.org/10.1080/02673843.2012.699880

Bangun, C. S., Purnama, S., & Panjaitan, A. S. (2022). Analysis of new business opportunities from online informal education mediamorphosis through digital platforms. International Transactions on Education Technology , 1 (1), 42–52. https://doi.org/10.33050/itee.v1i1.184

Bleakley, A., Bligh, J., & Browne, J. (2011). Socio-cultural learning theories. In A. Bleakley, J. Bligh, J. Browne (Eds.), Medical education for the future: Identity, power and Location (pp. 43–60). Springer. https://doi.org/10.1007/978-90-481-9692-0_4

Carliner, S. (2023). Informal learning basics . American Society for Training and Development.

Chang, C. T., Tu, C. S., & Hajiyev, J. (2019). Integrating academic type of social media activity with perceived academic performance: A role of task-related and nontask-related compulsive internet use. Computers & Education , 139 , 157–172. https://doi.org/10.1016/j.compedu.2019.05.011

Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology , 51 (5), 1441–1449. https://doi.org/10.1111/bjet.13015

Doebel, S. (2020). Rethinking executive function and its development. Perspectives on Psychological Science , 15 (4), 942–956. https://doi.org/10.1177/1745691620904771

Doleck, T., Lajoie, S. P., & Bazelais, P. (2019). Social networking and academic performance: A net benefits perspective. Education and Information Technologies , 24 (5), 3053–3073. https://doi.org/10.1007/s10639-019-09913-3

Dung, D. T. H. (2020). The advantages and disadvantages of virtual learning. IOSR Journal of Research & Method in Education , 10 (3), 45–48. https://doi.org/10.9790/7388-1003054548

Falloon, G. (2020). From digital literacy to digital competence: The teacher digital competency (TDC) framework. Educational Technology Research & Development , 68 (5), 2449–2472. https://doi.org/10.1007/s11423-020-09767-4

Farrokhnia, M., Pijeira-Díaz, H. J., Noroozi, O., & Hatami, J. (2019). Computer-supported collaborative concept mapping: The effects of different instructional designs on conceptual understanding and knowledge co-construction. Computers & Education , 142 , 103640. https://doi.org/10.1016/j.compedu.2019.103640

Giannakos, M., & Cukurova, M. (2023). The role of learning theory in multimodal learning analytics. British Journal of Educational Technology , 54 (5), 1246–1267. https://doi.org/10.1111/bjet.13320

Han, S., & Yi, Y. J. (2019). How does the smartphone usage of college students affect academic performance? Journal of Computer Assisted Learning , 35 (1), 13–22. https://doi.org/10.1111/jcal.12306

Hassanzadeh, M., Hatami, J., Latifi, S., Farrokhnia, M. R., & Saheb, T. (2016). Teaching science for understanding: The positive impact of simultaneous use of concept mapping and computer simulations. In A. Cañas, P. Reiska, & J. Novak (Eds.), Innovating with Concept Mapping: 7th International Conference on Concept Mapping, CMC 2016, Tallinn, Estonia, September 5–9, 2016, Proceedings 7 (pp. 192–202). Springer. https://doi.org/10.1007/978-3-319-45501-3_15

He, T., & Li, S. (2019). A comparative study of digital informal learning: The effects of digital competence and technology expectancy. British Journal of Educational Technology , 50 (4), 1744–1758. https://doi.org/10.1111/bjet.12778

Article   MathSciNet   Google Scholar  

He, T., Zhu, C., & Questier, F. (2018). Predicting digital informal learning: An empirical study among Chinese university students. Asia Pacific Education Review , 19 (1), 79–90. https://doi.org/10.1007/s12564-018-9517-x

Heidari, E., Salimi, G., & Mehrvarz, M. (2023). The influence of online social networks and online social capital on constructing a new graduate students’ professional identity. Interactive Learning Environments , 31 (1), 214–231. https://doi.org/10.1080/10494820.2020.1769682

Helsper, E. J., van Deursen, A. J. A. M., Eynon, R., & van Deursen, A. J. A. M. (2015). Tangible outcomes of internet use: From digital skills to tangible outcomes project report Retrieved September 12, 2023 from www.oii.ox.ac.uk/research/projects/?id=112

Hubbard, P. (2019). Leveraging technology to integrate informal language learning within classroom settings. In M. Dressman, & R. W. Sadler (Eds.), The handbook of Informal Language Learning (pp. 405–419). John Wiley & Sons Ltd. https://doi.org/10.1002/9781119472384.ch27

Jin, B., Kim, J., & Baumgartner, L. M. (2019). Informal learning of older adults in using mobile devices: A review of the literature. Adult Education Quarterly , 69 (2), 120–141. https://doi.org/10.1177/0741713619834726

Jurkovič, V. (2019). Online informal learning of English through smartphones in Slovenia. System , 80 , 27–37. https://doi.org/10.1016/j.system.2018.10.007

Karpenko, O. M., Lukyanova, A. V., Bugai, V. V., & Shchedrova, I. A. (2019). Individualization of learning: An investigation on educational technologies. Journal of History Culture and Art Research , 8 (3), 81–90. https://doi.org/10.7596/taksad.v8i3.2243

Kim, H., Hong, A., & Song, H. D. (2018). The relationships of family, perceived digital competence and attitude, and learning agility in sustainable student engagement in higher education. Sustainability , 10 (12), 4635. https://doi.org/10.3390/su10124635

Kress, G. (2009). Assessment in the perspective of a social semiotic theory of multimodal teaching and learning. In C. Wyatt-Smith, J. J. Cumming (Eds.), Educational assessment in the 21st century: Connecting theory and practice (pp. 19–41). Springer. https://doi.org/10.1007/978-1-4020-9964-9_2

Mangiri, H. S., Sofyan, H., Susanto, A., & Rohmantoro, D. (2019). The contribution of teacher’s digital competency to teacher’s professionalism at vocational high school. International Journal of Innovative Technology and Exploring Engineering , 9 (1), 1728–1731. https://doi.org/10.35940/ijitee.A5198.119119

Mehrvarz, M., Heidari, E., Farrokhnia, M., & Noroozi, O. (2021). The mediating role of digital informal learning in the relationship between students’ digital competence and their academic performance. Computers & Education , 167 , 104184. https://doi.org/10.1016/j.compedu.2021.104184

Mishra, S. (2020). Social networks, social capital, social support and academic success in higher education: A systematic review with a special focus on underrepresented students. Educational Research Review , 29 , 100307. https://doi.org/10.1016/j.edurev.2019.100307

Mohammed, S. H., & Kinyó, L. (2020). The role of constructivism in the enhancement of social studies education. Journal of Critical Reviews , 7 (7), 249–256. https://doi.org/10.31838/jcr.07.05.01

Nyikes, Z. (2018). Contemporary digital competency review. Interdisciplinary Description of Complex Systems , 16 (1), 124–131. https://doi.org/10.7906/indecs.16.1.9

Peters, M., & Romero, M. (2019). Lifelong learning ecologies in online higher education: Students’ engagement in the continuum between formal and informal learning. British Journal of Educational Technology , 50 (4), 1729–1743. https://doi.org/10.1111/bjet.12803

Rahmatirad, M. (2020). A review of socio-cultural theory. Siasat , 5 (3), 23–31. https://doi.org/10.33258/siasat.v4i3.66

Reinders, H., & Benson, P. (2017). Research agenda: Language learning beyond the classroom. Language Teaching , 50 (4), 561–578. https://doi.org/10.1017/S0261444817000192

Sackey, D. J., Nguyen, M. T., & Grabill, J. T. (2015). Constructing learning spaces: What we can learn from studies of informal learning online. Computers and Composition , 35 , 112–124. https://doi.org/10.1016/j.compcom.2015.01.004

Xu, F. (2019). Towards a rational constructivist theory of cognitive development. Psychological Review , 126 (6), 841–864. https://doi.org/10.1037/rev0000153

Yang, Z. (2020). A study on self-efficacy and its role in mobile-assisted language learning. Theory and Practice in Language Studies , 10 (4), 439–444. https://doi.org/10.17507/tpls.1004.13

Yazon, A. D., Ang-Manaig, K., Buama, C. A. C., & Tesoro, J. F. B. (2019). Digital literacy, digital competence and research productivity of educators. Universal Journal of Educational Research , 7 (8), 1734–1743. https://doi.org/10.13189/ujer.2019.070812

Download references

Acknowledgements

Natalia Mikhaylova has been supported by the RUDN University Strategic Academic Leadership Program.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Contributions

GT, NM, and BT contributed equally to the experimentation. GT, BT wrote and edited the article. BT and NM equally designed and conducted the experiment. GT, NM studied scientific literature about the topic. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Gulsara Tazhenova .

Consent to participate

Informed consent was signed by participants.

Competing interests

The authors report there are no competing interests to declare.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Tazhenova, G., Mikhaylova, N. & Turgunbayeva, B. Digital media in informal learning activities. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12687-y

Download citation

Received : 12 September 2023

Accepted : 10 April 2024

Published : 03 May 2024

DOI : https://doi.org/10.1007/s10639-024-12687-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Educational activities
• Informal education
• Interactive tools
• Pedagogical practice
• Technology-enhanced learning
• Find a journal
• Publish with us
• Track your research

How to Use methodology in a Sentence

Methodology.

• He blamed the failure of their research on poor methodology .

Some of these examples are programmatically compiled from various online sources to illustrate current usage of the word 'methodology.' Any opinions expressed in the examples do not represent those of Merriam-Webster or its editors. Send us feedback about these examples.

Can you solve 4 words at once?

Word of the day.

See Definitions and Examples »

Get Word of the Day daily email!

National Center for Science and Engineering Statistics

• All previous cycle years

The Survey of State Government Research and Development provides comprehensive, uniform statistics regarding the extent of R&D activity performed and funded by departments and agencies in each of the nation's 50 states, the District of Columbia, and Puerto Rico.

Survey Info

• tag for use when URL is provided --> Methodology
• tag for use when URL is provided --> Data
• tag for use when URL is provided --> Analysis

The Survey of State Government Research and Development measures the extent of R&D activity performed and funded by the governments of each of the nation’s 50 states, the District of Columbia, and Puerto Rico (collectively, states). By employing consistent, uniform definitions and collection techniques, the survey allows collection of state R&D expenditures data that are comparable nationwide. The survey is a census of state government departments, agencies, commissions, public authorities, and dependent entities with R&D activities.

Areas of Interest

• Government Funding for Science and Engineering
• Research and Development

Survey Administration

The survey was funded by the National Center for Science and Engineering Statistics within the National Science Foundation, and data collection was conducted by the Census Bureau.

Survey Details

• Survey Description (PDF 131 KB)
• Data Tables (PDF 1.6 MB)

Featured Survey Analysis

State Agencies’ R&D Increased 1% in FY 2021; Five States Account for Nearly 60% of All State Government R&D

Survey of State Government R&D Overview

Data highlights, state government agencies’ expenditures for r&d totaled nearly $2.5 billion in fy 2021..

State government funding of external R&D totaled $1.8 billion in FY 2021.

Methodology

Survey description, survey overview (fy 2021 survey cycle).

The Survey of State Government Research and Development (R&D) is the only source for comprehensive, uniform statistics regarding the extent of R&D activity performed and funded by departments and agencies in each of the nation’s 50 state governments, the government of the District of Columbia, and the government of Puerto Rico.

Data collection authority

The information is solicited under the authority of the National Science Foundation Act of 1950, as amended, and the America COMPETES Reauthorization Act of 2010. It is collected under Office of Management and Budget control number 0607–0933, expiration date 31 July 2023. The survey is conducted by the Census Bureau under Title 13, United States Code, § 8(b) for the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation.

Major changes to recent survey cycle

Key survey information, initial survey year, reference period.

State government fiscal years ending in 2021.

Response unit

State government departments, agencies, commissions, public authorities, institutions, and other entities that operate separately or somewhat autonomously from the central state government—but where the state government maintains administrative or fiscal control over their activities—with the capacity to perform or fund R&D; units are collectively referred to as agencies .

Sample or census

Population size.

498 agencies.

Sample size

Not applicable.

Key variables

• State government department or agency
• Total expenditures for R&D
• R&D expenditures by source of funds (federal, state, and other)
• Expenditures for intramural performance by source of funds
• Expenditures for intramural performance by type of work (basic research, applied research, and experimental development)
• Expenditures for extramural performance by source of funds
• Expenditures for extramural performance by type of performer (academic institutions, companies and individuals, and others)
• Federal funds for R&D by state and federal agency
• R&D expenditures by governmental function (agriculture, energy, environment and natural resources, health, transportation, and other)
• Capital outlays for state government R&D-related facilities
• R&D personnel and full-time equivalent by type (researchers, technicians, and support staff)

Survey Design

Target population.

The target population consists of all state departments, agencies, commissions, and dependent entities that funded R&D activities for state government fiscal years ending in 2021. Several industry-specific state commissions, which are generally chartered by state legislatures but are administered independently, are considered state agencies and included in the survey’s population of interest. Excluded are state-run colleges and universities, which are canvassed as part of NCSES’s Higher Education Research and Development (HERD) Survey. State-run laboratories or experiment stations controlled by state universities are also excluded from the respondent universe, as are any entities determined to be nonprofit or private as defined by the Census Bureau. Most state fiscal year periods begin 1 July and end the following 30 June. For example, FY 2021 is defined as the state fiscal period beginning on 1 July 2020 and ending on 30 June 2021. There are, however, five exceptions to the 30 June fiscal year end: New York (ends 31 March), Texas (ends 31 August), and Alabama, the District of Columbia, and Michigan (end 30 September). For comparability, all states, the District of Columbia, and Puerto Rico are surveyed at the same time.

Sampling frame

The total universe includes all state government-dependent units with the capacity to perform or fund R&D, including those for the District of Columbia and Puerto Rico, as defined by the Census Bureau’s Government Finance and Employment Classification Manual . All units were identified with the aid of a state coordinator who was appointed by the governor of each state, Puerto Rico, and the mayor of the District of Columbia. For FY 2021, e-mails were sent to the chief of staff for each Governor’s office asking them to appoint a state coordinator.

Sample design

The Survey of State Government R&D is a census. For the FY 2021 survey, state coordinators were provided with the list of agencies that were previously identified as having the potential to perform or fund R&D from the FY 2020 survey cycle. In addition, the list included agencies identified from a systematic review of state session laws and additional review of agencies reporting to the Census Bureau’s Census of Governments program by staff members from the Census Bureau and NCSES. State coordinators were asked to review this list and add agencies that they believed were involved with R&D and were not already identified. State agencies that have reported $0 R&D for the last three survey cycles were marked as inactive for this survey cycle. These will be reactivated for the FY 2023 survey cycle. State coordinators also adjusted the agency universe to remove agencies that have never had any qualifying R&D to report to NCSES, to add any agencies they thought were missing or could possibly have R&D to report, to address organizational changes within their respective states since the previous survey, and to provide updated agency contact information.

Data Collection and Processing

Data collection.

The survey was funded by NCSES. Data collection was conducted by the Census Bureau via an e-mail containing a survey form. The survey was launched in September 2021, and responses were collected through mid-June 2022. The respondent questionnaire consisted of one screening question intended to reduce the burden on agency respondents who did not have qualifying R&D expenditures during FY 2021, seven questions regarding R&D-related expenditures, and two questions regarding counts of employees.

Data processing

Data collected under the survey are subject to automated data correction procedures using a combination of logical edits incorporated into the survey form, as well as telephone and e-mail follow-up with survey respondents by staff members from NCSES and the Census Bureau for any other data anomalies.

Estimation techniques

All state and national totals are summations of reported state agency data.

Survey Quality Measures

Sampling error, coverage error.

In addition to a Census Bureau review of state session laws to identify agencies with the capacity to fund R&D, NCSES utilizes the expertise of an appointed state coordinator to assist in identifying state government agencies that have the capacity to perform or fund R&D. State coordinators are also offered the opportunity to review survey responses from their respective state agencies before results are finalized for data release. In cases where the state coordinator refused to cooperate or where some agencies did not respond to the survey, it is possible there may be an undercount of state government R&D activities. The undercount may occur despite efforts by staff members of NCSES and the Census Bureau to conduct additional queries and conduct outreach with state agencies that did not appoint a state coordinator. In other instances, the appointed state coordinator could misinterpret the NCSES definition and examples of qualifying R&D activities and not identify all state government-dependent units with the capacity to perform or fund R&D.

Nonresponse error

Of the 498 agencies in the survey universe, 476 (95.6%) responded to the survey. Of the 476 respondents, 399 (83.8%) reported having R&D activities in FY 2021. A mathematical imputation method was used to impute three nonresponding agencies known to have R&D in the past, namely the Kentucky Department of Fish and Wildlife Resources, New Mexico Department of Game and Fish, and Tennessee Department of Education. The agency in Tennessee was imputed despite having responded due to some reporting discrepancies. Puerto Rico did not participate in this year’s survey. No statistical methods were used to account for the other nonresponding agencies, which historically have reported not having R&D.

Measurement error

All responses, including the initial agency data submissions and final state coordinator reviews, were received via e-mail or phone. Census Bureau staff performed basic logical edit checks and reviewed respondent comments, allowing staff to detect errors and work with respondents to correct them. Despite these efforts, some of the data reported could include expenditures for non-R&D activities, such as commercialization, environmental testing, or routine survey work. Similarly, some state data may also exclude minor R&D expenditure amounts from agencies not surveyed.

Data Availability and Comparability

Data availability.

Data presented in trend tables in this report are from the most recently completed survey cycle. Agency-level data are available beginning with FY 2009. No survey of state governments’ FY 2008 R&D activity was conducted.

Data comparability

References to data prior to FY 2021 should be restricted to those published in this report for three reasons: (1) when completing the current-year survey, survey respondents may revise their prior-year data; (2) state coordinators may identify additional agencies to be canvassed that were not initially surveyed during the prior survey cycle, and many of these agencies will provide prior-year data during the current survey collection cycle; and (3) NCSES reviews data from prior years for consistency with current-year responses and, if necessary, may revise these data in consultation with respondents.

For FYs 1995, 1988, and 1987, data collections of state government R&D were conducted by nonfederal organizations that were supported by NCSES grants. Prior to those efforts, NCSES collected state government R&D data for FYs 1977, 1973, 1972, 1968, 1967, 1965, and 1964 in collaboration with the Census Bureau’s Census of Governments and related programs. Because of differences in the survey populations, in definitions of covered R&D activities, and in collection methods over time, the results of these historical surveys are not comparable with the statistics collected for FY 2006 and subsequent Surveys of State Government R&D.

Data Products

Data from the Survey of State Government R&D are published in NCSES InfoBriefs and data tables available at https://www.nsf.gov/statistics/srvystaterd/ . Data from the Survey of State Government R&D are also used in the annual report National Patterns of R&D and the biennial report Science and Engineering Indicators .

Technical Notes

Purpose. The Survey of State Government Research and Development is the only source for comprehensive uniform statistics regarding the extent of R&D activity performed and funded by departments and agencies in each of the nation’s 50 state governments, the government of the District of Columbia, and the government of Puerto Rico.

Data collection a uthority. The information is solicited under the National Science Foundation (NSF) Act of 1950, as amended; the America COMPETES Reauthorization Act of 2010; and Title 13, United States Code, § 8(b). It is collected under Office of Management and Budget control number 0607-0933, expiration date 31 July 2023.

Survey c on tractor. The Census Bureau, under NSF interagency agreement number NCSE-2114762, collected, processed, and tabulated the statistics in this report.

Survey s ponsor. The National Center for Science and Engineering Statistics (NCSES) within NSF.

Frequency. Annual.

Initial survey y ear. 2006.

Reference p eriod. State government fiscal years ending in 2021.

Response unit. State government departments, agencies, commissions, public authorities, institutions, and other entities that operate separately or somewhat autonomously from the central state government—but where the state government maintains administrative or fiscal control over their activities—with the capacity to perform or fund R&D; units are collectively referred to as agencies .

Sample or census. Census.

Population size. The population comprised 498 agencies from the 50 state governments, the District of Columbia, and Puerto Rico with the capacity to perform or fund R&D during FY 2021. This year, agencies that reported $0 R&D the last three survey cycles were marked as inactive for this survey cycle. These will be reactivated for the FY 2023 survey cycle.

Sample size. Not applicable.

Target population. State government departments, agencies, commissions, public authorities, institutions, and other entities that operate separately or somewhat autonomously from the central state government but where the state government maintains administrative or fiscal control over their activities, as defined by the Census Bureau’s Government Finance and Employment Classification Manual (see chapter 1), and that funded or performed R&D for state government FY 2021. Several industry-specific state commissions, which are generally chartered by state legislatures but are administered independently, are considered state agencies and are included in the survey’s population. State-run colleges and universities, which are canvassed as part of NCSES’s Higher Education Research and Development (HERD) Survey, are excluded from the survey frame. State-run laboratories or experiment stations controlled by state universities are also excluded from the respondent universe, as are any entities determined to be nonprofit or private, as defined by the Census Bureau government classification criteria. However, because agricultural experiment stations in Connecticut are legally organized as a state government-dependent agency and because they are not affiliated with any university system, they are included in the survey’s population.

Sampl ing frame. The total universe includes all state government-dependent units, including those for the District of Columbia and Puerto Rico, with the capacity to perform and fund R&D, identified with the aid of a state coordinator who is appointed by the governor of each state.

Sample design. The Survey of State Government Research and Development is a census. For the FY 2021 survey, state coordinators were provided with a list of agencies that were previously identified from the FY 2020 survey cycle as having the potential to perform or fund R&D. In addition, these lists included agencies identified from a systematic review of state session laws and additional review of agencies reporting to the Census Bureau’s Census of Governments program by Census Bureau and NCSES staff. Coordinators were asked to review this list and add agencies that they believed were involved with R&D and were not already identified. State coordinators also adjusted the agency universe to remove agencies that have never had any qualifying R&D to report to NCSES, to add any agencies that were missing or could have R&D to report, to address organizational changes within their respective states since the previous survey, and to provide updated agency contact information.

Data Collection and Processing Methods

Data c ollection. For FY 2021, due to the continued COVID pandemic, letters were not mailed to the office of each state governor, to the governor of Puerto Rico, or to the mayor of the District of Columbia, asking them to appoint a state coordinator who would provide updates to the universe of state government-dependent agencies that had the capacity to perform or fund R&D during FY 2021 (see “Appendix”). Instead, e-mails were sent to the chief of staff for all governors asking them to appoint a state coordinator. Census Bureau staff also reached out to prior state coordinators asking if they would be willing to continue to serve as the coordinators for 2021. On a flow basis, state coordinators were sent a spreadsheet of agencies and contacts that were surveyed for the FY 2020 Survey of State Government Research and Development and asked to add agencies that might have some R&D, remove agencies from the survey universe that no longer perform or fund R&D or have been reorganized, and update agency points of contact. Once the state coordinators completed updates to the list of active agencies to be surveyed, they then sent introductory e-mails to the agency respondents stating that respondents would be receiving an e-mail with instructions and the survey form to be completed and e-mailed back to the Census Bureau. The state coordinators sent the updated spreadsheet of agencies and contacts back to the Census Bureau. After receiving the list of updated agencies, agencies identified as having the potential to perform or fund R&D were then e-mailed the survey form. Upon completion by all agencies within a state, the state coordinators were provided with a spreadsheet of agency responses to review the survey results before they were provided to NCSES for final analysis and dissemination.

Mode. State agencies were e-mailed a fillable form with auto summations and edits built in and were asked to complete the form. State coordinators were given a spreadsheet of potential state agencies and contact information to review and revise as necessary to add agencies to be surveyed, remove others from the survey as inactive, or make corrections to agency points of contact.

Response rates. Response rates were calculated for the FY 2021 Survey of State Government Research and Development and are available in table A-1 .

All 50 state governments and the District of Columbia participated in the survey. A total of 35 of 52 state coordinators updated their spreadsheet. However, only 10 of 52 state coordinators officially responded to verify the final aggregate data for their states. For those agencies that did not have a coordinator appointed, final aggregate data files were sent to staff at NCSES for review. Some or all agencies submitted data in those states where the coordinator did not verify data officially. A coordinator was not appointed in Colorado, Delaware, New York, Pennsylvania, and Rhode Island. Historically, NCSES has partnered with the Puerto Rico Institute of Statistics to collect information on R&D spending from Puerto Rico agencies. The Institute conducts its own survey of R&D activities in the territory and uses the NCSES survey questions for its collections of government agencies and provides the results to NCSES. Although the governor’s office did appoint a coordinator for this year’s survey, they were unable to collect the data on behalf of NCSES in time for this survey cycle. Even though NCSES communicated with Puerto Rico this year, no data were received for FY 2021 from the 17 agencies in Puerto Rico that previously provided data to NCSES in previous years.

The final agency response was 476 of 498 agencies (95.6%). If agencies were unable to answer the survey request, staff from the Census Bureau and NCSES compiled R&D estimates from publicly available financial statements in lieu of a direct response when such materials were available. This includes data for the following state agency: Massachusetts Massventures.

In addition to the 17 agencies in Puerto Rico, the following agencies did not respond to the survey and as a result may contribute to an undercount in their states’ public expenditures for R&D activities:

• California Arts Council
• Minnesota Department of Public Safety
• Ohio Industrial Commission

Of the 476 agencies that responded, 399 (83.8%) reported having some R&D activity in FY 2021.

Data editing. Initial agency data submissions were received via fillable forms e-mailed to the Census Bureau analyst. Basic logical edit checks, review of respondent comments, and comparisons of data from previous surveys allowed Census Bureau and NCSES staff to detect data errors and work with respondents to correct them. Census Bureau and NCSES staff also conducted follow-up calls to agencies with major data changes in R&D between FY 2020 and FY 2021 to ensure the accuracy of the survey data. Major data changes are dependent on the state and type of agency. After all the agencies in a state submitted responses, a spreadsheet of aggregated agency data was sent to the coordinators. They were asked to perform a final verification of aggregated agency data.

Imputation. Data was imputed for three agencies. Kentucky Department of Fish and Wildlife Resources was imputed using a ratio of historic state data. New Mexico Department of Game and Fish was imputed by bringing forward prior-year data based on historic reporting pattern. Tennessee Department of Education was imputed using a ratio of historic Department of Education data. The reported total for Midwest Dairy was broken out across the divisions in Arkansas, Illinois, Iowa, Kansas, Minnesota, Nebraska, North Dakota, Oklahoma, and South Dakota based on percentages each division contributed as per the annual report. All state and national totals are aggregates of reported agency data. Each state government’s organizational structure, laws, and delegation of powers within its purview are unique. There are no universally applied methods of imputation that can be used across all state government agencies and still account for these structural differences. This is consistent with basic statistical methods used by the Census Bureau’s Census of Governments, Survey of State Government Finance. Therefore, R&D expenditures may be underestimated in states where agencies failed to respond. When imputation methods are needed, they are handled on an individual state-by-state basis. When agency financial reports were publicly available, some agency research expenditures were compiled from these reports by staff at the Census Bureau and NCSES.

Weighting. Not applicable.

Variance estimation. Not applicable.

Sampling error . Not applicable.

Coverage error. In addition to a Census Bureau review of state session laws to identify agencies with the capacity to fund R&D, NCSES utilizes the expertise of an appointed state coordinator to assist in identifying state government agencies that have the capacity to perform or fund R&D. State coordinators are also offered the opportunity to review survey responses from their respective state agencies before results are finalized for data release. In cases where the state coordinator refused to cooperate or where some agencies failed to respond to the survey, it is possible there may be an undercount of state government R&D activities. The undercount may occur despite efforts by NCSES and Census Bureau staff to conduct additional queries and outreach with state agencies that did not appoint a state coordinator. In other instances, the appointed state coordinator could misinterpret the survey definition and examples of qualifying R&D activities and thus fail to identify all state government-dependent units with the capacity to perform or fund R&D.

Nonresponse error. Of the 498 agencies in the survey universe, 476 (95.6%) responded to the survey. Of the 476 respondents, 399 (83.8%) reported having R&D activities in FY 2021. A mathematical statistical method was used to impute three nonresponding larger agencies known to have R&D in the past, namely the Kentucky Department of Fish and Wildlife Resources, New Mexico Department Game and Fish, and the Tennessee Department of Education. The agency in Tennessee was imputed despite having responded due to some reporting discrepancies. No other imputation was used for the other nonresponding agencies because the other agencies historically have reported not having R&D.

Measurement error . The most common form of nonsampling error in the Survey of State Government Research and Development is in respondents’ interpretation of the survey definition of qualifying R&D activities. To mitigate any potential misinterpretations, several steps were taken. NCSES provided a series of examples specific to the types of activities performed or funded by state government agencies in the survey questionnaire’s definitions and examples. All responses, including the initial agency data submissions and final state coordinator verifications, were received via e-mail or phone. Census Bureau staff performed basic logical edit checks and reviewed respondent comments, allowing staff to detect errors and work with state respondents to correct them. Despite these efforts, some of the data reported could include expenditures for non-R&D activities, such as non-R&D salaries, commercialization, environmental testing, or routine survey or monitoring work. Similarly, some state data may also exclude minor R&D expenditure amounts from agencies not surveyed.

Data Comparability

State government R&D totals can display considerable volatility between survey cycles. For example, state agency expenditures are influenced by several national and state-specific factors, and large changes (either increases or decreases) are not unusual, especially for discretionary spending items such as R&D. States often will create special funds to support specific research activities for a limited time. These funds may have a one-time appropriation from the legislature and expire within 2–5 fiscal years; state agencies obligate those funds for specific R&D projects, depending on availability and expiration of funding authority, as well as other program-specific and administrative considerations. Data reported are agency direct expenditures for R&D in a given fiscal year, not obligations. As such, in the case of multiyear grants to extramural performers, an agency’s expenditures for that fiscal year may be greater than its obligations because expenditures may include spending from the previous year’s appropriations, depending on the specific budget authority granted by the legislature. It is likely that some portion of the reported changes reflects measurement and coverage errors. In the case of R&D funds for extramural performers, some agencies were able to report only multiyear obligations rather than single-year expenditures.

The survey asked about state agencies’ expenditures for R&D at the end of FY 2021. Most states and Puerto Rico have a fiscal year that begins 1 July and ends the following 30 June. For example, FY 2021 is the state fiscal period beginning on 1 July 2020 and ending on 30 June 2021. There are, however, five exceptions to the 30 June fiscal year end: New York (ends 31 March), Texas (ends 31 August), and Alabama, the District of Columbia, and Michigan (end 30 September). For comparability, all states, the District of Columbia, and Puerto Rico are surveyed at the same time.

A state’s R&D priorities may be shaped by the state’s unique legislative and budgeting processes. State budget practices vary considerably due to both political and historical reasons. Nineteen states enact biennial budgets. Of these states, Montana, Nevada, North Dakota, and Texas have both biennial legislative sessions and biennial budgets. The remaining 15 states of Connecticut, Hawaii, Indiana, Kentucky, Maine, Minnesota, Nebraska, New Hampshire, North Carolina, Ohio, Oregon, Virginia, Washington, Wisconsin, and Wyoming hold annual legislative sessions but maintain biennial budgeting. Only North Dakota and Wyoming enact consolidated 2-year budgets; other biennial budget states enact two annual budgets at one time. As such, the nature of a state’s budget priorities for R&D may be determined on a biennial basis in some states; in others, however, it may be determined on an annual basis. In states with biennial budgets, the legislatures will often make supplemental appropriations to the second-year budget, which may result in further changes to the initial funding priorities.

The data exclude R&D expenditures by state governments that did not flow through state agencies’ budgets. The state totals do not include direct appropriations from state legislatures to colleges and universities. Higher education institutions’ expenditures from state government appropriations are available from NCSES’s HERD Survey ( https://www.nsf.gov/statistics/srvyherd/ ). For FY 2021, state government agencies reported $960 million in expenditures used to support R&D performance by academic institutions. A major factor for the difference between totals reported in NCSES’s HERD Survey and the Survey of State Government Research and Development is direct appropriations to state-run universities are included in the former but not in the latter. Another likely factor is the exclusion of R&D at agricultural experiment stations from the state survey totals because they are generally associated with land-grant colleges and universities and are canvassed on the HERD Survey.

Direct comparison of state agency expenditures should also be viewed with caution because state governments often reorganize departments and agencies such that some divisions and offices that were part of one agency may be moved to another agency. In other instances, entire departments may be reorganized into newly created departments. Although the FY 2021 Survey of State Government Research and Development encountered several instances of these organizational changes in several states, the survey itself is not designed to measure specific changes in state government organization. To account for these and other changes in the data, Census Bureau and NCSES staff conducted follow-up calls for agencies with major data changes depending on the type of agency and state, between FY 2020 and FY 2021, to ensure the accuracy of the survey data.

Data specific to state government agencies were first released with the FY 2009 survey results and are also included in the FY 2021 data tables. Specific agency-level data for FY 2006 and FY 2007 are not available.

The current Survey of State Government Research and Development has been conducted for FY 2006, FY 2007, FY 2009, FYs 2010–11, FYs 2012–13, FYs 2014–15, FY 2016, FY 2017, FY 2018, FY 2019, FY 2020, and FY 2021. (No survey was conducted for state governments for FY 2008.) Data presented in trend tables in this report are from the most recently completed survey cycle. References to prior-year data should be restricted to those published in this report for two reasons: (1) when completing the current year’s survey, survey respondents may revise their prior year’s data, and (2) NCSES reviews data for prior years for consistency with current-year responses and, if necessary, may revise these data in consultation with respondents.

NCSES has collected state government R&D data for FY 1964, FY 1965, FY 1967, FY 1968, FY 1972, FY 1973, and FY 1977 in collaboration with the Census Bureau’s Census of Governments and related programs. For FY 1987, FY 1988, and FY 1995, data collections of state government R&D were conducted by nonfederal organizations that were supported by NSF grants. As a result of differences in the survey populations, in definitions of covered R&D activities, and in collection methods over time, the results of these historical surveys are not comparable with the statistics collected for the FY 2006 and subsequent Surveys of State Government Research and Development.

Changes in survey coverage and population . Each year, state coordinators update the universe of agencies most likely to have funded or performed R&D based on changes in funding authority, organization changes within the government, or other initiatives by the legislature. No survey was conducted for state governments for FY 2008. Beginning with the FY 2009 survey cycle, state coordinators were no longer able to overwrite the aggregate R&D data reported by state agencies to correct or modify the state total. Any changes or revisions were now required to be made at the state government agency level.

Changes in questionnaire .

• FY 2009. The FY 2009 questionnaire was the first to collect state government R&D activities by governmental functions of agriculture, environment and natural resources, health, transportation, and other.
• FYs 2010 and 2011. The survey was reorganized as a biennial survey and collected 2 fiscal years of data on one questionnaire. In addition, the energy category was added to the list of specific government functions of R&D.
• FYs 2012 and 2013. A minor change to the instructions in question 1 for extramural performers was made from “R&D done for your department/agency” to “R&D funded by your department/agency” to ensure that all R&D-related projects that the agency funds regardless of the end result (i.e., grants) were properly included.
• FYs 2014 and 2015. The survey was revised to collect additional details about R&D funding and performance to better align with the Organisation for Economic Co-operation and Development (OECD) 2002 Frascati Manual , the most recent edition available at survey launch. These changes include source of funds for extramural R&D performance supported by federal funds, state funds, or other funds. For all federal funds received for both intramural and extramural R&D, respondents were asked how much was received from specific federal agencies. For intramural R&D performance, respondents were asked how much of federal, state, and other funding was classified as basic research, applied research, or experimental development.
• FY 2016. Survey reporting period changed from a biennial to annual survey. Questions remained the same with some minor additions to examples and wording changes. A remarks box was added for respondents to provide comments.
• FY 2017. The “other” category for Internal Sources of R&D was split into multiple categories: nonfederal government funds, nonprofit organizations, businesses, and higher education institutions.
• FY 2018. No changes were made.
• FY 2019. A question asking about the amount of time it takes to complete survey. This question was added for administrative purposes.
• FY 2020. Two new questions were added asking for the number of intramural R&D personnel and the full-time equivalents for researchers, technicians, and support staff. Data from these new questions are not available this year.
• FY 2021. The survey was restructured to group related questions together. For example, questions on expenditures for R&D performed internally by type of R&D, internal R&D employees, and internal full-time equivalent R&D personnel were moved to follow the internal R&D expenditures question. Similarly, questions on expenditures for R&D performed externally and expenditures for R&D performed externally by type of entity were moved to follow the internal R&D questions. All questions asking for crosscuts on the total R&D expenditures were moved to follow both internal and external R&D-related questions.

Changes in reporting procedures or classification .

• FY 2018. Online reporting was replaced with a fillable PDF with automated summations and edits built into the survey instrument.
• FYs 2019–21. No changes were made.

Definitions

Applied research. Original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily toward a specific, practical aim or objective.

Basic research . Experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view.

Construction and acquisition of facilities used primarily for R&D. Includes the acquisition of, construction of, and major repairs or alterations to structures, works, equipment, facilities, or land for use in R&D activities. Construction and acquisition of land and facilities used primarily for R&D includes major costs for construction and purchase of buildings to be primarily used as R&D facilities.

Experimental development. Systematic work, drawing on knowledge gained from research and practical experience and producing additional knowledge, which is directed to producing new products or processes or to improving existing products or processes.

Performers, ext ramural . Those outside the department or agency who perform R&D under the administrative oversight or control of that department or agency. This may include projects for the department or agency as well as the department’s or agency’s extramural research programs. Extramural performers include the following:

• Academic institutions. Public or private universities and colleges.
• Companies and individuals. Performers under contract for research projects or that received grants for research projects.
• Other. Nonprofit organizations, including foundations; federal government departments and agencies; other departments or agencies within the state; other state governments; county, city, special district, or regional local governments.

Performers, int ramural . Department’s or agency’s own employees who perform R&D, which includes R&D performed by those employees and services performed by others in support of an internal R&D project (e.g., laboratory testing).

Research and development. Comprise creative and systematic work undertaken in order to increase the stock of knowledge—including knowledge of humankind, culture, and society—and to devise new applications of available knowledge. Sources and examples of R&D funding include the following:

• Federal government . Grants, contracts, awards, and appropriations from the U.S. government.
• State. Appropriations from the state legislature, agricultural commodity assessments, bond funds, general funds, restricted funds, revenue funds, state grants, tobacco settlement funds, lottery proceeds, funds from other agencies within the state, and revenue from charges, fees, or fines.
• Nonprofit organizations . Includes funding from foundations.
• Nonfederal governmen t. Funding from other state governments, county, city, regional, or other local governments.
• Businesses . Grants and contracts from companies.
• Higher education institutions . Funding from public or private universities and colleges.
• R&D e mployee count .
• Researchers . For example, biologists, psychologists, research scientists and engineers, primary investigators, and R&D managers.
• Technicians . Equivalent staff includes wildlife technicians, lab technicians, and field staff.
• Support s taff . For example, accountants, facilities management, grant specialist, and clerical staff.
• Full-time equivalent R&D personnel . Calculated as the total working hours spent working on R&D during your state government’s fiscal year divided by the number of hours representing a full-time schedule within the same period.

Technical Tables

Questionnaires, view archived questionnaires, key data tables.

Recommended data tables

State agency expenditures for R&D and R&D plant, by state: FY 2021

State government expenditures for r&d, by state, state government expenditures for r&d, by state department or agency, state government intramural r&d personnel, by state department or agency, data tables, general notes.

These tables present the results of the FY 2021 Survey of State Government Research and Development (R&D), conducted by the Census Bureau under an interagency agreement with the National Center for Science and Engineering Statistics within the National Science Foundation. The Census Bureau employed a methodology similar to one they use to collect data from state and local governments on other censuses and surveys.

The survey was distributed to departments, agencies, commissions, public authorities, and other state-run entities within the 50 states, the District of Columbia, and Puerto Rico that were considered likely to conduct or fund R&D activities. (The District of Columbia and Puerto Rico are referred to as states and treated as state equivalents throughout this report.) Respondents that did not have a qualifying R&D activity were not required to complete the questionnaire beyond a screening question. All 50 states and the District of Columbia participated in the survey. Puerto Rico did not participate this year. Respondents were asked to provide statistics on their total R&D expenditures and amounts for intramural and extramural performers; the amount of intramural R&D devoted to basic research, applied research, and experimental development activities by federal and nonfederal funds; the amount for which the federal government was an original source of funds for both intramural and extramural R&D and from which federal agency the funding came; and the amount of money allocated to particular government functions. An additional question was asked about the amount of expenditures for the construction and acquisition of R&D facilities and major equipment. Additional questions were asked about the number of employees (researchers, technicians, and support staff) and full-time equivalent personnel for intramural R&D.

Acknowledgments and Suggested Citation

Acknowledgments.

Christopher V. Pece of the National Center for Science and Engineering Statistics (NCSES) developed and coordinated this report under the guidance of Gary Anderson, NCSES Program Director, and under the leadership of Emilda B. Rivers, NCSES Director; Vipin Arora, NCSES Deputy Director; and John Finamore, NCSES Chief Statistician.

Under NCSES interagency agreement with the Census Bureau, Vicki Kuppala and Millicent Grant, under the supervision of Michael Flaherty, Chief, Research, Development, and Innovation Surveys Branch at the Census Bureau, compiled the tables in this report. Elizabeth Willhide and Millicent Grant at the Census Bureau also conducted legal review of state sessions laws and agency founding legislation to update the survey frame.

NCSES thanks the program and budget offices at the state agencies that provided information for this report.

Suggested Citation

National Center for Science and Engineering Statistics (NCSES). 2022. Survey of State Government Research and Development: FY 20 2 1 . NSF 23-302. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/pubs/nsf23302 .

Featured Analysis

Related content, related collections, survey contact.

For additional information about this survey or the methodology, contact

Get e-mail updates from NCSES

NCSES is an official statistical agency. Subscribe below to receive our latest news and announcements.