example of method in research paper

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2023 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

How to Write a Methods Section for a Psychology Paper

Tips and Examples of an APA Methods Section

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

Verywell / Brianna Gilmartin 

The methods section of an APA format psychology paper provides the methods and procedures used in a research study or experiment . This part of an APA paper is critical because it allows other researchers to see exactly how you conducted your research.

Method refers to the procedure that was used in a research study. It included a precise description of how the experiments were performed and why particular procedures were selected. While the APA technically refers to this section as the 'method section,' it is also often known as a 'methods section.'

The methods section ensures the experiment's reproducibility and the assessment of alternative methods that might produce different results. It also allows researchers to replicate the experiment and judge the study's validity.

This article discusses how to write a methods section for a psychology paper, including important elements to include and tips that can help.

What to Include in a Method Section

So what exactly do you need to include when writing your method section? You should provide detailed information on the following:

• Research design
• Participants
• Participant behavior

The method section should provide enough information to allow other researchers to replicate your experiment or study.

Components of a Method Section

The method section should utilize subheadings to divide up different subsections. These subsections typically include participants, materials, design, and procedure.

Participants 

In this part of the method section, you should describe the participants in your experiment, including who they were (and any unique features that set them apart from the general population), how many there were, and how they were selected. If you utilized random selection to choose your participants, it should be noted here.

For example: "We randomly selected 100 children from elementary schools near the University of Arizona."

At the very minimum, this part of your method section must convey:

• Basic demographic characteristics of your participants (such as sex, age, ethnicity, or religion)
• The population from which your participants were drawn
• Any restrictions on your pool of participants
• How many participants were assigned to each condition and how they were assigned to each group (i.e., randomly assignment , another selection method, etc.)
• Why participants took part in your research (i.e., the study was advertised at a college or hospital, they received some type of incentive, etc.)

Information about participants helps other researchers understand how your study was performed, how generalizable the result might be, and allows other researchers to replicate the experiment with other populations to see if they might obtain the same results.

In this part of the method section, you should describe the materials, measures, equipment, or stimuli used in the experiment. This may include:

• Testing instruments
• Technical equipment
• Any psychological assessments that were used
• Any special equipment that was used

For example: "Two stories from Sullivan et al.'s (1994) second-order false belief attribution tasks were used to assess children's understanding of second-order beliefs."

For standard equipment such as computers, televisions, and videos, you can simply name the device and not provide further explanation.

Specialized equipment should be given greater detail, especially if it is complex or created for a niche purpose. In some instances, such as if you created a special material or apparatus for your study, you might need to include an illustration of the item in the appendix of your paper.

In this part of your method section, describe the type of design used in the experiment. Specify the variables as well as the levels of these variables. Identify:

• The independent variables
• Dependent variables
• Control variables
• Any extraneous variables that might influence your results.

Also, explain whether your experiment uses a  within-groups  or between-groups design.

For example: "The experiment used a 3x2 between-subjects design. The independent variables were age and understanding of second-order beliefs."

The next part of your method section should detail the procedures used in your experiment. Your procedures should explain:

• What the participants did
• How data was collected
• The order in which steps occurred

For example: "An examiner interviewed children individually at their school in one session that lasted 20 minutes on average. The examiner explained to each child that he or she would be told two short stories and that some questions would be asked after each story. All sessions were videotaped so the data could later be coded."

Keep this subsection concise yet detailed. Explain what you did and how you did it, but do not overwhelm your readers with too much information.

Tips for How to Write a Methods Section

In addition to following the basic structure of an APA method section, there are also certain things you should remember when writing this section of your paper. Consider the following tips when writing this section:

• Use the past tense : Always write the method section in the past tense.
• Be descriptive : Provide enough detail that another researcher could replicate your experiment, but focus on brevity. Avoid unnecessary detail that is not relevant to the outcome of the experiment.
• Use an academic tone : Use formal language and avoid slang or colloquial expressions. Word choice is also important. Refer to the people in your experiment or study as "participants" rather than "subjects."
• Use APA format : Keep a style guide on hand as you write your method section. The Publication Manual of the American Psychological Association is the official source for APA style.
• Make connections : Read through each section of your paper for agreement with other sections. If you mention procedures in the method section, these elements should be discussed in the results and discussion sections.
• Proofread : Check your paper for grammar, spelling, and punctuation errors.. typos, grammar problems, and spelling errors. Although a spell checker is a handy tool, there are some errors only you can catch.

After writing a draft of your method section, be sure to get a second opinion. You can often become too close to your work to see errors or lack of clarity. Take a rough draft of your method section to your university's writing lab for additional assistance.

A Word From Verywell

The method section is one of the most important components of your APA format paper. The goal of your paper should be to clearly detail what you did in your experiment. Provide enough detail that another researcher could replicate your study if they wanted.

Finally, if you are writing your paper for a class or for a specific publication, be sure to keep in mind any specific instructions provided by your instructor or by the journal editor. Your instructor may have certain requirements that you need to follow while writing your method section.

Frequently Asked Questions

While the subsections can vary, the three components that should be included are sections on the participants, the materials, and the procedures.

• Describe who the participants were in the study and how they were selected.
• Define and describe the materials that were used including any equipment, tests, or assessments
• Describe how the data was collected

To write your methods section in APA format, describe your participants, materials, study design, and procedures. Keep this section succinct, and always write in the past tense. The main heading of this section should be labeled "Method" and it should be centered, bolded, and capitalized. Each subheading within this section should be bolded, left-aligned and in title case.

The purpose of the methods section is to describe what you did in your experiment. It should be brief, but include enough detail that someone could replicate your experiment based on this information. Your methods section should detail what you did to answer your research question. Describe how the study was conducted, the study design that was used and why it was chosen, and how you collected the data and analyzed the results.

Erdemir F. How to write a materials and methods section of a scientific article ? Turk J Urol . 2013;39(Suppl 1):10-5. doi:10.5152/tud.2013.047

Kallet RH. How to write the methods section of a research paper . Respir Care . 2004;49(10):1229-32. PMID: 15447808.

American Psychological Association.  Publication Manual of the American Psychological Association  (7th ed.). Washington DC: The American Psychological Association; 2019.

American Psychological Association. APA Style Journal Article Reporting Standards . Published 2020.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What Is a Research Methodology? | Steps & Tips

What Is a Research Methodology? | Steps & Tips

Published on 25 February 2019 by Shona McCombes . Revised on 10 October 2022.

Your research methodology discusses and explains the data collection and analysis methods you used in your research. A key part of your thesis, dissertation, or research paper, the methodology chapter explains what you did and how you did it, allowing readers to evaluate the reliability and validity of your research.

It should include:

• The type of research you conducted
• How you collected and analysed your data
• Any tools or materials you used in the research
• Why you chose these methods
• Your methodology section should generally be written in the past tense .
• Academic style guides in your field may provide detailed guidelines on what to include for different types of studies.
• Your citation style might provide guidelines for your methodology section (e.g., an APA Style methods section ).

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

How to write a research methodology, why is a methods section important, step 1: explain your methodological approach, step 2: describe your data collection methods, step 3: describe your analysis method, step 4: evaluate and justify the methodological choices you made, tips for writing a strong methodology chapter, frequently asked questions about methodology.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Your methods section is your opportunity to share how you conducted your research and why you chose the methods you chose. It’s also the place to show that your research was rigorously conducted and can be replicated .

It gives your research legitimacy and situates it within your field, and also gives your readers a place to refer to if they have any questions or critiques in other sections.

You can start by introducing your overall approach to your research. You have two options here.

Option 1: Start with your “what”

What research problem or question did you investigate?

• Aim to describe the characteristics of something?
• Explore an under-researched topic?
• Establish a causal relationship?

And what type of data did you need to achieve this aim?

• Quantitative data , qualitative data , or a mix of both?
• Primary data collected yourself, or secondary data collected by someone else?
• Experimental data gathered by controlling and manipulating variables, or descriptive data gathered via observations?

Option 2: Start with your “why”

Depending on your discipline, you can also start with a discussion of the rationale and assumptions underpinning your methodology. In other words, why did you choose these methods for your study?

• Why is this the best way to answer your research question?
• Is this a standard methodology in your field, or does it require justification?
• Were there any ethical considerations involved in your choices?
• What are the criteria for validity and reliability in this type of research ?

Once you have introduced your reader to your methodological approach, you should share full details about your data collection methods .

Quantitative methods

In order to be considered generalisable, you should describe quantitative research methods in enough detail for another researcher to replicate your study.

Here, explain how you operationalised your concepts and measured your variables. Discuss your sampling method or inclusion/exclusion criteria, as well as any tools, procedures, and materials you used to gather your data.

Surveys Describe where, when, and how the survey was conducted.

• How did you design the questionnaire?
• What form did your questions take (e.g., multiple choice, Likert scale )?
• Were your surveys conducted in-person or virtually?
• What sampling method did you use to select participants?
• What was your sample size and response rate?

Experiments Share full details of the tools, techniques, and procedures you used to conduct your experiment.

• How did you design the experiment ?
• How did you recruit participants?
• How did you manipulate and measure the variables ?
• What tools did you use?

Existing data Explain how you gathered and selected the material (such as datasets or archival data) that you used in your analysis.

• Where did you source the material?
• How was the data originally produced?
• What criteria did you use to select material (e.g., date range)?

The survey consisted of 5 multiple-choice questions and 10 questions measured on a 7-point Likert scale.

The goal was to collect survey responses from 350 customers visiting the fitness apparel company’s brick-and-mortar location in Boston on 4–8 July 2022, between 11:00 and 15:00.

Here, a customer was defined as a person who had purchased a product from the company on the day they took the survey. Participants were given 5 minutes to fill in the survey anonymously. In total, 408 customers responded, but not all surveys were fully completed. Due to this, 371 survey results were included in the analysis.

Qualitative methods

In qualitative research , methods are often more flexible and subjective. For this reason, it’s crucial to robustly explain the methodology choices you made.

Be sure to discuss the criteria you used to select your data, the context in which your research was conducted, and the role you played in collecting your data (e.g., were you an active participant, or a passive observer?)

Interviews or focus groups Describe where, when, and how the interviews were conducted.

• How did you find and select participants?
• How many participants took part?
• What form did the interviews take ( structured , semi-structured , or unstructured )?
• How long were the interviews?
• How were they recorded?

Participant observation Describe where, when, and how you conducted the observation or ethnography .

• What group or community did you observe? How long did you spend there?
• How did you gain access to this group? What role did you play in the community?
• How long did you spend conducting the research? Where was it located?
• How did you record your data (e.g., audiovisual recordings, note-taking)?

Existing data Explain how you selected case study materials for your analysis.

• What type of materials did you analyse?
• How did you select them?

In order to gain better insight into possibilities for future improvement of the fitness shop’s product range, semi-structured interviews were conducted with 8 returning customers.

Here, a returning customer was defined as someone who usually bought products at least twice a week from the store.

Surveys were used to select participants. Interviews were conducted in a small office next to the cash register and lasted approximately 20 minutes each. Answers were recorded by note-taking, and seven interviews were also filmed with consent. One interviewee preferred not to be filmed.

Mixed methods

Mixed methods research combines quantitative and qualitative approaches. If a standalone quantitative or qualitative study is insufficient to answer your research question, mixed methods may be a good fit for you.

Mixed methods are less common than standalone analyses, largely because they require a great deal of effort to pull off successfully. If you choose to pursue mixed methods, it’s especially important to robustly justify your methods here.

Next, you should indicate how you processed and analysed your data. Avoid going into too much detail: you should not start introducing or discussing any of your results at this stage.

In quantitative research , your analysis will be based on numbers. In your methods section, you can include:

• How you prepared the data before analysing it (e.g., checking for missing data , removing outliers , transforming variables)
• Which software you used (e.g., SPSS, Stata or R)
• Which statistical tests you used (e.g., two-tailed t test , simple linear regression )

In qualitative research, your analysis will be based on language, images, and observations (often involving some form of textual analysis ).

Specific methods might include:

• Content analysis : Categorising and discussing the meaning of words, phrases and sentences
• Thematic analysis : Coding and closely examining the data to identify broad themes and patterns
• Discourse analysis : Studying communication and meaning in relation to their social context

Mixed methods combine the above two research methods, integrating both qualitative and quantitative approaches into one coherent analytical process.

Above all, your methodology section should clearly make the case for why you chose the methods you did. This is especially true if you did not take the most standard approach to your topic. In this case, discuss why other methods were not suitable for your objectives, and show how this approach contributes new knowledge or understanding.

In any case, it should be overwhelmingly clear to your reader that you set yourself up for success in terms of your methodology’s design. Show how your methods should lead to results that are valid and reliable, while leaving the analysis of the meaning, importance, and relevance of your results for your discussion section .

• Quantitative: Lab-based experiments cannot always accurately simulate real-life situations and behaviours, but they are effective for testing causal relationships between variables .
• Qualitative: Unstructured interviews usually produce results that cannot be generalised beyond the sample group , but they provide a more in-depth understanding of participants’ perceptions, motivations, and emotions.
• Mixed methods: Despite issues systematically comparing differing types of data, a solely quantitative study would not sufficiently incorporate the lived experience of each participant, while a solely qualitative study would be insufficiently generalisable.

Remember that your aim is not just to describe your methods, but to show how and why you applied them. Again, it’s critical to demonstrate that your research was rigorously conducted and can be replicated.

1. Focus on your objectives and research questions

The methodology section should clearly show why your methods suit your objectives  and convince the reader that you chose the best possible approach to answering your problem statement and research questions .

2. Cite relevant sources

Your methodology can be strengthened by referencing existing research in your field. This can help you to:

• Show that you followed established practice for your type of research
• Discuss how you decided on your approach by evaluating existing research
• Present a novel methodological approach to address a gap in the literature

3. Write for your audience

Consider how much information you need to give, and avoid getting too lengthy. If you are using methods that are standard for your discipline, you probably don’t need to give a lot of background or justification.

Regardless, your methodology should be a clear, well-structured text that makes an argument for your approach, not just a list of technical details and procedures.

Methodology refers to the overarching strategy and rationale of your research. Developing your methodology involves studying the research methods used in your field and the theories or principles that underpin them, in order to choose the approach that best matches your objectives.

Methods are the specific tools and procedures you use to collect and analyse data (e.g. interviews, experiments , surveys , statistical tests ).

In a dissertation or scientific paper, the methodology chapter or methods section comes after the introduction and before the results , discussion and conclusion .

Depending on the length and type of document, you might also include a literature review or theoretical framework before the methodology.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

A sample is a subset of individuals from a larger population. Sampling means selecting the group that you will actually collect data from in your research.

For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

Statistical sampling allows you to test a hypothesis about the characteristics of a population. There are various sampling methods you can use to ensure that your sample is representative of the population as a whole.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, October 10). What Is a Research Methodology? | Steps & Tips. Scribbr. Retrieved 29 April 2024, from https://www.scribbr.co.uk/thesis-dissertation/methodology/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a literature review | guide, template, & examples, what is a theoretical framework | a step-by-step guide.

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

• PLOS Biology
• PLOS Climate
• PLOS Complex Systems
• PLOS Computational Biology
• PLOS Digital Health
• PLOS Genetics
• PLOS Global Public Health
• PLOS Medicine
• PLOS Mental Health
• PLOS Neglected Tropical Diseases
• PLOS Pathogens
• PLOS Sustainability and Transformation
• PLOS Collections
• How to Write Your Methods

Ensure understanding, reproducibility and replicability

What should you include in your methods section, and how much detail is appropriate?

Why Methods Matter

The methods section was once the most likely part of a paper to be unfairly abbreviated, overly summarized, or even relegated to hard-to-find sections of a publisher’s website. While some journals may responsibly include more detailed elements of methods in supplementary sections, the movement for increased reproducibility and rigor in science has reinstated the importance of the methods section. Methods are now viewed as a key element in establishing the credibility of the research being reported, alongside the open availability of data and results.

A clear methods section impacts editorial evaluation and readers’ understanding, and is also the backbone of transparency and replicability.

For example, the Reproducibility Project: Cancer Biology project set out in 2013 to replicate experiments from 50 high profile cancer papers, but revised their target to 18 papers once they understood how much methodological detail was not contained in the original papers.

What to include in your methods section

What you include in your methods sections depends on what field you are in and what experiments you are performing. However, the general principle in place at the majority of journals is summarized well by the guidelines at PLOS ONE : “The Materials and Methods section should provide enough detail to allow suitably skilled investigators to fully replicate your study. ” The emphases here are deliberate: the methods should enable readers to understand your paper, and replicate your study. However, there is no need to go into the level of detail that a lay-person would require—the focus is on the reader who is also trained in your field, with the suitable skills and knowledge to attempt a replication.

A constant principle of rigorous science

A methods section that enables other researchers to understand and replicate your results is a constant principle of rigorous, transparent, and Open Science. Aim to be thorough, even if a particular journal doesn’t require the same level of detail . Reproducibility is all of our responsibility. You cannot create any problems by exceeding a minimum standard of information. If a journal still has word-limits—either for the overall article or specific sections—and requires some methodological details to be in a supplemental section, that is OK as long as the extra details are searchable and findable .

Imagine replicating your own work, years in the future

As part of PLOS’ presentation on Reproducibility and Open Publishing (part of UCSF’s Reproducibility Series ) we recommend planning the level of detail in your methods section by imagining you are writing for your future self, replicating your own work. When you consider that you might be at a different institution, with different account logins, applications, resources, and access levels—you can help yourself imagine the level of specificity that you yourself would require to redo the exact experiment. Consider:

• Which details would you need to be reminded of? 
• Which cell line, or antibody, or software, or reagent did you use, and does it have a Research Resource ID (RRID) that you can cite?
• Which version of a questionnaire did you use in your survey? 
• Exactly which visual stimulus did you show participants, and is it publicly available? 
• What participants did you decide to exclude? 
• What process did you adjust, during your work? 

Tip: Be sure to capture any changes to your protocols

You yourself would want to know about any adjustments, if you ever replicate the work, so you can surmise that anyone else would want to as well. Even if a necessary adjustment you made was not ideal, transparency is the key to ensuring this is not regarded as an issue in the future. It is far better to transparently convey any non-optimal methods, or methodological constraints, than to conceal them, which could result in reproducibility or ethical issues downstream.

Visual aids for methods help when reading the whole paper

Consider whether a visual representation of your methods could be appropriate or aid understanding your process. A visual reference readers can easily return to, like a flow-diagram, decision-tree, or checklist, can help readers to better understand the complete article, not just the methods section.

Ethical Considerations

In addition to describing what you did, it is just as important to assure readers that you also followed all relevant ethical guidelines when conducting your research. While ethical standards and reporting guidelines are often presented in a separate section of a paper, ensure that your methods and protocols actually follow these guidelines. Read more about ethics .

Existing standards, checklists, guidelines, partners

While the level of detail contained in a methods section should be guided by the universal principles of rigorous science outlined above, various disciplines, fields, and projects have worked hard to design and develop consistent standards, guidelines, and tools to help with reporting all types of experiment. Below, you’ll find some of the key initiatives. Ensure you read the submission guidelines for the specific journal you are submitting to, in order to discover any further journal- or field-specific policies to follow, or initiatives/tools to utilize.

Tip: Keep your paper moving forward by providing the proper paperwork up front

Be sure to check the journal guidelines and provide the necessary documents with your manuscript submission. Collecting the necessary documentation can greatly slow the first round of peer review, or cause delays when you submit your revision.

Randomized Controlled Trials – CONSORT The Consolidated Standards of Reporting Trials (CONSORT) project covers various initiatives intended to prevent the problems of  inadequate reporting of randomized controlled trials. The primary initiative is an evidence-based minimum set of recommendations for reporting randomized trials known as the CONSORT Statement . 

Systematic Reviews and Meta-Analyses – PRISMA The Preferred Reporting Items for Systematic Reviews and Meta-Analyses ( PRISMA ) is an evidence-based minimum set of items focusing  on the reporting of  reviews evaluating randomized trials and other types of research.

Research using Animals – ARRIVE The Animal Research: Reporting of In Vivo Experiments ( ARRIVE ) guidelines encourage maximizing the information reported in research using animals thereby minimizing unnecessary studies. (Original study and proposal , and updated guidelines , in PLOS Biology .) 

Laboratory Protocols Protocols.io has developed a platform specifically for the sharing and updating of laboratory protocols , which are assigned their own DOI and can be linked from methods sections of papers to enhance reproducibility. Contextualize your protocol and improve discovery with an accompanying Lab Protocol article in PLOS ONE .

Consistent reporting of Materials, Design, and Analysis – the MDAR checklist A cross-publisher group of editors and experts have developed, tested, and rolled out a checklist to help establish and harmonize reporting standards in the Life Sciences . The checklist , which is available for use by authors to compile their methods, and editors/reviewers to check methods, establishes a minimum set of requirements in transparent reporting and is adaptable to any discipline within the Life Sciences, by covering a breadth of potentially relevant methodological items and considerations. If you are in the Life Sciences and writing up your methods section, try working through the MDAR checklist and see whether it helps you include all relevant details into your methods, and whether it reminded you of anything you might have missed otherwise.

Summary Writing tips

The main challenge you may find when writing your methods is keeping it readable AND covering all the details needed for reproducibility and replicability. While this is difficult, do not compromise on rigorous standards for credibility!

• Keep in mind future replicability, alongside understanding and readability.
• Follow checklists, and field- and journal-specific guidelines.
• Consider a commitment to rigorous and transparent science a personal responsibility, and not just adhering to journal guidelines.
• Establish whether there are persistent identifiers for any research resources you use that can be specifically cited in your methods section.
• Deposit your laboratory protocols in Protocols.io, establishing a permanent link to them. You can update your protocols later if you improve on them, as can future scientists who follow your protocols.
• Consider visual aids like flow-diagrams, lists, to help with reading other sections of the paper.
• Be specific about all decisions made during the experiments that someone reproducing your work would need to know.

Don’t

• Summarize or abbreviate methods without giving full details in a discoverable supplemental section.
• Presume you will always be able to remember how you performed the experiments, or have access to private or institutional notebooks and resources.
• Attempt to hide constraints or non-optimal decisions you had to make–transparency is the key to ensuring the credibility of your research.
• How to Write a Great Title
• How to Write an Abstract
• How to Report Statistics
• How to Write Discussions and Conclusions
• How to Edit Your Work

The contents of the Peer Review Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

The contents of the Writing Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

There’s a lot to consider when deciding where to submit your work. Learn how to choose a journal that will help your study reach its audience, while reflecting your values as a researcher…

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 6. The Methodology
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The methods section describes actions taken to investigate a research problem and the rationale for the application of specific procedures or techniques used to identify, select, process, and analyze information applied to understanding the problem, thereby, allowing the reader to critically evaluate a study’s overall validity and reliability. The methodology section of a research paper answers two main questions: How was the data collected or generated? And, how was it analyzed? The writing should be direct and precise and always written in the past tense.

Kallet, Richard H. "How to Write the Methods Section of a Research Paper." Respiratory Care 49 (October 2004): 1229-1232.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

• Readers need to know how the data was obtained because the method you chose affects the results and, by extension, how you interpreted their significance in the discussion section of your paper.
• Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and, as a consequence, undermines the value of your analysis of the findings.
• In most cases, there are a variety of different methods you can choose to investigate a research problem. The methodology section of your paper should clearly articulate the reasons why you have chosen a particular procedure or technique.
• The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a multiple choice questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
• The method must be appropriate to fulfilling the overall aims of the study. For example, you need to ensure that you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
• The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring. For any problems that do arise, you must describe the ways in which they were minimized or why these problems do not impact in any meaningful way your interpretation of the findings.
• In the social and behavioral sciences, it is important to always provide sufficient information to allow other researchers to adopt or replicate your methodology. This information is particularly important when a new method has been developed or an innovative use of an existing method is utilized.

Bem, Daryl J. Writing the Empirical Journal Article. Psychology Writing Center. University of Washington; Denscombe, Martyn. The Good Research Guide: For Small-Scale Social Research Projects . 5th edition. Buckingham, UK: Open University Press, 2014; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I.  Groups of Research Methods

There are two main groups of research methods in the social sciences:

• The e mpirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences . This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for formulating hypotheses that need to be tested. This approach is focused on explanation.
• The i nterpretative group of methods is focused on understanding phenomenon in a comprehensive, holistic way . Interpretive methods focus on analytically disclosing the meaning-making practices of human subjects [the why, how, or by what means people do what they do], while showing how those practices arrange so that it can be used to generate observable outcomes. Interpretive methods allow you to recognize your connection to the phenomena under investigation. However, the interpretative group requires careful examination of variables because it focuses more on subjective knowledge.

II.  Content

The introduction to your methodology section should begin by restating the research problem and underlying assumptions underpinning your study. This is followed by situating the methods you used to gather, analyze, and process information within the overall “tradition” of your field of study and within the particular research design you have chosen to study the problem. If the method you choose lies outside of the tradition of your field [i.e., your review of the literature demonstrates that the method is not commonly used], provide a justification for how your choice of methods specifically addresses the research problem in ways that have not been utilized in prior studies.

The remainder of your methodology section should describe the following:

• Decisions made in selecting the data you have analyzed or, in the case of qualitative research, the subjects and research setting you have examined,
• Tools and methods used to identify and collect information, and how you identified relevant variables,
• The ways in which you processed the data and the procedures you used to analyze that data, and
• The specific research tools or strategies that you utilized to study the underlying hypothesis and research questions.

In addition, an effectively written methodology section should:

• Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
• Indicate how the approach fits the overall research design . Your methods for gathering data should have a clear connection to your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is not suitable to achieving the stated objective of your paper.
• Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom. Also be sure to explain how older data is still relevant to investigating the current research problem.
• Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors? Describe how you plan to obtain an accurate assessment of relationships, patterns, trends, distributions, and possible contradictions found in the data.
• Provide background and a rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
• Provide a justification for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of data being used? If other data sources exist, explain why the data you chose is most appropriate to addressing the research problem.
• Provide a justification for case study selection . A common method of analyzing research problems in the social sciences is to analyze specific cases. These can be a person, place, event, phenomenon, or other type of subject of analysis that are either examined as a singular topic of in-depth investigation or multiple topics of investigation studied for the purpose of comparing or contrasting findings. In either method, you should explain why a case or cases were chosen and how they specifically relate to the research problem.
• Describe potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE :   Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic. If necessary, consider using appendices for raw data.

ANOTHER NOTE : If you are conducting a qualitative analysis of a research problem , the methodology section generally requires a more elaborate description of the methods used as well as an explanation of the processes applied to gathering and analyzing of data than is generally required for studies using quantitative methods. Because you are the primary instrument for generating the data [e.g., through interviews or observations], the process for collecting that data has a significantly greater impact on producing the findings. Therefore, qualitative research requires a more detailed description of the methods used.

YET ANOTHER NOTE :   If your study involves interviews, observations, or other qualitative techniques involving human subjects , you may be required to obtain approval from the university's Office for the Protection of Research Subjects before beginning your research. This is not a common procedure for most undergraduate level student research assignments. However, i f your professor states you need approval, you must include a statement in your methods section that you received official endorsement and adequate informed consent from the office and that there was a clear assessment and minimization of risks to participants and to the university. This statement informs the reader that your study was conducted in an ethical and responsible manner. In some cases, the approval notice is included as an appendix to your paper.

III.  Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but concise. Do not provide any background information that does not directly help the reader understand why a particular method was chosen, how the data was gathered or obtained, and how the data was analyzed in relation to the research problem [note: analyzed, not interpreted! Save how you interpreted the findings for the discussion section]. With this in mind, the page length of your methods section will generally be less than any other section of your paper except the conclusion.

Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. An exception to this rule is if you select an unconventional methodological approach; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall process of discovery.

Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data, or, gaps will exist in existing data or archival materials. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose.

Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in and of itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. "How to Write a Scientific Paper: Writing the Methods Section." Revista Portuguesa de Pneumologia 17 (2011): 232-238; Blair Lorrie. “Choosing a Methodology.” In Writing a Graduate Thesis or Dissertation , Teaching Writing Series. (Rotterdam: Sense Publishers 2016), pp. 49-72; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Kallet, Richard H. “How to Write the Methods Section of a Research Paper.” Respiratory Care 49 (October 2004):1229-1232; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Rudestam, Kjell Erik and Rae R. Newton. “The Method Chapter: Describing Your Research Plan.” In Surviving Your Dissertation: A Comprehensive Guide to Content and Process . (Thousand Oaks, Sage Publications, 2015), pp. 87-115; What is Interpretive Research. Institute of Public and International Affairs, University of Utah; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University; Methods and Materials. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

To locate data and statistics, GO HERE .

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between the application of theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics . Part 1, Chapter 3. Boise State University; The Theory-Method Relationship. S-Cool Revision. United Kingdom.

Yet Another Writing Tip

Methods and the Methodology

Do not confuse the terms "methods" and "methodology." As Schneider notes, a method refers to the technical steps taken to do research . Descriptions of methods usually include defining and stating why you have chosen specific techniques to investigate a research problem, followed by an outline of the procedures you used to systematically select, gather, and process the data [remember to always save the interpretation of data for the discussion section of your paper].

The methodology refers to a discussion of the underlying reasoning why particular methods were used . This discussion includes describing the theoretical concepts that inform the choice of methods to be applied, placing the choice of methods within the more general nature of academic work, and reviewing its relevance to examining the research problem. The methodology section also includes a thorough review of the methods other scholars have used to study the topic.

Bryman, Alan. "Of Methods and Methodology." Qualitative Research in Organizations and Management: An International Journal 3 (2008): 159-168; Schneider, Florian. “What's in a Methodology: The Difference between Method, Methodology, and Theory…and How to Get the Balance Right?” PoliticsEastAsia.com. Chinese Department, University of Leiden, Netherlands.

• << Previous: Scholarly vs. Popular Publications
• Next: Qualitative Methods >>
• Last Updated: Apr 29, 2024 1:49 PM
• URL: https://libguides.usc.edu/writingguide

How To Write The Methodology Chapter

The what, why & how explained simply (with examples).

By: Jenna Crossley (PhD) | Reviewed By: Dr. Eunice Rautenbach | September 2021 (Updated April 2023)

So, you’ve pinned down your research topic and undertaken a review of the literature – now it’s time to write up the methodology section of your dissertation, thesis or research paper . But what exactly is the methodology chapter all about – and how do you go about writing one? In this post, we’ll unpack the topic, step by step .

Overview: The Methodology Chapter

• The purpose  of the methodology chapter
• Why you need to craft this chapter (really) well
• How to write and structure the chapter
• Methodology chapter example
• Essential takeaways

What (exactly) is the methodology chapter?

The methodology chapter is where you outline the philosophical underpinnings of your research and outline the specific methodological choices you’ve made. The point of the methodology chapter is to tell the reader exactly how you designed your study and, just as importantly, why you did it this way.

Importantly, this chapter should comprehensively describe and justify all the methodological choices you made in your study. For example, the approach you took to your research (i.e., qualitative, quantitative or mixed), who  you collected data from (i.e., your sampling strategy), how you collected your data and, of course, how you analysed it. If that sounds a little intimidating, don’t worry – we’ll explain all these methodological choices in this post .

Why is the methodology chapter important?

The methodology chapter plays two important roles in your dissertation or thesis:

Firstly, it demonstrates your understanding of research theory, which is what earns you marks. A flawed research design or methodology would mean flawed results. So, this chapter is vital as it allows you to show the marker that you know what you’re doing and that your results are credible .

Secondly, the methodology chapter is what helps to make your study replicable. In other words, it allows other researchers to undertake your study using the same methodological approach, and compare their findings to yours. This is very important within academic research, as each study builds on previous studies.

The methodology chapter is also important in that it allows you to identify and discuss any methodological issues or problems you encountered (i.e., research limitations ), and to explain how you mitigated the impacts of these. Every research project has its limitations , so it’s important to acknowledge these openly and highlight your study’s value despite its limitations . Doing so demonstrates your understanding of research design, which will earn you marks. We’ll discuss limitations in a bit more detail later in this post, so stay tuned!

Need a helping hand?

How to write up the methodology chapter

First off, it’s worth noting that the exact structure and contents of the methodology chapter will vary depending on the field of research (e.g., humanities, chemistry or engineering) as well as the university . So, be sure to always check the guidelines provided by your institution for clarity and, if possible, review past dissertations from your university. Here we’re going to discuss a generic structure for a methodology chapter typically found in the sciences.

Before you start writing, it’s always a good idea to draw up a rough outline to guide your writing. Don’t just start writing without knowing what you’ll discuss where. If you do, you’ll likely end up with a disjointed, ill-flowing narrative . You’ll then waste a lot of time rewriting in an attempt to try to stitch all the pieces together. Do yourself a favour and start with the end in mind .

Section 1 – Introduction

As with all chapters in your dissertation or thesis, the methodology chapter should have a brief introduction. In this section, you should remind your readers what the focus of your study is, especially the research aims . As we’ve discussed many times on the blog, your methodology needs to align with your research aims, objectives and research questions. Therefore, it’s useful to frontload this component to remind the reader (and yourself!) what you’re trying to achieve.

In this section, you can also briefly mention how you’ll structure the chapter. This will help orient the reader and provide a bit of a roadmap so that they know what to expect. You don’t need a lot of detail here – just a brief outline will do.

Section 2 – The Methodology

The next section of your chapter is where you’ll present the actual methodology. In this section, you need to detail and justify the key methodological choices you’ve made in a logical, intuitive fashion. Importantly, this is the heart of your methodology chapter, so you need to get specific – don’t hold back on the details here. This is not one of those “less is more” situations.

Let’s take a look at the most common components you’ll likely need to cover. 

Methodological Choice #1 – Research Philosophy

Research philosophy refers to the underlying beliefs (i.e., the worldview) regarding how data about a phenomenon should be gathered , analysed and used . The research philosophy will serve as the core of your study and underpin all of the other research design choices, so it’s critically important that you understand which philosophy you’ll adopt and why you made that choice. If you’re not clear on this, take the time to get clarity before you make any further methodological choices.

While several research philosophies exist, two commonly adopted ones are positivism and interpretivism . These two sit roughly on opposite sides of the research philosophy spectrum.

Positivism states that the researcher can observe reality objectively and that there is only one reality, which exists independently of the observer. As a consequence, it is quite commonly the underlying research philosophy in quantitative studies and is oftentimes the assumed philosophy in the physical sciences.

Contrasted with this, interpretivism , which is often the underlying research philosophy in qualitative studies, assumes that the researcher performs a role in observing the world around them and that reality is unique to each observer . In other words, reality is observed subjectively .

These are just two philosophies (there are many more), but they demonstrate significantly different approaches to research and have a significant impact on all the methodological choices. Therefore, it’s vital that you clearly outline and justify your research philosophy at the beginning of your methodology chapter, as it sets the scene for everything that follows.

Methodological Choice #2 – Research Type

The next thing you would typically discuss in your methodology section is the research type. The starting point for this is to indicate whether the research you conducted is inductive or deductive .

Inductive research takes a bottom-up approach , where the researcher begins with specific observations or data and then draws general conclusions or theories from those observations. Therefore these studies tend to be exploratory in terms of approach.

Conversely , d eductive research takes a top-down approach , where the researcher starts with a theory or hypothesis and then tests it using specific observations or data. Therefore these studies tend to be confirmatory in approach.

Related to this, you’ll need to indicate whether your study adopts a qualitative, quantitative or mixed  approach. As we’ve mentioned, there’s a strong link between this choice and your research philosophy, so make sure that your choices are tightly aligned . When you write this section up, remember to clearly justify your choices, as they form the foundation of your study.

Methodological Choice #3 – Research Strategy

Next, you’ll need to discuss your research strategy (also referred to as a research design ). This methodological choice refers to the broader strategy in terms of how you’ll conduct your research, based on the aims of your study.

Several research strategies exist, including experimental , case studies , ethnography , grounded theory, action research , and phenomenology . Let’s take a look at two of these, experimental and ethnographic, to see how they contrast.

Experimental research makes use of the scientific method , where one group is the control group (in which no variables are manipulated ) and another is the experimental group (in which a specific variable is manipulated). This type of research is undertaken under strict conditions in a controlled, artificial environment (e.g., a laboratory). By having firm control over the environment, experimental research typically allows the researcher to establish causation between variables. Therefore, it can be a good choice if you have research aims that involve identifying causal relationships.

Ethnographic research , on the other hand, involves observing and capturing the experiences and perceptions of participants in their natural environment (for example, at home or in the office). In other words, in an uncontrolled environment.  Naturally, this means that this research strategy would be far less suitable if your research aims involve identifying causation, but it would be very valuable if you’re looking to explore and examine a group culture, for example.

As you can see, the right research strategy will depend largely on your research aims and research questions – in other words, what you’re trying to figure out. Therefore, as with every other methodological choice, it’s essential to justify why you chose the research strategy you did.

Methodological Choice #4 – Time Horizon

The next thing you’ll need to detail in your methodology chapter is the time horizon. There are two options here: cross-sectional and longitudinal . In other words, whether the data for your study were all collected at one point in time (cross-sectional) or at multiple points in time (longitudinal).

The choice you make here depends again on your research aims, objectives and research questions. If, for example, you aim to assess how a specific group of people’s perspectives regarding a topic change over time , you’d likely adopt a longitudinal time horizon.

Another important factor to consider is simply whether you have the time necessary to adopt a longitudinal approach (which could involve collecting data over multiple months or even years). Oftentimes, the time pressures of your degree program will force your hand into adopting a cross-sectional time horizon, so keep this in mind.

Methodological Choice #5 – Sampling Strategy

Next, you’ll need to discuss your sampling strategy . There are two main categories of sampling, probability and non-probability sampling.

Probability sampling involves a random (and therefore representative) selection of participants from a population, whereas non-probability sampling entails selecting participants in a non-random  (and therefore non-representative) manner. For example, selecting participants based on ease of access (this is called a convenience sample).

The right sampling approach depends largely on what you’re trying to achieve in your study. Specifically, whether you trying to develop findings that are generalisable to a population or not. Practicalities and resource constraints also play a large role here, as it can oftentimes be challenging to gain access to a truly random sample. In the video below, we explore some of the most common sampling strategies.

Methodological Choice #6 – Data Collection Method

Next up, you’ll need to explain how you’ll go about collecting the necessary data for your study. Your data collection method (or methods) will depend on the type of data that you plan to collect – in other words, qualitative or quantitative data.

Typically, quantitative research relies on surveys , data generated by lab equipment, analytics software or existing datasets. Qualitative research, on the other hand, often makes use of collection methods such as interviews , focus groups , participant observations, and ethnography.

So, as you can see, there is a tight link between this section and the design choices you outlined in earlier sections. Strong alignment between these sections, as well as your research aims and questions is therefore very important.

Methodological Choice #7 – Data Analysis Methods/Techniques

The final major methodological choice that you need to address is that of analysis techniques . In other words, how you’ll go about analysing your date once you’ve collected it. Here it’s important to be very specific about your analysis methods and/or techniques – don’t leave any room for interpretation. Also, as with all choices in this chapter, you need to justify each choice you make.

What exactly you discuss here will depend largely on the type of study you’re conducting (i.e., qualitative, quantitative, or mixed methods). For qualitative studies, common analysis methods include content analysis , thematic analysis and discourse analysis . In the video below, we explain each of these in plain language.

For quantitative studies, you’ll almost always make use of descriptive statistics , and in many cases, you’ll also use inferential statistical techniques (e.g., correlation and regression analysis). In the video below, we unpack some of the core concepts involved in descriptive and inferential statistics.

In this section of your methodology chapter, it’s also important to discuss how you prepared your data for analysis, and what software you used (if any). For example, quantitative data will often require some initial preparation such as removing duplicates or incomplete responses . Similarly, qualitative data will often require transcription and perhaps even translation. As always, remember to state both what you did and why you did it.

Section 3 – The Methodological Limitations

With the key methodological choices outlined and justified, the next step is to discuss the limitations of your design. No research methodology is perfect – there will always be trade-offs between the “ideal” methodology and what’s practical and viable, given your constraints. Therefore, this section of your methodology chapter is where you’ll discuss the trade-offs you had to make, and why these were justified given the context.

Methodological limitations can vary greatly from study to study, ranging from common issues such as time and budget constraints to issues of sample or selection bias . For example, you may find that you didn’t manage to draw in enough respondents to achieve the desired sample size (and therefore, statistically significant results), or your sample may be skewed heavily towards a certain demographic, thereby negatively impacting representativeness .

In this section, it’s important to be critical of the shortcomings of your study. There’s no use trying to hide them (your marker will be aware of them regardless). By being critical, you’ll demonstrate to your marker that you have a strong understanding of research theory, so don’t be shy here. At the same time, don’t beat your study to death . State the limitations, why these were justified, how you mitigated their impacts to the best degree possible, and how your study still provides value despite these limitations .

Section 4 – Concluding Summary

Finally, it’s time to wrap up the methodology chapter with a brief concluding summary. In this section, you’ll want to concisely summarise what you’ve presented in the chapter. Here, it can be a good idea to use a figure to summarise the key decisions, especially if your university recommends using a specific model (for example, Saunders’ Research Onion ).

Importantly, this section needs to be brief – a paragraph or two maximum (it’s a summary, after all). Also, make sure that when you write up your concluding summary, you include only what you’ve already discussed in your chapter; don’t add any new information.

Methodology Chapter Example

In the video below, we walk you through an example of a high-quality research methodology chapter from a dissertation. We also unpack our free methodology chapter template so that you can see how best to structure your chapter.

Wrapping Up

And there you have it – the methodology chapter in a nutshell. As we’ve mentioned, the exact contents and structure of this chapter can vary between universities , so be sure to check in with your institution before you start writing. If possible, try to find dissertations or theses from former students of your specific degree program – this will give you a strong indication of the expectations and norms when it comes to the methodology chapter (and all the other chapters!).

Also, remember the golden rule of the methodology chapter – justify every choice ! Make sure that you clearly explain the “why” for every “what”, and reference credible methodology textbooks or academic sources to back up your justifications.

If you need a helping hand with your research methodology (or any other component of your research), be sure to check out our private coaching service , where we hold your hand through every step of the research journey. Until next time, good luck!

Psst... there’s more!

This post was based on one of our popular Research Bootcamps . If you're working on a research project, you'll definitely want to check this out ...

You Might Also Like:

51 Comments

highly appreciated.

This was very helpful!

This was helpful

Thanks ,it is a very useful idea.

Thanks ,it is very useful idea.

Thank you so much, this information is very useful.

Thank you very much. I must say the information presented was succinct, coherent and invaluable. It is well put together and easy to comprehend. I have a great guide to create the research methodology for my dissertation.

Highly clear and useful.

I understand a bit on the explanation above. I want to have some coach but I’m still student and don’t have any budget to hire one. A lot of question I want to ask.

Thank you so much. This concluded my day plan. Thank you so much.

Thanks it was helpful

Great information. It would be great though if you could show us practical examples.

Thanks so much for this information. God bless and be with you

Thank you so so much. Indeed it was helpful

This is EXCELLENT!

I was totally confused by other explanations. Thank you so much!.

justdoing my research now , thanks for the guidance.

Thank uuuu! These contents are really valued for me!

This is powerful …I really like it

Highly useful and clear, thank you so much.

Highly appreciated. Good guide

That was helpful. Thanks

This is very useful.Thank you

Very helpful information. Thank you

This is exactly what I was looking for. The explanation is so detailed and easy to comprehend. Well done and thank you.

Great job. You just summarised everything in the easiest and most comprehensible way possible. Thanks a lot.

Thank you very much for the ideas you have given this will really help me a lot. Thank you and God Bless.

Such great effort …….very grateful thank you

Please accept my sincere gratitude. I have to say that the information that was delivered was congruent, concise, and quite helpful. It is clear and straightforward, making it simple to understand. I am in possession of an excellent manual that will assist me in developing the research methods for my dissertation.

Thank you for your great explanation. It really helped me construct my methodology paper.

thank you for simplifieng the methodoly, It was realy helpful

Very helpful!

Thank you for your great explanation.

The explanation I have been looking for. So clear Thank you

Thank you very much .this was more enlightening.

helped me create the in depth and thorough methodology for my dissertation

Thank you for the great explaination.please construct one methodology for me

I appreciate you for the explanation of methodology. Please construct one methodology on the topic: The effects influencing students dropout among schools for my thesis

This helped me complete my methods section of my dissertation with ease. I have managed to write a thorough and concise methodology!

its so good in deed

wow …what an easy to follow presentation. very invaluable content shared. utmost important.

Peace be upon you, I am Dr. Ahmed Khedr, a former part-time professor at Al-Azhar University in Cairo, Egypt. I am currently teaching research methods, and I have been dealing with your esteemed site for several years, and I found that despite my long experience with research methods sites, it is one of the smoothest sites for evaluating the material for students, For this reason, I relied on it a lot in teaching and translated most of what was written into Arabic and published it on my own page on Facebook. Thank you all… Everything I posted on my page is provided with the names of the writers of Grad coach, the title of the article, and the site. My best regards.

A remarkably simple and useful guide, thank you kindly.

I real appriciate your short and remarkable chapter summary

Bravo! Very helpful guide.

Only true experts could provide such helpful, fantastic, and inspiring knowledge about Methodology. Thank you very much! God be with you and us all!

highly appreciate your effort.

This is a very well thought out post. Very informative and a great read.

THANKS SO MUCH FOR SHARING YOUR NICE IDEA

I love you Emma, you are simply amazing with clear explanations with complete information. GradCoach really helped me to do my assignment here in Auckland. Mostly, Emma make it so simple and enjoyable

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Writing a scientific paper.

• Writing a lab report
• INTRODUCTION

Writing a "good" methods section

"methods checklist" from: how to write a good scientific paper. chris a. mack. spie. 2018..

• LITERATURE CITED
• Bibliography of guides to scientific writing and presenting
• Peer Review
• Presentations
• Lab Report Writing Guides on the Web

The purpose is to provide enough detail that a competent worker could repeat the experiment. Many of your readers will skip this section because they already know from the Introduction the general methods you used. However careful writing of this section is important because for your results to be of scientific merit they must be reproducible. Otherwise your paper does not represent good science.

• Exact technical specifications and quantities and source or method of preparation
• Describe equipment used and provide illustrations where relevant.
• Chronological presentation (but related methods described together)
• Questions about "how" and "how much" are answered for the reader and not left for them to puzzle over
• Discuss statistical methods only if unusual or advanced
• When a large number of components are used prepare tables for the benefit of the reader
• Do not state the action without stating the agent of the action
• Describe how the results were generated with sufficient detail so that an independent researcher (working in the same field) could reproduce the results sufficiently to allow validation of the conclusions.
• Can the reader assess internal validity (conclusions are supported by the results presented)?
• Can the reader assess external validity (conclusions are properly generalized beyond these specific results)?
• Has the chosen method been justified?
• Are data analysis and statistical approaches justified, with assumptions and biases considered?
• Avoid: including results in the Method section; including extraneous details (unnecessary to enable reproducibility or judge validity); treating the method as a chronological history of events; unneeded references to commercial products; references to “proprietary” products or processes unavailable to the reader. 
• << Previous: INTRODUCTION
• Next: RESULTS >>
• Last Updated: Aug 4, 2023 9:33 AM
• URL: https://guides.lib.uci.edu/scientificwriting

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

The Plagiarism Checker Online For Your Academic Work

Start Plagiarism Check

Editing & Proofreading for Your Research Paper

Get it proofread now

Online Printing & Binding with Free Express Delivery

Configure binding now

• Academic essay overview
• The writing process
• Structuring academic essays
• Types of academic essays
• Academic writing overview
• Sentence structure
• Academic writing process
• Improving your academic writing
• Titles and headings
• APA style overview
• APA citation & referencing
• APA structure & sections
• Citation & referencing
• Structure and sections
• APA examples overview
• Commonly used citations
• Other examples
• British English vs. American English
• Chicago style overview
• Chicago citation & referencing
• Chicago structure & sections
• Chicago style examples
• Citing sources overview
• Citation format
• Citation examples
• College essay overview
• Application
• How to write a college essay
• Types of college essays
• Commonly confused words
• Definitions
• Dissertation overview
• Dissertation structure & sections
• Dissertation writing process
• Graduate school overview
• Application & admission
• Study abroad
• Master degree
• Harvard referencing overview
• Language rules overview
• Grammatical rules & structures
• Parts of speech
• Punctuation
• Methodology overview
• Analyzing data
• Experiments
• Observations
• Inductive vs. Deductive
• Qualitative vs. Quantitative
• Types of validity
• Types of reliability
• Sampling methods
• Theories & Concepts
• Types of research studies
• Types of variables
• MLA style overview
• MLA examples
• MLA citation & referencing
• MLA structure & sections
• Plagiarism overview
• Plagiarism checker
• Types of plagiarism
• Printing production overview
• Research bias overview
• Types of research bias
• Example sections
• Types of research papers
• Research process overview
• Problem statement
• Research proposal
• Research topic
• Statistics overview
• Levels of measurment
• Frequency distribution
• Measures of central tendency
• Measures of variability
• Hypothesis testing
• Parameters & test statistics
• Types of distributions
• Correlation
• Effect size
• Hypothesis testing assumptions
• Types of ANOVAs
• Types of chi-square
• Statistical data
• Statistical models
• Spelling mistakes
• Tips overview
• Academic writing tips
• Dissertation tips
• Sources tips
• Working with sources overview
• Evaluating sources
• Finding sources
• Including sources
• Types of sources

Your Step to Success

Plagiarism Check within 10min

Printing & Binding with 3D Live Preview

APA Methods Section – How To Write It With Examples

How do you like this article cancel reply.

Save my name, email, and website in this browser for the next time I comment.

The APA methods section is a very important part of your academic paper, displaying how you conducted your research by providing a precise description of the methods and procedures you used for the study. This section ensures transparency, allowing other researchers to see exactly how you conducted your experiments. In APA style , the methods section usually includes subsections on participants, materials or measures, and procedures. This article discusses the APA methods section in detail.

Inhaltsverzeichnis

• 1 APA Methods Section – In a Nutshell
• 2 Definition: APA Methods Section
• 3 APA Methods Section: Structure
• 4 APA Methods Section: Participants
• 5 APA Methods Section: Materials
• 6 APA Methods Section: Procedure

APA Methods Section – In a Nutshell

• The APA methods section covers the participants, materials, and procedures.
• Under the ‘Participants’ heading of the APA methods section, you should state the relevant demographic characteristics of your participants.
• Accurately reporting the facts of the study can help other researchers determine how much the results can be generalized.

Definition: APA Methods Section

The APA methods section describes the procedures you used to carry out your research and explains why particular processes were selected. It allows other researchers to replicate the study and make their own conclusions on the validity of the experiment.

APA Methods Section: Structure

• The main heading of the APA methods section should be written in bold and should be capitalized. It also has to be centered.
• All subheadings should be aligned to the left and must be boldfaced. You should select subheadings that are suitable for your essay, and the most commonly used include ‘Participants’, ‘Materials’, and ‘Procedure’.

Heading formats:

APA format has certain requirements for reporting different research designs. You should go through these guidelines to determine what you should mention for research using longitudinal designs , replication studies, and experimental designs .

APA Methods Section: Participants

Under this subheading, you will have to report on the sample characteristics, the procedures used to collect samples, and the sample size selected.

Subject or Participant Characteristics

In academic studies, ‘participants’ refers to the people who take part in a study. If animals are used instead of human beings, the researcher can use the term ‘subjects’. In this subheading of the APA methods section, you have to describe the demographic characteristics of the participants, including their age, sex, race, ethnic group, education level, and gender identity. Depending on the nature of the study, other characteristics may be important. Some of these include:

• Education levels
• Language preference
• Immigration status

By describing the characteristics of the participants, readers will be able to determine how much the results can be generalized. Make sure you use bias-free language when writing this part of the APA methods section.

The study included 100 homosexual men and 100 homosexual women aged between 30 and 50 years from the city of London, UK.

Sampling Procedures

When selecting participants for your study, you will have to use certain sampling procedures. If the study could access all members of the population, you can say that you used random sampling methods. This section of the APA methods section should cover the percentage of respondents who participated in the research, and how they were chosen. You also need to state how participants were compensated and the ethical standard followed.

• Transgender male students from London were invited to participate in a study.
• Invites were sent to the students via email, social media posts, and posters in the schools.
• Each participant received $10 for the time spent in the study.
• The research obtained ethical approval before the participants were recruited.

Sample Size and Statistical Power

In this part of the APA methods section, you should give details on the sample size and statistical power you aimed at achieving. You should mention whether the final sample was the same as the intended sample. This section should show whether your research had enough statistical power to find any effects.

• The study aimed at a statistical power of 75% to detect an effect of 10% with an alpha of .05.
• 200 participants were required, and the study fulfilled these conditions.

APA Methods Section: Materials

Readers also need to know the materials you used for the study. This part of the APA methods section will give other researchers a good picture of the methods used to conduct the study.

Primary and secondary measures

Here, you should indicate the instruments used in the study, as well as the constructs they were meant to measure. Some of these are inventories, scales, tests, software, and hardware. Make sure you cover the following aspects:

• Reliability
• The Traumatic Stress Schedule (TSS) was used to measure the exposure to traumatic events.
• This 10-item chart requires participants to report lifelong exposure to traumatic stress.
• For example, they could indicate whether they suffered the traumatic death of a loved one.
• The Davidson Trauma Scale was also used to assess the symptoms of trauma.

Under this subheading of the APA methods section, you should also mention covariates or additional variables that can explain the outcomes.

Quality of measurements

You can mention the strategies you applied to ensure data integrity and reliability. These may include:

• Training the interviewers
• Establishing clear data nominalization procedures
• Rigorous data handling and analysis processes
• Having multiple people assess the data

If the data was subjectively coded, you should indicate the interrater reliability scores in the APA methods section.

• ✓ Post a picture on Instagram
• ✓ Get the most likes on your picture
• ✓ Receive up to $300 cash back

APA Methods Section: Procedure

This part of the APA methods section indicates the methods you used to carry out the research, process the data, and analyze the results.

Research Design and Data Collection Methods

Data collection is the systematic gathering of observations and measurements, and you have to describe all procedures used in this process. You can use supplementary materials to describe long and complicated data collection methods.

When reporting the research design, you should mention the framework of the study. This could be experimental, longitudinal, correlational, or descriptive. Additionally, you should mention whether you used a between-subjects design or within-subjects design .

In this part of the APA methods section, you should also mention whether any masking methods were used to hide condition assignments from the participants.

• Participants are told the research takes an hour covers their personal experiences in school.
• They were assured that the reports would be confidential and were asked to give consent.
• The participants were asked to fill in questionnaires .
• The control group was given an unrelated filler task, after which they filled a questionnaire.
• It was determined the experiences of homosexual and CIS-gendered students varied.

Data diagnostics

This part of the APA method section outlines the steps taken to process the data. It includes:

• Methods of identifying and controlling outliers
• Data transformation procedures
• Methods of compensating for missing values

Analytic strategies

This subheading of the APA methods section describes the analytic strategies used, but you shouldn’t mention the outcomes. The primary and secondary hypotheses use past studies or theoretical frameworks , while exploratory hypotheses focus on the data in the study.

We started by assessing the demographic differences between the two groups. We also performed an independent samples t-test on the test scores .

What are the parts of an APA methods section?

In this section, you should include the study participants, the methods used, and the procedures.

What is included in the APA methods section?

The methods section covers the participants or subject characteristics, the sampling procedures, the sample size, the measures used, the data collection methods, the research design, the data analysis strategy, and the data processing method.

Should I use the Oxford comma when writing the APA methods section?

Yes, the serial comma is required when writing the APA methods section.

Should I use the first person to write the APA methods section?

Yes, the APA language guidelines encourage researchers to use first-person pronouns when writing the methods section.

We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.

• External Media

Individual Privacy Preferences

Cookie Details Privacy Policy Imprint

Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

Accept all Save

Essential cookies enable basic functions and are necessary for the proper function of the website.

Show Cookie Information Hide Cookie Information

Statistics cookies collect information anonymously. This information helps us to understand how our visitors use our website.

Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

Privacy Policy Imprint

• Privacy Policy

Home » Research Methods – Types, Examples and Guide

Research Methods – Types, Examples and Guide

Table of Contents

Research Methods

Definition:

Research Methods refer to the techniques, procedures, and processes used by researchers to collect , analyze, and interpret data in order to answer research questions or test hypotheses. The methods used in research can vary depending on the research questions, the type of data that is being collected, and the research design.

Types of Research Methods

Types of Research Methods are as follows:

Qualitative research Method

Qualitative research methods are used to collect and analyze non-numerical data. This type of research is useful when the objective is to explore the meaning of phenomena, understand the experiences of individuals, or gain insights into complex social processes. Qualitative research methods include interviews, focus groups, ethnography, and content analysis.

Quantitative Research Method

Quantitative research methods are used to collect and analyze numerical data. This type of research is useful when the objective is to test a hypothesis, determine cause-and-effect relationships, and measure the prevalence of certain phenomena. Quantitative research methods include surveys, experiments, and secondary data analysis.

Mixed Method Research

Mixed Method Research refers to the combination of both qualitative and quantitative research methods in a single study. This approach aims to overcome the limitations of each individual method and to provide a more comprehensive understanding of the research topic. This approach allows researchers to gather both quantitative data, which is often used to test hypotheses and make generalizations about a population, and qualitative data, which provides a more in-depth understanding of the experiences and perspectives of individuals.

Key Differences Between Research Methods

The following Table shows the key differences between Quantitative, Qualitative and Mixed Research Methods

Examples of Research Methods

Examples of Research Methods are as follows:

Qualitative Research Example:

A researcher wants to study the experience of cancer patients during their treatment. They conduct in-depth interviews with patients to gather data on their emotional state, coping mechanisms, and support systems.

Quantitative Research Example:

A company wants to determine the effectiveness of a new advertisement campaign. They survey a large group of people, asking them to rate their awareness of the product and their likelihood of purchasing it.

Mixed Research Example:

A university wants to evaluate the effectiveness of a new teaching method in improving student performance. They collect both quantitative data (such as test scores) and qualitative data (such as feedback from students and teachers) to get a complete picture of the impact of the new method.

Applications of Research Methods

Research methods are used in various fields to investigate, analyze, and answer research questions. Here are some examples of how research methods are applied in different fields:

• Psychology : Research methods are widely used in psychology to study human behavior, emotions, and mental processes. For example, researchers may use experiments, surveys, and observational studies to understand how people behave in different situations, how they respond to different stimuli, and how their brains process information.
• Sociology : Sociologists use research methods to study social phenomena, such as social inequality, social change, and social relationships. Researchers may use surveys, interviews, and observational studies to collect data on social attitudes, beliefs, and behaviors.
• Medicine : Research methods are essential in medical research to study diseases, test new treatments, and evaluate their effectiveness. Researchers may use clinical trials, case studies, and laboratory experiments to collect data on the efficacy and safety of different medical treatments.
• Education : Research methods are used in education to understand how students learn, how teachers teach, and how educational policies affect student outcomes. Researchers may use surveys, experiments, and observational studies to collect data on student performance, teacher effectiveness, and educational programs.
• Business : Research methods are used in business to understand consumer behavior, market trends, and business strategies. Researchers may use surveys, focus groups, and observational studies to collect data on consumer preferences, market trends, and industry competition.
• Environmental science : Research methods are used in environmental science to study the natural world and its ecosystems. Researchers may use field studies, laboratory experiments, and observational studies to collect data on environmental factors, such as air and water quality, and the impact of human activities on the environment.
• Political science : Research methods are used in political science to study political systems, institutions, and behavior. Researchers may use surveys, experiments, and observational studies to collect data on political attitudes, voting behavior, and the impact of policies on society.

Purpose of Research Methods

Research methods serve several purposes, including:

• Identify research problems: Research methods are used to identify research problems or questions that need to be addressed through empirical investigation.
• Develop hypotheses: Research methods help researchers develop hypotheses, which are tentative explanations for the observed phenomenon or relationship.
• Collect data: Research methods enable researchers to collect data in a systematic and objective way, which is necessary to test hypotheses and draw meaningful conclusions.
• Analyze data: Research methods provide tools and techniques for analyzing data, such as statistical analysis, content analysis, and discourse analysis.
• Test hypotheses: Research methods allow researchers to test hypotheses by examining the relationships between variables in a systematic and controlled manner.
• Draw conclusions : Research methods facilitate the drawing of conclusions based on empirical evidence and help researchers make generalizations about a population based on their sample data.
• Enhance understanding: Research methods contribute to the development of knowledge and enhance our understanding of various phenomena and relationships, which can inform policy, practice, and theory.

When to Use Research Methods

Research methods are used when you need to gather information or data to answer a question or to gain insights into a particular phenomenon.

Here are some situations when research methods may be appropriate:

• To investigate a problem : Research methods can be used to investigate a problem or a research question in a particular field. This can help in identifying the root cause of the problem and developing solutions.
• To gather data: Research methods can be used to collect data on a particular subject. This can be done through surveys, interviews, observations, experiments, and more.
• To evaluate programs : Research methods can be used to evaluate the effectiveness of a program, intervention, or policy. This can help in determining whether the program is meeting its goals and objectives.
• To explore new areas : Research methods can be used to explore new areas of inquiry or to test new hypotheses. This can help in advancing knowledge in a particular field.
• To make informed decisions : Research methods can be used to gather information and data to support informed decision-making. This can be useful in various fields such as healthcare, business, and education.

Advantages of Research Methods

Research methods provide several advantages, including:

• Objectivity : Research methods enable researchers to gather data in a systematic and objective manner, minimizing personal biases and subjectivity. This leads to more reliable and valid results.
• Replicability : A key advantage of research methods is that they allow for replication of studies by other researchers. This helps to confirm the validity of the findings and ensures that the results are not specific to the particular research team.
• Generalizability : Research methods enable researchers to gather data from a representative sample of the population, allowing for generalizability of the findings to a larger population. This increases the external validity of the research.
• Precision : Research methods enable researchers to gather data using standardized procedures, ensuring that the data is accurate and precise. This allows researchers to make accurate predictions and draw meaningful conclusions.
• Efficiency : Research methods enable researchers to gather data efficiently, saving time and resources. This is especially important when studying large populations or complex phenomena.
• Innovation : Research methods enable researchers to develop new techniques and tools for data collection and analysis, leading to innovation and advancement in the field.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Get science-backed answers as you write with Paperpal's Research feature

How to Write a Hypothesis? Types and Examples 

All research studies involve the use of the scientific method, which is a mathematical and experimental technique used to conduct experiments by developing and testing a hypothesis or a prediction about an outcome. Simply put, a hypothesis is a suggested solution to a problem. It includes elements that are expressed in terms of relationships with each other to explain a condition or an assumption that hasn’t been verified using facts. 1 The typical steps in a scientific method include developing such a hypothesis, testing it through various methods, and then modifying it based on the outcomes of the experiments.  

A research hypothesis can be defined as a specific, testable prediction about the anticipated results of a study. 2 Hypotheses help guide the research process and supplement the aim of the study. After several rounds of testing, hypotheses can help develop scientific theories. 3 Hypotheses are often written as if-then statements. 

Here are two hypothesis examples: 

Dandelions growing in nitrogen-rich soils for two weeks develop larger leaves than those in nitrogen-poor soils because nitrogen stimulates vegetative growth. 4  

If a company offers flexible work hours, then their employees will be happier at work. 5  

Table of Contents

• What is a hypothesis? 
• Types of hypotheses 
• Characteristics of a hypothesis 
• Functions of a hypothesis 
• How to write a hypothesis 
• Hypothesis examples 
• Frequently asked questions 

What is a hypothesis?

A hypothesis expresses an expected relationship between variables in a study and is developed before conducting any research. Hypotheses are not opinions but rather are expected relationships based on facts and observations. They help support scientific research and expand existing knowledge. An incorrectly formulated hypothesis can affect the entire experiment leading to errors in the results so it’s important to know how to formulate a hypothesis and develop it carefully.

A few sources of a hypothesis include observations from prior studies, current research and experiences, competitors, scientific theories, and general conditions that can influence people. Figure 1 depicts the different steps in a research design and shows where exactly in the process a hypothesis is developed. 4  

There are seven different types of hypotheses—simple, complex, directional, nondirectional, associative and causal, null, and alternative. 

Types of hypotheses

The seven types of hypotheses are listed below: 5 , 6,7  

• Simple : Predicts the relationship between a single dependent variable and a single independent variable. 

Example: Exercising in the morning every day will increase your productivity.  

• Complex : Predicts the relationship between two or more variables. 

Example: Spending three hours or more on social media daily will negatively affect children’s mental health and productivity, more than that of adults.  

• Directional : Specifies the expected direction to be followed and uses terms like increase, decrease, positive, negative, more, or less. 

Example: The inclusion of intervention X decreases infant mortality compared to the original treatment.  

• Non-directional : Does not predict the exact direction, nature, or magnitude of the relationship between two variables but rather states the existence of a relationship. This hypothesis may be used when there is no underlying theory or if findings contradict prior research. 

Example: Cats and dogs differ in the amount of affection they express.  

• Associative and causal : An associative hypothesis suggests an interdependency between variables, that is, how a change in one variable changes the other.  

Example: There is a positive association between physical activity levels and overall health.  

A causal hypothesis, on the other hand, expresses a cause-and-effect association between variables. 

Example: Long-term alcohol use causes liver damage.  

• Null : Claims that the original hypothesis is false by showing that there is no relationship between the variables. 

Example: Sleep duration does not have any effect on productivity.  

• Alternative : States the opposite of the null hypothesis, that is, a relationship exists between two variables. 

Example: Sleep duration affects productivity.  

Characteristics of a hypothesis

So, what makes a good hypothesis? Here are some important characteristics of a hypothesis. 8,9  

• Testable : You must be able to test the hypothesis using scientific methods to either accept or reject the prediction. 
• Falsifiable : It should be possible to collect data that reject rather than support the hypothesis. 
• Logical : Hypotheses shouldn’t be a random guess but rather should be based on previous theories, observations, prior research, and logical reasoning. 
• Positive : The hypothesis statement about the existence of an association should be positive, that is, it should not suggest that an association does not exist. Therefore, the language used and knowing how to phrase a hypothesis is very important. 
• Clear and accurate : The language used should be easily comprehensible and use correct terminology. 
• Relevant : The hypothesis should be relevant and specific to the research question. 
• Structure : Should include all the elements that make a good hypothesis: variables, relationship, and outcome. 

Functions of a hypothesis

The following list mentions some important functions of a hypothesis: 1  

• Maintains the direction and progress of the research. 
• Expresses the important assumptions underlying the proposition in a single statement. 
• Establishes a suitable context for researchers to begin their investigation and for readers who are referring to the final report. 
• Provides an explanation for the occurrence of a specific phenomenon. 
• Ensures selection of appropriate and accurate facts necessary and relevant to the research subject. 

To summarize, a hypothesis provides the conceptual elements that complete the known data, conceptual relationships that systematize unordered elements, and conceptual meanings and interpretations that explain the unknown phenomena. 1  

How to write a hypothesis

Listed below are the main steps explaining how to write a hypothesis. 2,4,5  

• Make an observation and identify variables : Observe the subject in question and try to recognize a pattern or a relationship between the variables involved. This step provides essential background information to begin your research.  

For example, if you notice that an office’s vending machine frequently runs out of a specific snack, you may predict that more people in the office choose that snack over another. 

• Identify the main research question : After identifying a subject and recognizing a pattern, the next step is to ask a question that your hypothesis will answer.  

For example, after observing employees’ break times at work, you could ask “why do more employees take breaks in the morning rather than in the afternoon?” 

• Conduct some preliminary research to ensure originality and novelty : Your initial answer, which is your hypothesis, to the question is based on some pre-existing information about the subject. However, to ensure that your hypothesis has not been asked before or that it has been asked but rejected by other researchers you would need to gather additional information.  

For example, based on your observations you might state a hypothesis that employees work more efficiently when the air conditioning in the office is set at a lower temperature. However, during your preliminary research you find that this hypothesis was proven incorrect by a prior study. 

• Develop a general statement : After your preliminary research has confirmed the originality of your proposed answer, draft a general statement that includes all variables, subjects, and predicted outcome. The statement could be if/then or declarative.  
• Finalize the hypothesis statement : Use the PICOT model, which clarifies how to word a hypothesis effectively, when finalizing the statement. This model lists the important components required to write a hypothesis. 

P opulation: The specific group or individual who is the main subject of the research 

I nterest: The main concern of the study/research question 

C omparison: The main alternative group 

O utcome: The expected results  

T ime: Duration of the experiment 

Once you’ve finalized your hypothesis statement you would need to conduct experiments to test whether the hypothesis is true or false. 

Hypothesis examples

The following table provides examples of different types of hypotheses. 10 ,11  

Key takeaways  

Here’s a summary of all the key points discussed in this article about how to write a hypothesis. 

• A hypothesis is an assumption about an association between variables made based on limited evidence, which should be tested. 
• A hypothesis has four parts—the research question, independent variable, dependent variable, and the proposed relationship between the variables.   
• The statement should be clear, concise, testable, logical, and falsifiable. 
• There are seven types of hypotheses—simple, complex, directional, non-directional, associative and causal, null, and alternative. 
• A hypothesis provides a focus and direction for the research to progress. 
• A hypothesis plays an important role in the scientific method by helping to create an appropriate experimental design. 

Frequently asked questions

Hypotheses and research questions have different objectives and structure. The following table lists some major differences between the two. 9  

Here are a few examples to differentiate between a research question and hypothesis. 

Yes, here’s a simple checklist to help you gauge the effectiveness of your hypothesis. 9   1. When writing a hypothesis statement, check if it:  2. Predicts the relationship between the stated variables and the expected outcome.  3. Uses simple and concise language and is not wordy.  4. Does not assume readers’ knowledge about the subject.  5. Has observable, falsifiable, and testable results. 

As mentioned earlier in this article, a hypothesis is an assumption or prediction about an association between variables based on observations and simple evidence. These statements are usually generic. Research objectives, on the other hand, are more specific and dictated by hypotheses. The same hypothesis can be tested using different methods and the research objectives could be different in each case.     For example, Louis Pasteur observed that food lasts longer at higher altitudes, reasoned that it could be because the air at higher altitudes is cleaner (with fewer or no germs), and tested the hypothesis by exposing food to air cleaned in the laboratory. 12 Thus, a hypothesis is predictive—if the reasoning is correct, X will lead to Y—and research objectives are developed to test these predictions. 

Null hypothesis testing is a method to decide between two assumptions or predictions between variables (null and alternative hypotheses) in a statistical relationship in a sample. The null hypothesis, denoted as H 0 , claims that no relationship exists between variables in a population and any relationship in the sample reflects a sampling error or occurrence by chance. The alternative hypothesis, denoted as H 1 , claims that there is a relationship in the population. In every study, researchers need to decide whether the relationship in a sample occurred by chance or reflects a relationship in the population. This is done by hypothesis testing using the following steps: 13   1. Assume that the null hypothesis is true.  2. Determine how likely the sample relationship would be if the null hypothesis were true. This probability is called the p value.  3. If the sample relationship would be extremely unlikely, reject the null hypothesis and accept the alternative hypothesis. If the relationship would not be unlikely, accept the null hypothesis. 

To summarize, researchers should know how to write a good hypothesis to ensure that their research progresses in the required direction. A hypothesis is a testable prediction about any behavior or relationship between variables, usually based on facts and observation, and states an expected outcome.  

We hope this article has provided you with essential insight into the different types of hypotheses and their functions so that you can use them appropriately in your next research project. 

References  

• Dalen, DVV. The function of hypotheses in research. Proquest website. Accessed April 8, 2024. https://www.proquest.com/docview/1437933010?pq-origsite=gscholar&fromopenview=true&sourcetype=Scholarly%20Journals&imgSeq=1  
• McLeod S. Research hypothesis in psychology: Types & examples. SimplyPsychology website. Updated December 13, 2023. Accessed April 9, 2024. https://www.simplypsychology.org/what-is-a-hypotheses.html  
• Scientific method. Britannica website. Updated March 14, 2024. Accessed April 9, 2024. https://www.britannica.com/science/scientific-method  
• The hypothesis in science writing. Accessed April 10, 2024. https://berks.psu.edu/sites/berks/files/campus/HypothesisHandout_Final.pdf  
• How to develop a hypothesis (with elements, types, and examples). Indeed.com website. Updated February 3, 2023. Accessed April 10, 2024. https://www.indeed.com/career-advice/career-development/how-to-write-a-hypothesis  
• Types of research hypotheses. Excelsior online writing lab. Accessed April 11, 2024. https://owl.excelsior.edu/research/research-hypotheses/types-of-research-hypotheses/  
• What is a research hypothesis: how to write it, types, and examples. Researcher.life website. Published February 8, 2023. Accessed April 11, 2024. https://researcher.life/blog/article/how-to-write-a-research-hypothesis-definition-types-examples/  
• Developing a hypothesis. Pressbooks website. Accessed April 12, 2024. https://opentext.wsu.edu/carriecuttler/chapter/developing-a-hypothesis/  
• What is and how to write a good hypothesis in research. Elsevier author services website. Accessed April 12, 2024. https://scientific-publishing.webshop.elsevier.com/manuscript-preparation/what-how-write-good-hypothesis-research/  
• How to write a great hypothesis. Verywellmind website. Updated March 12, 2023. Accessed April 13, 2024. https://www.verywellmind.com/what-is-a-hypothesis-2795239  
• 15 Hypothesis examples. Helpfulprofessor.com Published September 8, 2023. Accessed March 14, 2024. https://helpfulprofessor.com/hypothesis-examples/ 
• Editage insights. What is the interconnectivity between research objectives and hypothesis? Published February 24, 2021. Accessed April 13, 2024. https://www.editage.com/insights/what-is-the-interconnectivity-between-research-objectives-and-hypothesis  
• Understanding null hypothesis testing. BCCampus open publishing. Accessed April 16, 2024. https://opentextbc.ca/researchmethods/chapter/understanding-null-hypothesis-testing/#:~:text=In%20null%20hypothesis%20testing%2C%20this,said%20to%20be%20statistically%20significant  

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• What is a Literature Review? How to Write It (with Examples)
• What are Journal Guidelines on Using Generative AI Tools

Measuring Academic Success: Definition & Strategies for Excellence

What are scholarly sources and where can you find them , you may also like, 4 ways paperpal encourages responsible writing with ai, what are scholarly sources and where can you..., what is academic writing: tips for students, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift..., how to use paperpal to generate emails &....

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

Writing Survey Questions

Perhaps the most important part of the survey process is the creation of questions that accurately measure the opinions, experiences and behaviors of the public. Accurate random sampling will be wasted if the information gathered is built on a shaky foundation of ambiguous or biased questions. Creating good measures involves both writing good questions and organizing them to form the questionnaire.

Questionnaire design is a multistage process that requires attention to many details at once. Designing the questionnaire is complicated because surveys can ask about topics in varying degrees of detail, questions can be asked in different ways, and questions asked earlier in a survey may influence how people respond to later questions. Researchers are also often interested in measuring change over time and therefore must be attentive to how opinions or behaviors have been measured in prior surveys.

Surveyors may conduct pilot tests or focus groups in the early stages of questionnaire development in order to better understand how people think about an issue or comprehend a question. Pretesting a survey is an essential step in the questionnaire design process to evaluate how people respond to the overall questionnaire and specific questions, especially when questions are being introduced for the first time.

For many years, surveyors approached questionnaire design as an art, but substantial research over the past forty years has demonstrated that there is a lot of science involved in crafting a good survey questionnaire. Here, we discuss the pitfalls and best practices of designing questionnaires.

Question development

There are several steps involved in developing a survey questionnaire. The first is identifying what topics will be covered in the survey. For Pew Research Center surveys, this involves thinking about what is happening in our nation and the world and what will be relevant to the public, policymakers and the media. We also track opinion on a variety of issues over time so we often ensure that we update these trends on a regular basis to better understand whether people’s opinions are changing.

At Pew Research Center, questionnaire development is a collaborative and iterative process where staff meet to discuss drafts of the questionnaire several times over the course of its development. We frequently test new survey questions ahead of time through qualitative research methods such as  focus groups , cognitive interviews, pretesting (often using an  online, opt-in sample ), or a combination of these approaches. Researchers use insights from this testing to refine questions before they are asked in a production survey, such as on the ATP.

Measuring change over time

Many surveyors want to track changes over time in people’s attitudes, opinions and behaviors. To measure change, questions are asked at two or more points in time. A cross-sectional design surveys different people in the same population at multiple points in time. A panel, such as the ATP, surveys the same people over time. However, it is common for the set of people in survey panels to change over time as new panelists are added and some prior panelists drop out. Many of the questions in Pew Research Center surveys have been asked in prior polls. Asking the same questions at different points in time allows us to report on changes in the overall views of the general public (or a subset of the public, such as registered voters, men or Black Americans), or what we call “trending the data”.

When measuring change over time, it is important to use the same question wording and to be sensitive to where the question is asked in the questionnaire to maintain a similar context as when the question was asked previously (see  question wording  and  question order  for further information). All of our survey reports include a topline questionnaire that provides the exact question wording and sequencing, along with results from the current survey and previous surveys in which we asked the question.

The Center’s transition from conducting U.S. surveys by live telephone interviewing to an online panel (around 2014 to 2020) complicated some opinion trends, but not others. Opinion trends that ask about sensitive topics (e.g., personal finances or attending religious services ) or that elicited volunteered answers (e.g., “neither” or “don’t know”) over the phone tended to show larger differences than other trends when shifting from phone polls to the online ATP. The Center adopted several strategies for coping with changes to data trends that may be related to this change in methodology. If there is evidence suggesting that a change in a trend stems from switching from phone to online measurement, Center reports flag that possibility for readers to try to head off confusion or erroneous conclusions.

Open- and closed-ended questions

One of the most significant decisions that can affect how people answer questions is whether the question is posed as an open-ended question, where respondents provide a response in their own words, or a closed-ended question, where they are asked to choose from a list of answer choices.

For example, in a poll conducted after the 2008 presidential election, people responded very differently to two versions of the question: “What one issue mattered most to you in deciding how you voted for president?” One was closed-ended and the other open-ended. In the closed-ended version, respondents were provided five options and could volunteer an option not on the list.

When explicitly offered the economy as a response, more than half of respondents (58%) chose this answer; only 35% of those who responded to the open-ended version volunteered the economy. Moreover, among those asked the closed-ended version, fewer than one-in-ten (8%) provided a response other than the five they were read. By contrast, fully 43% of those asked the open-ended version provided a response not listed in the closed-ended version of the question. All of the other issues were chosen at least slightly more often when explicitly offered in the closed-ended version than in the open-ended version. (Also see  “High Marks for the Campaign, a High Bar for Obama”  for more information.)

Researchers will sometimes conduct a pilot study using open-ended questions to discover which answers are most common. They will then develop closed-ended questions based off that pilot study that include the most common responses as answer choices. In this way, the questions may better reflect what the public is thinking, how they view a particular issue, or bring certain issues to light that the researchers may not have been aware of.

When asking closed-ended questions, the choice of options provided, how each option is described, the number of response options offered, and the order in which options are read can all influence how people respond. One example of the impact of how categories are defined can be found in a Pew Research Center poll conducted in January 2002. When half of the sample was asked whether it was “more important for President Bush to focus on domestic policy or foreign policy,” 52% chose domestic policy while only 34% said foreign policy. When the category “foreign policy” was narrowed to a specific aspect – “the war on terrorism” – far more people chose it; only 33% chose domestic policy while 52% chose the war on terrorism.

In most circumstances, the number of answer choices should be kept to a relatively small number – just four or perhaps five at most – especially in telephone surveys. Psychological research indicates that people have a hard time keeping more than this number of choices in mind at one time. When the question is asking about an objective fact and/or demographics, such as the religious affiliation of the respondent, more categories can be used. In fact, they are encouraged to ensure inclusivity. For example, Pew Research Center’s standard religion questions include more than 12 different categories, beginning with the most common affiliations (Protestant and Catholic). Most respondents have no trouble with this question because they can expect to see their religious group within that list in a self-administered survey.

In addition to the number and choice of response options offered, the order of answer categories can influence how people respond to closed-ended questions. Research suggests that in telephone surveys respondents more frequently choose items heard later in a list (a “recency effect”), and in self-administered surveys, they tend to choose items at the top of the list (a “primacy” effect).

Because of concerns about the effects of category order on responses to closed-ended questions, many sets of response options in Pew Research Center’s surveys are programmed to be randomized to ensure that the options are not asked in the same order for each respondent. Rotating or randomizing means that questions or items in a list are not asked in the same order to each respondent. Answers to questions are sometimes affected by questions that precede them. By presenting questions in a different order to each respondent, we ensure that each question gets asked in the same context as every other question the same number of times (e.g., first, last or any position in between). This does not eliminate the potential impact of previous questions on the current question, but it does ensure that this bias is spread randomly across all of the questions or items in the list. For instance, in the example discussed above about what issue mattered most in people’s vote, the order of the five issues in the closed-ended version of the question was randomized so that no one issue appeared early or late in the list for all respondents. Randomization of response items does not eliminate order effects, but it does ensure that this type of bias is spread randomly.

Questions with ordinal response categories – those with an underlying order (e.g., excellent, good, only fair, poor OR very favorable, mostly favorable, mostly unfavorable, very unfavorable) – are generally not randomized because the order of the categories conveys important information to help respondents answer the question. Generally, these types of scales should be presented in order so respondents can easily place their responses along the continuum, but the order can be reversed for some respondents. For example, in one of Pew Research Center’s questions about abortion, half of the sample is asked whether abortion should be “legal in all cases, legal in most cases, illegal in most cases, illegal in all cases,” while the other half of the sample is asked the same question with the response categories read in reverse order, starting with “illegal in all cases.” Again, reversing the order does not eliminate the recency effect but distributes it randomly across the population.

Question wording

The choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent and ensuring that all respondents interpret the question the same way. Even small wording differences can substantially affect the answers people provide.

[View more Methods 101 Videos ]

An example of a wording difference that had a significant impact on responses comes from a January 2003 Pew Research Center survey. When people were asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule,” 68% said they favored military action while 25% said they opposed military action. However, when asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule  even if it meant that U.S. forces might suffer thousands of casualties, ” responses were dramatically different; only 43% said they favored military action, while 48% said they opposed it. The introduction of U.S. casualties altered the context of the question and influenced whether people favored or opposed military action in Iraq.

There has been a substantial amount of research to gauge the impact of different ways of asking questions and how to minimize differences in the way respondents interpret what is being asked. The issues related to question wording are more numerous than can be treated adequately in this short space, but below are a few of the important things to consider:

First, it is important to ask questions that are clear and specific and that each respondent will be able to answer. If a question is open-ended, it should be evident to respondents that they can answer in their own words and what type of response they should provide (an issue or problem, a month, number of days, etc.). Closed-ended questions should include all reasonable responses (i.e., the list of options is exhaustive) and the response categories should not overlap (i.e., response options should be mutually exclusive). Further, it is important to discern when it is best to use forced-choice close-ended questions (often denoted with a radio button in online surveys) versus “select-all-that-apply” lists (or check-all boxes). A 2019 Center study found that forced-choice questions tend to yield more accurate responses, especially for sensitive questions.  Based on that research, the Center generally avoids using select-all-that-apply questions.

It is also important to ask only one question at a time. Questions that ask respondents to evaluate more than one concept (known as double-barreled questions) – such as “How much confidence do you have in President Obama to handle domestic and foreign policy?” – are difficult for respondents to answer and often lead to responses that are difficult to interpret. In this example, it would be more effective to ask two separate questions, one about domestic policy and another about foreign policy.

In general, questions that use simple and concrete language are more easily understood by respondents. It is especially important to consider the education level of the survey population when thinking about how easy it will be for respondents to interpret and answer a question. Double negatives (e.g., do you favor or oppose  not  allowing gays and lesbians to legally marry) or unfamiliar abbreviations or jargon (e.g., ANWR instead of Arctic National Wildlife Refuge) can result in respondent confusion and should be avoided.

Similarly, it is important to consider whether certain words may be viewed as biased or potentially offensive to some respondents, as well as the emotional reaction that some words may provoke. For example, in a 2005 Pew Research Center survey, 51% of respondents said they favored “making it legal for doctors to give terminally ill patients the means to end their lives,” but only 44% said they favored “making it legal for doctors to assist terminally ill patients in committing suicide.” Although both versions of the question are asking about the same thing, the reaction of respondents was different. In another example, respondents have reacted differently to questions using the word “welfare” as opposed to the more generic “assistance to the poor.” Several experiments have shown that there is much greater public support for expanding “assistance to the poor” than for expanding “welfare.”

We often write two versions of a question and ask half of the survey sample one version of the question and the other half the second version. Thus, we say we have two  forms  of the questionnaire. Respondents are assigned randomly to receive either form, so we can assume that the two groups of respondents are essentially identical. On questions where two versions are used, significant differences in the answers between the two forms tell us that the difference is a result of the way we worded the two versions.

One of the most common formats used in survey questions is the “agree-disagree” format. In this type of question, respondents are asked whether they agree or disagree with a particular statement. Research has shown that, compared with the better educated and better informed, less educated and less informed respondents have a greater tendency to agree with such statements. This is sometimes called an “acquiescence bias” (since some kinds of respondents are more likely to acquiesce to the assertion than are others). This behavior is even more pronounced when there’s an interviewer present, rather than when the survey is self-administered. A better practice is to offer respondents a choice between alternative statements. A Pew Research Center experiment with one of its routinely asked values questions illustrates the difference that question format can make. Not only does the forced choice format yield a very different result overall from the agree-disagree format, but the pattern of answers between respondents with more or less formal education also tends to be very different.

One other challenge in developing questionnaires is what is called “social desirability bias.” People have a natural tendency to want to be accepted and liked, and this may lead people to provide inaccurate answers to questions that deal with sensitive subjects. Research has shown that respondents understate alcohol and drug use, tax evasion and racial bias. They also may overstate church attendance, charitable contributions and the likelihood that they will vote in an election. Researchers attempt to account for this potential bias in crafting questions about these topics. For instance, when Pew Research Center surveys ask about past voting behavior, it is important to note that circumstances may have prevented the respondent from voting: “In the 2012 presidential election between Barack Obama and Mitt Romney, did things come up that kept you from voting, or did you happen to vote?” The choice of response options can also make it easier for people to be honest. For example, a question about church attendance might include three of six response options that indicate infrequent attendance. Research has also shown that social desirability bias can be greater when an interviewer is present (e.g., telephone and face-to-face surveys) than when respondents complete the survey themselves (e.g., paper and web surveys).

Lastly, because slight modifications in question wording can affect responses, identical question wording should be used when the intention is to compare results to those from earlier surveys. Similarly, because question wording and responses can vary based on the mode used to survey respondents, researchers should carefully evaluate the likely effects on trend measurements if a different survey mode will be used to assess change in opinion over time.

Question order

Once the survey questions are developed, particular attention should be paid to how they are ordered in the questionnaire. Surveyors must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions. Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow (these effects are called “order effects”).

One kind of order effect can be seen in responses to open-ended questions. Pew Research Center surveys generally ask open-ended questions about national problems, opinions about leaders and similar topics near the beginning of the questionnaire. If closed-ended questions that relate to the topic are placed before the open-ended question, respondents are much more likely to mention concepts or considerations raised in those earlier questions when responding to the open-ended question.

For closed-ended opinion questions, there are two main types of order effects: contrast effects ( where the order results in greater differences in responses), and assimilation effects (where responses are more similar as a result of their order).

An example of a contrast effect can be seen in a Pew Research Center poll conducted in October 2003, a dozen years before same-sex marriage was legalized in the U.S. That poll found that people were more likely to favor allowing gays and lesbians to enter into legal agreements that give them the same rights as married couples when this question was asked after one about whether they favored or opposed allowing gays and lesbians to marry (45% favored legal agreements when asked after the marriage question, but 37% favored legal agreements without the immediate preceding context of a question about same-sex marriage). Responses to the question about same-sex marriage, meanwhile, were not significantly affected by its placement before or after the legal agreements question.

Another experiment embedded in a December 2008 Pew Research Center poll also resulted in a contrast effect. When people were asked “All in all, are you satisfied or dissatisfied with the way things are going in this country today?” immediately after having been asked “Do you approve or disapprove of the way George W. Bush is handling his job as president?”; 88% said they were dissatisfied, compared with only 78% without the context of the prior question.

Responses to presidential approval remained relatively unchanged whether national satisfaction was asked before or after it. A similar finding occurred in December 2004 when both satisfaction and presidential approval were much higher (57% were dissatisfied when Bush approval was asked first vs. 51% when general satisfaction was asked first).

Several studies also have shown that asking a more specific question before a more general question (e.g., asking about happiness with one’s marriage before asking about one’s overall happiness) can result in a contrast effect. Although some exceptions have been found, people tend to avoid redundancy by excluding the more specific question from the general rating.

Assimilation effects occur when responses to two questions are more consistent or closer together because of their placement in the questionnaire. We found an example of an assimilation effect in a Pew Research Center poll conducted in November 2008 when we asked whether Republican leaders should work with Obama or stand up to him on important issues and whether Democratic leaders should work with Republican leaders or stand up to them on important issues. People were more likely to say that Republican leaders should work with Obama when the question was preceded by the one asking what Democratic leaders should do in working with Republican leaders (81% vs. 66%). However, when people were first asked about Republican leaders working with Obama, fewer said that Democratic leaders should work with Republican leaders (71% vs. 82%).

The order questions are asked is of particular importance when tracking trends over time. As a result, care should be taken to ensure that the context is similar each time a question is asked. Modifying the context of the question could call into question any observed changes over time (see  measuring change over time  for more information).

A questionnaire, like a conversation, should be grouped by topic and unfold in a logical order. It is often helpful to begin the survey with simple questions that respondents will find interesting and engaging. Throughout the survey, an effort should be made to keep the survey interesting and not overburden respondents with several difficult questions right after one another. Demographic questions such as income, education or age should not be asked near the beginning of a survey unless they are needed to determine eligibility for the survey or for routing respondents through particular sections of the questionnaire. Even then, it is best to precede such items with more interesting and engaging questions. One virtue of survey panels like the ATP is that demographic questions usually only need to be asked once a year, not in each survey.

U.S. Surveys

Other research methods, sign up for our weekly newsletter.

Fresh data delivered Saturday mornings

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy

Data Science Journal

• Download PDF (English) XML (English)
• Alt. Display

Practice Papers

Detailed implementation of a reproducible machine learning-enabled workflow.

• Kenneth E. Schackart III
• Heidi J. Imker
• Charles E. Cook

Machine learning (ML) and advanced computational methods are powerful tools for processing and deriving value from large data volumes. These methods are being developed and deployed rapidly, but best practices are still evolving regarding code and data standards, leading to irreproducibility of ML-enabled research. In this Practice Paper, we describe our efforts to make a ML-enabled research project to create a global inventory of biodata resources open and reproducible. To contribute to community conversations on evolving norms and expectations, we present our experiences as a practical, real-world case study that includes the implementation details as well as our overall approach and subsequent decisions. Our goal in openly sharing this experience is to provide a concrete example that others may consider as they look to vet, adapt, and adopt similar strategies to make their own work open and reproducible.

• computational reproducibility
• open science
• research software
• machine learning workflow
• biodata resource inventory

1. Introduction

There is broad concern over the lack of reproducibility in science ( Baggerly and Coombes 2009 ; Peng and Hicks 2021 ), with many believing there is a crisis ( Baker 2016 ). While the extent is contested ( Fanelli 2018 ; Leek and Jager 2017 ), concerns about scientific reproducibility are ongoing, and flawed study designs and irreproducible analyses play a role. There have been efforts to encourage better practices, such as pre-publication of study protocols, analysis plans, and all code ( Haring and Bell 2018 ). However, as argued in Haring ( 2018 ), while the different biases in production and reporting of research are largely identifiable and modifiable, continued methodological training for early career researchers is also crucial.

Use of machine learning (ML) in biosciences has proliferated so rapidly that it is difficult for adoption of good practices and proper training to keep pace. Open Science practices, such as public release of code and data, aim to remedy this ( Walters 2020 ). While access to code and data are necessary for reproduction of computational results, such access does not guarantee that results can be reproduced. Indeed, the recent Ten Years Reproducibility Challenge investigated the ability to rerun code and reproduce results from projects ten years or older, and the issues involved resulted in a useful ‘reproducibility checklist’ ( Perkel 2020 ). Additionally, efforts have been made to set standards for reproducible code, including for ML, and they serve as rubrics for assessing reproducibility ( Heil et al. 2021 ). What seems lacking, however, are detailed examples of practical implementations. This work provides such an example by explaining how a ML-enabled study was planned and executed with reproducibility as an explicit goal from the onset of the project.

In our example, the study is a ML-enabled inventory of biodata resources identified from the scientific literature. Biodata resources are biological, life sciences, and biomedical databases that archive research data generated by scientists, serving as the repositories of record for particular data types; as well as knowledge bases that add value by aggregation, processing, and expert curation. These resources are connected through extensive exchanges of data and form a distributed global infrastructure. They are crucial for the entire life science research endeavor and are used ubiquitously.

However, the infrastructure is not well-described. A number of existing resource registries, such as re3data and FAIRsharing, have done a commendable job of cataloging resources either through self-registration by the resource owner or through addition by a curator. However, neither the number of resources nor their location has been systematically explored. A better understanding of the scale of the infrastructure, as provided by this inventory, will aid funders and other stakeholders in addressing challenges to sustainability faced by the infrastructure. The methods and results of creating this inventory are fully described elsewhere ( Imker et al. 2023 ). However, during preparation of that manuscript we realized that there were many additional details to share about how we attempted to design and implement a reproducible workflow—details we wish we had found in the literature ourselves.

As context for this reproducibility case study, the following provides an outline of the research project ( Figure 1 ), and we invite readers to access the openly available article referenced above for additional details. Briefly, the study first utilized the API of Europe PMC ( europepmc.org ) ( The Europe PMC Consortium 2015 ), which is a data resource that archives a large corpus of medical and life sciences publications ( Ferguson et al. 2021 ). Europe PMC provides both individual (browser-based) and automated (API-based) queries. Our workflow started with a targeted query to the Europe PMC API to retrieve the titles and abstracts of publications for which both a URL and the word ‘data,’ ‘database,’ or ‘resource’ are present in the title and/or abstract. The results of the query represented publications that might describe a biological (biodata) resource. A 10% random subset of publications from this initial result was manually classified as describing or not describing a biodata resource (see Imker et al. 2023 and additional documentation in Imker and Schackart 2023 ). Those that did describe a biodata resource were curated to label the resource’s common name (e.g., PDB) and full name (e.g., Protein Data Bank) ( Berman et al. 2000 ). Recently, BERT (Bidirectional Encoder Representations from Transformers) performed well on NLP tasks ( Wolf et al. 2020 ). Several BERT models pre-trained on biomedical corpora (e.g., SciBERT, PubMedBERT, BioMed-RoBERTa-RCT, etc.) were selected from huggingface.co and fine-tuned for the classification (predicting if the article describes a biodata resource) and named-entity recognition (predicting common and full name) tasks. Further downstream processing was performed, including URL extraction and HTTP status checking, before finalizing the inventory.

Flowchart of overall study design to identify biodata resources from the scientific literature. The fine-tuning procedure is not shown. Reproduced unmodified from ( Imker et al. 2023 ) under Creative Commons Attribution License.

During the study, a strong emphasis was placed on Open Science, reproducibility, and robustness of the codebase and documentation for both philosophical reasons (in support of Open Science) and practical reasons (enabling future updating of the inventory). The entire process, from data splitting, model training and selection, to all downstream processing, is encapsulated in a Snakemake workflow ( Köster and Rahmann 2012 ). This allows reproduction of the entire analysis with a single command. Strong standards of code quality were developed and are enforced through the use of static code checking and automated testing. Additionally, significant efforts were made to make all data products findable, accessible, interoperable, and reusable (FAIR) ( Wilkinson et al. 2016 ).

When we began the project, we turned to the literature for robust examples of reproducibility that implemented both open data practices and code standards. Several articles contain excellent conceptual overviews (e.g., Wilson et al. 2017 ; Gruning et al. 2018 ; and a recent synthesis in Ziemann et al. 2023 ), and examples of efforts to implement Open Science practices, including open data and/or computational reproducibility, have been reported from many domains (e.g. Bush et al. 2022 in neuroimaging ; Figueiredo et al. 2022 in ecology ; and Kim et al. 2018 in bioinformatics ). These examples show how reports often focus on a few critical aspects of implementing Open Science practices; for example, although Bush et al.’s work didn’t provide the explicit code details we were interested in, it provides excellent administrative considerations like accounting for trade-offs. Figueiredo et al. provides a clear and detailed ‘kit’ for using computational notebooks in order to both show the value of reproducible workflows as well as enable their adoption. In Kim et al.’s article, they first describe their efforts to reproduce a study in which the original authors had taken steps towards reproducibility, the challenges faced despite those steps, and then their own iteration towards greater reproducibility. While there is similarity between these efforts and our goals, when it comes to implementation, there are many details which are inherently different, if described at all, because of variation in the nature of the work and relevant packages and tools. Not surprisingly, we were unable to locate implementation details that mapped exactly to our project and goals, so we adapted to fit our scenario. As a ML project, we found Heil et al.’s rubrics especially helpful in providing a framework for us to consider and specific goals to aim towards. We recognize that there are other ways of attaining these goals, and projects that have subsequently cited Heil et al.’s standards show this diversity (e.g., Wanner et al. 2023 ; Kaczmarzyk et al. 2023 ; and Heil et al. 2023 ). We offer our experience as just one example of how to make a computationally heavy study reproducible and open. We provide the reasoning behind the various considerations, which may be applicable to other research projects. We also provide specific examples of how those were realized in this study.

2. Have a Plan

‘A goal without a plan is just a wish,’ wrote Antoine de Saint-Exupéry in The Little Prince ( de Saint-Exupéry 1943 ). As with any other part of a research project, planning ahead makes the path to achieving reproducibility as smooth as possible. To this end, early in the project we developed an Open Science Implementation Plan ( Imker and Schackart 2022 ). In this document, we outlined the goals for reproducibility and how we planned to achieve them. These goals were organized into four groups: reproducibility of methods, code standards, data standards, and external review/validation ( Figure 2 ).

Graphical overview of the objectives of the study and the tools and methods used to address them regarding reproducibility, code quality, and data standards. The execution of these objectives was assessed by external review and validation.

By considering these topics early in the project, we explicitly defined what expectations we had for our Open Science goals. Keeping these goals in mind helped ensure that the effort and resources required to obtain them was anticipated and considered a core aspect of the project. This minimized the accumulation of technical debt that would have been time-consuming and difficult to address near the end of the project.

3. Reproducibility of Methods

We found the reproducibility standards (bronze, silver, gold) defined by Heil et al. ( 2021 ) useful for ranking reproducibility levels. In our case, bronze alone was not acceptable (data published and downloadable, models published and downloadable, source code published and downloadable). Obtaining silver was acceptable (bronze + dependencies set up in a single command, key analysis details recorded, analysis components set to deterministic), but the gold standard was our goal (silver + entire analysis reproducible with a single command).

3.1. Meeting the bronze standard

The bronze standard of reproducibility is characterized by having the following published and downloadable: all data necessary for reproduction, trained models, and source code.

Data availability and, more broadly, FAIRness (findability, accessibility, interoperability, and reusability) will be further discussed in a later section. To address the minimum requirements of the bronze standard, all data are available for download from the project’s Github and Zenodo repositories.

Model availability is addressed in a few ways. All of the models used in this project were pre-trained by other groups and made available on HuggingFaceHub (HFHub, https://huggingface.co/ ). As part of model training, these pretrained models were fine-tuned to various tasks (sequence classification and token classification). These fine-tuned models are made available on HFHub.

All source code is stored in two places. First, GitHub serves as a ‘living’ repository. An important aspect of Open Science is providing a place for open discussion (and criticism) of methods. The GitHub Issues system permits and encourages free and open commentary of computational methods. However, GitHub repositories are not immutable. It is important to have the methods, as described in the original publication, preserved and available, so the source code used to obtain the results in the associated full publication mentioned above has been archived as a code release on GitHub and also deposited into the Zenodo archive unmodified.

3.2. Meeting the silver standard

The silver standard requires, in addition to those aspects listed in the bronze standard, that all dependencies can be installed and set up with a single command, key analysis details are recorded, and all analysis components are deterministic (not random).

A common challenge for reproducibility is having simple installation procedures. To reach the silver standard in this regard we wanted it to be possible to install all dependencies with a single command. For Python-based projects that is often possible with the command ‘pip install -r requirements.txt’ ( pypi, n.d. ). However, sometimes other dependencies not covered by pip need to be installed. To simplify this step, we utilized Make (GNU Make v42.1) ( GNU Make 1988 ). While Make is a powerful tool intended for the control of executable files, we use it only for effectively creating aliases for shell commands. In the case of installation, we provide a Make target called ‘setup’. By doing so, the user can simply type ‘make setup’ and shell commands are executed to install all dependencies, including running pip (v21.1.2) for installing Python dependencies ( pypi, n.d. ) and renv (v0.14.0) for installing R dependencies ( Ushey 2022 ).

In addition to providing a simple pip install procedure we created a conda installation procedure ( Conda 2017 ). While using pip to install dependencies at the user level is sufficient in isolated environments, such as Google Colab ( https://colab.research.google.com/ ), it can lead to conflicts on other systems if a virtual environment is not used. Conda (v22.9.0) provides an isolated environment in which the project-specific dependencies are installed. By providing a conda environment description (yaml) file, it is possible to recreate the conda environment in a single command.

Beyond virtual environments, containers such as Docker ( Merkel 2014 ) are often used for documenting and sharing computational environments directly. However, containers can be challenging to use in certain environments. We wanted this project to be reusable for people with a wide range of technical skills, including those who may not have ready access to a robust computational infrastructure. This is especially important when thinking of potential users on a global scale, whose access to resources will be highly variable. This dependence on access to computational resources has been noted as an important part of data democratization ( Hook and Porter 2021 ). Here, we designed this project to be run on Google Colab for its low barrier to entry and its provision of graphics processing units (GPUs) for free use. Unfortunately, Colab does not natively support common container services such as Docker. However, by providing several options for dependency installation we hope that future users can find one to suit their needs.

Sufficient documentation of ‘key analysis details’ is subjective. To satisfy this requirement, in addition to an overview README that describes the entire repository, we provide README files in every directory within the repository. These explain what the various files/scripts are and how they relate to each other. Since 2021 GitHub supports the use of Mermaid, a JavaScript-based diagramming and charting tool ( Sveidqvist 2014 ), in markdown files, which we leverage to create informative flowcharts illustrating workflow logic.

An often overlooked key to reproducibility in computational methods, particularly ML methods, is seeding pseudo-random processes such that they are deterministic ( Ahmed and Lofstead 2022 ; Heil et al. 2021 ). The random numbers generated by pseudo-random number generators can have significant effects on the trained model and model performance ( Ahmed and Lofstead 2022 ). So, to make the process reproducible, we added options to use seeding to make the processes deterministic.

3.3. Meeting the gold standard

The gold standard implies that the entire analysis can be run with a single command ( Heil et al. 2021 ). Such single-command analyses require the use of a workflow manager, of which there are several options. We utilize Snakemake (v7.1.1), which facilitates automation through the definition of ‘rules’ or steps that take inputs and generate outputs. By stating what outputs are desired Snakemake creates a directed acyclic graph of which rules must be executed to create the specified output. For instance, in this project we specify that we would like the final output file to contain the classified articles along with extracted metadata. If the final output is not present, Snakemake executes all necessary steps in the pipeline including data splitting, model training and comparison, classification and Named Entity Recognition (NER), and all downstream processing. With the help of a Make alias, the Snakemake workflow for reproducing all results can be run with the single command ‘make train_and_predict’.

It is important to be able to reproduce all results from the raw data to final results, including model training. However, model training is resource intensive, and may require the use of specialized hardware such as a GPU for training to be performed in a reasonable amount of time. Requiring that all models be trained to reproduce results may be a practical challenge to reproducibility. To minimize the computational resources necessary for reproduction all fine-tuned models are available in HFHub. If the fine-tuned models are downloaded and present when Snakemake is run then Snakemake will not execute model training.

4. Beyond Reproducibility

The goal of reproducibility is to allow anyone to reproduce the results of published research. We have provided, as described above, a system that allows the results of the inventory of global biodata resources to be reproduced. However, this project was also designed to allow the entire analysis to be rerun periodically. Strictly speaking, this goes beyond reproduction since the underlying data is expected to change as more publications are added to the corpus of literature archived in Europe PMC, so the methods developed need to be generalizable. Generalizability benefits from the same considerations as reproducibility but tends to include additional challenges.

We approached generalizability with the same standards as reproducibility and wanted to make updating the inventory possible with a single command. To this end we designed a second Snakemake workflow for periodically updating the inventory. For this process the trained models can be automatically obtained from Zenodo using the setup command. The previously best performing models for each task are used, which eliminates the need for retraining and evaluation.

5. Code Standards

We’ve taken the philosophy that the results of a computational research project are no more trustworthy than the code used to produce them. Trustworthiness of code is dependent on code quality, including considerations such as readability and robustness. In this section we will describe the measures taken to ensure code quality such as code formatting, static code checking, and automated testing.

5.1. Code formatting

To accomplish Open Science, accessibility of code should not be limited to code being publicly available. True accessibility requires that code also be readable and well documented. A good first step is to utilize a code formatter, which all modern programming languages have. We used yapf v0.31.0 to format all of the Python code in this project ( Google Inc. ). Similarly, Snakemake files were formatted with snakefmt v0.6.0, and R files were formatted with styler v1.7.0 ( Hall and Letcher 2020 ; Müller et al. 2021 ). These steps are meant to ensure that all components of the project are readably formatted and documented to maximize their ease of use for others.

5.2. Static code checking

Another measure taken to increase code robustness is static code checking. Again, the code checking tools available will depend on the language. We utilize the linters pylint v2.8.2 and flake8 v.3.9.2 to check all Python code to ensure that community code standards are upheld and to detect code smell (patterns indicative of potential problems) ( Thénault 2001 ; Ziade and Cordasco 2011, p. 8 ). Many of the items that these linters consider can greatly improve code quality and readability. Some examples of considerations of the linters are: line lengths must be limited to predefined thresholds, within any context (e.g., a function) there should not be too many variables, and all functions should have docstrings. These, and many other requirements, encourage developers to write cleaner, more readable code.

Additionally, while type annotations are not required in the Python community, we implemented them as they provide a number of benefits. Type annotations provide built-in documentation by defining the data types of all inputs and outputs of functions. A lesser discussed benefit of type annotations is that they provide an enhanced integrated development environment (IDE) experience since the IDE has more knowledge of the variables and can give better help messages, syntax highlighting, and autocompletion. The final benefit of type annotations is prevention of unforeseen bugs when they are used in conjunction with a static type checker. We used mypy v0.812 to check type compatibility within all our Python code ( Lehtosalo 2012 ). This can significantly reduce the chances of encountering bugs that occur not at compile time (since Python is interpreted and dynamically typed), but instead at runtime, which can be more difficult to resolve and may not show up until running the code at a later time.

While static code checking has many benefits, programmers need not strictly adhere to all suggestions made by the code checkers. Luckily, most tools are configurable. Importantly, the user can disable certain warnings. To ensure portability of these configurations, most code checkers allow for configurations to be defined in a resource configuration (rc) file rather than in global or user settings. Accordingly, we have included our rc files in the GitHub repository so that when someone else runs the code checkers on our published code they yield the same results.

5.3. Testing

A crucial software engineering practice that is often absent from research code is testing. Testing in all of its forms: unit, integration, and end-to-end, defines the specifications of a piece of software and ensures that the software meets those specifications when the tests pass. This has numerous benefits that cannot be understated.

One of the primary benefits is that tests serve as a contract, which is a form of documentation. A unit test of a function explicitly states what kinds of input are expected and what kinds of outputs will be produced. For documentation, the only thing better than telling what a function does (through comments and docstrings) is showing through tests (asserting that when certain inputs are provided, the expected output is returned). While the descriptions provided in docstrings and comments are what the developer intends the software to do, a passing test demonstrates that it indeed does what was intended. Conversely, anything not covered in the test cases is where the contract ends. Tests ensure that the code can do what it says.

From an Open Science perspective testing is particularly valuable. Not only does testing provide more detailed documentation than could ever be provided in an article’s methods section, but it facilitates community feedback and contributions. Making changes to software always poses the risk of disrupting previous functionality. When considering applying community feedback or contributions this is problematic. However, with strong test coverage, developers can have more confidence that updates do not introduce breaking changes, as long as all previously passed tests still pass. Indeed, they provide a clear avenue for addressing bugs which may be caught by the community. Developers can add another test case that exposes the bug, then modify the code such that the new test and all previous ones pass. This is effectively amending the contract provided by the tests so that it is more comprehensive. Without tests in place developers would have to check that the code still behaves as described manually. Such checking is so error prone that many researchers may be hesitant to implement changes suggested by others.

Of course, adding strong test coverage does require more work than, for instance, implementing static code checks or formatting. Without tests, though, code must be manually assessed to ensure that a given piece of software is able to perform its intended task, and there is a barrier to implementing community feedback. Further, a lack of tests is a form of technical debt, and the price is paid when trying to refactor or fix bugs.

Pytest v6.2.4 was used as a testing framework for all Python code in this project ( Krekel 2004 ). Pytest plugins for flake8, pylint, and mypy are used to include static code checks of each file as part of the test suite (pytest-flake8 v1.0.7, pytest-pylint v0.18.0, pytest-mypy v0.8.1) ( Bader 2016 ; Gee 2015 ; Lockhert, 2015 ). This makes it such that the test suite cannot pass without all static checks passing. Additionally, most functions have associated tests, and most scripts also have end-to-end tests that ensure that they properly reject bad inputs and produce correct output when given good input. While we aim to have good test coverage, some functions and scripts are not comprehensively tested. This is generally the case for functions/scripts that take a very long time to run, such as the actual process of model training. Additionally, the Snakemake workflows developed are not formally tested using an automated testing framework, although it would be best to do so and we may implement this at a later time.

5.4. Configurability

Our aim was that the users of code, whether for reproducibility, generalization, or separate implementation, would not need to edit source code to change its behavior within the intended use cases. Parameters that may change could be supplied as inputs/arguments instead. Often, this means that paths to input files should not be hard-coded but rather passed in when calling a script. In terms of ML projects, this also often applies to hyperparameters.

One solution to this is to use parameterization extensively and, in order to make the analyses reproducible, to store the parameters used in configuration (config) files. By doing so, others can see what parameters were used to generate the results. This process additionally gives future users a clear indication of what parameters are likely okay to change, all without them having to edit any source code.

We store a large number of parameters in config files such as input/output directories, training parameters, and locations of fine-tuned models. To train a new model and compare its performance to existing models, a new row need simply be added to a tab-separated config file. The README file in the config/ directory describes the acceptable ranges of values allowed in the config files, such as a description of what kind of models are compatible with the existing workflow.

Snakemake also makes extensive use of config files, and the config files described here are formatted such that Snakemake can utilize them when executing the workflow. So, to change the behavior of the workflow (again, within the expected range of uses), only config files need to be edited.

6. Data Standards

6.1. source selection.

Both code and data were integral components of this project and both required consideration for reproducible outcomes. To create an open inventory as a product we aimed to reuse and create data that aligned with the FAIR guiding principles ( Wilkinson et al. 2016 ). The primary data source needed was bibliographic metadata. There are several commercial sources of bibliographic metadata such as Dimensions (Digital Science), Scopus (Elsevier), and Web of Science (Clarivate Analytics). However, these resources require a subscription which would limit others’ ability to reproduce and reuse our workflow and neither are they openly licensed. Therefore, we opted to use the open metadata available from Europe PMC as the data source for creating the inventory. Although not as exhaustive as the commercial options mentioned, Europe PMC covers a large swath of the life sciences; as of October 2023, high quality, interoperable metadata, including titles and abstracts, was available for over 40 million articles. Additionally, Europe PMC offers robust and well-documented APIs that facilitate access and are especially useful for a reproducible pipeline. Although we know that some biodata resources will be missed due to articles being published outside of the ~4000 journals available in Europe PMC, we felt that this tradeoff was justified in order to optimize openness and reproducibility.

6.2. Addressing data findability and accessibility

Depending on context, anyone interested in reusing the data from this project might wish to start at different points. We therefore offer multiple options. The exact query string we used can be rerun to obtain results from Europe PMC. Additionally, since bibliographic databases may change slightly over time (e.g. records added, removed, or corrected), query results themselves (PMID, title, abstract) may be of use to reproduce our results using the exact same data. There is also the labeled training data that was used to train the various models, a preliminary inventory that is subjected to selective review by a curator, and, finally, the primary data product for this project is the final inventory itself. The query string, query results, training data, preliminary inventory, and the final inventory are all available within the project’s GitHub repository and were archived for long-term preservation and persistent reference in an associated Zenodo deposition once the article was accepted for publication. Zenodo provides a DOI and relies on the DataCite metadata schema, which allows the dataset to be found within Zenodo’s search interface, DataCite’s central metadata store, and via internet search engines such as Google.

6.3. Addressing data interoperability

For the final inventory, we retained unique article identifiers (PMIDs) to allow easy extraction of additional metadata or for access to the full text, when available, from either Europe PMC or PubMed Central. Additionally, we logged URL status codes per specification RFC 9110 ( Fielding et al. 2022 ), extracted countries from author affiliations following ISO 3166 ( ISO 3166 n.d. ), and retained geo coordinates for IP address look-ups, when available. While it would have been ideal to include a persistent identifier for the biodata resources located (e.g., ROR ID or DOI), most resources do not have an identifier, which perfectly illustrates the challenge of trying to locate these resources in the first place.

6.4. Addressing data reusability

In addition to the efforts towards interoperability described above, we also maintained a structured format throughout and used the CSV format for preservability and to ensure ease of reuse. These files are accompanied by a plaintext README file that includes a description of each variable as well as data collection details and licensing. By using open data from Europe PMC, we were able to release the data with CC0 licensing, thus allowing the broadest reuse possible. Together, this documentation, the repository’s Github history, and Zenodo’s commitment to long-term archiving all provide provenance.

Finally, to further extend the potential for reuse, we plan to provide identified biodata resources to Europe PMC as community annotations. This will allow easy bulk access to the identified resources as well as their associated articles. The annotations can be used for several purposes, for example, mining articles with full text available or analysis of the intersection between these annotations and the many other annotation types available within Europe PMC.

7. External Review/Validation

In the Open Science Implementation Plan that we drafted (see Section 2 above), we also included a desire to have a party external to the team review the products of the study. Working within a team inherently provides a mechanism for internal feedback, but review by another person outside of the project helps reveal implicit knowledge developed during the project that would otherwise remain hidden to potential reusers. For example, team members may, without realizing it, adopt terms or abbreviations that are not well-known outside of the project.

This section of the Open Science Implementation Plan was not particularly well-developed beyond acknowledging that such a review would be ideal, as noted by others ( Coburn and Johnston 2020 ; Heil et al. 2021 ), and that this role is included in the CRediT taxonomy ( Allen et al. 2019 ). As we moved closer to having products finalized, we had a better sense of what sorts of reviews will be most valuable. We recruited an individual who reviewed the code and documentation in detail and ran nearly all the code available in the open archive. We budgeted 40 hours for this work, which was easily consumed given review effort required. Others may wish to allocate even more resources to this activity, which we found extremely helpful for identifying errors and pointing out gaps in our documentation. We formally acknowledge this effort here as well as in the associated article.

8. Discussion

Here we have described the efforts that were taken to develop a methodology for obtaining and updating a biodata resource inventory with Heil et al.’s gold standard of reproducibility, a robust codebase, and complying with FAIR data standards.

8.1 From Principles to practice

We, and many others, are committed to Open Science and see the imperative of reproducibility. Putting these principles into practice on a complex project presented an opportunity for us to work through philosophical, organizational, and technical details. We were successful in meeting the goals outlined in the Open Science Implementation Plan established at the beginning of the project. Installation of dependencies and reproduction of the entire analysis can each be performed with a single command, and analysis steps are fully documented. All code passes static code checks for formatting, linting, and type compatibility. Much of the code was formally tested with unit and integration tests. The core data products, such as the labeled training data and preliminary inventory, are present in GitHub and in Zenodo, with accompanying documentation.

The methodologies used in this work are not novel on their own. Wherever possible, we looked to existing tools and practices. The automation employed to make reproduction simple relies on the widely used Snakemake workflow manager. It is also common practice in software engineering disciplines to leverage static code checking and testing as we have done. Regarding data standards, we looked to the FAIR principles. The purpose of this report is to provide an example of how a research project that utilizes computational methods, particularly ML, can be implemented to maintain robustness and strive for a high level of reproducibility. However, we recognize that there are numerous ways to accomplish this and do not mean to claim our implementation is failproof.

8.2 From details to decisions

When we began the project, we were especially interested in finding implementation details. How exactly does one make it possible to re-run an entire analysis with a single command? How exactly does one make data ‘interoperable’? Although we knew these details would be different in our case, concrete examples can provide clarity and inspiration. As the project progressed and we learned by doing, our questions evolved to focus on the choices that must be made. One example is the tradeoff of using only open data versus a more extensive commercial data source, which would likely have yielded a larger, but in our estimation less useful, inventory. Many of the trickiest decisions involved accounting for the diverse interests of, and the resources available to, potential reusers, now and into the future.

There were also ambitions that we had at the start of the study that are now future directions because we chose to devote time developing a robust workflow instead. This required principled project management and caused, even as we write this, some amount of wistfulness. In the end, we could not do it ‘all’, and we fully appreciate that others must decide for themselves where to place their efforts. Such decisions required us, and will require others, to devote a substantial amount of time to think through and implement. We were able to do this only because of our team’s collective belief that these efforts were worth the resources invested.

8.3 Limitations

Certain improvements could be made, such as using a more robust package manager like poetry and using git hooks to automatically run tests upon committing to git. Importantly, test coverage is lacking in some areas, especially for portions that involve heavy computation such as model training. Still, the current test coverage is enough to increase confidence in the code’s behavior. As Peng ( 2011 ) noted, ‘Given the barriers to reproducible research, it is tempting to wait for a comprehensive solution to arrive.’ Thus we thought our experiences may be helpful to share.

Possibly the greatest limitation, or threat to long-term reproducibility, was the decision to not use containers as a trade-off to be compatible with Google Colaboratory. In the current configuration, all dependencies are listed in a requirements.txt file and must be installed to run the code. However, it is possible that dependencies become unavailable or incompatible in future. Containers mitigate this problem by packaging all dependencies with the code, eliminating this concern.

A key consideration is how generalizable the efforts and methods toward reproducibility presented here are to other research projects, methods, and domains. Fortunately, most of the methods and tools here are not specific to natural language processing pipelines, and therefore generalize well to most computational research tasks. For example, workflow managers such as Snakemake can be applied to data analysis pipelines in general. Additionally, the more conceptual steps, like creating the Open Science Implementation Plan at the start of a project, could be broadly applied.

9. Conclusion

Through articulating our goals early on and dedicating time and resources, we were able to accomplish our Open Science and reproducibility goals. Throughout this case study, we provided details on the steps we took to make the code clean and robust and the data FAIR. We invested considerable effort into ensuring reproducibility, with the intent that both the methods and outputs would be of use to us and others. Our first update of the inventory, initiated approximately one year after project completion, only required modification to the Colab notebooks to account for Google Colaboratory changes, but otherwise functioned as expected. With this promising, albeit early, success, we remain cautiously optimistic that the work is durable. By presenting our experiences, we hope this Practice Paper provides a helpful example for others to consider as they work to build greater reproducibility in their research.

Data Accessibility Statement

Code and data generated during the course of the project are archived in Zenodo along with associated documentation ( https://zenodo.org/doi/10.5281/zenodo.10105161 ). The final inventory and associated data dictionary are available as a separate Zenodo deposit ( https://zenodo.org/doi/10.5281/zenodo.10105947 ). Readers may visit HuggingFaceHub ( https://huggingface.co/globalbiodata/inventory_2022_all_models/tree/main ) to access the fine-tuned models. Additionally, all materials are available on GitHub, which may be updated after this publication ( https://github.com/globalbiodata/inventory_2022/ ). All other software used is openly available and shown Table 1 .

Glossary of Software.

Acknowledgements

The authors would like to thank Ana-Maria Istrate with the Chan Zuckerberg Initiative for her contributions to developing the machine learning methods used in the project as well as CZI colleagues Dario Taraborelli, Donghui Li, and Gully Burns for their support and feedback on early versions of the study. We also thank Ken Youens-Clark formerly at The University of Arizona, Alise Ponsero at The University of Helsinki, and Bonnie Hurwitz at The University of Arizona for their mentorship of Kenneth Schackart. Additionally, we thank the Europe PMC team, especially Aravind Venkatesan, Mohamed Selim, and Melissa Harrison, for their guidance and expertise. Finally, we would like to acknowledge Jodie Forbes for detailed review of the associated code and documentation.

Funding Information

This work was funded by the Global Biodata Coalition ( globalbiodata.org ), a coalition of research funding organizations working towards sustainability of biodata resources worldwide.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

KES – Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

HJI – Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing

CEC – Conceptualization, Data curation, Funding acquisition, Supervision, Writing – original draft, Writing – review & editing

Ahmed, H and Lofstead, J 2022 Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems . New York, NY, USA: Association for Computing Machinery. pp. 15–20. DOI: https://doi.org/10.1145/3526062.3536353  

Allen, L, O’Connell, A and Kiermer, V 2019 How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship. Learned Publishing , 32(1): 71–74. DOI: https://doi.org/10.1002/leap.1210  

Bader, D 2016 pytest-mypy: Mypy static type checker plugin for Pyest . Available at https://github.com/realpython/pytest-mypy [Last accessed 22 November 2022].  

Baggerly, K A and Coombes, K R 2009 Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. The Annals of Applied Statistics , 3(4): 1309–1334. DOI: https://doi.org/10.1214/09-AOAS291  

Baker, M 2016 1,500 scientists lift the lid on reproducibility. Nature , 533(7604): 452–454. DOI: https://doi.org/10.1038/533452a  

Berman, H M, Westbrook, J, Feng, Z, Gilliland, G, Bhat, T N, Weissig, H, Shindyalov, I N and Bourne, P E 2000 The Protein Data Bank. Nucleic Acids Research , 28(1): 235–242. DOI: https://doi.org/10.1093/nar/28.1.235  

Bush, K A, Calvert, M L and Kilts, C D 2022 Lessons learned: A neuroimaging research center’s transition to open and reproducible science. Frontiers in Big Data , 5. DOI: https://doi.org/10.3389/fdata.2022.988084  

Coburn, E and Johnston, L 2020 Testing our assumptions: Preliminary results from the Data Curation Network. Journal of eScience Librarianship , 9(1). DOI: https://doi.org/10.7191/jeslib.2020.1186  

Conda 2017 Available at https://www.anaconda.com [Last accessed 22 November 2022].  

Country Codes – ISO 3166 n.d. Available at https://www.iso.org/iso-3166-country-codes.html .  

de Saint-Exupéry, A 1943 Le petit prince [The little prince] . Verenigde State van Amerika: Reynal & Hitchkock (US), Gallimard (FR).  

Fanelli, D 2018 Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences , 115(11): 2628–2631. DOI: https://doi.org/10.1073/pnas.1708272114  

Ferguson, C, Araújo, D, Faulk, L, Gou, Y, Hamelers, A, Huang, Z, Ide-Smith, M, Levchenko, M, Marinos, N, Nambiar, R, Nassar, M, Parkin, M, Pi, X, Rahman, F, Rogers, F, Roochun, Y, Saha, S, Selim, M, Shafique, Z, Sharma, S, Stephenson, D, Talo’, F, Thouvenin, A, Tirunagari, S, Vartak, V, Venkatesan, A, Yang, X and McEntyre, J 2021 Europe PMC in 2020. Nucleic Acids Research , 49(D1): D1507–D1514. DOI: https://doi.org/10.1093/nar/gkaa994  

Fielding, R, Nottingham, M and Reschke, J 2022 RFC 9910 HTTP Semantics. Internet Engineering Task Force. Available at https://www.doi.org/10.17487/RFC9110 .  

Figueiredo, L, Scherer, C and Sarmento Cabral, J 2022 A simple kit to use computational notebooks for more openness, reproducibility, and productivity in research. PLOS Computational Biology , 18(9): e1010356. DOI: https://doi.org/10.1371/journal.pcbi.1010356  

Gee, C 2015 pytest-pylint: pytest plugin for running pylint against your codebase . Available at https://github.com/carsongee/pytest-pylint [Last accessed 22 November 2022].  

GNU Make 1988. Available at https://www.gnu.org/software/make/ [Last accessed 22 November 2022].  

Google Inc. yapf: A formatter for Python files , 2004. Available at https://github.com/google/yapf [Last accessed 22 November 2022].  

Grüning, B, Chilton, J, Köster, J, Dale, R, Soranzo, N, van den Beek, M, Goecks, J, Backofen, R, Nekrutenko, A and Taylor, J 2018 Practical computational reproducibility in the life sciences. Cell Systems , 6(6): 631–35. DOI: https://doi.org/10.1016/j.cels.2018.03.014  

Hall, M and Letcher, B 2020 Snakefmt: The uncompromising Snakemake code formatter . Available at https://github.com/snakemake/snakefmt [Last accessed 22 November 2022].  

Haring, R and Bell, R J 2018 Lack of research reproducibility, the rise of open science and the need for continuing education in research methods. Climacteric , 21(5): 413–414. DOI: https://doi.org/10.1080/13697137.2018.1476968  

Heil, B J, Crawford J and Greene, C S 2023 The effect of non-linear signal in classification problems using gene expression. PLoS Computational Biology , 19(3): e1010984. DOI: https://doi.org/10.1371/journal.pcbi.1010984  

Heil, B J, Hoffman, M M, Markowetz, F, Lee, S-I, Greene, C S and Hicks, S C 2021 Reproducibility standards for machine learning in the life sciences. Nature Methods , 18(10): 1132–1135. DOI: https://doi.org/10.1038/s41592-021-01256-7  

Hook, D W and Porter, S J 2021 Scaling scientometrics: Dimensions on Google BigQuery as an infrastructure for large-scale analysis. Frontiers in Research Metrics and Analytics , 6. Available at https://www.frontiersin.org/articles/10.3389/frma.2021.656233 [Last accessed 3 February 2023].  

Imker, H J and Schackart, K E 2022 Open Science implementation plan for the biodata resource inventory. Zenodo . DOI: https://doi.org/10.5281/zenodo.7392518  

Imker, H J and Schackart, K E 2023 Manual review process for the biodata resource inventory. Zenodo . DOI: https://doi.org/10.5281/zenodo.7768363  

Imker, H J, Schackart III, K E, Istrate, A-M and Cook, C E 2023 A machine learning-enabled open biodata resource inventory from the scientific literature. PLOS ONE , 18(11): 1–28. DOI: https://doi.org/10.1371/journal.pone.0294812  

Kaczmarzyk, J R, Gupta, R, Kurc, T M, Abousamra, S, Saltz, J H and Koo, P K 2023 ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification. Computer Methods and Programs in Biomedicine , 239. DOI: https://doi.org/10.1016/j.cmpb.2023.107631  

Kim, Y-M, Poline, J-B and Dumas, G 2018 Experimenting with reproducibility: A case study of robustness in bioinformatics. GigaScience , 7(7). DOI: https://doi.org/10.1093/gigascience/giy077  

Köster, J and Rahmann, S 2012 Snakemake—a scalable bioinformatics workflow engine. Bioinformatics , 28(19): 2520–2522. DOI: https://doi.org/10.1093/bioinformatics/bts480  

Krekel, H 2004 pytest: The pytest framework makes it easy to write small tests, yet scales to support complex functional testing . Available at https://github.com/pytest-dev/pytest [Last accessed 22 November 2022].  

Leek, J T and Jager, L R 2017 Is Most Published Research Really False? Annual Review of Statistics and Its Application , 4(1): 109–122. DOI: https://doi.org/10.1146/annurev-statistics-060116-054104  

Lehtosalo, J 2012 mypy: Optional static typing for Python . Available at https://github.com/python/mypy [Last accessed 22 November 2022].  

Lockhert, T 2015 pytest-flake8: Pytest plugin to run flake8 . Available at https://github.com/tholo/pytest-flake8 [Last accessed 22 November 2022].  

Merkel, D 2014 Docker: Lightweight linux containers for consistent development and deployment. Linux j , 239(2): 2.  

Müller, K, Walthert, L and Patil, I 2021 styler: Non-invasive pretty printing of R code . Available at https://github.com/r-lib/styler [Last accessed 22 November 2022].  

Peng, R D 2011 Reproducible Research in Computational Science. Science , 334(6060): 1226–1227. DOI: https://doi.org/10.1126/science.1213847  

Peng, R D and Hicks, S C 2021 Reproducible Research: A Retrospective. Annual Review of Public Health , 42(1): 79–93. DOI: https://doi.org/10.1146/annurev-publhealth-012420-105110  

Perkel, J M 2020 Challenge to scientists: does your ten-year-old code still run? Nature , 584(7822): 656–658. DOI: https://doi.org/10.1038/d41586-020-02462-7  

pypi n.d. Python Package Index – PyPI . Available at https://pypi.org/ [Last accessed 22 November 2022].  

Sveidqvist, K 2014 Mermaid: Generation of diagrams like flowcharts or sequence diagrams from text in a similar manner as markdown . Available at https://github.com/mermaid-js/mermaid/ [Last accessed 22 November 2022].  

The Europe PMC Consortium 2015 Europe PMC: a full-text literature database for the life sciences and platform for innovation. Nucleic Acids Research , 43(D1): D1042–D1048. DOI: https://doi.org/10.1093/nar/gku1061  

Thénault, S 2001 Pylint: It’s not just a linter that annoys you! Available at https://github.com/PyCQA/pylint [Last accessed 22 November 2022].  

Ushey, K 2022 renv: Project Environments . Available at https://rstudio.github.io/renv/ [Last accessed 6 January 2023].  

Walters, W P 2020 Code sharing in the Open Science era. Journal of Chemical Information and Modeling , 60(10): 4417–4420. DOI: https://doi.org/10.1021/acs.jcim.0c01000  

Wanner, J, Cuellar, L K, Rausch, L, Berendzen, K W, Wanke, F, Gabernet, G, Harter, K and Nahnsen, S 2023 nf-root: A best-practice pipeline for deep learning-based analysis of apoplastic pH in microscopy images of developmental zones in plant root tissue. bioRxiv , 2023.01.16.524272. DOI: https://doi.org/10.1101/2023.01.16.524272  

Wilkinson, M D, Dumontier, M, Aalbersberg, Ij J, Appleton, G, Axton, M, Baak, A, Blomberg, N, Boiten, J-W, da Silva Santos, L B, Bourne, P E, Bouwman, J, Brookes, A J, Clark, T, Crosas, M, Dillo, I, Dumon, O, Edmunds, S, Evelo, C T, Finkers, R, Gonzalez-Beltran, A, Gray, A J G, Groth, P, Goble, C, Grethe, J S, Heringa, J, ’t Hoen, P A C, Hooft, R, Kuhn, T, Kok, R, Kok, J, Lusher, S J, Martone, M E, Mons, A, Packer, A L, Persson, B, Rocca-Serra, P, Roos, M, van Schaik, R, Sansone, S-A, Schultes, E, Sengstag, T, Slater, T, Strawn, G, Swertz, M A, Thompson, M, van der Lei, J, van Mulligen, E, Velterop, J, Waagmeester, A, Wittenburg, P, Wolstencroft, K, Zhao, J and Mons, B 2016 The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data , 3(1): 160018. DOI: https://doi.org/10.1038/sdata.2016.18  

Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L and Teal, T K 2017 Good enough practices in scientific computing. PLoS Computational Biology , 13(6): e1005510. DOI: https://doi.org/10.1371/journal.pcbi.1005510  

Wolf, T, Debut, L, Sanh, V, Chaumond, J, Delangue, C, Moi, A, Cistac, P, Rault, T, Louf, R, Funtowicz, M, Davison, J, Shleifer, S, von Platen, P, Ma, C, Jernite, Y, Plu, J, Xu, C, Le Scao, T, Gugger, S, Drame, M, Lhoest, Q and Rush, A 2020 Transformers: state-of-the-art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online: Association for Computational Linguistics. pp. 38–45. DOI: https://doi.org/10.18653/v1/2020.emnlp-demos.6  

Ziade, T and Cordasco, I 2011 Flake8: Your tool for style guide enforcement . Available at https://github.com/PyCQA/flake8 [Last accessed 22 November 2022].  

Ziemann, M, Poulain, P and Bora, A 2023 The five pillars of computational reproducibility: bioinformatics and beyond. Briefings in Bioinformatics , 24(6). DOI: https://doi.org/10.1093/bib/bbad375  

METHODS article

This article is part of the research topic.

Integrated Energy System Planning, Optimization, Trading and Benefit Assessment

Study on Low-carbon Service Mode of Park-level Integrated Energy System with Flexible Supply and Demand Balance Provisionally Accepted

• 1 Shanghai University of Electric Power, China

The final, formatted version of the article will be published soon.

Against the backdrop of "carbon peaking and carbon neutrality" goals in China, the park-level integrated energy system is a significant approach to meeting the energy consumption demand of users and improving the utilization rate of new energy. To take into account both the interests of integrated energy services (IES) providers and the current power market rules in China, targeting at low-carbon energy consumption, a balancing service mode for integrated energy services providers is created herein based on the idea of balancing multiple interests of regional integrated energy services providers. It aims to break down the market barriers and address the tricky problem that the integrated energy services industry fails to establish a sophisticated industrial system under the original mode. Combining the integrated energy services with the auxiliary market of power balance, the paper analyzes the characteristics of energy flow within the system, and puts forward the optimization scheme of regional integrated energy balancing services under the low-carbon background. The objective is solved by Mayfly algorithm and compared for analysis, and the results show that the optimal daily operating cost is 2632.59 yuan and the daily carbon emission is 3869.90 kg under the typical industrial scenario. The examples provided here show that the optimization plan for integrated energy balancing services can reduce carbon emissions from the park-level integrated energy system and boost the revenue for integrated energy service providers. It provides theoretical support for the green transformation of energy enterprises and promotes the healthy development of the global integrated energy industry.

Keywords: green innovation, integrated energy system services provider, Balancing service, mechanism design, Low-carbon transition

Received: 01 Mar 2024; Accepted: 30 Apr 2024.

Copyright: © 2024 Hua, Yanji, Huimin and Rong. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mx. LI Yanji, Shanghai University of Electric Power, Shanghai, 130012, Shanghai Municipality, China

People also looked at