one shot case study design example

Child Care and Early Education Research Connections

Pre-experimental designs.

Pre-experiments are the simplest form of research design. In a pre-experiment either a single group or multiple groups are observed subsequent to some agent or treatment presumed to cause change.

Types of Pre-Experimental Design

One-shot case study design, one-group pretest-posttest design, static-group comparison.

A single group is studied at a single point in time after some treatment that is presumed to have caused change. The carefully studied single instance is compared to general expectations of what the case would have looked like had the treatment not occurred and to other events casually observed. No control or comparison group is employed.

A single case is observed at two time points, one before the treatment and one after the treatment. Changes in the outcome of interest are presumed to be the result of the intervention or treatment. No control or comparison group is employed.

A group that has experienced some treatment is compared with one that has not. Observed differences between the two groups are assumed to be a result of the treatment.

Validity of Results

An important drawback of pre-experimental designs is that they are subject to numerous threats to their  validity . Consequently, it is often difficult or impossible to dismiss rival hypotheses or explanations. Therefore, researchers must exercise extreme caution in interpreting and generalizing the results from pre-experimental studies.

One reason that it is often difficult to assess the validity of studies that employ a pre-experimental design is that they often do not include any control or comparison group. Without something to compare it to, it is difficult to assess the significance of an observed change in the case. The change could be the result of historical changes unrelated to the treatment, the maturation of the subject, or an artifact of the testing.

Even when pre-experimental designs identify a comparison group, it is still difficult to dismiss rival hypotheses for the observed change. This is because there is no formal way to determine whether the two groups would have been the same if it had not been for the treatment. If the treatment group and the comparison group differ after the treatment, this might be a reflection of differences in the initial recruitment to the groups or differential mortality in the experiment.

Advantages and Disadvantages

As exploratory approaches, pre-experiments can be a cost-effective way to discern whether a potential explanation is worthy of further investigation.

Disadvantages

Pre-experiments offer few advantages since it is often difficult or impossible to rule out alternative explanations. The nearly insurmountable threats to their validity are clearly the most important disadvantage of pre-experimental research designs.

One-Group Posttest Only Design: An Introduction

The one-group posttest-only design (a.k.a. one-shot case study ) is a type of quasi-experiment in which the outcome of interest is measured only once after exposing a non-random group of participants to a certain intervention.

The objective is to evaluate the effect of that intervention which can be:

• A training program
• A policy change
• A medical treatment, etc.

As in other quasi-experiments, the group of participants who receive the intervention is selected in a non-random way (for example according to their choosing or that of the researcher).

The one-group posttest-only design is especially characterized by having:

• No control group
• No measurements before the intervention

It is the simplest and weakest of the quasi-experimental designs in terms of level of evidence as the measured outcome cannot be compared to a measurement before the intervention nor to a control group.

So the outcome will be compared to what we assume will happen if the intervention was not implemented. This is generally based on expert knowledge and speculation.

Next we will discuss cases where this design can be useful and its limitations in the study of a causal relationship between the intervention and the outcome.

Advantages and Limitations of the one-group posttest-only design

Advantages of the one-group posttest-only design, 1. advantages related to the non-random selection of participants:.

• Ethical considerations: Random selection of participants is considered unethical when the intervention is believed to be harmful (for example exposing people to smoking or dangerous chemicals) or on the contrary when it is believed to be so beneficial that it would be malevolent not to offer it to all participants (for example a groundbreaking treatment or medical operation).
• Difficulty to adequately randomize subjects and locations: In some cases where the intervention acts on a group of people at a given location, it becomes infeasible to adequately randomize subjects (ex. an intervention that reduces pollution in a given area).

2. Advantages related to the simplicity of this design:

• Feasible with fewer resources than most designs: The one-group posttest-only design is especially useful when the intervention must be quickly introduced and we do not have enough time to take pre-intervention measurements. Other designs may also require a larger sample size or a higher cost to account for the follow-up of a control group.
• No temporality issue: Since the outcome is measured after the intervention, we can be certain that it occurred after it, which is important for inferring a causal relationship between the two.

Limitations of the one-group posttest-only design

1. selection bias:.

Because participants were not chosen at random, it is certainly possible that those who volunteered are not representative of the population of interest on which we intend to draw our conclusions.

2. Limitation due to maturation:

Because we don’t have a control group nor a pre-intervention measurement of the variable of interest, the post-intervention measurement will be compared to what we believe or assume would happen was there no intervention at all.

The problem is when the outcome of interest has a natural fluctuation pattern (maturation effect) that we don’t know about.

So since certain factors are essentially hard to predict and since 1 measurement is certainly not enough to understand the natural pattern of an outcome, therefore with the one-group posttest-only design, we can hardly infer any causal relationship between intervention and outcome.

3. Limitation due to history:

The idea here is that we may have a historical event, which may also influence the outcome, occurring at the same time as the intervention.

The problem is that this event can now be an alternative explanation of the observed outcome. The only way out of this is if the effect of this event on the outcome is well-known and documented in order to account for it in our data analysis.

This is why most of the time we prefer other designs that include a control group (made of people who were exposed to the historical event but not to the intervention) as it provides us with a reference to compare to.

Example of a study that used the one-group posttest-only design

In 2018, Tsai et al. designed an exercise program for older adults based on traditional Chinese medicine ideas, and wanted to test its feasibility, safety and helpfulness.

So they conducted a one-group posttest-only study as a pilot test with 31 older adult volunteers. Then they evaluated these participants (using open-ended questions) after receiving the intervention (the exercise program).

The study concluded that the program was safe, helpful and suitable for older adults.

What can we learn from this example?

1. work within the design limitations:.

Notice that the outcome measured was the feasibility of the program and not its health effects on older adults.

The purpose of the study was to design an exercise program based on the participants’ feedback. So a pilot one-group posttest-only study was enough to do so.

For studying the health effects of this program on older adults a randomized controlled trial will certainly be necessary.

2. Be careful with generalization when working with non-randomly selected participants:

For instance, participants who volunteered to be in this study were all physically active older adults who exercise regularly.

Therefore, the study results may not generalize to all the elderly population.

• Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference . 2nd Edition. Cengage Learning; 2001.
• Campbell DT, Stanley J. Experimental and Quasi-Experimental Designs for Research . 1st Edition. Cengage Learning; 1963.

Further reading

• Understand Quasi-Experimental Design Through an Example
• Experimental vs Quasi-Experimental Design
• Static-Group Comparison Design
• One-Group Pretest-Posttest Design

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons

Margin Size

• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

7.4: Pre-Experimental Designs

• Last updated
• Save as PDF
• Page ID 124576

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

\( \newcommand{\Span}{\mathrm{span}}\)

\( \newcommand{\id}{\mathrm{id}}\)

\( \newcommand{\kernel}{\mathrm{null}\,}\)

\( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\)

\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\)

\( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

\( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vectorC}[1]{\textbf{#1}} \)

\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

Learning Objectives

• Discuss when is the appropriate time to use a pre-experimental Design.
• Identify and describe the various types of pre-experimental designs.

What is it and When to Use it?

Time, other resources such as funding, and even one’s topic may limit a researcher’s ability to use a solid experimental design such a a between subject (which includes the classical experiment) or a within subject design. For researchers in the medical and health sciences, conducting one of these more solid designs could require denying needed treatment to patients, which is a clear ethical violation. Even those whose research may not involve the administration of needed medications or treatments may be limited in their ability to conduct a classic experiment. In social scientific experiments, for example, it might not be equitable or ethical to provide a large financial or other reward only to members of the experimental group. When random assignment of participants into experimental and control groups (using either randomization or matching) is not feasible, researchers may turn to a pre-experimental design (Campbell & Stanley, 1963).Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research . Chicago, IL: Rand McNally. However, this type of design comes with some unique disadvantages, which we’ll describe as we review the pre-experimental designs available.

If we wished to measure the impact of some natural disaster, for example, Hurricane Katrina, we might conduct a pre-experiment by identifying an experimental group from a community that experienced the hurricane and a control group from a similar community that had not been hit by the hurricane. This study design, called a static group comparison , has the advantage of including a comparison control group that did not experience the stimulus (in this case, the hurricane) but the disadvantage of containing experimental and control groups that were determined by a factor or factors other than random assignment. As you might have guessed from our example, static group comparisons are useful in cases where a researcher cannot control or predict whether, when, or how the stimulus is administered, as in the case of natural disasters.

In cases where the administration of the stimulus is quite costly or otherwise not possible, a one-shot case study design might be used. In this instance, no pretest is administered, nor is a control group present. In our example of the study of the impact of Hurricane Katrina, a researcher using this design would test the impact of Katrina only among a community that was hit by the hurricane and not seek out a comparison group from a community that did not experience the hurricane. Researchers using this design must be extremely cautious about making claims regarding the effect of the stimulus, though the design could be useful for exploratory studies aimed at testing one’s measures or the feasibility of further study.

Finally, if a researcher is unlikely to be able to identify a sample large enough to split into multiple groups, or if he or she simply doesn’t have access to a control group, the researcher might use a one-group pre-/posttest design. In this instance, pre- and posttests are both taken but, as stated, there is no control group to which to compare the experimental group. We might be able to study of the impact of Hurricane Katrina using this design if we’d been collecting data on the impacted communities prior to the hurricane. We could then collect similar data after the hurricane. Applying this design involves a bit of serendipity and chance. Without having collected data from impacted communities prior to the hurricane, we would be unable to employ a one-group pre-/posttest design to study Hurricane Katrina’s impact.

Table 7.2 summarizes each of the preceding examples of pre-experimental designs.

As implied by the preceding examples where we considered studying the impact of Hurricane Katrina, experiments do not necessarily need to take place in the controlled setting of a lab. In fact, many applied researchers rely on experiments to assess the impact and effectiveness of various programs and policies.

KEY TAKEAWAYS

• Pre-experimental designs are not ideal, but have to be done under certain circumstances.
• There are three major types of this design.

• Survey Software The world’s leading omnichannel survey software
• Online Survey Tools Create sophisticated surveys with ease.
• Mobile Offline Conduct efficient field surveys.
• Text Analysis
• Close The Loop
• Automated Translations
• NPS Dashboard
• CATI Manage high volume phone surveys efficiently
• Cloud/On-premise Dialer TCPA compliant Cloud & on-premise dialer
• IVR Survey Software Boost productivity with automated call workflows.
• Analytics Analyze survey data with visual dashboards
• Panel Manager Nurture a loyal community of respondents.
• Survey Portal Best-in-class user friendly survey portal.
• Voxco Audience Conduct targeted sample research in hours.

Find the best survey software for you! (Along with a checklist to compare platforms)

Get Buyer’s Guide

• 40+ question types
• Drag-and-drop interface
• Skip logic and branching
• Multi-lingual survey
• Text piping
• Question library
• CSS customization
• White-label surveys
• Customizable ‘Thank You’ page
• Customizable survey theme
• Reminder send-outs
• Survey rewards
• Social media
• Website surveys
• Correlation analysis
• Cross-tabulation analysis
• Trend analysis
• Real-time dashboard
• Customizable report
• Email address validation
• Recaptcha validation
• SSL security

Take a peek at our powerful survey features to design surveys that scale discoveries.

Download feature sheet.

• Hospitality
• Financial Services
• Academic Research
• Customer Experience
• Employee Experience
• Product Experience
• Market Research
• Social Research
• Data Analysis

Explore Voxco 

Need to map Voxco’s features & offerings? We can help!

Watch a Demo 

Download Brochures 

Get a Quote

• NPS Calculator
• CES Calculator
• A/B Testing Calculator
• Margin of Error Calculator
• Sample Size Calculator
• CX Strategy & Management Hub
• Market Research Hub
• Patient Experience Hub
• Employee Experience Hub
• Market Research Guide
• Customer Experience Guide
• The Voxco Guide to Customer Experience
• NPS Knowledge Hub
• Survey Research Guides
• Survey Template Library
• Webinars and Events
• Feature Sheets
• Try a sample survey
• Professional services

Find the best customer experience platform

Uncover customer pain points, analyze feedback and run successful CX programs with the best CX platform for your team.

Get the Guide Now

We’ve been avid users of the Voxco platform now for over 20 years. It gives us the flexibility to routinely enhance our survey toolkit and provides our clients with a more robust dataset and story to tell their clients.

VP Innovation & Strategic Partnerships, The Logit Group

• Client Stories
• Voxco Reviews
• Why Voxco Research?
• Careers at Voxco
• Vulnerabilities and Ethical Hacking

Explore Regional Offices

• Cloud/On-premise Dialer TCPA compliant Cloud on-premise dialer
• Predictive Analytics
• Customer 360
• Customer Loyalty
• Fraud & Risk Management
• AI/ML Enablement Services
• Credit Underwriting

Get Buyer’s Guide

• 100+ question types
• SMS surveys
• Banking & Financial Services
• Retail Solution
• Risk Management
• Customer Lifecycle Solutions
• Net Promoter Score
• Customer Behaviour Analytics
• Customer Segmentation
• Data Unification

Explore Voxco 

Watch a Demo 

Download Brochures 

• CX Strategy & Management Hub
• Blogs & White papers
• Case Studies

VP Innovation & Strategic Partnerships, The Logit Group

• Why Voxco Intelligence?
• Our clients
• Client stories
• Featuresheets

Pre-experimental Design: Definition, Types & Examples

• October 1, 2021

SHARE THE ARTICLE ON

Experimental research is conducted to analyze and understand the effect of a program or a treatment. There are three types of experimental research designs – pre-experimental designs, true experimental designs, and quasi-experimental designs . 

In this blog, we will be talking about pre-experimental designs. Let’s first explain pre-experimental research. 

What is Pre-experimental Research?

As the name suggests, pre- experimental research happens even before the true experiment starts. This is done to determine the researchers’ intervention on a group of people. This will help them tell if the investment of cost and time for conducting a true experiment is worth a while. Hence, pre-experimental research is a preliminary step to justify the presence of the researcher’s intervention. 

The pre-experimental approach helps give some sort of guarantee that the experiment can be a full-scale successful study. 

What is Pre-experimental Design?

The pre-experimental design includes one or more than one experimental groups to be observed against certain treatments. It is the simplest form of research design that follows the basic steps in experiments. 

The pre-experimental design does not have a comparison group. This means that while a researcher can claim that participants who received certain treatment have experienced a change, they cannot conclude that the change was caused by the treatment itself. 

The research design can still be useful for exploratory research to test the feasibility for further study. 

Let us understand how pre-experimental design is different from the true and quasi-experiments:

The above table tells us pretty much about the working of the pre-experimental designs. So we can say that it is actually to test treatment, and check whether it has the potential to cause a change or not. For the same reasons, it is advised to perform pre-experiments to define the potential of a true experiment.

See Voxco survey software in action with a Free demo.

Types of Pre-experimental Designs

Assuming now you have a better understanding of what the whole pre-experimental design concept is, it is time to move forward and look at its types and their working:

One-shot case study design

• This design practices the treatment of a single group.
• It only takes a single measurement after the experiment.
• A one-shot case study design only analyses post-test results.

The one-shot case study compares the post-test results to the expected results. It makes clear what the result is and how the case would have looked if the treatment wasn’t done. 

A team leader wants to implement a new soft skills program in the firm. The employees can be measured at the end of the first month to see the improvement in their soft skills. The team leader will know the impact of the program on the employees.

One-group pretest-posttest design

• Like the previous one, this design also works on just one experimental group.
• But this one takes two measures into account. 
• A pre-test and a post-test are conducted. 

As the name suggests, it includes one group and conducts pre-test and post-test on it. The pre-test will tell how the group was before they were put under treatment. Whereas post-test determines the changes in the group after the treatment. 

This sounds like a true experiment , but being a pre-experiment design, it does not have any control group. 

Following the previous example, the team leader here will conduct two tests. One before the soft skill program implementation to know the level of employees before they were put through the training. And a post-test to know their status after the training.

Now that he has a frame of reference, he knows exactly how the program helped the employees. 

Static-group comparison

• This compares two experimental groups.
• One group is exposed to the treatment.
• The other group is not exposed to the treatment.
• The difference between the two groups is the result of the experiment.

As the name suggests, it has two groups, which means it involves a control group too. 

In static-group comparison design, the two groups are observed as one goes through the treatment while the other does not. They are then compared to each other to determine the outcome of the treatment.

The team lead decides one group of employees to get the soft skills training while the other group remains as a control group and is not exposed to any program. He then compares both the groups and finds out the treatment group has evolved in their soft skills more than the control group. 

Due to such working, static-group comparison design is generally perceived as a quasi-experimental design too. 

Characteristics of Pre-experimental Designs

In this section, let us point down the characteristics of pre-experimental design:

• Generally uses only one group for treatment which makes observation simple and easy.
• Validates the experiment in the preliminary phase itself. 
• Pre-experimental design tells the researchers how their intervention will affect the whole study. 
• As they are conducted in the beginning, pre-experimental designs give evidence for or against their intervention.
• It does not involve the randomization of the participants. 
• It generally does not involve the control group, but in some cases where there is a need for studying the control group against the treatment group, static-group comparison comes into the picture. 
• The pre-experimental design gives an idea about how the treatment is going to work in case of actual true experiments.  

Validity of results in Pre-experimental Designs

Validity means a level to which data or results reflect the accuracy of reality. And in the case of pre-experimental research design, it is a tough catch. The reason being testing a hypothesis or dissolving a problem can be quite a difficult task, let’s say close to impossible. This being said, researchers find it challenging to generalize the results they got from the pre-experimental design, over the actual experiment. 

As pre-experimental design generally does not have any comparison groups to compete for the results with, that makes it pretty obvious for the researchers to go through the trouble of believing its results. Without comparison, it is hard to tell how significant or valid the result is. Because there is a chance that the result occurred due to some uncalled changes in the treatment, maturation of the group, or is it just sheer chance. 

Let’s say all the above parameters work just in favor of your experiment, you even have a control group to compare it with, but that still leaves us with one problem. And that is what “kind” of groups we get for the true experiments. It is possible that the subjects in your pre-experimental design were a lot different from the subjects you have for the true experiment. If this is the case, even if your treatment is constant, there is still going to be a change in your results. 

Advantages of Pre-experimental Designs

• Cost-effective due to its easy process. 
• Very simple to conduct.
• Efficient to conduct in the natural environment. 
• It is also suitable for beginners. 
• Involves less human intervention. 
• Determines how your treatment is going to affect the true experiment. 

Disadvantages of Pre-experimental Designs

• It is a weak design to determine causal relationships between variables. 
• Does not have any control over the research. 
• Possess a high threat to internal validity. 
• Researchers find it tough to examine the results’ integrity. 
• The absence of a control group makes the results less reliable. 

This sums up the basics of pre-experimental design and how it differs from other experimental research designs. Curious to learn how you can use survey software to conduct your experimental research, book a meeting with us . 

Explore all the survey question types possible on Voxco

Pre-experimental design is a research method that happens before the true experiment and determines how the researcher’s intervention will affect the experiment.

An example of a pre-experimental design would be a gym trainer implementing a new training schedule for a trainee.

Characteristics of pre-experimental design include its ability to determine the significance of treatment even before the true experiment is performed.

Researchers want to know how their intervention is going to affect the experiment. So even before the true experiment starts, they carry out a pre-experimental research design to determine the possible results of the true experiment.

The pre-experimental design deals with the treatment’s effect on the experiment and is carried out even before the true experiment takes place. While a true experiment is an actual experiment, it is important to conduct its pre-experiment first to see how the intervention is going to affect the experiment.

The true experimental design carries out the pre-test and post-test on both the treatment group as well as a control group. whereas in pre-experimental design, control group and pre-test are options. it does not always have the presence of those two and helps the researcher determine how the real experiment is going to happen.

The main difference between a pre-experimental design and a quasi-experimental design is that pre-experimental design does not use control groups and quasi-experimental design does. Quasi always makes use of the pre-test post-test model of result comparison while pre-experimental design mostly doesn’t.

Non-experimental research methods majorly fall into three categories namely: Cross-sectional research, correlational research and observational research.

Explore Voxco Survey Software

+ Omnichannel Survey Software 

+ Online Survey Software 

+ CATI Survey Software 

+ IVR Survey Software 

+ Market Research Tool

+ Customer Experience Tool 

+ Product Experience Software 

+ Enterprise Survey Software 

What is Customer Experience

What is Customer Experience? Free Download: Enhance NPS® Scores using our NPS® guide. Download Now SHARE THE ARTICLE ON Table of Contents Customer experience (CX)

Market intelligence : how external data impacts business working

Market Intelligence Research: The Key to Business Success See what question types are possible with a sample survey! Try a Sample Survey Table of Contents

Brain Research

Case Study: Brain Research SHARE THE ARTICLE ON   Brain Research observes 60% efficiency gains with Voxco Offline. “60% of the CAPI software we have

Customer 360 data model

Customer 360 data model SHARE THE ARTICLE ON Table of Contents What is customer 360? When it comes to data management and data warehousing, many

Employee Engagement Action Plan

Employee Engagement Action Plan SHARE THE ARTICLE ON Share on facebook Share on twitter Share on linkedin Table of Contents What is an Employee Engagement

Non-Probability Sampling: Definition, Method and Examples

Enhancing Response Rates in CATI Surveys: Strategies and Techniques SHARE THE ARTICLE ON Table of Contents Research studies often try to find correlations, differences, or

We use cookies in our website to give you the best browsing experience and to tailor advertising. By continuing to use our website, you give us consent to the use of cookies. Read More

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

8.2 Quasi-experimental and pre-experimental designs

Learning objectives.

• Identify and describe the various types of quasi-experimental designs
• Distinguish true experimental designs from quasi-experimental and pre-experimental designs
• Identify and describe the various types of quasi-experimental and pre-experimental designs

As we discussed in the previous section, time, funding, and ethics may limit a researcher’s ability to conduct a true experiment. For researchers in the medical sciences and social work, conducting a true experiment could require denying needed treatment to clients, which is a clear ethical violation. Even those whose research may not involve the administration of needed medications or treatments may be limited in their ability to conduct a classic experiment. When true experiments are not possible, researchers often use quasi-experimental designs.

Quasi-experimental designs

Quasi-experimental designs are similar to true experiments, but they lack random assignment to experimental and control groups. Quasi-experimental designs have a comparison group that is similar to a control group except assignment to the comparison group is not determined by random assignment. The most basic of these quasi-experimental designs is the nonequivalent comparison groups design (Rubin & Babbie, 2017).  The nonequivalent comparison group design looks a lot like the classic experimental design, except it does not use random assignment. In many cases, these groups may already exist. For example, a researcher might conduct research at two different agency sites, one of which receives the intervention and the other does not. No one was assigned to treatment or comparison groups. Those groupings existed prior to the study. While this method is more convenient for real-world research, it is less likely that that the groups are comparable than if they had been determined by random assignment. Perhaps the treatment group has a characteristic that is unique–for example, higher income or different diagnoses–that make the treatment more effective.

Quasi-experiments are particularly useful in social welfare policy research. Social welfare policy researchers often look for what are termed natural experiments , or situations in which comparable groups are created by differences that already occur in the real world. Natural experiments are a feature of the social world that allows researchers to use the logic of experimental design to investigate the connection between variables. For example, Stratmann and Wille (2016) were interested in the effects of a state healthcare policy called Certificate of Need on the quality of hospitals. They clearly could not randomly assign states to adopt one set of policies or another. Instead, researchers used hospital referral regions, or the areas from which hospitals draw their patients, that spanned across state lines. Because the hospitals were in the same referral region, researchers could be pretty sure that the client characteristics were pretty similar. In this way, they could classify patients in experimental and comparison groups without dictating state policy or telling people where to live.

Matching is another approach in quasi-experimental design for assigning people to experimental and comparison groups. It begins with researchers thinking about what variables are important in their study, particularly demographic variables or attributes that might impact their dependent variable. Individual matching involves pairing participants with similar attributes. Then, the matched pair is split—with one participant going to the experimental group and the other to the comparison group. An ex post facto control group , in contrast, is when a researcher matches individuals after the intervention is administered to some participants. Finally, researchers may engage in aggregate matching , in which the comparison group is determined to be similar on important variables.

Time series design

There are many different quasi-experimental designs in addition to the nonequivalent comparison group design described earlier. Describing all of them is beyond the scope of this textbook, but one more design is worth mentioning. The time series design uses multiple observations before and after an intervention. In some cases, experimental and comparison groups are used. In other cases where that is not feasible, a single experimental group is used. By using multiple observations before and after the intervention, the researcher can better understand the true value of the dependent variable in each participant before the intervention starts. Additionally, multiple observations afterwards allow the researcher to see whether the intervention had lasting effects on participants. Time series designs are similar to single-subjects designs, which we will discuss in Chapter 15.

Pre-experimental design

When true experiments and quasi-experiments are not possible, researchers may turn to a pre-experimental design (Campbell & Stanley, 1963).  Pre-experimental designs are called such because they often happen as a pre-cursor to conducting a true experiment.  Researchers want to see if their interventions will have some effect on a small group of people before they seek funding and dedicate time to conduct a true experiment. Pre-experimental designs, thus, are usually conducted as a first step towards establishing the evidence for or against an intervention. However, this type of design comes with some unique disadvantages, which we’ll describe below.

A commonly used type of pre-experiment is the one-group pretest post-test design . In this design, pre- and posttests are both administered, but there is no comparison group to which to compare the experimental group. Researchers may be able to make the claim that participants receiving the treatment experienced a change in the dependent variable, but they cannot begin to claim that the change was the result of the treatment without a comparison group.   Imagine if the students in your research class completed a questionnaire about their level of stress at the beginning of the semester.  Then your professor taught you mindfulness techniques throughout the semester.  At the end of the semester, she administers the stress survey again.  What if levels of stress went up?  Could she conclude that the mindfulness techniques caused stress?  Not without a comparison group!  If there was a comparison group, she would be able to recognize that all students experienced higher stress at the end of the semester than the beginning of the semester, not just the students in her research class.

In cases where the administration of a pretest is cost prohibitive or otherwise not possible, a one- shot case study design might be used. In this instance, no pretest is administered, nor is a comparison group present. If we wished to measure the impact of a natural disaster, such as Hurricane Katrina for example, we might conduct a pre-experiment by identifying  a community that was hit by the hurricane and then measuring the levels of stress in the community.  Researchers using this design must be extremely cautious about making claims regarding the effect of the treatment or stimulus. They have no idea what the levels of stress in the community were before the hurricane hit nor can they compare the stress levels to a community that was not affected by the hurricane.  Nonetheless, this design can be useful for exploratory studies aimed at testing a measures or the feasibility of further study.

In our example of the study of the impact of Hurricane Katrina, a researcher might choose to examine the effects of the hurricane by identifying a group from a community that experienced the hurricane and a comparison group from a similar community that had not been hit by the hurricane. This study design, called a static group comparison , has the advantage of including a comparison group that did not experience the stimulus (in this case, the hurricane). Unfortunately, the design only uses for post-tests, so it is not possible to know if the groups were comparable before the stimulus or intervention.  As you might have guessed from our example, static group comparisons are useful in cases where a researcher cannot control or predict whether, when, or how the stimulus is administered, as in the case of natural disasters.

As implied by the preceding examples where we considered studying the impact of Hurricane Katrina, experiments, quasi-experiments, and pre-experiments do not necessarily need to take place in the controlled setting of a lab. In fact, many applied researchers rely on experiments to assess the impact and effectiveness of various programs and policies. You might recall our discussion of arresting perpetrators of domestic violence in Chapter 2, which is an excellent example of an applied experiment. Researchers did not subject participants to conditions in a lab setting; instead, they applied their stimulus (in this case, arrest) to some subjects in the field and they also had a control group in the field that did not receive the stimulus (and therefore were not arrested).

Key Takeaways

• Quasi-experimental designs do not use random assignment.
• Comparison groups are used in quasi-experiments.
• Matching is a way of improving the comparability of experimental and comparison groups.
• Quasi-experimental designs and pre-experimental designs are often used when experimental designs are impractical.
• Quasi-experimental and pre-experimental designs may be easier to carry out, but they lack the rigor of true experiments.
• Aggregate matching – when the comparison group is determined to be similar to the experimental group along important variables
• Comparison group – a group in quasi-experimental design that does not receive the experimental treatment; it is similar to a control group except assignment to the comparison group is not determined by random assignment
• Ex post facto control group – a control group created when a researcher matches individuals after the intervention is administered
• Individual matching – pairing participants with similar attributes for the purpose of assignment to groups
• Natural experiments – situations in which comparable groups are created by differences that already occur in the real world
• Nonequivalent comparison group design – a quasi-experimental design similar to a classic experimental design but without random assignment
• One-group pretest post-test design – a pre-experimental design that applies an intervention to one group but also includes a pretest
• One-shot case study – a pre-experimental design that applies an intervention to only one group without a pretest
• Pre-experimental designs – a variation of experimental design that lacks the rigor of experiments and is often used before a true experiment is conducted
• Quasi-experimental design – designs lack random assignment to experimental and control groups
• Static group design – uses an experimental group and a comparison group, without random assignment and pretesting
• Time series design – a quasi-experimental design that uses multiple observations before and after an intervention

Image attributions

cat and kitten   matching avocado costumes on the couch looking at the camera by Your Best Digs CC-BY-2.0

Foundations of Social Work Research Copyright © 2020 by Rebecca L. Mauldin is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Our websites may use cookies to personalize and enhance your experience. By continuing without changing your cookie settings, you agree to this collection. For more information, please see our University Websites Privacy Notice .

Neag School of Education

Educational Research Basics by Del Siegle

Experimental research designs.

R=Random Assignment X= Treatment O=Observation (Assessment)

R = Random Assignment X = Treatment O = Observation (test or measurement of some type)

Del Siegle, Ph.D. Neag School of Education – University of Connecticut [email protected] www.delsiegle.com

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Time series experimental design under one-shot sampling: The importance of condition diversity

Xiaohan Kang

1 Coordinated Science Laboratory and Department of Electrical and Computer Engineering, University of Illinois at Urbana–Champaign, Urbana, Illinois, United States of America

Bruce Hajek

2 Department of Biology, California State University, Northridge, Northridge, California, United States of America

Yoshie Hanzawa

Associated data.

The computer simulation code is available at https://github.com/Veggente/one-shot-sampling .

Many biological data sets are prepared using one-shot sampling, in which each individual organism is sampled at most once. Time series therefore do not follow trajectories of individuals over time. However, samples collected at different times from individuals grown under the same conditions share the same perturbations of the biological processes, and hence behave as surrogates for multiple samples from a single individual at different times. This implies the importance of growing individuals under multiple conditions if one-shot sampling is used. This paper models the condition effect explicitly by using condition-dependent nominal mRNA production amounts for each gene, it quantifies the performance of network structure estimators both analytically and numerically, and it illustrates the difficulty in network reconstruction under one-shot sampling when the condition effect is absent. A case study of an Arabidopsis circadian clock network model is also included.

Introduction

Time series data is important for studying biological processes in organisms because of the dynamic nature of the biological systems. Ideally it is desirable to use time series with multi-shot sampling , where each individual (such as a plant, animal, or microorganism) is sampled multiple times to produce the trajectory of the biological process, as in Fig 1 . Then the natural biological variations in different individuals can be leveraged for statistical inference, and thus inference can be made even if the samples are prepared under the same experimental condition.

Each plant is observed four times.

However, in many experiments multi-shot sampling is not possible. Due to stress response of the organisms and/or the large amount of cell tissue required for accurate measurements, the dynamics of the relevant biological process in an individual of the organism cannot be observed at multiple times without interference. For example, in an RNA-seq experiment an individual plant is often only sampled once in its entire life, leaving the dynamics unobserved at other times. See the Discussion section for a review of literature on this subject. We call the resulting time series data, as illustrated in Fig 2 , a time series with one-shot sampling . Because the time series with one-shot sampling do not follow the trajectories of the same individuals, they do not capture all the correlations in the biological processes. For example, the trajectory of observations on plants 1–2–3–4 and that on 1–6–7–4 in Fig 2 are statistically identical. The resulting partial observation renders some common models for the biological system dynamics inaccurate or even irrelevant.

Each plant is observed once.

To address this problem, instead of getting multi-shot time series of single individuals, one can grow multiple individuals under each condition with a variety of conditions, and get one-shot time series of the single conditions. The one-shot samples from the same condition then become a surrogate for multi-shot samples for a single individual, as illustrated in Fig 3 . In essence, if we view the preparation condition of each sample as being random, then there should be a positive correlation among samples grown under the same condition. We call this correlation the condition variation effect . It is similar to the effect of biological variation of a single individual sampled at different times, if such sampling were possible.

For each condition, the one-shot samples at different times are also complemented by biological replicates , which are samples from independent individuals taken at the same time used to reduce technical and/or biological variations. See the Discussion section for a review on how replicates are used for biological inference. With a budget over the number of samples, a balance must be kept between the number of conditions, the number of sampling times and the number of replicates.

To illustrate and quantify the effect of one-shot sampling in biological inference, we introduce a simple dynamic gene expression model with a condition variation effect. We consider a hypothesis testing setting and model the dynamics of the gene expression levels at different sampling times by a dynamic Bayesian network (DBN), where the randomness comes from nominal (or basal) biological and condition variations for each gene. The nominal condition-dependent variation of gene j is the same for all plants in that condition and the remaining variation is biological and is independent across the individuals in the condition. In contrast to GeneNetWeaver [ 1 ], where the effect of a condition is modeled by a random perturbation to the network coefficients, in our model the condition effect is characterized by correlation in the nominal variation terms of the dynamics. Note in both models samples from different individuals under the same condition are statistically independent given the randomness associated with the condition.

The contributions of this paper are threefold.

• A composite hypothesis testing problem on the gene regulatory network is formulated and a gene expression dynamic model that explicitly captures the per-gene condition effect and the gene regulatory interactions is proposed.
• The performance of gene regulatory network structure estimators is analyzed for both multi-shot and one-shot sampling, with focus on two algorithms. Furthermore, single-gene and multi-gene simulation results indicate that multiple-condition experiments can somewhat mitigate the shortcomings of one-shot sampling.
• The difficulty of network reconstruction under one-shot sampling with no condition effect is illustrated. This difficulty connects network analysis and differential expression analysis, two common tasks in large-scale genomics studies, in the sense that the part of network involving non-differentially expressed genes may be harder to reconstruct.

The simulation code for generating the figures is available at [ 2 ].

Materials and methods

Stochastic model of time-series samples.

This section formulates the hypothesis testing problem of learning the structure of the gene regulatory network (GRN) from gene expression data with one-shot or multi-shot sampling. The GRN is characterized by an unknown adjacency matrix. Given the GRN, a dynamic Bayesian network model is used for the gene expression evolution with time. Two parameters σ co, j and σ bi, j are used for each gene j , with the former explicitly capturing the condition variation effect and the latter capturing the biological variation level.

For any positive integer n , let [ n ] = {1, 2, …, n }. We use ( f ( x ) ) x ∈ I to denote the family of elements in the set { f ( x ) : x ∈ I } indexed by the index set I . The indicator function on a statement or a set P is denoted by 1 P . The n -by- n identity matrix is denoted by I n . The transpose of matrix A is denoted by A *.

Model for gene regulatory network topology

Let n be the number of genes and let A ∈ A ⊆ R n × n be the unknown adjacency matrix of the GRN. The sign of the entry a ij of A for i ≠ j indicates the type of regulation of j by i , and the absolute value the strength of the regulation. A zero entry a ij = 0 with i ≠ j indicates no regulation of j by i . The diagonal of A characterizes protein concentration passed from the previous time, protein degradation, and gene autoregulation. Let S = { S 1 , S 2 , … , S | S | } be a finite set of network structures and let D be a mapping from A to S ; D ( A ) represents the network structure of an adjacency matrix A . Then A is partitioned by the associated network structures. Fix a loss function l : S 2 → R . Let Y ∈ Y be the random observation and let δ : Y → S be an estimator for the structure. The performance of an estimator is evaluated by the expected loss E l ( D ( A ) , δ ( Y ) ) . This is a hypothesis testing problem with composite hypotheses { D - 1 ( S ) : S ∈ S } . This paper considers network reconstruction up to regulation type with D ( A ) = ( sgn ( A i j ) ) ( i , j ) ∈ [ n ] 2 , where sgn ( s ) = 1 { s > 0 } - 1 { s < 0 } . In other words, the ternary value of the edge signs (positive, negative, or no edge) are to be recovered. A structure S has the form S = ( S i j ) ( i , j ) ∈ [ n ] 2 with S ij ∈ {0, 1, −1}, and it can be interpreted as a directed graph with possible self-loops. Some examples of loss functions are as follows.

• Ternary false discovery rate (FDR) l FDR ( S , S ′ ) = 1 - ∑ i = 1 n ∑ j = 1 n 1 { S i j = S i j ′ ≠ 0 } ∑ i = 1 n ∑ j = 1 n 1 { S i j ′ ≠ 0 } .
• Ternary false negative rate (FNR) l FNR ( S , S ′ ) = 1 - ∑ i = 1 n ∑ j = 1 n 1 { S i j = S i j ′ ≠ 0 } ∑ i = 1 n ∑ j = 1 n 1 { S i j ≠ 0 } .
• Ternary false positive rate (FPR) l FPR ( S , S ′ ) = 1 - ∑ i = 1 n ∑ j = 1 n 1 { S i j = S i j ′ = 0 } ∑ i = 1 n ∑ j = 1 n 1 { S i j = 0 } .
• Ternary error rate l E ( S , S ′ ) = 1 n 2 ∑ i = 1 n ∑ j = 1 n 1 { S i j ≠ S i j ′ } .

Note the FDR and the FNR are well-defined when S ′ and S contains nonzero elements, respectively, and the FPR is well-defined when S contains zeros. The error rate is always well-defined. It can be seen that l FDR ( S , S ′) = l FNR ( S ′, S ). Also if S does not contain zeros then l FNR ( S , S ′) = l E ( S , S ′). Similarly if S ′ does not contain zeros then l FNR ( S , S ′) = l E ( S , S ′). As an example, for a random guessing algorithm with probabilities of S i j ′ = 0 , 1 , - 1 being 1 − q , q /2, q /2 and a network prior with probabilities of S ij = 0, 1, −1 being 1 − p , p /2, p /2, l FDR = 1 − p /2, l FNR = 1 − q /2, and l FPR = q .

Model for gene expression dynamics

This section models the gene expression dynamics of individuals by a dynamic Bayesian networks with parameters σ co, j and σ bi, j as the condition variation level and biological variation level for gene j .

Let K , T and C be the number of individuals, sampling times, and conditions, respectively. Let X j k ( t ) ∈ R be the expression level of gene j ∈ [ n ] in individual k ∈ [ K ] at time t ∈ [ T ], and let c k ∈ [ C ] be the label that indicates the condition for individual k . Here we assume X j k ( t ) represents both the mRNA abundance and the protein concentration. The gene expression levels evolve according to the Gaussian linear model (GLM) with initial condition X j k ( 0 ) = 0 for any j ∈ [ n ], k ∈ [ K ] and the following recursion (note the values of X can be the expression levels after a logarithm transform, in which case lowly expressed genes have negative X values)

for j ∈ [ n ], k ∈ [ K ], and t ∈ {0, 1, …, T −1}, where ( W co , j c ( t ) ) ( c , j , t ) ∈ [ C ] × [ n ] × [ T ] and ( W bi , k j ( t ) ) ( j , k , t ) ∈ [ n ] × [ K ] × [ T ] are collections of independent standard Gaussian random variables that are used to drive the dynamics. Here the last two terms in ( 1 ) denote the condition variation and biological variation, respectively. To write ( 1 ) in matrix form, we let X ( t ) = ( X j k ( t ) ) ( k , j ) ∈ [ K ] × [ n ] and W ( t ) = ( W j k ( t ) ) ( k , j ) ∈ [ K ] × [ n ] be K -by- n matrices, where W j k ( t ) = σ co , j W co , j c k ( t ) + σ bi , j W bi , j k ( t ) . Then

The variable W j k ( t ) is the nominal mRNA production amount for target gene j , individual k at time t that would occur in the absence of regulation by other genes.

Model for sampling method

In this section two sampling methods are described: one-shot sampling and multi-shot sampling. For simplicity, throughout this paper we consider a full factorial design with CRT samples obtained under C conditions, R replicates and T sampling times, denoted by Y = ( Y c , r , t ) ( c , r , t )∈[ C ]×[ R ]×[ T ] . In other words, instead of X we observe Y , a noisy and possibly partial observation of X . Here the triple index for each sample indicates the condition, replicate, and time. As we will see in the Discussion at the end of this section, for either sampling method, the biological variation level σ bi, j can be reduced by combining multiple individuals to form a single sample.

Multi-shot sampling

Assume an individual can be sampled multiple times. This sampling model corresponds to K = CR and c k = ⌈ k R ⌉ ∈ [ C ] for all k ∈ [ K ]. Equivalently, multi-index ( c , r ) can be used to determine the individual instead of k for X and W with c denoting the condition and r the replicate. Then ( 1 ) for multi-shot sampling can be rewritten as

and the observation for condition c , replicate r and time t is

with ( Z j c , r , t ) ( j , c , r , t ) ∈ [ n ] × [ C ] × [ R ] × [ T ] being a collection of independent standard Gaussian random variables modeling the observation noise, and σ te, j is the technical variance level of gene j . We see that for fixed c and r the observations at different times are from the same individual with the multi-index ( c , r ). As a result, with multi-shot sampling Y is a noisy full observation of X .

One-shot sampling

Assume an individual can be sampled only once. This model corresponds to K = CRT and c k = ⌈ k R T ⌉ ∈ [ C ] for all k ∈ [ K ]. Equivalently, with multi-index ( c , r , s ) denoting the condition, the replicate, and the target sampling time, the evolution ( 1 ) for one-shot sampling can be rewritten as

and the observation is

Again σ te, j is the observation noise level of gene j and the Z ’s are independent standard Gaussian random variables. Note that for fixed c and r the observations at different times are from different individuals because the triple indices are different. Hence with one-shot sampling, Y is a noisy partial observation of X (to see this, note for gene 1 and the individual indexed by condition 1, replicate 1, and target sampling time 1, X 1 1 , 1 , 1 ( 1 ) , which is the expression level at time 1, is observed through Y 1 1 , 1 , 1 but X 1 1 , 1 , 1 ( 2 ) , which is the expression level at time 2, is not observed).

Discussion on sources of variance

The σ co , j W co , j c ( t ) terms measure the condition-dependent nominal production level as global driving noise terms that are shared across individuals under the same condition. They are independent and identically distributed (i.i.d.) across conditions. The σ bi , j W bi , j k ( t ) terms measure the biological nominal production level of individuals as local driving noise terms. They are i.i.d. across individuals. The σ te , j Z j c , r , t terms measure the technical variation of samples as additive observational noise terms that are not in the evolution of X . They are i.i.d. across samples. We then have the following observations.

• If the samples of the individuals under many different conditions are averaged and treated as a single sample, then effectively σ co, j , σ bi, j and σ te, j are reduced.
• If the samples of R individuals under same conditions (biological replicates) are averaged and treated as a single sample, then effectively σ bi , j 2 and σ te , j 2 are reduced by a factor of R while σ co , j 2 remains unchanged.
• If material from multiple individuals grown under the same condition is combined into a composite sample before measuring, then effectively σ bi, j is reduced while σ co, j and σ te, j remain unchanged. Note for microorganisms a sample may consist of millions of individuals and the biological variation is practically eliminated ( σ bi, j ≈ 0).
• If the samples from same individuals (technical replicates) are averaged and treated as a single sample, then effectively σ te, j is reduced while σ co, j and σ bi, j remain unchanged.

Note this sampling model with hierarchical driving and observational noises can also be used for single-cell RNA sequencing (scRNAseq) in addition to bulk RNA sequencing and microarray experiments. For scRNAseq, σ co, j can model the tissue-dependent variation (the global effect) and σ bi, j the per-cell variation (the local effect).

Performance evaluation of network structure estimators

This section studies the performance of network structure estimators with multi-shot and one-shot sampling data. First, general properties of the two sampling methods are obtained. Then two algorithms, the generalized likelihood-ratio test (GLRT) and the basic sparse linear regression (BSLR), are studied. The former is a standard decision rule for composite hypothesis testing problems and is shown to have some properties but is computationally infeasible for even a small number of genes. The latter is an algorithm based on linear regression, and is feasible for a moderate number of genes. Finally simulation results for a single-gene network with GLRT and for a multi-gene network with BSLR are shown.

General properties

By ( 3 ), ( 4 ) and ( 5 ), the samples Y are jointly Gaussian with zero mean. The covariance of the random tensor Y is derived under the two sampling methods in the following.

Under multi-shot sampling, the samples under different conditions are independent and hence uncorrelated. Consider Y c , r , t and Y c , r ′, t ′ , which are two samples under the same condition and collected at times t and t ′. The covariance matrix between Y c , r , t and Y c , r ′, t ′ is the sum of the covariance matrices of their common variations at times τ for 1 ≤ τ ≤ t ∧ t ′ multiplied by ( A *) t − τ on the left and A t ′− τ on the right, plus covariance for the observation noise. Let Σ co = diag ( σ co , 1 2 , σ co , 2 2 , … , σ co , n 2 ) , Σ bi = diag ( σ bi , 1 2 , σ bi , 2 2 , … , σ bi , n 2 ) , and Σ te = diag ( σ te , 1 2 , σ te , 2 2 , … , σ te , n 2 ) . Then the covariance matrix of the variations is Σ co + Σ bi if the two samples are from the same individual (i.e., r = r ′), and Σ co otherwise. This yields:

Under one-shot sampling the only difference compared with multi-shot sampling is that two samples indexed by ( c , r , t ) and ( c , r , t ′) are from different individuals if t ≠ t ′. So

For any fixed network structure estimator:

• If Σ bi = 0 and C , R and T are fixed, the joint distribution of the data is the same for both types of sampling. So the performance of the estimator would be the same for multi-shot and one-shot sampling.
• If Σ bi = 0 and Σ te = 0 (no observation noise) and C , T are fixed, the joint distribution of the data is the same for both types of sampling (as noted in item 1) and any replicates beyond the first are identical to the first. So the performance of the estimator can be obtained even if all replicates beyond the first are discarded.
• Under multi-shot sampling, when C , R , T are fixed with R = 1, the joint distribution of the data depends on Σ co and Σ bi only through their sum. So the performance of the estimator would be the same for all Σ co and Σ bi such that Σ co + Σ bi is the same.
• In the homogeneous gene case with σ co, j = σ co , σ bi, j = σ bi , σ te, j = σ te for all j with σ co * + σ bi * + σ te * > 0 , suppose that the estimator δ is based on replicate averages y = ( y c , t ) ( c , t )∈[ C ]×[ T ] with y c , t = 1 R ∑ r = 1 R Y c , r , t , and that δ is scale-invariant (i.e., δ ( Y ) = δ ( c 0 Y ) for any c 0 ≠ 0 and Y ). Then under multi-shot sampling, δ ’s performance depends on σ co , σ bi , σ te and R only through the ratio σ te 2 / R σ co 2 + σ bi 2 / R + σ te 2 / R . Under one-shot sampling, the estimator’s performance depends on σ co , σ bi , σ te and R only through the ratios σ te 2 / R σ co 2 + σ bi 2 / R + σ te 2 / R and σ co 2 σ co 2 + σ bi 2 / R (through the latter only when σ co 2 + σ bi 2 > 0 ).

To see 4), recall from observation 2 above that averaging reduces the variance of the biological variation and that of the observation noise by a factor of R due to independence, but preserves the condition variation because it is identical across replicates. Hence the variance of the driving noise in the averages is σ co 2 + σ bi 2 / R and the variance of the observation noise of the averages is σ te 2 / R . Then the averages are essentially single-replicate data, and the performance under multi-shot sampling depends only on the ratio of the new driving noise variance to the new observational noise variance. For one-shot sampling the ratio between the condition variation and the biological variation also matters for the single-replicate data when the condition variation and the biological variation are not both zero, so the performance also depends on σ co 2 σ co 2 + σ bi 2 / R .

Network reconstruction algorithms

In this section we introduce GLRT and BSLR. GLRT is a standard choice in composite hypothesis testing setting. We observe some properties for GLRT under one-shot and multi-shot sampling. However, GLRT involves optimizing the likelihood over the entire parameter space, which grows exponentially with the square of the number of genes. Hence GLRT is hard to compute for multiple-gene network reconstruction. In contrast, BSLR is an intuitive algorithm based on linear regression, and will be shown in simulations to perform reasonably well for multi-gene scenarios.

The GLRT (see, e.g., page 38, Chapter II.E in [ 3 ]) is given by δ ( y ) = D ( A ^ ML ( y ) ) , where A ^ ML ( y ) is the maximum-likelihood estimate for A based on the covariance of Y given the observation Y = y .

Proposition 1 GLRT ( with the knowledge of Σ co , Σ bi and Σ te ) has the following properties .

• For a fixed σ 2 , under multi-shot sampling with Σ te = 0 ( no observation noise ), σ co, j = σ co , σ bi, j = σ bi , and σ co 2 + σ bi 2 = σ 2 , the performance of GLRT for sign estimation is the same for all ( R , σ co , σ bi ) excluding ( R ≥ 2, σ bi = 0).
• Under one-shot sampling and Σ co = 0, the log likelihood of the data as a function of A ( i . e . the log likelihood function ) is invariant with respect to replacing A by − A . So , for the single-gene n = 1 case , the log likelihood function is an even function of A , and thus the GLRT will do no better than random guessing .

For 2 it suffices to notice in ( 6 ) the covariance is invariant with respect to changing A to − A . A proof of 1 is given below.

Proof of 1) We first prove it for the case of a single gene with constant T and a constant number of individuals, CR . To do that we need to look at the likelihood function closely.

We may assume σ 2 = 1. Because the trajectories for different conditions are independent (for given parameters ( A , σ co 2 ) ), we shall first consider the case with a single condition; i.e., C = 1. There are hence R trajectories of length T . Then the covariance matrix of the length- R driving vector used at time t for the trajectories is

When σ co > 0, Σ is not the identity matrix multiplied by some constant; i.e., the noise vector W ( t ) is colored across replicates. It can be checked when σ co < 1 (i.e., σ bi > 0) the matrix Σ is positive definite. Then there exists an orthogonal matrix U and a diagonal matrix Λ with positive diagonal elements such that Σ = U Λ U *. Let Σ −1/2 = U Λ −1/2 U * and let

for all t ∈ [ T ]. Then the trajectories for the R replicates in a single condition become:

It can be checked that after the linear transformation by Σ −1/2 , which does not depend on A , the new driving vectors are white (i.e., Cov ( W ˜ ( t ) ) = I R ). It follows that the distribution of X ˜ | ( A , σ co 2 ) is the same as the distribution of X |( A , 0) (i.e. σ co = 0). Therefore, for x = ( x r ( t ) ) ( r , t ) ∈ [ R ] × [ T ] , if we let L X ( x | A , σ co 2 ) denote the likelihood of X = x for parameters A , σ co 2 , then

where d ( R , T , σ co 2 ) = ( det Σ ) - T / 2 is a function of R , T and σ co 2 , and x ˜ ( t ) = Σ - 1 / 2 x ( t ) .

Now consider the likelihood function for all CRT samples with general C . It is the product of C likelihood functions for the samples prepared under the C different conditions. It is thus equal to d ( R , T , σ co 2 ) C times the likelihood of the transformed expression levels x ˜ , which is the likelihood function for σ co = 0 and a total of CRT samples. The form of the product depends on C and R only through CR , because under the transformation, all CR trajectories are independent. Hence, for fixed A , σ co 2 , C , R , T the distribution of the maximum likelihood estimate of A , when the samples are generated using a given σ co > 0 (so the R individuals under each condition are correlated) and the likelihood function also uses σ co 2 , is the same as the distribution of the maximum likelihood estimate of A when σ co = 0 (in which case the CR individual trajectories are i.i.d.). Formally,

where E σ co denotes that the condition variation level of the random elements X and Y is σ co 2 . The above fails if σ co = 1 (i.e., σ bi = 0) and R ≥ 2 because then Σ is singular. It also fails if σ co and σ bi are unknown to the GLRT.

For the general model with multiple genes, if σ co, j is the same for each gene j , 1) holds as before—for the proof, apply left multiplication by Σ - 1 2 for each gene, time, and condition to all R samples in the condition. View ( 2 ) as an update equation for an R × n matrix for each group of R samples in one condition. One column of length R per gene, and one row per sample.

In BSLR, replicates are averaged and the average gene expression levels at different times under different conditions are fitted in a linear regression model with best-subset sparse model selection, followed by a Granger causality test to eliminate the false discoveries. BSLR is similar to other two-stage linear regression–based network reconstruction algorithms, notably oCSE [ 4 ] and CaSPIAN [ 5 ]. Both oCSE and CaSPIAN use greedy algorithms in the first build-up stage, making them more suitable for large-scale problems. In contrast, BSLR uses best subset selection, which is conceptually simpler but computationally expensive for large n . For the tear-down stage both BSLR and CaSPIAN use the Granger causality test, while oCSE uses a permutation test.

Build-up stage

In the first stage BSLR finds potential regulatory interactions using a linear regression model. Let Y j c ( t ) = 1 R ∑ r = 1 R Y c , r , t and let Y ( t ) = ( Y j c ( t ) ) ( c , j ) ∈ [ C ] × [ n ] denote the C -by- n matrix. Let

For each target gene j ∈ [ n ], BSLR solves the following best subset selection problem with a subset size k < n :

Denote the solution by ( A *, b *, d *). The output of the first stage is then A *.

A naive algorithm to solve the above optimization has a computational complexity of O ( n k +1 ) for fixed k as n → ∞. Faster near-optimal alternatives exist [ 6 ].

Tear-down stage

The second stage is the same as that of CaSPIAN. For each j ∈ [ n ] and each i ∈ supp ( A · j * ) , let the unrestricted residual sum of squares be

and the restricted residual sum of squares

The F -statistic is given by

The potential parent i of j is removed in the tear-down stage if the p -value of the F -statistic with degrees of freedom (1, C ( T − 1) − k − 2) is above the preset significance level (e.g., 0.05). Note the tests are done for all parents in A ⋅ j simultaneously; both the restricted and the unrestricted models contain the other potential parents regardless of the results of the tests on them.

Simulations on single-gene network reconstruction

The GLM is used to simulate one-shot sampling data with a single gene. The goal is to determine the type of autoregulation of the single gene (activation or repression). The protein concentration passed from the previous time is ignored so the type of autoregulation is represented by the sign of the scalar A . In order to compare one-shot and multi-shot sampling, we view the main expense to be proportional to the number of samples to prepare as opposed to the number of individuals to grow. We thus fix a total budget of CRT = 180 samples and consider full factorial design with C and R varying with CR = 30, and T = 6 with 10 000 simulations. We assume the knowledge of the existence of the autoregulation (i.e., A ≠ 0), in which case the FDR, the FNR and the error rate coincide, so we only look at error rates. The results are plotted in Fig 4 . The four plots on the left are for one-shot sampling and the four on the right are for multi-shot sampling. Consider the homogeneous case with σ co, j = σ co , σ bi, j = σ bi and σ te, j = σ te for all j and let γ = σ co 2 σ co 2 + σ bi 2 be the fraction of condition variation in the driving noise. For each plot the observed probability of sign (of A ) error is shown for γ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0} and for R ranging over the divisors of 30 from smallest to largest. Fig 4a–4d show the performance for the GLRT algorithm assuming no observation noise ( σ te = 0), known γ and known total driving variation σ 2 = σ co 2 + σ bi 2 = 1 . Fig 4e–4h show the performance for the GLRT algorithm assuming known driving noise level σ = 1 and observational noise level σ te = 1, while both γ and A are unknown to the algorithm.

The numerical simulations reflect the following observations implied by the analytical model.

• Under one-shot sampling, when γ = 0, the GLRT is equivalent to random guessing.
• The GLRT performs the same under one-shot and multi-shot sampling when γ = 1.
• Under no observation noise, the performance for multi-shot sampling is the same for all γ < 1.

Some empirical observations are in order.

• Multi-shot sampling outperforms one-shot sampling (unless γ = 1, where they have the same error probability).
• For one-shot sampling, the performance improves as γ increases. Regarding the number of replicates R per condition, if γ = 0.2 (small condition effect), a medium number of replicates (2 to 5) outperforms the single replicate strategy. For larger γ , one replicate per condition is the best.
• For multi-shot sampling, performance worsens as γ increases. One replicate per condition ( R = 1) is best.
• Comparing Fig 4a–4d vs. Fig 4e–4h , we observe that the performance degrades with the addition of observation noise, though for moderate noise ( σ te = 1.0) the effect of observation noise on the sign error is not large. Also, the effect of the algorithm not knowing γ is not large.

Simulations on multi-gene network reconstruction

This subsection studies the case when multiple genes interact through the GRN. The goal is to compare one-shot vs. multi-shot sampling for BSLR under a variety of scenarios, including different homogeneous γ values, varying number of replicates, varying observation noise level, and heterogeneous γ values.

The performance evaluation for multi-gene network reconstruction is trickier than the single-gene case because of the many degrees of freedom introduced by the number of genes. First, the network adjacency matrix A is now an n -by- n matrix. While some notion of “size” of A (like the spectral radius or the matrix norm) may be important, potentially every entry of A may affect the reconstruction result. So instead of fixing a ground truth A as in Fig 4 , we fix a prior distribution of A with split Gaussian prior described in S2 Appendix (note we assume the knowledge of no autoregulation), and choose A i.i.d. from the prior distribution with d max = 3. Second, because the prior of A can be subject to sparsity constraints and thus far from a uniform distribution, multiple loss functions that are more meaningful than the ternary error rate can be considered for performance. So we consider ternary FDR, ternary FNR and ternary FPR for the multi-gene case. In the simulations we have 20 genes and d max = 3 with in-degree uniformly distributed over {0, 1, …, d max }, so the average in-degree is 1.5. The number of sampling times is T = 6 and CR = 30.

Varying γ , R and σ te

In this set of simulations we fix the observation noise level and vary the number of replicates R and the condition correlation coefficient γ . The performance of BSLR under one-shot and multi-shot sampling is shown in Fig 5 ( σ te = 0) and Fig 6 ( σ te = 1). Note BSLR does not apply to a single condition with 30 replicates due to the constraint that the degrees of freedom C ( T − 1) − k − 2 in the second stage must be at least 1.

For one-shot sampling, when γ = 0, we see in both Figs ​ Figs5 5 and ​ and6 6 that BSLR is no different from random guessing, with an FDR close to 1 - 1 2 1 . 5 19 ≈ 0 . 96 and an FNR and an FPR such that l FNR + 1 2 l FPR ≈ 1 (recall the example of random guessing at the end of the section of the model for gene regulatory network topology). When γ = 1, BSLR performs similarly with one-shot or multi-shot sampling, which is consistent with property 1 in the section on general properties. As γ increases from 0 to 1, under one-shot sampling for a fixed number of replicates, the FDR and FNR reduce greatly. For example, as γ increases from 0.2 to 1, the FDR for single replicate under one-shot sampling decreases from 0.74 to 0.31 with noiseless data ( Fig 5 ), and from 0.79 to 0.36 with noisy data ( Fig 6 ), while the FNR decreases from 0.70 to 0.00 with noiseless data, and from 0.78 to 0.04 with noisy data. This decrease is more pronounced for smaller number of replicates. Note the trend of the performance of BSLR under one-shot sampling as a function of R and γ is very similar to that of GLRT in Fig 4e and 4g .

For multi-shot sampling, in the noiseless case, we see all three losses are invariant with respect to different γ for fixed R , which is consistent with property 4 in the section on general properties because BSLR is an average-based scale-invariant algorithm (note CR is a constant so for different R the performance is different due to the change in C ). In the noisy case, the FDR and FNR slightly decrease as γ increases, which is an opposite trend compared with Fig 4f and 4h .

In summary, the main conclusions from Figs ​ Figs5 5 and ​ and6 6 are the following.

• The performance of BSLR under multi-shot sampling is consistently better than that under one-shot sampling.
• The performance of BSLR under one-shot sampling varies with γ , from random guessing performance at γ = 0 to the same performance as multi-shot sampling at γ = 1.
• By comparing Figs ​ Figs5 5 with ​ with6, 6 , we see the observation noise of σ te = 1 has only a small effect on the performance with the two sampling methods.

Reduced number of directly differentially expressed genes

In the above simulations we have assumed all genes are equally directly differentially expressed. In other words, we took σ co , j 2 + σ bi , j 2 = 1 and σ co, j = σ co for all j . To test what happens more generally, we conducted simulations such that only half of the genes are directly differentially expressed genes (DDEGs), while the other half are non-DDEGs. To do so, we assign σ co , j 2 = 0 . 8 and σ bi , j 2 = 0 . 2 for 1 ≤ j ≤ 10, and σ co , j 2 = 0 and σ bi , j 2 = 1 for 11 ≤ j ≤ 20. The results for R = 3 are pictured in Fig 7 . We see that with one-shot sampling the edges coming out of the DDEGs are reconstructed with lower FDR and FNR compared to those coming out of non-DDEGs. However, under one-shot sampling, even the edges from the non-DDEGs in Fig 7 are recovered with much lower FDR and FNR, as compared to one-shot sampling in Fig 6 with γ = 0 and R = 3 (both FDR and FNR are around 0.95). The results indicate that the performance of BSLR under one-shot sampling benefits from diversity in conditions even when not all genes are directly differentially expressed.

We summarize the simulations performed in Table 1 . Note the last row is a summary of Table 2 in the Discussion section.

OS and MS stand for the losses of one-shot sampling and multi-shot sampling. RG stands for random guessing. * indicates mathematically proved results.

The errors are estimated using Hoeffding’s inequality over 1000 simulations with significance level 0.05.

Information limitation for reconstruction under one shot sampling without condition effect

In the previous section it is shown that both GLRT and BSLR are close to random guessing under one-shot sampling when σ co, j = 0 for all j . This leads us to the following question: is the network reconstruction with no condition effect ( σ co, j = 0 for all j ) information theoretically possible? In this section we examine this question under general estimator-independent settings. Note in this case the trajectories of all individuals are independent given A regardless of ( c k ) k ∈[ K ] .

As we have seen in Proposition 1 part 2, when Σ co = 0, the distribution of the observed data Y is invariant under adjacency matrix A or − A , implying any estimator will have a sign error probability no better than random guessing for the average or worst case over A and − A . Here, instead of sign error probability, we consider the estimation for A itself.

The extreme case with infinite number of samples available for network reconstruction is considered to give a lower bound on the accuracy for the finite data case. Note that with infinite number of samples a sufficient statistic for the estimation of the parameter A is the marginal distributions of X 1 ( t ); no information on the correlation of ( X 1 ( t )) t ∈[ T ] across time t can be obtained. A similar observation is made in [ 7 ] for sampling stochastic differential equations.

We first consider the transient case with X (0) = 0 as stated in the section of the model for gene expression dynamics. With infinite data the covariance matrix Σ t ≜ Cov ( X ( t ) ) = ∑ τ = 1 t ( A * ) t - τ A t - τ can be recovered for t ∈ [ T ]. Now we want to solve A from (Σ t ) t ∈[ T ] . As a special case, if A * A = ρI n (i.e., ρ −1/2 A is orthogonal) then we will have Σ t = ∑ τ = 0 t - 1 ρ τ I n . As a result, given (Σ t ) t ∈[ T ] in the above form, no more information of A can be obtained other than ρ −1/2 A being orthogonal, with n ( n - 1 ) 2 degrees of freedom remaining. In general case it is not clear if A can be recovered from (Σ t ) t ∈[ T ] .

Now consider the case where X k is in steady state; i.e., X (0) is random such that Cov( X ( t )) is invariant with t . With infinite amount of data we can get the covariance matrix Σ, which satisfies Σ = A *Σ A + I . Since covariance matrices are symmetric, we have n ( n + 1 ) 2 equations for n 2 variables in A . Thus A is in general not determined by the equation uniquely. In fact, note that Σ is positive definite. Then by eigendecomposition Σ = Q Λ Q *, where Q is an orthogonal matrix and Λ the diagonal matrix of the eigenvalues of Σ. Then Λ = ( Q * AQ )*Λ( Q * AQ ) + I . Let B = QAQ *. Then Λ = B *Λ B . By the Gram–Schmidt process, B can be determined with n ( n - 1 ) 2 degrees of freedom. So the network cannot be recovered from the stationary covariance matrix.

In summary, the recovery of the matrix A is generally not possible in the stationary case, and also not possible in the transient case at least when A is orthogonal. To reconstruct A , further constraints (like sparsity) may be required.

One-shot sampling in the literature

This section reviews the sampling procedures reported in several papers measuring gene expression levels in biological organisms with samples collected at different times to form time series data. In all cases, the sampling is one-shot, in the sense that a single plant or cell is only sampled at one time.

Microorganisms

In the transcriptional network inference challenge from DREAM5 [ 8 ], three compendia of biological data sets were provided based on microorganisms ( E. coli , S. aureus , and S. cerevisiae ), and some of the data corresponded to different sampling times in a time series. Being based on microorganisms, the expression level measurements involved multiple individuals per sample, a form of one-shot sampling.

In [ 9 ], the plants are exposed to nitrate, which serves as a synchronizing event, and samples are taken from three to twenty minutes after the synchronizing event. The statement “… each replicate is independent of all microarrays preceding and following in time” suggests the experiments are based on one-shot sampling. In contrast, the state-space model with correlation between transcription factors in an earlier time and the regulated genes in a later time fits multi-shot sampling. [ 10 ] studied the gene expression difference between leaves at different developmental stages in rice. The 12th, 11th and 10th leaf blades were collected every 3 days for 15 days starting from the day of the emergence of the 12th leaves. While a single plant could provide multiple samples, namely three different leaves at a given sampling time, no plant was sampled at two different times. Thus, from the standpoint of producing time series data, the sampling in this paper was one-shot sampling. [ 11 ] devised the phenol-sodium dodecyl sulfate (SDS) method for isolating total RNA from Arabidopsis . It reports the relative level of mRNA of several genes for five time points ranging up to six hours after exposure to a synchronizing event, namely being sprayed by a hormone trans -zeatin. The samples were taken from the leaves of plants. It is not clear from the paper whether the samples were collected from different leaves of the same plant, or from leaves of different plants.

[ 12 ] likely used one-shot sampling for their −24, 60, 120, 168 hour time series data from mouse skeletal muscle C2C12 cells without specifying whether the samples are all taken from different individuals. [ 13 ] produced time series data by extracting cells from a human, seeding the cells on plates, and producing samples in triplicate, at a series of six times, for each of five conditions. Multiple cells are used for each sample with different sets of cells being used for different samples, so this is an example of one-shot sampling of in vitro experiment in the sense that each plate of cells is one individual. The use of (five) multiple conditions can serve as a surrogate for a single individual set of cells to gain the effect of multi-shot sampling. Similarly, the data sets produced by [ 14 ] involving the plating of HeLa S3 cells can be classified as one-shot samples because different samples are made from different sets of individual cells. Interestingly, the samples are prepared under one set of conditions, so the use of different conditions is not adopted as a surrogate for multi-shot sampling. However, a particular line of cells was selected (HeLa S3) for which cells can be highly synchronized. Also, the paper does not attempt to determine causal interactions.

The three in silico benchmark suites described in the GeneNetWeaver paper on performance profiling of network inference methods [ 1 ] consisted of steady state, and therefore one-shot, samples from dynamical models. However, the GeneNetWeaver software can be used to generate multi-shot time series data, and some of that was included in the network inference challenges, DREAM3, DREAM4, and DREAM5 [ 1 , 8 ].

On biological replicates

In many biological experiments, independent biological replicates are used to reduce the variation in the measurements and to consequently increase the power of the statistical tests. It turns out that both how to use biological replicates, and the power of biological replicates, depend on whether the sampling is one-shot or multi-shot. To focus on this issue we first summarize how replicates have traditionally been used for the more common problem of gene differential expression analysis, before turning to the use of replicates for recovery of gene regulatory networks.

The following summarizes the use of replicates for gene differential expression analysis. A recent survey [ 15 ] suggests a minimum of three replicates for RNA-seq experiments whenever sample availability allows. Briggs et al. [ 16 ] studies the effect of biological replication together with dye switching in microarray experiments and recommends biological replication when precision in the measurements is desired. Liu et al. [ 17 ] studies the tradeoff between biological replication and sequencing depth under a sequencing budget limit in RNA-seq differential expression (DE) analysis. It proposes a metric for cost effectiveness that suggests a sequencing depth of 10 million reads per library of human breast cells and 2–6 biological replicates for optimal RNA-seq DE design. Schurch et al. [ 18 ] studies the number of necessary biological replicates in RNA-seq differential expression experiments on S. cerevisiae quantitatively with various statistical tools and concludes with the usage of a minimum of six biological replicates.

The choice of replication strategy depends on how the statistical algorithm uses the replicate data. In many differential analysis software packages replicates are treated as independent samples with identical experimental conditions. For example, in edgeR [ 19 ] and sleuth [ 20 ] the logarithm of the abundance of gene i in sample m is assumed to be x m * β i , where x m is the column vector of design characteristics with respect to p variates for sample m and β i the column vector of the associated effects of the p variates to gene i . Replicate samples can then be used to expand the design matrix x with identical columns. Note that, as a result, replicates are not necessary for edgeR and sleuth because samples with different design characteristics can all contribute to the estimation of β . It is then not clear whether it is better to have more replicates under the same condition, or to have more conditions, for a fixed total number of samples.

For regulatory network reconstruction there is even less consensus on how replicates should be used. One straightforward way is to reduce the replicates into a single set of data by averaging either directly or after a random resampling of the original replicated data. In this case the mean of the replicates are used as a better estimate of the population than each single replicate, while higher moments of the empirical distribution of the replicates are practically ignored. Another way adopted in [ 9 ] is to account for all four potential transitions between two replicates in two adjacent sampling times in their machine learning algorithm due to the one-shot nature of the replicates. In the next section, we illustrate why replicates should be used differently for one-shot and multi-shot sampling, in the context of recovering a circadian clock network model.

A case study on Arabidopsis circadian clock network

As we have discussed earlier, the current expression datasets are prominently one-shot, making a direct comparison between one-shot and multi-shot sampling in real biological data difficult. The lack of a well-accepted ground truth of the gene regulatory network also makes performance evaluation hard, if not impossible. To test the applicability of the sampling models on real biological data, we generate expression data from a most-accepted Arabidopsis circadian clock model using stochastic differential equation (SDE) model similar to GeneNetWeaver with condition-dependent Brownian motions, and evaluate the performance of BSLR.

To extend the sampling models in this paper to the more biologically plausible SDE models, we model the individual and condition-dependent variations by independent and coupled Brownian motions. Following the Arabidopsis clock network in [ 21 ], we let genes 1, 2, 3, and 4 be LHY , TOC1 , X and Y, and construct the following group of SDEs (the dark condition in [ 21 ] is assumed here).

Here x i , t k , y i , t k , and z i , t k denote the mRNA abundance, and cytoplasmic and nuclear protein concentrations of gene i at time t in plant k . The B terms are independent standard Brownian motions (Wiener processes). Note the linear diffusion terms attenuate the Brownian motions as the processes get close to 0 and consequently keep the processes nonnegative. Compared to the GLM analyzed in this paper, this SDE model captures the nonlinearity of the regulatory interactions, the continuous-time nature of the dynamics, and the detailed diffusion from mRNA to cytoplasmic and nuclear protein. So the SDE model is much more complicated and considered a basic version of the state-of-the-art circadian clock model (see [ 22 , 23 ]). Nevertheless, the SDE model shares a property with the GLM that allows the gene regulatory interactions to be summarized by a single signed directed graph: the effect of increasing the mRNA abundance of one gene on that of another has a constant sign regardless of the mRNA abundances and the protein concentrations of any genes. For example, the drift of x 1 (mRNA of LHY ) would have a tendency of increasing with an increased x 3 (mRNA of X) through the equations for y 3 (cytoplasmic protein of X) and z 3 (nuclear protein of X) regardless of the specific values of all the x ’s, y ’s and z ’s. Fig 8 shows the signed directed graph for the SDE model of Locke network. Now we have a ground-truth network based on the SDE model.

We choose the parameters in the drift coefficients (the terms in front of the d t ’s) accordingly to the supplementary material of [ 21 ], where the authors optimized the parameters to fit experimental results. We sample the SDE at times 0, 2, 4, 6, 8, and 10 for a single condition ( C = 1) with three replicates ( R = 3), σ co = 0.3 and σ bi = 0.4, and obtain the performance of BSLR in Table 2 . The estimates and the errors are based on 1000 simulations for each sampling method. Here the significance level of the Granger causality test in BSLR is set to 0.5. Note a random guess would have FDR = 0.75 and FNR + 1 2 FPR = 1 . We see from Table 2 that BSLR without replicate averaging on one-shot sampling data is no different from random guessing, while BSLR with replicate averaging doing slightly better in FDR and FNR. This is because replicate averaging of one-shot data practically increases the condition effect by reducing the biological variation, and thus gets a better temporal correlation between one-shot samples of adjacent times. BSLR performs better on multi-shot data compared to one-shot data because the biological variations of the previous times also contribute to the temporal correlation. In particular, BSLR without replicate averaging on the multi-shot data has the best performance because it allows tracking the individual replicates rather than merely tracking their averages. Although the performance numbers appear far from ideal, this demonstrates remarkable improvement from BSLR with replicate averaging on one-shot data to BSLR without replicate averaging on multi-shot data, especially considering the highly nonlinear SDE model, the unobserved protein concentrations levels, the very limited number of 18 samples (3 replicates with 6 times) and the fact that BSLR does not use any knowledge of the (around 60) parameters or the form of the equations, highlighting the difference in the statistical power of one-shot and multi-shot data and its implication in downstream statistical analysis decisions (replicate averaging vs. no replicate averaging).

In summary, we demonstrated a setting of the biologically plausible Arabidopsis circadian clock network with a single condition, where the BSLR performs similarly to a random guessing algorithm under one-shot sampling, and performs significantly better under multi-shot sampling. We also show that whether replicate averaging should be done or not varies with the sampling method.

Conclusions

One-shot sampling can miss a lot of potentially useful correlation information. Often gene expression data collected from plants is prepared under one-shot sampling. One factor that can partially mitigate the shortcomings of one-shot sampling is to prepare samples under a variety of conditions or perturbations. One-shot samples grown under the same condition can then be thought of as a surrogate for the multi-shot samples of an individual plant.

To clarify issues and take a step towards quantifying them, we proposed a gene expression dynamic model for gene regulatory network reconstruction that explicitly captures the condition variation effect. We show analytically and numerically the performance of two algorithms for single-gene and multi-gene settings. We also demonstrate the difficulty of network reconstruction without condition variation effect.

There is little agreement across the biology literature about how to model the impact of condition on the gene regulatory network. In some cases, it is not even clear that we are observing the same network structure as conditions vary. Nevertheless, our results suggest that the preparation of samples under different conditions can partially compensate for the shortcomings of one-shot sampling.

Supporting information

S1 appendix.

The parameters A , γ , σ , and σ te are assumed unknown and jointly estimated in GLRT.

S2 Appendix

The random network prior distribution used to generate the multi-gene network.

Funding Statement

This work was supported by the Plant Genome Research Program from the National Science Foundation (NSF-IOS-PGRP-1823145) to B.H. and Y.H.

Data Availability

• PLoS One. 2019; 14(10): e0224577.

Decision Letter 0

21 Aug 2019

PONE-D-19-16969

Dear Dr. Kang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

In the reviewer comments below, I would emphasize the exploration of actual biological data and the comparison of performance to one or two other methods.

We would appreciate receiving your revised manuscript by Oct 05 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Steven M. Abel, Ph.D.

Academic Editor

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This papers explores the effect of condition variance and biological variance under one-shot and multi-shot sampling. It provides a general model for network representation when these parameters are known. Further, the authors describe the relationship between the sampling regimes and their representation in the model described. Finally, the authors simulation from this model and describe the performance of two-estimators under different configurations.

Things I would like to see:

- A table describing the simulations performed and summarizing the behavior

- A description of how the adjacency matrix A was chosen from simulations

- An application on "real" data

- A comparison of performance to other methods on some of the DREAM data

The section "On biological replicates" is a bit odd and seems tacked on. Particularly, the discussion of differential expression tools seems odd. The goal in those tools is not to infer gene regulation, but to infer whether the expression of some set of genes is changed given some experimental perturbation. Not sure what you mean here (line 551):

> It is then not clear whether replicates bring more benefit than the sheer additional amount of data compared to samples under different conditions.

6. PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at gro.solp@serugif . Please note that Supporting Information files do not need this step.

Author response to Decision Letter 0

See the response letter.

Submitted filename: response.pdf

Decision Letter 1

17 Oct 2019

PONE-D-19-16969R1

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/ , click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at gro.solp@gnillibrohtua .

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact gro.solp@sserpeno .

With kind regards,

Acceptance letter

22 Oct 2019

Dear Dr. Kang:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact gro.solp@sserpeno .

For any other questions or concerns, please email gro.solp@enosolp .

Thank you for submitting your work to PLOS ONE.

PLOS ONE Editorial Office Staff

on behalf of

Dr. Steven M. Abel

• Experimental Research Designs: Types, Examples & Methods

Experimental research is the most familiar type of research design for individuals in the physical sciences and a host of other fields. This is mainly because experimental research is a classical scientific experiment, similar to those performed in high school science classes.

Imagine taking 2 samples of the same plant and exposing one of them to sunlight, while the other is kept away from sunlight. Let the plant exposed to sunlight be called sample A, while the latter is called sample B.

If after the duration of the research, we find out that sample A grows and sample B dies, even though they are both regularly wetted and given the same treatment. Therefore, we can conclude that sunlight will aid growth in all similar plants.

What is Experimental Research?

Experimental research is a scientific approach to research, where one or more independent variables are manipulated and applied to one or more dependent variables to measure their effect on the latter. The effect of the independent variables on the dependent variables is usually observed and recorded over some time, to aid researchers in drawing a reasonable conclusion regarding the relationship between these 2 variable types.

The experimental research method is widely used in physical and social sciences, psychology, and education. It is based on the comparison between two or more groups with a straightforward logic, which may, however, be difficult to execute.

Mostly related to a laboratory test procedure, experimental research designs involve collecting quantitative data and performing statistical analysis on them during research. Therefore, making it an example of quantitative research method .

What are The Types of Experimental Research Design?

The types of experimental research design are determined by the way the researcher assigns subjects to different conditions and groups. They are of 3 types, namely; pre-experimental, quasi-experimental, and true experimental research.

Pre-experimental Research Design

In pre-experimental research design, either a group or various dependent groups are observed for the effect of the application of an independent variable which is presumed to cause change. It is the simplest form of experimental research design and is treated with no control group.

Although very practical, experimental research is lacking in several areas of the true-experimental criteria. The pre-experimental research design is further divided into three types

• One-shot Case Study Research Design

In this type of experimental study, only one dependent group or variable is considered. The study is carried out after some treatment which was presumed to cause change, making it a posttest study.

• One-group Pretest-posttest Research Design: 

This research design combines both posttest and pretest study by carrying out a test on a single group before the treatment is administered and after the treatment is administered. With the former being administered at the beginning of treatment and later at the end.

• Static-group Comparison: 

In a static-group comparison study, 2 or more groups are placed under observation, where only one of the groups is subjected to some treatment while the other groups are held static. All the groups are post-tested, and the observed differences between the groups are assumed to be a result of the treatment.

Quasi-experimental Research Design

  The word “quasi” means partial, half, or pseudo. Therefore, the quasi-experimental research bearing a resemblance to the true experimental research, but not the same.  In quasi-experiments, the participants are not randomly assigned, and as such, they are used in settings where randomization is difficult or impossible.

 This is very common in educational research, where administrators are unwilling to allow the random selection of students for experimental samples.

Some examples of quasi-experimental research design include; the time series, no equivalent control group design, and the counterbalanced design.

True Experimental Research Design

The true experimental research design relies on statistical analysis to approve or disprove a hypothesis. It is the most accurate type of experimental design and may be carried out with or without a pretest on at least 2 randomly assigned dependent subjects.

The true experimental research design must contain a control group, a variable that can be manipulated by the researcher, and the distribution must be random. The classification of true experimental design include:

• The posttest-only Control Group Design: In this design, subjects are randomly selected and assigned to the 2 groups (control and experimental), and only the experimental group is treated. After close observation, both groups are post-tested, and a conclusion is drawn from the difference between these groups.
• The pretest-posttest Control Group Design: For this control group design, subjects are randomly assigned to the 2 groups, both are presented, but only the experimental group is treated. After close observation, both groups are post-tested to measure the degree of change in each group.
• Solomon four-group Design: This is the combination of the pretest-only and the pretest-posttest control groups. In this case, the randomly selected subjects are placed into 4 groups.

The first two of these groups are tested using the posttest-only method, while the other two are tested using the pretest-posttest method.

Examples of Experimental Research

Experimental research examples are different, depending on the type of experimental research design that is being considered. The most basic example of experimental research is laboratory experiments, which may differ in nature depending on the subject of research.

Administering Exams After The End of Semester

During the semester, students in a class are lectured on particular courses and an exam is administered at the end of the semester. In this case, the students are the subjects or dependent variables while the lectures are the independent variables treated on the subjects.

Only one group of carefully selected subjects are considered in this research, making it a pre-experimental research design example. We will also notice that tests are only carried out at the end of the semester, and not at the beginning.

Further making it easy for us to conclude that it is a one-shot case study research. 

Employee Skill Evaluation

Before employing a job seeker, organizations conduct tests that are used to screen out less qualified candidates from the pool of qualified applicants. This way, organizations can determine an employee’s skill set at the point of employment.

In the course of employment, organizations also carry out employee training to improve employee productivity and generally grow the organization. Further evaluation is carried out at the end of each training to test the impact of the training on employee skills, and test for improvement.

Here, the subject is the employee, while the treatment is the training conducted. This is a pretest-posttest control group experimental research example.

Evaluation of Teaching Method

Let us consider an academic institution that wants to evaluate the teaching method of 2 teachers to determine which is best. Imagine a case whereby the students assigned to each teacher is carefully selected probably due to personal request by parents or due to stubbornness and smartness.

This is a no equivalent group design example because the samples are not equal. By evaluating the effectiveness of each teacher’s teaching method this way, we may conclude after a post-test has been carried out.

However, this may be influenced by factors like the natural sweetness of a student. For example, a very smart student will grab more easily than his or her peers irrespective of the method of teaching.

What are the Characteristics of Experimental Research?  

Experimental research contains dependent, independent and extraneous variables. The dependent variables are the variables being treated or manipulated and are sometimes called the subject of the research.

The independent variables are the experimental treatment being exerted on the dependent variables. Extraneous variables, on the other hand, are other factors affecting the experiment that may also contribute to the change.

The setting is where the experiment is carried out. Many experiments are carried out in the laboratory, where control can be exerted on the extraneous variables, thereby eliminating them. 

Other experiments are carried out in a less controllable setting. The choice of setting used in research depends on the nature of the experiment being carried out.

• Multivariable

Experimental research may include multiple independent variables, e.g. time, skills, test scores, etc.

Why Use Experimental Research Design?  

Experimental research design can be majorly used in physical sciences, social sciences, education, and psychology. It is used to make predictions and draw conclusions on a subject matter. 

Some uses of experimental research design are highlighted below.

• Medicine: Experimental research is used to provide the proper treatment for diseases. In most cases, rather than directly using patients as the research subject, researchers take a sample of the bacteria from the patient’s body and are treated with the developed antibacterial

The changes observed during this period are recorded and evaluated to determine its effectiveness. This process can be carried out using different experimental research methods.

• Education: Asides from science subjects like Chemistry and Physics which involves teaching students how to perform experimental research, it can also be used in improving the standard of an academic institution. This includes testing students’ knowledge on different topics, coming up with better teaching methods, and the implementation of other programs that will aid student learning.
• Human Behavior: Social scientists are the ones who mostly use experimental research to test human behaviour. For example, consider 2 people randomly chosen to be the subject of the social interaction research where one person is placed in a room without human interaction for 1 year.

The other person is placed in a room with a few other people, enjoying human interaction. There will be a difference in their behaviour at the end of the experiment.

• UI/UX: During the product development phase, one of the major aims of the product team is to create a great user experience with the product. Therefore, before launching the final product design, potential are brought in to interact with the product.

For example, when finding it difficult to choose how to position a button or feature on the app interface, a random sample of product testers are allowed to test the 2 samples and how the button positioning influences the user interaction is recorded.

What are the Disadvantages of Experimental Research?  

• It is highly prone to human error due to its dependency on variable control which may not be properly implemented. These errors could eliminate the validity of the experiment and the research being conducted.
• Exerting control of extraneous variables may create unrealistic situations. Eliminating real-life variables will result in inaccurate conclusions. This may also result in researchers controlling the variables to suit his or her personal preferences.
• It is a time-consuming process. So much time is spent on testing dependent variables and waiting for the effect of the manipulation of dependent variables to manifest.
• It is expensive. 
• It is very risky and may have ethical complications that cannot be ignored. This is common in medical research, where failed trials may lead to a patient’s death or a deteriorating health condition.
• Experimental research results are not descriptive.
• Response bias can also be supplied by the subject of the conversation.
• Human responses in experimental research can be difficult to measure. 

What are the Data Collection Methods in Experimental Research?  

Data collection methods in experimental research are the different ways in which data can be collected for experimental research. They are used in different cases, depending on the type of research being carried out.

1. Observational Study

This type of study is carried out over a long period. It measures and observes the variables of interest without changing existing conditions.

When researching the effect of social interaction on human behavior, the subjects who are placed in 2 different environments are observed throughout the research. No matter the kind of absurd behavior that is exhibited by the subject during this period, its condition will not be changed.

This may be a very risky thing to do in medical cases because it may lead to death or worse medical conditions.

2. Simulations

This procedure uses mathematical, physical, or computer models to replicate a real-life process or situation. It is frequently used when the actual situation is too expensive, dangerous, or impractical to replicate in real life.

This method is commonly used in engineering and operational research for learning purposes and sometimes as a tool to estimate possible outcomes of real research. Some common situation software are Simulink, MATLAB, and Simul8.

Not all kinds of experimental research can be carried out using simulation as a data collection tool . It is very impractical for a lot of laboratory-based research that involves chemical processes.

A survey is a tool used to gather relevant data about the characteristics of a population and is one of the most common data collection tools. A survey consists of a group of questions prepared by the researcher, to be answered by the research subject.

Surveys can be shared with the respondents both physically and electronically. When collecting data through surveys, the kind of data collected depends on the respondent, and researchers have limited control over it.

Formplus is the best tool for collecting experimental data using survey s. It has relevant features that will aid the data collection process and can also be used in other aspects of experimental research.

Differences between Experimental and Non-Experimental Research 

1. In experimental research, the researcher can control and manipulate the environment of the research, including the predictor variable which can be changed. On the other hand, non-experimental research cannot be controlled or manipulated by the researcher at will.

This is because it takes place in a real-life setting, where extraneous variables cannot be eliminated. Therefore, it is more difficult to conclude non-experimental studies, even though they are much more flexible and allow for a greater range of study fields.

2. The relationship between cause and effect cannot be established in non-experimental research, while it can be established in experimental research. This may be because many extraneous variables also influence the changes in the research subject, making it difficult to point at a particular variable as the cause of a particular change

3. Independent variables are not introduced, withdrawn, or manipulated in non-experimental designs, but the same may not be said about experimental research.

Conclusion  

Experimental research designs are often considered to be the standard in research designs. This is partly due to the common misconception that research is equivalent to scientific experiments—a component of experimental research design.

In this research design, one or more subjects or dependent variables are randomly assigned to different treatments (i.e. independent variables manipulated by the researcher) and the results are observed to conclude. One of the uniqueness of experimental research is in its ability to control the effect of extraneous variables.

Experimental research is suitable for research whose goal is to examine cause-effect relationships, e.g. explanatory research. It can be conducted in the laboratory or field settings, depending on the aim of the research that is being carried out. 

Connect to Formplus, Get Started Now - It's Free!

• examples of experimental research
• experimental research methods
• types of experimental research
• busayo.longe

You may also like:

What is Experimenter Bias? Definition, Types & Mitigation

In this article, we will look into the concept of experimental bias and how it can be identified in your research

Simpson’s Paradox & How to Avoid it in Experimental Research

In this article, we are going to look at Simpson’s Paradox from its historical point and later, we’ll consider its effect in...

Experimental Vs Non-Experimental Research: 15 Key Differences

Differences between experimental and non experimental research on definitions, types, examples, data collection tools, uses, advantages etc.

Response vs Explanatory Variables: Definition & Examples

In this article, we’ll be comparing the two types of variables, what they both mean and see some of their real-life applications in research

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Time series experimental design under one-shot sampling: The importance of condition diversity

Roles Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Coordinated Science Laboratory and Department of Electrical and Computer Engineering, University of Illinois at Urbana–Champaign, Urbana, Illinois, United States of America

Roles Writing – review & editing

Affiliation Department of Biology, California State University, Northridge, Northridge, California, United States of America

• Xiaohan Kang, 
• Bruce Hajek, 
• Faqiang Wu, 
• Yoshie Hanzawa

• Published: October 31, 2019
• https://doi.org/10.1371/journal.pone.0224577
• See the preprint
• Peer Review
• Reader Comments

Many biological data sets are prepared using one-shot sampling, in which each individual organism is sampled at most once. Time series therefore do not follow trajectories of individuals over time. However, samples collected at different times from individuals grown under the same conditions share the same perturbations of the biological processes, and hence behave as surrogates for multiple samples from a single individual at different times. This implies the importance of growing individuals under multiple conditions if one-shot sampling is used. This paper models the condition effect explicitly by using condition-dependent nominal mRNA production amounts for each gene, it quantifies the performance of network structure estimators both analytically and numerically, and it illustrates the difficulty in network reconstruction under one-shot sampling when the condition effect is absent. A case study of an Arabidopsis circadian clock network model is also included.

Citation: Kang X, Hajek B, Wu F, Hanzawa Y (2019) Time series experimental design under one-shot sampling: The importance of condition diversity. PLoS ONE 14(10): e0224577. https://doi.org/10.1371/journal.pone.0224577

Editor: Steven M. Abel, University of Tennessee, UNITED STATES

Received: June 14, 2019; Accepted: October 16, 2019; Published: October 31, 2019

Copyright: © 2019 Kang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The computer simulation code is available at https://github.com/Veggente/one-shot-sampling .

Funding: This work was supported by the Plant Genome Research Program from the National Science Foundation (NSF-IOS-PGRP-1823145) to B.H. and Y.H.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Time series data is important for studying biological processes in organisms because of the dynamic nature of the biological systems. Ideally it is desirable to use time series with multi-shot sampling , where each individual (such as a plant, animal, or microorganism) is sampled multiple times to produce the trajectory of the biological process, as in Fig 1 . Then the natural biological variations in different individuals can be leveraged for statistical inference, and thus inference can be made even if the samples are prepared under the same experimental condition.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

Each plant is observed four times.

https://doi.org/10.1371/journal.pone.0224577.g001

However, in many experiments multi-shot sampling is not possible. Due to stress response of the organisms and/or the large amount of cell tissue required for accurate measurements, the dynamics of the relevant biological process in an individual of the organism cannot be observed at multiple times without interference. For example, in an RNA-seq experiment an individual plant is often only sampled once in its entire life, leaving the dynamics unobserved at other times. See the Discussion section for a review of literature on this subject. We call the resulting time series data, as illustrated in Fig 2 , a time series with one-shot sampling . Because the time series with one-shot sampling do not follow the trajectories of the same individuals, they do not capture all the correlations in the biological processes. For example, the trajectory of observations on plants 1–2–3–4 and that on 1–6–7–4 in Fig 2 are statistically identical. The resulting partial observation renders some common models for the biological system dynamics inaccurate or even irrelevant.

Each plant is observed once.

https://doi.org/10.1371/journal.pone.0224577.g002

To address this problem, instead of getting multi-shot time series of single individuals, one can grow multiple individuals under each condition with a variety of conditions, and get one-shot time series of the single conditions. The one-shot samples from the same condition then become a surrogate for multi-shot samples for a single individual, as illustrated in Fig 3 . In essence, if we view the preparation condition of each sample as being random, then there should be a positive correlation among samples grown under the same condition. We call this correlation the condition variation effect . It is similar to the effect of biological variation of a single individual sampled at different times, if such sampling were possible.

https://doi.org/10.1371/journal.pone.0224577.g003

For each condition, the one-shot samples at different times are also complemented by biological replicates , which are samples from independent individuals taken at the same time used to reduce technical and/or biological variations. See the Discussion section for a review on how replicates are used for biological inference. With a budget over the number of samples, a balance must be kept between the number of conditions, the number of sampling times and the number of replicates.

To illustrate and quantify the effect of one-shot sampling in biological inference, we introduce a simple dynamic gene expression model with a condition variation effect. We consider a hypothesis testing setting and model the dynamics of the gene expression levels at different sampling times by a dynamic Bayesian network (DBN), where the randomness comes from nominal (or basal) biological and condition variations for each gene. The nominal condition-dependent variation of gene j is the same for all plants in that condition and the remaining variation is biological and is independent across the individuals in the condition. In contrast to GeneNetWeaver [ 1 ], where the effect of a condition is modeled by a random perturbation to the network coefficients, in our model the condition effect is characterized by correlation in the nominal variation terms of the dynamics. Note in both models samples from different individuals under the same condition are statistically independent given the randomness associated with the condition.

The contributions of this paper are threefold.

• A composite hypothesis testing problem on the gene regulatory network is formulated and a gene expression dynamic model that explicitly captures the per-gene condition effect and the gene regulatory interactions is proposed.
• The performance of gene regulatory network structure estimators is analyzed for both multi-shot and one-shot sampling, with focus on two algorithms. Furthermore, single-gene and multi-gene simulation results indicate that multiple-condition experiments can somewhat mitigate the shortcomings of one-shot sampling.
• The difficulty of network reconstruction under one-shot sampling with no condition effect is illustrated. This difficulty connects network analysis and differential expression analysis, two common tasks in large-scale genomics studies, in the sense that the part of network involving non-differentially expressed genes may be harder to reconstruct.

The simulation code for generating the figures is available at [ 2 ].

Materials and methods

Stochastic model of time-series samples.

This section formulates the hypothesis testing problem of learning the structure of the gene regulatory network (GRN) from gene expression data with one-shot or multi-shot sampling. The GRN is characterized by an unknown adjacency matrix. Given the GRN, a dynamic Bayesian network model is used for the gene expression evolution with time. Two parameters σ co, j and σ bi, j are used for each gene j , with the former explicitly capturing the condition variation effect and the latter capturing the biological variation level.

Model for gene regulatory network topology.

Model for gene expression dynamics.

This section models the gene expression dynamics of individuals by a dynamic Bayesian networks with parameters σ co, j and σ bi, j as the condition variation level and biological variation level for gene j .

Model for sampling method.

In this section two sampling methods are described: one-shot sampling and multi-shot sampling. For simplicity, throughout this paper we consider a full factorial design with CRT samples obtained under C conditions, R replicates and T sampling times, denoted by Y = ( Y c , r , t ) ( c , r , t )∈[ C ]×[ R ]×[ T ] . In other words, instead of X we observe Y , a noisy and possibly partial observation of X . Here the triple index for each sample indicates the condition, replicate, and time. As we will see in the Discussion at the end of this section, for either sampling method, the biological variation level σ bi, j can be reduced by combining multiple individuals to form a single sample.

Multi-shot sampling.

One-shot sampling.

Discussion on sources of variance.

• If the samples of the individuals under many different conditions are averaged and treated as a single sample, then effectively σ co, j , σ bi, j and σ te, j are reduced.

• If material from multiple individuals grown under the same condition is combined into a composite sample before measuring, then effectively σ bi, j is reduced while σ co, j and σ te, j remain unchanged. Note for microorganisms a sample may consist of millions of individuals and the biological variation is practically eliminated ( σ bi, j ≈ 0).
• If the samples from same individuals (technical replicates) are averaged and treated as a single sample, then effectively σ te, j is reduced while σ co, j and σ bi, j remain unchanged.

Note this sampling model with hierarchical driving and observational noises can also be used for single-cell RNA sequencing (scRNAseq) in addition to bulk RNA sequencing and microarray experiments. For scRNAseq, σ co, j can model the tissue-dependent variation (the global effect) and σ bi, j the per-cell variation (the local effect).

Performance evaluation of network structure estimators

This section studies the performance of network structure estimators with multi-shot and one-shot sampling data. First, general properties of the two sampling methods are obtained. Then two algorithms, the generalized likelihood-ratio test (GLRT) and the basic sparse linear regression (BSLR), are studied. The former is a standard decision rule for composite hypothesis testing problems and is shown to have some properties but is computationally infeasible for even a small number of genes. The latter is an algorithm based on linear regression, and is feasible for a moderate number of genes. Finally simulation results for a single-gene network with GLRT and for a multi-gene network with BSLR are shown.

General properties.

By ( 3 ), ( 4 ) and ( 5 ), the samples Y are jointly Gaussian with zero mean. The covariance of the random tensor Y is derived under the two sampling methods in the following.

• If Σ bi = 0 and C , R and T are fixed, the joint distribution of the data is the same for both types of sampling. So the performance of the estimator would be the same for multi-shot and one-shot sampling.
• If Σ bi = 0 and Σ te = 0 (no observation noise) and C , T are fixed, the joint distribution of the data is the same for both types of sampling (as noted in item 1) and any replicates beyond the first are identical to the first. So the performance of the estimator can be obtained even if all replicates beyond the first are discarded.
• Under multi-shot sampling, when C , R , T are fixed with R = 1, the joint distribution of the data depends on Σ co and Σ bi only through their sum. So the performance of the estimator would be the same for all Σ co and Σ bi such that Σ co + Σ bi is the same.

Network reconstruction algorithms.

In this section we introduce GLRT and BSLR. GLRT is a standard choice in composite hypothesis testing setting. We observe some properties for GLRT under one-shot and multi-shot sampling. However, GLRT involves optimizing the likelihood over the entire parameter space, which grows exponentially with the square of the number of genes. Hence GLRT is hard to compute for multiple-gene network reconstruction. In contrast, BSLR is an intuitive algorithm based on linear regression, and will be shown in simulations to perform reasonably well for multi-gene scenarios.

Proposition 1 GLRT ( with the knowledge of Σ co , Σ bi and Σ te ) has the following properties .

• Under one-shot sampling and Σ co = 0, the log likelihood of the data as a function of A ( i . e . the log likelihood function ) is invariant with respect to replacing A by − A . So , for the single-gene n = 1 case , the log likelihood function is an even function of A , and thus the GLRT will do no better than random guessing .

For 2 it suffices to notice in ( 6 ) the covariance is invariant with respect to changing A to − A . A proof of 1 is given below.

Proof of 1) We first prove it for the case of a single gene with constant T and a constant number of individuals, CR . To do that we need to look at the likelihood function closely.

In BSLR, replicates are averaged and the average gene expression levels at different times under different conditions are fitted in a linear regression model with best-subset sparse model selection, followed by a Granger causality test to eliminate the false discoveries. BSLR is similar to other two-stage linear regression–based network reconstruction algorithms, notably oCSE [ 4 ] and CaSPIAN [ 5 ]. Both oCSE and CaSPIAN use greedy algorithms in the first build-up stage, making them more suitable for large-scale problems. In contrast, BSLR uses best subset selection, which is conceptually simpler but computationally expensive for large n . For the tear-down stage both BSLR and CaSPIAN use the Granger causality test, while oCSE uses a permutation test.

Build-up stage.

A naive algorithm to solve the above optimization has a computational complexity of O ( n k +1 ) for fixed k as n → ∞. Faster near-optimal alternatives exist [ 6 ].

Tear-down stage.

Simulations on single-gene network reconstruction.

https://doi.org/10.1371/journal.pone.0224577.g004

The numerical simulations reflect the following observations implied by the analytical model.

• Under one-shot sampling, when γ = 0, the GLRT is equivalent to random guessing.
• The GLRT performs the same under one-shot and multi-shot sampling when γ = 1.
• Under no observation noise, the performance for multi-shot sampling is the same for all γ < 1.

Some empirical observations are in order.

• Multi-shot sampling outperforms one-shot sampling (unless γ = 1, where they have the same error probability).
• For one-shot sampling, the performance improves as γ increases. Regarding the number of replicates R per condition, if γ = 0.2 (small condition effect), a medium number of replicates (2 to 5) outperforms the single replicate strategy. For larger γ , one replicate per condition is the best.
• For multi-shot sampling, performance worsens as γ increases. One replicate per condition ( R = 1) is best.
• Comparing Fig 4a–4d vs. Fig 4e–4h , we observe that the performance degrades with the addition of observation noise, though for moderate noise ( σ te = 1.0) the effect of observation noise on the sign error is not large. Also, the effect of the algorithm not knowing γ is not large.

Simulations on multi-gene network reconstruction.

This subsection studies the case when multiple genes interact through the GRN. The goal is to compare one-shot vs. multi-shot sampling for BSLR under a variety of scenarios, including different homogeneous γ values, varying number of replicates, varying observation noise level, and heterogeneous γ values.

The performance evaluation for multi-gene network reconstruction is trickier than the single-gene case because of the many degrees of freedom introduced by the number of genes. First, the network adjacency matrix A is now an n -by- n matrix. While some notion of “size” of A (like the spectral radius or the matrix norm) may be important, potentially every entry of A may affect the reconstruction result. So instead of fixing a ground truth A as in Fig 4 , we fix a prior distribution of A with split Gaussian prior described in S2 Appendix (note we assume the knowledge of no autoregulation), and choose A i.i.d. from the prior distribution with d max = 3. Second, because the prior of A can be subject to sparsity constraints and thus far from a uniform distribution, multiple loss functions that are more meaningful than the ternary error rate can be considered for performance. So we consider ternary FDR, ternary FNR and ternary FPR for the multi-gene case. In the simulations we have 20 genes and d max = 3 with in-degree uniformly distributed over {0, 1, …, d max }, so the average in-degree is 1.5. The number of sampling times is T = 6 and CR = 30.

Varying γ , R and σ te .

In this set of simulations we fix the observation noise level and vary the number of replicates R and the condition correlation coefficient γ . The performance of BSLR under one-shot and multi-shot sampling is shown in Fig 5 ( σ te = 0) and Fig 6 ( σ te = 1). Note BSLR does not apply to a single condition with 30 replicates due to the constraint that the degrees of freedom C ( T − 1) − k − 2 in the second stage must be at least 1.

https://doi.org/10.1371/journal.pone.0224577.g005

https://doi.org/10.1371/journal.pone.0224577.g006

For multi-shot sampling, in the noiseless case, we see all three losses are invariant with respect to different γ for fixed R , which is consistent with property 4 in the section on general properties because BSLR is an average-based scale-invariant algorithm (note CR is a constant so for different R the performance is different due to the change in C ). In the noisy case, the FDR and FNR slightly decrease as γ increases, which is an opposite trend compared with Fig 4f and 4h .

In summary, the main conclusions from Figs 5 and 6 are the following.

• The performance of BSLR under multi-shot sampling is consistently better than that under one-shot sampling.
• The performance of BSLR under one-shot sampling varies with γ , from random guessing performance at γ = 0 to the same performance as multi-shot sampling at γ = 1.
• By comparing Figs 5 with 6 , we see the observation noise of σ te = 1 has only a small effect on the performance with the two sampling methods.

Reduced number of directly differentially expressed genes.

https://doi.org/10.1371/journal.pone.0224577.g007

We summarize the simulations performed in Table 1 . Note the last row is a summary of Table 2 in the Discussion section.

OS and MS stand for the losses of one-shot sampling and multi-shot sampling. RG stands for random guessing. * indicates mathematically proved results.

https://doi.org/10.1371/journal.pone.0224577.t001

The errors are estimated using Hoeffding’s inequality over 1000 simulations with significance level 0.05.

https://doi.org/10.1371/journal.pone.0224577.t002

Information limitation for reconstruction under one shot sampling without condition effect

In the previous section it is shown that both GLRT and BSLR are close to random guessing under one-shot sampling when σ co, j = 0 for all j . This leads us to the following question: is the network reconstruction with no condition effect ( σ co, j = 0 for all j ) information theoretically possible? In this section we examine this question under general estimator-independent settings. Note in this case the trajectories of all individuals are independent given A regardless of ( c k ) k ∈[ K ] .

As we have seen in Proposition 1 part 2, when Σ co = 0, the distribution of the observed data Y is invariant under adjacency matrix A or − A , implying any estimator will have a sign error probability no better than random guessing for the average or worst case over A and − A . Here, instead of sign error probability, we consider the estimation for A itself.

The extreme case with infinite number of samples available for network reconstruction is considered to give a lower bound on the accuracy for the finite data case. Note that with infinite number of samples a sufficient statistic for the estimation of the parameter A is the marginal distributions of X 1 ( t ); no information on the correlation of ( X 1 ( t )) t ∈[ T ] across time t can be obtained. A similar observation is made in [ 7 ] for sampling stochastic differential equations.

In summary, the recovery of the matrix A is generally not possible in the stationary case, and also not possible in the transient case at least when A is orthogonal. To reconstruct A , further constraints (like sparsity) may be required.

One-shot sampling in the literature

This section reviews the sampling procedures reported in several papers measuring gene expression levels in biological organisms with samples collected at different times to form time series data. In all cases, the sampling is one-shot, in the sense that a single plant or cell is only sampled at one time.

Microorganisms.

In the transcriptional network inference challenge from DREAM5 [ 8 ], three compendia of biological data sets were provided based on microorganisms ( E. coli , S. aureus , and S. cerevisiae ), and some of the data corresponded to different sampling times in a time series. Being based on microorganisms, the expression level measurements involved multiple individuals per sample, a form of one-shot sampling.

In [ 9 ], the plants are exposed to nitrate, which serves as a synchronizing event, and samples are taken from three to twenty minutes after the synchronizing event. The statement “… each replicate is independent of all microarrays preceding and following in time” suggests the experiments are based on one-shot sampling. In contrast, the state-space model with correlation between transcription factors in an earlier time and the regulated genes in a later time fits multi-shot sampling. [ 10 ] studied the gene expression difference between leaves at different developmental stages in rice. The 12th, 11th and 10th leaf blades were collected every 3 days for 15 days starting from the day of the emergence of the 12th leaves. While a single plant could provide multiple samples, namely three different leaves at a given sampling time, no plant was sampled at two different times. Thus, from the standpoint of producing time series data, the sampling in this paper was one-shot sampling. [ 11 ] devised the phenol-sodium dodecyl sulfate (SDS) method for isolating total RNA from Arabidopsis . It reports the relative level of mRNA of several genes for five time points ranging up to six hours after exposure to a synchronizing event, namely being sprayed by a hormone trans -zeatin. The samples were taken from the leaves of plants. It is not clear from the paper whether the samples were collected from different leaves of the same plant, or from leaves of different plants.

[ 12 ] likely used one-shot sampling for their −24, 60, 120, 168 hour time series data from mouse skeletal muscle C2C12 cells without specifying whether the samples are all taken from different individuals. [ 13 ] produced time series data by extracting cells from a human, seeding the cells on plates, and producing samples in triplicate, at a series of six times, for each of five conditions. Multiple cells are used for each sample with different sets of cells being used for different samples, so this is an example of one-shot sampling of in vitro experiment in the sense that each plate of cells is one individual. The use of (five) multiple conditions can serve as a surrogate for a single individual set of cells to gain the effect of multi-shot sampling. Similarly, the data sets produced by [ 14 ] involving the plating of HeLa S3 cells can be classified as one-shot samples because different samples are made from different sets of individual cells. Interestingly, the samples are prepared under one set of conditions, so the use of different conditions is not adopted as a surrogate for multi-shot sampling. However, a particular line of cells was selected (HeLa S3) for which cells can be highly synchronized. Also, the paper does not attempt to determine causal interactions.

The three in silico benchmark suites described in the GeneNetWeaver paper on performance profiling of network inference methods [ 1 ] consisted of steady state, and therefore one-shot, samples from dynamical models. However, the GeneNetWeaver software can be used to generate multi-shot time series data, and some of that was included in the network inference challenges, DREAM3, DREAM4, and DREAM5 [ 1 , 8 ].

On biological replicates

In many biological experiments, independent biological replicates are used to reduce the variation in the measurements and to consequently increase the power of the statistical tests. It turns out that both how to use biological replicates, and the power of biological replicates, depend on whether the sampling is one-shot or multi-shot. To focus on this issue we first summarize how replicates have traditionally been used for the more common problem of gene differential expression analysis, before turning to the use of replicates for recovery of gene regulatory networks.

The following summarizes the use of replicates for gene differential expression analysis. A recent survey [ 15 ] suggests a minimum of three replicates for RNA-seq experiments whenever sample availability allows. Briggs et al. [ 16 ] studies the effect of biological replication together with dye switching in microarray experiments and recommends biological replication when precision in the measurements is desired. Liu et al. [ 17 ] studies the tradeoff between biological replication and sequencing depth under a sequencing budget limit in RNA-seq differential expression (DE) analysis. It proposes a metric for cost effectiveness that suggests a sequencing depth of 10 million reads per library of human breast cells and 2–6 biological replicates for optimal RNA-seq DE design. Schurch et al. [ 18 ] studies the number of necessary biological replicates in RNA-seq differential expression experiments on S. cerevisiae quantitatively with various statistical tools and concludes with the usage of a minimum of six biological replicates.

For regulatory network reconstruction there is even less consensus on how replicates should be used. One straightforward way is to reduce the replicates into a single set of data by averaging either directly or after a random resampling of the original replicated data. In this case the mean of the replicates are used as a better estimate of the population than each single replicate, while higher moments of the empirical distribution of the replicates are practically ignored. Another way adopted in [ 9 ] is to account for all four potential transitions between two replicates in two adjacent sampling times in their machine learning algorithm due to the one-shot nature of the replicates. In the next section, we illustrate why replicates should be used differently for one-shot and multi-shot sampling, in the context of recovering a circadian clock network model.

A case study on Arabidopsis circadian clock network

As we have discussed earlier, the current expression datasets are prominently one-shot, making a direct comparison between one-shot and multi-shot sampling in real biological data difficult. The lack of a well-accepted ground truth of the gene regulatory network also makes performance evaluation hard, if not impossible. To test the applicability of the sampling models on real biological data, we generate expression data from a most-accepted Arabidopsis circadian clock model using stochastic differential equation (SDE) model similar to GeneNetWeaver with condition-dependent Brownian motions, and evaluate the performance of BSLR.

https://doi.org/10.1371/journal.pone.0224577.g008

In summary, we demonstrated a setting of the biologically plausible Arabidopsis circadian clock network with a single condition, where the BSLR performs similarly to a random guessing algorithm under one-shot sampling, and performs significantly better under multi-shot sampling. We also show that whether replicate averaging should be done or not varies with the sampling method.

Conclusions

One-shot sampling can miss a lot of potentially useful correlation information. Often gene expression data collected from plants is prepared under one-shot sampling. One factor that can partially mitigate the shortcomings of one-shot sampling is to prepare samples under a variety of conditions or perturbations. One-shot samples grown under the same condition can then be thought of as a surrogate for the multi-shot samples of an individual plant.

To clarify issues and take a step towards quantifying them, we proposed a gene expression dynamic model for gene regulatory network reconstruction that explicitly captures the condition variation effect. We show analytically and numerically the performance of two algorithms for single-gene and multi-gene settings. We also demonstrate the difficulty of network reconstruction without condition variation effect.

There is little agreement across the biology literature about how to model the impact of condition on the gene regulatory network. In some cases, it is not even clear that we are observing the same network structure as conditions vary. Nevertheless, our results suggest that the preparation of samples under different conditions can partially compensate for the shortcomings of one-shot sampling.

Supporting information

S1 appendix. joint estimation for single-gene autoregulation recovery..

The parameters A , γ , σ , and σ te are assumed unknown and jointly estimated in GLRT.

https://doi.org/10.1371/journal.pone.0224577.s001

S2 Appendix. Split Gaussian network prior.

The random network prior distribution used to generate the multi-gene network.

https://doi.org/10.1371/journal.pone.0224577.s002

• View Article
• PubMed/NCBI
• Google Scholar
• 2. Kang X. One-shot sampling simulations; 2019. Available from: https://github.com/Veggente/one-shot-sampling .
• 3. Poor HV. An Introduction to Signal Detection and Estimation. Springer-Verlag New York; 1994.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

2. The “One Shot” Case Study Revisited

Method. Thousand Oaks, CA: Pine Forge Press.———.(2000). Fuzzy Set Social Science. Chicago, IL: University of Chicago Press. Ragin, Charles C. and Howard Becker, eds.(1992). What Is a Case? Exploring the Foundations of Social Inquiry. Cambridge, UK: Cambridge University Press. Savolainen, Jukka.(1994).“The Rationality of Drawing Big Conclusions Based on Small Samples: In Defense of Mill's Methods.” Social Forces 72 (2), 1217–1224. Stinchcombe, Arthur.(1968). Constructing Social Theories.

Related Papers

Papers Revista De Sociologia

Mihaela Vancea

Journal of Politics

Bear Braumoeller

Papers: revista de sociologia

Erasmus Journal for Philosophy and Economics

Attilia Ruzzene

Mohammed Faruque Uddin

This article exemplifies Yager’s theory of fuzzy logic for interpersonal communication to the area of social research. Taking the dilemma between qualitative and quantitative approaches into the account, there is an anticipation to make a merge between these two. There is an enormous prospect to turn up scientists’ philanthropic innovations if they could use fuzzy logic in social science researches! However, by using fuzzy logic in sociological research there is a great deal of opportunity to study the social facts related to poverty, consumption, employment, intersubjectivity, social capital, environment, gender etc. How can we use Yager’s theory of Fuzzy Logic to analyze the relationship between social capital and labor market partcicpation? From the experiential connection in Bangladesh society, I try to seek this answer using a hypothetical quantification of attributes.

David Collier

Analysts who developed the set-theoretic comparative method (STCM) have formulated admirable goals for researchers who work in the qualitative and multi-method tradition. STCM includes above all Charles Ragin’s innovative approach of Qualitative Comparative Analysis (QCA). However, the analytic tools employed by STCM have in many ways become an obstacle to achieving these goals. For example, the system of fuzzy-set scoring appears to be problematic, poorly matched to a standard under-standing of conceptual structure, and perhaps unnecessary in its present form. Computer simulations suggest that findings suffer from serious problems of stability and validity; and while the choice of simulations that appropriately evaluate the method is a matter of some controversy, the cumulative weight of simulation results raises major concerns about STCM’s algorithms—i.e., its basic, formalized analytic procedures. Questions also arise about the cumbersome formulation of findings in what is often a remarkably large number of causal paths. Relatedly, some scholars question the STCM’s rejection of the parsimonious findings, in the form of “net effects,” routinely reported in other methodological traditions. Regarding applications, readily available software has encouraged publication of dozens of articles that appear to abandon key foundations of the method and rely far too heavily on these algorithms. Finally, STCM appears inattentive to the major, recent rethinking of standards and procedures for causal inference from observational data. These problems raise the concern that the set-theoretic comparative method, as applied and practiced, has become disconnected from the underlying analytic goals that motivated Charles Ragin to create it.

American Journal of Sociology

Edgar Ojeda

Sujay Rao Mandavilli

The Sociological Ninety ten rules that are proposed in this paper, are based on the fundamental premise that various branches of social sciences like sociology, anthropology and economics are human-centric and are therefore inexact, and vary fundamentally from the more precise and exact sciences like physics, chemistry and mathematics which are characterized by precision and exactitude. A high degree of precision and certainty may not therefore manifest themselves in various branches of the social sciences, even if they at times make use of mathematical models or statistical techniques. Therefore, for every postulated rule in most fields in the social sciences, there are likely to be many different exceptions. These may be described as cultural variations and cultural exceptions, and exceptions over time or space. The name 'Ninety ten' is only an easy-to-understand and easy-to-use nomenclature. Real world exceptions to any given observation could be twenty per cent, five per cent, or take on any other value, but the above nomenclature is chosen for convenience. Variations across or within cultures or within or in between socio-cultural groups, socioeconomic groups, occupational groups or any other parameter must be assessed based on the principles that we propose. This may be a basis for splitting up such groups if necessary for further study and evaluation, and the prerogative for this lies with the researcher. Thus, not only rule-based reasoning but also case-based reasoning must be used for various fields in the social sciences. Therefore, a fundamental premise of this paper is that exceptions must be sought actively, as these will lead to better research and hypothesis formulation. Thus, every researcher must think of rules and exceptions to those rules, and this must become a mindset. If exceptions are significant, they may warrant an altogether different line of research. This process will also greatly aid in inductive analysis, nomothetic rule-building and theorization, and play a major role in the 'Globalization of science', particularly social sciences.

Sociological Methods & Research

Charles Ragin , Paul Pennings

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Instructions for Authors
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Single-arm clinical trials: design, ethics, principles
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Minyan Wang ,
• Haojie Ni ,
• Chu Qin and
• http://orcid.org/0000-0002-8329-8927 Conghua Ji
• Zhejiang Chinese Medical University , Hangzhou , Zhejiang , China
• Correspondence to Professor Conghua Ji, Zhejiang Chinese Medical University, Hangzhou, Zhejiang 310053, China; jchi2005{at}126.com

Although randomised controlled trials are considered the gold standard in clinical research, they are not always feasible due to limitations in the study population, challenges in obtaining evidence, high costs and ethical considerations. As a result, single-arm trial designs have emerged as one of the methods to address these issues. Single-arm trials are commonly applied to study advanced-stage cancer, rare diseases, emerging infectious diseases, new treatment methods and medical devices. Single-arm trials have certain ethical advantages over randomised controlled trials, such as providing equitable treatment, respecting patient preferences, addressing rare diseases and timely management of adverse events. While single-arm trials do not adhere to the principles of randomisation and blinding in terms of scientific rigour, they still incorporate principles of control, balance and replication, making the design scientifically reasonable. Compared with randomised controlled trials, single-arm trials require fewer sample sizes and have shorter trial durations, which can help save costs. Compared with cohort studies, single-arm trials involve intervention measures and reduce external interference, resulting in higher levels of evidence. However, single-arm trials also have limitations. Without a parallel control group, there may be biases in interpreting the results. In addition, single-arm trials cannot meet the requirements of randomisation and blinding, thereby limiting their evidence capacity compared with randomised controlled trials. Therefore, researchers consider using single-arm trials as a trial design method only when randomised controlled trials are not feasible.

• Clinical assessment
• Clinical decisions

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/spcare-2024-004984

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Single-arm trials refer to a design method in which only one experimental group is established, without a control group. And, more and more studies have adopted the single-arm test design.

WHAT THIS STUDY ADDS

In some cases, single-arm trials have certain ethical advantages over randomised controlled trials, but researchers need to be careful to remain strictly scientific. Single-arm trial satisfies the principle of control, repetition and balance in design.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

In clinical practice, for refractory diseases and rare diseases, suitable treatment schemes can be found through single-arm trial. In terms of study design, when randomised controlled trials are not feasible, single-arm trial design can be used as one of the alternative methods.

Introduction

Careful trial design is key in clinical research. A good trial design leads to the successful implementation of a clinical trial and the production of clear results. 1 The high-quality evidence sought during clinical trials is often achieved through randomised controlled trials (RCTs), 2 which have long been regarded as the gold standard for evidence generation in the medical field because they provide a fair and accurate assessment of the effectiveness of treatments without the influence of confounding factors. 3 Ideally, medical treatment should be subjected to rigorous large-sample RCTs. However, limitations associated with study populations, challenges in obtaining evidence, high costs and ethical considerations hinder this, making RCTs infeasible in some cases. For example, there are still more than 30 000 ongoing clinical studies in the field of cardiovascular disease that face challenges, such as difficulties recruiting patients, resulting in delays in their completion or timely publication. 4 Traditional trial designs must be adapted to the current needs of rapidly evolving genomics, immunology and precision medicine. 5 Consequently, an increasing number of innovative trial designs, such as single-arm trials, 6 cluster randomised trials 7 and adaptive trials, 8 have been developed.

Single-arm trials refer to a design method in clinical research where only one experimental group is established, without the inclusion of a parallel control group. 9 It can be considered as an alternative when the design of an RCT is not feasible. The study design is open and does not involve randomisation or blinding. This type of experimental design is mostly used in the early stages of drug research, especially in the field of antitumoural therapy, for several critical illnesses, rare diseases and diseases for which no treatments are currently effective. In recent years, more and more single-arm trials have been applied to clinical research, 10 11 and relevant departments have also added single-arm trial design to new drug research guidelines. 9 12 These guidelines and standards aim for standardisation and consistency in single-arm trials to improve their quality and reliability.

Single-arm trials meet ethical requirements more easily and require lower sample sizes, lower trial costs and shorter trial times than RCTs. However, owing to the lack of parallel control groups, many confounding factors are difficult to control, and the conclusions of the trials may be difficult to interpret and are only used for evidence and decision-making when the effect size is clinically significant. Currently, single-arm trials are mainly used for the initial exploration of clinical trials for tumours, rare diseases, significant effects and high-demand drugs. 13 14

This article mainly analyses the advantages and disadvantages of single-arm trials in terms of scientific and ethical considerations in clinical research and provides reference suggestions for their application.

Overview of the study design

Clinical trial staging.

Single-arm trials are a commonly used clinical trial design that can be applied in all four phases of clinical trials. In the phase I clinical trials, single-arm trials are employed to investigate the mechanism of action, pharmacokinetic properties and safety of a new drug, providing foundational data for subsequent clinical trials. 15 The phase II trials aim to evaluate the efficacy and preliminary safety of a treatment and provide evidence for proceeding to a phase III trial. 16 Phase III trials typically involve large-scale RCTs designed to provide sufficient evidence to support the efficacy and safety of a new drug or treatment method. 17 However, due to ethical limitations and challenges in patient recruitment, 18 single-arm trial designs are also used in phases II and III clinical research. Guidelines and regulations pertaining to single-arm trials are continuously evolving, 10 11 particularly in phase II trials, as they can yield faster results, saving time and costs. Phase IV trials, also known as postmarketing studies, are conducted to monitor and assess the long-term effectiveness and safety of a drug, as well as explore new indications. In certain cases, single-arm trial designs have also been employed in phase IV trials.

Application scenarios

The wide variety of disease areas covered by the application of single-arm trial designs include oncological, 19 circulatory, immune, 20 haematological 21 and infectious 22 diseases among others. Most interventions studied in single-arm trials are pharmacological interventions and new therapeutic regimens, such as targeted therapy, gene therapy or multidrug combinations. Other types of interventions include surgery, 23 minimally invasive radiofrequency therapy 24 and the use of medical devices. 25 In addition, single-arm trials are also applied in rare diseases, new drug research, adverse reaction monitoring and other areas.

Oncological diseases

In oncology, single-arm trials are used to design protocols for treating refractory, recurrent and metastatic cancers. These conditions are usually very severe, and the patients have a short survival period; therefore, new and more effective treatments are needed. Owing to the lack of effective treatments, single-arm trials have become an important method for these patients to gain access to new drugs. Over recent years, an increasing number of findings from single-arm trials have been used to support the introduction of new drugs to the market. For example, by 31 December 2020, 125 of the 254 new drug studies approved by the US. Food and Drug Administration’s Accelerated Approval Programme were single-arm trials, representing 49% of the total, 26 while 11 of the 54 clinical trials on antineoplastic drugs approved in Europe between 2014 and 2016 were single-arm trials. 27 In a study on refractory thymic carcinoma, Sato et al conducted an efficacy analysis of a therapeutic drug using an external control single-arm trial design. 28

Obtaining approval for large-scale RCTs in patients with advanced cancer can be ethically challenging because of the potential for additional suffering and risk to patients. 29 30 Therefore, when RCTs are not feasible, single-arm trials become one of the solutions.

Rare diseases

Rare diseases are highly specific and involve a relatively small number of patients, making it difficult to find a sufficient number of control participants for the study. 31 Owing to the scarcity of patients, conducting controlled trials and recruiting patients becomes exceptionally difficult. 32 In such cases, finding a sufficient number of patients and control participants tends to be time-consuming and costly. However, the single-arm trial design does not include a parallel control group and only involves treating patients, conducting follow-up and analysing the data. 32 Thus, single-arm trials are more suitable for rare disease research because of their smaller sample size while aiding in understanding the pathogenesis, pathophysiological characteristics and treatment and prevention of rare diseases. For example, Wagner et al analysed the therapeutic efficacy of malignant perivascular epithelioid cell tumour through self-controlled single-arm trial. 33 Using a single-arm trial allows all participants to receive the intervention under investigation and also enhances patients’ willingness and satisfaction to participate in the trial, thus facilitating the smooth progress of the research.

Outbreaks of infectious diseases

Treatment selection is crucial during sudden outbreaks of infectious diseases. RCTs, although more scientifically sound, require longer research cycles to identify effective treatments and single-arm trials with shorter research cycles have become an important option to identify treatments as quickly as possible for diseases characterised by rapid spread, widespread impact and high severity. 34 35 In addition, single-arm trials are more advantageous given the willingness and level of participation of the patients. Many patients are willing to participate in trials and try new treatments because of the urgency and severity of their condition, and their active participation provides strong support for the research. For example, for COVID-19 outbreak, the protocols of single-arm trials were rapidly developed and adopted to find effective treatment options as quickly as possible. 34 36 Significant results were obtained from these trials within a short period, and valuable references for clinical treatment were obtained. These results helped to ensure better treatment for patients with COVID-19 infection at that time and provided important experience and support for future responses to similar outbreaks.

New treatment programmes

Diseases for which no effective treatment has been developed and for which a gold standard treatment is yet to be established are also suitable for single-arm trials. This approach has an important role in clinical research, particularly in the exploratory studies of rare or novel diseases. 37 Researchers can use single-arm trials to evaluate the effectiveness of new treatment strategies or drugs, addressing the challenges encountered in patient recruitment and facilitating the timely identification of suitable treatment regimens or drugs. The single-arm trial design can provide scientific evidence in specific circumstances for diseases where a definitive treatment is lacking. Cheng et al conducted a study on a new treatment approach for psoriasis using a single-arm trial. 38

Medical equipment

Single-arm trials are invaluable in medical device-related research, particularly in areas requiring surgical intervention. This trial design provides physicians with a convenient way to evaluate the safety and efficacy of new medical devices, thereby helping them to choose better treatment options. 39 40 For tricuspid regurgitation symptoms, Nickenig et al designed a single-arm trial to demonstrate the efficacy of the TriClip system. 41 In some cases, single-arm trials are viable. For example, in medical device-related studies, RCTs may not be applicable because of ethical or safety concerns.

Ethical principle embodied in the single-arm trials

Ethical principles.

To ensure that the trial complies with ethical principles and protects the rights of participants, several ethical requirements need to be met during the design and conduct of single-arm trials. These requirements include:

Informed consent: Like any other clinical trial, a single-arm trial requires obtaining informed consent from participants. Researchers must provide comprehensive trial information, including the purpose of the trial, intervention measures, potential risks and benefits, and any available alternative treatments or choices. The process of obtaining informed consent should adhere to ethical guidelines and ensure that participants understand the nature of the trial and its potential consequences.

Balancing risks and benefits: Since single-arm trials inherently lack a control group for comparison, researchers must carefully assess and inform patients about the potential risks associated with the trial intervention, ensuring their full understanding. During trial design, potential benefits and possible adverse effects should be carefully weighed, and efforts should be made to ensure that the potential benefits of the intervention outweigh the risks that participants may face.

Participant safety: Protecting the safety of participants during the trial is of utmost importance. Appropriate safety measures and monitoring protocols should be implemented to minimise potential harm. Regular assessments of participants’ physical conditions should be conducted, and any potential adverse events should be promptly addressed.

Data collection and analysis: Scientific rigour and transparency should be followed during data collection and analysis. Researchers should adhere to standardised protocols and ensure transparency in reporting results. This helps accurately interpret the trial results and facilitates evidence-based decision-making.

Ethical review and oversight: Similar to any other clinical trial, single-arm trials also require ethical review and oversight by research ethics committees or institutional review boards. These entities evaluate the trial design, informed consent process, participant safety measures and overall ethical considerations to ensure that the trial meets ethical standards.

Ethical advantages

Compared with RCTs, single-arm trials have certain ethical advantages in some situations:

Equity in treatment: In a single-arm trial, all participants receive the experimental intervention, eliminating the potential disparity that arises from randomisation. This ensures that every participant has an equal opportunity to receive the treatment being investigated, which is particularly important when the control group receives a placebo or standard treatment. However, it is important to note that patients in single-arm trials also face similar potential risks.

Respect for patient choice: Single-arm trials provide an option for patients who do not wish to be assigned to a control group receiving a placebo or standard treatment. It respects patients’ autonomy by allowing them to access the experimental intervention without being randomised to a non-intervention group.

Opportunity for access to new treatments: Single-arm trials offer patients the opportunity to access new treatment methods or interventions that may not be available outside of the trial. This is particularly important in the case of rare diseases or limited treatment options. It allows patients to potentially benefit from innovative therapies, improving their quality of life or even providing life-saving benefits.

Feasibility in rare diseases: Conducting RCTs in rare diseases can be challenging due to the limited number of eligible patients. Single-arm trials require smaller sample sizes, making them more feasible in such cases. By using a single-arm design, researchers can gather important preliminary data on the effectiveness and safety of the intervention, which can inform future research and treatment decisions.

Ethical considerations in control group use: In some cases, using a control group in an RCT may raise ethical concerns. For example, if an existing treatment has already been proven highly effective, it may be considered unethical to assign participants to a control group and deny them the known treatment. In such cases, single-arm trials can provide an ethical alternative by evaluating the effectiveness of the intervention without compromising the ethical principles of patient care.

Timely handling of adverse events: Single-arm trials are non-blinded trials, allowing researchers to provide better follow-up and monitor participants’ physical conditions. If adverse events occur, researchers can promptly provide relevant treatment, saving unblinding time and avoiding irreversible consequences.

It is important to consider these ethical advantages in the context of each specific study and balance them with scientific rigour to ensure the validity and ethical integrity of the research.

Scientific principles embodied in the single-arm trials

A single-arm trial is a unique research design that cannot fully realise the principles of randomisation and blinding. Therefore, the study design must follow the principles of control, replication and equilibrium as much as possible to ensure accuracy and reliability. This paper summarises the different ways that RCTs and single-arm trials meet the scientific principle.

Principle of control

Although a single-arm trial lacks the use of a parallel control in its experimental design, a control group is typically present in the form of an external control, such as a target value or historical study control. Target value control sets a goal for the utility to be achieved in the experiment based on the best-effect value obtained from a previous study or an industry-standard treatment. A historical control study uses the results of a high-quality study as controls, where the characteristics of the historical study population and the evaluated effect indicators were consistent with those of the current study. Therefore, the selection of controls outside the single-arm trial is also a critical aspect. If not appropriately selected, the effects of confounding factors will be substantially elevated. The increased availability of electronic medical records has made historical control data readily accessible. However, the selection of appropriate controls for baseline patient characteristics and diagnostic criteria is crucial because of a high degree of bias. Propensity score matching or stratified analyses can be used to reduce confounding biases.

Principle of balance

The principle of balance is very important when designing single-arm clinical trials. Researchers need to carefully consider inclusion and exclusion criteria, which can affect the conduct of trials, the interpretation of results and the safety of patients. Strict inclusion and exclusion criteria can result in a study population that is not representative of the broader population, affecting the ability to extrapolate results and determine the effectiveness of treatments. Therefore, the internal and external validity of the study needs to be weighed when developing inclusion and exclusion criteria. In addition, balanced comparability between the two groups can be achieved by matching the control group. The effect index of matching control group should be representative, objective and referential. Representativeness means that the evaluation indicators can best represent the effect of the intervention and meet the accepted standards of the industry. Objectivity means that the indicators used (such as laboratory tests, imaging tests and key clinical events) are objective and not influenced by subjective factors. Referrability indicates that this index has been widely used as the main therapeutic effect evaluation index in similar studies. When matching the control group, care should be taken to avoid overmatching. Overmatching may result in a smaller sample size, limiting the statistical power and generalisations of the study. Therefore, it is necessary to carefully select matching indicators and matching algorithms when matching the control group to ensure balanced comparability between groups while maintaining sufficient sample size. In short, single-arm clinical trials meet the balanced comparability between groups through strict exclusion criteria and/or control matching to ensure the reliability and validity of the study. At the same time, it is necessary to weigh internal and external validity and ensure that the matching process meets the requirements of representation, objectivity and referencability.

Principle of repetition

Sample size calculation for a single-arm test is a critical aspect of the study design process. Single-arm trials require small sample sizes, making it particularly important to calculate a sufficient sample size to ensure test efficacy. Researchers must clearly define the primary endpoint of the study as a meaningful difference before and after the test. This clinical decision must be statistically significant in the design of the number of patients required in the study. For example, in a study of acute myeloid leukaemia treatment, where the primary endpoint was the overall remission rate, a review of the literature revealed that the objective response rate (ORR) of the historical study was approximately 35%, whereas the current study was projected to have an ORR of approximately 60%. 42 These rates can be used to define the study hypotheses and determine the sample size required to conduct the study. The ORR data obtained from historical studies are the target value of the effect, which is a set of widely recognised evaluation criteria obtained from a large amount of historical data. The ORR predicted by the study was the target value, which was the level expected to be achieved by the evaluation criteria for the efficacy of the intervention to be studied. The key to determining the target value was to establish a clinically meaningful superiority threshold; that is, how much higher the level of efficacy of the current study must be than the target value before it is considered clinically meaningful. Other methods of sample size estimation for survival analysis can also be used in single-arm trials. 43

Statistical analyses of the single-arm trials are mostly descriptive and exploratory. Appropriate datasets are selected for analysis to statistically describe baseline patient data. Appropriate statistical methods are selected to perform hypothesis testing on the study outcomes and controls to compare the effects of the interventions. Since relevant survival analysis indicators are included in the outcome metrics, survival curves can be plotted to visualise the prognostic progress of patients after the intervention. Several studies conduct post hoc statistical tests, such as grouping according to the efficacy and analysing and exploring the relevant factors affecting the efficacy.

Comparison with other study designs

Single-arm trials, RCTs and cohort studies are the three common research design methods with advantages and applications in different research scenarios. Figure 1 illustrates a basic diagram of these research methods and provides an initial understanding of their distinctions. Although single-arm trials are non-RCTs, they have advantages over cohort studies. However, their evidence is considered to be less scientifically robust compared with that of RCTs. Figure 2 illustrates the differences among the three research design approaches in terms of the principles of science.

• Download figure
• Open in new tab
• Download powerpoint

Flow chart of the three research design methods. (A) Single-arm trial; (B) randomised controlled trial; (C) conhort study.

Scientific principles embodied in the three research methods.

Medical technology is advancing rapidly and has successfully addressed many challenging diseases. However, the number of rare and difficult-to-treat diseases is increasing. Although RCTs have been considered the gold standard for demonstrating treatment efficacy, they are not always feasible or ethical to be conducted. 44 Single-arm trials are usually used when a control group cannot be found to match patients in the trial group or is unsuitable for placebo and blank controls due to ethical concerns. In these cases, single-arm trials are considered to be more easily aligned with ethical principles. The absence of a control group ensures that all patients have access to the potential benefits of the trial intervention, which is ethically advantageous. However, it must be acknowledged that all patients also face a consistent risk of harm. Unlike RCTs, single-arm trials do not require randomisation of groups, thus avoiding the ethical and implementation difficulties that may arise. In addition, sample size requirements can be reduced, saving time and resources because a single-arm trial requires only one group. They also allow for more flexibility in adjusting treatment regimens and observational metrics and better reflect real-life clinical practice. 6 Single-arm trials enable acquisition of preliminary efficacy data. Treatment response data can be collected quickly because of the requirement for only one group. This has resulted in a more rapid assessment of drug efficacy and safety. This approach can be especially valuable in emergencies or for treating rare diseases because it enables healthcare providers to offer effective treatment options to patients more quickly. Consequently, regulatory authorities may accept results from single-arm studies in certain circumstances, such as with rare diseases or specific disease subtypes with small patient populations where no effective standard treatment exists, or support expedited development using accelerated approval, conditional approval or other regulatory pathways. 45 46

Cohort and single-arm trials are commonly conducted in clinical research. However, single-arm trials are more persuasive in their evidence as non-RCTs than are cohort studies. Cohort studies can provide evidence for disease aetiology, prevention and treatment by observing and analysing the occurrence and prognosis of diseases in a population. 47 However, cohort studies may be affected by selection and measurement biases, which can cause inaccurate results. 48 49 Conversely, single-arm trials can assess the effect of an intervention by imposing this on a test group and comparing this with an external control. The study design can better control the interference of external variables and improve the accuracy and reliability of the results. Furthermore, a balanced and comparable population can be achieved in the test and control groups by selecting an external control. 6 50 Therefore, the outcomes of a single-arm trial may be more convincing and provide physicians and patients with a more reliable basis for decision-making. Although both cohort studies and single-arm trials are crucial methods in clinical research, the evidence from single-arm trials can be more persuasive and can offer more precise and reliable data for better direct clinical practice.

However, single-arm trials have certain limitations. RCTs can provide more scientific evidence. The absence of a control group means that the interpretation of results cannot rely solely on the intervention, thus reducing the reliability of the evidence. A single-arm trial does have a control but is not drawn from the same period or study as the subjects but is instead drawn from an external historical control or contemporaneous cohort study control. This introduces some bias as the subjects in the trial and external control groups could be from different populations, and therefore, less comparable. When selecting a control group, it is important to carefully screen the patients based on their baseline characteristics and diagnostic criteria. As parallel controls are lacking, comparisons could only be made with external historical data to evaluate the validity and safety of the study population. However, finding historical data that fully align with the current study design is challenging, and distinguishing the differences between studies makes it difficult to evaluate the results.

Because only one test group is present in a single-arm trial, the principles of randomisation and blinding cannot be applied, and this may introduce bias in the results. Therefore, extra care must be taken in designing and executing the trial to minimise bias and error and improve the accuracy and reliability of the results. In highly sensitive medical fields, it is essential to strive for higher levels of evidence. RCTs have always been the first choice in the design of clinical trials. Single-arm trials are only considered when it is not feasible to conduct an RCT.

In certain specific circumstances, a single-arm trial design is a feasible research design method. This is primarily because it involves only one experimental group without a control group, allowing all participants to receive treatment. However, researchers must fully acknowledge the limitations of single-arm trials in their design and implementation. These limitations may include the absence of a parallel control group, weaker interpretation of results, and the potential for selection bias in external comparison groups. To overcome these limitations and enhance the reliability and interpretability of single-arm trials, researchers must exercise caution and meticulousness when selecting external controls, determining sample sizes and setting effect measures. In conclusion, the use of single-arm trials should be carefully considered and should only be chosen when it is not possible to conduct an RCT.

Ethics statements

Patient consent for publication.

Not applicable.

• Woodcock J ,
• Minneci PC ,
• Krittanawong C ,
• Johnson KW ,
• Subbiah V ,
• Musoro JZ , et al
• Coggan S , et al
• Thorlund K ,
• Haggstrom J ,
• Park JJ , et al
• Cucherat M ,
• Laporte S ,
• Delaitre O , et al
• Ladanie A ,
• Briel M , et al
• Hatswell AJ ,
• Berlin JA , et al
• Gonzalez-Velez M ,
• Lalwani S ,
• Parekh S , et al
• Jiang J , et al
• Nangaku M ,
• Takabe S , et al
• Kaizer AM ,
• Wang J , et al
• Yeung RSM ,
• Cleary G , et al
• Santoro C ,
• Le PQ , et al
• Terada-Hirashima J ,
• Takamatsu Y ,
• Shimizu Y , et al
• Glassman AR , et al
• Jacobowitz O ,
• Sugiura J ,
• Ueda T , et al
• Ribeiro TB ,
• Colunga-Lozano LE ,
• Araujo APV , et al
• Savović J , et al
• Satouchi M ,
• Itoh S , et al
• Downing NS ,
• Aminawung JA ,
• Shah ND , et al
• Goldsmith JC ,
• Wagner AJ ,
• Riedel RF , et al
• He P , et al
• Mancher M ,
• Cuff PA , et al
• Ogbe A , et al
• Yu M , et al
• Peng C , et al
• Strueber M ,
• O’Driscoll G ,
• Jansz P , et al
• Lawson JH ,
• Glickman MH ,
• Ilzecki M , et al
• Nickenig G ,
• Lurz P , et al
• Li M-Y , et al
• Nagashima K ,
• Sato Y , et al
• Baumfeld Andre E ,
• Reynolds R ,
• Caubel P , et al
• Mahendraratnam N ,
• Chatterjee A , et al
• Grimson F ,
• Layton D , et al
• Carlson MDA ,
• Morrison RS
• Ivashynka A ,
• Logroscino G , et al
• Dagenais S ,
• Madsen A , et al

Contributors CJ and MW researched the literature and conceived the study. MW wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Funding This work was supported by Research Center for the Development of Medicine and Health Science and Technology of the National Health Commission (No. 2023YFC3606200), the Department of Science and Technology of Zhejiang Province (No. 2023C25012) and Zhejiang Provincial Health Commission (No. 2023ZL360 and No. 2024KY1195).

Competing interests None declared.

Provenance and peer review Not commissioned; internally peer reviewed.

Read the full text or download the PDF: