a scientist conducting a research study on sleep and learning

The Impact of Sleep on Learning and Memory

By Kelly Cappello, B.A.

For many students, staying awake all night to study is common practice. According to Medical News Today , around 20 percent of students pull all-nighters at least once a month, and about 35 percent stay up past three in the morning once or more weekly.

That being said, staying up all night to study is one of the worst things students can do for their grades. In October of 2019, two MIT professors found a correlation between sleep and test scores : The less students slept during the semester, the worse their scores.

So, why is it that sleep is so important for test scores? While the answer seems simple, that students simply perform better when they’re not mentally or physically tired, the truth may be far more complicated and interesting.

In the last 20 years, scientists have found that sleep impacts more than just students’ ability to perform well; it improves their ability to learn, memorize, retain, recall, and use their new knowledge to solve problems creatively. All of which contribute to better test scores.

Let’s take a look at some of the most interesting research regarding the impact of sleep on learning and memory.

How does sleep improve the ability to learn?

When learning facts and information, most of what we learn is temporarily stored in a region of the brain called the hippocampus. Some scientists hypothesize that , like most storage centers, the hippocampus has limited storage capacity. This means, if the hippocampus is full, and we try to learn more information, we won’t be able to.

Fortunately, many scientists also hypothesize that sleep, particularly Stages 2 and 3 sleep, plays a role in replenishing our ability to learn. In one study, a group of 44 participants underwent two rigorous sessions of learning, once at noon and again at 6:00 PM. Half of the group was allowed to nap between sessions, while the other half took part in standard activities. The researchers found that the group that napped between learning sessions learned just as easily at 6:00 PM as they did at noon. The group that didn’t nap, however, experienced a significant decrease in learning ability [1].

How does sleep improve the ability to recall information?

Humans have known about the benefits of sleep for memory recall for thousands of years. In fact, the first record of this revelation is from the first century AD. Rhetorician Quintilian stated, “It is a curious fact, of which the reason is not obvious, that the interval of a single night will greatly increase the strength of the memory.”

In the last century, scientists have tested this theory many times, often finding that sleep improves memory retention and recall by between 20 and 40 percent. Recent research has led scientists to hypothesize that Stage 3 (deep non-Rapid Eye Movement sleep, or Slow Wave Sleep) may be especially important for the improvement of memory retention and recall [2].

How does sleep improve long-term memory? 

Scientists hypothesize that sleep also plays a major role in forming long-term memories. According to Matthew Walker, professor of neuroscience and psychology at UC Berkeley, MRI scans indicate that the slow brain waves of stage 3 sleep (deep NREM sleep) “serve as a courier service,” transporting memories from the hippocampus to other more permanent storage sites [3].

How does sleep improve the ability to solve problems creatively?

Many tests are designed to assess critical thinking and creative problem-solving skills. Recent research has led scientists to hypothesize that sleep, particularly REM sleep, plays a role in strengthening these skills. In one study, scientists tested the effect of REM sleep on the ability to solve anagram puzzles (word scrambles like “EOUSM” for “MOUSE”), an ability that requires strong creative thinking and problem-solving skills.

In the study, participants solved a couple of anagram puzzles before going to sleep in a sleep laboratory with electrodes placed on their heads. The subjects were woken up four times during the night to solve anagram puzzles, twice during NREM sleep and twice during REM sleep.

The researchers found that when participants were woken up during REM sleep, they could solve 15 to 35 percent more puzzles than they could when woken up from NREM sleep. They also performed 15 to 35 percent better than they did in the middle of the day [4]. It seems that REM sleep may play a major role in improving the ability to solve complex problems.

So, what’s the point?

Sleep research from the last 20 years indicates that sleep does more than simply give students the energy they need to study and perform well on tests. Sleep actually helps students learn, memorize, retain, recall, and use their new knowledge to come up with creative and innovative solutions.

It’s no surprise that the MIT study previously mentioned revealed no improvement in scores for those who only prioritized their sleep the night before a big test. In fact, the MIT researchers concluded that if students want to see an improvement in their test scores, they have to prioritize their sleep during the entire learning process. Staying up late to study just doesn’t pay off.

Interested in learning more about the impact of sleep on learning and memory? Check out this Student Sleep Guide .

Author Biography

Kelly Cappello graduated from East Stroudsburg University of Pennsylvania with a B.A. in Interdisciplinary Studies in 2015. She is now a writer, specialized in researching complex topics and writing about them in simple English. She currently writes for Recharge.Energy , a company dedicated to helping the public improve their sleep and improve their lives.

• Mander, Bryce A., et al. “Wake Deterioration and Sleep Restoration of Human Learning.” Current Biology, vol. 21, no. 5, 2011, doi:10.1016/j.cub.2011.01.019.
• Walker M. P. (2009). The role of slow wave sleep in memory processing. Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine, 5(2 Suppl), S20–S26.
• Walker, Matthew. Why We Sleep. Scribner, 2017.
• Walker, Matthew P, et al. “Cognitive Flexibility across the Sleep–Wake Cycle: REM-Sleep Enhancement of Anagram Problem Solving.” Cognitive Brain Research, vol. 14, no. 3, 2002, pp. 317–324., doi:10.1016/s0926-6410(02)00134-9.

Posted on Dec 21, 2020 | Tagged: learning and memory

© The Trustees of the University of Pennsylvania | Site best viewed in a supported browser . | Report Accessibility Issues and Get Help | Privacy Policy | Site Design: PMACS Web Team. | Sitemap

Featured Topics

Featured series.

A series of random questions answered by Harvard experts.

Explore the Gazette

Read the latest.

Amazon butterfly evolved from hybrids

‘Harvard Thinking’: Is AI friend or foe? Wrong question.

Getting ahead of dyslexia

Brain practices new tasks while we sleep.

MGH News and Public Affairs

Scientists believe it plays a role in how we learn and form long-term memories

Why do we sleep? Scientists have debated this question for millennia, but a new study adds fresh clues for solving this mystery.

The findings, published in the  Journal of Neuroscience, may help explain how humans form memories and learn, and could eventually aid the development of assistive tools for people affected by neurologic disease or injury. The study was conducted by Massachusetts General Hospital in collaboration with colleagues at Brown University, the Department of Veterans Affairs, and several other institutions.

Scientists studying laboratory animals long ago discovered a phenomenon known as “replay” that occurs during sleep, explains neurologist Daniel Rubin of the MGH Center for Neurotechnology and Neurorecovery, the lead author of the study. Replay is theorized to be a strategy the brain uses to remember new information. If a mouse is trained to find its way through a maze, monitoring devices can show that a specific pattern of brain cells, or neurons, will light up as it traverses the correct route. “Then, later on while the animal is sleeping, you can see that those neurons will fire again in that same order,” says Rubin.

Scientists believe that this replay of neuronal firing during sleep is how the brain practices newly learned information, which allows a memory to be consolidated — that is, converted from a short-term memory to a long-term one.

However, replay has only been convincingly shown in lab animals. “There’s been an open question in the neuroscience community: To what extent is this model for how we learn things true in humans? And is it true for different kinds of learning?” asks neurologist Sydney S. Cash, co-director of the Center for Neurotechnology and Neurorecovery at MGH and co-senior author of the study. Importantly, says Cash, understanding whether replay occurs with the learning of motor skills could help guide the development of new therapies and tools for people with neurologic diseases and injuries.

To study whether replay occurs in the human motor cortex — the brain region that governs movement — Rubin, Cash, and their colleagues enlisted a 36-year-old man with tetraplegia (also called quadriplegia), meaning he is unable to move his upper and lower limbs, in his case due to a spinal cord injury. The man, identified in the study as T11, is a participant in a clinical trial of a brain-computer interface device that allows him to use a computer cursor and keyboard on a screen. The investigational device is being developed by the BrainGate consortium, a collaborative effort involving clinicians, neuroscientists, and engineers at several institutions with the goal of creating technologies to restore communication, mobility, and independence for people with neurologic disease, injury, or limb loss. The consortium is directed by Leigh R. Hochberg of MGH, Brown University, and the Department of Veterans Affairs.

In the study, T11 was asked to perform a memory task similar to the electronic game Simon, in which a player observes a pattern of flashing colored lights, then has to recall and reproduce that sequence. He controlled the cursor on the computer screen simply by thinking about the movement of his own hand. Sensors implanted in T11’s motor cortex measured patterns of neuronal firing, which reflected his intended hand movement, allowing him to move the cursor around on the screen and click it at his desired locations. These brain signals were recorded and wirelessly transmitted to a computer.

That night, while T11 slept at home, activity in his motor cortex was recorded and wirelessly transmitted to a computer. “What we found was pretty incredible,” says Rubin. “He was basically playing the game overnight in his sleep.” On several occasions, says Rubin, T11’s patterns of neuronal firing during sleep exactly matched patterns that occurred while he performed the memory-matching game earlier that day.

“This is the most direct evidence of replay from motor cortex that’s ever been seen during sleep in humans,” says Rubin. Most of the replay detected in the study occurred during slow-wave sleep, a phase of deep slumber. Interestingly, replay was much less likely to be detected while T11 was in REM sleep, the phase most commonly associated with dreaming. Rubin and Cash see this work as a foundation for learning more about replay and its role in learning and memory in humans.

“Our hope is that we can leverage this information to help build better brain-computer interfaces and come up with paradigms that help people learn more quickly and efficiently in order to regain control after an injury,” says Cash, noting the significance of moving this line of inquiry from animals to human subjects. “This kind of research benefits enormously from the close interaction we have with our participants,” he adds, with gratitude to T11 and other participants in the BrainGate clinical trial.

Hochberg concurs. “Our incredible BrainGate participants provide not only helpful feedback toward the creation of a system to restore communication and mobility, but they also give us the rare opportunity to advance fundamental human neuroscience — to understand how the human brain works at the level of circuits of individual neurons,” he says, “and to use that information to build next-generation restorative neurotechnologies.”

Rubin is also an instructor in neurology at Harvard Medical School. Cash is an associate professor of neurology at HMS. Hochberg is a senior lecturer on neurology at HMS and professor of engineering at Brown University.

This work was supported by the Department of Veterans Affairs, the National Institute of Neurologic Disease and Stroke, the National Institute of Mental Health, Conquer Paralysis Now, the MGH-Deane Institute, the American Academy of Neurology, and the Howard Hughes Medical Institute at Stanford University.  

Share this article

You might like.

Genomic findings challenge thinking on what makes a species

In podcast, a lawyer, computer scientist, and statistician debate ethics of artificial intelligence

Harvard lab’s research suggests at-risk kids can be identified before they ever struggle in school

Harvard announces return to required testing

Leading researchers cite strong evidence that testing expands opportunity

When will patients see personalized cancer vaccines?

Sooner than you may think, says researcher who recently won Sjöberg Prize for pioneering work in field

For all the other Willie Jacks

‘Reservation Dogs’ star Paulina Alexis offers behind-the-scenes glimpse of hit show, details value of Native representation

Subscribe for Free!

We'll send you our print magazine 6x per year!

Subscribe Now

Prefer email? Sign-up for our email newsletter

What Researchers Are Learning About Brain Health by Studying Sleep

Research on sleep disorders and the importance of regular shut-eye has deepened our understanding of the link between sleep and brain health..

Connor Reiff was 11 when doctors determined he had narcolepsy with cataplexy, a rare condition that manifests in extreme daytime sleepiness and the sudden loss of muscle control. After months of trial and error with different prescriptions, participation in a clinical trial, and countless conflicts with insurance providers, the Reiff family thought they had Connor's sleep issues under control.

Then, when Connor turned 13, he began sleeping for days at a time, only rising when his parents woke him to eat or use the bathroom. He'd sometimes walk in a sleep stupor, physically pushing away family members he encountered. “Our son was disappearing before our eyes,” says Connor's mother, Sarah Reiff, who lives with her family in Noblesville, IN. “This is a kid who plays soccer, who loves school, who is very involved in activities—and suddenly he isn't doing any of that.”

The eventual diagnosis was Kleine-Levin syndrome (KLS), a disorder marked by episodes of excessive sleep—up to 20 hours a day. Specialists believe the condition may be related to malfunctioning of the thalamus and hypothalamus, areas of the brain that govern appetite and sleep. More than two thirds of KLS patients are teenage boys.

Studying rare sleep disorders such as KLS has given scientists insight into why we sleep and how we can do it better. Overall, there are more than 80 sleep disorders, ranging from the mildly annoying to the potentially deadly. The best known is probably insomnia; about 10 percent of the general population has chronic insomnia, an inability to fall asleep for multiple nights over a period of months.

Addressing sleep disorders “is paramount to not only protecting the brain down the road but also on a day-to-day basis,” says Daniel Barone, MD, associate medical director of the Weill Cornell Center for Sleep Medicine in New York City and co-author of The Story of Sleep: From A to Zzz (Rowman & Littlefied, 2023). “One of the best ways to take care of our brains is by getting quality sleep.”

Intriguing Clues

Research on sleep disorders has led to improvements in treatment for a variety of sleep and neurologic conditions. Case in point: Studies in the late 1990s on the causes of narcolepsy with cataplexy—the condition Connor was initially diagnosed with—led to the development of dual orexin receptor agonists, drugs now commonly prescribed to treat insomnia. The researchers discovered that people with narcolepsy with cataplexy often had low levels of hypocretins (orexins), brain chemicals that sustain alertness and prevent REM from happening at the wrong time.

“Once they found out, ‘If I take away your hypocretin, it makes you sleepy,’ there was a new idea of how to make a sleeping pill,” says Rafael Pelayo, MD, clinical professor of psychiatry and behavioral sciences at Stanford University School of Medicine in California and a sleep specialist at the university's Sleep Medicine Center.

In another case, the research finding that REM sleep behavior disorder (RBD)—which causes people to enact their dreams—may be an initial manifestation of various disorders, including Parkinson's disease , Lewy body dementia, and multiple system atrophy , has driven further studies on RBD and neuroprotective treatments that could arrest or postpone the onset of these disorders. “Recognizing the link between RBD and future Parkinson's and dementia risk offers a window of opportunity for neuroprotection, so that once RBD is accurately diagnosed, we could treat patients in the short term to control injury risk, and in the long term to stop neurodegeneration of the brain,” says Erik K. St. Louis, MD, director of the Sleep Behavior and Neurophysiology Research Lab at Mayo Clinic in Rochester, MN.

“We don't have the treatment yet, but we remain hopeful that a safe and effective neuroprotective treatment or intervention will be identified soon, so stay tuned,” adds Dr. St. Louis, who's also associate professor of neurology at Mayo Clinic. RBD occurs in 1 to 2 percent of the general population and between 7 and 13 percent of adults age 60 and older.

Sleep disorders can disrupt not only nighttime behavior but also activities throughout the day. One of Dr. St. Louis' patients, who enjoys walleye fishing on Minnesota lakes, has type 1 narcolepsy with cataplexy—which means he can have sudden attacks of weakness, especially when experiencing strong emotions. He had several attacks triggered by the feel of a fish biting at the line and would get so excited, Dr. St. Louis says, that he often fell overboard. “We talked about safety, like always having a life preserver on. There were many times his family had to pull him out of the water. Once his condition was controlled, he safely landed several big catches,” says Dr. St. Louis.

Connor Reiff's condition, KLS, can be associated with overeating, irritability, and hallucinations, according to the National Institute of Neurological Disorders and Stroke . He twice tumbled out of his second-floor bedroom window while asleep, and though he was not seriously injured either time, he now has a full-time aide and keeps an alarm on his bed that goes off if he gets up.

Now 19, Connor experiences fewer periods of extreme sleep, thanks to six daily prescription medications, including sodium oxybate (Xyrem) and pitolisant (Wakix), both of which prevent the sudden onset of daytime sleepiness and muscle paralysis. For those with KLS, episodes generally decrease in frequency and intensity over the course of eight to 12 years. “His trajectory is going to be different from that of other kids, but it's still a path to success,” Connor's mother says. “He's going to do well in this world. It's just going to take a little longer.”

Some disorders that aren't necessarily dangerous can indicate a more serious underlying problem. For example, sleep talking (somniloquy) is considered a normal variant of sleep, but doctors take note when patients 50 years and older begin doing it because it could be an early sign of RBD. Sleepwalking, or somnambulism, is usually benign, but it too can signal the onset of RBD, especially if the sleeper seems to be acting out a dream.

Protective Factor

Inadequate sleep has both short- and long-term effects. People who sleep less than seven to eight hours each night may experience daytime sleepiness, forgetfulness, or an inability to concentrate the next day. Over the long haul, lack of sleep may lead to an accumulation of waste products in the brain. Research published in Science in November 2019 found that cerebrospinal fluid appears to synchronize with brain waves during sleep, clearing the brain of metabolic waste products in rhythmic, pulsing patterns. The project involved 13 participants ages 23 to 33, prompting researchers to wonder what a study involving older participants would reveal. Older adults can have sleep problems because of age-related conditions or medications. They also have reduced non-REM slow wave activity, which could lead to more accumulation of neurotoxic products in the brain, Dr. St. Louis says.

The study authors note that it's unclear if there's a link between the cerebrospinal fluid's work and the slow waves in neural activity that contribute to memory consolidation. “Sleep specialists are certain that during sleep, especially deep sleep, the brain clears out the byproducts of daily living,” says Dr. Barone. “The converse of that is that if we don't sleep well over many years and decades, the brain may not be able to clear out these bad products. The belief is that the buildup of waste products over this long time span can contribute to Alzheimer's disease and other degenerative illnesses.”

A 2021 study in Neurology found an association between robust function of the glymphatic sysem—which clears waste from the central nervous system—and better language scores, word recall, and higher gray matter volume in older adults.

Poor sleep also may contribute to mood problems. An article in the May 2021 edition of Frontiers in Psychiatry noted there was mounting evidence of “a close relationship between sleep disturbance and mood disorders, including major depressive disorder and bipolar disorder.”

We spend about a third of our lives sleeping, but even after decades of research, questions persist about what happens while we slumber. “We commonly say that sleep is restorative. You surely feel refreshed after a good night's sleep, but what is actually being restored?” asks Dr. Pelayo. “[Learning more about] the function of sleep is ultimately going to tell us how our brains really work.”

Additional important physiological functions that occur during sleep include building memories, processing information and emotions, restoring metabolic balance, and rebooting the immune system, according to Dr. St. Louis. In 2009, researchers discovered that a lot of bone growth occurs in children while they sleep, in part because lying down removes pressure and allows their bones to elongate. Another study, published in 2011 in the Journal of Clinical Densitometry , found that shorter sleep could impair bone mass.

Valuable Asset

Some people treat not getting enough sleep as a point of pride—a sign of hard work or an active social life—despite the potential health risks. “There used to be an attitude of ‘I'll sleep when I'm dead,’ and there's an increasing recognition that a lack of sleep may actually hasten that result,” says Brendan P. Lucey, MD, associate professor of neurology and head of sleep medicine at Washington University School of Medicine in St. Louis.

Too little sleep has been associated with multiple negative health outcomes, including obesity, high blood pressure, diabetes, and metabolic dysfunction. And untreated sleep apnea can contribute to cognitive problems, says Dr. Lucey.

Dr. Barone admits to pulling all-nighters in college and medical school, “but the more we learn, the more we realize that we shouldn't be doing that.” There's also something to the expression “sleep on it,” says Dr. Pelayo, who remembers reading his medical textbooks until the words on the page became blurry, falling asleep, and then waking up feeling he had a better grasp of the material he'd last read.

Positive trends in the study of sleep include the ever improving therapeutics that can treat a broad range of sleep disorders, Dr. St. Louis says. Another is that sleep is now recognized as a “multi-disciplinary team sport,” with primary care physicians, pulmonologists, psychiatrists, ENT specialists, and others joining neurologists in its study.

“There's still an endemic problem of sleep deprivation in our society. That's hard to make go away,” Dr. St. Louis says. “But over the last decade we're paying more attention to sleep, which is a good thing, and there's more messaging that sleep is one of the three pillars of health. Diet, exercise, and sleep are the three things we can all control to try to optimize our performance, longevity, and quality of life.”

Five Ways to Sleep Well

Rest Resources

• American Academy of Neurology: BrainandLife.org
• American Academy of Sleep Medicine: aasm.org ; 630-737-9700
• KLS Foundation: klsfoundation.org ; 714-394-9817
• Narcolepsy Network: narcolepsynetwork.org ; 888-292-6522
• National Institute of Neurological Disorders and Stroke: ninds.nih.gov ; 800-352-9424
• National Organization for Rare Disorders:  rarediseases.org ; 617-249-7300
• National Sleep Foundation: thensf.org ; 703-243-1697
• Sleep Foundation:  sleepfoundation.org ; 877-672-8966
• Sleep Research Society: sleepresearchsociety.org ; 630-737-9702
• World Sleep Society: worldsleepsociety.org ; 507-316-0084

• ABOUT THE PROJECT
• ONGOING PROJECTS
• PUBLICATIONS
• SURVEY METHODS
• SPOTLIGHT ON:
• DRUGS AND ALCOHOL
• PHYSICAL ACTIVITY
• SEDENTARY BEHAVIOR
• SEE ALL FINDINGS
• WOMEN’S FINANCIAL SECURITY
• HOME OWNERSHIP
• EMERGENCY FUNDS
• HEALTH INSURANCE
• INVESTMENTS
• LIFE INSURANCE
• LONG TERM CARE
• VOLUNTEERISM
• CIVIC ENGAGEMENT
• RELIGIOUS ORGANIZATIONS
• WORKING FOR PAY
• AFFILIATE INTERVIEWS
• DIRECTORS’ BLOG POSTS
• EXPERT PERSPECTIVES
• PERSONAL PERSPECTIVES
• STANFORD CENTER ON LONGEVITY >
• Research Update on Diet
• Research Update on Sleep

Research Update 1

By Marie Conley Smith

I n a world full of opportunities, stressors, inequalities, and distractions, maintaining a healthy lifestyle can be challenging, and sleep is often the first habit to suffer. Good sleep hygiene is a huge commitment: it takes up about a third of the day, every day, and works best when kept on a consistent schedule. It does not help that the primary short-term symptoms of insufficient sleep can be self-medicated away with caffeine. However, the effects of sleep loss can range from inconvenient to downright dangerous; people have trouble learning and being productive, take risks more readily, and are more likely to get into accidents. These effects also last longer than it takes to get them, as recovering from each night of poor sleep takes multiple days. When it comes to sleep, every night counts. In this update, we will discuss what Stanford researchers have to say about sleep and why we need it, who is getting too little of it, and some of the latest findings that may help us sleep better.

We have not cracked the code on sleep

Despite this progress, scientists have not been able to crack the code of why sleep is critical to brain function. There is also little consensus about how sleep stages actually affect quality of sleep and how they affect us when we are awake.

Part of the challenge of cracking the code on sleep is how difficult it is to study. The gold standard of sleep study, polysomnography, developed by Dement in the 1960s, 1 is the most reliable tool for measuring many sleep characteristics and detecting sleep disorders such as obstructive sleep apnea and narcolepsy. However, it is expensive and time-consuming to run, which means that usually only a night or two is recorded. This snapshot of sleep may not reflect what normally occurs for a given person, and makes it difficult to draw conclusions about their behavior and performance in the days surrounding the sleep measurement.

The recent explosion in consumer wearable devices is a promising trend for researchers because of their potential to measure thousands of people’s sleep in their natural environments. They have not yet been widely adopted as measurement tools by scientists, however, as it is unclear if they provide the level of precision and measurement consistency required for a scientific study. Researchers at Stanford have called for these devices to be cleared by the FDA before using them to assign a diagnosis. 2 The “holy grail” would be a wearable device that could track sleep accurately while also providing performance information about the rest of the day, which would allow researchers to recognize more nuanced relationships between how people sleep and how it affects their lives.

The short- and long-term effects of insufficient sleep

We all know anecdotally what it is like to get too little sleep; it might be described with words and phrases like “tired,” “cranky,” “sluggish,” and “need caffeine.” Review of the scientific literature reveals how wide-ranging these effects can be. With too little sleep, people have a harder time learning 3 and concentrating, and are more likely to take risks. 4,5 The likelihood of getting into an auto accident increases. 6 Sleep deprivation has a bidirectional relationship with depression, 7,8 in that insomnia often both precedes and follows a depressive episode. Short sleep also interferes with other Healthy Living behaviors: people are more likely to crave sweet and fatty foods 9 and to choose foods that are calorically dense, 10 are more prone to injury during exercise, 11 and have an increased risk of obesity. 12

Sleep deprivation can even affect mundane daily activities. In 2017, then Stanford PhD candidate Tim Althoff and Professor Jamie Zeitzer of the Stanford Center for Sleep Sciences and Medicine took up the sleep measurement challenge by collaborating with Microsoft Research to examine the effects of sleep deprivation through a common daily activity: using an online search engine. 13 They paired users’ Microsoft Band sleep data with their Bing searches among users who had agreed to share their activity for study. By linking quantity and timing of sleep with typing speed during the searches, they were able to draw a number of conclusions about how sleep quality affects performance.

In this study, the researchers captured the sleep duration and search engine interactions of over 31,000 people. The researchers measured the amount of time between keystrokes as people typed their search engine entries, and used this as a measure of daily performance (that is, how well people did after a night of sleep). They were able to track the people who had multiple nights of insufficient sleep (defined as 6 hours of sleep or fewer) to see if their typing speed changed. They found that, on average, one night of insufficient sleep resulted in worse performance for three days, and two nights of insufficient sleep negatively impacted performance for six days. In other words, it took people almost an entire week to recover their performance after two consecutive nights of insufficient sleep. The implication is that the impact of sleep loss can persist for days.

Recent Stanford solutions for better sleep

Ongoing research at Stanford has led both to treatments for sleep disorders and to recommendations for best sleep practices for the public.

There are a few clinics and organizations that offer CBTI remotely in an effort to give more people access. There are apps such as SleepRate , which features content designed by Stanford researchers, Somryst , which was recently approved by the FDA, and Sleepio , which is offered by several large employers as an employee benefit. The Cleveland Sleep Clinic offers a 6-week online program called “ Go! to Sleep ,” and the U.S. Department of Veterans Affairs offers one of the same duration called “ Path to Better Sleep .” A physician should be consulted before starting any of these programs to ensure there are not any underlying disorders that need to be addressed.

Ultrashort light flash therapy Professor Jamie Zeitzer was interested in helping people who had a hard time sleeping because their circadian rhythm was not in sync with their desired sleep schedule. He discovered that ultrashort bursts of light directed into a person’s closed eyes while they were sleeping was very effective at shifting the time a person starts getting sleepy. Sleep doctors had already been using continuous light to help people reset their internal clock while they were awake; this new short-flash method shows great promise not only because of its effectiveness, but because it can be administered passively while people are sleeping. The approach involves wearing a sleep mask that emits the bright flashes and has been shown to only wake individuals who are particularly sensitive to light.

Lumos Sleep Mask

Professor Zeitzer and his team administered these ultrashort light flashes to teenagers, whose natural circadian systems have shifted so that their sleep and wake times are considerably later than children or adults. The time structure of our society, and schools in particular, does not take this into account. Professor Zeitzer administered the light flashes to see if it would help teens go to bed earlier. 20 They found that, while the teenagers were getting sleepy earlier, the light flashes alone were not enough to get the teenagers to bed earlier. With a second group of teens, they combined the light therapy with cognitive behavioral therapy (CBT) sessions. The CBT sessions served to inform the teens about sleep health and hygiene and helped them schedule their activities to allow for their desired sleep hours. After this combined therapy trial, the teens went to bed an average of 50 minutes earlier, getting an average of 43 more minutes of sleep per night. The researchers found the CBT component to be integral to behavior change – without the added education and support, the teens were not motivated enough to change their behavior and would simply push past their sleepiness.

This ultrashort light flash therapy can be used by anyone who may want to shift their sleep schedule; for example, to rebound from jet lag or to cope with a consistent graveyard shift at work. There is no evidence that other groups would require accompanying CBT like the teens, as long as they are self-motivated to change their sleep schedule. Zeitzer plans to test this technology next with older adults who wish to push their sleep time later. A company has spun out of this work, which Zeitzer advises but in which he has no financial interest, called Lumos . They are currently developing their product, and are hoping to make this intervention widely available.

Data Spotlight on: Black Americans

While most Americans have seen improvements in sleep over the past decade, Black Americans continue to sleep significantly less than other groups. This trend has been examined both by researchers and the popular press. 21,22 Researchers have found that Black Americans, in addition to getting shorter sleep, are also more likely to get poor quality sleep – spending less time in the most restorative stages of sleep 23,24 – and to develop obstructive sleep apnea. 25 Black Americans are also disproportionately affected by diseases that have been associated with poor sleep, such as obesity, diabetes, 26 and cardiovascular disease. 25

The exact reason(s) for Black Americans’ poor sleep is still unclear, though researchers have proposed potential contributing factors, largely related to the social inequality Black Americans face in the U.S.:

Experiences of discrimination : the stress of racial discrimination has been associated with spending lesstime in deep sleep and more time in light sleep among Black Americans. 24

Living environment : neighborhood quality has been linked to sleep quality, 27 and Stanford researchersfound that racial and income disparities persist in neighborhoods. 28 They found that while middle-income white families are more likely to live in resource-rich neighborhoods with other middle-income families, middle-income black families tend to live in markedly lower-income, resource-poorneighborhoods.

Work and income inequality : for example, shift work can cause irregular working hours. This leadspeople to suffer “social jetlag,”; a discrepancy in sleep hours between work and free days, 29 leading tosymptoms of sleep deprivation.

Lack of access to resources : particularly sleep-related healthcare and education.

Some of these factors are being addressed directly. Professor Girardin Jean-Louis from New York University and his team have devoted themselves to addressing the access to healthcare and education issue among local black communities in New York by tailoring online materials about obstructive sleep apnea to the culture, language, and barriers of specific communities. 30 Professor Jamie Zeitzer and his team at Stanford recently completed an initial clinical trial of a drug (suvorexant), which was found to help people who work at night get three more hours of sleep during the day. 31 Professor Zeitzer’s ultrashort light flash therapy (discussed above) may also help with shift work. These interventions could help to improve sleep for Black Americans, but they may not make up the whole picture; it could be that the underlying social inequality needs to be addressed in order to fully close the sleep gap.

Thanks to Jamie Zeitzer and Ken Smith for their insights and edits on this report.

• Deak, M., & Epstein, L. J. (2009). The history of polysomnography. Sleep Medicine Clinics , 4 (3), 313–321.
• Cheung, J., Zeitzer, J. M., Lu, H., & Mignot, E. (2018). Validation of minute-to-minute scoring for sleep and wake periods in a consumer wearable device compared to an actigraphy device. Sleep Science and Practice , 2 (1), 11. https://doi.org/10.1186/s41606-018-0029-8
• Gao, C., Terlizzese, T., & Scullin, M. K. (2019). Short sleep and late bedtimes are detrimental to educational learning and knowledge transfer: An investigation of individual differences in susceptibility. Chronobiology International , 36 (3), 307–318. https://doi.org/10.1080/07420528.2018.1539401
• O’Brien, E. M., & Mindell, J. A. (2005). Sleep and risk-taking behavior in adolescents. Behavioral Sleep Medicine , 3 (3), 113–133. https://doi.org/10.1207/s15402010bsm0303_1
• Rusnac, N., Spitzenstetter, F., & Tassi, P. (2019). Chronic sleep loss and risk-taking behavior: Does the origin of sleep loss matter? Behavioral Sleep Medicine , 17 (6), 729–739. https://doi.org/10.1080/15402002.2018.1483368
• Bioulac, S., Micoulaud-Franchi, J.-A., Arnaud, M., Sagaspe, P., Moore, N., Salvo, F., & Philip, P. (2017). Risk of motor vehicle accidents related to sleepiness at the wheel: A systematic review and meta-analysis. Sleep , 40 (10). https://doi.org/10.1093/sleep/zsx134
• Franzen, P. L., & Buysse, D. J. (2008). Sleep disturbances and depression: Risk relationships for subsequent depression and therapeutic implications. Dialogues in Clinical Neuroscience , 10 (4), 473–481.
• Tsuno, N., & Ritchie, K. (2005). Sleep and Depression. The Journal of Clinical Psychiatry , 16.
• Lv, W., Finlayson, G., & Dando, R. (2018). Sleep, food cravings and taste. Appetite , 125 , 210–216. https://doi.org/10.1016/j.appet.2018.02.013
• Pardi, D., Buman, M., Black, J., Lammers, G. J., & Zeitzer, J. M. (2017). Eating decisions based on alertness levels after a single night of sleep manipulation: A randomized clinical trial. Sleep , 40 (2). https://doi.org/10.1093/sleep/zsw039
• Chennaoui, M., Arnal, P. J., Sauvet, F., & Léger, D. (2015). Sleep and exercise: A reciprocal issue? Sleep Medicine Reviews , 20 , 59–72. https://doi.org/10.1016/j.smrv.2014.06.008
• Cappuccio, F. P., Taggart, F. M., Kandala, N. B., Currie, A., Peile, E., Stranges, S., & Miller, M. A. (2008). Meta-analysis of short sleep duration and obesity in children and adults.  Sleep ,  31 (5), 619-626.
• Althoff, T., Horvitz, E., White, R. W., & Zeitzer, J. (2017, April). Harnessing the web for population-scale physiological sensing: A case study of sleep and performance. In  Proceedings of the 26th international conference on World Wide Web (pp. 113-122).
• Roth, T. (2007). Insomnia: Definition, prevalence, etiology, and consequences. Journal of Clinical Sleep Medicine , 3 (5 Suppl), S7–S10.
• Qaseem, A., Kansagara, D., Forciea, M. A., Cooke, M., & Denberg, T. D. (2016). Management of chronic insomnia disorder in adults: A clinical practice guideline from the American College of Physicians. Annals of Internal Medicine , 165 (2), 125–133. https://doi.org/10.7326/M15-2175
• Jacobs, G. D., Pace-Schott, E. F., Stickgold, R., & Otto, M. W. (2004). Cognitive behavior therapy and pharmacotherapy for insomnia: A randomized controlled trial and direct comparison. Archives of Internal Medicine , 164 (17), 1888–1896. https://doi.org/10.1001/archinte.164.17.1888
• Manber, R., Bei, B., Simpson, N., Asarnow, L., Rangel, E., Sit, A., & Lyell, D. (2019). Cognitive behavioral therapy for prenatal insomnia: A randomized controlled trial. Obstetrics & Gynecology , 133 (5), 911–919. https://doi.org/10.1097/AOG.0000000000003216
• Ong, J. C., Crawford, M. R., Dawson, S. C., Fogg, L. F., Turner, A. D., Wyatt, J. K., Crisostomo, M. I., Chhangani, B. S., Kushida, C. A., Edinger, J. D., Abbott, S. M., Malkani, R. G., Attarian, H. P., & Zee, P. C. (2020). A randomized controlled trial of CBT-I and PAP for obstructive sleep apnea and comorbid insomnia: Main outcomes from the MATRICS study. Sleep . https://doi.org/10.1093/sleep/zsaa041
• Karlin, B. E., Trockel, M., Taylor, C. B., Gimeno, J., & Manber, R. (20130415). National dissemination of cognitive behavioral therapy for insomnia in veterans: Therapist- and patient-level outcomes. Journal of Consulting and Clinical Psychology , 81 (5), 912. https://doi.org/10.1037/a0032554
• Kaplan, K. A., Mashash, M., Williams, R., Batchelder, H., Starr-Glass, L., & Zeitzer, J. M. (2019). Effect of light flashes vs sham therapy during sleep with adjunct cognitive behavioral therapy on sleep quality among adolescents: A randomized clinical trial. JAMA Network Open , 2 (9), e1911944. https://doi.org/10.1001/jamanetworkopen.2019.11944
• Resnick, B. (2015, October 27). The Racial Inequality of Sleep . The Atlantic. https://www.theatlantic.com/health/archive/2015/10/the-sleep-gap-and-racial-inequality/412405/
• Resnick, B., & Barton, G. (2018, April 12). Black Americans don’t sleep as well as white Americans. That’s a problem. Vox. https://www.vox.com/science-and-health/2018/4/12/17224328/sleep-gap-black-white-minority-america-health-consequences
• Beatty, D. L., Hall, M. H., Kamarck, T. A., Buysse, D. J., Owens, J. F., Reis, S. E., Mezick, E. J., Strollo, P. J., & Matthews, K. A. (2011). Unfair treatment is associated with poor sleep in African American and Caucasian adults: Pittsburgh SleepSCORE project. Health Psychology , 30 (3), 351–359. https://doi.org/10.1037/a0022976
• Tomfohr, L., Pung, M. A., Edwards, K. M., & Dimsdale, J. E. (2012). Racial differences in sleep architecture: The role of ethnic discrimination. Biological Psychology , 89 (1), 34–38. https://doi.org/10.1016/j.biopsycho.2011.09.002
• Olafiranye, O., Akinboboye, O., Mitchell, J., Ogedegbe, G., & Jean-Louis, G. (2013). Obstructive sleep apnea and cardiovascular disease in blacks: A call to action from association of black cardiologists. American Heart Journal , 165 (4), 468–476. https://doi.org/10.1016/j.ahj.2012.12.018
• Jackson, C. L., Redline, S., Kawachi, I., & Hu, F. B. (2013). Association between sleep duration and diabetes in black and white adults. Diabetes Care , 36 (11), 3557–3565. https://doi.org/10.2337/dc13-0777
• Hale, L., Hill, T. D., & Burdette, A. M. (2010). Does sleep quality mediate the association between neighborhood disorder and self-rated physical health? Preventive Medicine , 51 (3–4), 275–278. https://doi.org/10.1016/j.ypmed.2010.06.017
• Reardon, S. F., Fox, L., & Townsend, J. (2015). Neighborhood income composition by household race and income, 1990–2009. The ANNALS of the American Academy of Political and Social Science , 660 (1), 78–97. https://doi.org/10.1177/0002716215576104
• Wittmann, M., Dinich, J., Merrow, M., & Roenneberg, T. (2006). Social jetlag: Misalignment of biological and social time. Chronobiology International , 23 (1–2), 497–509. https://doi.org/10.1080/07420520500545979
• Jean-Louis, G., Robbins, R., Williams, N. J., Allegrante, J. P., Rapoport, D. M., Cohall, A., & Ogedegbe, G. (2020). Tailored Approach to Sleep Health Education (TASHE): A randomized controlled trial of a web-based application. Journal of Clinical Sleep Medicine . https://doi.org/10.5664/jcsm.8510
• Zeitzer, J. M., Joyce, D. S., McBean, A., Quevedo, Y. L., Hernandez, B., & Holty, J.-E. (2020). Effect of suvorexant vs placebo on total daytime sleep hours in shift workers: A randomized clinical trial. JAMA Network Open , 3 (6), e206614. https://doi.org/10.1001/jamanetworkopen.2020.6614

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

Important advances in sleep research in 2021

Leslie c west.

a Division of Sleep Medicine, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Palo Alto, CA 94305, USA

Clete A Kushida

Advances in sleep research in 2021 have brought about clinical developments for the next decade. Additionally, sleep telemedicine services have expanded rapidly, driven by the COVID-19 pandemic, to best serve patients with sleep disorders. 1 Here, we will explore some of the most impactful clinical studies from this field in 2021.

Progress has been made in evaluating the relationship between obstructive sleep apnoea and incidence of Alzheimer's disease. In a large cohort study of 53 321 patients with obstructive sleep apnoea, patients treated with continuous positive airway pressure (CPAP) were compared with those who were not. In adjusted models, those who were treated had a significantly lower odds of incident diagnosis of Alzheimer's disease or dementia (odds ratio 0·78, 95% CI 0·69–0·89). 2 This finding adds population-based evidence to suggest a temporal association between obstructive sleep apnoea treatment and dementia risk. In a study that analysed autopsy brains from 34 Icelandic patients with clinically verified obstructive sleep apnoea, Owen and colleagues 3 found that, as the severity of obstructive sleep apnoea increased, the burden of amyloid β in the hippocampus increased too, even after controlling for age, sex, body-mass index, and CPAP use. These findings, although in a small sample size, provide further neuropathological evidence for an association between Alzheimer's disease and obstructive sleep apnoea.

Throughout the COVID-19 pandemic, there have been reports of more frequent sleep disturbances. Morin and colleagues 4 conducted a multicentre survey of 22 330 adults from 13 countries and found that more than 35% of respondents reported insomnia symptoms, with about 17% meeting criteria for a probable insomnia disorder during the first months of the pandemic. These data provide an indication of the need for programmes to tackle insomnia during a global crisis. Arnedt and colleagues 5 reported findings of a placebo-controlled randomised trial of 65 adults assigned to telemedicine or in-person delivery of cognitive behavioural therapy for insomnia. They showed that cognitive behaviour therapy delivered via telemedicine was non-inferior to therapy done in-person, according to the reduction in insomnia severity index score.

Dual orexin receptor antagonists have been evaluated in clinical research as an additional treatment option for insomnia. 12-month results from the phase 3 analysis of the Study of the Efficacy and Safety of Lemborexant in Subjects 55 and Older with Insomnia Disorder (SUNRISE 2) were published in 2021. 6 This international, multicentre, randomised, double-blind, placebo-controlled, parallel-group study compared two doses of lemborexant (5 mg and 10 mg) with placebo, and enrolled 949 participants, including 318 who received placebo. Patients randomly assigned to treatment showed a benefit, with shorter sleep onset latency, increased subjective total sleep time, and decreased subjective wake after sleep onset, at 12-month follow-up.

Dauvilliers and colleagues 7 conducted a phase 3 randomised controlled trial of lower-sodium oxybate for the treatment of patients with idiopathic hypersomnia. They found a significant change in the Epworth Sleepiness Scale compared with placebo (difference of –6·5, 95% CI –8·0 to –5·0), indicating a clinically meaningful effect on excessive daytime sleepiness. Overall idiopathic hypersomnia symptom severity showed significant improvement in patients randomly assigned to lower-sodium oxybate compared with placebo.

Gaspar and colleagues 8 examined the effect of obstructive sleep apnoea on disruption of circadian rhythms. Their study was the first to compare the effect of CPAP in the short term (4 months) and long term (2 years) on circadian characteristics in a case-control study of 34 male patients with obstructive sleep apnoea and seven age-matched and sex-matched healthy controls. The investigators found that long-term, but not short-term, CPAP treatment was able to re-establish levels of major clock circadian outputs, including plasma melatonin, cortisol, and body temperature. The authors found that CPAP does not fully re-establish the expression profile of clock genes, but leads to evident improvements.

Regarding public health policy, further evidence was published from the Changing Start Times: Longitudinal Effects Study (CaSTLES) evaluating school start times at or after 0830 h for middle-school and high-school students. 9 The findings showed that, when start times were 40–70 min later, middle-school students obtained an extra 2·4 h of sleep per week and high-school students an extra 3·8 h of sleep per week, with maintenance of this effect over 2 years.

In basic sleep science research, Konkoly and colleagues 10 provided proof-of-concept evidence that individuals can be interviewed about their dreams while they are dreaming. In four independent laboratories in France, Germany, the Netherlands, and the USA, healthy participants and one patient with narcolepsy provided evidence for real-time communication during rapid eye movement (REM) sleep. This example of bidirectional communication during REM sleep showed the use of working memory, with yet-to-be explored potential clinical applications for cognitive improvement, learning, and beyond.

In conclusion, the advances during this year have included a broad landscape of investigations, while researchers and clinical teams adapted to the challenge of conducting studies during a pandemic, and provided far-reaching new insights for sleep medicine.

CAK reports personal fees related to consultancies or participation in medical advisory boards from XW Pharma, Idorsia, Merck, Jazz Pharmaceuticals, Avadel Pharmaceuticals, Cerebra, and VIVOS; grants or contracts with the National Institutes of Health, Avadel Pharmaceuticals, AmCad BioMed Corporation, Jazz Pharmaceuticals, Merck Sharp & Dohme, Parexel International Corporation, Asate, Inspire Medical Systems, and Respironics; lecture honoraria from ResMed Asia and Itamar Medical; and stock or stock options from M3 Public Benefit Corporation, Restful Robotics, VIVOS, and Koko. LCW declares no competing interests.

Research uncovers differences between men and women in sleep, circadian rhythms and metabolism

A new review of research evidence has explored the key differences in how women and men sleep, variations in their body clocks, and how this affects their metabolism.

Published in Sleep Medicine Reviews , the paper highlights the crucial role sex plays in understanding these factors and suggests a person's biological sex should be considered when treating sleep, circadian rhythm and metabolic disorders.

Differences in sleep

The review found women rate their sleep quality lower than men's and report more fluctuations in their quality of sleep, corresponding to changes throughout the menstrual cycle.

"Lower sleep quality is associated with anxiety and depressive disorders, which are twice as common in women as in men," says Dr Sarah L. Chellappa from the University of Southampton and senior author of the paper. "Women are also more likely than men to be diagnosed with insomnia, although the reasons are not entirely clear. Recognising and comprehending sex differences in sleep and circadian rhythms is essential for tailoring approaches and treatment strategies for sleep disorders and associated mental health conditions."

The paper's authors also found women have a 25 to 50 per cent higher likelihood of developing restless legs syndrome and are up to four times as likely to develop sleep-related eating disorder, where people eat repeatedly during the night.

Meanwhile, men are three times more likely to be diagnosed with obstructive sleep apnoea (OSA). OSA manifests differently in women and men, which might explain this disparity. OSA is associated with a heightened risk of heart failure in women, but not men.

Sleep lab studies found women sleep more than men, spending around 8 minutes longer in non-REM (Rapid Eye Movement) sleep, where brain activity slows down. While the time we spend in NREM declines with age, this decline is more substantial in older men. Women also entered REM sleep, characterised by high levels of brain activity and vivid dreaming, earlier than men.

Variations in body clocks

The team of all women researchers from the University of Southampton in the UK, and Stanford University and Harvard University in the United States, found differences between the sexes are also present in our circadian rhythms.

They found melatonin, a hormone that helps with the timing of circadian rhythms and sleep, is secreted earlier in women than men. Core body temperature, which is at its highest before sleep and its lowest a few hours before waking, follows a similar pattern, reaching its peak earlier in women than in men.

Corresponding to these findings, other studies suggest women's intrinsic circadian periods are shorter than men's by around six minutes.

Dr Renske Lok from Stanford University, who led the review, says: "While this difference may be small, it is significant. The misalignment between the central body clock and the sleep/wake cycle is approximately five times larger in women than in men. Imagine if someone's watch was consistently running six minutes faster or slower. Over the course of days, weeks, and months, this difference can lead to a noticeable misalignment between the internal clock and external cues, such as light and darkness.

"Disruptions in circadian rhythms have been linked to various health problems, including sleep disorders, mood disorders and impaired cognitive function. Even minor differences in circadian periods can have significant implications for overall health and well-being."

Men tend to be later chronotypes, preferring to go to bed and wake up later than women. This may lead to social jet lag, where their circadian rhythm doesn't align with social demands, like work. They also have less consistent rest-activity schedules than women on a day-to-day basis.

Impact on metabolism

The research team also investigated if the global increase in obesity might be partially related to people not getting enough sleep -- with 30 per cent of 30- to 64-year-olds sleeping less than six hours a night in the United States, with similar numbers in Europe.

There were big differences between how women's and men's brains responded to pictures of food after sleep deprivation. Brain networks associated with cognitive (decision making) and affective (emotional) processes were twice as active in women than in men. Another study found women had a 1.5 times higher activation in the limbic region (involved in emotion processing, memory formation, and behavioural regulation) in response to images of sweet food compared to men.

Despite this difference in brain activity, men tend to overeat more than women in response to sleep loss. Another study found more fragmented sleep, taking longer to get to sleep, and spending more time in bed trying to get to sleep were only associated with more hunger in men.

Both women and men nightshift workers are more likely to develop type 2 diabetes, but this risk is higher in men. Sixty-six per cent of women nightshift workers experienced emotional eating and another study suggests they are around 1.5 times more likely to be overweight or obese compared to women working day shifts.

The researchers also found emerging evidence on how women and men respond differently to treatments for sleep and circadian disorders. For example, weight loss was more successful in treating women with OSA than men, while women prescribed zolpidem (an insomnia medication) may require a lower dosage than men to avoid lingering sleepiness the next morning.

Dr Chellappa added: "Most of sleep and circadian interventions are a newly emerging field with limited research on sex differences. As we understand more about how women and men sleep, differences in their circadian rhythms and how these affect their metabolism, we can move towards more precise and personalised healthcare which enhances the likelihood of positive outcomes."

The research was funded by the Alexander Von Humboldt Foundation, the US Department of Defense and the National Institute of Health.

• Sleep Disorder Research
• Insomnia Research
• Gender Difference
• Sleep Disorders
• Obstructive Sleep Apnea
• Circadian rhythm sleep disorder
• Glutamic acid
• Sleep deprivation

Story Source:

Materials provided by University of Southampton . Note: Content may be edited for style and length.

Journal Reference :

• Renske Lok, Jingyi Qian, Sarah L. Chellappa. Sex differences in sleep, circadian rhythms, and metabolism: Implications for precision medicine . Sleep Medicine Reviews , 2024; 75: 101926 DOI: 10.1016/j.smrv.2024.101926

Cite This Page :

Explore More

• Largest Known Marine Reptile
• Neolithic Humans Lived in Lava Tube Caves
• Imminent Carbon Release from the Tundra
• How Working Memory Reallly Works
• Substantial Global Cost of Climate Inaction
• Paradox of Extreme Cold Events in Warming World
• Plastic Pollution Kills Ocean Embryos
• Most Massive Stellar Black Hole in Our Galaxy
• Coffee's Prehistoric Origin and It's Future
• Can Animals Count? New Rat Study

Trending Topics

Strange & offbeat.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

This is a member publication of National University of Singapore

Comparing Open-Access Database and Traditional Intensive Care Studies Using Machine Learning: Bibliometric Analysis Study

Authors of this article:

Original Paper

• Yuhe Ke 1 * , MBBS   ; 
• Rui Yang 2 * , MSc   ; 
• Nan Liu 2 , PhD  

1 Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore

2 Centre for Quantitative Medicine, Duke-NUS Medical School, National University of Singapore, Singapore, Singapore

*these authors contributed equally

Corresponding Author:

Nan Liu, PhD

Centre for Quantitative Medicine

Duke-NUS Medical School

National University of Singapore

8 College Road

Singapore, 169857

Phone: 65 66016503

Email: [email protected]

Background: Intensive care research has predominantly relied on conventional methods like randomized controlled trials. However, the increasing popularity of open-access, free databases in the past decade has opened new avenues for research, offering fresh insights. Leveraging machine learning (ML) techniques enables the analysis of trends in a vast number of studies.

Objective: This study aims to conduct a comprehensive bibliometric analysis using ML to compare trends and research topics in traditional intensive care unit (ICU) studies and those done with open-access databases (OADs).

Methods: We used ML for the analysis of publications in the Web of Science database in this study. Articles were categorized into “OAD” and “traditional intensive care” (TIC) studies. OAD studies were included in the Medical Information Mart for Intensive Care (MIMIC), eICU Collaborative Research Database (eICU-CRD), Amsterdam University Medical Centers Database (AmsterdamUMCdb), High Time Resolution ICU Dataset (HiRID), and Pediatric Intensive Care database. TIC studies included all other intensive care studies. Uniform manifold approximation and projection was used to visualize the corpus distribution. The BERTopic technique was used to generate 30 topic-unique identification numbers and to categorize topics into 22 topic families.

Results: A total of 227,893 records were extracted. After exclusions, 145,426 articles were identified as TIC and 1301 articles as OAD studies. TIC studies experienced exponential growth over the last 2 decades, culminating in a peak of 16,378 articles in 2021, while OAD studies demonstrated a consistent upsurge since 2018. Sepsis, ventilation-related research, and pediatric intensive care were the most frequently discussed topics. TIC studies exhibited broader coverage than OAD studies, suggesting a more extensive research scope.

Conclusions: This study analyzed ICU research, providing valuable insights from a large number of publications. OAD studies complement TIC studies, focusing on predictive modeling, while TIC studies capture essential qualitative information. Integrating both approaches in a complementary manner is the future direction for ICU research. Additionally, natural language processing techniques offer a transformative alternative for literature review and bibliometric analysis.

Introduction

The start of critical care as a medical subspecialty can be traced back to a polio epidemic during which a substantial number of patients needed prolonged mechanical ventilation [ 1 ]. Over time, the field of critical care has experienced significant growth and continual evolution. Research in this field has played a pivotal role in unraveling the complexities of numerous diseases and treatment modalities, driving substantial advancements in clinical practice over the past decades [ 2 ]. Groundbreaking studies have investigated critical areas such as sepsis, mechanical ventilation, acute lung and kidney injuries, intensive care unit (ICU) delirium, and sedation in critically ill patients [ 3 ].

These research studies have often been conducted in traditional ways such as prospective and randomized controlled trials [ 4 ], cohort and observational studies, clinical trials [ 5 ], and clinical and translational research [ 6 ]. These traditional methods have revolutionized patient care and improved outcomes significantly. For instance, the implementation of protocol-driven, goal-directed management of sepsis and appropriate fluid therapy has led to remarkable reductions in mortality rates [ 7 , 8 ], and these findings have been integral in developing evidence-based practice guidelines that are now the gold standard [ 9 , 10 ].

Despite their undeniable merits, traditional research methods in intensive care also come with several limitations [ 11 ]. Clinical trials are known for their high costs [ 12 ], stringent standardization requirements, and ethical oversight [ 13 ]. Data collection can be laborious, prone to human errors, and constrained in terms of quantity and granularity [ 14 ]. Moreover, obtaining patient consent for most randomized controlled trials in the ICU poses challenges [ 15 ], necessitating alternative consent models. These limitations have become increasingly apparent as medical complexity continues to grow exponentially [ 16 ].

The advent of electronic health records (EHRs) has heralded a new era in clinical research by facilitating the digitization of health care systems [ 17 ]. In this era of data science, a more integrated approach can be adopted, using machine learning (ML) algorithms to tackle the complexity of critical illness [ 18 ]. Open-access databases (OADs), such as the Medical Information Mart for Intensive Care (MIMIC) database [ 19 ] and the Philips eICU Collaborative Research Database (eICU-CRD) [ 20 ], have played a transformative role by enabling free data sharing.

The concept of free and open databases plays a pivotal role in promoting data sharing and advancing medical knowledge in accordance with the findable, accessible, interoperable, and reusable (FAIR) guiding principle. The FAIR principles, which emphasize that data should be findable, accessible, interoperable, and reusable, are essential for fostering a collaborative and transparent scientific research environment [ 21 , 22 ]. By removing barriers to access, free, and open databases allow researchers, regardless of their affiliations or resources, to contribute to and benefit from the collective pool of information. Accessibility fosters inclusivity and diversity in research, promoting a broader range of perspectives and approaches to medical challenges. This democratization of knowledge leads to a more equitable distribution of information. Researchers can now leverage these vast repositories of information for ML and artificial intelligence studies, marking a departure from traditional intensive care (TIC) research approaches.

Conducting a literature review [ 23 ] to investigate the disparities between traditional ICU research and studies based on open-access data sets holds significant importance as it provides a comprehensive understanding of the strengths and limitations of the latter. However, conventional methods of literature reviews and bibliometric analysis have their limitations, especially when dealing with large-scale literature due to computational complexity and the labor-intensive nature of manual interpretations [ 24 - 26 ]. To address these challenges, natural language processing (NLP) offers a promising avenue, while topic modeling techniques can be used to extract various topic themes from extensive data sets [ 27 , 28 ].

Built on the foundations of bidirectional encoder representations from transformers (BERT), BERTopic introduces a novel approach to topic modeling [ 29 , 30 ]. Unlike traditional unsupervised models like latent Dirichlet allocation, which rely on “bag-of-words” model [ 31 ], BERTopic overcomes the problem of semantic information loss, significantly enhancing the accuracy of generated topics, and providing more interpretable compositions for each topic, which greatly facilitates the classification of topics.

With the aid of BERTopic, this study aims to shed light on the disparities and commonalities between studies conducted through OADs and TIC research. By analyzing the overall trends and patterns in these 2 groups, we seek to identify knowledge gaps and explore avenues for complementary contributions between these research approaches.

Data Filtering

We performed an ML-based analysis of research abstracts in the Web of Science (WoS) database to automatically categorize the research papers to conduct this literature mapping analysis. There was no limit to the year of publication of the articles. The search query consisted of the following keyword to identify all the studies that were published under the umbrella of intensive care: (“ICU” OR “intensive care”). The search terms were deliberately left to be broad to cover broad spectrums of journals in the field.

The inclusion criteria were as follows: (1) written in English, (2) articles that had keywords related to intensive care, (3) articles that had the article type of “article” or “review.” We excluded articles with incomplete data fields (eg, title, abstract, publication year, and paper citation). The articles included were then further processed to identify if they were studies using OADs. These articles were labeled as “open-access database,” while the rest of the articles extracted were labeled as “traditional intensive care.”

The search used for this study was performed on January 18, 2023, from WoS. This generated 227,893 search results, which were subsequently reselected using Python. An advanced search from PubMed was done based on the broad search terms of ICU studies used from previous Cochrane ICU literature review [ 32 ] to ensure the accuracy of the results. The numbers corroborated with a discrepancy of 4.9% (227,893 WoS keyword search vs 239,748 PubMed ICU keyword search).

Selection Criteria for OADs

A title search using keywords from all currently existing OADs was conducted to identify OAD studies. These include (1) MIMIC [ 19 ], (2) eICU-CRD [ 20 ], (3) Amsterdam University Medical Centers Database (AmsterdamUMCdb) [ 33 ], (4) High Time Resolution ICU Dataset (HiRID) [ 34 ], and (5) Pediatric Intensive Care database [ 35 ]. We avoided including only keywords in the search and restricted the search years by the year that the OAD was made publicly available to reduce the inadvertent inclusion of incorrect articles due to keywords. For instance, the search term for OADs published with the MIMIC database included title keyword search with the terms (“MIMIC-IV” OR “MIMIC-III” OR “MIMIC-II” OR “MIMIC Dataset” OR “medical information mart for intensive care” OR “MIMIC IV” OR “MIMIC III” OR “MIMIC II”) in studies that were published after 2003. The title keyword search for the searches and the year of cutoff for each OAD are presented in Multimedia Appendix 1 .

Furthermore, to ensure the accuracy of the supervised keyword classification, a manual review of the classification by 2 critical care physicians was done for 100 articles from each category that were randomly selected. The review was done independently with the physicians labeling the extract publications into OAD and TICs. An accuracy of 99% was achieved on independent reviews, and full agreement was achieved after discussion on the discrepancy. The final results were matched with the supervised keyword classification.

We performed a bibliometric analysis by directly extracting publication details from the WoS database using Python (Python Software Foundation). The analysis involved assessing the number of articles published per year, calculating total citation counts, and identifying the top journals that published intensive care-related articles. Comprehensive results are presented in Multimedia Appendix 2 .

Data Analysis

Uniform manifold approximation and projection.

Uniform manifold approximation and projection (UMAP) is a manifold learning technique for dimension reduction, which can identify key structures in high-dimensional data space and map them to low-dimensional space to accomplish dimensionality reduction. Compared to other dimensionality reduction algorithms, such as principal component analysis [ 36 ], UMAP can retain more global features [ 37 ]. In this paper, we constructed a corpus consisting of abstract words from all studies. However, due to the massive size of the corpus, visualizing and analyzing the high-dimensional data to explore the differences in the vocabulary patterns between the OAD and TIC studies is a challenge. The UMAP package in Python, which implements the UMAP algorithm, was used to project the high-dimensional corpus to 4 dimensions. By cross plotting each dimension, we were able to investigate underlying differences in corpus distribution between OAD and TIC studies.

Topic modeling can help us explore the similarities and differences between research topics in OAD and TIC studies. Unlike conventional topic modeling models, BERTopic uses the BERT framework for embeddings, enabling a deeper understanding of semantic relationships [ 30 ]. The BERTopic model was implemented by the BERTopic package in Python and divided 146,727 studies into 30 topic IDs. We also performed latent Dirichlet allocation topic modeling through Python’s LdaModel package for comparison. Through the review of topic keywords by 2 critical care physicians, BERTopic exhibited superior accuracy and sophistication in topic identification, with enhanced interpretability and scientific rigor.

Consequently, the BERTopic model was used for the final analysis. Each of these topics was given a corresponding clinical research category. The overlapping categories were merged into topic families for easier comparisons. By using these advanced techniques, we were able to uncover hidden patterns and relationships within the literature and provide insights into the current state of intensive care research.

A total of 227,893 records were identified from the WoS database on January 18, 2023, of which 195,463 full records were subsequently processed. Records were excluded if they are not “article” or “review” or if they do not contain keywords related to intensive care. After exclusions, 145,426 articles were identified as TIC studies and 1301 articles were categorized as OAD ( Figure 1 ).

We examined the number of articles published per year to analyze the trends in TIC and OAD studies ( Figure 2 ). Over the past 2 decades, TIC studies have experienced exponential growth, culminating in a peak of 16,378 articles in 2021. A subsequent decline in the number of publications occurred in 2022, likely attributable to delayed indexing within the WoS database and a reduction in COVID-19–related studies as the pandemic stabilized [ 38 ]. In contrast, the first OAD study emerged in 2003, with its popularity experiencing a consistent upsurge since 2018. Nonetheless, the number of OAD publications remains markedly lower in comparison to TIC publications.

The OAD studies were published most frequently in new open-access journals such as Frontiers in Medicine , Frontiers in Cardiovascular Medicine , and Scientific Reports while the TIC studies were published most frequently in established journals like Critical Care Medicine , Intensive Care Medicine , and Critical Care ( Multimedia Appendix 2 ). Further analysis of keywords from the abstracts showed 2.4% (3492/145,426) TIC studies were meta-analyses or systematic reviews, while only 0.08% (1/1301) OAD study was in this category. There were 5.61% (73/1301) OAD studies, and 7.43% (10,799/145,426) TIC studies that had the keyword of “cost.” Examples of the data fields that are available within OADs such as MIMIC and eICU-CRD are listed in Textbox 1 . Some information fields such as end-of-life goals and values and health care provider psychology are not available within the current EHRs extracted for OADs.

Examples of information that is available in current OADs

• Patient information: demographics and social set-up
• Hospital context: admission time and discharge time, intensive care unit (ICU) and hospital admissions, and pre-ICU admission
• Diagnosis: physician-curated ICU diagnosis and data-driven phenotypes
• Intervention: medications, procedures, and organ support
• Diagnostics: blood test, microbiology, and scans
• Clinical texts: clinical notes and diagnostic reports
• Physiological monitoring: basic monitoring and waveforms

Examples of information that is not readily available in current OADs

• Patient information: family set up and visiting, financial information, and special populations
• Hospital context: post-ICU discharge details, delayed admission or discharge, and health personnel psychology
• Diagnosis: pre-ICU history and diagnosis requiring clinical symptoms
• Intervention: indications for interventions, complications, and intraoperative and postoperative
• Diagnostics: pathology photographs, imaging, and molecular or genetic studies
• Clinical texts: patient narratives, end-of-life goals and patient value, and health personnel behavior
• Physiological monitoring: advanced monitoring

The UMAP algorithm was used to project the high-dimension corpus to 4 dimensions and allowed exploration of the vocabulary patterns between the OAD and TIC studies ( Figure 3 ). The projection values are represented by the x-axis, while the densities are represented by the y-axis. The presence of considerable overlap between TIC studies and OAD studies suggests that they share a substantial number of common terminologies, which may correspond to similar research topics. Nonetheless, TIC studies exhibit a more extensive coverage than OAD studies, which may stem from broader research scope and extended research duration.

Subsequently, the BERTopic model was then used to generate 30 topic IDs ( Figure 4 ). The internal commonalities of each topic ID were reviewed by critical care physicians and assigned a specific subtopic in intensive care research. The model was able to automatically classify the topics with high interpretability and the topic components were interpreted with ease. For instance, components in topic ID 5 consist of, in decreasing order of weightage: “learning,” “model,” “machine,” “machine learning,” “models,” “data,” “prediction,” and “performance.” This topic was consequently labeled “predictive model” (topic ID 5 in Multimedia Appendix 3 ).

The overall topic distribution in TIC studies was more uniform, while the OAD studies tended to be concentrated on several topics including topic ID 2 (kidney injury), 5 (predictive model), and 13 (sepsis). Some topics that were missing in OAD studies included 6 (pediatrics care), 21 (viral infections), 23 (health personnel and psychology), and 28 (nutrition and rehabilitation).

The similarity matrix shows that there was little overlap between the topics ( Multimedia Appendix 4 ). To facilitate the interpretability of the categories, the overlapping topic IDs were merged to form the final 22 topic families ( Multimedia Appendix 3 ).

Topics such as “healthcare associated infection,” “thoracic surgeries,” and “pregnancy related” research were among the more frequently discussed 15 topics in TIC studies but have limited publications in OAD studies. The topics of “predictive model,” “obesity,” and “fungal infections” were popular in OAD studies but not the TIC studies. Overall, the topic distributions of the TIC studies were distributed more evenly with the topic family of sepsis accounting for a quarter of the studies, while publications in the OAD studies were heavily skewed toward the predictive model (>40%) and sepsis (>30%; Figure 5 ).

Principal Results

This study conducted a comprehensive review and bibliometric analysis of OAD and TIC studies. NLP was used to facilitate this large-scale literature review. Studies using OADs mainly concentrated on a few topics, such as predictive modeling, while TIC studies covered a wider range of topics with a more balanced distribution.

Advantages of OAD Studies

OAD studies offer several advantages that have contributed to their increasing popularity in intensive care research. The granularity of data and easy access to large-cohort databases, such as MIMIC [ 39 ], has enabled researchers to perform predictive modeling and conduct various secondary analyses efficiently [ 40 , 41 ]. This accessibility has provided valuable opportunities for exploring specific aspects of patient care, evident in studies investigating phenomena like “weekend effects” and circadian rhythms in ICU patients before discharge [ 42 - 46 ]. The vast amount of longitudinal and time series data available in OADs has also facilitated the implementation of complex ML and deep learning methods [ 47 ].

Limitations of OAD Studies

However, it is crucial to acknowledge the retrospective nature of OAD data, which inherently limits the assessment of confounding factors and the ability to draw strong causal conclusions. The observational design of OAD studies may result in lower-quality evidence according to the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) framework [ 48 , 49 ], and thus, the research from OAD studies has yet to be fully integrated into existing evidence-based guidelines, as exemplified by the omission of OAD studies in the 2021 sepsis guidelines [ 50 ]. Nevertheless, OADs remain a valuable resource for supplementing and complementing TIC studies, providing unique insights and enhanced predictive scores for intensive care settings.

Furthermore, approximately 50% of the studies using OADs published focused on predictive modeling. The increased usage of ML methods in predictive modeling has not been without critique. Some medical prediction problems inherently possess linear characteristics, and the selection of features may predominantly focus on already known strong predictors, leading to limited improvements in prediction accuracy with ML [ 51 ]. Additionally, interstudy heterogeneity poses a challenge in comparing results obtained from different ML models applied to the same data sets [ 52 ]. The ethical implications of relying solely on ML models to make high-risk health care decisions instead of involving clinical expertise are also relevant considerations [ 51 , 53 ].

While OADs provide comprehensive patient data, there are certain limitations in their ability to capture specific information essential for certain critical care research areas. Notably, data fields related to qualitative aspects such as ethics and end-of-life care [ 54 , 55 ], and health care personnel psychology [ 56 ] may be challenging, if not impossible, to obtain through OADs generated from EHRs. Consequently, TIC studies have played a pivotal role in addressing these limitations by capturing critical information that is integral to understanding ethical considerations, patient experiences, and health care provider psychology in intensive care [ 57 , 58 ].

Synergy Between OAD and TIC Studies

The synergy between OAD and TIC studies is a promising approach to enhance the comprehensiveness and robustness of intensive care research. OADs, with their large cohort sizes, can serve as external validation cohorts for ML models developed from TIC studies, potentially reducing the sample sizes required for prospective research. Furthermore, OAD studies can corroborate the results of TIC studies, benefiting from larger sample sizes and real-world data, thus providing more practical insights for implementation in intensive care settings [ 43 ]. The integration of OAD and TIC studies presents an opportunity to bridge the gaps in data availability and research methodologies, ultimately enriching the understanding and practice of critical care medicine.

Potential Impact of NLP

The usage of large language models such as BERTopic has proven to be a valuable tool for large-scale literature review and topic extraction [ 58 ]. This approach has enabled accurate, reliable, and granular topic generation, offering clinicians a more effective means of interpreting data compared to traditional bag-of-words models [ 59 ]. The potential of NLP to analyze scientific articles and identify trends and knowledge gaps holds promise for shaping the future of research in critical care medicine. As the volume of publications in critical care continues to grow and large language modeling continues to advance in health care [ 60 ], AI technology will be crucial in efficiently identifying and predicting emerging trends.

Future Directions

Future research in the field of critical care can explore novel applications of ML beyond predictive modeling. For instance, using ML to study patterns in how papers are cited, shared, and discussed on the web could help predict their potential impact on the scientific community. This analysis can aid in identifying highly influential papers and understanding the factors that contribute to their recognition. Additionally, investigations into methods for enhancing the interpretability and transparency of ML algorithms in critical care research would further facilitate the ethical and responsible use of AI technologies.

Strengths and Limitations

The study’s application of NLP-driven in analyzing scientific articles and identifying trends highlights the potential impact of AI technologies in streamlining literature reviews and identifying emerging trends more efficiently.

Another notable strength of this study is the usage of the WoS database, the world’s oldest and most extensively used repository of research publications and citations, encompassing approximately 34,000 journals [ 61 ]. The comprehensiveness of this database provides a robust representation of the literature in the field of intensive care research. Nevertheless, it is essential to acknowledge that some articles published in nonindexed journals might not have been captured, and future studies could benefit from considering additional databases to supplement our findings.

One other limitation lies in the classification of OAD and TIC studies, which may be subject to variations in the interpretation of keywords. However, we optimized the keyword combinations during the search process in the WoS database and implemented Python filtering techniques, resulting in a relatively high level of accuracy in our classifications. The number of studies was further corroborated with a manual search on PubMed and a review of the classifications of the studies was done by critical care physicians.

Although there were no specific language restrictions, the nature of the search term being in English inadvertently excluded valuable contributions from non-English research. This may potentially limit the generalizability of our findings to a broader international audience. In future investigations, the inclusion of articles from various languages could offer a more comprehensive and diverse perspective on intensive care research.

Conclusions

This study has provided valuable insights into the expanding landscape of intensive care research through a comprehensive bibliometric analysis of a large number of publications by leveraging NLP technologies. While OAD studies have demonstrated significant promise, it is essential to view them as a complementary approach rather than a replacement for TIC studies. The unique strength of TIC studies lies in their ability to capture crucial qualitative information, which is essential for comprehensive and ethical decision-making. The integration of both OAD and TIC studies offers a synergistic approach to enriching our understanding of critical care medicine and advancing patient care outcomes. As NLP technology continues to advance, it holds the potential to offer a feasible and transformative alternative for literature review and bibliometric analysis.

Acknowledgments

We thank Dr Nicholas Brian Shannon for assistance with the manual review of the supervised keyword classification. This work was supported by the Duke-NUS Signature Research Programme, funded by the Ministry of Health, Singapore.

Data Availability

The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable request. The complete set of code used in this study is readily available for download on GitHub [ 62 ].

Authors' Contributions

YK and NL played key roles in the conceptualization of the project. RY was responsible for formalizing the methodology and conducting data curation with the advisory of YK. YK contributed to the validation of the data, ensuring its relevance to the research objectives. RY took the lead in visualizing the data. Both YK and RY drafted the original manuscript. NL served as the project supervisor, overseeing the implementation, and providing valuable input in the writing, review, and editing phases.

Conflicts of Interest

None declared.

Search terms for open-access database (OAD) studies with the cutoff by the years of publications.

Top 20 journals ranked by total citation in which the open-access database and traditional intensive care studies were published. The average citation per article was obtained with the total citation/total number of articles. The citation counts were obtained from Web of Science.

Topic ID and topic family and the components and weightage in each of the categories.

Similarity matrix of 30 topics.

• Kelly FE, Fong K, Hirsch N, Nolan JP. Intensive care medicine is 60 years old: the history and future of the intensive care unit. Clin Med (Lond). 2014;14(4):376-379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cook D, Brower R, Cooper J, Brochard L, Vincent JL. Multicenter clinical research in adult critical care. Crit Care Med. 2002;30(7):1636-1643. [ CrossRef ] [ Medline ]
• Rosenberg AL, Tripathi RS, Blum J. The most influential articles in critical care medicine. J Crit Care. 2010;25(1):157-170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Granholm A, Alhazzani W, Derde LPG, Angus DC, Zampieri FG, Hammond NE, et al. Randomised clinical trials in critical care: past, present and future. Intensive Care Med. 2022;48(2):164-178. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Markey KA, Ottridge R, Mitchell JL, Rick C, Woolley R, Ives N, et al. Assessing the efficacy and safety of an 11β-hydroxysteroid dehydrogenase type 1 inhibitor (AZD4017) in the idiopathic intracranial hypertension drug trial, IIH:DT: clinical methods and design for a phase II randomized controlled trial. JMIR Res Protoc. 2017;6(9):e181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Verdonk F, Feyaerts D, Badenes R, Bastarache JA, Bouglé A, Ely W, et al. Upcoming and urgent challenges in critical care research based on COVID-19 pandemic experience. Anaesth Crit Care Pain Med. 2022;41(5):101121. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gurnani PK, Patel GP, Crank CW, Vais D, Lateef O, Akimov S, et al. Impact of the implementation of a sepsis protocol for the management of fluid-refractory septic shock: a single-center, before-and-after study. Clin Ther. 2010;32(7):1285-1293. [ CrossRef ] [ Medline ]
• Wang JL, Chin CS, Chang MC, Yi CY, Shih SJ, Hsu JY, et al. Key process indicators of mortality in the implementation of protocol-driven therapy for severe sepsis. J Formos Med Assoc. 2009;108(10):778-787. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Levy MM, Evans LE, Rhodes A. The surviving sepsis campaign bundle: 2018 update. Crit Care Med. 2018;46(6):997-1000. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fan E, Del Sorbo L, Goligher EC, Hodgson CL, Munshi L, Walkey AJ, et al. An official American Thoracic Society/European Society of Intensive Care Medicine/Society of Critical Care Medicine clinical practice guideline: mechanical ventilation in adult patients with acute respiratory distress syndrome. Am J Respir Crit Care Med. 2017;195(9):1253-1263. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Goldfrad C, Vella K, Bion JF, Rowan KM, Black NA. Research priorities in critical care medicine in the UK. Intensive Care Med. 2000;26(10):1480-1488. [ CrossRef ] [ Medline ]
• Moore TJ, Heyward J, Anderson G, Alexander GC. Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015-2017: a cross-sectional study. BMJ Open. 2020;10(6):e038863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Umscheid CA, Margolis DJ, Grossman CE. Key concepts of clinical trials: a narrative review. Postgrad Med. 2011;123(5):194-204. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maré IA, Kramer B, Hazelhurst S, Nhlapho MD, Zent R, Harris PA, et al. Electronic data capture system (REDCap) for health care research and training in a resource-constrained environment: technology adoption case study. JMIR Med Inform. 2022;10(8):e33402. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Hearn K, Gibson J, Krewulak K, Porteous R, Saigle V, Sampson M, et al. Consent models in Canadian critical care randomized controlled trials: a scoping review. Can J Anaesth. 2022;69(4):513-526. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ghassemi M, Celi LA, Stone DJ. State of the art review: the data revolution in critical care. Crit Care. 2015;19(1):118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood). 2014;33(7):1123-1131. [ CrossRef ] [ Medline ]
• Mlodzinski E, Wardi G, Viglione C, Nemati S, Crotty Alexander L, Malhotra A. Assessing barriers to implementation of machine learning and artificial intelligence-based tools in critical care: web-based survey study. JMIR Perioper Med. 2023;6:e41056. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Johnson AEW, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10(1):1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU collaborative research database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Inau ET, Sack J, Waltemath D, Zeleke AA. Initiatives, concepts, and implementation practices of FAIR (findable, accessible, interoperable, and reusable) data principles in health data stewardship practice: protocol for a scoping review. JMIR Res Protoc. 2021;10(2):e22505. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilkinson MD, Dumontier M, Aalbersberg IJJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3:160018. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bahadoran Z, Mirmiran P, Kashfi K, Ghasemi A. Importance of systematic reviews and meta-analyses of animal studies: challenges for animal-to-human translation. J Am Assoc Lab Anim Sci. 2020;59(5):469-477. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29-37. [ FREE Full text ] [ Medline ]
• Thompson DF, Walker CK. A descriptive and historical review of bibliometrics with applications to medical sciences. Pharmacotherapy. 2015;35(6):551-559. [ CrossRef ] [ Medline ]
• Donthu N, Kumar S, Mukherjee D, Pandey N, Lim WM. How to conduct a bibliometric analysis: an overview and guidelines. J Bus Res. 2021;133:285-296. [ FREE Full text ] [ CrossRef ]
• Zhao W, Chen JJ, Perkins R, Liu Z, Ge W, Ding Y, et al. A heuristic approach to determine an appropriate number of topics in topic modeling. BMC Bioinformatics. 2015;16(Suppl 13):S8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Doanvo A, Qian X, Ramjee D, Piontkivska H, Desai A, Majumder M. Machine learning maps research needs in COVID-19 literature. Patterns (N Y). 2020;1(9):100123. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. ArXiv. Preprint posted online on May 24 2019. 2018. [ CrossRef ]
• Grootendorst M. BERTopic: neural topic modeling with a class-based TF-IDF procedure. ArXiv. Preprint posted online on March 11 2022. 2022. [ CrossRef ]
• Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993-1022. [ FREE Full text ]
• Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ. A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care. 2005;9(5):R575-R582. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thoral PJ, Peppink JM, Driessen RH, Sijbrands EJG, Kompanje EJO, Kaplan L, et al. Sharing ICU patient data responsibly under the Society of Critical Care Medicine/European Society of Intensive Care Medicine joint data science collaboration: the Amsterdam university medical centers database (AmsterdamUMCdb) example. Crit Care Med. 2021;49(6):e563-e577. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Faltys M, Zimmermann M, Lyu X, Hüser M, Hyland S, Rätsch G, et al. HiRID, a high time-resolution ICU dataset. PhysioNet. 2021. URL: https://physionet.org/content/hirid/1.1.1/ [accessed 2024-04-02]
• Zeng X, Yu G, Lu Y, Tan L, Wu X, Shi S, et al. PIC, a paediatric-specific intensive care database. Sci Data. 2020;7(1):14. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maćkiewicz A, Ratajczak W. Principal components analysis (PCA). Comput Geosci. 1993;19(3):303-342. [ CrossRef ]
• McInnes L, Healy J, Melville J. UMAP: uniform manifold approximation and projection for dimension reduction. ArXiv. Preprint posted online on September 18 2020. 2018. [ CrossRef ]
• Murray CJL. COVID-19 will continue but the end of the pandemic is near. Lancet. 2022;399(10323):417-419. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mark R. The story of MIMIC. In: Secondary Analysis of Electronic Health Records. Cham, Switzerland. Springer International Publishing; 2016;43-49.
• Alghatani K, Ammar N, Rezgui A, Shaban-Nejad A. Predicting intensive care unit length of stay and mortality using patient vital signs: machine learning model development and validation. JMIR Med Inform. 2021;9(5):e21347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu D, Zheng M, Sepulveda NA. Using artificial neural network condensation to facilitate adaptation of machine learning in medical settings by reducing computational burden: model design and evaluation study. JMIR Form Res. 2021;5(12):e20767. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Ho KM, Hong Y. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit Care. 2019;23(1):112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, Sündermann SH, et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018;6(12):905-914. [ CrossRef ] [ Medline ]
• Chen H, Zhu Z, Zhao C, Guo Y, Chen D, Wei Y, et al. Central venous pressure measurement is associated with improved outcomes in septic patients: an analysis of the MIMIC-III database. Crit Care. 2020;24(1):433. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Faust L, Feldman K, Chawla NV. Examining the weekend effect across ICU performance metrics. Crit Care. 2019;23(1):207. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davidson S, Villarroel M, Harford M, Finnegan E, Jorge J, Young D, et al. Vital-sign circadian rhythms in patients prior to discharge from an ICU: a retrospective observational analysis of routinely recorded physiological data. Crit Care. 2020;24(1):181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xie F, Yuan H, Ning Y, Ong MEH, Feng M, Hsu W, et al. Deep learning for temporal data representation in electronic health records: a systematic review of challenges and methodologies. J Biomed Inform. 2022;126:103980. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ, et al. GRADE Working Group. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith CM, French C, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021. Crit Care Med. 2021;49(11):e1063-e1143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Volovici V, Syn NL, Ercole A, Zhao JJ, Liu N. Steps to avoid overuse and misuse of machine learning in clinical research. Nat Med. 2022;28(10):1996-1999. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020;46(3):383-400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yoon CH, Torrance R, Scheinerman N. Machine learning in medicine: should the pursuit of enhanced interpretability be abandoned? J Med Ethics. 2022;48(9):581-585. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gillett GR. Intensive care unit research ethics and trials on unconscious patients. Anaesth Intensive Care. 2015;43(3):309-312. [ CrossRef ] [ Medline ]
• Aulisio MP, Chaitin E, Arnold RM. Ethics and palliative care consultation in the intensive care unit. Crit Care Clin. 2004;20(3):505-523, x-xi. [ CrossRef ] [ Medline ]
• Raudenská J, Steinerová V, Javůrková A, Urits I, Kaye AD, Viswanath O, et al. Occupational burnout syndrome and post-traumatic stress among healthcare professionals during the novel coronavirus disease 2019 (COVID-19) pandemic. Best Pract Res Clin Anaesthesiol. 2020;34(3):553-560. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davidson JE, Jones C, Bienvenu OJ. Family response to critical illness: postintensive care syndrome-family. Crit Care Med. 2012;40(2):618-624. [ CrossRef ] [ Medline ]
• White DB, Angus DC, Shields AM, Buddadhumaruk P, Pidro C, Paner C, et al. A randomized trial of a family-support intervention in intensive care units. N Engl J Med. 2018;378(25):2365-2375. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Popoff B, Occhiali É, Grangé S, Bergis A, Carpentier D, Tamion F, et al. Trends in major intensive care medicine journals: a machine learning approach. J Crit Care. 2022;72:154163. [ CrossRef ] [ Medline ]
• Yang R, Tan TF, Lu W, Thirunavukarasu AJ, Ting DSW, Liu N. Large language models in health care: development, applications, and challenges. Health Care Sci. 2023;2(4):255-263. [ FREE Full text ] [ CrossRef ]
• Birkle C, Pendlebury DA, Schnell J, Adams J. Web of science as a data source for research on scientific and scholarly activity. Quant Sci Stud. 2020;1(1):363-376. [ FREE Full text ] [ CrossRef ]
• GitHub. URL: https://github.com/YangRui525/Comparing-OAD-and-TIC-Studies

Abbreviations

Edited by A Mavragani; submitted 19.04.23; peer-reviewed by D Chrimes, S Pesälä; comments to author 14.07.23; revised version received 01.08.23; accepted 14.01.24; published 17.04.24.

©Yuhe Ke, Rui Yang, Nan Liu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.