research method of observational

• Privacy Policy

Home » Observational Research – Methods and Guide

Observational Research – Methods and Guide

Table of Contents

Observational Research

Definition:

Observational research is a type of research method where the researcher observes and records the behavior of individuals or groups in their natural environment. In other words, the researcher does not intervene or manipulate any variables but simply observes and describes what is happening.

Observation

Observation is the process of collecting and recording data by observing and noting events, behaviors, or phenomena in a systematic and objective manner. It is a fundamental method used in research, scientific inquiry, and everyday life to gain an understanding of the world around us.

Types of Observational Research

Observational research can be categorized into different types based on the level of control and the degree of involvement of the researcher in the study. Some of the common types of observational research are:

Naturalistic Observation

In naturalistic observation, the researcher observes and records the behavior of individuals or groups in their natural environment without any interference or manipulation of variables.

Controlled Observation

In controlled observation, the researcher controls the environment in which the observation is taking place. This type of observation is often used in laboratory settings.

Participant Observation

In participant observation, the researcher becomes an active participant in the group or situation being observed. The researcher may interact with the individuals being observed and gather data on their behavior, attitudes, and experiences.

Structured Observation

In structured observation, the researcher defines a set of behaviors or events to be observed and records their occurrence.

Unstructured Observation

In unstructured observation, the researcher observes and records any behaviors or events that occur without predetermined categories.

Cross-Sectional Observation

In cross-sectional observation, the researcher observes and records the behavior of different individuals or groups at a single point in time.

Longitudinal Observation

In longitudinal observation, the researcher observes and records the behavior of the same individuals or groups over an extended period of time.

Data Collection Methods

Observational research uses various data collection methods to gather information about the behaviors and experiences of individuals or groups being observed. Some common data collection methods used in observational research include:

Field Notes

This method involves recording detailed notes of the observed behavior, events, and interactions. These notes are usually written in real-time during the observation process.

Audio and Video Recordings

Audio and video recordings can be used to capture the observed behavior and interactions. These recordings can be later analyzed to extract relevant information.

Surveys and Questionnaires

Surveys and questionnaires can be used to gather additional information from the individuals or groups being observed. This method can be used to validate or supplement the observational data.

Time Sampling

This method involves taking a snapshot of the observed behavior at pre-determined time intervals. This method helps to identify the frequency and duration of the observed behavior.

Event Sampling

This method involves recording specific events or behaviors that are of interest to the researcher. This method helps to provide detailed information about specific behaviors or events.

Checklists and Rating Scales

Checklists and rating scales can be used to record the occurrence and frequency of specific behaviors or events. This method helps to simplify and standardize the data collection process.

Observational Data Analysis Methods

Observational Data Analysis Methods are:

Descriptive Statistics

This method involves using statistical techniques such as frequency distributions, means, and standard deviations to summarize the observed behaviors, events, or interactions.

Qualitative Analysis

Qualitative analysis involves identifying patterns and themes in the observed behaviors or interactions. This analysis can be done manually or with the help of software tools.

Content Analysis

Content analysis involves categorizing and counting the occurrences of specific behaviors or events. This analysis can be done manually or with the help of software tools.

Time-series Analysis

Time-series analysis involves analyzing the changes in behavior or interactions over time. This analysis can help identify trends and patterns in the observed data.

Inter-observer Reliability Analysis

Inter-observer reliability analysis involves comparing the observations made by multiple observers to ensure the consistency and reliability of the data.

Multivariate Analysis

Multivariate analysis involves analyzing multiple variables simultaneously to identify the relationships between the observed behaviors, events, or interactions.

Event Coding

This method involves coding observed behaviors or events into specific categories and then analyzing the frequency and duration of each category.

Cluster Analysis

Cluster analysis involves grouping similar behaviors or events into clusters based on their characteristics or patterns.

Latent Class Analysis

Latent class analysis involves identifying subgroups of individuals or groups based on their observed behaviors or interactions.

Social network Analysis

Social network analysis involves mapping the social relationships and interactions between individuals or groups based on their observed behaviors.

The choice of data analysis method depends on the research question, the type of data collected, and the available resources. Researchers should choose the appropriate method that best fits their research question and objectives. It is also important to ensure the validity and reliability of the data analysis by using appropriate statistical tests and measures.

Applications of Observational Research

Observational research is a versatile research method that can be used in a variety of fields to explore and understand human behavior, attitudes, and preferences. Here are some common applications of observational research:

• Psychology : Observational research is commonly used in psychology to study human behavior in natural settings. This can include observing children at play to understand their social development or observing people’s reactions to stress to better understand how stress affects behavior.
• Marketing : Observational research is used in marketing to understand consumer behavior and preferences. This can include observing shoppers in stores to understand how they make purchase decisions or observing how people interact with advertisements to determine their effectiveness.
• Education : Observational research is used in education to study teaching and learning in natural settings. This can include observing classrooms to understand how teachers interact with students or observing students to understand how they learn.
• Anthropology : Observational research is commonly used in anthropology to understand cultural practices and beliefs. This can include observing people’s daily routines to understand their culture or observing rituals and ceremonies to better understand their significance.
• Healthcare : Observational research is used in healthcare to understand patient behavior and preferences. This can include observing patients in hospitals to understand how they interact with healthcare professionals or observing patients with chronic illnesses to better understand their daily routines and needs.
• Sociology : Observational research is used in sociology to understand social interactions and relationships. This can include observing people in public spaces to understand how they interact with others or observing groups to understand how they function.
• Ecology : Observational research is used in ecology to understand the behavior and interactions of animals and plants in their natural habitats. This can include observing animal behavior to understand their social structures or observing plant growth to understand their response to environmental factors.
• Criminology : Observational research is used in criminology to understand criminal behavior and the factors that contribute to it. This can include observing criminal activity in a particular area to identify patterns or observing the behavior of inmates to understand their experience in the criminal justice system.

Observational Research Examples

Here are some real-time observational research examples:

• A researcher observes and records the behaviors of a group of children on a playground to study their social interactions and play patterns.
• A researcher observes the buying behaviors of customers in a retail store to study the impact of store layout and product placement on purchase decisions.
• A researcher observes the behavior of drivers at a busy intersection to study the effectiveness of traffic signs and signals.
• A researcher observes the behavior of patients in a hospital to study the impact of staff communication and interaction on patient satisfaction and recovery.
• A researcher observes the behavior of employees in a workplace to study the impact of the work environment on productivity and job satisfaction.
• A researcher observes the behavior of shoppers in a mall to study the impact of music and lighting on consumer behavior.
• A researcher observes the behavior of animals in their natural habitat to study their social and feeding behaviors.
• A researcher observes the behavior of students in a classroom to study the effectiveness of teaching methods and student engagement.
• A researcher observes the behavior of pedestrians and cyclists on a city street to study the impact of infrastructure and traffic regulations on safety.

How to Conduct Observational Research

Here are some general steps for conducting Observational Research:

• Define the Research Question: Determine the research question and objectives to guide the observational research study. The research question should be specific, clear, and relevant to the area of study.
• Choose the appropriate observational method: Choose the appropriate observational method based on the research question, the type of data required, and the available resources.
• Plan the observation: Plan the observation by selecting the observation location, duration, and sampling technique. Identify the population or sample to be observed and the characteristics to be recorded.
• Train observers: Train the observers on the observational method, data collection tools, and techniques. Ensure that the observers understand the research question and objectives and can accurately record the observed behaviors or events.
• Conduct the observation : Conduct the observation by recording the observed behaviors or events using the data collection tools and techniques. Ensure that the observation is conducted in a consistent and unbiased manner.
• Analyze the data: Analyze the observed data using appropriate data analysis methods such as descriptive statistics, qualitative analysis, or content analysis. Validate the data by checking the inter-observer reliability and conducting statistical tests.
• Interpret the results: Interpret the results by answering the research question and objectives. Identify the patterns, trends, or relationships in the observed data and draw conclusions based on the analysis.
• Report the findings: Report the findings in a clear and concise manner, using appropriate visual aids and tables. Discuss the implications of the results and the limitations of the study.

When to use Observational Research

Here are some situations where observational research can be useful:

• Exploratory Research: Observational research can be used in exploratory studies to gain insights into new phenomena or areas of interest.
• Hypothesis Generation: Observational research can be used to generate hypotheses about the relationships between variables, which can be tested using experimental research.
• Naturalistic Settings: Observational research is useful in naturalistic settings where it is difficult or unethical to manipulate the environment or variables.
• Human Behavior: Observational research is useful in studying human behavior, such as social interactions, decision-making, and communication patterns.
• Animal Behavior: Observational research is useful in studying animal behavior in their natural habitats, such as social and feeding behaviors.
• Longitudinal Studies: Observational research can be used in longitudinal studies to observe changes in behavior over time.
• Ethical Considerations: Observational research can be used in situations where manipulating the environment or variables would be unethical or impractical.

Purpose of Observational Research

Observational research is a method of collecting and analyzing data by observing individuals or phenomena in their natural settings, without manipulating them in any way. The purpose of observational research is to gain insights into human behavior, attitudes, and preferences, as well as to identify patterns, trends, and relationships that may exist between variables.

The primary purpose of observational research is to generate hypotheses that can be tested through more rigorous experimental methods. By observing behavior and identifying patterns, researchers can develop a better understanding of the factors that influence human behavior, and use this knowledge to design experiments that test specific hypotheses.

Observational research is also used to generate descriptive data about a population or phenomenon. For example, an observational study of shoppers in a grocery store might reveal that women are more likely than men to buy organic produce. This type of information can be useful for marketers or policy-makers who want to understand consumer preferences and behavior.

In addition, observational research can be used to monitor changes over time. By observing behavior at different points in time, researchers can identify trends and changes that may be indicative of broader social or cultural shifts.

Overall, the purpose of observational research is to provide insights into human behavior and to generate hypotheses that can be tested through further research.

Advantages of Observational Research

There are several advantages to using observational research in different fields, including:

• Naturalistic observation: Observational research allows researchers to observe behavior in a naturalistic setting, which means that people are observed in their natural environment without the constraints of a laboratory. This helps to ensure that the behavior observed is more representative of the real-world situation.
• Unobtrusive : Observational research is often unobtrusive, which means that the researcher does not interfere with the behavior being observed. This can reduce the likelihood of the research being affected by the observer’s presence or the Hawthorne effect, where people modify their behavior when they know they are being observed.
• Cost-effective : Observational research can be less expensive than other research methods, such as experiments or surveys. Researchers do not need to recruit participants or pay for expensive equipment, making it a more cost-effective research method.
• Flexibility: Observational research is a flexible research method that can be used in a variety of settings and for a range of research questions. Observational research can be used to generate hypotheses, to collect data on behavior, or to monitor changes over time.
• Rich data : Observational research provides rich data that can be analyzed to identify patterns and relationships between variables. It can also provide context for behaviors, helping to explain why people behave in a certain way.
• Validity : Observational research can provide high levels of validity, meaning that the results accurately reflect the behavior being studied. This is because the behavior is being observed in a natural setting without interference from the researcher.

Disadvantages of Observational Research

While observational research has many advantages, it also has some limitations and disadvantages. Here are some of the disadvantages of observational research:

• Observer bias: Observational research is prone to observer bias, which is when the observer’s own beliefs and assumptions affect the way they interpret and record behavior. This can lead to inaccurate or unreliable data.
• Limited generalizability: The behavior observed in a specific setting may not be representative of the behavior in other settings. This can limit the generalizability of the findings from observational research.
• Difficulty in establishing causality: Observational research is often correlational, which means that it identifies relationships between variables but does not establish causality. This can make it difficult to determine if a particular behavior is causing an outcome or if the relationship is due to other factors.
• Ethical concerns: Observational research can raise ethical concerns if the participants being observed are unaware that they are being observed or if the observations invade their privacy.
• Time-consuming: Observational research can be time-consuming, especially if the behavior being observed is infrequent or occurs over a long period of time. This can make it difficult to collect enough data to draw valid conclusions.
• Difficulty in measuring internal processes: Observational research may not be effective in measuring internal processes, such as thoughts, feelings, and attitudes. This can limit the ability to understand the reasons behind behavior.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Case Study – Methods, Examples and Guide

Quantitative Research – Methods, Types and...

Qualitative Research Methods

Explanatory Research – Types, Methods, Guide

Survey Research – Types, Methods, Examples

6.5 Observational Research

Learning objectives.

• List the various types of observational research methods and distinguish between each
• Describe the strengths and weakness of each observational research method. 

What Is Observational Research?

The term observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded. The goal of observational research is to describe a variable or set of variables. More generally, the goal is to obtain a snapshot of specific characteristics of an individual, group, or setting. As described previously, observational research is non-experimental because nothing is manipulated or controlled, and as such we cannot arrive at causal conclusions using this approach. The data that are collected in observational research studies are often qualitative in nature but they may also be quantitative or both (mixed-methods). There are several different types of observational research designs that will be described below.

Naturalistic Observation

Naturalistic observation  is an observational method that involves observing people’s behavior in the environment in which it typically occurs. Thus naturalistic observation is a type of field research (as opposed to a type of laboratory research). Jane Goodall’s famous research on chimpanzees is a classic example of naturalistic observation. Dr.  Goodall spent three decades observing chimpanzees in their natural environment in East Africa. She examined such things as chimpanzee’s social structure, mating patterns, gender roles, family structure, and care of offspring by observing them in the wild. However, naturalistic observation  could more simply involve observing shoppers in a grocery store, children on a school playground, or psychiatric inpatients in their wards. Researchers engaged in naturalistic observation usually make their observations as unobtrusively as possible so that participants are not aware that they are being studied. Such an approach is called disguised naturalistic observation.  Ethically, this method is considered to be acceptable if the participants remain anonymous and the behavior occurs in a public setting where people would not normally have an expectation of privacy. Grocery shoppers putting items into their shopping carts, for example, are engaged in public behavior that is easily observable by store employees and other shoppers. For this reason, most researchers would consider it ethically acceptable to observe them for a study. On the other hand, one of the arguments against the ethicality of the naturalistic observation of “bathroom behavior” discussed earlier in the book is that people have a reasonable expectation of privacy even in a public restroom and that this expectation was violated. 

In cases where it is not ethical or practical to conduct disguised naturalistic observation, researchers can conduct  undisguised naturalistic observation where the participants are made aware of the researcher presence and monitoring of their behavior. However, one concern with undisguised naturalistic observation is  reactivity. Reactivity  refers to when a measure changes participants’ behavior. In the case of undisguised naturalistic observation, the concern with reactivity is that when people know they are being observed and studied, they may act differently than they normally would. For instance, you may act much differently in a bar if you know that someone is observing you and recording your behaviors and this would invalidate the study. So disguised observation is less reactive and therefore can have higher validity because people are not aware that their behaviors are being observed and recorded. However, we now know that people often become used to being observed and with time they begin to behave naturally in the researcher’s presence. In other words, over time people habituate to being observed. Think about reality shows like Big Brother or Survivor where people are constantly being observed and recorded. While they may be on their best behavior at first, in a fairly short amount of time they are, flirting, having sex, wearing next to nothing, screaming at each other, and at times acting like complete fools in front of the entire nation.

Participant Observation

Another approach to data collection in observational research is participant observation. In  participant observation , researchers become active participants in the group or situation they are studying. Participant observation is very similar to naturalistic observation in that it involves observing people’s behavior in the environment in which it typically occurs. As with naturalistic observation, the data that is collected can include interviews (usually unstructured), notes based on their observations and interactions, documents, photographs, and other artifacts. The only difference between naturalistic observation and participant observation is that researchers engaged in participant observation become active members of the group or situations they are studying. The basic rationale for participant observation is that there may be important information that is only accessible to, or can be interpreted only by, someone who is an active participant in the group or situation. Like naturalistic observation, participant observation can be either disguised or undisguised. In disguised participant observation, the researchers pretend to be members of the social group they are observing and conceal their true identity as researchers. In contrast with undisguised participant observation,  the researchers become a part of the group they are studying and they disclose their true identity as researchers to the group under investigation. Once again there are important ethical issues to consider with disguised participant observation.  First no informed consent can be obtained and second passive deception is being used. The researcher is passively deceiving the participants by intentionally withholding information about their motivations for being a part of the social group they are studying. But sometimes disguised participation is the only way to access a protective group (like a cult). Further,  disguised participant observation is less prone to reactivity than undisguised participant observation. 

Rosenhan’s study (1973) [1]   of the experience of people in a psychiatric ward would be considered disguised participant observation because Rosenhan and his pseudopatients were admitted into psychiatric hospitals on the pretense of being patients so that they could observe the way that psychiatric patients are treated by staff. The staff and other patients were unaware of their true identities as researchers.

Another example of participant observation comes from a study by sociologist Amy Wilkins (published in  Social Psychology Quarterly ) on a university-based religious organization that emphasized how happy its members were (Wilkins, 2008) [2] . Wilkins spent 12 months attending and participating in the group’s meetings and social events, and she interviewed several group members. In her study, Wilkins identified several ways in which the group “enforced” happiness—for example, by continually talking about happiness, discouraging the expression of negative emotions, and using happiness as a way to distinguish themselves from other groups.

One of the primary benefits of participant observation is that the researcher is in a much better position to understand the viewpoint and experiences of the people they are studying when they are apart of the social group. The primary limitation with this approach is that the mere presence of the observer could affect the behavior of the people being observed. While this is also a concern with naturalistic observation when researchers because active members of the social group they are studying, additional concerns arise that they may change the social dynamics and/or influence the behavior of the people they are studying. Similarly, if the researcher acts as a participant observer there can be concerns with biases resulting from developing relationships with the participants. Concretely, the researcher may become less objective resulting in more experimenter bias.

Structured Observation

Another observational method is structured observation. Here the investigator makes careful observations of one or more specific behaviors in a particular setting that is more structured than the settings used in naturalistic and participant observation. Often the setting in which the observations are made is not the natural setting, rather the researcher may observe people in the laboratory environment. Alternatively, the researcher may observe people in a natural setting (like a classroom setting) that they have structured some way, for instance by introducing some specific task participants are to engage in or by introducing a specific social situation or manipulation. Structured observation is very similar to naturalistic observation and participant observation in that in all cases researchers are observing naturally occurring behavior, however, the emphasis in structured observation is on gathering quantitative rather than qualitative data. Researchers using this approach are interested in a limited set of behaviors. This allows them to quantify the behaviors they are observing. In other words, structured observation is less global than naturalistic and participant observation because the researcher engaged in structured observations is interested in a small number of specific behaviors. Therefore, rather than recording everything that happens, the researcher only focuses on very specific behaviors of interest.

Structured observation is very similar to naturalistic observation and participant observation in that in all cases researchers are observing naturally occurring behavior, however, the emphasis in structured observation is on gathering quantitative rather than qualitative data. Researchers using this approach are interested in a limited set of behaviors. This allows them to quantify the behaviors they are observing. In other words, structured observation is less global than naturalistic and participant observation because the researcher engaged in structured observations is interested in a small number of specific behaviors. Therefore, rather than recording everything that happens, the researcher only focuses on very specific behaviors of interest.

Researchers Robert Levine and Ara Norenzayan used structured observation to study differences in the “pace of life” across countries (Levine & Norenzayan, 1999) [3] . One of their measures involved observing pedestrians in a large city to see how long it took them to walk 60 feet. They found that people in some countries walked reliably faster than people in other countries. For example, people in Canada and Sweden covered 60 feet in just under 13 seconds on average, while people in Brazil and Romania took close to 17 seconds. When structured observation  takes place in the complex and even chaotic “real world,” the questions of when, where, and under what conditions the observations will be made, and who exactly will be observed are important to consider. Levine and Norenzayan described their sampling process as follows:

“Male and female walking speed over a distance of 60 feet was measured in at least two locations in main downtown areas in each city. Measurements were taken during main business hours on clear summer days. All locations were flat, unobstructed, had broad sidewalks, and were sufficiently uncrowded to allow pedestrians to move at potentially maximum speeds. To control for the effects of socializing, only pedestrians walking alone were used. Children, individuals with obvious physical handicaps, and window-shoppers were not timed. Thirty-five men and 35 women were timed in most cities.” (p. 186).  Precise specification of the sampling process in this way makes data collection manageable for the observers, and it also provides some control over important extraneous variables. For example, by making their observations on clear summer days in all countries, Levine and Norenzayan controlled for effects of the weather on people’s walking speeds.  In Levine and Norenzayan’s study, measurement was relatively straightforward. They simply measured out a 60-foot distance along a city sidewalk and then used a stopwatch to time participants as they walked over that distance.

As another example, researchers Robert Kraut and Robert Johnston wanted to study bowlers’ reactions to their shots, both when they were facing the pins and then when they turned toward their companions (Kraut & Johnston, 1979) [4] . But what “reactions” should they observe? Based on previous research and their own pilot testing, Kraut and Johnston created a list of reactions that included “closed smile,” “open smile,” “laugh,” “neutral face,” “look down,” “look away,” and “face cover” (covering one’s face with one’s hands). The observers committed this list to memory and then practiced by coding the reactions of bowlers who had been videotaped. During the actual study, the observers spoke into an audio recorder, describing the reactions they observed. Among the most interesting results of this study was that bowlers rarely smiled while they still faced the pins. They were much more likely to smile after they turned toward their companions, suggesting that smiling is not purely an expression of happiness but also a form of social communication.

When the observations require a judgment on the part of the observers—as in Kraut and Johnston’s study—this process is often described as  coding . Coding generally requires clearly defining a set of target behaviors. The observers then categorize participants individually in terms of which behavior they have engaged in and the number of times they engaged in each behavior. The observers might even record the duration of each behavior. The target behaviors must be defined in such a way that different observers code them in the same way. This difficulty with coding is the issue of interrater reliability, as mentioned in Chapter 4. Researchers are expected to demonstrate the interrater reliability of their coding procedure by having multiple raters code the same behaviors independently and then showing that the different observers are in close agreement. Kraut and Johnston, for example, video recorded a subset of their participants’ reactions and had two observers independently code them. The two observers showed that they agreed on the reactions that were exhibited 97% of the time, indicating good interrater reliability.

One of the primary benefits of structured observation is that it is far more efficient than naturalistic and participant observation. Since the researchers are focused on specific behaviors this reduces time and expense. Also, often times the environment is structured to encourage the behaviors of interested which again means that researchers do not have to invest as much time in waiting for the behaviors of interest to naturally occur. Finally, researchers using this approach can clearly exert greater control over the environment. However, when researchers exert more control over the environment it may make the environment less natural which decreases external validity. It is less clear for instance whether structured observations made in a laboratory environment will generalize to a real world environment. Furthermore, since researchers engaged in structured observation are often not disguised there may be more concerns with reactivity.

Case Studies

A  case study  is an in-depth examination of an individual. Sometimes case studies are also completed on social units (e.g., a cult) and events (e.g., a natural disaster). Most commonly in psychology, however, case studies provide a detailed description and analysis of an individual. Often the individual has a rare or unusual condition or disorder or has damage to a specific region of the brain.

Like many observational research methods, case studies tend to be more qualitative in nature. Case study methods involve an in-depth, and often a longitudinal examination of an individual. Depending on the focus of the case study, individuals may or may not be observed in their natural setting. If the natural setting is not what is of interest, then the individual may be brought into a therapist’s office or a researcher’s lab for study. Also, the bulk of the case study report will focus on in-depth descriptions of the person rather than on statistical analyses. With that said some quantitative data may also be included in the write-up of a case study. For instance, an individuals’ depression score may be compared to normative scores or their score before and after treatment may be compared. As with other qualitative methods, a variety of different methods and tools can be used to collect information on the case. For instance, interviews, naturalistic observation, structured observation, psychological testing (e.g., IQ test), and/or physiological measurements (e.g., brain scans) may be used to collect information on the individual.

HM is one of the most notorious case studies in psychology. HM suffered from intractable and very severe epilepsy. A surgeon localized HM’s epilepsy to his medial temporal lobe and in 1953 he removed large sections of his hippocampus in an attempt to stop the seizures. The treatment was a success, in that it resolved his epilepsy and his IQ and personality were unaffected. However, the doctors soon realized that HM exhibited a strange form of amnesia, called anterograde amnesia. HM was able to carry out a conversation and he could remember short strings of letters, digits, and words. Basically, his short term memory was preserved. However, HM could not commit new events to memory. He lost the ability to transfer information from his short-term memory to his long term memory, something memory researchers call consolidation. So while he could carry on a conversation with someone, he would completely forget the conversation after it ended. This was an extremely important case study for memory researchers because it suggested that there’s a dissociation between short-term memory and long-term memory, it suggested that these were two different abilities sub-served by different areas of the brain. It also suggested that the temporal lobes are particularly important for consolidating new information (i.e., for transferring information from short-term memory to long-term memory).

www.youtube.com/watch?v=KkaXNvzE4pk

The history of psychology is filled with influential cases studies, such as Sigmund Freud’s description of “Anna O.” (see Note 6.1 “The Case of “Anna O.””) and John Watson and Rosalie Rayner’s description of Little Albert (Watson & Rayner, 1920) [5] , who learned to fear a white rat—along with other furry objects—when the researchers made a loud noise while he was playing with the rat.

The Case of “Anna O.”

Sigmund Freud used the case of a young woman he called “Anna O.” to illustrate many principles of his theory of psychoanalysis (Freud, 1961) [6] . (Her real name was Bertha Pappenheim, and she was an early feminist who went on to make important contributions to the field of social work.) Anna had come to Freud’s colleague Josef Breuer around 1880 with a variety of odd physical and psychological symptoms. One of them was that for several weeks she was unable to drink any fluids. According to Freud,

She would take up the glass of water that she longed for, but as soon as it touched her lips she would push it away like someone suffering from hydrophobia.…She lived only on fruit, such as melons, etc., so as to lessen her tormenting thirst. (p. 9)

But according to Freud, a breakthrough came one day while Anna was under hypnosis.

[S]he grumbled about her English “lady-companion,” whom she did not care for, and went on to describe, with every sign of disgust, how she had once gone into this lady’s room and how her little dog—horrid creature!—had drunk out of a glass there. The patient had said nothing, as she had wanted to be polite. After giving further energetic expression to the anger she had held back, she asked for something to drink, drank a large quantity of water without any difficulty, and awoke from her hypnosis with the glass at her lips; and thereupon the disturbance vanished, never to return. (p.9)

Freud’s interpretation was that Anna had repressed the memory of this incident along with the emotion that it triggered and that this was what had caused her inability to drink. Furthermore, her recollection of the incident, along with her expression of the emotion she had repressed, caused the symptom to go away.

As an illustration of Freud’s theory, the case study of Anna O. is quite effective. As evidence for the theory, however, it is essentially worthless. The description provides no way of knowing whether Anna had really repressed the memory of the dog drinking from the glass, whether this repression had caused her inability to drink, or whether recalling this “trauma” relieved the symptom. It is also unclear from this case study how typical or atypical Anna’s experience was.

Figure 10.1 Anna O. “Anna O.” was the subject of a famous case study used by Freud to illustrate the principles of psychoanalysis. Source: http://en.wikipedia.org/wiki/File:Pappenheim_1882.jpg

Case studies are useful because they provide a level of detailed analysis not found in many other research methods and greater insights may be gained from this more detailed analysis. As a result of the case study, the researcher may gain a sharpened understanding of what might become important to look at more extensively in future more controlled research. Case studies are also often the only way to study rare conditions because it may be impossible to find a large enough sample to individuals with the condition to use quantitative methods. Although at first glance a case study of a rare individual might seem to tell us little about ourselves, they often do provide insights into normal behavior. The case of HM provided important insights into the role of the hippocampus in memory consolidation. However, it is important to note that while case studies can provide insights into certain areas and variables to study, and can be useful in helping develop theories, they should never be used as evidence for theories. In other words, case studies can be used as inspiration to formulate theories and hypotheses, but those hypotheses and theories then need to be formally tested using more rigorous quantitative methods.

The reason case studies shouldn’t be used to provide support for theories is that they suffer from problems with internal and external validity. Case studies lack the proper controls that true experiments contain. As such they suffer from problems with internal validity, so they cannot be used to determine causation. For instance, during HM’s surgery, the surgeon may have accidentally lesioned another area of HM’s brain (indeed questioning into the possibility of a separate brain lesion began after HM’s death and dissection of his brain) and that lesion may have contributed to his inability to consolidate new information. The fact is, with case studies we cannot rule out these sorts of alternative explanations. So as with all observational methods case studies do not permit determination of causation. In addition, because case studies are often of a single individual, and typically a very abnormal individual, researchers cannot generalize their conclusions to other individuals. Recall that with most research designs there is a trade-off between internal and external validity, with case studies, however, there are problems with both internal validity and external validity. So there are limits both to the ability to determine causation and to generalize the results. A final limitation of case studies is that ample opportunity exists for the theoretical biases of the researcher to color or bias the case description. Indeed, there have been accusations that the woman who studied HM destroyed a lot of her data that were not published and she has been called into question for destroying contradictory data that didn’t support her theory about how memories are consolidated. There is a fascinating New York Times article that describes some of the controversies that ensued after HM’s death and analysis of his brain that can be found at: https://www.nytimes.com/2016/08/07/magazine/the-brain-that-couldnt-remember.html?_r=0

Archival Research

Another approach that is often considered observational research is the use of  archival research  which involves analyzing data that have already been collected for some other purpose. An example is a study by Brett Pelham and his colleagues on “implicit egotism”—the tendency for people to prefer people, places, and things that are similar to themselves (Pelham, Carvallo, & Jones, 2005) [7] . In one study, they examined Social Security records to show that women with the names Virginia, Georgia, Louise, and Florence were especially likely to have moved to the states of Virginia, Georgia, Louisiana, and Florida, respectively.

As with naturalistic observation, measurement can be more or less straightforward when working with archival data. For example, counting the number of people named Virginia who live in various states based on Social Security records is relatively straightforward. But consider a study by Christopher Peterson and his colleagues on the relationship between optimism and health using data that had been collected many years before for a study on adult development (Peterson, Seligman, & Vaillant, 1988) [8] . In the 1940s, healthy male college students had completed an open-ended questionnaire about difficult wartime experiences. In the late 1980s, Peterson and his colleagues reviewed the men’s questionnaire responses to obtain a measure of explanatory style—their habitual ways of explaining bad events that happen to them. More pessimistic people tend to blame themselves and expect long-term negative consequences that affect many aspects of their lives, while more optimistic people tend to blame outside forces and expect limited negative consequences. To obtain a measure of explanatory style for each participant, the researchers used a procedure in which all negative events mentioned in the questionnaire responses, and any causal explanations for them were identified and written on index cards. These were given to a separate group of raters who rated each explanation in terms of three separate dimensions of optimism-pessimism. These ratings were then averaged to produce an explanatory style score for each participant. The researchers then assessed the statistical relationship between the men’s explanatory style as undergraduate students and archival measures of their health at approximately 60 years of age. The primary result was that the more optimistic the men were as undergraduate students, the healthier they were as older men. Pearson’s  r  was +.25.

This method is an example of  content analysis —a family of systematic approaches to measurement using complex archival data. Just as structured observation requires specifying the behaviors of interest and then noting them as they occur, content analysis requires specifying keywords, phrases, or ideas and then finding all occurrences of them in the data. These occurrences can then be counted, timed (e.g., the amount of time devoted to entertainment topics on the nightly news show), or analyzed in a variety of other ways.

Key Takeaways

• There are several different approaches to observational research including naturalistic observation, participant observation, structured observation, case studies, and archival research.
• Naturalistic observation is used to observe people in their natural setting, participant observation involves becoming an active member of the group being observed, structured observation involves coding a small number of behaviors in a quantitative manner, case studies are typically used to collect in-depth information on a single individual, and archival research involves analysing existing data.
• Describe one problem related to internal validity.
• Describe one problem related to external validity.
• Generate one hypothesis suggested by the case study that might be interesting to test in a systematic single-subject or group study.
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵
• Wilkins, A. (2008). “Happier than Non-Christians”: Collective emotions and symbolic boundaries among evangelical Christians. Social Psychology Quarterly, 71 , 281–301. ↵
• Levine, R. V., & Norenzayan, A. (1999). The pace of life in 31 countries. Journal of Cross-Cultural Psychology, 30 , 178–205. ↵
• Kraut, R. E., & Johnston, R. E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of Personality and Social Psychology, 37 , 1539–1553. ↵
• Watson, J. B., & Rayner, R. (1920). Conditioned emotional reactions. Journal of Experimental Psychology, 3 , 1–14. ↵
• Freud, S. (1961).  Five lectures on psycho-analysis . New York, NY: Norton. ↵
• Pelham, B. W., Carvallo, M., & Jones, J. T. (2005). Implicit egotism. Current Directions in Psychological Science, 14 , 106–110. ↵
• Peterson, C., Seligman, M. E. P., & Vaillant, G. E. (1988). Pessimistic explanatory style is a risk factor for physical illness: A thirty-five year longitudinal study. Journal of Personality and Social Psychology, 55 , 23–27. ↵

Share This Book

• Increase Font Size

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Non-Experimental Research

32 Observational Research

Learning objectives.

• List the various types of observational research methods and distinguish between each.
• Describe the strengths and weakness of each observational research method. 

What Is Observational Research?

The term observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded. The goal of observational research is to describe a variable or set of variables. More generally, the goal is to obtain a snapshot of specific characteristics of an individual, group, or setting. As described previously, observational research is non-experimental because nothing is manipulated or controlled, and as such we cannot arrive at causal conclusions using this approach. The data that are collected in observational research studies are often qualitative in nature but they may also be quantitative or both (mixed-methods). There are several different types of observational methods that will be described below.

Naturalistic Observation

Naturalistic observation  is an observational method that involves observing people’s behavior in the environment in which it typically occurs. Thus naturalistic observation is a type of field research (as opposed to a type of laboratory research). Jane Goodall’s famous research on chimpanzees is a classic example of naturalistic observation. Dr.  Goodall spent three decades observing chimpanzees in their natural environment in East Africa. She examined such things as chimpanzee’s social structure, mating patterns, gender roles, family structure, and care of offspring by observing them in the wild. However, naturalistic observation  could more simply involve observing shoppers in a grocery store, children on a school playground, or psychiatric inpatients in their wards. Researchers engaged in naturalistic observation usually make their observations as unobtrusively as possible so that participants are not aware that they are being studied. Such an approach is called disguised naturalistic observation .  Ethically, this method is considered to be acceptable if the participants remain anonymous and the behavior occurs in a public setting where people would not normally have an expectation of privacy. Grocery shoppers putting items into their shopping carts, for example, are engaged in public behavior that is easily observable by store employees and other shoppers. For this reason, most researchers would consider it ethically acceptable to observe them for a study. On the other hand, one of the arguments against the ethicality of the naturalistic observation of “bathroom behavior” discussed earlier in the book is that people have a reasonable expectation of privacy even in a public restroom and that this expectation was violated. 

In cases where it is not ethical or practical to conduct disguised naturalistic observation, researchers can conduct  undisguised naturalistic observation where the participants are made aware of the researcher presence and monitoring of their behavior. However, one concern with undisguised naturalistic observation is  reactivity. Reactivity refers to when a measure changes participants’ behavior. In the case of undisguised naturalistic observation, the concern with reactivity is that when people know they are being observed and studied, they may act differently than they normally would. This type of reactivity is known as the Hawthorne effect . For instance, you may act much differently in a bar if you know that someone is observing you and recording your behaviors and this would invalidate the study. So disguised observation is less reactive and therefore can have higher validity because people are not aware that their behaviors are being observed and recorded. However, we now know that people often become used to being observed and with time they begin to behave naturally in the researcher’s presence. In other words, over time people habituate to being observed. Think about reality shows like Big Brother or Survivor where people are constantly being observed and recorded. While they may be on their best behavior at first, in a fairly short amount of time they are flirting, having sex, wearing next to nothing, screaming at each other, and occasionally behaving in ways that are embarrassing.

Participant Observation

Another approach to data collection in observational research is participant observation. In  participant observation , researchers become active participants in the group or situation they are studying. Participant observation is very similar to naturalistic observation in that it involves observing people’s behavior in the environment in which it typically occurs. As with naturalistic observation, the data that are collected can include interviews (usually unstructured), notes based on their observations and interactions, documents, photographs, and other artifacts. The only difference between naturalistic observation and participant observation is that researchers engaged in participant observation become active members of the group or situations they are studying. The basic rationale for participant observation is that there may be important information that is only accessible to, or can be interpreted only by, someone who is an active participant in the group or situation. Like naturalistic observation, participant observation can be either disguised or undisguised. In disguised participant observation , the researchers pretend to be members of the social group they are observing and conceal their true identity as researchers.

In a famous example of disguised participant observation, Leon Festinger and his colleagues infiltrated a doomsday cult known as the Seekers, whose members believed that the apocalypse would occur on December 21, 1954. Interested in studying how members of the group would cope psychologically when the prophecy inevitably failed, they carefully recorded the events and reactions of the cult members in the days before and after the supposed end of the world. Unsurprisingly, the cult members did not give up their belief but instead convinced themselves that it was their faith and efforts that saved the world from destruction. Festinger and his colleagues later published a book about this experience, which they used to illustrate the theory of cognitive dissonance (Festinger, Riecken, & Schachter, 1956) [1] .

In contrast with undisguised participant observation ,  the researchers become a part of the group they are studying and they disclose their true identity as researchers to the group under investigation. Once again there are important ethical issues to consider with disguised participant observation.  First no informed consent can be obtained and second deception is being used. The researcher is deceiving the participants by intentionally withholding information about their motivations for being a part of the social group they are studying. But sometimes disguised participation is the only way to access a protective group (like a cult). Further, disguised participant observation is less prone to reactivity than undisguised participant observation. 

Rosenhan’s study (1973) [2]   of the experience of people in a psychiatric ward would be considered disguised participant observation because Rosenhan and his pseudopatients were admitted into psychiatric hospitals on the pretense of being patients so that they could observe the way that psychiatric patients are treated by staff. The staff and other patients were unaware of their true identities as researchers.

Another example of participant observation comes from a study by sociologist Amy Wilkins on a university-based religious organization that emphasized how happy its members were (Wilkins, 2008) [3] . Wilkins spent 12 months attending and participating in the group’s meetings and social events, and she interviewed several group members. In her study, Wilkins identified several ways in which the group “enforced” happiness—for example, by continually talking about happiness, discouraging the expression of negative emotions, and using happiness as a way to distinguish themselves from other groups.

One of the primary benefits of participant observation is that the researchers are in a much better position to understand the viewpoint and experiences of the people they are studying when they are a part of the social group. The primary limitation with this approach is that the mere presence of the observer could affect the behavior of the people being observed. While this is also a concern with naturalistic observation, additional concerns arise when researchers become active members of the social group they are studying because that they may change the social dynamics and/or influence the behavior of the people they are studying. Similarly, if the researcher acts as a participant observer there can be concerns with biases resulting from developing relationships with the participants. Concretely, the researcher may become less objective resulting in more experimenter bias.

Structured Observation

Another observational method is structured observation . Here the investigator makes careful observations of one or more specific behaviors in a particular setting that is more structured than the settings used in naturalistic or participant observation. Often the setting in which the observations are made is not the natural setting. Instead, the researcher may observe people in the laboratory environment. Alternatively, the researcher may observe people in a natural setting (like a classroom setting) that they have structured some way, for instance by introducing some specific task participants are to engage in or by introducing a specific social situation or manipulation.

Structured observation is very similar to naturalistic observation and participant observation in that in all three cases researchers are observing naturally occurring behavior; however, the emphasis in structured observation is on gathering quantitative rather than qualitative data. Researchers using this approach are interested in a limited set of behaviors. This allows them to quantify the behaviors they are observing. In other words, structured observation is less global than naturalistic or participant observation because the researcher engaged in structured observations is interested in a small number of specific behaviors. Therefore, rather than recording everything that happens, the researcher only focuses on very specific behaviors of interest.

Researchers Robert Levine and Ara Norenzayan used structured observation to study differences in the “pace of life” across countries (Levine & Norenzayan, 1999) [4] . One of their measures involved observing pedestrians in a large city to see how long it took them to walk 60 feet. They found that people in some countries walked reliably faster than people in other countries. For example, people in Canada and Sweden covered 60 feet in just under 13 seconds on average, while people in Brazil and Romania took close to 17 seconds. When structured observation  takes place in the complex and even chaotic “real world,” the questions of when, where, and under what conditions the observations will be made, and who exactly will be observed are important to consider. Levine and Norenzayan described their sampling process as follows:

“Male and female walking speed over a distance of 60 feet was measured in at least two locations in main downtown areas in each city. Measurements were taken during main business hours on clear summer days. All locations were flat, unobstructed, had broad sidewalks, and were sufficiently uncrowded to allow pedestrians to move at potentially maximum speeds. To control for the effects of socializing, only pedestrians walking alone were used. Children, individuals with obvious physical handicaps, and window-shoppers were not timed. Thirty-five men and 35 women were timed in most cities.” (p. 186).

Precise specification of the sampling process in this way makes data collection manageable for the observers, and it also provides some control over important extraneous variables. For example, by making their observations on clear summer days in all countries, Levine and Norenzayan controlled for effects of the weather on people’s walking speeds.  In Levine and Norenzayan’s study, measurement was relatively straightforward. They simply measured out a 60-foot distance along a city sidewalk and then used a stopwatch to time participants as they walked over that distance.

As another example, researchers Robert Kraut and Robert Johnston wanted to study bowlers’ reactions to their shots, both when they were facing the pins and then when they turned toward their companions (Kraut & Johnston, 1979) [5] . But what “reactions” should they observe? Based on previous research and their own pilot testing, Kraut and Johnston created a list of reactions that included “closed smile,” “open smile,” “laugh,” “neutral face,” “look down,” “look away,” and “face cover” (covering one’s face with one’s hands). The observers committed this list to memory and then practiced by coding the reactions of bowlers who had been videotaped. During the actual study, the observers spoke into an audio recorder, describing the reactions they observed. Among the most interesting results of this study was that bowlers rarely smiled while they still faced the pins. They were much more likely to smile after they turned toward their companions, suggesting that smiling is not purely an expression of happiness but also a form of social communication.

In yet another example (this one in a laboratory environment), Dov Cohen and his colleagues had observers rate the emotional reactions of participants who had just been deliberately bumped and insulted by a confederate after they dropped off a completed questionnaire at the end of a hallway. The confederate was posing as someone who worked in the same building and who was frustrated by having to close a file drawer twice in order to permit the participants to walk past them (first to drop off the questionnaire at the end of the hallway and once again on their way back to the room where they believed the study they signed up for was taking place). The two observers were positioned at different ends of the hallway so that they could read the participants’ body language and hear anything they might say. Interestingly, the researchers hypothesized that participants from the southern United States, which is one of several places in the world that has a “culture of honor,” would react with more aggression than participants from the northern United States, a prediction that was in fact supported by the observational data (Cohen, Nisbett, Bowdle, & Schwarz, 1996) [6] .

When the observations require a judgment on the part of the observers—as in the studies by Kraut and Johnston and Cohen and his colleagues—a process referred to as   coding is typically required . Coding generally requires clearly defining a set of target behaviors. The observers then categorize participants individually in terms of which behavior they have engaged in and the number of times they engaged in each behavior. The observers might even record the duration of each behavior. The target behaviors must be defined in such a way that guides different observers to code them in the same way. This difficulty with coding illustrates the issue of interrater reliability, as mentioned in Chapter 4. Researchers are expected to demonstrate the interrater reliability of their coding procedure by having multiple raters code the same behaviors independently and then showing that the different observers are in close agreement. Kraut and Johnston, for example, video recorded a subset of their participants’ reactions and had two observers independently code them. The two observers showed that they agreed on the reactions that were exhibited 97% of the time, indicating good interrater reliability.

One of the primary benefits of structured observation is that it is far more efficient than naturalistic and participant observation. Since the researchers are focused on specific behaviors this reduces time and expense. Also, often times the environment is structured to encourage the behaviors of interest which again means that researchers do not have to invest as much time in waiting for the behaviors of interest to naturally occur. Finally, researchers using this approach can clearly exert greater control over the environment. However, when researchers exert more control over the environment it may make the environment less natural which decreases external validity. It is less clear for instance whether structured observations made in a laboratory environment will generalize to a real world environment. Furthermore, since researchers engaged in structured observation are often not disguised there may be more concerns with reactivity.

Case Studies

A  case study   is an in-depth examination of an individual. Sometimes case studies are also completed on social units (e.g., a cult) and events (e.g., a natural disaster). Most commonly in psychology, however, case studies provide a detailed description and analysis of an individual. Often the individual has a rare or unusual condition or disorder or has damage to a specific region of the brain.

Like many observational research methods, case studies tend to be more qualitative in nature. Case study methods involve an in-depth, and often a longitudinal examination of an individual. Depending on the focus of the case study, individuals may or may not be observed in their natural setting. If the natural setting is not what is of interest, then the individual may be brought into a therapist’s office or a researcher’s lab for study. Also, the bulk of the case study report will focus on in-depth descriptions of the person rather than on statistical analyses. With that said some quantitative data may also be included in the write-up of a case study. For instance, an individual’s depression score may be compared to normative scores or their score before and after treatment may be compared. As with other qualitative methods, a variety of different methods and tools can be used to collect information on the case. For instance, interviews, naturalistic observation, structured observation, psychological testing (e.g., IQ test), and/or physiological measurements (e.g., brain scans) may be used to collect information on the individual.

HM is one of the most notorious case studies in psychology. HM suffered from intractable and very severe epilepsy. A surgeon localized HM’s epilepsy to his medial temporal lobe and in 1953 he removed large sections of his hippocampus in an attempt to stop the seizures. The treatment was a success, in that it resolved his epilepsy and his IQ and personality were unaffected. However, the doctors soon realized that HM exhibited a strange form of amnesia, called anterograde amnesia. HM was able to carry out a conversation and he could remember short strings of letters, digits, and words. Basically, his short term memory was preserved. However, HM could not commit new events to memory. He lost the ability to transfer information from his short-term memory to his long term memory, something memory researchers call consolidation. So while he could carry on a conversation with someone, he would completely forget the conversation after it ended. This was an extremely important case study for memory researchers because it suggested that there’s a dissociation between short-term memory and long-term memory, it suggested that these were two different abilities sub-served by different areas of the brain. It also suggested that the temporal lobes are particularly important for consolidating new information (i.e., for transferring information from short-term memory to long-term memory).

The history of psychology is filled with influential cases studies, such as Sigmund Freud’s description of “Anna O.” (see Note 6.1 “The Case of “Anna O.””) and John Watson and Rosalie Rayner’s description of Little Albert (Watson & Rayner, 1920) [7] , who allegedly learned to fear a white rat—along with other furry objects—when the researchers repeatedly made a loud noise every time the rat approached him.

The Case of “Anna O.”

Sigmund Freud used the case of a young woman he called “Anna O.” to illustrate many principles of his theory of psychoanalysis (Freud, 1961) [8] . (Her real name was Bertha Pappenheim, and she was an early feminist who went on to make important contributions to the field of social work.) Anna had come to Freud’s colleague Josef Breuer around 1880 with a variety of odd physical and psychological symptoms. One of them was that for several weeks she was unable to drink any fluids. According to Freud,

She would take up the glass of water that she longed for, but as soon as it touched her lips she would push it away like someone suffering from hydrophobia.…She lived only on fruit, such as melons, etc., so as to lessen her tormenting thirst. (p. 9)

But according to Freud, a breakthrough came one day while Anna was under hypnosis.

[S]he grumbled about her English “lady-companion,” whom she did not care for, and went on to describe, with every sign of disgust, how she had once gone into this lady’s room and how her little dog—horrid creature!—had drunk out of a glass there. The patient had said nothing, as she had wanted to be polite. After giving further energetic expression to the anger she had held back, she asked for something to drink, drank a large quantity of water without any difficulty, and awoke from her hypnosis with the glass at her lips; and thereupon the disturbance vanished, never to return. (p.9)

Freud’s interpretation was that Anna had repressed the memory of this incident along with the emotion that it triggered and that this was what had caused her inability to drink. Furthermore, he believed that her recollection of the incident, along with her expression of the emotion she had repressed, caused the symptom to go away.

As an illustration of Freud’s theory, the case study of Anna O. is quite effective. As evidence for the theory, however, it is essentially worthless. The description provides no way of knowing whether Anna had really repressed the memory of the dog drinking from the glass, whether this repression had caused her inability to drink, or whether recalling this “trauma” relieved the symptom. It is also unclear from this case study how typical or atypical Anna’s experience was.

Case studies are useful because they provide a level of detailed analysis not found in many other research methods and greater insights may be gained from this more detailed analysis. As a result of the case study, the researcher may gain a sharpened understanding of what might become important to look at more extensively in future more controlled research. Case studies are also often the only way to study rare conditions because it may be impossible to find a large enough sample of individuals with the condition to use quantitative methods. Although at first glance a case study of a rare individual might seem to tell us little about ourselves, they often do provide insights into normal behavior. The case of HM provided important insights into the role of the hippocampus in memory consolidation.

However, it is important to note that while case studies can provide insights into certain areas and variables to study, and can be useful in helping develop theories, they should never be used as evidence for theories. In other words, case studies can be used as inspiration to formulate theories and hypotheses, but those hypotheses and theories then need to be formally tested using more rigorous quantitative methods. The reason case studies shouldn’t be used to provide support for theories is that they suffer from problems with both internal and external validity. Case studies lack the proper controls that true experiments contain. As such, they suffer from problems with internal validity, so they cannot be used to determine causation. For instance, during HM’s surgery, the surgeon may have accidentally lesioned another area of HM’s brain (a possibility suggested by the dissection of HM’s brain following his death) and that lesion may have contributed to his inability to consolidate new information. The fact is, with case studies we cannot rule out these sorts of alternative explanations. So, as with all observational methods, case studies do not permit determination of causation. In addition, because case studies are often of a single individual, and typically an abnormal individual, researchers cannot generalize their conclusions to other individuals. Recall that with most research designs there is a trade-off between internal and external validity. With case studies, however, there are problems with both internal validity and external validity. So there are limits both to the ability to determine causation and to generalize the results. A final limitation of case studies is that ample opportunity exists for the theoretical biases of the researcher to color or bias the case description. Indeed, there have been accusations that the woman who studied HM destroyed a lot of her data that were not published and she has been called into question for destroying contradictory data that didn’t support her theory about how memories are consolidated. There is a fascinating New York Times article that describes some of the controversies that ensued after HM’s death and analysis of his brain that can be found at: https://www.nytimes.com/2016/08/07/magazine/the-brain-that-couldnt-remember.html?_r=0

Archival Research

Another approach that is often considered observational research involves analyzing archival data that have already been collected for some other purpose. An example is a study by Brett Pelham and his colleagues on “implicit egotism”—the tendency for people to prefer people, places, and things that are similar to themselves (Pelham, Carvallo, & Jones, 2005) [9] . In one study, they examined Social Security records to show that women with the names Virginia, Georgia, Louise, and Florence were especially likely to have moved to the states of Virginia, Georgia, Louisiana, and Florida, respectively.

As with naturalistic observation, measurement can be more or less straightforward when working with archival data. For example, counting the number of people named Virginia who live in various states based on Social Security records is relatively straightforward. But consider a study by Christopher Peterson and his colleagues on the relationship between optimism and health using data that had been collected many years before for a study on adult development (Peterson, Seligman, & Vaillant, 1988) [10] . In the 1940s, healthy male college students had completed an open-ended questionnaire about difficult wartime experiences. In the late 1980s, Peterson and his colleagues reviewed the men’s questionnaire responses to obtain a measure of explanatory style—their habitual ways of explaining bad events that happen to them. More pessimistic people tend to blame themselves and expect long-term negative consequences that affect many aspects of their lives, while more optimistic people tend to blame outside forces and expect limited negative consequences. To obtain a measure of explanatory style for each participant, the researchers used a procedure in which all negative events mentioned in the questionnaire responses, and any causal explanations for them were identified and written on index cards. These were given to a separate group of raters who rated each explanation in terms of three separate dimensions of optimism-pessimism. These ratings were then averaged to produce an explanatory style score for each participant. The researchers then assessed the statistical relationship between the men’s explanatory style as undergraduate students and archival measures of their health at approximately 60 years of age. The primary result was that the more optimistic the men were as undergraduate students, the healthier they were as older men. Pearson’s  r  was +.25.

This method is an example of  content analysis —a family of systematic approaches to measurement using complex archival data. Just as structured observation requires specifying the behaviors of interest and then noting them as they occur, content analysis requires specifying keywords, phrases, or ideas and then finding all occurrences of them in the data. These occurrences can then be counted, timed (e.g., the amount of time devoted to entertainment topics on the nightly news show), or analyzed in a variety of other ways.

Media Attributions

• What happens when you remove the hippocampus? – Sam Kean by TED-Ed licensed under a standard YouTube License
• Pappenheim 1882  by unknown is in the  Public Domain .
• Festinger, L., Riecken, H., & Schachter, S. (1956). When prophecy fails: A social and psychological study of a modern group that predicted the destruction of the world. University of Minnesota Press. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵
• Wilkins, A. (2008). “Happier than Non-Christians”: Collective emotions and symbolic boundaries among evangelical Christians. Social Psychology Quarterly, 71 , 281–301. ↵
• Levine, R. V., & Norenzayan, A. (1999). The pace of life in 31 countries. Journal of Cross-Cultural Psychology, 30 , 178–205. ↵
• Kraut, R. E., & Johnston, R. E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of Personality and Social Psychology, 37 , 1539–1553. ↵
• Cohen, D., Nisbett, R. E., Bowdle, B. F., & Schwarz, N. (1996). Insult, aggression, and the southern culture of honor: An "experimental ethnography." Journal of Personality and Social Psychology, 70 (5), 945-960. ↵
• Watson, J. B., & Rayner, R. (1920). Conditioned emotional reactions. Journal of Experimental Psychology, 3 , 1–14. ↵
• Freud, S. (1961).  Five lectures on psycho-analysis . New York, NY: Norton. ↵
• Pelham, B. W., Carvallo, M., & Jones, J. T. (2005). Implicit egotism. Current Directions in Psychological Science, 14 , 106–110. ↵
• Peterson, C., Seligman, M. E. P., & Vaillant, G. E. (1988). Pessimistic explanatory style is a risk factor for physical illness: A thirty-five year longitudinal study. Journal of Personality and Social Psychology, 55 , 23–27. ↵

Research that is non-experimental because it focuses on recording systemic observations of behavior in a natural or laboratory setting without manipulating anything.

An observational method that involves observing people’s behavior in the environment in which it typically occurs.

When researchers engage in naturalistic observation by making their observations as unobtrusively as possible so that participants are not aware that they are being studied.

Where the participants are made aware of the researcher presence and monitoring of their behavior.

Refers to when a measure changes participants’ behavior.

In the case of undisguised naturalistic observation, it is a type of reactivity when people know they are being observed and studied, they may act differently than they normally would.

Researchers become active participants in the group or situation they are studying.

Researchers pretend to be members of the social group they are observing and conceal their true identity as researchers.

Researchers become a part of the group they are studying and they disclose their true identity as researchers to the group under investigation.

When a researcher makes careful observations of one or more specific behaviors in a particular setting that is more structured than the settings used in naturalistic or participant observation.

A part of structured observation whereby the observers use a clearly defined set of guidelines to "code" behaviors—assigning specific behaviors they are observing to a category—and count the number of times or the duration that the behavior occurs.

An in-depth examination of an individual.

A family of systematic approaches to measurement using qualitative methods to analyze complex archival data.

Research Methods in Psychology Copyright © 2019 by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

6.6: Observational Research

• Last updated
• Save as PDF
• Page ID 19655

• Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton
• Kwantlen Polytechnic U., Washington State U., & Texas A&M U.—Texarkana

Learning Objectives

• List the various types of observational research methods and distinguish between each.
• Describe the strengths and weakness of each observational research method.

What Is Observational Research?

The term observational research is used to refer to several different types of non-experimental studies in which behavior is systematically observed and recorded. The goal of observational research is to describe a variable or set of variables. More generally, the goal is to obtain a snapshot of specific characteristics of an individual, group, or setting. As described previously, observational research is non-experimental because nothing is manipulated or controlled, and as such we cannot arrive at causal conclusions using this approach. The data that are collected in observational research studies are often qualitative in nature but they may also be quantitative or both (mixed-methods). There are several different types of observational methods that will be described below.

Naturalistic Observation

Naturalistic observation is an observational method that involves observing people’s behavior in the environment in which it typically occurs. Thus naturalistic observation is a type of field research (as opposed to a type of laboratory research). Jane Goodall’s famous research on chimpanzees is a classic example of naturalistic observation. Dr. Goodall spent three decades observing chimpanzees in their natural environment in East Africa. She examined such things as chimpanzee’s social structure, mating patterns, gender roles, family structure, and care of offspring by observing them in the wild. However, naturalistic observation could more simply involve observing shoppers in a grocery store, children on a school playground, or psychiatric inpatients in their wards. Researchers engaged in naturalistic observation usually make their observations as unobtrusively as possible so that participants are not aware that they are being studied. Such an approach is called disguised naturalistic observation. Ethically, this method is considered to be acceptable if the participants remain anonymous and the behavior occurs in a public setting where people would not normally have an expectation of privacy. Grocery shoppers putting items into their shopping carts, for example, are engaged in public behavior that is easily observable by store employees and other shoppers. For this reason, most researchers would consider it ethically acceptable to observe them for a study. On the other hand, one of the arguments against the ethicality of the naturalistic observation of “bathroom behavior” discussed earlier in the book is that people have a reasonable expectation of privacy even in a public restroom and that this expectation was violated.

In cases where it is not ethical or practical to conduct disguised naturalistic observation, researchers can conduct undisguised naturalistic observation where the participants are made aware of the researcher presence and monitoring of their behavior. However, one concern with undisguised naturalistic observation is reactivity. Reactivity refers to when a measure changes participants’ behavior. In the case of undisguised naturalistic observation, the concern with reactivity is that when people know they are being observed and studied, they may act differently than they normally would. This type of reactivity is known as the Hawthorne effect . For instance, you may act much differently in a bar if you know that someone is observing you and recording your behaviors and this would invalidate the study. So disguised observation is less reactive and therefore can have higher validity because people are not aware that their behaviors are being observed and recorded. However, we now know that people often become used to being observed and with time they begin to behave naturally in the researcher’s presence. In other words, over time people habituate to being observed. Think about reality shows like Big Brother or Survivor where people are constantly being observed and recorded. While they may be on their best behavior at first, in a fairly short amount of time they are flirting, having sex, wearing next to nothing, screaming at each other, and occasionally behaving in ways that are embarrassing.

Participant Observation

Another approach to data collection in observational research is participant observation. In participant observation , researchers become active participants in the group or situation they are studying. Participant observation is very similar to naturalistic observation in that it involves observing people’s behavior in the environment in which it typically occurs. As with naturalistic observation, the data that are collected can include interviews (usually unstructured), notes based on their observations and interactions, documents, photographs, and other artifacts. The only difference between naturalistic observation and participant observation is that researchers engaged in participant observation become active members of the group or situations they are studying. The basic rationale for participant observation is that there may be important information that is only accessible to, or can be interpreted only by, someone who is an active participant in the group or situation. Like naturalistic observation, participant observation can be either disguised or undisguised. In disguised participant observation, the researchers pretend to be members of the social group they are observing and conceal their true identity as researchers.

In a famous example of disguised participant observation, Leon Festinger and his colleagues infiltrated a doomsday cult known as the Seekers, whose members believed that the apocalypse would occur on December 21, 1954. Interested in studying how members of the group would cope psychologically when the prophecy inevitably failed, they carefully recorded the events and reactions of the cult members in the days before and after the supposed end of the world. Unsurprisingly, the cult members did not give up their belief but instead convinced themselves that it was their faith and efforts that saved the world from destruction. Festinger and his colleagues later published a book about this experience, which they used to illustrate the theory of cognitive dissonance (Festinger, Riecken, & Schachter, 1956) [1] .

In contrast with undisguised participant observation, the researchers become a part of the group they are studying and they disclose their true identity as researchers to the group under investigation. Once again there are important ethical issues to consider with disguised participant observation. First no informed consent can be obtained and second deception is being used. The researcher is deceiving the participants by intentionally withholding information about their motivations for being a part of the social group they are studying. But sometimes disguised participation is the only way to access a protective group (like a cult). Further, disguised participant observation is less prone to reactivity than undisguised participant observation.

Rosenhan’s study (1973) [2] of the experience of people in a psychiatric ward would be considered disguised participant observation because Rosenhan and his pseudopatients were admitted into psychiatric hospitals on the pretense of being patients so that they could observe the way that psychiatric patients are treated by staff. The staff and other patients were unaware of their true identities as researchers.

Another example of participant observation comes from a study by sociologist Amy Wilkins on a university-based religious organization that emphasized how happy its members were (Wilkins, 2008) [3] . Wilkins spent 12 months attending and participating in the group’s meetings and social events, and she interviewed several group members. In her study, Wilkins identified several ways in which the group “enforced” happiness—for example, by continually talking about happiness, discouraging the expression of negative emotions, and using happiness as a way to distinguish themselves from other groups.

One of the primary benefits of participant observation is that the researchers are in a much better position to understand the viewpoint and experiences of the people they are studying when they are a part of the social group. The primary limitation with this approach is that the mere presence of the observer could affect the behavior of the people being observed. While this is also a concern with naturalistic observation, additional concerns arise when researchers become active members of the social group they are studying because that they may change the social dynamics and/or influence the behavior of the people they are studying. Similarly, if the researcher acts as a participant observer there can be concerns with biases resulting from developing relationships with the participants. Concretely, the researcher may become less objective resulting in more experimenter bias.

Structured Observation

Another observational method is structured observation . Here the investigator makes careful observations of one or more specific behaviors in a particular setting that is more structured than the settings used in naturalistic or participant observation. Often the setting in which the observations are made is not the natural setting. Instead, the researcher may observe people in the laboratory environment. Alternatively, the researcher may observe people in a natural setting (like a classroom setting) that they have structured some way, for instance by introducing some specific task participants are to engage in or by introducing a specific social situation or manipulation.

Structured observation is very similar to naturalistic observation and participant observation in that in all three cases researchers are observing naturally occurring behavior; however, the emphasis in structured observation is on gathering quantitative rather than qualitative data. Researchers using this approach are interested in a limited set of behaviors. This allows them to quantify the behaviors they are observing. In other words, structured observation is less global than naturalistic or participant observation because the researcher engaged in structured observations is interested in a small number of specific behaviors. Therefore, rather than recording everything that happens, the researcher only focuses on very specific behaviors of interest.

Researchers Robert Levine and Ara Norenzayan used structured observation to study differences in the “pace of life” across countries (Levine & Norenzayan, 1999) [4] . One of their measures involved observing pedestrians in a large city to see how long it took them to walk 60 feet. They found that people in some countries walked reliably faster than people in other countries. For example, people in Canada and Sweden covered 60 feet in just under 13 seconds on average, while people in Brazil and Romania took close to 17 seconds. When structured observation takes place in the complex and even chaotic “real world,” the questions of when, where, and under what conditions the observations will be made, and who exactly will be observed are important to consider. Levine and Norenzayan described their sampling process as follows:

“Male and female walking speed over a distance of 60 feet was measured in at least two locations in main downtown areas in each city. Measurements were taken during main business hours on clear summer days. All locations were flat, unobstructed, had broad sidewalks, and were sufficiently uncrowded to allow pedestrians to move at potentially maximum speeds. To control for the effects of socializing, only pedestrians walking alone were used. Children, individuals with obvious physical handicaps, and window-shoppers were not timed. Thirty-five men and 35 women were timed in most cities.” (p. 186).

Precise specification of the sampling process in this way makes data collection manageable for the observers, and it also provides some control over important extraneous variables. For example, by making their observations on clear summer days in all countries, Levine and Norenzayan controlled for effects of the weather on people’s walking speeds. In Levine and Norenzayan’s study, measurement was relatively straightforward. They simply measured out a 60-foot distance along a city sidewalk and then used a stopwatch to time participants as they walked over that distance.

As another example, researchers Robert Kraut and Robert Johnston wanted to study bowlers’ reactions to their shots, both when they were facing the pins and then when they turned toward their companions (Kraut & Johnston, 1979) [5] . But what “reactions” should they observe? Based on previous research and their own pilot testing, Kraut and Johnston created a list of reactions that included “closed smile,” “open smile,” “laugh,” “neutral face,” “look down,” “look away,” and “face cover” (covering one’s face with one’s hands). The observers committed this list to memory and then practiced by coding the reactions of bowlers who had been videotaped. During the actual study, the observers spoke into an audio recorder, describing the reactions they observed. Among the most interesting results of this study was that bowlers rarely smiled while they still faced the pins. They were much more likely to smile after they turned toward their companions, suggesting that smiling is not purely an expression of happiness but also a form of social communication.

In yet another example (this one in a laboratory environment), Dov Cohen and his colleagues had observers rate the emotional reactions of participants who had just been deliberately bumped and insulted by a confederate after they dropped off a completed questionnaire at the end of a hallway. The confederate was posing as someone who worked in the same building and who was frustrated by having to close a file drawer twice in order to permit the participants to walk past them (first to drop off the questionnaire at the end of the hallway and once again on their way back to the room where they believed the study they signed up for was taking place). The two observers were positioned at different ends of the hallway so that they could read the participants’ body language and hear anything they might say. Interestingly, the researchers hypothesized that participants from the southern United States, which is one of several places in the world that has a “culture of honor,” would react with more aggression than participants from the northern United States, a prediction that was in fact supported by the observational data (Cohen, Nisbett, Bowdle, & Schwarz, 1996) [6] .

When the observations require a judgment on the part of the observers—as in the studies by Kraut and Johnston and Cohen and his colleagues—a process referred to as coding is typically required . Coding generally requires clearly defining a set of target behaviors. The observers then categorize participants individually in terms of which behavior they have engaged in and the number of times they engaged in each behavior. The observers might even record the duration of each behavior. The target behaviors must be defined in such a way that guides different observers to code them in the same way. This difficulty with coding illustrates the issue of interrater reliability, as mentioned in Chapter 4. Researchers are expected to demonstrate the interrater reliability of their coding procedure by having multiple raters code the same behaviors independently and then showing that the different observers are in close agreement. Kraut and Johnston, for example, video recorded a subset of their participants’ reactions and had two observers independently code them. The two observers showed that they agreed on the reactions that were exhibited 97% of the time, indicating good interrater reliability.

One of the primary benefits of structured observation is that it is far more efficient than naturalistic and participant observation. Since the researchers are focused on specific behaviors this reduces time and expense. Also, often times the environment is structured to encourage the behaviors of interest which again means that researchers do not have to invest as much time in waiting for the behaviors of interest to naturally occur. Finally, researchers using this approach can clearly exert greater control over the environment. However, when researchers exert more control over the environment it may make the environment less natural which decreases external validity. It is less clear for instance whether structured observations made in a laboratory environment will generalize to a real world environment. Furthermore, since researchers engaged in structured observation are often not disguised there may be more concerns with reactivity.

Case Studies

A case study is an in-depth examination of an individual. Sometimes case studies are also completed on social units (e.g., a cult) and events (e.g., a natural disaster). Most commonly in psychology, however, case studies provide a detailed description and analysis of an individual. Often the individual has a rare or unusual condition or disorder or has damage to a specific region of the brain.

Like many observational research methods, case studies tend to be more qualitative in nature. Case study methods involve an in-depth, and often a longitudinal examination of an individual. Depending on the focus of the case study, individuals may or may not be observed in their natural setting. If the natural setting is not what is of interest, then the individual may be brought into a therapist’s office or a researcher’s lab for study. Also, the bulk of the case study report will focus on in-depth descriptions of the person rather than on statistical analyses. With that said some quantitative data may also be included in the write-up of a case study. For instance, an individual’s depression score may be compared to normative scores or their score before and after treatment may be compared. As with other qualitative methods, a variety of different methods and tools can be used to collect information on the case. For instance, interviews, naturalistic observation, structured observation, psychological testing (e.g., IQ test), and/or physiological measurements (e.g., brain scans) may be used to collect information on the individual.

HM is one of the most notorious case studies in psychology. HM suffered from intractable and very severe epilepsy. A surgeon localized HM’s epilepsy to his medial temporal lobe and in 1953 he removed large sections of his hippocampus in an attempt to stop the seizures. The treatment was a success, in that it resolved his epilepsy and his IQ and personality were unaffected. However, the doctors soon realized that HM exhibited a strange form of amnesia, called anterograde amnesia. HM was able to carry out a conversation and he could remember short strings of letters, digits, and words. Basically, his short term memory was preserved. However, HM could not commit new events to memory. He lost the ability to transfer information from his short-term memory to his long term memory, something memory researchers call consolidation. So while he could carry on a conversation with someone, he would completely forget the conversation after it ended. This was an extremely important case study for memory researchers because it suggested that there’s a dissociation between short-term memory and long-term memory, it suggested that these were two different abilities sub-served by different areas of the brain. It also suggested that the temporal lobes are particularly important for consolidating new information (i.e., for transferring information from short-term memory to long-term memory),

The history of psychology is filled with influential cases studies, such as Sigmund Freud’s description of “Anna O.” (see Note 6.1 “The Case of “Anna O.””) and John Watson and Rosalie Rayner’s description of Little Albert (Watson & Rayner, 1920) [7] , who allegedly learned to fear a white rat—along with other furry objects—when the researchers repeatedly made a loud noise every time the rat approached him.

The Case of “Anna O.”

Sigmund Freud used the case of a young woman he called “Anna O.” to illustrate many principles of his theory of psychoanalysis (Freud, 1961) [8] . (Her real name was Bertha Pappenheim, and she was an early feminist who went on to make important contributions to the field of social work.) Anna had come to Freud’s colleague Josef Breuer around 1880 with a variety of odd physical and psychological symptoms. One of them was that for several weeks she was unable to drink any fluids. According to Freud,

She would take up the glass of water that she longed for, but as soon as it touched her lips she would push it away like someone suffering from hydrophobia.…She lived only on fruit, such as melons, etc., so as to lessen her tormenting thirst. (p. 9)

But according to Freud, a breakthrough came one day while Anna was under hypnosis.

[S]he grumbled about her English “lady-companion,” whom she did not care for, and went on to describe, with every sign of disgust, how she had once gone into this lady’s room and how her little dog—horrid creature!—had drunk out of a glass there. The patient had said nothing, as she had wanted to be polite. After giving further energetic expression to the anger she had held back, she asked for something to drink, drank a large quantity of water without any difficulty, and awoke from her hypnosis with the glass at her lips; and thereupon the disturbance vanished, never to return. (p.9)

Freud’s interpretation was that Anna had repressed the memory of this incident along with the emotion that it triggered and that this was what had caused her inability to drink. Furthermore, he believed that her recollection of the incident, along with her expression of the emotion she had repressed, caused the symptom to go away.

As an illustration of Freud’s theory, the case study of Anna O. is quite effective. As evidence for the theory, however, it is essentially worthless. The description provides no way of knowing whether Anna had really repressed the memory of the dog drinking from the glass, whether this repression had caused her inability to drink, or whether recalling this “trauma” relieved the symptom. It is also unclear from this case study how typical or atypical Anna’s experience was.

Case studies are useful because they provide a level of detailed analysis not found in many other research methods and greater insights may be gained from this more detailed analysis. As a result of the case study, the researcher may gain a sharpened understanding of what might become important to look at more extensively in future more controlled research. Case studies are also often the only way to study rare conditions because it may be impossible to find a large enough sample of individuals with the condition to use quantitative methods. Although at first glance a case study of a rare individual might seem to tell us little about ourselves, they often do provide insights into normal behavior. The case of HM provided important insights into the role of the hippocampus in memory consolidation.

However, it is important to note that while case studies can provide insights into certain areas and variables to study, and can be useful in helping develop theories, they should never be used as evidence for theories. In other words, case studies can be used as inspiration to formulate theories and hypotheses, but those hypotheses and theories then need to be formally tested using more rigorous quantitative methods. The reason case studies shouldn’t be used to provide support for theories is that they suffer from problems with both internal and external validity. Case studies lack the proper controls that true experiments contain. As such, they suffer from problems with internal validity, so they cannot be used to determine causation. For instance, during HM’s surgery, the surgeon may have accidentally lesioned another area of HM’s brain (a possibility suggested by the dissection of HM’s brain following his death) and that lesion may have contributed to his inability to consolidate new information. The fact is, with case studies we cannot rule out these sorts of alternative explanations. So, as with all observational methods, case studies do not permit determination of causation. In addition, because case studies are often of a single individual, and typically an abnormal individual, researchers cannot generalize their conclusions to other individuals. Recall that with most research designs there is a trade-off between internal and external validity. With case studies, however, there are problems with both internal validity and external validity. So there are limits both to the ability to determine causation and to generalize the results. A final limitation of case studies is that ample opportunity exists for the theoretical biases of the researcher to color or bias the case description. Indeed, there have been accusations that the woman who studied HM destroyed a lot of her data that were not published and she has been called into question for destroying contradictory data that didn’t support her theory about how memories are consolidated. There is a fascinating New York Times article that describes some of the controversies that ensued after HM’s death and analysis of his brain that can be found at: https://www.nytimes.com/2016/08/07/magazine/the-brain-that-couldnt-remember.html?_r=0

Archival Research

Another approach that is often considered observational research involves analyzing archival data that have already been collected for some other purpose. An example is a study by Brett Pelham and his colleagues on “implicit egotism”—the tendency for people to prefer people, places, and things that are similar to themselves (Pelham, Carvallo, & Jones, 2005) [9] . In one study, they examined Social Security records to show that women with the names Virginia, Georgia, Louise, and Florence were especially likely to have moved to the states of Virginia, Georgia, Louisiana, and Florida, respectively.

As with naturalistic observation, measurement can be more or less straightforward when working with archival data. For example, counting the number of people named Virginia who live in various states based on Social Security records is relatively straightforward. But consider a study by Christopher Peterson and his colleagues on the relationship between optimism and health using data that had been collected many years before for a study on adult development (Peterson, Seligman, & Vaillant, 1988) [10] . In the 1940s, healthy male college students had completed an open-ended questionnaire about difficult wartime experiences. In the late 1980s, Peterson and his colleagues reviewed the men’s questionnaire responses to obtain a measure of explanatory style—their habitual ways of explaining bad events that happen to them. More pessimistic people tend to blame themselves and expect long-term negative consequences that affect many aspects of their lives, while more optimistic people tend to blame outside forces and expect limited negative consequences. To obtain a measure of explanatory style for each participant, the researchers used a procedure in which all negative events mentioned in the questionnaire responses, and any causal explanations for them were identified and written on index cards. These were given to a separate group of raters who rated each explanation in terms of three separate dimensions of optimism-pessimism. These ratings were then averaged to produce an explanatory style score for each participant. The researchers then assessed the statistical relationship between the men’s explanatory style as undergraduate students and archival measures of their health at approximately 60 years of age. The primary result was that the more optimistic the men were as undergraduate students, the healthier they were as older men. Pearson’s r was +.25.

This method is an example of content analysis —a family of systematic approaches to measurement using complex archival data. Just as structured observation requires specifying the behaviors of interest and then noting them as they occur, content analysis requires specifying keywords, phrases, or ideas and then finding all occurrences of them in the data. These occurrences can then be counted, timed (e.g., the amount of time devoted to entertainment topics on the nightly news show), or analyzed in a variety of other ways.

• Festinger, L., Riecken, H., & Schachter, S. (1956). When prophecy fails: A social and psychological study of a modern group that predicted the destruction of the world. University of Minnesota Press. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵
• Wilkins, A. (2008). “Happier than Non-Christians”: Collective emotions and symbolic boundaries among evangelical Christians. Social Psychology Quarterly, 71 , 281–301. ↵
• Levine, R. V., & Norenzayan, A. (1999). The pace of life in 31 countries. Journal of Cross-Cultural Psychology, 30 , 178–205. ↵
• Kraut, R. E., & Johnston, R. E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of Personality and Social Psychology, 37 , 1539–1553. ↵
• Cohen, D., Nisbett, R. E., Bowdle, B. F., & Schwarz, N. (1996). Insult, aggression, and the southern culture of honor: An "experimental ethnography." Journal of Personality and Social Psychology, 70 (5), 945-960. ↵
• Watson, J. B., & Rayner, R. (1920). Conditioned emotional reactions. Journal of Experimental Psychology, 3 , 1–14. ↵
• Freud, S. (1961). Five lectures on psycho-analysis . New York, NY: Norton. ↵
• Pelham, B. W., Carvallo, M., & Jones, J. T. (2005). Implicit egotism. Current Directions in Psychological Science, 14 , 106–110. ↵
• Peterson, C., Seligman, M. E. P., & Vaillant, G. E. (1988). Pessimistic explanatory style is a risk factor for physical illness: A thirty-five year longitudinal study. Journal of Personality and Social Psychology, 55 , 23–27. ↵

The Ultimate Guide to Qualitative Research - Part 1: The Basics

• Introduction and overview
• What is qualitative research?
• What is qualitative data?
• Examples of qualitative data
• Qualitative vs. quantitative research
• Mixed methods
• Qualitative research preparation
• Theoretical perspective
• Theoretical framework
• Literature reviews
• Research question
• Conceptual framework
• Conceptual vs. theoretical framework

Data collection

• Qualitative research methods
• Focus groups

What is observational research?

Uses for observational research, observations in research, the different types of observational research, conducting observational studies, uses with other methods, challenges of observational studies.

• Case studies
• Ethnographical research
• Ethical considerations
• Confidentiality and privacy
• Power dynamics
• Reflexivity

Observational research

Observational research is a social research technique that involves the direct observation of phenomena in their natural setting.

An observational study is a non-experimental method to examine how research participants behave. Observational research is typically associated with qualitative methods , where the data ultimately require some reorganization and analysis .

Contemporary research is often associated with controlled experiments or randomized controlled trials, which involve testing or developing a theory in a controlled setting. Such an approach is appropriate for many physical and material sciences that rely on objective concepts such as the melting point of substances or the mass of objects. On the other hand, observational studies help capture socially constructed or subjective phenomena whose fundamental essence might change when taken out of their natural setting.

What is an example of observational research?

For example, imagine a study where you want to understand the actions and behaviors of single parents taking care of children. A controlled experiment might prove challenging, given the possibility that the behaviors of parents and their children will change if you isolate them in a lab or an otherwise unfamiliar context.

Instead, researchers pursuing such inquiries can observe participants in their natural environment, collecting data on what people do, say, and behave in interaction with others. Non-experimental research methods like observation are less about testing theories than learning something new to contribute to theories.

The goal of the observational study is to collect data about what people do and say. Observational data is helpful in several fields:

• market research
• health services research
• educational research
• user research

Observational studies are valuable in any domain where researchers want to learn about people's actions and behaviors in a natural setting. For example, observational studies in market research might seek out information about the target market of a product or service by identifying the needs or problems of prospective consumers. In medical contexts, observers might be interested in how patients cope with a particular medical treatment or interact with doctors and nurses under certain conditions.

Researchers may still be hung up on science being all about experiments to the point where they may overlook the empirical contribution that observations bring to research and theory. With that in mind, let's look at the strengths and weaknesses of observations in research .

Strengths of observational research

Observational research, especially those conducted in natural settings, can generate more insightful knowledge about social processes or rituals that one cannot fully understand by reading a plain-text description in a book or an online resource. Think about a cookbook with recipes, then think about a series of videos showing a cook making the same recipes. Both are informative, but the videos are often easier to understand as the cook can describe the recipe and show how to follow the steps at the same time. When you can observe what is happening, you can emulate the process for yourself.

Observing also allows researchers to create rich data about phenomena that cannot be explained through numbers. The quality of a theatrical performance, for example, cannot easily be reduced to a set of numbers. Qualitatively, a researcher can analyze aspects gleaned from observing that performance and create a working theory about the quality of that performance. Through data analysis, the researcher can identify patterns related to the aesthetics and creativity of the performance to provide a framework to judge the quality of other performances.

Weaknesses of observational research

Science is about organizing knowledge for the purposes of identifying the aspects of a concept or of determining cause-and-effect relationships between different phenomena. Experiments look to empirically accomplish these tasks by controlling certain variables to determine how other variables change under changing conditions. Those conducting observational research, on the other hand, exert no such control, which makes replication by other researchers difficult or even impossible when observing dynamic environments.

Observational studies take on various forms. There are various types of observational research, each of which has strengths and weaknesses. These types are organized below by the extent to which an experimenter intrudes upon or controls the environment.

Naturalistic observation

Naturalistic observation refers to a method where researchers study participants in their natural environment without manipulating variables or intervening in any way. It provides a realistic snapshot of behavior as it occurs in real-life settings, thereby enhancing ecological validity.

Examples of naturalistic observation include people-watching in public places, observing animal behaviors in the wild, and longitudinally studying children's social development at school. This method can reveal insights about behavior and relationships that might not surface in experimental designs, such as patterns of social interaction, routines, or responses to environmental changes.

Participant observation

Participant observation is similar to naturalistic observation, except that the researcher is part of the natural environment they are observing. In such studies, the researcher is also interested in rituals or cultural practices where they can only determine their value by actually experiencing them firsthand. For example, any individual can understand the basic rules of baseball by watching a game or following a team. Participant observation, on the other hand, allows for direct participation to develop a better sense of team dynamics and relationships among fellow players.

Most commonly, this process involves the researcher inserting themselves into a group to observe behavior that otherwise would not be accessible by observing from afar. Participant observation can capture rich data from the interactions with those who are observed to the reflections of the researchers themselves.

Controlled observation

A more structured observation involves capturing the behaviors of research participants in an isolated environment. Case-control studies have a greater resemblance to experimental research while still relying on observational research methods. Researchers may utilize a case-control study when they want to establish the causation of a particular phenomenon.

For example, a researcher may want to establish a structured observation of a control group and an experimental group, each with randomly assigned research participants, to observe the effects of variables such as distractions on people completing a particular task. By subjecting the experimental group to distractions such as noise and lights, researchers can observe the time it takes participants to complete a task and determine causation accordingly.

Longitudinal study

Among the different types of observational research, this observational method is quite arduous and time-consuming as it requires observation of people or events over extended periods. Researchers should consider longitudinal observations when their inquiry involves variables that can only be observed over time. After all, variables such as literacy development or weight loss cannot be fully captured in any particular moment of observation. Longitudinal studies keep track of the same research participants or events through multiple observations to document changes to or patterns in behavior.

A cohort study is a specific type of longitudinal study where researchers observe participants with similar traits (e.g., a similar risk factor or biological characteristic). Cohort studies aim to observe multiple participants over time to identify a relationship between observed phenomena and a common characteristic.

All forms of observational or field research benefit extensively from the special capabilities of qualitative research tools like ATLAS.ti . Our software can accommodate the major forms of data , such as text, audio, video, and images . The ATLAS.ti platform can help you organize all your observations , whatever method you employ.

Whatever your research, make it happen with ATLAS.ti.

Powerful analytical tools at your fingertips. Try for free by clicking here.

Like any other study design, observational studies begin by posing research questions . Inquiries common when employing observational methods include the study of different cultures, interactions between people from different communities, or people in particular circumstances warranting further study (e.g., people coping with a rare disease).

Generally, a research question that seeks to learn more about a relatively unfamiliar phenomenon would be best suited for observational research. On the other hand, quantitative methods or experimental research methods may be more suitable for inquiries where the theory about a social phenomenon is fairly established.

Study design

Study design for observational research involves thinking about who to observe, where they should be observed, and what the researcher should look for during observation. Many events can occur in a natural, dynamic environment in a short period, so it is challenging to document everything. If the researcher knows what they want to observe, they can pursue a structured observation which involves taking notes on a limited set of phenomena.

The actual data collection for an observational study can take several forms. Note-taking is common in observational research, where the researcher writes down what they see during the course of their observation. The goal of this method is to provide a record of the events that are observed to determine patterns and themes useful for theoretical development.

Observation can also involve taking pictures or recording audio for a richer understanding of social phenomena. Video recorded from observations can also provide data that the researcher can use to document the facial expressions, gestures, and other body language of research participants.

Note that there are ethical considerations when conducting observational research. Researchers should respect the privacy and confidentiality of their research participants to ensure they are not adversely affected by the research. Researchers should obtain informed consent from participants before any observation where possible.

Observational studies can be supplemented with other methods to further contextualize the research inquiry. Researchers can conduct interviews or focus groups with research participants to gather data about what they recall about their actions and behaviors in a natural setting. Focus groups, in particular, provide further opportunities to observe participants interacting with each other. In both cases, these research methods are ideal where the researcher needs to follow up with research participants about the evidence they've collected regarding their behaviors or actions.

As with many other methods in qualitative research , conducting an observational study is time-consuming. While experimental methods can quickly generate data , observational research relies on documenting events and interactions in detail that can be analyzed for theoretical development.

Unstructured data

One common critique of observational research is that it lacks the structure inherent to experimental research, which has concepts such as selection bias and interrater reliability to ensure research quality. On the other hand, qualitative research relies on the assumption that the study and its data are presented transparently and honestly . Under this principle, researchers are responsible for convincing their audiences that the assertions they make are connected empirically to the observations they have made and the data they have collected.

Researcher bias

In most qualitative research, but especially in observational research, the most important data collection instrument is the researcher themselves. This raises issues of bias and subjectivity influencing the collection and interpretation of the data.

Later in this guide, there will be discussion of reflexivity , a concept where the researcher comprehensively accounts for their place in the research relative to others in the environment. For now, it's important to know that social science researchers can and do adequately address critiques of researcher bias to maintain the empirical nature of their observational research.

Conduct your observational study with ATLAS.ti

From the inception of your study to disseminating to your research audience, get it all done with our powerful software platform. Start with a free trial.

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature

Bibliography

• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

15 Observational Methods

Jamie M. Ostrov, Department of Psychology, University at Buffalo, The State University of New York, Buffalo, NY

Emily J. Hart, Department of Psychology, University at Buffalo, The State University of New York, Buffalo, NY

• Published: 01 October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Systematic observational methods require clearly defined codes, structured sampling and recording procedures, and are subject to rigorous psychometric analysis. We review best practices in each of these areas with attention to the application of these methods for addressing empirical questions that quantitative researchers may posit. Special focus is placed on the selection of appropriate observational methods and coding systems as well as on the analysis of reliability and validity. The use of technology to facilitate the collection and analysis of observational data is discussed. Ethical considerations and future directions are raised.

Introduction

Systematic observational methods have been a common technique employed by psychologists studying human and animal behavior since the inception of our field, and yet best practices for the use of observational instruments ( see Table 15.1 ) are often not known or adopted by researchers in our field. As such, the quality of observational research varies widely, and thus, it is our goal in the present chapter to review and explicitly define the standards of practice for this important methodological tool in the psychological sciences. Bakeman and Gottman (1987) have previously defined observational methods to include the a priori use of operationally defined behavioral codes by observers who have achieved interobserver reliability. Importantly, the setting or context is not what defines a method as

being systematic ( Pellegrini, 2004 ). That is, systematic observations may be conducted in the laboratory, schools, workplace, public spaces and coded

live or via recordings/transcripts. Therefore, having clear definitions and sampling/recording rules as well as reliable codes delineates informal, unsystematic observation from systematic observation. We also distinguish between the use of nonsystematic field notes and other data collection techniques that are often used in qualitative studies by ethologists and educational practitioners in naturalistic contexts and only include a review and analysis of systematic observational methods (Pellegrini, Ostrov, Roseth, Solberg, & Dupuis, in press).

Nonsystematic sampling techniques such as Ad libitum (i.e., ad lib) in which there are no a priori systematic sampling or recording rules are often used by researchers as a part of pilot testing and help to inform the development of systematic observational coding systems ( Pellegrini, 2004 ). Thus, ad lib sampling approaches are important to understand the context and nature of the behaviors under study, but they will not be discussed further in this review. Observational methods may be used in a variety of designs from correlational and quasi-experimental to experimental and even randomized trial designs ( Bakeman & Gnisci, 2006 ). However, it is more typical to find systematic observational methods used outside the laboratory to maximize ecological validity and, thus, less likely as part of experimental manipulations ( Bakeman & Gnisci, 2006 ). The current review will be relevant to all research designs with a focus on those methods that are well designed for quantitative data analysis.

History of Observational Methods

The use of systematic observational methods has been used extensively by psychologists throughout the history of our field to examine various empirical questions ( see   Langfeld, 1913 ). One of the first documented cases of systematic observational methods in the extant literature was from a study by Goodenough (1930) and was part of an increasing trend in the systematic study of young children as part of the Child Welfare Movement in the United States, which was supported by the National Research Council (for review, see   Arrington, 1943 ). In fact, her seminal work was also one of the first studies in psychology to be published using time sampling ( see Sampling section below) observational procedures ( Arrington, 1943 ). In her classic work (appearing in the first issue of Child Development ), Florence L. Goodenough reported on several observational studies conducted in her laboratory at the Institute of Child Welfare (now Institute of Child Development) of the University of Minnesota. This study highlights several best practices that are still endorsed today. For example, careful pilot testing of the observational codes was conducted, and revisions were made to generate mutually exclusive codes ( see Coding section below) and reliable distinctions between the categories. In addition, observations of each child’s physical activity were conducted only once per day and only by one observer at a time so that observations of behavior were conducted independent of one another. Goodenough (1930) carefully defined the a priori categories or observational codes and demonstrated interobserver reliability for each of these codes. Finally, Goodenough (1930) described the justification for her observational procedures and discussed alternative techniques (e.g., the optimum duration for an interval within a time-sampling procedure). There are other well-known examples of systematic observation conducted by contemporaries of Goodenough, including Parten’s (1932) study of young children’s play behavior, which also illustrate best practices (e.g., clearly defined, mutually exclusive observational codes; rules designed to maintain independence of sampling and decrease observer error). Some of the earliest observational studies focused on either children or non-human animals (e.g., Crawford, 1942 ), as other techniques for studying behavior (and often social domains of study) were either not as well suited for the research questions or not available at the time. Today, systematic observational methods are used in research and applied settings ( Pellegrini, 2001 ) and relevant for training in all domains and subdisciplines of the social and behavioral sciences ( Krehbiel & Lewis, 1994 ).

Sampling and Recording Rules

Systematic observational systems follow various sampling and recording rules that are designed for different contexts and research questions. The following section includes a review of the central sampling and recording rules that quantitative scholars would use for conducting systematic observations ( see Table 15.2 for a summary of the strengths and weaknesses of each approach). Recently adopted best practices for direct systematic observation are relevant for each of these types of observational methods, and they are briefly reviewed here. These practices, which were first introduced by Hintze, Volpe, and Shapiro (2002) , include (1) the observational system is designed to measure well-defined behaviors; (2) the behaviors are operationally defined a priori ; (3) observations are recorded using objective, standardized (i.e., manualized training protocols) sampling procedures and recording rules; (4) the context and timing of sampling is explicitly determined; and (5) scoring and coding of data are conducted in a standardized fashion ( see Leff & Lakin, 2005 , p. 476).

Time Sampling

A time-dependent observational procedure in which the researcher a priori divides the behavior stream into discrete intervals and each time interval is scored for the presence or absence of the behavior in question is defined as a time sampling observational approach. That is, the time interval is the unit coded ( Bakeman & Gottman, 1987 ). Time sampling procedures may be conceptualized as either 0/1 (i.e., absent/present or nonoccurrence/occurrence) or continuous in nature. A time sampling procedure is an efficient method of sampling, as multiple data points may be collected from a single participant in a short period of time. Time sampling is well suited for measuring rather discrete behaviors, such as overt behaviors (e.g., on task and off task behavior in classrooms), or with behaviors that are frequently occurring. For example, a recent study of the frequency of various behaviors (e.g., off task behavior, noncompliance) during several naturalistic activities in 30 children with various psychiatric diagnoses used a reliable 0/1 time sampling approach with a 15-second interval ( Quake-Rapp, Miller, Ananthan, & Chiu, 2008 ). Alternatively, time sampling is not well designed for infrequently occurring events or events that are long in duration ( Slee, 1987 ). A clear advantage is that time sampling is relatively inexpensive because it is an efficient use of the research assistant ( Bakeman & Gottman, 1987 ). Further, 0/1 sampling is also easier for the observer than alternatives such as instantaneous sampling, in which the research assistant notes if the behavior is present at a precise moment in time rather than it occurring during a larger interval of time. A major disadvantage of the time sampling approach is that the researcher delineates the particular time interval and therefore arbitrarily categorizes the behavior into discrete artificial units of time that may or may not be meaningful ( Slee, 1987 ). Moreover, some behaviors may exceed the often brief interval of time that is selected for the sampling. Thus, it is crucial to carefully justify the interval that is selected. The intervals are often brief and the behaviors in question should be readily apparent and easily observable by trained research assistants. If frequency estimates are to be obtained, then the interval in question needs to be sufficiently brief so that an accurate assessment can be made. That is, typically with an interval approach, a maximum of one behavior is recorded during an interval even if the behavior independently occurs more frequently during this interval ( Slee, 1987 ). Thus, special attention needs to be given to the pilot testing of the observational scheme and various durations of the interval if frequency assessments are desired.

Time sampling procedures are used in a range of settings and studies to test various empirical questions that often have applied significance. For example, Macintosh and Dissanayake (2006) adopted a 0/1 time sampling technique to assess spontaneous social interactions in school-aged children with high-functioning autism or Asperger’s disorder as well as typically developing children. Observations were conducted in the schoolyard. For each timed interval of 30 seconds, one type of behavior (e.g., parallel play) from a particular behavioral domain (e.g., social participation) was coded. For reliability purposes, a second observer made independent ratings for 20% of the entire sample. Intraclass correlation reliability coefficients were all acceptable for each type of behavior (0.78–0.99) with the exception of nonverbal interaction (i.e., gestures; 0.58), which are often difficult to reliably assess in live settings ( see also   Ostrov & Keating, 2004 ). Results meaningfully distinguished between the typically developing children and the clinical groups and revealed few differences between the two clinical groups, supporting the use of time sampling as a means to discriminate between clinical and nonclinical groups ( Macintosh & Dissanayake, 2006 ). Time sampling procedures have several other applications and clinical considerations. For example, time sampling methods may differentially affect how treatment effects are interpreted ( Meany-Daboul, Roscoe, Bourret, & Ahearn, 2007 ) and may be appropriate for classroom-based research that tests adherence to educational policies intended to aid students with special needs ( Jackson & Neel, 2006 ; Soukup, Wehmeyer, Bashinski, & Boyaird, 2007 ).

Event Sampling

Event-based sampling is also known as behavior sampling and permits a researcher to study the frequency, duration, latency, and intensity of the behavior under study ( Pellegrini, 2004 ). Essentially, unlike time sampling, event sampling is a type of observational sampling in which the events are time-independent and the behavior is the unit of analysis ( Bakeman & Gottman, 1987 ). Event sampling allows the behavior to remain as part of the naturally occurring phenomenon and may unfold in a manner generally consistent with the timing of the behavior in the natural setting. This type of sampling also can be efficient in terms of the total amount of time needed for observations. Unlike other sampling techniques (e.g., time sampling), a third advantage is that event sampling may be used when the construct under study is either frequently or infrequently occurring ( Slee, 1987 ). There are some clear disadvantages to event-based sampling procedures, and this may be a reason that it is less commonly seen in the literature. First, it is sometimes challenging to delineate the independence of events—that is, the researcher must specify when one event ends and the next event begins. Second, event sampling does not lend itself well to coding of dyadic interactions such as parent–child or romantic partner relations in which there is a fair amount of interdependence between the participants ( Slee, 1987 ).

Event sampling also has wide applicability and has even been used to understand the propensity to violence at sporting events. For example, Bowker et al. (2009) used an event-sampling approach to examine spectator comments at youth hockey games in a large Canadian city. A group of five observers attended 69 hockey games played by youth in two age groups: 11–12 years and 13–14 years. Verbal comments were coded as positive, negative, corrective, or neutral and rated for intensity. Most of the comments elicited by spectators were positively toned. The valence of spectator comments was influenced by gender (i.e., the gender of the children playing) and the purpose for which the game was being played (i.e., competitive or recreational). These results support the utility of event sampling at social and athletic events, where particular behaviors are likely to occur during a finite period of time. Time sampling may not be appropriate in such circumstances because of the presence of a high concentration of individuals in a single setting and many potential interruptions arising from the nature of the activity.

Participant Observation

Although participant observation has been more frequently used with nonsystematic field observation and in disciplines that focus on qualitative methods, it is possible to conduct systematic participant observation as part of quantitative studies. Systematic participant observation has been the method of choice for behaviors of interest that require “an insider’s perspective” ( Pellegrini, 2004 , p. 288) or for contexts in which the sampling period may be long and informal. Moreover, this method is well suited for the use of more global observational ratings that sample events. This procedure has wide applicability, and participant observation has an extensive history of successful use from studies of children with behavioral problems at summer camps in clinical psychology (e.g., Newcomb, 1931 ; Pelham et al., 2000 ) to worker stress in organizational psychology (e.g., Länsisalmi, Peiró, & Kivimäki, 2000 ). For example, a recent study of children diagnosed with disruptive behavioral disorders and enrolled in a summer treatment program used staff counselors to complete daily participant observations of social behaviors of the children while they engaged in various camp activities ( Lopez-Williams et al., 2005 ). A second study of social competence among reunited adolescents ( M a g e = 1 5 . 5 years) who had attended a research-based summer camp when they were 10 years old revealed the predictive validity of participant observer (i.e., camp counselor) ratings of social skills ( Englund, Levy, Hyson, & Sroufe, 2000 ). The validity of the participant observations of social competence when the participants were 10 years old was determined by revealing significant prospective correlations with a group-problem solving task that was videotaped and coded by two independent raters along several dimensions (e.g., self-confidence, agency, overall social competence) when the participants were 15 years old. The results support the use of participant observations in studying the development and stability of complex, multifaceted constructs like social competence.

Focal Sampling

Focal person sampling involves selecting (typically at random from a roster of participants) one participant and observing the individual for a defined time period. For each sampling interval (ranges vary depending on the question of interest), the observer records all relevant behaviors of the focal person. As we have previously discussed ( see Pellegrini et al., in press), for studies of dyads or small groups, the sampling interval should be as long as the typical interaction or displayed behavior of interest. For example, in our work, we study the display of relational aggression (i.e., the use of the relationship as the means of harm via social exclusion, withdrawing friendship, spreading malicious rumors), and given the nature of these behaviors, we have found that an interval of 10 minutes is a reasonable interval for assessing the intent for harm as well as the subtle nature of these peer interactions ( Ostrov, 2008 ; Ostrov & Keating, 2004 ).

Focal sampling may technically use continuous (e.g., Fagot & Hagan, 1985 ; Laursen & Hartup, 1989 ), 0/1 (e.g., Hall & McGregor, 2000 ; Harrist & Bradley, 2003 ), or instantaneous recording rules ( see   Pellegrini, 2004 ). However, focal sampling often uses continuous recording procedures because it permits the simultaneous coding of various behaviors, sequences of behaviors, and interactions with multiple partners in a live setting (e.g., Arsenio & Lover, 1997 ; Keating & Heltman, 1994 ). For example, in our observational studies of relational aggression among young children, we always have used focal sampling with continuous recording given the somewhat covert nature of the behaviors we have targeted for observation, which require a longer period of direct assessment to decipher and appropriately record the behaviors ( Ostrov & Keating, 2004 ). Focal participant sampling is often conducted across multiple days and contexts to better capture the true nature of the behavior rather than any state-dependent artifacts. Given the amount of time and the continuous nature of the recordings, this technique permits the recording of behavior that is a close approximation to real-time recording, and a researcher may recreate the behavior of the focal participants with a high degree of accuracy (Pellegrini et al., in press). For example, we observe children in their naturally occurring play contexts on 8 separate days, and they are only ever observed once per day to maintain independence of the data. Thus, in our work, each participant is observed for 80 minutes (8 sessions at 10 minutes each session). More specifically, a study of 120 children resulted in more than 370 hours of observation across the two time-points of the short-term longitudinal study ( Ostrov, 2008 ). Therefore, time is a major cost of focal sampling because of the large number of independent observations typically conducted with this approach. Focal sampling may also be used with 0/1 or instantaneous sampling as recording procedures, but this is rarely done. As previously mentioned, both of these recording procedures require an a priori specified time interval, which is usually relatively brief (i.e., 1–10 seconds). Instantaneous recording is typically used only with scan sampling procedures ( see Scan Sampling section below). 0/1 time sampling is not usually used with focal sampling because we are often interested in assessing the true frequency of behaviors that may not be obtained with this procedure (i.e., an independent behavior could occur once or more than once during a set interval, but with 0/1 coding only one point is scored).

Despite the emphasis on the use of these methods for studying basic social behavior, focal sampling procedures may be used in a wide range of studies. It is common in the literature to find focal participant sampling studies on a range of social behavior topics: social dominance in children ( Keating & Heltman, 1994 ) and adults ( Ostrov & Collins, 2007 ), play behavior ( Pellegrini, 1989 ), emotion and aggression ( Arsenio & Lover, 1997 ), conflict ( Laursen & Hartup, 1989 ), and peer relations with young children and non-human primates (e.g., Hinde, Easton, & Meller, 1984 ; Silk, Cheney, & Seyfarth, 1996 ). However, there are many practical applications of focal participant sampling ( see   Leff & Lakin, 2005 ; Pellegrini, 2001 ). For example, applied studies have been conducted that have used these observational techniques for examining the adjustment of children with special needs in elementary schools ( Hall & McGregor, 2000 ), peer victimization in early adolescence ( Pellegrini & Bartini, 2000 ), and for testing the efficacy of randomized behavioral interventions (e.g., Harrist & Bradley, 2003 ; Ostrov et al., 2009 ).

Scan Sampling

Instantaneous or scan sampling is a more efficient observational procedure than focal sampling. Scan sampling exclusively relies on instantaneous recording rules ( Pellegrini, 2001 ). With this procedure the observer scans the entire observation field for a possible behavior or event for a particular period of time. If an event is noted during that scan, then it is recorded. Typically, a number of discrete scans occur across a number of days to maximize the independence of the data. A participant’s data is usually summed across the scans to yield a behavioral score for the construct of interest. A concern with this approach is that it may not accurately assess the true frequency of behaviors if spacing is not adequate between the scans ( Pellegrini, 2004 ). Moreover, given the typical approach in which scans are conducted on an entire reference group in their natural context, behaviors that are selected for this approach must be readily apparent, discrete, and overt behaviors that require typically only a few seconds to observe. In our own field, McNeilly-Choque, Hart, Robinson, Nelson, and Olsen (1996) conducted a study of young children’s aggressive behavior in which they used a random scan sampling method that yielded 100 five-second scans during a 5- to 7-week period, resulting in 8 minutes of total observation per participant ( McNeilly-Choque, Hart, Robinson, Nelson, & Olsen, 1996 ). Thus, this study demonstrated the feasibility and efficiency of systematic scan sampling observations of aggressive behavior on the playground.

Semi-Structured Observations

Analog tasks or semi-structured observations, involving controlled simulations or analog situations, are observational tasks designed to mimic naturalistic conditions. Semi-structured observational procedures are another observational paradigm well suited for low base rate events. The recording and coding procedures are often identical to the procedures an observer would use in a naturalistic setting; however, the context in which the behaviors emerge is different. Often analog tasks are completed in a laboratory or similarly controlled setting and are videotaped for subsequent coding by unaware observers. Thus, analog observational paradigms permit a great deal of experimental control/standardization of procedures, and with the use of videotapes, observers are able to objectively code the session using the same recording rules as permitted in other contexts. A clear advantage of these procedures is that they are efficient and require less cost and time spent observing participants. If the study is not designed well, then a major disadvantage is a lack of ecological validity (i.e., degree to which the context in which the research is conducted parallels the real-life experience of the participants), and poor generalizability of the findings is possible. Moreover, a relatively small sampling of behavior does not provide for a true frequency of behavior or for a representative sample of behavior with many interaction partners (i.e., the researcher is not able to examine individual–partner interactions). Other researchers have addressed this concern by using a “round robin” approach in which each participant completes an analog session with several (or all) other member of the reference group, which may improve the validity of the approach but, of course, adds a great deal of time and expense ( see   Hawley & Little, 1999 ).

In our own research we have used a semi-structured observational paradigm to provide an efficient estimate of young children’s aggressive behavior. To this end, we created a brief (9-minute) analog situation to observe various aggressive and prosocial behaviors (i.e., within dyads or triads) in early childhood ( Ostrov & Keating, 2004 ; Ostrov, Woods, Jansen, Casas, & Crick, 2004 ). The procedures and a review of the psychometric findings are described extensively elsewhere (e.g., Ostrov & Godleski, 2007 ), but essentially, each assessment includes three trials of 3 minutes each. For each trial, the children are given the same developmentally appropriate picture to color (e.g., Winnie the Pooh). For triads, three crayons are placed on the table equidistant from all participants, and only one crayon is the functional instrument (e.g., orange crayon for Winnie the Pooh) and two are functionally useless white crayons. At the end of the trial, a new picture and new crayons are placed on the table. This procedure is designed to produce mild conflict among the children and was developed to permit the children to engage in a variety of behaviors: prosocial behavior (e.g., sharing the one functional crayon or breaking into pieces to share), relational aggression (e.g., telling the child they will not be their friend anymore unless they give them the crayon), and physical aggression (e.g., taking the crayon away from someone else). The analog task was designed to be developmentally appropriate and resemble everyday conflict interactions concerning limited resources that young children experience in their typical preschool classroom. Highly trained research assistants monitored the entire session and intervened if needed to guarantee the safety of all participants and reduce the likelihood of participant distress. Moreover, at the end of the session, the children were each individually given access to a full box of crayons to diminish any distress and they were praised for their performance ( see   Ostrov et al., 2004 ). This paradigm is thus designed to elicit the behavioral constructs of interest in a more controlled environment than free play yet ensures the ethical treatment of participants.

One way to demonstrate the ecological validity of semi-structured observations is to correlate behaviors observed in a semi-structured context with behaviors observed in a more naturalistic context. For example, Coie and Kupersmidt (1983) found that social status in experimentally contrived playgroups comprised of unfamiliar peers matched social status in the classroom, supporting the validity of a contrived playgroup paradigm for studying social development ( see also   Dodge, 1983 ). Similarly, our own brief semi-structured observational paradigm (i.e., coloring task) has been shown to significantly predict observational scores collected from concurrently assessed naturalistic (i.e., classroom and playground free play) focal child observations with continuous recording ( r = 0 . 4 8 ) and to predict future (i.e., 12 months later) behavior in naturalistic contexts at moderate levels ( see   Ostrov et al., 2004 ).

Methods of Recording

Various methods of recording (i.e., checklist, detailed records, or observation forms) vary widely and should be based on the type of recording procedures that a researcher adopts. For example, time sampling (i.e., 0/1) and instantaneous or scan sampling procedures are well suited for checklist forms in which the prescribed intervals simply receive a check or a precise code indicating the occurrence or absence of the behavior in question. However, focal participant sampling often requires observation forms that permit greater detail and several codes that are recorded either simultaneously or in close temporal proximity, and, as such, a form that includes the behaviors or events of interest with space for recording the behavior in detail may be needed (for example forms and templates, see Pellegrini et al., in press). A general concern here is that the more time spent writing details about the behavior/event removes the observer’s attention from the participants and important details may be lost. Some observational procedures like time sampling provide the observer with a set period of time after the interval for recording behavior. In general, the easier the observation form is to complete, the less room there is for error. With that said, checklists often do not permit systematic reviews for accuracy of codes by the master trainer. For example, observers that are observing the same participant as part of a reliability check could both code a behavior as “PA” for physical aggression when in fact one research assistant observed a “hit” and the other observed a “kick,” which, depending on the observational system, may be different and might not warrant a positive match or agreement. Thus, depending on the coding scheme and intentions of the researcher, these may artificially match for reliability purposes when in fact they were closely related but discrete behaviors. Finally, if observers record some written details about the event, they may inform subsequent decision rules concerning whether a recorded behavior from observer 1 matches or does not match observer 2 for reliability assessments.

Coding Considerations

The development of a reliable coding scheme is crucial for appropriately capturing the behaviors in question and testing the experimenter’s a priori hypotheses ( Bakeman & Gottman, 1987 ). There are three types of coding categories that are often included in observational systems: physical description codes, consequence codes, and relational or environmental relations codes ( Pellegrini, 2004 ). Physical description is believed to be the most “objective” type of codes because these describe “muscle contraction” ( Pellegrini, 2004 , p. 108) and might, for example, be involved in recording a participant’s social dominance or submissiveness (e.g., direct eye contact, rigid posture, arms akimbo; see   Ostrov & Collins, 2007 ). The second type of codes is for those of consequence in which a constellation of behaviors are part of a single code if they lead to the same outcome ( Pellegrini, 2004 ). For example, if we were interested in studying social dominance, then we might code taking objects away from others that result in a submissive posture on the part of the nonfocal participant to be an indicator of social dominance ( Ostrov & Collins, 2007 ). The third type of codes includes categories in which participants are described in relation to the context in which they are observed ( Pellegrini, 2004 ). An example of a relational observational category would be a coding scheme that accounted for where and with whom an individual was socially dominant. In terms of costs and benefits, it is clear that physical description codes are often easier to train and therefore potentially more reliable. It is possible that consequence codes may be unreliable given a misunderstanding of the sequence of events ( Pellegrini, 2004 ). Relational codes involve the appropriate documentation of multiple factors and therefore create more possibilities of error (for discussion, see   Pellegrini, 2004 ; Bakeman & Gottman, 1987 ). Overall, the level of analysis from micro- to macro-coding schemes is important to consider and the most objective and reliable system for addressing a researcher’s particular research question should be adopted.

A second consideration is the determination of whether to use mutually exclusive and exhaustive codes. Mutually exclusive codes are used when a single behavior may be recorded under one and only one code. In our observational studies, our coding scheme includes mutually exclusive codes such that a single behavior may be coded as either physical aggression or relational aggression, but not both. Exhaustive coding schemes are designed such that for any given behavior of a theoretical construct, there is an appropriate code for that behavior. For example, in our work we have codes for physical, relational, verbal, or nonverbal aggression as well as aggression not otherwise specified. Thus, if we determine a behavior is an act of aggression, then it may be coded as one of our behaviors in our scheme. Often schemes include mutually exclusive and exhaustive codes because there are several benefits to this approach ( see   Bakeman & Gottman, 1987 ). Having mutually exclusive codes means that researchers are not violating assumptions of independence, which are often needed for parametric statistics. For example, if a single behavior may be coded as both physical and relational aggression, then that may violate our assumption that the data are independent and come from independent behavioral interactions ( Pellegrini, 2004 ). Having exhaustive codes also speaks to the content validity of a coding scheme. That is, if the overall construct appropriately measures all facets of that construct, then the behavior in question should be included in the observational system, and exhaustive schemes guarantee this occurrence. It is important to recall that the larger the coding scheme, the more taxing the observational procedures will be for observers and the greater the possibility of observer error.

Scoring of observational data is similar to the scoring of any quantitative data within the social and behavioral sciences, and it often depends on the convention within a particular field and the type of observational sampling and recording techniques that are adopted. For example, for focal participant sampling with continuous recording, frequency counts are often generated by summing each independently recorded behavior across the various sessions. In our own research, that would mean that an individual participant would get a score for each of the constructs (i.e., physical aggression, relational aggression, verbal aggression, etc.) by summing all the behaviors within a construct (e.g., all physical aggression behaviors) across all eight sessions ( Ostrov & Keating, 2004 ). If the number of sessions is different for each participant because of missing data, then it is often common practice to divide by the number of sessions completed to generate an average rate of behavior per session ( see   Crick, Ostrov, Burr et al., 2006 ). Occasionally it is apparent that an error was made in the original coding of behaviors. Best practices have not been established for addressing these concerns, but as long as these errors are not systematic, the adopted solutions are often not a concern. To avoid problems with potential scoring biases, the observers and coders should always be unaware of the participant’s condition and/or past history. In addition, whenever possible, observers and coders should be unaware of the study hypotheses.

Psychometric Properties

Reliability.

Reliability is often conceptualized as consistency within or between individuals (i.e., intra-observer or inter-observer), within measures (internal consistency), or across time (i.e., test–retest). Arguably, for observational methods, the most important measure of consistency is inter-observer reliability, or the degree to which two sets of observations from two independent observers agree ( Stangor, 2011 ). In the present review, we will first address intra-observer reliability and then focus on the assessment of inter-observer reliability.

Intra-observer, or within-observer, reliability is defined as a situation in which two sets of observations by the same research assistant agree or are consistent. Essentially, intra-observer reliability is assessing how consistent a particular observer is when coding specific behaviors either between sessions (i.e., across time) or within a single session. As Pellegrini (2004) has discussed in more detail, we may conceptualize and test (e.g., Pearson’s Product-Moment Correlation Coefficient) intra-observer reliability in ways similar to test–retest reliability, and thus, intra-observer reliability is essentially the temporal stability of the observational measure for a given observer between testing sessions. We might desire to know the degree to which the observational score on a given behavioral construct for the same observer is stable across time to test for observer drift (a threat to the validity of the observational data), or the likelihood that observers are deviating from initial training procedures over time and modifying the definitions of the constructs under study ( Smith, 1986 ). Intra-observer reliability or consistency within an observer may also be conceptualized as the reliability of an observer’s scores within a single session, and in this case the test is analogous to assessments of internal consistency (e.g., Cronbach’s α ). As Pellegrini (2004) has stated, we assume an observer is first reliable or consistent in their scoring/recording by themselves prior to testing if they agree with an independent observer (i.e., inter-observer reliability).

As mentioned, inter-observer reliability or consistency between observers is the gold standard for observational research. Essentially, inter-observer reliability involves comparing the independent codes of the observers with other trained observers. There are several ways to assess this psychometric property ( see   Pellegrini, 2004 ), but the key task is comparing agreement across all of the observers. An important best practice for inter-observer reliability procedures is to ensure that observers are sampling/recording the same behaviors independently. Independent coding may be conducted with the use of video and private coding sessions without discussion until all codes have been completed. Inter-observer reliability may be assessed live in the field if the observers take precautions to avoid conveying to their partner how (and, in some cases, when) they are recording the behavior in question. A second best practice is to assess for reliability across the study to help avoid various biases (e.g., observer drift) and coding/recording errors from corrupting the integrity of the data. That is, observers should be checked against a master coder at the start of the study just after training ends, and each observer should pass an a priori reliability threshold (e.g., Cohen’s κ 〉 0 . 7 0 ). Next, their observations should be compared against other independent reliable observers throughout the duration of the study, and the trainer should provide constructive feedback for any deviations from the training protocol. Finally, an important consideration is for what percentage of time inter-observer reliability will be checked. This percentage should be a function of the number of cases or possible events that will be recorded, but typically 15% to 30% of a randomly selected sample of the possible sessions is coded by more than one observer for assessing inter-observer reliability. To avoid potential biases, a best practice is for each observer to conduct reliability observations with all other observers in a round-robin format.

There are several ways to statistically measure inter-observer reliability. In the past, authors relied on zero-order correlations (Pearson’s r ) but that problematic practice is not seen as often in the recent literature. A second statistical method that is still reported in peer-reviewed journals is percent agreement. Percent agreement may be expressed in Equation 1 :

where P o b s is the proportion of agreement observed, N A is the total number of agreements, and N D is the total number of disagreements. Percent agreement is not currently best practice, as it is influenced by the number of cases (i.e., it may be biased by relatively few cases) and because it is not compared against a standard threshold ( Bakeman & Gottman, 1987 ). Finally, one of the central concerns with percent agreement (as well as Pearson’s r ) as a measure of inter-observer reliability is that it does not control for chance agreement ( Bakeman & Gottman, 1987 ).

Cohen’s (1960)   κ is a preferred statistic for inter-observer reliability because it does control for chance agreements and is a more “stringent statistic,” allowing greater precision in assessing reliability at a specific moment in time or for particular events rather than overall summaries of association ( Bakeman & Gottman, 1987 , p. 836). Importantly, κ may only be used when coders use a categorical scale ( Bakeman & Gottman, 1987 ) and when a 2 x 2 matrix may be created to depict the proportion of agreements/disagreements for occurrences/nonoccurrences of behavior for any two observers ( Pellegrini, 2004 ). When calculating the rate of agreement, it is important to a priori indicate any time parameters (i.e., within what period of time must both observers note the occurrence of a behavior, also known as the tolerance interval). Some experts caution that extremely short tolerance intervals (e.g., 1 sec) may be overly stringent and artificially reduce the degree of agreement given typical reaction times of observers ( see   Bakeman & Gnisci, 2006 ). If time sampling is being used, then observers should be signaled by an external source (e.g., audible tone from an electronic device) to indicate when they should record the behavior ( see   Pellegrini, 2004 ). κ may be expressed in Equation 2 :

where P o b s is the proportion of agreement observed, and P e x p is the expected proportion of agreement by chance ( Bakeman & Gnisci, 2006 ). Equation 2 indicates that agreement anticipated as a result of chance is subtracted from both the numerator and denominator, thus κ provides the proportion of agreement corrected for chance agreements ( Bakeman & Gnisci, 2006 ). The range for κ is from - 1 . 0 0 to + 1 . 0 0 , with a value of “0” indicating that obtained agreement is equivalent to agreement anticipated by chance, and greater than chance agreement would yield positive values with +1.00 equal to perfect agreement between the observers ( Cohen, 1960 ). Interestingly, Cohen (1960) revealed that negative values (less than 0) were rare and suggested agreement at less than chance levels. It is possible to test if κ is significantly different from 0, but statistical significance is often not used as a threshold for determining an “adequate” or “good” criterion ( Bakeman & Gottman, 1987 ). Initially, Landis and Koch (1977) provided an index of the strength of agreement or “benchmarks” and reported the following standards: κ of < 0 . 0 0 was “poor,” 0 . 0 0 - 0 . 2 0 was “slight,” 0 . 2 1 - 0 . 4 0 was “fair,” 0 . 4 1 - 0 . 6 0 was “moderate,” 0 . 6 1 - 0 . 8 0 was “substantial,” and 〉 0 . 8 1 was “almost perfect” (p. 165). However, Bakeman and Gottman (1987) reported that a significant κ of less than 0.70 may be a reason for concern. Other scholars have noted that the conservative nature of κ permits one to use a slightly lower threshold for adequate levels of reliability than the typical convention of 0.70 and suggest that a κ coefficient of 0.60 or higher is “acceptable” and 0.80 or above is considered “good” ( Pellegrini, 2001 ).

Under circumstances when a κ coefficient may not be calculated (e.g., when noncategorical data is used or quadrants of the aforementioned occurrence matrix may not be available given the recording rules of the adopted observational procedure), scholars have suggested that an intraclass correlation coefficient (ICC) be computed between independent raters on the continuous data ( Bartko, 1976 ; McGraw & Wong, 1996 ; Shrout & Fleiss, 1979 ). There are several possible ICC formulas that could be depicted that are beyond the scope of the present review, and as such the interested reader is referred to the prior literature on this topic ( Shrout & Fleiss, 1979 ; McGraw & Wong, 1996 ). Intra-class correlation coefficients may be expressed as a function of either the reliability for a single rating (i.e., the reliability of a typical, single observer compared to another observer) or the average rating of the observations across all the raters ( McGraw & Wong, 1996 ). The average rating ICC uses the Spearman-Brown correction to indicate the reliability for all the observers averaged together ( Bartko, 1976 ). The absolute value of an ICC assessing average ratings will be greater or equal to the ICC for a single rater ( Bartko, 1976 ). Intra-class correlation coefficients may also be calculated as an index of “consistency” or as a measure of “absolute agreement.” Essentially, if systematic differences among observers are of interest, then the “absolute agreement” formula accounts for observer variability in the denominator of the ICC estimate, and this is not included for ICCs that measure “consistency” (for further detail, see   McGraw & Wong, 1996 ). Intra-class correlation coefficients range from –1.00 to +1.00, where negative values indicate a lack of reliability and +1.00 would indicate perfect agreement ( Bartko, 1976 ). An advantage to ICCs is that confidence intervals may be calculated ( see   McGraw & Wong, 1996 ). Typically, acceptable levels of reliability for ICCs are similar to other criteria in the field, and as such, levels greater than or equal to 0.70 are considered “acceptable” (e.g., Ostrov, 2008 ; NICHD Early Child Care Research Network, 2004 ).

In using observational research methods, an assessment of validity is equally as important as an assessment of reliability. Different types of validity should be considered to strengthen the inferences drawn from a particular method, with construct validity being most fundamental to any empirical inquiry. Construct validity is the degree to which the construct being studied actually measures the concept that a researcher intends to study ( Stangor, 2011 ). Construct validity is often established through assessments designed to measure convergent and discriminant validity. Convergent validity rests on the assumption that if a construct is truly being measured, then alternative assessments of the same construct should be correlated with each other ( Stangor, 2011 ). For example, an observational method intended to measure disruptive behaviors in the classroom should be correlated with teacher reports of disruptive behaviors. Alternatively, discriminant validity suggests that the construct being studied should not be correlated with other variables unrelated to the construct ( Stangor, 2011 ). Should the expected convergent and discriminant associations not be observed, then it is unclear what an instrument or observational system is measuring.

Other types of validity that are secondary yet still important to the establishment of a psychometrically sound observational system include content validity and criterion validity. Content validity refers to the extent to which a measure adequately assesses the full breadth of the construct being studied ( Stangor, 2011 ). For example, an observational study of children’s play behavior should code for different types of play, given that it is a diverse construct. To ensure that all facets of a construct are included in an observational system, correspondence with experts and focus groups/review panels may be used. Criterion validity involves an assessment of whether a study variable is associated with a theoretically relevant outcome measure. If observations are associated with an outcome that is measured at the same point in time at which observations are conducted, then concurrent validity is demonstrated. If observations are associated with an outcome that is measured at a future point in time, then predictive validity is demonstrated. For example, concurrent validity would be confirmed by associations between classroom observations of disruptive behavior and teacher report of rejection by peers, and predictive validity would be confirmed by associations between classroom observations of disruptive behavior and future parent -report of academic performance.

threats to validity: sources of bias and error

There are numerous biases for which observational methods are susceptible. A key bias is the aforementioned observer drift, and it is paramount that investigators monitor for this threat to the validity of the data by carefully assessing observational records and calculating reliability coefficients for the duration of the study. Importantly, in addition to the aforementioned discussion about intra-observer reliability, observer drift may also be indicated if there is a drop in inter-observer reliability among the phases of training and data collection ( Smith, 1986 ). A second strategy to mitigate observer drift is to regularly retrain observers. In instances where particular observers demonstrate problematic coding patterns, retraining should be individualized and should target the particular area of concern. In general, retraining is a practice that is beneficial for every observer because it reinforces proper coding procedures and observer behavior, thereby ensuring the integrity of the study.

A second type of distortion that must be considered results from participant reactivity, which is also a threat to the validity of the observational data. Reactivity occurs when the individuals under study alter their behavior because of the presence or influence of an observer. Consequently, the behavior observed does not provide a true representation of the construct being measured. If participants avoid a particular location within a setting or modify their behavior because they know they are being recorded, this is a major concern for the validity of the data ( Stangor, 2011 ). Depending on the nature of the study, reactivity may be more probable. For example, when observers need to remain within earshot of a focal participant to hear and see the behavioral interactions, it is crucial that the observers remain unobtrusive (e.g., Pellegrini, 1989 ). Researchers should explicitly address reactivity by training observers in the field to have a minimally responsive manner ( Pellegrini, 2004 ). Essentially, observers should use neutral facial expressions and control their nonverbal behavior, posture, movement, and reactions to events during live coding. It is also possible that participants may be reactive to cameras and other recording devices, and efforts should be made to habituate participants to this equipment ( see Use of Technology and Software section below) and monitor for this occurrence. Thus, this habituation process should occur prior to the actual collection of data ( Pellegrini, 2004 ). In our studies, we spend a minimum of several days in the observational environment (and will do so for as long as needed) simulating our observations, which provide the participants an opportunity to habituate to our presence and reduce reactivity prior to actual data collection. Therefore, regardless of live or videotaped coding, researchers should observe for participant reactivity and report the degree of reactivity in their studies (e.g., Atlas & Pepler, 1998 ). We define participant reactivity as any direct eye contact between the focal participant and observer, comments from the focal participant to the observer about our presence, or comments about our presence to others in the environment ( Ostrov, 2008 ). Our training procedures and careful monitoring has resulted in relatively low levels of reactivity in several studies (e.g., 1.5–2.5 times per focal participant during 80 min of observation; Crick, Ostrov, Burr et al., 2006 ).

Observer expectancy effects are a third bias ( Hartmann & Pelzel, 2005 ), which is essentially when observers form expectations about the nature of the data based on their knowledge or assumptions about the study goals and hypotheses, which is why best practice is to use unaware observers, when possible, and to use unaware observers for reliability purposes, at a minimum.

A final source of bias that we will discuss is gender bias as this is a well-documented concern with observational methods ( Ostrov, Crick, & Keating, 2005 ). Past research has documented that untrained observers maintain gender biases when observing, for example, physical aggression ( Lyons & Serbin, 1986 ; see also   Condry & Ross; 1985 ; Susser & Keating, 1990 ). That is, men tend to rate boys as more physically aggressive than girls, even when boys and girls are displaying comparable levels of aggression ( Lyons & Serbin, 1986 ). Moreover, male and female college students have shown documented gender biases based on knowledge about gender of young children in past experimental studies ( Gurwtiz & Dodge, 1975 ). Finally, in our own research, we have documented that male college students are less likely to correctly identify relational aggression or prosocial behavior than their female peers ( Ostrov et al., 2005 ). Please note that although the examples were related to our field of study (i.e., aggression), gender biases may be present for a variety of topics of study. Importantly, it may be that when individuals are trained to recognize potential biases, they are more likely to be objective in their coding of behavior ( Lyons & Serbin, 1986 ).

Use of Technology and Software

Excellent detailed reviews of computer-assisted recording devices and observational software programs are available ( see   Hoch & Symons, 2004 ), and thus, the present goal of this section is to briefly review the current state of technology and software for assisting in systematic observations in the laboratory and field. The following will include a review of the three most common observational software programs as well as the use of handheld devices and remote audiovisual equipment. The commercially available programs vary widely in function and cost, but most permit the observer to define a coding scheme and corresponding letter or number codes that observers can quickly use when making observations live or when coding digital media in the laboratory. Overall, advances in technology have made observational methods more efficient (e.g., flexible data reduction procedures and automatic statistical analyses), accurate (i.e., automatic rewind and playback functions reduce errors in coding), and applicable to a wider range of settings and topics of study ( Bakeman & Gnisci, 2006 , p. 140).

The first software program and associated computer-assisted recording devices that we will discuss is the Observer ® ; system by Noldus Inc. ( Noldus, Trienes, Hendriksen, Jansen, & Jansen, 2000 ). The current version is Observer XT, which permits both time sampling as well as continuous event-based observational systems and has been used in both human and animal research ( see   http://www.noldus.com/the-observer-xt/observer-xt-research ). A notable feature is that this software permits an assessment of response latency of the time between the onset of a stimulus and the initiation of the response, which facilitates consequence coding ( see Coding Considerations section above). The software also permits the linking of data from multiple modalities (e.g., observational reports, physiological responses) with a continuous time synch. The software may be used in the field with durable handheld devices or in the laboratory with live streaming video linked directly with the coding program ( Noldus et al., 2000 ). Finally, the new version of the software permits searches of the data for particular comments, events, or behaviors, and data may be exported to various statistical software packages ( Noldus et al., 2000 ). Jonge, Kemner, Naber, and van Engeland (2009) used an earlier version of the Observer software to code data from a study on block design reconstruction in children with autism spectrum disorders and a group of comparison participants. The use of the videotaped sessions and later coding by unaware observers meant that the coders using the software were unaware of the child’s group status. The software permitted the coders to record the amount of time the children took to reconstruct the block design pattern as well as a range of errors ( Jonge et al., 2009 ). The program was used to calculate Cohen’s κ based on two independent coders ( Jonge et al., 2009 ), who could make independent evaluations of the behavior without biasing their coding partner.

The second observational software program that we examine is the Multi-Option Observation System for Experimental Studies (MOOSES; Tapp, Wehby, & Ellis, 1995 ) and the associated Procoder for Digital Video (PCDV; Tapp& Walden, 1993 ), which permits viewing and coding of digital media ( see   http://mooses.vueinnovations.com/overview ). The MOOSES and PCDV programs also permit event and time sampling and for the coding of real-time digital media files or verbatim transcripts of observational sessions ( Tapp & Walden, 1993 ; Tapp et al., 1995 ). In fact, data files may be exported to MOOSES for event coding or to another format known as the Systematic Analysis of Language Transcripts (SALT) for transcription data coding. MOOSES automatically timestamps events and may provide frequency and duration codes as well as basic reliability statistics (e.g., Cohen’s κ ), and MOOSES is designed for sequential analysis ( Tapp et al., 1995 ). A handheld version of MOOSES is available. MOOSES/PCDV has been described as a lower cost alternative to The Observer ( Hoch & Symons, 2004 ).

The third system we review is the Behavior Evaluation Strategies and Taxonomies (BEST; Sharpe & Koperwas, 2003 ). This computer system includes both the BEST Collection for capturing digital media files and the BEST Analysis program for both qualitative and quantitative analysis of the observational data ( Sidener, Shabani, & Carr, 2004 ). The BEST program may be used for examining the frequency or duration of events, and sophisticated sequential analysis may be conducted. Much like the more expensive alternatives, this program will calculate reliability statistics (e.g., Cohen’s κ ) and will summarize data in table or various graph formats. A review of this program suggests that BEST does not handle the collection of interval-based data well, but the BEST Analysis program will allow a researcher to analyze this type of observational data ( Sidener et al., 2004 ). A new platform permits video display for captured data from video files, and although the program was initially written for Windows ® ; , there are inexpensive Apple ® ; iPhone ® ; and iPod Touch ® ; applications available for data collection ( see   http://www.skware.com ).

Various types of technology (e.g., audio and video recordings) have an extensive history in the field and laboratory to assist researchers in better capturing verbal and nonverbal interactions (e.g., Abramovitch, Corter, Pepler, & Stanhope, 1986 ; Stauffacher & DeHart, 2005 ). Remote audiovisual recordings provided an opportunity to combine the benefits of both audio and video recording while also reducing reactivity to typical recording devices when participants were observed in naturally occurring settings ( Asher & Gabriel, 1993 ; Atlas & Pepler 1998 ; Pellegrini, 2004 ; Pepler & Craig, 1995 ; Pepler, Craig, & Roberts, 1998 ). That is, videotaping with a telephoto zoom lens from an unobtrusive location in the natural setting and recording audio via a system of wireless microphones provides an externally valid way to record behavior and a time-synched verbal record of the interaction ( Pepler & Craig, 1995 ). Thus, remote audiovisual observational recordings provide all the benefits of having a video for subsequent coding by unaware observers (i.e., the ability to pause, rewind, and analyze subtle nonverbal behaviors) as well as a complete verbal transcript, which helps to put the video data in proper context ( Asher & Gabriel, 1993 ; Pepler & Craig, 1995 ). Wireless microphones typically are housed within small vests or waist pouches that participants wear, and often only the focal participant has an active or live microphone, and others in the reference group have “dummy” microphones that resemble the weight and look of the real microphone. Importantly, observational codes made with the remote audiovisual equipment have demonstrated acceptable inter-observer reliability coefficients (e.g., κ = 0 . 7 6 ; Pepler & Craig, 1995 ). Moreover, this procedure as well as sufficient exposure to the equipment by the participants has been found to produce low levels of participant reactivity (e.g., <5%, Atlas & Pepler, 1998 ; see also   Asher & Gabriel, 1993 ). The benefits of a rich observational record with low levels of reactivity within settings of high ecological validity seem to outweigh the costs, which include additional training, equipment costs, and some ethical considerations. A central ethical consideration is that individuals without consent may be recorded indirectly. A possible solution is to temporarily store and then, after processing, discard film clips of individuals without consent ( Pepler & Craig, 1995 ), but this solution may violate the rights of nonparticipants. Alternatively, a researcher could restrict access to the observational setting to only those with consent, but this second approach is a threat to the ecological validity of the procedures ( Pepler & Craig, 1995 ). An additional concern is that third parties may wish to use the data as surveillance, which might limit the rights of participants being recorded. As such, policies related to confidentiality and any possible limits of confidentiality should be discussed with the participants and any other possible party that may desire access to the data ( see   Pepler & Craig, 1995 ). Importantly, to our knowledge, remote audiovisual observational methodology has only been used with school-aged children in the classroom ( Atlas & Pepler, 1998 ) and typically on the playground (e.g., Asher & Gabriel, 1993 ; Pepler, Craig, & Roberts, 1998 ); thus, it is not clear if older individuals would be more aware and reactive to the procedure and equipment ( Pepler & Craig, 1995 ).

Ethical Considerations

There are several ethical considerations with observational research. With naturally occurring phenomena, there may be a temptation to observe social interactions and behavior without obtaining informed consent. Although this practice may technically be exempt from most Institutional Review Board (IRB) review (i.e., if identifying information is not collected and video or audio recordings of the public behavior are not made), we strongly encourage researchers to obtain informed consent from participants and assent from legal minors to support their right for autonomy but also so that all risks (e.g., breaches of confidentiality) may be appropriately conveyed. To avoid these breaches of confidentiality, researchers conducting live observations typically use identification codes rather than identifying information about the participants on all observation forms and in data files. Access to video or audio recordings of observational sessions is typically restricted to only those individuals (e.g., coders) who must have access as part of the research study. Participants should be fully informed for how long the observational recordings will be maintained and when they will be destroyed. A final ethical consideration concerns intervention efforts or at what point the researcher or observers will intervene (for a discussion of duty to warn with observational methods, see   Pepler & Craig, 1995 ) and directly or indirectly act on the behalf of the participants. For example, in our observational studies, we have clearly established procedures for when we will notify a teacher that a child in the observation setting is in danger or in need of help (e.g., leaving the controlled area, serious injury). These procedures are discussed at the start of the study with school officials and are part of our consent process, which we believe are best practices.

An Overview of Procedures for a High-Quality Systematic Observational Study

The researcher begins by a priori selecting and operationally defining behaviors of interest. Next, the researcher adopts a coding scheme by selecting the most appropriate sampling and recording procedures given the nature of the behavior under study and the observational context ( see Table 15.2 ). Ethical considerations should be addressed during this development stage of the observational method and should be evaluated for the duration of the study. If the observational scheme is newly developed for the study, then it is imperative that pilot testing occur within a similar context and with a sample representing the target population. If it is not a new scheme or if pilot testing does not indicate any problems, then the investigator may begin training observers. If there are problems noted, then it is important to rectify these issues as quickly as possible to avoid further errors in the study. It is possible that modifications will be needed regarding the operational definition of the observed constructs or changes may be needed to the procedures and coding scheme given the nature of the context or sample under study. Once these changes are adopted, additional checks should be made to verify the solution has worked to ameliorate the original concerns. Training involves the use of a standardized manual, and initial reliability training assessments are conducted prior to the collection of data. Behavior is sampled in the lab or in the field in accordance with the adopted sampling and recording rules, and inter-observer reliability is collected for the duration of the study. Validity assessments are also conducted using alternative informants and methods. If reliability or validity problems are detected, then this may also yield further modifications to the coding scheme to address the problems. If no psychometric problems are noted, then coding and scoring of the observational data occurs using standardized procedures. Finally, the data are analyzed and reported, which concludes the systematic observational study ( see Fig. 15.1 ).

Systematic observational methods provide an opportunity to record the behavior of humans and animals in a relatively objective manner, without sacrificing ecological validity. In the present chapter, we have attempted to identify best practices as well as benefits and costs of various sampling and recording techniques. Quantitative researchers should be guided by a priori research questions and hypotheses when selecting the most appropriate sampling and recording procedure for the specific research setting. Systematic observations require careful attention to coding and scoring decisions and a focus on achieving acceptable levels of reliability and validity. As a field, we must work to establish more stringent standards of reliability (i.e., inter-observer) and validity (i.e., construct) for observational methods. Moreover, we must continue to address and reduce various sources of bias and error. The use of computer-assisted software and digital analysis technology provide some promising options for increasing the efficiency and appeal of systematic observations in the field. Attention must also be given to key ethical considerations to guide appropriate conduct as an observational researcher. Careful consideration of these issues may inform quality research in a wide variety of basic, clinical, and educational contexts.

Procedures for a high-quality systematic observational study.

Future Directions

Observational methods have been a part of the social and behavioral sciences since the early years of our field, and we anticipate that there is a bright future for observational methods within the quantitative scholar’s toolbox. We have defined seven questions and two remaining issues that we believe the field should work to address. This list is not exhaustive, but we hope these questions will generate future work using systematic observational methods.

1. What is the utility of observational methods above and beyond additional informants? Given the time and cost of observational methods, it is necessary to continue to demonstrate that observational methods have incremental predictive utility or may explain unique amounts of variance in relevant outcomes, above and beyond other informants and measures ( Doctoroff & Arnold, 2004 ; Shaw et al., 1998 ). For example, we have demonstrated that observations of relational and physical aggression account for a significant amount of unique variance above and beyond teacher reports of relational and physical aggression in the prediction of teacher-reported deceptive and lying behaviors ( Ostrov, Ries, Stauffacher, Godleski, & Mullins, 2008 ).

2. How does one best examine the construct validity of observational methods? To date, there is not wide consensus on the best approach for demonstrating the construct validity of observational systems. The typical approach is to compare observational data to other “gold standard” methods. For example, convergent evidence is achieved when high levels of association are found across methods such as between observations of aggression subtypes in classrooms, observations of aggression subtypes via semi-structured observations, and with various informants including teacher reports and parent reports of aggression subtypes (e.g., Crick, Ostrov, Burr, et al., 2006 ; Hinde et al., 1984 ; Ostrov & Bishop, 2008 ; Ostrov & Keating, 2004 ; Pellegrini & Bartini, 2000 ).

3. How do we detect observer biases? We believe the field has only begun to address the important issue of how to assess and identify observer biases. Much further work is needed to examine a host of possible biases from observer drift and observer expectancy effects to gender biases as well as other possible sources of distortion such as halo effects and potential expectancy biases derived from prior knowledge of participants in longitudinal studies ( Hartmann & Pelzel, 2005 ). In addition, more focus should be placed on assessing participant reactivity. Few studies report this source of error and threat to validity, and we encourage observational researchers to quantify the degree to which their participants are reactive to the observational procedures.

4. How do we eliminate observer biases and other sources of error? Once we identify observer biases, we need more evidence-based information on how to appropriately eliminate these biases and sources of error. The literature has indicated few possible solutions (e.g., increased training for individuals with identified biases). In addition, more emphasis should be placed on identifying best practices for reducing reactivity. It is clear that minimally responsive procedures and habituation practices have worked effectively to reduce reactivity to low levels (e.g., <5% of time), but our goal should be to eliminate this source of error from our data.

5. What is the sufficient amount of time for observational sampling? Too often the time interval for time sampling as well as the total duration of observed time for event-based coding systems is decided without sufficient justification, and greater work is needed to establish parameters and strategies for determining the most efficient and useful time intervals for various behaviors and settings.

6. How do we reduce the cost of observational methods? One of the biggest obstacles to greater adoption of systematic observational methods is the cost of observational procedures. Typically, large staffs of highly trained individuals are needed for observational work, and although volunteer research assistants may be used to address this concern, this is still a significant barrier to further work in this area. Moreover, the overall amount of time to conduct an observational study is potentially longer than comparable studies with other methods, and thus we must work to make training procedures, data collection, and coding processes more efficient. The use of computer-assisted software and coding technology will continue to greatly help in this regard.

7. How do we refine and create observational software so that it is compatible with all types of observational systems and more flexible as well as affordable? Although observational software and recording devices have advanced a great deal in recent years ( see   Hoch & Symons, 2004 ), the software must become more flexible to accommodate a greater range of observational sampling and recording procedures. Moreover, the financial cost of these programs and licenses are often prohibitive, and efforts must be made to develop high-quality, affordable, and flexible computer-assisted observational software programs.

8. A key remaining issue is that as a field we need to move away from the use of Pearson product moment correlations and percent agreement as a standard measure of assessing inter-observer reliability. Given what we know about the role of chance agreement from classic (e.g., Cohen, 1960 ) and modern sources ( Bakeman & Gottman, 1987 ; Pellegrini, 2004 ), it is not clear why some peer-reviewed manuscripts continue to only present either Pearson product moment correlations or percent agreement as strong evidence of inter-observer reliability.

9. A second remaining concern is that greater discussion of the ethical issues involved in observational methods is needed. For example, as we have discussed, it is not always clear when intervention is needed by observers in the field. Further, greater work needs to be conducted to examine how we may best ensure confidentiality of data with detailed observational records. Finally, we must focus on how we ensure confidentiality with the transfer of electronic observational data via handheld devices and other electronic technology.

Author Note

We wish to thank Jennifer Kane and members of the UB Social Development Laboratory for their assistance with the preparation of this chapter. Thanks to Dr. Leonard J. Simms for comments on an earlier draft. Special thanks to Dr. Anthony D. Pellegrini, who has greatly influenced the way we conceptualize systematic observational methods. The authors are affiliated with the Department of Psychology, University at Buffalo, The State University of New York. Please direct correspondence to the first author at [email protected] or 716-645-3680.

Abramovitch, R. , Corter, C. , Pepler, D. J. , & Stanhope, L. ( 1986 ). Sibling and peer interaction: A final follow-up and a comparison.   Child Development , 57 , 217–229.

Google Scholar

Arrington, R. E. ( 1943 ). Time sampling in studies of social behavior: A critical review of techniques and results with research suggestions.   Psychological Bulletin , 40 , 81–124.

Arsenio, W. F. , & Lover, A. ( 1997 ). Emotions, conflicts and aggression during preschoolers’ free play.   British Journal of Developmental Psychology , 15 , 531–542.

Asher, S. R. , & Gabriel, S. W. ( 1993 ). Using a wireless transmission system to observe conversation and social interaction on the playground. In C. H. Hart (Ed.). Children on playgrounds: Research perspectives and applications (pp. 184–209). Albany, NY: SUNY Press.

Google Preview

Atlas, R. S. , & Pepler, D. J. ( 1998 ). Observations of bullying in the classroom.   Journal of Educational Research , 92 , 86–99.

Bakeman, R. , & Gnisci, A. ( 2006 ). Sequential observational methods. In M. Eid & E. Diener (Eds.). Handbook of multimethod measurement in psychology (pp. 127–140). Washington DC: American Psychological Association.

Bakeman, R. , & Gottman, J. M. ( 1987 ). Applying observational methods: A systematic view. In J. D. Osofsky (Ed.). Handbook of infant development . (2nd ed., pp. 818–854). New York: John Wiley.

Bartko, J. J. ( 1976 ). On various intraclass correlation reliability coefficients.   Psychological Bulletin , 83 , 762–765.

Bowker, A. , Boekhoven, B. , Nolan, A. , Bauhaus, S. , Glover, P , Powell, T. , & Taylor, S. ( 2009 ). Naturalistic observations of spectator behavior at youth hockey games.   Applied Research , 23 , 301–316.

Cohen, J. ( 1960 ). A coefficient of agreement for nominal scales.   Educational and Psychological Measurement , 20 , 37–46.

Coie, J. D. & Kupersmidt, J. B. ( 1983 ). A behavioral analysis of emerging social status in boys’ groups.   Child Development , 54 , 1400–1416.

Condry, J. C. , & Ross, D. F. ( 1985 ). Sex and aggression: The influence of gender label on the perception of aggression in children.   Child Development , 56 , 225–233.

Crawford, M. P. ( 1942 ). Dominance and social behavior, for chimpanzees, in a non-competitive situation.   Journal of Comparative Psychology , 33 , 267–277.

Crick, N. R. , Ostrov, J. M. , Burr, J. E. , Jansen-Yeh, E. A. , Cullerton-Sen, C. , & Ralston, P. ( 2006 ). A longitudinal study of relational and physical aggression in preschool.   Journal of Applied Developmental Psychology , 27 , 254–268.

Doctoroff, G. L. , & Arnold, D. H. ( 2004 ). Parent-rated externalizing behavior in preschoolers: The predictive utility of structured interviews, teacher reports, and classroom observations.   Journal of Clinical Child and Adolescent Psychology , 4 , 813–818.

Dodge, K. A. ( 1983 ). Behavioral antecedents of peer social status.   Child Development , 54 , 1386–1399.

Englund, M. M. , Levy, A. K. , Hyson, D. M. , & Sroufe, L. A. ( 2000 ). Adolescent social competence: Effectiveness in a group setting.   Child Development , 71 , 1049–1060.

Fagot, B. T. , & Hagan, R. ( 1985 ). Aggression in toddlers: Responses to the assertive acts of boys and girls.   Sex Roles , 12 , 341–351.

Goodenough, F. L. ( 1930 ). Inter-relationships in the behavior of young children.   Child Development , 1 , 29–47.

Gurwtiz, S. B. , & Dodge, K. A. ( 1975 ). Adults’ evaluations of a child as a function of sex of adult and sex of child.   Journal of Personality and Social Psychology , 32 , 822–828.

Hall, L. J. , & McGregor, J. A. ( 2000 ). A follow-up study of the peer relationships of children with disabilities in an inclusive school.   The Journal of Special Education , 34 , 114–126.

Harrist, A. W. , & Bradley, K. D. ( 2003 ). You can’t say you can’t play: Intervening in the process of social exclusion in the kindergarten classroom.   Early Childhood Research Quarterly , 18 , 185–205.

Hartmann, D. P. , & Pelzel, K. E. ( 2005 ). Design, measurement, and analysis in developmental research. In M. H. Bornstein & M. E. Lamb (Eds.). Developmental science: An advanced textbook (5th ed., pp. 103–184). Mahwah, NJ: Lawrence Erlbaum Associates.

Hawley, P. H. , & Little, T. D. ( 1999 ). On winning some and losing some: A social relations approach to social dominance in toddlers.   Merrill-Palmer Quarterly , 45 , 185–214.

Hinde, R. A. , Easton, D. F. , & Meller, R. E. ( 1984 ). Teacher questionnaire compared with observational data on effects of sex and sibling status on preschool behavior.   Journal of Child Psychology and Psychiatry , 25 , 285–303.

Hintze, J. M. , Volpe, R. J. , & Shapiro, E. S. ( 2002 ). Best practices in the systematic direct observation of student behavior. In A. Thomas & J. Grimes (Eds.). Best practices in school psychology-IV (pp. 993–1006). Bethesda, MD: National Association of School Psychologists.

Hoch, J. , & Symons, F. J. ( 2004 ). Computer-assisted recording and observational software programs. In A. D. Pellegrini ’s Observing children in their natural worlds: A methodological primer . (2nd ed., pp. 214–222). Mahwah, NJ: Lawrence Erlbaum Associates

Jackson, H. G. & Neel, R. S. ( 2006 ). Observing mathematics: Do students with EBD have access to standards-based mathematics instruction?   Education and Treatment of Children , 29 , 593–614.

Jonge, M. de. , Kemner, C. , Naber, F. , & Engeland, H. van . ( 2009 ). Block design reconstruction skills: not a good candidate for an endophenotypic marker in autism research,   European Child & Adolescent Psychiatry , 18 , 197–205.

Keating, C. F. , & Heltman, K. R. ( 1994 ). Dominance and deception in children and adults: Are leaders the best misleaders?   Personality and Social Psychology Bulletin , 20 , 312–321.

Krehbiel, D. , & Lewis, P. T. ( 1994 ). An observational emphasis in undergraduate psychology laboratories.   Teaching of Psychology , 21 , 45–48.

Landis, J. R. , & Koch, G. G. ( 1977 ). The measurement of observer agreement for categorical data.   Biometrics , 33 , 159–174.

Langfeld, H. S. ( 1913 ). Text-books and general treatises.   Psychological Bulletin , 10 , 25–32.

Länsisalmi, H. , Peiró, J. M. , & Kivimäki, M. ( 2000 ). Collective stress and coping in the context of organizational culture.   European Journal of Work and Organizational Psychology , 9 , 527–559.

Laursen, B. , & Hartup, W. W. ( 1989 ). The dynamics of preschool children’s conflicts.   Merrill-Palmer Quarterly , 35 , 281–297.

Leff, S.S. , & Lakin, R. ( 2005 ). Playground-based observational systems: A review and implications for practitioners and researchers.   School Psychology Review , 34 (4), 475–489.

Lopez-Williams, A. , Chacko, A. , Wymbs, B. T. , Fabiano, G. A. , Seymour, K. E. , Gnagy, E. M. , et al. ( 2005 ). Athletic performance and social behavior as predictors of peer acceptance in children diagnosed with attention-deficit/hyperactivity disorder.   Journal of Emotional and Behavioral Disorders , 13 , 172–180.

Lyons, J. A. , & Serbin, L. A. ( 1986 ). Observer bias in scoring boys’ and girls’ aggression.   Sex Roles , 14 , 301–313.

Macintosh, K. & Dissanayake, C. ( 2006 ). A comparative study of the spontaneous social interactions of children with high-functioning autism and children with Asperger’s disorder.   Autism , 10 , 199–220.

McGraw, K. O. , & Wong, S. P. ( 1996 ). Forming inferences about some intra-class correlation coefficients.   Psychological Methods , 1 , 30–46.

McNeilly-Choque, M. K. , Hart, C. H. , & Robinson, C. C. , Nelson, L. , & Olsen, S. F. ( 1996 ). Overt and relational aggression on the playground: Correspondence among different informants.   Journal of Research in Childhood Education , 11 , 47–67.

Meany-Daboul, M. G. , Roscoe, E. M. , Bourret, J. C. , & Ahearn, W. H. ( 2007 ). A comparison of momentary time sampling and partial-interval recording for evaluating functional relations.   Journal of applied behavior analysis , 40 , 501–514.

Newcomb, T. ( 1931 ). An experiment designed to test the validity of a rating technique.   Journal of Educational Psychology , 22 , 279–289.

NICHD Early Child Care Research Network. ( 2004 ). Trajectories of physical aggression from toddlerhood to middle childhood.   Monographs of the Society for Research in Child Development , 69 , ( Serial No . 278).

Noldus, L. P. , Trienes, R. J. , Hendriksen, A. H. , Jansen, H. , & Jansen, R. G. ( 2000 ). The observer video-pro: New software for the collection, management, and presentation of time-structured data from videotapes and digital media files.   Behavior Research Methods, Instruments, and Computers , 32 , 197–206.

Ostrov, J. M. ( 2008 ). Forms of aggression and peer victimization during early childhood: A short-term longitudinal study.   Journal of Abnormal Child Psychology , 36 , 311–322.

Ostrov, J. M. & Bishop, C. M. ( 2008 ). Preschoolers’ aggression and parent-child conflict: A multiinformant and multimethod study.   Journal of Experimental Child Psychology , 99 , 309–322.

Ostrov, J. M. , & Collins, W. A. ( 2007 ). Social dominance in romantic relationships: A Prospective longitudinal study of non-verbal processes.   Social Development , 16 , 580–595.

Ostrov, J. M. , Crick, N. R. , & Keating, C. F. ( 2005 ). Gender-biased perceptions of preschoolers’ behavior: How much is aggression and prosocial behavior in the eye of the beholder?   Sex Roles , 52 , 393–398.

Ostrov, J. M. , & Godleski, S. A. ( 2007 ). Relational aggression, victimization, and language development: Implications for practice.   Topics in Language Disorders , 27 , 146–166.

Ostrov, J. M. , & Keating, C. F. ( 2004 ). Gender differences in preschool aggression during free play and structured interactions: An observational study.   Social Development , 13 , 255–277.

Ostrov, J. M. , Massetti, G. M. , Stauffacher, K. , Godleski, S. A. , Hart, K. C. , Karch, K. M. , Mullins, A. D. , et al. ( 2009 ). An intervention for relational and physical aggression in early childhood: A preliminary study.   Early Childhood Research Quarterly , 24 , 15–28.

Ostrov, J. M. , Ries, E. E. , Stauffacher, K. , Godleski, S. A. , & Mullins, A. D. ( 2008 ). Relational aggression, physical aggression and deception during early childhood: A multi-method, multi-informant short-term longitudinal study.   Journal of Clinical Child and Adolescent Psychology , 37 , 664–675.

Ostrov, J. M. , Woods, K. E. , Jansen, E. A. , Casas, J. F. , & Crick, N. R. ( 2004 ). An observational study of delivered and received aggression, gender, and social-psychological adjustment in preschool: “This white crayon doesn’t work …” Early Childhood Research Quarterly , 19 , 355–371.

Parten, M. B. ( 1932 ). Social participation among pre-school children.   The Journal of Abnormal and Social Psychology , 27 , 243–269.

Pelham, W. E. Jr. , Gnagy, E. M. , Greiner, A. R. , Hoza, B. , Hinshaw, S.P.   Swanson, J. M. , et al. ( 2000 ). Behavioral versus behavioral and pharmacological treatment in ADHD children attending a summer treatment program.   Journal of Abnormal Child Psychology , 28 , 507–525.

Pellegrini, A. D. ( 1989 ). Categorizing children’s rough-and-tumble play.   Play & Culture , 2 , 48–51.

Pellegrini, A. D. ( 2001 ). Practitioner review: The role of direct observation in the assessment of young children.   Journal of Child Psychology and Psychiatry , 42 , 861–869.

Pellegrini, A. D. ( 2004 ). Observing children in their natural worlds: A methodological primer . (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Pellegrini, A. D. , & Bartini, M. ( 2000 ). An empirical comparison of methods of sampling aggression and victimization in school settings.   Journal of Educational Psychology , 92 , 360–366.

Pellegrini, A. D. , Ostrov, J. M. , Roseth, C. , Solberg, D. , & Dupuis, D. ( in press ). Using observational methods to study children’s and adolescents’ development. In G. Melton , A. Ben-Arich , & J. Cashmore (Eds.). Handbook of child research . Beverly Hills, CA: Sage.

Pepler, D. J. , & Craig, W. M. ( 1995 ). A peek behind the fence: Naturalistic observations of aggressive children with remote audiovisual recording.   Developmental Psychology , 31 , 548–553.

Pepler, D. J. , Craig, W. M. , & Roberts, W. L. ( 1998 ). Observations of aggressive and nonaggressive children on the school playground.   Merrill-Palmer Quarterly , 44 , 55–76.

Quake-Rapp, C. , Miller, B. , Ananthan, G. , & Chiu, E-C. ( 2008 ). Direct observation as a means of assessing frequency of maladaptive behavior in youths with severe emotional and behavioral disorder.   The American Journal of Occupational Therapy , 62 , 206–211.

Sharpe, T. L. & Koperwas, J. ( 2003 ). Behavior and sequential analyses: Principles and practice . Thousand Oaks, CA: Sage Publications.

Shaw, D. S. , Winslow, E. B. , Owens, E. B. , Vondra, J. I. , Cohn, J. F. , & Bell, R. Q. ( 1998 ). The development of early externalizing problems among children from low-income families: A transformational perspective.   Journal of Abnormal Child Psychology , 26 , 95–107.

Shrout, P. E. , & Fleiss, J. L. ( 1979 ). Intraclass correlations: Uses in assessing rater reliability.   Psychological Bulletin , 86 , 420–428.

Sidener, T. M. , Shabani, D. B. , & Carr, J. E. ( 2004 ). A review of the Behavioral Evaluation Strategy and Taxonomy (BEST) Software Application.   Behavioral Interventions , 19 , 275–285.

Silk, J. B. , Cheney, D. L. , & Seyfarth, R. M. ( 1996 ). The form and function of post-conflict interactions between female baboons.   Animal Behaviour , 52 , 259–268.

Slee, P. T. ( 1987 ). Child observation skills . London, UK: Croom Helm.

Smith, G. A. ( 1986 ). Observer drift: A drifting definition.   The Behavior Analyst , 9 , 127–128.

Soukup, J. H. , Wehmeyer, M. L. , Bashinski, S. M. , & Boyaird, J. A. ( 2007 ). Classroom variables and access to the general curriculum for students with disabilities.   Exceptional Children , 24 , 101–120.

Stangor, C. ( 2011 ). Research methods for the behavioral sciences (4th ed) . Belmont CA: Wadsworth.

Stauffacher, K. , & DeHart, G. ( 2005 ). Preschoolers’ relational aggression with siblings and friends.   Early Education and Development , 16 , 185–206.

Susser, S. A. , & Keating, C. F. , ( 1990 ). Adult sex role orientation and perceptions of aggressive interactions between boys and girls.   Sex Roles , 23 , 147–155.

Tapp, J.T. , & Walden, T. ( 1993 ). PROCODER: A professional tape control, coding, and analysis system for behavioral research using videotape.   Behavior Research Methods, Instruments, & Computers , 25 , 53–56.

Tapp, J. T. , Wehby, J. H. , & Ellis, D. ( 1995 ). A Multi-option observation system for experimental studies: MOOSES . Behavior Research Methods, Instruments, & Computers , 27 , 25–31.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Observation

Observation Methods

• First Online: 14 December 2017

Cite this chapter

• Malgorzata Ciesielska 4 ,
• Katarzyna W. Boström 5 &
• Magnus Öhlander 6  

9075 Accesses

42 Citations

Observation may be seen as the very foundation of everyday social interaction: as people participate in social life, they are diligent observers and commentators of others’ behavior. Observation is also one of the most important research methods in social sciences and at the same time one of the most complex. It may be the main method in the project or one of several complementary qualitative methods. As a scientific method it is has to be carried out systematically, with a focus on specific research questions. Therefore, we start with practical guide on clarifying research objectives, accessing the research field, selecting subjects, observer’s roles, and tips on documenting the data collected. The observation comprises several techniques and approaches that can be combined in a variety of ways. Observation can be either participant or not, direct or indirect. Further in this chapter, the main characteristics of three types of observations are outlined (the fourth type—direct non-participant—is discussed in the chapter on shadowing). While participant observation follows the ideal of a long-time immersion in a specific culture as a marginal member, researcher conducting non-participant observation takes position of an outsider and tries to distance him/herself from the taken-for-granted categorizations and evaluations. In the case of indirect observation, the researcher relies on observations of others (e.g. other researchers), various types of documentation, or self-observation. The chapter discusses the differences between those types of observation, shows inspirational examples from previous studies, and summarizes the method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Adjam, M. (2017). Minnesspår. Hågkomstens rum och rörelse i skuggan av flykt . Höör: Brutus Östlings bokförlag Symposion.

Google Scholar  

Arvastson, G., & Ehn, B. (2009). Etnografiska observationer . Lund: Studentlitteratur.

Bernard, H. R. (2000). Social Research Methods: Qualitative and Quantitative Approaches . Thousand Oaks: Sage.

Bernard, H. R. (2006). Research Methods in Anthropology. Qualitative and Quantitative Approaches . Thousand Oaks: Sage.

Bowden, A., & Ciesielska, M. (2016). Ecomuseums as Cross-Sector Partnerships: Similarities and Dissimilarities in their Governance, Strategy and Leadership. Public Money and Management, 36 (1), 23–30.

Article   Google Scholar  

Ciesielska, M. (2010). Hybrid Organisations. A Case of the Open Source-Business Setting . Frederiksberg: Copenhagen Business School Press.

Ciesielska, M., & Westenholz, A. (2016). Dilemmas Within Commercial Involvement in Open Source Software. Journal of Organizational Change Management, 29 (3), 344–360.

Ciesielska, M., Wolanik Boström, K., & Öhlander, M. (2012). Obserwacja. In D. Jemielniak (Ed.), Badania Jakościowe. Podręcznik akademicki. Tom I: Podejścia, teorie, problemy (pp. 41–67). Warszawa: PWN.

D’Eredita, M. A., & Barreto, C. (2006). How Does Tacit Knowledge Proliferate? An Episode-Based Perspective Organization Studies, 27 (12), 1821–1841.

Drake, D. H., & Harvey, J. (2014). Performing the Role of Ethnographer. Processing and Managing the Emotional Dimensions of Prison Research. International Journal of Social Research Methodology, 17 (5), 489–501.

Emerson, R. M., Fretz, R. I., & Shaw, L. L. (1995). Writing Ethnographic Fieldnotes . Chicago: University of Chicago Press.

Book   Google Scholar  

Emerson, R. M., Fretz, R. I., & Shaw, L. L. (2001). Participant Observation and Fieldnotes. In P. Atkinson (Ed.), Handbook of Ethnography . London: Sage.

Fangen, K. (2001). Pride and Power. A Sociological Study of the Norwegian Radical Nationalist Underground Movement . Oslo: Akademika.

Hammersley, M., & Atkinson, P. (2007). Ethnography. Principles in Practice . London: Routledge.

Hine, C. (2000). Virtual Ethnography . London: Sage.

Jauregui, B. (2013). Dirty Anthropology. Epistemologies of Violence and Ethical Entanglements in Police Ethnography. In W. Garriot (Ed.), Policing and Contemporary Governance. The Anthropology of Police in Practice . London: Palgrave and Macmillan.

Klintberg, B. (1986). Råttan i pizzan . Stockholm: Norstedts Förlag.

Kostera, M. (2007). Organizational Ethnography: Methods and Inspirations . London: Sage.

Kozinetz, R. V. (2015). Netnography. Redefined . Thousand oaks: Sage.

Nakumara, K. (2013). Making Sense of Sensory Ethnography. The Sensual and the Multisensory. American Anthropologist, 115 (1), 132–144.

Pink, S. (2015). Doing Sensory Ethnography (2nd ed.). Los Angeles/London: Sage.

Pripps, O., & Öhlander, M. (2011). Observation. In L. Kaijser & M. Öhlander (Eds.), Etnologiskt fältarbete . Lund: Studentlitteratur.

Rathje, W. (2001). Integrated Archaeology. A Garbage Paradigm. In V. Buchli & G. Lucas (Eds.), Archaeologies of the Contemporary Past . London: Routledge.

Rathje, W., & Murphy, C. (1992). Rubbish! The Archaeology of Garbage . New York: HarperCollins Publishers.

Silow Kallenberg, K. (2015). Smutsig etnografi. En metoddiskussion. Kulturella perspektiv, 24 (2), 2–12.

Silow Kallenberg, K. (2016). Gränsland. Svensk ungdomsvård mellan vård och straff . Huddinge: Södertörns högskola.

Sotirin, P. (1999). Bringing the Outside in. Ethnography in/beyond the Classroom . Annual Meeting of the National Communication Association, Chicago, 4–7.

Spradley, J. P. (1980). Participant Observation . New York: Holt, Rinehart and Winston.

Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (1966). Unobtrusive Measures: Nonreactive Research in the Social Sciences . Chicago: Rand McNally.

Wolanik Boström, K., & Öhlander, M. (2015). Mobile Physicians Making Sense of Culture(s). On Mobile Everyday Ethnography. Ethnologia Europaea, 45 (1), 7–24.

Download references

Author information

Authors and affiliations.

Teesside University Business School, Teesside University, Middlesbrough, UK

Malgorzata Ciesielska

Umeå University, Umea, Sweden

Katarzyna W. Boström

Stockholm University, Stockholm, Sweden

Magnus Öhlander

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

Teesside University Business School, Teesside University, Middlesbrough, United Kingdom

Akademia Leona Koźmińskiego, Warsaw, Poland

Dariusz Jemielniak

Rights and permissions

Reprints and permissions

Copyright information

© 2018 The Author(s)

About this chapter

Ciesielska, M., Boström, K.W., Öhlander, M. (2018). Observation Methods. In: Ciesielska, M., Jemielniak, D. (eds) Qualitative Methodologies in Organization Studies. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-65442-3_2

Download citation

DOI : https://doi.org/10.1007/978-3-319-65442-3_2

Published : 14 December 2017

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-319-65441-6

Online ISBN : 978-3-319-65442-3

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Memberships

Observational Research Method explained

Observational Research Method: this article explains the concept of Observational Research Method in a practical way. The article begins with an introduction and the general definition of the term, followed by an explanation of why observational research is important, its advantages and disadvantages, and a practical example. Enjoy reading!

What is observational research?

Observational research is a method of collecting data by simply observing and recording the behavior of individuals, animals or objects in their natural environment.

It offers researchers insights into human and animal behavior, revealing patterns and dynamics that would otherwise go unnoticed.

This article explores the definition, types, advantages, and disadvantages of observational research. Several examples, including its application in market research, will show you how this approach improves our human understanding of the world.

Observational research: collecting insights unobtrusively

Definition of observational research.

Observational studies serve as a means of answering research questions through careful observation of subjects, without any interference or manipulation by the researcher.

Unlike traditional experiments, these studies lack control and treatment groups, allowing researchers to collect data in a natural setting without imposing predetermined conditions.

Observational studies are generally of a qualitative nature, with both exploratory and explanatory purposes, providing insight into the complexity of particular phenomena.

While quantitative observational studies also exist, they are less common compared to the qualitative studies.

Observational research is widely used in disciplines such as the exact sciences, medicine and social sciences.

Often, ethical or practical considerations prohibit researchers from conducting controlled experiments, leading them to opt for observational studies instead.

The lack of control and treatment groups can pose challenges in drawing conclusions. The risk of confounding variables and observer bias affecting the analysis is high, highlighting the importance of careful interpretation.

Types of observational research

Figure 1 – Types of observational research

Some common types of observational research are:

Naturalistic observation

In naturalistic observation, researchers observe participants in their natural environment, without any interference or disturbance. The aim is to study the behavior and interactions of individuals or groups as they occur in their natural environment.

Structured observation

In structured observation, a predetermined set of behaviors or variables is observed and systematically recorded.

The researchers use specific behavioral categories or measurement tools to collect data .

Participant observation

Participant observation means that the researcher actively participates in the activities or interactions of the participants while they are being observed.

This gives the researcher a deeper insight into the experience and perspectives of the participants.

Covert observation

In the case of a covert observational study, the researcher tries to make himself known to the participants as little as possible.

They observe and record behavior without the participants being aware of the observation. This minimizes the risk of deviant behaviour.

Cross-sectional study

In cross-sectional studies, data is collected at a single point in time or over a short period of time.

The goal is to get a snapshot of the behavior or phenomenon being studied.

Longitudinal study

Longitudinal studies involve following and observing participants over a longer period of time. This makes it possible to identify and analyze changes in behavior or patterns over time.

Choosing the right type of observational study depends on the research question, the aim of the study and the available resources and time. Each type has its own strengths and weaknesses and can be adapted to the specific needs of the research.

Research Methods For Business Students Course A-Z guide to writing a rockstar Research Paper with a bulletproof Research Methodology!   More information

Steps in observational research

Below you will find the steps that are followed when setting up an observational research.

Step 1: determine research topic and objectives

The first step involves determining the phenomenon to be observed and the reasons why it is important. Observational studies are especially suitable when an experiment is not an option for practical or ethical reasons. The research topic may also depend on natural behaviour.

As an example, let’s consider a researcher who is interested in the interactions of teens in their social situations. The researcher wants to investigate whether having a smartphone influences the social interactions of the teenagers. Conducting an experiment can be tricky because smartphone use should not be manipulated.

Step 2: choose the type of observation and techniques

Think about what needs to be observed. Does the researcher go in without preconceived notion? Is there another research method that makes more sense to use? Is it important for the analysis that the researcher is present during the observation? If so, a covert observation is already ruled out.

In the example described earlier, several options are possible. The observations could be performed by observing the teens in different situations. It may also be considered to have the observer join a social group and actively participate in their interactions while the group is being observed. Hidden cameras can also be used to record teens’ social interactions in a controlled environment.

Step 3: set up the observational study

There are a number of things to consider before starting the observation.

First, you need to plan ahead. If the participants are observed in a social setting such as community centers or schools, clear agreements should be made and permission should be given. Informed consent might be required. Decide in advance the observational research methods you will use for data collection. Are notes taken? Or video images or audio recordings?

Step 4: before the observation

Once the type of observation has been chosen, the research technique has been decided on and the correct time and place have been determined, it is time to conduct the observation.

In the example, it can be considered to observe two situations, for example one with smartphones and one without smartphones. When conducting the observation, it is important to take confounding variables into account.

Step 5: analyzing data

After completing the observation, it is important to immediately record the first clues, thoughts and impressions. If the observation has been recorded, this recording must be transcribed. Subsequently, a thematic or content analysis must be carried out.

Observations are often exploratory and have an open character. That is why this analysis fits well with this method.

Step 6: discuss next steps

Observational studies are generally exploratory in nature and therefore usually do not immediately yield definitive conclusions. This is mainly because of the risk of observational bias and confounding variables. If the researcher is satisfied with the conclusions that have been reached, it may be useful to switch to another research method, like an experiment.

Examples of observational research

Observational research has led to several revolutionary results that have forever changed our understanding of the world and human behavior.

Some examples of this are:

Development of Darwin’s theory of evolution

Charles Darwin used observational research during his travels on the ship HMS Beagle. Observations of various animal species in their natural environment, such as birds in the Galapagos Islands, allowed Darwin to gather evidence for his theory of evolution.

This revolutionary theory has completely changed the understanding of the origin and diversity of species of creatures.

Discovery of penicillin

Sir Alexander Fleming accidentally discovered the effect of penicillin, a revolutionary antibiotic, through observational research.

He observed that a fungus called Penicillium notatum destroyed bacteria in a petri dish.

This discovery laid the foundation for the development of modern antibiotics and has had an enormous impact on medicine and the treatment of infectious diseases.

Confirmation of Einstein’s theory of relativity

During a solar eclipse in 1919, Arthur Eddington and his team conducted observational research to test the predictions of Einstein’s general theory of relativity.

By observing the positions of stars during the eclipse, they were able to confirm the deflection of light by the sun’s gravity. This experimental evidence supported Einstein’s theory and marked a revolutionary breakthrough in physics.

Research into the effects of smoking on health

One of the most influential observational studies was the study of the relationship between smoking and health problems, particularly lung cancer.

By observing large groups of smokers over a long period of time and collecting data on their smoking behavior and health outcomes, it was shown that there is a strong association between smoking and the risk of lung cancer.

These findings have led to a better understanding of the harmful effects of smoking and have contributed to the promotion of anti-smoking measures and health education.

Pros and cons

Observational research has several advantages and disadvantages that need to be considered before choosing the right research approach.

Advantages of observational research

Authentic behaviour.

By observing people, animals or objects in their natural environment, researchers can study authentic behavior.

That means that the observations take place in real situations and not artificial laboratory conditions.

This allows researchers to study behavior as it actually occurs. This increases scientific validity.

Detailed information

Observational research offers the opportunity to collect detailed information about behaviour, interactions and context.

Researchers can observe specific behaviors such as nonverbal cues, responses to stimuli, and social dynamics. This leads to a deep understanding of the phenomenon being studied.

Flexibility

Observational research can be adapted to different research questions and contexts. Researchers can tailor the observations to the specific situations and variables they want to study. This gives them the flexibility to focus on specific aspects of behaviour, for example.

Disadvantages of observational research

Limited control.

In observational research, researchers have limited control over the conditions and variables they observe. They cannot perform experimental manipulations or control specific environmental factors.

Observer bias

Observer bias refers to the subjective interpretation of the observations by the researcher. Researchers may unconsciously project their own biases, expectations, or interpretations onto the observed behaviors. This could jeopardize the objectivity of the investigation.

Time consuming

Now it’s your turn

What do you think? Do you recognize the explanation about observational research? Are you familiar with observational research? What do you think are the main benefits of observational research? Have you ever read or experienced an observational study that has given you new insights? Do you have tips or other comments?

Share your experience and knowledge in the comments box below.

More information

• Barick, R. (2021). Research Methods For Business Students . Retrieved 02/16/2024 from Udemy.
• Rosenbaum, P. R. (2005). Observational study . Encyclopedia of statistics in behavioral science.
• Altmann, J. (1974). Observational study of behavior: sampling methods . Behaviour, 49(3-4), 227-266.
• Jepsen, P., Johnsen, S. P., Gillman, M. W., & Sørensen, H. T. (2004). Interpretation of observational studies . Heart, 90(8), 956-960.
• Ligthelm, R. J., Borzì, V., Gumprecht, J., Kawamori, R., Wenying, Y., & Valensi, P. (2007). Importance of observational studies in clinical practice . Clinical therapeutics , 29(6), 1284-1292.

How to cite this article: Janse, B. (2023). Observational Research Method . Retrieved [insert date] from Toolshero: https://www.toolshero.com/research/observational-research/

Original publication date: 10/17/2023 | Last update: 01/02/2024

Add a link to this page on your website: <a href=” https://www.toolshero.com/research/observational-research/”> Toolshero: Observational Research Method</a>

Did you find this article interesting?

Your rating is more than welcome or share this article via Social media!

Average rating 4 / 5. Vote count: 4

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Ben Janse is a young professional working at ToolsHero as Content Manager. He is also an International Business student at Rotterdam Business School where he focusses on analyzing and developing management models. Thanks to his theoretical and practical knowledge, he knows how to distinguish main- and side issues and to make the essence of each article clearly visible.

Related ARTICLES

Respondents: the definition, meaning and the recruitment

Market Research: the Basics and Tools

Gartner Magic Quadrant report and basics explained

Univariate Analysis: basic theory and example

Bivariate Analysis in Research explained

Contingency Table: the Theory and an Example

Also interesting.

Content Analysis explained plus example

Field Research explained

Research Ethics explained

Leave a reply cancel reply.

You must be logged in to post a comment.

BOOST YOUR SKILLS

Toolshero supports people worldwide ( 10+ million visitors from 100+ countries ) to empower themselves through an easily accessible and high-quality learning platform for personal and professional development.

By making access to scientific knowledge simple and affordable, self-development becomes attainable for everyone, including you! Join our learning platform and boost your skills with Toolshero.

POPULAR TOPICS

• Change Management
• Marketing Theories
• Problem Solving Theories
• Psychology Theories

ABOUT TOOLSHERO

• Free Toolshero e-book
• Memberships & Pricing

Observation

Observation, as the name implies, is a way of collecting data through observing. This data collection method is classified as a participatory study, because the researcher has to immerse herself in the setting where her respondents are, while taking notes and/or recording. Observation data collection method may involve watching, listening, reading, touching, and recording behavior and characteristics of phenomena.

Observation as a data collection method can be structured or unstructured. In structured or systematic observation, data collection is conducted using specific variables and according to a pre-defined schedule. Unstructured observation, on the other hand, is conducted in an open and free manner in a sense that there would be no pre-determined variables or objectives.

Moreover, this data collection method can be divided into overt or covert categories. In overt observation research subjects are aware that they are being observed. In covert observation, on the other hand, the observer is concealed and sample group members are not aware that they are being observed. Covert observation is considered to be more effective because in this case sample group members are likely to behave naturally with positive implications on the authenticity of research findings.

Advantages of observation data collection method include direct access to research phenomena, high levels of flexibility in terms of application and generating a permanent record of phenomena to be referred to later. At the same time, this method is disadvantaged with longer time requirements, high levels of observer bias, and impact of observer on primary data, in a way that presence of observer may influence the behaviour of sample group elements.

It is important to note that observation data collection method may be associated with certain ethical issues. As it is discussed further below in greater details, fully informed consent of research participant(s) is one of the basic ethical considerations to be adhered to by researchers. At the same time, the behaviour of sample group members may change with negative implications on the level of research validity if they are notified about the presence of the observer.

This delicate matter needs to be addressed by consulting with dissertation supervisor, and commencing the primary data collection process only after ethical aspects of the issue have been approved by the supervisor.

My e-book,  The Ultimate Guide to Writing a Dissertation in Business Studies: a step by step assistance  offers practical assistance to complete a dissertation with minimum or no stress. The e-book covers all stages of writing a dissertation starting from the selection to the research area to submitting the completed version of the work within the deadline.

John Dudovskiy

Research Methods In Psychology

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

Research methods in psychology are systematic procedures used to observe, describe, predict, and explain behavior and mental processes. They include experiments, surveys, case studies, and naturalistic observations, ensuring data collection is objective and reliable to understand and explain psychological phenomena.

Hypotheses are statements about the prediction of the results, that can be verified or disproved by some investigation.

There are four types of hypotheses :

• Null Hypotheses (H0 ) – these predict that no difference will be found in the results between the conditions. Typically these are written ‘There will be no difference…’
• Alternative Hypotheses (Ha or H1) – these predict that there will be a significant difference in the results between the two conditions. This is also known as the experimental hypothesis.
• One-tailed (directional) hypotheses – these state the specific direction the researcher expects the results to move in, e.g. higher, lower, more, less. In a correlation study, the predicted direction of the correlation can be either positive or negative.
• Two-tailed (non-directional) hypotheses – these state that a difference will be found between the conditions of the independent variable but does not state the direction of a difference or relationship. Typically these are always written ‘There will be a difference ….’

All research has an alternative hypothesis (either a one-tailed or two-tailed) and a corresponding null hypothesis.

Once the research is conducted and results are found, psychologists must accept one hypothesis and reject the other. 

So, if a difference is found, the Psychologist would accept the alternative hypothesis and reject the null.  The opposite applies if no difference is found.

Sampling techniques

Sampling is the process of selecting a representative group from the population under study.

A sample is the participants you select from a target population (the group you are interested in) to make generalizations about.

Representative means the extent to which a sample mirrors a researcher’s target population and reflects its characteristics.

Generalisability means the extent to which their findings can be applied to the larger population of which their sample was a part.

• Volunteer sample : where participants pick themselves through newspaper adverts, noticeboards or online.
• Opportunity sampling : also known as convenience sampling , uses people who are available at the time the study is carried out and willing to take part. It is based on convenience.
• Random sampling : when every person in the target population has an equal chance of being selected. An example of random sampling would be picking names out of a hat.
• Systematic sampling : when a system is used to select participants. Picking every Nth person from all possible participants. N = the number of people in the research population / the number of people needed for the sample.
• Stratified sampling : when you identify the subgroups and select participants in proportion to their occurrences.
• Snowball sampling : when researchers find a few participants, and then ask them to find participants themselves and so on.
• Quota sampling : when researchers will be told to ensure the sample fits certain quotas, for example they might be told to find 90 participants, with 30 of them being unemployed.

Experiments always have an independent and dependent variable .

• The independent variable is the one the experimenter manipulates (the thing that changes between the conditions the participants are placed into). It is assumed to have a direct effect on the dependent variable.
• The dependent variable is the thing being measured, or the results of the experiment.

Operationalization of variables means making them measurable/quantifiable. We must use operationalization to ensure that variables are in a form that can be easily tested.

For instance, we can’t really measure ‘happiness’, but we can measure how many times a person smiles within a two-hour period. 

By operationalizing variables, we make it easy for someone else to replicate our research. Remember, this is important because we can check if our findings are reliable.

Extraneous variables are all variables which are not independent variable but could affect the results of the experiment.

It can be a natural characteristic of the participant, such as intelligence levels, gender, or age for example, or it could be a situational feature of the environment such as lighting or noise.

Demand characteristics are a type of extraneous variable that occurs if the participants work out the aims of the research study, they may begin to behave in a certain way.

For example, in Milgram’s research , critics argued that participants worked out that the shocks were not real and they administered them as they thought this was what was required of them. 

Extraneous variables must be controlled so that they do not affect (confound) the results.

Randomly allocating participants to their conditions or using a matched pairs experimental design can help to reduce participant variables. 

Situational variables are controlled by using standardized procedures, ensuring every participant in a given condition is treated in the same way

Experimental Design

Experimental design refers to how participants are allocated to each condition of the independent variable, such as a control or experimental group.

• Independent design ( between-groups design ): each participant is selected for only one group. With the independent design, the most common way of deciding which participants go into which group is by means of randomization. 
• Matched participants design : each participant is selected for only one group, but the participants in the two groups are matched for some relevant factor or factors (e.g. ability; sex; age).
• Repeated measures design ( within groups) : each participant appears in both groups, so that there are exactly the same participants in each group.
• The main problem with the repeated measures design is that there may well be order effects. Their experiences during the experiment may change the participants in various ways.
• They may perform better when they appear in the second group because they have gained useful information about the experiment or about the task. On the other hand, they may perform less well on the second occasion because of tiredness or boredom.
• Counterbalancing is the best way of preventing order effects from disrupting the findings of an experiment, and involves ensuring that each condition is equally likely to be used first and second by the participants.

If we wish to compare two groups with respect to a given independent variable, it is essential to make sure that the two groups do not differ in any other important way. 

Experimental Methods

All experimental methods involve an iv (independent variable) and dv (dependent variable)..

• Field experiments are conducted in the everyday (natural) environment of the participants. The experimenter still manipulates the IV, but in a real-life setting. It may be possible to control extraneous variables, though such control is more difficult than in a lab experiment.
• Natural experiments are when a naturally occurring IV is investigated that isn’t deliberately manipulated, it exists anyway. Participants are not randomly allocated, and the natural event may only occur rarely.

Case studies are in-depth investigations of a person, group, event, or community. It uses information from a range of sources, such as from the person concerned and also from their family and friends.

Many techniques may be used such as interviews, psychological tests, observations and experiments. Case studies are generally longitudinal: in other words, they follow the individual or group over an extended period of time. 

Case studies are widely used in psychology and among the best-known ones carried out were by Sigmund Freud . He conducted very detailed investigations into the private lives of his patients in an attempt to both understand and help them overcome their illnesses.

Case studies provide rich qualitative data and have high levels of ecological validity. However, it is difficult to generalize from individual cases as each one has unique characteristics.

Correlational Studies

Correlation means association; it is a measure of the extent to which two variables are related. One of the variables can be regarded as the predictor variable with the other one as the outcome variable.

Correlational studies typically involve obtaining two different measures from a group of participants, and then assessing the degree of association between the measures. 

The predictor variable can be seen as occurring before the outcome variable in some sense. It is called the predictor variable, because it forms the basis for predicting the value of the outcome variable.

Relationships between variables can be displayed on a graph or as a numerical score called a correlation coefficient.

• If an increase in one variable tends to be associated with an increase in the other, then this is known as a positive correlation .
• If an increase in one variable tends to be associated with a decrease in the other, then this is known as a negative correlation .
• A zero correlation occurs when there is no relationship between variables.

After looking at the scattergraph, if we want to be sure that a significant relationship does exist between the two variables, a statistical test of correlation can be conducted, such as Spearman’s rho.

The test will give us a score, called a correlation coefficient . This is a value between 0 and 1, and the closer to 1 the score is, the stronger the relationship between the variables. This value can be both positive e.g. 0.63, or negative -0.63.

A correlation between variables, however, does not automatically mean that the change in one variable is the cause of the change in the values of the other variable. A correlation only shows if there is a relationship between variables.

Correlation does not always prove causation, as a third variable may be involved. 

Interview Methods

Interviews are commonly divided into two types: structured and unstructured.

A fixed, predetermined set of questions is put to every participant in the same order and in the same way. 

Responses are recorded on a questionnaire, and the researcher presets the order and wording of questions, and sometimes the range of alternative answers.

The interviewer stays within their role and maintains social distance from the interviewee.

There are no set questions, and the participant can raise whatever topics he/she feels are relevant and ask them in their own way. Questions are posed about participants’ answers to the subject

Unstructured interviews are most useful in qualitative research to analyze attitudes and values.

Though they rarely provide a valid basis for generalization, their main advantage is that they enable the researcher to probe social actors’ subjective point of view. 

Questionnaire Method

Questionnaires can be thought of as a kind of written interview. They can be carried out face to face, by telephone, or post.

The choice of questions is important because of the need to avoid bias or ambiguity in the questions, ‘leading’ the respondent or causing offense.

• Open questions are designed to encourage a full, meaningful answer using the subject’s own knowledge and feelings. They provide insights into feelings, opinions, and understanding. Example: “How do you feel about that situation?”
• Closed questions can be answered with a simple “yes” or “no” or specific information, limiting the depth of response. They are useful for gathering specific facts or confirming details. Example: “Do you feel anxious in crowds?”

Its other practical advantages are that it is cheaper than face-to-face interviews and can be used to contact many respondents scattered over a wide area relatively quickly.

Observations

There are different types of observation methods :

• Covert observation is where the researcher doesn’t tell the participants they are being observed until after the study is complete. There could be ethical problems or deception and consent with this particular observation method.
• Overt observation is where a researcher tells the participants they are being observed and what they are being observed for.
• Controlled : behavior is observed under controlled laboratory conditions (e.g., Bandura’s Bobo doll study).
• Natural : Here, spontaneous behavior is recorded in a natural setting.
• Participant : Here, the observer has direct contact with the group of people they are observing. The researcher becomes a member of the group they are researching.  
• Non-participant (aka “fly on the wall): The researcher does not have direct contact with the people being observed. The observation of participants’ behavior is from a distance

Pilot Study

A pilot  study is a small scale preliminary study conducted in order to evaluate the feasibility of the key s teps in a future, full-scale project.

A pilot study is an initial run-through of the procedures to be used in an investigation; it involves selecting a few people and trying out the study on them. It is possible to save time, and in some cases, money, by identifying any flaws in the procedures designed by the researcher.

A pilot study can help the researcher spot any ambiguities (i.e. unusual things) or confusion in the information given to participants or problems with the task devised.

Sometimes the task is too hard, and the researcher may get a floor effect, because none of the participants can score at all or can complete the task – all performances are low.

The opposite effect is a ceiling effect, when the task is so easy that all achieve virtually full marks or top performances and are “hitting the ceiling”.

Research Design

In cross-sectional research , a researcher compares multiple segments of the population at the same time

Sometimes, we want to see how people change over time, as in studies of human development and lifespan. Longitudinal research is a research design in which data-gathering is administered repeatedly over an extended period of time.

In cohort studies , the participants must share a common factor or characteristic such as age, demographic, or occupation. A cohort study is a type of longitudinal study in which researchers monitor and observe a chosen population over an extended period.

Triangulation means using more than one research method to improve the study’s validity.

Reliability

Reliability is a measure of consistency, if a particular measurement is repeated and the same result is obtained then it is described as being reliable.

• Test-retest reliability :  assessing the same person on two different occasions which shows the extent to which the test produces the same answers.
• Inter-observer reliability : the extent to which there is an agreement between two or more observers.

Meta-Analysis

A meta-analysis is a systematic review that involves identifying an aim and then searching for research studies that have addressed similar aims/hypotheses.

This is done by looking through various databases, and then decisions are made about what studies are to be included/excluded.

Strengths: Increases the conclusions’ validity as they’re based on a wider range.

Weaknesses: Research designs in studies can vary, so they are not truly comparable.

Peer Review

A researcher submits an article to a journal. The choice of the journal may be determined by the journal’s audience or prestige.

The journal selects two or more appropriate experts (psychologists working in a similar field) to peer review the article without payment. The peer reviewers assess: the methods and designs used, originality of the findings, the validity of the original research findings and its content, structure and language.

Feedback from the reviewer determines whether the article is accepted. The article may be: Accepted as it is, accepted with revisions, sent back to the author to revise and re-submit or rejected without the possibility of submission.

The editor makes the final decision whether to accept or reject the research report based on the reviewers comments/ recommendations.

Peer review is important because it prevent faulty data from entering the public domain, it provides a way of checking the validity of findings and the quality of the methodology and is used to assess the research rating of university departments.

Peer reviews may be an ideal, whereas in practice there are lots of problems. For example, it slows publication down and may prevent unusual, new work being published. Some reviewers might use it as an opportunity to prevent competing researchers from publishing work.

Some people doubt whether peer review can really prevent the publication of fraudulent research.

The advent of the internet means that a lot of research and academic comment is being published without official peer reviews than before, though systems are evolving on the internet where everyone really has a chance to offer their opinions and police the quality of research.

Types of Data

• Quantitative data is numerical data e.g. reaction time or number of mistakes. It represents how much or how long, how many there are of something. A tally of behavioral categories and closed questions in a questionnaire collect quantitative data.
• Qualitative data is virtually any type of information that can be observed and recorded that is not numerical in nature and can be in the form of written or verbal communication. Open questions in questionnaires and accounts from observational studies collect qualitative data.
• Primary data is first-hand data collected for the purpose of the investigation.
• Secondary data is information that has been collected by someone other than the person who is conducting the research e.g. taken from journals, books or articles.

Validity means how well a piece of research actually measures what it sets out to, or how well it reflects the reality it claims to represent.

Validity is whether the observed effect is genuine and represents what is actually out there in the world.

• Concurrent validity is the extent to which a psychological measure relates to an existing similar measure and obtains close results. For example, a new intelligence test compared to an established test.
• Face validity : does the test measure what it’s supposed to measure ‘on the face of it’. This is done by ‘eyeballing’ the measuring or by passing it to an expert to check.
• Ecological validit y is the extent to which findings from a research study can be generalized to other settings / real life.
• Temporal validity is the extent to which findings from a research study can be generalized to other historical times.

Features of Science

• Paradigm – A set of shared assumptions and agreed methods within a scientific discipline.
• Paradigm shift – The result of the scientific revolution: a significant change in the dominant unifying theory within a scientific discipline.
• Objectivity – When all sources of personal bias are minimised so not to distort or influence the research process.
• Empirical method – Scientific approaches that are based on the gathering of evidence through direct observation and experience.
• Replicability – The extent to which scientific procedures and findings can be repeated by other researchers.
• Falsifiability – The principle that a theory cannot be considered scientific unless it admits the possibility of being proved untrue.

Statistical Testing

A significant result is one where there is a low probability that chance factors were responsible for any observed difference, correlation, or association in the variables tested.

If our test is significant, we can reject our null hypothesis and accept our alternative hypothesis.

If our test is not significant, we can accept our null hypothesis and reject our alternative hypothesis. A null hypothesis is a statement of no effect.

In Psychology, we use p < 0.05 (as it strikes a balance between making a type I and II error) but p < 0.01 is used in tests that could cause harm like introducing a new drug.

A type I error is when the null hypothesis is rejected when it should have been accepted (happens when a lenient significance level is used, an error of optimism).

A type II error is when the null hypothesis is accepted when it should have been rejected (happens when a stringent significance level is used, an error of pessimism).

Ethical Issues

• Informed consent is when participants are able to make an informed judgment about whether to take part. It causes them to guess the aims of the study and change their behavior.
• To deal with it, we can gain presumptive consent or ask them to formally indicate their agreement to participate but it may invalidate the purpose of the study and it is not guaranteed that the participants would understand.
• Deception should only be used when it is approved by an ethics committee, as it involves deliberately misleading or withholding information. Participants should be fully debriefed after the study but debriefing can’t turn the clock back.
• All participants should be informed at the beginning that they have the right to withdraw if they ever feel distressed or uncomfortable.
• It causes bias as the ones that stayed are obedient and some may not withdraw as they may have been given incentives or feel like they’re spoiling the study. Researchers can offer the right to withdraw data after participation.
• Participants should all have protection from harm . The researcher should avoid risks greater than those experienced in everyday life and they should stop the study if any harm is suspected. However, the harm may not be apparent at the time of the study.
• Confidentiality concerns the communication of personal information. The researchers should not record any names but use numbers or false names though it may not be possible as it is sometimes possible to work out who the researchers were.

What would you like to do?

Category Select one Administrative Business Development Clinical Clinical Development College Job Compliance Corporate Services Engineering Finance Government Affairs Health Economics Human Resources Information Systems Law/Legal Logistics Maintenance Manufacturing Marketing Medical Affairs Medical Services Operations Postdoctoral Process Development Procurement Project Management Public Relations Quality Regulatory Research Safety Sales Sales & Marketing Operations Scientific Strategy and Innovation Supply Chain Training Value and Access

Job Type Select one Full time Part time

Where would you like to work?

City Select one Albany Albuquerque Algiers Atlanta Baltimore Bangkok Beijing Birmingham Birmingham Boca Raton Bogotá Boise Boston Bradenton Breda Bristol Bucharest Budapest Buenos Aires Burnaby Cairo Cambridge Cambridge Chandler Changchun Changsha Charleston Charlotte Chicago Chongqing Cincinnati Cleveland Dallas Danbury Daqing Dayton Denver Detroit Diegem Dongguan Dothan Dublin Duluth East Pensacola Heights Edinburgh Erie Fort Myers Gilbert Glasgow Greater Manchester Guangzhou Handan Harbin Hartford Holly Springs Hong Kong Houston Huntington Indianapolis Istanbul Jacksonville Jiaxing Juncos Jupiter Kansas City Kuala Lumpur Ladrido Lanzhou Leeds Lexington-Fayette Lisbon Little Rock London Los Angeles Madrid Manchester Melbourne Mexico City Miami Milan Minneapolis-St. Paul-Bloomington Mississauga Montreal Munich Nantong New Albany New Albany New Haven New York Newark Orlando Paris Parkersburg Philadelphia Phoenix Pittsburgh Portland Prague Raleigh Richmond Riyadh Rochester Rockville Saint Paul San Diego San Francisco Santa Clara São Paulo Seattle Seoul Shanghai Shenyang Singapore Søborg Sofia South San Francisco Southampton St Louis Suqian Sydney Taichung Taipei Taiyuan Tampa Thousand Oaks Tokyo Toledo Tucson Utrecht Uxbridge Uxbridge Vancouver Vienna Virginia Beach Warsaw Washington D.C. West Greenwich Xi'an Yangzhou Zhengzhou

Country Select one Algeria Argentina Australia Austria Belgium Brazil Bulgaria Canada China Colombia Czechia Denmark Egypt France Germany Hong Kong SAR Hungary Ireland Italy Japan Malaysia Mexico Netherlands Poland Portugal Puerto Rico Remote Romania Saudi Arabia Singapore South Korea Spain Taiwan Thailand Türkiye United Arab Emirates United Kingdom United States

State Select one Alabama Alberta Algiers Province Arizona Arkansas Bangkok Bavaria Beijing Municipality Bogota D.C. British Columbia București Budapest Buenos Aires F.D. Cairo Governorate California Capital Region Central and Western District Chongqing Municipality Colorado Connecticut District of Columbia Dubai England Flanders Florida Galicia Gansu Georgia Guangdong Hebei Heilongjiang Henan Hlavní město Praha Hunan Idaho Île-de-France Region Illinois Indiana Istanbul Jiangsu Jilin Juncos Kansas Kentucky Kuala Lumpur Leinster Liaoning Lisbon District Lombardy Madrid Maine Maryland Massachusetts Mazovia Mexico City Michigan Minnesota Missouri New Hampshire New Jersey New Mexico New South Wales New York North Brabant North Carolina Ohio Ontario Pennsylvania Quebec Rhode Island Riyadh Region São Paulo Scotland Seoul Shaanxi Shanghai Municipality Shanxi Singapore Sofia-grad Taiwan Texas Tokyo Utrecht Vienna Virginia Washington West Virginia Zhejiang

Observational Research Manager - US Remote

RADIUS Select Miles 5 miles 15 miles 25 miles 35 miles 50 miles

HOW MIGHT YOU DEFY IMAGINATION?

You’ve worked hard to become the professional you are today and are now ready to take the next step in your career. How will you put your skills, experience and passion to work toward your goals? At Amgen, our shared mission—to serve patients—drives all that we do. It is key to our becoming one of the world’s leading biotechnology companies, reaching over 10 million patients worldwide. Come do your best work alongside other innovative, driven professionals in this meaningful role.

Observational Research Manager

What you will do

Let’s do this. Let’s change the world. Are you searching for a new, dynamic opportunity working with multiple teams to generate real world evidence supporting a wide variety of business needs? If so, we invite you to explore Amgen’s Center for Observational Research.

With a constantly growing demand for information from regulatory and reimbursement agencies, Observational Research (OR) is a critical component in drug development and commercialization. Amgen’s Center for Observational Research (CfOR) partners with internal and external teams to generate real world evidence for multiple partners across the entire product lifecycle. We generate evidence to inform the frequency, distribution, clinical burden, natural history, and clinical course of disease, the design of clinical trials, health resource utilization, drug utilization patterns, and the safety and effectiveness of our medicines.

Let’s do this. Let’s change the world. In this vital role the Observational Research Manager will be a member of the Bone team within CfOR, and will be responsible for gathering insights and building collaborations and capabilities to advance the generation, use, and dissemination of real-world evidence (RWE) related to disease state epidemiology, treatment patterns, and medication safety and effectiveness to inform regulatory decision making and enhance access and use of Amgen’s medicines.

Responsibilities:

• Support the design, generation and delivery of RWE evaluating the safety and effectiveness of medicines to inform regulatory decision making (e.g., label expansion/change) and fulfill regulatory requirements across the globe.
• Managing research projects involving the analysis of multiple types of data including medical claims, electronic health records, and prospective observational cohort studies.
• Contribute to the development and implementation of innovative analytic methods, and leverage CfOR’s internal data and analytics capabilities and tools to enable rapid, scalable and reproducible RWE.
• Engage with cross-functional partners to promote the awareness, understanding, and use of observational research methods and enhance the use of RWE that can reduce the time and cost of drug development, and answer key business questions.
• Facilitate the dissemination of RWE through publications, congress presentations, trainings, and development of scientific/promotional resources targeting a broad base of external stakeholders (e.g., health care providers, payers, integrated delivery networks).
• Contribute to CfOR’s mission in progressing innovative epidemiological methods and analytical capabilities to support CfOR’s leadership role within Amgen and across industry.

What we expect of you

We are all different, yet we all use our unique contributions to serve patients. The Observational Research Manager professional we seek is a research leader with these qualifications.

Basic Qualifications:

Doctorate degree

Master’s degree and 3 years of scientific experience

Bachelor’s degree and 5 years of scientific experience

Associate’s degree and 10 years of scientific experience

High school diploma / GED and 12 years of scientific experience

Preferred Qualifications:

• Doctorate in Epidemiology or other subject with high observational research content.
• Experience in the design, execution and analysis of observation research studies within Pharmaceutical and/or Public Health settings.
• Experience in research to support drug development.
• Experience working with secondary data systems including administrative claims, EMR and registries.
• Experience in observational research project planning and management.
• Excellent communication, presentation and interpersonal skills.
• Experience working in multi-disciplinary teams.

What you can expect of us

As we work to develop treatments that take care of others, we also work to care for our teammates’ professional and personal growth and well-being.

The expected annual salary range for this role in the U.S. (excluding Puerto Rico) is posted. Actual salary will vary based on several factors including but not limited to, relevant skills, experience, and qualifications.

Amgen offers a Total Rewards Plan comprising health and welfare plans for staff and eligible dependents, financial plans with opportunities to save towards retirement or other goals, work/life balance, and career development opportunities including:

• Comprehensive employee benefits package, including a Retirement and Savings Plan with generous company contributions, group medical, dental and vision coverage, life and disability insurance, and flexible spending accounts.
• A discretionary annual bonus program, or for field sales representatives, a sales-based incentive plan
• Stock-based long-term incentives
• Award-winning time-off plans and bi-annual company-wide shutdowns
• Flexible work models, including remote work arrangements, where possible

for a career that defies imagination

Objects in your future are closer than they appear. Join us.

careers.amgen.com

Amgen is an Equal Opportunity employer and will consider you without regard to your race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or disability status.

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

SHARE THIS JOB

Sign up for job alerts.

Stay up to date on Amgen news and opportunities. Sign up to receive alerts about positions that suit your skills and career interests.

Category* Select one Administrative Business Development Clinical Clinical Development College Job Compliance Corporate Services Engineering Finance Government Affairs Health Economics Human Resources Information Systems Law/Legal Logistics Maintenance Manufacturing Marketing Medical Affairs Medical Services Operations Postdoctoral Process Development Procurement Project Management Public Relations Quality Regulatory Research Safety Sales Sales & Marketing Operations Scientific Strategy and Innovation Supply Chain Training Value and Access

• Clinical, Washington D.C., District of Columbia, United States Remove

Confirm Email

By submitting your information, you acknowledge that you have read our privacy policy (this content opens in new window)and consent to receive email communication from.

Related Content

Mission, Vision and Values

Diversity, Inclusion and Belonging

Amgen Stories

• Brief Report
• Open access
• Published: 23 April 2024

Case study observational research: inflammatory cytokines in the bronchial epithelial lining fluid of COVID-19 patients with acute hypoxemic respiratory failure

• Kazuki Sudo 1 , 2 ,
• Mao Kinoshita 1 ,
• Ken Kawaguchi 1 ,
• Kohsuke Kushimoto 1 ,
• Ryogo Yoshii 2 ,
• Keita Inoue 2 ,
• Masaki Yamasaki 2 , 3 ,
• Tasuku Matsuyama 4 ,
• Kunihiko Kooguchi 2 ,
• Yasuo Takashima 5 ,
• Masami Tanaka 5 ,
• Kazumichi Matsumoto 6 ,
• Kei Tashiro 5 ,
• Tohru Inaba 6 , 7 ,
• Bon Ohta 4 &
• Teiji Sawa 1 , 8  

Critical Care volume  28 , Article number:  134 ( 2024 ) Cite this article

215 Accesses

1 Altmetric

Metrics details

In this study, the concentrations of inflammatory cytokines were measured in the bronchial epithelial lining fluid (ELF) and plasma in patients with acute hypoxemic respiratory failure (AHRF) secondary to severe coronavirus disease 2019 (COVID-19).

We comprehensively analyzed the concentrations of 25 cytokines in the ELF and plasma of 27 COVID-19 AHRF patients. ELF was collected using the bronchial microsampling method through an endotracheal tube just after patients were intubated for mechanical ventilation.

Compared with those in healthy volunteers, the concentrations of interleukin (IL)-6 (median 27.6 pmol/L), IL-8 (1045.1 pmol/L), IL-17A (0.8 pmol/L), IL-25 (1.5 pmol/L), and IL-31 (42.3 pmol/L) were significantly greater in the ELF of COVID-19 patients than in that of volunteers. The concentrations of MCP-1 and MIP-1β were significantly greater in the plasma of COVID-19 patients than in that of volunteers. The ELF/plasma ratio of IL-8 was the highest among the 25 cytokines, with a median of 737, and the ELF/plasma ratio of IL-6 (median: 218), IL-1β (202), IL-31 (169), MCP-1 (81), MIP-1β (55), and TNF-α (47) were lower.

Conclusions

The ELF concentrations of IL-6, IL-8, IL-17A, IL-25, and IL-31 were significantly increased in COVID-19 patients. Although high levels of MIP-1 and MIP-1β were also detected in the blood samples collected simultaneously with the ELF samples, the results indicated that lung inflammation was highly compartmentalized. Our study demonstrated that a comprehensive analysis of cytokines in the ELF is a feasible approach for understanding lung inflammation and systemic interactions in patients with severe pneumonia.

Severe symptoms of coronavirus disease 2019 (COVID-19), such as acute hypoxemic respiratory failure (AHRF) and cytokine release syndrome, often lead to multiorgan failure and death [ 1 ]. Research has mainly focused on analyses of factors in the blood to study the impact of COVID-19 on immune function, especially in AHRF patients, due to the ease of access in using blood samples [ 2 , 3 , 4 , 5 ]. However, recent reports have shown that the bronchoalveolar immune response in COVID-19 patients presents a distinct local profile that significantly diverges from the immune response observed in the blood of these patients [ 6 , 7 , 8 , 9 ]. Notably, COVID-19 patients with AHRF were reported to have lower blood cytokine levels than those with bacterial sepsis [ 10 ]. These findings suggest a more complex and subtle immune mechanism in these patients, implying that a compartmentalized reaction within their lungs plays a significant role in the efficacy of therapeutic interventions [ 11 , 12 ].

In a study of 27 COVID-19 patients with AHRF who required mechanical ventilation (MV), bronchial epithelial lining fluid (ELF) was collected via bronchial microsampling (MS) [ 13 , 14 ]. This study analyzed 25 cytokine concentrations in both the ELF and plasma of COVID-19 patients using a multiplex bead-based assay and explored the relationship between lung injury severity, as depicted in chest computed tomography (CT) images, and disease duration influenced by viral mutations.

Materials and methods

From March 2021 to March 2022, 27 patients who needed MV for COVID-19-related AHRF at Kyoto Prefectural University of Medicine participated in this study. Their treatments, pneumonia severity index (PSI) [ 15 ], Charlson Comorbidity Index (CCI) [ 16 ], and other comorbidities were documented (Additional file 1 : Table S1, Additional file 2 : Table S2). All patients received high-flow nasal cannula therapy before MV. Four patients, including one who received extracorporeal membrane oxygenation (ECMO), did not survive. The control data for both the ELF and plasma cytokine levels were obtained from six healthy volunteers without lung injury who underwent an elective surgery under anesthesia and tracheal intubation.

Bronchial ELF

The bronchial ELF was collected using an MS probe (model BC-402C; Olympus, Tokyo, Japan) immediately after tracheal intubation [ 13 , 14 ] (Fig.  1 A). The probe was inserted into the segmental bronchus of the right lower lobe via an endotracheal tube; approximately 20 μL of ELF was retrieved from each probe; and this procedure was repeated nine times per patient.

A (1) The microsampling probe (model BC-402C, Olympus Tokyo, Japan) used for collecting bronchial epithelial lining fluid (ELF). (2) The probe tip is comprised of a 2.5-mm outer polyethylene sheath and a 1.9-mm inner polyester fiber rod probe, 20 mm in length, attached to a stainless-steel guide wire. (3) The process of extracting ELF by centrifugation. B The study analyzed cytokine concentrations in COVID-19 patients (normal) with acute hypoxemic respiratory failure (AHRF) compared to healthy volunteers (covid). (1) Cytokine concentrations in the ELF and (2) plasma are presented. C The ratio of bronchial ELF/plasma concentrations. The data included a box plot representing the 25th to 75th percentiles (interquartile range, IQR), the median (centerline), and whisker lines extending to the furthest data points within Q1–1.5 × IQR and Q3 + 1.5 × IQR. Outliers were identified beyond these limits. Significant differences (* p  < 0.05) between the normal and COVID-19 patient groups are marked with an asterisk and were assessed using the Kruskal‒Wallis test with Bonferroni correction

Cytokine measurement

To analyze 25 types of cytokines (Additional file 3 : Table S3), a multiplex bead-based assay (Bio-Plex Pro human cytokine GI-17-Plex for granulocyte colony-stimulating factor (G-CSF), granulocyte–macrophage colony-stimulating factor (GM-CSF), interferon-γ (IFN-γ), interleukin (IL)-1β, IL-2, IL-4, IL-5, IL-6, IL-7, IL-8, IL-10, IL12p70, monocyte chemotactic protein-1 (MCP-1), macrophage inflammatory protein-1β (MIP-1β), and tumor necrosis factor-α (TNF-α), and Th17-15-Plex panels for IL-17A, IL-17F, IL-21, IL-22, IL-23, IL-25, IL-31, IL-33, and soluble CD40 ligand (sCD40L), Bio-Rad, Hercules, CA, USA) was used. Values between zero and less than the manufacturer’s lower limit of quantification (LLOQ) were treated as the LLOQ.

Lung infiltration volume

All 27 patients received CT scans either before transfer or upon admission to the hospital. 3D Slicer software (ver.4.11) was used to calculate the lung infiltration volume (LIV) from chest CT images based on a recently reported method [ 17 , 18 ].

Statistical analyses

This study used SPSS Version 27 for Kruskal–Wallis nonparametric tests and χ 2 tests to compare group medians, with the data presented as medians with interquartile ranges (IQRs).

Cytokine concentrations

The ELF cytokines IL-5, IL-7, IL-12p70, IL-13, and GM-CSF were undetectable in the ELF of 26 patients (Fig.  1 B-1). There were increases in the other cytokines, with median levels of IL-8 at 1045.1 [IQR 178.4–11,688.0] pmol/L and IL-6 at 27.6 [5.2–151.2] pmol/L and measurable levels of IL-17A (0.8 [0.2–2.7] pmol/L), IL-25 (15.3 [0.4–3.9] pmol/L), and IL-31 (42.3 [2.4–85.9] pmol/L). The levels of typical inflammatory cytokines, such as MCP-1, MIP-1β, TNF-α, IL-1β, and IL-10, were not significantly greater in the patients than in healthy individuals.

The cytokines IFN-γ, sCD40L, IL-2, IL-4, IL-17A, IL-17F, IL-23, IL-25, and IL-31 were undetectable in the plasma of the 27 patients, and 26 patients had IL-7, IL-12p70, IL-21, and IL-22 levels that were below the detection limits (Fig.  1 B-2). The detected cytokines included MCP-1, MIP-1β, IL-8, and TNF-α, and only the levels of MCP-1 (2.5 [1.1–5.0] pmol/L) and MIP-1β (0.7 [0.6–1.6] pmol/L) were significantly greater in the patients than in healthy volunteers. Low plasma levels of G-CSF (0.3 [< 0.3–0.4] pmol/L), IL-6 (0.2 [0.0–0.6] pmol/L), IL-10 (0.1 [< 0.1–0.2] pmol/L), IL-13 (0.03 [< 0.03–0.03] pmol/L), and IL-33 (0.24 [< 0.24–0.24] pmol/L) were also detected.

The ratios of cytokine concentrations in the ELF to those in the plasma were calculated (Fig.  1 C). The ELF/plasma ratio was the highest for IL-8, at a median of 737 [IQR 262–11,688] with a detection frequency (%df) of 96.3%, in 27 patients, with the second highest ratios being those of IL-6 (218 [39–1206], %df = 74.1%), IL-1β (202 [21–6434], 3.7%), IL-31 (169 [9–394], 0.0%), MCP-1 (81 [13–323], 96.3%), MIP-1β (55 [0–1121], 96.3%), and TNF-α (47 [7–1560], 81.5%). These ratios underscore the significant disparity in the cytokine concentrations in the ELF and those in the plasma in COVID-19 patients.

Pneumonia severity and cytokine levels

This study assessed 27 COVID-19 patients with AHRF across three pandemic phases in Japan: the 4th wave with the original variant (Mar-Jun 2021), the 5th wave with the Delta variant (Jul–Sep 2021), and the 6th wave with the Omicron variant (Jan-Mar 2022) (Additional file 4 : Table S4, Additional file 5 : Fig. S1). Notably, the CCI and creatinine levels were lower in the patients of the Delta wave than in those of the 4th wave (Additional file 4 : Table S4). The PSI and ferritin levels were greater in the patients of the Omicron wave group than in those of the Delta wave group (Additional file 4 : Table S4, Fig.  2 B-1).

A The pneumonia severity index (PSI), lung infiltration volume (LIV), C-reactive protein (CRP) concentration in the blood, cytokine concentration in the ELF and plasma, provided therapies and comorbidities were recorded for individual patients in order of LIV, including deceased patients (marked by †). B Age, CRP in blood, total cytokine concentration in the ELF and plasma, LIV, and PSI were analyzed. (1) Comparisons among the chronological groups based on when they developed AHRF due to COVID-19. (2) Comparisons among the groups stratified by the severity of pneumonia using the PSI. (3) Comparison among the groups after patients were stratified by the severity of pneumonia using the LIV. The boxplots show the median, individual data points (colored dots), and whisker lines extending to Q1–1.5 × IQR and Q3 + 1.5 × IQR or the last data point within these values. Points outside these limits are considered outliers. Significance († p  < 0.05) was determined using the Kruskal‒Wallis test with Bonferroni correction for multiple comparisons. AHRF Acute hypoxemic respiratory failure, CRP C-reactive protein, ELF Epithelial lining fluid, IQR Interquartile range, LIV Lung infiltration volume (%) [ 17 , 18 ], PSI Pneumonia severity index [ 15 ]

In a study of 27 patients, subgroups were created based on the PSI and LIV. PSI was categorized as mild (PSI < 90), moderate (90  ≦  PSI < 130), or severe (130  ≦  PSI) (Fig. 2 B-2, Additional file 6 : Table S5), while LIV was divided into mild (LIV < 40%), moderate (40%  ≦  LIV < 50%), or severe (50%  ≦  LIV) (Fig. 2 A, B-3, Additional file 7 : Table S6). The present study revealed that there were more female patients in the moderate PSI group and more older patients in the severe PSI group. Interestingly, the sum of the 25 cytokine concentrations in the ELF and plasma samples did not significantly differ among the PSI groups. The severe LIV subgroup had higher C-reactive protein levels than did the moderate LIV subgroup. Notably, the sum of the 25 cytokine concentrations in the ELF was lower in the severe LIV subgroup than in the moderate group, but no significant difference in plasma cytokine concentrations was observed among the LIV subgroups, indicating that there are different patterns of inflammation based on lung injury severity.

In our recent study of 23 COVID-19 patients during Japan's third and fourth waves of the pandemic, including those on ECMO and those with severe AHRF, 109 cytokines were analyzed [ 19 ]. Significant increases in cytokines such as IL-11, M-CSF, stromal cell-derived factor-1 (SDF-1), and soluble tumor necrosis factor receptor 2 (sTNF-R2) were detected, suggesting a link between hematopoietic progenitor cell differentiation and Th1-derived hyperinflammation. Interestingly, the levels of traditional inflammatory cytokines such as IL-1β, IL-6, and TNF-α were not dramatically elevated. These findings indicate that cytokine storms in COVID-19 patients involve different cytokines than those typically associated with inflammation, which highlights the need to understand lung-specific inflammatory responses.

Bronchoscopic MS, a method for directly collecting ELF using a polyester fiber rod probe, was first reported by Ishizaka et al. in 2001 [ 13 ]. This technique has been applied for measuring antibiotics and conducting proteomic analyses of the patients’ ELF samples [ 14 , 20 ]. In this study, we used the MS method for comprehensive cytokine analysis of the ELF in COVID-19 patients. While BAL fluid (BALF) is traditionally used for lung cytokine measurement, obtaining BALF samples carries risks such as pathogen exposure and is challenging in severely ill patients because it can potentially cause hypoxemia and pulmonary edema. MS is less invasive for collecting undiluted ELF, but due to its localized sampling, ELF samples may not reflect all the conditions throughout the lung as comprehensively as BALF samples can.

The cytokine discrepancy between the ELF and blood suggests that the lung inflammation in COVID-19 patients is distinct from that in the systemic circulation of these patients, indicating that there is localized inflammation within the lungs of these patients rather than mere secondary spillover effects. This finding emphasizes the compartmentalization of pulmonary cytokines. However, this study was limited because measurements were taken just once after tracheal intubation, thus preventing observations of the cytokine changes over time. Intriguingly, patients with severe lung disease had lower total cytokine levels in their ELF than patients with other severity levels. Although the Omicron variant is less virulent and causes fewer severe cases, it significantly harms the lungs of immunocompromised elderly individuals. This underscores the inadequacy of solely using a blood cytokine analysis for understanding the pathology of lung injury and systemic manifestations in COVID-19 patients. ELF analysis via MS provides crucial insights into lung-specific inflammation, aiding in comprehending the complex pathology of COVID-19, particularly in patients with AHRF and worsening health.

Abbreviations

Coronavirus disease 2019

Acute hypoxemic respiratory failure

Mechanical ventilation

• Epithelial lining fluid

Microsampling

Computed tomography

Pneumonia severity index

Charlson Comorbidity Index

Extracorporeal membrane oxygenation

Lower limit of quantification

Granulocyte colony-stimulating factor

Granulocyte–macrophage colony-stimulating factor

Interferon-γ

Interleukin

Monocyte chemotactic protein-1

Macrophage inflammatory protein-1β

Tumor necrosis factor-α

Soluble CD40 ligand

Interquartile range

Detection frequency

Bronchoalveolar lavage fluid

Moore JB, June CH. Cytokine release syndrome in severe COVID-19. Science. 2020;368:473–4. https://doi.org/10.1126/science.abb8925 .

Article   CAS   PubMed   Google Scholar  

Chen LYC, Quach TTT. COVID-19 cytokine storm syndrome: a threshold concept. Lancet Microbe. 2021;2:e49–50. https://doi.org/10.1016/S2666-5247(20)30223-8 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Silva MJA, Ribeiro LR, Gouveia MIM, Marcelino BDR, Santos CSD, Lima KVB, et al. Hyperinflammatory response in COVID-19: a systematic review. Viruses. 2023. https://doi.org/10.3390/v15020553 .

Article   PubMed   PubMed Central   Google Scholar  

Ragab D, Salah Eldin H, Taeimah M, Khattab R, Salem R. The COVID-19 cytokine storm; what we know so far. Front Immunol. 2020;11:1446. https://doi.org/10.3389/fimmu.2020.01446 .

Calabrese F, Lunardi F, Baldasso E, Pezzuto F, Kilitci A, Olteanu GE, et al. Comprehensive bronchoalveolar lavage characterization in COVID-19 associated acute respiratory distress syndrome patients: a prospective cohort study. Respir Res. 2023;24:152. https://doi.org/10.1186/s12931-023-02464-9 .

Zeng HL, Chen D, Yan J, Yang Q, Han QQ, Li SS, et al. Proteomic characteristics of bronchoalveolar lavage fluid in critical COVID-19 patients. FEBS J. 2021;288:5190–200. https://doi.org/10.1111/febs.15609 .

Liao L, Yang GH. Clinical significance of cellular immunity function and inflammatory factors assays in alveolar lavage fluid for severe COVID-19 pneumonia. J Med Virol. 2021;93:2979–87. https://doi.org/10.1002/jmv.26827 .

Saris A, Reijnders TDY, Nossent EJ, Schuurman AR, Verhoeff J, van Asten S, et al. Distinct cellular immune profiles in the airways and blood of critically ill patients with COVID-19. Thorax. 2021;76:1010–9. https://doi.org/10.1136/thoraxjnl-2020-216256 .

Article   PubMed   Google Scholar  

de Brabander J, Boers LS, Kullberg RFJ, Zhang S, Nossent EJ, Heunks LMA, et al. Persistent alveolar inflammatory response in critically ill patients with COVID-19 is associated with mortality. Thorax. 2023;78:912–21. https://doi.org/10.1136/thorax-2023-219989 .

Kox M, Waalders NJB, Kooistra EJ, Gerretsen J, Pickkers P. Cytokine levels in critically ill patients with COVID-19 and other conditions. JAMA. 2020;324:1565–7. https://doi.org/10.1001/jama.2020.17052 .

Bendib I, Beldi-Ferchiou A, Schlemmer F, Surenaud M, Maitre B, Plonquet A, et al. Alveolar compartmentalization of inflammatory and immune cell biomarkers in pneumonia-related ARDS. Crit Care. 2021;25:23. https://doi.org/10.1186/s13054-020-03427-y .

Jouan Y, Baranek T, Si-Tahar M, Paget C, Guillon A. Lung compartmentalization of inflammatory biomarkers in COVID-19-related ARDS. Crit Care. 2021;25:120. https://doi.org/10.1186/s13054-021-03513-9 .

Ishizaka A, Watanabe M, Yamashita T, Ogawa Y, Koh H, Hasegawa N, et al. New bronchoscopic microsample probe to measure the biochemical constituents in epithelial lining fluid of patients with acute respiratory distress syndrome. Crit Care Med. 2001;29:896–8. https://doi.org/10.1097/00003246-200104000-00043 .

Kipnis E, Hansen K, Sawa T, Moriyama K, Zurawel A, Ishizaka A, et al. Proteomic analysis of undiluted lung epithelial lining fluid. Chest. 2008;134:338–45. https://doi.org/10.1378/chest.07-1643 .

Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, Singer DE, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336:243–50. https://doi.org/10.1056/NEJM199701233360402 .

Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40:373–83. https://doi.org/10.1016/0021-9681(87)90171-8 .

Lanza E, Muglia R, Bolengo I, Santonocito OG, Lisi C, Angelotti G, et al. Quantitative chest CT analysis in COVID-19 to predict the need for oxygenation support and intubation. Eur Radiol. 2020;30:6770–8. https://doi.org/10.1007/s00330-020-07013-2 .

Sudo K, Sawa T, Kushimoto K, Yoshii R, Yuasa K, Inoue K, et al. Choice of respiratory therapy for COVID-19 patients with acute hypoxemic respiratory failure: a retrospective case series study. PeerJ. 2023;11: e15174. https://doi.org/10.7717/peerj.15174 .

Takashima Y, Inaba T, Matsuyama T, Yoshii K, Tanaka M, Matsumoto K, et al. Potential marker subset of blood-circulating cytokines on hematopoietic progenitor-to-Th1 pathway in COVID-19. Front Med. 2024;2024:11. https://doi.org/10.3389/fmed.2024.1319980 .

Article   Google Scholar  

Yamazaki K, Ogura S, Ishizaka A, Oh-hara T, Nishimura M. Bronchoscopic microsampling method for measuring drug concentration in epithelial lining fluid. Am J Respir Crit Care Med. 2003;168:1304–7. https://doi.org/10.1164/rccm.200301-111OC .

Download references

Acknowledgements

On the basis of this clinical study, we would like to express our thanks to the clinicians in the intensive care unit, Infectious Disease Department, and emergency department, as well as the ward nurses and laboratory technicians at KPUM Hospital for their dedicated support in managing COVID-19 patients. We would like to express our gratitude to Prof. Akitoshi Ishizaka, former professor of the Department of Respiratory Medicine at Keio University, who has passed away, and to Prof. Satoru Hashimoto, former manager of the Critical Care Division at KPUM Hospital, for their technical advice on the microsampling method. We also extend our thanks to Dr. J. Ludovic Croxford from Edanz ( https://jp.edanz.com/ac ), Dr. T. Fernanda, and the team from Springer Nature Author Services ( https://authorservices.springernature.com/ ) for editing a draft of this manuscript.

This study was supported in part by the Japan Agency for Medical Research and Development [AMED Grant Number JP20fk0108270].

Author information

Authors and affiliations.

Department of Anesthesiology, Graduate School of Medical Science, Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Kazuki Sudo, Mao Kinoshita, Ken Kawaguchi, Kohsuke Kushimoto & Teiji Sawa

Division of Intensive Care, Hospital of Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Kazuki Sudo, Ryogo Yoshii, Keita Inoue, Masaki Yamasaki & Kunihiko Kooguchi

Department of Anesthesia, Kyoto First Red-Cross Hospital, Honmachi 15-749, Higashiyama, Kyoto, 605-0981, Japan

Masaki Yamasaki

Department of Emergency Medicine, Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Tasuku Matsuyama & Bon Ohta

Department of Genomic Medical Sciences, Graduate School of Medical Science, Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Yasuo Takashima, Masami Tanaka & Kei Tashiro

Division of Clinical Laboratory, Hospital of Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Kazumichi Matsumoto & Tohru Inaba

Department of Infection Control and Laboratory Medicine, Graduate School of Medical Science, Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

Tohru Inaba

Hospital of Kyoto Prefectural University of Medicine, Kajiicho 465, Kawaramachi-Hirokoji Agaru, Kamigyo, Kyoto, 602-8566, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

KS and TS developed the initial idea for this study and were responsible for selecting the survey. KS extracted the data and conducted a comprehensive search of the databases. MK, KKa, KKu, RY, KI, MY, TM, and BO participated in patient management. YT, MT, KM, KT, and TI supported the use of multiplex bead-based assays and laboratory examinations. All the authors contributed to the research design, interpretation of the results, and conception of the writing of the article. K.S. and T.S. analyzed the data and drafted the article. KK, KT, BO, and TI reviewed the article and provided suggestions for improvement. All the authors have carefully examined the manuscript and agreed with the ideas presented. All the authors meet the ICMJE authorship criteria and have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Teiji Sawa .

Ethics declarations

Ethics approval and consent to participate.

The study was conducted according to the guidelines of the Declaration of Helsinki. This retrospective observational study was conducted in combination with the Kyoto Prefectural University of Medicine (KPUM) COVID-19 Registry Study (ERB-C-1810-3; approved by the Institutional Review Board of KPUM on 3 September 2020), and cytokine concentrations were analyzed in tracheal and bronchial secretions from healthy adults (ERB-C-2179; approved by the Institutional Review Board of KPUM on 1 December 2021). Informed consent was obtained from all participants and/or their legal guardian(s), and all methods were performed according to the relevant guidelines and regulations.

Availability of data and materials

The dataset (DataSet_SudoK_etal.xlsx) used and/or analyzed during this study is downloadable as Additional file 8 .

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Major characteristics of the patients and volunteers.

Additional file 2: Table S2.

Specific medications used in the COVID-19 patients with acute hypoxemic respiratory failure.

Additional file 3: Table S3.

Measurement range and detection sensitivity of cytokines according to examples from the Bio-Rad BioPlex Pro ® manual.

Additional file 4: Table S4.

Major characteristics of the three chronological groups.

Additional file 5: Fig. S1.

The overview of the individual patients in chronological order.

Additional file 6: Table S5.

Major characteristics of the three PSI groups.

Additional file 7: Table S6.

Major characteristics of the three LIV groups.

Additional file 8:

Dataset file.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sudo, K., Kinoshita, M., Kawaguchi, K. et al. Case study observational research: inflammatory cytokines in the bronchial epithelial lining fluid of COVID-19 patients with acute hypoxemic respiratory failure. Crit Care 28 , 134 (2024). https://doi.org/10.1186/s13054-024-04921-3

Download citation

Received : 14 March 2024

Accepted : 17 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s13054-024-04921-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Bronchial microsampling
• Multiplex bead-based assay

Critical Care

ISSN: 1364-8535

• Submission enquiries: [email protected]

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

What the data says about crime in the U.S.

A growing share of Americans say reducing crime should be a top priority for the president and Congress to address this year. Around six-in-ten U.S. adults (58%) hold that view today, up from 47% at the beginning of Joe Biden’s presidency in 2021.

We conducted this analysis to learn more about U.S. crime patterns and how those patterns have changed over time.

The analysis relies on statistics published by the FBI, which we accessed through the Crime Data Explorer , and the Bureau of Justice Statistics (BJS), which we accessed through the  National Crime Victimization Survey data analysis tool .

To measure public attitudes about crime in the U.S., we relied on survey data from Pew Research Center and Gallup.

Additional details about each data source, including survey methodologies, are available by following the links in the text of this analysis.

With the issue likely to come up in this year’s presidential election, here’s what we know about crime in the United States, based on the latest available data from the federal government and other sources.

How much crime is there in the U.S.?

It’s difficult to say for certain. The  two primary sources of government crime statistics  – the Federal Bureau of Investigation (FBI) and the Bureau of Justice Statistics (BJS) – paint an incomplete picture.

The FBI publishes  annual data  on crimes that have been reported to law enforcement, but not crimes that haven’t been reported. Historically, the FBI has also only published statistics about a handful of specific violent and property crimes, but not many other types of crime, such as drug crime. And while the FBI’s data is based on information from thousands of federal, state, county, city and other police departments, not all law enforcement agencies participate every year. In 2022, the most recent full year with available statistics, the FBI received data from 83% of participating agencies .

BJS, for its part, tracks crime by fielding a  large annual survey of Americans ages 12 and older and asking them whether they were the victim of certain types of crime in the past six months. One advantage of this approach is that it captures both reported and unreported crimes. But the BJS survey has limitations of its own. Like the FBI, it focuses mainly on a handful of violent and property crimes. And since the BJS data is based on after-the-fact interviews with crime victims, it cannot provide information about one especially high-profile type of offense: murder.

All those caveats aside, looking at the FBI and BJS statistics side-by-side  does  give researchers a good picture of U.S. violent and property crime rates and how they have changed over time. In addition, the FBI is transitioning to a new data collection system – known as the National Incident-Based Reporting System – that eventually will provide national information on a much larger set of crimes , as well as details such as the time and place they occur and the types of weapons involved, if applicable.

Which kinds of crime are most and least common?

Property crime in the U.S. is much more common than violent crime. In 2022, the FBI reported a total of 1,954.4 property crimes per 100,000 people, compared with 380.7 violent crimes per 100,000 people.  

By far the most common form of property crime in 2022 was larceny/theft, followed by motor vehicle theft and burglary. Among violent crimes, aggravated assault was the most common offense, followed by robbery, rape, and murder/nonnegligent manslaughter.

BJS tracks a slightly different set of offenses from the FBI, but it finds the same overall patterns, with theft the most common form of property crime in 2022 and assault the most common form of violent crime.

How have crime rates in the U.S. changed over time?

Both the FBI and BJS data show dramatic declines in U.S. violent and property crime rates since the early 1990s, when crime spiked across much of the nation.

Using the FBI data, the violent crime rate fell 49% between 1993 and 2022, with large decreases in the rates of robbery (-74%), aggravated assault (-39%) and murder/nonnegligent manslaughter (-34%). It’s not possible to calculate the change in the rape rate during this period because the FBI  revised its definition of the offense in 2013 .

The FBI data also shows a 59% reduction in the U.S. property crime rate between 1993 and 2022, with big declines in the rates of burglary (-75%), larceny/theft (-54%) and motor vehicle theft (-53%).

Using the BJS statistics, the declines in the violent and property crime rates are even steeper than those captured in the FBI data. Per BJS, the U.S. violent and property crime rates each fell 71% between 1993 and 2022.

While crime rates have fallen sharply over the long term, the decline hasn’t always been steady. There have been notable increases in certain kinds of crime in some years, including recently.

In 2020, for example, the U.S. murder rate saw its largest single-year increase on record – and by 2022, it remained considerably higher than before the coronavirus pandemic. Preliminary data for 2023, however, suggests that the murder rate fell substantially last year .

How do Americans perceive crime in their country?

Americans tend to believe crime is up, even when official data shows it is down.

In 23 of 27 Gallup surveys conducted since 1993 , at least 60% of U.S. adults have said there is more crime nationally than there was the year before, despite the downward trend in crime rates during most of that period.

While perceptions of rising crime at the national level are common, fewer Americans believe crime is up in their own communities. In every Gallup crime survey since the 1990s, Americans have been much less likely to say crime is up in their area than to say the same about crime nationally.

Public attitudes about crime differ widely by Americans’ party affiliation, race and ethnicity, and other factors . For example, Republicans and Republican-leaning independents are much more likely than Democrats and Democratic leaners to say reducing crime should be a top priority for the president and Congress this year (68% vs. 47%), according to a recent Pew Research Center survey.

How does crime in the U.S. differ by demographic characteristics?

Some groups of Americans are more likely than others to be victims of crime. In the  2022 BJS survey , for example, younger people and those with lower incomes were far more likely to report being the victim of a violent crime than older and higher-income people.

There were no major differences in violent crime victimization rates between male and female respondents or between those who identified as White, Black or Hispanic. But the victimization rate among Asian Americans (a category that includes Native Hawaiians and other Pacific Islanders) was substantially lower than among other racial and ethnic groups.

The same BJS survey asks victims about the demographic characteristics of the offenders in the incidents they experienced.

In 2022, those who are male, younger people and those who are Black accounted for considerably larger shares of perceived offenders in violent incidents than their respective shares of the U.S. population. Men, for instance, accounted for 79% of perceived offenders in violent incidents, compared with 49% of the nation’s 12-and-older population that year. Black Americans accounted for 25% of perceived offenders in violent incidents, about twice their share of the 12-and-older population (12%).

As with all surveys, however, there are several potential sources of error, including the possibility that crime victims’ perceptions about offenders are incorrect.

How does crime in the U.S. differ geographically?

There are big geographic differences in violent and property crime rates.

For example, in 2022, there were more than 700 violent crimes per 100,000 residents in New Mexico and Alaska. That compares with fewer than 200 per 100,000 people in Rhode Island, Connecticut, New Hampshire and Maine, according to the FBI.

The FBI notes that various factors might influence an area’s crime rate, including its population density and economic conditions.

What percentage of crimes are reported to police? What percentage are solved?

Most violent and property crimes in the U.S. are not reported to police, and most of the crimes that  are  reported are not solved.

In its annual survey, BJS asks crime victims whether they reported their crime to police. It found that in 2022, only 41.5% of violent crimes and 31.8% of household property crimes were reported to authorities. BJS notes that there are many reasons why crime might not be reported, including fear of reprisal or of “getting the offender in trouble,” a feeling that police “would not or could not do anything to help,” or a belief that the crime is “a personal issue or too trivial to report.”

Most of the crimes that are reported to police, meanwhile,  are not solved , at least based on an FBI measure known as the clearance rate . That’s the share of cases each year that are closed, or “cleared,” through the arrest, charging and referral of a suspect for prosecution, or due to “exceptional” circumstances such as the death of a suspect or a victim’s refusal to cooperate with a prosecution. In 2022, police nationwide cleared 36.7% of violent crimes that were reported to them and 12.1% of the property crimes that came to their attention.

Which crimes are most likely to be reported to police? Which are most likely to be solved?

Around eight-in-ten motor vehicle thefts (80.9%) were reported to police in 2022, making them by far the most commonly reported property crime tracked by BJS. Household burglaries and trespassing offenses were reported to police at much lower rates (44.9% and 41.2%, respectively), while personal theft/larceny and other types of theft were only reported around a quarter of the time.

Among violent crimes – excluding homicide, which BJS doesn’t track – robbery was the most likely to be reported to law enforcement in 2022 (64.0%). It was followed by aggravated assault (49.9%), simple assault (36.8%) and rape/sexual assault (21.4%).

The list of crimes  cleared  by police in 2022 looks different from the list of crimes reported. Law enforcement officers were generally much more likely to solve violent crimes than property crimes, according to the FBI.

The most frequently solved violent crime tends to be homicide. Police cleared around half of murders and nonnegligent manslaughters (52.3%) in 2022. The clearance rates were lower for aggravated assault (41.4%), rape (26.1%) and robbery (23.2%).

When it comes to property crime, law enforcement agencies cleared 13.0% of burglaries, 12.4% of larcenies/thefts and 9.3% of motor vehicle thefts in 2022.

Are police solving more or fewer crimes than they used to?

Nationwide clearance rates for both violent and property crime are at their lowest levels since at least 1993, the FBI data shows.

Police cleared a little over a third (36.7%) of the violent crimes that came to their attention in 2022, down from nearly half (48.1%) as recently as 2013. During the same period, there were decreases for each of the four types of violent crime the FBI tracks:

• Police cleared 52.3% of reported murders and nonnegligent homicides in 2022, down from 64.1% in 2013.
• They cleared 41.4% of aggravated assaults, down from 57.7%.
• They cleared 26.1% of rapes, down from 40.6%.
• They cleared 23.2% of robberies, down from 29.4%.

The pattern is less pronounced for property crime. Overall, law enforcement agencies cleared 12.1% of reported property crimes in 2022, down from 19.7% in 2013. The clearance rate for burglary didn’t change much, but it fell for larceny/theft (to 12.4% in 2022 from 22.4% in 2013) and motor vehicle theft (to 9.3% from 14.2%).

Note: This is an update of a post originally published on Nov. 20, 2020.

• Criminal Justice

John Gramlich is an associate director at Pew Research Center

8 facts about Black Lives Matter

#blacklivesmatter turns 10, support for the black lives matter movement has dropped considerably from its peak in 2020, fewer than 1% of federal criminal defendants were acquitted in 2022, before release of video showing tyre nichols’ beating, public views of police conduct had improved modestly, most popular.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy

Find Info For

• Current Students
• Prospective Students
• Research and Partnerships
• Entrepreneurship and Commercialization

Quick Links

• Health and Life Sciences
• Info Security and AI
• Transformative Education
• Purdue Today
• Purdue Global
• Purdue in the News

April 22, 2024

Genetically engineering a treatment for incurable brain tumors

Sandro Matosevic, associate professor in the Department of Industrial and Molecular Pharmaceutics in Purdue’s College of Pharmacy, leads a team of researchers that is developing a novel immunotherapy to be used against glioblastoma. (Purdue University photo/Shambhavi Borde)

Purdue researchers develop fully off-the-shelf, stem cell-derived, natural killer cells against glioblastoma

WEST LAFAYETTE, Ind. — Purdue University researchers are developing and validating a patent-pending treatment for incurable glioblastoma brain tumors. Glioblastomas are almost always lethal with a median survival time of 14 months. Traditional methods used against other cancers, like chemotherapy and immunotherapy, are often ineffective on glioblastoma. 

Sandro Matosevic , associate professor in the Department of Industrial and Molecular Pharmaceutics in Purdue’s College of Pharmacy , leads a team of researchers that is developing a novel immunotherapy to be used against glioblastoma. Matosevic is also on the faculty of the Purdue Institute for Cancer Research and the Purdue Institute for Drug Discovery .

The Matosevic-led research has been published in the peer-reviewed journal Nature Communications .

The Purdue glioblastoma treatment

Matosevic said traditional cell therapies have almost exclusively been autologous, meaning taken from and returned to the same patient. Blood cells from a patient are engineered to better recognize and bind to proteins on cancer cells, then given back to the same patient to bind to and attack cancer cells. Unfortunately, these therapies have limited to no effect on glioblastoma. 

“By contrast, we are developing immunotherapy based on novel, genetically engineered, fully off-the-shelf or allogeneic immune cells. Allogeneic cells are not sourced from the same patient, but rather another source,” Matosevic said. “In our study, we sourced — or rather engineered — cells from induced pluripotent stem cells. So we eliminated the need for blood and instead differentiated stem cells into immune cells, or natural killer cells, and then genetically engineered those.”

Matosevic said novel Purdue immunotherapy can be considered to have a true off-the-shelf source.

“We can envision having unlimited supplies of these stem cells ready to be engineered,” Matosevic said. “This does not require blood to be sourced. And because these are human cells, they are directly usable in human patients.” 

Validation and next development steps

The research team tested its treatment by conducting animal studies with mice bearing human brain tumors, which were treated by direct injection of the newly engineered immune cells.

“Our preclinical studies showed these immune cells to be particularly remarkable in targeting and completely eliminating the growth of the tumors,” Matosevic said. “We found that we can engineer these cells at doses suitable for clinical use in humans. This is significant because one of the major hurdles to clinical translation of cell-based therapies to humans has been the poor expansion and lack of potency of cells that were sourced directly from patients. Using an off-the-shelf, fully synthetic approach breaks down significant barriers to the manufacturing of these cells.”

Matosevic said the next step to develop the glioblastoma treatment is to conduct clinical trials to treat patients with brain tumors, including those that were not successfully eliminated by surgery.

“Our ultimate goal is to bring this therapy to patients with brain tumors,” Matosevic said. “These patients urgently deserve better, and more effective, treatment options. We believe there is true potential for this therapy, and we have the motivation and capacity to bring it to the clinic.

“We are working with neurosurgical clinician collaborators to not only obtain funding, but also initiate clinical protocols,” he added. “We are also open to and always seeking new collaborations and partnerships with those who have interest in supporting our mission to translate this therapy to the clinic, where it is needed the most.”

Matosevic disclosed the innovative glioblastoma treatment to the Purdue Innovates Office of Technology Commercialization , which has applied for a patent from the U.S. Patent and Trademark Office to protect the intellectual property. Inquiries about the status of the intellectual property may be directed to Joe Kasper, assistant director of business development and licensing — life sciences, at [email protected] .

Matosevic and the research team received funding from the National Institutes of Health, the V Foundation for Cancer Research, the Purdue Institute for Cancer Research and industry partners.

About Purdue Innovates Office of Technology Commercialization  

The Purdue Innovates Office of Technology Commercialization operates one of the most comprehensive technology transfer programs among leading research universities in the U.S. Services provided by this office support the economic development initiatives of Purdue University and benefit the university’s academic activities through commercializing, licensing and protecting Purdue intellectual property. In fiscal year 2023, the office reported 150 deals finalized with 203 technologies signed, 400 disclosures received and 218 issued U.S. patents. The office is managed by the Purdue Research Foundation, which received the 2019 Innovation & Economic Prosperity Universities Award for Place from the Association of Public and Land-grant Universities. In 2020, IPWatchdog Institute ranked Purdue third nationally in startup creation and in the top 20 for patents. The Purdue Research Foundation is a private, nonprofit foundation created to advance the mission of Purdue University. Contact [email protected] for more information. 

About Purdue University

Purdue University is a public research institution demonstrating excellence at scale. Ranked among top 10 public universities and with two colleges in the top four in the United States, Purdue discovers and disseminates knowledge with a quality and at a scale second to none. More than 105,000 students study at Purdue across modalities and locations, including nearly 50,000 in person on the West Lafayette campus. Committed to affordability and accessibility, Purdue’s main campus has frozen tuition 13 years in a row. See how Purdue never stops in the persistent pursuit of the next giant leap — including its first comprehensive urban campus in Indianapolis, the new Mitchell E. Daniels, Jr. School of Business, and Purdue Computes — at https://www.purdue.edu/president/strategic-initiatives .

Writer/Media contact: Steve Martin, [email protected]

Source: Sandro Matosevic, [email protected]

Research Foundation News

Communication.

• OneCampus Portal
• Brightspace
• BoilerConnect
• Faculty and Staff
• Human Resources
• Colleges and Schools

Info for Staff

• Purdue Moves
• Board of Trustees
• University Senate
• Center for Healthy Living
• Information Technology
• Ethics & Compliance
• Campus Disruptions

Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, (765) 494-4600

© 2015-24 Purdue University | An equal access/equal opportunity university | Copyright Complaints | Maintained by Office of Strategic Communications

Trouble with this page? Disability-related accessibility issue? Please contact News Service at [email protected] .

• Rights Lab Home
• Research Projects
• News and Events
• Opportunities

Grant Award on Earth Observation for Social Sustainability

Professor Doreen Boyd (Geography) has won a grant from the British Council through the International Science Partnerships Fund (ISPF). The award is the Early Career Fellowships in Research and Innovation 2023/24 and will fund three year-long Early Career Fellowships. The scheme will enable early-career researchers (ECRs) to collaborate internationally and gain access to new research environments, facilities, knowledge and expertise, creating lasting benefits for the ECRs as well as the UK and international country/territory research communities through sustainable collaboration.

Professor Boyd will select three fellows from Brazil to work with her as a team focused on Earth Observation (EO) of potential forced labour sites. This work will likely focus on timber from the Brazilian Amazon: a known high-risk and priority supply chain to the UK, an industry that can be analysed from space, and a forced labour issue that exacerbates climate change and the displacement of traditional communities. The fellows will pioneer techniques for mapping high-risk sites and build on their country networks and knowledge as they develop and deploy EO techniques.

The fellows will be part of the Rights Lab's world-leading programme that is building on UK strengths in “space” to realise the power of EO research for human rights, sustainable supply chains and international development.

Following its work to estimate the prevalence of online child sexual exploitation in the Philippines in partnership with the International Justice Mission (IJM), the Rights Lab is now beginning two new prevalence estimation projects funded by IJM.

One, led by Professor Doreen Boyd (Geography) and with team members from Politics (Todd Landman, Kevin Fahey, Zoe Trodd) and Geography (Giles Foody), will estimate the prevalence of bonded labour in India. The team will establish prevalence, identify key industries with a high prevalence of bonded labour, and understand the factors leading to high prevalence. Delivered in partnership with IJM's India team, it will be the first geographically extensive prevalence estimation for bonded labour in India since the 1970s, and the largest-scale bonded labour estimate ever completed.

A second, led by Professor Todd Landman (Politics) and with team members from Politics (Kevin Fahey) and Geography (Doreen Boyd), will estimate the prevalence of cross-border forced labour and labour trafficking in Malaysia. The team will establish the scale of trafficking for cross-border labour in Malaysia, understand unique destination-side vulnerabilities, and identify key hubs and channels for trafficking. 

Delivered in partnership with IJM's Malaysia team, it will be the largest prevalence estimation study on forced labour among cross-border migrants in Malaysia conducted to date.

Both studies will inform new programmes and strategies for IJM, in India and Malaysia. The world's largest anti-slavery/anti-trafficking NGO, IJM works in 16 countries around the world to combat trafficking and slavery.

World-class research at the University of Nottingham

Legal information

• Terms and conditions
• Posting rules
• Accessibility
• Freedom of information
• Charity gateway
• Cookie policy

Connect with the University of Nottingham through social media and our blogs .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Observational Research Opportunities and Limitations

Edward j. boyko.

Epidemiologic Research and Information Center, VA Puget Sound Health Care System, Seattle, WA USA. University of Washington School of Medicine, Seattle, WA

Medical research continues to progress in its ability to identify treatments and characteristics associated with benefits and adverse outcomes. The principle engine for the evaluation of treatment efficacy is the randomized controlled trial (RCT). Due to the cost and other considerations, RCTs cannot address all clinically important decisions. Observational research often is used to address issues not addressed or not addressable by RCTs. This article provides an overview of the benefits and limitations of observational research to serve as a guide to the interpretation of this category of research designs in diabetes investigations. The potential for bias is higher in observational research but there are design and analysis features that can address these concerns although not completely eliminate them. Pharmacoepidemiologic research may provide important information regarding relative safety and effectiveness of diabetes pharmaceuticals. Such research must effectively address the important issue of confounding by indication in order to produce clinically meaningful results. Other methods such as instrumental variable analysis are being employed to enable stronger causal inference but these methods also require fulfillment of several key assumptions that may or may not be realistic. Nearly all clinical decisions involve probabilistic reasoning and confronting uncertainly, so a realistic goal for observational research may not be the high standard set by RCTs but instead the level of certainty needed to influence a diagnostic or treatment decision.

A major focus of medical research is the identification of causes of health outcomes, good and bad. The current gold standard method to accomplish this aim is the randomized controlled trial (RCT) ( Meldrum, 2000 ). The performance of a RCT requires strict specification of study conditions related to all aspects of its conduct, such as participant selection, treatment and control assignment arms, inclusion/exclusion criteria, randomization method, outcome measurement, and many other considerations. Such trials are difficult to mount due to the expense in terms of both time and money, and often lead to results that may be difficult to apply to a real-world setting due to either the rigor or complexity of the intervention or the selection process for participants that yields a population dissimilar from that seen in general clinical practice. A randomized controlled trial focuses on an assessment of the validity of its results at the expense of generalizability. For example, the Diabetes Prevention Program screened 158,177 subjects to yield 3,819 subjects who were eventually randomized to one of the four original arms ( Rubin et al., 2002 ). Other limitations of RCTs include a focus on treatment effects and not the ability to detect rarer adverse reactions; restrictions on diabetes duration at the time of trial entry, thereby yielding results that may not apply to persons with a different diabetes duration at the initiation of treatment; and high costs that limits the number of therapeutic comparisons. Regarding this last point, assessment of a new treatment for hyperglycemia requires comparison to existing accepted treatments, but the control population usually is restricted to fewer treatments than in current use, thereby limiting the ability to compare the new treatment to all existing treatments.

Given these considerations, observational research is often used to address important clinical questions in the absence of randomized clinical trial data, but may also make important potential contributions even when RCTs have been conducted. Examples include monitoring for long-term adverse events that did not appear during the time interval over which the RCT was conducted, or to assess whether the trial findings apply to a different population excluded from the trial due to younger or older age, gender, presence of comorbid conditions, or other factors. Observational research often also addresses other questions not suitable for randomized clinical trials, such as an exposure known to be harmful or in other ways unacceptable to participants or whose administration is inconsistent with ethical principles. Also, observational research can address other exposures that are not potentially under the control of the investigator, such as, for example, eye color, blood type, presence of a specific genetic marker, or elevations of blood pressure or plasma glucose concentration. Observational research may also provide preliminary data to justify the performance of a clinical trial, which might not have received sufficient funding support without the existence of such results.

This paper will review observational research methods applied to addressing questions of causation in diabetes research, with a particular focus on pharmacoepidemiology as an area of research where many important questions may be addressed regarding the relative merits of multiple pharmaceuticals for a given condition. There have been an increasing number of observational studies of the association between diabetes treatments and hard outcomes, such as death or CVD events. The increase in such studies likely has been facilitated by the availability of big data in general and specifically large pharmaceutical databases created by national health plans, large health care systems, or mail-order pharmacy providers ( Sobek et al., 2011 ). In addition, the ongoing development of diabetes pharmacotherapies approved based on ability to achieve an improvement in glycemic control but without data on hard outcomes may also provide the impetus to use such large databases for research on comparative safety and efficacy.

Observational Research Study Designs

Cohort and case-control studies.

The two most popular designs for investigating causal hypotheses are the cohort and case-control studies. Features are shown in Table 1 . The major difference between the two is that the cohort study begins with identification of the exposure status, whereas the case-control study begins with the identification of the outcome. A cohort study can be prospective, where exposed and non-exposed subjects are followed for the development of the outcome, or retrospective, where collected data can be used to identify both the exposure status at some past time point and the subsequent development of the outcome. A case-control study, on the other hand, can only look back in time for occurrence of the exposure. There are of course exceptions to these general statements. It is possible in some case-control studies to measure the exposure after the outcome in time if the exposure is invariant and if it is not related to a greater loss to follow-up among persons with the outcome due to mortality or other reasons. Examples of such exposures include genetic markers or an unchanging characteristic of adults such as femur length, eye color, or red blood cell type. Variations in these study designs include the case-cohort and case-only studies, which are described in detail elsewhere, and which a description of which will not be provided here ( DiPietro, 2010 ). Also, the relative merits of these study designs will not be discussed here but are covered in standard epidemiology texts.

Observational Study Designs for Assessment of Causal Relationships

Weaker Observational Research Designs

Other research designs are often used in studies reported in the medical literature. These include cross-sectional, case-series, and case-reports. The cross-sectional study has limited value in assessing a potential causal relationship since it may not be possible to determine whether the potential exposure preceded the outcome, except when the exposure does not vary over one’s life history, such as in the case of a genotype, ABO blood group, or eye color. Case-series and case reports are even more limited since it is not possible to assess if the outcome occurred more frequently among the persons included compared to a control population. Case reports do though have potential value in pharmaceutical safety research by generating potential signals that signify unexpected adverse events. Such monitoring is employed in the Food and Drug Administration’s Adverse Event Reporting System, and has led to changes in product labeling as well as restriction or outright removal of pharmaceuticals from the market due to safety concerns ( Wysowski & Swartz, 2005 ). Over 2 million case reports of adverse reactions were submitted between 1969–2002, resulting in only about 1% of marketed drugs being withdrawn or restricted. Therefore the noise-to-signal ratio for this method of surveillance is exceedingly high and presents an opportunity for other observational methods to better address this issue.

Observation Research for Causal Inference

Causal associations will always involve correlation, but the presence of a correlation does not imply causation. The challenge of observational research is to assess whether a correlation is present and then determine whether it may be due to a causal association. A list of criteria was developed by Dr. Austin Bradford Hill decades ago that is still referred to frequently today ( Hill, 1965 ), although reexamination of these criteria more recently has led to the conclusion that only one of the nine original features is really necessary for a causal relationship in a observational study ( Phillips & Goodman, 2004 ; Rothman & Greenland, 2005 ). The magnitude of the observed association, another Hill criterion, often figures into determinations about the presence of bias, with those of greater magnitude considered less likely to be due to bias and more likely due to a causal process ( Grimes & Schulz, 2002 ).

Examination of the features of an RCT provide some insight into the limitations of observational research in assessing causal associations. The randomization process provides the opportunity for equal distribution of risk factors for the outcome among persons assigned to the treatment and control. Thus any difference in the outcome between these two groups will not likely be due to unequal distribution of risk factors by treatment assignment. The use of randomization provides a way to approach the problem of not having complete knowledge about predictors of all clinically important outcomes. If we did have such knowledge then groups with exactly equal risks of the outcome could be assembled by the investigator. As we do not have such knowledge, the process of randomization utilizes chance to distribute both known and more importantly unknown risk factors for the outcome, and is most likely to achieve this aim with larger sample size ( Efird, 2011 ). Randomization, though, does not guarantee that the treatment and control group will have the same risk of the outcome. Accidents of randomization have occurred for known risk factors for outcomes as in the UGDP, where older subjects with a higher prevalence of cardiovascular disease risk factors were disproportionately assigned to the tolbutamide treatment arm ( Leibel, 1971 ). Such accidents also must occur for the unknown risk factors, although these would not be apparent to the investigator.

Bias in Observational Research

Confounding bias.

Observational research does not have the benefit of randomization to allocate by chance risk factors for an outcome of interest. Exposures to risk factors occur due to self-selection, medical provider prescription, in association with occupation, and for other reasons. When an exposure of interest is strongly associated with another exposure that is also related to the outcome, confounding bias is present, but methods exist to obtain an unbiased estimated of the exposure-disease association as long as the confounding factor is identified and measured accurately.

A cross-sectional study of a genetic marker (Gm haplotype Gm 3;5,13,14 ) and diabetes prevalence provides an example of confounding bias. Subjects included members of the Pima and Papago tribes of the Gila River Indian Community in Southern Arizona who underwent a medical history and examination every two years including assessment of diabetes status through oral glucose tolerance testing ( Knowler, Williams, Pettitt & Steinberg, 1988 ). Subjects were further characterized by degree of Indian heritage measured in eighths and referred to as “quantum.” A total of 4,640 subjects of either 0/8, 4/8, and 8/8 quantum were included in this analysis. There were 1,336 persons with and 3,304 persons without diabetes available for analysis, yielding a crude (unadjusted) overall odds ratio of 0.24 for the association between haplotype Gm 3;5,13,14 and diabetes prevalence ( Figure 1 , Panel A). This result supports a lower prevalence of diabetes in association with haplotype Gm 3;5,13,14 , but the unadjusted result represents a substantial overestimate due to confounding by Quantum. In Figure 1 panel B, subjects were divided by the three Quantum categories found in the sample, and within each of these the odds ratio is closer to 1.0 and therefore of smaller magnitude than the crude result. Note that collapsing the three tables in Panel B by summing the cells yields the single overall table shown in Panel A. Adjustment for these Quantum categories yields an odds ratio of 0.59, which is of smaller magnitude than the result seen in the unadjusted analysis ( Figure 1 , Panel B). Although the odds ratios vary across Quantum categories, a test for heterogeneity across these strata was non-significant (p=0.295). Therefore the null hypothesis that the odds ratios differed across Quantum strata could not be rejected.

Cross-sectional study of Native Americans of the Pima and Papago Indian tribes in Southern Arizona on the associations between the GM haplotype Gm 3;5,13,14 , native quantum, and diabetes mellitus prevalence. Panel A displays all participants combined with Native quantum of either 0/8, 4/8 or 8/8 by presence of diabetes mellitus in relation to Gm 3;5,13,14 presence or absence. The overall (crude) odds ratio for the association is shown. Panel B displays all participants from Panel A stratified by Native quantum, demonstrating confounding by Native quantum as judged by the discordance between the crude and stratified or Quantum-adjusted results. Panel C demonstrates that Quantum meets the criterion as a confounding variable due to its negative association with Gm 3;5,13,14 and positive association with diabetes prevalence.

Examination of the frequency of haplotype Gm 3;5,13,14 and diabetes prevalence across Indian heritage Quantum reveals the reason for the overestimation of the association in the unadjusted analysis. Diabetes occurred more frequently while the haplotype Gm 3;5,13,14 occurred less frequently among subjects with greater Indian heritage ( Figure 1 , Panel C). Adjustment for the imbalance in Quantum by haplotype Gm 3;5,13,14 in this specific example and in general any accurately measured confounding factor yields a less biased odds ratio that is closer to the true magnitude of the association between this haplotype and diabetes prevalence.

Another more recent example of confounding can be seen in a case-cohort European study of the association between artificially sweetened soft drinks and the risk of developing type 2 diabetes ( 2013 ). The unadjusted hazard ratio for the daily consumption of ≥ 250 g of this beverage type was 1.84 (95% CI 1.52 to 2.23) representing a statistically significant elevation in risk. After adjustment for daily energy intake and BMI, the hazard ratio diminished to 1.13 (95% CI 0.85 to 1.52) and was no longer statistically significant (p=0.24). The investigators concluded that consumption of artificially sweetened soft drinks was not associated with type 2 diabetes risk in their population.

Multiple methods exist to remove the bias from recognized, accurately measured confounding factors, but unfortunately there is no widely accepted option for handling unmeasured confounding factors and adjusting for this bias. In this regard observational research is unable to match the ability of a RCT to account for this potential bias. Methods have been developed to better assess whether associations represent causal pathways that will be described later in this paper.

Information Bias

Observational research can be susceptible to other types of bias. Information bias refers to inaccurate assessment of the outcome, the exposure, or potential confounding variables. An example includes measurement of nutritional intake, which is often assessed by research subjects completing a food frequency survey or 24-hour dietary recall. Even if subjects report these intakes correctly, the likelihood is low that this will reflect long-term dietary intake exactly. Attempts have been made to reduce the error of these measurements through biomarker calibration that in one study was based on a urinary nitrogen protocol to estimate daily protein consumption over a 24-hour period ( Tinker et al., 2011 ). This analysis revealed a slight increase in risk of incident diabetes in association with a 20% higher protein intake in grams (Hazard Ratio 1.05, 95% CI 1.03–1.07). Recalibrated results based on the results of the urinary nitrogen protocol yielded a substantially higher diabetes hazard ratio of 1.82 (95% CI 1.56–2.12) that after adjustment for BMI was reduced to 1.16 (95% CI 1.05–2.28). In this example, reduction of measurement error yielded a difference of greater magnitude than see in the analysis based on dietary self-reports only without objective validation, although theoretically more accurate measurements may yield smaller differences, depending on the type and magnitude of measurement error.

Selection Bias

Selection bias may produce factitious exposure-disease associations if the study population fails to mirror the target population of interest. For example, selection of control subjects from among hospitalized patients as might be the case in a study based on administrative data may not accurately depict smoking prevalence among controls, given that smoking is related to multiple diseases that would increase the risk for hospitalization. Effective observational research must recognize the potential for bias and attempt to minimize it both in the design and analysis, as well as accurately describing limitations of these data and the implications for study validity in reports of results.

Agreement and Discrepancies between Observational and Clinical Trial Research

One way to assess whether the potential biases of observational studies result in failure to detect true associations is by comparison of observational versus RCT results on the same questions. Since observational studies of treatments often precede definitive clinical trials, several authors have assessed agreement between similar hypotheses tested using the gold standard compared to observational designs, concluding that agreement between the two is high. A comparison of 136 reports published between 1985 to 1998 on 19 different treatments found excellent agreement, with the combined magnitude of the effect in observational studies lying within the 95% confidence intervals of the combined magnitude of the effect in RCTs for 17 of the 19 hypotheses tested ( Benson & Hartz, 2000 ). Another comparison focused on comparing the results of meta-analyses of observation and clinical trial research on five clinical questions that were identified through a search of five major medical journals from 1991 to 1995 ( Concato, Shah & Horwitz, 2000 ). These investigators concluded that average results of these studies were “remarkably similar.”

In contrast, other research has demonstrated discrepancies between RCT and observational designs. The Women’s Health Initiative (WHI) was a RCT of dietary and menopausal hormone interventions to assess these effects on mortality, cardiovascular disease, and cancer risk ( Prentice et al., 2005 ). Perhaps unique to this study was the establishment of a concurrent observational study accompanying the randomized clinical trial, thereby permitting direct comparison of reported associations by type of research design within the same study framework. In the trial/observational study of estrogen plus progestin for menopausal hormone replacement, marked differences were seen between the treatment and control groups by participation in the RCT or observational study ( Table 2 ). In the RCT, no important differences were seen by treatment assignment for race, educational level, BMI, or current smoking status. This was not true by estrogen-progestin exposure in the observational study, where exposed women were more likely to be White, having completed a college degree or higher, and less likely to be current smokers or obese. Outcomes occurred more frequently in the estrogen-progestin arm of the RCT, but less frequently in the corresponding arm of the observational study, except for venous thromboembolism ( Table 2 ). Hazard ratios for these comparisons adjusted for imbalances in baseline potential confounding factors show a harmful effect of estrogen-progestin use that is statistically significantly elevated in 2 of 3 outcomes and a discordance with the observational results due to null, somewhat protective hazard ratios or in the case of venous thromboembolism, an elevated hazard ratio of considerably smaller magnitude than in the clinical trial. Although good agreement between clinical trials and observational research occurs often, the example of the WHI prevents having complete confidence in the results of observational studies.

Comparison of baseline characteristics and outcomes in the randomized controlled trial and observational study of estrogen-progestin treatment in the Women’s Health Initiative (1994–2002).

Achievements of Observational Research

Despite the limitations of observational research design, many well-accepted causal associations in medicine are supported entirely or in part due to this type of investigation. Several examples include the association between hyperglycemia and diabetes complications including retinopathy, nephropathy, peripheral neuropathy, and ischemic heart disease ( 2013 ). Other well known examples include hypertension and stroke, smoking and lung cancer, asbestosis and mesothelioma, and LDL and HDL cholesterol concentrations and risk of ischemic heart disease ( Churg, 1988 ; Gordon, Kannel, Castelli & Dawber, 1981 ; Kannel, Wolf, Verter & McNamara, 1970 ; Pirie, Peto, Reeves, Green & Beral, 2013 ). In the case of complications due to hyperglycemia, high LDL-cholesterol concentration, and hypertension, clinical trials to reduce these levels have resulted in reductions in the rate of these outcomes, further supporting a causal association ( 1991 ; 1994 ; 1998 ; 1998 ). For many associations that involve an exposure that cannot be controlled by the investigator or should not be modified for ethical reasons, observational research may be the only avenue for direct testing of these associations in humans.

Causal Inference from Observational Research

The results of an observational research study are never interpreted in an information vacuum. Given the potential for bias with this study design, a number of other factors should be considered when weighing the strength of this evidence. First and foremost would be the replication of the finding in other observational research studies. Additional evidence to bolster the potential causal association would be support from the biological understanding of underlying mechanisms, animal experiments confirming that the exposure results in a similar outcome, and trend data in disease incidence following changes in exposure prevalence. For example, in the UK Million Women Study where median age was reported at 55 years, women who quit smoking completely at ages 25–34 or 35–44 years had only 3% and 10% of the excess mortality, respectively, seen among women who were continuing smokers ( Pirie, Peto, Reeves, Green & Beral, 2013 ). Coronary heart disease deaths in the U.S. declined by approximately 50% between 1980 to 2000. One analysis that addressed the reasons for this decline concluded that change in risk factors (reductions in total cholesterol concentration, systolic blood pressure, smoking, and physical inactivity) accounted for approximately 47% of this decrease ( Ford et al., 2007 ). These trends provide support for a causal association between smoking and lung cancer, and multiple cardiovascular disease risk factors and coronary death risk.

Pharmacoepidemiology

Many questions regarding the use of pharmaceuticals may never be answered through use of RCTs, thereby creating a need to address knowledge gaps using observational research. The specialized field of pharmacoepidemiology directly addresses these needs. The earliest appearance of the term “pharmacoepidemiology” on PubMed.com is in an article written in 1984 ( Lawson, 1984 ). The field of pharmacoepidemiology encompasses the use of observational research to assess pharmaceutical safety and effectiveness. For example, diabetes pharmaceuticals have received FDA approval based on efficacy at lowering glucose and safety, without the need to prove efficacy at preventing long-term complications. The sulfonylurea hypoglycemic agents glyburide and glipizide are in widespread use to manage the hyperglycemia of diabetes, but it is not clear whether one is associated with a greater reduction in hard outcomes such as mortality or diabetes complications, as this has not been tested in a clinical trial. Use of such surrogate endpoints as opposed to the hard outcomes one wishes to prevent has been criticized as an ineffective and potentially harmful approach to medication approval ( Fleming & DeMets, 1996 ; Psaty et al., 1999 ). Design of clinical trials to address hard as opposed to surrogate endpoints typically requires larger sample size, longer follow-up, and greater costs.

Observational research may also identify adverse effects associated with the use of pharmaceuticals that were not anticipated based on research conducted in support of the drug approval process. The withdrawal of the thiazolidinedione agent troglitazone from the U.S. market in 2000 followed reports on cases of severe liver toxicity during post-marketing surveillance. Similar data on a high number of reported cases of severe myopathies in cerivastatin users led to its withdrawal from the worldwide market in 2001 ( Furberg & Pitt, 2001 ). An observational study using administrative claims databases to assess the relative safety of lipid lowering medications in the U.S. between 2000–2004 reported a much higher risk for hospitalization for treatment of myopathy among cerivastatin users compared to users of other statin and non-stain lipid lowering agents ( Cziraky et al., 2006 ).

Confounding by Indication

As with other observational research designs, there are limitations to pharmacoepidemiology due to biases previous described, but in addition to these is the vexing phenomenon of confounding by indication, also referred to as channeling bias ( McMahon & MacDonald, 2000 ; Petri & Urquhart, 1991 ). This refers to an observed benefit (or harm) associated with a pharmaceutical due to the indications for treatment with it and not a medication effect. A hypothetical example of how confounding by indication results in outcome differences not due to medication effect is shown in Figure 2 , which provides an example of how the choice of a diabetes pharmaceutical may depend on the existence of a condition (higher serum creatinine reflecting lower GFR) associated with higher mortality risk ( Fox et al., 2012 ).

A hypothetical population of 2000 identical persons with type 2 diabetes differing only by renal function as measured by serum creatinine and assigned to either metformin or glipizide based on the serum creatinine level. The active treatment, though, is never dispensed, and instead substituted with a identical placebo. An expected difference in mortality is seen between the two groups given the association between poorer renal function and mortality in the glipizide group. This difference cannot be explained by the effect of the active pharmaceutical (since there was none) and therefore represents an example of confounding by indication.

Several approaches exist to the problem of confounding by indication. If there is no association between the indication for the pharmaceutical and the outcome of interest, then no bias will occur, since an association must also be present between both the indication and the outcome to yield a biased result. This same principle applies to all confounding factors ( van Stralen, Dekker, Zoccali & Jager, 2010 ). If the conditions for confounding are fulfilled, then statistical adjustment techniques are available to produce unbiased estimates of effect. Commonly used methods in biomedical research include linear regression analysis for continuous outcomes, logistic regression for categorical outcomes, and the Cox proportional hazards model for time-to-event outcomes. In addition, propensity scores have risen in popularity over the past decade. An “all fields” search of Pubmed conducted January 15, 2012 using the search term “propensity score” yielded 2,895 hits for the immediate past 5 years, and only 715 hits for the previous 5 years. The propensity score method models the probability of exposure in relation to predictor variables, and therefore estimates the likelihood, in the case of a pharmacoepidemiology study, of a subject receiving a particular pharmaceutical based on his or her characteristics ( Rubin, 2010 ). An additional step is required which uses standard previously mentioned adjustment methods to remove the bias associated with varying likelihood of receiving the pharmaceutical. Despite the rising popularity of this method, it has been demonstrated to be merely equivalent and sometimes inferior to standard multivariate adjustment methods ( Shah, Laupacis, Hux & Austin, 2005 ; Sturmer et al., 2006 ). Furthermore, propensity scores cannot address the issue of unmeasured confounding ( Cummings, 2008 ). So if the indications for the pharmaceutical cannot be determined from the other measured factors, neither multivariate adjustment or propensity scores will allow for adjustment and removal of bias.

Several design features of observational studies may increase the likelihood of confounding by indication but if recognized may be amenable to correction in the design or analysis phases of a study. Assessing outcomes for pharmaceuticals prescribed for different indications or by a comparison of populations who differ with regard to the presence of medication contraindications may introduce bias into comparisons. An assessment of the mortality risk associated with beta-blocker use compared to other antihypertensive medications should exclude participants in whom beta blockers but not other antihypertensive medications are prescribed for other indications, such as migraine headache or stage fright prophylaxis, as these conditions may be associated with better outcomes and lead to over-optimistic survival benefit. Also, failure to consider medication contraindications may lead to risk of the outcome differing by medication used, as seen in the example in Figure 2 which would lead to a higher frequency of subjects with renal insufficiency in the glipizide treatment group for hyperglycemia. To account for this potential bias, subjects with contraindications for use of any of the pharmaceuticals of interest in the comparison should be eliminated from the study. For example, recent studies of mortality and cardiovascular events among users of sulfonylurea or metformin monotherapy for treatment of diabetes in the Veterans Health Administration system excluded patients with serious medical conditions at baseline that might influence the prescription of diabetes medication ( Roumie et al., 2012 ; Wheeler et al., 2013 ). For example, some items on the list of exclusions were congestive heart failure, serum creatinine concentration of 1.5 mg/dl or greater, HIV, and other conditions described in this publication. Despite these design features and adjustment methods to correct for factors associated with a particular prescription that may also be associated with a different outcome risk, there will always be some uncertainty about the presence of bias due to residual confounding by indication.

Methods to Improve Causal Inference from Observational Research

Instrumental variables analysis has been promoted as a method to overcome the inability to exclude undetected confounding in observational research. This method involves identification of a factor that strongly predicts treatment (or exposure in an epidemiologic study not involving a pharmaceutical). This factor is referred to as an “instrument,” and it is used in a manner analogous to the intention to treat analysis employed in RCTs ( Thomas & Conti, 2004 ). A Mendelian Randomization study is a type of instrumental variable analysis that uses a genetic marker as the instrument ( Thomas & Conti, 2004 ). Although intriguing in concept, the difficulty is in the application, as this relies on finding an “instrument” that is (1) causally related to treatment but not unobserved risk factors for the outcome, and (2) influences the outcome only through its effect on treatment ( Hernan & Robins, 2006 ). This method is being explored in pharmacoepidemiologic investigations, with one example being use of physician prescribing preference for types of NSAIDS in the evaluation of the gastrointestinal toxicity of COX-2 inhibitors versus non-COX-2 inhibitor NSAIDS ( Brookhart, Wang, Solomon & Schneeweiss, 2006 ). This analysis reported a protective association with COX-2 inhibitors only in the instrumental variable analysis, leading the authors to conclude that this analysis resulted in a reduction in unmeasured confounding. Examples can also be found in the diabetes epidemiology literature, such as the lack of association between serum uric acid level and type 2 diabetes risk ( Pfister et al., 2011 ), and higher risk associated with lower sex hormone-binding globulin concentration ( Ding et al., 2009 ).

Conclusions

As it will not be possible to assess efficacy of all possible treatment comparisons in all possible groups of interest, or identify adverse (or unexpected beneficial) outcomes requiring longer follow-up or greater sample size using RCTs, observational research stands prepared to step forward to address these knowledge gaps. Much medical knowledge and practice currently rests on a foundation of observational research. Perhaps this is not noticed due to the gloss and novelty of recently completed RCTs. Little research has been conducted comparing results from observational and clinical trial designs, but that which has been completed finds generally good agreement in these findings. With any observational research finding, though, comes less certainly due to the inability to completely exclude the possibility of residual confounding, or in the case of a pharmaceutical, confounding by indication. However, the expectation of absolute certainty is unrealistic and inconsistent with the current practice of medicine, where decisions are made probabilistically, with the threshold for actions such as further testing or treatment varying widely depending on the comparative costs and benefits of true and false positive and negative decisions ( Boland & Lehmann, 2010 ; Pauker & Kassirer, 1980 ; Plasencia, Alderman, Baron, Rolfs & Boyko, 1992 ). Observational research definitely has had and will continue to have an important role in providing the information needed to improve medical decision-making. There is always room for improvement and the hope that the future will bring better methods to further reduce the uncertainty surrounding the validity of its results.

Acknowledgments

Grant Support: VA Epidemiologic Research and Information Center; the Diabetes Research Center at the University of Washington (DK-017047)

Thanks for James S. Floyd MD for his careful review of this manuscript. The work was supported by the VA Epidemiologic Research and Information Center; the Diabetes Research Center at the University of Washington (DK-017047); and VA Puget Sound Health Care System.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

• Prevention of stroke by antihypertensive drug treatment in older persons with isolated systolic hypertension. Final results of the Systolic Hypertension in the Elderly Program (SHEP). SHEP Cooperative Research Group. JAMA. 1991; 265 :3255–3264. [ PubMed ] [ Google Scholar ]
• 1994 Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S) Lancet. 344 :1383–1389. [ PubMed ] [ Google Scholar ]
• Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34). UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998; 352 :854–865. [ PubMed ] [ Google Scholar ]
• Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998; 352 :837–853. [ PubMed ] [ Google Scholar ]
• Consumption of sweet beverages and type 2 diabetes incidence in European adults: results from EPIC-InterAct. Diabetologia. 2013; 56 :1520–1530. [ PubMed ] [ Google Scholar ]
• Diagnosis and classification of diabetes mellitus. Diabetes Care. 2013; 36 (Suppl 1):S67–74. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000; 342 :1878–1886. [ PubMed ] [ Google Scholar ]
• Boland MV, Lehmann HP. A new method for determining physician decision thresholds using empiric, uncertain recommendations. BMC Med Inform Decis Mak. 2010; 10 :20. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Brookhart MA, Wang PS, Solomon DH, Schneeweiss S. Evaluating short-term drug effects using a physician-specific prescribing preference as an instrumental variable. Epidemiology. 2006; 17 :268–275. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Churg A. Chrysotile, tremolite, and malignant mesothelioma in man. Chest. 1988; 93 :621–628. [ PubMed ] [ Google Scholar ]
• Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000; 342 :1887–1892. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cummings P. Propensity scores. Arch Pediatr Adolesc Med. 2008; 162 :734–737. [ PubMed ] [ Google Scholar ]
• Cziraky MJ, Willey VJ, McKenney JM, Kamat SA, Fisher MD, Guyton JR, et al. Statin safety: an assessment using an administrative claims database. Am J Cardiol. 2006; 97 :61C–68C. [ PubMed ] [ Google Scholar ]
• Ding EL, Song Y, Manson JE, Hunter DJ, Lee CC, Rifai N, et al. Sex hormone-binding globulin and risk of type 2 diabetes in women and men. N Engl J Med. 2009; 361 :1152–1163. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• DiPietro NA. Methods in epidemiology: observational study designs. Pharmacotherapy. 2010; 30 :973–984. [ PubMed ] [ Google Scholar ]
• Efird J. Blocked randomization with randomly selected block sizes. Int J Environ Res Public Health. 2011; 8 :15–20. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled? Ann Intern Med. 1996; 125 :605–613. [ PubMed ] [ Google Scholar ]
• Ford ES, Ajani UA, Croft JB, Critchley JA, Labarthe DR, Kottke TE, et al. Explaining the decrease in U.S. deaths from coronary disease, 1980–2000. N Engl J Med. 2007; 356 :2388–2398. [ PubMed ] [ Google Scholar ]
• Fox CS, Matsushita K, Woodward M, Bilo HJ, Chalmers J, Heerspink HJ, et al. Associations of kidney disease measures with mortality and end-stage renal disease in individuals with and without diabetes: a meta-analysis. Lancet. 2012; 380 :1662–1673. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Furberg CD, Pitt B. Withdrawal of cerivastatin from the world market. Curr Control Trials Cardiovasc Med. 2001; 2 :205–207. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Gordon T, Kannel WB, Castelli WP, Dawber TR. Lipoproteins, cardiovascular disease, and death. The Framingham study. Arch Intern Med. 1981; 141 :1128–1131. [ PubMed ] [ Google Scholar ]
• Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002; 359 :248–252. [ PubMed ] [ Google Scholar ]
• Hernan MA, Robins JM. Instruments for causal inference: an epidemiologist’s dream? Epidemiology. 2006; 17 :360–372. [ PubMed ] [ Google Scholar ]
• Hill AB. The Environment and Disease: Association or Causation? Proceedings of the Royal Society of Medicine. 1965; 58 :295–300. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kannel WB, Wolf PA, Verter J, McNamara PM. Epidemiologic assessment of the role of blood pressure in stroke. The Framingham study. JAMA. 1970; 214 :301–310. [ PubMed ] [ Google Scholar ]
• Knowler WC, Williams RC, Pettitt DJ, Steinberg AG. Gm3;5,13,14 and type 2 diabetes mellitus: an association in American Indians with genetic admixture. American journal of human genetics. 1988; 43 :520–526. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lawson DH. Pharmacoepidemiology: a new discipline. Br Med J (Clin Res Ed) 1984; 289 :940–941. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Leibel B. An analysis of the University Group Diabetes Study Program: data results and conslusions. Can Med Assoc J. 1971; 105 :292–294. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• McMahon AD, MacDonald TM. Design issues for drug epidemiology. Br J Clin Pharmacol. 2000; 50 :419–425. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Meldrum ML. A brief history of the randomized controlled trial. From oranges and lemons to the gold standard. Hematol Oncol Clin North Am. 2000; 14 :745–760. vii. [ PubMed ] [ Google Scholar ]
• Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980; 302 :1109–1117. [ PubMed ] [ Google Scholar ]
• Petri H, Urquhart J. Channeling bias in the interpretation of drug effects. Stat Med. 1991; 10 :577–581. [ PubMed ] [ Google Scholar ]
• Pfister R, Barnes D, Luben R, Forouhi NG, Bochud M, Khaw KT, et al. No evidence for a causal link between uric acid and type 2 diabetes: a Mendelian randomisation approach. Diabetologia. 2011; 54 :2561–2569. [ PubMed ] [ Google Scholar ]
• Phillips CV, Goodman KJ. The missed lessons of Sir Austin Bradford Hill. Epidemiol Perspect Innov. 2004; 1 :3. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Pirie K, Peto R, Reeves GK, Green J, Beral V. The 21st century hazards of smoking and benefits of stopping: a prospective study of one million women in the UK. Lancet. 2013; 381 :133–141. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Plasencia CM, Alderman BW, Baron AE, Rolfs RT, Boyko EJ. A method to describe physician decision thresholds and its application in examining the diagnosis of coronary artery disease based on exercise treadmill testing. Med Decis Making. 1992; 12 :204–212. [ PubMed ] [ Google Scholar ]
• Prentice RL, Langer R, Stefanick ML, Howard BV, Pettinger M, Anderson G, et al. Combined postmenopausal hormone therapy and cardiovascular disease: toward resolving the discrepancy between observational studies and the Women’s Health Initiative clinical trial. Am J Epidemiol. 2005; 162 :404–414. [ PubMed ] [ Google Scholar ]
• Psaty BM, Weiss NS, Furberg CD, Koepsell TD, Siscovick DS, Rosendaal FR, et al. Surrogate end points, health outcomes, and the drug-approval process for the treatment of risk factors for cardiovascular disease. JAMA. 1999; 282 :786–790. [ PubMed ] [ Google Scholar ]
• Rothman KJ, Greenland S. Causation and causal inference in epidemiology. Am J Public Health. 2005; 95 (Suppl 1):S144–150. [ PubMed ] [ Google Scholar ]
• Roumie CL, Hung AM, Greevy RA, Grijalva CG, Liu X, Murff HJ, et al. Comparative effectiveness of sulfonylurea and metformin monotherapy on cardiovascular events in type 2 diabetes mellitus: a cohort study. Ann Intern Med. 2012; 157 :601–610. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Rubin DB. Propensity score methods. Am J Ophthalmol. 2010; 149 :7–9. [ PubMed ] [ Google Scholar ]
• Rubin RR, Fujimoto WY, Marrero DG, Brenneman T, Charleston JB, Edelstein SL, et al. The Diabetes Prevention Program: recruitment methods and results. Control Clin Trials. 2002; 23 :157–171. [ PubMed ] [ Google Scholar ]
• Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J Clin Epidemiol. 2005; 58 :550–559. [ PubMed ] [ Google Scholar ]
• Sobek M, Cleveland L, Flood S, Hall PK, King ML, Ruggles S, et al. Big Data: Large-Scale Historical Infrastructure from the Minnesota Population Center. Hist Methods. 2011; 44 :61–68. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Sturmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol. 2006; 59 :437–447. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Thomas DC, Conti DV. Commentary: the concept of ‘Mendelian Randomization’ Int J Epidemiol. 2004; 33 :21–25. [ PubMed ] [ Google Scholar ]
• Tinker LF, Sarto GE, Howard BV, Huang Y, Neuhouser ML, Mossavar-Rahmani Y, et al. Biomarker-calibrated dietary energy and protein intake associations with diabetes risk among postmenopausal women from the Women’s Health Initiative. Am J Clin Nutr. 2011; 94 :1600–1606. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• van Stralen KJ, Dekker FW, Zoccali C, Jager KJ. Confounding. Nephron Clin Pract. 2010; 116 :c143–147. [ PubMed ] [ Google Scholar ]
• Wheeler S, Moore K, Forsberg CW, Riley K, Floyd JS, Smith NL, et al. Mortality among veterans with type 2 diabetes initiating metformin, sulfonylurea or rosiglitazone monotherapy. Diabetologia 2013 [ PubMed ] [ Google Scholar ]
• Wysowski DK, Swartz L. Adverse drug event surveillance and drug withdrawals in the United States, 1969–2002: the importance of reporting suspected reactions. Arch Intern Med. 2005; 165 :1363–1369. [ PubMed ] [ Google Scholar ]

African farmers look to the past and the future to address climate change

HARARE, Zimbabwe — From ancient fertilizer methods in Zimbabwe to new greenhouse technology in Somalia, farmers across the heavily agriculture-reliant African continent are looking to the past and future to respond to climate change.

Africa, with the world’s youngest population, faces the worst effects of a warming planet while contributing the least to the problem. Farmers are scrambling to make sure the booming population is fed.

With over 60% of the world’s uncultivated land, Africa should be able to feed itself, some experts say. And yet three in four people across the continent cannot afford a healthy diet, according to a report last year by the African Union and United Nations agencies. Reasons include conflict and lack of investment.

In Zimbabwe, where the El Nino phenomenon has worsened a drought , small-scale farmer James Tshuma has lost hope of harvesting anything from his fields. It’s a familiar story in much of the country, where the government has declared a $2 billion state of emergency and millions of people face hunger.

But a patch of green vegetables is thriving in a small garden the 65-year-old Tshuma is keeping alive with homemade organic manure and fertilizer. Previously discarded items have again become priceless.

“This is how our fathers and forefathers used to feed the earth and themselves before the introduction of chemicals and inorganic fertilizers,” Tshuma said.

He applies livestock droppings, grass, plant residue, remains of small animals, tree leaves and bark, food scraps and other biodegradable items like paper. Even the bones of animals that are dying in increasing numbers due to the drought are burned before being crushed into ash for their calcium.

Climate change is compounding much of sub-Saharan Africa’s longstanding problem of poor soil fertility, said Wonder Ngezimana, an associate professor of crop science at Zimbabwe’s Marondera University of Agricultural Sciences and Technology.

“The combination is forcing people to re-look at how things were done in the past like nutrient recycling, but also blending these with modern methods,” said Ngezimana, whose institution is researching the combination of traditional practices with new technologies.

Apart from being rich in nitrogen, organic fertilizers help increase the soil’s carbon and ability to retain moisture, Ngezimana said. “Even if a farmer puts synthetic fertilizer into the soil, they are likely to suffer the consequences of poor moisture as long as there is a drought,” he said.

Other moves to traditional practices are under way. Drought-resistant millets , sorghum and legumes, staples until the early 20th century when they were overtaken by exotic white corn, have been taking up more land space in recent years.

Leaves of drought-resistant plants that were once a regular dish before being cast off as weeds are returning to dinner tables. They even appear on elite supermarket shelves and are served at classy restaurants, as are millet and sorghum.

This could create markets for the crops even beyond drought years, Ngezimana said.

A GREENHOUSE REVOLUTION IN SOMALIA

In conflict-prone Somalia in East Africa, greenhouses are changing the way some people live, with shoppers filling up carts with locally produced vegetables and traditionally nomadic pastoralists under pressure to settle down and grow crops.

“They are organic, fresh and healthy,” shopper Sucdi Hassan said in the capital, Mogadishu. “Knowing that they come from our local farms makes us feel secure.”

Her new shopping experience is a sign of relative calm after three decades of conflict and the climate shocks of drought and flooding.

Urban customers are now assured of year-round supplies, with more than 250 greenhouses dotted across Mogadishu and its outskirts producing fruit and vegetables. It is a huge leap.

“In the past, even basic vegetables like cucumbers and tomatoes were imported, causing logistical problems and added expenses,” said Somalia’s minister of youth and sports, Mohamed Barre.

The greenhouses also create employment in a country where about 75% of the population is people under 30 years old, many of them jobless.

About 15 kilometers (9 miles) from the capital, Mohamed Mahdi, an agriculture graduate, inspected produce in a greenhouse where he works.

“Given the high unemployment rate, we are grateful for the chance to work in our chosen field of expertise,” the 25-year-old said.

Meanwhile, some pastoralist herders are being forced to change their traditional ways after watching livestock die by the thousands.

“Transitioning to greenhouse farming provides pastoralists with a more resilient and sustainable livelihood option,” said Mohamed Okash, director of the Institute of Climate and Environment at SIMAD University in Mogadishu.

He called for larger investments in smart farming to combat food insecurity.

A MORE RESILIENT BEAN IN KENYA

In Kenya, a new climate-smart bean variety is bringing hope to farmers in a region that had recorded reduced rainfall in six consecutive rainy seasons.

The variety, called “Nyota” or “star” in Swahili, is the result of a collaboration between scientists from the Kenya Agricultural and Livestock Research Organization, the Alliance of Bioversity International and research organization International Center for Tropical Agriculture.

The new bean variety is tailored for Kenya’s diverse climatic conditions. One focus is to make sure drought doesn’t kill them off before they have time to flourish.

The bean variety flowers and matures so quickly that it is ready for harvesting by the time rains disappear, said David Karanja, a bean breeder and national coordinator for grains and legumes at KALRO.

Hopes are that these varieties could bolster national bean production. The annual production of 600,000 metric tons falls short of meeting annual demand of 755,000 metric tons, Karanja said.

Farmer Benson Gitonga said his yield and profits are increasing because of the new bean variety. He harvests between nine and 12 bags from an acre of land, up from the previous five to seven bags.

One side benefit of the variety is a breath of fresh air.

“Customers particularly appreciate its qualities, as it boasts low flatulence levels, making it an appealing choice,” Gitonga said.

Tiro reported from Nairobi, Kenya and Faruk reported from Mogadishu, Somalia.

The Associated Press receives financial support for global health and development coverage in Africa from the Bill & Melinda Gates Foundation Trust. The AP is solely responsible for all content. Find AP’s standards for working with philanthropies, a list of supporters and funded coverage areas at AP.org .

AP Africa news: https://apnews.com/hub/africa