exploratory data analysis in r case study

• Case Study: Exploratory Data Analysis in R
• by Daniel Pinedo
• Last updated over 3 years ago
• Hide Comments (–) Share Hide Toolbars

Twitter Facebook Google+

Or copy & paste this link into an email or IM:

Statistics Made Easy

How to Perform Exploratory Data Analysis in R (With Example)

One of the first steps of any data analysis project is exploratory data analysis .

This involves exploring a dataset in three ways:

1. Summarizing a dataset using descriptive statistics.

2. Visualizing a dataset using charts.

3. Identifying missing values.

By performing these three actions, you can gain an understanding of how the values in a dataset are distributed and detect any problematic values before proceeding to perform a hypothesis test or perform statistical modeling.

The easiest way to perform exploratory data analysis in R is by using functions from the tidyverse packages.

The following step-by-step example shows how to use functions from these packages to perform exploratory data analysis on the diamonds dataset that comes built-in with the tidyverse packages.

Step 1: Load & View the Data

First, let’s use the data() function to load the diamonds dataset:

We can take a look at the first six rows of the dataset by using the head() function:

Step 2: Summarize the Data

We can use the summary() function to quickly summarize each variable in the dataset:

For each of the numeric variables we can see the following information:

• Min : The minimum value.
• 1st Qu : The value of the first quartile (25th percentile).
• Median : The median value.
• Mean : The mean value.
• 3rd Qu : The value of the third quartile (75th percentile).
• Max : The maximum value.

For the categorical variables in the dataset (cut, color, and clarity) we see a frequency count of each value.

 For example, for the cut variable:

• Fair : This value occurs 1,610 times.
• Good : This value occurs 4,906 times.
• Very Good : This value occurs 12,082 times.
• Premium : This value occurs 13,791 times.
• Ideal : This value occurs 21,551 times.

We can use the dim() function to get the dimensions of the dataset in terms of number of rows and number of columns:

We can see that the dataset has 53,940 rows and 10  columns.

Step 3: Visualize the Data

We can also create charts to visualize the values in the dataset.

For example, we can use the geom_histogram() function to create a histogram of the values for a certain variable:

We can also use the geom_point() function to create a scatterplot of any pairwise combination of variables:

We can also use the geom_boxplot() function to create a boxplot of one variable grouped by another variable:

We can also use the cor() function to create a correlation matrix to view the correlation coefficient between each pairwise combination of numeric variables in the dataset:

Related: What is Considered to Be a “Strong” Correlation?

Step 4: Identify Missing Values

We can use the following code to count the total number of missing values in each column of the dataset:

From the output we can see that there are zero missing values in each column.

In practice, you’ll likely encounter several missing values throughout your dataset.

This function will be useful for counting the total number of missing values.

Related: How to Impute Missing Values in R

Additional Resources

The following tutorials explain how to perform other common operations in R:

How to Use length() Function in R How to Use cat() Function in R How to Use substring() Function in R

Featured Posts

Hey there. My name is Zach Bobbitt. I have a Masters of Science degree in Applied Statistics and I’ve worked on machine learning algorithms for professional businesses in both healthcare and retail. I’m passionate about statistics, machine learning, and data visualization and I created Statology to be a resource for both students and teachers alike.  My goal with this site is to help you learn statistics through using simple terms, plenty of real-world examples, and helpful illustrations.

One Reply to “How to Perform Exploratory Data Analysis in R (With Example)”

really good and interestant, thank you.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Join the Statology Community

I have read and agree to the terms & conditions

10   Exploratory data analysis

10.1 introduction.

This chapter will show you how to use visualization and transformation to explore your data in a systematic way, a task that statisticians call exploratory data analysis, or EDA for short. EDA is an iterative cycle. You:

Generate questions about your data.

Search for answers by visualizing, transforming, and modelling your data.

Use what you learn to refine your questions and/or generate new questions.

EDA is not a formal process with a strict set of rules. More than anything, EDA is a state of mind. During the initial phases of EDA you should feel free to investigate every idea that occurs to you. Some of these ideas will pan out, and some will be dead ends. As your exploration continues, you will home in on a few particularly productive insights that you’ll eventually write up and communicate to others.

EDA is an important part of any data analysis, even if the primary research questions are handed to you on a platter, because you always need to investigate the quality of your data. Data cleaning is just one application of EDA: you ask questions about whether your data meets your expectations or not. To do data cleaning, you’ll need to deploy all the tools of EDA: visualization, transformation, and modelling.

10.1.1 Prerequisites

In this chapter we’ll combine what you’ve learned about dplyr and ggplot2 to interactively ask questions, answer them with data, and then ask new questions.

10.2 Questions

“There are no routine statistical questions, only questionable statistical routines.” — Sir David Cox

“Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.” — John Tukey

Your goal during EDA is to develop an understanding of your data. The easiest way to do this is to use questions as tools to guide your investigation. When you ask a question, the question focuses your attention on a specific part of your dataset and helps you decide which graphs, models, or transformations to make.

EDA is fundamentally a creative process. And like most creative processes, the key to asking quality questions is to generate a large quantity of questions. It is difficult to ask revealing questions at the start of your analysis because you do not know what insights can be gleaned from your dataset. On the other hand, each new question that you ask will expose you to a new aspect of your data and increase your chance of making a discovery. You can quickly drill down into the most interesting parts of your data—and develop a set of thought-provoking questions—if you follow up each question with a new question based on what you find.

There is no rule about which questions you should ask to guide your research. However, two types of questions will always be useful for making discoveries within your data. You can loosely word these questions as:

What type of variation occurs within my variables?

What type of covariation occurs between my variables?

The rest of this chapter will look at these two questions. We’ll explain what variation and covariation are, and we’ll show you several ways to answer each question.

10.3 Variation

Variation is the tendency of the values of a variable to change from measurement to measurement. You can see variation easily in real life; if you measure any continuous variable twice, you will get two different results. This is true even if you measure quantities that are constant, like the speed of light. Each of your measurements will include a small amount of error that varies from measurement to measurement. Variables can also vary if you measure across different subjects (e.g., the eye colors of different people) or at different times (e.g., the energy levels of an electron at different moments). Every variable has its own pattern of variation, which can reveal interesting information about how that it varies between measurements on the same observation as well as across observations. The best way to understand that pattern is to visualize the distribution of the variable’s values, which you’ve learned about in Chapter 1 .

We’ll start our exploration by visualizing the distribution of weights ( carat ) of ~54,000 diamonds from the diamonds dataset. Since carat is a numerical variable, we can use a histogram:

Now that you can visualize variation, what should you look for in your plots? And what type of follow-up questions should you ask? We’ve put together a list below of the most useful types of information that you will find in your graphs, along with some follow-up questions for each type of information. The key to asking good follow-up questions will be to rely on your curiosity (What do you want to learn more about?) as well as your skepticism (How could this be misleading?).

10.3.1 Typical values

In both bar charts and histograms, tall bars show the common values of a variable, and shorter bars show less-common values. Places that do not have bars reveal values that were not seen in your data. To turn this information into useful questions, look for anything unexpected:

Which values are the most common? Why?

Which values are rare? Why? Does that match your expectations?

Can you see any unusual patterns? What might explain them?

Let’s take a look at the distribution of carat for smaller diamonds.

This histogram suggests several interesting questions:

Why are there more diamonds at whole carats and common fractions of carats?

Why are there more diamonds slightly to the right of each peak than there are slightly to the left of each peak?

Visualizations can also reveal clusters, which suggest that subgroups exist in your data. To understand the subgroups, ask:

How are the observations within each subgroup similar to each other?

How are the observations in separate clusters different from each other?

How can you explain or describe the clusters?

Why might the appearance of clusters be misleading?

Some of these questions can be answered with the data while some will require domain expertise about the data. Many of them will prompt you to explore a relationship between variables, for example, to see if the values of one variable can explain the behavior of another variable. We’ll get to that shortly.

10.3.2 Unusual values

Outliers are observations that are unusual; data points that don’t seem to fit the pattern. Sometimes outliers are data entry errors, sometimes they are simply values at the extremes that happened to be observed in this data collection, and other times they suggest important new discoveries. When you have a lot of data, outliers are sometimes difficult to see in a histogram. For example, take the distribution of the y variable from the diamonds dataset. The only evidence of outliers is the unusually wide limits on the x-axis.

There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you’ll spot something). To make it easy to see the unusual values, we need to zoom to small values of the y-axis with coord_cartesian() :

coord_cartesian() also has an xlim() argument for when you need to zoom into the x-axis. ggplot2 also has xlim() and ylim() functions that work slightly differently: they throw away the data outside the limits.

This allows us to see that there are three unusual values: 0, ~30, and ~60. We pluck them out with dplyr:

The y variable measures one of the three dimensions of these diamonds, in mm. We know that diamonds can’t have a width of 0mm, so these values must be incorrect. By doing EDA, we have discovered missing data that was coded as 0, which we never would have found by simply searching for NA s. Going forward we might choose to re-code these values as NA s in order to prevent misleading calculations. We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don’t cost hundreds of thousands of dollars!

It’s good practice to repeat your analysis with and without the outliers. If they have minimal effect on the results, and you can’t figure out why they’re there, it’s reasonable to omit them, and move on. However, if they have a substantial effect on your results, you shouldn’t drop them without justification. You’ll need to figure out what caused them (e.g., a data entry error) and disclose that you removed them in your write-up.

10.3.3 Exercises

Explore the distribution of each of the x , y , and z variables in diamonds . What do you learn? Think about a diamond and how you might decide which dimension is the length, width, and depth.

Explore the distribution of price . Do you discover anything unusual or surprising? (Hint: Carefully think about the binwidth and make sure you try a wide range of values.)

How many diamonds are 0.99 carat? How many are 1 carat? What do you think is the cause of the difference?

Compare and contrast coord_cartesian() vs.  xlim() or ylim() when zooming in on a histogram. What happens if you leave binwidth unset? What happens if you try and zoom so only half a bar shows?

10.4 Unusual values

If you’ve encountered unusual values in your dataset, and simply want to move on to the rest of your analysis, you have two options.

Drop the entire row with the strange values:

We don’t recommend this option because one invalid value doesn’t imply that all the other values for that observation are also invalid. Additionally, if you have low quality data, by the time that you’ve applied this approach to every variable you might find that you don’t have any data left!

Instead, we recommend replacing the unusual values with missing values. The easiest way to do this is to use mutate() to replace the variable with a modified copy. You can use the if_else() function to replace unusual values with NA :

It’s not obvious where you should plot missing values, so ggplot2 doesn’t include them in the plot, but it does warn that they’ve been removed:

To suppress that warning, set na.rm = TRUE :

Other times you want to understand what makes observations with missing values different to observations with recorded values. For example, in nycflights13::flights 1 , missing values in the dep_time variable indicate that the flight was cancelled. So you might want to compare the scheduled departure times for cancelled and non-cancelled times. You can do this by making a new variable, using is.na() to check if dep_time is missing.

However this plot isn’t great because there are many more non-cancelled flights than cancelled flights. In the next section we’ll explore some techniques for improving this comparison.

10.4.1 Exercises

What happens to missing values in a histogram? What happens to missing values in a bar chart? Why is there a difference in how missing values are handled in histograms and bar charts?

What does na.rm = TRUE do in mean() and sum() ?

Recreate the frequency plot of scheduled_dep_time colored by whether the flight was cancelled or not. Also facet by the cancelled variable. Experiment with different values of the scales variable in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights.

10.5 Covariation

If variation describes the behavior within a variable, covariation describes the behavior between variables. Covariation is the tendency for the values of two or more variables to vary together in a related way. The best way to spot covariation is to visualize the relationship between two or more variables.

10.5.1 A categorical and a numerical variable

For example, let’s explore how the price of a diamond varies with its quality (measured by cut ) using geom_freqpoly() :

Note that ggplot2 uses an ordered color scale for cut because it’s defined as an ordered factor variable in the data. You’ll learn more about these in Section 16.6 .

The default appearance of geom_freqpoly() is not that useful here because the height, determined by the overall count, differs so much across cut s, making it hard to see the differences in the shapes of their distributions.

To make the comparison easier we need to swap what is displayed on the y-axis. Instead of displaying count, we’ll display the density , which is the count standardized so that the area under each frequency polygon is one.

Note that we’re mapping the density to y , but since density is not a variable in the diamonds dataset, we need to first calculate it. We use the after_stat() function to do so.

There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because frequency polygons are a little hard to interpret - there’s a lot going on in this plot.

A visually simpler plot for exploring this relationship is using side-by-side boxplots.

We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.

cut is an ordered factor: fair is worse than good, which is worse than very good and so on. Many categorical variables don’t have such an intrinsic order, so you might want to reorder them to make a more informative display. One way to do that is with fct_reorder() . You’ll learn more about that function in Section 16.4 , but we want to give you a quick preview here because it’s so useful. For example, take the class variable in the mpg dataset. You might be interested to know how highway mileage varies across classes:

To make the trend easier to see, we can reorder class based on the median value of hwy :

If you have long variable names, geom_boxplot() will work better if you flip it 90°. You can do that by exchanging the x and y aesthetic mappings.

10.5.1.1 Exercises

Use what you’ve learned to improve the visualization of the departure times of cancelled vs. non-cancelled flights.

Based on EDA, what variable in the diamonds dataset appears to be most important for predicting the price of a diamond? How is that variable correlated with cut? Why does the combination of those two relationships lead to lower quality diamonds being more expensive?

Instead of exchanging the x and y variables, add coord_flip() as a new layer to the vertical boxplot to create a horizontal one. How does this compare to exchanging the variables?

One problem with boxplots is that they were developed in an era of much smaller datasets and tend to display a prohibitively large number of “outlying values”. One approach to remedy this problem is the letter value plot. Install the lvplot package, and try using geom_lv() to display the distribution of price vs. cut. What do you learn? How do you interpret the plots?

Create a visualization of diamond prices vs. a categorical variable from the diamonds dataset using geom_violin() , then a faceted geom_histogram() , then a colored geom_freqpoly() , and then a colored geom_density() . Compare and contrast the four plots. What are the pros and cons of each method of visualizing the distribution of a numerical variable based on the levels of a categorical variable?

If you have a small dataset, it’s sometimes useful to use geom_jitter() to avoid overplotting to more easily see the relationship between a continuous and categorical variable. The ggbeeswarm package provides a number of methods similar to geom_jitter() . List them and briefly describe what each one does.

10.5.2 Two categorical variables

To visualize the covariation between categorical variables, you’ll need to count the number of observations for each combination of levels of these categorical variables. One way to do that is to rely on the built-in geom_count() :

The size of each circle in the plot displays how many observations occurred at each combination of values. Covariation will appear as a strong correlation between specific x values and specific y values.

Another approach for exploring the relationship between these variables is computing the counts with dplyr:

Then visualize with geom_tile() and the fill aesthetic:

If the categorical variables are unordered, you might want to use the seriation package to simultaneously reorder the rows and columns in order to more clearly reveal interesting patterns. For larger plots, you might want to try the heatmaply package, which creates interactive plots.

10.5.2.1 Exercises

How could you rescale the count dataset above to more clearly show the distribution of cut within color, or color within cut?

What different data insights do you get with a segmented bar chart if color is mapped to the x aesthetic and cut is mapped to the fill aesthetic? Calculate the counts that fall into each of the segments.

Use geom_tile() together with dplyr to explore how average flight departure delays vary by destination and month of year. What makes the plot difficult to read? How could you improve it?

10.5.3 Two numerical variables

You’ve already seen one great way to visualize the covariation between two numerical variables: draw a scatterplot with geom_point() . You can see covariation as a pattern in the points. For example, you can see a positive relationship between the carat size and price of a diamond: diamonds with more carats have a higher price. The relationship is exponential.

(In this section we’ll use the smaller dataset to stay focused on the bulk of the diamonds that are smaller than 3 carats)

Scatterplots become less useful as the size of your dataset grows, because points begin to overplot, and pile up into areas of uniform black, making it hard to judge differences in the density of the data across the 2-dimensional space as well as making it hard to spot the trend. You’ve already seen one way to fix the problem: using the alpha aesthetic to add transparency.

But using transparency can be challenging for very large datasets. Another solution is to use bin. Previously you used geom_histogram() and geom_freqpoly() to bin in one dimension. Now you’ll learn how to use geom_bin2d() and geom_hex() to bin in two dimensions.

geom_bin2d() and geom_hex() divide the coordinate plane into 2d bins and then use a fill color to display how many points fall into each bin. geom_bin2d() creates rectangular bins. geom_hex() creates hexagonal bins. You will need to install the hexbin package to use geom_hex() .

Another option is to bin one continuous variable so it acts like a categorical variable. Then you can use one of the techniques for visualizing the combination of a categorical and a continuous variable that you learned about. For example, you could bin carat and then for each group, display a boxplot:

cut_width(x, width) , as used above, divides x into bins of width width . By default, boxplots look roughly the same (apart from number of outliers) regardless of how many observations there are, so it’s difficult to tell that each boxplot summaries a different number of points. One way to show that is to make the width of the boxplot proportional to the number of points with varwidth = TRUE .

10.5.3.1 Exercises

Instead of summarizing the conditional distribution with a boxplot, you could use a frequency polygon. What do you need to consider when using cut_width() vs.  cut_number() ? How does that impact a visualization of the 2d distribution of carat and price ?

Visualize the distribution of carat , partitioned by price .

How does the price distribution of very large diamonds compare to small diamonds? Is it as you expect, or does it surprise you?

Combine two of the techniques you’ve learned to visualize the combined distribution of cut, carat, and price.

Two dimensional plots reveal outliers that are not visible in one dimensional plots. For example, some points in the following plot have an unusual combination of x and y values, which makes the points outliers even though their x and y values appear normal when examined separately. Why is a scatterplot a better display than a binned plot for this case?

Instead of creating boxes of equal width with cut_width() , we could create boxes that contain roughly equal number of points with cut_number() . What are the advantages and disadvantages of this approach?

10.6 Patterns and models

If a systematic relationship exists between two variables it will appear as a pattern in the data. If you spot a pattern, ask yourself:

Could this pattern be due to coincidence (i.e. random chance)?

How can you describe the relationship implied by the pattern?

How strong is the relationship implied by the pattern?

What other variables might affect the relationship?

Does the relationship change if you look at individual subgroups of the data?

Patterns in your data provide clues about relationships, i.e., they reveal covariation. If you think of variation as a phenomenon that creates uncertainty, covariation is a phenomenon that reduces it. If two variables covary, you can use the values of one variable to make better predictions about the values of the second. If the covariation is due to a causal relationship (a special case), then you can use the value of one variable to control the value of the second.

Models are a tool for extracting patterns out of data. For example, consider the diamonds data. It’s hard to understand the relationship between cut and price, because cut and carat, and carat and price are tightly related. It’s possible to use a model to remove the very strong relationship between price and carat so we can explore the subtleties that remain. The following code fits a model that predicts price from carat and then computes the residuals (the difference between the predicted value and the actual value). The residuals give us a view of the price of the diamond, once the effect of carat has been removed. Note that instead of using the raw values of price and carat , we log transform them first, and fit a model to the log-transformed values. Then, we exponentiate the residuals to put them back in the scale of raw prices.

Once you’ve removed the strong relationship between carat and price, you can see what you expect in the relationship between cut and price: relative to their size, better quality diamonds are more expensive.

We’re not discussing modelling in this book because understanding what models are and how they work is easiest once you have tools of data wrangling and programming in hand.

10.7 Summary

In this chapter you’ve learned a variety of tools to help you understand the variation within your data. You’ve seen techniques that work with a single variable at a time and with a pair of variables. This might seem painfully restrictive if you have tens or hundreds of variables in your data, but they’re the foundation upon which all other techniques are built.

In the next chapter, we’ll focus on the tools we can use to communicate our results.

Remember that when we need to be explicit about where a function (or dataset) comes from, we’ll use the special form package::function() or package::dataset . ↩︎

Exploratory Data Analysis in R: Case Study (DataCamp)

Ch. 1 - data cleaning and summarizing with dplyr, the united nations voting dataset, filtering rows, adding a year column, adding a country column, grouping and summarizing, summarizing the full dataset, summarizing by year, summarizing by country, sorting and filtering summarized data, sorting by percentage of “yes” votes, filtering summarized output, ch. 2 - data visualization with ggplot2, visualization with ggplot2, choosing an aesthetic, plotting a line over time, other ggplot2 layers, visualizing by country, summarizing by year and country, plotting just the uk over time, plotting multiple countries, faceting by country, faceting with free y-axis, choose your own countries, ch. 3 - tidy modeling with broom, linear regression, linear regression on the united states, finding the slope of a linear regression, finding the p-value of a linear regression, tidying models with broom, tidying a linear regression model, combining models for multiple countries, nesting for multiple models, nesting a data frame, list columns, fitting multiple models, performing linear regression on each nested dataset, tidy each linear regression model, unnesting a data frame, working with many tidy models, filtering model terms, filtering for significant countries, sorting by slope, ch. 4 - joining and tidying, joining datasets, joining datasets with inner_join, filtering the joined dataset, visualizing colonialism votes, tidy data observations, using gather to tidy a dataset, recoding the topics, summarize by country, year, and topic, visualizing trends in topics for one country, tidy modeling by topic and country, nesting by topic and country, interpreting tidy models, steepest trends by topic, checking models visually, about michael mallari.

Michael is a hybrid thinker and doer —a byproduct of being a StrengthsFinder “Learner” over time. With nearly 20 years of engineering, design, and product experience, he helps organizations identify market needs, mobilize internal and external resources, and deliver delightful digital customer experiences that align with business goals. He has been entrusted with problem-solving for brands—ranging from Fortune 500 companies to early-stage startups to not-for-profit organizations.

Michael earned his BS in Computer Science from New York Institute of Technology and his MBA from the University of Maryland, College Park. He is also a candidate to receive his MS in Applied Analytics from Columbia University.

LinkedIn | Twitter | michaelmallari.com

R news and tutorials contributed by hundreds of R bloggers

New course exploratory data analysis in r: case study.

Posted on January 13, 2017 by DataCamp Blog in R bloggers | 0 Comments

[social4i size="small" align="align-left"] --> [This article was first published on DataCamp Blog , and kindly contributed to R-bloggers ]. (You can report issue about the content on this page here ) Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

What you’ll learn

To leave a comment for the author, please follow the link and comment on their blog: DataCamp Blog . R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job . Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Copyright © 2022 | MH Corporate basic by MH Themes

Never miss an update! Subscribe to R-bloggers to receive e-mails with the latest R posts. (You will not see this message again.)

Exploratory data analysis

Tutorial description.

When your dataset is represented as a table or a database, it’s difficult to observe much about it beyond its size and the types of variables it contains. In this tutorial, you’ll learn how to use graphical and numerical techniques to begin uncovering the structure of your data. Which variables suggest interesting relationships? Which observations are unusual? By the end of the tutorial, you’ll be able to answer these questions and more, while generating graphics that are both insightful and beautiful.

Learning objectives

• Visualize categorical and numerical data using appropriate graphics.
• Create graphical representations of multiple variables.
• Describe the structure revealed by graphics in the language of distributions.
• Use statistics to summarize important aspects of data.
• Identify unusual observations

1 - Visualizing categorical data

Creating graphical and numerical summaries of two categorical variables, primarily using two R packages: ggplot2 and dplyr.

• Side-by-side bar charts
• Stacked bar charts
• To normalize or not to normalize
• Computation of margins
• Counts vs proportions
• Law of total probability
• Marginal vs conditional
• data integrity check for levels

2 - Visualizing numerical data

Learn useful statistics for describing distributions of data.

• Graphical representation of one categorical and one numerical variable
• Side-by-side boxplots
• Faceted histograms
• Colored density curves
• Graphical representation of one numerical variable
• Marginal vs conditioning
• Density plot
• outlier detection

3 - Summarizing with statistics

Statistics for describing distributions of data.

• Center: mean, median, mode
• Shape: skewness, modality
• Spread: range, IQR, SD, variance
• Unusual observations
• Transformations: Logarithm and sqrt to reduce skew in graphics and ease comparisons.

4 - Case study

Apply what you’ve learned to explore and summarize a real world dataset in this case study of email spam.

Additional references

• Unwin, Anthony. Graphical Data Analysis with R .
• Velleman, Paul and Hoaglin, David. Exploratory Data Analysis .

Andrew Bray

Reed college.

Andrew Bray is an Assistant Professor of Statistics at Reed College and lover of all things statistics and R.

Exploratory Data Analysis with R

14 the ggplot2 plotting system: part 1.

The ggplot2 package in R is an implementation of The Grammar of Graphics as described by Leland Wilkinson in his book. The package was originally written by Hadley Wickham while he was a graduate student at Iowa State University (he still actively maintains the packgae). The package implements what might be considered a third graphics system for R (along with base graphics and lattice ). The package is available from CRAN via install.packages() ; the latest version of the source can be found on the package’s GitHub Repository . Documentation of the package can be found at http://docs.ggplot2.org/current/

The grammar of graphics represents an abstraction of graphics ideas and objects. You can think of this as developing the verbs, nouns, and adjectives for data graphics. Developing such a grammar allows for a “theory” of graphics on which to build new graphics and graphics objects. To quote from Hadley Wickham’s book on ggplot2 , we want to “shorten the distance from mind to page”. In summary,

“…the grammar tells us that a statistical graphic is a mapping from data to aesthetic attributes (colour, shape, size) of geometric objects (points, lines, bars). The plot may also contain statistical transformations of the data and is drawn on a specific coordinate system” – from ggplot2 book

You might ask yourself “Why do we need a grammar of graphics?” Well, for much the same reasons that having a grammar is useful for spoken languages. The grammer allows for a more compact summary of the base components of a language, and it allows us to extend the language and to handle situations that we have not before seen.

If you think about making a plot with the base graphics system, the plot is constructed by calling a series of functions that either create or annotate a plot. There’s no convenient agreed-upon way to describe the plot, except to just recite the series of R functions that were called to create the thing in the first place. In a previous chapter, we described the base plotting system as a kind of “artist’s palette” model, where you start with blank “canvas” and build up from there.

For example, consider the following plot made using base graphics.

Figure 14.1: Scatterplot of Temperature and Ozone in New York (base graphics)

How would one describe the creation of this plot? Well, we could say that we called the plot() function and then added a loess smoother by calling the lines() function on the output of loess.smooth() .

The base plotting system is convenient and it often mirrors how we think of building plots and analyzing data. But a key drawback is that you can’t go back once plot has started (e.g. to adjust margins), so there is in fact a need to plan in advance. Furthermore, it is difficult to “translate” a plot to others because there’s no formal graphical language; each plot is just a series of R commands.

Here’s the same plot made using ggplot2 .

Figure 5.1: Scatterplot of Temperature and Ozone in New York (ggplot2)

Note that the output is roughly equivalent, and the amount of code is similar, but ggplot2 allows for a more elegant way of expressing the components of the plot. In this case, the plot is a dataset ( airquality ) with aesthetic mappings derived from the Temp and Ozone variables, a set of points , and a smoother . In a sense, the ggplot2 system takes many of the cues from the base plotting system and formalizes them a bit.

The ggplot2 system also takes some cues from lattice . With the lattice system, plots are created with a single function call ( xyplot , bwplot , etc.). Things like margins and spacing are set automatically because the entire plot is specified at once. The lattice system is most useful for conditioning types of plots and is good for putting many many plots on a screen. That said, it is sometimes awkward to specify an entire plot in a single function call because many different options have to be specified at once. Furthermore, annotation in plots is not intuitive and the use of panel functions and subscripts is difficult to wield and requires intense preparation.

The ggplot2 system essentially takes the good parts of both the base graphics and lattice graphics system. It automatically handles things like margins and spacing, and also has the concept of “themes” which provide a default set of plotting symbols and colors. While ggplot2 bears a superficial similarity to lattice , ggplot2 is generally easier and more intuitive to use. The default thems makes many choices for you, but you can customize the presentation if you want.

14.1 The Basics: qplot()

The qplot() function in ggplot2 is meant to get you going q uickly. It works much like the plot() function in base graphics system. It looks for variables to plot within a data frame, similar to lattice, or in the parent environment. In general, it’s good to get used to putting your data in a data frame and then passing it to qplot() .

Plots are made up of aesthetics (size, shape, color) and geoms (points, lines). Factors play an important role for indicating subsets of the data (if they are to have different properties) so they should be labeled properly. The qplot() hides much of what goes on underneath, which is okay for most operations, ggplot() is the core function and is very flexible for doing things qplot() cannot do.

14.2 Before You Start: Label Your Data

One thing that is always true, but is particularly useful when using ggplot2 , is that you should always use informative and descriptive labels on your data. More generally, your data should have appropriate metadata so that you can quickly look at a dataset and know

what the variables are

what the values of each variable mean

This means that each column of a data frame should have a meaningful (but concise) variable name that accurately reflects the data stored in that column. Also, non-numeric or categorical variables should be coded as factor variables and have meaningful labels for each level of the factor. For example, it’s common to code a binary variable as a “0” or a “1”, but the problem is that from quickly looking at the data, it’s impossible to know whether which level of that variable is represented by a “0” or a “1”. Much better to simply label each observation as what they are. If a variable represents temperature categories, it might be better to use “cold”, “mild”, and “hot” rather than “1”, “2”, and “3”.

While it’s sometimes a pain to make sure all of your data are properly labelled, this investment in time can pay dividends down the road when you’re trying to figure out what you were plotting. In other words, including the proper metadata can make your exploratory plots essentially self-documenting.

14.3 ggplot2 “Hello, world!”

This example dataset comes with the ggplot2 package and contains data on the fuel economy of 38 popular models of car from 1999 to 2008.

You can see from the str() output that all of the factor variables are appropriately coded with meaningful labels. This will come in handy when qplot() has to label different aspects of a plot. Also note that all of the columns/variables have meaningful (if sometimes abbreviated) names, rather than names like “X1”, and “X2”, etc.

We can make a quick scatterplot of the engine displacement ( displ ) and the highway miles per gallon ( hwy ).

Figure 5.3: Plot of engine displacement and highway mileage

Note that in the call to qplot() you must specify the data argument so that qplot() knows where to look up the variables.

14.4 Modifying aesthetics

We can introduce a third variable into the plot by modifying the color of the points based on the value of that third variable. Color is an aesthetic and the color of each point can be mapped to a variable. Note that the x-coordinates and y-coordinates are aesthetics too, and they got mapped to the displ and hwy variables, respectively. In this case we will map the color to the drv variable which indicates whether a car is front wheel drive, rear wheel drive, or 4-wheel drive.

Figure 5.4: Engine displacement and highway mileage by drive class

Now we can see that the front wheel drive cars tend to have lower displacement relative to the 4-wheel or rear wheel drive cars. Also, it’s clear that the 4-wheel drive cars have the lowest highway gas mileage.

14.5 Adding a geom

Sometimes it’s nice to add a smoother to a scatterplot ot highlight any trends. Trends can be difficult to see if the data are very noisy or there are many data points obscuring the view. A smooth is a “geom” that you can add along with your data points.

Figure 5.5: Engine displacement and highway mileage w/smoother

Note that previously, we didn’t have to specify geom = "point" because that was done automatically. But if you want the smoother overlayed with the points, then you need to specify both explicitly.

Here it seems that engine displacement and highway mileage have a nonlinear U-shaped relationship, but from the previous plot we know that this is largely due to confounding by the drive class of the car.

14.6 Histograms

The qplot() function can be used to be used to plot 1-dimensional data too. By specifying a single variable, qplot() will by default make a histogram. Here we make a histogram if the highway mileage data and stratify on the drive class. So technically this is three histograms overlayed on top of each other.

Figure 5.6: Histogram of highway mileage by drive class

Having the different colors for each drive class is nice, but the three histograms can be a bit difficult to separate out. Side-by-side boxplots is one solution to this problem.

Figure 5.7: Boxplots of highway mileage by drive class

Another solution is to plot the histograms in separate panels using facets.

14.7 Facets

Facets are a way to create multiple panels of plots based on the levels of categorical variable. Here, we want to see a histogram of the highway mileages and the categorical variable is the drive class variable. We can do that using the facets argument to qplot() .

The facets argument expects a formula type of input, with a ~ separating the left hand side variable and the right hand side variable. The left hand side variable indicates how the rows of the panels should be divided and the right hand side variable indicates how the columns of the panels should be divided. Here, we just want three rows of histograms (and just one column), one for each drive class, so we specify drv on the left hand side and . on the right hand side indicating that there’s no variable there (it’s empty).

Figure 5.8: Histogram of highway mileage by drive class

We could also look at more data using facets, so instead of histograms we could look at scatterplots of engine displacement and highway mileage by drive class. Here we put the drv variable on the right hand side to indicate that we want a column for each drive class (as opposed to splitting by rows like we did above).

Figure 5.9: Engine displacement and highway mileage by drive class

What if you wanted to add a smoother to each one of those panels? Simple, you literally just add the smoother as another geom.

Figure 6.1: Engine displacement and highway mileage by drive class w/smoother

You could have also used the “geom” argument to qplot() , as in

There’s more than one way to do it.

14.8 Case Study: MAACS Cohort

This case study will use data based on the Mouse Allergen and Asthma Cohort Study (MAACS). This study was aimed at characterizing the indoor (home) environment and its relationship with asthma morbidity amonst children aged 5–17 living in Baltimore, MD. The children all had persistent asthma, defined as having had an exacerbation in the past year. A representative publication of results from this study can be found in this paper by Lu, et al.

NOTE: Because the individual-level data for this study are protected by various U.S. privacy laws, we cannot make those data available. For the purposes of this chapter, we have simulated data that share many of the same features of the original data, but do not contain any of the actual measurements or values contained in the original dataset.

The key variables are:

mopos : an indicator of whether the subject is allergic to mouse allergen (yes/no)

pm25 : average level of PM2.5 over the course of 7 days (micrograms per cubic meter)

eno : exhaled nitric oxide

The outcome of interest for this analysis will be exhaled nitric oxide (eNO), which is a measure of pulmonary inflamation. We can get a sense of how eNO is distributed in this population by making a quick histogram of the variable. Here, we take the log of eNO because some right-skew in the data.

Figure 10.1: Histogram of log eNO

A quick glance suggests that the histogram is a bit “fat”, suggesting that there might be multiple groups of people being lumped together. We can stratify the histogram by whether they are allergic to mouse.

Figure 6.4: Histogram of log eNO by mouse allergic status

We can see from this plot that the non-allergic subjects are shifted slightly to the left, indicating a lower eNO and less pulmonary inflammation. That said, there is significant overlap between the two groups.

An alternative to histograms is a density smoother, which sometimes can be easier to visualize when there are multiple groups. Here is a density smooth of the entire study population.

Figure 6.5: Density smooth of log eNO

And here are the densities straitified by allergic status. We can map the color aesthetic to the mopos variable.

Figure 6.6: Density smooth of log eNO by mouse allergic status

These tell the same story as the stratified histograms, which sould come as no surprise.

Now we can examine the indoor environment and its relationship to eNO. Here, we use the level of indoor PM2.5 as a measure of indoor environment air quality. We can make a simple scatterplot of PM2.5 and eNO.

Figure 6.7: eNO and PM2.5

The relationship appears modest at best, as there is substantial noise in the data. However, one question that we might be interested in is whether allergic individuals are prehaps more sensitive to PM2.5 inhalation than non-allergic individuals. To examine that question we can stratify the data into two groups.

This first plot uses different plot symbols for the two groups and overlays them on a single canvas. We can do this by mapping the mopos variable to the shape aesthetic.

Figure 14.2: eNO and PM2.5 by mouse allergic status

Because there is substantial overlap in the data it is a bit challenging to discern the circles from the triangles. Part of the reason might be that all of the symbols are the same color (black).

We can plot each group a different color to see if that helps.

Figure 6.8: eNO and PM2.5 by mouse allergic status

This is slightly better but the substantial overlap makes it difficult to discern any trends in the data. For this we need to add a smoother of some sort. Here we add a linear regression line (a type of smoother) to each group to see if there’s any difference.

Here we see quite clearly that the red group and the green group exhibit rather different relationships between PM2.5 and eNO. For the non-allergic individuals, there appears to be a slightly negative relationship between PM2.5 and eNO and for the allergic individuals, there is a positive relationship. This suggests a strong interaction between PM2.5 and allergic status, an hypothesis perhaps worth following up on in greater detail than this brief exploratory analysis.

Another, and perhaps more clear, way to visualize this interaction is to use separate panels for the non-allergic and allergic individuals using the facets argument to qplot() .

14.9 Summary of qplot()

The qplot() function in ggplot2 is the analog of plot() in base graphics but with many built-in features that the traditionaly plot() does not provide. The syntax is somewhere in between the base and lattice graphics system. The qplot() function is useful for quickly putting data on the page/screen, but for ultimate customization, it may make more sense to use some of the lower level functions that we discuss later in the next chapter.

Case Study: Exploratory Data Analysis in R

Page Links:

Description

Use data manipulation and visualization skills to explore the historical voting of the United Nations General Assembly. Read more.

This resource is offered by an affiliate partner. If you pay for training, we may earn a commission to support this site.

Career Relevance by Data Role

The techniques and tools covered in Case Study: Exploratory Data Analysis in R are most similar to the requirements found in Data Analyst job advertisements.

Tools and Techniques

Subscribe for updates, similar opportunities, data visualization with tableau project.

Coursera - University of California, Davis

Essential Design Principles for Tableau

Network analysis in r, intermediate network analysis in python, exploratory data analysis.

Coursera - Johns Hopkins University

Data Science: Visualization

edX - Harvard University

Visualizing Data in the Tidyverse

Ai workflow: data analysis and hypothesis testing, case studies: network analysis in r.

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

data manipulation and visualization like dplyr and ggplot2

allanbeto/Exploratory_Data_Analysis_in_R_Case_Study

Folders and files.