Basics on Stats and R

Introduction to data manipulation in R with {dplyr}

Mon, 27 Nov 2023 00:00:00 +0000

Introduction

In a previous post, we showed how to manipulate data in R. In particular, we illustrated how to create and manipulate vectors, factors, lists and data frames. This served as an introduction to R and was aimed at beginners. Moreover, as long as it was possible, all manipulations were made in base R, that is, without having to load any package.

In this post, we would like to show again how to manipulate data in R, but this time using the {dplyr} package.

The {dplyr} package, developed by Hadley Wickham and colleagues at Posit, provides a complete set of functions that help you solve the most common data manipulation challenges such as:

filtering observations based on their values
extracting observations based on their values or positions
sampling observations based on a specific number or fraction of rows
sorting observations based on one or several variables
selecting variables based on their names or positions
renaming variables
adding new variables based on existing ones
summarizing observations or variables to a single descriptive measure
performing any operation by group
categorizing observations into two or more groups
etc.

More information about the package can be found at dplyr.tidyverse.org.

In this post, we will present the most common functions for data manipulation and data management using the {dplyr} package (illustrated on a data frame). This is however not an exhaustive list! It is likely that you will need other functions that the ones presented here. For the interested readers, see the end of this post for further resources.

A question I am often asked is whether it is best to first learn data manipulation with base R and then with {dplyr}, or directly learn {dplyr}.

Everyone may not agree, but here is what I tend to answer. The interest in terms of efficiency and clarity/readability of the code that {dplyr} brings is obvious. So even when I give a training targeted to statisticians, I present the {dplyr} package. However, I still do believe that learning data manipulation with base R is important for two reasons:

Everyone will, at some point, be confronted with code written in base R (either from collaboration with other R users, or from code found in textbooks or online).
Through the feedback I receive from my students, I notice that {dplyr} is relatively easy to learn when you are familiar with base R (which is quite rewarding for those who struggled at the beginning).

For these reasons, I tend to teach data manipulation with base R first and then slowly switch to {dplyr} (up to the point that for some advanced training courses, I hardly use base R at all by the end of the course). As a side note, this is the approach I follow for data visualization in R as well: I teach first how to plot data with base R, then I gradually teach them how to use {ggplot2}.

With this approach, some students may have the impression that they wasted their time learning base R. At least, some may have this impression during the training. However, as soon as the training is done and they have to learn R by themselves or work on real projects, they are grateful of having learned both.

I am curious to hear from other teachers regarding their approach, so feel free to share your opinion.

Data

To present the different functions, we will use the data frame penguins, available within the {palmerpenguins} package (Horst et al. 2020). Data are available by CC-0 license and can be downloaded from CRAN:

# install.packages("palmerpenguins")
library(palmerpenguins)

Before going further, we rename the data frame as dat:

dat <- penguins

I like to call data frames I am working on with a generic name such as dat for two reasons:

Every time I need to write the name of the data frame, it is usually shorter to write dat than to write the name of the data frame (which is in this case penguins).
If I need to do similar analyses or plots on different data frames, the code I wrote in the past can be reused with only a few modifications. With this very simple trick, most of the time I only have to edit the names of the variables, but the name of the data frame does not need to be changed (which saves me a lot of time).

The data frame contains data for 344 penguins and 8 variables describing the species, the island, some measurements of the size of the bill, flipper and body mass, the sex and the study year. More information about the data frame can be found by running ?penguins (after loading the {palmerpenguins} package).

For this post, we will focus only on the variables species, body_mass_g, sex and year.

Before proceeding with the different data manipulation techniques, let’s first inspect the data by displaying its structure, the first 6 rows and a summary of it:

str(dat) # structure of the data

## tibble [344 × 4] (S3: tbl_df/tbl/data.frame)
##  $ species    : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ body_mass_g: int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex        : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year       : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

head(dat) # display first 6 rows

## # A tibble: 6 × 4
##   species body_mass_g sex     year
##   <fct>         <int> <fct>  <int>
## 1 Adelie         3750 male    2007
## 2 Adelie         3800 female  2007
## 3 Adelie         3250 female  2007
## 4 Adelie           NA <NA>    2007
## 5 Adelie         3450 female  2007
## 6 Adelie         3650 male    2007

summary(dat) # summary

##       species     body_mass_g       sex           year     
##  Adelie   :152   Min.   :2700   female:165   Min.   :2007  
##  Chinstrap: 68   1st Qu.:3550   male  :168   1st Qu.:2007  
##  Gentoo   :124   Median :4050   NA's  : 11   Median :2008  
##                  Mean   :4202                Mean   :2008  
##                  3rd Qu.:4750                3rd Qu.:2009  
##                  Max.   :6300                Max.   :2009  
##                  NA's   :2

{dplyr} package

Without further ado, let’s illustrate the different functions for data manipulation available in the {dplyr} package in the following sections.

As for any package, we first need to install and load it before using it:

# install.packages("dplyr")
library(dplyr)

Note that all functions presented below requires tidy data, which means that:

each variable is in its own column,
each observation, or case, is in its own row, and
each value is in its own cell.

Rules of a tidy data frame: variables are columns, observations are rows, and values are cells. Source: R for Data Science (2e) by H. Wickham, M. Çetinkaya-Rundel and G. Grolemund.

Filter observations

Filtering observations based on their values can be done with the filter() function. This function works on both quantitative and qualitative variables:

# filter observations based on a quantitative variable
filter(dat, body_mass_g > 4000)

## # A tibble: 172 × 4
##    species body_mass_g sex    year
##    <fct>         <int> <fct> <int>
##  1 Adelie         4675 male   2007
##  2 Adelie         4250 <NA>   2007
##  3 Adelie         4400 male   2007
##  4 Adelie         4500 male   2007
##  5 Adelie         4200 male   2007
##  6 Adelie         4150 male   2007
##  7 Adelie         4650 male   2007
##  8 Adelie         4400 male   2007
##  9 Adelie         4600 male   2007
## 10 Adelie         4150 male   2007
## # ℹ 162 more rows

# filter observations based on a qualitative variable
filter(dat, sex == "female")

## # A tibble: 165 × 4
##    species body_mass_g sex     year
##    <fct>         <int> <fct>  <int>
##  1 Adelie         3800 female  2007
##  2 Adelie         3250 female  2007
##  3 Adelie         3450 female  2007
##  4 Adelie         3625 female  2007
##  5 Adelie         3200 female  2007
##  6 Adelie         3700 female  2007
##  7 Adelie         3450 female  2007
##  8 Adelie         3325 female  2007
##  9 Adelie         3400 female  2007
## 10 Adelie         3800 female  2007
## # ℹ 155 more rows

You can combine several conditions with & (if the conditions must be cumulative) or | (if the conditions are alternatives), for instance:

# filter observations based on 2 cumulative conditions
filter(dat, body_mass_g > 4000 & sex == "female")

## # A tibble: 58 × 4
##    species body_mass_g sex     year
##    <fct>         <int> <fct>  <int>
##  1 Gentoo         4500 female  2007
##  2 Gentoo         4450 female  2007
##  3 Gentoo         4550 female  2007
##  4 Gentoo         4800 female  2007
##  5 Gentoo         4400 female  2007
##  6 Gentoo         4650 female  2007
##  7 Gentoo         4650 female  2007
##  8 Gentoo         4200 female  2007
##  9 Gentoo         4150 female  2007
## 10 Gentoo         4800 female  2007
## # ℹ 48 more rows

Notice that variable names do not have to be written inside single nor double quotation marks ('' or ""). This is the case for all functions presented below.

The pipe operator

As you can see from the code above, the filter() functions requires the name of the data frame as first argument, then the condition (with the usual logical operators >, <, >=, <=, ==, !=, %in%, etc.) as second argument.

Specifying the name of the data frame as first argument is required for all functions presented in this list. However, there is a workaround to specifying the data frame’s name inside the functions: the pipe operator (|> or %>%).¹

The pipe operator allows to perform a sequence of several operations, that is, chain a sequence of calculations together. It is particularly useful when you are performing several operations on a data frame, and you do not want to save the output at each intermediate step. We will see below how to use the pipe operator with several operations, but for now I would like to introduce it with only one operation at a time.

As you can see with the filter() function, the pipe operator is not compulsory. However, I recommend it so much (even to beginners) for its easy of use, convenience, code readability and popularity that from now on functions available in {dplyr} will be presented together with the pipe operator.

So with the pipe operator, the code above becomes:

# filter observations based on a quantitative variable
dat |>
  filter(body_mass_g > 4000)

## # A tibble: 172 × 4
##    species body_mass_g sex    year
##    <fct>         <int> <fct> <int>
##  1 Adelie         4675 male   2007
##  2 Adelie         4250 <NA>   2007
##  3 Adelie         4400 male   2007
##  4 Adelie         4500 male   2007
##  5 Adelie         4200 male   2007
##  6 Adelie         4150 male   2007
##  7 Adelie         4650 male   2007
##  8 Adelie         4400 male   2007
##  9 Adelie         4600 male   2007
## 10 Adelie         4150 male   2007
## # ℹ 162 more rows

# filter observations based on a qualitative variable
dat |>
  filter(sex == "female")

## # A tibble: 165 × 4
##    species body_mass_g sex     year
##    <fct>         <int> <fct>  <int>
##  1 Adelie         3800 female  2007
##  2 Adelie         3250 female  2007
##  3 Adelie         3450 female  2007
##  4 Adelie         3625 female  2007
##  5 Adelie         3200 female  2007
##  6 Adelie         3700 female  2007
##  7 Adelie         3450 female  2007
##  8 Adelie         3325 female  2007
##  9 Adelie         3400 female  2007
## 10 Adelie         3800 female  2007
## # ℹ 155 more rows

# filter observations based on 2 cumulative conditions
dat |>
  filter(body_mass_g > 4000 & sex == "female")

## # A tibble: 58 × 4
##    species body_mass_g sex     year
##    <fct>         <int> <fct>  <int>
##  1 Gentoo         4500 female  2007
##  2 Gentoo         4450 female  2007
##  3 Gentoo         4550 female  2007
##  4 Gentoo         4800 female  2007
##  5 Gentoo         4400 female  2007
##  6 Gentoo         4650 female  2007
##  7 Gentoo         4650 female  2007
##  8 Gentoo         4200 female  2007
##  9 Gentoo         4150 female  2007
## 10 Gentoo         4800 female  2007
## # ℹ 48 more rows

The pipe operator simply takes the results of one operation into the next operation below it, making the code extremely easy to write and read.

This way, instead of specifying the data frame’s name as first argument in the filter() function (or any other function within the {dplyr} package), we simply specify the data frame’s name and then the desired function, combined together thanks to the pipe operator.

Extract observations

It is possible to extract observations based on:

their positions, or
their values.

Based on their positions

Extracting observations based on their positions can be done with the slice() function:

# extract rows 2, 5 and 37
dat |>
  slice(c(2, 5, 37))

## # A tibble: 3 × 4
##   species body_mass_g sex     year
##   <fct>         <int> <fct>  <int>
## 1 Adelie         3800 female  2007
## 2 Adelie         3450 female  2007
## 3 Adelie         3950 male    2007

Moreover, extracting the first or last rows can be done with slice_head() and slice_tail():

# extract first 3 rows
dat |>
  slice_head(n = 3)

## # A tibble: 3 × 4
##   species body_mass_g sex     year
##   <fct>         <int> <fct>  <int>
## 1 Adelie         3750 male    2007
## 2 Adelie         3800 female  2007
## 3 Adelie         3250 female  2007

# extract last 3 rows
dat |>
  slice_tail(n = 3)

## # A tibble: 3 × 4
##   species   body_mass_g sex     year
##   <fct>           <int> <fct>  <int>
## 1 Chinstrap        3775 male    2009
## 2 Chinstrap        4100 male    2009
## 3 Chinstrap        3775 female  2009

Based on their values

To extract observations based on values of a variable, use:

slice_min() to select rows with the lowest values (with a defined proportion), and
slice_max() to select rows with the highest values (with a defined proportion).

# extract observations with 25% lowest body mass
dat |>
  slice_min(body_mass_g, prop = 0.25)

## # A tibble: 89 × 4
##    species   body_mass_g sex     year
##    <fct>           <int> <fct>  <int>
##  1 Chinstrap        2700 female  2008
##  2 Adelie           2850 female  2008
##  3 Adelie           2850 female  2008
##  4 Adelie           2900 female  2008
##  5 Adelie           2900 female  2008
##  6 Adelie           2900 female  2009
##  7 Chinstrap        2900 female  2007
##  8 Adelie           2925 female  2009
##  9 Adelie           2975 <NA>    2007
## 10 Adelie           3000 female  2007
## # ℹ 79 more rows

# extract observations with 25% highest body mass
dat |>
  slice_max(body_mass_g, prop = 0.25)

## # A tibble: 90 × 4
##    species body_mass_g sex    year
##    <fct>         <int> <fct> <int>
##  1 Gentoo         6300 male   2007
##  2 Gentoo         6050 male   2007
##  3 Gentoo         6000 male   2008
##  4 Gentoo         6000 male   2009
##  5 Gentoo         5950 male   2008
##  6 Gentoo         5950 male   2009
##  7 Gentoo         5850 male   2007
##  8 Gentoo         5850 male   2007
##  9 Gentoo         5850 male   2009
## 10 Gentoo         5800 male   2008
## # ℹ 80 more rows

Sample observations

Sampling observations can be done in two ways:

Random sample of a number of rows with sample_n()
Random sample of a fraction of rows with sample_frac()

# random sample of 3 rows
dat |>
  sample_n(size = 3)

## # A tibble: 3 × 4
##   species   body_mass_g sex     year
##   <fct>           <int> <fct>  <int>
## 1 Adelie           3450 female  2007
## 2 Chinstrap        3675 female  2009
## 3 Gentoo           4500 female  2007

# random sample of half of the rows
dat |>
  sample_frac(size = 1 / 2)

## # A tibble: 172 × 4
##    species   body_mass_g sex     year
##    <fct>           <int> <fct>  <int>
##  1 Adelie           4150 male    2008
##  2 Gentoo           5800 male    2008
##  3 Adelie           3650 male    2009
##  4 Adelie           3500 male    2009
##  5 Adelie           3450 female  2007
##  6 Adelie           4300 male    2009
##  7 Chinstrap        3400 female  2008
##  8 Adelie           3950 male    2007
##  9 Chinstrap        3325 female  2009
## 10 Adelie           3950 male    2008
## # ℹ 162 more rows

Note that, as with the sample() function within base R, size can be greater than the size of the data frame. In this case, some rows will be duplicated, and you will need to specify the argument replace = TRUE.

Alternatively, it is possible to obtain a random sample of a number of rows or fraction or rows with slice_sample(). For this, use:

the argument n to select a number of rows, or
the argument prop to select a fraction of rows.

# random sample of 3 rows
dat |>
  slice_sample(n = 3)

## # A tibble: 3 × 4
##   species body_mass_g sex     year
##   <fct>         <int> <fct>  <int>
## 1 Adelie         3200 female  2007
## 2 Adelie         3800 female  2007
## 3 Gentoo         4800 female  2007

# random sample of half of the rows
dat |>
  slice_sample(prop = 1 / 2)

## # A tibble: 172 × 4
##    species body_mass_g sex     year
##    <fct>         <int> <fct>  <int>
##  1 Adelie         3900 male    2009
##  2 Adelie         3275 female  2009
##  3 Gentoo         5050 male    2008
##  4 Gentoo         4700 female  2009
##  5 Gentoo         4600 female  2008
##  6 Gentoo         4875 <NA>    2009
##  7 Adelie         3700 <NA>    2007
##  8 Gentoo         3950 female  2008
##  9 Gentoo         4550 female  2007
## 10 Adelie         3500 female  2008
## # ℹ 162 more rows

Sort observations

Sorting observations can be done with the arrange() function:

# sort observations based on body mass (ascending order)
dat |>
  arrange(body_mass_g)

## # A tibble: 344 × 4
##    species   body_mass_g sex     year
##    <fct>           <int> <fct>  <int>
##  1 Chinstrap        2700 female  2008
##  2 Adelie           2850 female  2008
##  3 Adelie           2850 female  2008
##  4 Adelie           2900 female  2008
##  5 Adelie           2900 female  2008
##  6 Adelie           2900 female  2009
##  7 Chinstrap        2900 female  2007
##  8 Adelie           2925 female  2009
##  9 Adelie           2975 <NA>    2007
## 10 Adelie           3000 female  2007
## # ℹ 334 more rows

By default, arrange() uses the ascending order. To sort in descending order, use desc() inside arrange():

# sort observations based on body mass (descending order)
dat |>
  arrange(desc(body_mass_g))

## # A tibble: 344 × 4
##    species body_mass_g sex    year
##    <fct>         <int> <fct> <int>
##  1 Gentoo         6300 male   2007
##  2 Gentoo         6050 male   2007
##  3 Gentoo         6000 male   2008
##  4 Gentoo         6000 male   2009
##  5 Gentoo         5950 male   2008
##  6 Gentoo         5950 male   2009
##  7 Gentoo         5850 male   2007
##  8 Gentoo         5850 male   2007
##  9 Gentoo         5850 male   2009
## 10 Gentoo         5800 male   2008
## # ℹ 334 more rows

As with filter(), arrange() can be used for several variables and works both on quantitative and qualitative variables:

# sort observations based on two variables
dat |>
  arrange(sex, body_mass_g)

## # A tibble: 344 × 4
##    species   body_mass_g sex     year
##    <fct>           <int> <fct>  <int>
##  1 Chinstrap        2700 female  2008
##  2 Adelie           2850 female  2008
##  3 Adelie           2850 female  2008
##  4 Adelie           2900 female  2008
##  5 Adelie           2900 female  2008
##  6 Adelie           2900 female  2009
##  7 Chinstrap        2900 female  2007
##  8 Adelie           2925 female  2009
##  9 Adelie           3000 female  2007
## 10 Adelie           3000 female  2009
## # ℹ 334 more rows

The code above sorts the observations first based on the sex (in alphabetical order) and then based on the body mass (in ascending order, so from lowest to highest).

Note that if the qualitative variable is defined as an ordered factor, the sorting is based on level order, not alphabetical order!

Select variables

Selecting variables can be done with the select() function, based on:

the position of the variable(s), or
the name(s) of the variable(s).

# select variables by their positions
dat |>
  select(c(2, 4))

## # A tibble: 344 × 2
##    body_mass_g  year
##          <int> <int>
##  1        3750  2007
##  2        3800  2007
##  3        3250  2007
##  4          NA  2007
##  5        3450  2007
##  6        3650  2007
##  7        3625  2007
##  8        4675  2007
##  9        3475  2007
## 10        4250  2007
## # ℹ 334 more rows

# select variables by their names
dat |>
  select(body_mass_g, year)

## # A tibble: 344 × 2
##    body_mass_g  year
##          <int> <int>
##  1        3750  2007
##  2        3800  2007
##  3        3250  2007
##  4          NA  2007
##  5        3450  2007
##  6        3650  2007
##  7        3625  2007
##  8        4675  2007
##  9        3475  2007
## 10        4250  2007
## # ℹ 334 more rows

Note that it is also possible to remove variables. For this, use the - sign in front of their positions or names:

# remove variables by their positions
dat |>
  select(-c(2, 4))

## # A tibble: 344 × 2
##    species sex   
##    <fct>   <fct> 
##  1 Adelie  male  
##  2 Adelie  female
##  3 Adelie  female
##  4 Adelie  <NA>  
##  5 Adelie  female
##  6 Adelie  male  
##  7 Adelie  female
##  8 Adelie  male  
##  9 Adelie  <NA>  
## 10 Adelie  <NA>  
## # ℹ 334 more rows

# remove variables by their names
dat |>
  select(-c(body_mass_g, year))

## # A tibble: 344 × 2
##    species sex   
##    <fct>   <fct> 
##  1 Adelie  male  
##  2 Adelie  female
##  3 Adelie  female
##  4 Adelie  <NA>  
##  5 Adelie  female
##  6 Adelie  male  
##  7 Adelie  female
##  8 Adelie  male  
##  9 Adelie  <NA>  
## 10 Adelie  <NA>  
## # ℹ 334 more rows

It is also possible to select variables with a sequence of names:

# select all variables from species to sex
dat |>
  select(species:sex)

## # A tibble: 344 × 3
##    species body_mass_g sex   
##    <fct>         <int> <fct> 
##  1 Adelie         3750 male  
##  2 Adelie         3800 female
##  3 Adelie         3250 female
##  4 Adelie           NA <NA>  
##  5 Adelie         3450 female
##  6 Adelie         3650 male  
##  7 Adelie         3625 female
##  8 Adelie         4675 male  
##  9 Adelie         3475 <NA>  
## 10 Adelie         4250 <NA>  
## # ℹ 334 more rows

Last but not least, select() can also be used as an easy way to rearrange columns in the desired order:

# put sex as first column, then all the others
dat |>
  select(sex, species:year)

## # A tibble: 344 × 4
##    sex    species body_mass_g  year
##    <fct>  <fct>         <int> <int>
##  1 male   Adelie         3750  2007
##  2 female Adelie         3800  2007
##  3 female Adelie         3250  2007
##  4 <NA>   Adelie           NA  2007
##  5 female Adelie         3450  2007
##  6 male   Adelie         3650  2007
##  7 female Adelie         3625  2007
##  8 male   Adelie         4675  2007
##  9 <NA>   Adelie         3475  2007
## 10 <NA>   Adelie         4250  2007
## # ℹ 334 more rows

Rename variables

To rename variables, use the rename() function:

# rename variables
dat |>
  rename(
    body_mass = body_mass_g, # rename body_mass_g into body_mass
    study_year = year # rename year into study_year
  )

## # A tibble: 344 × 4
##    species body_mass sex    study_year
##    <fct>       <int> <fct>       <int>
##  1 Adelie       3750 male         2007
##  2 Adelie       3800 female       2007
##  3 Adelie       3250 female       2007
##  4 Adelie         NA <NA>         2007
##  5 Adelie       3450 female       2007
##  6 Adelie       3650 male         2007
##  7 Adelie       3625 female       2007
##  8 Adelie       4675 male         2007
##  9 Adelie       3475 <NA>         2007
## 10 Adelie       4250 <NA>         2007
## # ℹ 334 more rows

This might not be intuitive (at least it was not for me at the time of learning this package), so bear in mind that you always need to write first the new name and then the old name (separated with the = sign).

Create or modify variables

You can create or modify certain variables of the data frame with mutate(), based on:

another variable, or
a vector of your choice.

# create a new variable based on an existing one
dat |>
  mutate(
    body_mass_kg = body_mass_g / 1000
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_kg
##    <fct>         <int> <fct>  <int>        <dbl>
##  1 Adelie         3750 male    2007         3.75
##  2 Adelie         3800 female  2007         3.8 
##  3 Adelie         3250 female  2007         3.25
##  4 Adelie           NA <NA>    2007        NA   
##  5 Adelie         3450 female  2007         3.45
##  6 Adelie         3650 male    2007         3.65
##  7 Adelie         3625 female  2007         3.62
##  8 Adelie         4675 male    2007         4.68
##  9 Adelie         3475 <NA>    2007         3.48
## 10 Adelie         4250 <NA>    2007         4.25
## # ℹ 334 more rows

# create a new variable from a vector of your choice
dat |>
  mutate(
    ID = 1:nrow(dat)
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year    ID
##    <fct>         <int> <fct>  <int> <int>
##  1 Adelie         3750 male    2007     1
##  2 Adelie         3800 female  2007     2
##  3 Adelie         3250 female  2007     3
##  4 Adelie           NA <NA>    2007     4
##  5 Adelie         3450 female  2007     5
##  6 Adelie         3650 male    2007     6
##  7 Adelie         3625 female  2007     7
##  8 Adelie         4675 male    2007     8
##  9 Adelie         3475 <NA>    2007     9
## 10 Adelie         4250 <NA>    2007    10
## # ℹ 334 more rows

Note that if you create a variable with a name which already exists in the data frame, the old variable will be erased and replaced by the new one.

Like rename(), mutate() requires the argument to be written as name = expression, where name is name of the column created or modified and expression is the formula for calculating the values.

Summarize observations

Often, you will want to summarize the data with some descriptive statistics. This can be done with the summarize() function, in addition to most functions used for descriptive statistics (mean(), median(), min(), max(), sd(), var(), etc.):

# compute mean and sd of body mass
dat |>
  summarize(
    body_mass_mean = mean(body_mass_g, na.rm = TRUE),
    body_mass_sd = sd(body_mass_g, na.rm = TRUE)
  )

## # A tibble: 1 × 2
##   body_mass_mean body_mass_sd
##            <dbl>        <dbl>
## 1          4202.         802.

Note that:

The na.rm = TRUE argument is used to omit missing values in the computation of the summary statistics.
summarize() and summarise() give the exact same results.

Identify distinct values

Identifying distinct values of a variable can be done with distinct():

# find the distinct species
dat |>
  distinct(species)

## # A tibble: 3 × 1
##   species  
##   <fct>    
## 1 Adelie   
## 2 Gentoo   
## 3 Chinstrap

Identifying distinct values is mostly done on qualitative or quantitative discrete variables, but it can be done on any type of variable and with several variables at the same time. If more than one variables is specified, it returns all the combinations of values of the variables.

For instance, with species and study year:

# combination of distinct species and year
dat |>
  distinct(species, year)

## # A tibble: 9 × 2
##   species    year
##   <fct>     <int>
## 1 Adelie     2007
## 2 Adelie     2008
## 3 Adelie     2009
## 4 Gentoo     2007
## 5 Gentoo     2008
## 6 Gentoo     2009
## 7 Chinstrap  2007
## 8 Chinstrap  2008
## 9 Chinstrap  2009

Connected operations

Another advantage of using the {dplyr} package is that several operations can be connected all at once, with great readability of the code. This can easily be done with the pipe operator (|> or %>%) introduced earlier.

Until now, we have always seen the same structure: we call a data frame, and then we apply an operation on that data frame. From now on, we will see how to combine more operations into one single chain of operations.

Group by

group_by() allows to modify the way the basic functions are performed. Instead of covering all the rows of the data frame, the operations will cover each of the groups of rows defined by the grouping command. In this way, aggregation operations, using summarize(), will produce statistics for each group rather than for all observations.

For example, we might be interested in computing some descriptive statistics of a quantitative variable, for each level of a qualitative variable (so by group).

In our case, suppose we would like to compute the mean and standard deviation of the body mass, but this time separately for each species:

# compute mean and sd of body mass by species
dat |>
  group_by(species) |> # group by species
  summarize(
    mean = mean(body_mass_g, na.rm = TRUE), # compute mean
    sd = sd(body_mass_g, na.rm = TRUE) # compute sd
  )

## # A tibble: 3 × 3
##   species    mean    sd
##   <fct>     <dbl> <dbl>
## 1 Adelie    3701.  459.
## 2 Chinstrap 3733.  384.
## 3 Gentoo    5076.  504.

Applied to the example above, here is how the pipe operator works:

The first operation (group_by()) groups observations by species.
Then the output of the first operation is used as the input for the second operation (summarize()): mean and standard deviation are computed on body mass.

As a result, we have the mean and standard deviation of body mass by group. As you can imagine, all previous operations can be connected to match your needs.

Also note that group_by() can be used for several grouping variables at the same time:

# compute mean and sd of body mass by species and sex
dat |>
  group_by(species, sex) |> # group by species and sex
  summarize(
    mean = mean(body_mass_g, na.rm = TRUE), # compute mean
    sd = sd(body_mass_g, na.rm = TRUE) # compute sd
  )

## # A tibble: 8 × 4
## # Groups:   species [3]
##   species   sex     mean    sd
##   <fct>     <fct>  <dbl> <dbl>
## 1 Adelie    female 3369.  269.
## 2 Adelie    male   4043.  347.
## 3 Adelie    <NA>   3540   477.
## 4 Chinstrap female 3527.  285.
## 5 Chinstrap male   3939.  362.
## 6 Gentoo    female 4680.  282.
## 7 Gentoo    male   5485.  313.
## 8 Gentoo    <NA>   4588.  338.

(Note that, as for all connected operations, the name of the data frame needs to be specified only in the first operation.)

Number of observations

Some operations can only be performed inside other operations.

This is the case with the number of observations n(), which can only be used inside summarize():

# number of observations
dat |>
  summarize(n_obs = n())

## # A tibble: 1 × 1
##   n_obs
##   <int>
## 1   344

To refer with the previous connected operation group_by(), we can compute the number of observations by group using the two operations n() and group_by() separated by the pipe operator:

# number of observations by species
dat |>
  group_by(species) |>
  summarize(n_obs = n())

## # A tibble: 3 × 2
##   species   n_obs
##   <fct>     <int>
## 1 Adelie      152
## 2 Chinstrap    68
## 3 Gentoo      124

Note that n() accepts no parameters and is therefore always written with empty parentheses.

Moreover, note also that the count() function is equivalent to summarize(n = n()):

# number of observations
dat |>
  count()

## # A tibble: 1 × 1
##       n
##   <int>
## 1   344

# number of observations by species
dat |>
  count(species)

## # A tibble: 3 × 2
##   species       n
##   <fct>     <int>
## 1 Adelie      152
## 2 Chinstrap    68
## 3 Gentoo      124

Number of distinct values

n_distinct(), which can also be used only inside summarize(), computes the number of different values/levels of a variable or combination of variables:

# number of distinct species
dat |>
  summarize(n_species = n_distinct(species))

## # A tibble: 1 × 1
##   n_species
##       <int>
## 1         3

# number of distinct species and year of study
dat |>
  summarize(n_species_year = n_distinct(species, year))

## # A tibble: 1 × 1
##   n_species_year
##            <int>
## 1              9

Note that you do not have to specify a name for the output. In that case, the name of the operation will be used. For example:

# number of distinct species
dat |>
  summarize(n_distinct(species))

## # A tibble: 1 × 1
##   `n_distinct(species)`
##                   <int>
## 1                     3

First, last or nth value

Also only available inside summarize(), the first, last or nth value can be found with the following commands:

# first value of the variable sex
dat |>
  summarize(first(sex))

## # A tibble: 1 × 1
##   `first(sex)`
##   <fct>       
## 1 male

# last value of the variable sex
dat |>
  summarize(last(sex))

## # A tibble: 1 × 1
##   `last(sex)`
##   <fct>      
## 1 female

# 37th value of the variable sex
dat |>
  summarize(nth(sex, n = 37))

## # A tibble: 1 × 1
##   `nth(sex, n = 37)`
##   <fct>             
## 1 male

Several interesting arguments exist within this function to deal with missing values. For the interested reader, see more information in the documentation of the function (run ?nth()).

If else

A very common data transformation is the well known “if else” technique. This technique is usually used to create, from an existing variable, another variable which can take two levels.

Suppose that we want to create a new variable called body_mass_cat, which takes the value “High” when body_mass_g is equal or greater than a certain threshold, “Low” otherwise. This transformation can be performed with the combination of mutate() and if_else():

# if else
dat |>
  mutate(
    body_mass_cat = if_else(body_mass_g >= 4000, # condition
      "High", # output if condition is true
      "Low" # output if condition is false
    )
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_cat
##    <fct>         <int> <fct>  <int> <chr>        
##  1 Adelie         3750 male    2007 Low          
##  2 Adelie         3800 female  2007 Low          
##  3 Adelie         3250 female  2007 Low          
##  4 Adelie           NA <NA>    2007 <NA>         
##  5 Adelie         3450 female  2007 Low          
##  6 Adelie         3650 male    2007 Low          
##  7 Adelie         3625 female  2007 Low          
##  8 Adelie         4675 male    2007 High         
##  9 Adelie         3475 <NA>    2007 Low          
## 10 Adelie         4250 <NA>    2007 High         
## # ℹ 334 more rows

The if_else() function works with 3 arguments:

The condition (in our case: body_mass_g >= 4000)
The output value when the condition is true (High in our case).
The output value when the conditions is false (Low in our case).

As you can see from the table above, when body mass is missing, if_else() also returns a missing value, which is often a good thing to prevent observations being classified erroneously.

Case when

If you want to categorize a variable into more than two levels, an if else is not the most appropriate tool. In these cases, a “case when” is more appropriate.

For your information, when I learned R, I used to write nested if else functions, that is, a secondary if else inside a primary if else. Most of the time it worked (with very often a waste of time trying to debug my code), but it is very easy to make a mistake. And even if you managed to make it work, the code is not easy to read at all!

So I highly recommend using this case when technique instead of several if else functions nested within each other.

Suppose we want to classify body mass into 3 categories: low, medium and high. For this illustration, we arbitrarily decide that body mass is low when it is strictly lower than 3500, high when it is strictly higher than 4750 and medium otherwise.

With nested if else functions, here is the code we would need to write:

# nested if else
dat |>
  mutate(
    body_mass_cat = if_else(body_mass_g < 3500, # first condition
      "Low", # output if first condition is true
      if_else(body_mass_g > 4750, # second condition when first condition is false
        "High", # output when second condition is true
        "Medium" # output when second condition is false
      )
    )
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_cat
##    <fct>         <int> <fct>  <int> <chr>        
##  1 Adelie         3750 male    2007 Medium       
##  2 Adelie         3800 female  2007 Medium       
##  3 Adelie         3250 female  2007 Low          
##  4 Adelie           NA <NA>    2007 <NA>         
##  5 Adelie         3450 female  2007 Low          
##  6 Adelie         3650 male    2007 Medium       
##  7 Adelie         3625 female  2007 Medium       
##  8 Adelie         4675 male    2007 Medium       
##  9 Adelie         3475 <NA>    2007 Low          
## 10 Adelie         4250 <NA>    2007 Medium       
## # ℹ 334 more rows

This code works as follows:

It evaluates the first condition body_mass_g < 3500.
If it is true, body_mass_cat is Low. On the contrary, if it is false, it evaluates the second condition body_mass_g > 4750.
If this second condition is true, body_mass_cat is High, otherwise it is Medium.

As you can see from the results above, it works. However, you will concede that it is easy to make coding mistakes, and that the code is not easy to write nor to read.

To improve this workflow, we now use the case when technique:

# case when, without a default option
dat |>
  mutate(
    body_mass_cat = case_when(
      body_mass_g < 3500 ~ "Low",
      body_mass_g >= 3500 & body_mass_g <= 4750 ~ "Medium",
      body_mass_g > 4750 ~ "High"
    )
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_cat
##    <fct>         <int> <fct>  <int> <chr>        
##  1 Adelie         3750 male    2007 Medium       
##  2 Adelie         3800 female  2007 Medium       
##  3 Adelie         3250 female  2007 Low          
##  4 Adelie           NA <NA>    2007 <NA>         
##  5 Adelie         3450 female  2007 Low          
##  6 Adelie         3650 male    2007 Medium       
##  7 Adelie         3625 female  2007 Medium       
##  8 Adelie         4675 male    2007 Medium       
##  9 Adelie         3475 <NA>    2007 Low          
## 10 Adelie         4250 <NA>    2007 Medium       
## # ℹ 334 more rows

This workflow is much simpler to code and read!

If there are no missing values in the variable(s) used for the condition(s), it can even be simplified to:

# case when, with a default option
dat |>
  mutate(
    body_mass_cat = case_when(
      body_mass_g < 3500 ~ "Low",
      body_mass_g > 4750 ~ "High",
      .default = "Medium" # default output
    )
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_cat
##    <fct>         <int> <fct>  <int> <chr>        
##  1 Adelie         3750 male    2007 Medium       
##  2 Adelie         3800 female  2007 Medium       
##  3 Adelie         3250 female  2007 Low          
##  4 Adelie           NA <NA>    2007 Medium       
##  5 Adelie         3450 female  2007 Low          
##  6 Adelie         3650 male    2007 Medium       
##  7 Adelie         3625 female  2007 Medium       
##  8 Adelie         4675 male    2007 Medium       
##  9 Adelie         3475 <NA>    2007 Low          
## 10 Adelie         4250 <NA>    2007 Medium       
## # ℹ 334 more rows

As you can see, a default output can be specified with .default for observations that do not match any of the conditions.

However, be careful if there are missing values! Indeed, if there is at least one missing value (as in our case), the code above is not correct because observations with missing body_mass_g will be misclassified as Medium.

Therefore, if you really want to specify a default output, I recommend using the code below which keeps missing values as NA:

# case when, with a default option and missing values
dat |>
  mutate(
    body_mass_cat = case_when(
      body_mass_g < 3500 ~ "Low",
      body_mass_g > 4750 ~ "High",
      is.na(body_mass_g) ~ NA, # keep missing values as NA
      .default = "Medium" # default output
    )
  )

## # A tibble: 344 × 5
##    species body_mass_g sex     year body_mass_cat
##    <fct>         <int> <fct>  <int> <chr>        
##  1 Adelie         3750 male    2007 Medium       
##  2 Adelie         3800 female  2007 Medium       
##  3 Adelie         3250 female  2007 Low          
##  4 Adelie           NA <NA>    2007 <NA>         
##  5 Adelie         3450 female  2007 Low          
##  6 Adelie         3650 male    2007 Medium       
##  7 Adelie         3625 female  2007 Medium       
##  8 Adelie         4675 male    2007 Medium       
##  9 Adelie         3475 <NA>    2007 Low          
## 10 Adelie         4250 <NA>    2007 Medium       
## # ℹ 334 more rows

I personally prefer to write all categories and not write a default option for improved code readability and robustness of my code, but it is more a personal opinion.

In all cases, no matter if you used an if else or a case when, it is a good practice to check the variable you just created to make sure that you obtain the intended results.

Conclusion and other resources

Thanks for reading.

This article introduced and illustrated the most common data manipulation and data management tools in R, using the {dplyr} package. We also introduced the pipe operator, well known to users of modern R packages.

The {dplyr} package offers many more functions for data manipulation (in particular to merge data frames with the family of join() functions). If you would like to learn more about this package, I recommend starting with the following resources:

dplyr.tidyverse.org
Chapter “Data transformation” in the book “R for Data Science”
Cheatsheet
Vignette
For those who are used to base R, a vignette comparing {dplyr} functions to their base R equivalents

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Horst, Allison Marie, Alison Presmanes Hill, and Kristen B Gorman. 2020. Palmerpenguins: Palmer Archipelago (Antarctica) Penguin Data. https://doi.org/10.5281/zenodo.3960218.

The keyboard shortcut for the pipe operator is ctrl + shift + m (Windows) or cmd + shift + m (Mac). It will print %>%, unless you specified to use the native pipe operator |> in the settings of RStudio.↩︎

Pearson, Spearman and Kendall correlation coefficients by hand

Tue, 05 Sep 2023 00:00:00 +0000

Introduction

In statistics, a correlation is used to evaluate the relationship between two variables.

In a previous post, we showed how to compute a correlation and perform a correlation test in R. In this post, we illustrate how to compute the Pearson, Spearman and Kendall correlation coefficients by hand and under two different scenarios (i.e., with and without ties).

Data

To illustrate the methods with and without ties, we consider two different datasets, one with ties and another without ties.

With ties

For the illustrations of the scenarios with ties, suppose we have the following sample of size 5:

As we can see, there are some ties since there are two identical observations in the variable $x$.

Without ties

For the scenarios which require no ties, we will consider the following sample of size 3:

Correlation coefficients by hand

The three most common correlation methods are:¹

Pearson, used for two quantitative continuous variables which have a linear relationship
Spearman, used for two quantitative variables if the link is partially linear, or for one qualitative ordinal variable and one quantitative variable
Kendall, often used for two qualitative ordinal variables

Each method is presented in the next sections.

Note that the aim of this post is to illustrate how to compute the three correlation coefficients by hand and under two different scenarios; we are not interested in verifying the underlying assumptions.

Pearson

Luckily, the procedure for computing the Pearson correlation coefficient is the same whether there are ties or not so we do not distinguish the two scenarios.

With and without ties

The Pearson correlation coefficient, denoted $r$ in the case of a sample, can be computed as follows²

\[ r = \frac{\sum^n_{i = 1} x_i y_i - n \bar{x} \bar{y}}{(n - 1) s_x s_y} \]

where:

$n$ is the sample size
$x_i$ and $y_i$ are the observations
$\bar{x}$ and $\bar{y}$ are the sample means of $x$ and $y$
$s_x$ and $s_y$ are the sample standard deviations of $x$ and $y$

We show how to compute it step by step.

Step 1.

As you can see, before computing the correlation coefficient we first need to compute the mean and the standard deviation for each of the two variables.

The mean of $x$ is computed as follows:

\[\bar{x} = \frac{1}{n}\sum^n_{i = 1} x_i\]

The standard deviation of $x$ is computed as follows:

\[s_x = \sqrt{\frac{1}{n - 1} \sum^n_{i = 1}(x_i - \bar{x})^2}\]

The formulas can be used analogously for the variable $y$.

We start by computing the means of the two variables:

$\bar{x} = \frac{-1 + 3 + 5 + 5 + 2}{5} = 2.8$
$\bar{y} = \frac{-3 + 1 + 0 + 2 - 1}{5} = -0.2$

Step 2.

We now need to compute the standard deviation of each variable. To ease the computations, it is best to use a table, starting with the observations:

To compute the standard deviations, we need the sum of the squared differences between each observation and its mean, that is, $\sum^n_{i = 1}(x_i - \bar{x})^2$ and $\sum^n_{i = 1}(y_i - \bar{y})^2$. We start by creating two new columns in the table, denoted x-xbar and y-ybar, corresponding to $(x_i - \bar{x})$ and $(y_i - \bar{y})$:

We take the square of these two new columns to have $(x_i - \bar{x})^2$ and $(y_i - \bar{y})^2$, denoted (x-xbar)^2 and (y-ybar)^2:

We then sum these two columns, which gives:

for $x$: $\sum^n_{i = 1}(x_i - \bar{x})^2 =$ 24.8
for $y$: $\sum^n_{i = 1}(y_i - \bar{y})^2 =$ 14.8

The standard deviations are thus:

$s_x = \sqrt{\frac{24.8}{5-1}} = 2.49$
$s_y = \sqrt{\frac{14.8}{5-1}} = 1.92$

Step 3.

We now need $\sum^n_{i = 1} x_i y_i$, so we add a new column in the table, corresponding to $x_iy_i$ and denoted x*y:

Step 4.

We take the sum of that last column, which gives $\sum^n_{i = 1} x_i y_i =$ 14.

Step 5.

Finally, the Pearson correlation coefficient can be computed by plugging values found above in the initial formula:

\[ \begin{split} r &= \frac{\sum^n_{i = 1} x_i y_i - n \bar{x} \bar{y}}{(n - 1) s_x s_y} \\ &= \frac{14 - (5\times2.8\times-0.2)}{(5-1) \times 2.49\times1.92} \\ &= 0.88 \end{split} \]

Note that there are other formulas to compute the Pearson correlation coefficient. For instance,

\[ \begin{split} r &= \frac{1}{n - 1} \sum^n_{i = 1} \left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{x}}{s_y}\right) \\ &= \frac{\sum^n_{i = 1}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum^n_{i = 1}(x_i - \bar{x})^2} \sqrt{\sum^n_{i = 1}(y_i - \bar{y})^2}} \end{split} \]

All formulas will of course give the exact same results.

For your information, squaring the Pearson correlation coefficient $r$ gives the coefficient of determination $R^2$ in the context of a simple linear regression.

Spearman

We now present the Spearman correlation coefficient, also referred as Spearman’s rank correlation coefficient. This coefficient is actually the same than Pearson coefficient, except that the computations are based on the ranked values rather than on the raw observations.

Again, we present how to compute it by hand step by step, but this time we distinguish two scenarios:

if there are ties
if there are no ties

With ties

The Spearman correlation coefficient (with ties), denoted $r_s$, is defined as follows

\[ r_s = \frac{\sum^n_{i = 1} Rx_i Ry_i - n \overline{Rx} \overline{Ry}}{(n - 1) s_{Rx} s_{Ry}} \]

where:

$n$ is the sample size
$Rx_i$ and $Ry_i$ are the ranks for the two variables
$\overline{Rx}$ and $\overline{Ry}$ are the sample means of the ranks for the two variables
$s_{Rx}$ and $s_{Ry}$ are the sample standard deviations of the ranks for the two variables

Here is how to compute it by hand.

Step 1.

As mentioned before, the Spearman coefficient is based on the ranks. So we first need to add the ranks of the observations (from lowest to highest), separately for each of the two variables.

For $y$, we see that:

-3 is the smallest value, so we assign it the rank 1
-1 is the second smallest value, so we assign it the rank 2
then comes 0, so we assign it the rank 3
then comes 1, so we assign it the rank 4
finally, 2 is the largest value, so we assign it the rank 5

The same goes for $x$, except that here we have two observations with the value 5 (so there will be ties in the ranks). In this case, we take the mean rank:

-1 is the smallest value, so we assign it the rank 1
2 is the second smallest value, so we assign it the rank 2
then comes 3, so we assign it the rank 3
finally, the two largest values belongs to rank 4 and 5, so we assign both of them the rank 4.5

We include the ranks in the table, denoted Rx and Ry:

Step 2.

From there, it is similar than the Pearson coefficient except that we work on the ranks and not on the initial observations anymore. To avoid any confusion in the remaining steps, we remove the initial observations from the table and we keep only the ranks:

We start with the means of the ranks:

$\overline{Rx} = \frac{1+3+4.5+4.5+2}{5} = 3$
$\overline{Ry} = \frac{1+4+3+5+2}{5} = 3$

For the standard deviations, we use the table as we did for the Pearson coefficient:

We sum the last two columns, which gives:

for $x$: $\sum^n_{i = 1}(Rx_i - \overline{Rx})^2 =$ 9.5
for $y$: $\sum^n_{i = 1}(Ry_i - \overline{Ry})^2 =$ 10

The standard deviations are thus:

$s_{Rx} = \sqrt{\frac{9.5}{5-1}} = 1.54$
$s_{Ry} = \sqrt{\frac{10}{5-1}} = 1.58$

Step 3.

We now need $\sum^n_{i = 1} Rx_i Ry_i$, so we add a new column in the table, corresponding to $Rx_iRy_i$ and denoted Rx*Ry:

Step 4.

We take the sum of that last column, which gives $\sum^n_{i = 1} Rx_i Ry_i =$ 53.

Step 5.

Finally, the Spearman correlation coefficient can be computed by plugging all values in the initial formula:

\[ \begin{split} r_s &= \frac{\sum^n_{i = 1} Rx_i Ry_i - n \overline{Rx} \overline{Ry}}{(n - 1) s_{Rx} s_{Ry}} \\ &= \frac{53 - (5\times3\times3)}{(5-1) \times 1.54\times1.58} \\ &= 0.82 \end{split} \]

Without ties

When all initial values are different within each variable, meaning that all ranks are distinct integers, there are no ties. In that specific case, the Spearman coefficient can be computed with the following shortened formula:

\[ r_s = 1 - \frac{6 \sum^n_{i = 1}(Rx_i - Ry_i)^2}{n (n^2 - 1)} \]

For example, suppose the sample without ties introduced at the beginning of the post:

Step 1.

All observations within each variable are different, so all ranks are distinct integers and there are no ties:

Step 2.

We only need to compute the difference between the two ranks of each row, denoted Rx-Ry:

Take the square of these differences, denoted (Rx-Ry)^2:

And then take the sum of that last column, which gives $\sum^n_{i = 1}(Rx_i - Ry_i)^2 =$ 2.

Step 3.

Finally, we can fill in the initial formula to find the Spearman coefficient:

\[ \begin{split} r_s &= 1 - \frac{6 \sum^n_{i = 1}(Rx_i - Ry_i)^2}{n (n^2 - 1)} \\ &= 1 - \frac{6 \times2}{3 (3^2 - 1)} \\ &= 1 - \frac{12}{24} \\ &= 0.5 \end{split} \]

Kendall

Kendall coefficient correlation, also known as Kendall’s $\tau$ coefficient, is similar than Spearman coefficient, except that it is often preferred for small samples and when many rank ties.

Here also we distinguish between two scenarios:

if there are no ties
if there are ties

Unlike Spearman coefficient, we first illustrate the scenario when there are no ties, and then when there are ties.

Without ties

When there are no ties, the Kendall coefficient, denoted $\tau_a$, is defined as follows

\[ \tau_a = \frac{C - D}{C + D} \]

where:

$C$ is the number of concordant pairs
$D$ is the number of discordant pairs

This coefficient, also referred as Kendall tau-a, does not make any adjustment for ties.

Let’s see what are concordant and discordant pairs using the data without ties (the same data than for Spearman correlation without ties):

Step 1.

We start by computing the ranks for each variable (Rx and Ry), as we did for the Spearman coefficient:

Step 2.

We then arbitrarily choose a reference variable among the two, Rx or Ry. Suppose we take Rx as the reference variable here.

We sort the dataset by this reference variable:

From now, we only look at the ranks of the second variable (the one which is not the reference level, here Ry), so to avoid any confusion in the remaining steps we keep only the Ry column:

Step 3.

We now take each row of Ry one by one and check whether the rows below it in the table are smaller or larger.

In our table, the first row of Ry is 1. We see that the value just below it is 3, which is larger than 1. Since 3 is larger than 1, this is called a concordant pair. We write it in the table:

The next row, 2, is also larger than 1, so it is also a concordant pair. We also write it in the table:

We start again with the second row of Ry, which is 3. Again, we look at the row below it and check whether it is larger or smaller. Here, the row below it is 2, which is smaller than 3, so we have a discordant pair. We add this information into the table:

Step 4.

We now compute the total number of concordant and discordant pairs. There are:

2 concordant pairs so $C = 2$, and
1 discordant pair so $D = 1$.

Step 5.

Finally, we plug the values we have just found in the initial formula:

\[ \begin{split} \tau_a &= \frac{C - D}{C + D}\\ &= \frac{2 - 1}{2 + 1}\\ &= \frac{1}{3}\\ &= 0.33 \end{split} \]

Alternatively, we can also use the following formulas

\[ \begin{split} \tau_a &= 1 - \frac{4D}{n(n - 1)}\\ &= \frac{4C}{n(n - 1)} - 1 \end{split} \]

where $n$ is still the sample size, and which both give the exact same results.

With ties

I must admit that the process with ties is slightly more complex than without ties.

The Kendall tau-b coefficient, which makes adjustments for ties, is defined as follows

\[ \tau_b = \frac{C - D}{\sqrt{(C^2_n - n_x)(C^2_n - n_y)}} \]

where:

$C$ and $D$ are still the number of concordant and discordant pairs, respectively
$C^2_n$ is the total number of possible pairs
$n_x$ is the number of possible pairs with a tie among the $x$ values
$n_y$ is the number of possible pairs with a tie among the $y$ values

Note that here, the letter $C$ in $C^2_n$ denotes “combination” and not “concordant”.

Let’s illustrate that scenario and the formula with the dataset used for the Pearson correlation and the Spearman correlation with ties, that is:

Step 1.

Similarly to with no ties, the process with ties is based on the ranks so we start by adding the ranks for each variable, denoted as usual as Rx and Ry:

Step 2.

One of the two variables ($x$ or $y$) must be selected as the reference variable. This time, we do not choose it arbitrarily, but we choose the one which does not have any ties. In our case, variable $x$ has ties, while variable $y$ does not have any ties. So $y$ will be our reference variable.

We then:

order the dataset by the reference variable, here Ry, and
we keep only the necessary columns to avoid any confusion in the remaining steps:

Step 3.

We check whether it is a concordant or discordant pair the same way we did without ties, but this time we do not count ties.

The first row in column Rx is 1. We compare all rows below it in the table with that value. All rows below 1 in the table are larger, so we write that they are concordant pairs:

We repeat the same process for each row.

For instance, for the second row of the column Rx, we have the value 2. Again, all rows below in the table are larger so we write that they are concordant pairs:

For the third row of the column Rx, we have the value 4.5. The row just below in the table (= 3) is smaller than 4, so we write D for discordant pair:

Now, as you can see, the last row is also 4.5. Therefore, since it is equal to the value we are comparing to, it is neither a concordant nor a discordant pair, so we write “T” in the table for “ties”:

Last, the fourth row in the column Rx is 3, which we compare to 4.5 to see that 4.5 is larger so it is a concordant pair:

Step 4.

We then sum the number of concordant and discordant pairs:

$C = 8$
$D = 1$

Step 5.

We now have all the information required to compute the numerator of $\tau_b$, but we still need to find $C^2_n$, $n_x$ and $n_y$ to compute the denominator.

As mentioned above, $C^2_n$ is the total number of possible pairs, thus corresponding to the number of combinations of two values. This number of pairs can be found with the formula of the combination, that is, $C^2_n = \frac{n!}{2!(n - 2)!}$ where $n$ is the sample size.

In our case, we have a sample of size 5, so we have:³

\[ \begin{split} C^2_n &= \frac{n!}{2!(n - 2)!}\\ &= \frac{5!}{2!(5-2)!}\\ &= 10 \end{split} \]

Furthermore, $n_x$ and $n_y$ are the number of possible pairs with a tie among the $x$ and $y$ variables, respectively.

When looking at Ry and Rx from the table above:

We see that there are:

2 identical values for the variable $x$
0 identical value for the variable $y$

This means that the number of possible pairs with a tie is equal to:⁴

1 for the variable $x$ (the only possible pair with a tie is the pair {4.5, 4.5})
0 for the variable $y$ (since all ranks are distinct, there exists no pair with a tie)

Therefore,

\[ \begin{split} n_x &= 1\\ n_y &= 0 \end{split} \]

And we now have all the necessary information to compute $\tau_b$!

Step 6.

By plugging the values found above in the initial formula, we have

\[ \begin{split} \tau_b &= \frac{C - D}{\sqrt{(C^2_n - n_x)(C^2_n - n_y)}}\\ &= \frac{8-1}{\sqrt{(10 - 1)(10 - 0)}}\\ &= \frac{7}{\sqrt{90}}\\ &= 0.74 \end{split} \]

Verification in R

For the sake of completeness, we verify our results with the help of R for each coefficient and scenario.

Pearson:

x <- c(-1, 3, 5, 5, 2)
y <- c(-3, 1, 0, 2, -1)

cor(x, y, method = "pearson")

## [1] 0.8769051

Spearman with ties:

cor(x, y, method = "spearman")

## [1] 0.8207827

Spearman without ties:

x2 <- c(3, 5, 2)
y2 <- c(5, 1, -1)

cor(x2, y2, method = "spearman")

## [1] 0.5

Kendall without ties:

cor(x2, y2, method = "kendall")

## [1] 0.3333333

Kendall with ties:

cor(x, y, method = "kendall")

## [1] 0.7378648

We indeed find the same results by hand than with R (any discrepancies is due to rounding)!

Conclusion

Remember that a correlation coefficient (no matter whether it is Pearson, Spearman or Kendall) ranges from -1 to 1, with 0 being no correlation and the closer to 1 in absolute terms, the stronger the correlation.

Broadly speaking, a positive correlation means that high values of one variable are associated with high values of the other variable (and vice versa). A negative correlation means that high values of one variable are associated with low values of the other variable.

Last but not least, remember that the conclusions drawn from a correlation coefficient computed within a sample cannot be generalized to the population without a proper statistical test, that is, a correlation test.

Thanks for reading.

I hope this article helped you to compute the Pearson, Spearman and Kendall correlation coefficients by hand (with and without ties).

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

For your information, the Pearson correlation coefficient is considered as a parametric procedure, whereas the Spearman and Kendall correlation coefficients are considered as non-parametric procedures.↩︎
Here, we suppose that we have a sample and not a population. In the case where the observations you have represent the entire population, the correlation coefficient is denoted $\rho$ and the formula differs slightly. See a recap of the difference between a sample and a population if needed.↩︎
Remember that the factorial of $n$, denoted $n!$, is $n! = n \times (n - 1) \times \cdots \times 1$. So for instance, $5! = 5\times4\times3\times2\times1 = 120$.↩︎
If you have many identical values and want to count the number of possible pairs thanks to a formula rather than by counting them manually, you can again use the formula of the combination, that is, $C^2_n = \frac{n!}{x!(n - x)!}$ where $n$ is the number of identical values in a given group of ties. Bear in mind that the number of combinations must be summed up for all group of ties. In our case, we have only one group of ties with 2 values, so $n_x = C^2_2 = \frac{2!}{2!(2 - 2)!} = 1$.↩︎

How to: one-way ANOVA by hand

Wed, 30 Aug 2023 00:00:00 +0000

Introduction

An ANOVA is a statistical test used to compare a quantitative variable between groups, to determine if there is a statistically significant difference between several population means. In practice, it is usually used to compare three or more groups. However, in theory, it can also be done with only two groups.¹

In a previous post, we showed how to perform a one-way ANOVA in R. In this post, we illustrate how to conduct a one-way ANOVA by hand, via what is usually called an “ANOVA table”.

Data and hypotheses

To illustrate the method, suppose we take a sample of 12 students, divided equally into three classes (A, B and C) and we observe their age. Here is the sample:

We are interested in comparing the population means between classes.

Remember that the null hypothesis of the ANOVA is that all means are equal (i.e., age is not significantly different between classes), whereas the alternative hypothesis is that at least one mean is different from the other two (i.e., age is significantly different in at least one class compared to the other two). Formally, we have:

$\mu_A = \mu_B = \mu_C$
at least one mean is different

ANOVA by hand

As mentioned above, we are going to do an ANOVA table to conclude the test.

Note that the ANOVA requires some assumptions (i.e., independence, equality of variances and normality). The aim of this post is to illustrate how to do an ANOVA by hand and not how to verify these assumptions, so we suppose they are met without any verification. See how to test these assumptions in R if you are interested.

Overall and group means

We first need to compute the mean age by class (referred as the group means):

class A: $\frac{24 + 31 + 26 + 23}{4} = 26$
class B: $\frac{24 + 21 + 19 + 24}{4} = 22$
class C: $\frac{15 + 21 + 18 + 18}{4} = 18$

and the mean age for the whole sample (referred as the overall mean):

\[\begin{equation} \begin{split} & \frac{24 + 31 + 26 + 23 + 24 + 21 + 19 }{12} \\ &\frac{+ 24 + 15 + 21 + 18 + 18}{12} = 22 \end{split} \end{equation} \]

SSR and SSE

We then need to compute the sum of squares regression (SSR), and the sum of squares error (SSE).

The SSR is computed by taking the square of the difference between the mean group and the overall mean, multiplied by the number of observations in the group:

and then taking the sum of all cells:

\[64+0+64 = 128 = SSR\]

The SSE is computed by taking the square of the difference between each observation and its group mean:

and then taking the sum of all cells:

\[\begin{equation} \begin{split} & 4+25+0+9+4+1+9+4 \\ & +9+9+0+0 = 74 = SSE \end{split} \end{equation} \]

For those interested in computing the sum of square total (SST), it is simply the sum of SSR and SSE, that is,

\[\begin{equation} \begin{split} SST &= SSR + SSE\\ &= 128 + 74 \\ & =202 \end{split} \end{equation} \]

ANOVA table

The ANOVA table looks as follows (we leave it empty and we are going to fill it in step by step):

We start to build the ANOVA table by plugging the SSR and SSE values found above into the table (in the “Sum.of.Sq.” column):

The “Df” column corresponds to the degrees of freedom, and is computed as follows:

for the line regression: number of groups - 1 = 3 - 1 = 2
for the line error: number of observations - number of groups = 12 - 3 = 9

With this information, the ANOVA table becomes:

The “Mean.Sq.” column corresponds to the Mean Square, and is equal to the sum of square divided by the degrees of freedom, so the “Sum.of.Sq.” column divided by the “Df” column:

Finally, the F-value corresponds to the ratio between the two mean squares, so $\frac{64}{8.222} = 7.78$:

This F-value gives the test statistic (also referred as $F_{obs}$), which needs to be compared with the critical value found in the Fisher table to conclude the test.

We find the critical value in the Fisher table based on the degrees of freedom (those used in the ANOVA table) and based on the significance level. Suppose we take a significance level $\alpha = 0.05$, the critical value can be found in the Fisher table as follows:

So we have

\[F_{2; 9; 0.05} = 4.26\]

If you are interested to find this value with R, it can be found with the qf() function, where 0.95 corresponds to $1 - \alpha$:

qf(0.95, 2, 9)

## [1] 4.256495

Conclusion of the test

The rejection rule says that, if:

$F_{obs} > F_{2; 9; 0.05} \Rightarrow$ we reject the null hypothesis
$F_{obs} \le F_{2; 9; 0.05} \Rightarrow$ we do not reject the null hypothesis

In our case,

\[F_{obs} = 7.78 > F_{2; 9; 0.05} = 4.26\]

$\Rightarrow$ We reject the null hypothesis that all means are equal. In other words, it means that at least one class is different than the other two in terms of age.²

To verify our results, here is the ANOVA table using R:

##             Df Sum Sq Mean Sq F value Pr(>F)  
## class        2    128   64.00   7.784 0.0109 *
## Residuals    9     74    8.22                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We found the same results by hand, but note that in R, the $p$-value is computed instead of comparing the $F_{obs}$ with the critical value. The $p$-value can easily be found in R based on the $F_{obs}$ and the degrees of freedom:

pf(7.78, 2, 9,
  lower.tail = FALSE
)

## [1] 0.010916

Conclusion

Thanks for reading.

I hope this article helped you to conduct a one-way ANOVA by hand. See this tutorial if you want to learn how to do it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

In that case, a Student’s t-test is usually preferred over an ANOVA, although both tests will lead to the exact same conclusions.↩︎
Remember that an ANOVA cannot tell you which group is different than the other in terms of the quantitative dependent variable, nor whether they are all different or if only one is different. To answer this question, post-hoc tests are required. This is beyond the scope of the present post, but it can easily be done in R (see this tutorial).↩︎

Top 10 errors in R and how to fix them

Tue, 07 Feb 2023 00:00:00 +0000

Introduction

If you are just starting with R, you will often encounter errors in your code which prevent it to run. I remember when I was just starting to use R, errors in my code were so frequent that I almost gave up learning this programming language. I even recall that I went back to Excel a few times to finish my analyses because I could not find what was causing the issue.

Fortunately, I forced myself to continue despite the difficulties of the beginning. And today, even if I still encounter errors almost every time I write R code, with experience and practice, it takes less and less time to fix them. If you are also struggling at the beginning, rest assured, it is normal: everyone experiences some frustration when learning a new programming language (and this is the case not only with R).

In this post, I highlight the 10 most common errors in R and how to fix them. Of course, errors depend on your code and your analyses, so it is impossible to cover all of them (and Google does it way better than me). However, I would like to focus on some common syntax mistakes that are frequent when learning R, and which can sometimes take a long time to be fixed before realizing that the solution is right in front of our eyes.

This collection is based on my personal experience and the errors encountered by my students when I teach R. This list being non-exhaustive, feel free to comment (at the end of the post) with errors you often face when using R.

For each error, I provide examples and solutions to fix them. I also mention a couple of warnings (which are, strictly speaking, not errors) at the end of the post.

1. Unmatched parentheses, curly braces, square brackets or quotes

One rather trivial but still quite frequent error is a missing parenthesis, curly brace, square bracket or quotation mark.

This type of error is applicable to many programming languages. In R, for instance:

## missing a closing parenthesis:
mean(c(1, 7, 13)
     
## missing a curly brace before "else":
x <- 7 
if(x > 10) {
  print("x > 10")
 else {
  print("x <= 10")
 }
  
## missing a square bracket:
summary(ggplot2::diamonds[, "price")

## missing a closing quote in 2nd element:
grp <- c("Group 1", "Group 2) 
grp

These errors are easy to detect when the code is basic, but can become much harder to spot with a more complex code, for instance:¹

for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(..method.., ", p-value = ", ..p.format..),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format")
    )
  }

Thankfully, if you use RStudio,² a closing parenthesis, curly brace, square bracket or quotation mark will automatically be written when you open one.

Bear in mind that when installing a package, you must use (single or double) quotation marks around the package’s name:

## NOT correct:
install.packages(ggplot2)

## Error in install.packages : object 'ggplot2' not found

Instead, write one of the two following options:

# install.packages("ggplot2")

# install.packages('ggplot2')

Solution

The solution of course is to simply match all opening parentheses, curly braces, square brackets and quotation marks with their closing counterparts:

mean(c(1, 7, 13))

## [1] 7

x <- 7
if (x > 10) {
  print("x > 10")
} else {
  print("x <= 10")
}

## [1] "x <= 10"

summary(ggplot2::diamonds[, "price"])

##      price      
##  Min.   :  326  
##  1st Qu.:  950  
##  Median : 2401  
##  Mean   : 3933  
##  3rd Qu.: 5324  
##  Max.   :18823

grp <- c("Group 1", "Group 2")
grp

## [1] "Group 1" "Group 2"

Also, make sure:

to correctly place commas:

## NOT correct (misplaced comma):
c("Group 1," "Group 2")

## Error: unexpected string constant in "c("Group 1," "Group 2""

## also NOT correct (missing comma):
c("Group 1" "Group 2")

## Error: unexpected string constant in "c("Group 1" "Group 2""

## correct:
c("Group 1", "Group 2")

you do not mix single and double quotation marks for the same element:

## NOT correct:
c("Group 1')

## correct:
c("Group 1")

## also correct:
c('Group 1')

Note that c('Group 1', "Group 2") does not throw an error but for consistency, it is not recommended to mix single and double quotes within the same vector.

2. Using a function that is not installed or loaded

If you encounter the following error: “Error in … : could not find function ‘…’”, for example:

it means you are trying to use a function belonging to a package which is not yet installed or loaded.

Solution

To solve this error, you have to install the package (if it is not installed yet) and load it with the install.packages() and library() functions, respectively:

## install package:
install.packages("ggplot2")

## load package:
library(ggplot2)

If you are not sure about the usage of these two functions, see more details about installing and loading a package in R.

3. Typos in function, variable, dataset, object or package names

Another common mistake is to misspell a function, a variable, a dataset, an object or a package’s name, for example:

## typo in function name:
maen(c(1, 7, 13))

## Error in maen(c(1, 7, 13)) : could not find function "maen"

## typo in variable name:
summary(ggplot2::diamonds[, "detph"])

## Error: Column `detph` doesn't exist

## typo in dataset name:
data <- data.frame(
  x = rnorm(10),
  y = rnorm(10)
)
summary(dta[, 2])

## Error in summary(dta[, 2]) : object 'dta' not found

## typo in object name:
test <- c(1, 7, 13)
mean(tset)

## Error in mean(tset) : object 'tset' not found

## typo in package name:
library("tydiverse")

## Error in library("tydiverse") : there is no package called ‘tydiverse’

Solution

Make sure that you correctly spell all your functions, variables, datasets, objects and packages:

Note that R is case sensitive; mean() is considered different than Mean() for R!

mean(c(1, 7, 13))

## [1] 7

summary(ggplot2::diamonds[, "depth"])

##      depth      
##  Min.   :43.00  
##  1st Qu.:61.00  
##  Median :61.80  
##  Mean   :61.75  
##  3rd Qu.:62.50  
##  Max.   :79.00

data <- data.frame(
  x = rnorm(10),
  y = rnorm(10)
)
data[, 2]

##  [1]  1.3048697  2.2866454 -1.3888607 -0.2787888 -0.1333213  0.6359504
##  [7] -0.2842529 -2.6564554 -2.4404669  1.3201133

test <- c(1, 7, 13)
mean(test)

## [1] 7

library(tidyverse)

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.3     ✔ readr     2.1.4
## ✔ forcats   1.0.0     ✔ stringr   1.5.0
## ✔ ggplot2   3.4.3     ✔ tibble    3.2.1
## ✔ lubridate 1.9.2     ✔ tidyr     1.3.0
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

If you are sure that you correctly spelled an object, a function or a dataset but you still have an error stating that “object ‘…’ is not found”, make sure that you defined your object/function/dataset before calling it!

It often happens that a student asks me to come to his/her computer because he/she runs the exact same code than me, but cannot make it work. Most of the time, if his/her code is indeed exactly the same than mine, he/she simply has not executed a object/function/dataset before running the code which includes that object/function/dataset. In other words, he/she simply tries to use an undefined object or variable.

Remember that writing code in a R script (contrarily to the console) does not mean it is compiled. You actually have to run it (by clicking on the Run button or using the keyboard shortcut) in order the code to be executed and used later. If you are still struggling with this, see the basics of R and RStudio.

4. Missing, incorrect or misspelled arguments in functions

Most R functions require arguments. For example, the rnorm() function requires at least the number of observations, specified via the argument n.

Your code will not run if you do not specify compulsory arguments, or if incorrectly specify an argument. Moreover, the result might not be what you expect if you misspell an argument:

## missing compulsory argument:
rnorm()

## Error in rnorm() : argument "n" is missing, with no default

## incorrect argument:
rnorm(n = 3, var = 1)

## Error in rnorm(n = 3, var = 1) : unused argument (var = 1)

## misspelled argument:
mean(c(1, 7, 13, NA), narm = TRUE)

## [1] NA

The last piece of code does not throw an error, but the result is not what we want.

Solution

To solve these errors, make sure to specify at least all compulsory arguments of the function, and the correct ones:

In rnorm(), it is the standard deviation, sd, which can be specified in addition to the number of observations n (instead of the variance var).
Removing NA is done with na.rm (instead of narm).

rnorm(n = 3, sd = 1)

## [1] -0.3066386 -1.7813084 -0.1719174

mean(c(1, 7, 13, NA), na.rm = TRUE)

## [1] 7

If you do not know the arguments of a function by heart, you can always check the documentation with ?function_name or help(function_name), for example:

?rnorm()

## or:
help(rnorm)

5. Wrong, inappropriate or inconsistent data types

There are several data types in R, the main ones being:

Numeric
Character
Factor
Logical

You know that some operations and analyses are possible and appropriate only with some specific types of data.

For example, it is not appropriate to compute the mean of a factor or character variable:

gender <- factor(c("female", "female", "male", "female", "male"))

mean(gender)

## Warning in mean.default(gender): argument is not numeric or logical: returning
## NA

## [1] NA

Likewise, although it is technically possible, it makes little sense to draw a barplot of a quantitative continuous variable because in most cases, the frequency will be 1 for each value:

barplot(table(rnorm(10)))

(By the way, if your data is not already displayed in the form of a table, do not forget to add table() inside the barplot() function.)

Solution

Make sure to use the appropriate operation and type of analysis depending on the variable(s) of interest.

For example:

for factor variables, it is more appropriate to compute frequencies and/or relative frequencies, and draw barplots
for quantitative continuous variables, it is more appropriate to compute the mean, median, etc. and draw histograms, boxplots, etc.
for logical variables, the mean,³ a frequency table and a barplot are appropriate
for character variables, word clouds are the most appropriate (unless the variable can be considered as a factor variable because there are not too many different levels)

We now illustrate the examples in R:⁴

## factor:
table(gender)

## gender
## female   male 
##      3      2

prop.table(table(gender))

## gender
## female   male 
##    0.6    0.4

barplot(table(gender))

## quantitative continuous:
x <- rnorm(100)

summary(x)

##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
## -2.99309 -0.74143  0.01809 -0.08570  0.58937  2.70189

par(mfrow = c(1, 2)) ## combine two plots
hist(x)
boxplot(x)

## logical:
x <- c(TRUE, FALSE, FALSE, TRUE, TRUE)

mean(x)

## [1] 0.6

table(x)

## x
## FALSE  TRUE 
##     2     3

barplot(table(x))

For the interested reader, see the most common descriptive statistics in R for different types of data.

Note that, as for descriptive statistics, the choice of the statistical test depends on the variable’s type. See this flowchart to help you in selecting the most appropriate statistical test depending on the number of variables and their types.

An error linked to the one mentioned above is inconsistent data type. See it in practice with the following example:

x <- c(2.4, 3.7, 5.1, 9.8)
class(x)

## [1] "numeric"

y <- c(2.4, 3.7, 5.1, "9.8")
class(y)

## [1] "character"

As you can see, vector x is numerical, whereas vector y is in the form of character. This is due to the fact that the last element of y is surrounded with quotation marks (and thus considered as a string instead of a numerical value), so the entire vector takes the character form.

This can happen when you import a dataset into R and one or several elements of a variable are not encoded correctly. This leads to the entire variable to be considered as a character variable by R.

To avoid this, it is a good practice to check the structure of your dataset (with str()) after importing it to make sure all your variables have the desired format. If not, you can either correct the values in the initial file or change the format in R (with as.numeric()).

6. Forgetting the + sign in ggplot2

If you just learned to use the ggplot2 package for your visualizations (and I highly recommend it!), a common mistake is to forget the + sign.

You know that a visualization made with ggplot2 is constructed by adding several layers:

## load package:
library(ggplot2)

## first layer, the dataset:
ggplot(data = diamonds) +
  ## second layer, the aesthetics:
  aes(x = cut, y = price) +
  ## third layer, the type of plot:
  geom_boxplot() +
  ## add more layers:
  theme_minimal()

Solution

For all your graphics with ggplot2, do not forget to add a + sign after each layer except the last one.

7. Misunderstanding between = and ==

Assignment in R can be done in three ways, from the most to the least common:

<-
=
assign()

The second method, that is =, should not be confused with ==.

Indeed, assigning an object (with any of the three above methods) is used to save something in R. For example, if we want to save the vector (1, 3, 7) and rename that vector x, we can write:

x <- c(1, 3, 7)

## or:
x = c(1, 3, 7)

## or:
assign("x", c(1, 3, 7))

When executing this piece of code, you will see that the vector x of size 3 appears in the tab “Environment” (the top right panel if you use the default view of RStudio):

From now on, we can use that vector simply by calling it by its name:

## [1] 1 3 7

By no means, you can assign an object with ==:

## NOT correct if we want to assign c(1, 3, 7) to x:
x == c(1, 3, 7)

So you are wondering, when would we need to use ==? Actually, it is used when you want to use an equal sign.

I understand that it may be abstract and confusing at the moment, so let’s suppose the following two scenarios as examples (which are the two most common cases when we use ==):

we want to check whether an assigned object or variable respects some conditions, and
we want to subset a dataframe based on one or several conditions.

For these examples, suppose a sample of 5 children:

## create dataframe:
dat <- data.frame(
  Name = c("Mary", "Linda", "James", "John", "Patricia"),
  Age = c(7, 10, 3, 9, 7),
  Gender = c("Girl", "Girl", "Boy", "Boy", "Girl")
)

## print dataframe:
dat

##       Name Age Gender
## 1     Mary   7   Girl
## 2    Linda  10   Girl
## 3    James   3    Boy
## 4     John   9    Boy
## 5 Patricia   7   Girl

Let’s now write different pieces of code for these two scenarios to illustrate them:

We want to check whether the variable Age is equal to the vector (1, 2, 3, 4, 5):

dat$Age == 1:5

## [1] FALSE FALSE  TRUE FALSE FALSE

With this code, we ask whether the first element of the variable Age is equal to 1, the second element of the variable Age is equal to 2, and so on. The answer is of course FALSE, FALSE, TRUE, FALSE and FALSE since only the third child has an age equal to 3 years.

We want to know which of our 5 sampled children are girls:

dat$Gender == "Girl"

## [1]  TRUE  TRUE FALSE FALSE  TRUE

The results show that the first, second and fifth children are girls, while the third and fourth children are not girls.

If you write any of these two lines:

## this overwrites Age and Gender:
dat$Age = 1:5
dat$Gender = "Girl"

You actually overwrite the Age and Gender variables, such that our 5 children will have an age from 1 to 5 (1 year for the first child, up to 5 years for the fifth child) and all of them will be girls.

Now suppose we want to subset our dataframe based on a condition, namely, we want to extract only the children who are 7 years old:

subset(dat, Age == 7)

##       Name Age Gender
## 1     Mary   7   Girl
## 5 Patricia   7   Girl

If you do not want to use the subset function, you can also use square brackets:

dat[dat$Age == 7, ]

##       Name Age Gender
## 1     Mary   7   Girl
## 5 Patricia   7   Girl

As you can see in the previous examples, we do not want to assign anything. Instead, we are asking “is this variable or vector equal to something else?”. For that specific need, we use ==.

So to sum up, for technical reasons and in order to distinguish between the two concepts, R uses = for assignments, and == for the equality sign. Make sure to understand the difference between the two to avoid any errors.

8. Undefined columns selected

If you are used to subset dataframes with square brackets, [], instead of the subset() or filter() functions, you may have faced the error “Error in [.data.frame(…) : undefined columns selected”.

This occurs when R does not understand the column you want to use while subsetting the dataset.

Considering the same sample of 5 children introduced earlier, the following code will throw an error:

dat[dat$Age == 7]

## Error in `[.data.frame`(dat, dat$Age == 7) : undefined columns selected

because it does not specify the column dimension.

Solution

Remember that dataframes in R have two dimensions:

the rows (one for each experimental unit), and
the columns (one for each variable)

and in that particular order (so row first, then column)!

Since dataframes have two dimensions, R expects two dimensions when you call dat[].

In particular, it expects the first and then the second dimension, separated by a comma:

dat[dat$Age == 7, ]

##       Name Age Gender
## 1     Mary   7   Girl
## 5 Patricia   7   Girl

This code means that we are extracting all rows where Age is equal to 7 (first dimension, i.e. before the comma), for all variables of the dataset (since we did not specify any column after the comma).

For the interested reader, see more ways to subset and manipulate data in R.

9. Problem when importing or using the wrong data file

Importing a dataset in R can be quite challenging for beginners, mainly due to the misunderstanding about the working directory.

When importing a file, R will not search for the file in all your folders of your computer. Instead, it will look only in one specific folder. If your dataset is not inside that folder, it will result in an error such as “cannot open file ‘…’: No such file or directory”:

To fix this, you must specify the path to the folder where your dataset is located. In other words, you need to tell R in which folder you want it to work, hence the name working directory.

Setting the working directory can be done with the setwd() function or via the “Files” tab in the lower right panel of RStudio:

Alternatively, you can move the dataset in the folder where R is currently working (this can be found with getwd()). See more details on importing a file into R and about the working directory.

Another related problem is to use the wrong file. This error is different than the previous ones in the sense that you will not encounter an error but your analyses will still be wrong.

It may sound trivial, but make sure to import and use the correct data file! This is particularly the case if you have files for different points in time and which have a common structure (for example weekly or monthly data files with the exact same variables). It happened to me that I reported results for the wrong week (fortunately, without much consequence).

Also, make sure that you actually use all the rows you want to include in your analyses. It happened to me that, in order to test a model (and avoid long computing times), I extracted a random sample of the original dataset, and almost forgot about this sampling when running my final analyses.

It is thus a good practice to remind you to remove sampling and filters after you have tested your code (and before interpreting the final results).

10. Problem when using the $ operator

For the last error of this top 10, I would like to focus on two related errors:

“$ operator is invalid for atomic vectors”, and
“object of type ‘closure’ is not subsettable”.

I gather them in one single section because they are linked to each other in the sense that they both involve the $ operator.

$ operator is invalid for atomic vectors

To understand this error, we first must recall that an atomic vector is a one-dimensional object (usually created with c()). This is different than dataframes or matrices which are two-dimensional (i.e., rows form the first dimension and columns correspond to the second dimension).

The error “$ operator is invalid for atomic vectors” occurs when we try to access an element of an atomic vector using the dollar operator ($):

## define atomic vector:
x <- c(1, 3, 7)

## set names:
names(x) <- LETTERS[1:3]

## print vector:
x

## A B C 
## 1 3 7

## access value of element C:
x$C

## Error in x$C : $ operator is invalid for atomic vectors

Solution

The $ operator cannot be used to extract elements in atomic vectors. Instead, we must use double brackets [[]] notation:

x[["C"]]

## [1] 7

Remember that the $ operator can be used with dataframes, so we can also fix this error by first converting the atomic vector to a dataframe,⁵ and then access an element by its name with the $ operator:

## convert atomic vector to dataframe:
x <- as.data.frame(t(x))

## print x:
x

##   A B C
## 1 1 3 7

## access value of element C:
x$C

## [1] 7

object of type ‘closure’ is not subsettable

Another error (which I must admit is quite obscure and confusing when learning R) is the following: “object of type ‘closure’ is not subsettable”.

This error occurs when we try to subset or access some elements of a function. An example with the well-known mean() function:

mean[1:3]

## Error in mean[1:3] : object of type 'closure' is not subsettable

In R, we can subset lists, vectors, matrices, dataframes, but not functions. So it throws an error because it is impossible to subset an object of type “closure”, and a function is of that type:

typeof(mean)

## [1] "closure"

Most of the times, you will not encounter this error when using a basic function such as the mean() function (because it is unlikely that your goal is really to subset a function…).

Indeed, you will most likely face this error when trying to subset a dataset named data, but this dataset is not defined in the environment (because it has not been imported or created properly for instance).

To understand the concept, see the following examples:

## create dataset:
data <- data.frame(
  x = rnorm(10),
  y = rnorm(10)
)

## print variable x:
data$x

##  [1]  1.12288964  1.43985574 -1.09711377 -0.11731956  1.20149840 -0.46972958
##  [7] -0.05246948 -0.08610730 -0.88767902 -0.44468400

So far so good. Now suppose we made a mistake when creating the dataset:

## create dataset (with mistake):
data <- data.frame(x = rnorm(10)
                   y = rnorm(10))

You will notice that a comma is missing between variables x and y. As a result, the dataset named data is not created and thus not defined.

Therefore, if we now try to access the variable x from that dataset data, R will actually try to subset the function named data instead of the dataset named data!

data$x

## Error in data$x : object of type 'closure' is not subsettable

This happens because, I repeat, the dataset data does not exist, so R looks for an object named data and find a function with that name:

class(data)

## [1] "function"

Warnings

Warnings are different than errors in the sense that they alert you about something, but it does not prevent you from running the code. It is a good practice to read these warnings as they may give you valuable information.

There are too many warnings to mention them all, but I would like to focus on two common ones:

“NAs introduced by coercion”, and
“Removed … rows containing non-finite values (stat_bin())”.

NAs introduced by coercion

This warning occurs when you try to convert a vector which includes at least one non-numerical value to a numeric vector:

x <- c(1, 3, 7, "Emma")

as.numeric(x)

## Warning: NAs introduced by coercion

## [1]  1  3  7 NA

You do not need to fix it since it is only a warning and not an error. R is simply informing you that at least one element in the initial vector was converted to NA because it could not be converted to a numeric value.

Removed … rows containing non-finite values (stat_bin())

This warning occurs when you use ggplot2 to draw plots. For instance:

ggplot(airquality) +
  aes(x = Ozone) +
  geom_histogram()

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## Warning: Removed 37 rows containing non-finite values (`stat_bin()`).

Again, as it is a warning you do not need to fix it. It is simply informing you that there are some missing values (NA) in the variable of interest and that these missing values are removed to construct the plot.

Conclusion

Thanks for reading.

I hope that this collection of errors prevented you from making some coding mistakes, or that it helped you in debugging your code.

If you still cannot fix your error, I would recommend to read the documentation of the function (if you struggle with a function in particular), or look online for the solution. Bear in mind that if you encounter an error, it is very likely that someone else posted the answer online (Stack Overflow is usually a good resource).

R has a steep learning curve, in particular if you are not familiar with another programming language. Nonetheless, with practice and time, you will make less and less coding errors, but more importantly, you will be more and more proficient in typing the right keywords in search engines, resulting in less time spent looking for the solution.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There are 2 mistakes in that piece of code, feel free to try to fix them as an exercise.↩︎
And I strongly recommend using RStudio and not just R. See the differences here.↩︎
Note that mean() applied to a logical variable gives the proportion of TRUE.↩︎
par(mfrow = c(1, 2)) is used to put two plots next to each other.↩︎
Note that we also need to take the transpose of the vector x in order to have it as 1 row, 3 columns.↩︎

One-sample Wilcoxon test in R

Thu, 07 Jul 2022 00:00:00 +0000

Introduction

In a previous article, we showed how to do a two-sample Wilcoxon test in R. Remember that there are actually two versions of this test:

The Mann-Whitney-Wilcoxon test (also referred as Wilcoxon rank sum test or Mann-Whitney U test), used to compare two independent samples. This test is the non-parametric version of the Student’s t-test for independent samples.
The Wilcoxon signed-rank test (also referred as Wilcoxon test for paired samples), used to compare two paired samples. This test is the non-parametric version of the Student’s t-test for paired samples.

In another article, we also showed how to do a one-sample t-test by hand and in R. This test is used to determine whether the mean of a measurement variable is different from a specified value (a value that you specify based on your beliefs or a theoretical expectation for example). Since it is a parametric test, the data should follow a normal distribution (or sample size should be large enough (i.e., above 30), thanks to the central limit theorem) for the results to be valid.

Unlike the one-sample t-test, the one-sample Wilcoxon test (also referred as the one-sample Wilcoxon signed-rank test) is a non-parametric test, meaning that it does not rely on data belonging to any particular parametric family of probability distributions. Non-parametric tests usually have the same goal as their parametric counterparts (in this case, compare data to a given value). Nonetheless, they do not require the assumption of normality and they can deal with outliers and Likert scales.

In this article, we show when to perform the one-sample Wilcoxon test, how to do it in R and how to interpret its results. We will also briefly show some appropriate visualizations.

When?

The one-sample Wilcoxon test is used to compare our observations to a given default value—a value that you specify based on your beliefs or a theoretical expectation for example. In other words, it is used to determine if a group is significantly different from a known or hypothesized population value on the variable of interest.

Since the test statistic is computed based on the ranks of the difference between the observed values and the default value (making it a non-parametric test), the one-sample Wilcoxon test is more appropriate than a one-sample t-test when the observations do not follow a normal distribution.

The goal of this test is to verify whether the observations are significantly different from our default value. In terms of null and alternative hypotheses, we have (for a two-tailed test):

$H_0:$ location of the data is equal to the chosen value
$H_1:$ location of the data is different from the chosen value

In other words, a significant result (i.e., a rejection of the null hypothesis) suggests that the location of the data is different from the chosen value.

Note that some authors suggest that this test is a test of the median, that is (for a two-tailed test):

$H_0:$ the median is equal to the chosen value
$H_1:$ the median is different from the chosen value

However, this is the case only if the data are symmetric. Without further assumptions about the distribution of the data, the one-sample Wilcoxon test is not a test of the median but a test about the location of the data.¹

Note that although the normality assumption is not required, the independence assumption must still be verified. This means that observations must be independent of one another (usually, random sampling is sufficient to have independence).

Data

For our illustration, suppose we want to test whether the scores at an exam differ from 10, that is:

$H_0:$ scores at the exam $= 10$
$H_1:$ scores at the exam $\ne 10$

To verify this, we have a sample of 15 students and their score at the exam:

dat

##    Student_ID Score
## 1           1    17
## 2           2     5
## 3           3     1
## 4           4    10
## 5           5     4
## 6           6    18
## 7           7    17
## 8           8    15
## 9           9     7
## 10         10     4
## 11         11     5
## 12         12    14
## 13         13    20
## 14         14    18
## 15         15    15

Scores between students are assumed to be independent (a student’s score is not impacted or influenced by the score of another student). Therefore, the independence assumption is met.

Moreover, sample size is small (n < 30) and based on the histogram the data do not follow a normal distribution:

# histogram
hist(dat$Score)

Note that we refrain from verifying the normality via a normality test (such as the Shapiro-Wilk test for instance) because for small sample sizes, normality tests have little power to reject the null hypothesis and therefore small samples most often pass normality tests (Öztuna et al. 2006; Ghasemi and Zahediasl 2012).

Note also that although we use a quantitative variable for the illustration, the one-sample Wilcoxon test is also appropriate for interval data and Likert scales.

How?

The one-sample Wilcoxon test can be done in R with the wilcox.test() function.

But first, it is a good practice to visualize our data in a boxplot and compute some descriptive statistics to compare our observations with our default value:

# boxplot
boxplot(dat$Score,
  ylab = "Score"
)

If like me you prefer to use the {ggplot2} package for your plots:

# boxplot
library(ggplot2)

ggplot(dat, aes(y = Score)) +
  geom_boxplot() +
  labs(y = "Score") +
  theme( # remove axis text and ticks
    axis.text.x = element_blank(),
    axis.ticks = element_blank()
  )

Some basic descriptive statistics (rounded to two decimals):

round(summary(dat$Score),
  digits = 2
)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    5.00   14.00   11.33   17.00   20.00

From the boxplot and the descriptive statistics above, we see that the mean and median of the scores in our sample are respectively 11.33 and 14.

The one-sample Wilcoxon test will tell us whether the scores are significantly different from 10 or not (and thus whether they are different from 10 in the population or not):

wilcox.test(dat$Score,
  mu = 10 # default value
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.3779
## alternative hypothesis: true location is not equal to 10

The output presents several information such as the:

title of the test
data
test statistic
$p$-value
alternative hypothesis

We focus on the $p$-value to interpret and conclude the test.

Interpretation:

Based on the results of the test, (at the significance level of 0.05) we do not reject the null hypothesis, so we do not reject the hypothesis that the scores at this exam are equal to 10, and we cannot conclude that the scores are significantly different from 10 ($p$-value = 0.378).

By default, it is a two-tailed test that is done. As for the t.test() function, we can specify that a one-sided test is required by using either the alternative = "greater" or alternative = "less argument in the wilcox.test() function.

For example, if we want to test that the scores are higher than 10:

wilcox.test(dat$Score,
  mu = 10, # default value
  alternative = "greater" # H1: scores > 10
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.189
## alternative hypothesis: true location is greater than 10

Interpretation:

In this case, we still do not reject the hypothesis that scores are equal to 10 and we cannot conclude that scores are significantly higher than 10 ($p$-value = 0.189).

For more information about the arguments available in the function, see ?wilcox.test.

Note that you may encounter the following warnings when using wilcox.test():

Warning messages:
1: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with ties
2: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with zeroes

It is a warning rather than an indication that your results are incorrect. R is informing you that it is reporting a $p$-value based on a normal approximation rather than an exact $p$-value based on the data because there are ties (some values are the same). Use the exact = FALSE option if you want to remove the warning.

Combine statistical test and plot

If you are a frequent user of the blog, you know that I like to present results of a test directly on a plot. This allows me to visualize the data and conclude the test in a concise manner.

This is possible thanks to the gghistostats() function within the {ggstatsplot} package:

# load package
library(ggstatsplot)

# combine plot and test
gghistostats(
  data = dat, # dataframe
  x = Score, # variable
  type = "nonparametric", # nonparametric = Wilcoxon, parametric = t-test
  test.value = 10 # default value
)

The histogram² shows the distribution of the scores and results of the test is shown in the title of the plot.

As you can see, results of the test are the same, that is, there is not enough evidence in the data to conclude that scores are significantly different from 10 ($p$-value = 0.378).

Conclusion

Thanks for reading.

I hope this article helped you to understand the one-sample Wilcoxon test and how to do it in R.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Ghasemi, Asghar, and Saleh Zahediasl. 2012. “Normality Tests for Statistical Analysis: A Guide for Non-Statisticians.” International Journal of Endocrinology and Metabolism 10 (2): 486.

Öztuna, Derya, Atilla Halil Elhan, and Ersöz Tüccar. 2006. “Investigation of Four Different Normality Tests in Terms of Type 1 Error Rate and Power Under Different Distributions.” Turkish Journal of Medical Sciences 36 (3): 171–76.

See more information in this article.↩︎
This histogram looks different than the previous one because the number of bins is different (4 versus 5 bins).↩︎

What statistical test should I do?

Thu, 02 Dec 2021 00:00:00 +0000

Introduction

Being a teaching assistant in statistics for students with diverse backgrounds, I have the chance to see what is globally not well understood by students.

I have realized that it is usually not a problem for students to do a specific statistical test when they are told which one to use (as long as they have good resources and they have been attentive during classes, of course). However, it appears that the task is much more difficult for them when they need to choose what test to do.

This article presents a flowchart to help students in selecting the most appropriate statistical test based on a couple of criteria.¹

Flowchart

Due to the large number of tests, the image is quite wide so it may not render well on all screens. In that case, you can see it in full screen by clicking directly on the image or by clicking on the following link:

Download in PDF

Notes

As you can see in the flowchart, the selection of the most appropriate test is based on:

the number of variables of interest: one, two or more than two variables
the variable type: quantitative or qualitative
in case of a qualitative variable, the number of groups and whether they are independent or paired (i.e., dependent)
whether you want the parametric or nonparametric version²

Summarizing so many tests in a single image is not an easy task. The goal of this flowchart is to provide students with a quick and easy way to select the most appropriate statistical test among the most common ones (or to see what are the alternatives).

Obviously, this flowchart is not exhaustive. There are many other tests but most of them have been omitted on purpose to keep it simple and readable. I decided to keep it simple so that the flowchart is not overwhelming, with the hope that it is still complete and precise enough for most students.

For the sake of completeness, here are a few additional remarks about this flowchart:

Tests for more than 2 variables are applicable to the case of 2 variables as well. For simplicity, I however tend to suggest the simplest test when more than one is possible. For instance, with two quantitative variables, both a correlation test and a simple linear regression can be done. In introductory statistics classes, I will most likely teach the concept of correlation but not necessarily the concept of linear regression. For this reason, I will most likely recommend a correlation test over a linear regression, unless the students have a more advanced level.
The Kolmogorov-Smirnov test, used to compare a sample with a reference probability distribution or to compare two samples, has been omitted because it is generally not taught in introductory classes. Keep in mind, however, that this test is useful both in the uni and bivariate cases.
Normality tests (such as, among others, the Shapiro-Wilk or Kolmogorov-Smirnov test) have also been omitted as they are part of another family of tests (they are used to answer the question “Is my dataset well-modeled by a normal distribution?”). Remember, nonetheless, that they are very useful to verify the normality assumption required in many hypothesis tests. For example, the Student’s t-test requires that the data follow approximately a normal distribution in case of small sample size. If this is not the case, the nonparametric version (i.e., the Wilcoxon test) should be preferred. The same goes for ANOVA and many other statistical tests.
The flowchart could be extended to include more advanced linear or non-linear models, but this is beyond its scope and goal. Remember that I created it to help non-experts to see more clearly and have a broad overview of the most common statistical tests, not to confuse them even more.
If you open the flowchart in PDF, you will be able to click on most of the tests. Clicking on the name of the test will redirect you to the corresponding article, which explains the test in further details. If the test is not clickable, it means I have not written about it yet. I will update the flowchart if I publish an article about one of the missing test.

Conclusion

Thanks for reading.

I hope this guide will help you in determining the right statistical test. Feel free to share it with all students who might be interested.

As always, if you have a question or a suggestion (for example, if I missed a test which you believe should be included), please add it as a comment so other readers can benefit from the discussion.

For those interested to draw flowcharts like this one, note that I used diagrams.net. At the time of writing this article, it is free and you can synchronize it with Google Drive, GitHub, etc.↩︎
A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. On the contrary, a nonparametric test does not rely on data belonging to any particular parametric family of probability distributions. Nonparametric tests have the same objective as their parametric counterparts. However, they have two advantages over parametric tests: (i) they do not require the assumption of normality of distributions and (ii) they can deal with outliers. The trade-off is that nonparametric tests are usually less powerful than their corresponding parametric version when the normality assumption holds. Therefore, all else being equal, with a nonparametric test you are less likely to reject the null hypothesis when it is false if the data follow a normal distribution. It is thus preferred to use the parametric version when the assumptions are met.↩︎

Hypothesis test by hand

Wed, 27 Jan 2021 00:00:00 +0000

Descriptive versus inferential statistics

Remember that descriptive statistics is the branch of statistics aiming at describing and summarizing a set of data in the best possible manner, that is, by reducing it down to a few meaningful key measures and visualizations—with as little loss of information as possible. In other words, the branch of descriptive statistics helps to have a better understanding and a clear image about a set of observations thanks to summary statistics and graphics. With descriptive statistics, there is no uncertainty because we describe only the group of observations that we decided to work on and no attempt is made to generalize the observed characteristics to another or to a larger group of observations.

Inferential statistics, one the other hand, is the branch of statistics that uses a random sample of data taken from a population to make inferences, i.e., to draw conclusions about the population of interest (see the difference between population and sample if you need a refresh of the two concepts). In other words, information from the sample is used to make generalizations about the parameter of interest in the population.

The two most important tools used in the domain of inferential statistics are:

hypothesis test (which is the main subject of the present article), and
confidence interval (which is briefly discussed in this section)

Motivations and limitations

Via my teaching tasks, I realized that many students (especially in introductory statistic classes) struggle to perform hypothesis tests and interpret the results. It seems to me that these students often encounter difficulties mainly because hypothesis testing is rather unclear and abstract to them.

One of the reason it looks abstract to them is because they do not understand the final goal of hypothesis testing—the “why” behind this tool. They often do inferential statistics without understanding the reasoning behind it, as if they were following a cooking recipe which does not require any thinking. However, as soon as they understand the principle underlying hypothesis testing, it is much easier for them to apply the concepts and solve the exercises.

For this reason, I though it would be useful to write an article on the goal of hypothesis tests (the “why?”), in which context they should be used (the “when?”), how they work (the “how?”) and how to interpret the results (the “so what?”). Like anything else in statistics, it becomes much easier to apply a concept in practice when we understand what we are testing or what we are trying to demonstrate beforehand.

In this article, I present—as comprehensibly as possible—the different steps required to perform and conclude a hypothesis test by hand.

These steps are illustrated with a basic example. This will build the theoretical foundations of hypothesis testing, which will in turn be of great help for the understanding of most statistical tests.

Hypothesis tests come in many forms and can be used for many parameters or research questions. The steps I present in this article are not applicable to all hypothesis test, unfortunately.

They are however, appropriate for at least the most common hypothesis tests—the tests on:

One mean: $\mu$
Two means:
- independent samples: $\mu_1$ and $\mu_2$
- paired samples: $\mu_D$
One proportion: $p$
Two proportions: $p_1$ and $p_2$
One variance: $\sigma^2$
Two variances: $\sigma^2_1$ and $\sigma^2_2$

The good news is that the principles behind these 6 statistical tests (and many more) are exactly the same. So if you understand the intuition and the process for one of them, all others pretty much follow.

Hypothesis test

Why?

Unlike descriptive statistics where we only describe the data at hand, hypothesis tests use a subset of observations, referred as a sample, to draw conclusions about a population.

One may wonder why we would try to “guess” or make inference about a parameter of a population based on a sample, instead of simply collecting data for the entire population, compute statistics we are interested in and take decisions based upon that.

The main reason we actually use a sample instead of the entire population is because, most of the time, collecting data on the entire population is practically impossible, too complex, too expensive, it would take too long, or a combination of any of these.¹

So the overall objective of a hypothesis test is to draw conclusions in order to confirm or refute a belief about a population, based on a smaller group of observations.

In practice, we take some measurements of the variable of interest—representing the sample(s)—and we check whether our measurements are likely or not given our assumption (our belief). Based on the probability of observing the sample(s) we have, we decide whether we can trust our belief or not.

When?

Hypothesis tests have many practical applications.

Here are different situations illustrating when the 6 tests mentioned above would be appropriate:

One mean: suppose that a health professional would like to test whether the mean weight of Belgian adults is different than 80 kg (176.4 lbs).
Two means:
- Independent samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) for patients in a control group and patients in a treatment group, where patients in the two groups are different.
- Paired samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) before and after a treatment, where patients are measured twice—before and after treatment, so patients are the same in the 2 samples.
One proportion: suppose that a political pundit would like to test whether the proportion of citizens who are going to vote for a specific candidate is smaller than 30%.
Two proportions: suppose that a doctor would like to test whether the proportion of smokers is different between professional and amateur athletes.
One variance: suppose that an engineer would like to test whether a voltmeter has a lower variability than what is imposed by the safety standards.
Two variances: suppose that, in a factory, two production lines work independently from each other. The financial manager would like to test whether the costs of the weekly maintenance of these two machines have the same variance. Note that a test on two variances is also often performed to verify the assumption of equal variances, which is required for several other statistical tests, such as the Student’s t-test for instance.

Of course, this is a non-exhaustive list of potential applications and many research questions can be answered thanks to a hypothesis test.

One important point to remember is that in hypothesis testing we are always interested in the population and not in the sample. The sample is used for the aim of drawing conclusions about the population, so we always test in terms of the population.

Usually, hypothesis tests are used to answer research questions in confirmatory analyses. Confirmatory analyses refer to statistical analyses where hypotheses—deducted from theory—are defined beforehand (preferably before data collection). In this approach, the researcher has a specific idea about the variables under consideration and she is trying to see if her idea, specified as hypotheses, is supported by data.

On the other hand, hypothesis tests are rarely used in exploratory analyses.² Exploratory analyses aims to uncover possible relationships between the variables under investigation. In this approach, the researcher does not have any clear theory-driven assumptions or ideas in mind before data collection. This is the reason exploratory analyses are sometimes referred as hypothesis-generating analyses—they are used to create some hypotheses, which in turn may be tested via confirmatory analyses at a later stage.

How?

There are, to my knowledge, 3 different methods to perform a hypothesis tests:

Although the process for these 3 approaches may slightly differ, they all lead to the exact same conclusions. Using one method or another is, therefore, more often than not a matter of personal choice or a matter of context. See this section to know which method I use depending on the context.

I present the 3 methods in the following sections, starting with, in my opinion, the most comprehensive one when it comes to doing it by hand: comparing the test statistic with the critical value.

For the three methods, I will explain the required steps to perform a hypothesis test from a general point of view and illustrate them with the following situation:³

Suppose a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg.

Note that, as for most hypothesis tests, the test we are going to use as example below requires some assumptions. Since the aim of the present article is to explain a hypothesis test, we assume that all assumptions are met. For the interested reader, see the assumptions (and how to verify them) for this type of hypothesis test in the article presenting the one-sample t-test.

Method A: Comparing the test statistic with the critical value

Method A, which consists in comparing the test statistic with the critical value, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Finding the critical value
Concluding and interpreting the results

Each step is detailed below.

Step #1: Stating the null and alternative hypothesis

As discussed before, a hypothesis test first requires an idea, that is, an assumption about a phenomenon. This assumption, referred as hypothesis, is derived from the theory and/or the research question.

Since a hypothesis test is used to confirm or refute a prior belief, we need to formulate our belief so that there is a null and an alternative hypothesis. Those hypotheses must be mutually exclusive, which means that they cannot be true at the same time. This is step #1.

In the context of our scenario, the null and alternative hypothesis are thus:

Null hypothesis $H_0: \mu = 80$
Alternative hypothesis $H_1: \mu \ne 80$

When stating the null and alternative hypothesis, bear in mind the following three points:

We are always interested in the population and not in the sample. This is the reason $H_0$ and $H_1$ will always be written in terms of the population and not in terms of the sample (in this case, $\mu$ and not $\bar{x}$).
The assumption we would like to test is often the alternative hypothesis. If the researcher wanted to test whether the mean weight of Belgian adults was less than 80 kg, she would have stated $H_0: \mu = 80$ (or equivalently, $H_0: \mu \ge 80$) and $H_1: \mu < 80$.⁴ Do not mix the null with the alternative hypothesis, or the conclusions will be diametrically opposed!
The null hypothesis is often the status quo. For instance, suppose that a doctor wants to test whether the new treatment A is more efficient than the old treatment B. The status quo is that the new and old treatments are equally efficient. Assuming a larger value is better, she will then write $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A > \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B > 0$). On the opposite, if the lower the better, she would have written $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A < \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B < 0$).

Step #2: Computing the test statistic

The test statistic (often called t-stat) is, in some sense, a metric indicating how extreme the observations are compared to the null hypothesis. The higher the t-stat (in absolute value), the more extreme the observations are.

There are several formulas to compute the t-stat, with one formula for each type of hypothesis test—one or two means, one or two proportions, one or two variances. This means that there is a formula to compute the t-stat for a hypothesis test on one mean, another formula for a test on two means, another for a test on one proportion, etc.⁵

The only difficulty in this second step is to choose the appropriate formula. As soon as you know which formula to use based on the type of test, you simply have to apply it to the data. For the interested reader, see the different formulas to compute the t-stat for the most common tests in this Shiny app.

Luckily, formulas for hypothesis tests on one and two means, and one and two proportions follow the same structure.

Computing the test statistic for these tests is similar than scaling a random variable (a process also knows as “standardization” or “normalization”) which consists in subtracting the mean from that random variable, and dividing the result by the standard deviation:

\[Z = \frac{X - \mu}{\sigma}\]

For these 4 hypothesis tests (one/two means and one/two proportions), computing the test statistic is like scaling the estimator (computed from the sample) corresponding to the parameter of interest (in the population). So we basically subtract the target parameter from the point estimator and then divide the result by the standard error (which is equivalent to the standard deviation but for an estimator).

If this is unclear, here is how the test statistic (denoted $t_{obs}$) is computed in our scenario (assuming that the variance of the population is unknown):

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}}\]

where:

$\bar{x}$ is the sample mean (i.e., the estimator)
$\mu$ is the mean under the null hypothesis (i.e., the target parameter)
$s$ is the sample standard deviation
$n$ is the sample size
($\frac{s}{\sqrt{n}}$ is the standard error)

Notice the similarity between the formula of this test statistic and the formula used to standardize a random variable. This structure is the same for a test on two means, one proportion and two proportions, except that the estimator, the parameter and the standard error are, of course, slightly different for each type of test.

Suppose that in our case we have a sample mean of 71 kg ($\bar{x}$ = 71), a sample standard deviation of 13 kg ($s$ = 13) and a sample size of 10 adults ($n$ = 10). Remember that the population mean (the mean under the null hypothesis) is 80 kg ($\mu$ = 80).

The t-stat is thus:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Although formulas are different depending on which parameter you are testing, the value found for the test statistic gives us an indication on how extreme our observations are.

We keep this value of -2.189 in mind because it will be used again in step #4.

Step #3: Finding the critical value

Although the t-stat gives us an indication of how extreme our observations are, we cannot tell whether this “score of extremity” is too extreme or not based on its value only.

So, at this point, we cannot yet tell whether our data are too extreme or not. For this, we need to compare our t-stat with a threshold—referred as critical value—given by the probability distribution tables (and which can, of course, also be found with R).

In the same way that the formula to compute the t-stat is different for each parameter of interest, the underlying probability distribution—and thus the statistical table—on which the critical value is based is also different for each target parameter. This means that, in addition to choosing the appropriate formula to compute the t-stat, we also need to select the appropriate probability distribution depending on the parameter we are testing.

Luckily, there are only 4 different probability distributions for the 6 hypothesis tests covered in this article (one/two means, one/two proportions and one/two variances):

Standard Normal distribution:
- test on one and two means with known population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is known
- test on one and two proportions (given that some assumptions are met)
Student distribution:
- test on one and two means with unknown population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is unknown
Chi-square distribution:
- test on one variance
Fisher distribution:
- test on two variances

Each probability distribution also has its own parameters (up to two parameters for the 4 distribution considered here), defining its shape and/or location. Parameter(s) of a probability distribution can be seen as its DNA; meaning that the distribution is entirely defined by its parameter(s).

Taking our initial scenario—a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg—as example.

The underlying probability distribution of a test on one mean is either the standard Normal or the Student distribution, depending on whether the variance of the population (not sample variance!) is known or unknown:⁶

If the population variance is known $\rightarrow$ the standard Normal distribution is used
If the population variance is unknown $\rightarrow$ the Student distribution is used

If no population variance is explicitly given, you can assume that it is unknown since you cannot compute it based on a sample. If you could compute it, that would mean you have access to the entire population and there is, in this case, no point in performing a hypothesis test (you could simply use some descriptive statistics to confirm or refute your belief).

In our example, no population variance is specified so it is assumed to be unknown. We therefore use the Student distribution.

The Student distribution has one parameter which defines it; the number of degrees of freedom. The number of degrees of freedom depends on the type of hypothesis test. For instance, the number of degrees of freedom for a test on one mean is equal to the number of observations minus one ($n$ - 1). Without going too far into the details, the - 1 comes from the fact that there is one quantity which is estimated (i.e., the mean).⁷ The sample size being equal to 10 in our example, the degrees of freedom is equal to $n$ - 1 = 10 - 1 = 9.

There is only one last element missing to find the critical value: the significance level. The significance level, denoted $\alpha$, is the probability of wrongly rejecting the null hypothesis, so the probability of rejecting the null hypothesis although it is in reality true. In this sense, it is an error (type I error, as opposed to the type II error⁸) that we accept to deal with, in order to be able to draw conclusions about a population based on a subset of it.

As you may have read in many statistical textbooks, the significance level is very often set to 5%.⁹ In some fields (such as medicine or engineering, among others), the significance level is also sometimes set to 1% to decrease the error rate.

It is best to specify the significance level before performing a hypothesis test to avoid the temptation to set the significance level in accordance to the results (the temptation is even bigger when the results are on the edge of being significant). As I always tell my students, you cannot “guess” nor compute the significance level. Therefore, if it is not explicitly specified, you can safely assume it is 5%. In our case, we did not indicate it, so we take $\alpha$ = 5% = 0.05.

Furthermore, in our example, we want to test whether the mean weight of Belgian adults is different than 80 kg. Since we do not specify the direction of the test, it is a two-sided test. If we wanted to test that the mean weight was less than 80 kg ($H_1: \mu <$ 80) or greater than 80 kg ($H_1: \mu >$ 80), we would have done a one-sided test.

Make sure that you perform the correct test (two-sided or one-sided) because it has an impact on how to find the critical value (see more in the following paragraphs).

So now that we know the appropriate distribution (Student distribution), its parameter (degrees of freedom (df) = 9), the significance level ($\alpha$ = 0.05) and the direction (two-sided), we have all we need to find the critical value in the statistical tables:

By looking at the row df = 9 and the column $t_.025$ in the Student’s distribution table, we find a critical value of:

\[t_{n-1; \alpha / 2} = t_{9; 0.025} = 2.262\]

One may wonder why we take $t_{\alpha/2} = t_.025$ and not $t_\alpha = t_.05$ since the significance level is 0.05. The reason is that we are doing a two-sided test ($H_1: \mu \ne$ 80), so the error rate of 0.05 must be divided in 2 to find the critical value to the right of the distribution. Since the Student’s distribution is symmetric, the critical value to the left of the distribution is simply: -2.262.

Visually, the error rate of 0.05 is partitioned into two parts:

0.025 to the left of -2.262 and
0.025 to the right of 2.262

We keep in mind these critical values of -2.262 and 2.262 for the fourth and last step.

Note that the red shaded areas in the previous plot are also known as the rejection regions. More on that in the following section.

These critical values can also be found in R, thanks to the qt() function:

qt(0.025, df = 9, lower.tail = TRUE)

## [1] -2.262157

qt(0.025, df = 9, lower.tail = FALSE)

## [1] 2.262157

The qt() function is used for the Student’s distribution (q stands for quantile and t for Student). There are other functions accompanying the different distributions:

qnorm() for the Normal distribution
qchisq() for the Chi-square distribution
qf() for the Fisher distribution

Step #4: Concluding and interpreting the results

In this fourth and last step, all we have to do is to compare the test statistic (computed in step #2) with the critical values (found in step #3) in order to conclude the hypothesis test.

The only two possibilities when concluding a hypothesis test are:

Rejection of the null hypothesis
Non-rejection of the null hypothesis

In our example of adult weight, remember that:

the t-stat is -2.189
the critical values are -2.262 and 2.262

Also remember that:

the t-stat gives an indication on how extreme our sample is compared to the null hypothesis
the critical values are the threshold from which the t-stat is considered as too extreme

To compare the t-stat with the critical values, I always recommend to plot them:

These two critical values form the rejection regions (the red shaded areas):

from $- \infty$ to -2.262, and
from 2.262 to $\infty$

If the t-stat lies within one of the rejection region, we reject the null hypothesis. On the contrary, if the t-stat does not lie within any of the rejection region, we do not reject the null hypothesis.

As we can see from the above plot, the t-stat is less extreme than the critical value and therefore does not lie within any of the rejection region. In conclusion, we do not reject the null hypothesis that $\mu = 80$.

This is the conclusion in statistical terms but they are meaningless without proper interpretation. So it is a good practice to also interpret the result in the context of the problem:

At the 5% significance level, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

Why don’t we accept $H_0$?

From a more philosophical (but still very important) perspective, note that we wrote “we do not reject the null hypothesis” and “we do not reject the hypothesis that the mean weight of Belgian adults is equal to 80 kg”. We did not write “we accept the null hypothesis” nor “the mean weight of Belgian adults is 80 kg”.

The reason is due to the fact that, in hypothesis testing, we conclude something about the population based on a sample. There is, therefore, always some uncertainty and we cannot be 100% sure that our conclusion is correct.

Perhaps it is the case that the mean weight of Belgian adults is in reality different than 80 kg, but we failed to prove it based on the data at hand. It may be the case that if we had more observations, we would have rejected the null hypothesis (since all else being equal, a larger sample size implies a more extreme t-stat). Or, it may be the case that even with more observations, we would not have rejected the null hypothesis because the mean weight of Belgian adults is in reality close to 80 kg. We cannot distinguish between the two.

So we can just say that we did not find enough evidence against the hypothesis that the mean weight of Belgian adults is 80 kg, but we do not conclude that the mean is equal to 80 kg.

If the difference is still not clear to you, the following example may help. Suppose a person is suspected of having committed a crime. This person is either innocent—the null hypothesis—or guilty—the alternative hypothesis. In the attempt to know if the suspect committed the crime, the police collects as much information and proof as possible. This is similar to the researcher collecting data to form a sample. And then the judge, based on the collected evidence, decides whether the suspect is considered as innocent or guilty. If there is enough evidence that the suspect committed the crime, the judge will conclude that the suspect is guilty. In other words, she will reject the null hypothesis of the suspect being innocent because there are enough evidence that the suspect committed the crime.

This is similar to the t-stat being more extreme than the critical value: we have enough information (based on the sample) to say that the null hypothesis is unlikely because our data would be too extreme if the null hypothesis were true. Since the sample cannot be “wrong” (it corresponds to the collected data), the only remaining possibility is that the null hypothesis is in fact wrong. This is the reason we write “we reject the null hypothesis”.

On the other hand, if there is not enough evidence that the suspect committed the crime (or no evidence at all), the judge will conclude that the suspect is considered as not guilty. In other words, she will not reject the null hypothesis of the suspect being innocent. But even if she concludes that the suspect is considered as not guilty, she will never be 100% sure that he is really innocent.

It may be the case that:

the suspect did not commit the crime, or
the suspect committed the crime but the police was not able to collect enough information against the suspect.

In the former case the suspect is really innocent, whereas in the latter case the suspect is guilty but the police and the judge failed to prove it because they failed to find enough evidence against him. Similar to hypothesis testing, the judge has to conclude the case by considering the suspect not guilty, without being able to distinguish between the two.

This is the main reason we write “we do not reject the null hypothesis” or “we fail to reject the null hypothesis” (you may even read in some textbooks conclusion such as “there is no sufficient evidence in the data to reject the null hypothesis”), and we do not write “we accept the null hypothesis”.

I hope this metaphor helped you to understand the reason why we reject the null hypothesis instead of accepting it.

In the following sections, we present two other methods used in hypothesis testing.

These methods will result in the exact same conclusion: non-rejection of the null hypothesis, that is, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg. It is thus presented only if you prefer to use these methods over the first one.

Method B: Comparing the p-value with the significance level $\alpha$

Method B, which consists in computing the p-value and comparing this p-value with the significance level $\alpha$, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Computing the p-value
Concluding and interpreting the results

In this second method which uses the p-value, the first and second steps are similar than in the first method.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the test statistic

Remember that the formula for the t-stat is different depending on the type of hypothesis test (one or two means, one or two proportions, one or two variances). In our case of one mean with unknown variance, we have:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Step #3: Computing the p-value

The p-value is the probability (so it goes from 0 to 1) of observing a sample at least as extreme as the one we observed if the null hypothesis were true. In some sense, it gives you an indication on how likely your null hypothesis is. It is also defined as the smallest level of significance for which the data indicate rejection of the null hypothesis.

For more information about the p-value, I recommend reading this note about the p-value and the significance level $\alpha$.

Formally, the p-value is the area beyond the test statistic. Since we are doing a two-sided test, the p-value is thus the sum of the area above 2.189 and below -2.189.

Visually, the p-value is the sum of the two blue shaded areas in the following plot:

The p-value can computed with precision in R with the pt() function:

p_val <- pt(-2.189, df = 9, lower.tail = TRUE) + pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

# which is equivalent than:
p_val <- 2 * pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

The p-value is 0.0563, which indicates that there is a 5.63% chance to observe a sample at least as extreme as the one observed if the null hypothesis were true. This already gives us a hint on whether our t-stat is too extreme or not (and thus whether our null hypothesis is likely or not), but we formally conclude in step #4.

Like the qt() function to find the critical value, we use pt() to find the p-value because the underlying distribution is the Student’s distribution.

Use pnorm(), pchisq() and pf() for the Normal, Chi-square and Fisher distribution, respectively. See also this Shiny app to compute the p-value given a certain t-stat for most probability distributions.

If you do not have access to a computer (during exams for example) you will not be able to compute the p-value precisely, but you can bound it using the statistical table referring to your test.

In our case, we use the Student distribution and we look at the row df = 9 (since df = n - 1):

The test statistic is -2.189
We take the absolute value, which gives 2.189
The value 2.189 is between 1.833 and 2.262 (highlighted in blue in the above table)
From the column names $t_{.050}$ and $t_{.025}$ related to 1.833 and 2.262, we know that:
- the area to the right of 1.833 is 0.05
- the area to the right of 2.262 is 0.025
So we know that the area to the right of 2.189 must be between 0.025 and 0.05
Since the Student distribution is symmetric, we know that the area to the left of -2.189 must also be between 0.025 and 0.05
Therefore, the sum of the two areas must be between 0.05 and 0.10
In other words, the p-value is between 0.05 and 0.10 (i.e., 0.05 < p-value < 0.10)

Although we could not compute it precisely, it is enough to conclude our hypothesis test in the last step.

Step #4: Concluding and interpreting the results

The final step is now to simply compare the p-value (computed in step #3) with the significance level $\alpha$. As for all statistical tests:

If the p-value is smaller than $\alpha$ (p-value < 0.05) $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the p-value is greater than or equal to $\alpha$ (p-value $\ge$ 0.05) $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

No matter if we take into consideration the exact p-value (i.e., 0.0563) or the bounded one (0.05 < p-value < 0.10), it is larger than 0.05, so we do not reject the null hypothesis.¹⁰ In the context of the problem, we do not reject the null hypothesis that the mean weight of Belgian adults is 80 kg.

Remember that rejecting (or not rejecting) a null hypothesis at the significance level $\alpha$ using the critical value method (method A) is equivalent to rejecting (or not rejecting) the null hypothesis when the p-value is lower (equal or greater) than $\alpha$ (method B).

This is the reason we find the exact same conclusion than with method A, and why you should too if you use both methods on the same data and with the same significance level.

Method C: Comparing the target parameter with the confidence interval

Method C, which consists in computing the confidence interval and comparing this confidence interval with the target parameter (the parameter under the null hypothesis), boils down to the following 3 steps:

Stating the null and alternative hypothesis
Computing the confidence interval
Concluding and interpreting the results

In this last method which uses the confidence interval, the first step is similar than in the first two methods.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the confidence interval

Like hypothesis testing, confidence intervals are a well-known tool in inferential statistics.

Confidence interval is an estimation procedure which produces an interval (i.e., a range of values) containing the true parameter with a certain—usually high—probability.

In the same way that there is a formula for each type of hypothesis test when computing the test statistics, there exists a formula for each type of confidence interval. Formulas for the different types of confidence intervals can be found in this Shiny app.

Here is the formula for a confidence interval on one mean $\mu$ (with unknown population variance):

\[ (1-\alpha)\text{% CI for } \mu = \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \]

where $t_{\alpha/2, n - 1}$ is found in the Student distribution table (and is similar to the critical value found in step #3 of method A).

Given our data and with $\alpha$ = 0.05, we have:

\[ \begin{aligned} 95\text{% CI for } \mu &= \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \\ &= 71 \pm 2.262 \frac{13}{\sqrt{10}} \\ &= [61.70; 80.30] \end{aligned} \]

The 95% confidence interval for $\mu$ is [61.70; 80.30] kg. But what does a 95% confidence interval mean?

We know that this estimation procedure has a 95% probability of producing an interval containing the true mean $\mu$. In other words, if we construct many confidence intervals (with different samples of the same size), 95% of them will, on average, include the mean of the population (the true parameter). So on average, 5% of these confidence intervals will not cover the true mean.

If you wish to decrease this last percentage, you can decrease the significance level (set $\alpha$ = 0.01 or 0.02 for instance). All else being equal, this will increase the range of the confidence interval and thus increase the probability that it includes the true parameter.

Step #3: Concluding and interpreting the results

The final step is simply to compare the confidence interval (constructed in step #2) with the value of the target parameter (the value under the null hypothesis, mentioned in step #1):

If the confidence interval does not include the hypothesized value $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the confidence interval includes the hypothesized value $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

In our example:

the hypothesized value is 80 (since $H_0: \mu$ = 80)
80 is included in the 95% confidence interval since it goes from 61.70 to 80.30 kg
So we do not reject the null hypothesis

In the terms of the problem, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

As you can see, the conclusion is equivalent than with the critical value method (method A) and the p-value method (method B). Again, this must be the case since we use the same data and the same significance level $\alpha$ for all three methods.

Which method to choose?

All three methods give the same conclusion. However, each method has its own advantage so I usually select the most convenient one depending on the situation:

Method A (comparing the test statistic with the critical value):
- It is, in my opinion, the easiest and most straightforward method of the three when I do not have access to R.
Method B (comparing the p-value with the significance level $\alpha$):
- In addition to being able to know whether the null hypothesis is rejected or not, computing the exact p-value can be very convenient so I tend to use this method if I have access to R.
Method C (comparing the target parameter with the confidence interval):
- If I need to test several hypothesized values, I tend to choose this method because I can construct one single confidence interval and compare it to as many values as I want. For example, with our 95% confidence interval [61.70; 80.30], I know that any value below 61.70 kg and above 80.30 kg will be rejected, without testing it for each value.

Summary

In this article, we reviewed the goals and when hypothesis testing is used. We then showed how to do a hypothesis test by hand through three different methods (A. critical value, B. p-value and C. confidence interval). We also showed how to interpret the results in the context of the initial problem.

Although all three methods give the exact same conclusion when using the same data and the same significance level (otherwise there is a mistake somewhere), I also presented my personal preferences when it comes to choosing one method over the other two.

Thanks for reading.

I hope this article helped you to understand the structure of a hypothesis by hand. I remind you that, at least for the 6 hypothesis tests covered in this article, the formulas are different, but the structure and the reasoning behind it remain the same. So you basically have to know which formulas to use, and simply follow the steps mentioned in this article.

For the interested reader, I created two accompanying Shiny apps:

Hypothesis testing and confidence intervals: after entering your data, the app illustrates all the steps in order to conclude the test and compute a confidence interval. See more information in this article.
How to read statistical tables: the app helps you to compute the p-value given a t-stat for most probability distributions. See more information in this article.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Suppose a researcher wants to test whether Belgian women are taller than French women. Suppose a health professional would like to know whether the proportion of smokers is different among athletes and non-athletes. It would take way too long to measure the height of all Belgian and French women and to ask all athletes and non-athletes their smoking habits. So most of the time, decisions are based on a representative sample of the population and not on the whole population. If we could measure the entire population in a reasonable time frame, we would not do any inferential statistics.↩︎
Don’t get me wrong, this does not mean that hypothesis tests are never used in exploratory analyses. It is just much less frequent in exploratory research than in confirmatory research.↩︎
You may see more or less steps in other articles or textbooks, depending on whether these steps are detailed or concise. Hypothesis testing should, however, follows the same process regardless of the number of steps.↩︎
For one-sided tests, writing $H_0: \mu = 80$ or $H_0: \mu \ge 80$ are both correct. The point is that the null and alternative hypothesis must be mutually exclusive since you are testing one hypothesis against the other, so both cannot be true at the same time.↩︎
To be complete, there are even different formulas within each type of test, depending on whether some assumptions are met or not. For the interested reader, see all the different scenarios and thus the different formulas for a test on one mean and on two means.↩︎
There are more uncertainty if the population variance is unknown than if it is known, and this greater uncertainty is taken into account by using the Student distribution instead of the standard Normal distribution. Also note that as the sample size increases, the degrees of freedom of the Student distribution increases and the two distributions become more and more similar. For large sample size (usually from $n >$ 30), the Student distribution becomes so close to the standard Normal distribution that, even if the population variance is unknown, the standard Normal distribution can be used.↩︎
For a test on two independent samples, the degrees of freedom is $n_1 + n_2 - 2$, where $n_1$ and $n_2$ are the size of the first and second sample, respectively. Note the - 2 due to the fact that in this case, two quantities are estimated.↩︎
The type II error is the probability of not rejecting the null hypothesis although it is in reality false.↩︎
Whether this is a good or a bad standard is a question that comes up often and is debatable. This is, however, beyond the scope of the article.↩︎
Again, p-values found via a statistical table or via R must be coherent.↩︎

ANOVA in R

Mon, 12 Oct 2020 00:00:00 +0000

Introduction

ANOVA (ANalysis Of VAriance) is a statistical test to determine whether two or more population means are different. In other words, it is used to compare two or more groups to see if they are significantly different.

In practice, however, the:

Student t-test is used to compare 2 groups;
ANOVA generalizes the t-test beyond 2 groups, so it is used to compare 3 or more groups.

Note that there are several versions of the ANOVA (e.g., one-way ANOVA, two-way ANOVA, mixed ANOVA, repeated measures ANOVA, etc.). In this article, we present the simplest form only—the one-way ANOVA¹—and we refer to it as ANOVA in the remaining of the article.

Although ANOVA is used to make inference about means of different groups, the method is called “analysis of variance”. It is called like this because it compares the “between” variance (the variance between the different groups) and the variance “within” (the variance within each group). If the between variance is significantly larger than the within variance, the group means are declared to be different. Otherwise, we cannot conclude one way or the other. The two variances are compared to each other by taking the ratio ($\frac{variance_{between}}{variance_{within}}$) and then by comparing this ratio to a threshold from the Fisher probability distribution (a threshold based on a specific significance level, usually 5%).

This is enough theory regarding the ANOVA method for now. In the remaining of this article, we discuss about it from a more practical point of view, and in particular we will cover the following points:

the aim of the ANOVA, when it should be used and the null/alternative hypothesis
the underlying assumptions of the ANOVA and how to check them
how to perform the ANOVA in R
how to interpret results of the ANOVA
understand the notion of post-hoc test and interpret the results
how to visualize results of ANOVA and post-hoc tests

Data

Data for the present article is the penguins dataset (an alternative to the well-known iris dataset), accessible via the {palmerpenguins} package:

# install.packages("palmerpenguins")
library(palmerpenguins)

The dataset contains data for 344 penguins of 3 different species (Adelie, Chinstrap and Gentoo). The dataset contains 8 variables, but we focus only on the flipper length and the species for this article, so we keep only those 2 variables:

library(tidyverse)

dat <- penguins %>%
  select(species, flipper_length_mm)

(If you are unfamiliar with the pipe operator (%>%), you can also select variables with penguins[, c("species", "flipper_length_mm")]. Learn more ways to select variables in the article about data manipulation.)

Below some basic descriptive statistics and a plot (made with the {ggplot2} package) of our dataset before we proceed to the goal of the ANOVA:

summary(dat)

##       species    flipper_length_mm
##  Adelie   :152   Min.   :172.0    
##  Chinstrap: 68   1st Qu.:190.0    
##  Gentoo   :124   Median :197.0    
##                  Mean   :200.9    
##                  3rd Qu.:213.0    
##                  Max.   :231.0    
##                  NA's   :2

Flipper length varies from 172 to 231 mm, with a mean of 200.9 mm. There are respectively 152, 68 and 124 penguins of the species Adelie, Chinstrap and Gentoo.

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm, color = species) +
  geom_jitter() +
  theme(legend.position = "none")

Here, the factor is the species variable which contains 3 modalities or groups (Adelie, Chinstrap and Gentoo).

Aim and hypotheses of ANOVA

As mentioned in the introduction, the ANOVA is used to compare groups (in practice, 3 or more groups). More generally, it is used to:

study whether measurements are similar across different modalities (also called levels or treatments in the context of ANOVA) of a categorical variable
compare the impact of the different levels of a categorical variable on a quantitative variable
explain a quantitative variable based on a qualitative variable

In this context and as an example, we are going to use an ANOVA to help us answer the question: “Is the length of the flippers different between the 3 species of penguins?”.

The null and alternative hypothesis of an ANOVA are:

$H_0$: $\mu_{Adelie} = \mu_{Chinstrap} = \mu_{Gentoo}$ ($\Rightarrow$ the 3 species are equal in terms of flipper length)
$H_1$: at least one mean is different ($\Rightarrow$ at least one species is different from the other 2 species in terms of flipper length)

Be careful that the alternative hypothesis is not that all means are different. The opposite of all means being equal ($H_0$) is that at least one mean is different from the others ($H_1$).

In this sense, if the null hypothesis is rejected, it means that at least one species is different from the other 2, but not necessarily that all 3 species are different from each other. It could be that flipper length for the species Gentoo is different than for the species Chinstrap and Adelie, but flipper length is similar between Chinstrap and Adelie. Other types of test (known as post-hoc tests and covered in this section) must be performed to test whether all 3 species differ.

Underlying assumptions of ANOVA

As for many statistical tests, there are some assumptions that need to be met in order to be able to interpret the results. When one or several assumptions are not met, although it is technically possible to perform these tests, it would be incorrect to interpret the results and trust the conclusions.

Below are the assumptions of the ANOVA, how to test them and which other tests exist if an assumption is not met:

Variable type: ANOVA requires a mix of one continuous quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with at least 2 levels which will determine the groups to compare).
Independence: the data, collected from a representative and randomly selected portion of the total population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if three measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before, (ii) during and (iii) after a treatment), the repeated measures ANOVA should be preferred in order to take into account the dependency between the samples.
Normality:
- In case of small samples, residuals² should follow approximately a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. If, even after a transformation of your data (e.g., logarithmic transformation, square root, Box-Cox, etc.), the residuals still do not follow approximately a normal distribution, the Kruskal-Wallis test can be applied (kruskal.test(variable ~ group, data = dat in R). This non-parametric test, robust to non normal distributions, has the same goal than the ANOVA—compare 3 or more groups—but it uses sample medians instead of sample means to compare groups.
- In case of large samples, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013).³ It is therefore not required to test the normality assumption when the number of observations in each group/sample is large (usually $n \ge 30$).
Equality of variances: the variances of the different groups should be equal in the populations (an assumption called homogeneity of the variances, or even sometimes referred as homoscedasticity, as opposed to heteroscedasticity if variances are different across groups). This assumption can be tested graphically (by comparing the dispersion in a boxplot or dotplot for instance), or more formally via the Levene’s test (leveneTest(variable ~ group) from the {car} package) or Bartlett’s test, among others. If the hypothesis of equal variances is rejected, another version of the ANOVA can be used: the Welch ANOVA (oneway.test(variable ~ group, var.equal = FALSE)). Note that the Welch ANOVA does not require homogeneity of the variances, but the distributions should still follow approximately a normal distribution. Note that the Kruskal-Wallis test does not require the assumptions of normality nor homoscedasticity of the variances.⁴
Outliers: An outlier is a value or an observation that is distant from the other observations. There should be no significant outliers in the different groups, or the conclusions of your ANOVA may be flawed. There are several methods to detect outliers in your data but in order to deal with them, it is your choice to either:
- use the non-parametric version (i.e., the Kruskal-Wallis test)
- transform your data (logarithmic or Box-Cox transformation, among others)
- or remove them (be careful)

Choosing the appropriate test depending on whether assumptions are met may be confusing so here is a brief summary:

Check that your observations are independent.
Sample sizes:
- In case of small samples, test the normality of residuals:
  - If normality is assumed, test the homogeneity of the variances:
    - If variances are equal, use ANOVA.
    - If variances are not equal, use the Welch ANOVA.
  - If normality is not assumed, use the Kruskal-Wallis test.
- In case of large samples normality is assumed, so test the homogeneity of the variances:
  - If variances are equal, use ANOVA.
  - If variances are not equal, use the Welch ANOVA.

Now that we have seen the underlying assumptions of the ANOVA, we review them specifically for our dataset before applying the appropriate version of the test.

Variable type

The dependent variable flipper_length_mm is a quantitative variable and the independent variable species is a qualitative one (with 3 levels corresponding to the 3 species). So we have a mix of the two types of variable and this assumption is met.

Independence

Independence of the observations is assumed as data have been collected from a randomly selected portion of the population and measurements within and between the 3 samples are not related.

The independence assumption is most often verified based on the design of the experiment and on the good control of experimental conditions, as it is the case here.

If you really want to test it more formally, you can, however, test it via a statistical test—the Durbin-Watson test (in R: durbinWatsonTest(res_lm) where res_lm is a linear model). The null hypothesis of this test specifies an autocorrelation coefficient = 0, while the alternative hypothesis specifies an autocorrelation coefficient $\ne$ 0.

Normality

Since the smallest sample size per group (i.e., per species) is 68, we have large samples. Therefore, we do not need to check normality.

Usually, we would directly test the homogeneity of the variances without testing normality. However, for the sake of illustration, we act as if the sample sizes were small in order to illustrate what would need to be done in that case.

Remember that normality of residuals can be tested visually via a histogram and a QQ-plot, and/or formally via a normality test (Shapiro-Wilk test for instance).

Before checking the normality assumption, we first need to compute the ANOVA (more on that in this section). We then save the results in res_aov :

res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

We can now check normality visually:

par(mfrow = c(1, 2)) # combine plots

# histogram
hist(res_aov$residuals)

# QQ-plot
library(car)
qqPlot(res_aov$residuals,
  id = FALSE # id = FALSE to remove point identification
)

From the histogram and QQ-plot above, we can already see that the normality assumption seems to be met. Indeed, the histogram roughly form a bell curve, indicating that the residuals follow a normal distribution. Furthermore, points in the QQ-plots roughly follow the straight line and most of them are within the confidence bands, also indicating that residuals follow approximately a normal distribution.

Some researchers stop here and assume that normality is met, while others also test the assumption via a formal normality test. It is your choice to test it (i) only visually, (ii) only via a normality test, or (iii) both visually AND via a normality test. Bear in mind, however, the two following points:

ANOVA is quite robust to small deviations from normality. This means that it is not an issue (from the perspective of the interpretation of the ANOVA results) if a small number of points deviates slightly from the normality,
normality tests are sometimes quite conservative, meaning that the null hypothesis of normality may be rejected due to a limited deviation from normality. This is especially the case with large samples as power of the test increases with the sample size.

In practice, I tend to prefer the (i) visual approach only, but again, this is a matter of personal choice and also depends on the context of the analysis.

Still for the sake of illustration, we also now test the normality assumption via a normality test. You can use the Shapiro-Wilk test or the Kolmogorov-Smirnov test, among others.

Remember that the null and alternative hypothesis of these tests are:

$H_0$: data come from a normal distribution
$H_1$: data do not come from a normal distribution

In R, we can test normality of the residuals with the Shapiro-Wilk test thanks to the shapiro.test() function:

shapiro.test(res_aov$residuals)

## 
## 	Shapiro-Wilk normality test
## 
## data:  res_aov$residuals
## W = 0.99452, p-value = 0.2609

P-value of the Shapiro-Wilk test on the residuals is larger than the usual significance level of $\alpha = 5\%$, so we do not reject the hypothesis that residuals follow a normal distribution (p-value = 0.261).

This result is in line with the visual approach. In our case, the normality assumption is thus met both visually and formally.

Side note: Remind that the p-value is the probability of having observations as extreme as the ones we have observed in the sample(s) given that the null hypothesis is true. If the p-value $< \alpha$ (indicating that it is not likely to observe the data we have in the sample given that the null hypothesis is true), the null hypothesis is rejected, otherwise the null hypothesis is not rejected. See more about p-value and significance level if you are unfamiliar with those important statistical concepts.

Remember that if the normality assumption was not reached, some transformation(s) would need to be applied on the raw data in the hope that residuals would better fit a normal distribution, or you would need to use the non-parametric version of the ANOVA—the Kruskal-Wallis test.

As pointed out by a reader (see comments at the very end of the article), the normality assumption can also be tested on the “raw” data (i.e., the observations) instead of the residuals. However, if you test the normality assumption on the raw data, it must be tested for each group separately as the ANOVA requires normality in each group.

Testing normality on all residuals or on the observations per group is equivalent, and will give similar results. Indeed, saying “The distribution of Y within each group is normally distributed” is the same as saying “The residuals are normally distributed”.

Remember that residuals are the distance between the actual value of Y and the mean value of Y for a specific value of X, so the grouping variable is induced in the computation of the residuals.

So in summary, in ANOVA you actually have two options for testing normality:

Checking normality separately for each group on the “raw” data (Y values)
Checking normality on all residuals (but not per group)

In practice, you will see that it is often easier to just use the residuals and check them all together, especially if you have many groups or few observations per group.

If you are still not convinced: remember that an ANOVA is a special case of a linear model. Suppose your independent variable is a continuous variable (instead of a categorical variable), the only option you have left is to check normality on the residuals, which is precisely what is done for testing normality in linear regression models.

Equality of variances - homogeneity

Assuming residuals follow a normal distribution, it is now time to check whether the variances are equal across species or not. The result will have an impact on whether we use the ANOVA or the Welch ANOVA.

This can again be verified visually—via a boxplot or dotplot—or more formally via a statistical test (Levene’s test, among others).

Visually, we have:

# Boxplot
boxplot(flipper_length_mm ~ species,
  data = dat
)

# Dotplot
library("lattice")

dotplot(flipper_length_mm ~ species,
  data = dat
)

Both the boxplot and the dotplot show a similar variance for the different species. In the boxplot, this can be seen by the fact that the boxes and the whiskers have a comparable size for all species.

There are a couple of outliers as shown by the points outside the whiskers, but this does not change the fact that the dispersion is more or less the same between the different species.

In the dotplot, this can be seen by the fact that points for all 3 species have more or less the same range, a sign of the dispersion and thus the variance being similar.

Like the normality assumption, if you feel that the visual approach is not sufficient, you can formally test for equality of the variances with a Levene’s or Bartlett’s test. Notice that the Levene’s test is less sensitive to departures from normal distribution than the Bartlett’s test.

The null and alternative hypothesis for both tests are:

$H_0$: variances are equal
$H_1$: at least one variance is different

In R, the Levene’s test can be performed thanks to the leveneTest() function from the {car} package:

# Levene's test
library(car)

leveneTest(flipper_length_mm ~ species,
  data = dat
)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   2  0.3306 0.7188
##       339

The p-value being larger than the significance level of 0.05, we do not reject the null hypothesis, so we cannot reject the hypothesis that variances are equal between species (p-value = 0.719).

This result is also in line with the visual approach, so the homogeneity of variances is met both visually and formally.

Another method to test normality and homogeneity

For your information, it is also possible to test the homogeneity of the variances and the normality of the residuals visually (and both at the same time) via the plot() function:

par(mfrow = c(1, 2)) # combine plots

# 1. Homogeneity of variances
plot(res_aov, which = 3)

# 2. Normality
plot(res_aov, which = 2)

Plot on the left hand side shows that there is no evident relationships between residuals and fitted values (the mean of each group), so homogeneity of variances is assumed. If homogeneity of variances was violated, the red line would not be flat (horizontal).

Plot on the right hand side shows that residuals follow approximately a normal distribution, so normality is assumed. If normality was violated, points would consistently deviate from the dashed line.

Outliers

There are several techniques to detect outliers. In this article, we focus on the most simple one (yet very efficient)—the visual approach via a boxplot:

boxplot(flipper_length_mm ~ species,
  data = dat
)

There is one outlier in the group Adelie, as defined by the interquartile range criterion. This point is, however, not seen as a significant outlier so we can assume that the assumption of no significant outliers is met.

ANOVA

We showed that all assumptions of the ANOVA are met.

We can thus proceed to the implementation of the ANOVA in R, but first, let’s do some preliminary analyses to better understand the research question.

Preliminary analyses

A good practice before actually performing the ANOVA in R is to visualize the data in relation to the research question. The best way to do so is to draw and compare boxplots of the quantitative variable flipper_length_mm for each species.

This can be done with the boxplot() function in base R (same code than the visual check of equal variances):

boxplot(flipper_length_mm ~ species,
  data = dat
)

Or with the {ggplot2} package:

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm) +
  geom_boxplot()

The boxplots above show that, at least for our sample, penguins of the species Gentoo seem to have the biggest flipper, and Adelie species the smallest flipper.

Besides a boxplot for each species, it is also a good practice to compute some descriptive statistics such as the mean and standard deviation by species.

This can be done, for instance, with the aggregate() function:

aggregate(flipper_length_mm ~ species,
  data = dat,
  function(x) round(c(mean = mean(x), sd = sd(x)), 2)
)

##     species flipper_length_mm.mean flipper_length_mm.sd
## 1    Adelie                 189.95                 6.54
## 2 Chinstrap                 195.82                 7.13
## 3    Gentoo                 217.19                 6.48

or with the summarise() and group_by() functions from the {dplyr} package:

library(dplyr)

group_by(dat, species) %>%
  summarise(
    mean = mean(flipper_length_mm, na.rm = TRUE),
    sd = sd(flipper_length_mm, na.rm = TRUE)
  )

## # A tibble: 3 × 3
##   species    mean    sd
##   <fct>     <dbl> <dbl>
## 1 Adelie     190.  6.54
## 2 Chinstrap  196.  7.13
## 3 Gentoo     217.  6.48

Mean is also the lowest for Adelie and highest for Gentoo. Boxplots and descriptive statistics are, however, not enough to conclude that flippers are significantly different in the 3 populations of penguins.

ANOVA in R

As you guessed by now, only the ANOVA can help us to make inference about the population given the sample at hand, and help us to answer the initial research question “Is the length of the flippers different between the 3 species of penguins?”.

ANOVA in R can be done in several ways, of which two are presented below:

With the oneway.test() function:

# 1st method:
oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = TRUE # assuming equal variances
)

## 
## 	One-way analysis of means
## 
## data:  flipper_length_mm and species
## F = 594.8, num df = 2, denom df = 339, p-value < 2.2e-16

With the summary() and aov() functions:

# 2nd method:
res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

summary(res_aov)

##              Df Sum Sq Mean Sq F value Pr(>F)    
## species       2  52473   26237   594.8 <2e-16 ***
## Residuals   339  14953      44                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 2 observations deleted due to missingness

As you can see from the two outputs above, the test statistic (F = in the first method and F value in the second one) and the p-value (p-value in the first method and Pr(>F) in the second one) are exactly the same for both methods, which means that in case of equal variances, results and conclusions will be unchanged.

The advantage of the first method is that it is easy to switch from the ANOVA (used when variances are equal) to the Welch ANOVA (used when variances are unequal). This can be done by replacing var.equal = TRUE by var.equal = FALSE, as presented below:

oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = FALSE # assuming unequal variances
)

## 
## 	One-way analysis of means (not assuming equal variances)
## 
## data:  flipper_length_mm and species
## F = 614.01, num df = 2.00, denom df = 172.76, p-value < 2.2e-16

The advantage of the second method, however, is that:

the full ANOVA table (with degrees of freedom, mean squares, etc.) is printed, which may be of interest in some (theoritical) cases
results of the ANOVA (res_aov) can be saved for later use (especially useful for post-hoc tests)

Interpretations of ANOVA results

Given that the p-value is smaller than 0.05, we reject the null hypothesis, so we reject the hypothesis that all means are equal. Therefore, we can conclude that at least one species is different than the others in terms of flippers length (p-value < 2.2e-16).

(For the sake of illustration, if the p-value was larger than 0.05: we cannot reject the null hypothesis that all means are equal, so we cannot reject the hypothesis that the 3 considered species of penguins are equal in terms of flippers length.)

A nice and easy way to report results of an ANOVA in R is with the report() function from the {report} package:

# install.packages("remotes")
# remotes::install_github("easystats/report") # You only need to do that once
library("report") # Load the package every time you start R

report(res_aov)

## The ANOVA (formula: flipper_length_mm ~ species) suggests that:
## 
##   - The main effect of species is statistically significant and large (F(2, 339)
## = 594.80, p < .001; Eta2 = 0.78, 95% CI [0.75, 1.00])
## 
## Effect sizes were labelled following Field's (2013) recommendations.

As you can see, the function interprets the results for you and indicates a large and significant main effect of the species on the flipper length (p-value < .001).

Note that the report() function can be used for other analyses. See more tips and tricks in R if you find this one useful.

What’s next?

If the null hypothesis is not rejected (p-value $\ge$ 0.05), it means that we do not reject the hypothesis that all groups are equal. The ANOVA more or less stops here.

Other types of analyses can be performed of course, but—given the data at hand—we could not prove that at least one group was different so we usually do not go further with the ANOVA.

On the contrary, if the null hypothesis is rejected (as it is our case since the p-value < 0.05), we proved that at least one group is different. We can decide to stop here if we are only interested to test whether all species are equal in terms of flippers length.

But most of the time, when we showed thanks to an ANOVA that at least one group is different, we are also interested in knowing which one(s) is(are) different. Results of an ANOVA, however, do NOT tell us which group(s) is(are) different from the others.

To test this, we need to use other types of test, referred as post-hoc tests (in Latin, “after this”, so after obtaining statistically significant ANOVA results) or multiple pairwise-comparison tests.⁵

This family of statistical tests is the topic of the following sections.

Post-hoc test

Issue of multiple testing

In order to see which group(s) is(are) different from the others, we need to compare groups 2 by 2. In practice, since there are 3 species, we are going to compare species 2 by 2 as follows:

Chinstrap versus Adelie
Gentoo vs. Adelie
Gentoo vs. Chinstrap

In theory, we could compare species thanks to 3 Student’s t-tests since we need to compare 2 groups and a t-test is used precisely in that case.

However, if several t-tests are performed, the issue of multiple testing (also referred as multiplicity) arises. In short, when several statistical tests are performed, some will have p-values less than $\alpha$ purely by chance, even if all null hypotheses are in fact true.

To demonstrate the problem, consider our case where we have 3 hypotheses to test and a desired significance level of 0.05.

The probability of observing at least one significant result (at least one p-value < 0.05) just due to chance is:

\[\begin{equation} \begin{split} P(\text{at least 1 sig. result}) & = 1 - P(\text{no sig. results}) \\ & = 1 - (1 - 0.05)^3 \\ & = 0.142625 \end{split} \end{equation}\]

So, with as few as 3 tests being considered, we already have a 14.26% chance of observing at least one significant result, even if all of the tests are actually not significant.

And as the number of groups increases, the number of comparisons increases as well, so the probability of having a significant result simply due to chance keeps increasing.

For example, with 10 groups we need to make 45 comparisons and the probability of having at least one significant result by chance becomes $1 - (1 - 0.05)^{45} = 90\%$. So it is very likely to observe a significant result just by chance when comparing 10 groups, and when we have 14 groups or more we are almost certain (99%) to have a false positive!

Post-hoc tests take into account that multiple tests are done and deal with the problem by adjusting $\alpha$ in some way, so that the probability of observing at least one significant result due to chance remains below our desired significance level.⁶

Post-hoc tests in R and their interpretation

Post-hoc tests are a family of statistical tests so there are several of them. The most common ones are:

Tukey HSD, used to compare all groups to each other (so all possible comparisons of 2 groups).
Dunnett, used to make comparisons with a reference group. For example, consider 2 treatment groups and one control group. If you only want to compare the 2 treatment groups with respect to the control group, and you do not want to compare the 2 treatment groups to each other, the Dunnett’s test is preferred.
Bonferroni correction if one has a set of planned comparisons to do.

The Bonferroni correction is simple: you simply divide the desired global $\alpha$ level by the number of comparisons.

In our example, we have 3 comparisons so if we want to keep a global $\alpha = 0.05$, we have $\alpha' = \frac{0.05}{3} = 0.0167$. We can then simply perform a Student’s t-test for each comparison, and compare the obtained $p$-values with this new $\alpha'$.

The other two post-hoc tests are presented in the next sections.

Note that variances are assumed to be equal for all three methods (unless you use the Welch’s t-test instead of the Student’s t-test with the Bonferroni correction). If variances are not equal, you can use the Games-Howell test, among others.

Tukey HSD test

In our case, since there is no “reference” species and we are interested in comparing all species, we are going to use the Tukey HSD test.

In R, the Tukey HSD test is done as follows. This is where the second method to perform the ANOVA comes handy because the results (res_aov) are reused for the post-hoc test:

library(multcomp)

# Tukey HSD test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Tukey")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 1.03e-08 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-08 ***
## Gentoo - Chinstrap == 0  21.3635     1.0036  21.286  < 1e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

In the output of the Tukey HSD test, we are interested in the table displayed after Linear Hypotheses:, and more precisely, in the first and last column of the table. The first column shows the comparisons which have been made; the last column (Pr(>|t|)) shows the adjusted⁷ p-values for each comparison (with the null hypothesis being the two groups are equal and the alternative hypothesis being the two groups are different).

It is these adjusted p-values that are used to test whether two groups are significantly different or not, and we can be confident that the entire set of comparisons collectively has an error rate of 0.05.

In our example, we tested:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)
Gentoo vs. Chinstrap (line Gentoo - Chinstrap == 0)

All three ajusted p-values are smaller than 0.05, so we reject the null hypothesis for all comparisons, which means that all species are significantly different in terms of flippers length.

The results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that all groups are significantly different.

Note that the Tukey HSD test can also be done in R with the TukeyHSD() function:

TukeyHSD(res_aov)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## $species
##                       diff       lwr       upr p adj
## Chinstrap-Adelie  5.869887  3.586583  8.153191     0
## Gentoo-Adelie    27.233349 25.334376 29.132323     0
## Gentoo-Chinstrap 21.363462 19.000841 23.726084     0

With this code, it is the column p adj (also the last column) which is of interest. Notice that the conclusions are the same than above: all species are significantly different in terms of flippers length.

The results can also be visualized with the plot() function:

plot(TukeyHSD(res_aov))

Dunnett’s test

We have seen in this section that as the number of groups increases, the number of comparisons also increases. And as the number of comparisons increases, the post-hoc analysis must lower the individual significance level even further, which leads to lower statistical power (so a difference between group means in the population is less likely to be detected).

One method to mitigate this and increase the statistical power is by reducing the number of comparisons. This reduction allows the post-hoc procedure to use a larger individual error rate to achieve the desired global error rate.

While comparing all possible groups with a Tukey HSD test is a common approach, many studies have a control group and several treatment groups. For these studies, you may need to compare the treatment groups only to the control group, which reduces the number of comparisons.

Dunnett’s test does precisely this—it only compares a group taken as reference to all other groups, but it does not compare all groups to each others.

So to recap:

the Tukey HSD test allows to compares all groups but at the cost of less power
the Dunnett’s test allows to only make comparisons with a reference group, but with the benefit of more power

Now, again for the sake of illustration, consider that the species Adelie is the reference species and we are only interested in comparing the reference species against the other 2 species. In that scenario, we would use the Dunnett’s test.

In R, the Dunnett’s test is done as follows (the only difference with the code for the Tukey HSD test is in the line linfct = mcp(species = "Dunnett")):

library(multcomp)

# Dunnett's test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 7.59e-09 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

The interpretation is the same as for the Tukey HSD test’s except that in the Dunett’s test we only compare:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)

Both adjusted p-values (displayed in the last column) are below 0.05, so we reject the null hypothesis for both comparisons.

This means that both the species Chinstrap and Gentoo are significantly different from the reference species Adelie in terms of flippers length. (Nothing can be said about the comparison between Chinstrap and Gentoo though.)

Again, the results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that both the species Gentoo and Chinstrap are significantly different from the reference species Adelie.

Note that in R, by default, the reference category for a factor variable is the first category in alphabetical order. This is the reason that, by default, the reference species is Adelie.

The reference category can be changed with the relevel() function (or with the {questionr} addin). Considering that we want Gentoo as the reference category instead of Adelie:

# Change reference category:
dat$species <- relevel(dat$species, ref = "Gentoo")

# Check that Gentoo is the reference category:
levels(dat$species)

## [1] "Gentoo"    "Adelie"    "Chinstrap"

Gentoo now being the first category of the three, it is indeed considered as the reference level.

In order to perform the Dunnett’s test with the new reference we first need to rerun the ANOVA to take into account the new reference:

res_aov2 <- aov(flipper_length_mm ~ species,
  data = dat
)

We can then run the Dunett’s test with the new results of the ANOVA:

# Dunnett's test:
post_test <- glht(res_aov2,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Adelie - Gentoo == 0    -27.2333     0.8067  -33.76   <1e-10 ***
## Chinstrap - Gentoo == 0 -21.3635     1.0036  -21.29   <1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

par(mar = c(3, 8, 3, 3))
plot(post_test)

From the results above we conclude that Adelie and Chinstrap species are significantly different from Gentoo species in terms of flippers length (adjusted p-values < 1e-10).

Note that even if your study does not have a reference group which you can compare to the other groups, it is still often better to do multiple comparisons determined by some research questions than to do all-pairwise tests. By reducing the number of post-hoc comparisons to what is necessary only, and no more, you maximize the statistical power.⁸

Other p-values adjustment methods

For the interested readers, note that you can use other p-values adjustment methods by using the pairwise.t.test() function:

pairwise.t.test(dat$flipper_length_mm, dat$species,
  p.adjust.method = "holm"
)

## 
## 	Pairwise comparisons using t tests with pooled SD 
## 
## data:  dat$flipper_length_mm and dat$species 
## 
##           Gentoo  Adelie 
## Adelie    < 2e-16 -      
## Chinstrap < 2e-16 3.8e-09
## 
## P value adjustment method: holm

By default, the Holm method is applied but other methods exist. See ?p.adjust for all available options.

Visualization of ANOVA and post-hoc tests on the same plot

If you are interested in including results of ANOVA and post-hoc tests on the same plot (directly on the boxplots), here are two pieces of code which may be of interest to you.

The first one is edited by me based on the code found in this article:

# Edit from here
x <- which(names(dat) == "species") # name of grouping variable
y <- which(
  names(dat) == "flipper_length_mm" # names of variables to test
)
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("Chinstrap", "Adelie"), c("Gentoo", "Adelie"), c("Gentoo", "Chinstrap")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
library(ggpubr)
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

And the second method is from the {ggstatsplot} package:

library(ggstatsplot)

ggbetweenstats(
  data = dat,
  x = species,
  y = flipper_length_mm,
  type = "parametric", # ANOVA or Kruskal-Wallis
  var.equal = TRUE, # ANOVA or Welch ANOVA
  plot.type = "box",
  pairwise.comparisons = TRUE,
  pairwise.display = "significant",
  centrality.plotting = FALSE,
  bf.message = FALSE
)

As you can see on the above plot, boxplots by species are presented together with p-values of the ANOVA (after p = in the subtitle of the plot) and p-values of the post-hoc tests (above each comparison).

Besides the fact that these methods can be used to combine a visual representation and statistical results on the same plot, they also have the advantage that you can perform multiple ANOVA tests at once. See more information in this article.

Summary

In this article, we reviewed the goals and hypotheses of an ANOVA, what are the assumptions which need to be verified before being able to trust the results (namely, independence, normality and homogeneity), we then showed how to do an ANOVA in R and how to interpret the results.

An article about ANOVA would not be complete without discussing about post-hoc tests, and in particular, the Tukey HSD—to compare all groups—and the Dunnett’s test—to compare a reference group to all other groups.

Last but not least, we showed how to visualize the data and the results of the ANOVA and post-hoc tests in the same plot.

Thanks for reading. See this tutorial if you would like to learn how to do an ANOVA by hand.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Hsu, Jason. 1996. Multiple Comparisons: Theory and Methods. CRC Press.

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Note that it is called one-way or one-factor ANOVA because the means relate to the different modalities of a single independent variable, or factor.↩︎
Residuals (denoted $\epsilon$) are the differences between the observed values of the dependent variable ($y$) and the predicted values ($\hat{y}$). In the context of ANOVA, residuals correspond to the differences between the observed values and the mean of all values for that group.↩︎
Stevens (2013) wrote, in p. 57, “Numerous studies have examined the effect of violations of assumptions in ANOVA, and an excellent summary of this literature has been provided by Glass, Peckham, and Sanders (1972). Their review indicates that non normality has only a slight effect on the type I error rate, even for very skewed or kurtotic distributions. For example, the actual $\alpha$s for some very non-normal populations were only .055 or .06: very minor deviations from the nominal level of .05. […] The basic reason is the Central Limit Theorem, which states that the sum of independent observations having any distribution whatsoever approaches a normal distribution as the number of observations increases. To be somewhat more specific, Bock (1975) notes,”even for distributions which depart markedly from normality, sums of 50 or more observations approximate to normality. For moderately non-normal distributions the approximation is good with as few as 10 to 20 observations” (p. 111). Now since the sums of independent observations approach normality rapidly, so do the means, and the sampling distribution of F is based on means. Thus the sampling distribution of F is only slightly affected, and therefore the critical values when sampling from normal and non-normal distributions will not differ by much. Lack of normality due to skewness also has only a slight effect on power (a few hundredths).”↩︎
As long as you use the Kruskal-Wallis test to, in fine, compare groups, homoscedasticity is not required. If you wish to compare medians, the Kruskal-Wallis test requires homoscedasticity. See more information about the difference in this article.↩︎
Note that, as discussed in the comments at the end of the article, post-hoc tests can under some circumstances be done directly (without an ANOVA). See the comments or Hsu (1996) for more details.↩︎
Note that you could in principle apply the Bonferroni correction to all tests. For example, in the example above, with 3 tests and a global desired significance level of $\alpha$ = 0.05, we would only reject a null hypothesis if the p-value is less than $\frac{0.05}{3}$ = 0.0167. This method is, however, known to be quite conservative, leading to a potentially high rate of false negatives.↩︎
The p-values are adjusted to keep the global significance level to the desired level.↩︎
Thanks Michael Friendly for this suggestion.↩︎

Correlation coefficient and correlation test in R

Thu, 28 May 2020 00:00:00 +0000

Introduction

Correlations between variables play an important role in a descriptive analysis. A correlation measures the relationship between two variables, that is, how they are linked to each other. In this sense, a correlation allows to know which variables evolve in the same direction, which ones evolve in the opposite direction, and which ones are independent.

In this article, I show how to compute correlation coefficients, how to perform correlation tests and how to visualize relationships between variables in R.

Correlation is usually computed on two quantitative variables, but it can also be computed on two qualitative ordinal variables.¹ See the Chi-square test of independence if you need to study the relationship between two qualitative nominal variables.

If you need to quantify the relationship between two variables, I refer you to the article about linear regression.

Data

In this article, we use the mtcars dataset (loaded by default in R):

# display first 5 observations
head(mtcars, 5)

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2

The variables vs and am are categorical variables, so they are removed for this article:

# remove vs and am variables
library(tidyverse)
dat <- mtcars %>%
  select(-vs, -am)

# display 5 first obs. of new dataset
head(dat, 5)

##                    mpg cyl disp  hp drat    wt  qsec gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02    3    2

Correlation coefficient

Between two variables

The correlation between 2 variables is found with the cor() function.

Suppose we want to compute the correlation between horsepower (hp) and miles per gallon (mpg):

# Pearson correlation between 2 variables
cor(dat$hp, dat$mpg)

## [1] -0.7761684

Note that the correlation between variables X and Y is equal to the correlation between variables Y and X so the order of the variables in the cor() function does not matter.

The Pearson correlation is computed by default with the cor() function. If you want to compute the Spearman correlation, add the argument method = "spearman" to the cor() function:

# Spearman correlation between 2 variables
cor(dat$hp, dat$mpg,
  method = "spearman"
)

## [1] -0.8946646

The most common correlation methods (Run ?cor for more information about the different methods available in the cor() function) are:

Pearson correlation is often used for quantitative continuous variables that have a linear relationship
Spearman correlation (which is actually similar to Pearson but based on the ranked values for each variable rather than on the raw data) is often used to evaluate relationships involving at least one qualitative ordinal variable or two quantitative variables if the link is partially linear
Kendall’s tau-b which is computed from the number of concordant and discordant pairs is often used for qualitative ordinal variables

Note that there exists the point-biserial correlation (which can be used to measure the association between a continuous variable and a nominal variable of two levels), but this correlation is not covered here.

Correlation matrix: correlations for all variables

Suppose now that we want to compute correlations for several pairs of variables. We can easily do so for all possible pairs of variables in the dataset, again with the cor() function:

# correlation for all variables
round(cor(dat),
  digits = 2 # rounded to 2 decimals
)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg   1.00 -0.85 -0.85 -0.78  0.68 -0.87  0.42  0.48 -0.55
## cyl  -0.85  1.00  0.90  0.83 -0.70  0.78 -0.59 -0.49  0.53
## disp -0.85  0.90  1.00  0.79 -0.71  0.89 -0.43 -0.56  0.39
## hp   -0.78  0.83  0.79  1.00 -0.45  0.66 -0.71 -0.13  0.75
## drat  0.68 -0.70 -0.71 -0.45  1.00 -0.71  0.09  0.70 -0.09
## wt   -0.87  0.78  0.89  0.66 -0.71  1.00 -0.17 -0.58  0.43
## qsec  0.42 -0.59 -0.43 -0.71  0.09 -0.17  1.00 -0.21 -0.66
## gear  0.48 -0.49 -0.56 -0.13  0.70 -0.58 -0.21  1.00  0.27
## carb -0.55  0.53  0.39  0.75 -0.09  0.43 -0.66  0.27  1.00

This correlation matrix gives an overview of the correlations for all combinations of two variables.

Interpretation of a correlation coefficient

First of all, correlation ranges from -1 to 1. It gives us an indication on two things:

The direction of the relationship between the 2 variables
The strength of the relationship between the 2 variables

Regarding the direction of the relationship: On the one hand, a negative correlation implies that the two variables under consideration vary in opposite directions, that is, if a variable increases the other decreases and vice versa. On the other hand, a positive correlation implies that the two variables under consideration vary in the same direction, i.e., if a variable increases the other one increases and if one decreases the other one decreases as well.

Regarding the strength of the relationship: The more extreme the correlation coefficient (the closer to -1 or 1), the stronger the relationship. This also means that a correlation close to 0 indicates that the two variables are independent, that is, as one variable increases, there is no tendency in the other variable to either decrease or increase.

As an illustration, the Pearson correlation between horsepower (hp) and miles per gallon (mpg) found above is -0.78, meaning that the 2 variables vary in opposite direction. This makes sense, cars with more horsepower tend to consume more fuel (and thus have a lower mileage per gallon). On the contrary, from the correlation matrix we see that the correlation between miles per gallon (mpg) and the time to drive 1/4 of a mile (qsec) is 0.42, meaning that fast cars (low qsec) tend to have a worse mileage per gallon (low mpg). This again makes sense as fast cars tend to consume more fuel.

Note that it is a good practice to visualize the type of the relationship between the two variables before interpreting the correlation coefficients. The reason is that the correlation coefficient could be biased due to an outlier or due to the type of link between the two variables.

For instance, see the two Pearson correlation coefficients (denoted by R in the following plots) when the outlier is excluded and included:

The Pearson correlation coefficient changes drastically due to a single point, and thus the interpretation. It goes from a negative correlation coefficient, indicating a negative relationship between the 2 variables, to a positive coefficient, indicating a positive relationship. We would have missed this insight if we had not visualized the data in a scatterplot (see how to draw a scatterplot in this section).

A correlation coefficient may also miss a non-linear link between two variables:

The Pearson correlation coefficient is equal to 0, indicating no relationship between the two variables, because it measures the linear relationship and it is clear from the plot that the link is non-linear.

So to recap, it is a good practice to visualize the data via a scatterplot before interpreting a correlation coefficient (it does not tell the whole story) and see how the correlation coefficient changes when using the parametric (Pearson) or nonparametric version (Spearman or Kendall’s tau-b).

Visualizations

The correlation matrix presented above is not easily interpretable, especially when the dataset is composed of many variables. In the following sections, we present some alternatives to the correlation matrix for better readability.

A scatterplot for 2 variables

A good way to visualize a correlation between 2 variables is to draw a scatterplot of the two variables of interest. Suppose we want to examine the relationship between horsepower (hp) and miles per gallon (mpg):

# scatterplot
library(ggplot2)

ggplot(dat) +
  aes(x = hp, y = mpg) +
  geom_point(colour = "#0c4c8a") +
  theme_minimal()

If you are unfamiliar with the {ggplot2} package, you can draw the scatterplot using the plot() function from R base graphics:

plot(dat$hp, dat$mpg)

or use the esquisse addin to easily draw plots using the {ggplot2} package.

Scatterplots for several pairs of variables

Suppose that instead of visualizing the relationship between only 2 variables, we want to visualize the relationship for several pairs of variables. This is possible thanks to the pair() function.

For this illustration, we focus only on miles per gallon (mpg), horsepower (hp) and weight (wt):

# multiple scatterplots
pairs(dat[, c("mpg", "hp", "wt")])

The figure indicates that weight (wt) and horsepower (hp) are positively correlated, whereas miles per gallon (mpg) seems to be negatively correlated with horsepower (hp) and weight (wt).

Another simple correlation matrix

This version of the correlation matrix presents the correlation coefficients in a slightly more readable way, i.e., by coloring the coefficients based on their sign. Applied to our dataset, we have:

# improved correlation matrix
library(corrplot)

corrplot(cor(dat),
  method = "number",
  type = "upper" # show only upper side
)

Correlation test

For 2 variables

Unlike a correlation matrix which indicates the correlation coefficients between some pairs of variables in the sample, a correlation test is used to test whether the correlation (denoted $\rho$) between 2 variables is significantly different from 0 or not in the population.

Actually, a correlation coefficient different from 0 in the sample does not mean that the correlation is significantly different from 0 in the population. This needs to be tested with a hypothesis test—and known as the correlation test.

The null and alternative hypothesis for the correlation test are as follows:

$H_0$: $\rho = 0$ (meaning that there is no linear relationship between the two variables)
$H_1$: $\rho \ne 0$ (meaning that there is a linear relationship between the two variables)

Via this correlation test, what we are actually testing is whether:

the sample contains sufficient evidence to reject the null hypothesis and conclude that the correlation coefficient does not equal 0, so the relationship exists in the population.
or on the contrary, the sample does not contain enough evidence that the correlation coefficient does not equal 0, so in this case we do not reject the null hypothesis of no relationship between the variables in the population.

Note that there are 2 assumptions for this test to be valid:

Independence of the data
For small sample sizes (usually $n < 30$), the two variables should follow a normal distribution

Suppose that we want to test whether the rear axle ratio (drat) is correlated with the time to drive a quarter of a mile (qsec):

# Pearson correlation test
test <- cor.test(dat$drat, dat$qsec)
test

## 
## 	Pearson's product-moment correlation
## 
## data:  dat$drat and dat$qsec
## t = 0.50164, df = 30, p-value = 0.6196
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.265947  0.426340
## sample estimates:
##        cor 
## 0.09120476

The p-value of the correlation test between these 2 variables is 0.62. At the 5% significance level, we do not reject the null hypothesis of no correlation. We therefore conclude that we do not reject the hypothesis that there is no linear relationship between the 2 variables.²

This test proves that even if the correlation coefficient is different from 0 (the correlation is 0.09 in the sample), it is actually not significantly different from 0 in the population.

Note that the p-value of a correlation test is based on the correlation coefficient and the sample size. The larger the sample size and the more extreme the correlation (closer to -1 or 1), the more likely the null hypothesis of no correlation will be rejected.

With a small sample size, it is thus possible to obtain a relatively large correlation in the sample (based on the correlation coefficient), but still find a correlation not significantly different from 0 in the population (based on the correlation test). For this reason, it is recommended to always perform a correlation test before interpreting a correlation coefficient to avoid flawed conclusions.

For several pairs of variables

Similar to the correlation matrix used to compute correlation for several pairs of variables, the rcorr() function (from the {Hmisc} package) allows to compute p-values of the correlation test for several pairs of variables at once. Applied to our dataset, we have:

# correlation tests for whole dataset
library(Hmisc)
res <- rcorr(as.matrix(dat)) # rcorr() accepts matrices only

# display p-values (rounded to 3 decimals)
round(res$P, 3)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg     NA 0.000 0.000 0.000 0.000 0.000 0.017 0.005 0.001
## cyl  0.000    NA 0.000 0.000 0.000 0.000 0.000 0.004 0.002
## disp 0.000 0.000    NA 0.000 0.000 0.000 0.013 0.001 0.025
## hp   0.000 0.000 0.000    NA 0.010 0.000 0.000 0.493 0.000
## drat 0.000 0.000 0.000 0.010    NA 0.000 0.620 0.000 0.621
## wt   0.000 0.000 0.000 0.000 0.000    NA 0.339 0.000 0.015
## qsec 0.017 0.000 0.013 0.000 0.620 0.339    NA 0.243 0.000
## gear 0.005 0.004 0.001 0.493 0.000 0.000 0.243    NA 0.129
## carb 0.001 0.002 0.025 0.000 0.621 0.015 0.000 0.129    NA

Only correlations with p-values smaller than the significance level (usually $\alpha = 0.05$) should be interpreted.

Combination of correlation coefficients and correlation tests

Now that we covered the concepts of correlation coefficients and correlation tests, let see if we can combine the two concepts.

If you need to do this for a few pairs of variables, I recommend using the ggscatterstats() function from the {ggstatsplot} package. Let’s see it in practice with one pair of variables—wt and mpg:

## plot with statistical results
library(ggstatsplot)

ggscatterstats(
  data = dat,
  x = wt,
  y = mpg,
  bf.message = FALSE,
  marginal = FALSE # remove histograms
)

Based on the result of the test, we conclude that there is a negative correlation between the weight and the number of miles per gallon ($r = - 0.87$, $p$-value < 0.001).

If you need to do it for many pairs of variables, I recommend using the the correlation function from the easystats {correlation} package.

This function allows to combine correlation coefficients and correlation tests for several pairs of variables, all in a single table (thanks to krzysiektr for pointing it out to me):

library(correlation)

correlation::correlation(dat,
  include_factors = TRUE, method = "auto"
)

## # Correlation Matrix (auto-method)
## 
## Parameter1 | Parameter2 |     r |         95% CI | t(30) |         p
## --------------------------------------------------------------------
## mpg        |        cyl | -0.85 | [-0.93, -0.72] | -8.92 | < .001***
## mpg        |       disp | -0.85 | [-0.92, -0.71] | -8.75 | < .001***
## mpg        |         hp | -0.78 | [-0.89, -0.59] | -6.74 | < .001***
## mpg        |       drat |  0.68 | [ 0.44,  0.83] |  5.10 | < .001***
## mpg        |         wt | -0.87 | [-0.93, -0.74] | -9.56 | < .001***
## mpg        |       qsec |  0.42 | [ 0.08,  0.67] |  2.53 | 0.137    
## mpg        |       gear |  0.48 | [ 0.16,  0.71] |  3.00 | 0.065    
## mpg        |       carb | -0.55 | [-0.75, -0.25] | -3.62 | 0.016*   
## cyl        |       disp |  0.90 | [ 0.81,  0.95] | 11.45 | < .001***
## cyl        |         hp |  0.83 | [ 0.68,  0.92] |  8.23 | < .001***
## cyl        |       drat | -0.70 | [-0.84, -0.46] | -5.37 | < .001***
## cyl        |         wt |  0.78 | [ 0.60,  0.89] |  6.88 | < .001***
## cyl        |       qsec | -0.59 | [-0.78, -0.31] | -4.02 | 0.007**  
## cyl        |       gear | -0.49 | [-0.72, -0.17] | -3.10 | 0.054    
## cyl        |       carb |  0.53 | [ 0.22,  0.74] |  3.40 | 0.027*   
## disp       |         hp |  0.79 | [ 0.61,  0.89] |  7.08 | < .001***
## disp       |       drat | -0.71 | [-0.85, -0.48] | -5.53 | < .001***
## disp       |         wt |  0.89 | [ 0.78,  0.94] | 10.58 | < .001***
## disp       |       qsec | -0.43 | [-0.68, -0.10] | -2.64 | 0.131    
## disp       |       gear | -0.56 | [-0.76, -0.26] | -3.66 | 0.015*   
## disp       |       carb |  0.39 | [ 0.05,  0.65] |  2.35 | 0.177    
## hp         |       drat | -0.45 | [-0.69, -0.12] | -2.75 | 0.110    
## hp         |         wt |  0.66 | [ 0.40,  0.82] |  4.80 | < .001***
## hp         |       qsec | -0.71 | [-0.85, -0.48] | -5.49 | < .001***
## hp         |       gear | -0.13 | [-0.45,  0.23] | -0.69 | > .999   
## hp         |       carb |  0.75 | [ 0.54,  0.87] |  6.21 | < .001***
## drat       |         wt | -0.71 | [-0.85, -0.48] | -5.56 | < .001***
## drat       |       qsec |  0.09 | [-0.27,  0.43] |  0.50 | > .999   
## drat       |       gear |  0.70 | [ 0.46,  0.84] |  5.36 | < .001***
## drat       |       carb | -0.09 | [-0.43,  0.27] | -0.50 | > .999   
## wt         |       qsec | -0.17 | [-0.49,  0.19] | -0.97 | > .999   
## wt         |       gear | -0.58 | [-0.77, -0.29] | -3.93 | 0.008**  
## wt         |       carb |  0.43 | [ 0.09,  0.68] |  2.59 | 0.132    
## qsec       |       gear | -0.21 | [-0.52,  0.15] | -1.19 | > .999   
## qsec       |       carb | -0.66 | [-0.82, -0.40] | -4.76 | < .001***
## gear       |       carb |  0.27 | [-0.08,  0.57] |  1.56 | 0.774    
## 
## p-value adjustment method: Holm (1979)
## Observations: 32

As you can see, it gives, among other useful information, the correlation coefficients (column r) and the result of the correlation test (column 95% CI for the confidence interval or p for the $p$-value) for all pairs of variables.

Correlograms

The table above is very useful and informative, but let see if it is possible to combine the concepts of correlation coefficients and correlations test in one single visualization. A visualization that would be easy to read and interpret.

Ideally, we would like to have a concise overview of correlations between all possible pairs of variables present in a dataset, with a clear distinction for correlations that are significantly different from 0.

The figure below, known as a correlogram and adapted from the corrplot() function, does precisely this:

# do not edit
corrplot2 <- function(data,
                      method = "pearson",
                      sig.level = 0.05,
                      order = "original",
                      diag = FALSE,
                      type = "upper",
                      tl.srt = 90,
                      number.font = 1,
                      number.cex = 1,
                      mar = c(0, 0, 0, 0)) {
  library(corrplot)
  data_incomplete <- data
  data <- data[complete.cases(data), ]
  mat <- cor(data, method = method)
  cor.mtest <- function(mat, method) {
    mat <- as.matrix(mat)
    n <- ncol(mat)
    p.mat <- matrix(NA, n, n)
    diag(p.mat) <- 0
    for (i in 1:(n - 1)) {
      for (j in (i + 1):n) {
        tmp <- cor.test(mat[, i], mat[, j], method = method)
        p.mat[i, j] <- p.mat[j, i] <- tmp$p.value
      }
    }
    colnames(p.mat) <- rownames(p.mat) <- colnames(mat)
    p.mat
  }
  p.mat <- cor.mtest(data, method = method)
  col <- colorRampPalette(c("#BB4444", "#EE9988", "#FFFFFF", "#77AADD", "#4477AA"))
  corrplot(mat,
    method = "color", col = col(200), number.font = number.font,
    mar = mar, number.cex = number.cex,
    type = type, order = order,
    addCoef.col = "black", # add correlation coefficient
    tl.col = "black", tl.srt = tl.srt, # rotation of text labels
    # combine with significance level
    p.mat = p.mat, sig.level = sig.level, insig = "blank",
    # hide correlation coefficients on the diagonal
    diag = diag
  )
}

# edit from here
corrplot2(
  data = dat,
  method = "pearson",
  sig.level = 0.05,
  order = "original",
  diag = FALSE,
  type = "upper",
  tl.srt = 75
)

The correlogram shows correlation coefficients for all pairs of variables (with more intense colors for more extreme correlations), and correlations not significantly different from 0 are represented by a white box.

To learn more about this plot and the code used, I invite you to read the article entitled “Correlogram in R: how to highlight the most correlated variables in a dataset”.

For those of you who are still not completely satisfied, I recently found two alternatives—one with the ggpairs() function from the {GGally} package and one with the ggcormat() function from the {ggstatsplot} package.

The two functions are illustrated with the variables mpg, hp and wt:

library(GGally)

ggpairs(dat[, c("mpg", "hp", "wt")])

The plot above combines correlation coefficients, correlation tests (via the asterisks next to the coefficients³) and scatterplots for all possible pairs of variables present in a dataset.

library(ggstatsplot)

ggcorrmat(
  data = dat[, c("mpg", "hp", "wt")],
  type = "parametric", # parametric for Pearson, nonparametric for Spearman's correlation
  colors = c("darkred", "white", "steelblue") # change default colors
)

The plot above also shows the correlation coefficients and if any, the non-significant correlations (by default at the 5% significance level with the Holm adjustment method) are shown by a big cross on the correlation coefficients.

The advantage of these two alternatives compared to the first one is that it is directly available within a package, so you do not need to run the code of the function first in order to draw the correlogram.

Correlation does not imply causation

I am pretty sure you have already heard the statement “Correlation does not imply causation” in statistics. An article about correlation would not be complete without discussing about causation.

A non-zero correlation between two variables does not necessarily mean that there is a cause and effect relationship between these two variables!

Indeed, a significant correlation between two variables means that changes in one variable are associated (positively or negatively) with changes in the other variable. Nonetheless, a significant correlation does not indicate that variations in one variable cause the variations in the other variable.

A non-zero correlation between X and Y can appear in several cases:

X causes Y
Y causes X
a third variable causes X and Y
a combination of these three reasons

Sometimes it is quite clear that there is a causal relationship between two variables. Take for example the correlation between the price of a consumer product such as milk and its consumption. It is quite obvious that there is a causal link between the two: if the price of milk increases, it is expected that its consumption will decrease.

However, this causal link is not always present even if the correlation is significant. Maurage et al. (2013) showed that, although there is a positive and significant correlation between chocolate consumption and the number of Nobel laureates, this correlation comes from the fact that a third variable, Gross Domestic Product (GDP), causes chocolate consumption and the number of Nobel laureates. They found that countries with higher GDP tend to have a higher level of chocolate consumption and scientific research (leading to more Nobel laureates).

This example shows that one must be very cautious when interpreting correlations and avoid over-interpreting a correlation as a causal relationship.

Conclusion

Thanks for reading.

I hope this article helped you to compute correlation coefficients and perform correlation tests in R. If you would like to learn how to compute the coefficients by hand, see this step-by-step tutorial.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Maurage, Pierre, Alexandre Heeren, and Mauro Pesenti. 2013. “Does Chocolate Consumption Really Boost Nobel Award Chances? The Peril of over-Interpreting Correlations in Health Studies.” The Journal of Nutrition 143 (6): 931–33.

It is true that there is the point-biserial correlation which can be used with a nominal variable (consisting of two factors). Nonetheless, this type of correlation is much less known and usually not covered in introductory statistics classes; with one continuous and one nominal variable, it is much more frequent to learn about the Student’s t-test (for a nominal variable with 2 groups) or ANOVA (for a nominal variable with 3 or more groups). More information about choosing the most appropriate measure of association depending on the type of variable can be found in this article.↩︎
It is important to remember that we tested for a linear relationship between the two variables since we used the Pearson’s correlation. It may be the case that there is a relationship between the two variables in the population, but this relation may not be linear.↩︎
One asterisk means that the coefficient is significant at the 5% level, 2 is at the 1% significance level, and 3 is at the 0.1% significance level. This is usually the case in R; the more asterisks, the more it is significant.↩︎

How to upload your R code on GitHub? An example with an R script on MacOS

Sun, 24 May 2020 00:00:00 +0000

Introduction

A few days ago, a colleague asked me how to upload some R code on GitHub in order to make it accessible to everyone. Due to the lockdown, I could not just go into his office and show him on his computer. So I sent him several screenshots showing, step by step, how to do so.

Right before I deleted the screenshots I’d just taken, I thought that perhaps they would be useful for other persons, so I wrote this article.

Note 1: The screenshots have been taken on MacOS and I have not tested it on Windows. Do not hesitate to let me know in the comments whether it is similar or not on other operating systems.

Note 2: There must be other ways to do it, but the method shown below is (in my opinion) easy and works well.

Prerequisite

In order to follow this guide and upload your R code on GitHub, you will need at least:

a GitHub account
the GitHub Desktop application installed on your computer

Step-by-step guide

For this guide, I use a R script created to plot the evolution of the hospital admissions due to COVID-19 in Belgium. See the repository on GitHub in case you want to see the final result before proceeding further.

You often hear

“A picture is worth a thousand words”

So here is without further ado how to upload your R script on GitHub in images:

Step 1: Go to github.com/login and sign in

Step 2: Go to your GitHub profile and create a new repository

Step 3: Set the name, description, audience and README file for the new repository

Be careful that if you choose to make it public, it will be visible to everyone. If you do not want to share the code, but still want it to be uploaded on GitHub, choose the private option.

Initializing the repo with a README file is not mandatory, but I strongly recommend it so you will be able to add information for this repo.

Step 4: You now see your new repository with only the README file

Step 5: Open the GitHub Desktop application and sign in with your GitHub account

Step 6 a: Clone the repository you just created on github.com to your computer

Step 6 b: Clone the repository you just created on github.com to your computer

Step 6 c: Clone the repository you just created on github.com to your computer

Step 7: Your new repo appears on the GitHub Desktop application

Step 8: Open an R script and write your code

(See how to install R and RStudio if you are unfamiliar with it.)

Step 9 a: Save your R script in the corresponding folder

You can see the path to the folder in step 6 c, under “Local Path”.

Step 9 b: Save your R script in the corresponding folder

Step 10: Reopen GitHub Desktop, edit the commit title (1) and click on the commit button (2)

In step 10, make sure that the current repository (see top left) is the repo you are currently working on.

Step 11: Push the commit to github.com

Step 12: On your GitHub profile, click on the newly created repository

Step 13: You now see your script which has been added to the repository

Your R code is now available online via the new repository on your GitHub profile. If you need to share it, you can simply share the URL of the repo.

Additional notes

If you need to edit your code:

Find the folder of the repo on your computer by clicking on the button “Show in Finder”:

Find the folder of your repo on your computer

You will be redirected to the folder of your repo on your computer. Open the R script you want to work on, edit it and save it:

Open the R script, edit it and save it

After your script is saved, do not forget to commit the changes and push the commit (see step 10 above).

In case you need to see an old version of your code, you can see all commits by clicking on the commits tab accessible via the repo on github.com:

See all commits

Conclusion

Thanks for reading.

I hope this article helped you to create a GitHub repository, and upload your R scripts so that it is available to anyone.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

One-proportion and chi-square goodness of fit test

Wed, 13 May 2020 00:00:00 +0000

Introduction

In a previous article, I presented the Chi-square test of independence in R which is used to test the independence between two categorical variables.

In this article, I show how to perform, first in R and then by hand, the:

one-proportion test (also referred as one-sample proportion test)
Chi-square goodness of fit test

The first test is used to compare an observed proportion to an expected proportion, when the qualitative variable has only two categories. The second test is used to compare multiple observed proportions to multiple expected proportions, in a situation where the qualitative variable has two or more categories.

Both tests allow to test the equality of proportions between the levels of the qualitative variable or to test the equality with given proportions. These given proportions could be determined arbitrarily or based on the theoretical probabilities of a known distribution.

In R

Data

For this section, we use the same dataset than in the article on descriptive statistics. It is the well-known iris dataset, to which we add the variable size. The variable size corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

# load iris dataset
dat <- iris

# create size variable
dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

# show first 5 observations
head(dat, n = 5)

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species  size
## 1          5.1         3.5          1.4         0.2  setosa small
## 2          4.9         3.0          1.4         0.2  setosa small
## 3          4.7         3.2          1.3         0.2  setosa small
## 4          4.6         3.1          1.5         0.2  setosa small
## 5          5.0         3.6          1.4         0.2  setosa small

One-proportion test

For this example, we have a sample of 150 flowers and we want to test whether the proportion of small flowers is different than the proportion of big flowers (measured by the variable size). Here are the number of flowers by size, and the corresponding proportions:

# barplot
library(ggplot2)
ggplot(dat) +
  aes(x = size) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by size
table(dat$size)

## 
##   big small 
##    77    73

# proportions by size, rounded to 2 decimals
round(prop.table(table(dat$size)), 2)

## 
##   big small 
##  0.51  0.49

Among the 150 flowers forming our sample, 51% and 49% are big and small, respectively. To test whether the proportions are different among both sizes, we use the prop.test() function which accepts the following arguments:

number of successes
number of observations/trials
expected probability (the one we want to test against)

The hypotheses in our example are:

$H_0$: proportions of big and small flowers are equal
$H_1$: proportions of big and small flowers are different

Considering (arbitrarily) that big is the success, we have:¹

# one-proportion test
test <- prop.test(
  x = 77, # number of successes
  n = 150, # total number of trials (77 + 73)
  p = 0.5 # we test for equal proportion so prob = 0.5 in each group
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  77 out of 150, null probability 0.5
## X-squared = 0.06, df = 1, p-value = 0.8065
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.4307558 0.5952176
## sample estimates:
##         p 
## 0.5133333

We obtain an output with:

the null probability (0.5),
the test statistic (X-squared = 0.06),
the degrees of freedom (df = 1),
the p-value (p-value = 0.8065),
the alternative hypothesis (true p is not equal to 0.5),
the 95% confidence interval (which can also be extracted with test$conf.int) and
the proportion in the sample (0.5133333).

The p-value is 0.806 so, at the 5% significance level, we do not reject the null hypothesis that the proportions of small and big flowers are the same.

An alternative is the ggpiestats() function from the {ggstatsplot} package:²

## plot with statistical results
library(ggstatsplot)
ggpiestats(
  data = dat,
  x = size,
  bf.message = FALSE
)

Note that the $p$-value (the value after p = in the subtitle of the plot) is slightly different because Yates’ continuity correction is not applied in ggpiestats() while it is applied by default in prop.test().

The conclusion remains however the same, that is, we do not reject the null hypothesis that proportions of big and small flowers are equal.

Assumption of `prop.test()` and `binom.test()`

Note that prop.test() uses a normal approximation to the binomial distribution. Therefore, one assumption of this test is that the sample size is large enough (usually, n > 30). If the sample size is small, it is recommended to use the exact binomial test.

The exact binomial test can be performed with the binom.test() function and accepts the same arguments as the prop.test() function.

For this example, suppose now that we have a sample of 12 big and 3 small flowers and we want to test whether the proportions are the same among both sizes:

# barplot
barplot(c(12, 3), # observed counts
  names.arg = c("big", "small"), # rename labels
  ylab = "Frequency", # y-axis label
  xlab = "Size" # x-axis label
)
abline(
  h = 15 / 2, # expected counts in each level
  lty = 2 # dashed line
)

# exact binomial test
test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5 # expected proportion
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.03516
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.5191089 0.9566880
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.035 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportions of small and big flowers are significantly different. This is equivalent than concluding that the proportion of big flowers is significantly different from 0.5 (since there are only two sizes).

If you want to test that the proportion of big flowers is greater than 50%, add the alternative = "greater" argument into the binom.test() function:³

test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5, # expected proportion
  alternative = "greater" # test that prop of big flowers is > 0.5
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.01758
## alternative hypothesis: true probability of success is greater than 0.5
## 95 percent confidence interval:
##  0.5602156 1.0000000
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.018 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of big flowers is significantly larger than 50%.

Chi-square goodness of fit test

Suppose now that the qualitative variable has more than two levels as it is the case for the variable Species:

# barplot
ggplot(dat) +
  aes(x = Species) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by Species
table(dat$Species)

## 
##     setosa versicolor  virginica 
##         50         50         50

The variable Species has 3 levels, with 50 observations in each level.

Suppose for this example that we want to test whether the 3 species are equally common. If they were equally common, they would be equally distributed and the expected proportions would be $\frac{1}{3}$ for each of the species.

The hypotheses are now:

$H_0$: proportions of each species are equal
$H_1$: there is at least one species with a different proportion⁴

This test can be done with the chisq.test() function, accepting the following arguments:

a numeric vector representing the observed proportions
a vector of probabilities (of the same length of the observed proportions) representing the expected proportions

Applied to our research question (i.e., are the 3 species equally common?), we have:

# chi-square goodness of fit test
test <- chisq.test(table(dat$Species), # observed proportions
  p = c(1 / 3, 1 / 3, 1 / 3) # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  table(dat$Species)
## X-squared = 0, df = 2, p-value = 1

The p-value is 1 so, at the 5% significance level, we do not reject the null hypothesis that the proportions are equal among all species.

This was quite obvious even before doing the statistical test given that there are exactly 50 flowers of each species, so it was easy to see that the species are equally common. We however still did the test to show how it works in practice.

Note that the alternative proposed by the {ggstatsplot} package can also be used for a Chi-square goodness of fit test:

## plot with statistical results
ggpiestats(
  data = dat,
  x = Species,
  bf.message = FALSE
)

Assumptions

One of the assumptions of the chi-square goodness of fit test is that the sample size is large enough in order for the chi-square approximation to be valid.

To be more precise, there must be at least 5 expected frequencies in each group of your categorical variable. This can be verified as follows:

chisq.test(table(dat$Species))$expected

##     setosa versicolor  virginica 
##         50         50         50

The assumption of sufficiently large sample size is met as all expected frequencies are above 5.

Does my distribution follow a given distribution?

In the previous section, we chose the proportions ourselves. The goodness of fit test is also particularly useful to compare observed proportions with expected proportions that are based on some known distribution.

Remember the hypotheses of the test:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

For this example, suppose that we measured the number of girls in 100 families of 5 children. We want to test whether the (observed) distribution of number girls follows a binomial distribution.

Observed frequencies

Here is the distribution of the number of girls per family in our sample of 100 families of 5 children:

And the corresponding frequencies and relative frequencies (remember that the relative frequency is the frequency divided by the total sample size):

# counts
dat

##   Girls Frequency Relative_freq
## 1     0         5          0.05
## 2     1        12          0.12
## 3     2        28          0.28
## 4     3        33          0.33
## 5     4        17          0.17
## 6     5         5          0.05

Expected frequencies

In order to compare the observed frequencies to a binomial distribution and see if both distributions match, we first need to determine the expected frequencies that would be obtained in case of a binomial distribution.

The expected frequencies assuming a probability of 0.5 of having a girl (for each of the 5 children) are as follows:

# create expected frequencies for a binomial distribution
x <- 0:5
df <- data.frame(
  Girls = factor(x),
  Expected_relative_freq = dbinom(x, size = 5, prob = 0.5)
)
df$Expected_freq <- df$Expected_relative_freq * 100 # *100 since there are 100 families

# create barplot
p <- ggplot(df, aes(x = Girls, y = Expected_freq)) +
  geom_bar(stat = "identity", fill = "#F8766D") +
  xlab("Number of girls per family") +
  ylab("Expected frequency") +
  labs(title = "Binomial distribution Bi(x, n = 5, p = 0.5)") +
  theme_minimal()
p

# expected relative frequencies and (absolute) frequencies
df

##   Girls Expected_relative_freq Expected_freq
## 1     0                0.03125         3.125
## 2     1                0.15625        15.625
## 3     2                0.31250        31.250
## 4     3                0.31250        31.250
## 5     4                0.15625        15.625
## 6     5                0.03125         3.125

Observed vs. expected frequencies

We now compare the observed frequencies to the expected frequencies to see whether the two differ significantly. If the two differ significantly, we reject the hypothesis that the number of girls per family of 5 children follows a binomial distribution. On the other hand, if the observed and expected frequencies are similar, we do not reject the hypothesis that the number of girls per family follows a binomial distribution.

Visually we have:

# create data
data <- data.frame(
  num_girls = factor(rep(c(0:5), times = 2)),
  Freq = c(dat$Freq, df$Expected_freq),
  obs_exp = c(rep("observed", 6), rep("expected", 6))
)

# create plot
ggplot() +
  geom_bar(
    data = data, aes(
      x = num_girls, y = Freq,
      fill = obs_exp
    ),
    position = "dodge", # bar next to each other
    stat = "identity"
  ) +
  ylab("Frequency") +
  xlab("Number of girls per family") +
  theme_minimal() +
  theme(legend.title = element_blank()) # remove legend title

We see that the observed and expected frequencies are quite similar, so we expect that the number of girls in families of 5 children follows a binomial distribution. However, only the goodness of fit test will confirm our belief:

# chi-square goodness of fit test
test <- chisq.test(dat$Freq, # observed frequencies
  p = df$Expected_relative_freq # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$Freq
## X-squared = 3.648, df = 5, p-value = 0.6011

The p-value is 0.601 so, at the 5% significance level, we do not reject the null hypothesis that the observed and expected frequencies are equal. This is equivalent than concluding that we cannot reject the hypothesis that the number of girls in families of 5 children follows a binomial distribution (since the expected frequencies were based on a binomial distribution).

Note that the chi-square goodness of fit test can of course be performed with other types of distribution than the binomial one. For instance, if you want to test whether an observed distribution follows a Poisson distribution, this test can be used to compare the observed frequencies with the expected proportions that would be obtained in case of a Poisson distribution.

By hand

Now that we showed how to perform the one-proportion and chi-square goodness of fit test in R, in this section we show how to do these tests by hand. We first illustrate the one-proportion test then the chi-square goodness of fit test.

One-proportion test

For this example, suppose that we tossed a coin 100 times and noted that it landed on heads 67 times. Following this, we want to test whether the coin is fair, that is, test whether the probability of landing on heads or tails is equal to 50%.

As for many hypothesis tests, we do it through 4 easy steps:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

In our example, the null and alternative hypotheses are:

$H_0$: $p_0 = 0.5$
$H_1$: $p_0 \ne 0.5$

where $p_0$ is the expected proportion of landing on heads.

Step 2.

The test statistic is:⁵

\[z_{obs} = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1 - p_0)}{n}}} = \frac{0.67 - 0.5}{\sqrt{\frac{0.5 \cdot (1 - 0.5)}{100}}} = 3.4\]

(See how to perform hypothesis tests in a Shiny app if you need more help in computing the test statistic.)

Step 3.

The rejection region is found via the normal distribution table. Assuming a significance level $\alpha = 0.05$, we have:

\[\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic lies within the rejection region (i.e., the grey shaded areas). Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of heads (and thus tails) is significantly different than 50%. In other words, still at the 5% significance level, we conclude that the coin is unfair.

If you prefer to compute the p-value instead of comparing the t-stat and the rejection region, you can use this Shiny app to easily compute p-values for different probability distributions. After having opened the app, set the t-stat, the corresponding alternative and you will find the p-value at the top of the page.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# one-proportion test
test <- prop.test(
  x = 67, # number of heads
  n = 100, # number of trials
  p = 0.5 # expected probability of heads
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  67 out of 100, null probability 0.5
## X-squared = 10.89, df = 1, p-value = 0.0009668
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.5679099 0.7588442
## sample estimates:
##    p 
## 0.67

The p-value is 0.001 so, at the 5% significance level, we reject the null hypothesis that the proportions of heads and tails are equal, and we conclude that the coin is biased. This is the same conclusion than the one found by hand.

Goodness of fit test

We now illustrate the chi-square goodness of fit test by hand with the following example.

Suppose that we toss a dice 100 times, we note how many times it lands on each face (1 to 6) and we test whether the dice is fair. Here are the observed counts by dice face:

## dice_face
##  1  2  3  4  5  6 
## 15 24 10 19 19 13

With a fair dice, we would expect it to land $\frac{100}{6} \approx 16.67$ times on each face (this expected value is represented by the dashed line in the above plot). Although the observed frequencies are different than the expected value of 16.67:

##   dice_face observed_freq expected_freq
## 1         1            15         16.67
## 2         2            24         16.67
## 3         3            10         16.67
## 4         4            19         16.67
## 5         5            19         16.67
## 6         6            13         16.67

we need to test whether they are significantly different. For this, we perform the appropriate hypothesis test following the 4 easy steps mentioned above:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

The null and alternative hypotheses of the chi-square goodness of fit test are:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

Applied to our example, we have:

$H_0$: all faces occur in the same proportion
$H_1$: at least one proportion is not equal to 1/6

Step 2.

The test statistic is:

\[\chi^2 = \sum_{i = 1}^k \frac{(O_i - E_i)^2}{E_i}\]

where $O_i$ is the observed frequency, $E_i$ is the expected frequency and $k$ is the number of categories (in our case, there are 6 categories, representing the 6 dice faces).

This $\chi^2$ statistic is obtained by calculating the difference between the observed number of cases and the expected number of cases in each category. This difference is squared (to avoid negative and positive differences being compensated) and divided by the expected number of cases in that category. These values are then summed for all categories, and the total is referred to as the $\chi^2$ statistic. Large values of this test statistic lead to the rejection of the null hypothesis, small values mean that the null hypothesis cannot be rejected.⁶

Given our data, we have:

\[\chi^2 = \frac{(15 - 16.67)^2}{16.67} + \frac{(24 - 16.67)^2}{16.67} + \\ \frac{(10 - 16.67)^2}{16.67} +\frac{(19 - 16.67)^2}{16.67} + \\ \frac{(19 - 16.67)^2}{16.67} + \frac{(13 - 16.67)^2}{16.67} = 7.52\]

Step 3.

Whether the $\chi^2$ test statistic is small or large depends on the rejection region. The rejection region is found via the $\chi^2$ distribution table. With a degrees of freedom equals to $k - 1$ (where $k$ is the number of categories) and assuming a significance level $\alpha = 0.05$, we have:

\[\chi^2_{\alpha; k-1} = \chi^2_{0.05; 5} = 11.0705\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic does not lie within the rejection region (i.e., the grey shaded area). Therefore, at the 5% significance level, we do not reject the null hypothesis that there is no significant difference between the observed and the expected frequencies. In other words, still at the 5% significance level, we cannot reject the hypothesis that the dice is fair.

Again, you can use the Shiny app to easily compute the p-value given the test statistic if you prefer this method over the comparison between the t-stat and the rejection region.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# chi-square goodness of fit test
test <- chisq.test(dat$observed_freq, # observed frequencies for each dice face
  p = rep(1 / 6, 6) # expected probabilities for each dice face
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$observed_freq
## X-squared = 7.52, df = 5, p-value = 0.1847

The test statistic and degrees of freedom are exactly the same than the ones found by hand. The p-value is 0.185 which, still at the 5% significance level, leads to the same conclusion than by hand (i.e., failing to reject the null hypothesis).

Conclusion

Thanks for reading.

I hope this article helped you to understand and perform the one-proportion and chi-square goodness of fit test in R and by hand. Learn more about the Chi-square test of independence in R and by hand if you want to analyze two categorical variables instead of one.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Choosing big or small as the success event gives the exact same conclusion.↩︎
Note that if possible, it is best to avoid pie charts and use bar charts instead. Unfortunately, the ggbarstats() function works only for the independence Chi-square test.↩︎
Similarly, this argument can also be added to the prop.test() function to test whether the observed proportion is larger than the expected proportion. Use alternative = "less" if you want to test whether the observed proportion is smaller than the expected one.↩︎
Be careful that the alternative hypothesis is not that all proportions are different. One different from the others is sufficient to reject the null hypothesis. It is, in some sense, similar to the alternative hypothesis of the ANOVA which says that at least one mean is different than another.↩︎
One assumption of this test is that $n \cdot p \ge 5$ and $n \cdot (1 - p) \ge 5$. The assumption is met so we can use the normal approximation to the binomial distribution.↩︎
Source: http://uregina.ca/~gingrich/ch10.pdf.↩︎

How to create a simple Coronavirus dashboard specific to your country in R?

Mon, 23 Mar 2020 00:00:00 +0000

Coronavirus dashboard: the case of Belgium

Introduction

The Novel COVID-19 Coronavirus is the hottest topic right now. Every day, the media and newspapers share the number of new cases and deaths in several countries, try to measure the impacts of the virus on citizens and remind us to stay home in order to stay safe. The Coronavirus is on everyone’s lips.

In addition to governments, media and companies discussing about it, data scientists and data professionals in general are putting their knowledge and time at the service of the virus. This leads to a proliferation of applications, dashboards, blog posts, videos, datasets and code analyzing, in one way or another, the expansion of the COVID-19 and how it spreads in the population.

Top R resources on Coronavirus

As a data lover myself, I discovered a multitude of great resources about the Coronavirus. However, these resources were spread all over the internet and were often hidden by the mass of information of another type (e.g., alarming headlines, names of infected celebrities, companies showing off how they helped health care agencies, etc.). To tackle this issue, I collected and then shared in a previous article the best R resources on the Coronavirus I came across.

Note that only resources on R are shared in this article as R is my favorite statistical program and the one I am most familiar with. The fact that I use this program almost daily makes it easier for me to realize the complexity and time put behind a resource, and appreciate its quality and its potential.

I am sure that there are other very interesting resources online (see for example the probably most popular dashboard by the Johns Hopkins Coronavirus Resource Center).¹ Nonetheless, a lot of people are in a better position than I am to judge the quality of resources made with programming languages for which I am not competent.

This article made me discover so many great resources about the Coronavirus and I keep receiving data visualizations and data analyses from scientists all over the world so that I include them in the collection. Thanks for that, it continuously improves the quality and completeness of the collection.

In addition to receiving R resources, a question often asked by readers was “How can I create a dashboard myself?” or “How can I build a dashboard specific to my country?”. I therefore thought it would serve some people if I created a dashboard specific to my country (Belgium) and detailed the steps on how to build it.

Questions on how to develop such dashboards came mostly from R beginners as advanced R users most probably know how to do one, or at least can easily use the resources I collected here as sources of inspiration for their own work. Furthermore, in response to the craze about the Coronavirus, interested users were quite in a hurry and wanted to have their own dashboard running as quickly as possible.

These questions led me to the idea of creating a simple (yet powerful and visually appealing) dashboard, as opposed to a Shiny app. Shiny apps have the advantage that they are interactive in the sense that users can edit the outputs and the visualizations by simply changing some inputs in a user-friendly way, while dashboards are static and cannot be modified by the final user. On the other hand, the advantage of a dashboard over a Shiny app is that it is much easier to code, especially if you are already proficient in R Markdown.

Coronavirus dashboard: the case of Belgium

Amongst all the visualizations I have seen so far, one is standing out by its simplicity and at the same time, by its completeness and by the quality of its visualizations. I thus decided to create a Coronavirus dashboard based on this already existing dashboard by Rami Krispin (which comes with a license that allows to be freely adapted and shared) and adapt it so that it is specific to Belgium. Note that I also removed some visualizations and tables from the initial dashboard to keep it really simple and straight to the point.

Before reading further, here is my Coronavirus dashboard adapted to Belgium and previews of the main sections below:

The dashboard is segmented into several sections that can be selected at the top:

The summary section provides key measures about the Coronavirus (total cases, active cases and deaths) and a plot displaying the cumulative number of active cases and deaths from January 22, 2020 to the latest available date.
The comparison section presents a comparison of the number of daily new cases (left panel) and the distribution of cases by type (right panel) with other European countries (you can also change these countries by replacing them in the code).
The map section shows a world map of the confirmed cases and deaths. You can uncheck one or several types of cases (top right corner) and zoom in or out (top left corner) to adapt the map to your needs.
The about section gives more information about the data, the dashboard in general and how often it is updated.

I believe this simple dashboard is easy enough to be adapted to any country (and by anyone from beginner to expert), and still communicate key measures about the virus throughout some visualizations.

A little extra which is worth mentioning is the fact that all plots are generated with the {plotly} package. This package allows to enhance plots by displaying additional relevant information when hovering over them (try by yourself!).

How to create your own Coronavirus dashboard

If you want to build your own dashboard specific to a country, follow these steps:

Open the dashboard here
See the entire code via the button “Source code” located in the top right corner of the dashboard, or see the code on GitHub. Copy that code.
Open a new R Markdown file (.Rmd), type any title and author (they will be replaced in the next step anyway), select HTML as the output format and click on OK:

Remove all the template code already present and paste the code you copied in step 1.
Make sure that the required packages are installed:

install.packages(c("devtools", "flexdashboard", "leaflet", "leafpop"))
devtools::install_github("RamiKrispin/coronavirus", force = TRUE)

If R asks you which package you would like to update, there should be no need to update them: type 3 for “None”.

In the code, replace Belgium with your country. Here is the list of all available countries in the dataset:

Canada, United Kingdom, China, Netherlands, Australia, New Zealand, Denmark, France, Afghanistan, Albania, Algeria, Andorra, Angola, Antarctica, Antigua and Barbuda, Argentina, Armenia, Austria, Azerbaijan, Bahamas, Bahrain, Bangladesh, Barbados, Belarus, Belgium, Belize, Benin, Bhutan, Bolivia, Bosnia and Herzegovina, Botswana, Brazil, Brunei, Bulgaria, Burkina Faso, Burma, Burundi, Cabo Verde, Cambodia, Cameroon, Central African Republic, Chad, Chile, Colombia, Comoros, Congo (Brazzaville), Congo (Kinshasa), Costa Rica, Cote d’Ivoire, Croatia, Cuba, Cyprus, Czechia, Diamond Princess, Djibouti, Dominica, Dominican Republic, Ecuador, Egypt, El Salvador, Equatorial Guinea, Eritrea, Estonia, Eswatini, Ethiopia, Fiji, Finland, Gabon, Gambia, Georgia, Germany, Ghana, Greece, Grenada, Guatemala, Guinea, Guinea-Bissau, Guyana, Haiti, Holy See, Honduras, Hungary, Iceland, India, Indonesia, Iran, Iraq, Ireland, Israel, Italy, Jamaica, Japan, Jordan, Kazakhstan, Kenya, Kiribati, Korea, North, Korea, South, Kosovo, Kuwait, Kyrgyzstan, Laos, Latvia, Lebanon, Lesotho, Liberia, Libya, Liechtenstein, Lithuania, Luxembourg, Madagascar, Malawi, Malaysia, Maldives, Mali, Malta, Marshall Islands, Mauritania, Mauritius, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Morocco, Mozambique, MS Zaandam, Namibia, Nepal, Nicaragua, Niger, Nigeria, North Macedonia, Norway, Oman, Pakistan, Palau, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Poland, Portugal, Qatar, Romania, Russia, Rwanda, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, Samoa, San Marino, Sao Tome and Principe, Saudi Arabia, Senegal, Serbia, Seychelles, Sierra Leone, Singapore, Slovakia, Slovenia, Solomon Islands, Somalia, South Africa, South Sudan, Spain, Sri Lanka, Sudan, Summer Olympics 2020, Suriname, Sweden, Switzerland, Syria, _Taiwan*_, Tajikistan, Tanzania, Thailand, Timor-Leste, Togo, Tonga, Trinidad and Tobago, Tunisia, Turkey, Uganda, Ukraine, United Arab Emirates, Uruguay, US, Uzbekistan, Vanuatu, Venezuela, Vietnam, West Bank and Gaza, Winter Olympics 2022, Yemen, Zambia and Zimbabwe

Note that if your country is spelled in two words or more, you will need to surround it by a backtick (but only at one specific line in the code, see an example with United Kingdom):

#----------------------------------------
# Plotting the data

daily_confirmed %>%
  plotly::plot_ly() %>%
  plotly::add_trace(
    x = ~date,
    y = ~`United Kingdom`,
    type = "scatter",
    mode = "lines+markers",
    name = "United Kingdom"
  ) %>%

Do not add backticks in the rest of the code as everywhere else the country’s name is surrounded by double quotes "".

Do not forget to also change the title and the author at the top of the document, and edit the about section at the bottom of the document. Last but not least, as you can see on the plot in the summary section, the arrows point to different (sad) “milestones” in Belgium (i.e., first case, first death and new containment measures).

You will need to adapt these milestones for your country (or remove them if you do not want to have any milestone displayed on the plot). Change this in the code following the plotly::add_annotations() functions.

Knit the document (see this article if you are unfamiliar with R Markdown). Your dashboard should appear in HTML format.

Following these 7 steps, you should already have a simple dashboard specific to your country. I have intentionally kept it simple so that everyone could copy it and have their own dashboard in a limited amount of time.

If you are familiar with the {flexdashboard}, {plotly} and {leaflet} packages for the dashboard interface and the visualizations, and the {dplyr} and {tidyr} packages for the data manipulation, feel free to edit the code according to your needs and improve your dashboard.

Additional notes

Data

The input data for this dashboard is the dataset available from the {coronavirus} R package. Make sure to download the development version of the package to have the latest data:

install.packages("devtools")
devtools::install_github("RamiKrispin/coronavirus")

To update your dashboard with the latest data, you have to manually update the data by reinstalling the {coronavirus} package with devtools::install_github("RamiKrispin/coronavirus", force = TRUE). Again, if R asks you whether you would like to update other packages, type 3 for “None”.

This question was often raised so I repeat, your dashboard will not update by itself every day, you need to manually update it. After updating the data, you may also need to restart your R session in order to have the last available data.

The raw data is pulled from the Johns Hopkins University Center for Systems Science and Engineering (JHU CCSE) Coronavirus repository.

Open source

This dashboard and the code available on GitHub are open source so feel free to copy it, adapt it and share it as much as you want.

Accuracy

Please note that this dashboard has been built mainly for educational purposes. I update the dashboard as often as possible to keep it accurate. However, there is some uncertainty concerning the COVID-19 case numbers and the testing methods vary between countries so the figures on this dashboard may be slightly different compared to other sources. Currently, the maintainer of the dataset updates it on a daily basis, but updates may become less frequent in the future.

Publish your dashboard

If you want to share your dashboard, you can either:

Upload it on your website if you have one (and I strongly suggest you to create one if you do not already have one)
Publish it through RPubs (it is free and easy to publish your work directly from RStudio)

Conclusion

Thanks for reading.

I hope this article helped you to build your first Coronavirus dashboard in R. See these top R resources on Coronavirus if you need inspiration to enhance further your dashboard.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

I would be glad to mention a collection of Python resources if someone is willing to create a collection of resources about the Coronavirus made with this programming language. Feel free to contact me if this is the case.↩︎

How to perform a one-sample t-test by hand and in R: test on one mean

Mon, 09 Mar 2020 00:00:00 +0000

Introduction

After having written an article on the Student’s t-test for two samples (independent and paired samples), I believe it is time to explain in details how to perform one-sample t-tests by hand and in R.

One-sample t-test is an important part of inferential statistics (probably one of the first statistical test that students learn). Remind that, unlike descriptive statistics, inferential statistics is a branch of statistics aiming at drawing conclusions about one or two populations, based on a subset (or two) of that population (called samples). In other words, we first collect a random set of observations from a population, and then some measurements are calculated in order to generalize to the population the information found through the sample.

In this context, the one-sample t-test is used to determine whether the mean of a measurement variable is different from a specified value (a belief or a theoretical expectation for example). It works as follows: if the mean of the sample is too distant from the specified value (the value under the null hypothesis), it is considered that the mean of the population is different from what is expected. On the contrary, if the mean of the sample is close to the specified value, we cannot reject the hypothesis that the population mean is equal to what is expected.

Like the Student’s t-test for two samples and the ANOVA (for 3 or more samples), there are also different versions of the one-sample t-test. Luckily, there are only two different versions for this test (the Student’s t-test for two samples has 5 versions!). The difference between the two versions of the one-sample t-test lies in the fact that one version is used when the variance of the population (not the variance of the sample!) is known, the other version being used when the variance of the population is unknown.

In this article, I will first detail step by step how to perform both versions of the one-sample t-test by hand. The analyses will be done on a small set of observations for the sake of illustration and easiness. I will then show how to perform this test in R with the exact same data in order to verify the results found by hand. Reminders about the reasoning behind hypothesis tests, interpretations of the p-value and the results, and assumptions of this test will also be presented.

Note that the aim of this article is to show how to compute the one-sample t-test by hand and in R, so we refrain from testing the assumptions and we assume all assumptions are met for this exercise. For completeness, we still mention the assumptions and how to test them. Interested readers are invited to have a look at the end of the article for more information about these assumptions.

Null and alternative hypothesis

Before diving into the computations of the one-sample t-test by hand, let’s recap the null and alternative hypotheses of this test:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu \ne \mu_0$

where $\mu$ is the population mean and $\mu_0$ is the known or hypothesized value of the mean in the population.

This is in the general case where we simply want to determine whether the population mean is different (in terms of the dependent variable) compared to the hypothesized value. In this sense, we have no prior belief about the population mean being larger or smaller than the hypothesized value. This type of test is referred as a two-sided or bilateral test.

If we have some prior beliefs about the population mean being larger or smaller than the hypothesized value, the one-sample t-test also allows to test the following hypotheses:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu > \mu_0$

$H_0$: $\mu = \mu_0$
$H_1$: $\mu < \mu_0$

In the first case, we want to test if the population mean is significantly larger than the hypothesized value, while in the latter case, we want to test if the population mean is significantly smaller than the hypothesized value. This type of test is referred as a one-sided or unilateral test.

Hypothesis testing

In statistics, many statistical tests is in the form of hypothesis tests. Hypothesis tests are used to determine whether a certain belief can be deemed as true (plausible) or not, based on the data at hand (i.e., the sample(s)). Most hypothesis tests boil down to the following 4 steps:¹

State the null and alternative hypothesis.
Compute the test statistic, denoted t-stat. Formulas to compute the test statistic differ among the different versions of the one-sample t-test but they have the same structure. See scenarios 1 and 2 below to see the different formulas.
Find the critical value given the theoretical statistical distribution of the test, the parameters of the distribution and the significance level $\alpha$. For the two versions of the one-sample t-test, it is either the normal or the Student’s t distribution (t denoting the Student distribution and z denoting the normal distribution).
Conclude by comparing the t-stat (found in step 2.) with the critical value (found in step. 3). If the t-stat lies in the rejection region (determined thanks to the critical value and the direction of the test), we reject the null hypothesis, otherwise we do not reject the null hypothesis. These two alternatives (reject or do not reject the null hypothesis) are the only two possible solutions, we never “accept” an hypothesis. It is also a good practice to always interpret the decision in the terms of the initial question.

For the interested reader, see these 4 steps of hypothesis testing in more details in this article.

Two versions of the one-sample t-test

There are two versions of the one-sample t-test, depending on whether the variance of the population (not the variance of the sample!) is known or unknown. This criteria is rather straightforward, we either know the variance of the population or we do not. The variance of the population cannot be computed because if you can compute the variance of a population, it means you have the data for the whole population, then there is no need to do a hypothesis test anymore…

So the variance of the population is either given in the statement (use them in that case), or there is no information about the variance and in that case, it is assumed that the variance is unknown. In practice, the variance of the population is most of the time unknown. However, we still illustrate how to do both versions of this test by hand and in R in the next sections following the 4 steps of a hypothesis test.

How to compute the one-sample t-test by hand?

Note that the data are artificial and do not represent any real variable. Furthermore, remind that the assumptions may or may not be met. The point of the article is to detail how to compute the different versions of the test by hand and in R, so all assumptions are assumed to be met. Moreover, we assume that for all tests the significance level, that is, the type I error is $\alpha = 5$%.

If you are interested in applying these tests by hand without having to do the computations yourself, here is a Shiny app which does it for you. You just need to enter the data and choose the appropriate version of the test thanks to the sidebar menu. There is also a graphical representation that helps you to visualize the test statistic and the rejection region. I hope you will find it useful!

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

value
0.9
-0.8
1.3
-0.3
1.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 0.56$
variance of the population: $\sigma^2 = 1$
$\mu_0 = 0$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 0$ and $H_1: \mu \ne 0$. ($\ne$ because we want to test whether the population mean is different from 0, we do not impose a direction in the test.)
Test statistic: \[z_{obs} = \frac{\bar{x} - \mu_0}{\frac{\sigma}{\sqrt{n}}} = \frac{0.56-0}{0.447} = 1.252\]
Critical value: $\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the population mean is equal to 0, or there is no sufficient evidence in the data to conclude that the population mean is different from 0.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

value
7.9
5.8
6.3
7.3
6.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 6.8$
standard deviation of the sample: $s = 0.825$
$\mu_0 = 5$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 5$ and $H_1: \mu > 5$. (> because we want to test whether the population mean is larger than 5.)
Test statistic: \[t_{obs} = \frac{\bar{x} - \mu_0}{\frac{s}{\sqrt{n}}} = \frac{6.8-5}{0.369} = 4.881\]
Critical value: $t_{\alpha, n - 1} = t_{0.05, 4} = 2.132$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection region is thus from 2.132 to $+\infty$. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the population mean is larger than 5.

This concludes how to perform the two versions of the one-sample t-test by hand. In the next sections, we detail how to perform the exact same tests in R.

Different underlying distributions for the critical value

As you may have noticed, the underlying probability distributions used to find the critical value are different depending on whether the variance of the population is known or unknown.

The underlying probability distribution when the variance is known (scenario 1) is the normal distribution, while the probability distribution in the case where the variance is unknown (scenario 2) is the Student’s t distribution. This difference is partially explained by the fact that when the variance of the population is unknown, there is more “uncertainty” in the data, so we need to use the Student’s t distribution instead of the normal distribution.

Note that when the sample size is large (usually when n > 30), the Student’s t distribution tends to a normal distribution. Using a normal distribution when the variance is known and a Student’s t distribution when the variance is unknown also applies to a t-test for two samples.

How to compute the one-sample t-test in R?

A good practice before doing t-tests in R is to visualize the data thanks to a boxplot (or eventually a histogram or a density plot). A boxplot gives a first indication on the location of the sample, and thus, a first indication on whether the null hypothesis is likely to be rejected or not. However, even if a boxplot or a density plot is great in showing the distribution of a sample, only a sound statistical test will confirm our first impression.

After a visualization of the data, we replicate in R the results found by hand. Note that we use the same data, the same assumptions and the same question for both scenarios to facilitate the comparison between the tests performed by hand and in R.

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

dat1 <- data.frame(
  value = c(0.9, -0.8, 1.3, -0.3, 1.7)
)

dat1

##   value
## 1   0.9
## 2  -0.8
## 3   1.3
## 4  -0.3
## 5   1.7

library(ggplot2)

ggplot(dat1) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Note that you can use the {esquisse} RStudio addin if you want to draw a boxplot with the package {ggplot2} without writing the code yourself. If you prefer the default graphics, use the boxplot() function:

boxplot(dat1$value)

The boxplot shows that the distribution of the sample is not distant from 0 (the hypothesized value), so we tend to believe that we will not be able to reject the null hypothesis that the population mean is equal to 0. However, only a formal statistical test will confirm this belief.

Below a function to perform a t-test with a known population variance, with arguments accepting the sample (x), the variance of the population (V), the mean under the null hypothesis (m0, default is 0), the significance level (alpha, default is 0.05) and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test2 <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test2(dat1$value,
  V = 1
)
test

## $mean
## [1] 0.56
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 1.252198
## 
## $p.value
## [1] 0.2104977
## 
## $LCL
## [1] -0.3165225
## 
## $UCL
## [1] 1.436523
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the sample mean, the hypothesized value and the population variance (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.2104977

The p-value is 0.21 so at the 5% significance level we do not reject the null hypothesis. There is no sufficient evidence in the data to reject the hypothesis that the population mean is equal to 0. This result confirms what we found by hand.

Note that a similar function exists in the {BSDA} package:²

library(BSDA)

z.test(dat1$value,
  alternative = "two.sided",
  mu = 0,
  sigma.x = 1,
  conf.level = 0.95
)

## 
## 	One-sample z-Test
## 
## data:  dat1$value
## z = 1.2522, p-value = 0.2105
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.3165225  1.4365225
## sample estimates:
## mean of x 
##      0.56

If you are unfamiliar with the concept of p-value, I invite you to read my note on p-value and significance level $\alpha$.

To sum up what have been said in that article about p-value and significance level $\alpha$:

If the p-value is smaller than the predetermined significance level $\alpha$ (usually 5%) so if p-value < 0.05, we reject the null hypothesis
If the p-value is greater than or equal to the predetermined significance level $\alpha$ (usually 5%) so if p-value $\ge$ 0.05, we do not reject the null hypothesis

This applies to all statistical tests without exception. Of course, the null and alternative hypotheses change depending on the test.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

dat2 <- data.frame(
  value = c(7.9, 5.8, 6.3, 7.3, 6.7)
)

dat2

##   value
## 1   7.9
## 2   5.8
## 3   6.3
## 4   7.3
## 5   6.7

ggplot(dat2) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the box is quite distant from the hypothesized value of 5. From this boxplot, we can expect the test to reject the null hypothesis of the population mean being equal to 5. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” t-test for one-sample. Note that in our case the alternative hypothesis is $H_1: \mu > 5$ so we need to add the arguments mu = 5 and alternative = "greater" to the function because the default arguments are mu = 0 and the two-sided test:

test <- t.test(dat2$value,
  mu = 5,
  alternative = "greater"
)

test

## 
## 	One Sample t-test
## 
## data:  dat2$value
## t = 4.8809, df = 4, p-value = 0.004078
## alternative hypothesis: true mean is greater than 5
## 95 percent confidence interval:
##  6.013814      Inf
## sample estimates:
## mean of x 
##       6.8

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used, the hypothesized value and the sample mean (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.004077555

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the population mean is significantly larger than 5. This result confirms what we found by hand.

Confidence interval

Note that the confidence interval can be extracted with $conf.int:

test$conf.int

## [1] 6.013814      Inf
## attr(,"conf.level")
## [1] 0.95

You can see that the 95% confidence interval for the population mean is $[6.01; \infty]$, meaning that, at the significance level $\alpha = 5$%, we reject the null hypothesis as long as the hypothesized value $\mu_0$ is below 6.01, otherwise the null hypothesis cannot be rejected.

Combination of plot and statistical test

After having written this article, I discovered the {ggstatsplot} package which I believe is worth mentioning here, in particular the gghistostats() function for one-sample Student’s t-test.

This function combines a histogram—representing the distribution—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for scenario 2. Unfortunately, the package does not allow to run the test for scenario 1.

Scenario 2: variance of the population is unknown

# load packages
library(ggstatsplot)
library(ggplot2)

# plot with stat test
gghistostats(
  data = dat2, # dataframe from which variable is to be taken
  x = value, # numeric variable whose distribution is of interest
  type = "parametric", # for student's t-test
  test.value = 5 # default value is 0
) +
  labs(caption = NULL) # remove caption

The p-value is displayed after p = in the subtitle of the plot. Based on this plot and the p-value being lower than 5% (p-value = 0.008), we reject the null hypothesis that the population mean is equal to 5.

Note that, the p-value is two times as large as the one obtained with the t.test() function because when we ran t.test() we specified alternative = "greater" (i.e., a one-sided test). In our plot with the gghistostats() function, it is a two-sided test that is performed by default, that is, alternative = "two.sided".

The point of this section was to illustrate how to easily draw plots together with statistical results, which is exactly the aim of the {ggstatsplot} package. See more details and examples in this article.

Assumptions

Variable type: The dependent variable (i.e., the measured variable) must be measured on a continuous or ordinal scale.
Independence: The data, collected from a representative and randomly selected portion of the population, should be independent of one another. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another). If not, it is most likely that you have independent samples.
Normality:
- With a small sample size (usually $n < 30$), observations should follow a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. Some transformations, such as among others, the logarithm, the square root or the Box-Cox transformation can be applied on the observations to transform you data to better fit the normal distribution. If, even after a transformation, your data still do not follow a normal distribution, the one-sample Wilcoxon test (wilcox.test(variable_name, data = dat in R) can be applied. This non-parametric test is robust to non normal distributions so it does not require normality of the data.
- With a large sample size ($n \ge 30$), normality of the data is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). It is therefore not required to test the normality assumption when the number of observations is large.
Outliers: There should be no outliers in your data. An observation slightly different from the others does not pose a problem, but it starts to be an issue when you have at least one extreme outlier. In presence of at least one extreme outlier, it is best to transform your data (with the logarithm transformation for instance, as you would do with a non normal distribution) or use the non-parametric one-sample Wilcoxon test.

Conclusion

Thanks for reading.

I hope this article helped you to understand how the different versions of the one-sample t-test work and how to perform them by hand and in R.

If you are interested, here is a Shiny app to perform these tests by hand easily (you just need to enter your data and select the appropriate version of the test thanks to the sidebar menu). Moreover, read this article if you would like to know how to compute the Student’s t-test but this time, for two samples—in order to compare two dependent or independent groups—or this article if you want to use an ANOVA to compare 3 or more groups.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

It is a least the case regarding parametric hypothesis tests. A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. In the case of the one-sample t-test, it is based on the Student’s t distribution with a single parameter, the degrees of freedom ($df = n - 1$ where $n$ is the sample size), or the normal distribution.↩︎
Thanks gmacar for pointing it out to me.↩︎

The 9 concepts and formulas in probability that every data scientist should know

Tue, 03 Mar 2020 00:00:00 +0000

What is probability?

A probability is a number that reflects the chance that a particular event will occur. In other words, it quantifies (on a scale from 0 to 1, or from 0% to 100%) how likely an event is to occur.

Probability is a branch of mathematics that provides models to describe random processes. These mathematical tools allow to establish theoretical models for random phenomena and to use them to make predictions. Like every model, the probabilistic model is a simplification of the world. However, the model is useful as soon as it captures the essential features.

In this article, we present 9 fundamental formulas and concepts in probability that every data scientist should understand and master in order to appropriately handle any project involving probabilities.

1. A probability is always between 0 and 1

The probability of an event is always between 0 and 1 (or 0% and 100%). If we denote the probability that an event A (which could be any event) occurs by $P(A)$, we have:

\[0 \le P(A) \le 1\]

If an event is impossible: $P(A) = 0$
If an event is certain: $P(A) = 1$

For example, throwing a 7 with a standard six-sided dice (with faces ranging from 1 to 6) is impossible so its probability is equal to 0. Throwing head or tail with a coin is certain, so its probability is equal to 1.

2. Compute a probability

If the elements of a sample space (the set of all possible results of a randomized experiment) are equiprobable (= all elements have the same probability), then the probability of an event occurring is equal to the number of favourable cases (number of ways it can happen) divided by the number of possible cases (total number of outcomes):

\[P(A) = \frac{\text{number of favourable cases}}{\text{number of possible cases}}\]

For example, all numbers of a six-sided dice are equiprobable since they all have the same probability of occurring. The probability of rolling a 3 with a dice is thus

\[P(3) = \frac{\text{number of favourable cases}}{\text{number of possible cases}} = \frac{1}{6}\]

because there is only one favourable case (there is only one face with a 3 on it), and there are 6 possible cases (because there are 6 faces altogether).

3. Complement of an event

The probability of the complement (or opposite) of an event is:

\[P(\text{not A}) = P(\bar{A}) = 1 - P(A)\]

For instance, the probability of not throwing a 3 with a dice is:

\[P(\bar{A}) = 1 - P(A) = 1 - \frac{1}{6} = \frac{5}{6}\]

4. Union of two events

The probability of the union of two events is the probability of either occurring:

\[\begin{align} P(\text{A or B)} &= P(A \cup B) \\ &= P(A) + P(B) - P(A \cap B) \end{align}\]

Suppose that the probability of a fire breaking out in two houses in a given year is:

in house A: 60%, so $P(A) = 0.6$
in house B: 45%, so $P(B) = 0.45$
in at least one of the two houses: 80%, so $P(A \cup B) = 0.8$

Graphically we have

The probability of a fire breaking out in house A or house B is

\[P(A \cup B) = P(A) + P(B) - P(A \cap B)\] \[= 0.6 + 0.45 - 0.25 = 0.8\]

By summing $P(A)$ and $P(B)$, the intersection of A and B, i.e. $P(A \cap B)$, is counted twice. This is the reason we subtract it to count it only once.

If two events are mutually exclusive (i.e., two events that cannot occur simultaneously), the probability of both events occurring $P(A \cap B)$ is equal to 0, so the above formula becomes

\[P(A \cup B) = P(A) + P(B)\]

For example, the event “rolling a 3” and the event “rolling a 6” on a six-sided dice are two mutually exclusive events since they cannot both occur at the same time. Since their joint probability is equal to 0, the probability of rolling a 3 or 6 on a six-sided dice is

\[P(3 \cup 6) = P(3) + P(6) = \frac{1}{6} + \frac{1}{6} = \frac{1}{3}\]

5. Intersection of two events

If two events are independent, the probability of the intersection of the two events (i.e., the joint probability) is the probability of the two events occurring:

\[P(\text{A and B)} = P(A \cap B) = P(A) \cdot P(B)\]

For instance, if two coins are flipped, the probability of both coins being tails is

\[P(T_1 \cap T_2) = P(T_1) \cdot P(T_2) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\]

where $T_1$ ($T_2$) denotes the event that the first (second) coin is tail.

Note that $P(A \cap B) = P(B \cap A)$.

If two events are mutually exclusive, their joint probability is equal to 0:

\[P(A \cap B) = 0\]

6. Independence of two events

Another important concept in probability is the independence of two events. Formally, the events A and B are independent if and only if

\[P(A \cap B) = P(A) \cdot P(B)\]

If the equality holds, the two events are said to be independent, otherwise the two events are said to be dependent.

In the example of the two coins:

\[P(T_1 \cap T_2) = \frac{1}{4}\]

and

\[P(T_1) \cdot P(T_2) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\]

so the following equality holds

\[P(T_1 \cap T_2) = P(T_1) \cdot P(T_2) = \frac{1}{4}\]

The two events are thus independent, denoted $T_1{\perp\!\!\!\perp}T_2$.

In the example of the fire breaking out in two houses (see section 4):

\[P(A \cap B) = 0.25\]

and

\[P(A) \cdot P(B) = 0.6 \cdot 0.45 = 0.27\]

so the following equality does not hold

\[P(A \cap B) \ne P(A) \cdot P(B)\]

The two events are thus dependent (or not independent), denoted $A \not\!\perp\!\!\!\perp B$.

7. Conditional probability

Suppose two events A and B and $P(B) > 0$. The conditional probability of A given (knowing) B is the likelihood of event A occurring given that event B has occurred:

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\] \[= \frac{P(B \cap A)}{P(B)} \text{ (since } P(A \cap B) = P(B \cap A))\]

Note that, in general, the probability of A given B is not equal to the probability of B given A, that is, $P(A | B) \ne P(B | A)$.

From the formula of the conditional probability, we can derive the multiplicative law:

\[P(A | B) = \frac{P(A \cap B)}{P(B)} \text{ (Eq. 1)}\] \[P(A | B) \cdot P(B) = \frac{P(A \cap B)}{P(B)} \cdot P(B)\] \[P(A | B) \cdot P(B) = P(A \cap B) \text{ (multiplicative law)}\]

If two events are independent, $P(A \cap B) = P(A) \cdot P(B)$, and:

$P(B) > 0$, the conditional probability becomes

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\] \[P(A | B) = \frac{P(A) \cdot P(B)}{P(B)}\] \[P(A | B) = P(A) \text{ (Eq. 2)}\]

$P(A) > 0$, the conditional probability becomes

\[P(B | A) = \frac{P(B \cap A)}{P(A)}\] \[P(B | A) = \frac{P(B) \cdot P(A)}{P(A)}\] \[P(B | A) = P(B) \text{ (Eq. 3)}\] Equations 2 and 3 mean that knowing that one event occurred does not influence the probability of the outcome of the other event. This is in fact the definition of the independence: if knowing that one event occurred does not help to predict (does not influence) the outcome of the other event, the two events are by essence independent.

Bayes’ theorem

From the formulas of the conditional probability and the multiplicative law, we can derive the Bayes’ theorem:

\[\begin{align} P(B | A) &= \frac{P(B \cap A)}{P(A)} \\ & \text{(from conditional probability)} \\ &= \frac{P(A \cap B)}{P(A)} \\ & \text{(since } P(A \cap B) = P(B \cap A)) \\ &= \frac{P(A | B) \cdot P(B)}{P(A)} \\ & \text{ (from multiplicative law)} \end{align}\]

which is equivalent to

\[\begin{align} P(B | A) &= \frac{P(B | A) \cdot P(A)}{P(B)} \\ & \text{(Bayes' theorem)} \end{align}\]

Example

In order to illustrate the conditional probability and the Bayes’ theorem, suppose the following problem:

In order to determine the presence of a disease in a person, a blood test is performed. When a person has the disease, the test can reveal the disease in 80% of cases. When the disease is not present, the test is negative in 90% of cases. Experience has shown that the probability of the disease being present is 10%. A researcher would like to know the probability that an individual has the disease given that the result of the test is positive.

To answer this question, the following events are defined:

P: the test result is positive
D: the person has the disease

Moreover, we use a tree diagram to illustrate the statement:

(The sum of all 4 scenarios must be equal to 1 since these 4 scenarios cover all possible cases.)

We are looking for the probability that an individual has the disease given that the result of the test is positive, $P(D | P)$.

Following the formula of the conditional probability (Eq. 1) we have:

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\]

In terms of our problem:

\[P(D | P) = \frac{P(D \cap P)}{P(P)}\] \[P(D | P) = \frac{0.08}{P(P)} \text{ (Eq. 4)}\]

From the tree diagram, we can see that a positive test result is possible under two scenarios:

when a person has the disease, or
when the person does not actually have the disease (because the test is not always correct).

In order to find the probability of a positive test result, $P(P)$, we need to sum up those two scenarios:

\[\begin{align} P(P) &= P(D \cap P) + P(\bar{D} \cap P) \\ &= 0.08 + 0.09 \\ &= 0.17 \end{align}\]

Eq. 4 then becomes

\[P(D | P) = \frac{0.08}{0.17} = 0.4706\]

The probability of having the disease given that the result of the test is positive is only 47.06%. This means that in this specific case (with the same percentages), an individual with a positive test has less than 1 chance out of 2 of having the disease!

This relatively small percentage is due to the facts that the disease is quite rare (only 10% of the population is affected) and that the test is not always correct (sometimes it detects the disease although it is not present, and sometimes it does not detect it although it is present).

As a consequence, a higher percentage of healthy people have a positive result (9%) compared to the percentage of people who have a positive result and who actually have the disease (8%). This explains why several diagnostic tests are often performed before announcing the diagnosis, especially for rare diseases.

8. Accuracy measures

Based on the example of the disease and the diagnostic test presented above, we explain the most common accuracy measures:

False negative (number and rate)
False positive (number and rate)
Sensitivity
Specificity
Positive predictive value
Negative predictive value

Before diving into the details of these accuracy measures, here is an overview of the measures and the tree diagram with the labels added for each of the 4 scenarios:

Adapted from Wikipedia

False negatives

The false negatives (FN) are the number of people incorrectly labeled as not having the disease or the condition, when in reality it is present. It is like telling a women who is 7 months pregnant that she is not pregnant.

From the tree diagram, we have:

\[FN = P(D \cap \bar{P}) = 0.02\]

Moreover, the false negative rate (FNR) is defined as

\[\begin{align} FNR &= \frac{FN}{FN + TP} \\ &= P(\bar{P} | D) \\ &= \frac{P(\bar{P} \cap D)}{P(D)} \\ &= \frac{0.02}{0.08 + 0.02} \\ &= 0.2 \end{align}\]

False positives

The false positives (FP) are the number of people incorrectly labeled as having the disease or the condition, when in reality it is not present. It is like telling a man he is pregnant.

From the tree diagram, we have:

\[FP = P(\bar{D} \cap P) = 0.09\]

Moreover, the false positive rate (FPR) is defined as

\[\begin{align} FPR &= \frac{FP}{FP + TN} \\ &= P(P | \bar{D}) \\ &= \frac{P(P \cap \bar{D})}{P(\bar{D})} \\ &= \frac{0.09}{0.09 + 0.81} \\ &= 0.1 \end{align}\]

Sensitivity

The sensitivity of a test, also referred as the recall, measures the ability of a test to detect the condition when the condition is present (the percentage of sick people who are correctly identified as having the disease):

\[ Sensitivity = \frac{TP}{TP + FN}\]

where TP is the true positives.

From the tree diagram, we have:

\[\begin{align} Sensitivity &= \frac{TP}{TP + FN} \\ &= P(P|D) \\ &= 0.8 \end{align}\]

Note also that $1 - sensitivity = FNR$ (false negative rate).

Specificity

The specificity of a test measures the ability of a test to correctly exclude the condition when the condition is absent (the percentage of healthy people who are correctly identified as not having the disease):

\[Specificity = \frac{TN}{TN + FP}\]

where TN is the true negatives.

From the tree diagram, we have:

\[\begin{align} Specificity &= \frac{TN}{TN + FP} \\ &= P(\bar{P} | \bar{D}) \\ &= 0.9 \end{align}\]

Note also that $1 - specificity = FPR$ (false positive rate).

Positive predictive value

The positive predictive value, also referred as the precision, is the proportion of positives that correspond to the presence of the condition, so the proportions of positive results that are true positive results:

\[PPV = \frac{TP}{TP+FP}\]

From the tree diagram, we have:

\[\begin{align} PPV &= \frac{TP}{TP+FP} \\ &= P(D | P) \\ &= \frac{P(D \cap P)}{P(P)} \\ &= \frac{0.08}{0.08+0.09} \\ &= 0.4706 \end{align}\]

Negative predictive value

The negative predictive value is the proportion of negatives that correspond to the absence of the condition, so the proportions of negative results that are true negative results:

\[NPV = \frac{TN}{TN + FN}\]

From the tree diagram, we have:

\[\begin{align} NPV &= \frac{TN}{TN + FN} \\ &= P(\bar{D} | \bar{P}) \\ &= \frac{P(\bar{D} \cap \bar{P})}{P(\bar{P})} \\ &= \frac{0.81}{0.81+0.02} \\ &= 0.9759 \end{align}\]

9. Counting techniques

In order to use the formula in section 2, one must know how to count the number of possible elements (both for favorable and possible cases).

There are 3 main counting techniques in probability:

Multiplication
Permutation
Combination

See below how to count the number of possible elements in case of equiprobable results.

Multiplication

The multiplication rule is as follows:

\[\#(A \times B) = (\#A) \times (\#B)\]

where $\#$ is the number of elements.

Example

In a restaurant, a customer has to choose a starter, a main course and a dessert. The restaurant offers 2 starters, 3 main courses and 2 desserts. How many different choices are possible?

There are $2 \cdot 3 \cdot 2 = 12$ different possible choices.

Permutation

The number of permutations is as follows:

\[\begin{align} P^r_n &= n \times (n - 1) \times \cdots \times (n - r + 1) \\ &= \frac{n !}{(n - r)!} \end{align}\]

with $r$ the length, $n$ the number of elements and $r \le n$. Note that $0! = 1$ and $k! = k \times (k - 1) \times (k - 2) \times \cdots \times 2 \times 1$ if $k = 1, 2, \dots$

The order is important in permutations!

Example

Count the permutations of length 2 of the set $A = \{a, b, c, d\}$, without a letter being repeated. How many permutations do you find?

By hand

\[P^4_2 = \frac{4!}{(4-2)!} = \frac{4\cdot3\cdot2\cdot1}{2\cdot1} = 12\]

In R

library(gtools)

x <- c("a", "b", "c", "d")

# See all different permutations
perms <- permutations(
  n = 4, r = 2, v = x,
  repeats.allowed = FALSE
)
perms

##       [,1] [,2]
##  [1,] "a"  "b" 
##  [2,] "a"  "c" 
##  [3,] "a"  "d" 
##  [4,] "b"  "a" 
##  [5,] "b"  "c" 
##  [6,] "b"  "d" 
##  [7,] "c"  "a" 
##  [8,] "c"  "b" 
##  [9,] "c"  "d" 
## [10,] "d"  "a" 
## [11,] "d"  "b" 
## [12,] "d"  "c"

# Count the number of permutations
nrow(perms)

## [1] 12

Combination

The number of combinations is as follows:

\[\begin{align} C^r_n &= \frac{P^r_n}{r!} \\ &= \frac{n !}{r!(n - r)!} \\ &= {n \choose r} \\ &= \frac{n}{r} \times \frac{n - 1}{r - 1} \times \dots \times \frac{n - r + 1}{1} \end{align}\]

with $r$ the length, $n$ the number of elements and $r \le n$.

The order is not important in combinations!

Example

In a family of 5 children, what is the probability that there are 3 girls and 2 boys? Assume that the probabilities of giving birth to a girl and a boy are equal.

By hand

Count of 3 girls and 2 boys (favourable cases): $C^3_5 = {5 \choose 3} = \frac{5!}{3!(5-3)!} = 10$
Count of possible cases: $2^5 = 32$

$\Rightarrow P(3 \text{ girls and 2 boys}) = \frac{\text{# of favourable cases}}{\text{# of possible cases}}$ \[= \frac{10}{32} = 0.3125\]

In R

Count of 3 girls and 2 boys:

choose(n = 5, k = 3)

## [1] 10

Count of possible cases:

2^5

## [1] 32

Probability of 3 girls and 2 boys:

choose(n = 5, k = 3) / 2^5

## [1] 0.3125

Conclusion

Thanks for reading.

I hope this article helped you to understand the most important formulas and concepts from probability theory.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Getting started in R markdown

Tue, 18 Feb 2020 00:00:00 +0000

If you have spent some time writing code in R, you probably have heard of generating dynamic reports incorporating R code, R outputs (results) and text or comments. In this article, I will explain how R Markdown works and give you the basic elements you need to get started easily in the production of these dynamic reports.

R Markdown: what, why and how?

R Markdown allows to generate a report (most of the time in PDF, HTML, Word or as a beamer presentation) that is automatically generated from a file written within RStudio. The generated documents can serve as a neat record of your analysis that can be shared and published in a detailed and complete report. Even if you never expect to present the results to someone else, it can also be used as a personal notebook to look back so you can see what you did at that time. A R Markdown file has the extension .Rmd, while a R script file has the extension .R.

The first main advantage of using R Markdown over R is that, in a R Markdown document, you can combine three important parts of any statistical analysis:

R code to show how the analyses have been done. For instance, the data and the functions you used. This allows readers to follow your code and to check that the analyses were correctly performed.
Results of the code, that is, the output of your analyses. For example, the output of your linear model, plots, or results of the hypothesis test you just coded. This allows readers to see the results of your analyses.
Text, comments and interpretations of the results. For instance, after computing the main descriptive statistics and plotting some graphs, you can interpret them in the context of your problem and highlight important findings. This enables readers to understand your results thanks to your interpretations and your comments, delivered as if you wrote a document explaining your work.

Another advantage of R Markdown is that the reports are dynamic and reproducible by anyone who has access to the .Rmd file (and the data if external data are used of course), making it perfectly suited to collaboration and dissemination of results. By dynamic, we mean that if your data changes, your results and your interpretations will change accordingly, without any work from your side.

The production of the reports is done in two stages:

The .Rmd file which contains blocks of R code (called chunks) and text is provided to the {knitr} package which will execute the R code to get the output, and create a document in markdown (.md) format. This document then contains the R code, the results (or outputs), and the text.
This .md file is then converted to the desired format (HTML, PDF or Word), by the markdown package based on pandoc (i.e., a document conversion tool).

Before you start

To create a new R Markdown document (.Rmd), you first need to install and load the following packages:

install.packages(c("knitr", "rmarkdown", "markdown"))

library(knitr)
library(rmarkdown)
library(markdown)

Then click on File -> New File -> R Markdown or click on the small white sheet with a green cross in the top left corner and select R Markdown:

Create a new R Markdown document

A window will open, choose the title and the author and click on OK. The default output format is HTML. It can be changed later to PDF or Word.

After you have clicked on OK, a new .Rmd file which serves as example has been created. We are going to use this file as starting point to our more complex and more personalized file.

To compile your R Markdown document into a HTML document, click on the Knit button located at the top:

Knit a R Markdown document

A preview of the HTML report appears and it is also saved in your working directory (see a reminder of what is a working directory).

Components of a `.Rmd` file

Below the main components of a R Markdown document:

These components are detailed in the following sections.

YAML header

A .Rmd file starts with the YAML header, enclosed by two series of ---. By default, this includes the title, author, date and the format of the report. If you want to generate the report in a PDF document, replace output: html_document byoutput: pdf_document. These information from the YAML header will appear at the top of the generated report after you compile it (i.e., after knitting the document).

To add a table of contents to your documents, replace output: html_document by

output:
  html_document:
    toc: true

Here are my usual settings regarding the format of a HTML document (remove everything after number_sections: true if you render the document in PDF, as PDF documents do not accept these options in the YAML header):

output:
  html_document:
    toc: true
    toc_depth: 6
    number_sections: true
    toc_float: true
    code_folding: hide
    theme: flatly
    code_download: true

In addition to adding a table of contents, it sets its depth, adds a section numbering, the table of contents is floating when scrolling down the document, the code is hidden by default, the flatly theme is used and it adds the possibility to download the .Rmd document.

You can visualize your table of contents even before knitting the document, or go directly to a specific section by clicking on the small icon in the top right corner. Your table of contents will appear, click on a section to go to this section in your .Rmd document:

Visualize your table of contents

In addition to this enhanced table of contents, I usually set the following date in the YAML header:

Dynamic date in R Markdown

This piece of code allows to write the current date, without having to change it myself. This is very convenient for projects that last several weeks or months to always have an updated date at the top of the document.

Code chunks

Below the YAML header, there is a first code chunk which is used for the setup options of your entire document. It is best to leave it like this at the moment, we can change it later if needed.

Code chunks in R Markdown documents are used to write R code. Every time you want to include R code, you will need to enclose it with three backwards apostrophes. For instance, to compute the mean of the values 1, 7 and 11, we first need to insert a R code chunk by clicking on the Insert button located at the top and select R (see below a picture), then we need to write the corresponding code inside the code chunk we just inserted:

Insert R code chunk in R Markdown

Example of code chunk

In the example file, you can see that the first R code chunk (except the setup code chunk) includes the function summary() of the preloaded dataset cars: summary(cars). If you look at the HTML document that is generated from this example file, you will see that the summary measures are displayed just after the code chunk.

The next code chunk in this example file is plot(pressure), which will produce a plot. Try writing other R codes and knit (i.e., compile the document by clicking on the knit button) the document to see if your code is generated correctly.

If you already wrote code in a R script and want to reuse it in your R Markdown document, you can simply copy paste your code inside code chunks. Do not forget to always include your code inside code chunks or R will throw an error when compiling your document.

As you can see, there are two additional arguments in the code chunk of the plot compared to my code chunk of the mean presented above. The first argument following the letter r (without comma between the two) is used to set the name of the chunk. In general, do not bother with this, it is mainly used to refer to a specific code chunk. You can remove the name of the chunk, but do not remove the letter r between the {} as it tells R that the code that follows corresponds to R code (yes you read it well, that also means you can include code from another programming language, e.g., Python, SQL, etc.).

After the name of the chunk (after pressure in the example file), you can see that there is an additional argument: echo = FALSE. This argument, called an option, indicates that you want to hide the code, and display only the output of the code. Try removing it (or change it to echo = TRUE), and you will see that after knitting the document, both the code AND the output will appear, while only the results appeared previously.

You can specify if you want to hide or display the code alongside the output of that code for each code chunk separately, for instance if you want to show the code for some code chunks, but not for others. Alternatively, if you want to always hide/display the code together with the output for the entire document, you can specify it in the setup code chunk located just after the YAML header. The options passed to this setup code chunk will determine the options for all code chunks, except for those that have been specifically modified.

By default, the only setup option when you open a new R Markdown file is knitr::opts_chunk$set(echo = TRUE), meaning that by default, all outputs will be accompanied by its corresponding code. If you want to display only the results without the code for the whole document, replace it by knitr::opts_chunk$set(echo = FALSE). Two other options often passed to this setup code chunk are warning = FALSE and message = FALSE to prevent warnings and messages to be displayed on the report. If you want to pass several options, do not forget to separate them with a comma:

Several options for a code chunk

You can also choose to display the code, but not the result. For this, pass the option results = "hide". Alternatively, with the option include = FALSE, you can prevent code and results from appearing in the finished file while R still runs the code in order to use it at a later stage. If you want to prevent the code and the results to appear, and do not want R to run the code, use eval = FALSE. To edit the width and height of figures, use the options fig.width and fig.height. Another very interesting option is the tidy = 'styler' option which automatically reformats the R code shown in the output.

See all options and their description here or see the list of the default options by running str(knitr::opts_chunk$get()).

Tip: When writing in R Markdown, you will very often need to insert new R code chunks. To insert a new R code chunk more rapidly, press CTRL + ALT + I on Windows or command + option + I on Mac. If you are interested in such shortcuts making you more efficient, see other tips and tricks in R Markdown.

Note that a code chunk can be run without the need to compile the entire document, if you want to check the results of a specific code chunk for instance. In order to run a specific code chunk, select the code and run it as you would do in a R script (.R), by clicking on run or by pressing CTRL + Enter on Windows or command + Enter on Mac. Results of this code chunk will be displayed directly in the R Markdown document, just below the code chunk.

Text

Text can be added everywhere outside code chunks. R Markdown documents use the Markdown syntax for the formatting of the text. In our example file just below the setup code chunk, some text has been inserted. To insert text, you simply write text without any enclosing. Try adding some sentences and knit the document to see how it appears in the HTML document.

Example of text below an example of R code chunk

Markdown syntax can be used to change the formatting of your text appearing in the output file, for example to format some text in italics, in bold, etc. Below some common formatting commands:

Title: # Title
Subtitle: ## Subtitle
Subsubtitle: ### Subsubtitle. These headings will automatically be included in the table of contents if you included one.
italics: *italics* or _italics_
bold: **bold** or __bold__
Link: [link](https://statsandr.com/) (do not forget https:// or http:// if it is an external URL)
Equations:

Enclose your equation (written in LaTeX) with one $ to have it in the text:

$A = \pi*r^{2}$

This is a well-know equation $A = \pi*r^{2}$.

Enclose your LaTeX equation with two $$ to have it centered on a new line:

$$A = \pi*r^{2}$$

This is another well-known equation:

\[E = mc^2\]

Lists:

Unordered list, item 1: * Unordered list, item 1
Unordered list, item 2: * Unordered list, item 2

Ordered list, item 1: 1. Ordered list, item 1
Ordered list, item 2: 2. Ordered list, item 2

Code inside text

Before going further, I would like to introduce an important feature of R Markdown. It is often the case that, when writing interpretations or detailing an analysis, we would like to refer to a result directly in our text. For instance, suppose we work on the iris dataset (preloaded in R). We may want to explain in words, that the mean of the length of the petal is a certain value, while the median is another value.

Without R Markdown, the user would need to compute the mean and median, and then report it manually. Thanks to R Markdown, it is possible to report these two descriptive statistics directly in the text, without manually encoding it. Even better, if the dataset happens to change because we removed some observations, the mean and median reported in the generated document will change automatically, without any change in the text from our side.

We can insert results directly in the interpretations (i.e., in the text) by placing a backward apostrophe, the letter r, a space, the code, and then close it with another backward apostrophe:

Inline code in R Markdown

Here is an illustration with the mean and median of the length of the sepal for the iris dataset integrated in a sentence:

Example of inline code and text

This combination of text and code will give the following output in the generated report:

The mean of the length of the sepal is 5.8433333 and the standard deviation is 0.8280661.

This technique, referred as inline code, allows you to insert results directly into the text of a R Markdown document. And as mentioned, if the dataset changes, the results incorporated inside the text (the mean and standard deviation in our case) will automatically be adjusted to the new dataset, exactly like the output of a code chunk is dynamically updated if the dataset changes.

This technique of inline code and the fact that it is possible to combine code, outputs of code, and text to comment the outputs makes R Markdown my favorite tool when it comes to statistical analyses. Since I discovered the power of R Markdown (and I am still learning as it has a huge amount of possibilities and features), I almost never write R code in scripts anymore. Every R code I write is supplemented by text and inline code in a R Markdown document, resulting in a professional and complete final document ready to be shared, published, or stored for future usage. If you are unfamiliar to this type of document, I invite you to learn more about it and to try it with your next analysis, you will most likely not go back to R scripts anymore.

Highlight text like it is code

Alternatively to the inline code technique, you may want to make some text appears as if it is a piece of code in the generated report, without actually running it.

For this, surround your text with back ticks (the same backward apostrophe used for inline code) without the letter r. Writing the following:

will produce this in the generated report:

For example, in this sentence I would like to highlight the variable name Species from the dataframe iris as if it is a piece of code.

The word “Species” and “iris” appear and are highlighted as if it is a piece of code.

Images

In addition to code, results and text, you can also insert images in your final document. To insert an image, place it in your current working directory, and outside a code chunk write:

![](path_to_your_image.jpg)

Note that the the file/url path is NOT quoted. To add an alt text to your image, add it between the square brackets []:

![alt text here](path_to_your_image.jpg)

Tables

There are two options to insert tables in R Markdown documents:

the kable() function from the {knitr} package
the pander() function from the {pander} package

Here are an example of a table without any formatting, and the same code with the two functions applied on the iris dataset:

# without formatting
summary(iris)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
##

# with kable()
library(knitr)
kable(summary(iris))

# with pander()
library(pander)
pander(summary(iris))

The advantage of pander() over kable() is that it can be used for many more different outputs than table. Try on your own code, with results of a linear regression or a simple vector for example.

Additional notes and useful resources

For more advanced users, R Markdown files can also be used to create Shiny apps, websites (this website is built thanks to R Markdown and the {blogdown} package), to write scientific papers based on templates from several international journals (with the {rticles} package), or even to write books (with the {bookdown} package).

To continue learning about R Markdown, see two complete cheat sheets from the R Studio team here and here, and a more complete guide here written by Yihui Xie, J. J. Allaire and Garrett Grolemund.

Conclusion

Thanks for reading.

I hope this article convinced you to use R Markdown for your future projects. See more tips and tricks in R Markdown to increase even further your efficiency in R Markdown.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Do my data follow a normal distribution? A note on the most widely used distribution and how to test for normality in R

Wed, 29 Jan 2020 00:00:00 +0000

What is a normal distribution?

The normal distribution is a function that defines how a set of measurements is distributed around the center of these measurements (i.e., the mean). Many natural phenomena in real life can be approximated by a bell-shaped frequency distribution known as the normal distribution or the Gaussian distribution.

The normal distribution is a mount-shaped, unimodal and symmetric distribution where most measurements gather around the mean. Moreover, the further a measure deviates from the mean, the lower the probability of occurring. In this sense, for a given variable, it is common to find values close to the mean, but less and less likely to find values as we move away from the mean. Last but not least, since the normal distribution is symmetric around its mean, extreme values in both tails of the distribution are equivalently unlikely. For instance, given that adult height follows a normal distribution, most adults are close to the average height and extremely short adults occur as infrequently as extremely tall adults.

In this article, the focus is on understanding the normal distribution, the associated empirical rule, its parameters and how to compute $Z$-scores to find probabilities under the curve (illustrated with examples). As it is a requirement in some statistical tests and hypothesis tests, we also show 4 complementary methods to test the normality assumption in R.

Empirical rule

Data possessing an approximately normal distribution have a definite variation, as expressed by the following empirical rule:

$\mu \pm \sigma$ includes approximately 68% of the observations
$\mu \pm 2 \cdot \sigma$ includes approximately 95% of the observations
$\mu \pm 3 \cdot \sigma$ includes almost all of the observations (99.7% to be more precise)

Normal distribution & empirical rule (68-95-99.7% rule)

where $\mu$ and $\sigma$ correspond to the population mean and population standard deviation, respectively.

The empirical rule, also known as the 68-95-99.7% rule, is illustrated by the following 2 examples.

Suppose that the scores of an exam in statistics given to all students in a Belgian university are known to have, approximately, a normal distribution with mean $\mu = 67$ and standard deviation $\sigma = 9$. It can then be deduced that approximately 68% of the scores are between 58 and 76, that approximately 95% of the scores are between 49 and 85, and that almost all of the scores (99.7%) are between 40 and 94. Thus, knowing the mean and the standard deviation gives us a fairly good picture of the distribution of scores.

Now suppose that a single university student is randomly selected from those who took the exam. What is the probability that her score will be between 49 and 85? Based on the empirical rule, we find that 0.95 is a reasonable answer to this probability question.

The utility and value of the empirical rule are due to the common occurrence of approximately normal distributions of measurements in nature. For example, IQ, shoe size, height, birth weight, etc. are approximately normally-distributed. You will find that approximately 95% of these measurements will be within $2\sigma$ of their mean (Wackerly, Mendenhall, and Scheaffer 2014).

Parameters

Like many probability distributions, the shape and probabilities of the normal distribution is defined entirely by some parameters. The normal distribution has two parameters:

the mean $\mu$, and
the variance $\sigma^2$ (i.e., the square of the standard deviation $\sigma$).

The mean $\mu$ locates the center of the distribution, that is, the central tendency of the observations, and the variance $\sigma^2$ defines the width of the distribution, that is, the spread of the observations.

The mean $\mu$ can take on any finite value (i.e., $-\infty < \mu < \infty$), whereas the variance $\sigma^2$ can assume any positive finite value (i.e., $\sigma^2 > 0$). The shape of the normal distribution changes based on these two parameters. Since there is an infinite number of combinations of the mean and variance, there is an infinite number of normal distributions, and thus an infinite number of forms.

For instance, see how the shapes of the normal distributions vary when the two parameters change:

As you can see on the second graph, when the variance (or the standard deviation) decreases, the observations are closer to the mean. On the contrary, when the variance (or standard deviation) increases, it is more likely that observations will be further away from the mean.

A random variable $X$ which follows a normal distribution with a mean of 430 and a variance of 17 is denoted $X ~ \sim \mathcal{N}(\mu = 430, \sigma^2 = 17)$.

We have seen that, although different normal distributions have different shapes, all normal distributions have common characteristics:

They are symmetric, 50% of the population is above the mean and 50% of the population is below the mean
The mean, median and mode are equal
The empirical rule detailed earlier is applicable to all normal distributions

Probabilities and standard normal distribution

Probabilities and quantiles for random variables with normal distributions are easily found using R via the functions pnorm() and qnorm(). Probabilities associated with a normal distribution can also be found using this Shiny app. However, before computing probabilities, we need to learn more about the standard normal distribution and the $Z$-score.

Although there are infinitely many normal distributions (since there is a normal distribution for every combination of mean and variance), we need only one table to find the probabilities under the normal curve: the standard normal distribution.

The normal standard distribution is a special case of the normal distribution where the mean is equal to 0 and the variance is equal to 1. A normal random variable $X$ can always be transformed to a standard normal random variable $Z$, a process known as “scaling” or “standardization”, by subtracting the mean from the observation, and dividing the result by the standard deviation. Formally:

\[Z = \frac{x - \mu}{\sigma}\]

where $x$ is the observation, $\mu$ and $\sigma$ the mean and standard deviation of the population from which the observation was drawn. So the mean of the standard normal distribution is 0, and its variance is 1, denoted $Z ~ \sim \mathcal{N}(\mu = 0, \sigma^2 = 1)$.

From this formula, we see that $Z$, referred as standard score or $Z$-score, allows to see how far away one specific observation is from the mean of all observations, with the distance expressed in standard deviations. In other words, the $Z$-score corresponds to the number of standard deviations an observation is away from the mean. A positive $Z$-score means that the specific observation is above the mean, whereas a negative $Z$-score means that the specific observation is below the mean. $Z$-scores are often used to compare an individual to her peers, or more generally, a measurement compared to its distribution.

For instance, suppose a student scoring 60 at a statistics exam with the mean score of the class being 40, and scoring 65 at an economics exam with the mean score of the class being 80. Given the “raw” scores, one would say that the student performed better in economics than in statistics. However, taking into consideration her peers, it is clear that the student performed relatively better in statistics than in economics.

Computing $Z$-scores allows to take into consideration all other students (i.e., the entire distribution) and gives a better measure of comparison. Let’s compute the $Z$-scores for the two exams, assuming that the score for both exams follow a normal distribution with the following parameters:

	Statistics	Economics
Mean	40	80
Standard deviation	8	12.5
Student’s score	60	65

$Z$-scores for:

Statistics: $z_{stat} = \frac{60 - 40}{8} = 2.5$
Economics: $z_{econ} = \frac{65 - 80}{12.5} = -1.2$

On the one hand, the $Z$-score for the exam in statistics is positive ($z_{stat} = 2.5$) which means that she performed better than average. On the other hand, her score for the exam in economics is negative ($z_{econ} = -1.2$) which means that she performed worse than average. Below an illustration of her grades in a standard normal distribution for better comparison:

Although the score in economics is better in absolute terms, the score in statistics is actually relatively better when comparing each score within its own distribution.

Furthermore, $Z$-score also enables to compare observations that would otherwise be impossible because they have different units for example. Suppose you want to compare a salary in € with a weight in kg. Without standardization, there is no way to conclude whether someone is more extreme in terms of her wage or in terms of her weight. Thanks to $Z$-scores, we can compare two values that were in the first place not comparable to each other.

Final remark regarding the interpretation of a $Z$-score: a rule of thumb is that an observation with a $z$-score between -3 and -2 or between 2 and 3 is considered as a rare value. An observation with a $z$-score smaller than -3 or larger than 3 is considered as an extremely rare value. A value with any other $z$-score is considered as not rare nor extremely rare.

Areas under the normal distribution in R and by hand

Now that we have covered the $Z$-score, we are going to use it to determine the area under the curve of a normal distribution.

Note that there are several ways to arrive at the solution in the following exercises. You may therefore use other steps than the ones presented to obtain the same result.

Ex. 1

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(Z > 1)$.

We actually look for the shaded area in the following figure:

Standard normal distribution: $P(Z > 1)$

In R

pnorm(1,
  mean = 0,
  sd = 1, # sd stands for standard deviation
  lower.tail = FALSE
)

## [1] 0.1586553

We look for the probability of $Z$ being larger than 1 so we set the argument lower.tail = FALSE. The default lower.tail = TRUE would give the result for $P(Z < 1)$. Note that $P(Z = 1) = 0$ so writing $P(Z > 1)$ or $P(Z \ge 1)$ is equivalent.

By hand

See that the random variable $Z$ has already a mean of 0 and a standard deviation of 1, so no transformation is required. To find the probabilities by hand, we need to refer to the standard normal distribution table shown below:

Standard normal distribution table (Wackerly, Mendenhall, and Scheaffer 2014).

From the illustration at the top of the table, we see that the values inside the table correspond to the area under the normal curve above a certain $z$. Since we are looking precisely at the probability above $z = 1$ (since we look for $P(Z > 1)$), we can simply proceed down the first ($z$) column in the table until $z = 1.0$. The probability is 0.1587. Thus, $P(Z > 1) = 0.1587$. This is similar to what we found using R, except that values in the table are rounded to 4 digits.

Ex. 2

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(−1 \le Z \le 1)$.

We are looking for the shaded area in the following figure:

Standard normal distribution: $P(−1 \le Z \le 1)$

In R

pnorm(1, lower.tail = TRUE) - pnorm(-1, lower.tail = TRUE)

## [1] 0.6826895

Note that the arguments by default for the mean and the standard deviation are mean = 0 and sd = 1. Since this is what we need, we can omit them.¹

By hand

For this exercise we proceed by steps:

The shaded area corresponds to the entire area under the normal curve minus the two white areas in both tails of the curve.
We know that the normal distribution is symmetric.
Therefore, the shaded area is the entire area under the curve minus two times the white area in the right tail of the curve, the white area in the right tail of the curve being $P(Z > 1)$.
We also know that the entire area under the normal curve is 1.
Thus, the shaded area is 1 minus 2 times $P(Z > 1)$:

\[P(−1 \le Z \le 1) = 1 - 2 \cdot P(Z > 1)\] \[= 1 - 2 \cdot 0.1587 = 0.6826\]

where $P(Z > 1) = 0.1587$ has been found in the previous exercise.

Ex. 3

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(0 \le Z \le 1.37)$.

We are looking for the shaded area in the following figure:

Standard normal distribution: $P(0 \le Z \le 1.37)$

In R

pnorm(0, lower.tail = FALSE) - pnorm(1.37, lower.tail = FALSE)

## [1] 0.4146565

By hand

Again we proceed by steps for this exercise:

We know that $P(Z > 0) = 0.5$ since the entire area under the curve is 1, half of it is 0.5.
The shaded area is half of the entire area under the curve minus the area from 1.37 to infinity.
The area under the curve from 1.37 to infinity corresponds to $P(Z > 1.37)$.
Therefore, the shaded area is $0.5 - P(Z > 1.37)$.
To find $P(Z > 1.37)$, proceed down the $z$ column in the table to the entry 1.3 and then across the top of the table to the column labeled .07 to read $P(Z > 1.37) = .0853$
Thus,

\[P(0 \le Z \le 1.37) = P(Z > 0) - P(Z > 1.37)\] \[ = 0.5 - 0.0853 = 0.4147\]

Ex. 4

Recap the example presented in the empirical rule: Suppose that the scores of an exam in statistics given to all students in a Belgian university are known to have a normal distribution with mean $\mu = 67$ and standard deviation $\sigma = 9$. What fraction of the scores lies between 70 and 80?

We are looking for the shaded area in the following figure:

$P(70 \le X \le 80)$ where $X \sim \mathcal{N}(\mu = 67, \sigma^2 = 9^2)$

In R

pnorm(70, mean = 67, sd = 9, lower.tail = FALSE) - pnorm(80, mean = 67, sd = 9, lower.tail = FALSE)

## [1] 0.2951343

By hand

Remind that we are looking for $P(70 \le X \le 80)$ where $X \sim \mathcal{N}(\mu = 67, \sigma^2 = 9^2)$. The random variable $X$ is in its “raw” format, meaning that it has not been standardized yet since the mean is 67 and the variance is $9^2$. We thus need to first apply the transformation to standardize the endpoints 70 and 80 with the following formula:

\[Z = \frac{X - \mu}{\sigma}\]

After the standardization, $x = 70$ becomes (in terms of $z$, so in terms of deviation from the mean expressed in standard deviation):

\[z = \frac{70 - 67}{9} = 0.3333\]

and $x = 80$ becomes:

\[z = \frac{80 - 67}{9} = 1.4444\]

The figure above in terms of $X$ is now in terms of $Z$:

$P(0.3333 \le Z \le 1.4444)$ where $Z \sim \mathcal{N}(\mu = 0, \sigma^2 = 1)$

Finding the probability $P(0.3333 \le Z \le 1.4444)$ is similar to exercises 1 to 3:

The shaded area corresponds to the area under the curve from $z = 0.3333$ to $z = 1.4444$.
In other words, the shaded area is the area under the curve from $z = 0.3333$ to infinity minus the area under the curve from $z = 1.4444$ to infinity.
From the table, $P(Z > 0.3333) = 0.3707$ and $P(Z > 1.4444) = 0.0749$
Thus:

\[P(0.3333 \le Z \le 1.4444)\] \[= P(Z > 0.3333) - P(Z > 1.4444)\] \[= 0.3707 - 0.0749 = 0.2958\]

The difference with the probability found using in R comes from the rounding.

To conclude this exercise, we can say that, given that the mean scores is 67 and the standard deviation is 9, 29.58% of the students scored between 70 and 80.

Ex. 5

See another example in a context here.

Why is the normal distribution so crucial in statistics?

The normal distribution is important for three main reasons:

Some statistical hypothesis tests assume that the data follow a normal distribution
The central limit theorem states that, for a large number of observations (usually $n > 30$), no matter what is the underlying distribution of the original variable, the distribution of the sample means ($\overline{X}_n$) and of the sum ($S_n = \sum_{i = 1}^n X_i$) may be approached by a normal distribution (Stevens 2013)
Linear and nonlinear regression assume that the residuals are normally-distributed (for small sample sizes)

It is therefore useful to know how to test for normality in R, which is the topic of next sections.

How to test the normality assumption

As mentioned above, some statistical tests require that the data follow a normal distribution, or the result of the test may be flawed.

In this section, we show 4 complementary methods to determine whether your data follow a normal distribution in R.

Histogram

A histogram displays the spread and shape of a distribution, so it is a good starting point to evaluate normality.

Let’s have a look at the histogram of a distribution that we would expect to follow a normal distribution, the height of 1,000 adults in cm:

The normal curve with the corresponding mean and variance has been added to the histogram. The histogram follows the normal curve so the data seems to follow a normal distribution.

Below the minimal code for a histogram in R with the dataset iris:

data(iris)
hist(iris$Sepal.Length)

In {ggplot2}:

ggplot(iris) +
  aes(x = Sepal.Length) +
  geom_histogram()

Histograms are however not sufficient, particularly in the case of small samples because the number of bins greatly change its appearance. Histograms are not recommended when the number of observations is less than 20 because it does not always correctly illustrate the distribution. See two examples below with datasets of 10 and 12 observations:

Can you tell whether these datasets follow a normal distribution? Surprisingly, both series are generated from a normal distribution!

In the remaining of the article, we will use the dataset of the 12 adults. If you would like to follow my code in your own script, here is how I generated the data:

set.seed(42)
dat_hist <- data.frame(
  value = rnorm(12, mean = 165, sd = 5)
)

The rnorm() function generates random numbers from a normal distribution (12 random numbers with a mean of 165 and standard deviation of 5 in this case). These 12 observations are then saved in the dataset called dat_hist under the variable value. Note that set.seed(42) is important to obtain the exact same data as me.²

Density plot

Density plots also provide a visual judgment about whether the data follow a normal distribution.

They are similar to histograms as they also allow to analyze the spread and the shape of the distribution. However, they are a smoothed version of the histogram.

Here is the density plot drawn from the dataset on the height of the 12 adults discussed above:

plot(density(dat_hist$value))

In {ggpubr}:

library(ggpubr) # package must be installed first
ggdensity(dat_hist$value,
  main = "Density plot of adult height",
  xlab = "Height (cm)"
)

Since it is hard to test for normality from histograms and density plots only, it is recommended to corroborate these graphs with a QQ-plot. QQ-plot, also known as normality plot, is the third method presented to evaluate normality.

QQ-plot

Like histograms and density plots, QQ-plots allow to visually evaluate the normality assumption.

Here is the QQ-plot drawn from the dataset on the height of the 12 adults discussed above:

library(car)
qqPlot(dat_hist$value)

## [1] 12  2

In {ggpubr}:

library(ggpubr)
ggqqplot(dat_hist$value)

Instead of looking at the spread of the data (as it is the case with histograms and density plots), with QQ-plots we only need to ascertain whether the data points follow the line (sometimes referred as Henry’s line).

If points are close to the reference line and within the confidence bands, the normality assumption can be considered as met. The bigger the deviation between the points and the reference line and the more they lie outside the confidence bands, the less likely that the normality condition is met. The height of these 12 adults seem to follow a normal distribution because points follow the line and all of them lie within the confidence bands.

When facing a non-normal distribution as shown by the QQ-plot below (systematic departure from the reference line), the first step is usually to apply the logarithm transformation on the data and recheck to see whether the log-transformed data are normally distributed.

If this is the case, the data are said to follow a log-normal distribution. Applying the logarithm transformation can be done in R with the log() function.

Note that QQ-plots are also a convenient way to assess whether residuals from linear regression follow a normal distribution.

Normality test

The 3 tools presented above were a visual inspection of the normality. Nonetheless, visual inspection may sometimes be unreliable so it is also possible to formally test whether the data follow a normal distribution with statistical tests.

These normality tests compare the distribution of the data to a normal distribution in order to assess whether observations show an important deviation from normality.

The two most common normality tests are Shapiro-Wilk’s test and Kolmogorov-Smirnov test. Both tests have the same hypotheses, that is:

$H_0$: the data follow a normal distribution
$H_1$: the data do not follow a normal distribution

Shapiro-Wilk test is recommended for normality test as it provides better power than Kolmogorov-Smirnov test.³ In R, the Shapiro-Wilk test of normality can be done with the function shapiro.test():⁴

shapiro.test(dat_hist$value)

## 
##  Shapiro-Wilk normality test
## 
## data:  dat_hist$value
## W = 0.93968, p-value = 0.4939

From the output, we see that the $p$-value $> 0.05$ implying that we do not reject the null hypothesis that the data follow a normal distribution. This test goes in the same direction than the QQ-plot, which showed no significant deviation from the normality (as all points lied within the confidence bands).

It is important to note that, in practice, normality tests are often considered as too conservative in the sense that for large sample size ($n > 50$), a small deviation from the normality may cause the normality condition to be violated.

A normality test is a hypothesis test, so as the sample size increases, their capacity of detecting smaller differences increases. So as the number of observations increases, the Shapiro-Wilk test becomes very sensitive even to a small deviation from normality. As a consequence, it happens that according to the normality test the data do not follow a normal distribution although the departures from the normal distribution is negligible so the data could in fact be considered to follow approximately a normal distribution. For this reason, it is often the case that the normality condition is verified based on a combination of all methods presented in this article, that is, visual inspections (with histograms and QQ-plots) and a formal inspection (with the Shapiro-Wilk test for instance).

I personally tend to prefer QQ-plots over histograms and normality tests so I do not have to bother about the sample size. This article showed the different methods that are available, your choice will of course depends on the type of your data and the context of your analyses.

Conclusion

Thanks for reading.

I hope the article helped you to learn more about the normal distribution and how to test for normality in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Wackerly, Dennis, William Mendenhall, and Richard L Scheaffer. 2014. Mathematical Statistics with Applications. Cengage Learning.

The argument lower.tail = TRUE is also the default so we could omit it as well. However, for clarity and to make sure I compute the propabilities in the correct side of the curve, I used to keep this argument explicit by writing it.↩︎
The set.seed() function accepts any numeric as argument. Generating random numbers (via rnorm() for instance) implies that R will generates different random numbers every time you generate these random numbers (so every time you run the function rnorm()). To make sure R generates the exact same numbers every time you run the function, a seed can be set with the function set.seed(). Setting a seed implies that R will generate random numbers, but these numbers will always be the same as long as the seed is the same. This allows to replicate results that are based on a random generation. Change the seed if you want to generate other random values.↩︎
The Shapiro-Wilk test is based on the correlation between the sample and the corresponding normal scores.↩︎
In R, the Kolmogorov-Smirnov test is performed with the function ks.test().↩︎

RStudio addins, or how to make your coding life easier?

Sun, 26 Jan 2020 00:00:00 +0000

What are RStudio addins?

Although I have been using RStudio for several years, I only recently discovered RStudio addins. Since then, I am using these addins almost every time I use RStudio.

What are RStudio addins? RStudio addins are extensions which provide a simple mechanism for executing advanced R functions from within RStudio. In simpler words, when executing an addin (by clicking a button in the Addins menu), the corresponding code is executed without you having to write the code. If it is still not clear, remember that for importing a dataset in RStudio, you have two options:

import it by writing the code (thanks to the read.csv() function for instance)
or you can import it by clicking on the “Import Dataset” button in the Environment pane, set the importing settings, then click on “Import”

An RStudio addin is exactly like the Import Dataset button but for other common functionalities. So you could write code as you can import a dataset by writing code, but thanks to RStudio addins you can execute code without actually writing the necessary code. By using the RStudio addins, RStudio will run the required code for you. RStudio addins can be as simple as a function that inserts a commonly used snippet of code, and as complex as a Shiny application that accepts input from the user to draw a plot. RStudio addins have the advantage that they allow you to execute complex and advanced code much more easily than if you would have to write it yourself.

I believe addins are worth trying for all R users. Beginners will have the possibility to use functions that they would not have used otherwise because the code is too complex, whereas advanced users may find them useful to speed up the writing of their code in some circumstances. For other tips in R, see the article “Tips and tricks in RStudio and R Markdown”.

Installation

RStudio Addins are distributed as R packages. So before being able to use them, you need to install them.

You can install an addin exactly the same way you install a package: install.packages("name_of_addin"). Once you have installed the R package that contains the addin, it will immediately become available within RStudio, via the Addins menu located at the top.

RStudio addins toolbar

See this guide if you want to install a personal package as addin. In short, you have to create an R package, put your functions in a specific file and RStudio will automatically discover and register these addins when your package is installed.

Addins

If you are still not convinced, see below a list of the addins I find most useful together with concrete examples in the following sections.

Note that this list is not exhaustive and you are likely to find others useful too depending on what type of analyses you do on RStudio. Feel free to comment at the end of the article to let me know (and other readers) addins you found worth using.

Esquisse

{esquisse} is an addin developed by a French company called dreamRs. Here is how they define it:

This addin allows you to interactively explore your data by visualizing it with the ggplot2 package. It allows you to draw bar plots, curves, scatter plots, histograms, boxplot and sf objects, then export the graph or retrieve the code to reproduce the graph.

With this addin you can easily create beautiful graphs from the {ggplot2} package and the best part according to me is that you can retrieve the code to reproduce the graph. Compared to the default {graphics} package, it is true that graphs from the {ggplot2} package look usually better but the code is also longer and more complex. With this addin, you can draw graphs from the {ggplot2} package by dragging and dropping variables of interest in an user-friendly and interactive window, and then use the generated code in your script.

For the sake of illustration, let’s say we want to create a scatter plot of the variables Sepal.Length and Petal.Length of the dataset iris and color the points by the variable Species. For this, follow these steps:

load the dataset and rename it:¹

dat <- iris

install the package {esquisse}. This must be done only once

install.packages("esquisse")

open the ‘ggplot2’ builder from the RStudio Addins menu:

Step 3: Open ‘ggplot2’ builder from the RStudio addins menu

Select the dataset you want to work on (in this case dat) and click on “Validate imported data” after checking that the number of observations and variables are correct (green box):

Step 4: Select dataset and validate the imported data

Drag and drop the variables of interest in the corresponding areas. In this case, we would like to draw a scatter plot of the variables Sepal.Length and Petal.Length and color points based on the variable Species:

Step 5: Drag and drop the variables in the corresponding areas

Click on “</> Export & code” at the bottom right of the window. You can either copy the code and paste it where you want to place it in your script, or you can click on “Insert code in script” to place the code where your cursor is located in your script:

Step 6: Retrieve the code to use it in your script

If you chose the second option, the following code should appear where your cursor was located:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length, colour = Species) +
  geom_point(shape = "circle", size = 1.5) +
  scale_color_hue(direction = 1) +
  theme_minimal()

Many different options and customizations are possible (e.g., axis labels, colors, legend position, theme, data filtering, etc.). For this, use the buttons located at the bottom of the window (“Labels & title”, “Plot options” and “Data”). You can see the changes instantly in the window, and when the plot corresponds to your needs, export the code into your script.

I will not go into more details regarding the different types of plots and the customizations, but make sure to try other types of plots by moving variables and customize it to see what is possible.

ggThemeAssist

Note that there is another addin called {ggThemeAssist} which helps you to edit the theme() layer of a {ggplot2} plot.

This layer allows you to modify the appearance of the background, grids, axes, labels, legend, (sub)title, caption, etc.

To use this addin:

Write the code of the plot in {ggplot2} (it does not work for a plot written in base R)
Highlight the code
Select the {ggThemeAssist} addin in the addins menu
Customize your plot according to your needs
Click on the button Done and your code will be edited with your changes:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length, colour = Species) +
  geom_point(shape = "circle", size = 1.5) +
  scale_color_hue(direction = 1) +
  theme_minimal() +
  theme(
    panel.grid.major = element_line(linetype = "blank"),
    panel.grid.minor = element_line(linetype = "blank"),
    legend.position = c(0.88, 0.22)
  ) +
  theme(plot.caption = element_text(face = "italic")) +
  labs(
    title = "Sepal & petal length by species",
    x = "Sepal length", y = "Petal length",
    caption = "Data: iris"
  )

Find more information about the theme() layer in this cheatsheet.

Questionr

The {questionr} addin is useful for survey analysis and when dealing with factor variables. With this addin, you can easily reorder and recode factor variables. The addin also allows to easily transform a numeric variable into factors (i.e., categorize a continuous variable) thanks to the cut() function. Like any other addin, after having installed the {questionr} package, you should see it appearing in the addins menu at the top. From the addins dropdown menu, choose whether you want to reorder or recode your factor variable, or categorize your numeric variable.

Recoding factors

Instead of writing the recode() function from the {dplyr} package, we can use the {questionr} addin.

For this example, let’s say we want to recode the Species variable to shorten the length of the factors, and then store this new variable as Species_rec:

Step 1: Select the variable to be recoded and the recoding settings

Step 2: Specify the names of the new factors

Step 3: Check the results thanks to the contingency table and use the code at the top

Reordering factors

Similar to recoding, we can reorder factors thanks to the {questionr} addin. Let’s say we want to reorder the 3 factors of the Species variable such that the order is versicolor then virginica and finally setosa. This can be done as follows:

Step 1: Select the variable to reorder and the new variable name

Step 2: Specify the order you want

Step 3: Use the code at the top in your script

Categorize a numeric variable

The {questionr} addin also allows to transform a numeric variable into a categorical variable. This is often done for the age, when age is transformed into age groups for instance. For this example, let’s say we want to create 3 categories of the variable Sepal.Length:

Step 1: Choose the variable to be transformed and the new variable name

Step 2: Set the number of breaks equal to 3

(Try by yourself the other cutting methods and see the results directly in the window.)

Step 3: Check the result thanks to the barplot at the bottom

Step 4: Use the code in your script

Remedy

If you often write in R Markdown, the {remedy} addins will greatly facilitate your work. The addins allow you to add bold, creating lists, urls, italics, title (H1 to H6), footnote, etc. in an efficient way. I reckon that some of these tasks can be done faster by using the code directly rather than by going through the addins menu and then selecting the transformations you want. However, I personally cannot remember the code for all transformations, and it is quicker to apply it through the menu than by searching for the answer on Google or on your Markdown cheat sheet.

Styler

The {styler} addin allows to reformat your code in a more readable format by running styler::tidyverse_style(). It works both for R scripts and R Markdown documents. You can either reformat the selected code, the active file or the active package. I find this addin particularly useful before sharing or publishing my code, so that it respects the most common styling guidelines for code.

For instance, a piece of code like this:

1+1
#this is a comment
  for(i in 1:10){if(!i%%2){next}
print(i)
 }

becomes much more neat and readable:

1 + 1
# this is a comment
for (i in 1:10) {
  if (!i %% 2) {
    next
  }
  print(i)
}

Snakecaser

The {snakecaser} addins converts a character string to the snake case styling. Snake case styling is the practice of writing character strings of several words separated by space into character strings with words separated with an underscore (_). Moreover, it replaces capital letters with lowercases. For instance, the following string:

This is the Test 1

will be transformed to:

this_is_the_test_1

The snake case styling is particularly useful (and even recommended by many R users) for variables, functions and file names, etc.

ViewPipeSteps

Thanks to a reader of this article, I discovered the ViewPipeSteps addin. This addin allows to print or view the output of your pipe chain after each step.

For instance, here is a chain with the dataset diamonds:

library(tidyverse)

diamonds %>%
  select(carat, cut, color, clarity, price) %>%
  group_by(color) %>%
  summarise(n = n(), price = mean(price)) %>%
  arrange(desc(color))

## # A tibble: 7 × 3
##   color     n price
##   <ord> <int> <dbl>
## 1 J      2808 5324.
## 2 I      5422 5092.
## 3 H      8304 4487.
## 4 G     11292 3999.
## 5 F      9542 3725.
## 6 E      9797 3077.
## 7 D      6775 3170.

If you are unsure about your pipe chain or want to debug it, you can view the output after each step by highlighting your entire chain then clicking on “View Pipe Chain Steps” from the addins menu:

From the addins menu, you can choose to either print the result to the console, or view the result in a new pane (as if you called the function View() after each step of the pipe). Clicking on View Pipe Chain Steps will open a new window with the output of each step:

Note that you have to use the following command to install the ViewPipeSteps addin:

devtools::install_github("daranzolin/ViewPipeSteps")
library(ViewPipeSteps)

Now, you have no more excuses to use this pipe operator!

Ymlthis

ymlthis addin to easily write YAML header

The ymlthis addin makes it easy to write YAML front matter for R Markdown and related documents. The addin will create YAML for you and put it in a file, such as an .Rmd file, or on your clipboard.

This addin is particularly useful if you want to write (more complicated) YAML header.

Reprex

If you often ask the R community for help, this addin may be very useful!

It is important to remember that when you ask someone for help, you have to help them help you by giving a precise and clear overview of your problem. This helps the community to quickly understand your issue and thus reduces the response time.

In most cases, this involves providing a reproducible example, that is to say a piece of code (as small and as readable as possible) reproducing the problem encountered.

The {reprex} addin allows you to transform your reproducible example so that it can easily be shared on platforms such as GitHub, Stack Overflow, RStudio community, etc. The layout of your reproducible example will be adapted to the platform, and you can even include information about your R session.²

Here is how to use it steps by steps:

First create your reproducible example in R (remember to keep it as short and readable as possible to save time to potential helpers):

Minimal reproducible example

Select the {reprex} addin in the list of addins:

Select the {reprex} addin

Select the platform on which you will post your issue (and check “Append session info” if you want to display your session info at the end of your reproducible example):

Set options in the {reprex} addin

You can now see the output of your reproducible example (right panel), but more importantly, it has been copied in your clipboard and it is now ready to be pasted on the platform of your choice:

Your reprex is copied in your clipboard

Paste your reprex on the platform of your choice (here, it is posted as an issue on GitHub):

Paste your reprex

Check the final result of your reproducible example:

Final result

(Visit this GitHub issue to see the final result.)

Blogdown

I put this addin at the end of the list because it will be of interest for only a limited number of RStudio users: people maintaining a blog written in R like this one (see why I recommend everyone to start a technical blog).

The most useful functionalities in this addin are the following:

New post: create a new post with blogdown::new_post(). It can be used to create new pages as well, not only posts
Insert image: insert an external image into a blog post
Update metadata: update the title, author, date, categories and tags of the current blog post
Serve site: run blogdown::serve_site() to live preview your website locally

Conclusion

Thanks for reading.

I hope that you will find these addins useful for your future R-related projects. See other tips and tricks in RStudio and R Markdown.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

You actually do not need to rename it. However, I often rename the datasets I work on with the same generic name dat so when I reuse codes from a previous project for a new project, I do not have to change the name of the dataset in the code.↩︎
Thanks to Josep for the suggestion.↩︎

Descriptive statistics in R

Wed, 22 Jan 2020 00:00:00 +0000

Introduction

This article explains how to compute the main descriptive statistics in R and how to present them graphically. To learn more about the reasoning behind each descriptive statistics, how to compute them by hand and how to interpret them, read the article “Descriptive statistics by hand”.

To briefly recap what have been said in that article, descriptive statistics (in the broad sense of the term) is a branch of statistics aiming at summarizing, describing and presenting a series of values or a dataset. Descriptive statistics is often the first step and an important part in any statistical analysis. It allows to check the quality of the data and it helps to “understand” the data by having a clear overview of it. If well presented, descriptive statistics is already a good starting point for further analyses. There exists many measures to summarize a dataset. They are divided into two types:

location measures and
dispersion measures

Location measures give an understanding about the central tendency of the data, whereas dispersion measures give an understanding about the spread of the data. In this article, we focus only on the implementation in R of the most common descriptive statistics and their visualizations (when deemed appropriate). See online or in the above mentioned article for more information about the purpose and usage of each measure.

Data

We use the dataset iris throughout the article. This dataset is imported by default in R, you only need to load it by running iris:

dat <- iris # load the iris dataset and renamed it dat

Below a preview of this dataset and its structure:

head(dat) # first 6 observations

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa

str(dat) # structure of dataset

## 'data.frame':	150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

The dataset contains 150 observations and 5 variables, representing the length and width of the sepal and petal and the species of 150 flowers. Length and width of the sepal and petal are numeric variables and the species is a factor with 3 levels (indicated by num and Factor w/ 3 levels after the name of the variables). See the different variables types in R if you need a refresh.

Regarding plots, we present the default graphs and the graphs from the well-known {ggplot2} package. Graphs from the {ggplot2} package usually have a better look but it requires more advanced coding skills (see the article “Graphics in R with ggplot2” to learn more). If you need to publish or share your graphs, I suggest using {ggplot2} if you can, otherwise the default graphics will do the job.

Tip: I recently discovered the ggplot2 builder from the {esquisse} addins. See how you can easily draw graphs from the {ggplot2} package without having to code it yourself.

All plots displayed in this article can be customized. For instance, it is possible to edit the title, x and y-axis labels, color, etc. However, customizing plots is beyond the scope of this article so all plots are presented without any customization. Interested readers will find numerous resources online.

Minimum and maximum

Minimum and maximum can be found thanks to the min() and max() functions:

min(dat$Sepal.Length)

## [1] 4.3

max(dat$Sepal.Length)

## [1] 7.9

Alternatively the range() function:

rng <- range(dat$Sepal.Length)
rng

## [1] 4.3 7.9

gives you the minimum and maximum directly. Note that the output of the range() function is actually an object containing the minimum and maximum (in that order). This means you can actually access the minimum with:

rng[1] # rng = name of the object specified above

## [1] 4.3

and the maximum with:

rng[2]

## [1] 7.9

This reminds us that, in R, there are often several ways to arrive at the same result. The method that uses the shortest piece of code is usually preferred as a shorter piece of code is less prone to coding errors and more readable.

Range

The range can then be easily computed, as you have guessed, by subtracting the minimum from the maximum:

max(dat$Sepal.Length) - min(dat$Sepal.Length)

## [1] 3.6

To my knowledge, there is no default function to compute the range. However, if you are familiar with writing functions in R , you can create your own function to compute the range:

range2 <- function(x) {
  range <- max(x) - min(x)
  return(range)
}

range2(dat$Sepal.Length)

## [1] 3.6

which is equivalent than $max - min$ presented above.

Mean

The mean can be computed with the mean() function:

mean(dat$Sepal.Length)

## [1] 5.843333

Tips:

if there is at least one missing value in your dataset, use mean(dat$Sepal.Length, na.rm = TRUE) to compute the mean with the NA excluded. This argument can be used for most functions presented in this article, not only the mean
for a truncated mean, use mean(dat$Sepal.Length, trim = 0.10) and change the trim argument to your needs

Median

The median can be computed thanks to the median() function:

median(dat$Sepal.Length)

## [1] 5.8

or with the quantile() function:

quantile(dat$Sepal.Length, 0.5)

## 50% 
## 5.8

since the quantile of order 0.5 ($q_{0.5}$) corresponds to the median.

First and third quartile

As the median, the first and third quartiles can be computed thanks to the quantile() function and by setting the second argument to 0.25 or 0.75:

quantile(dat$Sepal.Length, 0.25) # first quartile

## 25% 
## 5.1

quantile(dat$Sepal.Length, 0.75) # third quartile

## 75% 
## 6.4

You may have seen that the results above are slightly different than the results you would have found if you compute the first and third quartiles by hand. It is normal, there are many methods to compute them (R actually has 7 methods to compute the quantiles!). However, the methods presented here and in the article “descriptive statistics by hand” are the easiest and most “standard” ones. Furthermore, results do not dramatically change between the two methods.

Other quantiles

As you have guessed, any quantile can also be computed with the quantile() function. For instance, the $4^{th}$ decile or the $98^{th}$ percentile:

quantile(dat$Sepal.Length, 0.4) # 4th decile

## 40% 
## 5.6

quantile(dat$Sepal.Length, 0.98) # 98th percentile

## 98% 
## 7.7

Interquartile range

The interquartile range (i.e., the difference between the first and third quartile) can be computed with the IQR() function:

IQR(dat$Sepal.Length)

## [1] 1.3

or alternatively with the quantile() function again:

quantile(dat$Sepal.Length, 0.75) - quantile(dat$Sepal.Length, 0.25)

## 75% 
## 1.3

As mentioned earlier, when possible it is usually recommended to use the shortest piece of code to arrive at the result. For this reason, the IQR() function is preferred to compute the interquartile range.

Standard deviation and variance

The standard deviation and the variance is computed with the sd() and var() functions:

sd(dat$Sepal.Length) # standard deviation

## [1] 0.8280661

var(dat$Sepal.Length) # variance

## [1] 0.6856935

Remember from the article descriptive statistics by hand that the standard deviation and the variance are different whether we compute it for a sample or a population (see the difference between sample and population). In R, the standard deviation and the variance are computed as if the data represent a sample (so the denominator is $n - 1$, where $n$ is the number of observations). To my knowledge, there is no function by default in R that computes the standard deviation or variance for a population.

Tip: to compute the standard deviation (or variance) of multiple variables at the same time, use lapply() with the appropriate statistics as second argument:

lapply(dat[, 1:4], sd)

## $Sepal.Length
## [1] 0.8280661
## 
## $Sepal.Width
## [1] 0.4358663
## 
## $Petal.Length
## [1] 1.765298
## 
## $Petal.Width
## [1] 0.7622377

The command dat[, 1:4] selects the variables 1 to 4 as the fifth variable is a qualitative variable and the standard deviation cannot be computed on such type of variable. See a recap of the different data types in R if needed.

Summary

You can compute the minimum, $1^{st}$ quartile, median, mean, $3^{rd}$ quartile and the maximum for all numeric variables of a dataset at once using summary():

summary(dat)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
##

Tip: if you need these descriptive statistics by group use the by() function:

by(dat, dat$Species, summary)

## dat$Species: setosa
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.300   Min.   :1.000   Min.   :0.100  
##  1st Qu.:4.800   1st Qu.:3.200   1st Qu.:1.400   1st Qu.:0.200  
##  Median :5.000   Median :3.400   Median :1.500   Median :0.200  
##  Mean   :5.006   Mean   :3.428   Mean   :1.462   Mean   :0.246  
##  3rd Qu.:5.200   3rd Qu.:3.675   3rd Qu.:1.575   3rd Qu.:0.300  
##  Max.   :5.800   Max.   :4.400   Max.   :1.900   Max.   :0.600  
##        Species  
##  setosa    :50  
##  versicolor: 0  
##  virginica : 0  
##                 
##                 
##                 
## ------------------------------------------------------------ 
## dat$Species: versicolor
##   Sepal.Length    Sepal.Width     Petal.Length   Petal.Width          Species  
##  Min.   :4.900   Min.   :2.000   Min.   :3.00   Min.   :1.000   setosa    : 0  
##  1st Qu.:5.600   1st Qu.:2.525   1st Qu.:4.00   1st Qu.:1.200   versicolor:50  
##  Median :5.900   Median :2.800   Median :4.35   Median :1.300   virginica : 0  
##  Mean   :5.936   Mean   :2.770   Mean   :4.26   Mean   :1.326                  
##  3rd Qu.:6.300   3rd Qu.:3.000   3rd Qu.:4.60   3rd Qu.:1.500                  
##  Max.   :7.000   Max.   :3.400   Max.   :5.10   Max.   :1.800                  
## ------------------------------------------------------------ 
## dat$Species: virginica
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.900   Min.   :2.200   Min.   :4.500   Min.   :1.400  
##  1st Qu.:6.225   1st Qu.:2.800   1st Qu.:5.100   1st Qu.:1.800  
##  Median :6.500   Median :3.000   Median :5.550   Median :2.000  
##  Mean   :6.588   Mean   :2.974   Mean   :5.552   Mean   :2.026  
##  3rd Qu.:6.900   3rd Qu.:3.175   3rd Qu.:5.875   3rd Qu.:2.300  
##  Max.   :7.900   Max.   :3.800   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    : 0  
##  versicolor: 0  
##  virginica :50  
##                 
##                 
##

where the arguments are the name of the dataset, the grouping variable and the summary function. Follow this order, or specify the name of the arguments if you do not follow this order.

If you need more descriptive statistics, use stat.desc() from the package {pastecs}:

library(pastecs)
stat.desc(dat)

##              Sepal.Length  Sepal.Width Petal.Length  Petal.Width Species
## nbr.val      150.00000000 150.00000000  150.0000000 150.00000000      NA
## nbr.null       0.00000000   0.00000000    0.0000000   0.00000000      NA
## nbr.na         0.00000000   0.00000000    0.0000000   0.00000000      NA
## min            4.30000000   2.00000000    1.0000000   0.10000000      NA
## max            7.90000000   4.40000000    6.9000000   2.50000000      NA
## range          3.60000000   2.40000000    5.9000000   2.40000000      NA
## sum          876.50000000 458.60000000  563.7000000 179.90000000      NA
## median         5.80000000   3.00000000    4.3500000   1.30000000      NA
## mean           5.84333333   3.05733333    3.7580000   1.19933333      NA
## SE.mean        0.06761132   0.03558833    0.1441360   0.06223645      NA
## CI.mean.0.95   0.13360085   0.07032302    0.2848146   0.12298004      NA
## var            0.68569351   0.18997942    3.1162779   0.58100626      NA
## std.dev        0.82806613   0.43586628    1.7652982   0.76223767      NA
## coef.var       0.14171126   0.14256420    0.4697441   0.63555114      NA

You can have even more statistics (i.e., skewness, kurtosis and normality test) by adding the argument norm = TRUE in the previous function. Note that the variable Species is not numeric, so descriptive statistics cannot be computed for this variable and NA are displayed.

Coefficient of variation

The coefficient of variation can be found with stat.desc() (see the line coef.var in the table above) or by computing manually (remember that the coefficient of variation is the standard deviation divided by the mean):

sd(dat$Sepal.Length) / mean(dat$Sepal.Length)

## [1] 0.1417113

Mode

To my knowledge there is no function to find the mode of a variable. However, we can easily find it thanks to the functions table() and sort():

tab <- table(dat$Sepal.Length) # number of occurrences for each unique value
sort(tab, decreasing = TRUE) # sort highest to lowest

## 
##   5 5.1 6.3 5.7 6.7 5.5 5.8 6.4 4.9 5.4 5.6   6 6.1 4.8 6.5 4.6 5.2 6.2 6.9 7.7 
##  10   9   9   8   8   7   7   7   6   6   6   6   6   5   5   4   4   4   4   4 
## 4.4 5.9 6.8 7.2 4.7 6.6 4.3 4.5 5.3   7 7.1 7.3 7.4 7.6 7.9 
##   3   3   3   3   2   2   1   1   1   1   1   1   1   1   1

table() gives the number of occurrences for each unique value, then sort() with the argument decreasing = TRUE displays the number of occurrences from highest to lowest. The mode of the variable Sepal.Length is thus 5. This code to find the mode can also be applied to qualitative variables such as Species:

sort(table(dat$Species), decreasing = TRUE)

## 
##     setosa versicolor  virginica 
##         50         50         50

or:

summary(dat$Species)

##     setosa versicolor  virginica 
##         50         50         50

Correlation

Another descriptive statistics is the correlation coefficient.

The correlation measures the linear relationship between two variables, and it can be computed with the cor() function:

cor(dat$Sepal.Length, dat$Sepal.Width)

## [1] -0.1175698

Computing correlation in R and interpreting the results deserve a detailed explanation, so I wrote an article covering correlation and correlation test.

Contingency table

table() introduced above can also be used on two qualitative variables to create a contingency table. The dataset iris has only one qualitative variable so we create a new qualitative variable just for this example. We create the variable size which corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

Here is a recap of the occurrences by size:

table(dat$size)

## 
##   big small 
##    77    73

We now create a contingency table of the two variables Species and size with the table() function:

table(dat$Species, dat$size)

##             
##              big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

or with the xtabs() function:

xtabs(~ dat$Species + dat$size)

##             dat$size
## dat$Species  big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

The contingency table gives the number of cases in each subgroup. For instance, there is only one big setosa flower, while there are 49 small setosa flowers in the dataset.

To go further, we can see from the table that setosa flowers seem to be smaller in size than virginica flowers. In order to check whether size is significantly associated with species, we could perform a Chi-square test of independence since both variables are categorical variables. See how to do this test by hand and in R.

Note that Species are in rows and size in column because we specified Species and then size in table(). Change the order if you want to switch the two variables.

Instead of having the frequencies (i.e.. the number of cases) you can also have the relative frequencies (i.e., proportions) in each subgroup by adding the table() function inside the prop.table() function:

prop.table(table(dat$Species, dat$size))

##             
##                      big       small
##   setosa     0.006666667 0.326666667
##   versicolor 0.193333333 0.140000000
##   virginica  0.313333333 0.020000000

Note that you can also compute the percentages by row or by column by adding a second argument to the prop.table() function: 1 for row, or 2 for column:

# percentages by row:
round(prop.table(table(dat$Species, dat$size), 1), 2) # round to 2 digits with round()

##             
##               big small
##   setosa     0.02  0.98
##   versicolor 0.58  0.42
##   virginica  0.94  0.06

# percentages by column:
round(prop.table(table(dat$Species, dat$size), 2), 2) # round to 2 digits with round()

##             
##               big small
##   setosa     0.01  0.67
##   versicolor 0.38  0.29
##   virginica  0.61  0.04

See the section on advanced descriptive statistics for more advanced contingency tables.

Mosaic plot

A mosaic plot allows to visualize a contingency table of two qualitative variables:

mosaicplot(table(dat$Species, dat$size),
  color = TRUE,
  xlab = "Species", # label for x-axis
  ylab = "Size" # label for y-axis
)

The mosaic plot shows that, for our sample, the proportion of big and small flowers is clearly different between the three species. In particular, the virginica species is the biggest, and the setosa species is the smallest of the three species (in terms of sepal length since the variable size is based on the variable Sepal.Length).

For your information, a mosaic plot can also be done via the mosaic() function from the {vcd} package:

library(vcd)

mosaic(~ Species + size,
  data = dat,
  direction = c("v", "h")
)

Barplot

Barplots can only be done on qualitative variables (see the difference with a quantitative variable here). A barplot is a tool to visualize the distribution of a qualitative variable. We draw a barplot of the qualitative variable size:

barplot(table(dat$size)) # table() is mandatory

You can also draw a barplot of the relative frequencies instead of the frequencies by adding prop.table() as we did earlier:

barplot(prop.table(table(dat$size)))

In {ggplot2}:

library(ggplot2) # needed each time you open RStudio
# The package ggplot2 must be installed first

ggplot(dat) +
  aes(x = size) +
  geom_bar()

Histogram

A histogram gives an idea about the distribution of a quantitative variable. The idea is to break the range of values into intervals and count how many observations fall into each interval. Histograms are a bit similar to barplots, but histograms are used for quantitative variables whereas barplots are used for qualitative variables. To draw a histogram in R, use hist():

hist(dat$Sepal.Length)

Add the arguments breaks = inside the hist() function if you want to change the number of bins. A rule of thumb (known as the square-root rule) is that the number of bins should be the rounded value of the square root of the number of observations. The dataset includes 150 observations so in this case the number of bins can be set to 12.

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length) +
  geom_histogram()

By default, the number of bins is 30. You can change this value with geom_histogram(bins = 12) for instance.

Boxplot

Boxplots are really useful in descriptive statistics and are often underused (mostly because it is not well understood by the public). A boxplot graphically represents the distribution of a quantitative variable by visually displaying five common location summary (minimum, median, first/third quartiles and maximum) and any observation that was classified as a suspected outlier using the interquartile range (IQR) criterion.

The IQR criterion means that all observations above $q_{0.75} + 1.5 \cdot IQR$ or below $q_{0.25} - 1.5 \cdot IQR$ (where $q_{0.25}$ and $q_{0.75}$ correspond to first and third quartile respectively) are considered as potential outliers by R. The minimum and maximum in the boxplot are represented without these suspected outliers.

Seeing all these information on the same plot help to have a good first overview of the dispersion and the location of the data. Before drawing a boxplot of our data, see below a graph explaining the information present on a boxplot:

How to interpret a boxplot? Source: LFSAB1105

Now an example with our dataset:

boxplot(dat$Sepal.Length)

Boxplots are even more informative when presented side-by-side for comparing and contrasting distributions from two or more groups. For instance, we compare the length of the sepal across the different species:

boxplot(dat$Sepal.Length ~ dat$Species)

In {ggplot2}:

ggplot(dat) +
  aes(x = Species, y = Sepal.Length) +
  geom_boxplot()

Dotplot

A dotplot is more or less similar than a boxplot, except that:

observations are represented as points
it does not easily tell us about the median, first and third quartiles.

library(lattice)

dotplot(dat$Sepal.Length ~ dat$Species)

In {ggplot2}:

ggplot(dat) +
  aes(x = Species, y = Sepal.Length) +
  geom_dotplot(binaxis = "y", stackdir = "center")

The advantage of using {ggplot2} over {lattice} for this plot is that we can easily see the mode.

Note that a dotplot is particularly useful when there are a limited number of observations, whereas a boxplot is more appropriate with large datasets.

Scatterplot

Scatterplots allow to check whether there is a potential link between two quantitative variables. For this reason, scatterplots are often used to visualize a potential correlation between two variables. For instance, when drawing a scatterplot of the length of the sepal and the length of the petal:

plot(dat$Sepal.Length, dat$Petal.Length)

There seems to be a positive association between the two variables.

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length) +
  geom_point()

Like boxplots, scatterplots are even more informative when differentiating the points according to a factor, in this case the species:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length, colour = Species) +
  geom_point() +
  scale_color_hue()

Line plot

Line plots, particularly useful in time series or finance, can be created by adding the type = "l" argument in the plot() function:

plot(dat$Sepal.Length,
  type = "l"
) # "l" for line

QQ-plot

For a single variable

In order to check the normality assumption of a variable (normality means that the data follow a normal distribution, also known as a Gaussian distribution), we usually use histograms and/or QQ-plots.¹ See an article discussing about the normal distribution and how to evaluate the normality assumption in R if you need a refresh on that subject.

Histograms have been presented earlier, so here is how to draw a QQ-plot:

# Draw points on the qq-plot:
qqnorm(dat$Sepal.Length)
# Draw the reference line:
qqline(dat$Sepal.Length)

Or a QQ-plot with confidence bands with the qqPlot() function from the {car} package:

library(car) # package must be installed first
qqPlot(dat$Sepal.Length)

## [1] 132 118

If points are close to the reference line (sometimes referred as Henry’s line) and within the confidence bands, the normality assumption can be considered as met. The bigger the deviation between the points and the reference line and the more they lie outside the confidence bands, the less likely that the normality condition is met. The variable Sepal.Length does not seem to follow a normal distribution because several points lie outside the confidence bands. When facing a non-normal distribution, the first step is usually to apply the logarithm transformation on the data and recheck to see whether the log-transformed data are normally distributed. Applying the logarithm transformation can be done with the log() function.

In {ggpubr}:

library(ggpubr)
ggqqplot(dat$Sepal.Length)

By groups

For some statistical tests, the normality assumption is required in all groups. One solution is to draw a QQ-plot for each group by manually splitting the dataset into different groups and then draw a QQ-plot for each subset of the data (with the methods shown above). Another (easier) solution is to draw a QQ-plot for each group automatically with the argument groups = in the function qqPlot() from the {car} package:

qqPlot(dat$Sepal.Length, groups = dat$size)

In {ggplot2}:

qplot(
  sample = Sepal.Length, data = dat,
  col = size, shape = size
)

It is also possible to differentiate groups by only shape or color. For this, remove one of the argument col or shape in the qplot() function above.

Density plot

Density plot is a smoothed version of the histogram and is used in the same concept, that is, to represent the distribution of a numeric variable. The functions plot() and density() are used together to draw a density plot:

plot(density(dat$Sepal.Length))

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length) +
  geom_density()

Correlation plot

The last type of descriptive plot is a correlation plot, also called a correlogram. This type of graph is more complex than the ones presented above, so it is detailed in a separate article. See how to draw a correlogram to highlight the most correlated variables in a dataset.

Advanced descriptive statistics

We covered the main functions to compute the most common and basic descriptive statistics. There are, however, many more functions and packages to perform more advanced descriptive statistics in R. In this section, I present some of them with applications to our dataset.

`{summarytools}` package

One package for descriptive statistics I often use for my projects in R is the {summarytools} package. The package is centered around 4 functions:

freq() for frequencies tables
ctable() for cross-tabulations
descr() for descriptive statistics
dfSummary() for dataframe summaries

A combination of these 4 functions is usually more than enough for most descriptive analyses. Moreover, the package has been built with R Markdown in mind, meaning that outputs render well in HTML reports. And for non-English speakers, built-in translations exist for French, Portuguese, Spanish, Russian and Turkish.

I illustrate each of the 4 functions in the following sections. Outputs that follow display much better in R Markdown reports, but in this article I limit myself to the raw outputs as the goal is to show how the functions work, not how to make them render well. See the setup settings in the vignette of the package if you want to print the outputs in a nice way in R Markdown.²

Frequency tables with `freq()`

The freq() function produces frequency tables with frequencies, proportions, as well as missing data information.

library(summarytools)
freq(dat$Species)

## Frequencies  
## dat$Species  
## Type: Factor  
## 
##                    Freq   % Valid   % Valid Cum.   % Total   % Total Cum.
## ---------------- ------ --------- -------------- --------- --------------
##           setosa     50     33.33          33.33     33.33          33.33
##       versicolor     50     33.33          66.67     33.33          66.67
##        virginica     50     33.33         100.00     33.33         100.00
##             <NA>      0                               0.00         100.00
##            Total    150    100.00         100.00    100.00         100.00

If you do not need information about missing values, add the report.nas = FALSE argument:

freq(dat$Species,
  report.nas = FALSE # remove NA information
)

## Frequencies  
## dat$Species  
## Type: Factor  
## 
##                    Freq        %   % Cum.
## ---------------- ------ -------- --------
##           setosa     50    33.33    33.33
##       versicolor     50    33.33    66.67
##        virginica     50    33.33   100.00
##            Total    150   100.00   100.00

And for a minimalist output with only counts and proportions:

freq(dat$Species,
  report.nas = FALSE, # remove NA information
  totals = FALSE, # remove totals
  cumul = FALSE, # remove cumuls
  headings = FALSE # remove headings
)

## 
##                    Freq       %
## ---------------- ------ -------
##           setosa     50   33.33
##       versicolor     50   33.33
##        virginica     50   33.33

Cross-tabulations with `ctable()`

The ctable() function produces cross-tabulations (also known as contingency tables) for pairs of categorical variables. Using the two categorical variables in our dataset:

ctable(
  x = dat$Species,
  y = dat$size
)

## Cross-Tabulation, Row Proportions  
## Species * size  
## Data Frame: dat  
## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 2.0%)   49 (98.0%)    50 (100.0%)
##   versicolor          29 (58.0%)   21 (42.0%)    50 (100.0%)
##    virginica          47 (94.0%)    3 ( 6.0%)    50 (100.0%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------

Row proportions are shown by default. To display column or total proportions, add the prop = "c" or prop = "t" arguments, respectively:

ctable(
  x = dat$Species,
  y = dat$size,
  prop = "t" # total proportions
)

## Cross-Tabulation, Total Proportions  
## Species * size  
## Data Frame: dat  
## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 0.7%)   49 (32.7%)    50 ( 33.3%)
##   versicolor          29 (19.3%)   21 (14.0%)    50 ( 33.3%)
##    virginica          47 (31.3%)    3 ( 2.0%)    50 ( 33.3%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------

To remove proportions altogether, add the argument prop = "n". Furthermore, to display only the bare minimum, add the totals = FALSE and headings = FALSE arguments:

ctable(
  x = dat$Species,
  y = dat$size,
  prop = "n", # remove proportions
  totals = FALSE, # remove totals
  headings = FALSE # remove headings
)

## 
## ------------ ------ ----- -------
##                size   big   small
##      Species                     
##       setosa            1      49
##   versicolor           29      21
##    virginica           47       3
## ------------ ------ ----- -------

This is equivalent than table(dat$Species, dat$size) and xtabs(~ dat$Species + dat$size) performed in the section on contingency tables.

To display results of the Chi-square test of independence, add the chisq = TRUE argument:³

ctable(
  x = dat$Species,
  y = dat$size,
  chisq = TRUE, # display results of Chi-square test of independence
  headings = FALSE # remove headings
)

## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 2.0%)   49 (98.0%)    50 (100.0%)
##   versicolor          29 (58.0%)   21 (42.0%)    50 (100.0%)
##    virginica          47 (94.0%)    3 ( 6.0%)    50 (100.0%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------
## 
## ----------------------------
##  Chi.squared   df   p.value 
## ------------- ---- ---------
##    86.0345     2       0    
## ----------------------------

The p-value is close to 0 so we reject the null hypothesis of independence between the two variables. In our context, this indicates that species and size are dependent and that there is a significant relationship between the two variables.

It is also possible to create a contingency table for each level of a third categorical variable thanks to the combination of the stby() and ctable() functions. There are only 2 categorical variables in our dataset, so let’s use the tabacco dataset which has 4 categorical variables (i.e., gender, age group, smoker, diseased). For this example, we would like to create a contingency table of the variables smoker and diseased, and this for each gender:

stby(
  list(
    x = tobacco$smoker, # smoker and diseased
    y = tobacco$diseased
  ),
  INDICES = tobacco$gender, # for each gender
  FUN = ctable # ctable for cross-tabulation
)

## Cross-Tabulation, Row Proportions  
## smoker * diseased  
## Data Frame: tobacco  
## Group: gender = F  
## 
## -------- ---------- ------------- ------------- --------------
##            diseased           Yes            No          Total
##   smoker                                                      
##      Yes               62 (42.2%)    85 (57.8%)   147 (100.0%)
##       No               49 (14.3%)   293 (85.7%)   342 (100.0%)
##    Total              111 (22.7%)   378 (77.3%)   489 (100.0%)
## -------- ---------- ------------- ------------- --------------
## 
## Group: gender = M  
## 
## -------- ---------- ------------- ------------- --------------
##            diseased           Yes            No          Total
##   smoker                                                      
##      Yes               63 (44.1%)    80 (55.9%)   143 (100.0%)
##       No               47 (13.6%)   299 (86.4%)   346 (100.0%)
##    Total              110 (22.5%)   379 (77.5%)   489 (100.0%)
## -------- ---------- ------------- ------------- --------------

Descriptive statistics with `descr()`

The descr() function produces descriptive (univariate) statistics with common central tendency statistics and measures of dispersion. (See the difference between a measure of central tendency and dispersion if you need a reminder.)

A major advantage of this function is that it accepts single vectors as well as data frames. If a data frame is provided, all non-numerical columns are ignored so you do not have to remove them yourself before running the function.

The descr() function allows to display:

only a selection of descriptive statistics of your choice, with the stats = c("mean", "sd") argument for mean and standard deviation for example
the minimum, first quartile, median, third quartile and maximum with stats = "fivenum"
the most common descriptive statistics (mean, standard deviation, minimum, median, maximum, number and percentage of valid observations), with stats = "common":

descr(dat,
  headings = FALSE, # remove headings
  stats = "common" # most common descriptive statistics
)

## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           3.76          1.20           5.84          3.06
##         Std.Dev           1.77          0.76           0.83          0.44
##             Min           1.00          0.10           4.30          2.00
##          Median           4.35          1.30           5.80          3.00
##             Max           6.90          2.50           7.90          4.40
##         N.Valid         150.00        150.00         150.00        150.00
##               N         150.00        150.00         150.00        150.00
##       Pct.Valid         100.00        100.00         100.00        100.00

Tip: if you have a large number of variables, add the transpose = TRUE argument for a better display.

In order to compute these descriptive statistics by group (e.g., Species in our dataset), use the descr() function in combination with the stby() function:

stby(
  data = dat,
  INDICES = dat$Species, # by Species
  FUN = descr, # descriptive statistics
  stats = "common" # most common descr. stats
)

## Descriptive Statistics  
## dat  
## Group: Species = setosa  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           1.46          0.25           5.01          3.43
##         Std.Dev           0.17          0.11           0.35          0.38
##             Min           1.00          0.10           4.30          2.30
##          Median           1.50          0.20           5.00          3.40
##             Max           1.90          0.60           5.80          4.40
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00
## 
## Group: Species = versicolor  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           4.26          1.33           5.94          2.77
##         Std.Dev           0.47          0.20           0.52          0.31
##             Min           3.00          1.00           4.90          2.00
##          Median           4.35          1.30           5.90          2.80
##             Max           5.10          1.80           7.00          3.40
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00
## 
## Group: Species = virginica  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           5.55          2.03           6.59          2.97
##         Std.Dev           0.55          0.27           0.64          0.32
##             Min           4.50          1.40           4.90          2.20
##          Median           5.55          2.00           6.50          3.00
##             Max           6.90          2.50           7.90          3.80
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00

Data frame summaries with `dfSummary()`

The dfSummary() function generates a summary table with statistics, frequencies and graphs for all variables in a dataset. The information shown depends on the type of the variables (character, factor, numeric, date) and also varies according to the number of distinct values.

dfSummary(dat)

## Data Frame Summary  
## dat  
## Dimensions: 150 x 6  
## Duplicates: 1  
## 
## -----------------------------------------------------------------------------------------------------------
## No   Variable       Stats / Values          Freqs (% of Valid)   Graph                 Valid      Missing  
## ---- -------------- ----------------------- -------------------- --------------------- ---------- ---------
## 1    Sepal.Length   Mean (sd) : 5.8 (0.8)   35 distinct values     . . : :             150        0        
##      [numeric]      min < med < max:                               : : : :             (100.0%)   (0.0%)   
##                     4.3 < 5.8 < 7.9                                : : : : :                               
##                     IQR (CV) : 1.3 (0.1)                           : : : : :                               
##                                                                  : : : : : : : :                           
## 
## 2    Sepal.Width    Mean (sd) : 3.1 (0.4)   23 distinct values           :             150        0        
##      [numeric]      min < med < max:                                     :             (100.0%)   (0.0%)   
##                     2 < 3 < 4.4                                        . :                                 
##                     IQR (CV) : 0.5 (0.1)                             : : : :                               
##                                                                  . . : : : : : :                           
## 
## 3    Petal.Length   Mean (sd) : 3.8 (1.8)   43 distinct values   :                     150        0        
##      [numeric]      min < med < max:                             :         . :         (100.0%)   (0.0%)   
##                     1 < 4.3 < 6.9                                :         : : .                           
##                     IQR (CV) : 3.5 (0.5)                         : :       : : : .                         
##                                                                  : :   . : : : : : .                       
## 
## 4    Petal.Width    Mean (sd) : 1.2 (0.8)   22 distinct values   :                     150        0        
##      [numeric]      min < med < max:                             :                     (100.0%)   (0.0%)   
##                     0.1 < 1.3 < 2.5                              :       . .   :                           
##                     IQR (CV) : 1.5 (0.6)                         :       : :   :   .                       
##                                                                  : :   : : : . : : :                       
## 
## 5    Species        1. setosa               50 (33.3%)           IIIIII                150        0        
##      [factor]       2. versicolor           50 (33.3%)           IIIIII                (100.0%)   (0.0%)   
##                     3. virginica            50 (33.3%)           IIIIII                                    
## 
## 6    size           1. big                  77 (51.3%)           IIIIIIIIII            150        0        
##      [character]    2. small                73 (48.7%)           IIIIIIIII             (100.0%)   (0.0%)   
## -----------------------------------------------------------------------------------------------------------

`describeBy()` from the `{psych}` package

The describeBy() function from the {psych} package allows to report several summary statistics (i.e., number of valid cases, mean, standard deviation, median, trimmed mean, mad: median absolute deviation (from the median), minimum, maximum, range, skewness and kurtosis) by a grouping variable.

library(psych)
describeBy(
  dat,
  dat$Species # grouping variable
)

## 
##  Descriptive statistics by group 
## group: setosa
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 5.01 0.35    5.0    5.00 0.30 4.3 5.8   1.5  0.11    -0.45
## Sepal.Width     2 50 3.43 0.38    3.4    3.42 0.37 2.3 4.4   2.1  0.04     0.60
## Petal.Length    3 50 1.46 0.17    1.5    1.46 0.15 1.0 1.9   0.9  0.10     0.65
## Petal.Width     4 50 0.25 0.11    0.2    0.24 0.00 0.1 0.6   0.5  1.18     1.26
## Species         5 50 1.00 0.00    1.0    1.00 0.00 1.0 1.0   0.0   NaN      NaN
## size            6 50 1.98 0.14    2.0    2.00 0.00 1.0 2.0   1.0 -6.65    43.12
##                se
## Sepal.Length 0.05
## Sepal.Width  0.05
## Petal.Length 0.02
## Petal.Width  0.01
## Species      0.00
## size         0.02
## ------------------------------------------------------------ 
## group: versicolor
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 5.94 0.52   5.90    5.94 0.52 4.9 7.0   2.1  0.10    -0.69
## Sepal.Width     2 50 2.77 0.31   2.80    2.78 0.30 2.0 3.4   1.4 -0.34    -0.55
## Petal.Length    3 50 4.26 0.47   4.35    4.29 0.52 3.0 5.1   2.1 -0.57    -0.19
## Petal.Width     4 50 1.33 0.20   1.30    1.32 0.22 1.0 1.8   0.8 -0.03    -0.59
## Species         5 50 2.00 0.00   2.00    2.00 0.00 2.0 2.0   0.0   NaN      NaN
## size            6 50 1.42 0.50   1.00    1.40 0.00 1.0 2.0   1.0  0.31    -1.94
##                se
## Sepal.Length 0.07
## Sepal.Width  0.04
## Petal.Length 0.07
## Petal.Width  0.03
## Species      0.00
## size         0.07
## ------------------------------------------------------------ 
## group: virginica
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 6.59 0.64   6.50    6.57 0.59 4.9 7.9   3.0  0.11    -0.20
## Sepal.Width     2 50 2.97 0.32   3.00    2.96 0.30 2.2 3.8   1.6  0.34     0.38
## Petal.Length    3 50 5.55 0.55   5.55    5.51 0.67 4.5 6.9   2.4  0.52    -0.37
## Petal.Width     4 50 2.03 0.27   2.00    2.03 0.30 1.4 2.5   1.1 -0.12    -0.75
## Species         5 50 3.00 0.00   3.00    3.00 0.00 3.0 3.0   0.0   NaN      NaN
## size            6 50 1.06 0.24   1.00    1.00 0.00 1.0 2.0   1.0  3.59    11.15
##                se
## Sepal.Length 0.09
## Sepal.Width  0.05
## Petal.Length 0.08
## Petal.Width  0.04
## Species      0.00
## size         0.03

`aggregate()` function

The aggregate() function allows to split the data into subsets and then to compute summary statistics for each. For instance, if we want to compute the mean for the variables Sepal.Length and Sepal.Width by Species and Size:

aggregate(cbind(Sepal.Length, Sepal.Width) ~ Species + size,
  data = dat,
  mean
)

##      Species  size Sepal.Length Sepal.Width
## 1     setosa   big     5.800000    4.000000
## 2 versicolor   big     6.282759    2.868966
## 3  virginica   big     6.663830    2.997872
## 4     setosa small     4.989796    3.416327
## 5 versicolor small     5.457143    2.633333
## 6  virginica small     5.400000    2.600000

`summaryBy()` from `{doBy}`

An alternative is the summaryBy() function from the {doBy} package:

# summary statistics by group
library(doBy)
summaryBy(Sepal.Length + Sepal.Width ~ Species,
  data = dat,
  FUN = summary
)

##      Species Sepal.Length.Min. Sepal.Length.1st Qu. Sepal.Length.Median
## 1     setosa               4.3                4.800                 5.0
## 2 versicolor               4.9                5.600                 5.9
## 3  virginica               4.9                6.225                 6.5
##   Sepal.Length.Mean Sepal.Length.3rd Qu. Sepal.Length.Max. Sepal.Width.Min.
## 1             5.006                  5.2               5.8              2.3
## 2             5.936                  6.3               7.0              2.0
## 3             6.588                  6.9               7.9              2.2
##   Sepal.Width.1st Qu. Sepal.Width.Median Sepal.Width.Mean Sepal.Width.3rd Qu.
## 1               3.200                3.4            3.428               3.675
## 2               2.525                2.8            2.770               3.000
## 3               2.800                3.0            2.974               3.175
##   Sepal.Width.Max.
## 1              4.4
## 2              3.4
## 3              3.8

If you are interested in some specific descriptive statistics, you can easily specify them via the FUN argument:

summaryBy(Sepal.Length + Sepal.Width ~ Species,
  data = dat,
  FUN = c(mean, var)
)

##      Species Sepal.Length.mean Sepal.Width.mean Sepal.Length.var
## 1     setosa             5.006            3.428        0.1242490
## 2 versicolor             5.936            2.770        0.2664327
## 3  virginica             6.588            2.974        0.4043429
##   Sepal.Width.var
## 1      0.14368980
## 2      0.09846939
## 3      0.10400408

`group_by()` and `summarise()` from `{dplyr}`

Another alternative is with the summarise() and group_by() functions from the {dplyr} package:

library(dplyr)

group_by(dat, Species) %>%
  summarise(
    mean = mean(Sepal.Length, na.rm = TRUE),
    sd = sd(Sepal.Length, na.rm = TRUE)
  )

## # A tibble: 3 × 3
##   Species     mean    sd
##   <fct>      <dbl> <dbl>
## 1 setosa      5.01 0.352
## 2 versicolor  5.94 0.516
## 3 virginica   6.59 0.636

Conclusion

Thanks for reading.

I hope this article helped you to do descriptive statistics in R. If you would like to do the same by hand or understand what these statistics represent, I invite you to read the article “Descriptive statistics by hand”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

Normality tests such as Shapiro-Wilk or Kolmogorov-Smirnov tests can also be used to test whether the data follow a normal distribution or not. However, in practice, normality tests are often considered as too conservative in the sense that for large sample size, a small deviation from the normality may cause the normality condition to be violated. For this reason, it is often the case that the normality condition is verified based on a combination of visual inspections (with histograms and QQ-plots) and formal test (Shapiro-Wilk test for instance).↩︎
Note that the plain.ascii and style arguments are needed for this package. In our examples, these arguments are added in the settings of each chunk so they are not visible.↩︎
Note that it is also possible to compute odds ratio and risk ratio. See the vignette of the package for more information on this matter as these ratios are beyond the scope of this article.↩︎

Descriptive statistics by hand

Sun, 19 Jan 2020 00:00:00 +0000

Introduction

This article explains how to compute the main descriptive statistics by hand and how to interpret them. To learn how to compute these measures in R, read the article “Descriptive statistics in R”.

Descriptive statistics (in the broad sense of the term) is a branch of statistics aiming at summarizing, describing and presenting a series of values or a dataset. Long series of values without any preparation or without any summary measures are often not informative due to the difficulty of recognizing any pattern in the data. Below an example with the height (in cm) of a population of 100 adults:

188.7, 169.4, 178.6, 181.3, 179, 173.9, 190.1, 174.1, 195.2, 174.4, 188, 197.9, 161.1, 172.2, 173.7, 181.4, 172.2, 148.4, 150.6, 188.2, 171.9, 157.2, 173.3, 187.1, 194, 170.7, 172.4, 157.4, 179.6, 168.6, 179.6, 182, 185.4, 168.9, 180, 157.8, 167.2, 166.5, 150.9, 175.4, 177.1, 171.4, 182.6, 167.7, 161.3, 179.3, 166.9, 189.4, 170.7, 181.6, 178.2, 167.2, 190.8, 181.4, 175.9, 177.8, 181.8, 175.9, 145.1, 177.8, 171.3, 176.9, 180.8, 189, 167.7, 188, 178.4, 185.4, 184.2, 182.2, 164.6, 174.1, 181.2, 165.5, 169.6, 180.8, 182.7, 179.6, 166.1, 164, 190.1, 177.6, 175.9, 173.8, 163.1, 181.1, 172.8, 173.2, 184.3, 183.2, 188.9, 170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

Facing this series, it is hard (not to say impossible) for anyone to understand the data and have a clear view of the size of these adults in a reasonable amount of time. Descriptive statistics allow to summarize, and thus have a better overview of the data. Of course, by summarizing data through one or several measures, some information will inevitably be lost. However, in many cases it is generally better to lose some information but in return gain an overview.

Descriptive statistics is often the first step and an important part in any statistical analysis. It allows to check the quality of the data by detecting potential outliers (i.e., data points that appear to be separated from the rest of the data), collection or encoding errors. It also helps to “understand” the data and if well presented, descriptive statistics is already a good starting point for further analyses.

Location versus dispersion measures

Several different measures (called statistics if we are analyzing a sample) are used to summarize the data. Some of them give an understanding about the location of the data, others give and understanding about the dispersion of the data. In practice, both types of measures are often used together in order to summarize the data in the most concise but complete way.

We illustrate this point with the graph below, representing the height (in cm) of 100 persons divided into two groups (50 persons in each group):

The black line corresponds to the mean. The mean height (in cm) is similar in both groups. However, it is clear that the dispersion of heights are very different in the two groups. For this reason, location or dispersion measures are often not enough if presented individually and it is a good practice to present several statistics from both types of measures.

In the following sections, we detail the most common location and dispersion measures and illustrate them with examples. Note that for the sake of simplicity, we consider only series of values (i.e., univariate data) and not bivariate or multivariate data, and we do not consider the case of series grouped in classes.

Location

Location measures allow to see “where” the data are located, around which values. In other words, location measures give an understanding on what is the central tendency, the “position” of the data as a whole. It includes the following statistics (others exist but we focus on the most common ones):

minimum
maximum
mean
median
first quartile
third quartile
mode

We detail and compute by hand each of them in the following sections.

Minimum and maximum

Minimum ($min$) and maximum ($max$) are simply the lowest and largest values, respectively. Given the height (in cm) of a sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

The minimum is 169.4 cm and the maximum is 188.7 cm. These two basic statistics give a clear idea about the size of the smallest and tallest of these 6 adults.

Mean

The mean, also known as average, is probably the most common statistics. It gives an idea on what is the average value, that is, the central value of the data or in other words the center of gravity:¹

The mean is found by summing all values and dividing the total by the number of observations (denoted $n$):

\[mean = \bar{x} = \frac{\text{sum of all values}}{\text{number of values}} = \frac{1}{n}\sum^{n}_{i = 1} x_i\]

Given our sample of 6 adults presented above, the mean is:

\[\bar{x} = \frac{188.7 + 169.4 + 178.6}{6}\] \[\frac{+ 181.3 + 179 + 173.9}{6}\\ = 178.4833\]

In conclusion, the mean size, that is, the average size of our sample of 6 adults is 178.48 cm (rounded to 2 decimals).

Median

The median is another measure of location so it also gives an idea about the central tendency of the data. The interpretation of the median is that there are as many observations below as above the median. In other words, 50% of the observations lie below the median, and 50% of the observations lie above the median. Below a visual representation of the median:

The easiest way to compute the median is by first sorting the data from lowest to highest (i.e., in ascending order) then take the middle point as the median. From the sorted values, for an odd number of observations, the middle point is easy to find: it is the value with as many observations below as above. Still from the sorted values, for an even number of observations, the middle point is exactly between the two middle values. Formally, after sorting, the median is:

if $n$ (number of observations) is odd: \[med(x) = x_{\frac{n+1}{2}}\]
if $n$ is even: \[med(x) = \frac{1}{2}\big(x_{\frac{n}{2}} + x_{\frac{n}{2} + 1}\big)\]

where the subscript of $x$ denotes the numbering of the sorted data. The formulas look harder than they really are, so let’s see with two concrete examples.

Odd number of observations

Given the height of a sample of 7 adults taken from the 100 adults presented in the introduction:

188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

We first sort the order from lowest to highest:

160.4, 163.7, 163.9, 166.4, 175.8, 181.5 and 188.9

Given that the number of observations $n$ is odd (since $n = 7$), the median is

\[med(x) = x_\frac{7 + 1}{2} = x_4\]

So we take the fourth value from the sorted values, which corresponds to 166.4. In conclusion, the median size of these 7 adults is 166.4 cm. As you can see, there are 3 observations below 166.4 and 3 observations above 166.4 cm.

Even number of observations

Now let’s see when the number of observations is even, which is slightly more complicated than when the number of observations is odd. Given the height of a sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

We sort the values in ascending order:

169.4, 173.9, 178.6, 179, 181.3 and 188.7

Given that the number of observations $n$ is even (since $n = 6$), the median is

\[med(x) = \frac{1}{2}\big(x_{\frac{6}{2}} + x_{\frac{6}{2} + 1}\big) = \frac{1}{2}\big(x_{3} + x_{4}\big)\]

So we sum the third and fourth values from the sorted values and divide the total by 2 (which is equivalent than taking the mean of these two middle values):

\[\frac{1}{2}(178.6 + 179) = 178.8\]

In conclusion, the median size of these 6 adults is 178.8 cm. Again, remark that there are as many observations below as above 178.8 cm.

Mean vs. median

Although the mean and median are often relatively close to each other (in particular when the distribution is symmetric) they should not be confused since they both have advantages and disadvantages in different contexts. Besides the fact that almost everyone knows (or at least have heard about) the mean, it has the advantage that it gives a unique picture for each different series of data. However, it has the disadvantage that the mean is sensible to outliers (i.e., extreme values). On the other hand, the advantage of the median is that it is resistant to outliers and the inconvenient is that it may be the exact same value for very different series of data (so not unique to the data).

To illustrate the “sensible to outlier” argument, consider 3 friends in a bar comparing their salaries. Their salaries are 1800, 2000 and 2100€, for an average (mean) salary of 1967€. A friend of them (who happens to be friend with Bill Gates as well) joins them in the bar. Their salaries are now 1800, 2000, 2100 and 1000000€. The average salary of the 4 friends is now 251475€, compared to 1967€ without the rich friend. Although it is statistically correct to say that the mean salary of the 4 friends is 251475€, you will concede that this measure does not represent a fair image of the salaries of the 4 friends, as 3 of them earn much less than the mean salary. As we have just seen, the mean is sensible to outliers. (Note: this example also shows how a large majority of citizens earn less than the mean salary reported in the news. For the french-speaking readers, see this video for more information.)

On the other hand, if we report the medians, we see that the median salary of the 3 first friends is 2000€, and the median salary of the 4 friends is 2050€. As you can see with this example, the median is not sensible to outliers and for series with such extreme value(s), the median is more appropriate compared to the mean as it often gives a better representation of the data.

Given the previous example, one may then choose to always use the median instead of the mean. However, the median has it own inconvenient which the mean does not have: the median is less unique and less specific to its underlying data than the mean. Consider the following data, representing the grades of 5 students taking a statistics and economics exam:

studentID	economics	statistics
1	10	5
2	10	7
3	10	10
4	18	10
5	20	11

The median of the grades is the same in economics and statistics (median = 10). Therefore, had we computed only the medians, we could have concluded that the students performed as well in economics as in statistics. However, although the medians are exactly the same for both classes, it is clear that students performed better in economics than in statistics (compare both grades for each student to see for yourself). In fact, the mean of the grades in economics is 13.6 and the mean of the grades in statistics is 8.6.

What we have just shown here is that the median is based only on one single value, the middle value, or on the two middle values if there are an even number of observations, while the mean is based on all values (and thus includes more information). The median is therefore not sensible to outliers, but it is also not unique (i.e., not specific) to different series of data, whereas the mean is much more likely to be different and unique for different series of data. This difference in terms of specificity and uniqueness between the two measures may make the mean more useful for data with no outlier.

In conclusion, depending on the context and the data, it is often more interesting to report the mean or the median, or both. As a last remark regarding the comparison between the two most important location measures, note that when the mean and median are equal, the distribution of your data can often be considered to follow a normal distribution (also referred as a Gaussian distribution).

$1^{st}$ and $3^{rd}$ quartiles

The first and third quartiles are similar to the median in the sense that they also divide the observations into two parts, except that these parts are not equal. Remind that the median divides the data into two equal parts (with 50% of the observations below and 50% above the median).

The first quartile cuts the observations such that there are 25% of the observations below and thus 75% above the first quartile. The third quartile, as you have guessed by now, represents the value with 75% of the observations below it and thus 25% of the observations above it. There exists several methods to compute the first and third quartile (which sometimes give slight differences, R for instance uses a different method), but here is I believe the easiest one when computing these statistics by hand:

sort the data in ascending order
compute $0.25 \cdot n$ and $0.75 \cdot n$ (i.e., 0.25 and 0.75 times the number of observations)
round up these two numbers to the next whole number
these two numbers represent the rank of the first and third quartile (in the sorted series), respectively

The steps are the same for both an odd and even number of observations. Here is an example with the following series, representing the height in cm of 9 adults:

170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

We first order from lowest to highest:

160.4, 163.7, 163.9, 166.4, 170.2, 175.8, 181.5, 181.5 and 188.9

There are 9 observations so \[0.25 \cdot 9 = 2.25\] and \[0.75 \cdot 9 = 6.75\]

Rounding up to the whole number gives 3 and 7, which represent the rank of the first and third quartiles, respectively. Therefore, the first quartile is 163.9 cm and the third quartile is 181.5 cm.

In conclusion, 25% of adults are less than 163.9 cm tall (and thus 75% of them are more than 163.9 cm tall), while 75% of adults are less than 181.5 cm tall (and thus 25% of them are more than 181.5 cm tall).

$q_{0.25}$, $q_{0.75}$ and $q_{0.5}$

Note that the first quartile is denoted $q_{0.25}$ and the third quartile is denoted $q_{0.75}$ (where $q$ stands for quartile). As you can see, the median is actually the second quartile and for this reason it is also sometimes denoted $q_{0.5}$.

A note on deciles and percentiles

Deciles and percentiles are similar to quartiles except that they cuts the data in 10 and 100 equal parts. For instance, the $4^{th}$ decile ($q_{0.4}$) is the value such that there are 40% of the observations below it and thus 60% of the observations above it.

Percentiles follow the same logic. For example, the $98^{th}$ percentile ($q_{0.98}$, also sometimes denoted $P98$) is the value such that there are 98% of the observations below it and thus 2% of the observations above it. Percentiles are often used for the weight and height of babies, giving precise information to the parents about where their child stands compared to other children of the same age.

Mode

The mode of a series is the value that appears most often. In other words, it is the value that has the highest number of occurrences.

Unlike some descriptive statistics that can only be computed for quantitative variables (the mean for instance), the mode can be computed for quantitative and qualitative variables (see a recap of the different types of variables if you do not remember the difference).

Quantitative variables

Given the height of 9 adults:

170, 168, 171, 170, 182, 165, 170, 189 and 167

The mode is 170 because it is the most common value with 3 occurrences. All other values appear only once.

Note that it is possible that a series has no mode or more than one mode:

the series 4, 7, 2 and 10 has no mode,
the series 4, 2, 2, 8, 11 and 11 has two modes.

Data with two modes are often called bimodal and data with more than two modes are often called multimodal, as opposed to series with one mode which are referred as unimodal.

Qualitative variables

Given the eye color of the 9 adults presented above:

brown, brown, brown, brown, blue, blue, blue, brown and green

Brown is the most frequent color, so the mode is brown.

Dispersion

All previous descriptive statistics helps to get a sense of the location and position of the data. We now present the most common dispersion measures, which help to get a sense of the dispersion and the variability of the data (to which extent a distribution is squeezed or stretched):

range
standard deviation
variance
interquartile range
coefficient of variation

As for location measures, we detail and compute by hand each of these statistics one by one.

Range

The range is the difference between the maximum and the minimum value:

\[range = max - min\]

Given the height (in cm) of our sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

The range is 188.7 $-$ 169.4 $=$ 19.3 cm. The advantage of the range is that it is extremely easy to compute it and it gives a precise idea about the “length” of the data. The disadvantage is that it relies on the two most extreme values only, so it is highly sensible to outliers.

Standard deviation

The standard deviation is the most common dispersion measure in statistics. Like the mean for the location measures, if we have to present one statistics which summarizes the spread of the data, it is usually the standard deviation.

As its name suggests, the standard deviation tells what is the “normal” deviation of the data. It actually computes the mean deviation from the global mean. The larger the standard deviation, the more scattered the data are. On the contrary, the smaller the standard deviation, the more the data are centered around the mean. Below a visual representation of the standard deviation:

The standard deviation is a bit more complex than the previous statistics in the sense that there are two formulas depending on whether we face a sample or a population. A population includes all members from a specified group, all possible outcomes or measurements that are of interest. A sample consists of some observations drawn from the population, so a part or a subset of the population. For instance, the population may be “all people living in Belgium” and the sample may be “some people living in Belgium”. Read this article on the difference between population and sample if you want to learn more.

Standard deviation for a population

The standard deviation for a population, denoted $\sigma$, is:

\[\sigma = \sqrt{\frac{1}{n}\sum^n_{i = 1}(x_i - \mu)^2}\]

As you can see from the formula, the standard deviation is actually the mean deviation of the data from the global mean $\mu$. Note the square for the difference between the observations ($x_i$) and the mean ($\mu$) to avoid that negative differences are compensated by positive differences.

For the sake of easiness, imagine a population of only 3 adults (the steps are the same with a large population, the computation is just longer). Below their heights (in cm):

160.4, 175.8 and 181.5

The mean is 172.6 (rounded to 1 decimal). The standard deviation is thus:

\[\sigma = \sqrt{\frac{1}{3} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big]}\] \[\sigma = 8.91\]

In conclusion, the standard deviation for the heights of these 3 adults is 8.91 cm. This means that, on average, the height of the adults in this population deviates from the mean by 8.91 cm.

Standard deviation for a sample

The standard deviation for a sample is similar to the standard deviation for a population except that we divide by $n -1$ instead of $n$ and it is denoted $s$:

\[s = \sqrt{\frac{1}{n-1}\sum^n_{i = 1}(x_i - \bar{x})^2}\]

Imagine now that the 3 adults presented in the previous section is a sample instead of a population:

160.4, 175.8 and 181.5

The mean is still 172.6 (rounded to 1 decimal) since the mean is the same whether it is a population or a sample. The standard deviation is now:

\[s = \sqrt{\frac{1}{3 - 1} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big]}\] \[s = 10.92\]

In conclusion, the standard deviation for the heights of these 3 adults is 10.92 cm. The interpretation is the same than for a population.

Variance

The variance is simply the square of the standard deviation. Put it another way, the standard deviation is the square root of the variance. We also distinguish between the variance for a population and for a sample in the next sections.

Variance for a population

The variance for a population, denoted $\sigma^2$, is:

\[\sigma^2 = \frac{1}{n}\sum^n_{i = 1}(x_i - \mu)^2\]

As you can see, the formula for variance is the same than for standard deviation, except that the square root is removed for the variance. Remember the heights of our population of 3 adults:

160.4, 175.8 and 181.5

The standard deviation was 8.91 cm, so the variance of the height of these adults is $8.91^2 = 79.39$ $cm^2$ (see below why the unit of a variance is $unit^2$).

If you did not know the standard deviation of the population and needed to compute the variance of the population by hand, here is how:

\[\sigma^2 = \frac{1}{3} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big] \\ = 79.43\]

(The difference with the above result is due to rounding.)

Variance for a sample

Again, the variance for a sample is similar to the variance for a population except that we divide by $n - 1$ instead of $n$ and it is denoted $s^2$:

\[s^2 = \frac{1}{n-1}\sum^n_{i = 1}(x_i - \bar{x})^2\]

Imagine again that the 3 adults in the previous section is a sample instead of a population:

160.4, 175.8 and 181.5

The standard deviation for this sample was 10.92 cm, so the variance of the height of these adults is $10.92^2 = 119.25$ $cm^2$.

If you did not know the standard deviation of the sample and needed to compute the variance of the sample by hand, here is how:

\[s^2 = \frac{1}{3 - 1} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big] \\ = 119.15\]

(The difference with the above result is due to rounding.)

Standard deviation vs. variance

Standard deviation and variance are often used interchangeably and both quantify the spread of a given dataset by measuring how far the observations are from their mean. However, the standard deviation can be more easily interpreted because the unit for the standard deviation is the same than the unit of measurement of the data (while it is the $unit^2$ for the variance).

Following our example of adult heights in cm, the standard deviation is measured in cm while the variance is measured in $cm^2$. The fact that the standard deviation keeps the same unit than the initial unit of measurement makes it more interpretable and thus more often used in practice.

Notations

For completeness, below a table showing the different notations for variance and standard deviation in case of population and sample:

	Population	Sample
Standard deviation	$\sigma$	$s$
Variance	$\sigma^2$	$s^2$

Interquartile range

Remember the first $q_{0.25}$ and third quartile $q_{0.75}$ presented earlier (see this section). The interquartile range is another measure of dispersion of the data, using the quartiles. It is the difference between the third and first quartile:

\[IQR = q_{0.75} - q_{0.25}\]

Considering the height of the 9 adults presented in the section about the first and third quartile:

170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

The first quartile was 163.9 cm and the third quartile was 181.5 cm. The IQR is thus:

\[IQR = 181.5 - 163.9 = 17.6\]

In conclusion, the interquartile range is 17.6 cm. The interquartile range is actually the range (since it is the difference between a higher and a lower value) of the middle data. The graph below may help to understand better the IQR and the quartiles:

Coefficient of variation

The last dispersion measure is the coefficient of variation. The coefficient of variation, denoted $CV$, is the standard deviation divided by the mean. Formally:

\[CV = \frac{s}{\bar{x}}\]

Consider the height of a sample of 4 adults:

163.7, 160.4, 175.8 and 181.5

The mean is $\bar{x} =$ 170.35 cm and the standard deviation is $s =$ 9.95 cm. (Find the same values as an exercise!) The coefficient of variation is

\[CV = \frac{9.95 \text{ cm}}{170.35 \text{ cm}} = 0.058 = 5.8\%\]

In conclusion, the coefficient of variation is 5.8%. Note that, as a rule of thumb, a coefficient of variation greater than 15% usually means that the data are heterogeneous while a coefficient of variation equal to or less than 15% means that the data are homogeneous. Given that the coefficient of variation equals 5.8% in this case, we can conclude that these 4 adults are homogeneous in terms of height.

Note that the coefficient of variation for a population follows the same formula, except that notations for the mean and the standard deviation differ:

\[CV = \frac{\sigma}{\mu}\]

The interpretation is also the same as for a sample.

Coefficient of variation vs. standard deviation

Although the coefficient of variation is rather unknown to the public, it is, in fact, worth presenting when making descriptive statistics.

The standard deviation should always be understood in the context of the mean of the data and is dependent on its unit. The standard deviation has the advantage that it tells by how far on average the data is from the mean in terms of unit in which the data has been measured. Standard deviation is useful when considering variables with same units and approximately same means. However, standard deviation becomes less useful when comparing variables with different units or widely different means. For instance, a variable with a standard deviation of 10 cm cannot be compared to a variable with a standard deviation of 12€ to conclude which one of the two is the most dispersed.

The coefficient of variation is a ratio of two statistics with the same units. It has thus no unit and is independent of the unit in which the data has been measured. Being unit-free, coefficients of variation computed on datasets or variables with different units or widely different means can be compared to conclude, in fine, which data or variables is more (or less) dispersed. For instance, consider a sample of 10 women with their heights in cm and their salaries in €. We cannot compare the dispersion of their weights with the dispersion of their salaries because it is not measured on the same unit/scale. Now suppose that the coefficients of variation are 0.032 and 0.061 respectively for the height and the salary. Based on that, we can conclude that, relative to their respective average, their salaries vary more than their heights for these women. This is the case because the coefficient of variation is larger for the salary compared to the coefficient variation for the height, and a coefficient of variation has no unit (it is a ratio).

Conclusion

Remember that descriptive statistics are useful to describe and present a series of observations in a concise and informative way. There are two families of descriptive statistics: location and dispersion measures. Location measures give information about the position of the data, whereas dispersion measures give information about the variability of the data.

We showed how to compute the most common descriptive statistics by hand with concrete examples. We also discussed differences between some measures and when it is more appropriate to use one or the other depending on the context and the data at hand.

This concludes a relatively long article, thanks for reading! If you would like to learn how to compute these measures in R, read the article “Descriptive statistics in R”. A book I recommend for further reading is “The Art of Statistics” by David Spiegelhalter. It is a great book for beginners in statistics and covers a wide range of topics.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

This image and the following ones are taken from the LFSAB1105 course syllabus at UCLouvain.↩︎

What is the difference between population and sample?

Sat, 18 Jan 2020 00:00:00 +0000

Introduction

People often fail to properly distinguish between population and sample. It is however essential in any statistical analysis, starting from descriptive statistics with different formulas for variance and standard deviation depending on whether we face a sample or a population.

Moreover, the branch of statistics called inferential statistics is often defined as the science of drawing conclusions about a population from observations made on a representative sample of that population. It is therefore crucial to properly distinguish between the two concepts. So, what exactly is the difference between population and sample?

Sample vs. population

Population versus sample. Source: towardsdatascience.com

A population includes all members from a specified group, all possible outcomes or measurements that are of interest. The exact population will depend on the scope of the study. For example, say you would like to know whether there is an association between job performance and the amount of home working hours per week in the specific case of Belgian data scientists. In this case, the population might be Belgian data scientists. However, if the scope of the study is more narrow (e.g., the study focuses on french-speaking Belgian data scientists who live at least 30km away from their workplace), then the population will be more specific and include only workers who meet the criteria. The point is that the population should only include people to whom the results will apply.

A sample consists of some observations drawn from the population, so a part or a subset of the population. The sample is the group of elements who actually participated in the study.

Members and elements are defined in the broad sense of the term. It may be human. For instance, the population may be “all people living in Belgium” and the sample may be “some people living in Belgium”. It can be anything else too. Say you are testing the effect of a new fertilizer on crop yield. All the crop fields represent your population, whereas the 10 crop fields you tested correspond to your sample. Since a sample is a subset of a population, a sample is always smaller than the population.¹ Note that, a population must not necessarily be large. It might be the case that you study such a narrow population (e.g., first-year male bachelor students from your university who passed the statistics exam in June and for whom their parents have been divorced for more than 5 years), that the size of the population is actually rather small.

Why a sample?

As mentioned at the beginning of this article, one of the main concern in statistics is being able to draw conclusions about a population from a representative sample. Why using a sample of the population and not directly the population? In general it is almost always impossible to carry out measurements for the entire study population because:

the population is too large. Example: the population of pregnant women. If we want to take measurements on all pregnant women in the world, it will most likely either take too long or cost too much
the population is virtual. In this case “virtual” population is understood as a “hypothetical” population: it is unlimited in size. Example: for an experimental study, we focus on men with prostate cancer treated with a new treatment. We do not know how many people will be treated, so the population varies, is infinite and uncountable at the present time, and therefore virtual
the population is not easily reachable. Example: the population of homeless persons in Belgium

For these reasons, measurements are made on a subgroup of observations from the population, i.e., on a sample of our population. These measures are then used to draw conclusions about the population of interest. With an appropriate methodology and a sufficiently large sample size, the results obtained on a sample are often almost as accurate as those that would be obtained on the entire population.

Representative sample

Of course, the sample must be selected to be representative of the population under study. If participants are included in a study on a voluntary basis, there is a serious concern that the resulting sample may not be representative of the population. It may be the case that volunteers are different in terms of the parameter² of interest, leading to a selection bias. Another selection bias can occur when, for instance, a researcher collects citizens’ wage, by the means of internet. It might be the case that people having access to internet have different wages than people who do not have access.

The gold standard to select a sample representative of the population under study is by selecting a random sample. A random sample is a sample selected at random from the population so that each member of the population has an equal chance of being selected. A random sample is usually an unbiased sample, that is, a sample whose randomness is not in doubt.

In some situations or domains (e.g., in medicine, psychology, etc.) it is complicated or even impossible to obtain a random sample of the population. In such cases, it will be important to consider how representative the resulting sample will be.

Paired samples

Last but not least, paired samples are samples in which groups (often pairs) of experimental units are linked together by the same experimental conditions. For example, one may measure the hours of sleep for 20 patients before taking a sleeping pill (forming sample A), and then repeat the measurements on the same individuals after they have taken a sleeping pill (forming sample B). Another example is the grades of 25 students at their statistics and economics exam. Among each student, grades for these two exams are undoubtedly associated (to some extent) to each other.

Paired samples are usually formed when a variable of interest is measured at different times on the same experimental unit. But, paired samples can also occur even when measurements are made at one specific point in time. For example, if we measure the strength in the right and left arm of 30 athletes. Strength in the right and left arm for the same individual are linked to each other, so sample A (for the right arm) and sample B (for the left arm) are paired samples.

The two measurements for each individual (hours of sleep before and after the sleeping pill, grades at two different exams, or the strength in the right and left arm) and the two samples are of course related. Statistical tools accounting for a relation between the samples exist and should be preferred in that case.

Conclusion

To summarize, the sample is the group of individuals who participated in the study and the population is the broader group to whom the results will apply. Measurements on the entire population is often too complex or impossible, so representative samples are used to draw conclusions about the population. Samples based on a random selection are often the most representative samples.

Thanks for reading.

I hope the article helped you to understand the difference between population and sample. For the interested reader, see the most common descriptive statistics that can be made on a sample or a population.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There can be, however, many different samples from the same population. This is beyond the scope of this article, and at the moment we assume there is only one sample from a specified population.↩︎
The tools used to describe a population are called parameters, whereas the tools used to describe a sample are referred as statistics. See the most common descriptive statistics for a sample and a population.↩︎

A guide on how to read statistical tables

Mon, 06 Jan 2020 00:00:00 +0000

Below a Shiny app to help you read the main statistical tables:

Statistics-101

This Shiny app helps you to compute probabilities for the main probability distributions.

How to use this app?

Open the app via this link
Choose the distribution
Set the parameter(s) of the distribution (the parameters depend of course on the chosen distribution)
Select whether you want to find the lower tail, upper tail or an interval
Choose the value of x

On the right panel (or below depending on the size of your screen) you will see:

a recap of the data you just entered
the numerical solution (i.e., the probability)
a visualization of the solution
the probability density function together with the mean, the standard deviation and the variance

Example

Here is an example with the most common distribution: the normal distribution.

Suppose the following problem: The cost of weekly maintenance and repair of a business has been observed over a long period of time and turns out to be distributed according to a normal distribution with an average of 402€ and a standard deviation of 22€. Having set a budget of 439€ for next week, what is the probability that the cost exceeds this budget?

To solve this problem, follow these steps in the app:

Choose the normal distribution, as it is said that the costs follow a normal distribution
Set the mean $\mu$ equal to 402, as it is said that the average cost is 402€
In the statement, the standard deviation is given (and not the variance) so select “Standard deviation $\sigma$” and set it equal to 22
We are asked what is the probability the the cost exceeds the budget. Therefore, we look for the probability above a certain x, so select upper tail $P(X > x)$
We are now asked to find the probability that the cost exceeds 439€, so set x equal to 439

The solution panel gives a recap of the data:

\[X ∼ \mathcal{N}(\mu = 402, \sigma^2 = 484)\] where $484 = 22^2$, and the solution: \[P(X > 439) = P(Z > 1.68) = 0.0463\] where $Z = \frac{X - \mu}{\sigma} = \frac{439 - 402}{22} = 1.68$ and $Z ∼ \mathcal{N}(\mu = 0, \sigma^2 = 1)$ (known as the standard normal distribution). Thus, the probability that the cost next week exceeds the budget of 439€ is 0.0463, or 4.63%.

It also shows the normal distribution (with $\mu = 402$ and $\sigma^2 = 484$) with the shaded area corresponding to the probability we are looking for. It then gives some details about the density function, the mean, the standard deviation and the variance.

Code

See the last version on GitHub.

Note that the link may not work if the app has hit the monthly usage limit. Try again later if that is the case.

Conclusion

Thanks for reading.

I hope you will find this app useful to compute probabilities for the main distributions.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Data types in R

Mon, 30 Dec 2019 00:00:00 +0000

This article presents the different data types in R. To learn about the different variable types from a statistical point of view, read “Variable types and examples”.

What data types exist in R?

There are the 6 most common data types in R:

Numeric
Integer
Complex
Character
Factor
Logical

Datasets in R are often a combination of these 6 different data types. Below we explore in more detail each data types one by one, except the data type “complex” as we focus on the main ones and this data type is rarely used in practice.

Numeric

The most common data type in R is numeric. A variable or a series will be stored as numeric data if the values are numbers or if the values contains decimals. For example, the following two series are stored as numeric by default:

# numeric series without decimals
num_data <- c(3, 7, 2)
num_data

## [1] 3 7 2

class(num_data)

## [1] "numeric"

# numeric series with decimals
num_data_dec <- c(3.4, 7.1, 2.9)
num_data_dec

## [1] 3.4 7.1 2.9

class(num_data_dec)

## [1] "numeric"

# also possible to check the class thanks to str()
str(num_data_dec)

##  num [1:3] 3.4 7.1 2.9

In other words, if you assign one or several numbers to an object in R, it will be stored as numeric by default (numbers with decimals), unless specified otherwise.

Integer

Integer data type is actually a special case of numeric data. Integers are numeric data without decimals. It can be used if you are sure that the numbers you store will never contains decimals. For example, let’s say you are interested in the number of children in a sample of 10 families. This variable is a discrete variable (see a reminder on the variable types if you do not remember what is a discrete variable) and will never have decimals. Therefore, it can be stored as integer data thanks to the as.integer() command:

children

##  [1] 1 3 2 2 4 4 1 1 1 4

children <- as.integer(children)
class(children)

## [1] "integer"

Note that if your variable does not have decimals, R will automatically set the type as integers instead of numeric.

Character

The data type character is used when storing text, known as strings in R. The simplest ways to store data under the character format is by using "" around the piece of text:

char <- "some text"
char

## [1] "some text"

class(char)

## [1] "character"

If you want to force any kind of data to be stored as character, you can do it by using the command as.character():

char2 <- as.character(children)
char2

##  [1] "1" "3" "2" "2" "4" "4" "1" "1" "1" "4"

class(char2)

## [1] "character"

Note that everything inside "" will be considered as character, no matter if it looks like character or not. For example:

chars <- c("7.42")
chars

## [1] "7.42"

class(chars)

## [1] "character"

Furthermore, as soon as there is at least one character value inside a variable or vector, the whole variable or vector will be considered as character:

char_num <- c("text", 1, 3.72, 4)
char_num

## [1] "text" "1"    "3.72" "4"

class(char_num)

## [1] "character"

Last but not least, although space does not matter in numeric data, it does matter for character data:

num_space <- c(1)
num_nospace <- c(1)
# is num_space equal to num_nospace?
num_space == num_nospace

## [1] TRUE

char_space <- "text "
char_nospace <- "text"
# is char_space equal to char_nospace?
char_space == char_nospace

## [1] FALSE

As you can see from the results above, a space within character data (i.e., within "") makes it a different string in R!

Factor

Factor variables are a special case of character variables in the sense that it also contains text. However, factor variables are used when there are a limited number of unique character strings. It often represents a categorical variable. For instance, the gender will usually take on only two values, “female” or “male” (and will be considered as a factor variable) whereas the name will generally have lots of possibilities (and thus will be considered as a character variable). To create a factor variable use the factor() function:

gender <- factor(c("female", "female", "male", "female", "male"))
gender

## [1] female female male   female male  
## Levels: female male

To know the different levels of a factor variable, use levels():

levels(gender)

## [1] "female" "male"

By default, the levels are sorted alphabetically. You can reorder the levels with the argument levels in the factor() function:

gender <- factor(gender, levels = c("male", "female"))
levels(gender)

## [1] "male"   "female"

Character strings can be converted to factors with as.factor():

text <- c("test1", "test2", "test1", "test1") # create a character vector
class(text) # to know the class

## [1] "character"

text_factor <- as.factor(text) # transform to factor
class(text_factor) # recheck the class

## [1] "factor"

The character strings have been transformed to factors, as shown by its class of the type factor.

Logical

A logical variable is a variable with only two values; TRUE or FALSE:

value1 <- 7
value2 <- 9

# is value1 greater than value2?
greater <- value1 > value2
greater

## [1] FALSE

class(greater)

## [1] "logical"

# is value1 less than or equal to value2?
less <- value1 <= value2
less

## [1] TRUE

class(less)

## [1] "logical"

It is also possible to transform logical data into numeric data. After the transformation from logical to numeric with the as.numeric() command, FALSE values equal to 0 and TRUE values equal to 1:

greater_num <- as.numeric(greater)
sum(greater)

## [1] 0

less_num <- as.numeric(less)
sum(less)

## [1] 1

Conversely, numeric data can be converted to logical data, with FALSE for all values equal to 0 and TRUE for all other values.

x <- 0
as.logical(x)

## [1] FALSE

y <- 5
as.logical(y)

## [1] TRUE

Conclusion

Thanks for reading.

I hope this article helped you to understand the basic data types in R and their particularities. If you would like to learn more about the different variable types from a statistical point of view, read the article “Variable types and examples”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Variable types and examples

Mon, 30 Dec 2019 00:00:00 +0000

Introduction

If you happen to work with datasets frequently, you probably know that each row of your dataset represents a different experimental unit (also called observation) and each column represents a different characteristic (called variable):

Structure of a dataset. Source: R for Data Science by Hadley Wickham & Garrett Grolemund

If you do some research on the weight and height of 100 students of your university, for example, you will most likely have a dataset containing 100 rows and 3 columns:

one for the student’s ID (could be anonymized or not),
one for the weight,
and one for the height.

These three columns represent three characteristics of the 100 students. They are called variables.

In this article, we are going to focus on variables, and in particular on the different types of variable that exist in statistics. (To learn about the different data types in R, read “Data types in R”.)

Different types of variables for different types of statistical analysis

First, one may wonder why we are interested in defining the types of our variables of interest.

The reason why we often class variables into different types is because not all statistical analyses can be performed on all variable types. For instance, it is impossible to compute the mean of the variable “hair color” as you cannot sum brown and blond hair.

On the other hand, finding the mode of a continuous variable does not really make any sense because most of the time there will not be two exact same values, so there will be no mode. And even in the case there is a mode, there will be very few observations with this value. As an example, try finding the mode of the height of the students in your class. If you are lucky, a couple of students will have the same size. However, most of the time, every student will have a different size (especially if heights have been measured in millimeters) and thus there will be no mode. To see what kind of analysis is possible on each type of variable, see more details in the articles “Descriptive statistics by hand” and “Descriptive statistics in R”.

Similarly, some statistical tests can only be performed on certain type of variables. For example, the Pearson correlation is usually computed on two quantitative variables, while a Chi-square test of independence is done with two qualitative variables, and a Student t-test or ANOVA requires a mix of one quantitative and one qualitative variable.

Big picture

In statistics, variables are classified into 4 different types:

We present each type together with examples in the following sections.

Quantitative

A quantitative variable is a variable that reflects a notion of magnitude, that is, if the values it can take are numbers. A quantitative variable represents thus a measure and is numerical.

Quantitative variables are divided into two types: discrete and continuous. The difference is explained in the following two sections.

Discrete

Quantitative discrete variables are variables for which the values it can take are countable and have a finite number of possibilities. The values are often (but not always) integers. Here are some examples of discrete variables:

Number of children per family
Number of students in a class
Number of citizens of a country

Even if it would take a long time to count the citizens of a large country, it is still technically doable. Moreover, for all examples, the number of possibilities is finite. Whatever the number of children in a family, it will never be 3.58 or 7.912 so the number of possibilities is a finite number and thus countable.

Continuous

On the other hand, quantitative continuous variables are variables for which the values are not countable and have an infinite number of possibilities. For example:

Age
Weight
Height

For simplicity, we usually referred to years, kilograms (or pounds) and centimeters (or feet and inches) for age, weight and height respectively. However, a 28-year-old man could actually be 28 years, 7 months, 16 days, 3 hours, 4 minutes, 5 seconds, 31 milliseconds, 9 nanoseconds old.

For all measurements, we usually stop at a standard level of granularity, but nothing (except our measurement tools) prevents us from going deeper, leading to an infinite number of potential values. The fact that the values can take an infinite number of possibilities makes it uncountable.

Qualitative

In opposition to quantitative variables, qualitative variables (also referred as categorical variables or factors in R) are variables that are not numerical and which values fit into categories.

In other words, a qualitative variable is a variable which takes as its values modalities, categories or even levels, in contrast to quantitative variables which measure a quantity on each individual.

Qualitative variables are divided into two types: nominal and ordinal.

Nominal

A qualitative nominal variable is a qualitative variable where no ordering is possible or implied in the levels.

For example, the variable gender is nominal because there is no order in the levels (no matter how many levels you consider for the gender—only two with female/male, or more than two with female/male/ungendered/others, levels are unordered). Eye color is another example of a nominal variable because there is no order among blue, brown or green eyes.

A nominal variable can have:

two levels (e.g., do you smoke? Yes/No, or are you pregnant? Yes/No), or
a large number of levels (what is your college major? Each major is a level in that case).

Note that a qualitative variable with exactly 2 levels is also referred as a binary or dichotomous variable.

Ordinal

On the other hand, a qualitative ordinal variable is a qualitative variable with an order implied in the levels. For instance, if the severity of road accidents has been measured on a scale such as light, moderate and fatal accidents, this variable is a qualitative ordinal variable because there is a clear order in the levels.

Another good example is health, which can take values such as poor, reasonable, good, or excellent. Again, there is a clear order in these levels so health is in this case a qualitative ordinal variable.

Variable transformations

There are two main variable transformations:

From a continuous to a discrete variable
From a quantitative to a qualitative variable

From continuous to discrete

Let’s say we are interested in babies’ ages. The data collected is the age of the babies, so a quantitative continuous variable. However, we may work with only the number of weeks since birth and thus transforming the age into a discrete variable. The variable age remains a quantitative continuous variable but the variable we are working on (i.e., the number of weeks since birth) can be seen as a quantitative discrete variable.

From quantitative to qualitative

Let’s say we are interested in the Body Mass Index (BMI). For this, a researcher collects data on height and weight of individuals and computes the BMI. The BMI is a quantitative continuous variable but the researcher may want to turn it into a qualitative variable by categorizing individuals below a certain threshold as underweight, above a certain threshold as overweight and the rest as normal weight. The raw BMI is a quantitative continuous variable but the categorization of the BMI makes the transformed variable a qualitative (ordinal) variable, where the levels are in this case underweight < normal < overweight.

Same goes for age when age is transformed to a qualitative ordinal variable with levels such as minors, adults and seniors. It is also often the case (especially in surveys) that the variable salary (quantitative continuous) is transformed into a qualitative ordinal variable with different range of salaries (e.g., < 1000€, 1000 - 2000€, > 2000€).

Additional notes

Misleading data encoding

Last but not least, in datasets it is very often the case that numbers are used for qualitative variables. For instance, a researcher may assign the number “1” to women and the number “2” to men (or “0” to the answer “No” and “1” to the answer “Yes”). Despite the numerical classification, the variable gender is still a qualitative variable and not a discrete variable as it may look. The numerical classification is only used to facilitate data collection and data management. It is indeed easier to write the number “1” or “2” instead of “women” or “men”, and thus less prone to encoding errors.

The same goes for the identification of each observation. Suppose you collected information on 100 students. You may use their student’s ID to identify them in the dataset (so that you can trace them back). Most of the time, students’ ID (or ID in general) are encoded as numeric values. At first sight, it may thus look like a quantitative variable (because it goes from 1 to 100 for example). However, ID is clearly not a quantitative variable because it actually corresponds to an anonymized version of the student’s first and last name. If you think about it, it would make no sense to compute the mean or median on the IDs, as it does not represent a numerical measurement (but rather just an easier way to identify students than with their names).

If you face this kind of setup, do not forget to transform your variable into the right type before performing any statistical analyses. Usually, a basic descriptive analysis (and knowledge about the variables which have been measured) prior to the main statistical analyses is enough to check that all variable types are correct.

Conclusion

Thanks for reading.

I hope this article helped you to understand the different types of variable. If you would like to learn more about the different data types in R, read the article “Data types in R”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Data manipulation in R

Tue, 24 Dec 2019 00:00:00 +0000

Note that this article is inspired from a workshop entitled “Introduction to data analysis with R”, given by UCLouvain’s Statistical Methodology and Computing Service. See all their workshops on their website.

Introduction

Not all data frames are as clean and tidy as you would expect. Therefore, after importing your data frame into RStudio, most of the time you will need to prepare it before performing any statistical analyses. Data manipulation can even sometimes take longer than the actual analyses when the quality of the data is poor.

Data manipulation include a broad range of tools and techniques. We present here in details the manipulations that you will most likely need for your projects in R. Do not hesitate to let me know (as a comment at the end of this article for example) if you find other data manipulations essential so that I can add them.

In this article we show the main functions to manipulate data in R. We first illustrate these functions on vectors, factors and lists. We then illustrate the main functions to manipulate data frames and dates/times in R.

For those who are interested in going further, see also an introduction to data manipulation in R with the {dplyr} package.

Vectors

Concatenation

We can concatenate (i.e., combine) numbers or strings with c():

c(2, 4, -1)

## [1]  2  4 -1

c(1, 5 / 6, 2^3, -0.05)

## [1]  1.0000000  0.8333333  8.0000000 -0.0500000

Note that by default R displays 7 decimals. You can modify it with options(digits = 2) (two decimals).

It is also possible to create a sequence of consecutive integers:

1:10

##  [1]  1  2  3  4  5  6  7  8  9 10

# is the same than
c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

##  [1]  1  2  3  4  5  6  7  8  9 10

# or
c(1:10)

##  [1]  1  2  3  4  5  6  7  8  9 10

`seq()` and `rep()`

seq() allows to make a vector defined by a sequence. You can either choose the increment:

seq(from = 2, to = 5, by = 0.5)

## [1] 2.0 2.5 3.0 3.5 4.0 4.5 5.0

or its length:

seq(from = 2, to = 5, length.out = 7)

## [1] 2.0 2.5 3.0 3.5 4.0 4.5 5.0

On the other hand, rep() creates a vector which is the repetition of numbers or strings:

rep(1, times = 3)

## [1] 1 1 1

rep(c("A", "B", "C"), times = c(3, 1, 2))

## [1] "A" "A" "A" "B" "C" "C"

You can also create a vector which is the repetition of numbers and strings:

rep(c("A", 2, "C"), times = c(3, 1, 2))

## [1] "A" "A" "A" "2" "C" "C"

but in that case, the number 2 will be considered as a string too (and not as a numeric) since there is at least one string in the vector.

Assignment

There are three ways to assign an object in R:

<-
=
assign()

# 1st method
x <- c(2.1, 5, -4, 1, 5)
x

## [1]  2.1  5.0 -4.0  1.0  5.0

# 2nd method
x2 = c(2.1, 5, -4, 1, 5)
x2

## [1]  2.1  5.0 -4.0  1.0  5.0

# 3rd method (much less common)
assign("x3", c(2.1, 5, -4, 1, 5))
x3

## [1]  2.1  5.0 -4.0  1.0  5.0

You can also assign a vector to another vector, for example:

y <- c(x, 10, 1 / 4)
y

## [1]  2.10  5.00 -4.00  1.00  5.00 10.00  0.25

Elements of a vector

We can select one or several elements of a vector by specifying its position between square brackets:

# select one element
x[3]

## [1] -4

# select more than one element with c()
x[c(1, 3, 4)]

## [1]  2.1 -4.0  1.0

Note that in R the numbering of the indices starts at 1 (and no 0 like other programming languages) so x[1] gives the first element of the vector x.

We can also use booleans (i.e., TRUE or FALSE) to select some elements of a vector. This method selects only the elements corresponding to TRUE:

x[c(TRUE, FALSE, TRUE, TRUE, FALSE)]

## [1]  2.1 -4.0  1.0

Or we can give the elements to withdraw:

x[-c(2, 4)]

## [1]  2.1 -4.0  5.0

Type and length

The main types of a vector are numeric, logical and character. For more details on each type, see the different data types in R.

class() gives the vector type:

x <- c(2.1, 5, -4, 1, 5, 0)
class(x)

## [1] "numeric"

y <- c(x, "Hello")
class(y)

## [1] "character"

As you can see above, the class of a vector will be numeric only if all of its elements are numeric. As soon as one element is a character, the class of the vector will be a character.

z <- c(TRUE, FALSE, FALSE)
class(z)

## [1] "logical"

length() gives the length of a vector:

length(x)

## [1] 6

So to select the last element of a vector (in a dynamic way), we can use a combination of length() and []:

x[length(x)]

## [1] 0

Finding the vector type

We can find the type of a vector with the family of is.type functions:

is.numeric(x)

## [1] TRUE

is.logical(x)

## [1] FALSE

is.character(x)

## [1] FALSE

Or in a more generic way with the is() function:

is(x)

## [1] "numeric" "vector"

Modifications of type and length

We can change the type of a vector with the as.numeric(), as.logical() and as.character() functions:

x_character <- as.character(x)
x_character

## [1] "2.1" "5"   "-4"  "1"   "5"   "0"

is.character(x_character)

## [1] TRUE

x_logical <- as.logical(x)
x_logical

## [1]  TRUE  TRUE  TRUE  TRUE  TRUE FALSE

is.logical(x_logical)

## [1] TRUE

It is also possible to change its length:

length(x) <- 4
x

## [1]  2.1  5.0 -4.0  1.0

As you can see, the first elements of the vector are conserved while all others are removed. In this case, the first 4 since we specified a length of 4.

Numerical operators

The basic numerical operators such as +, -, *, / and ^ can be applied to vectors:

x <- c(2.1, 5, -4, 1)
y <- c(0, -7, 1, 1 / 4)

x + y

## [1]  2.10 -2.00 -3.00  1.25

x * y

## [1]   0.00 -35.00  -4.00   0.25

x^y

## [1]  1.00e+00  1.28e-05 -4.00e+00  1.00e+00

It is also possible to compute the minimum, maximum, sum, product, cumulative sum and cumulative product of a vector:

min(x)

## [1] -4

max(x)

## [1] 5

sum(x)

## [1] 4.1

prod(x)

## [1] -42

cumsum(x)

## [1] 2.1 7.1 3.1 4.1

cumprod(x)

## [1]   2.1  10.5 -42.0 -42.0

The following mathematical operations can be applied too:

sqrt() (square root)
cos() (cosine)
sin() (sine)
tan() (tangent)
log() (logarithm)
log10() (base 10 logarithm)
exp() (exponential)
abs() (absolute value)

cos(x)

## [1] -0.5048461  0.2836622 -0.6536436  0.5403023

exp(x)

## [1]   8.16616991 148.41315910   0.01831564   2.71828183

If you need to round a number, you can use the round(), floor() and ceiling() functions:

round(cos(x), digits = 3) # round to 3 decimals

## [1] -0.505  0.284 -0.654  0.540

floor(cos(x)) # largest integer not greater than x

## [1] -1  0 -1  0

ceiling(cos(x)) # smallest integer not less than x

## [1] 0 1 0 1

Logical operators

The most common logical operators in R are:

Negation: !
Comparisons: <, <=, >=, >, == (equality), != (difference)
And: &
Or: |

## [1]  2.1  5.0 -4.0  1.0

x <= c(1, 6, 3, 4)

## [1] FALSE  TRUE  TRUE  TRUE

x <= 1

## [1] FALSE FALSE  TRUE  TRUE

(x == 1 | x > 4)

## [1] FALSE  TRUE FALSE  TRUE

!(x == 1 | x > 4)

## [1]  TRUE FALSE  TRUE FALSE

`all()` and `any()`

As the names suggest, all() return TRUE if conditions are met for all elements, whereas any() returns TRUE if conditions are met for any of the element of a vector:

## [1]  2.1  5.0 -4.0  1.0

x <= 1

## [1] FALSE FALSE  TRUE  TRUE

all(x <= 1)

## [1] FALSE

any(x <= 1)

## [1] TRUE

Operations on character strings vector

You can paste two vectors (or more) together:

code <- paste(c("BE", "BE", "FR", "EN", "BE"), 1:5, sep = "/")
code

## [1] "BE/1" "BE/2" "FR/3" "EN/4" "BE/5"

The argument sep stands for separator and allows to specify the character(s) or symbol(s) used to separate each character strings.

If you do not want to specify a separator, you can use sep = "" or the paste0() function:

paste(c("BE", "BE", "FR", "EN", "BE"), 1:5, sep = "")

## [1] "BE1" "BE2" "FR3" "EN4" "BE5"

paste0(c("BE", "BE", "FR", "EN", "BE"), 1:5)

## [1] "BE1" "BE2" "FR3" "EN4" "BE5"

To find the positions of the elements containing a given string, use the grep() function:

grep("BE", code)

## [1] 1 2 5

To extract a character string based on the beginning and the end positions, we can use the substr() function:

# extract characters 1 to 3
substr(code,
  start = 1,
  stop = 3
)

## [1] "BE/" "BE/" "FR/" "EN/" "BE/"

Replace a character string by another one if it exists in the vector by using the sub() function:

sub(
  pattern = "BE", # find BE
  replacement = "BEL", # replace it with BEL
  code
)

## [1] "BEL/1" "BEL/2" "FR/3"  "EN/4"  "BEL/5"

Split a character string based on a specific symbol with the strsplit() function:

strsplit(c("Rafael Nadal", "Roger Federer", "Novak Djokovic"),
  split = " "
)

## [[1]]
## [1] "Rafael" "Nadal" 
## 
## [[2]]
## [1] "Roger"   "Federer"
## 
## [[3]]
## [1] "Novak"    "Djokovic"

strsplit(code,
  split = "/"
)

## [[1]]
## [1] "BE" "1" 
## 
## [[2]]
## [1] "BE" "2" 
## 
## [[3]]
## [1] "FR" "3" 
## 
## [[4]]
## [1] "EN" "4" 
## 
## [[5]]
## [1] "BE" "5"

To transform a character vector to uppercase and lowercase:

toupper(c("Rafael Nadal", "Roger Federer", "Novak Djokovic"))

## [1] "RAFAEL NADAL"   "ROGER FEDERER"  "NOVAK DJOKOVIC"

tolower(c("Rafael Nadal", "Roger Federer", "Novak Djokovic"))

## [1] "rafael nadal"   "roger federer"  "novak djokovic"

Orders and vectors

We can sort the elements of a vector from smallest to largest, or from largest to smallest:

x <- c(2.1, 5, -4, 1, 1)
sort(x) # smallest to largest

## [1] -4.0  1.0  1.0  2.1  5.0

sort(x, decreasing = TRUE) # largest to smallest

## [1]  5.0  2.1  1.0  1.0 -4.0

order() gives the permutation to apply to the vector in order to sort its elements:

order(x)

## [1] 3 4 5 1 2

As you can see, the third element of the vector is the smallest and the second element is the largest. This is indicated by the 3 at the beginning of the output, and the 2 at the end of the output.

Like sort() the decreasing = TRUE argument can also be added:

order(x, decreasing = TRUE)

## [1] 2 1 4 5 3

In this case, the 2 in the output indicates that the second element of the vector is the largest, while the 3 indicates that the third element is the smallest.

rank() gives the ranks of the elements:

rank(x)

## [1] 4.0 5.0 1.0 2.5 2.5

The two last elements of the vector have a rank of 2.5 because they are equal and they come after the first but before the fourth rank.

We can also reverse the elements (from the last one to the first one):

## [1]  2.1  5.0 -4.0  1.0  1.0

rev(x)

## [1]  1.0  1.0 -4.0  5.0  2.1

Factors

Factors in R are vectors with a list of levels, also referred as categories. Factors are useful for qualitative data such as the gender, civil status, eye color, etc.

Creating factors

We create factors with the factor() function (do not forget the c()):

f1 <- factor(c("T1", "T3", "T1", "T2"))
f1

## [1] T1 T3 T1 T2
## Levels: T1 T2 T3

We can of course create a factor from an existing vector:

v <- c(1, 1, 0, 1, 0)
v2 <- factor(v,
  levels = c(0, 1),
  labels = c("bad", "good")
)
v2

## [1] good good bad  good bad 
## Levels: bad good

We can also specify that the levels are ordered by adding the ordered = TRUE argument:

v2 <- factor(v,
  levels = c(0, 1),
  labels = c("bad", "good"),
  ordered = TRUE
)
v2

## [1] good good bad  good bad 
## Levels: bad < good

Note that the order of the levels will follow the order that is specified in the labels argument.

Properties

To know the names of the levels:

levels(f1)

## [1] "T1" "T2" "T3"

For the number of levels:

nlevels(f1)

## [1] 3

In R, the first level is always the reference level. This reference level can be modified with relevel():

relevel(f1, ref = "T3")

## [1] T1 T3 T1 T2
## Levels: T3 T1 T2

You see that “T3” is now the first and thus the reference level. Changing the reference level has an impact on the order they are displayed or treated in statistical analyses. Compare, for instance, boxplots with different reference levels.

Handling

To know the frequencies for each level:

table(f1)

## f1
## T1 T2 T3 
##  2  1  1

# or
summary(f1)

## T1 T2 T3 
##  2  1  1

Note that the relative frequencies (i.e., the proportions) can be found with the combination of prop.table() and table() or summary():

prop.table(table(f1))

## f1
##   T1   T2   T3 
## 0.50 0.25 0.25

# or
prop.table(summary(f1))

##   T1   T2   T3 
## 0.50 0.25 0.25

Remember that a factor is coded in R as a numeric vector even though it looks like a character one. We can transform a factor into its numerical equivalent with the as.numeric() function:

f1

## [1] T1 T3 T1 T2
## Levels: T1 T2 T3

as.numeric(f1)

## [1] 1 3 1 2

And a numeric vector can be transformed into a factor with the as.factor() or factor() function:

num <- 1:4
fac <- as.factor(num)
fac

## [1] 1 2 3 4
## Levels: 1 2 3 4

fac2 <- factor(num)
fac2

## [1] 1 2 3 4
## Levels: 1 2 3 4

The advantage of factor() is that it is possible to specify a name for each level:

fac2 <- factor(num,
  labels = c("bad", "neutral", "good", "very good")
)
fac2

## [1] bad       neutral   good      very good
## Levels: bad neutral good very good

Lists

A list is a vector whose elements can be of different natures: a vector, a list, a factor, numeric or character, etc.

Creating lists

The function list() allows to create lists:

tahiti <- list(
  plane = c("Airbus", "Boeing"),
  departure = c("Brussels", "Milan", "Paris"),
  duration = c(15, 11, 14)
)
tahiti

## $plane
## [1] "Airbus" "Boeing"
## 
## $departure
## [1] "Brussels" "Milan"    "Paris"   
## 
## $duration
## [1] 15 11 14

Handling

There are several methods to extract elements from a list:

tahiti$departure

## [1] "Brussels" "Milan"    "Paris"

# or
tahiti$de

## [1] "Brussels" "Milan"    "Paris"

# or
tahiti[[2]]

## [1] "Brussels" "Milan"    "Paris"

# or
tahiti[["departure"]]

## [1] "Brussels" "Milan"    "Paris"

tahiti[[2]][c(1, 2)]

## [1] "Brussels" "Milan"

To transform a list into a vector:

v <- unlist(tahiti)
v

##     plane1     plane2 departure1 departure2 departure3  duration1  duration2 
##   "Airbus"   "Boeing" "Brussels"    "Milan"    "Paris"       "15"       "11" 
##  duration3 
##       "14"

is.vector(v)

## [1] TRUE

Getting details on an object

attributes() gives the names of the elements (it can be used on every R object):

attributes(tahiti)

## $names
## [1] "plane"     "departure" "duration"

str() gives a short description about the elements (it can also be used on every R object):

str(tahiti)

## List of 3
##  $ plane    : chr [1:2] "Airbus" "Boeing"
##  $ departure: chr [1:3] "Brussels" "Milan" "Paris"
##  $ duration : num [1:3] 15 11 14

Data frames

Every imported file in R is a data frame (at least if you do not use a package to import your data in R). A data frame is a mix of a list and a matrix: it has the shape of a matrix but the columns can have different classes.

Remember that the gold standard for a data frame is that:

columns represent variables
lines correspond to observations and
each value must have its own cell

Structure of a data frame. Source: R for Data Science by Hadley Wickham & Garrett Grolemund

In this article, we use the data frame cars to illustrate the main data manipulation techniques. Note that the data frame is installed by default in RStudio (so you do not need to import it) and I use the generic name dat as the name of the data frame throughout the article (see here why I always use a generic name instead of more specific names).

Here is the whole data frame:

dat <- cars # rename the cars data frame with a generic name
dat # display the entire data frame

##    speed dist
## 1      4    2
## 2      4   10
## 3      7    4
## 4      7   22
## 5      8   16
## 6      9   10
## 7     10   18
## 8     10   26
## 9     10   34
## 10    11   17
## 11    11   28
## 12    12   14
## 13    12   20
## 14    12   24
## 15    12   28
## 16    13   26
## 17    13   34
## 18    13   34
## 19    13   46
## 20    14   26
## 21    14   36
## 22    14   60
## 23    14   80
## 24    15   20
## 25    15   26
## 26    15   54
## 27    16   32
## 28    16   40
## 29    17   32
## 30    17   40
## 31    17   50
## 32    18   42
## 33    18   56
## 34    18   76
## 35    18   84
## 36    19   36
## 37    19   46
## 38    19   68
## 39    20   32
## 40    20   48
## 41    20   52
## 42    20   56
## 43    20   64
## 44    22   66
## 45    23   54
## 46    24   70
## 47    24   92
## 48    24   93
## 49    24  120
## 50    25   85

This data frame has 50 observations with 2 variables (speed and distance).

You can check the number of observations and variables with nrow() and ncol() respectively, or both at the same time with dim():

nrow(dat) # number of rows/observations

## [1] 50

ncol(dat) # number of columns/variables

## [1] 2

dim(dat) # dimension: number of rows and number of columns

## [1] 50  2

Line and column names

Before manipulating a data frame, it is interesting to know the line and column names:

dimnames(dat)

## [[1]]
##  [1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14" "15"
## [16] "16" "17" "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30"
## [31] "31" "32" "33" "34" "35" "36" "37" "38" "39" "40" "41" "42" "43" "44" "45"
## [46] "46" "47" "48" "49" "50"
## 
## [[2]]
## [1] "speed" "dist"

To know only the column names:

names(dat)

## [1] "speed" "dist"

# or
colnames(dat)

## [1] "speed" "dist"

And to know only the row names:

rownames(dat)

##  [1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14" "15"
## [16] "16" "17" "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30"
## [31] "31" "32" "33" "34" "35" "36" "37" "38" "39" "40" "41" "42" "43" "44" "45"
## [46] "46" "47" "48" "49" "50"

Subset a data frame

First or last observations

To keep only the first 10 observations:

head(dat, n = 10)

##    speed dist
## 1      4    2
## 2      4   10
## 3      7    4
## 4      7   22
## 5      8   16
## 6      9   10
## 7     10   18
## 8     10   26
## 9     10   34
## 10    11   17

To keep only the last 5 observations:

tail(dat, n = 5)

##    speed dist
## 46    24   70
## 47    24   92
## 48    24   93
## 49    24  120
## 50    25   85

Random sample of observations

To draw a sample of 4 observations without replacement:

library(dplyr)
sample_n(dat, 4, replace = FALSE)

##   speed dist
## 1    24  120
## 2    19   46
## 3     4    2
## 4    15   26

Based on row or column numbers

If you know what observation(s) or column(s) you want to keep, you can use the row or column number(s) to subset your data frame. We illustrate this with several examples:

keep all the variables for the $3^{rd}$ observation:

dat[3, ]

keep the $2^{nd}$ variable for all observations:

dat[, 2]

You can mix the two above methods to keep only the $2^{nd}$ variable of the $3^{rd}$ observation:

dat[3, 2]

## [1] 4

keep several observations; for example observations $1$ to $5$, the $10^{th}$ and the $15^{th}$ observation for all variables:

dat[c(1:5, 10, 15), ] # do not forget c()

##    speed dist
## 1      4    2
## 2      4   10
## 3      7    4
## 4      7   22
## 5      8   16
## 10    11   17
## 15    12   28

remove observations 5 to 45:

dat[-c(5:45), ]

##    speed dist
## 1      4    2
## 2      4   10
## 3      7    4
## 4      7   22
## 46    24   70
## 47    24   92
## 48    24   93
## 49    24  120
## 50    25   85

tip: to keep only the last observation, use nrow() instead of the row number:

dat[nrow(dat), ] # nrow() gives the number of rows

##    speed dist
## 50    25   85

This way, no matter the number of observations, you will always select the last one. This technique of using a piece of code instead of a specific value is to avoid “hard coding”. Hard coding is generally not recommended (unless you want to specify a parameter that you are sure will never change) because if your data frame changes, you will need to manually edit your code.

As you probably figured out by now, you can select observations and/or variables of a dataset by running dataset_name[row_number, column_number]. When the row (column) number is left empty, the entire row (column) is selected.

Note that all examples presented above also work for matrices:

mat <- matrix(c(-1, 2, 0, 3), ncol = 2, nrow = 2)
mat

##      [,1] [,2]
## [1,]   -1    0
## [2,]    2    3

mat[1, 2]

## [1] 0

Based on variable names

To select one variable of the dataset based on its name rather than on its column number, use dataset_name$variable_name:

dat$speed

##  [1]  4  4  7  7  8  9 10 10 10 11 11 12 12 12 12 13 13 13 13 14 14 14 14 15 15
## [26] 15 16 16 17 17 17 18 18 18 18 19 19 19 20 20 20 20 20 22 23 24 24 24 24 25

Accessing variables inside a data frame with this second method is strongly recommended compared to the first if you intend to modify the structure of your database. Indeed, if a column is added or removed in the data frame, the numbering will change. Therefore, variables are generally referred to by its name rather than by its position (column number). In addition, it is easier to understand and interpret code with the name of the variable written (another reason to call variables with a concise but clear name). There is only one reason why I would still use the column number; if the variables names are expected to change while the structure of the data frame will not change.

To select variables, it is also possible to use the select() command from the powerful dplyr package (for compactness only the first 6 observations are displayed thanks to the head() command):

head(select(dat, speed))

##   speed
## 1     4
## 2     4
## 3     7
## 4     7
## 5     8
## 6     9

This is equivalent than removing the distance variable:

head(select(dat, -dist))

##   speed
## 1     4
## 2     4
## 3     7
## 4     7
## 5     8
## 6     9

Based on one or multiple criterion

Instead of subsetting a data frame based on row/column numbers or variable names, you can also subset it based on one or multiple criterion:

keep only observations with speed larger than 20. The first argument refers to the name of the data frame, while the second argument refers to the subset criteria:

subset(dat, dat$speed > 20)

##    speed dist
## 44    22   66
## 45    23   54
## 46    24   70
## 47    24   92
## 48    24   93
## 49    24  120
## 50    25   85

keep only observations with distance smaller than or equal to 50 and speed equal to 10. Note the == (and not =) for the equal criteria:

subset(dat, dat$dist <= 50 & dat$speed == 10)

##   speed dist
## 7    10   18
## 8    10   26
## 9    10   34

use | to keep only observations with distance smaller than 20 or speed equal to 10:

subset(dat, dat$dist < 20 | dat$speed == 10)

##    speed dist
## 1      4    2
## 2      4   10
## 3      7    4
## 5      8   16
## 6      9   10
## 7     10   18
## 8     10   26
## 9     10   34
## 10    11   17
## 12    12   14

to filter out some observations, use !=. For instance, to keep observations with speed not equal to 24 and distance not equal to 120 (for compactness only the last 6 observations are displayed thanks to the tail() command):

tail(subset(dat, dat$speed != 24 & dat$dist != 120))

##    speed dist
## 41    20   52
## 42    20   56
## 43    20   64
## 44    22   66
## 45    23   54
## 50    25   85

Note that it is also possible to subset a data frame with split():

split(dat, dat$factor_variable)

The above code will split your data frame into several lists, one for each level of the factor variable.

Create a new variable

Often, a data frame can be enhanced by creating new variables based on other variables from the initial data frame, or simply by adding a new variable manually.

In this example, we create two new variables; one being the speed times the distance (which we call speed_dist) and the other being a categorization of the speed (which we call speed_cat). We then display the first 6 observations of this new data frame with the 4 variables:

# create new variable speed_dist
dat$speed_dist <- dat$speed * dat$dist

# create new variable speed_cat
# with ifelse(): if dat$speed > 7, then speed_cat is "high speed", otherwise it is "low_speed"
dat$speed_cat <- factor(ifelse(dat$speed > 7,
  "high speed", "low speed"
))

# display first 6 observations
head(dat) # 6 is the default in head()

##   speed dist speed_dist  speed_cat
## 1     4    2          8  low speed
## 2     4   10         40  low speed
## 3     7    4         28  low speed
## 4     7   22        154  low speed
## 5     8   16        128 high speed
## 6     9   10         90 high speed

Note than in programming, a character string is generally surrounded by quotes (e.g., "character string") and R is not an exception.

Transform a continuous variable into a categorical variable

To transform a continuous variable into a categorical variable (also known as qualitative variable):

dat$speed_quali <- cut(dat$speed,
  breaks = c(0, 12, 15, 19, 26), # cut points
  right = FALSE # closed on the left, open on the right
)

dat[c(1:2, 23:24, 49:50), ] # display some observations

##    speed dist speed_dist  speed_cat speed_quali
## 1      4    2          8  low speed      [0,12)
## 2      4   10         40  low speed      [0,12)
## 23    14   80       1120 high speed     [12,15)
## 24    15   20        300 high speed     [15,19)
## 49    24  120       2880 high speed     [19,26)
## 50    25   85       2125 high speed     [19,26)

This transformation is for example often done on age, when the age (a continuous variable) is transformed into a qualitative variable representing different age groups.

Sum and mean in rows

In survey with Likert scale (used in psychology, among others), it is often the case that we need to compute a score for each respondents based on multiple questions. The score is usually the mean or the sum of all the questions of interest.

This can be done with rowMeans() and rowSums(). For instance, let’s compute the mean and the sum of the variables speed, dist and speed_dist (variables must be numeric of course as a sum and a mean cannot be computed on qualitative variables!) for each row and store them under the variables mean_score and total_score:

dat$mean_score <- rowMeans(dat[, 1:3]) # variables speed, dist and speed_dist correspond to variables 1 to 3
dat$total_score <- rowSums(dat[, 1:3])

head(dat)

##   speed dist speed_dist  speed_cat speed_quali mean_score total_score
## 1     4    2          8  low speed      [0,12)   4.666667          14
## 2     4   10         40  low speed      [0,12)  18.000000          54
## 3     7    4         28  low speed      [0,12)  13.000000          39
## 4     7   22        154  low speed      [0,12)  61.000000         183
## 5     8   16        128 high speed      [0,12)  50.666667         152
## 6     9   10         90 high speed      [0,12)  36.333333         109

Sum and mean in column

It is also possible to compute the mean and sum by column with colMeans() and colSums():

colMeans(dat[, 1:3])

##      speed       dist speed_dist 
##      15.40      42.98     769.64

colSums(dat[, 1:3])

##      speed       dist speed_dist 
##        770       2149      38482

This is equivalent than:

mean(dat$speed)

## [1] 15.4

sum(dat$speed)

## [1] 770

but it allows to do it for several variables at a time.

Categorical variables and labels management

For categorical variables, it is a good practice to use the factor format and to name the different levels of the variables.

for this example, let’s create another new variable called dist_cat based on the distance and then change its format from numeric to factor (while also specifying the labels of the levels):

# create new variable dist_cat
dat$dist_cat <- ifelse(dat$dist < 15,
  1, 2
)

# change from numeric to factor and specify the labels
dat$dist_cat <- factor(dat$dist_cat,
  levels = c(1, 2),
  labels = c("small distance", "big distance") # follow the order of the levels
)

head(dat)

##   speed dist speed_dist  speed_cat speed_quali mean_score total_score
## 1     4    2          8  low speed      [0,12)   4.666667          14
## 2     4   10         40  low speed      [0,12)  18.000000          54
## 3     7    4         28  low speed      [0,12)  13.000000          39
## 4     7   22        154  low speed      [0,12)  61.000000         183
## 5     8   16        128 high speed      [0,12)  50.666667         152
## 6     9   10         90 high speed      [0,12)  36.333333         109
##         dist_cat
## 1 small distance
## 2 small distance
## 3 small distance
## 4   big distance
## 5   big distance
## 6 small distance

to check the format of a variable:

class(dat$dist_cat)

## [1] "factor"

# or
str(dat$dist_cat)

##  Factor w/ 2 levels "small distance",..: 1 1 1 2 2 1 2 2 2 2 ...

This will be sufficient if you need to format only a limited number of variables. However, if you need to do it for a large amount of categorical variables, it quickly becomes time consuming to write the same code many times. As you can imagine, it possible to format many variables without having to write the entire code for each variable one by one by using the within() command:

dat <- within(dat, {
  speed_cat <- factor(speed_cat, labels = c(
    "high speed",
    "low speed"
  ))
  dist_cat <- factor(dist_cat, labels = c(
    "small distance",
    "big distance"
  ))
})

head(dat)

##   speed dist speed_dist  speed_cat speed_quali mean_score total_score
## 1     4    2          8  low speed      [0,12)   4.666667          14
## 2     4   10         40  low speed      [0,12)  18.000000          54
## 3     7    4         28  low speed      [0,12)  13.000000          39
## 4     7   22        154  low speed      [0,12)  61.000000         183
## 5     8   16        128 high speed      [0,12)  50.666667         152
## 6     9   10         90 high speed      [0,12)  36.333333         109
##         dist_cat
## 1 small distance
## 2 small distance
## 3 small distance
## 4   big distance
## 5   big distance
## 6 small distance

str(dat)

## 'data.frame':	50 obs. of  8 variables:
##  $ speed      : num  4 4 7 7 8 9 10 10 10 11 ...
##  $ dist       : num  2 10 4 22 16 10 18 26 34 17 ...
##  $ speed_dist : num  8 40 28 154 128 90 180 260 340 187 ...
##  $ speed_cat  : Factor w/ 2 levels "high speed","low speed": 2 2 2 2 1 1 1 1 1 1 ...
##  $ speed_quali: Factor w/ 4 levels "[0,12)","[12,15)",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ mean_score : num  4.67 18 13 61 50.67 ...
##  $ total_score: num  14 54 39 183 152 109 208 296 384 215 ...
##  $ dist_cat   : Factor w/ 2 levels "small distance",..: 1 1 1 2 2 1 2 2 2 2 ...

Alternatively, if you want to transform several numeric variables into categorical variables without changing the labels, it is best to use the transform() function. We illustrate this function with the mpg data frame from the {ggplot2} package:

library(ggplot2)
mpg <- transform(mpg,
  cyl = factor(cyl),
  drv = factor(drv),
  fl = factor(fl),
  class = factor(class)
)

Recode categorical variables

It is possible to recode labels of a categorical variable if you are not satisfied with the current labels. In this example, we change the labels as follows:

“small distance” becomes “short distance”
“big distance” becomes “large distance”

dat$dist_cat <- recode(dat$dist_cat,
  "small distance" = "short distance",
  "big distance" = "large distance"
)

head(dat)

##   speed dist speed_dist  speed_cat speed_quali mean_score total_score
## 1     4    2          8  low speed      [0,12)   4.666667          14
## 2     4   10         40  low speed      [0,12)  18.000000          54
## 3     7    4         28  low speed      [0,12)  13.000000          39
## 4     7   22        154  low speed      [0,12)  61.000000         183
## 5     8   16        128 high speed      [0,12)  50.666667         152
## 6     9   10         90 high speed      [0,12)  36.333333         109
##         dist_cat
## 1 short distance
## 2 short distance
## 3 short distance
## 4 large distance
## 5 large distance
## 6 short distance

Change reference level

For some analyses, you might want to change the order of the levels. For example, if you are analyzing data about a control group and a treatment group, you may want to set the control group as the reference group. By default, levels are ordered by alphabetical order or by its numeric value if it was transformed from numeric to factor.

to check the current order of the levels (the first level being the reference):

levels(dat$dist_cat)

## [1] "short distance" "large distance"

In this case, “short distance” being the first level it is the reference level. It is the first level because it was initially set with a value equal to 1 when creating the variable.

to change the reference level:

dat$dist_cat <- relevel(dat$dist_cat, ref = "large distance")

levels(dat$dist_cat)

## [1] "large distance" "short distance"

Large distance is now the first and thus the reference level.

Rename variable names

To rename variable names as follows:

dist $\rightarrow$ distance
speed_dist $\rightarrow$ speed_distance
dist_cat $\rightarrow$ distance_cat

use the rename() command from the dplyr package:

dat <- rename(dat,
  distance = dist,
  speed_distance = speed_dist,
  distance_cat = dist_cat
)

names(dat) # display variable names

## [1] "speed"          "distance"       "speed_distance" "speed_cat"     
## [5] "speed_quali"    "mean_score"     "total_score"    "distance_cat"

Create a data frame manually

Although most analyses are performed on an imported data frame, it is also possible to create a data frame directly in R:

# Create the data frame named dat with 2 variables
dat <- data.frame(
  "variable1" = c(6, 12, NA, 3), # presence of 1 missing value (NA)
  "variable2" = c(3, 7, 9, 1)
)

# Print the data frame
dat

##   variable1 variable2
## 1         6         3
## 2        12         7
## 3        NA         9
## 4         3         1

Merging two data frames

By default, the merge is done on the common variables (variables that have the same name). However, if they do not have the same name, it is still possible to merge the two data frames by specifying their names:

dat1 <- data.frame(
  person = c(1:4),
  treatment = c("T1", "T2")
)

dat1

##   person treatment
## 1      1        T1
## 2      2        T2
## 3      3        T1
## 4      4        T2

dat2 <- data.frame(
  patient = c(1:4),
  age = c(56, 23, 32, 19),
  gender = c("M", "F", "F", "M")
)

dat2

##   patient age gender
## 1       1  56      M
## 2       2  23      F
## 3       3  32      F
## 4       4  19      M

We want to merge the two data frames by the subject number, but this number is referred as person in the first data frame and patient in the second data frame, so we need to indicate it:

merge(
  x = dat1, y = dat2,
  by.x = "person", by.y = "patient",
  all = TRUE
)

##   person treatment age gender
## 1      1        T1  56      M
## 2      2        T2  23      F
## 3      3        T1  32      F
## 4      4        T2  19      M

Add new observations from another data frame

In order to add new observations from another data frame, the two data frames need to have the same column names (but they can be in a different order):

dat1

##   person treatment
## 1      1        T1
## 2      2        T2
## 3      3        T1
## 4      4        T2

dat3 <- data.frame(
  person = 5:8,
  treatment = c("T3")
)

dat3

##   person treatment
## 1      5        T3
## 2      6        T3
## 3      7        T3
## 4      8        T3

rbind(dat1, dat3) # r stands for row, so we bind data frames by row

##   person treatment
## 1      1        T1
## 2      2        T2
## 3      3        T1
## 4      4        T2
## 5      5        T3
## 6      6        T3
## 7      7        T3
## 8      8        T3

As you can see, data for persons 5 to 8 have been added at the end of the data frame dat1 (because dat1 comes before dat3 in the rbind() function).

Add new variables from another data frame

It is also possible to add new variables to a data frame with the cbind() function. Unlike rbind(), column names do not have to be the same since they are added next to each other:

dat2

##   patient age gender
## 1       1  56      M
## 2       2  23      F
## 3       3  32      F
## 4       4  19      M

dat3

##   person treatment
## 1      5        T3
## 2      6        T3
## 3      7        T3
## 4      8        T3

cbind(dat2, dat3) # c stands for column, so we bind data frames by column

##   patient age gender person treatment
## 1       1  56      M      5        T3
## 2       2  23      F      6        T3
## 3       3  32      F      7        T3
## 4       4  19      M      8        T3

If you want to add only a specific variable from another data frame:

dat_cbind <- cbind(dat2, dat3$treatment)

dat_cbind

##   patient age gender dat3$treatment
## 1       1  56      M             T3
## 2       2  23      F             T3
## 3       3  32      F             T3
## 4       4  19      M             T3

names(dat_cbind)[4] <- "treatment"

dat_cbind

##   patient age gender treatment
## 1       1  56      M        T3
## 2       2  23      F        T3
## 3       3  32      F        T3
## 4       4  19      M        T3

or more simply with the data.frame() function:

data.frame(dat2,
  treatment = dat3$treatment
)

##   patient age gender treatment
## 1       1  56      M        T3
## 2       2  23      F        T3
## 3       3  32      F        T3
## 4       4  19      M        T3

Missing values

Missing values (represented by NA in RStudio, for “Not Applicable”) are often problematic for many analyses because many computations including a missing value has a missing value for result.

For instance, the mean of a series or variable with at least one NA will give a NA as a result. The data frame dat created in the previous section is used for this example:

dat

##   variable1 variable2
## 1         6         3
## 2        12         7
## 3        NA         9
## 4         3         1

mean(dat$variable1)

## [1] NA

The na.omit() function avoids the NA result, doing as if there was no missing value:

mean(na.omit(dat$variable1))

## [1] 7

Moreover, most basic functions include an argument to deal with missing values:

mean(dat$variable1, na.rm = TRUE)

## [1] 7

is.na() indicates if an element is a missing value or not:

is.na(dat)

##      variable1 variable2
## [1,]     FALSE     FALSE
## [2,]     FALSE     FALSE
## [3,]      TRUE     FALSE
## [4,]     FALSE     FALSE

Note that “NA” as a string is not considered as a missing value:

y <- c("NA", "2")

is.na(y)

## [1] FALSE FALSE

To check whether there is at least one missing value in a vector or data frame:

anyNA(dat$variable2) # check for NA in variable2

## [1] FALSE

anyNA(dat) # check for NA in the whole data frame

## [1] TRUE

# or
any(is.na(dat)) # check for NA in the whole data frame

## [1] TRUE

Nonetheless, data frames with NAs are still problematic for some types of analysis. Several alternatives exist to remove or impute missing values.

Remove NAs

A simple solution is to remove all observations (i.e., rows) containing at least one missing value. This is done by keeping only observations with complete cases:

dat_complete <- dat[complete.cases(dat), ]
dat_complete

##   variable1 variable2
## 1         6         3
## 2        12         7
## 4         3         1

Be careful when removing observations with missing values, especially if missing values are not “missing at random”. It is not because it is possible (and easy) to remove them, that you should do it in all cases. This is, however, beyond the scope of the present article.

Impute NAs

Instead of removing observations with at least one NA, it is possible to impute them, that is, replace them by some values such as the median or the mode of the variable. This can be done easily with the command impute() from the package Hmisc:

library(Hmisc)

# impute NA with default method (median/mode)
variable1 <- impute(dat$variable1)

# create data frame with imputed data
dat_imputed <- data.frame(variable1,
  variable2 = dat$variable2
)

dat_imputed

##   variable1 variable2
## 1         6         3
## 2        12         7
## 3         6         9
## 4         3         1

When the median/mode method is used (the default), character vectors and factors are imputed with the mode. Numeric and integer vectors are imputed with the median. Again, use imputations carefully. Other packages offer more advanced imputation techniques. However, we keep it simple and straightforward for this article as advanced imputations is beyond the scope of introductory data manipulations in R.

Scale

Scaling (also referred as standardizing) a variable is often used before a Principal Component Analysis (PCA)¹ when variables of a data frame have different units. Remember that scaling a variable means that it will compute the mean and the standard deviation of that variable. Then each value (so each row) of that variable is “scaled” by subtracting the mean and dividing by the standard deviation of that variable. Formally:

\[z = \frac{x - \bar{x}}{s}\]

where $\bar{x}$ and $s$ are the mean and the standard deviation of the variable, respectively.

To scale one or more variables in R use scale():

dat_scaled <- scale(dat_imputed)

head(dat_scaled)

##    variable1  variable2
## 1 -0.1986799 -0.5477226
## 2  1.3907590  0.5477226
## 3 -0.1986799  1.0954451
## 4 -0.9933993 -1.0954451

Dates and times

Dates

In R the default date format follows the rules of the ISO 8601 international standard which expresses a day as “2001-02-13” (yyyy-mm-dd).²

Date can be defined by a string of characters or a number. For example, October 1st, 2016:

as.Date("01/10/16", format = "%d/%m/%y")

## [1] "2016-10-01"

as.Date(274, origin = "2016-01-01") # there are 274 days between the origin and October 1st, 2016

## [1] "2016-10-01"

Times

An example with date and time vectors:

dates <- c("02/27/92", "02/27/99", "01/14/92")
times <- c("23:03:20", "22:29:56", "01:03:30")

x <- paste(dates, times)
x

## [1] "02/27/92 23:03:20" "02/27/99 22:29:56" "01/14/92 01:03:30"

strptime(x,
  format = "%m/%d/%y %H:%M:%S"
)

## [1] "1992-02-27 23:03:20 CET" "1999-02-27 22:29:56 CET"
## [3] "1992-01-14 01:03:30 CET"

Find more information on how to express a date and time format with help(strptime).

Extraction from dates

We can extract:

weekdays
months
quarters
years

y <- strptime(x,
  format = "%m/%d/%y %H:%M:%S"
)

y

## [1] "1992-02-27 23:03:20 CET" "1999-02-27 22:29:56 CET"
## [3] "1992-01-14 01:03:30 CET"

weekdays(y, abbreviate = FALSE)

## [1] "Thursday" "Saturday" "Tuesday"

months(y, abbreviate = FALSE)

## [1] "February" "February" "January"

quarters(y, abbreviate = FALSE)

## [1] "Q1" "Q1" "Q1"

format(y, "%Y") # 4-digit year

## [1] "1992" "1999" "1992"

format(y, "%y") # 2-digit year

## [1] "92" "99" "92"

Exporting and saving

If a copy-paste is not sufficient, you can save an object in R format with save():

save(dat, file = "dat.Rdata")

or using write.table(), write.csv() or write.xlsx():

# in a text format
write.table(dat, "dat.txt", row = FALSE, sep = "\t", quote = FALSE)

# in csv
write.csv(dat, file = "dat.csv", row.names = FALSE, quote = FALSE)

# in excel
# install.packages("openxlsx")
library(openxlsx)
write.xlsx(dat, file = "dat.xlsx")

If you need to send every results into a file instead of the console:

sink("filename")

(Don’t forget to stop it with sink().)

Looking for help

You can always find some help about:

a function: ?function or help(function)
a package: help(package = packagename)
a concept: help.search("concept") or apropos("concept")

Otherwise, Google is your best friend!

Conclusion

Thanks for reading.

I hope this article helped you to manipulate your data in RStudio. For those who are interested in going further, see also an introduction to data manipulation in R with the {dplyr} package.

Now that you know how to import a data frame into R and how to manipulate it, the next step would probably be to learn how to perform descriptive statistics in R. If you are looking for more advanced statistical analyses using R, see all articles about R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Principal Component Analysis (PCA) is a useful technique for exploratory data analysis, allowing a better visualization of the variation present in a data frame with a large number of variables. When there are many variables, the data cannot easily be illustrated in their raw format. To counter this, the PCA takes a data frame with many variables and simplifies it by transforming the original variables into a smaller number of “principal components”. The first dimension contains the most variance in the data frame and so on, and the dimensions are uncorrelated. Note that PCA is done on quantitative variables.↩︎
For your information, note that this date format is not the same for every software! Excel, for instance, uses a different format.↩︎

How to import an Excel file in RStudio?

Wed, 18 Dec 2019 00:00:00 +0000

Introduction

As we have seen in this article on how to install R and RStudio, R is useful for many kind of computational tasks and statistical analyses. However, it would not be so powerful and useful without the possibility to import datasets into R. As you will most likely use R with your own data, being able to import it into R is crucial for any user.

In this article I present two different ways to import an Excel file; (i) via the text editor and (ii) in a more “user-friendly” way. I also discuss about the main advantages and disadvantages of both methods. Note that:

How to import a dataset often depends on the format of the file (Excel, CSV, text, SPSS, Stata, etc.). I focus here only on Excel files as it is the most common type of file for a dataset
There are several other ways to import an Excel file (probably even some I am not aware of), but I present the two most simple yet robust ways to import such files
No matter what type of file and how you import it, there is one gold standard regarding how datasets are structured: columns correspond to variables, rows correspond to observations (in the broad sense of the term) and each value must have its own cell (known as tidy format):

Structure of a dataset. Source: R for Data Science by Hadley Wickham & Garrett Grolemund

Transform an Excel file to a CSV file

Before dealing with the importation, the first thing is to change the format of your Excel file to a CSV format.¹ CSV format is the standard when working with datasets and programming languages as it is a more robust format compared to Excel.

If your file is already in the CSV format (with the extension .csv), you can skip this section. If the file is not in the CSV format (for example the extension is .xlsx) you can easily transform it to CSV by following these steps:

Open your Excel file
Click on File > Save as
Choose the format .csv
Click on Save

Check that your file finishes with the extension .csv. If that is the case, your file is now ready to be imported. But first, let me introduce an important concept when importing datasets into RStudio, the working directory.

R working directory

Although programming languages may be very powerful, it often needs our help and importing a dataset is not an exception. Indeed, before importing your data, you must tell RStudio where your file is located (so let RStudio know in which folder to look for your dataset). But before this, let me introduce the working directory. The working directory is the location (in your computer) of where RStudio is currently working (in fact RStudio is not working across your entire computer; it is working inside one folder of your computer). Concerning this working directory, there are two functions that we will need:

getwd() (wd stands for working directory)
setwd()

Get working directory

In most cases, when you open RStudio, the working directory (so where it is currently working) is different than where your dataset is located. To know what is the working directory RStudio is currently using, run getwd(). On MacOS, this function will most likely render a location such as "/Users/yourname/", while on Windows it will most likely render "c:/Documents/". Do not worry if your working directory is different, the most important is to set the working directory correctly (so where your file is located) and not where it is now.

Set working directory

As mentioned earlier, your dataset is most likely located in a different location than your working directory. Without any action from you, RStudio will never be able to import your file as it is not looking in the correct folder (you will encounter the following error in the console: cannot open file ‘data.csv’: No such file or directory). Now, in order to specify the correct location of your file (that is, to tell RStudio in which folder it should look for your dataset), you have three options:

the user-friendly method
via the console
via the text editor (see below why it is my preferred option)

User-friendly method

To set the correct folder, so to set the working directory equal to the folder where your file is located, follow these steps:

In the lower right pane of RStudio, click on the tab “Files”
Click on “Home” next to the house icon
Go to the folder where your dataset is located
Click on “More”
Click on “Set As Working Directory”

Set working directory in RStudio (user-friendly method)

Alternatively, you can also set the working directory by clicking on Session > Set Working Directory > Choose Directory…

Set working directory in RStudio (user-friendly method)

As you can see in the console, any of the two methods will actually execute the code setwd() with the path to the folder you specified. So by clicking on the buttons you actually asked RStudio to write a line of code for you. This method has the advantage that you do not need to remember the code and that you will not make a mistake in the name of the path to your folder. The disadvantage is that if you leave RStudio and open it again later, you will have to specify the working directory again as RStudio did not save your actions via the buttons.

Via the console

You can specify the working directory by running setwd(path/to/folder) directly in the console, with path/to/folder being the path to the folder containing your dataset. However, you will need to run the command again when reopening RStudio.

Via the text editor

This method is actually a combination of the two above:

Set the working directory by following the exact same steps than for the user-friendly method (via the buttons)
Copy the code executed in the console and paste it in the text editor (i.e., your script)

I recommend this method for several reasons. First, you do not need to remember the setwd() function. Second, you will not make typos in the path of your folder (path which can sometimes be quite long if you have folders inside folders). Third, when saving your script (which I assume you do otherwise you would lose all your work), you also save the actions you just made via the buttons. So when you reopen your script in the future, no matter what is the current directory, by executing your script (which now include the line of code for setting the working directory), you will at the same time specify the working directory you selected for this project.

Import your dataset

Now that you have transformed your Excel file into a CSV file and you have specified the folder containing your data by setting the working directory, you are now ready to actually import your dataset. Remind that there are two methods to import a file:

in a user-friendly way
via the text editor (see also below why it is my preferred option)

No matter which method you choose, it is a good practice to first open your file in TextEdit (on Mac) or Notepad (on Windows) in order to see the raw data. If you open the file in Excel you will see the data already formatted and thus miss some important information needed for the importation. Below an example of raw data:

Example of raw data

There are a few things we need to look for in order to properly import our dataset:

Are the variables names present?
How are the values separated? Comma, semicolon, whitespace, tab?
Is the decimal a point or a comma?
How are specified missing values? Empty cells, NA, null, O, other?

User-friendly way

As shown below, simply click on the file > Import Dataset…

Import dataset in RStudio

A window which looks like this will open:

Import window in RStudio

From this window, you can have a preview of your data, and more importantly, check whether your data seems to have been imported correctly. If your data have been correctly imported, you can click on “Import”. If this is not the case, you can change the import options at the bottom of the window (below the data preview) corresponding to the information you gathered when looking at the raw data. Below, the import options you will most likely use:

Name: set the name of your data set (default is the name of the file). Avoid special characters and long names (as you will have to type the name of your dataset several times). I personally rename my datasets with a generic name such as “dat”, others use “df” (for dataframe), “data”, or even “my_data”. You could use more explicit names such as “tennis_data” if you are using data on tennis matches for example. However, the main drawback with using specific names for datasets is that if, for instance, you want to reuse the code you created while analysing tennis data on other datasets, you will need to edit your code by replacing all occurrences of “tennis_data” by the name of your new dataset
Skip: specify the number of top rows you want to skip (default is 0). Most of the time, 0 is fine. However, if your file contains some blank rows at the top (or information you want to disregard), set the number of rows to skip
First Row as Names: specify whether the variables names are present or not (default is that variables names are present)
Delimiter: the character which separate the values. From our raw data above, you can see that the delimiter is a comma (“,”). Change it to semicolon if your values are separated by “;”
NA: how missing values are specified (default is empty cells). From our raw data above, you can see that missing values are simply empty cells, so leave NA to default or change it to “empty”. Change this option if missing values in your raw data are coded as “NA” or “0” (tip: do not code yourself missing values as “0”, otherwise you will not be able to distinguish the true zero values and the missing values)

After changing the import options corresponding to your data, click on “Import”. You should now see your dataset in a new window and from there you can start analyzing your data.

This user-friendly method has the advantage that you do not need to remember the code (see the next section for the entire code). However, the main drawback is that your import options will not be saved for a future usage so you will need to import your dataset manually each time you open RStudio.

Via the text editor

Similarly to setting the working directory, I also recommend using the text editor instead of the user-friendly method for the simple reason that you can save your import options when using the text editor (and not when using the user-friendly method). Saving your import options in your script (thanks to a line of code) allows you to quickly import your dataset the exact same way without having to repeat all the necessary steps every time you import your dataset. The command to import a CSV file is read.csv() (or read.csv2() which is equivalent but with other default import options). Here is an example with the same file than in the user-friendly method:

dat <- read.csv(
  file = "data.csv",
  header = TRUE,
  sep = ",",
  dec = ".",
  stringsAsFactors = TRUE
)

dat <-: name of the dataset in RStudio. This means that after importation, I will need to refer to the dataset by calling dat
file =: name of the file in the working directory. Do not forget “” around the name, the extension .csv at the end and the fact that RStudio is case sensitive ("Data.csv" will give an error) and space sensitive inside “” ("data .csv" will also throw an error). In our case the file is named “data.csv” so file = "data.csv"
header =: are variables names present? The default is TRUE, change it to FALSE if it is not the case in your dataset (TRUE and FALSE are always in capital letters, true will not work!)
sep =: separator. Equivalent to delimiter in the user-friendly method. Do not forget the ““. In our dataset the separator of the values is a comma so sep = ","
dec =: decimal. Do not forget the ““. In our dataset, the decimal for the numeric values is a point, so dec = "."
stringsAsFactors =: should character vectors be converted to factors? The default option used to be TRUE, but since R version 4.0.0 it is FALSE by default. If all your character vectors are actually qualitative variables (so factors in R), set it to TRUE
I do not write that missing values are coded as empty cells in my dataset because it is the default
Last but not least, do not forget that the arguments are separated by a comma

Other arguments exist, run ?read.csv to see all of them.

After the importation you can check whether your data have been correctly imported by running View(dat) where dat is the name you chose for your data. A window, similar than for the user-friendly method, will display your data. Alternatively you can also run head(dat) to see the first 6 rows and check that it corresponds to your Excel file. If something is not correct, edit the import options and check again. If your dataset has been correctly imported, you can now start analyzing your data. See other articles on R if you want to learn how.

The advantage of importing your dataset directly via the code in the text editor is that your import options will be saved for a future usage, preventing you from importing it manually every time you open your script. You will, however, need to remember the function read.csv() (not the arguments since you can always check them in the help documentation).

Import SPSS (.sav) files

Only Excel files are covered in details here. However, SPSS files (.sav) can also be read in R by using the following command:

library(foreign)
dat <- read.spss(
  file = "filename.sav",
  use.value.labels = TRUE,
  to.data.frame = TRUE
)

The read.spss() function outputs a data table which retrieves all the characteristics of the .sav file, including the names given for the different levels of the categorical variables and the characteristics of the variables. If you need more information about this command, see the help documentation (library(foreign) then ?read.spss).

Conclusion

Thanks for reading.

I hope this article helped you to import an Excel file in RStudio. Now that your dataset is correctly imported, learn how to manipulate it or how to perform descriptive statistics in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

I am aware that it is possible to import an Excel directly into R without converting it in a CSV file, with the read_excel() function from the {readxl} package for instance. However, CSV format is the standard and more importantly, importing a CSV does not require to install and load a package (which is sometimes confusing for beginners).↩︎

How to install R and RStudio?

Tue, 17 Dec 2019 00:00:00 +0000

Note that this article is inspired from the lecture notes of Prof. Johan Segers and my personal notes as teaching assistant for his course entitled “Multivariate statistical analysis” given at UCLouvain.

What is R and RStudio?

R

The statistical program R is nothing more than a programming language, mainly used for data manipulation and to perform statistical analyses. At the time of writing, this language is (one of) the leading program in statistics, although not the only programming language used by statisticians.

In order to use R, we need two things:

a text editor in which to write our code
a place to run this code

RStudio

This is where RStudio comes handy.

RStudio is an integrated development environment (IDE) for R. R and RStudio work together. R is a program that runs all your code, and RStudio is another program that allows you to control R in a more comfortable and friendly way. RStudio has the advantage of offering both a powerful text editor for writing your code and a place to run the code written in this editor.

For these reasons, I highly recommend using RStudio instead of R. I use RStudio (and not R) on a daily basis and you will see that all articles on this blog is written in RStudio.

Note that RStudio requires the prior installation of the R software provided by CRAN in order to be able to function properly. Just installing RStudio on your personal computer is not enough. See the next section on how to install both.

How to install R and RStudio?

You can download R at https://cran.r-project.org/mirrors.html. Select the CRAN mirror site closest to your country. If there are more than one links for your country, simply select one:

Select the CRAN mirror site closest to your country

Then in the box labeled “Download and Install R” (located at the top), click on the link corresponding to your operating system (Windows, Mac or Linux):

Click on the link corresponding to your operating system

Now that R is installed on your computer, you can download RStudio. You can download the free version of RStudio (which is totally enough for most users, including me!) on their website.

The main components of RStudio

Now that both programs are installed on your computer, let’s dive into the main components of RStudio.

By default, the RStudio window has three panes:

The console (red pane)
The environment (green pane)
Files, plots, help, etc. (blue pane)

RStudio window

The console (red pane) is where you can execute your code (more information on the red and blue panes later). By default, the text editor does not open automatically. To open it, click on File > New File > R Script or click on the button representing a white sheet marked with a small green cross in the upper left corner, then on R Script:

New R script in RStudio

A new pane (in orange below), also known as the text editor, opens in which you will be able to write your code. The code will be executed and the results displayed in the console (red pane).

Write and execute your code in RStudio

Note that you can also write code in the console (red pane). However, I strongly recommend writing your code in the text editor (orange pane) because you can save the code written in the text editor (and thus execute it again later), while you cannot save the code written in the console.

To execute code written in the text editor (orange pane), you have two options:

Type your code and then press the “Run” button (see below) or use the keyboard shortcut CTRL + Enter (cmd + Enter on Mac). Only the chunk of code where your cursor is located will then be executed.
Type your code and select in the text editor the part you want to execute and then press the “Run” button or use the keyboard shortcut CTRL + Enter (cmd + Enter on Mac). All the selected code will be executed

Run code in RStudio

For example, try typing 1+1 in the text editor and execute it by clicking on “Run” (or CTRL/cmd + Enter). You should see the result 2 in the console, as shown in the screenshot below:

Example of code executed in RStudio

The text editor and the console are the panes you will use most often. The two other panes (the blue and green panes introduced earlier) will however still be very useful when using RStudio.

The environment (green pane) displays all values stored by RStudio. For example, if you type and execute the code a = 1, RStudio will store the value 1 for a, as shown in the screenshot below:

Environment pane in RStudio

This means that you can now perform any computations with a, such that if you execute a + 1, RStudio will render 2 in the console. In this pane you can also see a tab with a history of the code executed and a button to import a dataset (more on importing a dataset in RStudio).

The last pane (blue) is where you will find everything else such as your files, the plots, the packages, the help documentation, etc. I discuss about the Files tab in more detail here so let’s discuss about the other tabs:

Plot: where you will see the rendered plots. For instance, run plot(1:10) and you should see it in this tab. If you plotted more than one plots, you can navigate between them by clicking on the arrows. You can open the plot in a new window by clicking on Zoom and export your plot by clicking on Export. Those buttons are located just under the Plot tab (see figure below)
Packages: where you see all your installed packages. Only fundamental functionalities come with R. Everything else must be installed from packages. Remind that R is open source; everyone can write code and publish it as a package. You are then able to use this package (and all functions built inside this package) for free. Some packages are installed by default, all others must be installed by running install.packages("name of the package") (do not forget "" around the name of the package!). Once the package is installed, you must load the package and only after it has been loaded you can use all the functions it contains. To load a package, run library(name of the package) (this time "" around the name of the package are optional, but can still be used if you wish). You also have the possibility to install and load packages via the buttons under the Packages tab. For this, click on the button Install under Packages, type the name of the package you want to install and then click on Install. You will see that the code appears in the console. To load the package, find the package you want to load in the Packages window (you can use the search box), then click on the checkbox next to the name of the package. Again, the code is run in the console. See the figures below if needed. Note that you will need to install packages only once,¹ but load packages each time you open RStudio. Furthermore, note that an internet connection is required to install a package, while it is not required to load a package
Help: documentation about all functions written for R. To access the help of a function, run help("name of the function") or simply ?name of the function. For example, to see the help about the mean function, run ?mean. You can also press F1 while having your cursor on a function

Plot buttons in RStudio

Step 1: click on the Install button

Step 2: type the name of the package and click on Install

Step 3: load your package by clicking on the box next to the package’s name

Examples of code

Now that you have installed R and RStudio and you know its main components, below are some examples of basic code.

More advanced code and analyses are presented in other articles about R, and in particular in this article about data manipulation in R.

Calculator

Compute $5*5$

5 * 5

## [1] 25

Compute $\frac{1}{\sqrt{50\pi}}\, e^{-\frac{(10 - 11)^2}{50}}$

1 / sqrt(50 * pi) * exp(-(10 - 11)^2 / 50)

## [1] 0.07820854

As you can see, some values like $\pi$ are stored by default so you do not need to specify its value. Note that RStudio is case sensitive, but not space sensitive. This means that pi is different than Pi but 5*5 gives the same result than 5 * 5.

Comments

To add comments in your code, use # before the code:

# A comment
# Another comment
1 + 1

## [1] 2

Store and print values

Note that in order to store a value inside an object, using = or <- is equivalent. I however recommend using <- to follow the guidelines of R programming. You can name your objects (A and B in our case) as you like. However, it is recommended to use short and concise names (as you will most likely type them several times) and avoid special characters.

A <- 5
B <- 6

When storing values, RStudio does not display it on the console. To store a value AND print it in the console, use:

(A <- 5)

## [1] 5

or:

A <- 5
A

## [1] 5

Vectors

It is also possible to store more than one value inside an object via the function c() (c stands for combine).

A <- c(1 / 2, -1, 0)
A

## [1]  0.5 -1.0  0.0

Matrices

Or create a matrix via matrix():

my_mat <- matrix(c(-1, 2, 0, 3), ncol = 2, nrow = 2)
my_mat

##      [,1] [,2]
## [1,]   -1    0
## [2,]    2    3

You can access the help of this function via ?matrix or help("matrix"). Note that inside a function, you can have multiple arguments separated by a comma. Inside matrix(), the first argument is the vector c(-1, 2, 0, 3), the second is ncol = 2 and the third is nrow = 2. For all functions in RStudio, you can specify an argument by its order inside the function or by the name of the argument. If you specify the name of the argument, the order does not matter anymore, so matrix(c(-1, 2, 0, 3), ncol = 2, nrow = 2) is equivalent to matrix(c(-1, 2, 0, 3), nrow = 2, ncol = 2):

my_mat2 <- matrix(c(-1, 2, 0, 3), nrow = 2, ncol = 2)
my_mat2

##      [,1] [,2]
## [1,]   -1    0
## [2,]    2    3

my_mat == my_mat2 # is my_mat equal to my_mat2?

##      [,1] [,2]
## [1,] TRUE TRUE
## [2,] TRUE TRUE

Generate random values

To generate 10 values based on a normal distribution with mean $\mu = 400$ and standard deviation $\sigma = 10$:

my_vec <- rnorm(10, mean = 400, sd = 10)
# Display only the first 5 values:
head(my_vec, 5)

## [1] 413.7096 394.3530 403.6313 406.3286 404.0427

# Display only the last 5 values:
tail(my_vec, 5)

## [1] 398.9388 415.1152 399.0534 420.1842 399.3729

You will have different values than mine due to the fact that they are randomly generated. If you want to make sure to have always the same random values, use set.seed() (with any numeric inside the brackets). For instance, with the following code, you should have the exact same values, no matter where and when you run it:

set.seed(42)
rnorm(3, mean = 10, sd = 2)

## [1] 12.741917  8.870604 10.726257

Plot

plot(my_vec,
  type = "l", # "l" stands for line
  main = "Plot title",
  ylab = "Y-axis label",
  xlab = "X-axis label"
)

Conclusion

Thanks for reading.

I hope this article helped you to install R and RStudio.

This is only a very limited introduction to the possibilities of RStudio. If you want to learn more, I recommend that you read other articles related to R, starting with how to import an Excel file or how to manipulate a dataset in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Actually you will need to reinstall your packages for each new R update. However, if you work on the same R version, you need to install your packages only once but load them everytime you open RStudio.↩︎

	Population	Sample
Standard deviation	\(\sigma\)	\(s\)
Variance	\(\sigma^2\)	\(s^2\)