Hypothesis test on Stats and R

McNemar's test in R

Wed, 19 Aug 2026 00:00:00 +0000

Introduction

In a previous article, we showed how to perform the Chi-square test of independence in R in order to test whether two qualitative variables are related. As mentioned in that article (and in the one showing how to do the Chi-square test of independence by hand), this test requires that observations are independent. When observations are dependent, that is, when the two measurements are collected on the same subjects (paired samples), the McNemar’s or Cochran’s Q tests should be used instead.

This article is dedicated to the first one: the McNemar’s test. It is used to compare two related (paired) proportions measured on a qualitative variable with only two possible levels. In practice, it is mostly used when the same subjects are measured twice (typically before and after an intervention), or when two raters or two conditions are applied to the same subjects.

In a way, the McNemar’s test is to two paired proportions what the paired Student’s t-test is to two paired means: in both cases we take advantage of the fact that the two measurements belong to the same individuals, the difference being that here the variable of interest is binary instead of quantitative.

Note also that the McNemar’s test is limited to exactly two related measurements. If you have more than two (for example, the same question asked at three different time points), the appropriate extension is the Cochran’s Q test, of which the McNemar’s test is the special case for two measurements. If you are unsure about which test is appropriate for your own data, see this overview of the most common statistical tests.

In the remaining of the article, we present the data used for the illustration, the aim, hypotheses and assumptions of the test, and finally how to perform it in R and how to interpret its results.

Data

A dataset with a paired binary structure is not so easy to find among the datasets shipped with R, so we simulate our own data for this article.

Suppose that we ask 200 randomly selected citizens whether they are in favor of a new policy in their city (answer “Yes” or “No”), that we then have them watch a public debate on this policy, and that we ask them exactly the same question again right after the debate:

# number of respondents
n <- 200

# opinion before the debate
before <- sample(c("Yes", "No"),
  size = n,
  replace = TRUE,
  prob = c(0.4, 0.6)
)

# opinion after the debate (respondents who were in favor
# tend to keep their opinion, while those who were against
# are more likely to change their mind)
after <- ifelse(before == "Yes",
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.9, 0.1)),
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.4, 0.6))
)

# dataset
dat <- data.frame(
  respondent = 1:n,
  before = factor(before, levels = c("Yes", "No")),
  after = factor(after, levels = c("Yes", "No"))
)

head(dat)

##   respondent before after
## 1          1    Yes   Yes
## 2          2    Yes   Yes
## 3          3     No   Yes
## 4          4    Yes   Yes
## 5          5    Yes   Yes
## 6          6     No    No

(Note that a seed has been set in the background with set.seed(42), so the simulated data and all results below are reproducible.)

Each row corresponds to one respondent and contains two measurements of the same binary variable: the opinion before and the opinion after the debate. The two samples are thus paired, since the two answers on a given row belong to the same person.

As always, it is a good practice to start with some descriptive statistics. Here, the proportion of respondents in favor of the policy at each of the two time points:

# install.packages("dplyr")
library(dplyr)

dat %>%
  summarise(
    prop_before = mean(before == "Yes"),
    prop_after = mean(after == "Yes")
  )

##   prop_before prop_after
## 1        0.46       0.61

In our sample, the proportion of respondents in favor of the policy went from 46% before the debate to 61% after the debate.

These two proportions are computed on the same people, so comparing them as if they came from two independent groups would ignore the pairing. What matters for the McNemar’s test is the way each respondent moved (or did not move) from one answer to the other, and this information is contained in the 2 $\times$ 2 contingency table of the paired answers:

tab <- table(dat$before, dat$after,
  dnn = c("Before", "After")
)

tab

##       After
## Before Yes No
##    Yes  81 11
##    No   41 67

This table must be read pair by pair, and not cell by cell as we usually do:

the two cells on the diagonal are the concordant pairs: 81 respondents answered “Yes” twice and 67 answered “No” twice, so these 148 respondents did not change their mind,
the two cells outside the diagonal are the discordant pairs: 11 respondents were in favor before the debate but against after, while 41 were against before but in favor after.

Only the discordant pairs carry information about a change of opinion (a respondent who gave twice the same answer tells us nothing about the effect of the debate), and this is precisely what the McNemar’s test is built on.

The same information can be visualized with a simple barplot of the paired counts:

# install.packages("ggplot2")
library(ggplot2)

ggplot(dat) +
  aes(x = before, fill = after) +
  geom_bar(position = "dodge") +
  labs(
    x = "Opinion before the debate",
    y = "Number of respondents",
    fill = "Opinion after the debate"
  )

From the table and the plot, we see that the changes of opinion do not balance out: many more respondents switched from “No” to “Yes” than the opposite. The question is now whether this imbalance is large enough to be declared significant, or whether it could reasonably be explained by chance alone (that is, by sampling fluctuations).

McNemar’s test

Aim and hypotheses

The McNemar’s test is used to compare two related proportions, so it allows to determine whether the proportion of subjects belonging to a given category changed between two dependent measurements.

The null and alternative hypotheses of the McNemar’s test are:

$H_0$: the two related proportions are equal (marginal homogeneity, that is, there is no systematic change between the two measurements)
$H_1$: the two related proportions are different (there is a significant change between the two measurements)

Since concordant pairs bring no information about a change, the test is based only on the two discordant cells. Denoting by $b$ the number of subjects who answered “Yes” then “No”, and by $c$ the number of subjects who answered “No” then “Yes”, the hypotheses can equivalently be written as:

$H_0: p_b = p_c$
$H_1: p_b \ne p_c$

where $p_b$ and $p_c$ are the probabilities of the two possible types of change. Under the null hypothesis, a change in one direction is as likely as a change in the other direction, so the test statistic

\[\chi^2 = \frac{(b - c)^2}{b + c}\]

follows a Chi-square distribution with 1 degree of freedom. By default, R applies a continuity correction (see more on this below), which replaces the numerator by $(|b - c| - 1)^2$.

In the context of our example, the McNemar’s test helps us to answer the following question: “Did the public debate significantly change the proportion of citizens in favor of the new policy?”.

Rejecting $H_0$ would mean that the proportion of citizens in favor of the policy is significantly different before and after the debate, so that the changes of opinion observed in our sample are unlikely to be due to chance only. On the contrary, not rejecting $H_0$ would mean that we do not have enough evidence to conclude that opinions changed: the switches observed in the two directions would then be compatible with random fluctuations.

Note that, as for many tests, the McNemar’s test does not indicate the direction of the change. The direction must be read from the contingency table or from the marginal proportions computed in the previous section.

Assumptions

For the results of the McNemar’s test to be valid, the following assumptions must be met:

Paired measurements on a binary variable. The two measurements must be collected on the same subjects, or on matched pairs (twins, or patients matched on age and sex for instance), and the variable of interest must be qualitative with exactly two levels (“Yes”/“No”, success/failure, present/absent, etc.). If the two samples are independent instead of paired, use the Chi-square test of independence.
Data organized in a 2 $\times$ 2 contingency table of the paired outcomes. Each subject contributes to one and only one cell of the table, so the sum of the four cells equals the number of subjects (200 in our case), and not twice this number.
Pairs are independent of each other. Within a pair, the two measurements are of course dependent, and this is precisely the reason why we use this test. Between pairs, however, independence is required: one subject’s answers must not influence another subject’s answers. As for many statistical tests, this assumption is usually verified based on the design of the experiment rather than via a formal test. A random and representative sample of the population of interest is generally sufficient. In our example, respondents have been selected at random and answered the question individually, so we consider this assumption as met.
Enough discordant pairs. The $p$-value returned by mcnemar.test() is based on a Chi-square approximation, which is reliable only if the number of discordant pairs is large enough. A common rule of thumb is that $b + c$ should be at least 25. In our sample, $b + c$ = 52, so the approximation can be used safely.

When the number of discordant pairs is small, it is preferable to use the exact version of the test, which is based on a binomial distribution instead of the Chi-square approximation. It boils down to testing whether, among the discordant pairs, changes in one direction are as frequent as changes in the other direction, so it can be performed in base R with the binom.test() function:

# exact version of the McNemar's test
binom.test(tab[1, 2], tab[1, 2] + tab[2, 1], p = 0.5)

## 
## 	Exact binomial test
## 
## data:  tab[1, 2] and tab[1, 2] + tab[2, 1]
## number of successes = 11, number of trials = 52, p-value = 3.589e-05
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.1106115 0.3470376
## sample estimates:
## probability of success 
##              0.2115385

Note that the {exact2x2} package also provides a dedicated mcnemar.exact() function, which returns the same $p$-value together with a confidence interval for the odds ratio.

In R

The McNemar’s test can be performed in R with the mcnemar.test() function, applied on the contingency table of the paired outcomes:

mcnemar.test(tab)

## 
## 	McNemar's Chi-squared test with continuity correction
## 
## data:  tab
## McNemar's chi-squared = 16.173, df = 1, p-value = 5.781e-05

The test can also be applied directly on the two variables, without building the contingency table first (results are of course identical):

mcnemar.test(dat$before, dat$after)

## 
## 	McNemar's Chi-squared test with continuity correction
## 
## data:  dat$before and dat$after
## McNemar's chi-squared = 16.173, df = 1, p-value = 5.781e-05

The output shows:

the title of the test, together with the mention that a continuity correction has been applied,
the data which have been used,
the test statistic (McNemar's chi-squared),
the degrees of freedom (always equal to 1 for a 2 $\times$ 2 table) and
the $p$-value.

As mentioned above, R applies a continuity correction by default. This correction makes the test slightly more conservative (that is, it gives a larger $p$-value), and it can be removed thanks to the correct = FALSE argument:

mcnemar.test(tab, correct = FALSE)

## 
## 	McNemar's Chi-squared test
## 
## data:  tab
## McNemar's chi-squared = 17.308, df = 1, p-value = 3.179e-05

With 52 discordant pairs, both versions lead to the same conclusion. The correction really matters only when the number of discordant pairs is small, and in that case the exact version presented in the previous section is a better option anyway.

It is the $p$-value which is of interest to conclude the test. If you are not familiar with $p$-values, I invite you to read this section.

Interpretations

Based on the McNemar’s test, we reject the null hypothesis and we conclude that the proportion of citizens in favor of the new policy is significantly different before and after the debate ($p$-value < 0.001).

$\Rightarrow$ In our context, rejecting the null hypothesis means that the debate is associated with a significant change of opinion. Looking at the direction of this change, the proportion of citizens in favor of the policy increased from 46% before the debate to 61% after the debate.

(For the sake of illustration, if the $p$-value had been larger than the significance level $\alpha = 0.05$: we could not have rejected the null hypothesis, so we could not have concluded that the proportion of citizens in favor of the policy was different before and after the debate.)

Contrary to the tests comparing three groups or more, no post-hoc test is required after a significant McNemar’s test: only two related measurements are compared, so a significant result already tells us which two proportions differ. Post-hoc comparisons become relevant again with more than two related measurements, in which case you should turn to the Cochran’s Q test.

Summary

In this article, we reviewed the aim and the hypotheses of the McNemar’s test, which is used to compare two related proportions measured on the same subjects, together with its underlying assumptions (paired measurements on a binary variable, independence between pairs and a sufficient number of discordant pairs). We then showed how to perform it in R with the mcnemar.test() function, applied either on the 2 $\times$ 2 contingency table of the paired answers or directly on the two variables, and how to interpret its results by comparing the $p$-value with the significance level $\alpha$. Remember that it is the special case of the Cochran’s Q test for exactly two related measurements, and that with independent samples the Chi-square test of independence should be preferred.

Thanks for reading.

I hope this article helped you to understand the McNemar’s test and how to perform it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

How to: one-way ANOVA by hand

Wed, 30 Aug 2023 00:00:00 +0000

Introduction

An ANOVA is a statistical test used to compare a quantitative variable between groups, to determine if there is a statistically significant difference between several population means. In practice, it is usually used to compare three or more groups. However, in theory, it can also be done with only two groups.¹

In a previous post, we showed how to perform a one-way ANOVA in R. In this post, we illustrate how to conduct a one-way ANOVA by hand, via what is usually called an “ANOVA table”.

Data and hypotheses

To illustrate the method, suppose we take a sample of 12 students, divided equally into three classes (A, B and C) and we observe their age. Here is the sample:

We are interested in comparing the population means between classes.

Remember that the null hypothesis of the ANOVA is that all means are equal (i.e., age is not significantly different between classes), whereas the alternative hypothesis is that at least one mean is different from the other two (i.e., age is significantly different in at least one class compared to the other two). Formally, we have:

$\mu_A = \mu_B = \mu_C$
at least one mean is different

ANOVA by hand

As mentioned above, we are going to do an ANOVA table to conclude the test.

Note that the ANOVA requires some assumptions (i.e., independence, equality of variances and normality). The aim of this post is to illustrate how to do an ANOVA by hand and not how to verify these assumptions, so we suppose they are met without any verification. See how to test these assumptions in R if you are interested.

Overall and group means

We first need to compute the mean age by class (referred as the group means):

class A: $\frac{24 + 31 + 26 + 23}{4} = 26$
class B: $\frac{24 + 21 + 19 + 24}{4} = 22$
class C: $\frac{15 + 21 + 18 + 18}{4} = 18$

and the mean age for the whole sample (referred as the overall mean):

\[\begin{equation} \begin{split} & \frac{24 + 31 + 26 + 23 + 24 + 21 + 19 }{12} \\ &\frac{+ 24 + 15 + 21 + 18 + 18}{12} = 22 \end{split} \end{equation} \]

SSR and SSE

We then need to compute the sum of squares regression (SSR), and the sum of squares error (SSE).

The SSR is computed by taking the square of the difference between the mean group and the overall mean, multiplied by the number of observations in the group:

and then taking the sum of all cells:

\[64+0+64 = 128 = SSR\]

The SSE is computed by taking the square of the difference between each observation and its group mean:

and then taking the sum of all cells:

\[\begin{equation} \begin{split} & 4+25+0+9+4+1+9+4 \\ & +9+9+0+0 = 74 = SSE \end{split} \end{equation} \]

For those interested in computing the sum of square total (SST), it is simply the sum of SSR and SSE, that is,

\[\begin{equation} \begin{split} SST &= SSR + SSE\\ &= 128 + 74 \\ & =202 \end{split} \end{equation} \]

ANOVA table

The ANOVA table looks as follows (we leave it empty and we are going to fill it in step by step):

We start to build the ANOVA table by plugging the SSR and SSE values found above into the table (in the “Sum.of.Sq.” column):

The “Df” column corresponds to the degrees of freedom, and is computed as follows:

for the line regression: number of groups - 1 = 3 - 1 = 2
for the line error: number of observations - number of groups = 12 - 3 = 9

With this information, the ANOVA table becomes:

The “Mean.Sq.” column corresponds to the Mean Square, and is equal to the sum of square divided by the degrees of freedom, so the “Sum.of.Sq.” column divided by the “Df” column:

Finally, the F-value corresponds to the ratio between the two mean squares, so $\frac{64}{8.222} = 7.78$:

This F-value gives the test statistic (also referred as $F_{obs}$), which needs to be compared with the critical value found in the Fisher table to conclude the test.

We find the critical value in the Fisher table based on the degrees of freedom (those used in the ANOVA table) and based on the significance level. Suppose we take a significance level $\alpha = 0.05$, the critical value can be found in the Fisher table as follows:

So we have

\[F_{2; 9; 0.05} = 4.26\]

If you are interested to find this value with R, it can be found with the qf() function, where 0.95 corresponds to $1 - \alpha$:

qf(0.95, 2, 9)

## [1] 4.256495

Conclusion of the test

The rejection rule says that, if:

$F_{obs} > F_{2; 9; 0.05} \Rightarrow$ we reject the null hypothesis
$F_{obs} \le F_{2; 9; 0.05} \Rightarrow$ we do not reject the null hypothesis

In our case,

\[F_{obs} = 7.78 > F_{2; 9; 0.05} = 4.26\]

$\Rightarrow$ We reject the null hypothesis that all means are equal. In other words, it means that at least one class is different than the other two in terms of age.²

To verify our results, here is the ANOVA table using R:

##             Df Sum Sq Mean Sq F value Pr(>F)  
## class        2    128   64.00   7.784 0.0109 *
## Residuals    9     74    8.22                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We found the same results by hand, but note that in R, the $p$-value is computed instead of comparing the $F_{obs}$ with the critical value. The $p$-value can easily be found in R based on the $F_{obs}$ and the degrees of freedom:

pf(7.78, 2, 9,
  lower.tail = FALSE
)

## [1] 0.010916

Conclusion

Thanks for reading.

I hope this article helped you to conduct a one-way ANOVA by hand. See this tutorial if you want to learn how to do it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

In that case, a Student’s t-test is usually preferred over an ANOVA, although both tests will lead to the exact same conclusions.↩︎
Remember that an ANOVA cannot tell you which group is different than the other in terms of the quantitative dependent variable, nor whether they are all different or if only one is different. To answer this question, post-hoc tests are required. This is beyond the scope of the present post, but it can easily be done in R (see this tutorial).↩︎

Two-way ANOVA in R

Mon, 19 Jun 2023 00:00:00 +0000

Introduction

The two-way ANOVA (analysis of variance) is a statistical method that allows to evaluate the simultaneous effect of two categorical variables on a quantitative continuous variable.

The two-way ANOVA is an extension of the one-way ANOVA since it allows to evaluate the effects on a numerical response of two categorical variables instead of one.

The advantage of a two-way ANOVA over a one-way ANOVA is that we test the relationship between two variables, while taking into account the effect of a third variable. Moreover, it also allows to include the possible interaction of the two categorical variables on the response to evaluate whether or not they act jointly on the response variable.

The advantage of a two-way over a one-way ANOVA is quite similar to the advantage of a multiple linear regression over a correlation:

The correlation measures the relationship between two quantitative variables. The multiple linear regression also measures the relationship between two variables, but this time taking into account the potential effect of other covariates.
The one-way ANOVA tests whether a quantitative variable is different between groups. The two-way ANOVA also tests whether a quantitative variable is different between groups, but this time taking into account the effect of another qualitative variable.

Previously, we have discussed about one-way ANOVA in R. Now, we show when, why and how to perform a two-way ANOVA in R.

Before going further, I would like to mention and briefly describe some related statistical methods and tests in order to avoid any confusion:

A Student’s t-test is used to evaluate the effect of one categorical variable on a quantitative continuous variable, when the categorical variable has exactly 2 levels:
- Student’s t-test for independent samples if the observations are independent (for example: if we compare the age between women and men)
- Student’s t-test for paired samples if the observations are dependent, that is, when they come in pairs (it is the case when the same subjects are measured twice, at two different points in time, before and after a treatment for example)
To evaluate the effect of one categorical variable on a quantitative variable, when the categorical variable has 3 or more levels:¹
- one-way ANOVA (often simply referred as ANOVA) if the groups are independent (for example a group of patients who received treatment A, another group of patients who received treatment B, and the last group of patients who received no treatment or a placebo)
- repeated measures ANOVA if the groups are dependent (when the same subjects are measured three times, at three different points in time, before, during and after a treatment for example)
A two-way ANOVA is used to evaluate the effects of 2 categorical variables (and their potential interaction) on a quantitative continuous variable. This is the topic of the post.
Linear regression is used to evaluate the relationship between a quantitative continuous dependent variable and one or several independent variables:
- simple linear regression if there is only one independent variable (which can be quantitative or qualitative)
- multiple linear regression if there is at least two independent variables (which can be quantitative, qualitative, or a mix of both)
An ANCOVA (analysis of covariance) is used to evaluate the effect of a categorical variable on a quantitative variable, while controlling for the effect of another quantitative variable (known as covariate). ANCOVA is actually a special case of multiple linear regression with a mix of one qualitative and one quantitative independent variable.
A mixed ANOVA is used to test differences between two or more groups whilst subjecting participants to repeated measures: one factor (a fixed effects factor) is a between-subjects variable (for example, treatment A and B, with patients receiving only one of the two treatments) and the other (a random effects factor) is a within-subjects variable (for example, measurements are made on day 1, day 2 and day 3 on all subjects).

In this post, we start by explaining when and why a two-way ANOVA is useful, we then do some preliminary descriptive analyses and present how to conduct a two-way ANOVA in R. Finally, we show how to interpret and visualize the results. We also briefly mention and illustrate how to verify the underlying assumptions.

Data

To illustrate how to perform a two-way ANOVA in R, we use the penguins dataset, available from the {palmerpenguins} package.

We do not need to import the dataset, but we need to load the package first and then call the dataset:

# install.packages("palmerpenguins")
library(palmerpenguins)

dat <- penguins # rename dataset
str(dat) # structure of dataset

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

The dataset contains 8 variables for 344 penguins, summarized below:

summary(dat)

##       species          island    bill_length_mm  bill_depth_mm  
##  Adelie   :152   Biscoe   :168   Min.   :32.10   Min.   :13.10  
##  Chinstrap: 68   Dream    :124   1st Qu.:39.23   1st Qu.:15.60  
##  Gentoo   :124   Torgersen: 52   Median :44.45   Median :17.30  
##                                  Mean   :43.92   Mean   :17.15  
##                                  3rd Qu.:48.50   3rd Qu.:18.70  
##                                  Max.   :59.60   Max.   :21.50  
##                                  NA's   :2       NA's   :2      
##  flipper_length_mm  body_mass_g       sex           year     
##  Min.   :172.0     Min.   :2700   female:165   Min.   :2007  
##  1st Qu.:190.0     1st Qu.:3550   male  :168   1st Qu.:2007  
##  Median :197.0     Median :4050   NA's  : 11   Median :2008  
##  Mean   :200.9     Mean   :4202                Mean   :2008  
##  3rd Qu.:213.0     3rd Qu.:4750                3rd Qu.:2009  
##  Max.   :231.0     Max.   :6300                Max.   :2009  
##  NA's   :2         NA's   :2

In this post, we will focus on the following three variables:

species: the species of the penguin (Adelie, Chinstrap or Gentoo)
sex: sex of the penguin (female and male)
body_mass_g: body mass of the penguin (in grams)

If needed, more information about this dataset can be found by running ?penguins in R.

body_mass_g is the quantitative continuous variable and will be the dependent variable, whereas species and sex are both qualitative variables.

Those two last variables will be our independent variables, also referred as factors. Make sure that they are read as factors by R. If it is not the case, they will need to be transformed to factors.

Aim and hypotheses of a two-way ANOVA

As mentioned above, a two-way ANOVA is used to evaluate simultaneously the effect of two categorical variables on one quantitative continuous variable.

It is referred as two-way ANOVA because we are comparing groups which are formed by two independent categorical variables.

Here, we would like to know if body mass depends on species and/or sex. In particular, we are interested in:

measuring and testing the relationship between species and body mass,
measuring and testing the relationship between sex and body mass, and
potentially check whether the relationship between species and body mass is different for females and males (which is equivalent than checking whether the relationship between sex and body mass depends on the species)

The first two relationships are referred as main effects, while the third point is known as the interaction effect.

The main effects test whether at least one group is different from another one (while controlling for the other independent variable). On the other hand, the interaction effect aims at testing whether the relationship between two variables differs depending on the level of a third variable. In other words, if the evolution between the response and the first categorical variable does not depend on the modalities of the second categorical variable, then there is no interaction between the two variables. If, on the contrary, there is a modification of this evolution, either by an increase in the effect of the first variable, or by a decrease, then there is an interaction.

When performing a two-way ANOVA, testing the interaction effect is not mandatory. However, omitting an interaction effect may lead to erroneous conclusions if the interaction effect is present.

If we go back to our example, we have the following hypothesis tests:

Main effect of sex on body mass:
- $H_0$: mean body mass is equal between females and males
- $H_1$: mean body mass is different between females and males
Main effect of species on body mass:
- $H_0$: mean body mass is equal between all 3 species
- $H_1$: mean body mass is different for at least one species
Interaction between sex and species:
- $H_0$: there is no interaction between sex and species, meaning that the relationship between species and body mass is the same for females and males (similarly, the relationship between sex and body mass is the same for all 3 species)
- $H_1$: there is an interaction between sex and species, meaning that the relationship between species and body mass is different for females than for males (similarly, the relationship between sex and body mass depends on the species)

Assumptions of a two-way ANOVA

Most statistical tests require some assumptions for the results to be valid, and a two-way ANOVA is not an exception.

Assumptions of a two-way ANOVA are similar than for a one-way ANOVA. To summarize:

Variable type: the dependent variable must be quantitative continuous, while the two independent variables must be categorical (with at least two levels).
Independence: the observations should be independent between groups and within each group.
Normality:
- For small samples, data should follow approximately a normal distribution
- For large samples (usually $n \ge 30$ in each group/sample), normality is not required (thanks to the central limit theorem)
Equality of variances: variances should be equal across groups.
Outliers: There should be no significant outliers in any group.

More details about these assumptions can be found in the assumptions of a one-way ANOVA.

Now that we have seen the underlying assumptions of the two-way ANOVA, we review them specifically for our dataset before applying the test and interpreting the results.

Variable type

The dependent variable body mass is quantitative continuous, while both independent variables sex and species are qualitative variables (with at least 2 levels).

Therefore, this assumption is met.

If your dependent variable is quantitative discrete, this is count data, which, strictly speaking, should be analyzed using a generalized linear model, not an ANOVA.

Independence

Independence is usually checked based on the design of the experiment and how data have been collected.²

To keep it simple, observations are usually:

independent if each experimental unit (here a penguin) has been measured only once and the observations are collected from a representative and randomly selected portion of the population, or
dependent if each experimental unit has been measured at least twice (as it is often the case in the medical field for example, with two measurements on the same subjects; one before and one after the treatment).

In our case, body mass has been measured only once on each penguin, and on a representative and random sample of the population, so the independence assumption is met.

Note that if your data correspond to observations made several times on the same experimental units (for example, if one of the factors is a treatment (A or B) and the second factor is time (day 1, day 2 and day 3), and measurements are made at each time point on the same subjects), a two-way mixed ANOVA should be used.

Normality

We have a large sample in all subgroups (each combination of the levels of the two factors, called cell):

table(dat$species, dat$sex)

##            
##             female male
##   Adelie        73   73
##   Chinstrap     34   34
##   Gentoo        58   61

so normality does not need to be checked.

For completeness, we still show how to verify normality, as if we had a small samples.

There are several methods to test the normality assumption. The most common methods being:

a QQ-plot by group or on the residuals, and/or
a histogram by group or on the residuals, and/or
a normality test (Shapiro-Wilk test for instance) by group or on the residuals.

The easiest/shortest way is to verify the normality with a QQ-plot on the residuals. To draw this plot, we first need to save the model:

# save model
mod <- aov(body_mass_g ~ sex * species,
  data = dat
)

This piece of code will be explained further.

Now we can draw the QQ-plot on the residuals. We show two ways to do so, first with the plot() function and second with the qqPlot() function from the {car} package:

# method 1
plot(mod, which = 2)

# method 2
library(car)

qqPlot(mod$residuals,
  id = FALSE # remove point identification
)

Code for method 1 is slightly shorter, but it misses the confidence interval around the reference line.

If points follow the straight line (called Henry’s line) and fall within the confidence band, we can assume normality. This is the case here.

If you prefer to verify the normality based on a histogram of the residuals, here is the code:

# histogram
hist(mod$residuals)

The histogram of the residuals show a gaussian distribution, which is in line with the conclusion from the QQ-plot.

Although the QQ-plot and histogram is largely enough to verify the normality, if you want to test it more formally with a statistical test, the Shapiro-Wilk test can be applied on the residuals as well:

# normality test
shapiro.test(mod$residuals)

## 
## 	Shapiro-Wilk normality test
## 
## data:  mod$residuals
## W = 0.99776, p-value = 0.9367

$\Rightarrow$ We do not reject the null hypothesis that the residuals follow a normal distribution ($p$-value = 0.937).

From the QQ-plot, histogram and Shapiro-Wilk test, we conclude that we do not reject the null hypothesis of normality of the residuals.

The normality assumption is thus verified, we can now check the equality of the variances.

Note that if the normality assumption is not met, many transformations can be applied on the dependent variable to improve it, the most common ones being the logarithmic (log() function in R) and the Box-Cox transformations. If the normality assumption is still not met on the transformed data, the non-parametric version of the two-way ANOVA, the Scheirer–Ray–Hare test, can be used. Alternatively, a permutation test can also be used.

Homogeneity of variances

Equality of variances, also referred as homogeneity of variances or homoscedasticity, can be verified visually with the plot() function:

plot(mod, which = 3)

Since the spread of the residuals is constant, the red smooth line is horizontal and flat, so it looks like the constant variance assumption is satisfied here.

The diagnostic plot above is sufficient, but if you prefer it can also be tested more formally with the Levene’s test (also from the {car} package):³

leveneTest(mod)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   5  1.3908 0.2272
##       327

$\Rightarrow$ We do not reject the null hypothesis that the variances are equal ($p$-value = 0.227).

Both the visual and formal approaches give the same conclusion; we do not reject the hypothesis of homogeneity of the variances.

Note that, as for the normality, the logarithmic and Box-Cox transformations may improve homogeneity of the residuals.

Outliers

The easiest and most common way to detect outliers is visually thanks to boxplots by groups.

For females and males:

library(ggplot2)

# boxplots by sex
ggplot(dat) +
  aes(x = sex, y = body_mass_g) +
  geom_boxplot()

For the three species:

# boxplots by species
ggplot(dat) +
  aes(x = species, y = body_mass_g) +
  geom_boxplot()

There are, as defined by the interquartile range criterion, two outliers for the species Chinstrap. These points are, nonetheless, not extreme enough to bias results.

Therefore, we consider that the assumption of no significant outliers is met.

Two-way ANOVA

We have shown that all assumptions are met, so we can now proceed to the implementation of the two-way ANOVA in R.

This will allow us to answer the following research questions:

Controlling for the species, is body mass significantly different between the two sexes?
Controlling for the sex, is body mass significantly different for at least one species?
Is the relationship between species and body mass different between female and male penguins?

Preliminary analyses

Before performing any statistical test, it is a good practice to make some descriptive statistics in order to have a first overview of the data, and perhaps, have a glimpse of the results to be expected.

This can be done via descriptive statistics or plots.

Descriptive statistics

If we want to keep it simple, we can compute only the mean for each subgroup:

# mean by group
aggregate(body_mass_g ~ species + sex,
  data = dat,
  FUN = mean
)

##     species    sex body_mass_g
## 1    Adelie female    3368.836
## 2 Chinstrap female    3527.206
## 3    Gentoo female    4679.741
## 4    Adelie   male    4043.493
## 5 Chinstrap   male    3938.971
## 6    Gentoo   male    5484.836

Or eventually, the mean and standard deviation for each subgroup using the {dplyr} package:

# mean and sd by group
library(dplyr)

group_by(dat, sex, species) %>%
  summarise(
    mean = round(mean(body_mass_g, na.rm = TRUE)),
    sd = round(sd(body_mass_g, na.rm = TRUE))
  )

## # A tibble: 8 × 4
## # Groups:   sex [3]
##   sex    species    mean    sd
##   <fct>  <fct>     <dbl> <dbl>
## 1 female Adelie     3369   269
## 2 female Chinstrap  3527   285
## 3 female Gentoo     4680   282
## 4 male   Adelie     4043   347
## 5 male   Chinstrap  3939   362
## 6 male   Gentoo     5485   313
## 7 <NA>   Adelie     3540   477
## 8 <NA>   Gentoo     4588   338

Plots

If you are a frequent reader of the blog, you know that I like to draw plots to visualize the data at hand before interpreting results of a test.

The most appropriate plot when we have one quantitative and two qualitative variables is a boxplot by group. This can easily be made with the {ggplot2} package:

# boxplot by group
library(ggplot2)

ggplot(dat) +
  aes(x = species, y = body_mass_g, fill = sex) +
  geom_boxplot()

Some observations are missing for the sex, we can remove them to have a more concise plot:

dat %>%
  filter(!is.na(sex)) %>%
  ggplot() +
  aes(x = species, y = body_mass_g, fill = sex) +
  geom_boxplot()

Note that we could also have made the following plot:

dat %>%
  filter(!is.na(sex)) %>%
  ggplot() +
  aes(x = sex, y = body_mass_g, fill = species) +
  geom_boxplot()

But for a more readable plot, I tend to prefer putting the variable with the smallest number of levels as color (which is in fact the argument fill in the aes() layer) and the variable with the largest number of categories on the x-axis (i.e., the argument x in the aes() layer).

From the means and the boxplots by subgroup, we can already see that, in our sample:

female penguins tend to have a lower body mass than males, and that is the case for all the considered species, and
body mass is higher for Gentoo penguins than for the other two species.

Bear in mind that these conclusions are only valid within our sample! To generalize these conclusions to the population, we need to perform the two-way ANOVA and check the significance of the explanatory variables. This is the aim of the next section.

Two-way ANOVA in R

As mentioned earlier, including an interaction effect in a two-way ANOVA is not compulsory. However, in order to avoid flawed conclusions, it is recommended to first check whether the interaction is significant or not, and depending on the results, include it or not.

If the interaction is not significant, it is safe to remove it from the final model. On the contrary, if the interaction is significant, it should be included in the final model which will be used to interpret results.

We thus start with a model which includes the two main effects (i.e., sex and species) and the interaction:

# Two-way ANOVA with interaction
# save model
mod <- aov(body_mass_g ~ sex * species,
  data = dat
)

# print results
summary(mod)

##              Df    Sum Sq  Mean Sq F value   Pr(>F)    
## sex           1  38878897 38878897 406.145  < 2e-16 ***
## species       2 143401584 71700792 749.016  < 2e-16 ***
## sex:species   2   1676557   838278   8.757 0.000197 ***
## Residuals   327  31302628    95727                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 11 observations deleted due to missingness

Similar to a one-way ANOVA, the principle of a two-way ANOVA is based on the total dispersion of the data, and its decomposition into four components:

the share attributable to the first factor
the share attributable to the second factor
the share attributable to the interaction of the 2 factors
the unexplained, or residual portion.

The sum of squares (column Sum Sq) shows these four components. The two-way ANOVA consists of using a statistical test to determine whether each of the dispersion component (attributable to the 2 factors studied and to their interaction) is significantly greater than the residual component. If this is the case, we conclude that the effect considered (factor A, factor B or the interaction) is significant.

We see that the species explain a large part of the variability of body mass. It is the most important factor in explaining this variability.

The $p$-values are displayed in the last column of the output above (Pr(>F)). From these $p$-values, we conclude that, at the 5% significance level:

controlling for the species, body mass is significantly different between the two sexes,
controlling for the sex, body mass is significantly different for at least one species, and
the interaction between sex and species (displayed at the line sex:species in the output above) is significant.

So from the significant interaction effect, we have just seen that the relationship between body mass and species is different between males and females. Since it is significant, we have to keep it in the model and we should interpret results from that model.

If, on the contrary, the interaction was not significant (that is, if the $p$-value $\ge$ 0.05) we would have removed this interaction effect from the model. For illustrative purposes, below the code for a two-way ANOVA without interaction, referred as an additive model:

# Two-way ANOVA without interaction
aov(body_mass_g ~ sex + species,
  data = dat
)

For the readers who are used to perform linear regressions in R, you will notice that the structure of the code for a two-way ANOVA is in fact similar:

the formula is dependent variable ~ independent variables
the + sign is used to include independent variables without an interaction⁴
the * sign is used to include independent variables with an interaction

The resemblance with a linear regression is not a surprise because a two-way ANOVA, like all ANOVA, is actually a linear model.

Note that the following code works as well, and give the same results:⁵

# method 2
mod2 <- lm(body_mass_g ~ sex * species,
  data = dat
)

Anova(mod2)

## Anova Table (Type II tests)
## 
## Response: body_mass_g
##                Sum Sq  Df F value    Pr(>F)    
## sex          37090262   1 387.460 < 2.2e-16 ***
## species     143401584   2 749.016 < 2.2e-16 ***
## sex:species   1676557   2   8.757 0.0001973 ***
## Residuals    31302628 327                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Note that the aov() function assumes a balanced design, meaning that we have equal sample sizes within levels of our independent grouping variables. Moreover, aov() uses the type I sums of squares, so we can obtain different $p$-values when we write y ~ A * B and y ~ B * A.

For unbalanced design, that is, unequal numbers of subjects in each subgroup, the recommended methods are:

the type II ANOVA when there is no significant interaction, which can be done in R with Anova(mod, type = "II") or Anova(mod, type = 2), where mod is the name of your saved model, and
the type III ANOVA when there is a significant interaction, which can be done in R with Anova(mod, type = "III") or Anova(mod, type = 3).

This is beyond the scope of the post and we assume a balanced design here. For the interested reader, see this detailed discussion about type I, type II and type III ANOVA.

Pairwise comparisons

Through the two main effects being significant, we concluded that:

controlling for the species, body mass is different between females and males, and
controlling for the sex, body mass is different for at least one species.

If body mass is different between the two sexes, given that there are exactly two sexes, it must be because body mass is significantly different between females and males.

If one wants to know which sex has the highest body mass, it can be deduced from the means and/or boxplots by subgroup. Here, it is clear that males have a significantly higher body mass than females.

However, it is not so straightforward for the species. Let me explain why it is not as easy as for the sexes.

There are three species (Adelie, Chinstrap and Gentoo), so there are 3 pairs of species:

Adelie and Chinstrap
Adelie and Gentoo
Chinstrap and Gentoo

If body mass is significantly different for at least one species, it could be that:

body mass is significantly different between Adelie and Chinstrap but not significantly different between Adelie and Gentoo, and not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Adelie and Gentoo but not significantly different between Adelie and Chinstrap, and not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Chinstrap and Gentoo but not significantly different between Adelie and Chinstrap, and not significantly different between Adelie and Gentoo.

Or, it could also be that:

body mass is significantly different between Adelie and Chinstrap, and between Adelie and Gentoo, but not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Adelie and Chinstrap, and between Chinstrap and Gentoo, but not significantly different between Adelie and Gentoo, or
body mass is significantly different between Chinstrap and Gentoo, and between Adelie and Gentoo, but not significantly different between Adelie and Chinstrap.

Last, it could also be that body mass is significantly different between all species.

As for a one-way ANOVA, we cannot, at this stage, know precisely which species is different from which one in terms of body mass. To know this, we need to compare each species two by two thanks to post-hoc tests (also known as pairwise comparisons).

There are several post-hoc tests, the most common ones being the Tukey HSD which tests all possible pairs of groups, and the Dunett’s test which compares all groups to a reference group. As mentioned earlier, these tests should not be done on the sex variable because there are only two levels.

In this post, we show only the Tukey HSD test. For the interested reader, the Dunnett’s test is illustrated here.

As for the one-way ANOVA, the Tukey HSD test can be done in R as follows:

# method 1
TukeyHSD(mod,
  which = "species"
)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = body_mass_g ~ sex * species, data = dat)
## 
## $species
##                        diff       lwr       upr     p adj
## Chinstrap-Adelie   26.92385  -80.0258  133.8735 0.8241288
## Gentoo-Adelie    1377.65816 1287.6926 1467.6237 0.0000000
## Gentoo-Chinstrap 1350.73431 1239.9964 1461.4722 0.0000000

or using the {multcomp} package:

# method 2
library(multcomp)

res_tukey <- glht(
  aov(body_mass_g ~ sex + species,
    data = dat
  ),
  linfct = mcp(species = "Tukey")
)

summary(res_tukey)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: aov(formula = body_mass_g ~ sex + species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0    26.92      46.48   0.579     0.83    
## Gentoo - Adelie == 0     1377.86      39.10  35.236   <1e-05 ***
## Gentoo - Chinstrap == 0  1350.93      48.13  28.067   <1e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

or using the pairwise.t.test() function using the $p$-value adjustment method of your choice:⁶

# method 3
pairwise.t.test(dat$body_mass_g, dat$species,
  p.adjust.method = "BH"
)

## 
## 	Pairwise comparisons using t tests with pooled SD 
## 
## data:  dat$body_mass_g and dat$species 
## 
##           Adelie Chinstrap
## Chinstrap 0.63   -        
## Gentoo    <2e-16 <2e-16   
## 
## P value adjustment method: BH

Note that when using the second method, it is the model without the interaction that needs to be specified into the glht() function, even if the interaction is significant. Moreover, do not forget to replace mod and species in my code with the name of your model and the name of your independent variable.

Both methods give the same results, that is:

body mass is not significantly different between Chinstrap and Adelie (adjusted $p$-value = 0.83),
body mass is significantly different between Gentoo and Adelie (adjusted $p$-value < 0.001), and
body mass is significantly different between Gentoo and Chinstrap (adjusted $p$-value < 0.001).

Remember that it is the adjusted $p$-values that are reported, to prevent the issue of multiple testing which occurs when comparing several pairs of groups.

If you would like to compare all combinations of groups, it can be done with the TukeyHSD() function and specifying the interaction in the which argument:

# all combinations of sex and species
TukeyHSD(mod,
  which = "sex:species"
)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = body_mass_g ~ sex * species, data = dat)
## 
## $`sex:species`
##                                      diff       lwr       upr     p adj
## male:Adelie-female:Adelie        674.6575  527.8486  821.4664 0.0000000
## female:Chinstrap-female:Adelie   158.3703  -25.7874  342.5279 0.1376213
## male:Chinstrap-female:Adelie     570.1350  385.9773  754.2926 0.0000000
## female:Gentoo-female:Adelie     1310.9058 1154.8934 1466.9181 0.0000000
## male:Gentoo-female:Adelie       2116.0004 1962.1408 2269.8601 0.0000000
## female:Chinstrap-male:Adelie    -516.2873 -700.4449 -332.1296 0.0000000
## male:Chinstrap-male:Adelie      -104.5226 -288.6802   79.6351 0.5812048
## female:Gentoo-male:Adelie        636.2482  480.2359  792.2606 0.0000000
## male:Gentoo-male:Adelie         1441.3429 1287.4832 1595.2026 0.0000000
## male:Chinstrap-female:Chinstrap  411.7647  196.6479  626.8815 0.0000012
## female:Gentoo-female:Chinstrap  1152.5355  960.9603 1344.1107 0.0000000
## male:Gentoo-female:Chinstrap    1957.6302 1767.8040 2147.4564 0.0000000
## female:Gentoo-male:Chinstrap     740.7708  549.1956  932.3460 0.0000000
## male:Gentoo-male:Chinstrap      1545.8655 1356.0392 1735.6917 0.0000000
## male:Gentoo-female:Gentoo        805.0947  642.4300  967.7594 0.0000000

Or with the HSD.test() function from the {agricolae} package, which denotes subgroups that are not significantly different from each other with the same letter:

library(agricolae)

HSD.test(mod,
  trt = c("sex", "species"),
  console = TRUE # print results
)

## 
## Study: mod ~ c("sex", "species")
## 
## HSD Test for body_mass_g 
## 
## Mean Square Error:  95726.69 
## 
## sex:species,  means
## 
##                  body_mass_g      std  r  Min  Max
## female:Adelie       3368.836 269.3801 73 2850 3900
## female:Chinstrap    3527.206 285.3339 34 2700 4150
## female:Gentoo       4679.741 281.5783 58 3950 5200
## male:Adelie         4043.493 346.8116 73 3325 4775
## male:Chinstrap      3938.971 362.1376 34 3250 4800
## male:Gentoo         5484.836 313.1586 61 4750 6300
## 
## Alpha: 0.05 ; DF Error: 327 
## Critical Value of Studentized Range: 4.054126 
## 
## Groups according to probability of means differences and alpha level( 0.05 )
## 
## Treatments with the same letter are not significantly different.
## 
##                  body_mass_g groups
## male:Gentoo         5484.836      a
## female:Gentoo       4679.741      b
## male:Adelie         4043.493      c
## male:Chinstrap      3938.971      c
## female:Chinstrap    3527.206      d
## female:Adelie       3368.836      d

If you have many groups to compare, plotting them might be easier to interpret:

# set axis margins so labels do not get cut off
par(mar = c(4.1, 13.5, 4.1, 2.1))

# create confidence interval for each comparison
plot(TukeyHSD(mod, which = "sex:species"),
  las = 2 # rotate x-axis ticks
)

From the outputs and plot above, we conclude that all combinations of sex and species are significantly different, except between female Chinstrap and female Adelie ($p$-value = 0.138) and male Chinstrap and male Adelie ($p$-value = 0.581).

These results, which are by the way in line with the boxplots shown above and which will be confirmed with the visualizations below, concludes the two-way ANOVA in R.

Visualizations

If you would like to visualize results in a different way to what has already been presented in the preliminary analyses, below are some ideas of useful plots.

First, with the mean and standard error of the mean by subgroup using the allEffects() function from the {effects} package:

# method 1
library(effects)

plot(allEffects(mod))

Or using the {ggpubr} package:

# method 2
library(ggpubr)

ggline(subset(dat, !is.na(sex)), # remove NA level for sex
  x = "species",
  y = "body_mass_g",
  color = "sex",
  add = c("mean_se") # add mean and standard error
) +
  labs(y = "Mean of body mass (g)")

Alternatively, using {Rmisc} and {ggplot2}:

library(Rmisc)

# compute mean and standard error of the mean by subgroup
summary_stat <- summarySE(dat,
  measurevar = "body_mass_g",
  groupvars = c("species", "sex")
)

# plot mean and standard error of the mean
ggplot(
  subset(summary_stat, !is.na(sex)), # remove NA level for sex
  aes(x = species, y = body_mass_g, colour = sex)
) +
  geom_errorbar(aes(ymin = body_mass_g - se, ymax = body_mass_g + se), # add error bars
    width = 0.1 # width of error bars
  ) +
  geom_point() +
  labs(y = "Mean of body mass (g)")

Second, if you prefer to draw only the mean by subgroup:

with(
  dat,
  interaction.plot(species, sex, body_mass_g)
)

Last but not least, for those of you who are familiar with GraphPad, you are most likely familiar with plotting means and error bars as follows:

# plot mean and standard error of the mean as barplots
ggplot(
  subset(summary_stat, !is.na(sex)), # remove NA level for sex
  aes(x = species, y = body_mass_g, fill = sex)
) +
  geom_bar(position = position_dodge(), stat = "identity") +
  geom_errorbar(aes(ymin = body_mass_g - se, ymax = body_mass_g + se), # add error bars
    width = 0.25, # width of error bars
    position = position_dodge(.9)
  ) +
  labs(y = "Mean of body mass (g)")

Conclusion

In this post, we started with a few reminders of the different tests that exist to compare a quantitative variable across groups. We then focused on the two-way ANOVA, starting from its goal and hypotheses to its implementation in R, together with the interpretations and some visualizations. We also briefly mentioned its underlying assumptions and one post-hoc test to compare all subgroups.

All this was illustrated with the penguins dataset available from the {palmerpenguins} package.

Thanks for reading.

I hope this article will help you in conducting a two-way ANOVA with your data.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

In theory, a one-way ANOVA can also be used to compare 2 groups, and not only 3 or more. Nonetheless, in practice, it is often the case that a Student’s t-test is performed to compare 2 groups, and a one-way ANOVA to compare 3 or more groups. Conclusions obtained via a Student’s t-test for independent samples and a one-way ANOVA with 2 groups will be similar.↩︎
If you really want to test the independence, you can do so visually with a plot of the residuals vs. fitted values. This plot can be done in R with plot(mod, which = 1), where mod corresponds to the name of your model. Or you can do so with the Durbin-Watson test. In R, it can be done with the durbinWatsonTest() function from the {car} package.↩︎
Note that the Bartlett’s and Fligner-Killeen tests are also appropriate to test the assumption of equal variances.↩︎
An additive model makes the assumption that the 2 explanatory variables are independent; they do not interact with each other.↩︎
To not be confused with the anova() function because it provides sequential results that depend on the order in which the variables appear in the model.↩︎
Here, we use the Benjamini & Hochberg (1995) correction, but you can choose between several methods. See ?p.adjust for more details.↩︎

What is survival analysis? Examples by hand and in R

Thu, 22 Dec 2022 00:00:00 +0000

Note that this article is inspired from:

the lecture notes of Prof. Van Keilegom and my personal notes as teaching assistant for her course entitled “Analysis of Survival and Duration Data” given at UCLouvain
the lecture notes of Prof. Legrand for her course entitled “Statistics in clinical trials” given at UCLouvain

Introduction

For the last post of the year, I would like to present a rather unknown (yet important) statistical method–survival analysis.

Although survival analysis is a branch of statistics, it is usually not covered in introductory statistics courses and it is rather unknown to the general public. It is mostly taught in biostatistics courses or advanced statistics study programs.

In this article, I will explain what is survival analysis, in which context and how it is used. I will explain the main tools and methods used by biostatisticians to analyze survival data and how to estimate and interpret survival curves.

I will show in detail how to apply these techniques in R with concrete examples. In practice, survival analysis is almost always done via a statistical program and never done by hand. However, as for any statistical concept, I believe that doing it by hand allows to really understand the concepts and what these programs actually do. For this reason, I will also show a brief example on how to perform a basic survival analysis by hand.

What is survival analysis?

Survival analysis (also called time-to-event analysis or duration analysis) is a branch of statistics aimed at analyzing the duration of time from a well-defined time origin until one or more events happen, called survival times or duration times.

In other words, in survival analysis, we are interested in a certain event and want to analyze the time until the event happens.

While the event of interest is often death (in this case we study the time to death for patients having a specific disease) or recurrence (in this case we study the time to relapse of a certain disease), it is not limited to the medical field or epidemiology.

In fact, it can be used in many domains. For example, we may also analyze the time until:

getting cured from a certain disease
finding a new job after a period of unemployment
being arrested again after having been released from jail
the first pregnancy
the failure of a mechanical system or a machine
a bank or a company goes bankrupt
a customer buys a new product or stops its current subscription
a letter is delivered
a taxi picks you up after having called the taxi company
an employee leaves the company
etc.

As you can see, the event of interest does not necessarily have to be the death or the occurrence of a disease, but in all situations we are interested in analyzing the time until a specific event occurs.

Why do we need special methods for survival analysis?

Survival data, also referred as time-to-event data, requires a special set of statistical methods for three main reasons:

Duration times are always positive: the time until an event of interest occurs cannot be less than 0. Moreover, the distribution of survival times is right-skewed.
Different measures are of interest depending on the research question, context, etc. For instance, we could be interested in:
- The probability that a cancer patient survives longer than 5 years after diagnosis?
- The typical waiting time for a cab to arrive after having called the taxi company?
- How many, out of 100 unemployed people, are expected to have a job again after 2 months of unemployment?
Censoring is almost always an issue:
- When the event occurred before the end of the study, the survival time is known.
- However, sometimes, the event is not yet observed at the end of the study. Suppose that we study the time until death of patients with breast cancer. Luckily, some patients will not die before the end of the study.
- It can also happen that the patient withdraws from the study or moves to another country before the end of the study (known as lost to follow up or drop out).
- In all situations, his or her survival time cannot be observed because the event is not observed for the duration of the study.
- Censoring can be seen, in some sense, as a type of missing data.
- For these reasons, many “standard” statistical tools such as descriptive statistics, hypothesis tests and regression models are not appropriate for this kind of data. Specific statistical methods are required to take into account the fact that the exact survival duration for some patients is missing. It is known that they survived a certain amount of time (until the end of the study or until the time of withdrawal), but their exact survival time is unknown.

For your information, there are three types of censoring:

right-censoring (the most frequent),
left-censoring (the least frequent) and
interval-censoring.

When the event is not yet observed at the end of the study (i.e., the survival time is greater than the observed duration), this is referred as right-censoring. Left-censoring occurs if a participant is entered into the study when the event of interest occurred prior to study entry but we do not know exactly when. Interval-censoring implies that the event occurred within a time interval (between two known dates, two visits, etc.); the exact moment of occurrence is not known. The goal is of course to analyze all available data, including information about censored patients.

The goal of survival analysis is thus to model and describe time-to-event data in an appropriate way, taking the particularities of this type of data into account.

Common functions in survival analysis

We are not going to go to much into the details, but it is important to lay the foundation with the most common functions in survival analysis.

Let $T$ be a non-negative continuous random variable, representing the time until the event of interest. We consider the following functions:

Survival function
Cumulative hazard function
Hazard function

Survival function

The most common one is the survival function.

Let $T$ be a non-negative continuous random variable, representing the time until the event of interest. The survival function $S(t)$ is the probability that a randomly chosen individual is still at risk at time $t$, where $0 \le t \le +\infty$. For each $t$, it is given by

\[ \begin{align*} S(t) &= P(T > t)\\ &= 1 - P(T \le t)\\ &= 1 - F(t)\\ &= 1 - \int^t_0 f(u)\text{d}u, \end{align*} \]

where $f(\cdot)$ and $F(\cdot)$ are the density and the cumulative distribution functions of $T$, respectively.

$S(t)$ represents, for each time $t$, the probability that the time until the event is greater than this time $t$. In other words, it models the probability that the event of interest happens after $t$.

In the context of our examples mentioned above, it gives the probability that:

a randomly selected patient will survive beyond time $t$ or the proportion of patients still alive after time $t$,
a cab takes more than $t$ minutes to arrive, or
an unemployed person take more than $t$ months to find a new job.

The survival function $S(t)$ is:

a decreasing function,
taking values in $[0, 1]$ (since it is a probability), and
equal to 1 at $t = 0$ (i.e., $S(0) = 1$) and 0 at $t = \infty$ (i.e., $S(\infty) = 0$).

Visually we have:

The curve shows the proportion of individuals (or experimental units) who, as time goes on, have not experienced the event of interest. As time progresses, events occur, so the proportion who have not experienced the event decreases.

Cumulative hazard function

The cumulative hazard function, which is the total hazard experienced up to time $t$, is defined as:

\[H(t) = -log\left(S(t)\right)\]

and has the following properties:

increasing function,
taking value in $[0, +\infty]$, and
$S(t) = exp(-H(t))$.

Hazard function

The hazard function $h(t)$, or hazard rate, defines the instantaneous event rate at time $t$ for an individual still at risk at that time. It can be obtained by

\[ \begin{align*} h(t) &= \lim_{\Delta t \rightarrow 0} \frac{P(t \le T < t + \Delta t | T \ge t)}{\Delta t}\\ &= \frac{d}{dt} H(t)\\ &= \frac{f(t)}{S(t)}. \end{align*} \]

and has the following properties:

positive function (not necessarily increasing or decreasing)
the hazard function $h(t)$ can have many different shapes and is therefore a useful tool to summarize survival data

In the context of cancer research when death is the event of interest, $h(t)$ measures the instantaneous risk of dying right after time $t$ given the individual is alive at time $t$.

To link the hazard rate with the survival function; the survival curve represents the hazard rates. A steeper slope indicates a higher hazard rate because events happen more frequently, reducing the proportion of individuals who have not experienced the event at a faster rate. On the contrary, a gradual and flatter slope indicates a lower hazard rate because events occur less frequently, reducing the proportion of individuals who have not experiences the event at a slower rate. More formally:

\[S(t) = \exp\left(-\int^t_0 h(u) \text{d}u\right).\]

Note that, in contrast to the survival function which focuses on not having an event, the hazard function focuses on the event occurring.

Estimation

To estimate the survival function, we need to use an estimator which is able to deal with censoring. The most common one is the nonparametric Kaplan and Meier (1958) estimator (also sometimes referred as the product-limit estimator, or more simply, the K-M estimator).

The advantages of the Kaplan-Meier estimator are that:

it is simple and straightforward to use and interpret
it is a nonparametric estimator, so it constructs a survival curve from the data and no assumptions is made about the shape of the underlying distribution
it gives a graphical representation of the survival function(s), useful for illustrative purposes

The principle behind this estimator is that surviving beyond time $t_i$ implies surviving beyond time $t_{i-1}$ and surviving at time $t_i$. Note that an important assumption for the estimation to hold is that censoring is independent of the occurrence of events. We say that censoring is non-informative, that is, censored subjects have the same survival prospects as subjects who are not censored and who continue to be followed.

By hand

To understand how it works, let’s first estimate it by hand on the following dataset:¹

subject	time	event
1	3	0
2	5	1
3	7	1
4	2	1
5	18	0
6	16	1
7	2	1
8	9	1
9	16	1
10	5	0

where:

subject is the individual’s identifier
time is the time to event (in years)²
event is the event status (0 = censored, 1 = event happened)

Remember that for each subject, we need to know at least 2 pieces of information:

the time until the event of interest or the time until the censoring, and
whether we have observed the event of interest or if we have observed censoring.

We first need to count the number of distinct event times. Ignoring censored observations, we have 5 distinct event times:

2, 5, 7, 9 and 16

The easiest way to do the calculation by hand is by filling the following table (a table with 5 rows since there are 5 distinct event times):

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1
2
3
4
5

We fill columns one by one:

$y_{(j)}$ = the ordered distinct event times:

2, 5, 7, 9 and 16

So the table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1	2
2	5
3	7
4	9
5	16

$d_{(j)}$ = the number of observations for each distinct event time. For this, the frequency for each distinct event time is useful:

## time
##  2  5  7  9 16 
##  2  1  1  1  2

The table becomes:

$j$	$y_{(j)}$	$d_{(j)}$
1	2	2
2	5	1
3	7	1
4	9	1
5	16	2

$R_{(j)}$ = the remaining number of individuals at risk. For this, the distribution of time (censored and not censored) is useful:

## time
##  2  3  5  7  9 16 18 
##  2  1  2  1  1  2  1

We see that:

At the beginning there are 10 subjects
Just before time $t = 5$, there are 7 subjects left (10 subjects - 2 who had the event - 1 who is censored)
Just before time $t = 7$, there are 5 subjects left (= 10 - 2 - 1 - 2)
Just before time $t = 9$, there are 4 subjects left (= 10 - 2 - 1 - 2 - 1)
Just before time $t = 16$, there are 3 subjects left (= 10 - 2 - 1 - 2 - 1 - 1)

The table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$
1	2	2	10
2	5	1	7
3	7	1	5
4	9	1	4
5	16	2	3

$1 - \frac{d_{(j)}}{R_{(j)}}$ is straightforward, so the table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1.00	2.00	2.00	10.00	0.80
2.00	5.00	1.00	7.00	0.86
3.00	7.00	1.00	5.00	0.80
4.00	9.00	1.00	4.00	0.75
5.00	16.00	2.00	3.00	0.33

The Kaplan-Meier estimator is:

\[\hat{S}_{KM}(t) = \prod_{j:y_{(j)} \le t} \left(1 - \frac{d_{(j)}}{R_{(j)}} \right)\]

For each $j$, we thus take the cumulative product:

$j_1 = 0.8$
$j_2 = 0.8 \cdot 0.857 = 0.6856$
$j_3 = 0.6856 \cdot 0.8 = 0.54848$
$j_4 = 0.54848 \cdot 0.75 = 0.41136$
$j_5 = 0.41136 \cdot 0.333 = 0.1369829$

So finally, we have the survival probabilities (rounded to 3 digits):

$j$	$1 - \frac{d_{(j)}}{R_{(j)}}$	$\hat{S}_{KM}(t)$
1.00	0.80	0.80
2.00	0.86	0.69
3.00	0.80	0.55
4.00	0.75	0.41
5.00	0.33	0.14

We can now represent graphically the Kaplan-Meier estimator:

To draw this survival curve, remember that:

the x-axis corresponds to the time variable in the initial dataset, and
the y-axis corresponds to the survival probabilities found above.

In R

We now compare our results with the results found in R.

We first create the dataset with the time and event variables:

# create dataset
dat <- data.frame(
  time = c(3, 5, 7, 2, 18, 16, 2, 9, 16, 5),
  event = c(0, 1, 1, 1, 0, 1, 1, 1, 1, 0)
)

We then run the Kaplan-Meier estimator with the survfit() and Surv() functions:

# KM
library(survival)

km <- survfit(Surv(time, event) ~ 1,
  data = dat
)

Notice that the Surv() function accepts two arguments:

the time variable, and
the event variable.

The ~ 1 in the survfit() function indicates that we estimate the Kaplan-Meier without any grouping. See more on this later in the post.

Finally, we display the results and draw the Kaplan-Meier plot:

# results
summary(km)

## Call: survfit(formula = Surv(time, event) ~ 1, data = dat)
## 
##  time n.risk n.event survival std.err lower 95% CI upper 95% CI
##     2     10       2    0.800   0.126       0.5868        1.000
##     5      7       1    0.686   0.151       0.4447        1.000
##     7      5       1    0.549   0.172       0.2963        1.000
##     9      4       1    0.411   0.176       0.1782        0.950
##    16      3       2    0.137   0.126       0.0225        0.834

# plot
plot(km,
  xlab = "Time",
  ylab = "Survival probability",
  conf.int = FALSE
)

The survival probabilities can be found in the survival column. Remark that results by hand and in R are similar (any difference with the results by hand is due to rounding).

Alternatively, we can use the ggsurvplot() function within the {survminer} package:

library(survminer)

# plot
ggsurvplot(km,
  conf.int = FALSE,
  legend = "none"
)

Note that the crosses on the survival curve denote the censored observations.

The advantage with the ggsurvplot() function is that it is easy to draw the median survival directly on the plot:³

ggsurvplot(km,
  conf.int = FALSE,
  surv.median.line = "hv",
  legend = "none"
)

To find the median survival:⁴

summary(km)$table["median"]

## median 
##      9

# or more simply
km

## Call: survfit(formula = Surv(time, event) ~ 1, data = dat)
## 
##       n events median 0.95LCL 0.95UCL
## [1,] 10      7      9       5      NA

Suppose that the event of interest is death:

At time zero, the survival probability is 1 (100% of the subjects are alive).
The median indicates that the median survival time is 9 years.⁵ This is the time at which the survival $S(t)$ is 50%. In other words, is the time after which half of the subjects are expected to have died.
From the plot, we also see that $S(5) = P(T > 5 \text{ years}) =$ Probability of survival of more than 5 years for these subjects = 68.6%. This means that 68.6% of all subjects survive longer than 5 years, and that 31.4% of all subjects die within the first 5 years.

For the sake of completeness, let’s do another example with a much larger dataset; the tongue dataset within the {KMsurv} package.⁶

# load data
library(KMsurv)
data(tongue)

# preview data
head(tongue)

##   type time delta
## 1    1    1     1
## 2    1    3     1
## 3    1    3     1
## 4    1    4     1
## 5    1   10     1
## 6    1   13     1

type is the tumor DNA profile (1 = aneuploid tumor, 2 = diploid tumor)
time is the time to death or on-study time (in weeks)
delta is the death indicator (0 = alive, 1 = dead)

For this example, we focus on the aneuploid type:

anaploid <- subset(tongue, type == 1)

We can now plot the estimated survival function and estimate the median time to death. Since it is an estimator, we can also construct a confidence interval for the estimated survival at each time $t$ and for the estimated median survival time.⁷

# results
fit <- survfit(Surv(time, delta) ~ 1,
  data = anaploid,
  conf.type = "log-log"
)

fit

## Call: survfit(formula = Surv(time, delta) ~ 1, data = anaploid, conf.type = "log-log")
## 
##       n events median 0.95LCL 0.95UCL
## [1,] 52     31     93      65     157

# plot
ggsurvplot(fit,
  surv.median.line = "hv",
  legend = "none"
)

The median survival time is estimated to be 93 weeks, with a 95% confidence interval between 65 and 157 weeks.

Kaplan-Meier curves can be seen as descriptive statistics for survival data. We now focus on the second branch of statistics, hypothesis testing which allows to draw conclusions on the population based on a sample (see a quick reminder about the difference between population and sample if you need).

Hypothesis testing

Hypothesis testing in the field of survival analysis mostly concerns:

The hazard function of one population: in this case we test whether a censored sample comes from a population with a known hazard function $h_0(t)$. For example, we may be interested to compare survival in a sample of patients to the survival in the overall population (derived from the life tables).
The comparison of the hazard function of two or more populations: in this case we are interested in assessing whether there are differences in survival among different groups of subjects. For example:
- 2 groups: we are interested in comparing survival for female and male colon cancer patients
- 3 groups or more: we are interested in comparing survival for melanoma cancer patients according to their treatments (with treatments A, B and C for example)⁸

Log-rank test

In this article, we focus on comparing survival between two groups using the log-rank test (also known as Mantel-Cox test). This test is the most common hypothesis test to compare survival between two groups.

The intuition behind the test is that if the two groups have different hazard rates, the two survival curves (so their slopes) will differ. More precisely, the log-rank test compares the observed number of events in each group to what would be expected if the survival curves were identical (i.e., if the null hypothesis were true).

Note that, as for the Kaplan-Meier estimator, the log-rank test is a nonparametric test, which makes no assumptions about the survival distributions.

For this example, consider the following dataset:

##    patient group time event
## 1        1     1  4.1     1
## 2        2     1  7.8     0
## 3        3     1 10.0     1
## 4        4     1 10.0     1
## 5        5     1 12.3     0
## 6        6     1 17.2     1
## 7        7     2  9.7     1
## 8        8     2 10.0     1
## 9        9     2 11.1     0
## 10      10     2 13.1     0
## 11      11     2 19.7     1
## 12      12     2 24.1     0

where:

patient is the patient’s identifier
group is the group (group 1 or 2)
time is the time to death (in years)⁹
event is the event status (0 = censored, 1 = death)

Suppose we are interested in comparing group 1 and 2 in terms of survival, that is, we compare survival curves between the 2 groups:

$H_0 : S_1(t) = S_2(t)$ for all $t$
$H_1 : S_1(t) \ne S_2(t)$ for some $t$

It is a statistical test, so if the $p$-value < $\alpha$ (usually 0.05), we reject the null hypothesis and we conclude that survival (or the time to event) is significantly different between the two groups considered.

To perform the log-rank test, the following test statistic will be useful:

\[\begin{eqnarray} U &=& \sum_{j=1}^r w(y_{(j)})\left(O_j - E_j\right) \\ &=& \sum_{j=1}^r w(y_{(j)})\left( d_{(j)1} - \frac{d_{(j)}R_{(j)1}}{R_{(j)}}\right) \end{eqnarray}\]

with $U^{obs} = \frac{U}{\sqrt{Var(U)}} \sim N(0,1)$ and¹⁰

\[\begin{eqnarray} Var(U) &=& \sum_{j=1}^r w^2(y_{(j)}) \frac{N_{(j)}}{ D_{(j)} }\\ &=& \sum_{j=1}^r w^2(y_{(j)}) \frac{ d_{(j)} \frac{R_{(j)1}}{R_{(j)} } \left( 1 - \frac{R_{(j)1}}{R_{(j)} } \right) \left( R_{(j)} - d_{(j)}\right) }{ R_{(j)} - 1 } \end{eqnarray}\]

By hand

As for the Kaplan-Meier estimator by hand, it is best to also fill in a table for the log-rank test by hand.

Let’s present the final table and comment below on how to fill it, column by column:

$j$	$y_{(j)}$	$d_{(j)1}$	$R_{(j)1}$	$d_{(j)2}$	$R_{(j)2}$	$d_{(j)}$	$R_{(j)}$	$E_{j}$	$O_{j}$	$O_{j} - E_{j}$	$N_{(j)}$	$D_{(j)}$	$N_{(j)}/D_{(j)}$
1	4.1	1	6	0	6	1	12	0.5	1	0.5	2.75	11	0.25
2	9.7	0	4	1	6	1	10	0.4	0	-0.4	2.16	9	0.24
3	10	2	4	1	5	3	9	1.333	2	0.667	4.44	8	0.555
4	17.2	1	1	0	2	1	3	0.333	1	0.667	0.44	2	0.22
5	19.7	0	0	1	2	1	2	0	0	0	0.00	1	0.00
$Total$		4		3		7		2.566		1.433			1.265

Column $j$ is the number of distinct event times. We see that there are 5 (ignoring censored observations), so we write 1 to 5 in the table.

Column $y_{(j)}$ is the ordered distinct event times:

4.1, 9.7, 10, 17.2 and 19.7

Column $d_{(j)1}$ is the number of observations for each distinct event time, for group 1:

## time
##  4.1   10 17.2 
##    1    2    1

When there is no event, we simply write 0 in the table.

Column $R_{(j)1}$ is the remaining number of patients at risk, for group 1. For this, the distribution of time (censored and not censored, for group 1) is useful:

## time
##  4.1  7.8   10 12.3 17.2 
##    1    1    2    1    1

We see that:

At the beginning, there are 6 patients
Before time 9.7, there are 4 patients left (6 - 1 who had the event at time 4.1 - 1 who was censored at time 7.8)
Before time 10, there are 4 patients left (6 - 2)
Before time 17.2, there are 1 patient left (6 - 5)
Before time 19.7, there are 0 patient left (6 - 6)

Columns $d_{(j)2}$ and $R_{(j)2}$ follow the same principle, but for group 2 this time. So we have, respectively for $d_{(j)2}$ and $R_{(j)2}$:

## time
##  9.7   10 19.7 
##    1    1    1

## time
##  9.7   10 11.1 13.1 19.7 24.1 
##    1    1    1    1    1    1

Columns $d_{(j)}$ and $R_{(j)}$ also follow the same principle, but this time considering both groups. So we have, respectively for $d_{(j)}$ and $R_{(j)}$:

## time
##  4.1  9.7   10 17.2 19.7 
##    1    1    3    1    1

## time
##  4.1  7.8  9.7   10 11.1 12.3 13.1 17.2 19.7 24.1 
##    1    1    1    3    1    1    1    1    1    1

Column $E_{j}$ is the expected number of events in the first group assuming that $h_1 \equiv h_2$. It is obtained as follows

\[ E_{j} = \frac{d_{(j)}R_{(j)1}}{R_{(j)}}\]

Column $O_{j}$ is the observed number of events in the first group, so it is equal to the $d_{(j)1}$ column.

Column $O_{j} - E_{j}$ is straightforward.

Column $N_{(j)}$ is defined as follows

\[N_{(j)} = d_{(j)} \frac{R_{(j)1}}{R_{(j)} } \left( 1 - \frac{R_{(j)1}}{R_{(j)} } \right) \left( R_{(j)} - d_{(j)}\right)\]

Column $D_{(j)}$ is $R_{(j)} - 1$.

Column $N_{(j)}/D_{(j)}$ is straightforward.

Since $w(y_{(j)}) = w^2(y_{(j)}) = 1$ for a log-rank test, we have¹¹

\[ U^{obs} = \frac{U}{\sqrt{Var(U)}} = \frac{1.434}{\sqrt{1.265}} = 1.275.\]

We reject $H_0$ if $|U^{obs}|>z_{1-\alpha/2}$, so at the 5% significance level we reject $H_0$ if $|U^{obs}|>z_{0.975}=1.96$.

We have $|U^{obs}| = 1.275 < z_{0.975}=1.96$. Hence, at the 5% significance level we do not reject $H_0$. This means that, based on the data, we are not able to conclude that survival is different between the two groups (which is equivalent than saying that we do not reject the hypothesis that survival is equal between the two groups).

If you are interested in computing the $p$-value:

$p$-value $= 2\times P(Z>1.275) = 2 \times 0.101 = 0.202 > 0.05$.

In R

We now compare our results in R with the survdiff() function:

dat <- data.frame(
  group = c(rep(1, 6), rep(2, 6)),
  time = c(4.1, 7.8, 10, 10, 12.3, 17.2, 9.7, 10, 11.1, 13.1, 19.7, 24.1),
  event = c(1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0)
)

dat

##    group time event
## 1      1  4.1     1
## 2      1  7.8     0
## 3      1 10.0     1
## 4      1 10.0     1
## 5      1 12.3     0
## 6      1 17.2     1
## 7      2  9.7     1
## 8      2 10.0     1
## 9      2 11.1     0
## 10     2 13.1     0
## 11     2 19.7     1
## 12     2 24.1     0

survdiff(Surv(time, event) ~ group,
  data = dat
)

## Call:
## survdiff(formula = Surv(time, event) ~ group, data = dat)
## 
##         N Observed Expected (O-E)^2/E (O-E)^2/V
## group=1 6        4     2.57     0.800      1.62
## group=2 6        3     4.43     0.463      1.62
## 
##  Chisq= 1.6  on 1 degrees of freedom, p= 0.2

Alternatively, we can use the ggsurvplot() function to draw the survival curves and perform the log-rank test at the same time:

fit <- survfit(Surv(time, event) ~ group, data = dat)

ggsurvplot(fit,
  pval = TRUE,
  pval.method = TRUE
)

As we can see, the $p$-values and the conclusions are the same (any difference with the results by hand is due to rounding).

As for the Kaplan-Meier estimation, we do another example on a larger dataset. Consider the data on the times until staphylococcus infection of burn patients, also available in the {KMsurv}:¹²

# load data
data(burn)

# preview data
head(burn)

##   Obs Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8 Z9 Z10 Z11 T1 D1 T2 D2 T3 D3
## 1   1  0  0  0 15  0  0  1  1  0   0   2 12  0 12  0 12  0
## 2   2  0  0  1 20  0  0  1  0  0   0   4  9  0  9  0  9  0
## 3   3  0  0  1 15  0  0  0  1  1   0   2 13  0 13  0  7  1
## 4   4  0  0  0 20  1  0  1  0  0   0   2 11  1 29  0 29  0
## 5   5  0  0  1 70  1  1  1  1  0   0   2 28  1 31  0  4  1
## 6   6  0  0  1 20  1  0  1  0  0   0   4 11  0 11  0  8  1

Using the log-rank test, we want to test the hypothesis of difference in the time to staphylococcus infection (T3 variable) between patients whose burns were cared for with a routine bathing care method (Z1 = 0) versus those whose body cleansing was initially performed using 4% chlorhexidine gluconate (Z1 = 1). The event indicator is in variable D3.

For this test, we use a two-sided alternative and a 5% significance level.

# fit
fit <- survfit(Surv(T3, D3) ~ Z1, data = burn)

# plot with log-rank test
ggsurvplot(fit,
  pval = TRUE,
  pval.method = TRUE
)

In the sample, it seems that the time to infection for patients with routine bathing (Z1 = 0) is smaller than for patients with body cleansing (Z1 = 1). This is the case because the percentage of patients who have not experienced the infection decreases more quickly, so the hazard rate is greater.

However, this conclusion cannot be generalized to the population without performing a sound statistical test. And based on the result of the log-rank test, we do not reject the hypothesis that time to infection is the same between the two groups of patients ($p$-value = 0.051).

To go further

In this article, we have presented what is survival analysis, when, why and how to use it. We discussed about censoring and survival curves. We showed how to estimate the survival function via the Kaplan-Meier estimator and how to test survival between two groups via the log-rank test. We illustrated these approaches both by hand and in R.

As you noticed, we did not show how to model survival data. There are several regression models that can be applied to survival data, the most common one being the semiparametric Cox Proportional Hazards model (1972). It originated from the medical area to investigate and assess the relationship between the survival times of patients and their corresponding predictor variables.

We have seen that the Kaplan-Meier estimator is useful to visualize survival between groups and the log-rank test to test whether survival significantly differs between groups (so both approaches use a categorical variable as predictor). However, it does not work well for assessing the effect of quantitative predictor. The Cox model has the advantage that it works for both quantitative as well as for categorical predictors, and for several risk factors at the same time (so it can model the effect of multiple variables at once).

With the Cox model, we model the impact of different factors $X_1, X_2, \ldots, X_q$ on survival via their impact on the hazard function:

\[h(t|\textbf{X}) = h_0 (t) exp(\beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_q X_q),\]

where:

$h(t|\textbf{X})$ is the instantaneous death rate conditional on having survived up to time $t$.
$h_0 (t)$ is the population-level baseline hazard – the underlying hazard function. It describes how the average person’s risk evolves over time.
$exp(\beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_q X_q)$ describes how covariates affect the hazard. In particular, a unit increase in $x_i$ leads to an increase of the hazard by a factor of $\exp(\beta_i)$.

This post aimed at presenting the introductory concepts in survival analysis, so this model will be developed in another post. In the meantime, if you would like to learn more about modeling survival data (thanks to the Cox model and other models), see this post from Joseph Rickert.

Thanks for reading.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Cox, David R. 1972. “Regression Models and Life-Tables.” Journal of the Royal Statistical Society: Series B (Methodological) 34 (2): 187–202.

Kaplan, Edward L, and Paul Meier. 1958. “Nonparametric Estimation from Incomplete Observations.” Journal of the American Statistical Association 53 (282): 457–81.

Note that in survival analysis, the precision of the estimators (and the power of the tests) does not depend on the number of patients but on the number of events. So it is best to have many observations where the event does occur for the analyses to be effective. Here we work on a small sample for the sake of illustration.↩︎
Note that the time variable can be expressed in other units, such as seconds, days, weeks, months, etc.↩︎
Median is preferred over mean in survival analysis because survival functions are often skewed to the right. The mean is often influenced by outliers, whereas the median is not. See a discussion comparing the two in this section.↩︎
Note that if the survival curve does not cross 50% (because survival is greater than 50% at the last time point), then the median survival cannot be computed and is simply undefined.↩︎
Note that the median survival is expressed in the same unit than the unit of the time variable in the initial dataset. So if the time unit was months, the median survival time would be 9 months.↩︎
More information about the dataset can be found on CRAN or with ?tongue.↩︎
See the reason we use log-log for the confidence interval in this thread.↩︎
Note that if the groups to compare have a natural ordering (such as the educational level; none, low, medium, high), tests that take it into consideration have more power to detect significant effects. These tests are referred as tests for trend.↩︎
Remember that the time unit can be different than years.↩︎
This is the case for large samples. The example described here does not meet this condition, but we still show it as an illustration.↩︎
Note that other weights can be considered, but this is beyond the scope of this article.↩︎
More information about the dataset can be found on CRAN or with ?burn.↩︎

One-sample Wilcoxon test in R

Thu, 07 Jul 2022 00:00:00 +0000

Introduction

In a previous article, we showed how to do a two-sample Wilcoxon test in R. Remember that there are actually two versions of this test:

The Mann-Whitney-Wilcoxon test (also referred as Wilcoxon rank sum test or Mann-Whitney U test), used to compare two independent samples. This test is the non-parametric version of the Student’s t-test for independent samples.
The Wilcoxon signed-rank test (also referred as Wilcoxon test for paired samples), used to compare two paired samples. This test is the non-parametric version of the Student’s t-test for paired samples.

In another article, we also showed how to do a one-sample t-test by hand and in R. This test is used to determine whether the mean of a measurement variable is different from a specified value (a value that you specify based on your beliefs or a theoretical expectation for example). Since it is a parametric test, the data should follow a normal distribution (or sample size should be large enough (i.e., above 30), thanks to the central limit theorem) for the results to be valid.

Unlike the one-sample t-test, the one-sample Wilcoxon test (also referred as the one-sample Wilcoxon signed-rank test) is a non-parametric test, meaning that it does not rely on data belonging to any particular parametric family of probability distributions. Non-parametric tests usually have the same goal as their parametric counterparts (in this case, compare data to a given value). Nonetheless, they do not require the assumption of normality and they can deal with outliers and Likert scales.

In this article, we show when to perform the one-sample Wilcoxon test, how to do it in R and how to interpret its results. We will also briefly show some appropriate visualizations.

When?

The one-sample Wilcoxon test is used to compare our observations to a given default value—a value that you specify based on your beliefs or a theoretical expectation for example. In other words, it is used to determine if a group is significantly different from a known or hypothesized population value on the variable of interest.

Since the test statistic is computed based on the ranks of the difference between the observed values and the default value (making it a non-parametric test), the one-sample Wilcoxon test is more appropriate than a one-sample t-test when the observations do not follow a normal distribution.

The goal of this test is to verify whether the observations are significantly different from our default value. In terms of null and alternative hypotheses, we have (for a two-tailed test):

$H_0:$ location of the data is equal to the chosen value
$H_1:$ location of the data is different from the chosen value

In other words, a significant result (i.e., a rejection of the null hypothesis) suggests that the location of the data is different from the chosen value.

Note that some authors suggest that this test is a test of the median, that is (for a two-tailed test):

$H_0:$ the median is equal to the chosen value
$H_1:$ the median is different from the chosen value

However, this is the case only if the data are symmetric. Without further assumptions about the distribution of the data, the one-sample Wilcoxon test is not a test of the median but a test about the location of the data.¹

Note that although the normality assumption is not required, the independence assumption must still be verified. This means that observations must be independent of one another (usually, random sampling is sufficient to have independence).

Data

For our illustration, suppose we want to test whether the scores at an exam differ from 10, that is:

$H_0:$ scores at the exam $= 10$
$H_1:$ scores at the exam $\ne 10$

To verify this, we have a sample of 15 students and their score at the exam:

dat

##    Student_ID Score
## 1           1    17
## 2           2     5
## 3           3     1
## 4           4    10
## 5           5     4
## 6           6    18
## 7           7    17
## 8           8    15
## 9           9     7
## 10         10     4
## 11         11     5
## 12         12    14
## 13         13    20
## 14         14    18
## 15         15    15

Scores between students are assumed to be independent (a student’s score is not impacted or influenced by the score of another student). Therefore, the independence assumption is met.

Moreover, sample size is small (n < 30) and based on the histogram the data do not follow a normal distribution:

# histogram
hist(dat$Score)

Note that we refrain from verifying the normality via a normality test (such as the Shapiro-Wilk test for instance) because for small sample sizes, normality tests have little power to reject the null hypothesis and therefore small samples most often pass normality tests (Öztuna et al. 2006; Ghasemi and Zahediasl 2012).

Note also that although we use a quantitative variable for the illustration, the one-sample Wilcoxon test is also appropriate for interval data and Likert scales.

How?

The one-sample Wilcoxon test can be done in R with the wilcox.test() function.

But first, it is a good practice to visualize our data in a boxplot and compute some descriptive statistics to compare our observations with our default value:

# boxplot
boxplot(dat$Score,
  ylab = "Score"
)

If like me you prefer to use the {ggplot2} package for your plots:

# boxplot
library(ggplot2)

ggplot(dat, aes(y = Score)) +
  geom_boxplot() +
  labs(y = "Score") +
  theme( # remove axis text and ticks
    axis.text.x = element_blank(),
    axis.ticks = element_blank()
  )

Some basic descriptive statistics (rounded to two decimals):

round(summary(dat$Score),
  digits = 2
)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    5.00   14.00   11.33   17.00   20.00

From the boxplot and the descriptive statistics above, we see that the mean and median of the scores in our sample are respectively 11.33 and 14.

The one-sample Wilcoxon test will tell us whether the scores are significantly different from 10 or not (and thus whether they are different from 10 in the population or not):

wilcox.test(dat$Score,
  mu = 10 # default value
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.3779
## alternative hypothesis: true location is not equal to 10

The output presents several information such as the:

title of the test
data
test statistic
$p$-value
alternative hypothesis

We focus on the $p$-value to interpret and conclude the test.

Interpretation:

Based on the results of the test, (at the significance level of 0.05) we do not reject the null hypothesis, so we do not reject the hypothesis that the scores at this exam are equal to 10, and we cannot conclude that the scores are significantly different from 10 ($p$-value = 0.378).

By default, it is a two-tailed test that is done. As for the t.test() function, we can specify that a one-sided test is required by using either the alternative = "greater" or alternative = "less argument in the wilcox.test() function.

For example, if we want to test that the scores are higher than 10:

wilcox.test(dat$Score,
  mu = 10, # default value
  alternative = "greater" # H1: scores > 10
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.189
## alternative hypothesis: true location is greater than 10

Interpretation:

In this case, we still do not reject the hypothesis that scores are equal to 10 and we cannot conclude that scores are significantly higher than 10 ($p$-value = 0.189).

For more information about the arguments available in the function, see ?wilcox.test.

Note that you may encounter the following warnings when using wilcox.test():

Warning messages:
1: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with ties
2: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with zeroes

It is a warning rather than an indication that your results are incorrect. R is informing you that it is reporting a $p$-value based on a normal approximation rather than an exact $p$-value based on the data because there are ties (some values are the same). Use the exact = FALSE option if you want to remove the warning.

Combine statistical test and plot

If you are a frequent user of the blog, you know that I like to present results of a test directly on a plot. This allows me to visualize the data and conclude the test in a concise manner.

This is possible thanks to the gghistostats() function within the {ggstatsplot} package:

# load package
library(ggstatsplot)

# combine plot and test
gghistostats(
  data = dat, # dataframe
  x = Score, # variable
  type = "nonparametric", # nonparametric = Wilcoxon, parametric = t-test
  test.value = 10 # default value
)

The histogram² shows the distribution of the scores and results of the test is shown in the title of the plot.

As you can see, results of the test are the same, that is, there is not enough evidence in the data to conclude that scores are significantly different from 10 ($p$-value = 0.378).

Conclusion

Thanks for reading.

I hope this article helped you to understand the one-sample Wilcoxon test and how to do it in R.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Ghasemi, Asghar, and Saleh Zahediasl. 2012. “Normality Tests for Statistical Analysis: A Guide for Non-Statisticians.” International Journal of Endocrinology and Metabolism 10 (2): 486.

Öztuna, Derya, Atilla Halil Elhan, and Ersöz Tüccar. 2006. “Investigation of Four Different Normality Tests in Terms of Type 1 Error Rate and Power Under Different Distributions.” Turkish Journal of Medical Sciences 36 (3): 171–76.

See more information in this article.↩︎
This histogram looks different than the previous one because the number of bins is different (4 versus 5 bins).↩︎

Kruskal-Wallis test, or the nonparametric version of the ANOVA

Thu, 24 Mar 2022 00:00:00 +0000

Introduction

In a previous article, we showed how to do an ANOVA in R to compare three or more groups.

Remember that, as for many statistical tests, the one-way ANOVA requires that some assumptions are satisfied in order to be able to use and interpret the results. In particular, the ANOVA requires that residuals follow approximately a normal distribution.¹

Luckily, if the normality assumption is not satisfied, there is the nonparametric version of the ANOVA: the Kruskal-Wallis test.

In the rest of the article, we show how to perform the Kruskal-Wallis test in R and how to interpret its results. We will also briefly show how to do post-hoc tests and how to present all necessary statistical results directly on a plot.

Data

Data for the present article is based on the penguins dataset (an alternative to the well-known iris dataset), accessible via the {palmerpenguins} package:

# install.packages("palmerpenguins")
library(palmerpenguins)

The original dataset contains data for 344 penguins of 3 different species (Adelie, Chinstrap and Gentoo).

It contains 8 variables, but we focus only on the flipper length and the species for this article, so we keep only those 2 variables:

library(tidyverse)

dat <- penguins %>%
  select(species, flipper_length_mm)

(If you are unfamiliar with the pipe operator (%>%), you can also select variables with penguins[, c("species", "flipper_length_mm")]. Learn more ways to select variables in the article about data manipulation.)

It is always a good practice to do some descriptive statistics for the entire sample and by group before doing the test, so we have a broad overview of the data at hand.

# entire sample
summary(dat)

##       species    flipper_length_mm
##  Adelie   :152   Min.   :172.0    
##  Chinstrap: 68   1st Qu.:190.0    
##  Gentoo   :124   Median :197.0    
##                  Mean   :200.9    
##                  3rd Qu.:213.0    
##                  Max.   :231.0    
##                  NA's   :2

# by group
library(doBy)
summaryBy(flipper_length_mm ~ species,
  data = dat,
  FUN = median,
  na.rm = TRUE
)

## # A tibble: 3 × 2
##   species   flipper_length_mm.median
##   <fct>                        <dbl>
## 1 Adelie                         190
## 2 Chinstrap                      196
## 3 Gentoo                         216

# boxplot by species
ggplot(dat) +
  aes(x = species, y = flipper_length_mm, fill = species) +
  geom_boxplot() +
  theme(legend.position = "none")

Based on the boxplots and the summary statistics, we already see that, in our sample, penguins from the Adelie species seem to have the smallest flippers, while those from the Gentoo species seem to have the biggest flippers. However, only a sound statistical test will tell us whether we can infer this conclusion to our population.

Kruskal-Wallis test

Aim and hypotheses

As mentioned earlier, the Kruskal-Wallis test allows to compare three or more groups. More precisely, it is used to compare three or more groups in terms of a quantitative variable.

It can be seen as the extension to the Mann-Whitney test which allows to compare 2 groups under the non-normality assumption.

In the context of our example, we are going to use the Kruskal-Wallis test to help us answer the following question: “Is the length of the flippers different between the 3 species of penguins?”.

The null and alternative hypotheses of the Kruskal-Wallis test are:

$H_0$: The 3 species are equal in terms of flipper length
$H_1$: At least one species is different from the other 2 species in terms of flipper length

Be careful that, as for the ANOVA, the alternative hypothesis is not that all species are different in terms of flipper length. The opposite of all species being equal ($H_0$) is that at least one species is different from the others ($H_1$).

In this sense, if the null hypothesis is rejected, it means that at least one species is different from the other 2, but not necessarily that all 3 species are different from each other. It could be that flipper length for the species Gentoo is different than for the species Chinstrap and Adelie, but flipper length is similar between Chinstrap and Adelie. Other types of test (known as post-hoc tests and covered later) must be performed to test whether all 3 species differ.

Assumptions

First, the Kruskal-Wallis test compares several groups in terms of a quantitative variable. So there must be one quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with at least 2 levels which will determine the groups to compare).²

Second, remember that the Kruskal-Wallis test is a nonparametric test, so the normality assumption is not required. However, the independence assumption still holds.

This means that the data, collected from a representative and randomly selected portion of the total population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if three measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before, (ii) during and (iii) after a treatment), the Friedman test should be preferred in order to take into account the dependency between the samples.

Regarding the homoscedasticity (i.e., equality of the variances): As long as you use the Kruskal-Wallis test to, in fine, compare groups, homoscedasticity is not required. If you wish to compare medians, the Kruskal-Wallis test requires homoscedasticity.³

In our example, independence is assumed and we do not need to compare medians (we are only interested in comparing groups), so we can proceed to how to do the test in R. Note that the normality assumption may or may not hold, but for this article we assume it is not satisfied.

In R

The Kruskal-Wallis test in R can be done with the kruskal.test() function:

kruskal.test(flipper_length_mm ~ species,
  data = dat
)

## 
## 	Kruskal-Wallis rank sum test
## 
## data:  flipper_length_mm by species
## Kruskal-Wallis chi-squared = 244.89, df = 2, p-value < 2.2e-16

The most important result in this output is the p-value. We show how to interpret it in the next section.

Interpretations

Based on the Kruskal-Wallis test, we reject the null hypothesis and we conclude that at least one species is different in terms of flippers length (p-value < 0.001).

(For the sake of illustration, if the p-value was larger than the significance level $\alpha = 0.05$: we cannot reject the null hypothesis so we cannot reject the hypothesis that the 3 considered species of penguins are equal in terms of flippers length.)

Post-hoc tests

We have just showed that at least one species is different from the others in terms of flippers length. Nonetheless, here comes the limitations of the Kruskal-Wallis test: it does not say which group(s) is(are) different from the others.

To know this, we need to use other types of test, referred as post-hoc tests (in Latin, “after this”, so after obtaining statistically significant Kruskal-Wallis results) or multiple pairwise-comparison tests. For the interested reader, a more detailed explanation of post-hoc tests can be found here.

The most common post-hoc tests after a significant Kruskal-Wallis test are:

Dunn test
Conover test
Nemenyi test
Pairwise Wilcoxon test

The Dunn test being the most common one, here is how to do it in R.⁴

Dunn test

library(FSA)

dunnTest(flipper_length_mm ~ species,
  data = dat,
  method = "holm"
)

##           Comparison          Z      P.unadj        P.adj
## 1 Adelie - Chinstrap  -3.629336 2.841509e-04 2.841509e-04
## 2    Adelie - Gentoo -15.476612 4.990733e-54 1.497220e-53
## 3 Chinstrap - Gentoo  -8.931938 4.186100e-19 8.372200e-19

It is the last column (the adjusted p-values, adjusted for multiple comparisons) that is of interest. These p-values should be compared to your desired significance level (usually 5%).

Based on the output, we conclude that:

Adelie and Chinstrap differ significantly (p < 0.001)
Adelie and Gentoo differ significantly (p < 0.001)
Chinstrap and Gentoo differ significantly (p < 0.001)

Therefore, based on the Dunn test, we can now conclude that all 3 species differ in terms of flipper length.

Combination of statistical results and plot

A very good alternative for performing a Kruskal-Wallis and the post-hoc tests in R is with the ggbetweenstats() function from the {ggstatsplot} package:

library(ggstatsplot)

ggbetweenstats(
  data = dat,
  x = species,
  y = flipper_length_mm,
  type = "nonparametric", # ANOVA or Kruskal-Wallis
  plot.type = "box",
  pairwise.comparisons = TRUE,
  pairwise.display = "significant",
  centrality.plotting = FALSE,
  bf.message = FALSE
)

This method has the advantage that all necessary statistical results are displayed directly on the plot.

The results of the Kruskal-Wallis test are shown in the subtitle above the plot (the p-value is after p =). Moreover, the results of the post-hoc test are displayed between each group via accolades, and the boxplots allow to visualize the distribution for each species.

Summary

In this post, we reviewed the aim and hypotheses of the Kruskal-Wallis test and its underlying assumptions. We then showed how to do the test in R and how to interpret the results.

We also showed the most common post-hoc test after a significant Kruskal-Wallis test—the Dunn test.

Last but not least, we presented a concise way to present both the data by group and all the statistical results on the same plot.

Thanks for reading.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

This is the case for small sample sizes. For large sample sizes, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). See more details in this note.↩︎
Note that in theory, Kruskal-Wallis test can also be used for only two groups. However, in practice we use the Mann-Whitney test for two groups and Kruskal-Wallis for three or more groups.↩︎
See more information about the difference in this article.↩︎
Note that there are other p-value adjustment methods. See ?dunnTest for more options.↩︎

What statistical test should I do?

Thu, 02 Dec 2021 00:00:00 +0000

Introduction

Being a teaching assistant in statistics for students with diverse backgrounds, I have the chance to see what is globally not well understood by students.

I have realized that it is usually not a problem for students to do a specific statistical test when they are told which one to use (as long as they have good resources and they have been attentive during classes, of course). However, it appears that the task is much more difficult for them when they need to choose what test to do.

This article presents a flowchart to help students in selecting the most appropriate statistical test based on a couple of criteria.¹

Flowchart

Due to the large number of tests, the image is quite wide so it may not render well on all screens. In that case, you can see it in full screen by clicking directly on the image or by clicking on the following link:

Download in PDF

Notes

As you can see in the flowchart, the selection of the most appropriate test is based on:

the number of variables of interest: one, two or more than two variables
the variable type: quantitative or qualitative
in case of a qualitative variable, the number of groups and whether they are independent or paired (i.e., dependent)
whether you want the parametric or nonparametric version²

Summarizing so many tests in a single image is not an easy task. The goal of this flowchart is to provide students with a quick and easy way to select the most appropriate statistical test among the most common ones (or to see what are the alternatives).

Obviously, this flowchart is not exhaustive. There are many other tests but most of them have been omitted on purpose to keep it simple and readable. I decided to keep it simple so that the flowchart is not overwhelming, with the hope that it is still complete and precise enough for most students.

For the sake of completeness, here are a few additional remarks about this flowchart:

Tests for more than 2 variables are applicable to the case of 2 variables as well. For simplicity, I however tend to suggest the simplest test when more than one is possible. For instance, with two quantitative variables, both a correlation test and a simple linear regression can be done. In introductory statistics classes, I will most likely teach the concept of correlation but not necessarily the concept of linear regression. For this reason, I will most likely recommend a correlation test over a linear regression, unless the students have a more advanced level.
The Kolmogorov-Smirnov test, used to compare a sample with a reference probability distribution or to compare two samples, has been omitted because it is generally not taught in introductory classes. Keep in mind, however, that this test is useful both in the uni and bivariate cases.
Normality tests (such as, among others, the Shapiro-Wilk or Kolmogorov-Smirnov test) have also been omitted as they are part of another family of tests (they are used to answer the question “Is my dataset well-modeled by a normal distribution?”). Remember, nonetheless, that they are very useful to verify the normality assumption required in many hypothesis tests. For example, the Student’s t-test requires that the data follow approximately a normal distribution in case of small sample size. If this is not the case, the nonparametric version (i.e., the Wilcoxon test) should be preferred. The same goes for ANOVA and many other statistical tests.
The flowchart could be extended to include more advanced linear or non-linear models, but this is beyond its scope and goal. Remember that I created it to help non-experts to see more clearly and have a broad overview of the most common statistical tests, not to confuse them even more.
If you open the flowchart in PDF, you will be able to click on most of the tests. Clicking on the name of the test will redirect you to the corresponding article, which explains the test in further details. If the test is not clickable, it means I have not written about it yet. I will update the flowchart if I publish an article about one of the missing test.

Conclusion

Thanks for reading.

I hope this guide will help you in determining the right statistical test. Feel free to share it with all students who might be interested.

As always, if you have a question or a suggestion (for example, if I missed a test which you believe should be included), please add it as a comment so other readers can benefit from the discussion.

For those interested to draw flowcharts like this one, note that I used diagrams.net. At the time of writing this article, it is free and you can synchronize it with Google Drive, GitHub, etc.↩︎
A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. On the contrary, a nonparametric test does not rely on data belonging to any particular parametric family of probability distributions. Nonparametric tests have the same objective as their parametric counterparts. However, they have two advantages over parametric tests: (i) they do not require the assumption of normality of distributions and (ii) they can deal with outliers. The trade-off is that nonparametric tests are usually less powerful than their corresponding parametric version when the normality assumption holds. Therefore, all else being equal, with a nonparametric test you are less likely to reject the null hypothesis when it is false if the data follow a normal distribution. It is thus preferred to use the parametric version when the assumptions are met.↩︎

Hypothesis test by hand

Wed, 27 Jan 2021 00:00:00 +0000

Descriptive versus inferential statistics

Remember that descriptive statistics is the branch of statistics aiming at describing and summarizing a set of data in the best possible manner, that is, by reducing it down to a few meaningful key measures and visualizations—with as little loss of information as possible. In other words, the branch of descriptive statistics helps to have a better understanding and a clear image about a set of observations thanks to summary statistics and graphics. With descriptive statistics, there is no uncertainty because we describe only the group of observations that we decided to work on and no attempt is made to generalize the observed characteristics to another or to a larger group of observations.

Inferential statistics, one the other hand, is the branch of statistics that uses a random sample of data taken from a population to make inferences, i.e., to draw conclusions about the population of interest (see the difference between population and sample if you need a refresh of the two concepts). In other words, information from the sample is used to make generalizations about the parameter of interest in the population.

The two most important tools used in the domain of inferential statistics are:

hypothesis test (which is the main subject of the present article), and
confidence interval (which is briefly discussed in this section)

Motivations and limitations

Via my teaching tasks, I realized that many students (especially in introductory statistic classes) struggle to perform hypothesis tests and interpret the results. It seems to me that these students often encounter difficulties mainly because hypothesis testing is rather unclear and abstract to them.

One of the reason it looks abstract to them is because they do not understand the final goal of hypothesis testing—the “why” behind this tool. They often do inferential statistics without understanding the reasoning behind it, as if they were following a cooking recipe which does not require any thinking. However, as soon as they understand the principle underlying hypothesis testing, it is much easier for them to apply the concepts and solve the exercises.

For this reason, I though it would be useful to write an article on the goal of hypothesis tests (the “why?”), in which context they should be used (the “when?”), how they work (the “how?”) and how to interpret the results (the “so what?”). Like anything else in statistics, it becomes much easier to apply a concept in practice when we understand what we are testing or what we are trying to demonstrate beforehand.

In this article, I present—as comprehensibly as possible—the different steps required to perform and conclude a hypothesis test by hand.

These steps are illustrated with a basic example. This will build the theoretical foundations of hypothesis testing, which will in turn be of great help for the understanding of most statistical tests.

Hypothesis tests come in many forms and can be used for many parameters or research questions. The steps I present in this article are not applicable to all hypothesis test, unfortunately.

They are however, appropriate for at least the most common hypothesis tests—the tests on:

One mean: $\mu$
Two means:
- independent samples: $\mu_1$ and $\mu_2$
- paired samples: $\mu_D$
One proportion: $p$
Two proportions: $p_1$ and $p_2$
One variance: $\sigma^2$
Two variances: $\sigma^2_1$ and $\sigma^2_2$

The good news is that the principles behind these 6 statistical tests (and many more) are exactly the same. So if you understand the intuition and the process for one of them, all others pretty much follow.

Hypothesis test

Why?

Unlike descriptive statistics where we only describe the data at hand, hypothesis tests use a subset of observations, referred as a sample, to draw conclusions about a population.

One may wonder why we would try to “guess” or make inference about a parameter of a population based on a sample, instead of simply collecting data for the entire population, compute statistics we are interested in and take decisions based upon that.

The main reason we actually use a sample instead of the entire population is because, most of the time, collecting data on the entire population is practically impossible, too complex, too expensive, it would take too long, or a combination of any of these.¹

So the overall objective of a hypothesis test is to draw conclusions in order to confirm or refute a belief about a population, based on a smaller group of observations.

In practice, we take some measurements of the variable of interest—representing the sample(s)—and we check whether our measurements are likely or not given our assumption (our belief). Based on the probability of observing the sample(s) we have, we decide whether we can trust our belief or not.

When?

Hypothesis tests have many practical applications.

Here are different situations illustrating when the 6 tests mentioned above would be appropriate:

One mean: suppose that a health professional would like to test whether the mean weight of Belgian adults is different than 80 kg (176.4 lbs).
Two means:
- Independent samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) for patients in a control group and patients in a treatment group, where patients in the two groups are different.
- Paired samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) before and after a treatment, where patients are measured twice—before and after treatment, so patients are the same in the 2 samples.
One proportion: suppose that a political pundit would like to test whether the proportion of citizens who are going to vote for a specific candidate is smaller than 30%.
Two proportions: suppose that a doctor would like to test whether the proportion of smokers is different between professional and amateur athletes.
One variance: suppose that an engineer would like to test whether a voltmeter has a lower variability than what is imposed by the safety standards.
Two variances: suppose that, in a factory, two production lines work independently from each other. The financial manager would like to test whether the costs of the weekly maintenance of these two machines have the same variance. Note that a test on two variances is also often performed to verify the assumption of equal variances, which is required for several other statistical tests, such as the Student’s t-test for instance.

Of course, this is a non-exhaustive list of potential applications and many research questions can be answered thanks to a hypothesis test.

One important point to remember is that in hypothesis testing we are always interested in the population and not in the sample. The sample is used for the aim of drawing conclusions about the population, so we always test in terms of the population.

Usually, hypothesis tests are used to answer research questions in confirmatory analyses. Confirmatory analyses refer to statistical analyses where hypotheses—deducted from theory—are defined beforehand (preferably before data collection). In this approach, the researcher has a specific idea about the variables under consideration and she is trying to see if her idea, specified as hypotheses, is supported by data.

On the other hand, hypothesis tests are rarely used in exploratory analyses.² Exploratory analyses aims to uncover possible relationships between the variables under investigation. In this approach, the researcher does not have any clear theory-driven assumptions or ideas in mind before data collection. This is the reason exploratory analyses are sometimes referred as hypothesis-generating analyses—they are used to create some hypotheses, which in turn may be tested via confirmatory analyses at a later stage.

How?

There are, to my knowledge, 3 different methods to perform a hypothesis tests:

Although the process for these 3 approaches may slightly differ, they all lead to the exact same conclusions. Using one method or another is, therefore, more often than not a matter of personal choice or a matter of context. See this section to know which method I use depending on the context.

I present the 3 methods in the following sections, starting with, in my opinion, the most comprehensive one when it comes to doing it by hand: comparing the test statistic with the critical value.

For the three methods, I will explain the required steps to perform a hypothesis test from a general point of view and illustrate them with the following situation:³

Suppose a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg.

Note that, as for most hypothesis tests, the test we are going to use as example below requires some assumptions. Since the aim of the present article is to explain a hypothesis test, we assume that all assumptions are met. For the interested reader, see the assumptions (and how to verify them) for this type of hypothesis test in the article presenting the one-sample t-test.

Method A: Comparing the test statistic with the critical value

Method A, which consists in comparing the test statistic with the critical value, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Finding the critical value
Concluding and interpreting the results

Each step is detailed below.

Step #1: Stating the null and alternative hypothesis

As discussed before, a hypothesis test first requires an idea, that is, an assumption about a phenomenon. This assumption, referred as hypothesis, is derived from the theory and/or the research question.

Since a hypothesis test is used to confirm or refute a prior belief, we need to formulate our belief so that there is a null and an alternative hypothesis. Those hypotheses must be mutually exclusive, which means that they cannot be true at the same time. This is step #1.

In the context of our scenario, the null and alternative hypothesis are thus:

Null hypothesis $H_0: \mu = 80$
Alternative hypothesis $H_1: \mu \ne 80$

When stating the null and alternative hypothesis, bear in mind the following three points:

We are always interested in the population and not in the sample. This is the reason $H_0$ and $H_1$ will always be written in terms of the population and not in terms of the sample (in this case, $\mu$ and not $\bar{x}$).
The assumption we would like to test is often the alternative hypothesis. If the researcher wanted to test whether the mean weight of Belgian adults was less than 80 kg, she would have stated $H_0: \mu = 80$ (or equivalently, $H_0: \mu \ge 80$) and $H_1: \mu < 80$.⁴ Do not mix the null with the alternative hypothesis, or the conclusions will be diametrically opposed!
The null hypothesis is often the status quo. For instance, suppose that a doctor wants to test whether the new treatment A is more efficient than the old treatment B. The status quo is that the new and old treatments are equally efficient. Assuming a larger value is better, she will then write $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A > \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B > 0$). On the opposite, if the lower the better, she would have written $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A < \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B < 0$).

Step #2: Computing the test statistic

The test statistic (often called t-stat) is, in some sense, a metric indicating how extreme the observations are compared to the null hypothesis. The higher the t-stat (in absolute value), the more extreme the observations are.

There are several formulas to compute the t-stat, with one formula for each type of hypothesis test—one or two means, one or two proportions, one or two variances. This means that there is a formula to compute the t-stat for a hypothesis test on one mean, another formula for a test on two means, another for a test on one proportion, etc.⁵

The only difficulty in this second step is to choose the appropriate formula. As soon as you know which formula to use based on the type of test, you simply have to apply it to the data. For the interested reader, see the different formulas to compute the t-stat for the most common tests in this Shiny app.

Luckily, formulas for hypothesis tests on one and two means, and one and two proportions follow the same structure.

Computing the test statistic for these tests is similar than scaling a random variable (a process also knows as “standardization” or “normalization”) which consists in subtracting the mean from that random variable, and dividing the result by the standard deviation:

\[Z = \frac{X - \mu}{\sigma}\]

For these 4 hypothesis tests (one/two means and one/two proportions), computing the test statistic is like scaling the estimator (computed from the sample) corresponding to the parameter of interest (in the population). So we basically subtract the target parameter from the point estimator and then divide the result by the standard error (which is equivalent to the standard deviation but for an estimator).

If this is unclear, here is how the test statistic (denoted $t_{obs}$) is computed in our scenario (assuming that the variance of the population is unknown):

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}}\]

where:

$\bar{x}$ is the sample mean (i.e., the estimator)
$\mu$ is the mean under the null hypothesis (i.e., the target parameter)
$s$ is the sample standard deviation
$n$ is the sample size
($\frac{s}{\sqrt{n}}$ is the standard error)

Notice the similarity between the formula of this test statistic and the formula used to standardize a random variable. This structure is the same for a test on two means, one proportion and two proportions, except that the estimator, the parameter and the standard error are, of course, slightly different for each type of test.

Suppose that in our case we have a sample mean of 71 kg ($\bar{x}$ = 71), a sample standard deviation of 13 kg ($s$ = 13) and a sample size of 10 adults ($n$ = 10). Remember that the population mean (the mean under the null hypothesis) is 80 kg ($\mu$ = 80).

The t-stat is thus:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Although formulas are different depending on which parameter you are testing, the value found for the test statistic gives us an indication on how extreme our observations are.

We keep this value of -2.189 in mind because it will be used again in step #4.

Step #3: Finding the critical value

Although the t-stat gives us an indication of how extreme our observations are, we cannot tell whether this “score of extremity” is too extreme or not based on its value only.

So, at this point, we cannot yet tell whether our data are too extreme or not. For this, we need to compare our t-stat with a threshold—referred as critical value—given by the probability distribution tables (and which can, of course, also be found with R).

In the same way that the formula to compute the t-stat is different for each parameter of interest, the underlying probability distribution—and thus the statistical table—on which the critical value is based is also different for each target parameter. This means that, in addition to choosing the appropriate formula to compute the t-stat, we also need to select the appropriate probability distribution depending on the parameter we are testing.

Luckily, there are only 4 different probability distributions for the 6 hypothesis tests covered in this article (one/two means, one/two proportions and one/two variances):

Standard Normal distribution:
- test on one and two means with known population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is known
- test on one and two proportions (given that some assumptions are met)
Student distribution:
- test on one and two means with unknown population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is unknown
Chi-square distribution:
- test on one variance
Fisher distribution:
- test on two variances

Each probability distribution also has its own parameters (up to two parameters for the 4 distribution considered here), defining its shape and/or location. Parameter(s) of a probability distribution can be seen as its DNA; meaning that the distribution is entirely defined by its parameter(s).

Taking our initial scenario—a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg—as example.

The underlying probability distribution of a test on one mean is either the standard Normal or the Student distribution, depending on whether the variance of the population (not sample variance!) is known or unknown:⁶

If the population variance is known $\rightarrow$ the standard Normal distribution is used
If the population variance is unknown $\rightarrow$ the Student distribution is used

If no population variance is explicitly given, you can assume that it is unknown since you cannot compute it based on a sample. If you could compute it, that would mean you have access to the entire population and there is, in this case, no point in performing a hypothesis test (you could simply use some descriptive statistics to confirm or refute your belief).

In our example, no population variance is specified so it is assumed to be unknown. We therefore use the Student distribution.

The Student distribution has one parameter which defines it; the number of degrees of freedom. The number of degrees of freedom depends on the type of hypothesis test. For instance, the number of degrees of freedom for a test on one mean is equal to the number of observations minus one ($n$ - 1). Without going too far into the details, the - 1 comes from the fact that there is one quantity which is estimated (i.e., the mean).⁷ The sample size being equal to 10 in our example, the degrees of freedom is equal to $n$ - 1 = 10 - 1 = 9.

There is only one last element missing to find the critical value: the significance level. The significance level, denoted $\alpha$, is the probability of wrongly rejecting the null hypothesis, so the probability of rejecting the null hypothesis although it is in reality true. In this sense, it is an error (type I error, as opposed to the type II error⁸) that we accept to deal with, in order to be able to draw conclusions about a population based on a subset of it.

As you may have read in many statistical textbooks, the significance level is very often set to 5%.⁹ In some fields (such as medicine or engineering, among others), the significance level is also sometimes set to 1% to decrease the error rate.

It is best to specify the significance level before performing a hypothesis test to avoid the temptation to set the significance level in accordance to the results (the temptation is even bigger when the results are on the edge of being significant). As I always tell my students, you cannot “guess” nor compute the significance level. Therefore, if it is not explicitly specified, you can safely assume it is 5%. In our case, we did not indicate it, so we take $\alpha$ = 5% = 0.05.

Furthermore, in our example, we want to test whether the mean weight of Belgian adults is different than 80 kg. Since we do not specify the direction of the test, it is a two-sided test. If we wanted to test that the mean weight was less than 80 kg ($H_1: \mu <$ 80) or greater than 80 kg ($H_1: \mu >$ 80), we would have done a one-sided test.

Make sure that you perform the correct test (two-sided or one-sided) because it has an impact on how to find the critical value (see more in the following paragraphs).

So now that we know the appropriate distribution (Student distribution), its parameter (degrees of freedom (df) = 9), the significance level ($\alpha$ = 0.05) and the direction (two-sided), we have all we need to find the critical value in the statistical tables:

By looking at the row df = 9 and the column $t_.025$ in the Student’s distribution table, we find a critical value of:

\[t_{n-1; \alpha / 2} = t_{9; 0.025} = 2.262\]

One may wonder why we take $t_{\alpha/2} = t_.025$ and not $t_\alpha = t_.05$ since the significance level is 0.05. The reason is that we are doing a two-sided test ($H_1: \mu \ne$ 80), so the error rate of 0.05 must be divided in 2 to find the critical value to the right of the distribution. Since the Student’s distribution is symmetric, the critical value to the left of the distribution is simply: -2.262.

Visually, the error rate of 0.05 is partitioned into two parts:

0.025 to the left of -2.262 and
0.025 to the right of 2.262

We keep in mind these critical values of -2.262 and 2.262 for the fourth and last step.

Note that the red shaded areas in the previous plot are also known as the rejection regions. More on that in the following section.

These critical values can also be found in R, thanks to the qt() function:

qt(0.025, df = 9, lower.tail = TRUE)

## [1] -2.262157

qt(0.025, df = 9, lower.tail = FALSE)

## [1] 2.262157

The qt() function is used for the Student’s distribution (q stands for quantile and t for Student). There are other functions accompanying the different distributions:

qnorm() for the Normal distribution
qchisq() for the Chi-square distribution
qf() for the Fisher distribution

Step #4: Concluding and interpreting the results

In this fourth and last step, all we have to do is to compare the test statistic (computed in step #2) with the critical values (found in step #3) in order to conclude the hypothesis test.

The only two possibilities when concluding a hypothesis test are:

Rejection of the null hypothesis
Non-rejection of the null hypothesis

In our example of adult weight, remember that:

the t-stat is -2.189
the critical values are -2.262 and 2.262

Also remember that:

the t-stat gives an indication on how extreme our sample is compared to the null hypothesis
the critical values are the threshold from which the t-stat is considered as too extreme

To compare the t-stat with the critical values, I always recommend to plot them:

These two critical values form the rejection regions (the red shaded areas):

from $- \infty$ to -2.262, and
from 2.262 to $\infty$

If the t-stat lies within one of the rejection region, we reject the null hypothesis. On the contrary, if the t-stat does not lie within any of the rejection region, we do not reject the null hypothesis.

As we can see from the above plot, the t-stat is less extreme than the critical value and therefore does not lie within any of the rejection region. In conclusion, we do not reject the null hypothesis that $\mu = 80$.

This is the conclusion in statistical terms but they are meaningless without proper interpretation. So it is a good practice to also interpret the result in the context of the problem:

At the 5% significance level, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

Why don’t we accept $H_0$?

From a more philosophical (but still very important) perspective, note that we wrote “we do not reject the null hypothesis” and “we do not reject the hypothesis that the mean weight of Belgian adults is equal to 80 kg”. We did not write “we accept the null hypothesis” nor “the mean weight of Belgian adults is 80 kg”.

The reason is due to the fact that, in hypothesis testing, we conclude something about the population based on a sample. There is, therefore, always some uncertainty and we cannot be 100% sure that our conclusion is correct.

Perhaps it is the case that the mean weight of Belgian adults is in reality different than 80 kg, but we failed to prove it based on the data at hand. It may be the case that if we had more observations, we would have rejected the null hypothesis (since all else being equal, a larger sample size implies a more extreme t-stat). Or, it may be the case that even with more observations, we would not have rejected the null hypothesis because the mean weight of Belgian adults is in reality close to 80 kg. We cannot distinguish between the two.

So we can just say that we did not find enough evidence against the hypothesis that the mean weight of Belgian adults is 80 kg, but we do not conclude that the mean is equal to 80 kg.

If the difference is still not clear to you, the following example may help. Suppose a person is suspected of having committed a crime. This person is either innocent—the null hypothesis—or guilty—the alternative hypothesis. In the attempt to know if the suspect committed the crime, the police collects as much information and proof as possible. This is similar to the researcher collecting data to form a sample. And then the judge, based on the collected evidence, decides whether the suspect is considered as innocent or guilty. If there is enough evidence that the suspect committed the crime, the judge will conclude that the suspect is guilty. In other words, she will reject the null hypothesis of the suspect being innocent because there are enough evidence that the suspect committed the crime.

This is similar to the t-stat being more extreme than the critical value: we have enough information (based on the sample) to say that the null hypothesis is unlikely because our data would be too extreme if the null hypothesis were true. Since the sample cannot be “wrong” (it corresponds to the collected data), the only remaining possibility is that the null hypothesis is in fact wrong. This is the reason we write “we reject the null hypothesis”.

On the other hand, if there is not enough evidence that the suspect committed the crime (or no evidence at all), the judge will conclude that the suspect is considered as not guilty. In other words, she will not reject the null hypothesis of the suspect being innocent. But even if she concludes that the suspect is considered as not guilty, she will never be 100% sure that he is really innocent.

It may be the case that:

the suspect did not commit the crime, or
the suspect committed the crime but the police was not able to collect enough information against the suspect.

In the former case the suspect is really innocent, whereas in the latter case the suspect is guilty but the police and the judge failed to prove it because they failed to find enough evidence against him. Similar to hypothesis testing, the judge has to conclude the case by considering the suspect not guilty, without being able to distinguish between the two.

This is the main reason we write “we do not reject the null hypothesis” or “we fail to reject the null hypothesis” (you may even read in some textbooks conclusion such as “there is no sufficient evidence in the data to reject the null hypothesis”), and we do not write “we accept the null hypothesis”.

I hope this metaphor helped you to understand the reason why we reject the null hypothesis instead of accepting it.

In the following sections, we present two other methods used in hypothesis testing.

These methods will result in the exact same conclusion: non-rejection of the null hypothesis, that is, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg. It is thus presented only if you prefer to use these methods over the first one.

Method B: Comparing the p-value with the significance level $\alpha$

Method B, which consists in computing the p-value and comparing this p-value with the significance level $\alpha$, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Computing the p-value
Concluding and interpreting the results

In this second method which uses the p-value, the first and second steps are similar than in the first method.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the test statistic

Remember that the formula for the t-stat is different depending on the type of hypothesis test (one or two means, one or two proportions, one or two variances). In our case of one mean with unknown variance, we have:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Step #3: Computing the p-value

The p-value is the probability (so it goes from 0 to 1) of observing a sample at least as extreme as the one we observed if the null hypothesis were true. In some sense, it gives you an indication on how likely your null hypothesis is. It is also defined as the smallest level of significance for which the data indicate rejection of the null hypothesis.

For more information about the p-value, I recommend reading this note about the p-value and the significance level $\alpha$.

Formally, the p-value is the area beyond the test statistic. Since we are doing a two-sided test, the p-value is thus the sum of the area above 2.189 and below -2.189.

Visually, the p-value is the sum of the two blue shaded areas in the following plot:

The p-value can computed with precision in R with the pt() function:

p_val <- pt(-2.189, df = 9, lower.tail = TRUE) + pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

# which is equivalent than:
p_val <- 2 * pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

The p-value is 0.0563, which indicates that there is a 5.63% chance to observe a sample at least as extreme as the one observed if the null hypothesis were true. This already gives us a hint on whether our t-stat is too extreme or not (and thus whether our null hypothesis is likely or not), but we formally conclude in step #4.

Like the qt() function to find the critical value, we use pt() to find the p-value because the underlying distribution is the Student’s distribution.

Use pnorm(), pchisq() and pf() for the Normal, Chi-square and Fisher distribution, respectively. See also this Shiny app to compute the p-value given a certain t-stat for most probability distributions.

If you do not have access to a computer (during exams for example) you will not be able to compute the p-value precisely, but you can bound it using the statistical table referring to your test.

In our case, we use the Student distribution and we look at the row df = 9 (since df = n - 1):

The test statistic is -2.189
We take the absolute value, which gives 2.189
The value 2.189 is between 1.833 and 2.262 (highlighted in blue in the above table)
From the column names $t_{.050}$ and $t_{.025}$ related to 1.833 and 2.262, we know that:
- the area to the right of 1.833 is 0.05
- the area to the right of 2.262 is 0.025
So we know that the area to the right of 2.189 must be between 0.025 and 0.05
Since the Student distribution is symmetric, we know that the area to the left of -2.189 must also be between 0.025 and 0.05
Therefore, the sum of the two areas must be between 0.05 and 0.10
In other words, the p-value is between 0.05 and 0.10 (i.e., 0.05 < p-value < 0.10)

Although we could not compute it precisely, it is enough to conclude our hypothesis test in the last step.

Step #4: Concluding and interpreting the results

The final step is now to simply compare the p-value (computed in step #3) with the significance level $\alpha$. As for all statistical tests:

If the p-value is smaller than $\alpha$ (p-value < 0.05) $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the p-value is greater than or equal to $\alpha$ (p-value $\ge$ 0.05) $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

No matter if we take into consideration the exact p-value (i.e., 0.0563) or the bounded one (0.05 < p-value < 0.10), it is larger than 0.05, so we do not reject the null hypothesis.¹⁰ In the context of the problem, we do not reject the null hypothesis that the mean weight of Belgian adults is 80 kg.

Remember that rejecting (or not rejecting) a null hypothesis at the significance level $\alpha$ using the critical value method (method A) is equivalent to rejecting (or not rejecting) the null hypothesis when the p-value is lower (equal or greater) than $\alpha$ (method B).

This is the reason we find the exact same conclusion than with method A, and why you should too if you use both methods on the same data and with the same significance level.

Method C: Comparing the target parameter with the confidence interval

Method C, which consists in computing the confidence interval and comparing this confidence interval with the target parameter (the parameter under the null hypothesis), boils down to the following 3 steps:

Stating the null and alternative hypothesis
Computing the confidence interval
Concluding and interpreting the results

In this last method which uses the confidence interval, the first step is similar than in the first two methods.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the confidence interval

Like hypothesis testing, confidence intervals are a well-known tool in inferential statistics.

Confidence interval is an estimation procedure which produces an interval (i.e., a range of values) containing the true parameter with a certain—usually high—probability.

In the same way that there is a formula for each type of hypothesis test when computing the test statistics, there exists a formula for each type of confidence interval. Formulas for the different types of confidence intervals can be found in this Shiny app.

Here is the formula for a confidence interval on one mean $\mu$ (with unknown population variance):

\[ (1-\alpha)\text{% CI for } \mu = \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \]

where $t_{\alpha/2, n - 1}$ is found in the Student distribution table (and is similar to the critical value found in step #3 of method A).

Given our data and with $\alpha$ = 0.05, we have:

\[ \begin{aligned} 95\text{% CI for } \mu &= \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \\ &= 71 \pm 2.262 \frac{13}{\sqrt{10}} \\ &= [61.70; 80.30] \end{aligned} \]

The 95% confidence interval for $\mu$ is [61.70; 80.30] kg. But what does a 95% confidence interval mean?

We know that this estimation procedure has a 95% probability of producing an interval containing the true mean $\mu$. In other words, if we construct many confidence intervals (with different samples of the same size), 95% of them will, on average, include the mean of the population (the true parameter). So on average, 5% of these confidence intervals will not cover the true mean.

If you wish to decrease this last percentage, you can decrease the significance level (set $\alpha$ = 0.01 or 0.02 for instance). All else being equal, this will increase the range of the confidence interval and thus increase the probability that it includes the true parameter.

Step #3: Concluding and interpreting the results

The final step is simply to compare the confidence interval (constructed in step #2) with the value of the target parameter (the value under the null hypothesis, mentioned in step #1):

If the confidence interval does not include the hypothesized value $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the confidence interval includes the hypothesized value $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

In our example:

the hypothesized value is 80 (since $H_0: \mu$ = 80)
80 is included in the 95% confidence interval since it goes from 61.70 to 80.30 kg
So we do not reject the null hypothesis

In the terms of the problem, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

As you can see, the conclusion is equivalent than with the critical value method (method A) and the p-value method (method B). Again, this must be the case since we use the same data and the same significance level $\alpha$ for all three methods.

Which method to choose?

All three methods give the same conclusion. However, each method has its own advantage so I usually select the most convenient one depending on the situation:

Method A (comparing the test statistic with the critical value):
- It is, in my opinion, the easiest and most straightforward method of the three when I do not have access to R.
Method B (comparing the p-value with the significance level $\alpha$):
- In addition to being able to know whether the null hypothesis is rejected or not, computing the exact p-value can be very convenient so I tend to use this method if I have access to R.
Method C (comparing the target parameter with the confidence interval):
- If I need to test several hypothesized values, I tend to choose this method because I can construct one single confidence interval and compare it to as many values as I want. For example, with our 95% confidence interval [61.70; 80.30], I know that any value below 61.70 kg and above 80.30 kg will be rejected, without testing it for each value.

Summary

In this article, we reviewed the goals and when hypothesis testing is used. We then showed how to do a hypothesis test by hand through three different methods (A. critical value, B. p-value and C. confidence interval). We also showed how to interpret the results in the context of the initial problem.

Although all three methods give the exact same conclusion when using the same data and the same significance level (otherwise there is a mistake somewhere), I also presented my personal preferences when it comes to choosing one method over the other two.

Thanks for reading.

I hope this article helped you to understand the structure of a hypothesis by hand. I remind you that, at least for the 6 hypothesis tests covered in this article, the formulas are different, but the structure and the reasoning behind it remain the same. So you basically have to know which formulas to use, and simply follow the steps mentioned in this article.

For the interested reader, I created two accompanying Shiny apps:

Hypothesis testing and confidence intervals: after entering your data, the app illustrates all the steps in order to conclude the test and compute a confidence interval. See more information in this article.
How to read statistical tables: the app helps you to compute the p-value given a t-stat for most probability distributions. See more information in this article.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Suppose a researcher wants to test whether Belgian women are taller than French women. Suppose a health professional would like to know whether the proportion of smokers is different among athletes and non-athletes. It would take way too long to measure the height of all Belgian and French women and to ask all athletes and non-athletes their smoking habits. So most of the time, decisions are based on a representative sample of the population and not on the whole population. If we could measure the entire population in a reasonable time frame, we would not do any inferential statistics.↩︎
Don’t get me wrong, this does not mean that hypothesis tests are never used in exploratory analyses. It is just much less frequent in exploratory research than in confirmatory research.↩︎
You may see more or less steps in other articles or textbooks, depending on whether these steps are detailed or concise. Hypothesis testing should, however, follows the same process regardless of the number of steps.↩︎
For one-sided tests, writing $H_0: \mu = 80$ or $H_0: \mu \ge 80$ are both correct. The point is that the null and alternative hypothesis must be mutually exclusive since you are testing one hypothesis against the other, so both cannot be true at the same time.↩︎
To be complete, there are even different formulas within each type of test, depending on whether some assumptions are met or not. For the interested reader, see all the different scenarios and thus the different formulas for a test on one mean and on two means.↩︎
There are more uncertainty if the population variance is unknown than if it is known, and this greater uncertainty is taken into account by using the Student distribution instead of the standard Normal distribution. Also note that as the sample size increases, the degrees of freedom of the Student distribution increases and the two distributions become more and more similar. For large sample size (usually from $n >$ 30), the Student distribution becomes so close to the standard Normal distribution that, even if the population variance is unknown, the standard Normal distribution can be used.↩︎
For a test on two independent samples, the degrees of freedom is $n_1 + n_2 - 2$, where $n_1$ and $n_2$ are the size of the first and second sample, respectively. Note the - 2 due to the fact that in this case, two quantities are estimated.↩︎
The type II error is the probability of not rejecting the null hypothesis although it is in reality false.↩︎
Whether this is a good or a bad standard is a question that comes up often and is debatable. This is, however, beyond the scope of the article.↩︎
Again, p-values found via a statistical table or via R must be coherent.↩︎

ANOVA in R

Mon, 12 Oct 2020 00:00:00 +0000

Introduction

ANOVA (ANalysis Of VAriance) is a statistical test to determine whether two or more population means are different. In other words, it is used to compare two or more groups to see if they are significantly different.

In practice, however, the:

Student t-test is used to compare 2 groups;
ANOVA generalizes the t-test beyond 2 groups, so it is used to compare 3 or more groups.

Note that there are several versions of the ANOVA (e.g., one-way ANOVA, two-way ANOVA, mixed ANOVA, repeated measures ANOVA, etc.). In this article, we present the simplest form only—the one-way ANOVA¹—and we refer to it as ANOVA in the remaining of the article.

Although ANOVA is used to make inference about means of different groups, the method is called “analysis of variance”. It is called like this because it compares the “between” variance (the variance between the different groups) and the variance “within” (the variance within each group). If the between variance is significantly larger than the within variance, the group means are declared to be different. Otherwise, we cannot conclude one way or the other. The two variances are compared to each other by taking the ratio ($\frac{variance_{between}}{variance_{within}}$) and then by comparing this ratio to a threshold from the Fisher probability distribution (a threshold based on a specific significance level, usually 5%).

This is enough theory regarding the ANOVA method for now. In the remaining of this article, we discuss about it from a more practical point of view, and in particular we will cover the following points:

the aim of the ANOVA, when it should be used and the null/alternative hypothesis
the underlying assumptions of the ANOVA and how to check them
how to perform the ANOVA in R
how to interpret results of the ANOVA
understand the notion of post-hoc test and interpret the results
how to visualize results of ANOVA and post-hoc tests

Data

Data for the present article is the penguins dataset (an alternative to the well-known iris dataset), accessible via the {palmerpenguins} package:

# install.packages("palmerpenguins")
library(palmerpenguins)

The dataset contains data for 344 penguins of 3 different species (Adelie, Chinstrap and Gentoo). The dataset contains 8 variables, but we focus only on the flipper length and the species for this article, so we keep only those 2 variables:

library(tidyverse)

dat <- penguins %>%
  select(species, flipper_length_mm)

Below some basic descriptive statistics and a plot (made with the {ggplot2} package) of our dataset before we proceed to the goal of the ANOVA:

summary(dat)

##       species    flipper_length_mm
##  Adelie   :152   Min.   :172.0    
##  Chinstrap: 68   1st Qu.:190.0    
##  Gentoo   :124   Median :197.0    
##                  Mean   :200.9    
##                  3rd Qu.:213.0    
##                  Max.   :231.0    
##                  NA's   :2

Flipper length varies from 172 to 231 mm, with a mean of 200.9 mm. There are respectively 152, 68 and 124 penguins of the species Adelie, Chinstrap and Gentoo.

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm, color = species) +
  geom_jitter() +
  theme(legend.position = "none")

Here, the factor is the species variable which contains 3 modalities or groups (Adelie, Chinstrap and Gentoo).

Aim and hypotheses of ANOVA

As mentioned in the introduction, the ANOVA is used to compare groups (in practice, 3 or more groups). More generally, it is used to:

study whether measurements are similar across different modalities (also called levels or treatments in the context of ANOVA) of a categorical variable
compare the impact of the different levels of a categorical variable on a quantitative variable
explain a quantitative variable based on a qualitative variable

In this context and as an example, we are going to use an ANOVA to help us answer the question: “Is the length of the flippers different between the 3 species of penguins?”.

The null and alternative hypothesis of an ANOVA are:

$H_0$: $\mu_{Adelie} = \mu_{Chinstrap} = \mu_{Gentoo}$ ($\Rightarrow$ the 3 species are equal in terms of flipper length)
$H_1$: at least one mean is different ($\Rightarrow$ at least one species is different from the other 2 species in terms of flipper length)

Be careful that the alternative hypothesis is not that all means are different. The opposite of all means being equal ($H_0$) is that at least one mean is different from the others ($H_1$).

In this sense, if the null hypothesis is rejected, it means that at least one species is different from the other 2, but not necessarily that all 3 species are different from each other. It could be that flipper length for the species Gentoo is different than for the species Chinstrap and Adelie, but flipper length is similar between Chinstrap and Adelie. Other types of test (known as post-hoc tests and covered in this section) must be performed to test whether all 3 species differ.

Underlying assumptions of ANOVA

As for many statistical tests, there are some assumptions that need to be met in order to be able to interpret the results. When one or several assumptions are not met, although it is technically possible to perform these tests, it would be incorrect to interpret the results and trust the conclusions.

Below are the assumptions of the ANOVA, how to test them and which other tests exist if an assumption is not met:

Variable type: ANOVA requires a mix of one continuous quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with at least 2 levels which will determine the groups to compare).
Independence: the data, collected from a representative and randomly selected portion of the total population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if three measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before, (ii) during and (iii) after a treatment), the repeated measures ANOVA should be preferred in order to take into account the dependency between the samples.
Normality:
- In case of small samples, residuals² should follow approximately a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. If, even after a transformation of your data (e.g., logarithmic transformation, square root, Box-Cox, etc.), the residuals still do not follow approximately a normal distribution, the Kruskal-Wallis test can be applied (kruskal.test(variable ~ group, data = dat in R). This non-parametric test, robust to non normal distributions, has the same goal than the ANOVA—compare 3 or more groups—but it uses sample medians instead of sample means to compare groups.
- In case of large samples, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013).³ It is therefore not required to test the normality assumption when the number of observations in each group/sample is large (usually $n \ge 30$).
Equality of variances: the variances of the different groups should be equal in the populations (an assumption called homogeneity of the variances, or even sometimes referred as homoscedasticity, as opposed to heteroscedasticity if variances are different across groups). This assumption can be tested graphically (by comparing the dispersion in a boxplot or dotplot for instance), or more formally via the Levene’s test (leveneTest(variable ~ group) from the {car} package) or Bartlett’s test, among others. If the hypothesis of equal variances is rejected, another version of the ANOVA can be used: the Welch ANOVA (oneway.test(variable ~ group, var.equal = FALSE)). Note that the Welch ANOVA does not require homogeneity of the variances, but the distributions should still follow approximately a normal distribution. Note that the Kruskal-Wallis test does not require the assumptions of normality nor homoscedasticity of the variances.⁴
Outliers: An outlier is a value or an observation that is distant from the other observations. There should be no significant outliers in the different groups, or the conclusions of your ANOVA may be flawed. There are several methods to detect outliers in your data but in order to deal with them, it is your choice to either:
- use the non-parametric version (i.e., the Kruskal-Wallis test)
- transform your data (logarithmic or Box-Cox transformation, among others)
- or remove them (be careful)

Choosing the appropriate test depending on whether assumptions are met may be confusing so here is a brief summary:

Check that your observations are independent.
Sample sizes:
- In case of small samples, test the normality of residuals:
  - If normality is assumed, test the homogeneity of the variances:
    - If variances are equal, use ANOVA.
    - If variances are not equal, use the Welch ANOVA.
  - If normality is not assumed, use the Kruskal-Wallis test.
- In case of large samples normality is assumed, so test the homogeneity of the variances:
  - If variances are equal, use ANOVA.
  - If variances are not equal, use the Welch ANOVA.

Now that we have seen the underlying assumptions of the ANOVA, we review them specifically for our dataset before applying the appropriate version of the test.

Variable type

The dependent variable flipper_length_mm is a quantitative variable and the independent variable species is a qualitative one (with 3 levels corresponding to the 3 species). So we have a mix of the two types of variable and this assumption is met.

Independence

Independence of the observations is assumed as data have been collected from a randomly selected portion of the population and measurements within and between the 3 samples are not related.

The independence assumption is most often verified based on the design of the experiment and on the good control of experimental conditions, as it is the case here.

If you really want to test it more formally, you can, however, test it via a statistical test—the Durbin-Watson test (in R: durbinWatsonTest(res_lm) where res_lm is a linear model). The null hypothesis of this test specifies an autocorrelation coefficient = 0, while the alternative hypothesis specifies an autocorrelation coefficient $\ne$ 0.

Normality

Since the smallest sample size per group (i.e., per species) is 68, we have large samples. Therefore, we do not need to check normality.

Usually, we would directly test the homogeneity of the variances without testing normality. However, for the sake of illustration, we act as if the sample sizes were small in order to illustrate what would need to be done in that case.

Remember that normality of residuals can be tested visually via a histogram and a QQ-plot, and/or formally via a normality test (Shapiro-Wilk test for instance).

Before checking the normality assumption, we first need to compute the ANOVA (more on that in this section). We then save the results in res_aov :

res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

We can now check normality visually:

par(mfrow = c(1, 2)) # combine plots

# histogram
hist(res_aov$residuals)

# QQ-plot
library(car)
qqPlot(res_aov$residuals,
  id = FALSE # id = FALSE to remove point identification
)

From the histogram and QQ-plot above, we can already see that the normality assumption seems to be met. Indeed, the histogram roughly form a bell curve, indicating that the residuals follow a normal distribution. Furthermore, points in the QQ-plots roughly follow the straight line and most of them are within the confidence bands, also indicating that residuals follow approximately a normal distribution.

Some researchers stop here and assume that normality is met, while others also test the assumption via a formal normality test. It is your choice to test it (i) only visually, (ii) only via a normality test, or (iii) both visually AND via a normality test. Bear in mind, however, the two following points:

ANOVA is quite robust to small deviations from normality. This means that it is not an issue (from the perspective of the interpretation of the ANOVA results) if a small number of points deviates slightly from the normality,
normality tests are sometimes quite conservative, meaning that the null hypothesis of normality may be rejected due to a limited deviation from normality. This is especially the case with large samples as power of the test increases with the sample size.

In practice, I tend to prefer the (i) visual approach only, but again, this is a matter of personal choice and also depends on the context of the analysis.

Still for the sake of illustration, we also now test the normality assumption via a normality test. You can use the Shapiro-Wilk test or the Kolmogorov-Smirnov test, among others.

Remember that the null and alternative hypothesis of these tests are:

$H_0$: data come from a normal distribution
$H_1$: data do not come from a normal distribution

In R, we can test normality of the residuals with the Shapiro-Wilk test thanks to the shapiro.test() function:

shapiro.test(res_aov$residuals)

## 
## 	Shapiro-Wilk normality test
## 
## data:  res_aov$residuals
## W = 0.99452, p-value = 0.2609

P-value of the Shapiro-Wilk test on the residuals is larger than the usual significance level of $\alpha = 5\%$, so we do not reject the hypothesis that residuals follow a normal distribution (p-value = 0.261).

This result is in line with the visual approach. In our case, the normality assumption is thus met both visually and formally.

Side note: Remind that the p-value is the probability of having observations as extreme as the ones we have observed in the sample(s) given that the null hypothesis is true. If the p-value $< \alpha$ (indicating that it is not likely to observe the data we have in the sample given that the null hypothesis is true), the null hypothesis is rejected, otherwise the null hypothesis is not rejected. See more about p-value and significance level if you are unfamiliar with those important statistical concepts.

Remember that if the normality assumption was not reached, some transformation(s) would need to be applied on the raw data in the hope that residuals would better fit a normal distribution, or you would need to use the non-parametric version of the ANOVA—the Kruskal-Wallis test.

As pointed out by a reader (see comments at the very end of the article), the normality assumption can also be tested on the “raw” data (i.e., the observations) instead of the residuals. However, if you test the normality assumption on the raw data, it must be tested for each group separately as the ANOVA requires normality in each group.

Testing normality on all residuals or on the observations per group is equivalent, and will give similar results. Indeed, saying “The distribution of Y within each group is normally distributed” is the same as saying “The residuals are normally distributed”.

Remember that residuals are the distance between the actual value of Y and the mean value of Y for a specific value of X, so the grouping variable is induced in the computation of the residuals.

So in summary, in ANOVA you actually have two options for testing normality:

Checking normality separately for each group on the “raw” data (Y values)
Checking normality on all residuals (but not per group)

In practice, you will see that it is often easier to just use the residuals and check them all together, especially if you have many groups or few observations per group.

If you are still not convinced: remember that an ANOVA is a special case of a linear model. Suppose your independent variable is a continuous variable (instead of a categorical variable), the only option you have left is to check normality on the residuals, which is precisely what is done for testing normality in linear regression models.

Equality of variances - homogeneity

Assuming residuals follow a normal distribution, it is now time to check whether the variances are equal across species or not. The result will have an impact on whether we use the ANOVA or the Welch ANOVA.

This can again be verified visually—via a boxplot or dotplot—or more formally via a statistical test (Levene’s test, among others).

Visually, we have:

# Boxplot
boxplot(flipper_length_mm ~ species,
  data = dat
)

# Dotplot
library("lattice")

dotplot(flipper_length_mm ~ species,
  data = dat
)

Both the boxplot and the dotplot show a similar variance for the different species. In the boxplot, this can be seen by the fact that the boxes and the whiskers have a comparable size for all species.

There are a couple of outliers as shown by the points outside the whiskers, but this does not change the fact that the dispersion is more or less the same between the different species.

In the dotplot, this can be seen by the fact that points for all 3 species have more or less the same range, a sign of the dispersion and thus the variance being similar.

Like the normality assumption, if you feel that the visual approach is not sufficient, you can formally test for equality of the variances with a Levene’s or Bartlett’s test. Notice that the Levene’s test is less sensitive to departures from normal distribution than the Bartlett’s test.

The null and alternative hypothesis for both tests are:

$H_0$: variances are equal
$H_1$: at least one variance is different

In R, the Levene’s test can be performed thanks to the leveneTest() function from the {car} package:

# Levene's test
library(car)

leveneTest(flipper_length_mm ~ species,
  data = dat
)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   2  0.3306 0.7188
##       339

The p-value being larger than the significance level of 0.05, we do not reject the null hypothesis, so we cannot reject the hypothesis that variances are equal between species (p-value = 0.719).

This result is also in line with the visual approach, so the homogeneity of variances is met both visually and formally.

Another method to test normality and homogeneity

For your information, it is also possible to test the homogeneity of the variances and the normality of the residuals visually (and both at the same time) via the plot() function:

par(mfrow = c(1, 2)) # combine plots

# 1. Homogeneity of variances
plot(res_aov, which = 3)

# 2. Normality
plot(res_aov, which = 2)

Plot on the left hand side shows that there is no evident relationships between residuals and fitted values (the mean of each group), so homogeneity of variances is assumed. If homogeneity of variances was violated, the red line would not be flat (horizontal).

Plot on the right hand side shows that residuals follow approximately a normal distribution, so normality is assumed. If normality was violated, points would consistently deviate from the dashed line.

Outliers

There are several techniques to detect outliers. In this article, we focus on the most simple one (yet very efficient)—the visual approach via a boxplot:

boxplot(flipper_length_mm ~ species,
  data = dat
)

There is one outlier in the group Adelie, as defined by the interquartile range criterion. This point is, however, not seen as a significant outlier so we can assume that the assumption of no significant outliers is met.

ANOVA

We showed that all assumptions of the ANOVA are met.

We can thus proceed to the implementation of the ANOVA in R, but first, let’s do some preliminary analyses to better understand the research question.

Preliminary analyses

A good practice before actually performing the ANOVA in R is to visualize the data in relation to the research question. The best way to do so is to draw and compare boxplots of the quantitative variable flipper_length_mm for each species.

This can be done with the boxplot() function in base R (same code than the visual check of equal variances):

boxplot(flipper_length_mm ~ species,
  data = dat
)

Or with the {ggplot2} package:

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm) +
  geom_boxplot()

The boxplots above show that, at least for our sample, penguins of the species Gentoo seem to have the biggest flipper, and Adelie species the smallest flipper.

Besides a boxplot for each species, it is also a good practice to compute some descriptive statistics such as the mean and standard deviation by species.

This can be done, for instance, with the aggregate() function:

aggregate(flipper_length_mm ~ species,
  data = dat,
  function(x) round(c(mean = mean(x), sd = sd(x)), 2)
)

##     species flipper_length_mm.mean flipper_length_mm.sd
## 1    Adelie                 189.95                 6.54
## 2 Chinstrap                 195.82                 7.13
## 3    Gentoo                 217.19                 6.48

or with the summarise() and group_by() functions from the {dplyr} package:

library(dplyr)

group_by(dat, species) %>%
  summarise(
    mean = mean(flipper_length_mm, na.rm = TRUE),
    sd = sd(flipper_length_mm, na.rm = TRUE)
  )

## # A tibble: 3 × 3
##   species    mean    sd
##   <fct>     <dbl> <dbl>
## 1 Adelie     190.  6.54
## 2 Chinstrap  196.  7.13
## 3 Gentoo     217.  6.48

Mean is also the lowest for Adelie and highest for Gentoo. Boxplots and descriptive statistics are, however, not enough to conclude that flippers are significantly different in the 3 populations of penguins.

ANOVA in R

As you guessed by now, only the ANOVA can help us to make inference about the population given the sample at hand, and help us to answer the initial research question “Is the length of the flippers different between the 3 species of penguins?”.

ANOVA in R can be done in several ways, of which two are presented below:

With the oneway.test() function:

# 1st method:
oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = TRUE # assuming equal variances
)

## 
## 	One-way analysis of means
## 
## data:  flipper_length_mm and species
## F = 594.8, num df = 2, denom df = 339, p-value < 2.2e-16

With the summary() and aov() functions:

# 2nd method:
res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

summary(res_aov)

##              Df Sum Sq Mean Sq F value Pr(>F)    
## species       2  52473   26237   594.8 <2e-16 ***
## Residuals   339  14953      44                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 2 observations deleted due to missingness

As you can see from the two outputs above, the test statistic (F = in the first method and F value in the second one) and the p-value (p-value in the first method and Pr(>F) in the second one) are exactly the same for both methods, which means that in case of equal variances, results and conclusions will be unchanged.

The advantage of the first method is that it is easy to switch from the ANOVA (used when variances are equal) to the Welch ANOVA (used when variances are unequal). This can be done by replacing var.equal = TRUE by var.equal = FALSE, as presented below:

oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = FALSE # assuming unequal variances
)

## 
## 	One-way analysis of means (not assuming equal variances)
## 
## data:  flipper_length_mm and species
## F = 614.01, num df = 2.00, denom df = 172.76, p-value < 2.2e-16

The advantage of the second method, however, is that:

the full ANOVA table (with degrees of freedom, mean squares, etc.) is printed, which may be of interest in some (theoritical) cases
results of the ANOVA (res_aov) can be saved for later use (especially useful for post-hoc tests)

Interpretations of ANOVA results

Given that the p-value is smaller than 0.05, we reject the null hypothesis, so we reject the hypothesis that all means are equal. Therefore, we can conclude that at least one species is different than the others in terms of flippers length (p-value < 2.2e-16).

(For the sake of illustration, if the p-value was larger than 0.05: we cannot reject the null hypothesis that all means are equal, so we cannot reject the hypothesis that the 3 considered species of penguins are equal in terms of flippers length.)

A nice and easy way to report results of an ANOVA in R is with the report() function from the {report} package:

# install.packages("remotes")
# remotes::install_github("easystats/report") # You only need to do that once
library("report") # Load the package every time you start R

report(res_aov)

## The ANOVA (formula: flipper_length_mm ~ species) suggests that:
## 
##   - The main effect of species is statistically significant and large (F(2, 339)
## = 594.80, p < .001; Eta2 = 0.78, 95% CI [0.75, 1.00])
## 
## Effect sizes were labelled following Field's (2013) recommendations.

As you can see, the function interprets the results for you and indicates a large and significant main effect of the species on the flipper length (p-value < .001).

Note that the report() function can be used for other analyses. See more tips and tricks in R if you find this one useful.

What’s next?

If the null hypothesis is not rejected (p-value $\ge$ 0.05), it means that we do not reject the hypothesis that all groups are equal. The ANOVA more or less stops here.

Other types of analyses can be performed of course, but—given the data at hand—we could not prove that at least one group was different so we usually do not go further with the ANOVA.

On the contrary, if the null hypothesis is rejected (as it is our case since the p-value < 0.05), we proved that at least one group is different. We can decide to stop here if we are only interested to test whether all species are equal in terms of flippers length.

But most of the time, when we showed thanks to an ANOVA that at least one group is different, we are also interested in knowing which one(s) is(are) different. Results of an ANOVA, however, do NOT tell us which group(s) is(are) different from the others.

To test this, we need to use other types of test, referred as post-hoc tests (in Latin, “after this”, so after obtaining statistically significant ANOVA results) or multiple pairwise-comparison tests.⁵

This family of statistical tests is the topic of the following sections.

Post-hoc test

Issue of multiple testing

In order to see which group(s) is(are) different from the others, we need to compare groups 2 by 2. In practice, since there are 3 species, we are going to compare species 2 by 2 as follows:

Chinstrap versus Adelie
Gentoo vs. Adelie
Gentoo vs. Chinstrap

In theory, we could compare species thanks to 3 Student’s t-tests since we need to compare 2 groups and a t-test is used precisely in that case.

However, if several t-tests are performed, the issue of multiple testing (also referred as multiplicity) arises. In short, when several statistical tests are performed, some will have p-values less than $\alpha$ purely by chance, even if all null hypotheses are in fact true.

To demonstrate the problem, consider our case where we have 3 hypotheses to test and a desired significance level of 0.05.

The probability of observing at least one significant result (at least one p-value < 0.05) just due to chance is:

\[\begin{equation} \begin{split} P(\text{at least 1 sig. result}) & = 1 - P(\text{no sig. results}) \\ & = 1 - (1 - 0.05)^3 \\ & = 0.142625 \end{split} \end{equation}\]

So, with as few as 3 tests being considered, we already have a 14.26% chance of observing at least one significant result, even if all of the tests are actually not significant.

And as the number of groups increases, the number of comparisons increases as well, so the probability of having a significant result simply due to chance keeps increasing.

For example, with 10 groups we need to make 45 comparisons and the probability of having at least one significant result by chance becomes $1 - (1 - 0.05)^{45} = 90\%$. So it is very likely to observe a significant result just by chance when comparing 10 groups, and when we have 14 groups or more we are almost certain (99%) to have a false positive!

Post-hoc tests take into account that multiple tests are done and deal with the problem by adjusting $\alpha$ in some way, so that the probability of observing at least one significant result due to chance remains below our desired significance level.⁶

Post-hoc tests in R and their interpretation

Post-hoc tests are a family of statistical tests so there are several of them. The most common ones are:

Tukey HSD, used to compare all groups to each other (so all possible comparisons of 2 groups).
Dunnett, used to make comparisons with a reference group. For example, consider 2 treatment groups and one control group. If you only want to compare the 2 treatment groups with respect to the control group, and you do not want to compare the 2 treatment groups to each other, the Dunnett’s test is preferred.
Bonferroni correction if one has a set of planned comparisons to do.

The Bonferroni correction is simple: you simply divide the desired global $\alpha$ level by the number of comparisons.

In our example, we have 3 comparisons so if we want to keep a global $\alpha = 0.05$, we have $\alpha' = \frac{0.05}{3} = 0.0167$. We can then simply perform a Student’s t-test for each comparison, and compare the obtained $p$-values with this new $\alpha'$.

The other two post-hoc tests are presented in the next sections.

Note that variances are assumed to be equal for all three methods (unless you use the Welch’s t-test instead of the Student’s t-test with the Bonferroni correction). If variances are not equal, you can use the Games-Howell test, among others.

Tukey HSD test

In our case, since there is no “reference” species and we are interested in comparing all species, we are going to use the Tukey HSD test.

In R, the Tukey HSD test is done as follows. This is where the second method to perform the ANOVA comes handy because the results (res_aov) are reused for the post-hoc test:

library(multcomp)

# Tukey HSD test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Tukey")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 1.03e-08 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-08 ***
## Gentoo - Chinstrap == 0  21.3635     1.0036  21.286  < 1e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

In the output of the Tukey HSD test, we are interested in the table displayed after Linear Hypotheses:, and more precisely, in the first and last column of the table. The first column shows the comparisons which have been made; the last column (Pr(>|t|)) shows the adjusted⁷ p-values for each comparison (with the null hypothesis being the two groups are equal and the alternative hypothesis being the two groups are different).

It is these adjusted p-values that are used to test whether two groups are significantly different or not, and we can be confident that the entire set of comparisons collectively has an error rate of 0.05.

In our example, we tested:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)
Gentoo vs. Chinstrap (line Gentoo - Chinstrap == 0)

All three ajusted p-values are smaller than 0.05, so we reject the null hypothesis for all comparisons, which means that all species are significantly different in terms of flippers length.

The results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that all groups are significantly different.

Note that the Tukey HSD test can also be done in R with the TukeyHSD() function:

TukeyHSD(res_aov)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## $species
##                       diff       lwr       upr p adj
## Chinstrap-Adelie  5.869887  3.586583  8.153191     0
## Gentoo-Adelie    27.233349 25.334376 29.132323     0
## Gentoo-Chinstrap 21.363462 19.000841 23.726084     0

With this code, it is the column p adj (also the last column) which is of interest. Notice that the conclusions are the same than above: all species are significantly different in terms of flippers length.

The results can also be visualized with the plot() function:

plot(TukeyHSD(res_aov))

Dunnett’s test

We have seen in this section that as the number of groups increases, the number of comparisons also increases. And as the number of comparisons increases, the post-hoc analysis must lower the individual significance level even further, which leads to lower statistical power (so a difference between group means in the population is less likely to be detected).

One method to mitigate this and increase the statistical power is by reducing the number of comparisons. This reduction allows the post-hoc procedure to use a larger individual error rate to achieve the desired global error rate.

While comparing all possible groups with a Tukey HSD test is a common approach, many studies have a control group and several treatment groups. For these studies, you may need to compare the treatment groups only to the control group, which reduces the number of comparisons.

Dunnett’s test does precisely this—it only compares a group taken as reference to all other groups, but it does not compare all groups to each others.

So to recap:

the Tukey HSD test allows to compares all groups but at the cost of less power
the Dunnett’s test allows to only make comparisons with a reference group, but with the benefit of more power

Now, again for the sake of illustration, consider that the species Adelie is the reference species and we are only interested in comparing the reference species against the other 2 species. In that scenario, we would use the Dunnett’s test.

In R, the Dunnett’s test is done as follows (the only difference with the code for the Tukey HSD test is in the line linfct = mcp(species = "Dunnett")):

library(multcomp)

# Dunnett's test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 7.59e-09 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

The interpretation is the same as for the Tukey HSD test’s except that in the Dunett’s test we only compare:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)

Both adjusted p-values (displayed in the last column) are below 0.05, so we reject the null hypothesis for both comparisons.

This means that both the species Chinstrap and Gentoo are significantly different from the reference species Adelie in terms of flippers length. (Nothing can be said about the comparison between Chinstrap and Gentoo though.)

Again, the results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that both the species Gentoo and Chinstrap are significantly different from the reference species Adelie.

Note that in R, by default, the reference category for a factor variable is the first category in alphabetical order. This is the reason that, by default, the reference species is Adelie.

The reference category can be changed with the relevel() function (or with the {questionr} addin). Considering that we want Gentoo as the reference category instead of Adelie:

# Change reference category:
dat$species <- relevel(dat$species, ref = "Gentoo")

# Check that Gentoo is the reference category:
levels(dat$species)

## [1] "Gentoo"    "Adelie"    "Chinstrap"

Gentoo now being the first category of the three, it is indeed considered as the reference level.

In order to perform the Dunnett’s test with the new reference we first need to rerun the ANOVA to take into account the new reference:

res_aov2 <- aov(flipper_length_mm ~ species,
  data = dat
)

We can then run the Dunett’s test with the new results of the ANOVA:

# Dunnett's test:
post_test <- glht(res_aov2,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Adelie - Gentoo == 0    -27.2333     0.8067  -33.76   <1e-10 ***
## Chinstrap - Gentoo == 0 -21.3635     1.0036  -21.29   <1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

par(mar = c(3, 8, 3, 3))
plot(post_test)

From the results above we conclude that Adelie and Chinstrap species are significantly different from Gentoo species in terms of flippers length (adjusted p-values < 1e-10).

Note that even if your study does not have a reference group which you can compare to the other groups, it is still often better to do multiple comparisons determined by some research questions than to do all-pairwise tests. By reducing the number of post-hoc comparisons to what is necessary only, and no more, you maximize the statistical power.⁸

Other p-values adjustment methods

For the interested readers, note that you can use other p-values adjustment methods by using the pairwise.t.test() function:

pairwise.t.test(dat$flipper_length_mm, dat$species,
  p.adjust.method = "holm"
)

## 
## 	Pairwise comparisons using t tests with pooled SD 
## 
## data:  dat$flipper_length_mm and dat$species 
## 
##           Gentoo  Adelie 
## Adelie    < 2e-16 -      
## Chinstrap < 2e-16 3.8e-09
## 
## P value adjustment method: holm

By default, the Holm method is applied but other methods exist. See ?p.adjust for all available options.

Visualization of ANOVA and post-hoc tests on the same plot

If you are interested in including results of ANOVA and post-hoc tests on the same plot (directly on the boxplots), here are two pieces of code which may be of interest to you.

The first one is edited by me based on the code found in this article:

# Edit from here
x <- which(names(dat) == "species") # name of grouping variable
y <- which(
  names(dat) == "flipper_length_mm" # names of variables to test
)
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("Chinstrap", "Adelie"), c("Gentoo", "Adelie"), c("Gentoo", "Chinstrap")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
library(ggpubr)
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

And the second method is from the {ggstatsplot} package:

library(ggstatsplot)

ggbetweenstats(
  data = dat,
  x = species,
  y = flipper_length_mm,
  type = "parametric", # ANOVA or Kruskal-Wallis
  var.equal = TRUE, # ANOVA or Welch ANOVA
  plot.type = "box",
  pairwise.comparisons = TRUE,
  pairwise.display = "significant",
  centrality.plotting = FALSE,
  bf.message = FALSE
)

As you can see on the above plot, boxplots by species are presented together with p-values of the ANOVA (after p = in the subtitle of the plot) and p-values of the post-hoc tests (above each comparison).

Besides the fact that these methods can be used to combine a visual representation and statistical results on the same plot, they also have the advantage that you can perform multiple ANOVA tests at once. See more information in this article.

Summary

In this article, we reviewed the goals and hypotheses of an ANOVA, what are the assumptions which need to be verified before being able to trust the results (namely, independence, normality and homogeneity), we then showed how to do an ANOVA in R and how to interpret the results.

An article about ANOVA would not be complete without discussing about post-hoc tests, and in particular, the Tukey HSD—to compare all groups—and the Dunnett’s test—to compare a reference group to all other groups.

Last but not least, we showed how to visualize the data and the results of the ANOVA and post-hoc tests in the same plot.

Thanks for reading. See this tutorial if you would like to learn how to do an ANOVA by hand.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Hsu, Jason. 1996. Multiple Comparisons: Theory and Methods. CRC Press.

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Note that it is called one-way or one-factor ANOVA because the means relate to the different modalities of a single independent variable, or factor.↩︎
Residuals (denoted $\epsilon$) are the differences between the observed values of the dependent variable ($y$) and the predicted values ($\hat{y}$). In the context of ANOVA, residuals correspond to the differences between the observed values and the mean of all values for that group.↩︎
Stevens (2013) wrote, in p. 57, “Numerous studies have examined the effect of violations of assumptions in ANOVA, and an excellent summary of this literature has been provided by Glass, Peckham, and Sanders (1972). Their review indicates that non normality has only a slight effect on the type I error rate, even for very skewed or kurtotic distributions. For example, the actual $\alpha$s for some very non-normal populations were only .055 or .06: very minor deviations from the nominal level of .05. […] The basic reason is the Central Limit Theorem, which states that the sum of independent observations having any distribution whatsoever approaches a normal distribution as the number of observations increases. To be somewhat more specific, Bock (1975) notes,”even for distributions which depart markedly from normality, sums of 50 or more observations approximate to normality. For moderately non-normal distributions the approximation is good with as few as 10 to 20 observations” (p. 111). Now since the sums of independent observations approach normality rapidly, so do the means, and the sampling distribution of F is based on means. Thus the sampling distribution of F is only slightly affected, and therefore the critical values when sampling from normal and non-normal distributions will not differ by much. Lack of normality due to skewness also has only a slight effect on power (a few hundredths).”↩︎
As long as you use the Kruskal-Wallis test to, in fine, compare groups, homoscedasticity is not required. If you wish to compare medians, the Kruskal-Wallis test requires homoscedasticity. See more information about the difference in this article.↩︎
Note that, as discussed in the comments at the end of the article, post-hoc tests can under some circumstances be done directly (without an ANOVA). See the comments or Hsu (1996) for more details.↩︎
Note that you could in principle apply the Bonferroni correction to all tests. For example, in the example above, with 3 tests and a global desired significance level of $\alpha$ = 0.05, we would only reject a null hypothesis if the p-value is less than $\frac{0.05}{3}$ = 0.0167. This method is, however, known to be quite conservative, leading to a potentially high rate of false negatives.↩︎
The p-values are adjusted to keep the global significance level to the desired level.↩︎
Thanks Michael Friendly for this suggestion.↩︎

Wilcoxon test in R: how to compare 2 groups under the non-normality assumption?

Sun, 07 Jun 2020 00:00:00 +0000

Introduction

In a previous article, we showed how to compare two groups under different scenarios using the Student’s t-test. The Student’s t-test requires that the distributions follow a normal distribution when in presence of small samples.¹

In this article, we show how to compare two groups when the normality assumption is violated, using the Wilcoxon test.

The Wilcoxon test is a non-parametric test, meaning that it does not rely on data belonging to any particular parametric family of probability distributions. Non-parametric tests have the same objective as their parametric counterparts. However, they have two advantages over parametric tests: they do not require the assumption of normality of distributions and they can deal with outliers.

A Student’s t-test for instance is only applicable if the data are Gaussian or if the sample size is large enough (usually $n \ge 30$, thanks to the central limit theorem). A non-parametric test should be used in other cases.

One may wonder why we would not always use a non-parametric test so we do not have to bother about testing for normality. The reason is that non-parametric tests are usually less powerful than corresponding parametric tests when the normality assumption holds.

Therefore, all else being equal, with a non-parametric test you are less likely to reject the null hypothesis when it is false if the data follows a normal distribution. It is thus preferred to use the parametric version of a statistical test when the assumptions are met.

In the remaining of the article, we present the two scenarios of the Wilcoxon test and how to perform them in R through two examples.

Two different scenarios

As for the Student’s t-test, the Wilcoxon test is used to compare two groups and see whether they are significantly different from each other in terms of the variable of interest.

The two groups to be compared are either:

independent, or
paired (i.e., dependent)

There are actually two versions of the Wilcoxon test:

The Mann-Whitney-Wilcoxon test (also referred as Wilcoxon rank sum test or Mann-Whitney U test) is performed when the samples are independent (so this test is the non-parametric equivalent to the Student’s t-test for independent samples).
The Wilcoxon signed-rank test (also sometimes referred as Wilcoxon test for paired samples) is performed when the samples are paired/dependent (so this test is the non-parametric equivalent to the Student’s t-test for paired samples).

Luckily, those two tests can be done in R with the same function: wilcox.test(). They are presented in the following sections.

Independent samples

For the Wilcoxon test with independent samples, suppose that we want to test whether grades at the statistics exam differ between female and male students.

We have collected grades for 24 students (12 girls and 12 boys):

dat <- data.frame(
  Sex = as.factor(c(rep("Girl", 12), rep("Boy", 12))),
  Grade = c(
    19, 18, 9, 17, 8, 7, 16, 19, 20, 9, 11, 18,
    16, 5, 15, 2, 14, 15, 4, 7, 15, 6, 7, 14
  )
)

dat

##     Sex Grade
## 1  Girl    19
## 2  Girl    18
## 3  Girl     9
## 4  Girl    17
## 5  Girl     8
## 6  Girl     7
## 7  Girl    16
## 8  Girl    19
## 9  Girl    20
## 10 Girl     9
## 11 Girl    11
## 12 Girl    18
## 13  Boy    16
## 14  Boy     5
## 15  Boy    15
## 16  Boy     2
## 17  Boy    14
## 18  Boy    15
## 19  Boy     4
## 20  Boy     7
## 21  Boy    15
## 22  Boy     6
## 23  Boy     7
## 24  Boy    14

Here are the distributions of the grades by sex (using {ggplot2}):

library(ggplot2)

ggplot(dat) +
  aes(x = Sex, y = Grade) +
  geom_boxplot(fill = "#0c4c8a") +
  theme_minimal()

We first check whether the 2 samples follow a normal distribution via a histogram and the Shapiro-Wilk test:

hist(subset(dat, Sex == "Girl")$Grade,
  main = "Grades for girls",
  xlab = "Grades"
)

hist(subset(dat, Sex == "Boy")$Grade,
  main = "Grades for boys",
  xlab = "Grades"
)

shapiro.test(subset(dat, Sex == "Girl")$Grade)

## 
## 	Shapiro-Wilk normality test
## 
## data:  subset(dat, Sex == "Girl")$Grade
## W = 0.84548, p-value = 0.0323

shapiro.test(subset(dat, Sex == "Boy")$Grade)

## 
## 	Shapiro-Wilk normality test
## 
## data:  subset(dat, Sex == "Boy")$Grade
## W = 0.84313, p-value = 0.03023

The histograms show that both distributions do not seem to follow a normal distribution and the p-values of the Shapiro-Wilk tests confirm it (since we reject the null hypothesis of normality for both distributions at the 5% significance level).

We just showed that normality assumption is violated for both groups so it is now time to see how to perform the Wilcoxon test in R.

Note that in order to use the Student’s t-test (the parametric version of the Wilcoxon test), it is required that both samples follow a normal distribution if samples are small.² Therefore, even if one sample follows a normal distribution (and the other does not follow a normal distribution), it is recommended to use the non-parametric test.

Remember that the null and alternative hypothesis of the Wilcoxon test are as follows:

$H_0$: the 2 groups are equal in terms of the variable of interest
$H_1$: the 2 groups are different in terms of the variable of interest

Applied to our research question, we have:

$H_0$: grades of girls and boys are equal
$H_1$: grades of girls and boys are different

test <- wilcox.test(dat$Grade ~ dat$Sex)
test

## 
## 	Wilcoxon rank sum test with continuity correction
## 
## data:  dat$Grade by dat$Sex
## W = 31.5, p-value = 0.02056
## alternative hypothesis: true location shift is not equal to 0

We obtain the test statistic, the p-value and a reminder of the hypothesis tested.³

The p-value is 0.021. Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that grades are significantly different between girls and boys.

Given the boxplot presented above showing the grades by sex, one may see that girls seem to perform better than boys. This can be tested formally by adding the alternative = "less" argument to the wilcox.test() function:⁴

test <- wilcox.test(dat$Grade ~ dat$Sex,
  alternative = "less"
)
test

## 
## 	Wilcoxon rank sum test with continuity correction
## 
## data:  dat$Grade by dat$Sex
## W = 31.5, p-value = 0.01028
## alternative hypothesis: true location shift is less than 0

The p-value is 0.01. Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that boys performed significantly worse than girls (which is equivalent than concluding that girls performed significantly better than boys).

Paired samples

For this second scenario, consider that we administered a math test in a class of 12 students at the beginning of a semester, and that we administered a similar test at the end of the semester to the exact same students. We have the following data:

dat2 <- data.frame(
  Beginning = c(16, 5, 15, 2, 14, 15, 4, 7, 15, 6, 7, 14),
  End = c(19, 18, 9, 17, 8, 7, 16, 19, 20, 9, 11, 18)
)

dat2

##    Beginning End
## 1         16  19
## 2          5  18
## 3         15   9
## 4          2  17
## 5         14   8
## 6         15   7
## 7          4  16
## 8          7  19
## 9         15  20
## 10         6   9
## 11         7  11
## 12        14  18

We transform the dataset to have it in a tidy format:

dat2 <- data.frame(
  Time = c(rep("Before", 12), rep("After", 12)),
  Grade = c(dat2$Beginning, dat2$End)
)
dat2

##      Time Grade
## 1  Before    16
## 2  Before     5
## 3  Before    15
## 4  Before     2
## 5  Before    14
## 6  Before    15
## 7  Before     4
## 8  Before     7
## 9  Before    15
## 10 Before     6
## 11 Before     7
## 12 Before    14
## 13  After    19
## 14  After    18
## 15  After     9
## 16  After    17
## 17  After     8
## 18  After     7
## 19  After    16
## 20  After    19
## 21  After    20
## 22  After     9
## 23  After    11
## 24  After    18

The distribution of the grades at the beginning and after the semester:

# Reordering dat2$Time
dat2$Time <- factor(dat2$Time,
  levels = c("Before", "After")
)

ggplot(dat2) +
  aes(x = Time, y = Grade) +
  geom_boxplot(fill = "#0c4c8a") +
  theme_minimal()

(See the {esquisse} and {questionr} addins to help you reorder levels of a factor variable and to easily draw plots with the {ggplot2} package.)

In this example, it is clear that the two samples are not independent since the same 12 students took the exam before and after the semester. Supposing also that the normality assumption is violated (and given the small sample size), we thus use the Wilcoxon test for paired samples, with the following hypotheses:⁵

$H_0$: grades before and after the semester are equal
$H_1$: grades before and after the semester are different

We add the paired = TRUE argument to the wilcox.test() function to take into consideration the dependency between the 2 samples:

before <- dat2$Grade[dat2$Time == "Before"]
after <- dat2$Grade[dat2$Time == "After"]
test <- wilcox.test(before, after, paired = TRUE)

test

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  before and after
## V = 21, p-value = 0.1692
## alternative hypothesis: true location shift is not equal to 0

We obtain the test statistic, the p-value and a reminder of the hypothesis tested.

The p-value is 0.169. Therefore, at the 5% significance level, we do not reject the null hypothesis that the grades are similar before and after the semester.

Combination of plot and statistical test

After having written this article, I discovered the {ggstatsplot} package which I believe is worth mentioning here, in particular the ggbetweenstats() and ggwithinstats() functions for independent and paired samples, respectively.

These two functions combine a boxplot—representing the distribution for each group—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for independent and paired samples, using the same data than previously.

Independent samples

For independent samples, it is the ggbetweenstats() function which is used:

# load package
library(ggstatsplot)

# plot with statistical results
ggbetweenstats( # independent samples
  data = dat,
  x = Sex,
  y = Grade,
  plot.type = "box", # for boxplot
  type = "nonparametric", # for wilcoxon
  centrality.plotting = FALSE # remove median
)

The p-value (displayed after p = in the subtitle of the plot) indicates that we reject the null hypothesis, and we conclude that grades are significantly different between girls and boys (p-value = 0.02).

Paired samples

For paired samples, it is the ggwithinstats() function which is used:

# load package
library(ggstatsplot)

# plot with statistical results
ggwithinstats( # paired samples
  data = dat2,
  x = Time,
  y = Grade,
  type = "nonparametric", # for wilcoxon
  centrality.plotting = FALSE # remove median
)

The p-value (displayed after p = in the subtitle of the plot) indicates that we do not reject the null hypothesis, so we do not reject the hypothesis that grades are equal before and after the semester (p-value = 0.17).

The point of this section was to illustrate how to easily draw plots together with statistical results, which is exactly the aim of the {ggstatsplot} package. See more details and examples in this article.

Assumption of equal variances

As written at the beginning of the article, the Wilcoxon test does not require the assumption of normality in case of small samples.

Regarding the assumption of equal variances, this assumption may or may not be needed depending on your goal. If you only want to compare the two groups, you do not have to test the equality of variances because the two distributions do not have to have the same shape. However, if your goal is to compare medians of the two groups, then you will need to make sure that the two distributions have the same shape (and thus, the same variance).⁶

So results of your test of equality of variances will change your interpretation: differences in the “distributions” of two groups or differences in the “medians” of two groups.

In this article I do not wish to compare medians, I only want compare the groups by determining whether there are differences in the distributions of the two groups. This is the reason I do not test for equality of variances.

Note that this is equivalent when performing the Kruskal-Wallis test to compare three groups or more (i.e., the non-parametric version of the ANOVA): if you only want to test whether there are differences in the groups you do not need homoscedasticity, whereas if you want to compare the medians this assumption must be met.

Conclusion

Thanks for reading.

I hope this article helped you to compare two groups that do not follow a normal distribution in R using the Wilcoxon test. See also:

the one-sample Wilcoxon test if you have only one group and want to compare it to a default given value,
the Student’s t-test if you need to perform the parametric version of the two-sample Wilcoxon test,
and the ANOVA if you need to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Remember that the normality assumption can be tested via 3 complementary methods: (i) histogram, (ii) QQ-plot and (iii) normality tests (with the most common being the Shapiro-Wilk test). See how to determine if a distribution follows a normal distribution if you need a refresh.↩︎
In case of large samples, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013).↩︎
Note that the presence of equal elements (ties) prevents an exact p-value calculation. This can be tackled by computing the exact or asymptotic Wilcoxon-Mann-Whitney test with adjustment for ties, using the wilcox_test() function from the {coin} package: wilcox_test(dat$Grade ~ dat$Sex, distribution = exact()) or wilcox_test(dat$Grade ~ dat$Sex). In our case, conclusions remain unchanged.↩︎
We add alternative = "less" (and not alternative = "greater") because we want to test that grades for boys are less than grade for girls. Using "less" or "greater" can be deducted from the reference level in the dataset.↩︎
Note that for paired samples (when in presence of a small sample), normality must be checked on the differences between the two paired samples, and not individually on the two samples like it is done for independent samples.↩︎
See these three articles for a more detailed discussion on the assumption of equal variances in Wilcoxon test: 1, 2 & 3.↩︎

Correlation coefficient and correlation test in R

Thu, 28 May 2020 00:00:00 +0000

Introduction

Correlations between variables play an important role in a descriptive analysis. A correlation measures the relationship between two variables, that is, how they are linked to each other. In this sense, a correlation allows to know which variables evolve in the same direction, which ones evolve in the opposite direction, and which ones are independent.

In this article, I show how to compute correlation coefficients, how to perform correlation tests and how to visualize relationships between variables in R.

Correlation is usually computed on two quantitative variables, but it can also be computed on two qualitative ordinal variables.¹ See the Chi-square test of independence if you need to study the relationship between two qualitative nominal variables.

If you need to quantify the relationship between two variables, I refer you to the article about linear regression.

Data

In this article, we use the mtcars dataset (loaded by default in R):

# display first 5 observations
head(mtcars, 5)

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2

The variables vs and am are categorical variables, so they are removed for this article:

# remove vs and am variables
library(tidyverse)
dat <- mtcars %>%
  select(-vs, -am)

# display 5 first obs. of new dataset
head(dat, 5)

##                    mpg cyl disp  hp drat    wt  qsec gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02    3    2

Correlation coefficient

Between two variables

The correlation between 2 variables is found with the cor() function.

Suppose we want to compute the correlation between horsepower (hp) and miles per gallon (mpg):

# Pearson correlation between 2 variables
cor(dat$hp, dat$mpg)

## [1] -0.7761684

Note that the correlation between variables X and Y is equal to the correlation between variables Y and X so the order of the variables in the cor() function does not matter.

The Pearson correlation is computed by default with the cor() function. If you want to compute the Spearman correlation, add the argument method = "spearman" to the cor() function:

# Spearman correlation between 2 variables
cor(dat$hp, dat$mpg,
  method = "spearman"
)

## [1] -0.8946646

The most common correlation methods (Run ?cor for more information about the different methods available in the cor() function) are:

Pearson correlation is often used for quantitative continuous variables that have a linear relationship
Spearman correlation (which is actually similar to Pearson but based on the ranked values for each variable rather than on the raw data) is often used to evaluate relationships involving at least one qualitative ordinal variable or two quantitative variables if the link is partially linear
Kendall’s tau-b which is computed from the number of concordant and discordant pairs is often used for qualitative ordinal variables

Note that there exists the point-biserial correlation (which can be used to measure the association between a continuous variable and a nominal variable of two levels), but this correlation is not covered here.

Correlation matrix: correlations for all variables

Suppose now that we want to compute correlations for several pairs of variables. We can easily do so for all possible pairs of variables in the dataset, again with the cor() function:

# correlation for all variables
round(cor(dat),
  digits = 2 # rounded to 2 decimals
)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg   1.00 -0.85 -0.85 -0.78  0.68 -0.87  0.42  0.48 -0.55
## cyl  -0.85  1.00  0.90  0.83 -0.70  0.78 -0.59 -0.49  0.53
## disp -0.85  0.90  1.00  0.79 -0.71  0.89 -0.43 -0.56  0.39
## hp   -0.78  0.83  0.79  1.00 -0.45  0.66 -0.71 -0.13  0.75
## drat  0.68 -0.70 -0.71 -0.45  1.00 -0.71  0.09  0.70 -0.09
## wt   -0.87  0.78  0.89  0.66 -0.71  1.00 -0.17 -0.58  0.43
## qsec  0.42 -0.59 -0.43 -0.71  0.09 -0.17  1.00 -0.21 -0.66
## gear  0.48 -0.49 -0.56 -0.13  0.70 -0.58 -0.21  1.00  0.27
## carb -0.55  0.53  0.39  0.75 -0.09  0.43 -0.66  0.27  1.00

This correlation matrix gives an overview of the correlations for all combinations of two variables.

Interpretation of a correlation coefficient

First of all, correlation ranges from -1 to 1. It gives us an indication on two things:

The direction of the relationship between the 2 variables
The strength of the relationship between the 2 variables

Regarding the direction of the relationship: On the one hand, a negative correlation implies that the two variables under consideration vary in opposite directions, that is, if a variable increases the other decreases and vice versa. On the other hand, a positive correlation implies that the two variables under consideration vary in the same direction, i.e., if a variable increases the other one increases and if one decreases the other one decreases as well.

Regarding the strength of the relationship: The more extreme the correlation coefficient (the closer to -1 or 1), the stronger the relationship. This also means that a correlation close to 0 indicates that the two variables are independent, that is, as one variable increases, there is no tendency in the other variable to either decrease or increase.

As an illustration, the Pearson correlation between horsepower (hp) and miles per gallon (mpg) found above is -0.78, meaning that the 2 variables vary in opposite direction. This makes sense, cars with more horsepower tend to consume more fuel (and thus have a lower mileage per gallon). On the contrary, from the correlation matrix we see that the correlation between miles per gallon (mpg) and the time to drive 1/4 of a mile (qsec) is 0.42, meaning that fast cars (low qsec) tend to have a worse mileage per gallon (low mpg). This again makes sense as fast cars tend to consume more fuel.

Note that it is a good practice to visualize the type of the relationship between the two variables before interpreting the correlation coefficients. The reason is that the correlation coefficient could be biased due to an outlier or due to the type of link between the two variables.

For instance, see the two Pearson correlation coefficients (denoted by R in the following plots) when the outlier is excluded and included:

The Pearson correlation coefficient changes drastically due to a single point, and thus the interpretation. It goes from a negative correlation coefficient, indicating a negative relationship between the 2 variables, to a positive coefficient, indicating a positive relationship. We would have missed this insight if we had not visualized the data in a scatterplot (see how to draw a scatterplot in this section).

A correlation coefficient may also miss a non-linear link between two variables:

The Pearson correlation coefficient is equal to 0, indicating no relationship between the two variables, because it measures the linear relationship and it is clear from the plot that the link is non-linear.

So to recap, it is a good practice to visualize the data via a scatterplot before interpreting a correlation coefficient (it does not tell the whole story) and see how the correlation coefficient changes when using the parametric (Pearson) or nonparametric version (Spearman or Kendall’s tau-b).

Visualizations

The correlation matrix presented above is not easily interpretable, especially when the dataset is composed of many variables. In the following sections, we present some alternatives to the correlation matrix for better readability.

A scatterplot for 2 variables

A good way to visualize a correlation between 2 variables is to draw a scatterplot of the two variables of interest. Suppose we want to examine the relationship between horsepower (hp) and miles per gallon (mpg):

# scatterplot
library(ggplot2)

ggplot(dat) +
  aes(x = hp, y = mpg) +
  geom_point(colour = "#0c4c8a") +
  theme_minimal()

If you are unfamiliar with the {ggplot2} package, you can draw the scatterplot using the plot() function from R base graphics:

plot(dat$hp, dat$mpg)

or use the esquisse addin to easily draw plots using the {ggplot2} package.

Scatterplots for several pairs of variables

Suppose that instead of visualizing the relationship between only 2 variables, we want to visualize the relationship for several pairs of variables. This is possible thanks to the pair() function.

For this illustration, we focus only on miles per gallon (mpg), horsepower (hp) and weight (wt):

# multiple scatterplots
pairs(dat[, c("mpg", "hp", "wt")])

The figure indicates that weight (wt) and horsepower (hp) are positively correlated, whereas miles per gallon (mpg) seems to be negatively correlated with horsepower (hp) and weight (wt).

Another simple correlation matrix

This version of the correlation matrix presents the correlation coefficients in a slightly more readable way, i.e., by coloring the coefficients based on their sign. Applied to our dataset, we have:

# improved correlation matrix
library(corrplot)

corrplot(cor(dat),
  method = "number",
  type = "upper" # show only upper side
)

Correlation test

For 2 variables

Unlike a correlation matrix which indicates the correlation coefficients between some pairs of variables in the sample, a correlation test is used to test whether the correlation (denoted $\rho$) between 2 variables is significantly different from 0 or not in the population.

Actually, a correlation coefficient different from 0 in the sample does not mean that the correlation is significantly different from 0 in the population. This needs to be tested with a hypothesis test—and known as the correlation test.

The null and alternative hypothesis for the correlation test are as follows:

$H_0$: $\rho = 0$ (meaning that there is no linear relationship between the two variables)
$H_1$: $\rho \ne 0$ (meaning that there is a linear relationship between the two variables)

Via this correlation test, what we are actually testing is whether:

the sample contains sufficient evidence to reject the null hypothesis and conclude that the correlation coefficient does not equal 0, so the relationship exists in the population.
or on the contrary, the sample does not contain enough evidence that the correlation coefficient does not equal 0, so in this case we do not reject the null hypothesis of no relationship between the variables in the population.

Note that there are 2 assumptions for this test to be valid:

Independence of the data
For small sample sizes (usually $n < 30$), the two variables should follow a normal distribution

Suppose that we want to test whether the rear axle ratio (drat) is correlated with the time to drive a quarter of a mile (qsec):

# Pearson correlation test
test <- cor.test(dat$drat, dat$qsec)
test

## 
## 	Pearson's product-moment correlation
## 
## data:  dat$drat and dat$qsec
## t = 0.50164, df = 30, p-value = 0.6196
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.265947  0.426340
## sample estimates:
##        cor 
## 0.09120476

The p-value of the correlation test between these 2 variables is 0.62. At the 5% significance level, we do not reject the null hypothesis of no correlation. We therefore conclude that we do not reject the hypothesis that there is no linear relationship between the 2 variables.²

This test proves that even if the correlation coefficient is different from 0 (the correlation is 0.09 in the sample), it is actually not significantly different from 0 in the population.

Note that the p-value of a correlation test is based on the correlation coefficient and the sample size. The larger the sample size and the more extreme the correlation (closer to -1 or 1), the more likely the null hypothesis of no correlation will be rejected.

With a small sample size, it is thus possible to obtain a relatively large correlation in the sample (based on the correlation coefficient), but still find a correlation not significantly different from 0 in the population (based on the correlation test). For this reason, it is recommended to always perform a correlation test before interpreting a correlation coefficient to avoid flawed conclusions.

For several pairs of variables

Similar to the correlation matrix used to compute correlation for several pairs of variables, the rcorr() function (from the {Hmisc} package) allows to compute p-values of the correlation test for several pairs of variables at once. Applied to our dataset, we have:

# correlation tests for whole dataset
library(Hmisc)
res <- rcorr(as.matrix(dat)) # rcorr() accepts matrices only

# display p-values (rounded to 3 decimals)
round(res$P, 3)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg     NA 0.000 0.000 0.000 0.000 0.000 0.017 0.005 0.001
## cyl  0.000    NA 0.000 0.000 0.000 0.000 0.000 0.004 0.002
## disp 0.000 0.000    NA 0.000 0.000 0.000 0.013 0.001 0.025
## hp   0.000 0.000 0.000    NA 0.010 0.000 0.000 0.493 0.000
## drat 0.000 0.000 0.000 0.010    NA 0.000 0.620 0.000 0.621
## wt   0.000 0.000 0.000 0.000 0.000    NA 0.339 0.000 0.015
## qsec 0.017 0.000 0.013 0.000 0.620 0.339    NA 0.243 0.000
## gear 0.005 0.004 0.001 0.493 0.000 0.000 0.243    NA 0.129
## carb 0.001 0.002 0.025 0.000 0.621 0.015 0.000 0.129    NA

Only correlations with p-values smaller than the significance level (usually $\alpha = 0.05$) should be interpreted.

Combination of correlation coefficients and correlation tests

Now that we covered the concepts of correlation coefficients and correlation tests, let see if we can combine the two concepts.

If you need to do this for a few pairs of variables, I recommend using the ggscatterstats() function from the {ggstatsplot} package. Let’s see it in practice with one pair of variables—wt and mpg:

## plot with statistical results
library(ggstatsplot)

ggscatterstats(
  data = dat,
  x = wt,
  y = mpg,
  bf.message = FALSE,
  marginal = FALSE # remove histograms
)

Based on the result of the test, we conclude that there is a negative correlation between the weight and the number of miles per gallon ($r = - 0.87$, $p$-value < 0.001).

If you need to do it for many pairs of variables, I recommend using the the correlation function from the easystats {correlation} package.

This function allows to combine correlation coefficients and correlation tests for several pairs of variables, all in a single table (thanks to krzysiektr for pointing it out to me):

library(correlation)

correlation::correlation(dat,
  include_factors = TRUE, method = "auto"
)

## # Correlation Matrix (auto-method)
## 
## Parameter1 | Parameter2 |     r |         95% CI | t(30) |         p
## --------------------------------------------------------------------
## mpg        |        cyl | -0.85 | [-0.93, -0.72] | -8.92 | < .001***
## mpg        |       disp | -0.85 | [-0.92, -0.71] | -8.75 | < .001***
## mpg        |         hp | -0.78 | [-0.89, -0.59] | -6.74 | < .001***
## mpg        |       drat |  0.68 | [ 0.44,  0.83] |  5.10 | < .001***
## mpg        |         wt | -0.87 | [-0.93, -0.74] | -9.56 | < .001***
## mpg        |       qsec |  0.42 | [ 0.08,  0.67] |  2.53 | 0.137    
## mpg        |       gear |  0.48 | [ 0.16,  0.71] |  3.00 | 0.065    
## mpg        |       carb | -0.55 | [-0.75, -0.25] | -3.62 | 0.016*   
## cyl        |       disp |  0.90 | [ 0.81,  0.95] | 11.45 | < .001***
## cyl        |         hp |  0.83 | [ 0.68,  0.92] |  8.23 | < .001***
## cyl        |       drat | -0.70 | [-0.84, -0.46] | -5.37 | < .001***
## cyl        |         wt |  0.78 | [ 0.60,  0.89] |  6.88 | < .001***
## cyl        |       qsec | -0.59 | [-0.78, -0.31] | -4.02 | 0.007**  
## cyl        |       gear | -0.49 | [-0.72, -0.17] | -3.10 | 0.054    
## cyl        |       carb |  0.53 | [ 0.22,  0.74] |  3.40 | 0.027*   
## disp       |         hp |  0.79 | [ 0.61,  0.89] |  7.08 | < .001***
## disp       |       drat | -0.71 | [-0.85, -0.48] | -5.53 | < .001***
## disp       |         wt |  0.89 | [ 0.78,  0.94] | 10.58 | < .001***
## disp       |       qsec | -0.43 | [-0.68, -0.10] | -2.64 | 0.131    
## disp       |       gear | -0.56 | [-0.76, -0.26] | -3.66 | 0.015*   
## disp       |       carb |  0.39 | [ 0.05,  0.65] |  2.35 | 0.177    
## hp         |       drat | -0.45 | [-0.69, -0.12] | -2.75 | 0.110    
## hp         |         wt |  0.66 | [ 0.40,  0.82] |  4.80 | < .001***
## hp         |       qsec | -0.71 | [-0.85, -0.48] | -5.49 | < .001***
## hp         |       gear | -0.13 | [-0.45,  0.23] | -0.69 | > .999   
## hp         |       carb |  0.75 | [ 0.54,  0.87] |  6.21 | < .001***
## drat       |         wt | -0.71 | [-0.85, -0.48] | -5.56 | < .001***
## drat       |       qsec |  0.09 | [-0.27,  0.43] |  0.50 | > .999   
## drat       |       gear |  0.70 | [ 0.46,  0.84] |  5.36 | < .001***
## drat       |       carb | -0.09 | [-0.43,  0.27] | -0.50 | > .999   
## wt         |       qsec | -0.17 | [-0.49,  0.19] | -0.97 | > .999   
## wt         |       gear | -0.58 | [-0.77, -0.29] | -3.93 | 0.008**  
## wt         |       carb |  0.43 | [ 0.09,  0.68] |  2.59 | 0.132    
## qsec       |       gear | -0.21 | [-0.52,  0.15] | -1.19 | > .999   
## qsec       |       carb | -0.66 | [-0.82, -0.40] | -4.76 | < .001***
## gear       |       carb |  0.27 | [-0.08,  0.57] |  1.56 | 0.774    
## 
## p-value adjustment method: Holm (1979)
## Observations: 32

As you can see, it gives, among other useful information, the correlation coefficients (column r) and the result of the correlation test (column 95% CI for the confidence interval or p for the $p$-value) for all pairs of variables.

Correlograms

The table above is very useful and informative, but let see if it is possible to combine the concepts of correlation coefficients and correlations test in one single visualization. A visualization that would be easy to read and interpret.

Ideally, we would like to have a concise overview of correlations between all possible pairs of variables present in a dataset, with a clear distinction for correlations that are significantly different from 0.

The figure below, known as a correlogram and adapted from the corrplot() function, does precisely this:

# do not edit
corrplot2 <- function(data,
                      method = "pearson",
                      sig.level = 0.05,
                      order = "original",
                      diag = FALSE,
                      type = "upper",
                      tl.srt = 90,
                      number.font = 1,
                      number.cex = 1,
                      mar = c(0, 0, 0, 0)) {
  library(corrplot)
  data_incomplete <- data
  data <- data[complete.cases(data), ]
  mat <- cor(data, method = method)
  cor.mtest <- function(mat, method) {
    mat <- as.matrix(mat)
    n <- ncol(mat)
    p.mat <- matrix(NA, n, n)
    diag(p.mat) <- 0
    for (i in 1:(n - 1)) {
      for (j in (i + 1):n) {
        tmp <- cor.test(mat[, i], mat[, j], method = method)
        p.mat[i, j] <- p.mat[j, i] <- tmp$p.value
      }
    }
    colnames(p.mat) <- rownames(p.mat) <- colnames(mat)
    p.mat
  }
  p.mat <- cor.mtest(data, method = method)
  col <- colorRampPalette(c("#BB4444", "#EE9988", "#FFFFFF", "#77AADD", "#4477AA"))
  corrplot(mat,
    method = "color", col = col(200), number.font = number.font,
    mar = mar, number.cex = number.cex,
    type = type, order = order,
    addCoef.col = "black", # add correlation coefficient
    tl.col = "black", tl.srt = tl.srt, # rotation of text labels
    # combine with significance level
    p.mat = p.mat, sig.level = sig.level, insig = "blank",
    # hide correlation coefficients on the diagonal
    diag = diag
  )
}

# edit from here
corrplot2(
  data = dat,
  method = "pearson",
  sig.level = 0.05,
  order = "original",
  diag = FALSE,
  type = "upper",
  tl.srt = 75
)

The correlogram shows correlation coefficients for all pairs of variables (with more intense colors for more extreme correlations), and correlations not significantly different from 0 are represented by a white box.

To learn more about this plot and the code used, I invite you to read the article entitled “Correlogram in R: how to highlight the most correlated variables in a dataset”.

For those of you who are still not completely satisfied, I recently found two alternatives—one with the ggpairs() function from the {GGally} package and one with the ggcormat() function from the {ggstatsplot} package.

The two functions are illustrated with the variables mpg, hp and wt:

library(GGally)

ggpairs(dat[, c("mpg", "hp", "wt")])

The plot above combines correlation coefficients, correlation tests (via the asterisks next to the coefficients³) and scatterplots for all possible pairs of variables present in a dataset.

library(ggstatsplot)

ggcorrmat(
  data = dat[, c("mpg", "hp", "wt")],
  type = "parametric", # parametric for Pearson, nonparametric for Spearman's correlation
  colors = c("darkred", "white", "steelblue") # change default colors
)

The plot above also shows the correlation coefficients and if any, the non-significant correlations (by default at the 5% significance level with the Holm adjustment method) are shown by a big cross on the correlation coefficients.

The advantage of these two alternatives compared to the first one is that it is directly available within a package, so you do not need to run the code of the function first in order to draw the correlogram.

Correlation does not imply causation

I am pretty sure you have already heard the statement “Correlation does not imply causation” in statistics. An article about correlation would not be complete without discussing about causation.

A non-zero correlation between two variables does not necessarily mean that there is a cause and effect relationship between these two variables!

Indeed, a significant correlation between two variables means that changes in one variable are associated (positively or negatively) with changes in the other variable. Nonetheless, a significant correlation does not indicate that variations in one variable cause the variations in the other variable.

A non-zero correlation between X and Y can appear in several cases:

X causes Y
Y causes X
a third variable causes X and Y
a combination of these three reasons

Sometimes it is quite clear that there is a causal relationship between two variables. Take for example the correlation between the price of a consumer product such as milk and its consumption. It is quite obvious that there is a causal link between the two: if the price of milk increases, it is expected that its consumption will decrease.

However, this causal link is not always present even if the correlation is significant. Maurage et al. (2013) showed that, although there is a positive and significant correlation between chocolate consumption and the number of Nobel laureates, this correlation comes from the fact that a third variable, Gross Domestic Product (GDP), causes chocolate consumption and the number of Nobel laureates. They found that countries with higher GDP tend to have a higher level of chocolate consumption and scientific research (leading to more Nobel laureates).

This example shows that one must be very cautious when interpreting correlations and avoid over-interpreting a correlation as a causal relationship.

Conclusion

Thanks for reading.

I hope this article helped you to compute correlation coefficients and perform correlation tests in R. If you would like to learn how to compute the coefficients by hand, see this step-by-step tutorial.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Maurage, Pierre, Alexandre Heeren, and Mauro Pesenti. 2013. “Does Chocolate Consumption Really Boost Nobel Award Chances? The Peril of over-Interpreting Correlations in Health Studies.” The Journal of Nutrition 143 (6): 931–33.

It is true that there is the point-biserial correlation which can be used with a nominal variable (consisting of two factors). Nonetheless, this type of correlation is much less known and usually not covered in introductory statistics classes; with one continuous and one nominal variable, it is much more frequent to learn about the Student’s t-test (for a nominal variable with 2 groups) or ANOVA (for a nominal variable with 3 or more groups). More information about choosing the most appropriate measure of association depending on the type of variable can be found in this article.↩︎
It is important to remember that we tested for a linear relationship between the two variables since we used the Pearson’s correlation. It may be the case that there is a relationship between the two variables in the population, but this relation may not be linear.↩︎
One asterisk means that the coefficient is significant at the 5% level, 2 is at the 1% significance level, and 3 is at the 0.1% significance level. This is usually the case in R; the more asterisks, the more it is significant.↩︎

One-proportion and chi-square goodness of fit test

Wed, 13 May 2020 00:00:00 +0000

Introduction

In a previous article, I presented the Chi-square test of independence in R which is used to test the independence between two categorical variables.

In this article, I show how to perform, first in R and then by hand, the:

one-proportion test (also referred as one-sample proportion test)
Chi-square goodness of fit test

The first test is used to compare an observed proportion to an expected proportion, when the qualitative variable has only two categories. The second test is used to compare multiple observed proportions to multiple expected proportions, in a situation where the qualitative variable has two or more categories.

Both tests allow to test the equality of proportions between the levels of the qualitative variable or to test the equality with given proportions. These given proportions could be determined arbitrarily or based on the theoretical probabilities of a known distribution.

In R

Data

For this section, we use the same dataset than in the article on descriptive statistics. It is the well-known iris dataset, to which we add the variable size. The variable size corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

# load iris dataset
dat <- iris

# create size variable
dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

# show first 5 observations
head(dat, n = 5)

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species  size
## 1          5.1         3.5          1.4         0.2  setosa small
## 2          4.9         3.0          1.4         0.2  setosa small
## 3          4.7         3.2          1.3         0.2  setosa small
## 4          4.6         3.1          1.5         0.2  setosa small
## 5          5.0         3.6          1.4         0.2  setosa small

One-proportion test

For this example, we have a sample of 150 flowers and we want to test whether the proportion of small flowers is different than the proportion of big flowers (measured by the variable size). Here are the number of flowers by size, and the corresponding proportions:

# barplot
library(ggplot2)
ggplot(dat) +
  aes(x = size) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by size
table(dat$size)

## 
##   big small 
##    77    73

# proportions by size, rounded to 2 decimals
round(prop.table(table(dat$size)), 2)

## 
##   big small 
##  0.51  0.49

Among the 150 flowers forming our sample, 51% and 49% are big and small, respectively. To test whether the proportions are different among both sizes, we use the prop.test() function which accepts the following arguments:

number of successes
number of observations/trials
expected probability (the one we want to test against)

The hypotheses in our example are:

$H_0$: proportions of big and small flowers are equal
$H_1$: proportions of big and small flowers are different

Considering (arbitrarily) that big is the success, we have:¹

# one-proportion test
test <- prop.test(
  x = 77, # number of successes
  n = 150, # total number of trials (77 + 73)
  p = 0.5 # we test for equal proportion so prob = 0.5 in each group
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  77 out of 150, null probability 0.5
## X-squared = 0.06, df = 1, p-value = 0.8065
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.4307558 0.5952176
## sample estimates:
##         p 
## 0.5133333

We obtain an output with:

the null probability (0.5),
the test statistic (X-squared = 0.06),
the degrees of freedom (df = 1),
the p-value (p-value = 0.8065),
the alternative hypothesis (true p is not equal to 0.5),
the 95% confidence interval (which can also be extracted with test$conf.int) and
the proportion in the sample (0.5133333).

The p-value is 0.806 so, at the 5% significance level, we do not reject the null hypothesis that the proportions of small and big flowers are the same.

An alternative is the ggpiestats() function from the {ggstatsplot} package:²

## plot with statistical results
library(ggstatsplot)
ggpiestats(
  data = dat,
  x = size,
  bf.message = FALSE
)

Note that the $p$-value (the value after p = in the subtitle of the plot) is slightly different because Yates’ continuity correction is not applied in ggpiestats() while it is applied by default in prop.test().

The conclusion remains however the same, that is, we do not reject the null hypothesis that proportions of big and small flowers are equal.

Assumption of `prop.test()` and `binom.test()`

Note that prop.test() uses a normal approximation to the binomial distribution. Therefore, one assumption of this test is that the sample size is large enough (usually, n > 30). If the sample size is small, it is recommended to use the exact binomial test.

The exact binomial test can be performed with the binom.test() function and accepts the same arguments as the prop.test() function.

For this example, suppose now that we have a sample of 12 big and 3 small flowers and we want to test whether the proportions are the same among both sizes:

# barplot
barplot(c(12, 3), # observed counts
  names.arg = c("big", "small"), # rename labels
  ylab = "Frequency", # y-axis label
  xlab = "Size" # x-axis label
)
abline(
  h = 15 / 2, # expected counts in each level
  lty = 2 # dashed line
)

# exact binomial test
test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5 # expected proportion
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.03516
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.5191089 0.9566880
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.035 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportions of small and big flowers are significantly different. This is equivalent than concluding that the proportion of big flowers is significantly different from 0.5 (since there are only two sizes).

If you want to test that the proportion of big flowers is greater than 50%, add the alternative = "greater" argument into the binom.test() function:³

test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5, # expected proportion
  alternative = "greater" # test that prop of big flowers is > 0.5
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.01758
## alternative hypothesis: true probability of success is greater than 0.5
## 95 percent confidence interval:
##  0.5602156 1.0000000
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.018 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of big flowers is significantly larger than 50%.

Chi-square goodness of fit test

Suppose now that the qualitative variable has more than two levels as it is the case for the variable Species:

# barplot
ggplot(dat) +
  aes(x = Species) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by Species
table(dat$Species)

## 
##     setosa versicolor  virginica 
##         50         50         50

The variable Species has 3 levels, with 50 observations in each level.

Suppose for this example that we want to test whether the 3 species are equally common. If they were equally common, they would be equally distributed and the expected proportions would be $\frac{1}{3}$ for each of the species.

The hypotheses are now:

$H_0$: proportions of each species are equal
$H_1$: there is at least one species with a different proportion⁴

This test can be done with the chisq.test() function, accepting the following arguments:

a numeric vector representing the observed proportions
a vector of probabilities (of the same length of the observed proportions) representing the expected proportions

Applied to our research question (i.e., are the 3 species equally common?), we have:

# chi-square goodness of fit test
test <- chisq.test(table(dat$Species), # observed proportions
  p = c(1 / 3, 1 / 3, 1 / 3) # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  table(dat$Species)
## X-squared = 0, df = 2, p-value = 1

The p-value is 1 so, at the 5% significance level, we do not reject the null hypothesis that the proportions are equal among all species.

This was quite obvious even before doing the statistical test given that there are exactly 50 flowers of each species, so it was easy to see that the species are equally common. We however still did the test to show how it works in practice.

Note that the alternative proposed by the {ggstatsplot} package can also be used for a Chi-square goodness of fit test:

## plot with statistical results
ggpiestats(
  data = dat,
  x = Species,
  bf.message = FALSE
)

Assumptions

One of the assumptions of the chi-square goodness of fit test is that the sample size is large enough in order for the chi-square approximation to be valid.

To be more precise, there must be at least 5 expected frequencies in each group of your categorical variable. This can be verified as follows:

chisq.test(table(dat$Species))$expected

##     setosa versicolor  virginica 
##         50         50         50

The assumption of sufficiently large sample size is met as all expected frequencies are above 5.

Does my distribution follow a given distribution?

In the previous section, we chose the proportions ourselves. The goodness of fit test is also particularly useful to compare observed proportions with expected proportions that are based on some known distribution.

Remember the hypotheses of the test:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

For this example, suppose that we measured the number of girls in 100 families of 5 children. We want to test whether the (observed) distribution of number girls follows a binomial distribution.

Observed frequencies

Here is the distribution of the number of girls per family in our sample of 100 families of 5 children:

And the corresponding frequencies and relative frequencies (remember that the relative frequency is the frequency divided by the total sample size):

# counts
dat

##   Girls Frequency Relative_freq
## 1     0         5          0.05
## 2     1        12          0.12
## 3     2        28          0.28
## 4     3        33          0.33
## 5     4        17          0.17
## 6     5         5          0.05

Expected frequencies

In order to compare the observed frequencies to a binomial distribution and see if both distributions match, we first need to determine the expected frequencies that would be obtained in case of a binomial distribution.

The expected frequencies assuming a probability of 0.5 of having a girl (for each of the 5 children) are as follows:

# create expected frequencies for a binomial distribution
x <- 0:5
df <- data.frame(
  Girls = factor(x),
  Expected_relative_freq = dbinom(x, size = 5, prob = 0.5)
)
df$Expected_freq <- df$Expected_relative_freq * 100 # *100 since there are 100 families

# create barplot
p <- ggplot(df, aes(x = Girls, y = Expected_freq)) +
  geom_bar(stat = "identity", fill = "#F8766D") +
  xlab("Number of girls per family") +
  ylab("Expected frequency") +
  labs(title = "Binomial distribution Bi(x, n = 5, p = 0.5)") +
  theme_minimal()
p

# expected relative frequencies and (absolute) frequencies
df

##   Girls Expected_relative_freq Expected_freq
## 1     0                0.03125         3.125
## 2     1                0.15625        15.625
## 3     2                0.31250        31.250
## 4     3                0.31250        31.250
## 5     4                0.15625        15.625
## 6     5                0.03125         3.125

Observed vs. expected frequencies

We now compare the observed frequencies to the expected frequencies to see whether the two differ significantly. If the two differ significantly, we reject the hypothesis that the number of girls per family of 5 children follows a binomial distribution. On the other hand, if the observed and expected frequencies are similar, we do not reject the hypothesis that the number of girls per family follows a binomial distribution.

Visually we have:

# create data
data <- data.frame(
  num_girls = factor(rep(c(0:5), times = 2)),
  Freq = c(dat$Freq, df$Expected_freq),
  obs_exp = c(rep("observed", 6), rep("expected", 6))
)

# create plot
ggplot() +
  geom_bar(
    data = data, aes(
      x = num_girls, y = Freq,
      fill = obs_exp
    ),
    position = "dodge", # bar next to each other
    stat = "identity"
  ) +
  ylab("Frequency") +
  xlab("Number of girls per family") +
  theme_minimal() +
  theme(legend.title = element_blank()) # remove legend title

We see that the observed and expected frequencies are quite similar, so we expect that the number of girls in families of 5 children follows a binomial distribution. However, only the goodness of fit test will confirm our belief:

# chi-square goodness of fit test
test <- chisq.test(dat$Freq, # observed frequencies
  p = df$Expected_relative_freq # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$Freq
## X-squared = 3.648, df = 5, p-value = 0.6011

The p-value is 0.601 so, at the 5% significance level, we do not reject the null hypothesis that the observed and expected frequencies are equal. This is equivalent than concluding that we cannot reject the hypothesis that the number of girls in families of 5 children follows a binomial distribution (since the expected frequencies were based on a binomial distribution).

Note that the chi-square goodness of fit test can of course be performed with other types of distribution than the binomial one. For instance, if you want to test whether an observed distribution follows a Poisson distribution, this test can be used to compare the observed frequencies with the expected proportions that would be obtained in case of a Poisson distribution.

By hand

Now that we showed how to perform the one-proportion and chi-square goodness of fit test in R, in this section we show how to do these tests by hand. We first illustrate the one-proportion test then the chi-square goodness of fit test.

One-proportion test

For this example, suppose that we tossed a coin 100 times and noted that it landed on heads 67 times. Following this, we want to test whether the coin is fair, that is, test whether the probability of landing on heads or tails is equal to 50%.

As for many hypothesis tests, we do it through 4 easy steps:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

In our example, the null and alternative hypotheses are:

$H_0$: $p_0 = 0.5$
$H_1$: $p_0 \ne 0.5$

where $p_0$ is the expected proportion of landing on heads.

Step 2.

The test statistic is:⁵

\[z_{obs} = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1 - p_0)}{n}}} = \frac{0.67 - 0.5}{\sqrt{\frac{0.5 \cdot (1 - 0.5)}{100}}} = 3.4\]

(See how to perform hypothesis tests in a Shiny app if you need more help in computing the test statistic.)

Step 3.

The rejection region is found via the normal distribution table. Assuming a significance level $\alpha = 0.05$, we have:

\[\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic lies within the rejection region (i.e., the grey shaded areas). Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of heads (and thus tails) is significantly different than 50%. In other words, still at the 5% significance level, we conclude that the coin is unfair.

If you prefer to compute the p-value instead of comparing the t-stat and the rejection region, you can use this Shiny app to easily compute p-values for different probability distributions. After having opened the app, set the t-stat, the corresponding alternative and you will find the p-value at the top of the page.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# one-proportion test
test <- prop.test(
  x = 67, # number of heads
  n = 100, # number of trials
  p = 0.5 # expected probability of heads
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  67 out of 100, null probability 0.5
## X-squared = 10.89, df = 1, p-value = 0.0009668
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.5679099 0.7588442
## sample estimates:
##    p 
## 0.67

The p-value is 0.001 so, at the 5% significance level, we reject the null hypothesis that the proportions of heads and tails are equal, and we conclude that the coin is biased. This is the same conclusion than the one found by hand.

Goodness of fit test

We now illustrate the chi-square goodness of fit test by hand with the following example.

Suppose that we toss a dice 100 times, we note how many times it lands on each face (1 to 6) and we test whether the dice is fair. Here are the observed counts by dice face:

## dice_face
##  1  2  3  4  5  6 
## 15 24 10 19 19 13

With a fair dice, we would expect it to land $\frac{100}{6} \approx 16.67$ times on each face (this expected value is represented by the dashed line in the above plot). Although the observed frequencies are different than the expected value of 16.67:

##   dice_face observed_freq expected_freq
## 1         1            15         16.67
## 2         2            24         16.67
## 3         3            10         16.67
## 4         4            19         16.67
## 5         5            19         16.67
## 6         6            13         16.67

we need to test whether they are significantly different. For this, we perform the appropriate hypothesis test following the 4 easy steps mentioned above:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

The null and alternative hypotheses of the chi-square goodness of fit test are:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

Applied to our example, we have:

$H_0$: all faces occur in the same proportion
$H_1$: at least one proportion is not equal to 1/6

Step 2.

The test statistic is:

\[\chi^2 = \sum_{i = 1}^k \frac{(O_i - E_i)^2}{E_i}\]

where $O_i$ is the observed frequency, $E_i$ is the expected frequency and $k$ is the number of categories (in our case, there are 6 categories, representing the 6 dice faces).

This $\chi^2$ statistic is obtained by calculating the difference between the observed number of cases and the expected number of cases in each category. This difference is squared (to avoid negative and positive differences being compensated) and divided by the expected number of cases in that category. These values are then summed for all categories, and the total is referred to as the $\chi^2$ statistic. Large values of this test statistic lead to the rejection of the null hypothesis, small values mean that the null hypothesis cannot be rejected.⁶

Given our data, we have:

\[\chi^2 = \frac{(15 - 16.67)^2}{16.67} + \frac{(24 - 16.67)^2}{16.67} + \\ \frac{(10 - 16.67)^2}{16.67} +\frac{(19 - 16.67)^2}{16.67} + \\ \frac{(19 - 16.67)^2}{16.67} + \frac{(13 - 16.67)^2}{16.67} = 7.52\]

Step 3.

Whether the $\chi^2$ test statistic is small or large depends on the rejection region. The rejection region is found via the $\chi^2$ distribution table. With a degrees of freedom equals to $k - 1$ (where $k$ is the number of categories) and assuming a significance level $\alpha = 0.05$, we have:

\[\chi^2_{\alpha; k-1} = \chi^2_{0.05; 5} = 11.0705\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic does not lie within the rejection region (i.e., the grey shaded area). Therefore, at the 5% significance level, we do not reject the null hypothesis that there is no significant difference between the observed and the expected frequencies. In other words, still at the 5% significance level, we cannot reject the hypothesis that the dice is fair.

Again, you can use the Shiny app to easily compute the p-value given the test statistic if you prefer this method over the comparison between the t-stat and the rejection region.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# chi-square goodness of fit test
test <- chisq.test(dat$observed_freq, # observed frequencies for each dice face
  p = rep(1 / 6, 6) # expected probabilities for each dice face
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$observed_freq
## X-squared = 7.52, df = 5, p-value = 0.1847

The test statistic and degrees of freedom are exactly the same than the ones found by hand. The p-value is 0.185 which, still at the 5% significance level, leads to the same conclusion than by hand (i.e., failing to reject the null hypothesis).

Conclusion

Thanks for reading.

I hope this article helped you to understand and perform the one-proportion and chi-square goodness of fit test in R and by hand. Learn more about the Chi-square test of independence in R and by hand if you want to analyze two categorical variables instead of one.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Choosing big or small as the success event gives the exact same conclusion.↩︎
Note that if possible, it is best to avoid pie charts and use bar charts instead. Unfortunately, the ggbarstats() function works only for the independence Chi-square test.↩︎
Similarly, this argument can also be added to the prop.test() function to test whether the observed proportion is larger than the expected proportion. Use alternative = "less" if you want to test whether the observed proportion is smaller than the expected one.↩︎
Be careful that the alternative hypothesis is not that all proportions are different. One different from the others is sufficient to reject the null hypothesis. It is, in some sense, similar to the alternative hypothesis of the ANOVA which says that at least one mean is different than another.↩︎
One assumption of this test is that $n \cdot p \ge 5$ and $n \cdot (1 - p) \ge 5$. The assumption is met so we can use the normal approximation to the binomial distribution.↩︎
Source: http://uregina.ca/~gingrich/ch10.pdf.↩︎

How to do a t-test or ANOVA for more than one variable at once in R?

Thu, 19 Mar 2020 00:00:00 +0000

Introduction

As part of my teaching assistant position in a Belgian university, students often ask me for some help in their statistical analyses for their master’s thesis.

A frequent question is how to compare groups of patients in terms of several quantitative continuous variables. Most of us know that:

To compare two groups, a Student’s t-test should be used¹
To compare three groups or more, an ANOVA should be performed

These two tests are quite basic and have been extensively documented online and in statistical textbooks so the difficulty is not in how to perform these tests.

In the past, I used to do the analyses by following these 3 steps:

Draw boxplots illustrating the distributions by group (with the boxplot() function or thanks to the {esquisse} R Studio addin if I wanted to use the {ggplot2} package)
Perform a t-test or an ANOVA depending on the number of groups to compare (with the t.test() and oneway.test() functions for t-test and ANOVA, respectively)
Repeat steps 1 and 2 for each variable

This was feasible as long as there were only a couple of variables to test. Nonetheless, most students came to me asking to perform these kind of tests not on one or two variables, but on multiples variables. So when there were more than one variable to test, I quickly realized that I was wasting my time and that there must be a more efficient way to do the job.

Note: you must be very careful with the issue of multiple testing (also referred as multiplicity) which can arise when you perform multiple tests. In short, when a large number of statistical tests are performed, some will have $p$-values less than 0.05 purely by chance, even if all null hypotheses are in fact really true. This is known as multiplicity or multiple testing. You can tackle this problem by using the Bonferroni correction, among others. The Bonferroni correction is a simple method that allows many t-tests to be made while still assuring an overall confidence level is maintained. For this, instead of using the standard threshold of $\alpha = 5$% for the significance level, you can use $\alpha = \frac{0.05}{m}$ where $m$ is the number of t-tests. For example, if you perform 20 t-tests with a desired $\alpha = 0.05$, the Bonferroni correction implies that you would reject the null hypothesis for each individual test when the $p$-value is smaller than $\alpha = \frac{0.05}{20} = 0.0025$.

Note also that there is no universally accepted approach for dealing with the problem of multiple comparisons. Usually, you should choose a p-value adjustment measure familiar to your audience or in your field of study. The Bonferroni correction is easy to implement. It is however not appropriate if you have a very large number of tests to perform (imagine you want to do 10,000 t-tests, a p-value would have to be less than $\frac{0.05}{10000} = 0.000005$ to be significant). A more powerful method is also to adjust the false discovery rate using the Benjamini-Hochberg or Holm procedure (McDonald 2014).

Another option is to use a multivariate ANOVA (MANOVA), if your independent variable has more than two levels. This is particularly useful when your dependent variables are correlated. Correlation between the dependent variables provides MANOVA the following advantages:

Identify patterns between several dependent variables: The independent variables can influence the relationship between dependent variables instead of influencing a single dependent variable.
Address the issue of multiple testing: with MANOVA, the error rate equals the significance level (with no p-value adjustment method needed).
Greater statistical power: When the dependent variables are correlated, MANOVA can identify effects that are too small for the ANOVA to detect.

Note that MANOVA is used if your independent variable has more than two levels. If your independent variable has only two levels, the multivariate equivalent of the t-test is Hotelling’s $T^2$.

This article aims at presenting a way to perform multiple t-tests and ANOVA from a technical point of view (how to implement it in R). Discussion on which adjustment method to use or whether there is a more appropriate model to fit the data is beyond the scope of this article (so be sure to understand the implications of using the code below for your own analyses). Make sure also to test the assumptions of the ANOVA before interpreting results.

Perform multiple tests at once

I thus wrote a piece of code that automated the process, by drawing boxplots and performing the tests on several variables at once. Below is the code I used, illustrating the process with the iris dataset. The Species variable has 3 levels, so let’s remove one, and then draw a boxplot and apply a t-test on all 4 continuous variables at once. Note that the continuous variables that we would like to test are variables 1 to 4 in the iris dataset.

dat <- iris

# remove one level to have only two groups
dat <- subset(dat, Species != "setosa")
dat$Species <- factor(dat$Species)

# boxplots and t-tests for the 4 variables at once
for (i in 1:4) { # variables to compare are variables 1 to 4
  boxplot(dat[, i] ~ dat$Species, # draw boxplots by group
    ylab = names(dat[i]), # rename y-axis with variable's name
    xlab = "Species"
  )
  print(t.test(dat[, i] ~ dat$Species)) # print results of t-test
}

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -5.6292, df = 94.025, p-value = 1.866e-07
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.8819731 -0.4220269
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    5.936                    6.588

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -3.2058, df = 97.927, p-value = 0.001819
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.33028364 -0.07771636
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    2.770                    2.974

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -12.604, df = 95.57, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -1.49549 -1.08851
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    4.260                    5.552

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -14.625, df = 89.043, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.7951002 -0.6048998
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    1.326                    2.026

As you can see, the above piece of code draws a boxplot and then prints results of the test for each continuous variable, all at once.

At some point in the past, I even wrote code to:

draw a boxplot
test for the equality of variances (thanks to the Levene’s test)
depending on whether the variances were equal or unequal, the appropriate test was applied: the Welch test if the variances were unequal and the Student’s t-test in the case the variances were equal (see more details about the different versions of the t-test for two samples)
apply steps 1 to 3 for all continuous variables at once

I had a similar code for ANOVA in case I needed to compare more than two groups.

The code was doing the job relatively well. Indeed, thanks to this code I was able to test several variables in an automated way in the sense that it compared groups for all variables at once.

The only thing I had to change from one project to another is that I needed to modify the name of the grouping variable and the numbering of the continuous variables to test (Species and 1:4 in the above code).

Concise and easily interpretable results

T-test

Although it was working quite well and applicable to different projects with only minor changes, I was still unsatisfied with another point.

Someone who is proficient in statistics and R can read and interpret the output of a t-test without any difficulty. However, as you may have noticed with your own statistical projects, most people do not know what to look for in the results and are sometimes a bit confused when they see so many graphs, code, output, results and numeric values in a document. They are quite easily overwhelmed by this mass of information and unable to extract the key message.

With my old R routine, the time I was saving by automating the process of t-tests and ANOVA was (partially) lost when I had to explain R outputs to my students so that they could interpret the results correctly. Although most of the time it simply boiled down to pointing out what to look for in the outputs (i.e., p-values), I was still losing quite a lot of time because these outputs were, in my opinion, too detailed for most real-life applications and for students in introductory classes. In other words, too much information seemed to be confusing for many people so I was still not convinced that it was the most optimal way to share statistical results to nonscientists.

Of course, they came to me for statistical advices, so they expected to have these results and I needed to give them answers to their questions and hypotheses. Nonetheless, I wanted to find a better way to communicate these results to this type of audience, with the minimum of information required to arrive at a conclusion. No more and no less than that.

After a long time spent online trying to figure out a way to present results in a more concise and readable way, I discovered the {ggpubr} package. This package allows to indicate the test used and the p-value of the test directly on a ggplot2-based graph. It also facilitates the creation of publication-ready plots for non-advanced statistical audiences.

After many refinements and modifications of the initial code (available in this article), I finally came up with a rather stable and robust process to perform t-tests and ANOVA for more than one variable at once, and more importantly, make the results concise and easily readable by anyone (statisticians or not).

A graph is worth a thousand words, so here are the exact same tests than in the previous section, but this time with my new R routine:

library(ggpubr)

# Edit from here #
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method <- "t.test" # one of "wilcox.test" or "t.test"
paired <- FALSE # if paired make sure that in the dataframe you have first all individuals at T1, then all individuals again at T2
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    if (paired == TRUE) {
      p <- ggpaired(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]), line.color = "gray", line.size = 0.4,
        palette = "npg",
        legend = "none",
        xlab = colnames(dat[j]),
        ylab = colnames(dat[i]),
        add = "jitter"
      )
    } else {
      p <- ggboxplot(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]),
        palette = "npg",
        legend = "none",
        add = "jitter"
      )
    }
    #  Add p-value
    print(p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
      method = method,
      paired = paired,
      # group.by = NULL,
      ref.group = NULL
    ))
  }
}

As you can see from the graphs above, only the most important information is presented for each variable:

a visual comparison of the groups thanks to boxplots
the name of the statistical test
the p-value of the test

Of course, experts may be interested in more advanced results. However, this simple yet complete graph, which includes the name of the test and the p-value, gives all the necessary information to answer the question: “Are the groups different?”.

In my experience, I have noticed that students and professionals (especially those from a less scientific background) understand way better these results than the ones presented in the previous section.

The only lines of code that need to be modified for your own project is the name of the grouping variable (Species in the above code), the names of the variables you want to test (Sepal.Length, Sepal.Width, etc.),² whether you want to apply a t-test (t.test) or Wilcoxon test (wilcox.test) and whether the samples are paired or not (FALSE if samples are independent, TRUE if they are paired).

Based on these graphs, it is easy, even for non-experts, to interpret the results and conclude that the versicolor and virginica species are significantly different in terms of all 4 variables (since all p-values $< \frac{0.05}{4} = 0.0125$ (remind that the Bonferroni correction is applied to avoid the issue of multiple testing, so we divide the usual $\alpha$ level by 4 because there are 4 t-tests)).

Additional p-value adjustment methods

If you would like to use another p-value adjustment method, you can use the p.adjust() function. Below are the raw p-values found above, together with p-values derived from the main adjustment methods (presented in a dataframe):

raw_pvalue <- numeric(length = length(1:4))
for (i in (1:4)) {
  raw_pvalue[i] <- t.test(dat[, i] ~ dat$Species,
    alternative = "two.sided"
  )$p.value
}

df <- data.frame(
  Variable = names(dat[, 1:4]),
  raw_pvalue = round(raw_pvalue, 3)
)

df$Bonferroni <-
  p.adjust(df$raw_pvalue,
    method = "bonferroni"
  )
df$BH <-
  p.adjust(df$raw_pvalue,
    method = "BH"
  )
df$Holm <-
  p.adjust(df$raw_pvalue,
    method = "holm"
  )
df$Hochberg <-
  p.adjust(df$raw_pvalue,
    method = "hochberg"
  )
df$Hommel <-
  p.adjust(df$raw_pvalue,
    method = "hommel"
  )
df$BY <-
  round(p.adjust(df$raw_pvalue,
    method = "BY"
  ), 3)
df

##       Variable raw_pvalue Bonferroni    BH  Holm Hochberg Hommel    BY
## 1 Sepal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 2  Sepal.Width      0.002      0.008 0.002 0.002    0.002  0.002 0.004
## 3 Petal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 4  Petal.Width      0.000      0.000 0.000 0.000    0.000  0.000 0.000

Regardless of the p-value adjustment method, the two species are different for all 4 variables. Note that the adjustment method should be chosen before looking at the results to avoid choosing the method based on the results.

Below another function that allows to perform multiple Student’s t-tests or Wilcoxon tests at once and choose the p-value adjustment method. The function also allows to specify whether samples are paired or unpaired and whether the variances are assumed to be equal or not. (The code has been adapted from Mark White’s article.)

t_table <- function(data, dvs, iv,
                    var_equal = TRUE,
                    p_adj = "none",
                    alpha = 0.05,
                    paired = FALSE,
                    wilcoxon = FALSE) {
  if (!inherits(data, "data.frame")) {
    stop("data must be a data.frame")
  }

  if (!all(c(dvs, iv) %in% names(data))) {
    stop("at least one column given in dvs and iv are not in the data")
  }

  if (!all(sapply(data[, dvs], is.numeric))) {
    stop("all dvs must be numeric")
  }

  if (length(unique(na.omit(data[[iv]]))) != 2) {
    stop("independent variable must only have two unique values")
  }

  out <- lapply(dvs, function(x) {
    if (paired == FALSE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]], var.equal = var_equal)
    } else if (paired == FALSE & wilcoxon == TRUE) {
      tres <- wilcox.test(data[[x]] ~ data[[iv]])
    } else if (paired == TRUE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]],
        var.equal = var_equal,
        paired = TRUE
      )
    } else {
      tres <- wilcox.test(data[[x]] ~ data[[iv]],
        paired = TRUE
      )
    }

    c(
      p_value = tres$p.value
    )
  })

  out <- as.data.frame(do.call(rbind, out))
  out <- cbind(variable = dvs, out)
  names(out) <- gsub("[^0-9A-Za-z_]", "", names(out))

  out$p_value <- p.adjust(out$p_value, p_adj)
  out$conclusion <- ifelse(out$p_value < alpha,
    paste0("Reject H0 at ", alpha * 100, "%"),
    paste0("Do not reject H0 at ", alpha * 100, "%")
  )
  out$p_value <- ifelse(out$p_value < 0.001,
    "<0.001",
    round(out$p_value, 3)
  )

  return(out)
}

Applied to our dataset, with no adjustment method for the p-values:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

And with the Holm (1979) adjustment method:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species",
  p_adj = "holm"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

Again, with the Holm’s adjustment method, we conclude that, at the 5% significance level, the two species are significantly different from each other in terms of all 4 variables.

ANOVA

Below the same process with an ANOVA. Note that we reload the dataset iris to include all three Species this time:

dat <- iris

# Edit from here
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("setosa", "versicolor"), c("setosa", "virginica"), c("versicolor", "virginica")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

Like the improved routine for the t-test, I have noticed that students and non-expert professionals understand ANOVA results presented this way much more easily compared to the default R outputs.

With one graph for each variable, it is easy to see that all species are different from each other in terms of all 4 variables.³

If you want to apply the same automated process to your data, you will need to modify the name of the grouping variable (Species), the names of the variables you want to test (Sepal.Length, etc.), whether you want to perform an ANOVA (anova) or Kruskal-Wallis test (kruskal.test) and finally specify the comparisons for the post-hoc tests.⁴

To go even further

As we have seen, these two improved R routines allow to:

Perform t-tests and ANOVA on a small or large number of variables with only minor changes to the code. I basically only have to replace the variable names and the name of the test I want to use. It takes almost the same time to test one or several variables so it is quite an improvement compared to testing one variable at a time.
Share test results in a much proper and cleaner way. This is possible thanks to a graph showing the observations by group and the p-value of the appropriate test included directly on the graph. This is particularly important when communicating results to a wider audience or to people from diverse backgrounds.

However, like most of my R routines, these two pieces of code are still a work in progress. Below are some additional features I have been thinking of and which could be added in the future to make the process of comparing two or more groups even more optimal:

Add the possibility to select variables by their numbering in the dataframe. For the moment it is only possible to do it via their names. This will allow to automate the process even further because instead of typing all variable names one by one, we could simply type 4:25 (to test variables 4 to 25 for instance).
Add the possibility to choose a p-value adjustment method. Currently, raw p-values are displayed in the graphs and I manually adjust them afterwards or adjust the $\alpha$.
When comparing more than two groups, it is only possible to apply an ANOVA or Kruskal-Wallis test at the moment. A major improvement would be to add the possibility to perform a repeated measures ANOVA (i.e., an ANOVA when the samples are dependent). It is currently already possible to do a t-test with two paired samples, but it is not yet possible to do the same with more than two groups.
Another less important (yet still nice) feature when comparing more than 2 groups would be to automatically apply post-hoc tests only in the case where the null hypothesis of the ANOVA or Kruskal-Wallis test is rejected (so when there is at least one group different from the others, because if the null hypothesis of equal groups is not rejected we do not apply a post-hoc test). At the present time, I manually add or remove the code that displays the p-values of post-hoc tests depending on the global p-value of the ANOVA or Kruskal-Wallis test.

I will try to add these features in the future, or I would be glad to help if the author of the {ggpubr} package needs help in including these features (I hope he will see this article!).

Last but not least, the following packages may be of interest to some readers:

If you want to report statistical results on a graph, I advise you to check the {ggstatsplot} package and in particular the ggbetweenstats() and ggwithinstats() functions. These functions allow to compare a continuous variable across multiple groups or conditions (for both independent and paired samples). Two advantages of the functions is that:
- it is very easy to switch from parametric to nonparametric tests and
- it automatically runs an ANOVA or t-test depending on the number of groups to compare

Note that many different statistical results are displayed on the graph, not only the name of the test and the p-value so a bit of simplicity and clarity is lost for more precision. However, it is still very convenient to be able to include tests results on a graph in order to combine the advantages of a visualization and a sound statistical analysis. Something that I still need to figure out is how to run the code on several variables at once.

The {compareGroups} package also provides a nice way to compare groups. It comes with a really complete Shiny app, available with:

# install.packages("compareGroups")
library(compareGroups)
cGroupsWUI()

Update with the `{ggstatsplot}` package

Several months after having written this article, I finally found a way to plot and run analyses on several variables at once with the package {ggstatsplot} (Patil 2021). This was the main feature I was missing and which prevented me from using it more often.

Although I still find that too much statistical details are displayed (in particular for non experts), I still believe the ggbetweenstats() and ggwithinstats() functions are worth mentioning in this article. I actually now use those two functions almost as often as my previous routines because:

I do not have to care about the number of groups to compare, the functions automatically choose the appropriate test according to the number of groups (ANOVA for 3 groups or more, and t-test for 2 groups)
I can select variables based on their column numbering, and not based on their names anymore (which prevents me from writing those variable names manually)
When comparing 3 or more groups (so for ANOVA, Kruskal-Wallis, repeated measure ANOVA or Friedman), $p$-values of the post-hoc tests within each dependent variable are by default the adjusted $p$-values (Holm is the default but many adjustment methods are available)
It is possible to compare both independent and paired samples, no matter the number of groups (remember that with the ggpubr package I could only do paired samples for two samples, not for 3 samples)
They allow to easily switch between the parametric and nonparametric version
All this in a more concise manner using the {purrr} package

For those of you who are interested, below my updated R routine which include these functions and applied this time on the penguins dataset.

library(palmerpenguins)

dat <- penguins
str(dat)

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

We illustrate the routine for two groups with the variables sex (two factors) as independent variable, and the 4 quantitative continuous variables bill_length_mm, bill_depth_mm, flipper_length_mm and body_mass_g as dependent variables:

library(ggstatsplot)
library(tibble)

# Comparison between sexes

# edit from here
x <- "sex"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

We now illustrate the routine for 3 groups or more with the variable species (three factors) as independent variable, and the 4 same dependent variables:

# Comparison between species

# edit from here
x <- "species"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

As you can see, I only have to specify:

the name of the grouping variable (sex and species),
the number of the dependent variables (variables 3 to 6 in the dataset),
whether I want to use the parametric or nonparametric version and
whether samples are independent (paired = FALSE) or paired (paired = TRUE).

Everything else is automated—the outputs show a graphical representation of what we are comparing, together with the details of the statistical analyses in the subtitle of the plot (the $p$-value among others).

Note that the code shown above is actually the same if I want to compare 2 groups or more than 2 groups. I wrote twice the same code (once for 2 groups and once again for 3 groups) for illustrative purposes only, but they are the same and should be treated as one for your projects.

I must admit I am quite satisfied with this routine, now that:

I can automate it on many variables at once and I do not need to write the variable names manually anymore,
at the same time, I can choose the appropriate test among all the available ones (depending on the number of groups, whether they are paired or not, and whether I want to use the parametric or nonparametric version).

Nonetheless, I must also admit that I am still not satisfied with the level of details of the statistical results. As already mentioned, many students get confused and get lost in front of so much information (except the $p$-value and the number of observations, most of the details are rather obscure to them because they are not covered in introductory statistic classes).

I saved time thanks to all improvements in comparison to my previous routine, but I definitely lose time when I have to point out to them what they should look for. After discussing with other professors, I noticed that they have the same problem.

For the moment, you can only print all results or none. I have opened an issue kindly requesting to add the possibility to display only a summary (with the $p$-value and the name of the test for instance).⁵ I will update again this article if the maintainer of the package includes this feature in the future. So stay tuned!

Conclusion

Thanks for reading.

I hope this article will help you to perform t-tests and ANOVA for multiple variables at once and make the results more easily readable and interpretable by non-scientists. Learn more about the t-test to compare two groups, or the ANOVA to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 65–70.

McDonald, JH. 2014. “Multiple Tests.” Handbook of Biological Statistics. 3rd Ed Baltimore, Maryland: Sparky House Publishing, 233–36.

Patil, Indrajeet. 2021. “Visualizations with statistical details: The ’ggstatsplot’ approach.” Journal of Open Source Software 6 (61): 3167. https://doi.org/10.21105/joss.03167.

In theory, an ANOVA can also be used to compare two groups as it will give the same results compared to a Student’s t-test, but in practice we use the Student’s t-test to compare two groups and the ANOVA to compare three groups or more.↩︎
Do not forget to separate the variables you want to test with |.↩︎
Do not forget to adjust the $p$-values or the significance level $\alpha$. If you use the Bonferroni correction, the adjusted $\alpha$ is simply the desired $\alpha$ level divided by the number of comparisons.↩︎
Post-hoc test is only the name used to refer to a specific type of statistical tests. Post-hoc test includes, among others, the Tukey HSD test, the Bonferroni correction, Dunnett’s test. Even if an ANOVA or a Kruskal-Wallis test can determine whether there is at least one group that is different from the others, it does not allow us to conclude which are different from each other. For this purpose, there are post-hoc tests that compare all groups two by two to determine which ones are different, after adjusting for multiple comparisons. Concretely, post-hoc tests are performed to each possible pair of groups after an ANOVA or a Kruskal-Wallis test has shown that there is at least one group which is different (hence “post” in the name of this type of test). The null and alternative hypotheses and the interpretations of these tests are similar to a Student’s t-test for two samples.↩︎
I am open to contribute to the package if I can help!↩︎

How to perform a one-sample t-test by hand and in R: test on one mean

Mon, 09 Mar 2020 00:00:00 +0000

Introduction

After having written an article on the Student’s t-test for two samples (independent and paired samples), I believe it is time to explain in details how to perform one-sample t-tests by hand and in R.

One-sample t-test is an important part of inferential statistics (probably one of the first statistical test that students learn). Remind that, unlike descriptive statistics, inferential statistics is a branch of statistics aiming at drawing conclusions about one or two populations, based on a subset (or two) of that population (called samples). In other words, we first collect a random set of observations from a population, and then some measurements are calculated in order to generalize to the population the information found through the sample.

In this context, the one-sample t-test is used to determine whether the mean of a measurement variable is different from a specified value (a belief or a theoretical expectation for example). It works as follows: if the mean of the sample is too distant from the specified value (the value under the null hypothesis), it is considered that the mean of the population is different from what is expected. On the contrary, if the mean of the sample is close to the specified value, we cannot reject the hypothesis that the population mean is equal to what is expected.

Like the Student’s t-test for two samples and the ANOVA (for 3 or more samples), there are also different versions of the one-sample t-test. Luckily, there are only two different versions for this test (the Student’s t-test for two samples has 5 versions!). The difference between the two versions of the one-sample t-test lies in the fact that one version is used when the variance of the population (not the variance of the sample!) is known, the other version being used when the variance of the population is unknown.

In this article, I will first detail step by step how to perform both versions of the one-sample t-test by hand. The analyses will be done on a small set of observations for the sake of illustration and easiness. I will then show how to perform this test in R with the exact same data in order to verify the results found by hand. Reminders about the reasoning behind hypothesis tests, interpretations of the p-value and the results, and assumptions of this test will also be presented.

Note that the aim of this article is to show how to compute the one-sample t-test by hand and in R, so we refrain from testing the assumptions and we assume all assumptions are met for this exercise. For completeness, we still mention the assumptions and how to test them. Interested readers are invited to have a look at the end of the article for more information about these assumptions.

Null and alternative hypothesis

Before diving into the computations of the one-sample t-test by hand, let’s recap the null and alternative hypotheses of this test:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu \ne \mu_0$

where $\mu$ is the population mean and $\mu_0$ is the known or hypothesized value of the mean in the population.

This is in the general case where we simply want to determine whether the population mean is different (in terms of the dependent variable) compared to the hypothesized value. In this sense, we have no prior belief about the population mean being larger or smaller than the hypothesized value. This type of test is referred as a two-sided or bilateral test.

If we have some prior beliefs about the population mean being larger or smaller than the hypothesized value, the one-sample t-test also allows to test the following hypotheses:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu > \mu_0$

$H_0$: $\mu = \mu_0$
$H_1$: $\mu < \mu_0$

In the first case, we want to test if the population mean is significantly larger than the hypothesized value, while in the latter case, we want to test if the population mean is significantly smaller than the hypothesized value. This type of test is referred as a one-sided or unilateral test.

Hypothesis testing

In statistics, many statistical tests is in the form of hypothesis tests. Hypothesis tests are used to determine whether a certain belief can be deemed as true (plausible) or not, based on the data at hand (i.e., the sample(s)). Most hypothesis tests boil down to the following 4 steps:¹

State the null and alternative hypothesis.
Compute the test statistic, denoted t-stat. Formulas to compute the test statistic differ among the different versions of the one-sample t-test but they have the same structure. See scenarios 1 and 2 below to see the different formulas.
Find the critical value given the theoretical statistical distribution of the test, the parameters of the distribution and the significance level $\alpha$. For the two versions of the one-sample t-test, it is either the normal or the Student’s t distribution (t denoting the Student distribution and z denoting the normal distribution).
Conclude by comparing the t-stat (found in step 2.) with the critical value (found in step. 3). If the t-stat lies in the rejection region (determined thanks to the critical value and the direction of the test), we reject the null hypothesis, otherwise we do not reject the null hypothesis. These two alternatives (reject or do not reject the null hypothesis) are the only two possible solutions, we never “accept” an hypothesis. It is also a good practice to always interpret the decision in the terms of the initial question.

For the interested reader, see these 4 steps of hypothesis testing in more details in this article.

Two versions of the one-sample t-test

There are two versions of the one-sample t-test, depending on whether the variance of the population (not the variance of the sample!) is known or unknown. This criteria is rather straightforward, we either know the variance of the population or we do not. The variance of the population cannot be computed because if you can compute the variance of a population, it means you have the data for the whole population, then there is no need to do a hypothesis test anymore…

So the variance of the population is either given in the statement (use them in that case), or there is no information about the variance and in that case, it is assumed that the variance is unknown. In practice, the variance of the population is most of the time unknown. However, we still illustrate how to do both versions of this test by hand and in R in the next sections following the 4 steps of a hypothesis test.

How to compute the one-sample t-test by hand?

Note that the data are artificial and do not represent any real variable. Furthermore, remind that the assumptions may or may not be met. The point of the article is to detail how to compute the different versions of the test by hand and in R, so all assumptions are assumed to be met. Moreover, we assume that for all tests the significance level, that is, the type I error is $\alpha = 5$%.

If you are interested in applying these tests by hand without having to do the computations yourself, here is a Shiny app which does it for you. You just need to enter the data and choose the appropriate version of the test thanks to the sidebar menu. There is also a graphical representation that helps you to visualize the test statistic and the rejection region. I hope you will find it useful!

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

value
0.9
-0.8
1.3
-0.3
1.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 0.56$
variance of the population: $\sigma^2 = 1$
$\mu_0 = 0$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 0$ and $H_1: \mu \ne 0$. ($\ne$ because we want to test whether the population mean is different from 0, we do not impose a direction in the test.)
Test statistic: \[z_{obs} = \frac{\bar{x} - \mu_0}{\frac{\sigma}{\sqrt{n}}} = \frac{0.56-0}{0.447} = 1.252\]
Critical value: $\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the population mean is equal to 0, or there is no sufficient evidence in the data to conclude that the population mean is different from 0.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

value
7.9
5.8
6.3
7.3
6.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 6.8$
standard deviation of the sample: $s = 0.825$
$\mu_0 = 5$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 5$ and $H_1: \mu > 5$. (> because we want to test whether the population mean is larger than 5.)
Test statistic: \[t_{obs} = \frac{\bar{x} - \mu_0}{\frac{s}{\sqrt{n}}} = \frac{6.8-5}{0.369} = 4.881\]
Critical value: $t_{\alpha, n - 1} = t_{0.05, 4} = 2.132$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection region is thus from 2.132 to $+\infty$. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the population mean is larger than 5.

This concludes how to perform the two versions of the one-sample t-test by hand. In the next sections, we detail how to perform the exact same tests in R.

Different underlying distributions for the critical value

As you may have noticed, the underlying probability distributions used to find the critical value are different depending on whether the variance of the population is known or unknown.

The underlying probability distribution when the variance is known (scenario 1) is the normal distribution, while the probability distribution in the case where the variance is unknown (scenario 2) is the Student’s t distribution. This difference is partially explained by the fact that when the variance of the population is unknown, there is more “uncertainty” in the data, so we need to use the Student’s t distribution instead of the normal distribution.

Note that when the sample size is large (usually when n > 30), the Student’s t distribution tends to a normal distribution. Using a normal distribution when the variance is known and a Student’s t distribution when the variance is unknown also applies to a t-test for two samples.

How to compute the one-sample t-test in R?

A good practice before doing t-tests in R is to visualize the data thanks to a boxplot (or eventually a histogram or a density plot). A boxplot gives a first indication on the location of the sample, and thus, a first indication on whether the null hypothesis is likely to be rejected or not. However, even if a boxplot or a density plot is great in showing the distribution of a sample, only a sound statistical test will confirm our first impression.

After a visualization of the data, we replicate in R the results found by hand. Note that we use the same data, the same assumptions and the same question for both scenarios to facilitate the comparison between the tests performed by hand and in R.

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

dat1 <- data.frame(
  value = c(0.9, -0.8, 1.3, -0.3, 1.7)
)

dat1

##   value
## 1   0.9
## 2  -0.8
## 3   1.3
## 4  -0.3
## 5   1.7

library(ggplot2)

ggplot(dat1) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Note that you can use the {esquisse} RStudio addin if you want to draw a boxplot with the package {ggplot2} without writing the code yourself. If you prefer the default graphics, use the boxplot() function:

boxplot(dat1$value)

The boxplot shows that the distribution of the sample is not distant from 0 (the hypothesized value), so we tend to believe that we will not be able to reject the null hypothesis that the population mean is equal to 0. However, only a formal statistical test will confirm this belief.

Below a function to perform a t-test with a known population variance, with arguments accepting the sample (x), the variance of the population (V), the mean under the null hypothesis (m0, default is 0), the significance level (alpha, default is 0.05) and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test2 <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test2(dat1$value,
  V = 1
)
test

## $mean
## [1] 0.56
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 1.252198
## 
## $p.value
## [1] 0.2104977
## 
## $LCL
## [1] -0.3165225
## 
## $UCL
## [1] 1.436523
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the sample mean, the hypothesized value and the population variance (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.2104977

The p-value is 0.21 so at the 5% significance level we do not reject the null hypothesis. There is no sufficient evidence in the data to reject the hypothesis that the population mean is equal to 0. This result confirms what we found by hand.

Note that a similar function exists in the {BSDA} package:²

library(BSDA)

z.test(dat1$value,
  alternative = "two.sided",
  mu = 0,
  sigma.x = 1,
  conf.level = 0.95
)

## 
## 	One-sample z-Test
## 
## data:  dat1$value
## z = 1.2522, p-value = 0.2105
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.3165225  1.4365225
## sample estimates:
## mean of x 
##      0.56

If you are unfamiliar with the concept of p-value, I invite you to read my note on p-value and significance level $\alpha$.

To sum up what have been said in that article about p-value and significance level $\alpha$:

If the p-value is smaller than the predetermined significance level $\alpha$ (usually 5%) so if p-value < 0.05, we reject the null hypothesis
If the p-value is greater than or equal to the predetermined significance level $\alpha$ (usually 5%) so if p-value $\ge$ 0.05, we do not reject the null hypothesis

This applies to all statistical tests without exception. Of course, the null and alternative hypotheses change depending on the test.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

dat2 <- data.frame(
  value = c(7.9, 5.8, 6.3, 7.3, 6.7)
)

dat2

##   value
## 1   7.9
## 2   5.8
## 3   6.3
## 4   7.3
## 5   6.7

ggplot(dat2) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the box is quite distant from the hypothesized value of 5. From this boxplot, we can expect the test to reject the null hypothesis of the population mean being equal to 5. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” t-test for one-sample. Note that in our case the alternative hypothesis is $H_1: \mu > 5$ so we need to add the arguments mu = 5 and alternative = "greater" to the function because the default arguments are mu = 0 and the two-sided test:

test <- t.test(dat2$value,
  mu = 5,
  alternative = "greater"
)

test

## 
## 	One Sample t-test
## 
## data:  dat2$value
## t = 4.8809, df = 4, p-value = 0.004078
## alternative hypothesis: true mean is greater than 5
## 95 percent confidence interval:
##  6.013814      Inf
## sample estimates:
## mean of x 
##       6.8

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used, the hypothesized value and the sample mean (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.004077555

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the population mean is significantly larger than 5. This result confirms what we found by hand.

Confidence interval

Note that the confidence interval can be extracted with $conf.int:

test$conf.int

## [1] 6.013814      Inf
## attr(,"conf.level")
## [1] 0.95

You can see that the 95% confidence interval for the population mean is $[6.01; \infty]$, meaning that, at the significance level $\alpha = 5$%, we reject the null hypothesis as long as the hypothesized value $\mu_0$ is below 6.01, otherwise the null hypothesis cannot be rejected.

Combination of plot and statistical test

After having written this article, I discovered the {ggstatsplot} package which I believe is worth mentioning here, in particular the gghistostats() function for one-sample Student’s t-test.

This function combines a histogram—representing the distribution—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for scenario 2. Unfortunately, the package does not allow to run the test for scenario 1.

Scenario 2: variance of the population is unknown

# load packages
library(ggstatsplot)
library(ggplot2)

# plot with stat test
gghistostats(
  data = dat2, # dataframe from which variable is to be taken
  x = value, # numeric variable whose distribution is of interest
  type = "parametric", # for student's t-test
  test.value = 5 # default value is 0
) +
  labs(caption = NULL) # remove caption

The p-value is displayed after p = in the subtitle of the plot. Based on this plot and the p-value being lower than 5% (p-value = 0.008), we reject the null hypothesis that the population mean is equal to 5.

Note that, the p-value is two times as large as the one obtained with the t.test() function because when we ran t.test() we specified alternative = "greater" (i.e., a one-sided test). In our plot with the gghistostats() function, it is a two-sided test that is performed by default, that is, alternative = "two.sided".

Assumptions

Variable type: The dependent variable (i.e., the measured variable) must be measured on a continuous or ordinal scale.
Independence: The data, collected from a representative and randomly selected portion of the population, should be independent of one another. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another). If not, it is most likely that you have independent samples.
Normality:
- With a small sample size (usually $n < 30$), observations should follow a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. Some transformations, such as among others, the logarithm, the square root or the Box-Cox transformation can be applied on the observations to transform you data to better fit the normal distribution. If, even after a transformation, your data still do not follow a normal distribution, the one-sample Wilcoxon test (wilcox.test(variable_name, data = dat in R) can be applied. This non-parametric test is robust to non normal distributions so it does not require normality of the data.
- With a large sample size ($n \ge 30$), normality of the data is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). It is therefore not required to test the normality assumption when the number of observations is large.
Outliers: There should be no outliers in your data. An observation slightly different from the others does not pose a problem, but it starts to be an issue when you have at least one extreme outlier. In presence of at least one extreme outlier, it is best to transform your data (with the logarithm transformation for instance, as you would do with a non normal distribution) or use the non-parametric one-sample Wilcoxon test.

Conclusion

Thanks for reading.

I hope this article helped you to understand how the different versions of the one-sample t-test work and how to perform them by hand and in R.

If you are interested, here is a Shiny app to perform these tests by hand easily (you just need to enter your data and select the appropriate version of the test thanks to the sidebar menu). Moreover, read this article if you would like to know how to compute the Student’s t-test but this time, for two samples—in order to compare two dependent or independent groups—or this article if you want to use an ANOVA to compare 3 or more groups.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

It is a least the case regarding parametric hypothesis tests. A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. In the case of the one-sample t-test, it is based on the Student’s t distribution with a single parameter, the degrees of freedom ($df = n - 1$ where $n$ is the sample size), or the normal distribution.↩︎
Thanks gmacar for pointing it out to me.↩︎

Student's t-test in R and by hand: how to compare two groups under different scenarios?

Fri, 28 Feb 2020 00:00:00 +0000

Introduction

One of the most important test within the branch of inferential statistics is the Student’s t-test.¹ The Student’s t-test for two samples is used to test whether two groups (two populations) are different in terms of a quantitative variable, based on the comparison of two samples drawn from these two groups. In other words, a Student’s t-test for two samples allows to determine whether the two populations from which your two samples are drawn are different (with the two samples being measured on a quantitative continuous variable).²

The reasoning behind this statistical test is that if your two samples are markedly different from each other, it can be assumed that the two populations from which the samples are drawn are different. On the contrary, if the two samples are rather similar, we cannot reject the hypothesis that the two populations are similar, so there is no sufficient evidence in the data at hand to conclude that the two populations from which the samples are drawn are different. Note that this statistical tool belongs to the branch of inferential statistics because conclusions drawn from the study of the samples are generalized to the population, even though we do not have the data on the entire population.

To compare two samples, it is usual to compare a measure of central tendency computed for each sample. In the case of the Student’s t-test, the mean is used to compare the two samples. However, in some cases, the mean is not appropriate to compare two samples so the median is used to compare them via the Wilcoxon test. This article being already quite long and complete, the Wilcoxon test is covered in a separate article, together with some illustrations on when to use one test or the other.

These two tests (Student’s t-test and Wilcoxon test) have the same final goal, that is, compare two samples in order to determine whether the two populations from which they were drawn are different or not. Note that the Student’s t-test is more powerful than the Wilcoxon test (i.e., it more often detects a significant difference if there is a true difference, so a smaller difference can be detected with the Student’s t-test) but the Student’s t-test is sensitive to outliers and data asymmetry. Furthermore, within each of these two tests, several versions exist, with each version using different formulas to arrive at the final result. It is thus necessary to understand the difference between the two tests and which version to use in order to carry out the appropriate analyses depending on the question and the data at hand.

In this article, I will first detail step by step how to perform all versions of the Student’s t-test for independent and paired samples by hand. The analyses will be done on a small set of observations for the sake of illustration and easiness. I will then show how to perform this test in R with the exact same data in order to verify the results found by hand. Reminders about the reasoning behind hypothesis testing, interpretations of the p-value and the results, and assumptions of this test will also be presented.

Note that the aim of this article is to show how to compute the Student’s t-test by hand and in R, so we refrain from testing the assumptions and we assume all of them are met for this exercise. For completeness, we still mention the assumptions, how to test them and what other tests exist if one is not met. Interested readers are invited to have a look at the end of the article for more information about these assumptions.

Null and alternative hypothesis

Before diving into the computations of the Student’s t-test by hand, let’s recap the null and alternative hypotheses of this test:

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 \ne \mu_2$

where $\mu_1$ and $\mu_2$ are the means of the two populations from which the samples were drawn.

As mentioned in the introduction, although technically the Student’s t-test is based on the comparison of the means of the two samples, the final goal of this test is actually to test the following hypotheses:

$H_0$: the two populations are similar
$H_1$: the two populations are different

This is in the general case where we simply want to determine whether the two populations are different or not (in terms of the dependent variable). In this sense, we have no prior belief about a particular population mean being larger or smaller than the other. This type of test is referred as a two-sided or bilateral test.

If we have some prior beliefs about one population mean being larger or smaller than the other, the Student’s t-test also allows to test the following hypotheses:

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 > \mu_2$

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 < \mu_2$

In the first case, we want to test if the mean of the first population is significantly larger than the mean of the second, while in the latter case, we want to test if the mean of the first population is significantly smaller than the mean of the second. This type of test is referred as a one-sided or unilateral test.

Some authors argue that one-sided tests should not be used in practice for the simple reason that, if a researcher is so sure that the mean of one population is larger (smaller) than the mean of the other and would never be smaller (larger) than the other, why would she needs to test for significance at all? This a rather philosophical question and it is beyond the scope of this article. Interested readers are invited to see part of the discussion in Rowntree (2000).

Hypothesis testing

State the null and alternative hypothesis.
Compute the test statistic, denoted t-stat. Formulas to compute the test statistic differ among the different versions of the Student’s t-test but they have the same structure. See scenarios 1 to 5 below to see the different formulas.
Find the critical value given the theoretical statistical distribution of the test, the parameters of the distribution and the significance level $\alpha$. For a Student’s t-test and its extended version, it is either the normal or the Student’s t distribution (t denoting the Student distribution and z denoting the normal distribution).
Conclude by comparing the t-stat (found in step 2.) with the critical value (found in step. 3). If the t-stat lies in the rejection region (determined thanks to the critical value and the direction of the test), we reject the null hypothesis, otherwise we do not reject the null hypothesis. These two alternatives (reject or do not reject the null hypothesis) are the only two possible solutions, we never “accept” an hypothesis. It is also a good practice to always interpret the decision in the terms of the initial question.

For the interested reader, see these 4 steps of hypothesis testing in more details in this article.

Different versions of the Student’s t-test

There are several versions of the Student’s t-test for two samples, depending on whether the samples are independent or paired and depending on whether the variances of the populations are (un)equal and/or (un)known:

On the one hand, independent samples means that the two samples are collected on different experimental units or different individuals, for instance when we are working on women and men separately, or working on patients who have been randomly assigned to a control and a treatment group (and a patient belongs to only one group). On the other hand, we face paired samples when measurements are collected on the same experimental units, same individuals. This is often the case, for example in medical studies, when testing the efficiency of a treatment at two different times. The same patients are measured twice, before and after the treatment, and the dependency between the two samples must be taken into account in the computation of the test statistic by working on the differences of measurements for each subject. Paired samples are usually the result of measurements at two different times, but not exclusively. Suppose we want to test the difference in vision between the left and right eyes of 50 athletes. Although the measurements are not made at two different time (before-after), it is clear that both eyes are dependent within each subject. Therefore, the Student’s t-test for paired samples should be used to account for the dependency between the two samples instead of the standard Student’s t-test for independent samples.

Another criteria for choosing the appropriate version of the Student’s t-test is whether the variances of the populations (not the variances of the samples!) are known or unknown and equal or unequal. This criteria is rather straightforward, we either know the variances of the populations or we do not. The variances of the populations cannot be computed because if you can compute the variance of a population, it means you have the data for the whole population, then there is no need to do a hypothesis test anymore… So the variances of the populations are either given in the statement (use them in that case), or there is no information about these variances and in this case, it is assumed that the variances are unknown. In practice, the variances of the populations are most of the time unknown and the only thing to do in order to choose the appropriate version of the test is to check whether the variances are equal or not. However, we still illustrate how to do all versions of this test by hand and in R in the next sections following the 4 steps of hypothesis testing.

How to compute Student’s t-test by hand?

Note that the data are artificial and do not represent any real variable. Furthermore, remind that the assumptions may or may not be met. The point of the article is to detail how to compute the different versions of the test by hand and in R, so all assumptions are assumed to be met. Moreover, assume that the significance level $\alpha = 5$% for all tests.

Scenario 1: Independent samples with 2 known variances

For the first scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances $\sigma^2 = 1$ in both populations and that we would like to test whether the two population means are different.

value	sample
0.9	1
-0.8	1
0.1	1
-0.3	1
0.2	1
0.8	2
-0.9	2
-0.1	2
0.4	2
0.1	2

So we have:

5 observations in each sample: $n_1 = n_2 = 5$
mean of sample 1: $\bar{x}_1 = 0.02$
mean of sample 2: $\bar{x}_2 = 0.06$
variances of both populations: $\sigma^2_1 = \sigma^2_2 = 1$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 \ne 0$. ($\ne$ because we want to test whether the two means are different, we do not impose a direction in the test.)
Test statistic: \[z_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2}}}\] \[= \frac{0.02-0.06-0}{0.632} = -0.063\]
Critical value: $\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the two population means are the same, or there is no sufficient evidence in the data to conclude that the two populations considered are different.

Scenario 2: Independent samples with 2 equal but unknown variances

For the second scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown but equal ($\sigma^2_1 = \sigma^2_1$) and that we would like to test whether the mean of population 1 is larger than the mean of population 2.

value	sample
1.78	1
1.5	1
0.9	1
0.6	1
0.8	1
1.9	1
0.8	2
-0.7	2
-0.1	2
0.4	2
0.1	2

So we have:

6 observations in sample 1: $n_1 = 6$
5 observations in sample 2: $n_2 = 5$
mean of sample 1: $\bar{x}_1 = 1.247$
mean of sample 2: $\bar{x}_2 = 0.1$
variance of sample 1: $s^2_1 = 0.303$
variance of sample 2: $s^2_1 = 0.315$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 > 0$. (> because we want to test if the mean of the first population is larger than the mean of the second population.)
Test statistic: \[t_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}\] where \[s_p = \sqrt{\frac{(n_1-1)s^2_1+ (n_2 - 1)s^2_2}{n_1 + n_2 - 2}} = 0.555\] so \[t_{obs} = \frac{1.247-0.1-0}{0.555 * 0.606} = 3.411\] (Note that as it is assumed the variances of the two populations are equal, a pooled (common) variance, denoted $s_p$, is computed.)
Critical value: $t_{\alpha, n_1 + n_2 - 2} = t_{0.05, 9} = 1.833$
Conclusion: The rejection region is thus from 1.833 to $+\infty$ (there is only one rejection region because it is a one-sided test). The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the mean of population 1 is larger than the mean of population 2.

Scenario 3: Independent samples with 2 unequal and unknown variances

For the third scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown and unequal ($\sigma^2_1 \ne \sigma^2_1$) and that we would like to test whether the mean of population 1 is smaller than the mean of population 2.

value	sample
0.8	1
0.7	1
0.1	1
0.4	1
0.1	1
1.78	2
1.5	2
0.9	2
0.6	2
0.8	2
1.9	2

So we have:

5 observations in sample 1: $n_1 = 5$
6 observations in sample 2: $n_2 = 6$
mean of sample 1: $\bar{x}_1 = 0.42$
mean of sample 2: $\bar{x}_2 = 1.247$
variance of sample 1: $s^2_1 = 0.107$
variance of sample 2: $s^2_1 = 0.303$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 < 0$. (< because we want to test if the mean of the first population is smaller than the mean of the second population.)
Test statistic: \[t_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}}\] \[= \frac{0.42-1.247-0}{0.268} = -3.084\]
Critical value: $-t_{\alpha, \upsilon}$ where \[\upsilon = \frac{\bigg(\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2} \bigg)^2}{\frac{\bigg(\frac{s^2_1}{n_1}\bigg)^2}{n_1 - 1} + \frac{\bigg(\frac{s^2_2}{n_2}\bigg)^2}{n_2 - 1}} = 8.28\] so \[-t_{0.05, 8.28} = -1.851\] Note: The degrees of freedom 8.28 does not exist in the standard Student distribution table, so simply take 8, or compute it in R with qt(p = 0.05, df = 8.28). For simplicity, this number of degrees of freedom is sometimes approximated as $df = min(n_1 - 1, n_2 - 1)$, so in this case it would be $df = 4$.
Conclusion: The rejection region is thus from $-\infty$ to -1.851. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the mean of population 1 is smaller than the mean of population 2.

Scenario 4: Paired samples where the variance of the differences is known

Student’s t-test with paired samples are a bit different than with independent samples, they are actually more similar to one sample Student’s t-test. Here is how it works. We actually compute the difference between the two samples for each pair of observations, and then we work on these differences as if we were doing a one sample Student’s t-test by computing the test statistic on these differences.

In case it is not clear, here is the fourth scenario as an illustration. Suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 ($\sigma^2_D = 1$) and that we would like to test whether the mean difference between the two populations is different than 0.

before	after
0.9	0.8
-0.8	-0.9
0.1	-0.1
-0.3	0.4
0.2	0.1

The first thing to do is to compute the differences for all pairs of observations:

before	after	difference
0.9	0.8	-0.1
-0.8	-0.9	-0.1
0.1	-0.1	-0.2
-0.3	0.4	0.7
0.2	0.1	-0.1

So we have:

number of pairs: $n = 5$
mean of the difference: $\bar{D} = 0.04$
variance of the difference in the population: $\sigma^2_D = 1$
standard deviation of the difference in the population: $\sigma_D = 1$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_D = 0$ and $H_1: \mu_D \ne 0$
Test statistic: \[z_{obs} = \frac{\bar{D} - \mu_0}{\frac{\sigma_D}{\sqrt{n}}} = \frac{0.04-0}{0.447} = 0.089\] (This formula is exactly the same than for one sample Student’s t-test with a known variance, except that we work on the mean of the differences.)
Critical value: $\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96$
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the mean difference between the two populations is equal to 0.

Scenario 5: Paired samples where the variance of the differences is unknown

For the fifth and final scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is unknown and that we would like to test whether a treatment is effective in increasing running capabilities (the higher the value, the better in terms of running capabilities).

before	after
9	16
8	11
1	15
3	12
2	9

The first thing to do is to compute the differences for all pairs of observations:

before	after	difference
9	16	7
8	11	3
1	15	14
3	12	9
2	9	7

So we have:

number of pairs: $n = 5$
mean of the difference: $\bar{D} = 8$
variance of the difference in the sample: $s^2_D = 16$
standard deviation of the difference in the sample: $s_D = 4$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_D = 0$ and $H_1: \mu_D > 0$ (> because we would like to test whether the treatment is effective, so whether the treatment has a positive impact on the running capabilities.)
Test statistic: \[t_{obs} = \frac{\bar{D} - \mu_0}{\frac{s_D}{\sqrt{n}}} = \frac{8-0}{1.789} = 4.472\] (This formula is exactly the same than for one sample Student’s t-test with an unknown variance, except that we work on the mean of the differences.)
Critical value: $t_{\alpha, n-1} = t_{0.05, 4} = 2.132$ (n is the number of pairs, not the number of observations!)
Conclusion: The rejection regions are thus from 2.132 to $+\infty$. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the treatment has a positive impact on the running capabilities (because the mean of the differences is greater than 0)

This concludes how to perform the different versions of the Student’s t-test for two samples by hand. In the next sections, we detail how to perform the exact same tests in R.

How to compute Student’s t-test in R?

A good practice before doing t-tests in R is to visualize the data by group thanks to a boxplot (or a density plot, or eventually both). A boxplot with the two boxes overlapping each other gives a first indication that the two samples are similar, and thus, that the null hypothesis of equal means may not be rejected. On the contrary, if the two boxes are not overlapping, it indicates that the two samples are not similar, and thus, that the populations may be different in terms of the considered variable. However, even if boxplots or density plots are great in showing a comparison between the two groups, only a sound statistical test will confirm our first impression.

After a visualization of the data by group, we replicate in R the results found by hand. We will see that for some versions of the t-test, there is no default function built in R (at least to my knowledge, do not hesitate to let me know in the comments if I’m mistaken). In these cases, a function is written to replicate the results by hand.

Note that we use the same data, the same assumptions and the same question for all 5 scenarios to facilitate the comparison between the tests performed by hand and in R.

Scenario 1: Independent samples with 2 known variances

dat1 <- data.frame(
  sample1 = c(0.9, -0.8, 0.1, -0.3, 0.2),
  sample2 = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat1

##   sample1 sample2
## 1     0.9     0.8
## 2    -0.8    -0.9
## 3     0.1    -0.1
## 4    -0.3     0.4
## 5     0.2     0.1

dat_ggplot <- data.frame(
  value = c(0.9, -0.8, 0.1, -0.3, 0.2, 0.8, -0.9, -0.1, 0.4, 0.1),
  sample = c(rep("1", 5), rep("2", 5))
)

library(ggplot2)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

boxplot(value ~ sample,
  data = dat_ggplot
)

The two boxes seem to overlap which illustrate that the two samples are quite similar, so we tend to believe that we will not be able to reject the null hypothesis that the two population means are similar. However, only a formal statistical test will confirm this belief.

Below a function to perform a t-test with known variances, with arguments accepting:

the two samples (x and y),
the two variances of the populations (V1 and V2),
the difference in means under the null hypothesis (m0, default is 0),
the significance level (alpha, default is 0.05)
and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_knownvar <- function(x, y, V1, V2, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M1 <- mean(x)
  M2 <- mean(y)
  n1 <- length(x)
  n2 <- length(y)
  sigma1 <- sqrt(V1)
  sigma2 <- sqrt(V2)
  S <- sqrt((V1 / n1) + (V2 / n2))
  statistic <- (M1 - M2 - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M1 - M2 - S * qnorm(1 - alpha / 2))
  UCL <- (M1 - M2 + S * qnorm(1 - alpha / 2))
  value <- list(mean1 = M1, mean2 = M2, m0 = m0, sigma1 = sigma1, sigma2 = sigma2, S = S, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_knownvar(dat1$sample1, dat1$sample2,
  V1 = 1, V2 = 1
)
test

## $mean1
## [1] 0.02
## 
## $mean2
## [1] 0.06
## 
## $m0
## [1] 0
## 
## $sigma1
## [1] 1
## 
## $sigma2
## [1] 1
## 
## $S
## [1] 0.6324555
## 
## $statistic
## [1] -0.06324555
## 
## $p.value
## [1] 0.949571
## 
## $LCL
## [1] -1.27959
## 
## $UCL
## [1] 1.19959
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the two sample means and the two variances of the populations (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.949571

The p-value is 0.95 so at the 5% significance level we do not reject the null hypothesis of equal means. There is no sufficient evidence in the data to reject the hypothesis that the two means in the populations are similar. This result confirms what we found by hand.

Note that a similar function exists in the {BSDA} package:⁴

library(BSDA)

z.test(dat1$sample1,
  dat1$sample2,
  alternative = "two.sided",
  mu = 0,
  sigma.x = 1,
  sigma.y = 1,
  conf.level = 0.95
)

## 
##  Two-sample z-Test
## 
## data:  dat1$sample1 and dat1$sample2
## z = -0.063246, p-value = 0.9496
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.27959  1.19959
## sample estimates:
## mean of x mean of y 
##      0.02      0.06

A note on p-value and significance level $\alpha$

For those unfamiliar with the concept of p-value, the p-value is a probability and as any probability it goes from 0 to 1. The p-value is the probability of having observations at least as extreme as the one we measured (via the samples) if the null hypothesis were true. In other words, it is the probability of having a test statistic at least as extreme as the one we computed, given that the null hypothesis is true. In some sense, it gives you an indication on how likely your null hypothesis is. It is also defined as the smallest level of significance for which the data indicate rejection of the null hypothesis.

If the observations are not so extreme—not unlikely to occur if the null hypothesis were true—we do not reject this null hypothesis because it is deemed plausible to be true. And if the observations are considered too extreme—too unlikely to happen under the null hypothesis—we reject the null hypothesis because it is deemed too implausible to be true. Note that it does not mean that we are 100% sure that it is too unlikely, it happens sometimes that the null hypothesis is rejected although it is true (see the significance level $\alpha$ later on).

In our example above, the observations are not really extreme and the difference between the two means is not extreme, so the test statistic is not extreme (since the test statistic is partially based on the difference of the means of the two samples). Having a test statistic which is not extreme is not unlikely and that is the reason why the p-value is quite high. The p-value of 0.95 actually tells us that the probability of having two samples with a difference in means of -0.04 (= 0.02 - 0.06), given that the difference in means in the populations is 0 (the null hypothesis), equals 95%. A probability of 95% is definitely considered as plausible, so we do not reject the null hypothesis of equal means in the populations.

One may then wonder, “What is too extreme for a test statistic?” Most of the time, we consider that a test statistic is too extreme to happen just by chance when the probability of having such an extreme test statistic given that the null hypothesis is true is below 5%. The threshold of 5% ($\alpha = 0.05$) that you very often see in statistic courses or textbooks is the threshold used in many fields. With a p-value under that threshold of 5%, we consider that the observations (and thus the test statistic) is too unlikely to happen just by chance if the null hypothesis were true, so the null hypothesis is rejected. With a p-value above that threshold of 5%, we consider that it is not really implausible to face the observations we have if the null hypothesis were true, and we therefore do not reject the null hypothesis.

Note that I wrote “we do not reject the null hypothesis”, and not “we accept the null hypothesis”. This is because it may be the case that the null hypothesis is in fact false, but we failed to prove it with the samples. Suppose the analogy of a suspect accused of murder and we do not know the truth. On the one hand, if we have collected enough evidence that the suspect committed the murder, he is considered guilty: we reject the null hypothesis that he is innocent. On the other hand, if we have not collected enough evidence against the suspect, he is presumed to be innocent although he may in fact have committed the crime: we failed to reject the null hypothesis of him being innocent. We are never sure that he did not commit the crime even if he is released, we just did not find sufficient evidence against the null hypothesis of the suspect being innocent. This is the reason why we do not reject the null hypothesis instead of accepting it, and why you will often read things like “there is no sufficient evidence in the data to reject the null hypothesis” or “based on the samples we fail to reject the null hypothesis”.

The significance level $\alpha$, derived from the threshold of 5% mentioned earlier, is the probability of rejecting the null hypothesis when it is in fact true. In this sense, it is an error (of 5%) that we accept to deal with, in order to be able to draw conclusions. If we would accept no error (an error of 0%), we would not be able to draw any conclusion about the population(s) since we only have access to a limited portion of the population(s) via the sample(s). As a consequence, we will never be 100% sure when interpreting the result of a hypothesis test unless we have access to the data for the entire population, but then there is no reason to do a hypothesis test anymore since we can simply compare the two populations. We usually allow this error (called Type I error) to be 5%, but in order to be a bit more certain when concluding that we reject the null hypothesis, the alpha level can also be set to 1% (or even to 0.1% in some rare cases).

To sum up what you need to remember about p-value and significance level $\alpha$:

If the p-value is smaller than the predetermined significance level $\alpha$ (usually 5%) so if p-value < 0.05 $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the p-value is greater than or equal to the predetermined significance level $\alpha$ (usually 5%) so if p-value $\ge$ 0.05 $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

This applies to all statistical tests without exception. Of course, the null and alternative hypotheses change depending on the test.

A rule of thumb is that, for most hypothesis tests, the alternative hypothesis is what you want to test and the null hypothesis is the status quo. Take this with extreme caution (!) because, even if it works for all versions of the Student’s t-test it does not apply to ALL statistical tests. For example, when testing for normality, you usually want to test whether your distribution follows a normal distribution. Following this piece of advice, you would write the alternative hypothesis $H_1:$ the distribution follows a normal distribution. Nonetheless, for normality tests such as the Shapiro-Wilk or Kolmogorov-Smirnov test, it is the opposite; the alternative hypothesis is $H_1:$ the distribution does not follow a normal distribution. So for every test, make sure to use the correct hypotheses, otherwise the conclusion and interpretation of your test will be wrong.

Last but not least, note that statistical significance is not equal to scientific significance. To this end, a result may be statistically significant (a p-value < $\alpha$), but of little or no interest from a scientific point of view (because the effect is so small that it is negligible and/or useless for instance).

Scenario 2: Independent samples with 2 equal but unknown variances

dat2 <- data.frame(
  sample1 = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample2 = c(0.8, -0.7, -0.1, 0.4, 0.1, NA)
)
dat2

##   sample1 sample2
## 1    1.78     0.8
## 2    1.50    -0.7
## 3    0.90    -0.1
## 4    0.60     0.4
## 5    0.80     0.1
## 6    1.90      NA

dat_ggplot <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the two boxes do not overlap which illustrates that the two samples are different from each other. From this boxplot, we can expect the test to reject the null hypothesis of equal means in the populations. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” Student’s t-test for two samples. Note that it is assumed that the variances of the two populations are equal so we need to specify it in the function with the argument var.equal = TRUE (the default is FALSE) and the alternative hypothesis is $H_1: \mu_1 - \mu_2 > 0$ so we need to add the argument alternative = "greater" as well:

test <- t.test(dat2$sample1, dat2$sample2,
  var.equal = TRUE, alternative = "greater"
)
test

## 
##  Two Sample t-test
## 
## data:  dat2$sample1 and dat2$sample2
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean of x mean of y 
##  1.246667  0.100000

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used and the two sample means (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.003866756

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis of equal means. This result confirms what we found by hand.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the mean of population 1 is larger than the mean of population 2.

A nice and easy way to report results of a Student’s t-test in R is with the report() function from the {report} package:

# install.packages("remotes")
# remotes::install_github("easystats/report") # You only need to do that once
library("report") # Load the package every time you start R

report(test)

## Effect sizes were labelled following Cohen's (1988) recommendations.
## 
## The Two Sample t-test testing the difference between dat2$sample1 and
## dat2$sample2 (mean of x = 1.25, mean of y = 0.10) suggests that the effect is
## positive, statistically significant, and large (difference = 1.15, 95% CI
## [0.53, Inf], t(9) = 3.41, p = 0.004; Cohen's d = 2.07, 95% CI [0.75, Inf])

As you can see, the function interprets the test (together with the p-value) for you.

Note that the report() function can be used for other analyses. See more tips and tricks in R if you find this one useful.

If your data is formatted in the long format (which is even better), simply use the tilde (~). For instance, imagine the exact same data presented like this:

dat2bis <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)
dat2bis

##    value sample
## 1   1.78      1
## 2   1.50      1
## 3   0.90      1
## 4   0.60      1
## 5   0.80      1
## 6   1.90      1
## 7   0.80      2
## 8  -0.70      2
## 9  -0.10      2
## 10  0.40      2
## 11  0.10      2

Here is how to perform the Student’s t-test in R with data in the long format:

test <- t.test(value ~ sample,
  data = dat2bis,
  var.equal = TRUE,
  alternative = "greater"
)
test

## 
##  Two Sample t-test
## 
## data:  value by sample
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means between group 1 and group 2 is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean in group 1 mean in group 2 
##        1.246667        0.100000

test$p.value

## [1] 0.003866756

The results are exactly the same.

Scenario 3: Independent samples with 2 unequal and unknown variances

dat3 <- data.frame(
  value = c(0.8, 0.7, 0.1, 0.4, 0.1, 1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample = c(rep("1", 5), rep("2", 6))
)
dat3

##    value sample
## 1   0.80      1
## 2   0.70      1
## 3   0.10      1
## 4   0.40      1
## 5   0.10      1
## 6   1.78      2
## 7   1.50      2
## 8   0.90      2
## 9   0.60      2
## 10  0.80      2
## 11  1.90      2

ggplot(dat3) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test as well, and it is simply the t.test() function with the var.equal = FALSE argument. FALSE is the default option for the var.equal argument so you actually do not need to specify it. This version of the test is actually the Welch Student’s test, used when the variances of the populations are unknown and unequal. To test if two population variances are equal, you can use the Levene’s test (leveneTest(dat3$value, dat3$sample) from the {car} package, or simply by comparing the dispersion of the two samples via a dotplot or a boxplot). Note that the alternative hypothesis is $H_1: \mu_1 - \mu_2 < 0$ so we need to add the argument alternative = "less" as well:

test <- t.test(value ~ sample,
  data = dat3,
  var.equal = FALSE,
  alternative = "less"
)
test

## 
##  Welch Two Sample t-test
## 
## data:  value by sample
## t = -3.0841, df = 8.2796, p-value = 0.007206
## alternative hypothesis: true difference in means between group 1 and group 2 is less than 0
## 95 percent confidence interval:
##        -Inf -0.3304098
## sample estimates:
## mean in group 1 mean in group 2 
##        0.420000        1.246667

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.00720603

The p-value is 0.007 so at the 5% significance level we reject the null hypothesis of equal means, meaning that we can conclude that the mean of population 1 is smaller than the mean of population 2. This result confirms what we found by hand.

Scenario 4: Paired samples where the variance of the differences is known

For the fourth scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 ($\sigma^2_D = 1$) and that we would like to test whether the mean difference between the two populations is different than 0.

dat4 <- data.frame(
  before = c(0.9, -0.8, 0.1, -0.3, 0.2),
  after = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat4

##   before after
## 1    0.9   0.8
## 2   -0.8  -0.9
## 3    0.1  -0.1
## 4   -0.3   0.4
## 5    0.2   0.1

dat4$difference <- dat4$after - dat4$before

ggplot(dat4) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

Since there is no function in R to perform a t-test with paired samples where the variance of the differences is known, here is one with arguments accepting:

the differences between the two samples (x),
the variance of the differences in the population (V),
the mean of the differences under the null hypothesis (m0, default is 0),
the significance level (alpha, default is 0.05)
and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_pairedknownvar <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_pairedknownvar(dat4$after - dat4$before,
  V = 1
)
test

## $mean
## [1] 0.04
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 0.08944272
## 
## $p.value
## [1] 0.9287301
## 
## $LCL
## [1] -0.8365225
## 
## $UCL
## [1] 0.9165225
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.9287301

The p-value is 0.929 so at the 5% significance level we do not reject the null hypothesis of the mean of the differences being equal to 0. There is no sufficient evidence in the data to reject the hypothesis that the mean difference between the two populations is equal to 0. This result confirms what we found by hand.

Scenario 5: Paired samples where the variance of the differences is unknown

dat5 <- data.frame(
  before = c(9, 8, 1, 3, 2),
  after = c(16, 11, 15, 12, 9)
)
dat5

##   before after
## 1      9    16
## 2      8    11
## 3      1    15
## 4      3    12
## 5      2     9

dat5$difference <- dat5$after - dat5$before

ggplot(dat5) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test, and it is simply the t.test() function with the paired = TRUE argument. This version of the test is actually the standard version of the Student’s t-test with paired samples. Note that the alternative hypothesis is $H_1: \mu_D > 0$ so we need to add the argument alternative = "greater" as well:

test <- t.test(dat5$after, dat5$before,
  alternative = "greater",
  paired = TRUE
)
test

## 
##  Paired t-test
## 
## data:  dat5$after and dat5$before
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true mean difference is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean difference 
##               8

Note that we wrote after and then before in this order. If you write before and then after, make sure to change the alternative to alternative = "less".

If your data is in the long format, use the tilde ~:

dat5 <- data.frame(
  value = c(9, 8, 1, 3, 2, 16, 11, 15, 12, 9),
  time = c(rep("before", 5), rep("after", 5))
)
dat5

##    value   time
## 1      9 before
## 2      8 before
## 3      1 before
## 4      3 before
## 5      2 before
## 6     16  after
## 7     11  after
## 8     15  after
## 9     12  after
## 10     9  after

test <- t.test(value ~ time,
  data = dat5,
  alternative = "greater",
  paired = TRUE
)
test

## 
##  Paired t-test
## 
## data:  value by time
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true mean difference is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean difference 
##               8

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.005528247

The p-value is 0.006 so at the 5% significance level we reject the null hypothesis of the mean of the differences being equal to 0, meaning that we can conclude that the treatment is effective in increasing the running capabilities (because the mean of the differences is greater than 0). This result confirms what we found by hand.

Combination of plot and statistical test

These two functions combine a boxplot—representing the distribution for each group—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for scenarios 2, 3 and 5. Unfortunately, the package does not allow to run tests for scenarios 1 and 4.

Scenario 2: Independent samples with 2 equal but unknown variances

The ggbetweenstats() function is used for independent samples:

# load package
library(ggstatsplot)
library(ggplot2)

# plot with statistical results
ggbetweenstats(
  data = dat2bis,
  x = sample,
  y = value,
  plot.type = "box", # for boxplot
  type = "parametric", # for student's t-test
  var.equal = TRUE, # equal variances
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

The p-value is displayed after p = in the subtitle of the plot. Based on this plot and the p-value being lower than 5% (p-value = 0.008), we reject the null hypothesis that the two population means are equal.

Note that, the p-value is two times as large as the one obtained with the t.test() function because when we ran t.test() we specified alternative = "greater" (i.e., a one-sided test). In our plot with the ggbetweenstats() function, it is a two-sided test that is performed by default, that is, alternative = "two.sided".

Scenario 3: Independent samples with 2 unequal and unknown variances

We also have independent samples so we use the ggbetweenstats() function again, but this time the two populations variances are not assumed to be equal so we specify the argument var.equal = FALSE:

# plot with statistical results
ggbetweenstats(
  data = dat3,
  x = sample,
  y = value,
  plot.type = "box", # for boxplot
  type = "parametric", # for student's t-test
  var.equal = FALSE, # unequal variances
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

Based on the output, we reject the null hypothesis that the two population means are equal (p-value = 0.01).

Note that the p-value displayed in the subtitle of the plot is also two times larger than with the t.test() function for the same reason than above.

Scenario 5: Paired samples where the variance of the differences is unknown

In this case, the samples are paired so we use the ggwithinstats() function:

ggwithinstats(
  data = dat5,
  x = time,
  y = value,
  type = "parametric", # for student's t-test
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

Based on the output, we reject the null hypothesis that mean of the differences between the two populations is equal to 0 (p-value = 0.01).

Again the p-value in the subtitle of the plot is twice the one obtained with the t.test() function for the same reason than above.

Assumptions

As for many statistical tests, there are some assumptions that need to be met in order to be able to interpret the results. When one or several of them are not met, although it is technically possible to perform these tests, it would be incorrect to interpret the results or trust the conclusions.

Below are the assumptions of the Student’s t-test for two samples, how to test them and which other tests exist if an assumption is not met:

Variable type: A Student’s t-test requires a mix of one quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with exactly 2 levels which will determine the groups to compare).
Independence: The data, collected from a representative and randomly selected portion of the population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if two measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before and (ii) after a treatment), the paired version of the Student’s t-test, called the Student’s t-test for paired samples, should be preferred in order to take into account the dependency between the two groups to be compared.
Normality:
- With small samples (usually $n < 30$), when the two samples are independent, observations in both samples should follow a normal distribution. When using the Student’s t-test for paired samples, it is the difference between the observations of the two samples that should follow a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. If, even after a transformation (e.g., logarithmic transformation, square root, etc.), your data still do not follow a normal distribution, the Wilcoxon test (wilcox.test(variable1 ~ variable2, data = dat in R) can be applied. This non-parametric test, robust to non normal distributions, compares the medians instead of the means in order to compare the two populations.
- With large samples (usually $n \ge 30$), normality of the data is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). It is therefore not required to test the normality assumption when the number of observations in each group/sample is large.
Equality of variances: When the two samples are independent, the variances of the two groups should be equal in the populations (an assumption called homogeneity of the variances, or even sometimes referred as homoscedasticity, as opposed to heteroscedasticity if variances are different across groups). This assumption can be tested graphically (by comparing the dispersion in a boxplot or dotplot for instance), or more formally via the Levene’s test (leveneTest(variable ~ group) from the {car} package) or via a F test (var.test(variable ~ group)). If the hypothesis of equal variances is rejected, another version of the Student’s t-test can be used: the Welch test (t.test(variable ~ group, var.equal = FALSE)). Note that the Welch test does not require homogeneity of the variances, but the distributions should still follow a normal distribution in case of small sample sizes. If your distributions are not normally distributed or the variances are unequal, the Wilcoxon test should be used. This test does not require the assumptions of normality nor homoscedasticity of the variances.
Outliers: An outlier is a value or an observation that is distant from the other observations. There should be no significant outliers in the two groups, or the conclusions of your t-test may be flawed. There are several methods to detect outliers in your data but in order to deal with them, it is your choice to either:
- use the non-parametric version (i.e., the Wilcoxon test)
- transform your data (logarithmic or Box-Cox transformation, among others)
- or remove them (be careful)

Conclusion

This concludes a relatively long article. Thanks for reading.

I hope this article helped you to understand how the different versions of the Student’s t-test for two samples work and how to perform them by hand and in R. If you are interested, here is a Shiny app to perform these tests by hand easily (you just need to enter your data and select the appropriate version of the test thanks to the sidebar menu).

Moreover, I invite you to read:

this article if you would like to know how to compute the Student’s t-test but this time, for one sample,
this article if you would like to compare 2 groups under the non-normality assumption, or
this article if you would like to use an ANOVA to compare 3 or more groups.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Rowntree, Derek. 2000. Statistics Without Tears.

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Remind that inferential statistics, as opposed to descriptive statistics, is a branch of statistics defined as the science of drawing conclusions about a population from observations made on a representative sample of that population. See the difference between population and sample.↩︎
For the rest of the present article, when we write Student’s t-test, we refer to the case of 2 samples. See one sample t-test if you want to compare only one sample.↩︎
It is a least the case for parametric hypothesis tests. A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. In the case of the Student’s t-test for two samples, it is based on the Student’s t distribution with a single parameter, the degrees of freedom ($df = n_1 + n_2 - 2$ where $n_1$ and $n_2$ are the two sample sizes), or the normal distribution.↩︎
Thanks gmacar for pointing it out to me.↩︎

Fisher's exact test in R: independence test for a small sample

Tue, 28 Jan 2020 00:00:00 +0000

Introduction

After presenting the Chi-square test of independence by hand and in R, this article focuses on the Fisher’s exact test.

Independence tests are used to determine if there is a significant relationship between two categorical variables. There exists two different types of independence test:

the Chi-square test (the most common)
the Fisher’s exact test

On the one hand, the Chi-square test is used when the sample is large enough (in this case the $p$-value is an approximation that becomes exact when the sample becomes infinite, which is the case for many statistical tests). On the other hand, the Fisher’s exact test is used when the sample is small (and in this case the $p$-value is exact and is not an approximation).

The literature indicates that the usual rule for deciding whether the $\chi^2$ approximation is good enough is that the Chi-square test is not appropriate when the expected values in one of the cells of the contingency table is less than 5, and in this case the Fisher’s exact test is preferred (McCrum-Gardner 2008; Bower 2003).

Hypotheses

The hypotheses of the Fisher’s exact test are the same than for the Chi-square test, that is:

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

Example

Data

For our example, we want to determine whether there is a statistically significant association between smoking and being a professional athlete. Smoking can only be “yes” or “no” and being a professional athlete can only be “yes” or “no”. The two variables of interest are qualitative variables and we collected data on 14 persons.¹

Observed frequencies

Our data are summarized in the contingency table below reporting the number of people in each subgroup:

dat <- data.frame(
  "smoke_no" = c(7, 0),
  "smoke_yes" = c(2, 5),
  row.names = c("Athlete", "Non-athlete"),
  stringsAsFactors = FALSE
)
colnames(dat) <- c("Non-smoker", "Smoker")

dat

##             Non-smoker Smoker
## Athlete              7      2
## Non-athlete          0      5

It is also a good practice to draw a mosaic plot to visually represent the data:

mosaicplot(dat,
  main = "Mosaic plot",
  color = TRUE
)

We can already see from the plot that the proportion of smokers in the sample is higher among non-athletes than athlete. The plot is however not sufficient to conclude that there is such a significant association in the population.

Like many statistical tests, this can be done via a hypothesis test. But before seeing how to do it in R, let’s see the concept of expected frequencies.

Expected frequencies

Remember that the Fisher’s exact test is used when there is at least one cell in the contingency table of the expected frequencies below 5. To retrieve the expected frequencies, use the chisq.test() function together with $expected:

chisq.test(dat)$expected

## Warning in chisq.test(dat): Chi-squared approximation may be incorrect

##             Non-smoker Smoker
## Athlete            4.5    4.5
## Non-athlete        2.5    2.5

The contingency table above confirms that we should use the Fisher’s exact test instead of the Chi-square test because there is at least one cell below 5.

Tip: although it is a good practice to check the expected frequencies before deciding between the Chi-square and the Fisher test, it is not a big issue if you forget. As you can see above, when doing the Chi-square test in R (with chisq.test()), a warning such as “Chi-squared approximation may be incorrect” will appear. This warning means that the smallest expected frequencies is lower than 5. Therefore, do not worry if you forgot to check the expected frequencies before applying the appropriate test to your data, R will warn you that you should use the Fisher’s exact test instead of the Chi-square test if that is the case.

(Remember that, as for the Chi-square test of independence, the observations must be independent in order for the Fisher’s exact test to be valid. See more details about the independence assumption in this section.)

Fisher’s exact test in R

To perform the Fisher’s exact test in R, use the fisher.test() function as you would do for the Chi-square test:

test <- fisher.test(dat)
test

## 
##  Fisher's Exact Test for Count Data
## 
## data:  dat
## p-value = 0.02098
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  1.449481      Inf
## sample estimates:
## odds ratio 
##        Inf

The most important in the output is the $p$-value. You can also retrieve the $p$-value with:

test$p.value

## [1] 0.02097902

Note that if your data is not already presented as a contingency table, you can simply use the following code:

fisher.test(table(dat$variable1, dat$variable2))

where dat is the name of your dataset, variable1 and variable2 correspond to the names of the two variables of interest.

Conclusion and interpretation

From the output and from test$p.value we see that the $p$-value is less than the significance level of 5%. Like any other statistical test, if the $p$-value is less than the significance level, we can reject the null hypothesis. If you are not familiar with $p$-values, I invite you to read this section.

$\Rightarrow$ In our context, rejecting the null hypothesis for the Fisher’s exact test of independence means that there is a significant relationship between the two categorical variables (smoking habits and being an athlete or not). Therefore, knowing the value of one variable helps to predict the value of the other variable.

Combination of plot and statistical test

It is possible print the results of the Fisher’s exact test directly on a barplot thanks to the ggbarstats() function from the {ggstatsplot} package (the function has been slightly edited to match our needs).

It is easier to work with the package when our data is not already in the form of a contingency table so we transform it to a data frame before plotting the results:

# create dataframe from contingency table
x <- c()
for (row in rownames(dat)) {
  for (col in colnames(dat)) {
    x <- rbind(x, matrix(rep(c(row, col), dat[row, col]), ncol = 2, byrow = TRUE))
  }
}
df <- as.data.frame(x)
colnames(df) <- c("Sport_habits", "Smoking_habits")
df

##    Sport_habits Smoking_habits
## 1       Athlete     Non-smoker
## 2       Athlete     Non-smoker
## 3       Athlete     Non-smoker
## 4       Athlete     Non-smoker
## 5       Athlete     Non-smoker
## 6       Athlete     Non-smoker
## 7       Athlete     Non-smoker
## 8       Athlete         Smoker
## 9       Athlete         Smoker
## 10  Non-athlete         Smoker
## 11  Non-athlete         Smoker
## 12  Non-athlete         Smoker
## 13  Non-athlete         Smoker
## 14  Non-athlete         Smoker

# Fisher's exact test with raw data
test <- fisher.test(table(df))

# combine plot and statistical test with ggbarstats
library(ggstatsplot)
ggbarstats(
  df, Smoking_habits, Sport_habits,
  results.subtitle = FALSE,
  subtitle = paste0(
    "Fisher's exact test", ", p-value = ",
    ifelse(test$p.value < 0.001, "< 0.001", round(test$p.value, 3))
  )
)

From the plot, it is clear that the proportion of smokers among non-athletes is higher than among athletes, suggesting that there is a relationship between the two variables.

This is confirmed thanks to the $p$-value displayed in the subtitle of the plot. As previously, we reject the null hypothesis and we conclude that the variables smoking habits and being an athlete or not are dependent ($p$-value = 0.021).

Conclusion

Thanks for reading.

I hope the article helped you to perform the Fisher’s exact test of independence in R and interpret its results. Learn more about the Chi-square test of independence by hand or in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Bower, Keith M. 2003. “When to Use Fisher’s Exact Test.” In American Society for Quality, Six Sigma Forum Magazine, 2:35–37. 4.

McCrum-Gardner, Evie. 2008. “Which Is the Correct Statistical Test to Use?” British Journal of Oral and Maxillofacial Surgery 46 (1): 38–41.

The data are the same than for the article covering the Chi-square test by hand, except that some observations have been removed to decrease the sample size.↩︎

Chi-square test of independence by hand

Mon, 27 Jan 2020 00:00:00 +0000

Introduction

Chi-square tests of independence test whether two qualitative variables are independent, that is, whether there exists a relationship between two categorical variables. In other words, this test is used to determine whether the values of one of the 2 qualitative variables depend on the values of the other qualitative variable.

If the test shows no association between the two variables (i.e., the variables are independent), it means that knowing the value of one variable gives no information about the value of the other variable. On the contrary, if the test shows a relationship between the variables (i.e., the variables are dependent), it means that knowing the value of one variable provides information about the value of the other variable.

This article focuses on how to perform a Chi-square test of independence by hand and how to interpret the results with a concrete example. To learn how to do this test in R, read the article “Chi-square test of independence in R”.

Hypotheses

The Chi-square test of independence is a hypothesis test so it has a null ($H_0$) and an alternative hypothesis ($H_1$):

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

How the test works?

The Chi-square test of independence works by comparing the observed frequencies (so the frequencies observed in your sample) to the expected frequencies if there was no relationship between the two categorical variables (so the expected frequencies if the null hypothesis was true).

If the difference between the observed frequencies and the expected frequencies is small, we cannot reject the null hypothesis of independence and thus we cannot reject the fact that the two variables are not related. On the other hand, if the difference between the observed frequencies and the expected frequencies is large, we can reject the null hypothesis of independence and thus we can conclude that the two variables are related.

The threshold between a small and large difference is a value that comes from the Chi-square distribution (hence the name of the test). This value, referred as the critical value, depends on the significance level $\alpha$ (usually set equal to 5%) and on the degrees of freedom. This critical value can be found in the statistical table of the Chi-square distribution. More on this critical value and the degrees of freedom later in the article.

Example

Note that we chose binary variables (binary variables = qualitative variables with two levels) for the sake of easiness, but the Chi-square test of independence can also be performed on qualitative variables with more than two levels. For instance, if the variable smoking had three levels: (i) non-smokers, (ii) moderate smokers and (iii) heavy smokers, the steps and the interpretation of the results of the test are similar than with two levels.

Observed frequencies

Our data are summarized in the contingency table below reporting the number of people in each subgroup, totals by row, by column and the grand total:

	Non-smoker	Smoker	Total
Athlete	14	4	18
Non-athlete	0	10	10
Total	14	14	28

Expected frequencies

Remember that for the Chi-square test of independence we need to determine whether the observed counts are significantly different from the counts that we would expect if there was no association between the two variables. We have the observed counts (see the table above), so we now need to compute the expected counts in the case the variables were independent. These expected frequencies are computed for each subgroup one by one with the following formula:

\[\text{exp. frequencies} = \frac{\text{total # of obs. for the row} \cdot \text{total # of obs. for the column}}{\text{total number of observations}}\]

where obs. correspond to observations. Given our table of observed frequencies above, below is the table of the expected frequencies computed for each subgroup:

	Non-smoker	Smoker	Total
Athlete	(18 * 14) / 28 = 9	(18 * 14) / 28 = 9	18
Non-athlete	(10 * 14) / 28 = 5	(10 * 14) / 28 = 5	10
Total	14	14	28

Note that the Chi-square test of independence should only be done when the expected frequencies in all groups are equal to or greater than 5. This assumption is met for our example as the minimum number of expected frequencies is 5. If the condition is not met, the Fisher’s exact test is preferred.

Talking about assumptions, the Chi-square test of independence requires that the observations are independent. This is usually not tested formally, but rather verified based on the design of the experiment and on the good control of experimental conditions. If you are not sure, ask yourself if one observation is related to another (if one observation has an impact on another). If not, it is most likely that you have independent observations.

If you have dependent observations (paired samples), the McNemar’s or Cochran’s Q tests should be used instead. The McNemar’s test is used when we want to know if there is a significant change in two paired samples (typically in a study with a measure before and after on the same subject) when the variables have only two categories. The Cochran’s Q tests is an extension of the McNemar’s test when we have more than two related measures.

Test statistic

We have the observed and expected frequencies. We now need to compare these frequencies to determine if they differ significantly. The difference between the observed and expected frequencies, referred as the test statistic (or t-stat) and denoted $\chi^2$, is computed as follows:

\[\chi^2 = \sum_{i, j} \frac{\big(O_{ij} - E_{ij}\big)^2}{E_{ij}}\]

where $O$ represents the observed frequencies and $E$ the expected frequencies. We use the square of the differences between the observed and expected frequencies to make sure that negative differences are not compensated by positive differences. The formula looks more complex than what it really is, so let’s illustrate it with our example. We first compute the difference in each subgroup one by one according to the formula:

in the subgroup of athlete and non-smoker: $\frac{(14 - 9)^2}{9} = 2.78$
in the subgroup of non-athlete and non-smoker: $\frac{(0 - 5)^2}{5} = 5$
in the subgroup of athlete and smoker: $\frac{(4 - 9)^2}{9} = 2.78$
in the subgroup of non-athlete and smoker: $\frac{(10 - 5)^2}{5} = 5$

and then we sum them all to obtain the test statistic:

\[\chi^2 = 2.78 + 5 + 2.78 + 5 = 15.56\]

Critical value

The test statistic alone is not enough to conclude for independence or dependence between the two variables. As previously mentioned, this test statistic (which in some sense is the difference between the observed and expected frequencies) must be compared to a critical value to determine whether the difference is large or small. One cannot tell that a test statistic is large or small without putting it in perspective with the critical value.

If the test statistic is above the critical value, it means that the probability of observing such a difference between the observed and expected frequencies is unlikely. On the other hand, if the test statistic is below the critical value, it means that the probability of observing such a difference is likely. If it is likely to observe this difference, we cannot reject the hypothesis that the two variables are independent, otherwise we can conclude that there exists a relationship between the variables.

The critical value can be found in the statistical table of the Chi-square distribution and depends on the significance level, denoted $\alpha$, and the degrees of freedom, denoted $df$. The significance level is usually set equal to 5%. The degrees of freedom for a Chi-square test of independence is found as follow:

\[df = (\text{number of rows} - 1) \cdot (\text{number of columns} - 1)\]

In our example, the degrees of freedom is thus $df = (2 - 1) \cdot (2 - 1) = 1$ since there are two rows and two columns in the contingency table (totals do not count as a row or column).

We now have all the necessary information to find the critical value in the Chi-square table ($\alpha = 0.05$ and $df = 1$). To find the critical value we need to look at the row $df = 1$ and the column $\chi^2_{0.050}$ (since $\alpha = 0.05$) in the picture below. The critical value is $3.84146$.¹

Chi-square table - Critical value for alpha = 5% and df = 1

Conclusion and interpretation

Now that we have the test statistic and the critical value, we can compare them to check whether the null hypothesis of independence of the variables is rejected or not. In our example,

\[\text{test statistic} = 15.56 > \text{critical value} = 3.84146\]

Like for many statistical tests, when the test statistic is larger than the critical value, we can reject the null hypothesis at the specified significance level.

In our case, we can therefore reject the null hypothesis of independence between the two categorical variables at the 5% significance level.

$\Rightarrow$ This means that there is a significant relationship between the smoking habit and being an athlete or not. Knowing the value of one variable helps to predict the value of the other variable.

Thanks for reading.

I hope the article helped you to perform the Chi-square test of independence by hand and interpret its results. If you would like to learn how to do this test in R, read the article “Chi-square test of independence in R”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

For readers that prefer to check the $p$-value in order to reject or not the null hypothesis, I also created a Shiny app to help you compute the $p$-value given a test statistic.↩︎

Chi-square test of independence in R

Mon, 27 Jan 2020 00:00:00 +0000

Introduction

This article explains how to perform the Chi-square test of independence in R and how to interpret its results. To learn more about how the test works and how to do it by hand, I invite you to read the article “Chi-square test of independence by hand”.

To briefly recap what have been said in that article, the Chi-square test of independence tests whether there is a relationship between two categorical variables. The null and alternative hypotheses are:

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

Data

For our example, let’s reuse the dataset introduced in the article “Descriptive statistics in R”. This dataset is the well-known iris dataset slightly enhanced. Since there is only one categorical variable and the Chi-square test of independence requires two categorical variables, we add the variable size which corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

dat <- iris

dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

We now create a contingency table of the two variables Species and size with the table() function:

table(dat$Species, dat$size)

##             
##              big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

The contingency table gives the observed number of cases in each subgroup. For instance, there is only one big setosa flower, while there are 49 small setosa flowers in the dataset.

It is also a good practice to draw a barplot to visually represent the data:

library(ggplot2)

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar()

If you prefer to visualize it in terms of proportions (so that bars all have a height of 1, or 100%):

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar(position = "fill")

This second barplot is particularly useful if there are a different number of observations in each level of the variable drawn on the $x$-axis because it allows to compare the two variables on the same ground.

If you prefer to have the bars next to each other:

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar(position = "dodge")

See the article “Graphics in R with ggplot2” to learn how to create this kind of barplot in {ggplot2}.

Chi-square test of independence in R

For this example, we are going to test in R if there is a relationship between the variables Species and size. For this, the chisq.test() function is used:

test <- chisq.test(table(dat$Species, dat$size))
test

## 
## 	Pearson's Chi-squared test
## 
## data:  table(dat$Species, dat$size)
## X-squared = 86.035, df = 2, p-value < 2.2e-16

Everything you need appears in this output:

the title of the test,
which variables have been used,
the test statistic,
the degrees of freedom and
the $p$-value of the test.

You can also retrieve the $\chi^2$ test statistic and the $p$-value with:

test$statistic # test statistic

## X-squared 
##  86.03451

test$p.value # p-value

## [1] 2.078944e-19

If you need to find the expected frequencies, use test$expected.

If a warning such as “Chi-squared approximation may be incorrect” appears, it means that the smallest expected frequencies is lower than 5. To avoid this issue, you can either:

gather some levels (especially those with a small number of observations) to increase the number of observations in the subgroups, or
use the Fisher’s exact test

The Fisher’s exact test does not require the assumption of a minimum of 5 expected counts in the contingency table. It can be applied in R thanks to the function fisher.test(). This test is similar to the Chi-square test in terms of hypothesis and interpretation of the results. Learn more about this test in this article dedicated to this type of test.

For your information, there are three other methods to perform the Chi-square test of independence in R:

with the summary() function
with the assocstats() function from the {vcd} package
with the ctable() function from the {summarytools} package

# second method:
summary(table(dat$Species, dat$size))

## Number of cases in table: 150 
## Number of factors: 2 
## Test for independence of all factors:
## 	Chisq = 86.03, df = 2, p-value = 2.079e-19

# third method:
library(vcd)

assocstats(table(dat$Species, dat$size))

##                      X^2 df P(> X^2)
## Likelihood Ratio 107.308  2        0
## Pearson           86.035  2        0
## 
## Phi-Coefficient   : NA 
## Contingency Coeff.: 0.604 
## Cramer's V        : 0.757

library(summarytools)
library(dplyr)

# fourth method:
dat %$%
  ctable(Species, size,
    prop = "r", chisq = TRUE, headings = FALSE
  ) %>%
  print(
    method = "render",
    style = "rmarkdown",
    footnote = NA
  )

As you can see all four methods give the same results.

If you do not have the same p-values with your data across the different methods, make sure to add the correct = FALSE argument in the chisq.test() function to prevent from applying the Yate’s continuity correction, which is applied by default in this method.¹

Conclusion and interpretation

$\Rightarrow$ In our context, rejecting the null hypothesis for the Chi-square test of independence means that there is a significant relationship between the species and the size. Therefore, knowing the value of one variable helps to predict the value of the other variable.

Combination of plot and statistical test

I recently discovered the mosaic() function from the {vcd} package. This function has the advantage that it combines a mosaic plot (to visualize a contingency table) and the result of the Chi-square test of independence:

library(vcd)

mosaic(~ Species + size,
  direction = c("v", "h"),
  data = dat,
  shade = TRUE
)

As you can see, the mosaic plot is similar to the barplot presented above, but the p-value of the Chi-square test is also displayed at the bottom right.

Moreover, this mosaic plot with colored cases shows where the observed frequencies deviates from the expected frequencies if the variables were independent. The red cases means that the observed frequencies are smaller than the expected frequencies, whereas the blue cases means that the observed frequencies are larger than the expected frequencies.

An alternative is the ggbarstats() function from the {ggstatsplot} package:

# load packages
library(ggstatsplot)
library(ggplot2)

# plot
ggbarstats(
  data = dat,
  x = size,
  y = Species
) +
  labs(caption = NULL) # remove caption

From the plot, it seems that big flowers are more likely to belong to the virginica species, while small flowers tend to belong to the setosa species. Species and size are thus expected to be dependent.

This is confirmed thanks to the statistical results displayed in the subtitle of the plot. There are several results, but we can in this case focus on the $p$-value which is displayed after p = at the top (in the subtitle of the plot).

As with the previous tests, we reject the null hypothesis and we conclude that species and size are dependent ($p$-value < 0.001).

Thanks for reading. I hope the article helped you to perform the Chi-square test of independence in R and interpret its results. If you would like to learn how to do this test by hand and how it works, read the article “Chi-square test of independence by hand”. If you want to go further and estimate the strength of the relationship between two categorical variables, see the binary logistic regression.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

Thanks Herivelto for pointing it out.↩︎

A Shiny app for inferential statistics by hand

Wed, 15 Jan 2020 00:00:00 +0000

Statistics is divided into four main branches:

Descriptive statistics
Inferential statistics
Predictive analysis
Exploratory analysis

Descriptive statistics provide a summary of the data; it helps explaining the data in a concise way without losing too much information. Data can be summarized numerically or graphically. See descriptive statistics by hand or in R to learn more about this branch of statistics.

The branch of predictive analysis aims at predicting a dependent variable based on one or several independent variables. Depending on the type of data to be predicted, it often encompasses methods such as linear regression or classification.

Exploratory analyses focus on using graphical approaches to delve into the data and identify the relationships that exist between the different variables in the dataset. They are therefore more akin to data visualization.

Inferential statistics uses a random sample of data taken from a population to make inferences, i.e., to draw conclusions about the population (see the difference between population and sample). In other words, information from the sample is used to make generalizations about the parameter of interest in the population.

The two major tools in inferential statistics are:

confidence intervals, and
hypothesis tests.

Here is a Shiny app which helps you to use these two tools:

Statistics-201

This Shiny app focuses on confidence intervals and hypothesis tests for:

1 and 2 means (with unpaired and paired samples)
1 and 2 proportions
1 and 2 variances

How to use this app?

Follow these steps:

Open the app via this link
Choose the parameter(s) you want to do inference for (i.e., mean(s), proportion(s) or variance(s))
Write your data in Sample. Observations are separated by a comma and the decimal is a point
Set the null and alternative hypothesis
Select the significance level (most of the time $\alpha = 0.05$)

In the results panel (on the right side or below depending on the size of your screen), you will see:

a recap of your sample together with some appropriate descriptive statistics
the confidence interval
the hypothesis test
the interpretation
and an illustration of the hypothesis test

All formulas, steps and computations to arrive at the final results are also provided.

Code

See the last version on GitHub.

Note that the link may not work if the app has hit the monthly usage limit. Try again later if that is the case.

Conclusion

Thanks for reading.

I hope you will find this app useful to do inferential statistics and in particular confidence interval and hypothesis testing by hand.

If you need to learn more about the structure of a hypothesis test by hand, see this article. See also this flowchart to have an overview of the most common statistical tests.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.