Statistics on Stats and R

McNemar's test in R

Wed, 19 Aug 2026 00:00:00 +0000

Introduction

In a previous article, we showed how to perform the Chi-square test of independence in R in order to test whether two qualitative variables are related. As mentioned in that article (and in the one showing how to do the Chi-square test of independence by hand), this test requires that observations are independent. When observations are dependent, that is, when the two measurements are collected on the same subjects (paired samples), the McNemar’s or Cochran’s Q tests should be used instead.

This article is dedicated to the first one: the McNemar’s test. It is used to compare two related (paired) proportions measured on a qualitative variable with only two possible levels. In practice, it is mostly used when the same subjects are measured twice (typically before and after an intervention), or when two raters or two conditions are applied to the same subjects.

In a way, the McNemar’s test is to two paired proportions what the paired Student’s t-test is to two paired means: in both cases we take advantage of the fact that the two measurements belong to the same individuals, the difference being that here the variable of interest is binary instead of quantitative.

Note also that the McNemar’s test is limited to exactly two related measurements. If you have more than two (for example, the same question asked at three different time points), the appropriate extension is the Cochran’s Q test, of which the McNemar’s test is the special case for two measurements. If you are unsure about which test is appropriate for your own data, see this overview of the most common statistical tests.

In the remaining of the article, we present the data used for the illustration, the aim, hypotheses and assumptions of the test, and finally how to perform it in R and how to interpret its results.

Data

A dataset with a paired binary structure is not so easy to find among the datasets shipped with R, so we simulate our own data for this article.

Suppose that we ask 200 randomly selected citizens whether they are in favor of a new policy in their city (answer “Yes” or “No”), that we then have them watch a public debate on this policy, and that we ask them exactly the same question again right after the debate:

# number of respondents
n <- 200

# opinion before the debate
before <- sample(c("Yes", "No"),
  size = n,
  replace = TRUE,
  prob = c(0.4, 0.6)
)

# opinion after the debate (respondents who were in favor
# tend to keep their opinion, while those who were against
# are more likely to change their mind)
after <- ifelse(before == "Yes",
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.9, 0.1)),
  sample(c("Yes", "No"), size = n, replace = TRUE, prob = c(0.4, 0.6))
)

# dataset
dat <- data.frame(
  respondent = 1:n,
  before = factor(before, levels = c("Yes", "No")),
  after = factor(after, levels = c("Yes", "No"))
)

head(dat)

##   respondent before after
## 1          1    Yes   Yes
## 2          2    Yes   Yes
## 3          3     No   Yes
## 4          4    Yes   Yes
## 5          5    Yes   Yes
## 6          6     No    No

(Note that a seed has been set in the background with set.seed(42), so the simulated data and all results below are reproducible.)

Each row corresponds to one respondent and contains two measurements of the same binary variable: the opinion before and the opinion after the debate. The two samples are thus paired, since the two answers on a given row belong to the same person.

As always, it is a good practice to start with some descriptive statistics. Here, the proportion of respondents in favor of the policy at each of the two time points:

# install.packages("dplyr")
library(dplyr)

dat %>%
  summarise(
    prop_before = mean(before == "Yes"),
    prop_after = mean(after == "Yes")
  )

##   prop_before prop_after
## 1        0.46       0.61

In our sample, the proportion of respondents in favor of the policy went from 46% before the debate to 61% after the debate.

These two proportions are computed on the same people, so comparing them as if they came from two independent groups would ignore the pairing. What matters for the McNemar’s test is the way each respondent moved (or did not move) from one answer to the other, and this information is contained in the 2 $\times$ 2 contingency table of the paired answers:

tab <- table(dat$before, dat$after,
  dnn = c("Before", "After")
)

tab

##       After
## Before Yes No
##    Yes  81 11
##    No   41 67

This table must be read pair by pair, and not cell by cell as we usually do:

the two cells on the diagonal are the concordant pairs: 81 respondents answered “Yes” twice and 67 answered “No” twice, so these 148 respondents did not change their mind,
the two cells outside the diagonal are the discordant pairs: 11 respondents were in favor before the debate but against after, while 41 were against before but in favor after.

Only the discordant pairs carry information about a change of opinion (a respondent who gave twice the same answer tells us nothing about the effect of the debate), and this is precisely what the McNemar’s test is built on.

The same information can be visualized with a simple barplot of the paired counts:

# install.packages("ggplot2")
library(ggplot2)

ggplot(dat) +
  aes(x = before, fill = after) +
  geom_bar(position = "dodge") +
  labs(
    x = "Opinion before the debate",
    y = "Number of respondents",
    fill = "Opinion after the debate"
  )

From the table and the plot, we see that the changes of opinion do not balance out: many more respondents switched from “No” to “Yes” than the opposite. The question is now whether this imbalance is large enough to be declared significant, or whether it could reasonably be explained by chance alone (that is, by sampling fluctuations).

McNemar’s test

Aim and hypotheses

The McNemar’s test is used to compare two related proportions, so it allows to determine whether the proportion of subjects belonging to a given category changed between two dependent measurements.

The null and alternative hypotheses of the McNemar’s test are:

$H_0$: the two related proportions are equal (marginal homogeneity, that is, there is no systematic change between the two measurements)
$H_1$: the two related proportions are different (there is a significant change between the two measurements)

Since concordant pairs bring no information about a change, the test is based only on the two discordant cells. Denoting by $b$ the number of subjects who answered “Yes” then “No”, and by $c$ the number of subjects who answered “No” then “Yes”, the hypotheses can equivalently be written as:

$H_0: p_b = p_c$
$H_1: p_b \ne p_c$

where $p_b$ and $p_c$ are the probabilities of the two possible types of change. Under the null hypothesis, a change in one direction is as likely as a change in the other direction, so the test statistic

\[\chi^2 = \frac{(b - c)^2}{b + c}\]

follows a Chi-square distribution with 1 degree of freedom. By default, R applies a continuity correction (see more on this below), which replaces the numerator by $(|b - c| - 1)^2$.

In the context of our example, the McNemar’s test helps us to answer the following question: “Did the public debate significantly change the proportion of citizens in favor of the new policy?”.

Rejecting $H_0$ would mean that the proportion of citizens in favor of the policy is significantly different before and after the debate, so that the changes of opinion observed in our sample are unlikely to be due to chance only. On the contrary, not rejecting $H_0$ would mean that we do not have enough evidence to conclude that opinions changed: the switches observed in the two directions would then be compatible with random fluctuations.

Note that, as for many tests, the McNemar’s test does not indicate the direction of the change. The direction must be read from the contingency table or from the marginal proportions computed in the previous section.

Assumptions

For the results of the McNemar’s test to be valid, the following assumptions must be met:

Paired measurements on a binary variable. The two measurements must be collected on the same subjects, or on matched pairs (twins, or patients matched on age and sex for instance), and the variable of interest must be qualitative with exactly two levels (“Yes”/“No”, success/failure, present/absent, etc.). If the two samples are independent instead of paired, use the Chi-square test of independence.
Data organized in a 2 $\times$ 2 contingency table of the paired outcomes. Each subject contributes to one and only one cell of the table, so the sum of the four cells equals the number of subjects (200 in our case), and not twice this number.
Pairs are independent of each other. Within a pair, the two measurements are of course dependent, and this is precisely the reason why we use this test. Between pairs, however, independence is required: one subject’s answers must not influence another subject’s answers. As for many statistical tests, this assumption is usually verified based on the design of the experiment rather than via a formal test. A random and representative sample of the population of interest is generally sufficient. In our example, respondents have been selected at random and answered the question individually, so we consider this assumption as met.
Enough discordant pairs. The $p$-value returned by mcnemar.test() is based on a Chi-square approximation, which is reliable only if the number of discordant pairs is large enough. A common rule of thumb is that $b + c$ should be at least 25. In our sample, $b + c$ = 52, so the approximation can be used safely.

When the number of discordant pairs is small, it is preferable to use the exact version of the test, which is based on a binomial distribution instead of the Chi-square approximation. It boils down to testing whether, among the discordant pairs, changes in one direction are as frequent as changes in the other direction, so it can be performed in base R with the binom.test() function:

# exact version of the McNemar's test
binom.test(tab[1, 2], tab[1, 2] + tab[2, 1], p = 0.5)

## 
## 	Exact binomial test
## 
## data:  tab[1, 2] and tab[1, 2] + tab[2, 1]
## number of successes = 11, number of trials = 52, p-value = 3.589e-05
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.1106115 0.3470376
## sample estimates:
## probability of success 
##              0.2115385

Note that the {exact2x2} package also provides a dedicated mcnemar.exact() function, which returns the same $p$-value together with a confidence interval for the odds ratio.

In R

The McNemar’s test can be performed in R with the mcnemar.test() function, applied on the contingency table of the paired outcomes:

mcnemar.test(tab)

## 
## 	McNemar's Chi-squared test with continuity correction
## 
## data:  tab
## McNemar's chi-squared = 16.173, df = 1, p-value = 5.781e-05

The test can also be applied directly on the two variables, without building the contingency table first (results are of course identical):

mcnemar.test(dat$before, dat$after)

## 
## 	McNemar's Chi-squared test with continuity correction
## 
## data:  dat$before and dat$after
## McNemar's chi-squared = 16.173, df = 1, p-value = 5.781e-05

The output shows:

the title of the test, together with the mention that a continuity correction has been applied,
the data which have been used,
the test statistic (McNemar's chi-squared),
the degrees of freedom (always equal to 1 for a 2 $\times$ 2 table) and
the $p$-value.

As mentioned above, R applies a continuity correction by default. This correction makes the test slightly more conservative (that is, it gives a larger $p$-value), and it can be removed thanks to the correct = FALSE argument:

mcnemar.test(tab, correct = FALSE)

## 
## 	McNemar's Chi-squared test
## 
## data:  tab
## McNemar's chi-squared = 17.308, df = 1, p-value = 3.179e-05

With 52 discordant pairs, both versions lead to the same conclusion. The correction really matters only when the number of discordant pairs is small, and in that case the exact version presented in the previous section is a better option anyway.

It is the $p$-value which is of interest to conclude the test. If you are not familiar with $p$-values, I invite you to read this section.

Interpretations

Based on the McNemar’s test, we reject the null hypothesis and we conclude that the proportion of citizens in favor of the new policy is significantly different before and after the debate ($p$-value < 0.001).

$\Rightarrow$ In our context, rejecting the null hypothesis means that the debate is associated with a significant change of opinion. Looking at the direction of this change, the proportion of citizens in favor of the policy increased from 46% before the debate to 61% after the debate.

(For the sake of illustration, if the $p$-value had been larger than the significance level $\alpha = 0.05$: we could not have rejected the null hypothesis, so we could not have concluded that the proportion of citizens in favor of the policy was different before and after the debate.)

Contrary to the tests comparing three groups or more, no post-hoc test is required after a significant McNemar’s test: only two related measurements are compared, so a significant result already tells us which two proportions differ. Post-hoc comparisons become relevant again with more than two related measurements, in which case you should turn to the Cochran’s Q test.

Summary

In this article, we reviewed the aim and the hypotheses of the McNemar’s test, which is used to compare two related proportions measured on the same subjects, together with its underlying assumptions (paired measurements on a binary variable, independence between pairs and a sufficient number of discordant pairs). We then showed how to perform it in R with the mcnemar.test() function, applied either on the 2 $\times$ 2 contingency table of the paired answers or directly on the two variables, and how to interpret its results by comparing the $p$-value with the significance level $\alpha$. Remember that it is the special case of the Cochran’s Q test for exactly two related measurements, and that with independent samples the Chi-square test of independence should be preferred.

Thanks for reading.

I hope this article helped you to understand the McNemar’s test and how to perform it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns

Tue, 16 Dec 2025 00:00:00 +0000

I am pleased to announce the publication of our paper “AssociationExplorer: A user-friendly Shiny application for exploring associations and visual patterns” in the journal SoftwareX, together with the official release of the AssociationExplorer2 R package on CRAN.

Both the paper and the software are part of an open-science effort aimed at making exploratory data analysis more accessible to non-technical users.

Why AssociationExplorer?

Exploring multivariate datasets is now central in social sciences, data journalism, and education. However, identifying and interpreting associations between variables often requires programming skills and a solid background in statistics, which can represent a substantial barrier for many users.

AssociationExplorer was designed to lower this barrier by providing an interactive, visual, and statistically grounded tool for exploring associations between quantitative and qualitative variables, without requiring users to write any code.

The application is primarily intended for:

journalists and data journalism practitioners,
teachers and students,
researchers in the exploratory phase of an analysis,
engaged citizens interested in understanding public or survey data.

What does the app do?

AssociationExplorer follows a simple and guided workflow:

Import data (CSV or Excel files)
Interactively select variables of interest
Automatically compute association measures adapted to variable types:
- Pearson’s $r$ correlation for numeric–numeric pairs,
- Cramer’s V for categorical–categorical pairs,
- the correlation ratio $\eta$ for mixed numeric–categorical pairs
Filter associations using user-defined thresholds
Visualize results through:
- an interactive correlation network,
- contextual bivariate visualizations (scatter plots, mean plots, and colored contingency tables)

This workflow is designed to support transparent, reactive, and interpretable exploratory data analysis.

A paper published in SoftwareX

The SoftwareX paper provides a detailed description of:

the motivation and intended audience of the tool,
the software architecture,
the methodological choices underlying the association measures,
an illustrative case study based on the European Social Survey,
and perspectives for future development.

Link to the paper: https://doi.org/10.1016/j.softx.2025.102483

In line with the journal’s standards, the code, documentation, and example data are fully open and reproducible.

The R package is also on CRAN

Alongside the paper, an R package is also available on CRAN, making installation and use straightforward:

install.packages("AssociationExplorer2")
library(AssociationExplorer2)
run_associationexplorer()

The CRAN release ensures:

standardized installation,
better integration with existing R workflows,
clearer versioning and dependency management.

Who is it for, and how can it be used?

AssociationExplorer is particularly useful for:

exploratory analysis prior to formal modeling,
teaching concepts related to association and dependence,
data storytelling and journalistic exploration,
the analysis of survey data and public datasets.

The goal is not to replace confirmatory statistical analyses, but to provide a robust tool for understanding the structure of the data before modeling.

Acknowledgements

This work was carried out in collaboration with Cédric Heuchenne, Arnaud Claes, and Antonin Descampe, whom I warmly thank.

The project is supported by the Walloon Region and SPW Recherche within the ODALON research project.

Useful links

📄 SoftwareX paper: https://doi.org/10.1016/j.softx.2025.102483
📦 CRAN package: https://CRAN.R-project.org/package=AssociationExplorer2
💻 GitHub repository:
- of the paper
- of the R package

As always, feedback, bug reports, and suggestions are very welcome.

Thanks for reading!

References

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2026). AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns. SoftwareX, 33(102483). https://doi.org/10.1016/j.softx.2025.102483

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2025). AssociationExplorer2: A User-Friendly ‘shiny’ Application for Exploring Associations and Visual Patterns. R package version 0.1.4, https://github.com/AntoineSoetewey/AssociationExplorer2

Nonparametric serial interval estimation

Mon, 18 Aug 2025 00:00:00 +0000

Motivation

Epidemiological delays inform about the time between two well-defined events related to a disease. The serial interval (SI) of an infectious disease is defined as the time between symptom onset in a primary case (infector) and symptom onset in a secondary case (infectee). It is a widely used epidemiological delay quantity and plays a central role in mathematical/statistical models of disease transmission. There exists a tight link between the reproduction number (average number of secondary infections generated by an infected individual) and the serial interval. Therefore, getting accurate knowledge about the SI distribution is key to gain a clear understanding of transmission dynamics during outbreaks. Timings of symptom onset for infector-infectee pairs can be obtained from line list data and observations usually consist of calendar dates. From a mathematical perspective, it is more convenient to work with numbers than with calendar dates and the latter are typically transformed to integers for the sake of statistical analysis.

The main challenge when working with SI data is censoring in the sense that exact symptom onset times are usually unobserved and only known to have occurred between two time points. If the time resolution of a reported timing of illness onset is a calendar day, for instance July 15, there is not enough information to determine the exact time of illness onset within that day. As such, symptom onset is assumed to have occurred between July 15 and July 16 and we say that serial interval data are interval-censored. The figure below illustrates the coarse structure of SI data that adds a layer of complexity to the estimation problem.

Source: Gressani O, Hens N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Comput Biol 21(8): e1013338.

A recent article by Gressani and Hens (2025) published in PLOS Computational Biology proposes a new estimator of the cumulative distribution function of the serial interval without making parametric assumptions regarding the underlying SI distribution. The estimator is based on mixtures of uniform distributions and only requires left and right bounds of serial interval windows of infector-infectee pairs as a main input ($s_{iL}$ and $s_{iR}$ in the above figure). Point estimates of different serial interval features are available in closed-form and the bootstrap is used to compute confidence intervals. The nonparametric methodology is relatively simple and computationally fast and stable. Moreover, a user-friendly routine is available in the EpiDelays package written in R. This post aims at giving users a simple first experience with this new nonparametric methodology for serial interval estimation. The package can be installed from GitHub (using devtools) as follows:

install.packages("devtools")
devtools::install_github("oswaldogressani/EpiDelays")

Simulated data

The estimSI() routine of the EpiDelays package can be used to compute nonparametric estimates (point estimates with standard errors and confidence intervals) of different serial interval features (e.g. the mean, median, standard deviation). The routine is simple to use and requires only two inputs:

x: A data frame with $n$ rows (corresponding to the number of transmission pairs for which illness onset data is available) and two columns containing the lower bound of the SI window $s_{iL}$ (first column) and the upper bound of the SI window $s_{iR}$ (second column).
nboot: An integer for the bootstrap sample size (default is 2000) used to construct ($90\%$ and $95\%$) confidence intervals (CIs).

We start by illustrating the use of estimSI() on simulated data. The simSI() routine can be used to simulate artificial serial interval data with SI windows having a width (coarseness) of at least two days. The underlying target SI distribution is assumed to have a Gaussian distribution with mean muS and standard deviation sdS that have to be specified by the user. The code below can be used to generate $n=15$ SI windows from a Gaussian distribution with a mean of $3$ days and standard deviation of $2$ days. More details regarding the data generating mechanism can be found in the article.

set.seed(2025)
simdata <- simSI(muS = 3, sdS = 2, n = 15)
gt::gt(round(simdata, 2))

s	sl	sr	sw
4.24	3	5	2
3.07	2	4	2
4.55	4	6	2
5.54	5	7	2
3.74	3	5	2
2.67	2	4	2
3.79	3	5	2
2.84	2	4	2
2.31	1	5	4
4.40	3	6	3
2.21	1	3	2
-0.51	-2	1	3
2.16	1	3	2
4.53	2	5	3
5.13	3	7	4

The first column of the simulated dataset contains the true (unobserved) serial interval value generated from the chosen Gaussian distribution. The second and third columns contain the left and right bound of the SI window (sl and sr). Finally, the last column contains the width of the observed SI window, i.e. sw=sr-sl. The underlying target SI distribution is specified to be Gaussian with a mean of $3$ days and standard deviation of $2$ days. The 5th, 25th, 75th and 95th quantiles of the latter distribution are:

round(qnorm(p = c(0.05, 0.25, 0.75, 0.95), mean = 3, sd = 2), 1)

## [1] -0.3  1.7  4.3  6.3

We now create a data frame containing sl and sr and use the latter as an input in the estimSI() routine. Nonparametric estimates of different SI features can be accessed with $npestim$ .

xdf <- data.frame(sl = simdata$sl, sr = simdata$sr)
SIfit <- estimSI(x = xdf, nboot = 2000)
gt::gt(round(SIfit$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	3.4	1.7	0.3	2.5	3.5	4.6	6.0
se	0.4	0.3	1.1	0.4	0.3	0.4	0.5
ci90l	2.8	1.1	-1.3	1.8	2.9	3.9	5.0
ci90r	4.0	2.1	2.1	3.2	4.1	5.2	6.6
ci95l	2.7	1.1	-1.3	1.5	2.8	3.8	4.9
ci95r	4.2	2.2	2.1	3.3	4.2	5.4	6.6

The output shows point estimates (point), standard errors (se) and confidence intervals bounds (ci) for the serial interval mean, standard deviation (sd) and 5th, 25th, 50th, 75th and 95th quantiles denoted by q0.05, q0.25, etc. We can also plot the estimated cumulative distribution function (cdf) obtained with the nonparametric approach and compare it with the target Gaussian cdf. The quality of the fit will typically depend on the sample size $n$ and on the degree of coarseness present in the data. Note that the nonparametric methodology naturally deals with negative SI values.

sl <- simdata$sl
sr <- simdata$sr
Fhat <- function(s) (1 / SIfit$n) * sum((s - sl) / (sr - sl) * (s >= sl & s <= sr) + (s > sr))
sf <- seq(-3, 8, length = 100)
plot(sf, sapply(sf, Fhat), type = "l", lwd = 2, xlab = "Serial interval", ylab = "cdf")
grid()
lines(sf, pnorm(sf, mean = 3, sd = 2), col = "blue", lwd = 2)
legend("topleft", c("Estimated cdf of SI", "Target cdf of SI"), col = c("black", "blue"), lwd = c(2, 2), bty = "n")

Real data

Lessler et al. (2009) share a dataset containing serial interval windows obtained from $n=16$ infector-infectee pairs for Influenza A (2009 H1N1 influenza) at a New York City school. The SI windows are directly available from the supplementary appendix of the latter reference and are encoded in a data frame xNY. Nonparametric estimates of serial interval features are then obtained with estimSI().

xNY <- data.frame(sl = c(1, 1, 1, 0, 0, 4, 2, 3, 0, 3, 0, 3, 4, 1, 3, 3), sr = c(3, 3, 3, 2, 2, 6, 4, 5, 2, 5, 2, 5, 6, 3, 5, 5))
set.seed(123)
SIfitNY <- estimSI(xNY, nboot = 2000)
gt::gt(round(SIfitNY$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	2.8	1.5	0.4	1.5	2.8	4.1	5.2
se	0.3	0.1	0.2	0.4	0.6	0.4	0.3
ci90l	2.2	1.3	0.2	1.1	1.8	3.2	4.7
ci90r	3.4	1.7	1.1	2.2	3.7	4.6	5.6
ci95l	2.1	1.2	0.2	1.0	1.8	3.0	4.6
ci95r	3.5	1.7	1.1	2.5	3.8	4.7	5.7

Interested readers can find more real data examples in Gressani and Hens (2025) and learn about the strengths and limitations of this new nonparametric methodology for serial interval estimation.

References

Gressani, O. and Hens, N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Computational Biology 21(8):e101338. https://doi.org/10.1371/journal.pcbi.1013338

Gressani; O. (2025). EpiDelays: A Software for Estimation of Epidemiological Delays (version 0.0.1). https://github.com/oswaldogressani/EpiDelays

Lessler, J., Reich, N. G., Cummings, D. A., and the New York City Department of Health and Mental Hygiene Swine Influenza Investigation Team. (2009). Outbreak of 2009 pandemic influenza A (H1N1) at a New York City school. New England Journal of Medicine 361(27), 2628-2636. https://www.nejm.org/doi/full/10.1056/NEJMoa0906089

Paper: 'Semi-Markov modeling for disease incidence risk and duration'

Mon, 16 Jun 2025 00:00:00 +0000

I’m happy to share that my latest research paper, “Semi-Markov modeling for disease incidence risk and duration” has been accepted for publication in the journal Biostatistics & Epidemiology (Soetewey et al., 2025).

Read the full paper here.

This work focuses on the use of a Semi-Markov illness-death model to estimate both:

The risk of cancer incidence over a future time period
The number of years of life lost (YLL) due to cancer, with a focus on loss before age 70

The analysis relies on real-world data from the Belgian Cancer Registry, covering over 160,000 cases of melanoma, thyroid, and female breast cancer diagnosed between 2004 and 2020. By modeling transitions between “healthy,” “ill,” and “dead” states, we provide a comprehensive framework to better understand disease burden over time, not just at diagnosis, but also for long-term survivors.

One key feature of this work is its application to non-homogeneous Semi-Markov processes, allowing us to account for the time since diagnosis when estimating survival and life expectancy. This adds a clinically meaningful dynamic dimension to traditional multi-state models.

Beyond its methodological contributions, this study has important implications for public health and insurance regulation. In particular, the results offer quantitative support for the right to be forgotten; a legal provision that allows cancer survivors to apply for credit or insurance products without being penalized once they’ve reached a certain number of years since the end of treatment.

Our results suggest that, for many patients who survive 10 years post-diagnosis, the expected loss in life years compared to the general population becomes minimal, sometimes even below one year. This is especially true for cancers like melanoma and thyroid cancer. These findings could contribute to more equitable and evidence-based insurance underwriting practices.

This research was a collaborative effort, and I would like to express my sincere thanks to my former PhD supervisors, Catherine Legrand, Michel Denuit (UCLouvain) and Geert Silversmit (Belgian Cancer Registry), for their invaluable guidance and support throughout the project.

Thanks for reading!

References

Soetewey, A., Legrand, C., Denuit, M., & Silversmit, G. (2025). Semi-Markov modeling for disease incidence risk and duration. Biostatistics & Epidemiology, 9(1). https://doi.org/10.1080/24709360.2025.2517916

Paper: 'Right to be forgotten for mortgage insurance issued to cancer survivors: critical assessment and new proposal'

Tue, 05 Nov 2024 00:00:00 +0000

I am happy to announce that our paper entitled “Right to be forgotten for mortgage insurance issued to cancer survivors: critical assessment and new proposal” has been accepted for publication in European Actuarial Journal (Soetewey et al., 2025).

In this paper, we propose an alternative method to determine the waiting period opening the right to be forgotten in insurance. This new method is based on a constraint imposed to the premium, which is then transposed into a target on the conditional observed survival. Furthermore, the paper also investigates the impact of the stage of the tumor at diagnosis on waiting periods.

Read more here.

Thanks to all co-authors for the great work. We are also thankful to the two anonymous reviewers for their input that has greatly helped shape the paper.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Soetewey, A., Legrand, C., Denuit, M. et al. Right to be forgotten for mortgage insurance issued to cancer survivors: critical assessment and new proposal. European Actuarial Journal 15, 15–43 (2025). https://doi.org/10.1007/s13385-024-00403-6

EpiLPS for estimation of incubation times

Thu, 01 Aug 2024 00:00:00 +0000

Motivation

A group of researchers from the Data Science Institute (DSI) at Hasselt University developed a new statistical model to estimate the incubation period of a pathogenic organism based on coarse data. The incubation period of an infectious disease (defined as the time elapsed between infection and the manifestation of first symptoms) is of great importance as it permits to shed light on the epidemic potential of a disease and to optimize the length of quarantine periods to freeze transmission. The article (Gressani et al. 2024) was recently published in the American Journal of Epidemiology with practical implementation of the methodology accessible through the EpiLPS package (Gressani et al. 2022).

Coarse data

What makes estimation of incubation times so challenging in the first place? The devil lies in the data. True infection times are stealthy and rarely observed. In information-theoretic jargon this phenomenon is called “imperfect information’’ but statisticians prefer to call it censoring. To be more precise, infection times are interval censored, i.e. one part of the collected data contains exposure intervals $\mathcal{E}=[t^{E_L},t^{E_R}]$ reported by individuals that are part of the study, where $t^{E_L}$ and $t^{E_R}$ stand for the left and right bound, respectively, of the exposure window. The other part of the data contains symptom onset times $t^{\mathcal{S}}$. This is a more easily accessible piece of information -people tend to remember the day when first symptoms appeared- and so the timing of symptom onset is assumed to be exactly observed. Subtracting the exposure bounds from the symptom onset time, one obtains the incubation interval $\mathcal{I}=[t^{\mathcal{I}_L}, t^{\mathcal{I}_R}]$ with lower bound $t^{\mathcal{I}_L}=t^{\mathcal{S}}-t^{E_R}$ and upper bound $t^{\mathcal{I}_R}=t^{\mathcal{S}}-t^{E_L}$, characterizing the coarse data structure which will be the main model input.

Simulated example

In EpiLPS, the estimIncub() routine is designed to compute an estimate of the incubation density based on the methodology of Gressani et al. (2024). Giving a detailed account of the methodology would be out of scope for this blog and the reader is redirected to the article for technicalities. In a nutshell, it is a Bayesian approach making use of (penalized) B-splines, Laplace approximations and Markov chain Monte Carlo (MCMC) methods to derive a semi-parametric estimate of the incubation density. An attractive feature of the estimIncub() routine for the end-user is the minimalistic input it requires to work, namely:

x: A data frame containing the lower and upper bound of the incubation interval.
K: An integer specifying the number of B-splines to smooth the incubation density.
niter: The number of MCMC samples required.

In practice, only the data frame x is required as the remaining inputs are assigned default values. Computationally, the routine requires a small amount of resources as costly subroutines are coded in C++ and integrated in R via the Rcpp package. The structure of x is quite simple. It should be a data frame with two columns containing the left bound $t^{\mathcal{I}_L}$ of the incubation interval (in the first column) and the right bound $t^{\mathcal{I}_R}$.

Let’s start with a simple example where x is simulated based on a data generating mechanism assuming a known incubation distribution. This can be achieved with the incubsim() routine in EpiLPS. We choose x to be generated according to a Lognormal incubation distribution with a mean of 5.5 days and a standard deviation of 2.1 days following Ferretti et al. (2020). Simulation of $n=40$ observations with an average exposure window of 2 days is implemented as follows:

set.seed(2024)
simdat <- incubsim(incubdist = "LogNormal", n = 40, coarseness = 2)
gt(head(simdat$Dobsincub, 5))

tL	tR
3.724500	5.510304
4.381377	6.654224
4.588020	5.483614
3.847835	6.003948
3.522282	5.033487

By typing simdat$Dobsincub, the user has access to the generated incubation intervals (expressed in days), corresponding here to a data frame with two columns and $n=40$ rows. This is the data frame that is injected in the estimIncub() routine:

fit <- estimIncub(x = simdat$Dobsincub, verbose = TRUE)

## ----------------------------------------------------------------
## Time elapsed: 1.339 seconds.
## Fitted density is Log-Normal with meanlog=1.617 and sdlog=0.317.
## Mean incubation period (days): 5.298 with 95% CI: 5.086-5.615.
## 95th percentile (days): 8.484 with 95% CI: 8.121-9.089.
## ----------------------------------------------------------------

The output in the R console can easily be interpreted. It tells us that the model chooses a Lognormal density fit for the incubation period with a mean of 5.3 days (95% CI: 5.0-5.6 days). More detailed summary statistics are accessible by typing fit$stats, such as the posterior standard deviation and additional percentiles. What happens under the hood? Basically, the model computes a semi-parametric fit to the data and compares it with classic parametric fits (Lognormal, Weibull and Gamma) used for incubation modeling. The candidate with the lowest Bayesian information criterion (BIC) wins the game and is finally selected (here the Lognormal distribution). The incubation windows and the fitted incubation density can be obtained by typing:

grid.arrange(plot(fit, typ = "incubwin"), plot(fit, type = "pdf"), nrow = 1)

Real data example

The flexible Bayesian methodology is illustrated on SARS-CoV-2 symptom onset and exposure window data extracted from cases in Vietnam. The dataset was analyzed in Bui et al. (2020) and is publicly available on the GitHub repository provided in the article (https://github.com/longbui/Covid19IncubVN; last accessed July 17, 2024). The dataset contains information about $n=19$ cases identified from January 23, 2020 to April 13, 2020. After continuity corrections (required to change calendar dates into continuous time points), the left and right incubation bounds are given by:

# Left incubation bound
tL <- c(0.504, 2.983, 5.343, 6.969, 6.990, 2.570, 6.870, 1.263, 0.693, 1.869, 1.151, 2.748, 1.209, 1.161, 4.982, 4.017, 2.170, 9.805, 1.659)

# Right incubation bound
tR <- c(1.491, 7.762, 11.777, 12.821, 11.359, 8.551, 24.954, 4.144, 6.408, 7.083, 4.191, 10.326, 9.942, 7.254, 12.690, 10.825, 14.478, 16.592, 18.649)

# Data set
dataVietnam <- data.frame(tL = tL, tR = tR)

The estimIncub() routine provides a Weibull fit with a mean incubation period of 6.7 days (95% CI: 5.8-7.4 days) and the standard deviation (extracted by typing incubfit$stats) is 3.3 days (95% CI: 3.0-3.8 days). A figure of the incubation windows and the fitted Weibull density (with 95% credible interval) is also provided.

set.seed(2024)
incubfit <- EpiLPS::estimIncub(x = dataVietnam, verbose = TRUE, tmax = 25)

## ----------------------------------------------------------------
## Time elapsed: 1.13 seconds.
## Fitted density is Weibull with shape=2.142 and scale=7.521.
## Mean incubation period (days): 6.661 with 95% CI: 5.753-7.375.
## 95th percentile (days): 12.552 with 95% CI: 11.931-14.087.
## ----------------------------------------------------------------

grid.arrange(plot(incubfit, typ = "incubwin"), plot(incubfit, type = "pdf"), nrow = 1)

References

Gressani, O., Torneri, A., Hens, N. and Faes, C. (2024). Flexible Bayesian estimation of incubation times. American Journal of Epidemiology (Accepted manuscript). https://doi.org/10.1093/aje/kwae192

Gressani, O., Wallinga, J., Althaus, C. L., Hens, N. and Faes, C. (2022). EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number. PLoS Comput Biol 18(10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618

Eddelbuettel, D. and Francois, R. (2011). Rcpp: Seamless R and C++ Integration. Journal of Statistical Software, 40(8), 1–18. https://doi.org/10.18637/jss.v040.i08

Ferretti, L. et al. (2020). Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science 368, eabb6936. https://doi.org/10.1126/science.abb6936

Bui LV, Nguyen HT, Levine H, Nguyen HN, Nguyen T-A, Nguyen TP, et al. (2020) Estimation of the incubation period of COVID-19 in Vietnam. PLoS ONE 15(12): e0243889. https://doi.org/10.1371/journal.pone.0243889

Binary logistic regression in R

Tue, 30 Jan 2024 00:00:00 +0000

Introduction

Regression is a common tool in statistics to test and quantify relationships between variables.

The two most common regressions are linear and logistic regressions. A linear regression is used when the dependent variable is quantitative, whereas a logistic regression is used when the dependent variable is qualitative.

Both linear and logistic regressions are divided into different types. Before detailing them, let’s first recap of which type a variable can be:

A quantitative variable measures a quantity, the values it can take are numbers. It is divided into:
- discrete: the values it can take are countable and have a finite number of possibilities (the values are often integers, for example the number of children), and
- continuous: the values it can take are not countable and have an infinite number of possibilities (the values are usually with decimals, or at least decimals are technically possible, for example the weight).
A qualitative variable (also known as categorical) is not numerical and its values fit into categories. It is also divided into two types:
- nominal: no ordering is possible or implied in the categories (for example the sex), and
- ordinal: an order is implied in the categories (for example the health status, such as poor/reasonable/good).

A binary variable, also known as dichotomous, is a special case of qualitative nominal variable when there are only two categories.

See more details and examples about variable types if needed.

Now that the types of a variable is clear, let’s summarize the different types of regression:

Linear regression:
- Simple linear regression is used when the goal is to estimate the relationship between a quantitative continuous dependent variable (also often called outcome or response variable) and only one independent variable (also often called explanatory variable, covariate or predictor) of any type.
- Multiple linear regression is used when the goal is to estimate the relationship between a quantitative continuous dependent variable and two or more independent variables (again, of any type).
Logistic regression:
- Binary logistic regression is used when the goal is to estimate the relationship between a binary dependent variable (= two outcomes), and one or more independent variables (of any type).
- Multinomial logistic regression is used when the goal is to estimate the relationship between a nominal dependent variable with three or more unordered outcomes, and one or more independent variables (of any type).
- Ordinal logistic regression is used when the goal is to estimate the relationship between an ordinal dependent variable with three or more ordered outcomes, and one or more independent variables (of any type).

Note that there exists another type of regression; the Poisson regression. This type of regression is used when the goal is to estimate the relationship between a dependent variable which is in the form of count data (number of occurrences of an event of interest over a given period of time or space, e.g., $0, 1, 2, \ldots$), and one or more independent variables.

Logistic regressions and poisson regressions are both part of a broader type of model called generalized linear models (abbreviated as GLM). The name “generalized linear models” comes from the fact that these models allow to “generalize” the classic linear model. Indeed, it can be used in many situations, for example when analyzing a dependent variable which is not necessarily quantitative continuous or when residuals are not normally distributed (which are prerequisites for a linear model).

Linear regression and its application in R have already been presented in this post. It is now time to present the logistic regression.

Binary logistic regression being the most common and the easiest one to interpret among the different types of logistic regression, this post will focus only on the binary logistic regression. Other types of regression (multinomial & ordinal logistic regressions, as well as Poisson regressions are left for future posts).

In this post, we will first explain when a logistic regression is more appropriate than a linear regression. We will then show how to perform a binary logistic regression in R, and how to interpret and report results. We will also present some plots in order to visualize results. Finally, we will cover the topics of model selection, quality of fit and underlying assumptions of a binary logistic regression. We will try to keep this tutorial as applied as possible by focusing on the applications in R and the interpretations. Mathematical details will be as concise as possible.

Linear versus logistic regression

We know that a linear regression is a convenient way to estimate the relationship between a quantitative continuous dependent variable, and one or more independent variables (of any type).

For instance, suppose we would like to estimate the relationship between two quantitative variables, $X$ and $Y$. Using the ordinary least squares method (the most common estimator used in linear regression), we obtain the following regression line:

Now suppose we are interested in estimating the impact of age on whether or not a patient has a certain disease. Age is considered as a quantitative continuous variable, while having the disease is binary (a patient is either ill or healthy).

Visually, we could have something like this:

If we fit a regression line (using the ordinary least square method) to the points, we obtain the following plot:

We see that the regression line goes below 0 and above 1 with respect to the $y$-axis. Since the dependent variable disease cannot take values below 0 (= healthy) nor above 1 (= ill), it is obvious that a linear regression is not appropriate for these data!

In addition to this limitation, the assumptions of normality and homoscedasticity, which are required in linear regression, are clearly not appropriate with these data since the dependent variable is binary and follows a Binomial distribution! R will not stop you from performing a linear regression on binary data, but this will produce a model of little interest.

This is where a logistic regression becomes handy as it takes into consideration these limitations.

Applied to our example, here is how the points are fitted using a binary logistic regression:

It is clear that this model is more appropriate.

The curve (known as a sigmoid) is obtained via a transformation of the predicted values. There are several possible choices for the link function, which aim is to constrain predicted values to be within the range of observed values.

The most widely used in practice is the logit function, which relates the probability of occurrence of an event (bounded between 0 and 1) to the linear combination of independent variables. The logit function also turns out to be the canonical link function for a Bernoulli or Binomial distribution. This transformation ensures that no matter in which range the $X$ values are located, $Y$ will only take numbers between 0 and 1.

One could say that the fitted values (= represented by the blue curve) taking values between 0 and 1 also does not seem to make sense since a patient can only be healthy or ill (and thus the dependent variable can only take the value 0 or 1, respectively). However, in a binary logistic regression it is not the outcome no disease/disease that is directly modeled, but the likelihood that a patient has the disease or not given his or her characteristics. This likelihood will be framed in terms of a probability to observe or not the disease in a patient, which is indeed included between 0 and 1, or 0% and 100%.

More generally, with a logistic regression we would like to model how the probability of success varies with the independent variables and determine whether or not these changes are statistically significant. We are actually going to model the logarithm of the odds, and the logistic regression model will be written as follows:

\[\begin{align} \log(odds(success)) &= logit(\pi) \\ &= \log\left(\frac{\pi}{1 - \pi}\right) \\ &= \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p \end{align}\]

where $\pi$ $(0 \le \pi \le 1)$ is the probability of an event happening (success) and denoted $\pi = P(success)$. We find the values of $\hat{\beta}_0$, $\hat{\beta}_1$, $\ldots$, $\hat{\beta}_p$, which are used as estimates for $\beta_0$, $\beta_1$, $\ldots$, $\beta_p$, using the maximum likelihood method. This method is one of several methods used in statistics to estimate parameters of a mathematical model. The goal of the estimator is to estimate the parameters $\beta_0$, $\beta_1$, $\ldots$, $\beta_p$ which maximize the log likelihood function. Different algorithms have been established over the years for this non-linear optimization, but this is beyond the scope of the post.

Logistic regressions are very common in the medical field, for example to:

estimate the risk factors associated with a disease or a harmful condition,
predict the risk of developing a disease based on a patient’s characteristics, or
determine the most important biological factors associated with a specific disease or condition.

However, logistic regressions are used in many other domains, for instance in:

banking sector: estimate a debtor’s creditworthiness based on his or her profile (income, assets, liabilities, etc.),
marketing: estimate a customer’s propensity to buy a product or service based on his or her profile (age, sex, salary, previous purchases, etc.),
sports: estimate the probability of a player winning against another player as a function of the characteristics of the two opponents,
politics: answer the question “Would a citizen vote for our political party at the next elections?”
etc.

Univariable versus multivariable logistic regression

Now that it is more clear when a binary logistic regression should be used, we show how to perform one in R. We start by presenting univariable binary logistic regressions, and then multivariable binary logistic regressions.

Remember that in both cases, the dependent variable must be a qualitative variable with two outcomes (hence the name binary logistic regression). The difference between a univariable and multivariable binary logistic regression lies in the fact that:

for a univariable binary logistic regression, there is only one independent variable, while
for a multivariable binary logistic regression, there are two ore more independent variables.

It is true that the term “univariable” may be confusing here because there are two variables in the model (i.e., one dependent variable and one independent variable). However, it is called univariable binary logistic regression to indicate that only one independent variable is considered in the model, as opposed to multivariable binary logistic regression where several independent variables are considered in the model.

To draw a parallel with linear regression:

a univariable binary logistic regression is the equivalent of a simple linear regression, whereas
a multivariable binary logistic regression is the equivalent of a multiple linear regression

when the dependent variable is binary instead of quantitative continuous. This is the reason a univariable binary logistic regression is sometimes called a simple binary logistic regression and a multivariable binary logistic regression sometimes called a multiple binary logistic regression.

The terms univariable/multivariable should not be confused with univariate/multivariate. The number of dependent variables characterizes the model as univariate or multivariate; univariate refers to a model with only one dependent variable, while multivariate refers to a model that simultaneously predicts more than one dependent variable. Usually, the intent is to differentiate models based on the number of independent variables. This distinction is made thanks to the terms univariable and multivariable. Multivariable refers to a model relating multiple predictor variables to a dependent variable, whereas univariable refers to a model relating one single independent variable to a dependent variable.

Data

For these illustrations, we use the “Heart Disease” dataset, available from the {kmed} R package. This data frame consists of 14 variables, of which only 5 of them are kept for this post:

age: age in years
sex: sex (FALSE = female, TRUE = male)
cp: chest pain type (1 = typical angina, 2 = atypical angina, 3 = non-anginal pain, 4 = asymptomatic)
thalach: maximum heart rate achieved
class: diagnosis of heart disease (divided into 4 classes)

# import and rename dataset
library(kmed)
dat <- heart

# select variables
library(dplyr)
dat <- dat |>
  select(
    age,
    sex,
    cp,
    thalach,
    class
  )

# print dataset's structure
str(dat)

## 'data.frame':	297 obs. of  5 variables:
##  $ age    : num  63 67 67 37 41 56 62 57 63 53 ...
##  $ sex    : logi  TRUE TRUE TRUE TRUE FALSE TRUE ...
##  $ cp     : Factor w/ 4 levels "1","2","3","4": 1 4 4 3 2 2 4 4 4 4 ...
##  $ thalach: num  150 108 129 187 172 178 160 163 147 155 ...
##  $ class  : int  0 2 1 0 0 0 3 0 2 1 ...
##  - attr(*, "na.action")= 'omit' Named int [1:6] 88 167 193 267 288 303
##   ..- attr(*, "names")= chr [1:6] "88" "167" "193" "267" ...

Note that the pipe operator |> and the {dplyr} package is used to select variables. See more data manipulation techniques using this package if you are interested.

For greater readability, we rename the variables cp, thalach and class with more informative names:

# rename variables
dat <- dat |>
  rename(
    chest_pain = cp,
    max_heartrate = thalach,
    heart_disease = class
  )

We transform the variables sex and chest_pain into factor and set the labels accordingly:

# recode sex
dat$sex <- factor(dat$sex,
  levels = c(FALSE, TRUE),
  labels = c("female", "male")
)

# recode chest_pain
dat$chest_pain <- factor(dat$chest_pain,
  levels = 1:4,
  labels = c("typical angina", "atypical angina", "non-anginal pain", "asymptomatic")
)

For a binary logistic regression in R, it is recommended that all the qualitative variables are transformed into factors.

In our case, heart_disease (our dependent variable) is currently encoded as integer with values ranging from 0 to 4. Therefore, we first classify it into 2 classes by setting 0 for 0 values and 1 for non-0 values, using the ifelse() function:

# recode heart_disease into 2 classes
dat$heart_disease <- ifelse(dat$heart_disease == 0,
  0,
  1
)

We then transform it into a factor and set the labels accordingly using the factor() function:

# set labels for heart_disease
dat$heart_disease <- factor(dat$heart_disease,
  levels = c(0, 1),
  labels = c("no disease", "disease")
)

Keep in mind the order of the levels for your dependent variable, as it will have an impact on the interpretations. In R, the first level given by levels() is always taken as the reference level.

In our case, the first level is the absence of the disease and the second level is the presence of the disease:

levels(dat$heart_disease)

## [1] "no disease" "disease"

This means that when we will build the models, we will estimate the impact of the independent variable(s) on the presence of the disease (and not the absence!).

This is the reason that, for dependent variables of the type no/yes, false/true, absence/presence of a condition, etc. it is recommended to set the level no, false, absence of the condition, etc. as the reference level. It is indeed usually easier to interpret the impact of an independent variable on the presence of a condition/disease than the opposite.

If you want to switch the reference level, this can be done with the relevel() function.

Here is a preview of the final data frame and some basic descriptive statistics:

# print first 6 observations
head(dat)

##   age    sex       chest_pain max_heartrate heart_disease
## 1  63   male   typical angina           150    no disease
## 2  67   male     asymptomatic           108       disease
## 3  67   male     asymptomatic           129       disease
## 4  37   male non-anginal pain           187    no disease
## 5  41 female  atypical angina           172    no disease
## 6  56   male  atypical angina           178    no disease

# basic descriptive statistics
summary(dat)

##       age            sex                 chest_pain  max_heartrate  
##  Min.   :29.00   female: 96   typical angina  : 23   Min.   : 71.0  
##  1st Qu.:48.00   male  :201   atypical angina : 49   1st Qu.:133.0  
##  Median :56.00                non-anginal pain: 83   Median :153.0  
##  Mean   :54.54                asymptomatic    :142   Mean   :149.6  
##  3rd Qu.:61.00                                       3rd Qu.:166.0  
##  Max.   :77.00                                       Max.   :202.0  
##     heart_disease
##  no disease:160  
##  disease   :137  
##                  
##                  
##                  
##

The data frame is now ready to be analyzed further through univariable and multivariable binary logistic regressions.

Binary logistic regression in R

Univariable binary logistic regression

As mentioned above, we start with a univariable binary logistic regression, that is, a binary logistic regression with only one independent variable.

In R, a binary logistic regression can be done with the glm() function and the family = "binomial" argument. Similar to linear regression, the formula used inside the function must be written as dependent variable ~ independent variable (in this order!).

While the dependent variable must be categorical with two levels, the independent variable can be of any type. However, interpretations differ depending on whether the independent variable is qualitative or quantitative.

For completeness, we illustrate this type of regression with both a quantitative and a qualitative independent variable, starting with a quantitative independent variable.

Quantitative independent variable

Suppose we want to estimate the impact of a patient’s age on the presence of heart disease. In this case, age is our independent variable and heart_disease is our dependent variable:

# save model
m1 <- glm(heart_disease ~ age,
  data = dat,
  family = "binomial"
)

Results of the model is saved under the object m1. Again, similar to linear regression, results can be accessed thanks to the summary() function:

# print results
summary(m1)

## 
## Call:
## glm(formula = heart_disease ~ age, family = "binomial", data = dat)
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -3.05122    0.76862  -3.970  7.2e-05 ***
## age          0.05291    0.01382   3.829 0.000128 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 409.95  on 296  degrees of freedom
## Residual deviance: 394.25  on 295  degrees of freedom
## AIC: 398.25
## 
## Number of Fisher Scoring iterations: 4

The most important results in this output are displayed in the table after Coefficients. The bottom part of the output summarizes the distribution of the deviance residuals. In a nutshell, deviance residuals measure how well the observations fit the model. The closer the residual to 0, the better the fit of the observation.

Within the Coefficients table, we focus on the first and last columns (the other two columns correspond to the standard error and the test statistic, which are both used to compute the $p$-value):

the column Estimate corresponds to the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, and
the column Pr(>|z|) corresponds to the $p$-values.

R performs a hypothesis test for each coefficient, that is, $H_0: \beta_j = 0$ versus $H_1: \beta_j \neq 0$ for $j = 0, 1$ via the Wald test, and print the $p$-values in the last column. We can thus compare these $p$-values to the chosen significance level (usually $\alpha = 0.05$) to conclude whether or not each of the coefficient is significantly different from 0. The lower the $p$-value, the more evidence that the coefficient is different from 0. This is similar to linear regression.

Coefficients are slightly harder to interpret in logistic regression than in linear regression because the relationship between dependent and independent variables is not linear.

Let’s first interpret the coefficient of age, $\hat{\beta}_1$, which is the most important coefficient of the two.

First, since the $p$-value of the test on the coefficient for age is < 0.05, we conclude that it is significantly different from 0, which means that age is significantly associated with the presence of heart disease (at the 5% significance level). Note that if the test was not significant (i.e., the $p$-value $\ge \alpha$), we would refrain from interpreting the coefficient since it means that, based on the data at hand, we are unable to conclude that age is associated with the presence of heart disease in the population.

Second, remember that:

when $\beta_1 = 0$, $X$ and $Y$ are independent,
when $\beta_1 > 0$, the probability that $Y = 1$ increases with $X$, and
when $\beta_1 < 0$, the probability that $Y = 1$ decreases with $X$.

In our context, we have:

when $\hat{\beta}_1 = 0$, the probability of developing a heart disease is independent of the age,
when $\hat{\beta}_1 > 0$, the probability of developing a heart disease increases with age, and
when $\hat{\beta}_1 < 0$, the probability of developing a heart disease decreases with age.

Here we have $\hat{\beta}_1 =$ 0.053 $> 0$, so we already know that the older the patient, the more likely he or she is to develop a heart disease. This makes sense.

Now that we know the direction of the relationship, we would like to quantify this relationship. This is easily done thanks to odds ratios (OR). OR are found by taking the exponential of the coefficients. In R, the exponential is done thanks to the exp() function.

OR can be interpreted as follows: the OR is the multiplicative change in the odds in favor of $Y = 1$ when $X$ increases by 1 unit.

Applied to our context, we compute the OR for the age by computing $\exp(\hat{\beta}_1) =$ exp(0.053).

Using R, this gives:

# OR for age
exp(coef(m1)["age"])

##      age 
## 1.054331

Based on this result, we can say that an extra year of age increases the odds (that is, the chance) of developing a heart disease by a factor of 1.054.

Therefore, the odds of developing a heart disease increases by (1.054 - 1) $\times$ 100 = 5.4% when a patient becomes one year older.

To sum up:

when the coefficient $\hat{\beta}_1 = 0 \Rightarrow$ OR $= \exp(\hat{\beta}_1) = 1 \Rightarrow P(Y = 1)$ is independent of $X \Rightarrow$ there is no relationship between $X$ and $Y$,
when the coefficient $\hat{\beta}_1 > 0 \Rightarrow$ OR $= \exp(\hat{\beta}_1) > 1 \Rightarrow P(Y = 1)$ increases with $X \Rightarrow$ there is a positive relationship between $X$ and $Y$, and
when the coefficient $\hat{\beta}_1 < 0 \Rightarrow$ OR $= \exp(\hat{\beta}_1) < 1 \Rightarrow P(Y = 1)$ decreases with $X \Rightarrow$ there is a negative relationship between $X$ and $Y$.

We now interpret the intercept $\hat{\beta}_0$.

First, we look at the $p$-value of the test on the intercept. This $p$-value being < 0.05, we conclude that the intercept is significantly different from 0 (at the 5% significance level).

Second, similar to linear regression, in order to obtain an interpretation of the intercept, we need to find a situation in which the other coefficient, $\beta_1$, vanishes.

In our case, it happens when a patient is 0 year old. We may or may not need to interpret results in a such a situation, and in many situations the interpretation does not make sense, so it is more a hypothetical interpretation. However, again for completeness we show how to interpret the intercept. Note that it is also possible to center the numeric variable so that the intercept has a more meaningful interpretation. This, however, goes beyond the scope of the post.

For a patient aged 0 year, the odds of developing a heart disease is $\exp(\hat{\beta}_0) =$ exp(-3.051) = 0.047. When interpreting an intercept, it often makes more sense to interpret it as the probability that $Y = 1$, which can be computed as follows:

\[\frac{\exp(\hat{\beta}_0)}{1 + \exp(\hat{\beta}_0)}.\]

In our case, it corresponds to the probability that a patient of age 0 develops a heart disease, which is equal to:

# prob(heart disease) for age = 0
exp(coef(m1)[1]) / (1 + exp(coef(m1)[1]))

## (Intercept) 
##  0.04516478

This means that, if we trust our model, a newborn is expected to develops a heart disease with a probability of 4.52%.

For your information, a confidence interval can be computed for any of the OR using the confint() function. For example, a 95% confidence interval for the OR for age:

# 95% CI for the OR for age
exp(confint(m1,
  parm = "age"
))

##    2.5 %   97.5 % 
## 1.026699 1.083987

Remember than when evaluating an OR, the null value is 1, not 0. An OR of 1 in this study would mean that there is no association between the age and the presence of heart disease. If the 95% confidence interval of the OR does not include 1, we conclude that there is a significant association between the age and the presence of heart disease. On the contrary, if it includes 1, we do not reject the hypothesis that there is no association between age and the presence of heart disease.

In our case, the 95% CI does not include 1, so we conclude, at the 5% significance level, that there is a significant association between age and the presence of heart disease.

You will notice that it is the same conclusion than with the $p$-value. This is normal, it will always be the case:

if the $p$-value < 0.05, 1 will not be included in the 95% CI, and
if the $p$-value $\ge$ 0.05, 1 will be included in the 95% CI.

This means that you can choose whether you draw your conclusion about the association between the two variables based on the 95% CI or the $p$-value.

Estimating the relationship between variables is the main reason for building models. Another goal is to predict the dependent variable based on newly observed values of the independent variable(s). This can be done with the predict() function.

Suppose we would like to predict the probability of developing a heart disease for a patient aged 30 years old:

# predict probability to develop heart disease
pred <- predict(m1,
  newdata = data.frame(age = c(30)),
  type = "response"
)

# print prediction
pred

##         1 
## 0.1878525

It is predicted that a 30-year-old patient has a 18.79% chance of developing a heart disease.

Note that if you would like to construct a confidence interval for this prediction, it can be done by adding the se = TRUE argument in the predict() function:

# predict probability to develop heart disease
pred <- predict(m1,
  newdata = data.frame(age = c(30)),
  type = "response",
  se = TRUE
)

# print prediction
pred$fit

##         1 
## 0.1878525

# 95% confidence interval for the prediction
lower <- pred$fit - (qnorm(0.975) * pred$se.fit)
upper <- pred$fit + (qnorm(0.975) * pred$se.fit)
c(lower, upper)

##          1          1 
## 0.07873357 0.29697138

If you are a frequent reader of the blog, you are probably know that I like visualizations. The plot_model() function available in the {sjPlot} R package does a good job of visualizing results of the model:

# load package
library(sjPlot)

# plot
plot_model(m1,
  type = "pred",
  terms = "age"
) +
  labs(y = "Prob(heart disease)")

For those of you who are familiar with the {ggplot2} package, you will have noticed that the function accepts layers from the {ggplot2} package. Note also that this function works with other types of model (such as linear models).

The above plot shows the probability of developing a heart disease in function of age, and confirms results found above:

We see that the probability of developing a heart disease increases with age (which was expected given that the OR for the coefficient of age is > 1),
and we also see that the probability of developing a heart disease for a 30-year-old patient is slightly below 20%.

Qualitative independent variable

Suppose now that we are interested in estimating the relationship between the probability of developing a heart disease and the sex (which is a qualitative variable).

Recall that when the independent variable was quantitative, $\exp(\hat{\beta}_1)$ was the multiplicative change in the odds in favor of $Y = 1$ as $X$ increases by 1 unit.

With $X$ being the sex, the only unit increase possible is from 0 to 1 (or from 1 to 2 if sex is encoded as a factor), so we can write an interpretation in terms of female/male:

$\exp(\hat{\beta}_1)$ is the multiplicative change of the odds in favor of $Y = 1$ as a female becomes a male.

Again, keep in mind what is the order of the level for the variable sex. In our case, the level female comes before the level male:

# levels for sex
levels(dat$sex)

## [1] "female" "male"

So it is indeed the multiplicative change of the odds in favor of $Y = 1$ as a female becomes a male. If the level male came before the level female in our dataset, it would have been the opposite.

You will concede that it is rather strange to interpret odds as a female becomes a male, or vice versa. Therefore, it is better to say:

$\exp(\hat{\beta}_1)$ is the multiplicative change of the odds in favor of $Y = 1$ for males versus females.

In our case, we obtain the following results:

# save model
m2 <- glm(heart_disease ~ sex,
  data = dat,
  family = "binomial"
)

# print results
summary(m2)

## 
## Call:
## glm(formula = heart_disease ~ sex, family = "binomial", data = dat)
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  -1.0438     0.2326  -4.488 7.18e-06 ***
## sexmale       1.2737     0.2725   4.674 2.95e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 409.95  on 296  degrees of freedom
## Residual deviance: 386.12  on 295  degrees of freedom
## AIC: 390.12
## 
## Number of Fisher Scoring iterations: 4

# OR for sex
exp(coef(m2)["sexmale"])

##  sexmale 
## 3.573933

which can be interpreted as follows:

The $p$-value of the test on the coefficient for sex is < 0.05, so we conclude that the sex is significantly associated with the presence of heart disease (at the 5% significance level).
Moreover, when looking at the coefficient for sex, $\hat{\beta}_1 =$ 1.274, we can say that:
- For males, the odds of developing a heart disease is multiplied by a factor of exp(1.274) = 3.574 compared to females.
- In other words, the odds of developing a heart disease for males are 3.574 times the odds for females.
- This means that, the odds of developing a heart disease are (3.574 - 1) $\times$ 100 = 257.4% higher for males than for females.

These results again make sense, as it known that men are more likely to have a heart disease than women.

The interpretation of the intercept $\hat{\beta}_0 =$ -1.044 is similar than in the previous section in the sense that it gives the probability of developing a heart disease when the other coefficient, $\beta_1$, is equal to 0.

In our case, $\beta_1 = 0$ simply means that the patient is a female. Therefore, the probability of developing a heart disease for a woman is:

# prob(disease) for sex = female
exp(coef(m2)[1]) / (1 + exp(coef(m2)[1]))

## (Intercept) 
##   0.2604167

The avid reader will notice that a univariable binary logistic regression with a qualitative independent variable will lead to the same conclusion than a Chi-square test of independence:

chisq.test(table(dat$heart_disease, dat$sex))

## 
## 	Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  table(dat$heart_disease, dat$sex)
## X-squared = 21.852, df = 1, p-value = 2.946e-06

Based on this test, we reject the null hypothesis of independence between the two variables and we thus conclude that there is a significant association between the sex and the presence of heart disease ($p$-value < 0.001).

The advantage of a univariable binary logistic regression over a Chi-square test of independence is that it not only tests whether or not there is a significant association between the two variables, but it also estimates the direction and strength of this relationship.

As for a univariable logistic regression with a quantitative independent variable, predictions can also be made with the predict() function. Suppose we would like to predict the probability of developing a heart disease for a male:

# predict probability to develop heart disease
pred <- predict(m2,
  newdata = data.frame(sex = c("male")),
  type = "response"
)

# print prediction
pred

##         1 
## 0.5572139

Based on this model, it is predicted that a male patient has 55.72% chance of developing a heart disease.

We can also visualize these results thanks to the plot_model() function:

# plot
plot_model(m2,
  type = "pred",
  terms = "sex"
) +
  labs(y = "Prob(heart disease)")

The points correspond to the predicted probabilities, and the bars correspond to their confidence intervals.

The plot of the results in terms of probabilities confirms what was found above, that is:

women are less likely to develop a heart disease than men,
the probability that a woman develops a heart disease is expected to be slightly above 25%, and
the probability that a man develops a heart disease is expected to be around 55%.

Multivariable binary logistic regression

The interpretation of the coefficients in multivariable logistic regression is similar to the interpretation in univariable regression, except that this time it estimates the multiplicative change in the odds in favor of $Y = 1$ when $X$ increases by 1 unit, while the other independent variables remain unchanged.

This is similar to multiple linear regression, where a coefficient gives the expected change of $Y$ for an increase of 1 unit of $X$, while keeping all other variables constant.

The main advantages of using a multivariable logistic regression compared to a univariable logistic regression are to:

consider the simultaneous (rather than isolated) effect of independent variables,
take into account potential confounding and/or interaction effects, and
improve predictions.

For this illustration, suppose we would like to estimate the relationship between heart disease and all variables present in the data frame, that is, age, sex, chest pain type and maximum heart rate achieved:

# save model
m3 <- glm(heart_disease ~ .,
  data = dat,
  family = "binomial"
)

# print results
summary(m3)

## 
## Call:
## glm(formula = heart_disease ~ ., family = "binomial", data = dat)
## 
## Coefficients:
##                             Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                -0.060150   1.962091  -0.031 0.975544    
## age                         0.042814   0.019009   2.252 0.024302 *  
## sexmale                     1.686330   0.349352   4.827 1.39e-06 ***
## chest_painatypical angina  -0.120481   0.641396  -0.188 0.851000    
## chest_painnon-anginal pain -0.124331   0.571093  -0.218 0.827658    
## chest_painasymptomatic      1.963723   0.548877   3.578 0.000347 ***
## max_heartrate              -0.030326   0.007975  -3.802 0.000143 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 409.95  on 296  degrees of freedom
## Residual deviance: 275.26  on 290  degrees of freedom
## AIC: 289.26
## 
## Number of Fisher Scoring iterations: 5

Note that the formula heart_disease ~ . is a shortcut to include all variables present in the data frame in the model as independent variables, except heart_disease.

Based on the $p$-values displayed in the last column of the coefficients table, we conclude that, at the 5% significance level, age, sex and maximum heart rate achieved are all significantly associated with heart disease ($p$-values < 0.05).

For the variables age, sex and max_heartrate, there is only one $p$-value (the result of the test on the nullity of the coefficient).

For the variable chest_pain, 3 $p$-values are displayed. This is normal: similar to linear regression when a categorical variable with more than two levels is included in the model, one test is performed for each comparison between the reference level and the other levels.

In our case, the reference level for the variable chest_pain is typical angina as it comes first:

levels(dat$chest_pain)

## [1] "typical angina"   "atypical angina"  "non-anginal pain" "asymptomatic"

Therefore, a test is performed for the comparison between:

typical angina and atypical angina,
typical angina and non-anginal pain, and
typical angina and asymptomatic.

But here we are not interested in comparing levels of the variable chest pain, we would like to test the overall effect of chest pain on heart disease. For this, we are going to compare two models via a likelihood ratio test (LRT):

a model which includes all the variables of interest and the variable chest_pain, and
the exact same model but which excludes the variable chest_pain.

The first one is referred as the full or complete model, whereas the second one is referred as the reduced model.

We compare these two models with the anova() function:

# save reduced model
m3_reduced <- glm(heart_disease ~ age + sex + max_heartrate,
  data = dat,
  family = "binomial"
)

# compare reduced with full model
anova(m3_reduced, m3,
  test = "LRT"
)

## Analysis of Deviance Table
## 
## Model 1: heart_disease ~ age + sex + max_heartrate
## Model 2: heart_disease ~ age + sex + chest_pain + max_heartrate
##   Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
## 1       293     325.12                          
## 2       290     275.26  3    49.86 8.558e-11 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Notice that the reduced model must come before the complete model in the anova() function. The null hypothesis of this test is that the two models are equivalent.

At the 5% significance level (see the $p$-value at the right of the R output), we reject the null hypothesis and we conclude that the complete model is significantly better than the reduced model at explaining the presence of heart disease. This means that chest pain is significantly associated with heart disease (which was expected since the comparison between typical angina and asymptomatic was found to be significant).

A comparison of two models via the LRT will be shown again later, when discussing about interactions.

Now that we have shown that all four independent variables were significantly associated with the presence of heart disease, we can interpret the coefficients in order to know the direction of the relationships and most importantly, quantify the strength of these relationships.

Like univariable binary logistic regression, it is easier to interpret these relationships through OR. But this time, we also print the 95% CI of the OR in addition to the OR (rounded to 3 decimals) so that we can easily see which ones are significantly different from 1:

# OR and 95% CI
round(exp(cbind(OR = coef(m3), confint(m3))), 3)

##                               OR 2.5 % 97.5 %
## (Intercept)                0.942 0.020 44.353
## age                        1.044 1.006  1.084
## sexmale                    5.400 2.776 10.971
## chest_painatypical angina  0.886 0.252  3.191
## chest_painnon-anginal pain 0.883 0.293  2.814
## chest_painasymptomatic     7.126 2.509 22.030
## max_heartrate              0.970 0.955  0.985

From the OR and their 95% CI computed above, we conclude that:

Age: the odds of having a heart disease are multiplied by a factor of 1.04 for each one-unit increase in age, all else being equal.
Sex: the odds of developing a heart disease for males are 5.4 times the odds for females, all else being equal.
Chest pain: the odds of developing a heart disease for people suffering from chest pain of the type “asymptomatic” are 7.13 times the odds for people suffering from chest pain of the type “typical angina”, all else being equal. We refrain from interpreting the other comparisons as they are not significant at the 5% significance level (1 is included in their 95% CI).
Maximum heart rate achieved: the odds of having a heart disease are multiplied by a factor of 0.97 for each one-unit increase in maximum heart rate achieved, all else being equal.
Intercept: we also refrain from interpreting the intercept as it is not significantly different from 0 at the 5% significance level (1 is included in the 95% CI).

If you are interested in printing only the OR for which the $p$-value of the coefficient is < 0.05, here is the code:

exp(coef(m3))[coef(summary(m3))[, "Pr(>|z|)"] < 0.05]

##                    age                sexmale chest_painasymptomatic 
##              1.0437437              5.3996293              7.1258054 
##          max_heartrate 
##              0.9701289

Remember that we can always write the interpretations in terms of the percentage increase/decrease in odds with the formula $(OR - 1) \times 100$, where OR corresponds to the odds ratio.

For instance, for the maximum heart rate achieved, the OR = 0.97, so the interpretation becomes: the odds of developing a heart disease increases by (0.97 $- 1) \times 100 =$ -3% for each one-unit increase in maximum heart rate achieved, which is equivalent to say that the odds of developing a heart disease decreases by 3% for each one-unit increase in maximum heart rate achieved.

For illustrative purposes, suppose now that we would like to predict the probability that a new patient develops a heart disease. Suppose that this patient is a 32-year-old woman, suffering from chest pain of the type non-anginal and she achieved a maximum heart rate of 150. The probability that she develops a heart disease is:

# create data frame of new patient
new_patient <- data.frame(
  age = 32,
  sex = "female",
  chest_pain = "non-anginal pain",
  max_heartrate = 150
)

# predict probability to develop heart disease
pred <- predict(m3,
  newdata = new_patient,
  type = "response"
)

# print prediction
pred

##          1 
## 0.03345948

If we trust our model, the probability that this new patient will develop a heart disease is predicted to be 3.35%.

We can also visualize the results thanks to the plot_model() function, three effects at the same time:

effect of age, sex and chest pain type on the predicted probability of developing a heart disease, and
effect of maximum heart rate achieved, sex and chest pain type on the predicted probability of developing a heart disease.

# 1. age, sex and chest pain on prob of disease
plot_model(m3,
  type = "pred",
  terms = c("age", "chest_pain", "sex"),
  ci.lvl = NA # remove confidence bands
) +
  labs(y = "Prob(heart disease)")

# 2. max heart rate, chest pain and sex on prob of disease
plot_model(m3,
  type = "pred",
  terms = c("max_heartrate", "chest_pain", "sex"),
  ci.lvl = NA # remove confidence bands
) +
  labs(y = "Prob(heart disease)")

For more clarity in the plots, confidence bands are removed thanks to ci.lvl = NA.

These plots confirm results obtained above, that is:

there is a positive relationship between age and the presence of heart disease,
there is a negative relationship between maximum heart rate achieved and the presence of heart disease,
the odds of developing a heart disease is higher for patients suffering from chest pain of the type asymptomatic and similar for the 3 other types of chest pain, and
the odds of developing a heart disease is higher for males than for females.

Interaction

In the previous sections, potential interaction effects were omitted.

An interaction occurs when the relationship between an independent variable and the outcome variable depends on the value or the level taken by another independent variable. On the contrary, if the relationship between an independent variable and the dependent variable remains unchanged no matter the value taken by another independent variable, we cannot conclude that there is an interaction effect.

In our case, there would be an interaction if for example the relationship between age and heart disease depends on the sex. There would be an interaction, for instance, if the relationship between age and heart disease was positive for females, and negative for males, or vice versa. Or if the relationship between age and heart disease was much stronger or much weaker for females than for males.

Let’s see if there is an interaction between age and sex, and more importantly, whether or not this interaction is significant. For this, we need to build two models:

one model containing only the main effects, so without the interaction, and
one model containing the main effects and the interaction.

# save model without interaction
m4 <- glm(heart_disease ~ age + sex,
  data = dat,
  family = "binomial"
)

# save model with interaction
m4_inter <- glm(heart_disease ~ age * sex,
  data = dat,
  family = "binomial"
)

We first assess the interaction visually via the plot_model() function:

# plot
plot_model(m4_inter,
  type = "pred",
  terms = c("age", "sex"),
  ci.lvl = NA # remove confidence bands
) +
  labs(y = "Prob(heart disease)")

Since the two curves of the predicted probabilities are relatively similar and follow the same pattern, the relationship between age and the presence of heart disease does not seem to depend on the sex, indicating that there may indeed be no interaction. However, we would like to test it more formally via a statistical test.

For this, we can compare the two models (the one without compared to the one with interaction) with a likelihood ratio test (LRT), using the anova() function:

anova(m4, m4_inter,
  test = "LRT"
)

## Analysis of Deviance Table
## 
## Model 1: heart_disease ~ age + sex
## Model 2: heart_disease ~ age * sex
##   Resid. Df Resid. Dev Df Deviance Pr(>Chi)
## 1       294     364.43                     
## 2       293     364.23  1  0.20741   0.6488

Remember that it is always the reduced model as the first argument in the anova() function, and then the more complex model as the second argument.

The test confirms what we supposed based on the plot: at the 5% significance level, we do not reject the null hypothesis that the two models are equivalent. Since the only difference between the two models is that an interaction term is added in the complete model, we do not reject the hypothesis that there is no interaction between the age and the sex ($p$-value = 0.649).

This conclusion could have also been obtained more simply with the drop1() function:

drop1(m4_inter,
  test = "LRT"
)

## Single term deletions
## 
## Model:
## heart_disease ~ age * sex
##         Df Deviance    AIC     LRT Pr(>Chi)
## <none>       364.23 372.23                 
## age:sex  1   364.43 370.43 0.20741   0.6488

As you can see, this gives the same $p$-value of 0.649.

In practice, a non-significant interaction is removed from the model before interpreting its results. In our case, only the main effects of age and sex would remain in the model.

This leads us to model selection, or which variables should be included in our final model. This is discussed in the next section.

Model selection

In practice, we often have several models, corresponding to the different combinations of independent variables and their interactions. Finding the best model is not easy.

In general, the best practice is to obtain a final model that is as parsimonious as possible, that is, with as few parameters as possible. A parsimonious model is easier to interpret and generalize, and also more powerful from a statistical point of view. On the other hand, it should not be too simple so that it still captures the variations or patterns in the data. In general, while more variables are often better than one, too many is often worse than a few.

The two most common approaches to obtain a final model are the following:

Adjust the model by removing the main effects and their interactions which are not significant with respect to their $p$-values, obtained by testing the nullity of the corresponding coefficients using a statistical test such as the likelihood ratio test or the Wald test. If there are several main effects or interactions which are not significant, interactions must be removed before removing any main effect. Moreover, it is recommended to remove interactions and independent variables one by one (starting with the one with the highest $p$-value), as removing a variable or an interaction may make another variable or interaction that was initially non-significant significant.
Adjust the model by using AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion). These procedures allow to select the best model (according to AIC or BIC) by finding an equilibrium between simplicity and complexity. These selection processes usually lead to a final model with as few parameters as possible, but which captures as much information in the data as possible. Note that only models with the same dependent variable can be compared using AIC or BIC. Models with different dependent variables cannot be compared using these criteria.

The first method requires that:

the underlying assumptions are valid,
the sample size is sufficiently large, and
the models are nested (i.e., the complete model includes at least all the variables included in the reduced model).

Moreover, the second method can be used with widely used criteria when selecting variables, and more importantly, in a completely autonomous way in R.

For this reason, the second method is recommended and more often used in practice.

This second method, referred as the stepwise selection, is divided into 3 types:

backward selection: we start from the most complete model (containing all independent variables and usually also their interactions), and the interactions/main effects are deleted at each step until the model cannot be improved,
forward selection: we start from the most basic model containing only the intercept, and the independent variables/interactions are added at each step until the model cannot be improved, or
mixed selection: we apply both the backward and forward selection to determine the best model according to the desired criterion.

We show how to select the best model according to AIC using the mixed stepwise selection, illustrated with all variables present in the data frame as independent variables and all possible second order interactions:

# save initial model
m5 <- glm(heart_disease ~ (age + sex + chest_pain + max_heartrate)^2,
  data = dat,
  family = "binomial"
)

# select best model according to AIC using mixed selection
m5_final <- step(m5,
  direction = "both", # both = mixed selection
  trace = FALSE # do not display intermediate steps
)

# display results of final model
summary(m5_final)

## 
## Call:
## glm(formula = heart_disease ~ age + sex + chest_pain + max_heartrate + 
##     age:max_heartrate, family = "binomial", data = dat)
## 
## Coefficients:
##                              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                19.4386591  8.1904201   2.373 0.017628 *  
## age                        -0.3050017  0.1414986  -2.156 0.031122 *  
## sexmale                     1.7055353  0.3507149   4.863 1.16e-06 ***
## chest_painatypical angina  -0.0086463  0.6547573  -0.013 0.989464    
## chest_painnon-anginal pain -0.0590333  0.5844000  -0.101 0.919538    
## chest_painasymptomatic      1.9724490  0.5649516   3.491 0.000481 ***
## max_heartrate              -0.1605658  0.0540933  -2.968 0.002994 ** 
## age:max_heartrate           0.0023314  0.0009433   2.471 0.013457 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 409.95  on 296  degrees of freedom
## Residual deviance: 268.60  on 289  degrees of freedom
## AIC: 284.6
## 
## Number of Fisher Scoring iterations: 5

According to the AIC (which is the default criterion when using the step() function), the best model is the one including:

age,
sex,
chest_pain,
max_heartrate, and
the interaction between age and max_heartrate.

For your information, you can also easily compare models manually using AIC or the pseudo-$R^2$ with the tab_model() function, also available in the {sjPlot} R package:

tab_model(m3, m4, m5_final,
  show.ci = FALSE, # remove CI
  show.aic = TRUE, # display AIC
  p.style = "numeric_stars" # display p-values and stars
)

	heart disease	heart disease	heart disease
Predictors	Odds Ratios	p	Odds Ratios	p	Odds Ratios	p
(Intercept)	0.94	0.976	0.01 ^***	<0.001	276759408.51 ^*	0.018
age	1.04 ^*	0.024	1.07 ^***	<0.001	0.74 ^*	0.031
sex [male]	5.40 ^***	<0.001	4.47 ^***	<0.001	5.50 ^***	<0.001
chest pain [atypical angina]	0.89	0.851			0.99	0.989
chest pain [non-anginal pain]	0.88	0.828			0.94	0.920
chest pain [asymptomatic]	7.13 ^***	<0.001			7.19 ^***	<0.001
max heartrate	0.97 ^***	<0.001			0.85 ^**	0.003
age × max heartrate					1.00 ^*	0.013
Observations	297	297	297
R² Tjur	0.393	0.142	0.409
AIC	289.263	370.435	284.599
* p<0.05 p<0.01 * p<0.001

Pseudo-$R^2$ is a generalization of the coefficient of determination $R^2$ often used in linear regression to judge the quality of a model. Like the $R^2$ in linear regression, the pseudo-$R^2$ varies from 0 to 1, and can be interpreted as the percentage of the null deviance explained by the independent variable(s). The higher the pseudo-$R^2$ and the lower the AIC, the better the model.

Note that there are several pseudo-$R^2$, such as:

Likelihood ratio $R^2_{L}$,
Cox and Snell $R^2_{CS}$,
Nagelkerke $R^2_{N}$,
McFadden $R^2_{McF}$, and
Tjur $R^2_{T}$.

The tab_model() function gives the Tjur $R^2_{T}$ by default.

Based on the AIC and the Tjur $R^2_{T}$, the last model is considered as the best one among the 3 considered.

Note that, even though a model is deemed the best one among the ones you have considered (based on one or several criteria), it does not necessarily mean that it fits the data well. There are several methods to check the quality of a model and to check if it is appropriate for the data at hand. This is the topic of the next section.

Quality of a model

Usually, the goal of building a model is to be able to predict, as precisely as possible, the response variable for new data.

In the next sections, we present some measures to judge the quality of a model, starting with the easiest and most intuitive one, followed by two widely used in the medical domain, and finally two other metrics common in the field of machine learning.

Validity of the predictions

Accuracy

A good way to judge the accuracy of a model is to monitor its performance on new data and count how often it predicts the correct outcome.

Unfortunately, when we have access to new data, we often do not know the real outcome and we cannot therefore check if the model does a good job in predicting the outcome. The trick is to:

train the model on the initial data frame,
test the model on the exact same data (just like if it was a complete different data frame for which we do not know the outcome), and then
compare the predictions made by the model to the real outcomes.

To illustrate this process, we take the model built in the previous section and test it on the initial data frame.

Moreover, suppose that if the probability for the patient to develop a heart disease is below 50%, we consider that the predicted outcome is the absence of the disease, otherwise the predicted outcome is the presence of the disease.

# create a vector of predicted probabilities
preds <- predict(m5_final,
  newdata = select(dat, -heart_disease), # remove real outcomes
  type = "response"
)

# if probability < threshold, patient is considered not to have the disease
preds_outcome <- ifelse(preds < 0.5,
  0,
  1
)

# transform predictions into factor and set labels
preds_outcome <- factor(preds_outcome,
  levels = c(0, 1),
  labels = c("no disease", "disease")
)

# compare observed vs. predicted outcome
tab <- table(dat$heart_disease, preds_outcome,
  dnn = c("observed", "predicted")
)

# print results
tab

##             predicted
## observed     no disease disease
##   no disease        132      28
##   disease            33     104

From the contingency table of the predicted and observed outcomes, we see that the model:

correctly predicted the absence of the disease for 132 patients,
incorrectly predicted the presence of the disease for 28 patients,
incorrectly predicted the absence of the disease for 33 patients, and
correctly predicted the presence of the disease for 104 patients.

The percentage of correct predictions, referred as the accuracy, is the sum of the correct predictions divided by the total number of predictions:

accuracy <- sum(diag(tab)) / sum(tab)
accuracy

## [1] 0.7946128

This model has an accuracy of 79.5%.

Although accuracy is the most intuitive and easiest way to measure a model’s predictive performance, it has some drawbacks, notably because we have to choose an arbitrary threshold beyond which we classify a new observation as 1 or 0. A more detailed discussion about this can be found on Frank Harrell’s blog.

In this illustration, we chose 50% as the threshold beyond which a patient was considered as having the disease. Nonetheless, we could have chosen another threshold and the results would have been different!

Sensitivity and specificity

If you work in the medical field, or if your research is related to medical sciences, you have probably already heard about sensitivity and specificity.

The sensitivity of a classifier, also referred as the recall, measures the ability of a classifier to detect the condition when the condition is present. In our case, it is the percentage of diseased people who are correctly identified as having the disease. Formally, we have:

\[ Sensitivity = \frac{\text{True positives}}{\text{True positives} + \text{False negatives}},\]

where true positives are people correctly diagnosed as ill and false negatives are people incorrectly diagnosed as healthy.

The specificity of a classifier measures the ability of a classifier to correctly exclude the condition when the condition is absent. In our case, it is the percentage of healthy people who are correctly identified as not having the disease. Formally, we have:

\[Specificity = \frac{\text{True negatives}}{\text{True negatives} + \text{False positives}},\]

where true negatives are people correctly diagnosed as healthy and false positives are people incorrectly diagnosed as ill.

In R, sensitivity and specificity can be computed as follows:

# sensitivity
sensitivity <- tab[2, 2] / (tab[2, 2] + tab[2, 1])
sensitivity

## [1] 0.7591241

# specificity
specificity <- tab[1, 1] / (tab[1, 1] + tab[1, 2])
specificity

## [1] 0.825

With our model, we obtain:

sensitivity = 75.9%, and
specificity = 82.5%.

The closer the sensitivity and the specificity are to 100%, the better the model.

AUC and ROC curve

We have already seen that the better the quality of the model, the better the predictions.

Another common and less arbitrary way to judge the quality of a model is by computing the AUC (Area Under the Curve) and plotting the ROC (Receiver Operating Characteristic) curve.

This can be achieved easily thanks to the {pROC} package:

# load package
library(pROC)

# save roc object
res <- roc(heart_disease ~ fitted(m5_final),
  data = dat
)

# plot ROC curve
ggroc(res, legacy.axes = TRUE)

# print AUC
res$auc

## Area under the curve: 0.87

As the ggroc() function works with layers from the {ggplot2} package, we can print the AUC directly in the title of the plot of the ROC curve:

# plot ROC curve with AUC in title
ggroc(res, legacy.axes = TRUE) +
  labs(title = paste0("AUC = ", round(res$auc, 2)))

These two quality metrics can be interpreted as follows:

in the plot, the closer the ROC curve is to the upper left-hand corner, the better the model, and
the closer the AUC is to 1, the better the model.

Based on the ROC curve and the AUC, we can say that this model is good to very good. This means that the model is appropriate for these data, and that it can be useful to predict whether or not a patient will develop a heart disease!

Reporting results

As we have seen before, odds ratios are useful when reporting results of binary logistic regressions.

Computing these odds ratios together with the confidence intervals is not particularly difficult. However, presenting them in a table for a publication or a report can quickly become time consuming, in particular if you have many models and many independent variables.

Luckily, there are two packages which saved me a lot of time and which I use almost every time I need to report results of a logistic regression.

The first package, called {gtsummary} is useful to report results of one regression at a time. The second one is the {finalfit} package.¹ This packages is more appropriate if you need to report results of several regressions at a time.

{gtsummary} package

Here is an example with one of the models built previously:

# load package
library(gtsummary)

# print table of results
tbl_regression(m5_final, exponentiate = TRUE)

Characteristic	OR¹	95% CI¹	p-value
age	0.74	0.55, 0.96	0.031
sex
female	—	—
male	5.50	2.82, 11.2	<0.001
chest_pain
typical angina	—	—
atypical angina	0.99	0.27, 3.65	>0.9
non-anginal pain	0.94	0.30, 3.07	>0.9
asymptomatic	7.19	2.44, 22.8	<0.001
max_heartrate	0.85	0.76, 0.94	0.003
age * max_heartrate	1.00	1.00, 1.00	0.013
¹ OR = Odds Ratio, CI = Confidence Interval

What I like with this package is its ease of use, and the fact that all results are nicely formatted in a table. This is a very good starting point when I need to create a table for a publication or a report, for one regression at a time.

The second package becomes interesting when you need to report results for several models at once.

{finalfit} package

The {finalfit} package allows to report odds ratios, their confidence intervals and the $p$-values in a very efficient way. Moreover, it is quite easy to do so for many regressions at the same time.

Let me present the package by reporting results of:

all univariable binary logistic regressions that are possible with the variables available in the data frame,
a multivariable binary logistic regression that includes all variables available in the data frame, and
a multivariable binary logistic regression that includes only some of the variables present in the data frame.

We start with all the possible univariable binary logistic regressions:

# load packages
library(tidyverse)
library(gt)
library(finalfit)

# set dependent and independent variables
dependent <- "heart_disease"
independent <- c("age", "sex", "chest_pain", "max_heartrate")

# save results of univariable logistic regressions
glmuni <- dat |>
  glmuni(dependent, independent) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Crude OR",
    estimate_suffix = " (95% CI)"
  )

# print results
glmuni |>
  gt()

Variables	Crude OR (95% CI)
age	1.05 (1.03-1.08, p<0.001)
sexmale	3.57 (2.12-6.18, p<0.001)
chest_painatypical angina	0.51 (0.16-1.66, p=0.255)
chest_painnon-anginal pain	0.63 (0.23-1.86, p=0.384)
chest_painasymptomatic	6.04 (2.39-16.76, p<0.001)
max_heartrate	0.96 (0.94-0.97, p<0.001)

A few remarks regarding this code:

glmuni() is used because we want to run univariable GLM.
explanatory_name = "Variables" is used to rename the first column (by default it is “explanatory”).
estimate_name = "Crude OR" is used to rename the second column and inform the reader that we are in the univariable case. In the univariable case, OR are often called crude OR because they are not adjusted for the effects of the other independent variables.
estimate_suffix = " (95% CI)" is used to specify that it is the 95% confidence intervals which are inside the parentheses.
The gt() layer at the end of the code is not compulsory. It is just to make the output appears in a nice table instead of the usual format of R outputs. See more information about the {gt} package in its documentation.

Here is how to report results of a multivariable binary logistic regression which includes all variables:

# save results of full model
glmmulti_full <- dat |>
  glmmulti(dependent, independent) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Adjusted OR - full model",
  )

# print results
glmmulti_full |>
  gt()

Variables	Adjusted OR - full model
age	1.04 (1.01-1.08, p=0.024)
sexmale	5.40 (2.78-10.97, p<0.001)
chest_painatypical angina	0.89 (0.25-3.19, p=0.851)
chest_painnon-anginal pain	0.88 (0.29-2.81, p=0.828)
chest_painasymptomatic	7.13 (2.51-22.03, p<0.001)
max_heartrate	0.97 (0.95-0.99, p<0.001)

A few remarks regarding this code:

glmmulti() is used because we want to run multivariable GLM.
estimate_name = "Adjusted OR - full model" is used to remind the reader that we are in the multivariable case with all variables included. In the multivariable case, OR are often called adjusted OR because they are adjusted for the effects of the other independent variables.

Here is how to report results of a multivariable binary logistic regression which includes only a selection of variables:

# select the variables to be included in the final model
independent_final <- c("age", "sex", "chest_pain")

# save results of final model
glmmulti_final <- dat |>
  glmmulti(dependent, independent_final) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Adjusted OR - final model",
    estimate_suffix = " (95% CI)"
  )

# print results
glmmulti_final |>
  gt()

Variables	Adjusted OR - final model (95% CI)
age	1.07 (1.03-1.11, p<0.001)
sexmale	5.52 (2.88-11.07, p<0.001)
chest_painatypical angina	0.82 (0.24-2.82, p=0.743)
chest_painnon-anginal pain	0.93 (0.32-2.90, p=0.903)
chest_painasymptomatic	9.48 (3.47-28.53, p<0.001)

Note that you have to manually select the variables (the package will not choose for you unfortunately). The selection could be done preliminary thanks to a stepwise procedure for example.

Now the most interesting part of this package:

we can combine all these results together,
in addition to some descriptive statistics for each level of the dependent variable!

Here are all results combined together and displayed in a table:

# save descriptive statistics
summary <- dat |>
  summary_factorlist(dependent, independent, fit_id = TRUE)

# save results of regressions
output <- summary |>
  finalfit_merge(glmuni) |>
  finalfit_merge(glmmulti_full) |>
  finalfit_merge(glmmulti_final)

# print all results
output |>
  dplyr::select(-fit_id, -index) |>
  dplyr::rename(
    Variables = label,
    " " = levels
  ) |>
  gt()

Variables		no disease	disease	Crude OR (95% CI)	Adjusted OR - full model	Adjusted OR - final model (95% CI)
age	Mean (SD)	52.6 (9.6)	56.8 (7.9)	1.05 (1.03-1.08, p<0.001)	1.04 (1.01-1.08, p=0.024)	1.07 (1.03-1.11, p<0.001)
sex	female	71 (44.4)	25 (18.2)	-	-	-
	male	89 (55.6)	112 (81.8)	3.57 (2.12-6.18, p<0.001)	5.40 (2.78-10.97, p<0.001)	5.52 (2.88-11.07, p<0.001)
chest_pain	typical angina	16 (10.0)	7 (5.1)	-	-	-
	atypical angina	40 (25.0)	9 (6.6)	0.51 (0.16-1.66, p=0.255)	0.89 (0.25-3.19, p=0.851)	0.82 (0.24-2.82, p=0.743)
	non-anginal pain	65 (40.6)	18 (13.1)	0.63 (0.23-1.86, p=0.384)	0.88 (0.29-2.81, p=0.828)	0.93 (0.32-2.90, p=0.903)
	asymptomatic	39 (24.4)	103 (75.2)	6.04 (2.39-16.76, p<0.001)	7.13 (2.51-22.03, p<0.001)	9.48 (3.47-28.53, p<0.001)
max_heartrate	Mean (SD)	158.6 (19.0)	139.1 (22.7)	0.96 (0.94-0.97, p<0.001)	0.97 (0.95-0.99, p<0.001)	-

A few remarks regarding this code:

summary_factorlist(dependent, independent, fit_id = TRUE) is used to compute the descriptive statistics by group of the dependent variable.
finalfit_merge() is used to merge results together.
dplyr::select(-fit_id, -index) is used to remove unnecessary columns.
dplyr::rename(Variables = label, " " = levels) is used to renames some columns.

And finally, a few remarks regarding the resulting table:

The first column gives the name of the variables.
The second column specifies:
- for qualitative variables: the levels
- for quantitative variables: that it is the mean and the standard deviation (SD) which will be computed in the next two columns
The third and fourth columns give the descriptive statistics for each level of the dependent variable:
- for qualitative variables: the number of cases, and in parentheses the frequencies by column
- for quantitative variables: the mean, and in parentheses the standard deviation
The last three columns give the OR, and in parentheses the 95% CI of the OR and the $p$-value (for the univariable and the two multivariable models, respectively).

For your convenience, here is the full code so you can copy paste it easily in case you want to reproduce the process:

# load packages
library(tidyverse)
library(gt)
library(finalfit)

# set variables
dependent <- "heart_disease"
independent <- c("age", "sex", "chest_pain", "max_heartrate")
independent_final <- c("age", "sex", "chest_pain")

# save descriptive statistics
summary <- dat |>
  summary_factorlist(dependent, independent, fit_id = TRUE)

# save results of univariable logistic regressions
glmuni <- dat |>
  glmuni(dependent, independent) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Crude OR",
    estimate_suffix = " (95% CI)"
  )

# save results of full model
glmmulti_full <- dat |>
  glmmulti(dependent, independent) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Adjusted OR - full model",
  )

# save results of final model
glmmulti_final <- dat |>
  glmmulti(dependent, independent_final) |>
  fit2df(
    explanatory_name = "Variables",
    estimate_name = "Adjusted OR - final model",
    estimate_suffix = " (95% CI)"
  )

# save merged results
output <- summary |>
  finalfit_merge(glmuni) |>
  finalfit_merge(glmmulti_full) |>
  finalfit_merge(glmmulti_final)

# print all results
output |>
  dplyr::select(-fit_id, -index) |>
  dplyr::rename(
    Variables = label,
    " " = levels
  ) |>
  gt()

Last but not least, the or_plot() function, also available from the {finalfit} package, is useful to visualize all odds ratio and their 95% confidence intervals:

dat |> or_plot(dependent, independent,
  table_text_size = 3.5 # reduce text size
)

Here is how to read this plot:

The squares represent the OR, and the whiskers their 95% CI.
When the 95% CI crosses the vertical dashed line, it means that the OR is not significantly different from 1 (at the 5% significance level). In these cases, we cannot reject the hypothesis of no association with the dependent variable.
When the 95% CI does not cross the vertical dashed line, it means that the OR is significantly different from 1. In these cases:
- if the square is located to the right of the vertical dashed line, there is a positive relationship between the outcome and the independent variable (known as a risk factor), and
- if the square is located to the left of the vertical dashed line, there is a negative relationship between the outcome and the independent variable (known as a protective factor).

The plot confirms what was obtained above:

age is a risk factor for heart disease,
maximum heart rate achieved is a protective factor for heart disease, and
being a male and suffering from asymptomatic chest pain are both risk factors of heart disease.

Be careful that sometimes the square is too big to see the whiskers of the 95% CI. This is the case for the variables max_heartrate and age. In these cases, it is better to check the significance of the OR thanks to their 95% CI or the $p$-values printed in parentheses.

Conditions of application

For results to be valid and interpretable, a binary logistic regression requires:

the dependent variable to be binary,
independence of the observations: no repeated measurements or matched data, otherwise generalize linear mixed effect models (GLMM) should be used,
linearity of continuous independent variables and the log-odds outcome: take age and heart disease as an example. If heart disease is more frequent or less frequent as age rises, the model will work well. However, if children and the elderly are at high risk of having a heart disease, but those in middle years are not, then the relationship is not linear, or not monotonic, meaning that the response does not only go in one direction,
a sufficiently large sample size (for confidence intervals and hypothesis tests to be valid), and
no multicollinearity: independent variables should not be highly correlated with each other, otherwise coefficients and OR can become unstable.

Here is how to verify each of them:

this is obvious; check if the dependent variable has indeed only two levels,
this is often not tested formally, but verified through the design of the experiment,
quantitative independent variables should have a linear relationship between their log-odds and their observed values. A visual check is sufficient, see below with age, maximum heart rate achieved and model m3 as example:

# linearity to the log-odds?
dat |>
  dplyr::select(age, max_heartrate) |>
  mutate(log_odds = predict(m3)) |>
  pivot_longer(-log_odds) |>
  ggplot(aes(log_odds, value)) +
  geom_point() +
  geom_smooth(method = "lm") +
  facet_wrap(~name)

in practice, it is recommended to have at least 10 times as many events as parameters in the model, and
the variance inflation factors (VIF) is a well known measure of multicollinearity. It should be below 10 or 5, depending on the field of research. VIF can be computed with the vif() function, available in the {car} package:

# load package
library(car)

# compute VIF for model m3
vif(m3)

##                   GVIF Df GVIF^(1/(2*Df))
## age           1.205246  1        1.097837
## sex           1.155071  1        1.074742
## chest_pain    1.113010  3        1.018005
## max_heartrate 1.143125  1        1.069170

Conclusion

Thanks for reading.

In this relatively long and detailed post, we covered several important points about binary logistic regression. First, when to use such models and what is the difference with linear models, how to implement it in R, and how to interpret and report results. We ended by discussing about model selection, how to judge the quality of fit of a logistic regression, and its underlying assumptions.

I now hope that (univariable and multivariable) binary logistic regressions in R no longer hold any secrets for you.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Thanks to Claire from DellaData.fr for introducing me to this package.↩︎

Pearson, Spearman and Kendall correlation coefficients by hand

Tue, 05 Sep 2023 00:00:00 +0000

Introduction

In statistics, a correlation is used to evaluate the relationship between two variables.

In a previous post, we showed how to compute a correlation and perform a correlation test in R. In this post, we illustrate how to compute the Pearson, Spearman and Kendall correlation coefficients by hand and under two different scenarios (i.e., with and without ties).

Data

To illustrate the methods with and without ties, we consider two different datasets, one with ties and another without ties.

With ties

For the illustrations of the scenarios with ties, suppose we have the following sample of size 5:

As we can see, there are some ties since there are two identical observations in the variable $x$.

Without ties

For the scenarios which require no ties, we will consider the following sample of size 3:

Correlation coefficients by hand

The three most common correlation methods are:¹

Pearson, used for two quantitative continuous variables which have a linear relationship
Spearman, used for two quantitative variables if the link is partially linear, or for one qualitative ordinal variable and one quantitative variable
Kendall, often used for two qualitative ordinal variables

Each method is presented in the next sections.

Note that the aim of this post is to illustrate how to compute the three correlation coefficients by hand and under two different scenarios; we are not interested in verifying the underlying assumptions.

Pearson

Luckily, the procedure for computing the Pearson correlation coefficient is the same whether there are ties or not so we do not distinguish the two scenarios.

With and without ties

The Pearson correlation coefficient, denoted $r$ in the case of a sample, can be computed as follows²

\[ r = \frac{\sum^n_{i = 1} x_i y_i - n \bar{x} \bar{y}}{(n - 1) s_x s_y} \]

where:

$n$ is the sample size
$x_i$ and $y_i$ are the observations
$\bar{x}$ and $\bar{y}$ are the sample means of $x$ and $y$
$s_x$ and $s_y$ are the sample standard deviations of $x$ and $y$

We show how to compute it step by step.

Step 1.

As you can see, before computing the correlation coefficient we first need to compute the mean and the standard deviation for each of the two variables.

The mean of $x$ is computed as follows:

\[\bar{x} = \frac{1}{n}\sum^n_{i = 1} x_i\]

The standard deviation of $x$ is computed as follows:

\[s_x = \sqrt{\frac{1}{n - 1} \sum^n_{i = 1}(x_i - \bar{x})^2}\]

The formulas can be used analogously for the variable $y$.

We start by computing the means of the two variables:

$\bar{x} = \frac{-1 + 3 + 5 + 5 + 2}{5} = 2.8$
$\bar{y} = \frac{-3 + 1 + 0 + 2 - 1}{5} = -0.2$

Step 2.

We now need to compute the standard deviation of each variable. To ease the computations, it is best to use a table, starting with the observations:

To compute the standard deviations, we need the sum of the squared differences between each observation and its mean, that is, $\sum^n_{i = 1}(x_i - \bar{x})^2$ and $\sum^n_{i = 1}(y_i - \bar{y})^2$. We start by creating two new columns in the table, denoted x-xbar and y-ybar, corresponding to $(x_i - \bar{x})$ and $(y_i - \bar{y})$:

We take the square of these two new columns to have $(x_i - \bar{x})^2$ and $(y_i - \bar{y})^2$, denoted (x-xbar)^2 and (y-ybar)^2:

We then sum these two columns, which gives:

for $x$: $\sum^n_{i = 1}(x_i - \bar{x})^2 =$ 24.8
for $y$: $\sum^n_{i = 1}(y_i - \bar{y})^2 =$ 14.8

The standard deviations are thus:

$s_x = \sqrt{\frac{24.8}{5-1}} = 2.49$
$s_y = \sqrt{\frac{14.8}{5-1}} = 1.92$

Step 3.

We now need $\sum^n_{i = 1} x_i y_i$, so we add a new column in the table, corresponding to $x_iy_i$ and denoted x*y:

Step 4.

We take the sum of that last column, which gives $\sum^n_{i = 1} x_i y_i =$ 14.

Step 5.

Finally, the Pearson correlation coefficient can be computed by plugging values found above in the initial formula:

\[ \begin{split} r &= \frac{\sum^n_{i = 1} x_i y_i - n \bar{x} \bar{y}}{(n - 1) s_x s_y} \\ &= \frac{14 - (5\times2.8\times-0.2)}{(5-1) \times 2.49\times1.92} \\ &= 0.88 \end{split} \]

Note that there are other formulas to compute the Pearson correlation coefficient. For instance,

\[ \begin{split} r &= \frac{1}{n - 1} \sum^n_{i = 1} \left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{x}}{s_y}\right) \\ &= \frac{\sum^n_{i = 1}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum^n_{i = 1}(x_i - \bar{x})^2} \sqrt{\sum^n_{i = 1}(y_i - \bar{y})^2}} \end{split} \]

All formulas will of course give the exact same results.

For your information, squaring the Pearson correlation coefficient $r$ gives the coefficient of determination $R^2$ in the context of a simple linear regression.

Spearman

We now present the Spearman correlation coefficient, also referred as Spearman’s rank correlation coefficient. This coefficient is actually the same than Pearson coefficient, except that the computations are based on the ranked values rather than on the raw observations.

Again, we present how to compute it by hand step by step, but this time we distinguish two scenarios:

if there are ties
if there are no ties

With ties

The Spearman correlation coefficient (with ties), denoted $r_s$, is defined as follows

\[ r_s = \frac{\sum^n_{i = 1} Rx_i Ry_i - n \overline{Rx} \overline{Ry}}{(n - 1) s_{Rx} s_{Ry}} \]

where:

$n$ is the sample size
$Rx_i$ and $Ry_i$ are the ranks for the two variables
$\overline{Rx}$ and $\overline{Ry}$ are the sample means of the ranks for the two variables
$s_{Rx}$ and $s_{Ry}$ are the sample standard deviations of the ranks for the two variables

Here is how to compute it by hand.

Step 1.

As mentioned before, the Spearman coefficient is based on the ranks. So we first need to add the ranks of the observations (from lowest to highest), separately for each of the two variables.

For $y$, we see that:

-3 is the smallest value, so we assign it the rank 1
-1 is the second smallest value, so we assign it the rank 2
then comes 0, so we assign it the rank 3
then comes 1, so we assign it the rank 4
finally, 2 is the largest value, so we assign it the rank 5

The same goes for $x$, except that here we have two observations with the value 5 (so there will be ties in the ranks). In this case, we take the mean rank:

-1 is the smallest value, so we assign it the rank 1
2 is the second smallest value, so we assign it the rank 2
then comes 3, so we assign it the rank 3
finally, the two largest values belongs to rank 4 and 5, so we assign both of them the rank 4.5

We include the ranks in the table, denoted Rx and Ry:

Step 2.

From there, it is similar than the Pearson coefficient except that we work on the ranks and not on the initial observations anymore. To avoid any confusion in the remaining steps, we remove the initial observations from the table and we keep only the ranks:

We start with the means of the ranks:

$\overline{Rx} = \frac{1+3+4.5+4.5+2}{5} = 3$
$\overline{Ry} = \frac{1+4+3+5+2}{5} = 3$

For the standard deviations, we use the table as we did for the Pearson coefficient:

We sum the last two columns, which gives:

for $x$: $\sum^n_{i = 1}(Rx_i - \overline{Rx})^2 =$ 9.5
for $y$: $\sum^n_{i = 1}(Ry_i - \overline{Ry})^2 =$ 10

The standard deviations are thus:

$s_{Rx} = \sqrt{\frac{9.5}{5-1}} = 1.54$
$s_{Ry} = \sqrt{\frac{10}{5-1}} = 1.58$

Step 3.

We now need $\sum^n_{i = 1} Rx_i Ry_i$, so we add a new column in the table, corresponding to $Rx_iRy_i$ and denoted Rx*Ry:

Step 4.

We take the sum of that last column, which gives $\sum^n_{i = 1} Rx_i Ry_i =$ 53.

Step 5.

Finally, the Spearman correlation coefficient can be computed by plugging all values in the initial formula:

\[ \begin{split} r_s &= \frac{\sum^n_{i = 1} Rx_i Ry_i - n \overline{Rx} \overline{Ry}}{(n - 1) s_{Rx} s_{Ry}} \\ &= \frac{53 - (5\times3\times3)}{(5-1) \times 1.54\times1.58} \\ &= 0.82 \end{split} \]

Without ties

When all initial values are different within each variable, meaning that all ranks are distinct integers, there are no ties. In that specific case, the Spearman coefficient can be computed with the following shortened formula:

\[ r_s = 1 - \frac{6 \sum^n_{i = 1}(Rx_i - Ry_i)^2}{n (n^2 - 1)} \]

For example, suppose the sample without ties introduced at the beginning of the post:

Step 1.

All observations within each variable are different, so all ranks are distinct integers and there are no ties:

Step 2.

We only need to compute the difference between the two ranks of each row, denoted Rx-Ry:

Take the square of these differences, denoted (Rx-Ry)^2:

And then take the sum of that last column, which gives $\sum^n_{i = 1}(Rx_i - Ry_i)^2 =$ 2.

Step 3.

Finally, we can fill in the initial formula to find the Spearman coefficient:

\[ \begin{split} r_s &= 1 - \frac{6 \sum^n_{i = 1}(Rx_i - Ry_i)^2}{n (n^2 - 1)} \\ &= 1 - \frac{6 \times2}{3 (3^2 - 1)} \\ &= 1 - \frac{12}{24} \\ &= 0.5 \end{split} \]

Kendall

Kendall coefficient correlation, also known as Kendall’s $\tau$ coefficient, is similar than Spearman coefficient, except that it is often preferred for small samples and when many rank ties.

Here also we distinguish between two scenarios:

if there are no ties
if there are ties

Unlike Spearman coefficient, we first illustrate the scenario when there are no ties, and then when there are ties.

Without ties

When there are no ties, the Kendall coefficient, denoted $\tau_a$, is defined as follows

\[ \tau_a = \frac{C - D}{C + D} \]

where:

$C$ is the number of concordant pairs
$D$ is the number of discordant pairs

This coefficient, also referred as Kendall tau-a, does not make any adjustment for ties.

Let’s see what are concordant and discordant pairs using the data without ties (the same data than for Spearman correlation without ties):

Step 1.

We start by computing the ranks for each variable (Rx and Ry), as we did for the Spearman coefficient:

Step 2.

We then arbitrarily choose a reference variable among the two, Rx or Ry. Suppose we take Rx as the reference variable here.

We sort the dataset by this reference variable:

From now, we only look at the ranks of the second variable (the one which is not the reference level, here Ry), so to avoid any confusion in the remaining steps we keep only the Ry column:

Step 3.

We now take each row of Ry one by one and check whether the rows below it in the table are smaller or larger.

In our table, the first row of Ry is 1. We see that the value just below it is 3, which is larger than 1. Since 3 is larger than 1, this is called a concordant pair. We write it in the table:

The next row, 2, is also larger than 1, so it is also a concordant pair. We also write it in the table:

We start again with the second row of Ry, which is 3. Again, we look at the row below it and check whether it is larger or smaller. Here, the row below it is 2, which is smaller than 3, so we have a discordant pair. We add this information into the table:

Step 4.

We now compute the total number of concordant and discordant pairs. There are:

2 concordant pairs so $C = 2$, and
1 discordant pair so $D = 1$.

Step 5.

Finally, we plug the values we have just found in the initial formula:

\[ \begin{split} \tau_a &= \frac{C - D}{C + D}\\ &= \frac{2 - 1}{2 + 1}\\ &= \frac{1}{3}\\ &= 0.33 \end{split} \]

Alternatively, we can also use the following formulas

\[ \begin{split} \tau_a &= 1 - \frac{4D}{n(n - 1)}\\ &= \frac{4C}{n(n - 1)} - 1 \end{split} \]

where $n$ is still the sample size, and which both give the exact same results.

With ties

I must admit that the process with ties is slightly more complex than without ties.

The Kendall tau-b coefficient, which makes adjustments for ties, is defined as follows

\[ \tau_b = \frac{C - D}{\sqrt{(C^2_n - n_x)(C^2_n - n_y)}} \]

where:

$C$ and $D$ are still the number of concordant and discordant pairs, respectively
$C^2_n$ is the total number of possible pairs
$n_x$ is the number of possible pairs with a tie among the $x$ values
$n_y$ is the number of possible pairs with a tie among the $y$ values

Note that here, the letter $C$ in $C^2_n$ denotes “combination” and not “concordant”.

Let’s illustrate that scenario and the formula with the dataset used for the Pearson correlation and the Spearman correlation with ties, that is:

Step 1.

Similarly to with no ties, the process with ties is based on the ranks so we start by adding the ranks for each variable, denoted as usual as Rx and Ry:

Step 2.

One of the two variables ($x$ or $y$) must be selected as the reference variable. This time, we do not choose it arbitrarily, but we choose the one which does not have any ties. In our case, variable $x$ has ties, while variable $y$ does not have any ties. So $y$ will be our reference variable.

We then:

order the dataset by the reference variable, here Ry, and
we keep only the necessary columns to avoid any confusion in the remaining steps:

Step 3.

We check whether it is a concordant or discordant pair the same way we did without ties, but this time we do not count ties.

The first row in column Rx is 1. We compare all rows below it in the table with that value. All rows below 1 in the table are larger, so we write that they are concordant pairs:

We repeat the same process for each row.

For instance, for the second row of the column Rx, we have the value 2. Again, all rows below in the table are larger so we write that they are concordant pairs:

For the third row of the column Rx, we have the value 4.5. The row just below in the table (= 3) is smaller than 4, so we write D for discordant pair:

Now, as you can see, the last row is also 4.5. Therefore, since it is equal to the value we are comparing to, it is neither a concordant nor a discordant pair, so we write “T” in the table for “ties”:

Last, the fourth row in the column Rx is 3, which we compare to 4.5 to see that 4.5 is larger so it is a concordant pair:

Step 4.

We then sum the number of concordant and discordant pairs:

$C = 8$
$D = 1$

Step 5.

We now have all the information required to compute the numerator of $\tau_b$, but we still need to find $C^2_n$, $n_x$ and $n_y$ to compute the denominator.

As mentioned above, $C^2_n$ is the total number of possible pairs, thus corresponding to the number of combinations of two values. This number of pairs can be found with the formula of the combination, that is, $C^2_n = \frac{n!}{2!(n - 2)!}$ where $n$ is the sample size.

In our case, we have a sample of size 5, so we have:³

\[ \begin{split} C^2_n &= \frac{n!}{2!(n - 2)!}\\ &= \frac{5!}{2!(5-2)!}\\ &= 10 \end{split} \]

Furthermore, $n_x$ and $n_y$ are the number of possible pairs with a tie among the $x$ and $y$ variables, respectively.

When looking at Ry and Rx from the table above:

We see that there are:

2 identical values for the variable $x$
0 identical value for the variable $y$

This means that the number of possible pairs with a tie is equal to:⁴

1 for the variable $x$ (the only possible pair with a tie is the pair {4.5, 4.5})
0 for the variable $y$ (since all ranks are distinct, there exists no pair with a tie)

Therefore,

\[ \begin{split} n_x &= 1\\ n_y &= 0 \end{split} \]

And we now have all the necessary information to compute $\tau_b$!

Step 6.

By plugging the values found above in the initial formula, we have

\[ \begin{split} \tau_b &= \frac{C - D}{\sqrt{(C^2_n - n_x)(C^2_n - n_y)}}\\ &= \frac{8-1}{\sqrt{(10 - 1)(10 - 0)}}\\ &= \frac{7}{\sqrt{90}}\\ &= 0.74 \end{split} \]

Verification in R

For the sake of completeness, we verify our results with the help of R for each coefficient and scenario.

Pearson:

x <- c(-1, 3, 5, 5, 2)
y <- c(-3, 1, 0, 2, -1)

cor(x, y, method = "pearson")

## [1] 0.8769051

Spearman with ties:

cor(x, y, method = "spearman")

## [1] 0.8207827

Spearman without ties:

x2 <- c(3, 5, 2)
y2 <- c(5, 1, -1)

cor(x2, y2, method = "spearman")

## [1] 0.5

Kendall without ties:

cor(x2, y2, method = "kendall")

## [1] 0.3333333

Kendall with ties:

cor(x, y, method = "kendall")

## [1] 0.7378648

We indeed find the same results by hand than with R (any discrepancies is due to rounding)!

Conclusion

Remember that a correlation coefficient (no matter whether it is Pearson, Spearman or Kendall) ranges from -1 to 1, with 0 being no correlation and the closer to 1 in absolute terms, the stronger the correlation.

Broadly speaking, a positive correlation means that high values of one variable are associated with high values of the other variable (and vice versa). A negative correlation means that high values of one variable are associated with low values of the other variable.

Last but not least, remember that the conclusions drawn from a correlation coefficient computed within a sample cannot be generalized to the population without a proper statistical test, that is, a correlation test.

Thanks for reading.

I hope this article helped you to compute the Pearson, Spearman and Kendall correlation coefficients by hand (with and without ties).

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

For your information, the Pearson correlation coefficient is considered as a parametric procedure, whereas the Spearman and Kendall correlation coefficients are considered as non-parametric procedures.↩︎
Here, we suppose that we have a sample and not a population. In the case where the observations you have represent the entire population, the correlation coefficient is denoted $\rho$ and the formula differs slightly. See a recap of the difference between a sample and a population if needed.↩︎
Remember that the factorial of $n$, denoted $n!$, is $n! = n \times (n - 1) \times \cdots \times 1$. So for instance, $5! = 5\times4\times3\times2\times1 = 120$.↩︎
If you have many identical values and want to count the number of possible pairs thanks to a formula rather than by counting them manually, you can again use the formula of the combination, that is, $C^2_n = \frac{n!}{x!(n - x)!}$ where $n$ is the number of identical values in a given group of ties. Bear in mind that the number of combinations must be summed up for all group of ties. In our case, we have only one group of ties with 2 values, so $n_x = C^2_2 = \frac{2!}{2!(2 - 2)!} = 1$.↩︎

How to: one-way ANOVA by hand

Wed, 30 Aug 2023 00:00:00 +0000

Introduction

An ANOVA is a statistical test used to compare a quantitative variable between groups, to determine if there is a statistically significant difference between several population means. In practice, it is usually used to compare three or more groups. However, in theory, it can also be done with only two groups.¹

In a previous post, we showed how to perform a one-way ANOVA in R. In this post, we illustrate how to conduct a one-way ANOVA by hand, via what is usually called an “ANOVA table”.

Data and hypotheses

To illustrate the method, suppose we take a sample of 12 students, divided equally into three classes (A, B and C) and we observe their age. Here is the sample:

We are interested in comparing the population means between classes.

Remember that the null hypothesis of the ANOVA is that all means are equal (i.e., age is not significantly different between classes), whereas the alternative hypothesis is that at least one mean is different from the other two (i.e., age is significantly different in at least one class compared to the other two). Formally, we have:

$\mu_A = \mu_B = \mu_C$
at least one mean is different

ANOVA by hand

As mentioned above, we are going to do an ANOVA table to conclude the test.

Note that the ANOVA requires some assumptions (i.e., independence, equality of variances and normality). The aim of this post is to illustrate how to do an ANOVA by hand and not how to verify these assumptions, so we suppose they are met without any verification. See how to test these assumptions in R if you are interested.

Overall and group means

We first need to compute the mean age by class (referred as the group means):

class A: $\frac{24 + 31 + 26 + 23}{4} = 26$
class B: $\frac{24 + 21 + 19 + 24}{4} = 22$
class C: $\frac{15 + 21 + 18 + 18}{4} = 18$

and the mean age for the whole sample (referred as the overall mean):

\[\begin{equation} \begin{split} & \frac{24 + 31 + 26 + 23 + 24 + 21 + 19 }{12} \\ &\frac{+ 24 + 15 + 21 + 18 + 18}{12} = 22 \end{split} \end{equation} \]

SSR and SSE

We then need to compute the sum of squares regression (SSR), and the sum of squares error (SSE).

The SSR is computed by taking the square of the difference between the mean group and the overall mean, multiplied by the number of observations in the group:

and then taking the sum of all cells:

\[64+0+64 = 128 = SSR\]

The SSE is computed by taking the square of the difference between each observation and its group mean:

and then taking the sum of all cells:

\[\begin{equation} \begin{split} & 4+25+0+9+4+1+9+4 \\ & +9+9+0+0 = 74 = SSE \end{split} \end{equation} \]

For those interested in computing the sum of square total (SST), it is simply the sum of SSR and SSE, that is,

\[\begin{equation} \begin{split} SST &= SSR + SSE\\ &= 128 + 74 \\ & =202 \end{split} \end{equation} \]

ANOVA table

The ANOVA table looks as follows (we leave it empty and we are going to fill it in step by step):

We start to build the ANOVA table by plugging the SSR and SSE values found above into the table (in the “Sum.of.Sq.” column):

The “Df” column corresponds to the degrees of freedom, and is computed as follows:

for the line regression: number of groups - 1 = 3 - 1 = 2
for the line error: number of observations - number of groups = 12 - 3 = 9

With this information, the ANOVA table becomes:

The “Mean.Sq.” column corresponds to the Mean Square, and is equal to the sum of square divided by the degrees of freedom, so the “Sum.of.Sq.” column divided by the “Df” column:

Finally, the F-value corresponds to the ratio between the two mean squares, so $\frac{64}{8.222} = 7.78$:

This F-value gives the test statistic (also referred as $F_{obs}$), which needs to be compared with the critical value found in the Fisher table to conclude the test.

We find the critical value in the Fisher table based on the degrees of freedom (those used in the ANOVA table) and based on the significance level. Suppose we take a significance level $\alpha = 0.05$, the critical value can be found in the Fisher table as follows:

So we have

\[F_{2; 9; 0.05} = 4.26\]

If you are interested to find this value with R, it can be found with the qf() function, where 0.95 corresponds to $1 - \alpha$:

qf(0.95, 2, 9)

## [1] 4.256495

Conclusion of the test

The rejection rule says that, if:

$F_{obs} > F_{2; 9; 0.05} \Rightarrow$ we reject the null hypothesis
$F_{obs} \le F_{2; 9; 0.05} \Rightarrow$ we do not reject the null hypothesis

In our case,

\[F_{obs} = 7.78 > F_{2; 9; 0.05} = 4.26\]

$\Rightarrow$ We reject the null hypothesis that all means are equal. In other words, it means that at least one class is different than the other two in terms of age.²

To verify our results, here is the ANOVA table using R:

##             Df Sum Sq Mean Sq F value Pr(>F)  
## class        2    128   64.00   7.784 0.0109 *
## Residuals    9     74    8.22                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We found the same results by hand, but note that in R, the $p$-value is computed instead of comparing the $F_{obs}$ with the critical value. The $p$-value can easily be found in R based on the $F_{obs}$ and the degrees of freedom:

pf(7.78, 2, 9,
  lower.tail = FALSE
)

## [1] 0.010916

Conclusion

Thanks for reading.

I hope this article helped you to conduct a one-way ANOVA by hand. See this tutorial if you want to learn how to do it in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

In that case, a Student’s t-test is usually preferred over an ANOVA, although both tests will lead to the exact same conclusions.↩︎
Remember that an ANOVA cannot tell you which group is different than the other in terms of the quantitative dependent variable, nor whether they are all different or if only one is different. To answer this question, post-hoc tests are required. This is beyond the scope of the present post, but it can easily be done in R (see this tutorial).↩︎

Two-way ANOVA in R

Mon, 19 Jun 2023 00:00:00 +0000

Introduction

The two-way ANOVA (analysis of variance) is a statistical method that allows to evaluate the simultaneous effect of two categorical variables on a quantitative continuous variable.

The two-way ANOVA is an extension of the one-way ANOVA since it allows to evaluate the effects on a numerical response of two categorical variables instead of one.

The advantage of a two-way ANOVA over a one-way ANOVA is that we test the relationship between two variables, while taking into account the effect of a third variable. Moreover, it also allows to include the possible interaction of the two categorical variables on the response to evaluate whether or not they act jointly on the response variable.

The advantage of a two-way over a one-way ANOVA is quite similar to the advantage of a multiple linear regression over a correlation:

The correlation measures the relationship between two quantitative variables. The multiple linear regression also measures the relationship between two variables, but this time taking into account the potential effect of other covariates.
The one-way ANOVA tests whether a quantitative variable is different between groups. The two-way ANOVA also tests whether a quantitative variable is different between groups, but this time taking into account the effect of another qualitative variable.

Previously, we have discussed about one-way ANOVA in R. Now, we show when, why and how to perform a two-way ANOVA in R.

Before going further, I would like to mention and briefly describe some related statistical methods and tests in order to avoid any confusion:

A Student’s t-test is used to evaluate the effect of one categorical variable on a quantitative continuous variable, when the categorical variable has exactly 2 levels:
- Student’s t-test for independent samples if the observations are independent (for example: if we compare the age between women and men)
- Student’s t-test for paired samples if the observations are dependent, that is, when they come in pairs (it is the case when the same subjects are measured twice, at two different points in time, before and after a treatment for example)
To evaluate the effect of one categorical variable on a quantitative variable, when the categorical variable has 3 or more levels:¹
- one-way ANOVA (often simply referred as ANOVA) if the groups are independent (for example a group of patients who received treatment A, another group of patients who received treatment B, and the last group of patients who received no treatment or a placebo)
- repeated measures ANOVA if the groups are dependent (when the same subjects are measured three times, at three different points in time, before, during and after a treatment for example)
A two-way ANOVA is used to evaluate the effects of 2 categorical variables (and their potential interaction) on a quantitative continuous variable. This is the topic of the post.
Linear regression is used to evaluate the relationship between a quantitative continuous dependent variable and one or several independent variables:
- simple linear regression if there is only one independent variable (which can be quantitative or qualitative)
- multiple linear regression if there is at least two independent variables (which can be quantitative, qualitative, or a mix of both)
An ANCOVA (analysis of covariance) is used to evaluate the effect of a categorical variable on a quantitative variable, while controlling for the effect of another quantitative variable (known as covariate). ANCOVA is actually a special case of multiple linear regression with a mix of one qualitative and one quantitative independent variable.
A mixed ANOVA is used to test differences between two or more groups whilst subjecting participants to repeated measures: one factor (a fixed effects factor) is a between-subjects variable (for example, treatment A and B, with patients receiving only one of the two treatments) and the other (a random effects factor) is a within-subjects variable (for example, measurements are made on day 1, day 2 and day 3 on all subjects).

In this post, we start by explaining when and why a two-way ANOVA is useful, we then do some preliminary descriptive analyses and present how to conduct a two-way ANOVA in R. Finally, we show how to interpret and visualize the results. We also briefly mention and illustrate how to verify the underlying assumptions.

Data

To illustrate how to perform a two-way ANOVA in R, we use the penguins dataset, available from the {palmerpenguins} package.

We do not need to import the dataset, but we need to load the package first and then call the dataset:

# install.packages("palmerpenguins")
library(palmerpenguins)

dat <- penguins # rename dataset
str(dat) # structure of dataset

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

The dataset contains 8 variables for 344 penguins, summarized below:

summary(dat)

##       species          island    bill_length_mm  bill_depth_mm  
##  Adelie   :152   Biscoe   :168   Min.   :32.10   Min.   :13.10  
##  Chinstrap: 68   Dream    :124   1st Qu.:39.23   1st Qu.:15.60  
##  Gentoo   :124   Torgersen: 52   Median :44.45   Median :17.30  
##                                  Mean   :43.92   Mean   :17.15  
##                                  3rd Qu.:48.50   3rd Qu.:18.70  
##                                  Max.   :59.60   Max.   :21.50  
##                                  NA's   :2       NA's   :2      
##  flipper_length_mm  body_mass_g       sex           year     
##  Min.   :172.0     Min.   :2700   female:165   Min.   :2007  
##  1st Qu.:190.0     1st Qu.:3550   male  :168   1st Qu.:2007  
##  Median :197.0     Median :4050   NA's  : 11   Median :2008  
##  Mean   :200.9     Mean   :4202                Mean   :2008  
##  3rd Qu.:213.0     3rd Qu.:4750                3rd Qu.:2009  
##  Max.   :231.0     Max.   :6300                Max.   :2009  
##  NA's   :2         NA's   :2

In this post, we will focus on the following three variables:

species: the species of the penguin (Adelie, Chinstrap or Gentoo)
sex: sex of the penguin (female and male)
body_mass_g: body mass of the penguin (in grams)

If needed, more information about this dataset can be found by running ?penguins in R.

body_mass_g is the quantitative continuous variable and will be the dependent variable, whereas species and sex are both qualitative variables.

Those two last variables will be our independent variables, also referred as factors. Make sure that they are read as factors by R. If it is not the case, they will need to be transformed to factors.

Aim and hypotheses of a two-way ANOVA

As mentioned above, a two-way ANOVA is used to evaluate simultaneously the effect of two categorical variables on one quantitative continuous variable.

It is referred as two-way ANOVA because we are comparing groups which are formed by two independent categorical variables.

Here, we would like to know if body mass depends on species and/or sex. In particular, we are interested in:

measuring and testing the relationship between species and body mass,
measuring and testing the relationship between sex and body mass, and
potentially check whether the relationship between species and body mass is different for females and males (which is equivalent than checking whether the relationship between sex and body mass depends on the species)

The first two relationships are referred as main effects, while the third point is known as the interaction effect.

The main effects test whether at least one group is different from another one (while controlling for the other independent variable). On the other hand, the interaction effect aims at testing whether the relationship between two variables differs depending on the level of a third variable. In other words, if the evolution between the response and the first categorical variable does not depend on the modalities of the second categorical variable, then there is no interaction between the two variables. If, on the contrary, there is a modification of this evolution, either by an increase in the effect of the first variable, or by a decrease, then there is an interaction.

When performing a two-way ANOVA, testing the interaction effect is not mandatory. However, omitting an interaction effect may lead to erroneous conclusions if the interaction effect is present.

If we go back to our example, we have the following hypothesis tests:

Main effect of sex on body mass:
- $H_0$: mean body mass is equal between females and males
- $H_1$: mean body mass is different between females and males
Main effect of species on body mass:
- $H_0$: mean body mass is equal between all 3 species
- $H_1$: mean body mass is different for at least one species
Interaction between sex and species:
- $H_0$: there is no interaction between sex and species, meaning that the relationship between species and body mass is the same for females and males (similarly, the relationship between sex and body mass is the same for all 3 species)
- $H_1$: there is an interaction between sex and species, meaning that the relationship between species and body mass is different for females than for males (similarly, the relationship between sex and body mass depends on the species)

Assumptions of a two-way ANOVA

Most statistical tests require some assumptions for the results to be valid, and a two-way ANOVA is not an exception.

Assumptions of a two-way ANOVA are similar than for a one-way ANOVA. To summarize:

Variable type: the dependent variable must be quantitative continuous, while the two independent variables must be categorical (with at least two levels).
Independence: the observations should be independent between groups and within each group.
Normality:
- For small samples, data should follow approximately a normal distribution
- For large samples (usually $n \ge 30$ in each group/sample), normality is not required (thanks to the central limit theorem)
Equality of variances: variances should be equal across groups.
Outliers: There should be no significant outliers in any group.

More details about these assumptions can be found in the assumptions of a one-way ANOVA.

Now that we have seen the underlying assumptions of the two-way ANOVA, we review them specifically for our dataset before applying the test and interpreting the results.

Variable type

The dependent variable body mass is quantitative continuous, while both independent variables sex and species are qualitative variables (with at least 2 levels).

Therefore, this assumption is met.

If your dependent variable is quantitative discrete, this is count data, which, strictly speaking, should be analyzed using a generalized linear model, not an ANOVA.

Independence

Independence is usually checked based on the design of the experiment and how data have been collected.²

To keep it simple, observations are usually:

independent if each experimental unit (here a penguin) has been measured only once and the observations are collected from a representative and randomly selected portion of the population, or
dependent if each experimental unit has been measured at least twice (as it is often the case in the medical field for example, with two measurements on the same subjects; one before and one after the treatment).

In our case, body mass has been measured only once on each penguin, and on a representative and random sample of the population, so the independence assumption is met.

Note that if your data correspond to observations made several times on the same experimental units (for example, if one of the factors is a treatment (A or B) and the second factor is time (day 1, day 2 and day 3), and measurements are made at each time point on the same subjects), a two-way mixed ANOVA should be used.

Normality

We have a large sample in all subgroups (each combination of the levels of the two factors, called cell):

table(dat$species, dat$sex)

##            
##             female male
##   Adelie        73   73
##   Chinstrap     34   34
##   Gentoo        58   61

so normality does not need to be checked.

For completeness, we still show how to verify normality, as if we had a small samples.

There are several methods to test the normality assumption. The most common methods being:

a QQ-plot by group or on the residuals, and/or
a histogram by group or on the residuals, and/or
a normality test (Shapiro-Wilk test for instance) by group or on the residuals.

The easiest/shortest way is to verify the normality with a QQ-plot on the residuals. To draw this plot, we first need to save the model:

# save model
mod <- aov(body_mass_g ~ sex * species,
  data = dat
)

This piece of code will be explained further.

Now we can draw the QQ-plot on the residuals. We show two ways to do so, first with the plot() function and second with the qqPlot() function from the {car} package:

# method 1
plot(mod, which = 2)

# method 2
library(car)

qqPlot(mod$residuals,
  id = FALSE # remove point identification
)

Code for method 1 is slightly shorter, but it misses the confidence interval around the reference line.

If points follow the straight line (called Henry’s line) and fall within the confidence band, we can assume normality. This is the case here.

If you prefer to verify the normality based on a histogram of the residuals, here is the code:

# histogram
hist(mod$residuals)

The histogram of the residuals show a gaussian distribution, which is in line with the conclusion from the QQ-plot.

Although the QQ-plot and histogram is largely enough to verify the normality, if you want to test it more formally with a statistical test, the Shapiro-Wilk test can be applied on the residuals as well:

# normality test
shapiro.test(mod$residuals)

## 
## 	Shapiro-Wilk normality test
## 
## data:  mod$residuals
## W = 0.99776, p-value = 0.9367

$\Rightarrow$ We do not reject the null hypothesis that the residuals follow a normal distribution ($p$-value = 0.937).

From the QQ-plot, histogram and Shapiro-Wilk test, we conclude that we do not reject the null hypothesis of normality of the residuals.

The normality assumption is thus verified, we can now check the equality of the variances.

Note that if the normality assumption is not met, many transformations can be applied on the dependent variable to improve it, the most common ones being the logarithmic (log() function in R) and the Box-Cox transformations. If the normality assumption is still not met on the transformed data, the non-parametric version of the two-way ANOVA, the Scheirer–Ray–Hare test, can be used. Alternatively, a permutation test can also be used.

Homogeneity of variances

Equality of variances, also referred as homogeneity of variances or homoscedasticity, can be verified visually with the plot() function:

plot(mod, which = 3)

Since the spread of the residuals is constant, the red smooth line is horizontal and flat, so it looks like the constant variance assumption is satisfied here.

The diagnostic plot above is sufficient, but if you prefer it can also be tested more formally with the Levene’s test (also from the {car} package):³

leveneTest(mod)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   5  1.3908 0.2272
##       327

$\Rightarrow$ We do not reject the null hypothesis that the variances are equal ($p$-value = 0.227).

Both the visual and formal approaches give the same conclusion; we do not reject the hypothesis of homogeneity of the variances.

Note that, as for the normality, the logarithmic and Box-Cox transformations may improve homogeneity of the residuals.

Outliers

The easiest and most common way to detect outliers is visually thanks to boxplots by groups.

For females and males:

library(ggplot2)

# boxplots by sex
ggplot(dat) +
  aes(x = sex, y = body_mass_g) +
  geom_boxplot()

For the three species:

# boxplots by species
ggplot(dat) +
  aes(x = species, y = body_mass_g) +
  geom_boxplot()

There are, as defined by the interquartile range criterion, two outliers for the species Chinstrap. These points are, nonetheless, not extreme enough to bias results.

Therefore, we consider that the assumption of no significant outliers is met.

Two-way ANOVA

We have shown that all assumptions are met, so we can now proceed to the implementation of the two-way ANOVA in R.

This will allow us to answer the following research questions:

Controlling for the species, is body mass significantly different between the two sexes?
Controlling for the sex, is body mass significantly different for at least one species?
Is the relationship between species and body mass different between female and male penguins?

Preliminary analyses

Before performing any statistical test, it is a good practice to make some descriptive statistics in order to have a first overview of the data, and perhaps, have a glimpse of the results to be expected.

This can be done via descriptive statistics or plots.

Descriptive statistics

If we want to keep it simple, we can compute only the mean for each subgroup:

# mean by group
aggregate(body_mass_g ~ species + sex,
  data = dat,
  FUN = mean
)

##     species    sex body_mass_g
## 1    Adelie female    3368.836
## 2 Chinstrap female    3527.206
## 3    Gentoo female    4679.741
## 4    Adelie   male    4043.493
## 5 Chinstrap   male    3938.971
## 6    Gentoo   male    5484.836

Or eventually, the mean and standard deviation for each subgroup using the {dplyr} package:

# mean and sd by group
library(dplyr)

group_by(dat, sex, species) %>%
  summarise(
    mean = round(mean(body_mass_g, na.rm = TRUE)),
    sd = round(sd(body_mass_g, na.rm = TRUE))
  )

## # A tibble: 8 × 4
## # Groups:   sex [3]
##   sex    species    mean    sd
##   <fct>  <fct>     <dbl> <dbl>
## 1 female Adelie     3369   269
## 2 female Chinstrap  3527   285
## 3 female Gentoo     4680   282
## 4 male   Adelie     4043   347
## 5 male   Chinstrap  3939   362
## 6 male   Gentoo     5485   313
## 7 <NA>   Adelie     3540   477
## 8 <NA>   Gentoo     4588   338

Plots

If you are a frequent reader of the blog, you know that I like to draw plots to visualize the data at hand before interpreting results of a test.

The most appropriate plot when we have one quantitative and two qualitative variables is a boxplot by group. This can easily be made with the {ggplot2} package:

# boxplot by group
library(ggplot2)

ggplot(dat) +
  aes(x = species, y = body_mass_g, fill = sex) +
  geom_boxplot()

Some observations are missing for the sex, we can remove them to have a more concise plot:

dat %>%
  filter(!is.na(sex)) %>%
  ggplot() +
  aes(x = species, y = body_mass_g, fill = sex) +
  geom_boxplot()

Note that we could also have made the following plot:

dat %>%
  filter(!is.na(sex)) %>%
  ggplot() +
  aes(x = sex, y = body_mass_g, fill = species) +
  geom_boxplot()

But for a more readable plot, I tend to prefer putting the variable with the smallest number of levels as color (which is in fact the argument fill in the aes() layer) and the variable with the largest number of categories on the x-axis (i.e., the argument x in the aes() layer).

From the means and the boxplots by subgroup, we can already see that, in our sample:

female penguins tend to have a lower body mass than males, and that is the case for all the considered species, and
body mass is higher for Gentoo penguins than for the other two species.

Bear in mind that these conclusions are only valid within our sample! To generalize these conclusions to the population, we need to perform the two-way ANOVA and check the significance of the explanatory variables. This is the aim of the next section.

Two-way ANOVA in R

As mentioned earlier, including an interaction effect in a two-way ANOVA is not compulsory. However, in order to avoid flawed conclusions, it is recommended to first check whether the interaction is significant or not, and depending on the results, include it or not.

If the interaction is not significant, it is safe to remove it from the final model. On the contrary, if the interaction is significant, it should be included in the final model which will be used to interpret results.

We thus start with a model which includes the two main effects (i.e., sex and species) and the interaction:

# Two-way ANOVA with interaction
# save model
mod <- aov(body_mass_g ~ sex * species,
  data = dat
)

# print results
summary(mod)

##              Df    Sum Sq  Mean Sq F value   Pr(>F)    
## sex           1  38878897 38878897 406.145  < 2e-16 ***
## species       2 143401584 71700792 749.016  < 2e-16 ***
## sex:species   2   1676557   838278   8.757 0.000197 ***
## Residuals   327  31302628    95727                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 11 observations deleted due to missingness

Similar to a one-way ANOVA, the principle of a two-way ANOVA is based on the total dispersion of the data, and its decomposition into four components:

the share attributable to the first factor
the share attributable to the second factor
the share attributable to the interaction of the 2 factors
the unexplained, or residual portion.

The sum of squares (column Sum Sq) shows these four components. The two-way ANOVA consists of using a statistical test to determine whether each of the dispersion component (attributable to the 2 factors studied and to their interaction) is significantly greater than the residual component. If this is the case, we conclude that the effect considered (factor A, factor B or the interaction) is significant.

We see that the species explain a large part of the variability of body mass. It is the most important factor in explaining this variability.

The $p$-values are displayed in the last column of the output above (Pr(>F)). From these $p$-values, we conclude that, at the 5% significance level:

controlling for the species, body mass is significantly different between the two sexes,
controlling for the sex, body mass is significantly different for at least one species, and
the interaction between sex and species (displayed at the line sex:species in the output above) is significant.

So from the significant interaction effect, we have just seen that the relationship between body mass and species is different between males and females. Since it is significant, we have to keep it in the model and we should interpret results from that model.

If, on the contrary, the interaction was not significant (that is, if the $p$-value $\ge$ 0.05) we would have removed this interaction effect from the model. For illustrative purposes, below the code for a two-way ANOVA without interaction, referred as an additive model:

# Two-way ANOVA without interaction
aov(body_mass_g ~ sex + species,
  data = dat
)

For the readers who are used to perform linear regressions in R, you will notice that the structure of the code for a two-way ANOVA is in fact similar:

the formula is dependent variable ~ independent variables
the + sign is used to include independent variables without an interaction⁴
the * sign is used to include independent variables with an interaction

The resemblance with a linear regression is not a surprise because a two-way ANOVA, like all ANOVA, is actually a linear model.

Note that the following code works as well, and give the same results:⁵

# method 2
mod2 <- lm(body_mass_g ~ sex * species,
  data = dat
)

Anova(mod2)

## Anova Table (Type II tests)
## 
## Response: body_mass_g
##                Sum Sq  Df F value    Pr(>F)    
## sex          37090262   1 387.460 < 2.2e-16 ***
## species     143401584   2 749.016 < 2.2e-16 ***
## sex:species   1676557   2   8.757 0.0001973 ***
## Residuals    31302628 327                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Note that the aov() function assumes a balanced design, meaning that we have equal sample sizes within levels of our independent grouping variables. Moreover, aov() uses the type I sums of squares, so we can obtain different $p$-values when we write y ~ A * B and y ~ B * A.

For unbalanced design, that is, unequal numbers of subjects in each subgroup, the recommended methods are:

the type II ANOVA when there is no significant interaction, which can be done in R with Anova(mod, type = "II") or Anova(mod, type = 2), where mod is the name of your saved model, and
the type III ANOVA when there is a significant interaction, which can be done in R with Anova(mod, type = "III") or Anova(mod, type = 3).

This is beyond the scope of the post and we assume a balanced design here. For the interested reader, see this detailed discussion about type I, type II and type III ANOVA.

Pairwise comparisons

Through the two main effects being significant, we concluded that:

controlling for the species, body mass is different between females and males, and
controlling for the sex, body mass is different for at least one species.

If body mass is different between the two sexes, given that there are exactly two sexes, it must be because body mass is significantly different between females and males.

If one wants to know which sex has the highest body mass, it can be deduced from the means and/or boxplots by subgroup. Here, it is clear that males have a significantly higher body mass than females.

However, it is not so straightforward for the species. Let me explain why it is not as easy as for the sexes.

There are three species (Adelie, Chinstrap and Gentoo), so there are 3 pairs of species:

Adelie and Chinstrap
Adelie and Gentoo
Chinstrap and Gentoo

If body mass is significantly different for at least one species, it could be that:

body mass is significantly different between Adelie and Chinstrap but not significantly different between Adelie and Gentoo, and not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Adelie and Gentoo but not significantly different between Adelie and Chinstrap, and not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Chinstrap and Gentoo but not significantly different between Adelie and Chinstrap, and not significantly different between Adelie and Gentoo.

Or, it could also be that:

body mass is significantly different between Adelie and Chinstrap, and between Adelie and Gentoo, but not significantly different between Chinstrap and Gentoo, or
body mass is significantly different between Adelie and Chinstrap, and between Chinstrap and Gentoo, but not significantly different between Adelie and Gentoo, or
body mass is significantly different between Chinstrap and Gentoo, and between Adelie and Gentoo, but not significantly different between Adelie and Chinstrap.

Last, it could also be that body mass is significantly different between all species.

As for a one-way ANOVA, we cannot, at this stage, know precisely which species is different from which one in terms of body mass. To know this, we need to compare each species two by two thanks to post-hoc tests (also known as pairwise comparisons).

There are several post-hoc tests, the most common ones being the Tukey HSD which tests all possible pairs of groups, and the Dunett’s test which compares all groups to a reference group. As mentioned earlier, these tests should not be done on the sex variable because there are only two levels.

In this post, we show only the Tukey HSD test. For the interested reader, the Dunnett’s test is illustrated here.

As for the one-way ANOVA, the Tukey HSD test can be done in R as follows:

# method 1
TukeyHSD(mod,
  which = "species"
)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = body_mass_g ~ sex * species, data = dat)
## 
## $species
##                        diff       lwr       upr     p adj
## Chinstrap-Adelie   26.92385  -80.0258  133.8735 0.8241288
## Gentoo-Adelie    1377.65816 1287.6926 1467.6237 0.0000000
## Gentoo-Chinstrap 1350.73431 1239.9964 1461.4722 0.0000000

or using the {multcomp} package:

# method 2
library(multcomp)

res_tukey <- glht(
  aov(body_mass_g ~ sex + species,
    data = dat
  ),
  linfct = mcp(species = "Tukey")
)

summary(res_tukey)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: aov(formula = body_mass_g ~ sex + species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0    26.92      46.48   0.579     0.83    
## Gentoo - Adelie == 0     1377.86      39.10  35.236   <1e-05 ***
## Gentoo - Chinstrap == 0  1350.93      48.13  28.067   <1e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

or using the pairwise.t.test() function using the $p$-value adjustment method of your choice:⁶

# method 3
pairwise.t.test(dat$body_mass_g, dat$species,
  p.adjust.method = "BH"
)

## 
## 	Pairwise comparisons using t tests with pooled SD 
## 
## data:  dat$body_mass_g and dat$species 
## 
##           Adelie Chinstrap
## Chinstrap 0.63   -        
## Gentoo    <2e-16 <2e-16   
## 
## P value adjustment method: BH

Note that when using the second method, it is the model without the interaction that needs to be specified into the glht() function, even if the interaction is significant. Moreover, do not forget to replace mod and species in my code with the name of your model and the name of your independent variable.

Both methods give the same results, that is:

body mass is not significantly different between Chinstrap and Adelie (adjusted $p$-value = 0.83),
body mass is significantly different between Gentoo and Adelie (adjusted $p$-value < 0.001), and
body mass is significantly different between Gentoo and Chinstrap (adjusted $p$-value < 0.001).

Remember that it is the adjusted $p$-values that are reported, to prevent the issue of multiple testing which occurs when comparing several pairs of groups.

If you would like to compare all combinations of groups, it can be done with the TukeyHSD() function and specifying the interaction in the which argument:

# all combinations of sex and species
TukeyHSD(mod,
  which = "sex:species"
)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = body_mass_g ~ sex * species, data = dat)
## 
## $`sex:species`
##                                      diff       lwr       upr     p adj
## male:Adelie-female:Adelie        674.6575  527.8486  821.4664 0.0000000
## female:Chinstrap-female:Adelie   158.3703  -25.7874  342.5279 0.1376213
## male:Chinstrap-female:Adelie     570.1350  385.9773  754.2926 0.0000000
## female:Gentoo-female:Adelie     1310.9058 1154.8934 1466.9181 0.0000000
## male:Gentoo-female:Adelie       2116.0004 1962.1408 2269.8601 0.0000000
## female:Chinstrap-male:Adelie    -516.2873 -700.4449 -332.1296 0.0000000
## male:Chinstrap-male:Adelie      -104.5226 -288.6802   79.6351 0.5812048
## female:Gentoo-male:Adelie        636.2482  480.2359  792.2606 0.0000000
## male:Gentoo-male:Adelie         1441.3429 1287.4832 1595.2026 0.0000000
## male:Chinstrap-female:Chinstrap  411.7647  196.6479  626.8815 0.0000012
## female:Gentoo-female:Chinstrap  1152.5355  960.9603 1344.1107 0.0000000
## male:Gentoo-female:Chinstrap    1957.6302 1767.8040 2147.4564 0.0000000
## female:Gentoo-male:Chinstrap     740.7708  549.1956  932.3460 0.0000000
## male:Gentoo-male:Chinstrap      1545.8655 1356.0392 1735.6917 0.0000000
## male:Gentoo-female:Gentoo        805.0947  642.4300  967.7594 0.0000000

Or with the HSD.test() function from the {agricolae} package, which denotes subgroups that are not significantly different from each other with the same letter:

library(agricolae)

HSD.test(mod,
  trt = c("sex", "species"),
  console = TRUE # print results
)

## 
## Study: mod ~ c("sex", "species")
## 
## HSD Test for body_mass_g 
## 
## Mean Square Error:  95726.69 
## 
## sex:species,  means
## 
##                  body_mass_g      std  r  Min  Max
## female:Adelie       3368.836 269.3801 73 2850 3900
## female:Chinstrap    3527.206 285.3339 34 2700 4150
## female:Gentoo       4679.741 281.5783 58 3950 5200
## male:Adelie         4043.493 346.8116 73 3325 4775
## male:Chinstrap      3938.971 362.1376 34 3250 4800
## male:Gentoo         5484.836 313.1586 61 4750 6300
## 
## Alpha: 0.05 ; DF Error: 327 
## Critical Value of Studentized Range: 4.054126 
## 
## Groups according to probability of means differences and alpha level( 0.05 )
## 
## Treatments with the same letter are not significantly different.
## 
##                  body_mass_g groups
## male:Gentoo         5484.836      a
## female:Gentoo       4679.741      b
## male:Adelie         4043.493      c
## male:Chinstrap      3938.971      c
## female:Chinstrap    3527.206      d
## female:Adelie       3368.836      d

If you have many groups to compare, plotting them might be easier to interpret:

# set axis margins so labels do not get cut off
par(mar = c(4.1, 13.5, 4.1, 2.1))

# create confidence interval for each comparison
plot(TukeyHSD(mod, which = "sex:species"),
  las = 2 # rotate x-axis ticks
)

From the outputs and plot above, we conclude that all combinations of sex and species are significantly different, except between female Chinstrap and female Adelie ($p$-value = 0.138) and male Chinstrap and male Adelie ($p$-value = 0.581).

These results, which are by the way in line with the boxplots shown above and which will be confirmed with the visualizations below, concludes the two-way ANOVA in R.

Visualizations

If you would like to visualize results in a different way to what has already been presented in the preliminary analyses, below are some ideas of useful plots.

First, with the mean and standard error of the mean by subgroup using the allEffects() function from the {effects} package:

# method 1
library(effects)

plot(allEffects(mod))

Or using the {ggpubr} package:

# method 2
library(ggpubr)

ggline(subset(dat, !is.na(sex)), # remove NA level for sex
  x = "species",
  y = "body_mass_g",
  color = "sex",
  add = c("mean_se") # add mean and standard error
) +
  labs(y = "Mean of body mass (g)")

Alternatively, using {Rmisc} and {ggplot2}:

library(Rmisc)

# compute mean and standard error of the mean by subgroup
summary_stat <- summarySE(dat,
  measurevar = "body_mass_g",
  groupvars = c("species", "sex")
)

# plot mean and standard error of the mean
ggplot(
  subset(summary_stat, !is.na(sex)), # remove NA level for sex
  aes(x = species, y = body_mass_g, colour = sex)
) +
  geom_errorbar(aes(ymin = body_mass_g - se, ymax = body_mass_g + se), # add error bars
    width = 0.1 # width of error bars
  ) +
  geom_point() +
  labs(y = "Mean of body mass (g)")

Second, if you prefer to draw only the mean by subgroup:

with(
  dat,
  interaction.plot(species, sex, body_mass_g)
)

Last but not least, for those of you who are familiar with GraphPad, you are most likely familiar with plotting means and error bars as follows:

# plot mean and standard error of the mean as barplots
ggplot(
  subset(summary_stat, !is.na(sex)), # remove NA level for sex
  aes(x = species, y = body_mass_g, fill = sex)
) +
  geom_bar(position = position_dodge(), stat = "identity") +
  geom_errorbar(aes(ymin = body_mass_g - se, ymax = body_mass_g + se), # add error bars
    width = 0.25, # width of error bars
    position = position_dodge(.9)
  ) +
  labs(y = "Mean of body mass (g)")

Conclusion

In this post, we started with a few reminders of the different tests that exist to compare a quantitative variable across groups. We then focused on the two-way ANOVA, starting from its goal and hypotheses to its implementation in R, together with the interpretations and some visualizations. We also briefly mentioned its underlying assumptions and one post-hoc test to compare all subgroups.

All this was illustrated with the penguins dataset available from the {palmerpenguins} package.

Thanks for reading.

I hope this article will help you in conducting a two-way ANOVA with your data.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

In theory, a one-way ANOVA can also be used to compare 2 groups, and not only 3 or more. Nonetheless, in practice, it is often the case that a Student’s t-test is performed to compare 2 groups, and a one-way ANOVA to compare 3 or more groups. Conclusions obtained via a Student’s t-test for independent samples and a one-way ANOVA with 2 groups will be similar.↩︎
If you really want to test the independence, you can do so visually with a plot of the residuals vs. fitted values. This plot can be done in R with plot(mod, which = 1), where mod corresponds to the name of your model. Or you can do so with the Durbin-Watson test. In R, it can be done with the durbinWatsonTest() function from the {car} package.↩︎
Note that the Bartlett’s and Fligner-Killeen tests are also appropriate to test the assumption of equal variances.↩︎
An additive model makes the assumption that the 2 explanatory variables are independent; they do not interact with each other.↩︎
To not be confused with the anova() function because it provides sequential results that depend on the order in which the variables appear in the model.↩︎
Here, we use the Benjamini & Hochberg (1995) correction, but you can choose between several methods. See ?p.adjust for more details.↩︎

What is survival analysis? Examples by hand and in R

Thu, 22 Dec 2022 00:00:00 +0000

Note that this article is inspired from:

the lecture notes of Prof. Van Keilegom and my personal notes as teaching assistant for her course entitled “Analysis of Survival and Duration Data” given at UCLouvain
the lecture notes of Prof. Legrand for her course entitled “Statistics in clinical trials” given at UCLouvain

Introduction

For the last post of the year, I would like to present a rather unknown (yet important) statistical method–survival analysis.

Although survival analysis is a branch of statistics, it is usually not covered in introductory statistics courses and it is rather unknown to the general public. It is mostly taught in biostatistics courses or advanced statistics study programs.

In this article, I will explain what is survival analysis, in which context and how it is used. I will explain the main tools and methods used by biostatisticians to analyze survival data and how to estimate and interpret survival curves.

I will show in detail how to apply these techniques in R with concrete examples. In practice, survival analysis is almost always done via a statistical program and never done by hand. However, as for any statistical concept, I believe that doing it by hand allows to really understand the concepts and what these programs actually do. For this reason, I will also show a brief example on how to perform a basic survival analysis by hand.

What is survival analysis?

Survival analysis (also called time-to-event analysis or duration analysis) is a branch of statistics aimed at analyzing the duration of time from a well-defined time origin until one or more events happen, called survival times or duration times.

In other words, in survival analysis, we are interested in a certain event and want to analyze the time until the event happens.

While the event of interest is often death (in this case we study the time to death for patients having a specific disease) or recurrence (in this case we study the time to relapse of a certain disease), it is not limited to the medical field or epidemiology.

In fact, it can be used in many domains. For example, we may also analyze the time until:

getting cured from a certain disease
finding a new job after a period of unemployment
being arrested again after having been released from jail
the first pregnancy
the failure of a mechanical system or a machine
a bank or a company goes bankrupt
a customer buys a new product or stops its current subscription
a letter is delivered
a taxi picks you up after having called the taxi company
an employee leaves the company
etc.

As you can see, the event of interest does not necessarily have to be the death or the occurrence of a disease, but in all situations we are interested in analyzing the time until a specific event occurs.

Why do we need special methods for survival analysis?

Survival data, also referred as time-to-event data, requires a special set of statistical methods for three main reasons:

Duration times are always positive: the time until an event of interest occurs cannot be less than 0. Moreover, the distribution of survival times is right-skewed.
Different measures are of interest depending on the research question, context, etc. For instance, we could be interested in:
- The probability that a cancer patient survives longer than 5 years after diagnosis?
- The typical waiting time for a cab to arrive after having called the taxi company?
- How many, out of 100 unemployed people, are expected to have a job again after 2 months of unemployment?
Censoring is almost always an issue:
- When the event occurred before the end of the study, the survival time is known.
- However, sometimes, the event is not yet observed at the end of the study. Suppose that we study the time until death of patients with breast cancer. Luckily, some patients will not die before the end of the study.
- It can also happen that the patient withdraws from the study or moves to another country before the end of the study (known as lost to follow up or drop out).
- In all situations, his or her survival time cannot be observed because the event is not observed for the duration of the study.
- Censoring can be seen, in some sense, as a type of missing data.
- For these reasons, many “standard” statistical tools such as descriptive statistics, hypothesis tests and regression models are not appropriate for this kind of data. Specific statistical methods are required to take into account the fact that the exact survival duration for some patients is missing. It is known that they survived a certain amount of time (until the end of the study or until the time of withdrawal), but their exact survival time is unknown.

For your information, there are three types of censoring:

right-censoring (the most frequent),
left-censoring (the least frequent) and
interval-censoring.

When the event is not yet observed at the end of the study (i.e., the survival time is greater than the observed duration), this is referred as right-censoring. Left-censoring occurs if a participant is entered into the study when the event of interest occurred prior to study entry but we do not know exactly when. Interval-censoring implies that the event occurred within a time interval (between two known dates, two visits, etc.); the exact moment of occurrence is not known. The goal is of course to analyze all available data, including information about censored patients.

The goal of survival analysis is thus to model and describe time-to-event data in an appropriate way, taking the particularities of this type of data into account.

Common functions in survival analysis

We are not going to go to much into the details, but it is important to lay the foundation with the most common functions in survival analysis.

Let $T$ be a non-negative continuous random variable, representing the time until the event of interest. We consider the following functions:

Survival function
Cumulative hazard function
Hazard function

Survival function

The most common one is the survival function.

Let $T$ be a non-negative continuous random variable, representing the time until the event of interest. The survival function $S(t)$ is the probability that a randomly chosen individual is still at risk at time $t$, where $0 \le t \le +\infty$. For each $t$, it is given by

\[ \begin{align*} S(t) &= P(T > t)\\ &= 1 - P(T \le t)\\ &= 1 - F(t)\\ &= 1 - \int^t_0 f(u)\text{d}u, \end{align*} \]

where $f(\cdot)$ and $F(\cdot)$ are the density and the cumulative distribution functions of $T$, respectively.

$S(t)$ represents, for each time $t$, the probability that the time until the event is greater than this time $t$. In other words, it models the probability that the event of interest happens after $t$.

In the context of our examples mentioned above, it gives the probability that:

a randomly selected patient will survive beyond time $t$ or the proportion of patients still alive after time $t$,
a cab takes more than $t$ minutes to arrive, or
an unemployed person take more than $t$ months to find a new job.

The survival function $S(t)$ is:

a decreasing function,
taking values in $[0, 1]$ (since it is a probability), and
equal to 1 at $t = 0$ (i.e., $S(0) = 1$) and 0 at $t = \infty$ (i.e., $S(\infty) = 0$).

Visually we have:

The curve shows the proportion of individuals (or experimental units) who, as time goes on, have not experienced the event of interest. As time progresses, events occur, so the proportion who have not experienced the event decreases.

Cumulative hazard function

The cumulative hazard function, which is the total hazard experienced up to time $t$, is defined as:

\[H(t) = -log\left(S(t)\right)\]

and has the following properties:

increasing function,
taking value in $[0, +\infty]$, and
$S(t) = exp(-H(t))$.

Hazard function

The hazard function $h(t)$, or hazard rate, defines the instantaneous event rate at time $t$ for an individual still at risk at that time. It can be obtained by

\[ \begin{align*} h(t) &= \lim_{\Delta t \rightarrow 0} \frac{P(t \le T < t + \Delta t | T \ge t)}{\Delta t}\\ &= \frac{d}{dt} H(t)\\ &= \frac{f(t)}{S(t)}. \end{align*} \]

and has the following properties:

positive function (not necessarily increasing or decreasing)
the hazard function $h(t)$ can have many different shapes and is therefore a useful tool to summarize survival data

In the context of cancer research when death is the event of interest, $h(t)$ measures the instantaneous risk of dying right after time $t$ given the individual is alive at time $t$.

To link the hazard rate with the survival function; the survival curve represents the hazard rates. A steeper slope indicates a higher hazard rate because events happen more frequently, reducing the proportion of individuals who have not experienced the event at a faster rate. On the contrary, a gradual and flatter slope indicates a lower hazard rate because events occur less frequently, reducing the proportion of individuals who have not experiences the event at a slower rate. More formally:

\[S(t) = \exp\left(-\int^t_0 h(u) \text{d}u\right).\]

Note that, in contrast to the survival function which focuses on not having an event, the hazard function focuses on the event occurring.

Estimation

To estimate the survival function, we need to use an estimator which is able to deal with censoring. The most common one is the nonparametric Kaplan and Meier (1958) estimator (also sometimes referred as the product-limit estimator, or more simply, the K-M estimator).

The advantages of the Kaplan-Meier estimator are that:

it is simple and straightforward to use and interpret
it is a nonparametric estimator, so it constructs a survival curve from the data and no assumptions is made about the shape of the underlying distribution
it gives a graphical representation of the survival function(s), useful for illustrative purposes

The principle behind this estimator is that surviving beyond time $t_i$ implies surviving beyond time $t_{i-1}$ and surviving at time $t_i$. Note that an important assumption for the estimation to hold is that censoring is independent of the occurrence of events. We say that censoring is non-informative, that is, censored subjects have the same survival prospects as subjects who are not censored and who continue to be followed.

By hand

To understand how it works, let’s first estimate it by hand on the following dataset:¹

subject	time	event
1	3	0
2	5	1
3	7	1
4	2	1
5	18	0
6	16	1
7	2	1
8	9	1
9	16	1
10	5	0

where:

subject is the individual’s identifier
time is the time to event (in years)²
event is the event status (0 = censored, 1 = event happened)

Remember that for each subject, we need to know at least 2 pieces of information:

the time until the event of interest or the time until the censoring, and
whether we have observed the event of interest or if we have observed censoring.

We first need to count the number of distinct event times. Ignoring censored observations, we have 5 distinct event times:

2, 5, 7, 9 and 16

The easiest way to do the calculation by hand is by filling the following table (a table with 5 rows since there are 5 distinct event times):

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1
2
3
4
5

We fill columns one by one:

$y_{(j)}$ = the ordered distinct event times:

2, 5, 7, 9 and 16

So the table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1	2
2	5
3	7
4	9
5	16

$d_{(j)}$ = the number of observations for each distinct event time. For this, the frequency for each distinct event time is useful:

## time
##  2  5  7  9 16 
##  2  1  1  1  2

The table becomes:

$j$	$y_{(j)}$	$d_{(j)}$
1	2	2
2	5	1
3	7	1
4	9	1
5	16	2

$R_{(j)}$ = the remaining number of individuals at risk. For this, the distribution of time (censored and not censored) is useful:

## time
##  2  3  5  7  9 16 18 
##  2  1  2  1  1  2  1

We see that:

At the beginning there are 10 subjects
Just before time $t = 5$, there are 7 subjects left (10 subjects - 2 who had the event - 1 who is censored)
Just before time $t = 7$, there are 5 subjects left (= 10 - 2 - 1 - 2)
Just before time $t = 9$, there are 4 subjects left (= 10 - 2 - 1 - 2 - 1)
Just before time $t = 16$, there are 3 subjects left (= 10 - 2 - 1 - 2 - 1 - 1)

The table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$
1	2	2	10
2	5	1	7
3	7	1	5
4	9	1	4
5	16	2	3

$1 - \frac{d_{(j)}}{R_{(j)}}$ is straightforward, so the table becomes:

$j$	$y_{(j)}$	$d_{(j)}$	$R_{(j)}$	$1 - \frac{d_{(j)}}{R_{(j)}}$
1.00	2.00	2.00	10.00	0.80
2.00	5.00	1.00	7.00	0.86
3.00	7.00	1.00	5.00	0.80
4.00	9.00	1.00	4.00	0.75
5.00	16.00	2.00	3.00	0.33

The Kaplan-Meier estimator is:

\[\hat{S}_{KM}(t) = \prod_{j:y_{(j)} \le t} \left(1 - \frac{d_{(j)}}{R_{(j)}} \right)\]

For each $j$, we thus take the cumulative product:

$j_1 = 0.8$
$j_2 = 0.8 \cdot 0.857 = 0.6856$
$j_3 = 0.6856 \cdot 0.8 = 0.54848$
$j_4 = 0.54848 \cdot 0.75 = 0.41136$
$j_5 = 0.41136 \cdot 0.333 = 0.1369829$

So finally, we have the survival probabilities (rounded to 3 digits):

$j$	$1 - \frac{d_{(j)}}{R_{(j)}}$	$\hat{S}_{KM}(t)$
1.00	0.80	0.80
2.00	0.86	0.69
3.00	0.80	0.55
4.00	0.75	0.41
5.00	0.33	0.14

We can now represent graphically the Kaplan-Meier estimator:

To draw this survival curve, remember that:

the x-axis corresponds to the time variable in the initial dataset, and
the y-axis corresponds to the survival probabilities found above.

In R

We now compare our results with the results found in R.

We first create the dataset with the time and event variables:

# create dataset
dat <- data.frame(
  time = c(3, 5, 7, 2, 18, 16, 2, 9, 16, 5),
  event = c(0, 1, 1, 1, 0, 1, 1, 1, 1, 0)
)

We then run the Kaplan-Meier estimator with the survfit() and Surv() functions:

# KM
library(survival)

km <- survfit(Surv(time, event) ~ 1,
  data = dat
)

Notice that the Surv() function accepts two arguments:

the time variable, and
the event variable.

The ~ 1 in the survfit() function indicates that we estimate the Kaplan-Meier without any grouping. See more on this later in the post.

Finally, we display the results and draw the Kaplan-Meier plot:

# results
summary(km)

## Call: survfit(formula = Surv(time, event) ~ 1, data = dat)
## 
##  time n.risk n.event survival std.err lower 95% CI upper 95% CI
##     2     10       2    0.800   0.126       0.5868        1.000
##     5      7       1    0.686   0.151       0.4447        1.000
##     7      5       1    0.549   0.172       0.2963        1.000
##     9      4       1    0.411   0.176       0.1782        0.950
##    16      3       2    0.137   0.126       0.0225        0.834

# plot
plot(km,
  xlab = "Time",
  ylab = "Survival probability",
  conf.int = FALSE
)

The survival probabilities can be found in the survival column. Remark that results by hand and in R are similar (any difference with the results by hand is due to rounding).

Alternatively, we can use the ggsurvplot() function within the {survminer} package:

library(survminer)

# plot
ggsurvplot(km,
  conf.int = FALSE,
  legend = "none"
)

Note that the crosses on the survival curve denote the censored observations.

The advantage with the ggsurvplot() function is that it is easy to draw the median survival directly on the plot:³

ggsurvplot(km,
  conf.int = FALSE,
  surv.median.line = "hv",
  legend = "none"
)

To find the median survival:⁴

summary(km)$table["median"]

## median 
##      9

# or more simply
km

## Call: survfit(formula = Surv(time, event) ~ 1, data = dat)
## 
##       n events median 0.95LCL 0.95UCL
## [1,] 10      7      9       5      NA

Suppose that the event of interest is death:

At time zero, the survival probability is 1 (100% of the subjects are alive).
The median indicates that the median survival time is 9 years.⁵ This is the time at which the survival $S(t)$ is 50%. In other words, is the time after which half of the subjects are expected to have died.
From the plot, we also see that $S(5) = P(T > 5 \text{ years}) =$ Probability of survival of more than 5 years for these subjects = 68.6%. This means that 68.6% of all subjects survive longer than 5 years, and that 31.4% of all subjects die within the first 5 years.

For the sake of completeness, let’s do another example with a much larger dataset; the tongue dataset within the {KMsurv} package.⁶

# load data
library(KMsurv)
data(tongue)

# preview data
head(tongue)

##   type time delta
## 1    1    1     1
## 2    1    3     1
## 3    1    3     1
## 4    1    4     1
## 5    1   10     1
## 6    1   13     1

type is the tumor DNA profile (1 = aneuploid tumor, 2 = diploid tumor)
time is the time to death or on-study time (in weeks)
delta is the death indicator (0 = alive, 1 = dead)

For this example, we focus on the aneuploid type:

anaploid <- subset(tongue, type == 1)

We can now plot the estimated survival function and estimate the median time to death. Since it is an estimator, we can also construct a confidence interval for the estimated survival at each time $t$ and for the estimated median survival time.⁷

# results
fit <- survfit(Surv(time, delta) ~ 1,
  data = anaploid,
  conf.type = "log-log"
)

fit

## Call: survfit(formula = Surv(time, delta) ~ 1, data = anaploid, conf.type = "log-log")
## 
##       n events median 0.95LCL 0.95UCL
## [1,] 52     31     93      65     157

# plot
ggsurvplot(fit,
  surv.median.line = "hv",
  legend = "none"
)

The median survival time is estimated to be 93 weeks, with a 95% confidence interval between 65 and 157 weeks.

Kaplan-Meier curves can be seen as descriptive statistics for survival data. We now focus on the second branch of statistics, hypothesis testing which allows to draw conclusions on the population based on a sample (see a quick reminder about the difference between population and sample if you need).

Hypothesis testing

Hypothesis testing in the field of survival analysis mostly concerns:

The hazard function of one population: in this case we test whether a censored sample comes from a population with a known hazard function $h_0(t)$. For example, we may be interested to compare survival in a sample of patients to the survival in the overall population (derived from the life tables).
The comparison of the hazard function of two or more populations: in this case we are interested in assessing whether there are differences in survival among different groups of subjects. For example:
- 2 groups: we are interested in comparing survival for female and male colon cancer patients
- 3 groups or more: we are interested in comparing survival for melanoma cancer patients according to their treatments (with treatments A, B and C for example)⁸

Log-rank test

In this article, we focus on comparing survival between two groups using the log-rank test (also known as Mantel-Cox test). This test is the most common hypothesis test to compare survival between two groups.

The intuition behind the test is that if the two groups have different hazard rates, the two survival curves (so their slopes) will differ. More precisely, the log-rank test compares the observed number of events in each group to what would be expected if the survival curves were identical (i.e., if the null hypothesis were true).

Note that, as for the Kaplan-Meier estimator, the log-rank test is a nonparametric test, which makes no assumptions about the survival distributions.

For this example, consider the following dataset:

##    patient group time event
## 1        1     1  4.1     1
## 2        2     1  7.8     0
## 3        3     1 10.0     1
## 4        4     1 10.0     1
## 5        5     1 12.3     0
## 6        6     1 17.2     1
## 7        7     2  9.7     1
## 8        8     2 10.0     1
## 9        9     2 11.1     0
## 10      10     2 13.1     0
## 11      11     2 19.7     1
## 12      12     2 24.1     0

where:

patient is the patient’s identifier
group is the group (group 1 or 2)
time is the time to death (in years)⁹
event is the event status (0 = censored, 1 = death)

Suppose we are interested in comparing group 1 and 2 in terms of survival, that is, we compare survival curves between the 2 groups:

$H_0 : S_1(t) = S_2(t)$ for all $t$
$H_1 : S_1(t) \ne S_2(t)$ for some $t$

It is a statistical test, so if the $p$-value < $\alpha$ (usually 0.05), we reject the null hypothesis and we conclude that survival (or the time to event) is significantly different between the two groups considered.

To perform the log-rank test, the following test statistic will be useful:

\[\begin{eqnarray} U &=& \sum_{j=1}^r w(y_{(j)})\left(O_j - E_j\right) \\ &=& \sum_{j=1}^r w(y_{(j)})\left( d_{(j)1} - \frac{d_{(j)}R_{(j)1}}{R_{(j)}}\right) \end{eqnarray}\]

with $U^{obs} = \frac{U}{\sqrt{Var(U)}} \sim N(0,1)$ and¹⁰

\[\begin{eqnarray} Var(U) &=& \sum_{j=1}^r w^2(y_{(j)}) \frac{N_{(j)}}{ D_{(j)} }\\ &=& \sum_{j=1}^r w^2(y_{(j)}) \frac{ d_{(j)} \frac{R_{(j)1}}{R_{(j)} } \left( 1 - \frac{R_{(j)1}}{R_{(j)} } \right) \left( R_{(j)} - d_{(j)}\right) }{ R_{(j)} - 1 } \end{eqnarray}\]

By hand

As for the Kaplan-Meier estimator by hand, it is best to also fill in a table for the log-rank test by hand.

Let’s present the final table and comment below on how to fill it, column by column:

$j$	$y_{(j)}$	$d_{(j)1}$	$R_{(j)1}$	$d_{(j)2}$	$R_{(j)2}$	$d_{(j)}$	$R_{(j)}$	$E_{j}$	$O_{j}$	$O_{j} - E_{j}$	$N_{(j)}$	$D_{(j)}$	$N_{(j)}/D_{(j)}$
1	4.1	1	6	0	6	1	12	0.5	1	0.5	2.75	11	0.25
2	9.7	0	4	1	6	1	10	0.4	0	-0.4	2.16	9	0.24
3	10	2	4	1	5	3	9	1.333	2	0.667	4.44	8	0.555
4	17.2	1	1	0	2	1	3	0.333	1	0.667	0.44	2	0.22
5	19.7	0	0	1	2	1	2	0	0	0	0.00	1	0.00
$Total$		4		3		7		2.566		1.433			1.265

Column $j$ is the number of distinct event times. We see that there are 5 (ignoring censored observations), so we write 1 to 5 in the table.

Column $y_{(j)}$ is the ordered distinct event times:

4.1, 9.7, 10, 17.2 and 19.7

Column $d_{(j)1}$ is the number of observations for each distinct event time, for group 1:

## time
##  4.1   10 17.2 
##    1    2    1

When there is no event, we simply write 0 in the table.

Column $R_{(j)1}$ is the remaining number of patients at risk, for group 1. For this, the distribution of time (censored and not censored, for group 1) is useful:

## time
##  4.1  7.8   10 12.3 17.2 
##    1    1    2    1    1

We see that:

At the beginning, there are 6 patients
Before time 9.7, there are 4 patients left (6 - 1 who had the event at time 4.1 - 1 who was censored at time 7.8)
Before time 10, there are 4 patients left (6 - 2)
Before time 17.2, there are 1 patient left (6 - 5)
Before time 19.7, there are 0 patient left (6 - 6)

Columns $d_{(j)2}$ and $R_{(j)2}$ follow the same principle, but for group 2 this time. So we have, respectively for $d_{(j)2}$ and $R_{(j)2}$:

## time
##  9.7   10 19.7 
##    1    1    1

## time
##  9.7   10 11.1 13.1 19.7 24.1 
##    1    1    1    1    1    1

Columns $d_{(j)}$ and $R_{(j)}$ also follow the same principle, but this time considering both groups. So we have, respectively for $d_{(j)}$ and $R_{(j)}$:

## time
##  4.1  9.7   10 17.2 19.7 
##    1    1    3    1    1

## time
##  4.1  7.8  9.7   10 11.1 12.3 13.1 17.2 19.7 24.1 
##    1    1    1    3    1    1    1    1    1    1

Column $E_{j}$ is the expected number of events in the first group assuming that $h_1 \equiv h_2$. It is obtained as follows

\[ E_{j} = \frac{d_{(j)}R_{(j)1}}{R_{(j)}}\]

Column $O_{j}$ is the observed number of events in the first group, so it is equal to the $d_{(j)1}$ column.

Column $O_{j} - E_{j}$ is straightforward.

Column $N_{(j)}$ is defined as follows

\[N_{(j)} = d_{(j)} \frac{R_{(j)1}}{R_{(j)} } \left( 1 - \frac{R_{(j)1}}{R_{(j)} } \right) \left( R_{(j)} - d_{(j)}\right)\]

Column $D_{(j)}$ is $R_{(j)} - 1$.

Column $N_{(j)}/D_{(j)}$ is straightforward.

Since $w(y_{(j)}) = w^2(y_{(j)}) = 1$ for a log-rank test, we have¹¹

\[ U^{obs} = \frac{U}{\sqrt{Var(U)}} = \frac{1.434}{\sqrt{1.265}} = 1.275.\]

We reject $H_0$ if $|U^{obs}|>z_{1-\alpha/2}$, so at the 5% significance level we reject $H_0$ if $|U^{obs}|>z_{0.975}=1.96$.

We have $|U^{obs}| = 1.275 < z_{0.975}=1.96$. Hence, at the 5% significance level we do not reject $H_0$. This means that, based on the data, we are not able to conclude that survival is different between the two groups (which is equivalent than saying that we do not reject the hypothesis that survival is equal between the two groups).

If you are interested in computing the $p$-value:

$p$-value $= 2\times P(Z>1.275) = 2 \times 0.101 = 0.202 > 0.05$.

In R

We now compare our results in R with the survdiff() function:

dat <- data.frame(
  group = c(rep(1, 6), rep(2, 6)),
  time = c(4.1, 7.8, 10, 10, 12.3, 17.2, 9.7, 10, 11.1, 13.1, 19.7, 24.1),
  event = c(1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0)
)

dat

##    group time event
## 1      1  4.1     1
## 2      1  7.8     0
## 3      1 10.0     1
## 4      1 10.0     1
## 5      1 12.3     0
## 6      1 17.2     1
## 7      2  9.7     1
## 8      2 10.0     1
## 9      2 11.1     0
## 10     2 13.1     0
## 11     2 19.7     1
## 12     2 24.1     0

survdiff(Surv(time, event) ~ group,
  data = dat
)

## Call:
## survdiff(formula = Surv(time, event) ~ group, data = dat)
## 
##         N Observed Expected (O-E)^2/E (O-E)^2/V
## group=1 6        4     2.57     0.800      1.62
## group=2 6        3     4.43     0.463      1.62
## 
##  Chisq= 1.6  on 1 degrees of freedom, p= 0.2

Alternatively, we can use the ggsurvplot() function to draw the survival curves and perform the log-rank test at the same time:

fit <- survfit(Surv(time, event) ~ group, data = dat)

ggsurvplot(fit,
  pval = TRUE,
  pval.method = TRUE
)

As we can see, the $p$-values and the conclusions are the same (any difference with the results by hand is due to rounding).

As for the Kaplan-Meier estimation, we do another example on a larger dataset. Consider the data on the times until staphylococcus infection of burn patients, also available in the {KMsurv}:¹²

# load data
data(burn)

# preview data
head(burn)

##   Obs Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8 Z9 Z10 Z11 T1 D1 T2 D2 T3 D3
## 1   1  0  0  0 15  0  0  1  1  0   0   2 12  0 12  0 12  0
## 2   2  0  0  1 20  0  0  1  0  0   0   4  9  0  9  0  9  0
## 3   3  0  0  1 15  0  0  0  1  1   0   2 13  0 13  0  7  1
## 4   4  0  0  0 20  1  0  1  0  0   0   2 11  1 29  0 29  0
## 5   5  0  0  1 70  1  1  1  1  0   0   2 28  1 31  0  4  1
## 6   6  0  0  1 20  1  0  1  0  0   0   4 11  0 11  0  8  1

Using the log-rank test, we want to test the hypothesis of difference in the time to staphylococcus infection (T3 variable) between patients whose burns were cared for with a routine bathing care method (Z1 = 0) versus those whose body cleansing was initially performed using 4% chlorhexidine gluconate (Z1 = 1). The event indicator is in variable D3.

For this test, we use a two-sided alternative and a 5% significance level.

# fit
fit <- survfit(Surv(T3, D3) ~ Z1, data = burn)

# plot with log-rank test
ggsurvplot(fit,
  pval = TRUE,
  pval.method = TRUE
)

In the sample, it seems that the time to infection for patients with routine bathing (Z1 = 0) is smaller than for patients with body cleansing (Z1 = 1). This is the case because the percentage of patients who have not experienced the infection decreases more quickly, so the hazard rate is greater.

However, this conclusion cannot be generalized to the population without performing a sound statistical test. And based on the result of the log-rank test, we do not reject the hypothesis that time to infection is the same between the two groups of patients ($p$-value = 0.051).

To go further

In this article, we have presented what is survival analysis, when, why and how to use it. We discussed about censoring and survival curves. We showed how to estimate the survival function via the Kaplan-Meier estimator and how to test survival between two groups via the log-rank test. We illustrated these approaches both by hand and in R.

As you noticed, we did not show how to model survival data. There are several regression models that can be applied to survival data, the most common one being the semiparametric Cox Proportional Hazards model (1972). It originated from the medical area to investigate and assess the relationship between the survival times of patients and their corresponding predictor variables.

We have seen that the Kaplan-Meier estimator is useful to visualize survival between groups and the log-rank test to test whether survival significantly differs between groups (so both approaches use a categorical variable as predictor). However, it does not work well for assessing the effect of quantitative predictor. The Cox model has the advantage that it works for both quantitative as well as for categorical predictors, and for several risk factors at the same time (so it can model the effect of multiple variables at once).

With the Cox model, we model the impact of different factors $X_1, X_2, \ldots, X_q$ on survival via their impact on the hazard function:

\[h(t|\textbf{X}) = h_0 (t) exp(\beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_q X_q),\]

where:

$h(t|\textbf{X})$ is the instantaneous death rate conditional on having survived up to time $t$.
$h_0 (t)$ is the population-level baseline hazard – the underlying hazard function. It describes how the average person’s risk evolves over time.
$exp(\beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_q X_q)$ describes how covariates affect the hazard. In particular, a unit increase in $x_i$ leads to an increase of the hazard by a factor of $\exp(\beta_i)$.

This post aimed at presenting the introductory concepts in survival analysis, so this model will be developed in another post. In the meantime, if you would like to learn more about modeling survival data (thanks to the Cox model and other models), see this post from Joseph Rickert.

Thanks for reading.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Cox, David R. 1972. “Regression Models and Life-Tables.” Journal of the Royal Statistical Society: Series B (Methodological) 34 (2): 187–202.

Kaplan, Edward L, and Paul Meier. 1958. “Nonparametric Estimation from Incomplete Observations.” Journal of the American Statistical Association 53 (282): 457–81.

Note that in survival analysis, the precision of the estimators (and the power of the tests) does not depend on the number of patients but on the number of events. So it is best to have many observations where the event does occur for the analyses to be effective. Here we work on a small sample for the sake of illustration.↩︎
Note that the time variable can be expressed in other units, such as seconds, days, weeks, months, etc.↩︎
Median is preferred over mean in survival analysis because survival functions are often skewed to the right. The mean is often influenced by outliers, whereas the median is not. See a discussion comparing the two in this section.↩︎
Note that if the survival curve does not cross 50% (because survival is greater than 50% at the last time point), then the median survival cannot be computed and is simply undefined.↩︎
Note that the median survival is expressed in the same unit than the unit of the time variable in the initial dataset. So if the time unit was months, the median survival time would be 9 months.↩︎
More information about the dataset can be found on CRAN or with ?tongue.↩︎
See the reason we use log-log for the confidence interval in this thread.↩︎
Note that if the groups to compare have a natural ordering (such as the educational level; none, low, medium, high), tests that take it into consideration have more power to detect significant effects. These tests are referred as tests for trend.↩︎
Remember that the time unit can be different than years.↩︎
This is the case for large samples. The example described here does not meet this condition, but we still show it as an illustration.↩︎
Note that other weights can be considered, but this is beyond the scope of this article.↩︎
More information about the dataset can be found on CRAN or with ?burn.↩︎

Paper: 'EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number'

Wed, 19 Oct 2022 00:00:00 +0000

Introduction

A colleague (and friend) of mine recently published a research paper entitled “EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number” in PLoS Computational Biology.

I am not in the habit of sharing research paper to which I did not contribute. Nevertheless, I would like to make an exception with this one because I strongly believe that the method developed in the paper deserves to be known, especially for anyone working in epidemiology.

Below is the motivation behind the article, as well as an illustration on simulated and real data (US hospitalization data). More information can be found in the paper and on the accompanying website.

Motivation

EpiLPS (Gressani et al. 2022) is a methodology for flexible Bayesian inference of the time-varying reproduction number $\mathcal{R}_t$; the average number of secondary cases generated by an infected agent at time $t$. This is a key epidemiological parameter that informs about the transmission potential of an infectious disease and can be used by public health authorities to gauge the effectiveness of interventions and propose an orientation for future control strategies.

This metric has gained in popularity during the SARS-CoV-2 pandemic with wide media coverage as its meaning is easily and intuitively grasped. Put simply, when $\mathcal{R} < 1$, the signal is encouraging as the epidemic is under control and will eventually vanish. On the contrary, a value of $\mathcal{R} > 1$ means that the disease keeps spreading and infections are witnessing an expansionary impact. Having a robust and reliable tool to compute the reproduction number from infectious disease data is therefore crucial.

A group of researchers in the EpiPose team from Hasselt University (Belgium), Leiden University (The Netherlands), and the University of Bern (Switzerland) have recently developed a new methodology for estimating the instantaneous reproduction number from incidence time series data for a given serial interval distribution (the time elapsed between the onset of symptoms in an infector and the onset of symptoms of secondary cases).

They termed their approach EpiLPS for “Epidemiological modeling with Laplacian-P-Splines” as Laplace approximations and P-splines smoothers are key ingredients that form the backbone of the proposed methodology.

The EpiLPS model assumes that the observed reported cases (by reporting date or date of symptom onset) are governed by a negative binomial distribution. As such, it allows to take the feature of overdispersion into account, contrary to a Poisson model. The epidemic curve is smoothed with P-splines (where posterior estimates of latent variables are computed via Laplace approximations) in a first step and a renewal equation model is used in a second step as a bridge between the reproduction number and the estimated spline coefficients through a “plug-in” method.

The authors also explain the main difference between EpiLPS and EpiEstim, a well established approached for estimating $\mathcal{R}_t$ in real-time developed by Cori et al. (2013) and make extensive comparisons between the two approaches under different epidemic scenarios.

An interesting feature of EpiLPS is that the user can choose between a fully “sampling-free” path, where model hyperparameters are fixed at their maximum a posteriori (LPSMAP) or a fully stochastic path (LPSMALA) based on a Metropolis-adjusted Langevin algorithm (LPSMALA). Talking about efficiency, routines for Laplace approximations and B-splines evaluations have been coded in C++ and integrated in R via the Rcpp package, so that the underlying algorithm can be executed in negligible time.

Below, we provide a short example of how to use the EpiLPS routines to estimate $\mathcal{R}_t$.

Getting started

The EpiLPS package is available from CRAN (see https://cran.r-project.org/web/packages/EpiLPS/index.html) and can be installed from the R console by typing:

install.packages("EpiLPS")

The package can then be loaded as follows:

library("EpiLPS")

The EpiLPS package structure is fairly simple as it consists in a few routines:

The function epilps() is the core routine for fitting the reproduction number.
With plot.epilps(), the user can plot the estimated epidemic curve and $\mathcal{R}_t$.
Finally, two ancillary routines, episim() and perfcheck() have been developed to essentially reproduce the simulation results of the associated paper.

A simulated example

A set of epidemic data can be simulated with the episim() routine by specifying a serial interval distribution and by choosing among a set of available patterns for the true reproduction number curve (here we choose pattern number 5 corresponding to a rather wiggly curve).

The simulated outbreak is for a duration of 40 days as specified in the endepi option. By setting the option plotsim = TRUE, the routine returns a figure summarizing the incidence time series, a bar plot for the specified serial interval distribution and the true underlying reproduction number curve.

set.seed(1234)

SI <- c(0.344, 0.316, 0.168, 0.104, 0.068)
simepidemic <- episim(
  serial_interval = SI,
  Rpattern = 5,
  plotsim = TRUE,
  verbose = TRUE,
  endepi = 40
)

## Chosen scenario: 5 'Wiggly then stable Rt'.
## Incidence of cases generated from a Poisson distribution. 
## Total number of days of epidemic: 40.

If you want to have an overview of the generated incidence time series, just type:

simepidemic$y

##  [1]  10   6  15  24  37  43  54  46  47  28  20   8  10  10   3   5   3   4   6
## [20]   6  15  21  44  75 135 217 329 409 453 487 457 443 297 290 255 246 246 339
## [39] 395 573

Smoothing the epidemic curve and estimating $\mathcal{R}_t$

Let us now use the epilps() routine to smooth the epidemic curve and estimate the reproduction number.

We will do this through LPSMAP (a fully sampling-free approach) and via LPSMALA (a fully stochastic approach relying on a MCMC algorithm with Langevin dynamics), where we specify a chain of length 10000 and a burn-in of size 4000.

LPSMAP_fit <- epilps(
  incidence = simepidemic$y,
  serial_interval = SI,
  tictoc = TRUE
)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.327.
## Mean 95% CI of Rt discarding first 7 days: (1.164,1.527) 
## Elapsed real time (wall clock time): 0.261 seconds.

LPSMALA_fit <- epilps(
  incidence = simepidemic$y, serial_interval = SI,
  method = "LPSMALA", chain_length = 10000, burn = 4000
)

## Inference method chosen: LPSMALA with chain length 10000 and warmup 4000.
## MCMC acceptance rate: 56.41%. 
## Geweke z-score < 2.33 for:  32 / 33  variables. 
## Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.326.
## Mean 95% CI of Rt discarding first 7 days: (1.117,1.555). 
## Timing of routine not requested.

After execution, each routine prints in the console a brief summary of the method that has been requested by the user.

For LPSMALA, it summarizes the chain length, the acceptance rate (should be around 56%) and other basic information. As can be seen from the printed output, the mean reproduction number for the simulated epidemic is around 1.33.

We can now use, say, the LPSMALA_fit object together with the plot() routine to obtain the smoothed epidemic curve and the estimated reproduction number (by default the credible interval is at a 5% level of significance but this can be changed by the user).

days <- seq(8, 40)

#--- Smoothed epidemic curve
gridExtra::grid.arrange(
  plot(LPSMALA_fit,
    plotout = "epicurve", incibars = TRUE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "Smoothed epidemic curve", titlesize = 13, barwidth = 0.25
  ),

  #--- Estimated reproduction number
  plot(LPSMALA_fit,
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "Estimated R (LPSMALA)"
  ),
  nrow = 1, ncol = 2
)

The figure can be customized in various ways:

Users can specify the theme under themetype. Available options are gray (the default), classic, light and dark.
Other choices, such as whether or not to show the incidence bars, the color of the credible interval envelope, the color of the smoothed epidemic curve and the estimated reproduction number are also available.

The figure above was generated within the ggplot2 package, but there is also another way of extracting information directly from the LPSMAP_fit and LPSMALA_fit objects. In fact, the estimated reproduction number values and their associated credible interval for each day can be extracted and plotted.

Below, we make the exercise and plot the estimated $\mathcal{R}_t$ obtained with LPSMAP and LPSMALA, respectively and compare it with the true underlying reproduction number curve. The fit is quite good.

par(mfrow = c(1, 2))

#--- LPSMAP vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMAP_fit$epifit$R95CI_low[8:40],
    rev(LPSMAP_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(0.23, 0.54, 1, 0.3), border = NA
)
lines(days, LPSMAP_fit$epifit$R_estim[8:40], type = "l", col = "cornflowerblue", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "blue", rgb(0.23, 0.54, 1, 0.3)),
  c("Target R", "LPSMAP", "LPSMAP 95% CI"), bty = "n", cex = 0.9
)

#--- LPSMALA vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMALA_fit$epifit$R95CI_low[8:40],
    rev(LPSMALA_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(1, 0.23, 0.31, 0.3), border = NA
)
lines(days, LPSMALA_fit$epifit$R_estim[8:40], type = "l", col = "red", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "red", rgb(1, 0.23, 0.31, 0.3)),
  c("Target R", "LPSMALA", "LPSMALA 95% CI"), bty = "n", cex = 0.9
)

You can access, say, the results of the last week of the epidemic by typing:

# Estimated R of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 1:4], 7), 3)

##    Date R_estim R95CI_low R95CI_up
## 34   34   0.708     0.663    0.756
## 35   35   0.724     0.676    0.775
## 36   36   0.809     0.755    0.868
## 37   37   0.971     0.908    1.039
## 38   38   1.205     1.135    1.279
## 39   39   1.461     1.384    1.541
## 40   40   1.671     1.557    1.794

# Estimated mean number of cases of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 5:7], 7))

##    mu_estim mu95CI_low mu95CI_up
## 34      284        225       357
## 35      254        202       319
## 36      248        196       312
## 37      267        211       338
## 38      319        253       404
## 39      411        325       520
## 40      552        394       774

USA hospitalization data

To illustrate EpiLPS on real data, we download hospitalization data from the COVID19 package for the USA in the period ranging from 2021-09-01 to 2022-09-01 and apply the epilps() routine to estimate the reproduction number.

install.packages("COVID19")
library("COVID19")

# Get data and specify serial interval distribution
USADat <- COVID19::covid19(
  country = "US", level = 1, start = "2021-09-01",
  end = "2022-09-01", verbose = FALSE
)

si <- c(0.344, 0.316, 0.168, 0.104, 0.068)

inciUSA <- USADat$hosp
dateUSA <- USADat$date

We use the epilps() routine with method LPSMAP (default) and plot the smoothed epidemic curve and the estimated reproduction number with a 95% credible interval.

epifit <- epilps(incidence = inciUSA, serial_interval = si, K = 20)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 366. 
## Mean Rt discarding first 7 days: 0.994.
## Mean 95% CI of Rt discarding first 7 days: (0.983,1.005) 
## Timing of routine not requested.

gridExtra::grid.arrange(
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "epicurve", incibars = FALSE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "USA smoothed epidemic curve", titlesize = 13
  ),
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "USA Estimated R"
  ),
  nrow = 1, ncol = 2
)

Conclusion

Thanks for reading.

I hope you will find the method developed in the paper as useful as I do. Feel free to reach out to me and to the authors if you happen to use it for your own research.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Cori, Anne, Neil M Ferguson, Christophe Fraser, and Simon Cauchemez. 2013. “A New Framework and Software to Estimate Time-Varying Reproduction Numbers During Epidemics.” American Journal of Epidemiology 178 (9): 1505–12.

Gressani, Oswaldo, Jacco Wallinga, Christian L Althaus, Niel Hens, and Christel Faes. 2022. “EpiLPS: A Fast and Flexible Bayesian Tool for Estimation of the Time-Varying Reproduction Number.” PLoS Computational Biology 18 (10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618.

One-sample Wilcoxon test in R

Thu, 07 Jul 2022 00:00:00 +0000

Introduction

In a previous article, we showed how to do a two-sample Wilcoxon test in R. Remember that there are actually two versions of this test:

The Mann-Whitney-Wilcoxon test (also referred as Wilcoxon rank sum test or Mann-Whitney U test), used to compare two independent samples. This test is the non-parametric version of the Student’s t-test for independent samples.
The Wilcoxon signed-rank test (also referred as Wilcoxon test for paired samples), used to compare two paired samples. This test is the non-parametric version of the Student’s t-test for paired samples.

In another article, we also showed how to do a one-sample t-test by hand and in R. This test is used to determine whether the mean of a measurement variable is different from a specified value (a value that you specify based on your beliefs or a theoretical expectation for example). Since it is a parametric test, the data should follow a normal distribution (or sample size should be large enough (i.e., above 30), thanks to the central limit theorem) for the results to be valid.

Unlike the one-sample t-test, the one-sample Wilcoxon test (also referred as the one-sample Wilcoxon signed-rank test) is a non-parametric test, meaning that it does not rely on data belonging to any particular parametric family of probability distributions. Non-parametric tests usually have the same goal as their parametric counterparts (in this case, compare data to a given value). Nonetheless, they do not require the assumption of normality and they can deal with outliers and Likert scales.

In this article, we show when to perform the one-sample Wilcoxon test, how to do it in R and how to interpret its results. We will also briefly show some appropriate visualizations.

When?

The one-sample Wilcoxon test is used to compare our observations to a given default value—a value that you specify based on your beliefs or a theoretical expectation for example. In other words, it is used to determine if a group is significantly different from a known or hypothesized population value on the variable of interest.

Since the test statistic is computed based on the ranks of the difference between the observed values and the default value (making it a non-parametric test), the one-sample Wilcoxon test is more appropriate than a one-sample t-test when the observations do not follow a normal distribution.

The goal of this test is to verify whether the observations are significantly different from our default value. In terms of null and alternative hypotheses, we have (for a two-tailed test):

$H_0:$ location of the data is equal to the chosen value
$H_1:$ location of the data is different from the chosen value

In other words, a significant result (i.e., a rejection of the null hypothesis) suggests that the location of the data is different from the chosen value.

Note that some authors suggest that this test is a test of the median, that is (for a two-tailed test):

$H_0:$ the median is equal to the chosen value
$H_1:$ the median is different from the chosen value

However, this is the case only if the data are symmetric. Without further assumptions about the distribution of the data, the one-sample Wilcoxon test is not a test of the median but a test about the location of the data.¹

Note that although the normality assumption is not required, the independence assumption must still be verified. This means that observations must be independent of one another (usually, random sampling is sufficient to have independence).

Data

For our illustration, suppose we want to test whether the scores at an exam differ from 10, that is:

$H_0:$ scores at the exam $= 10$
$H_1:$ scores at the exam $\ne 10$

To verify this, we have a sample of 15 students and their score at the exam:

dat

##    Student_ID Score
## 1           1    17
## 2           2     5
## 3           3     1
## 4           4    10
## 5           5     4
## 6           6    18
## 7           7    17
## 8           8    15
## 9           9     7
## 10         10     4
## 11         11     5
## 12         12    14
## 13         13    20
## 14         14    18
## 15         15    15

Scores between students are assumed to be independent (a student’s score is not impacted or influenced by the score of another student). Therefore, the independence assumption is met.

Moreover, sample size is small (n < 30) and based on the histogram the data do not follow a normal distribution:

# histogram
hist(dat$Score)

Note that we refrain from verifying the normality via a normality test (such as the Shapiro-Wilk test for instance) because for small sample sizes, normality tests have little power to reject the null hypothesis and therefore small samples most often pass normality tests (Öztuna et al. 2006; Ghasemi and Zahediasl 2012).

Note also that although we use a quantitative variable for the illustration, the one-sample Wilcoxon test is also appropriate for interval data and Likert scales.

How?

The one-sample Wilcoxon test can be done in R with the wilcox.test() function.

But first, it is a good practice to visualize our data in a boxplot and compute some descriptive statistics to compare our observations with our default value:

# boxplot
boxplot(dat$Score,
  ylab = "Score"
)

If like me you prefer to use the {ggplot2} package for your plots:

# boxplot
library(ggplot2)

ggplot(dat, aes(y = Score)) +
  geom_boxplot() +
  labs(y = "Score") +
  theme( # remove axis text and ticks
    axis.text.x = element_blank(),
    axis.ticks = element_blank()
  )

Some basic descriptive statistics (rounded to two decimals):

round(summary(dat$Score),
  digits = 2
)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    5.00   14.00   11.33   17.00   20.00

From the boxplot and the descriptive statistics above, we see that the mean and median of the scores in our sample are respectively 11.33 and 14.

The one-sample Wilcoxon test will tell us whether the scores are significantly different from 10 or not (and thus whether they are different from 10 in the population or not):

wilcox.test(dat$Score,
  mu = 10 # default value
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.3779
## alternative hypothesis: true location is not equal to 10

The output presents several information such as the:

title of the test
data
test statistic
$p$-value
alternative hypothesis

We focus on the $p$-value to interpret and conclude the test.

Interpretation:

Based on the results of the test, (at the significance level of 0.05) we do not reject the null hypothesis, so we do not reject the hypothesis that the scores at this exam are equal to 10, and we cannot conclude that the scores are significantly different from 10 ($p$-value = 0.378).

By default, it is a two-tailed test that is done. As for the t.test() function, we can specify that a one-sided test is required by using either the alternative = "greater" or alternative = "less argument in the wilcox.test() function.

For example, if we want to test that the scores are higher than 10:

wilcox.test(dat$Score,
  mu = 10, # default value
  alternative = "greater" # H1: scores > 10
)

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  dat$Score
## V = 67, p-value = 0.189
## alternative hypothesis: true location is greater than 10

Interpretation:

In this case, we still do not reject the hypothesis that scores are equal to 10 and we cannot conclude that scores are significantly higher than 10 ($p$-value = 0.189).

For more information about the arguments available in the function, see ?wilcox.test.

Note that you may encounter the following warnings when using wilcox.test():

Warning messages:
1: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with ties
2: In wilcox.test.default(dat$Score, mu = 10) :
  cannot compute exact p-value with zeroes

It is a warning rather than an indication that your results are incorrect. R is informing you that it is reporting a $p$-value based on a normal approximation rather than an exact $p$-value based on the data because there are ties (some values are the same). Use the exact = FALSE option if you want to remove the warning.

Combine statistical test and plot

If you are a frequent user of the blog, you know that I like to present results of a test directly on a plot. This allows me to visualize the data and conclude the test in a concise manner.

This is possible thanks to the gghistostats() function within the {ggstatsplot} package:

# load package
library(ggstatsplot)

# combine plot and test
gghistostats(
  data = dat, # dataframe
  x = Score, # variable
  type = "nonparametric", # nonparametric = Wilcoxon, parametric = t-test
  test.value = 10 # default value
)

The histogram² shows the distribution of the scores and results of the test is shown in the title of the plot.

As you can see, results of the test are the same, that is, there is not enough evidence in the data to conclude that scores are significantly different from 10 ($p$-value = 0.378).

Conclusion

Thanks for reading.

I hope this article helped you to understand the one-sample Wilcoxon test and how to do it in R.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Ghasemi, Asghar, and Saleh Zahediasl. 2012. “Normality Tests for Statistical Analysis: A Guide for Non-Statisticians.” International Journal of Endocrinology and Metabolism 10 (2): 486.

Öztuna, Derya, Atilla Halil Elhan, and Ersöz Tüccar. 2006. “Investigation of Four Different Normality Tests in Terms of Type 1 Error Rate and Power Under Different Distributions.” Turkish Journal of Medical Sciences 36 (3): 171–76.

See more information in this article.↩︎
This histogram looks different than the previous one because the number of bins is different (4 versus 5 bins).↩︎

Koh-Lanta 2022: the ambassadors probability problem

Mon, 16 May 2022 00:00:00 +0000

Introduction

There is a popular TV show broadcasted in France and the french-speaking part of Belgium called “Koh-Lanta”.

In this show, several adventurers are dropped off on a desert island with almost no food nor equipment (just a personal backpack with their clothes and a small portion of rice). They must learn to survive on the hostile island by building their hut, finding water and food, etc.

Each season, adventurers are divided into teams (called tribes), and the teams compete against each other in games involving ability, strength, thinking and endurance. Every other game, the winning tribe receives some food or survival equipment (something to fish or something to make a fire, for instance). The losing tribe receives nothing. For the other half of the games, each member of the losing tribe has to elect an adventurer. The adventurer with the most votes leaves the show definitely. The winning tribe goes back to its island with all of its members.

At some point during the show, the two competing tribes are grouped together into one single tribe, and it continues this time with each adventurer competing against each other (so they play individually). The winner of the show is the last one to “survive”.

The 2022 season started with 24 adventurers. Just before being grouped together, each tribe has to select an adventurer in the opposing tribe. The two selected adventurers become ambassadors of their tribe. The two ambassadors must then go to another island in order to choose the adventurer who is going to leaves the show definitely. The adventurer selected by the two ambassadors will not be part of the reunification of the two tribes and her adventure stops there.

Of course, both ambassadors want to eliminate a member of the opposing tribe (to arrive at the reunification with the most allies). If the two ambassadors cannot agree, they have to play a game that will determine which of the 2 ambassadors must leave the show. This game is entirely based on luck. For the rest of the article, we call this game the ambassadors’ game.

Before 2022

Here were the rules of the ambassadors’ game before the 2022 season:

There are two identical urns, one in front of each ambassador. Each urn contains exactly one black ball and one white ball. Each ambassador has to draw a ball among the 2 from his urn. Both urns are of course closed, so no one sees which ball is picked (nor which one is not picked). The winner of the game (remember that the winner stays in the show, the loser has to leave definitely) is the one who picks a white ball while the other ambassador draws a black ball. If both ambassadors draw the same ball (both black or both white), the balls are put back in the urns and the game start over (with the exact same conditions) until the two ambassadors draw a ball of different color.

For each draw, there are thus four possible results:

Ambassador from tribe A draws a white ball and ambassador from tribe B draws a black ball: ambassador from tribe A wins.
Ambassador from tribe A draws a black ball and ambassador from tribe B draws a white ball: ambassador from tribe B wins.
Both ambassadors draw a black ball: the game start over.
Both ambassadors draw a white ball: the game start over.

Let’s compute the probability for each result to occur. Since the two events are independent (the fact that ambassador A draws a white ball does not change the probability for ambassador B to draw a white ball makes the two events independent), we can multiply the probabilities to compute the joint probability of the two events.

With $P_A$ ($P_B$) denoting the probability that ambassador from tribe A (B) draws a white ball, we have:

$P_A \cdot (1 - P_B) = 0.5 \cdot 0.5 = 0.25$
$(1 - P_A) \cdot P_B = 0.5 \cdot 0.5 = 0.25$
$(1 - P_A) \cdot (1 - P_B) = 0.5 \cdot 0.5 = 0.25$
$P_A \cdot P_B = 0.5 \cdot 0.5 = 0.25$

The sum of the 4 probabilities gives 1 (i.e., 100%), which makes sense since it covers all possible outcomes.

This means that, on the first draw, each ambassador has a probability of 25% to win the game (outcome 1 for ambassador A, outcome 2 for ambassador B).

Of course, since the game is repeated until there is a winner, probabilities for outcomes 3 and 4 tend, in the long run, to decrease until it becomes null (0%). If this statement is not straightforward to you, think about it like this: if you play that game with your friend up to 100 times, what is the probability that there is still no winner, meaning that you and your friend drew the same ball (never a different color) 100 times in a row. You conceive that it is highly unlikely.

In this context, since both urns are identical and the game is played indefinitely until there is a winner, ambassadors have exactly the same probability of winning that game. It is indeed a 50-50 chance for each of them.

In 2022

Of course, I would not write an article about that ambassadors’ game if it is was this easy and straightforward.

The 2022 season differs from the previous ones in the sense that for each game, the losing tribe receives an additional punishment (called a curse in the show). As a consequence, this year, the two urns at the ambassadors’ game were not identical.

To give you some context, the red tribe won against the yellow tribe in the last game before the ambassadors’ negotiation. The punishment for the yellow tribe was the following:

The yellow tribe (with its ambassador Colin) had an urn with 2 black balls and 1 white ball.
The red tribe (with its ambassador Louana) had an urn with 1 black ball and 1 white ball.

This is indeed a punishment for the yellow tribe since the game is not fair anymore: the ambassador of the red tribe clearly has a higher chance of winning that game compared to the ambassador of the yellow tribe.

For the curious among you, here is how it happened:

The two ambassadors were informed by the presenter Denis Brogniart about the composition of the urns for each tribe.
Knowing that the odds were not in his favour, Colin (ambassador of the yellow tribe), chose not to play the game.¹
They finally agreed on the name of the adventurer who would leave the show (an adventurer of the yellow tribe).

You guessed it by now, the reason for writing this article is of course not to explain you what actually happened—the news sites do it better and much faster than me. The reason is that I wanted to compute the chance of winning the game for each ambassador if they had not agreed on an adventurer to eliminate.²

Moreover, I wanted to compute these probabilities in R through simulation. And why not, reuse the code in case organizers of Koh-Lanta decide to change the rules again in the future.

Even if you do not watch Koh-Lanta (because it is not broadcasted in your country, or you do not like the show), it could be of interest to those of you who want to see:

how a real life example can be transferred into R,
and how a for loop and a function can be used to answer the initial question.

Probabilities computation in R

For the remaining of this article, we denote:

$p_c$, the probability that Colin (ambassador of the yellow tribe) draws a white ball,
$p_l$, the probability that Louana (ambassador of the red tribe) draws a white ball,
$q_c$, the probability that Colin draws a black ball,
$q_l$, the probability that Louana draws a black ball.

Based on the composition of the urns given by Denis Brogniart, we have:

p_c <- 1 / 3
p_l <- 1 / 2
q_c <- (1 - p_c)
q_l <- (1 - p_l)

First draw

To start easy, let’s first compute the winning probabilities for each ambassador on the first draw only. Remember that to have a winner, balls must be of different colors.

Louana wins if and only if Colin draws a black ball and Louana draws a white ball:

# Louana winning on first draw
l_win <- q_c * p_l
l_win

## [1] 0.3333333

Louana has a 33.33% chance of winning on the first draw.

On the other hand, Colin wins if and only if Louana draws a black ball and Colin draws a white ball:

# Colin winning on first draw
c_win <- q_l * p_c
c_win

## [1] 0.1666667

Colin has a 16.67% chance of winning on the first draw.

You can already see that the game is in favour of Louana, as expected. Let’s see now how it evolves when playing several times.

Second draw

To win exactly on the second draw, it must be a tie on the first draw.

# Louana winning on second draw
tie <- (p_c * p_l) + (q_c * q_l)

l_win2 <- tie * l_win
l_win2

## [1] 0.1666667

# Colin winning on second draw
c_win2 <- tie * c_win
c_win2

## [1] 0.08333333

Louana has a 16.67% chance of winning on the second draw.
Colin has a 8.33% chance of winning on the second draw.

Third draw

To win exactly on the third draw, it must be a tie on the first two draws.

# Louana winning on third draw
l_win3 <- (tie^2) * l_win
l_win3

## [1] 0.08333333

# Colin winning on third draw
c_win3 <- (tie^2) * c_win
c_win3

## [1] 0.04166667

Louana has a 8.33% chance of winning on the third draw.
Colin has a 4.17% chance of winning on the third draw.

A pattern seems to emerge in the code.

Game limited to 3 draws

We can already compute the probabilities of winning for each ambassador as if the game was limited to three draws, by summing the probabilities for each of the first three draws:

# Louana winning on draw 1, 2 or 3
l_win_tot <- l_win + l_win2 + l_win3
l_win_tot

## [1] 0.5833333

# Colin winning on draw 1, 2 or 3
c_win_tot <- c_win + c_win2 + c_win3
c_win_tot

## [1] 0.2916667

Louana has a 58.33% chance of winning if the game is limited to three draws.
Colin has a 29.17% chance of winning if the game is limited to three draws.

Game limited to 5 draws

Now if we compute the probabilities as if the game was limited to 5 draws and generalize the computation to see the pattern even more clearly, we have:

# Louana winning
l_win_tot <- ((tie^0) * l_win) +
  ((tie^1) * l_win) +
  ((tie^2) * l_win) +
  ((tie^3) * l_win) +
  ((tie^4) * l_win)

l_win_tot

## [1] 0.6458333

# Colin winning
c_win_tot <- ((tie^0) * c_win) +
  ((tie^1) * c_win) +
  ((tie^2) * c_win) +
  ((tie^3) * c_win) +
  ((tie^4) * c_win)

c_win_tot

## [1] 0.3229167

Louana has a 64.58% chance of winning if the game is limited to 5 draws.
Colin has a 32.29% chance of winning if the game is limited to 5 draws.

Game limited to 100 draws

We could continue like this for a long time, but for now let’s compute it for up to 100 draws, using a for loop. Using a for loop is necessary here in order to avoid to copy-paste our computations a hundred times.

For the ease of illustration, we compute only the probability of winning for Louana. We will show later on how the probability for Colin can easily be computed.

n_draws <- 100 # number of draws
l_win_tot <- c() # set empty vector

for (i in 1:n_draws) {
  l_win_tot[i] <- ((tie^(i - 1)) * l_win) # prob of Louana winning up to n_draws
  print(paste0("Draw ", i, ": ", sum(l_win_tot))) # print sum of winning up to n_draws
}

## [1] "Draw 1: 0.333333333333333"
## [1] "Draw 2: 0.5"
## [1] "Draw 3: 0.583333333333333"
## [1] "Draw 4: 0.625"
## [1] "Draw 5: 0.645833333333333"
## [1] "Draw 6: 0.65625"
## [1] "Draw 7: 0.661458333333333"
## [1] "Draw 8: 0.6640625"
## [1] "Draw 9: 0.665364583333333"
## [1] "Draw 10: 0.666015625"
## [1] "Draw 11: 0.666341145833333"
## [1] "Draw 12: 0.66650390625"
## [1] "Draw 13: 0.666585286458333"
## [1] "Draw 14: 0.6666259765625"
## [1] "Draw 15: 0.666646321614583"
## [1] "Draw 16: 0.666656494140625"
## [1] "Draw 17: 0.666661580403646"
## [1] "Draw 18: 0.666664123535156"
## [1] "Draw 19: 0.666665395100911"
## [1] "Draw 20: 0.666666030883789"
## [1] "Draw 21: 0.666666348775228"
## [1] "Draw 22: 0.666666507720947"
## [1] "Draw 23: 0.666666587193807"
## [1] "Draw 24: 0.666666626930237"
## [1] "Draw 25: 0.666666646798452"
## [1] "Draw 26: 0.666666656732559"
## [1] "Draw 27: 0.666666661699613"
## [1] "Draw 28: 0.66666666418314"
## [1] "Draw 29: 0.666666665424903"
## [1] "Draw 30: 0.666666666045785"
## [1] "Draw 31: 0.666666666356226"
## [1] "Draw 32: 0.666666666511446"
## [1] "Draw 33: 0.666666666589056"
## [1] "Draw 34: 0.666666666627862"
## [1] "Draw 35: 0.666666666647264"
## [1] "Draw 36: 0.666666666656965"
## [1] "Draw 37: 0.666666666661816"
## [1] "Draw 38: 0.666666666664241"
## [1] "Draw 39: 0.666666666665454"
## [1] "Draw 40: 0.66666666666606"
## [1] "Draw 41: 0.666666666666364"
## [1] "Draw 42: 0.666666666666515"
## [1] "Draw 43: 0.666666666666591"
## [1] "Draw 44: 0.666666666666629"
## [1] "Draw 45: 0.666666666666648"
## [1] "Draw 46: 0.666666666666657"
## [1] "Draw 47: 0.666666666666662"
## [1] "Draw 48: 0.666666666666664"
## [1] "Draw 49: 0.666666666666666"
## [1] "Draw 50: 0.666666666666666"
## [1] "Draw 51: 0.666666666666666"
## [1] "Draw 52: 0.666666666666667"
## [1] "Draw 53: 0.666666666666667"
## [1] "Draw 54: 0.666666666666667"
## [1] "Draw 55: 0.666666666666667"
## [1] "Draw 56: 0.666666666666667"
## [1] "Draw 57: 0.666666666666667"
## [1] "Draw 58: 0.666666666666667"
## [1] "Draw 59: 0.666666666666667"
## [1] "Draw 60: 0.666666666666667"
## [1] "Draw 61: 0.666666666666667"
## [1] "Draw 62: 0.666666666666667"
## [1] "Draw 63: 0.666666666666667"
## [1] "Draw 64: 0.666666666666667"
## [1] "Draw 65: 0.666666666666667"
## [1] "Draw 66: 0.666666666666667"
## [1] "Draw 67: 0.666666666666667"
## [1] "Draw 68: 0.666666666666667"
## [1] "Draw 69: 0.666666666666667"
## [1] "Draw 70: 0.666666666666667"
## [1] "Draw 71: 0.666666666666667"
## [1] "Draw 72: 0.666666666666667"
## [1] "Draw 73: 0.666666666666667"
## [1] "Draw 74: 0.666666666666667"
## [1] "Draw 75: 0.666666666666667"
## [1] "Draw 76: 0.666666666666667"
## [1] "Draw 77: 0.666666666666667"
## [1] "Draw 78: 0.666666666666667"
## [1] "Draw 79: 0.666666666666667"
## [1] "Draw 80: 0.666666666666667"
## [1] "Draw 81: 0.666666666666667"
## [1] "Draw 82: 0.666666666666667"
## [1] "Draw 83: 0.666666666666667"
## [1] "Draw 84: 0.666666666666667"
## [1] "Draw 85: 0.666666666666667"
## [1] "Draw 86: 0.666666666666667"
## [1] "Draw 87: 0.666666666666667"
## [1] "Draw 88: 0.666666666666667"
## [1] "Draw 89: 0.666666666666667"
## [1] "Draw 90: 0.666666666666667"
## [1] "Draw 91: 0.666666666666667"
## [1] "Draw 92: 0.666666666666667"
## [1] "Draw 93: 0.666666666666667"
## [1] "Draw 94: 0.666666666666667"
## [1] "Draw 95: 0.666666666666667"
## [1] "Draw 96: 0.666666666666667"
## [1] "Draw 97: 0.666666666666667"
## [1] "Draw 98: 0.666666666666667"
## [1] "Draw 99: 0.666666666666667"
## [1] "Draw 100: 0.666666666666667"

The probabilities found up to draw 1, 3 and 5 are coherent with what we found in the previous sections. Moreover, we see that from draw 52 onwards, the probability of Louana winning remains constant at 66.67%. This is referred as the limit.

Game limited to the number of necessary draws

Without printing the results of the above for loop, we do not know how many draws are necessary to reach the limit.

Let’s now try to include the information about the number of necessary draws in order to avoid computing unnecessary draws:

n_draws <- 9999 # initial number of draws, set intentionally to a high number
p_tie <- c() # set empty vector for prob of ties
l_win_tot <- c() # set empty vector for prob of Louana winning

# find number of necessary draws:
for (i in 1:n_draws) {
  p_tie[i] <- tie^i # prob of tie for each draw
  limit_ndraws <- sum(p_tie > 2.2e-16) # number of necessary draws
}

# compute Louana winning probabilities with the smallest number of necessary draws
for (i in 1:limit_ndraws) {
  l_win_tot[i] <- ((tie^(i - 1)) * l_win) # prob of Louana winning up to limited number of draws
  print(paste0("Draw ", i, ": ", sum(l_win_tot))) # sum of winning up to limited number of draws
}

## [1] "Draw 1: 0.333333333333333"
## [1] "Draw 2: 0.5"
## [1] "Draw 3: 0.583333333333333"
## [1] "Draw 4: 0.625"
## [1] "Draw 5: 0.645833333333333"
## [1] "Draw 6: 0.65625"
## [1] "Draw 7: 0.661458333333333"
## [1] "Draw 8: 0.6640625"
## [1] "Draw 9: 0.665364583333333"
## [1] "Draw 10: 0.666015625"
## [1] "Draw 11: 0.666341145833333"
## [1] "Draw 12: 0.66650390625"
## [1] "Draw 13: 0.666585286458333"
## [1] "Draw 14: 0.6666259765625"
## [1] "Draw 15: 0.666646321614583"
## [1] "Draw 16: 0.666656494140625"
## [1] "Draw 17: 0.666661580403646"
## [1] "Draw 18: 0.666664123535156"
## [1] "Draw 19: 0.666665395100911"
## [1] "Draw 20: 0.666666030883789"
## [1] "Draw 21: 0.666666348775228"
## [1] "Draw 22: 0.666666507720947"
## [1] "Draw 23: 0.666666587193807"
## [1] "Draw 24: 0.666666626930237"
## [1] "Draw 25: 0.666666646798452"
## [1] "Draw 26: 0.666666656732559"
## [1] "Draw 27: 0.666666661699613"
## [1] "Draw 28: 0.66666666418314"
## [1] "Draw 29: 0.666666665424903"
## [1] "Draw 30: 0.666666666045785"
## [1] "Draw 31: 0.666666666356226"
## [1] "Draw 32: 0.666666666511446"
## [1] "Draw 33: 0.666666666589056"
## [1] "Draw 34: 0.666666666627862"
## [1] "Draw 35: 0.666666666647264"
## [1] "Draw 36: 0.666666666656965"
## [1] "Draw 37: 0.666666666661816"
## [1] "Draw 38: 0.666666666664241"
## [1] "Draw 39: 0.666666666665454"
## [1] "Draw 40: 0.66666666666606"
## [1] "Draw 41: 0.666666666666364"
## [1] "Draw 42: 0.666666666666515"
## [1] "Draw 43: 0.666666666666591"
## [1] "Draw 44: 0.666666666666629"
## [1] "Draw 45: 0.666666666666648"
## [1] "Draw 46: 0.666666666666657"
## [1] "Draw 47: 0.666666666666662"
## [1] "Draw 48: 0.666666666666664"
## [1] "Draw 49: 0.666666666666666"
## [1] "Draw 50: 0.666666666666666"
## [1] "Draw 51: 0.666666666666666"
## [1] "Draw 52: 0.666666666666667"

Final winning probabilities

Remember that the initial question was:

What is the probability of winning the game for each ambassador?

The probability that Louana wins the game can easily be extracted as follows:

sum(l_win_tot)

## [1] 0.6666667

And since the game only stops when there is a winner, the sum of the winning probabilities for Louana and Colin must be equal to 1, so the probability that Colin wins the game is:

c_win_tot <- 1 - sum(l_win_tot)
c_win_tot

## [1] 0.3333333

To summarize:

Louana has 66.67% chance to win the ambassadors’ game.
Colin has 33.33% chance to win the ambassadors’ game.

Visual representations

To visualize the probabilities for each ambassador, we miss the probabilities of winning for Colin so let’s compute them first:

c_win_tot <- c() # set empty vector for prob of Colin winning

# compute Colin winning probabilities with the smallest number of necessary draws
for (i in 1:limit_ndraws) {
  c_win_tot[i] <- ((tie^(i - 1)) * c_win) # prob of Colin winning
  print(paste0("Draw ", i, ": ", sum(c_win_tot))) # print sum of winning
}

## [1] "Draw 1: 0.166666666666667"
## [1] "Draw 2: 0.25"
## [1] "Draw 3: 0.291666666666667"
## [1] "Draw 4: 0.3125"
## [1] "Draw 5: 0.322916666666667"
## [1] "Draw 6: 0.328125"
## [1] "Draw 7: 0.330729166666667"
## [1] "Draw 8: 0.33203125"
## [1] "Draw 9: 0.332682291666667"
## [1] "Draw 10: 0.3330078125"
## [1] "Draw 11: 0.333170572916667"
## [1] "Draw 12: 0.333251953125"
## [1] "Draw 13: 0.333292643229167"
## [1] "Draw 14: 0.33331298828125"
## [1] "Draw 15: 0.333323160807292"
## [1] "Draw 16: 0.333328247070312"
## [1] "Draw 17: 0.333330790201823"
## [1] "Draw 18: 0.333332061767578"
## [1] "Draw 19: 0.333332697550456"
## [1] "Draw 20: 0.333333015441895"
## [1] "Draw 21: 0.333333174387614"
## [1] "Draw 22: 0.333333253860474"
## [1] "Draw 23: 0.333333293596904"
## [1] "Draw 24: 0.333333313465118"
## [1] "Draw 25: 0.333333323399226"
## [1] "Draw 26: 0.33333332836628"
## [1] "Draw 27: 0.333333330849806"
## [1] "Draw 28: 0.33333333209157"
## [1] "Draw 29: 0.333333332712452"
## [1] "Draw 30: 0.333333333022892"
## [1] "Draw 31: 0.333333333178113"
## [1] "Draw 32: 0.333333333255723"
## [1] "Draw 33: 0.333333333294528"
## [1] "Draw 34: 0.333333333313931"
## [1] "Draw 35: 0.333333333323632"
## [1] "Draw 36: 0.333333333328483"
## [1] "Draw 37: 0.333333333330908"
## [1] "Draw 38: 0.333333333332121"
## [1] "Draw 39: 0.333333333332727"
## [1] "Draw 40: 0.33333333333303"
## [1] "Draw 41: 0.333333333333182"
## [1] "Draw 42: 0.333333333333258"
## [1] "Draw 43: 0.333333333333295"
## [1] "Draw 44: 0.333333333333314"
## [1] "Draw 45: 0.333333333333324"
## [1] "Draw 46: 0.333333333333329"
## [1] "Draw 47: 0.333333333333331"
## [1] "Draw 48: 0.333333333333332"
## [1] "Draw 49: 0.333333333333333"
## [1] "Draw 50: 0.333333333333333"
## [1] "Draw 51: 0.333333333333333"
## [1] "Draw 52: 0.333333333333333"

We create a dataset with the probabilities for both ambassadors:

dat <- data.frame(
  Draw = rep(1:limit_ndraws, 2),
  Probability = c(cumsum(l_win_tot), cumsum(c_win_tot)),
  Ambassador = c(rep("Louana", limit_ndraws), rep("Colin", limit_ndraws))
)

We can now visualize these probabilities up to the number of necessary draws:

# load package
library(ggplot2)

# plot
ggplot(dat) +
  aes(x = Draw, y = Probability, colour = Ambassador) +
  geom_line(linewidth = 2L) +
  labs(
    y = "Probability of winning",
    caption = "Source: Koh-Lanta 2022"
  ) +
  scale_y_continuous(labels = scales::percent_format(accuracy = 1), limits = c(0, 1)) +
  theme_minimal()

But I believe the most appropriate plot to answer the initial question is with a barplot:

# create dataset
dat_barplot <- data.frame(
  Ambassador = c("Louana", "Colin"),
  Probability = c(sum(l_win_tot), sum(c_win_tot))
)

# plot
ggplot(data = dat_barplot, aes(x = Ambassador, y = Probability)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  labs(
    y = "Probability of winning",
    caption = "Source: Koh-Lanta 2022"
  ) +
  scale_y_continuous(labels = scales::percent_format(accuracy = 1), limits = c(0, 1)) +
  theme_minimal()

Coded into a function

Let’s try to implement this problem in a function to be able to reuse it with other initial probabilities.

With:

p_a and p_b denoting, respectively, the probability that ambassador A and ambassador B draw a white ball,
n_draws denoting the maximum number of draws that is allowed (default = 9999),

we have the following function:

ambassadors_game <- function(p_a, p_b, n_draws = 9999) {
  q_a <- (1 - p_a)
  q_b <- (1 - p_b)

  a_win <- q_b * p_a
  b_win <- q_a * p_b

  tie <- (p_a * p_b) + (q_a * q_b)

  p_tie <- c() # set empty vector for prob of ties
  a_win_tot <- c() # set empty vector for prob of A winning
  b_win_tot <- c() # set empty vector for prob of B winning

  # find number of necessary draws:
  for (i in 1:n_draws) {
    p_tie[i] <- tie^i # prob of tie for each draw
    limit_ndraws <- sum(p_tie > 2.2e-16) # number of necessary draws
  }

  # compute A and B winning probabilities with the smallest number of necessary draws:
  for (i in 1:limit_ndraws) {
    a_win_tot[i] <- ((tie^(i - 1)) * a_win) # prob of A winning up to limited number of draws
    b_win_tot[i] <- ((tie^(i - 1)) * b_win) # prob of B winning up to limited number of draws
  }

  # save P(A), P(B) and number of necessary draws:
  res <- list(
    "p_a" = sum(a_win_tot),
    "p_b" = sum(b_win_tot),
    "ndraws" = limit_ndraws
  )
  # print results:
  return(res)
}

We test the function to see if it matches results found above.

First, if both ambassadors have identical urns as it was the case before the 2022 season:

ambassadors_game(
  p_a = 1 / 2,
  p_b = 1 / 2
)

## $p_a
## [1] 0.5
## 
## $p_b
## [1] 0.5
## 
## $ndraws
## [1] 52

The game is indeed fair, with a 50% chance of winning for each ambassador.

Second, with the urns presented to Louana and Colin, but for the first draw only (setting arbitrarily that Louana is ambassador A and Colin is ambassador B):

# first draw only
ambassadors_game(
  p_a = 1 / 2,
  p_b = 1 / 3,
  n_draws = 1
)

## $p_a
## [1] 0.3333333
## 
## $p_b
## [1] 0.1666667
## 
## $ndraws
## [1] 1

Third, still with the urns presented to Louana and Colin, but for a game limited to exactly 3 and 5 draws:

# up to 3 draws
ambassadors_game(
  p_a = 1 / 2,
  p_b = 1 / 3,
  n_draws = 3
)

## $p_a
## [1] 0.5833333
## 
## $p_b
## [1] 0.2916667
## 
## $ndraws
## [1] 3

# up to 5 draws
ambassadors_game(
  p_a = 1 / 2,
  p_b = 1 / 3,
  n_draws = 5
)

## $p_a
## [1] 0.6458333
## 
## $p_b
## [1] 0.3229167
## 
## $ndraws
## [1] 5

And now, the final verification with the real situation of Louana and Colin in Koh-Lanta 2022:

# Koh-Lanta 2022 situation
out <- ambassadors_game(
  p_a = 1 / 2,
  p_b = 1 / 3
)

out

## $p_a
## [1] 0.6666667
## 
## $p_b
## [1] 0.3333333
## 
## $ndraws
## [1] 52

All results match the ones presented above.

(Note that the probabilities for each ambassador can be extracted as follows:)

# prob ambassador A
out$p_a

## [1] 0.6666667

# prob ambassador B
out$p_b

## [1] 0.3333333

Conclusion

The initial question, coming from the television show Koh-Lanta, was “what is the probability of winning for each participant if they play the ambassadors’ game?”.

In this article, we have shown how to compute these probabilities. Moreover, we have illustrated the process of how a simple probability problem could be generalized to suit many real life situations, and how a for loop and a function could be used to implement a real life situation into R.

Last but not least, I would like to focus on something Denis Brogniart (the well-known presenter of the show) said just after the two ambassadors came back to the island to announce their choice to the other adventurers. He mentioned that, due to the punishment afflicted to the yellow tribe, there was a difference of chances of 16% between Louana and Colin.

This comes naturally from the following two situations:

First situation:

Louana has a 50% chance of picking a white ball.
Colin has a 33.33% chance of picking a white ball.
The difference is 50 - 33.33 = 16.67%, truncated to 16%.

Or, if the game is limited to the first draw only:

Second situation:

Louana has 33.33% chance of winning the game.
Colin has a 16.67% chance of winning the game.
The difference is 33.33 - 16.67 = 16.66%, truncated to 16%.

It is true that, seeing the problem from those two angles, the difference is ~16%. Denis Brogniart is right in mentioning this 16% difference.

However, the difference of chances between the two ambassadors is larger if we see the game from a broader perspective. The rules of the game say that it stops only when there is a winner. From that point of view, as demonstrated above:

Louana has 66.67% chance of winning the game.
Colin has 33.33% chance of winning the game.

In this situation, the difference in probability of winning between the 2 ambassadors is 66.67 - 33.33 = 33.34%!

I must admit that I was not expecting such a large difference between the two ambassadors. Seeing the ambassadors’ game from that perspective looks different to what we were told, or to what we thought before actually computing the probabilities. (Denis Brogniart, if you happen to read this, feel free to let me know whether this relatively large difference was intended or not.)

For those of you who do not watch the show: upon the return of the ambassadors on the island with all adventurers, Colin has been heavily criticized by the members of his tribe. His decision not to play the ambassadors’ game (and the decision to eliminate one member of his tribe with the aim of saving himself) was seen as a betrayal. As a consequence of this, he was eliminated by the reunited tribe directly after that episode.

I am going to conclude this article with the following question:

If you were an ambassador in the 2022 Koh-Lanta season and given that now you know the exact probabilites of winning for each tribe, what would you have done?

Thanks for reading.

As always, if you have any question related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

We will never know if he would have played the game if the chances of winning were equal for both ambassadors. Since I started to watch this television show, I have never seen any ambassadors’ negotiation leading to the ambassadors’ game, they all ended with the designation of an adventurer.↩︎
And to be honest, I wanted to compute these probabilities because while I was watching the show with my girlfriend, she looked at me and asked “what are the probabilities for each of them?”. I am now able to give her a precise answer.↩︎

Paper: 'Semi-Markov modeling for cancer insurance'

Wed, 06 Apr 2022 00:00:00 +0000

I am happy to announce that our paper entitled “Semi-Markov modeling for cancer insurance” has been accepted for publication in the European Actuarial Journal (Soetewey et al. 2022).

Advancements in medicine and biostatistics have already resulted in a better access to insurance for people diagnosed with cancer. This materializes into the “right to be forgotten” adopted in several EU member states, granting access to insurance after a waiting period of at most 10 years starting at the end of the successful therapeutic protocol.

This paper concentrates on insurance covers on a market where such a right has been implemented. Stand-alone products are considered, as well as guarantees included as a rider in an existing package. The cost of offering standard premium rates to all applicants in mortgage insurance related to property loans is also evaluated.

The 3-state (healthy—ill—dead) Semi-Markov hierarchical model developed in Denuit et al. (2019) for long-term care insurance is adopted here for actuarial calculations. Semi-Markov transition intensities are estimated from cancer cases recorded by the Belgian Cancer Registry. The obtained results suggest that a new offer could develop, targeting the particular needs of cancer patients.

Read more here.

Note that this paper is written jointly with Prof. Legrand and Prof. Denuit—my PhD supervisors at UCLouvain—and Dr. Silversmit from the Belgian Cancer Registry.

Thanks for reading.

I hope this paper will, to some extent, be helpful for your research.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Soetewey, Antoine, Catherine Legrand, Michel Denuit, and Geert Silversmit. 2022. “Semi-Markov Modeling for Cancer Insurance.” European Actuarial Journal, 1–25. https://doi.org/10.1007/s13385-022-00308-2.

Kruskal-Wallis test, or the nonparametric version of the ANOVA

Thu, 24 Mar 2022 00:00:00 +0000

Introduction

In a previous article, we showed how to do an ANOVA in R to compare three or more groups.

Remember that, as for many statistical tests, the one-way ANOVA requires that some assumptions are satisfied in order to be able to use and interpret the results. In particular, the ANOVA requires that residuals follow approximately a normal distribution.¹

Luckily, if the normality assumption is not satisfied, there is the nonparametric version of the ANOVA: the Kruskal-Wallis test.

In the rest of the article, we show how to perform the Kruskal-Wallis test in R and how to interpret its results. We will also briefly show how to do post-hoc tests and how to present all necessary statistical results directly on a plot.

Data

Data for the present article is based on the penguins dataset (an alternative to the well-known iris dataset), accessible via the {palmerpenguins} package:

# install.packages("palmerpenguins")
library(palmerpenguins)

The original dataset contains data for 344 penguins of 3 different species (Adelie, Chinstrap and Gentoo).

It contains 8 variables, but we focus only on the flipper length and the species for this article, so we keep only those 2 variables:

library(tidyverse)

dat <- penguins %>%
  select(species, flipper_length_mm)

(If you are unfamiliar with the pipe operator (%>%), you can also select variables with penguins[, c("species", "flipper_length_mm")]. Learn more ways to select variables in the article about data manipulation.)

It is always a good practice to do some descriptive statistics for the entire sample and by group before doing the test, so we have a broad overview of the data at hand.

# entire sample
summary(dat)

##       species    flipper_length_mm
##  Adelie   :152   Min.   :172.0    
##  Chinstrap: 68   1st Qu.:190.0    
##  Gentoo   :124   Median :197.0    
##                  Mean   :200.9    
##                  3rd Qu.:213.0    
##                  Max.   :231.0    
##                  NA's   :2

# by group
library(doBy)
summaryBy(flipper_length_mm ~ species,
  data = dat,
  FUN = median,
  na.rm = TRUE
)

## # A tibble: 3 × 2
##   species   flipper_length_mm.median
##   <fct>                        <dbl>
## 1 Adelie                         190
## 2 Chinstrap                      196
## 3 Gentoo                         216

# boxplot by species
ggplot(dat) +
  aes(x = species, y = flipper_length_mm, fill = species) +
  geom_boxplot() +
  theme(legend.position = "none")

Based on the boxplots and the summary statistics, we already see that, in our sample, penguins from the Adelie species seem to have the smallest flippers, while those from the Gentoo species seem to have the biggest flippers. However, only a sound statistical test will tell us whether we can infer this conclusion to our population.

Kruskal-Wallis test

Aim and hypotheses

As mentioned earlier, the Kruskal-Wallis test allows to compare three or more groups. More precisely, it is used to compare three or more groups in terms of a quantitative variable.

It can be seen as the extension to the Mann-Whitney test which allows to compare 2 groups under the non-normality assumption.

In the context of our example, we are going to use the Kruskal-Wallis test to help us answer the following question: “Is the length of the flippers different between the 3 species of penguins?”.

The null and alternative hypotheses of the Kruskal-Wallis test are:

$H_0$: The 3 species are equal in terms of flipper length
$H_1$: At least one species is different from the other 2 species in terms of flipper length

Be careful that, as for the ANOVA, the alternative hypothesis is not that all species are different in terms of flipper length. The opposite of all species being equal ($H_0$) is that at least one species is different from the others ($H_1$).

In this sense, if the null hypothesis is rejected, it means that at least one species is different from the other 2, but not necessarily that all 3 species are different from each other. It could be that flipper length for the species Gentoo is different than for the species Chinstrap and Adelie, but flipper length is similar between Chinstrap and Adelie. Other types of test (known as post-hoc tests and covered later) must be performed to test whether all 3 species differ.

Assumptions

First, the Kruskal-Wallis test compares several groups in terms of a quantitative variable. So there must be one quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with at least 2 levels which will determine the groups to compare).²

Second, remember that the Kruskal-Wallis test is a nonparametric test, so the normality assumption is not required. However, the independence assumption still holds.

This means that the data, collected from a representative and randomly selected portion of the total population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if three measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before, (ii) during and (iii) after a treatment), the Friedman test should be preferred in order to take into account the dependency between the samples.

Regarding the homoscedasticity (i.e., equality of the variances): As long as you use the Kruskal-Wallis test to, in fine, compare groups, homoscedasticity is not required. If you wish to compare medians, the Kruskal-Wallis test requires homoscedasticity.³

In our example, independence is assumed and we do not need to compare medians (we are only interested in comparing groups), so we can proceed to how to do the test in R. Note that the normality assumption may or may not hold, but for this article we assume it is not satisfied.

In R

The Kruskal-Wallis test in R can be done with the kruskal.test() function:

kruskal.test(flipper_length_mm ~ species,
  data = dat
)

## 
## 	Kruskal-Wallis rank sum test
## 
## data:  flipper_length_mm by species
## Kruskal-Wallis chi-squared = 244.89, df = 2, p-value < 2.2e-16

The most important result in this output is the p-value. We show how to interpret it in the next section.

Interpretations

Based on the Kruskal-Wallis test, we reject the null hypothesis and we conclude that at least one species is different in terms of flippers length (p-value < 0.001).

(For the sake of illustration, if the p-value was larger than the significance level $\alpha = 0.05$: we cannot reject the null hypothesis so we cannot reject the hypothesis that the 3 considered species of penguins are equal in terms of flippers length.)

Post-hoc tests

We have just showed that at least one species is different from the others in terms of flippers length. Nonetheless, here comes the limitations of the Kruskal-Wallis test: it does not say which group(s) is(are) different from the others.

To know this, we need to use other types of test, referred as post-hoc tests (in Latin, “after this”, so after obtaining statistically significant Kruskal-Wallis results) or multiple pairwise-comparison tests. For the interested reader, a more detailed explanation of post-hoc tests can be found here.

The most common post-hoc tests after a significant Kruskal-Wallis test are:

Dunn test
Conover test
Nemenyi test
Pairwise Wilcoxon test

The Dunn test being the most common one, here is how to do it in R.⁴

Dunn test

library(FSA)

dunnTest(flipper_length_mm ~ species,
  data = dat,
  method = "holm"
)

##           Comparison          Z      P.unadj        P.adj
## 1 Adelie - Chinstrap  -3.629336 2.841509e-04 2.841509e-04
## 2    Adelie - Gentoo -15.476612 4.990733e-54 1.497220e-53
## 3 Chinstrap - Gentoo  -8.931938 4.186100e-19 8.372200e-19

It is the last column (the adjusted p-values, adjusted for multiple comparisons) that is of interest. These p-values should be compared to your desired significance level (usually 5%).

Based on the output, we conclude that:

Adelie and Chinstrap differ significantly (p < 0.001)
Adelie and Gentoo differ significantly (p < 0.001)
Chinstrap and Gentoo differ significantly (p < 0.001)

Therefore, based on the Dunn test, we can now conclude that all 3 species differ in terms of flipper length.

Combination of statistical results and plot

A very good alternative for performing a Kruskal-Wallis and the post-hoc tests in R is with the ggbetweenstats() function from the {ggstatsplot} package:

library(ggstatsplot)

ggbetweenstats(
  data = dat,
  x = species,
  y = flipper_length_mm,
  type = "nonparametric", # ANOVA or Kruskal-Wallis
  plot.type = "box",
  pairwise.comparisons = TRUE,
  pairwise.display = "significant",
  centrality.plotting = FALSE,
  bf.message = FALSE
)

This method has the advantage that all necessary statistical results are displayed directly on the plot.

The results of the Kruskal-Wallis test are shown in the subtitle above the plot (the p-value is after p =). Moreover, the results of the post-hoc test are displayed between each group via accolades, and the boxplots allow to visualize the distribution for each species.

Summary

In this post, we reviewed the aim and hypotheses of the Kruskal-Wallis test and its underlying assumptions. We then showed how to do the test in R and how to interpret the results.

We also showed the most common post-hoc test after a significant Kruskal-Wallis test—the Dunn test.

Last but not least, we presented a concise way to present both the data by group and all the statistical results on the same plot.

Thanks for reading.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

This is the case for small sample sizes. For large sample sizes, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). See more details in this note.↩︎
Note that in theory, Kruskal-Wallis test can also be used for only two groups. However, in practice we use the Mann-Whitney test for two groups and Kruskal-Wallis for three or more groups.↩︎
See more information about the difference in this article.↩︎
Note that there are other p-value adjustment methods. See ?dunnTest for more options.↩︎

What statistical test should I do?

Thu, 02 Dec 2021 00:00:00 +0000

Introduction

Being a teaching assistant in statistics for students with diverse backgrounds, I have the chance to see what is globally not well understood by students.

I have realized that it is usually not a problem for students to do a specific statistical test when they are told which one to use (as long as they have good resources and they have been attentive during classes, of course). However, it appears that the task is much more difficult for them when they need to choose what test to do.

This article presents a flowchart to help students in selecting the most appropriate statistical test based on a couple of criteria.¹

Flowchart

Due to the large number of tests, the image is quite wide so it may not render well on all screens. In that case, you can see it in full screen by clicking directly on the image or by clicking on the following link:

Download in PDF

Notes

As you can see in the flowchart, the selection of the most appropriate test is based on:

the number of variables of interest: one, two or more than two variables
the variable type: quantitative or qualitative
in case of a qualitative variable, the number of groups and whether they are independent or paired (i.e., dependent)
whether you want the parametric or nonparametric version²

Summarizing so many tests in a single image is not an easy task. The goal of this flowchart is to provide students with a quick and easy way to select the most appropriate statistical test among the most common ones (or to see what are the alternatives).

Obviously, this flowchart is not exhaustive. There are many other tests but most of them have been omitted on purpose to keep it simple and readable. I decided to keep it simple so that the flowchart is not overwhelming, with the hope that it is still complete and precise enough for most students.

For the sake of completeness, here are a few additional remarks about this flowchart:

Tests for more than 2 variables are applicable to the case of 2 variables as well. For simplicity, I however tend to suggest the simplest test when more than one is possible. For instance, with two quantitative variables, both a correlation test and a simple linear regression can be done. In introductory statistics classes, I will most likely teach the concept of correlation but not necessarily the concept of linear regression. For this reason, I will most likely recommend a correlation test over a linear regression, unless the students have a more advanced level.
The Kolmogorov-Smirnov test, used to compare a sample with a reference probability distribution or to compare two samples, has been omitted because it is generally not taught in introductory classes. Keep in mind, however, that this test is useful both in the uni and bivariate cases.
Normality tests (such as, among others, the Shapiro-Wilk or Kolmogorov-Smirnov test) have also been omitted as they are part of another family of tests (they are used to answer the question “Is my dataset well-modeled by a normal distribution?”). Remember, nonetheless, that they are very useful to verify the normality assumption required in many hypothesis tests. For example, the Student’s t-test requires that the data follow approximately a normal distribution in case of small sample size. If this is not the case, the nonparametric version (i.e., the Wilcoxon test) should be preferred. The same goes for ANOVA and many other statistical tests.
The flowchart could be extended to include more advanced linear or non-linear models, but this is beyond its scope and goal. Remember that I created it to help non-experts to see more clearly and have a broad overview of the most common statistical tests, not to confuse them even more.
If you open the flowchart in PDF, you will be able to click on most of the tests. Clicking on the name of the test will redirect you to the corresponding article, which explains the test in further details. If the test is not clickable, it means I have not written about it yet. I will update the flowchart if I publish an article about one of the missing test.

Conclusion

Thanks for reading.

I hope this guide will help you in determining the right statistical test. Feel free to share it with all students who might be interested.

As always, if you have a question or a suggestion (for example, if I missed a test which you believe should be included), please add it as a comment so other readers can benefit from the discussion.

For those interested to draw flowcharts like this one, note that I used diagrams.net. At the time of writing this article, it is free and you can synchronize it with Google Drive, GitHub, etc.↩︎
A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. On the contrary, a nonparametric test does not rely on data belonging to any particular parametric family of probability distributions. Nonparametric tests have the same objective as their parametric counterparts. However, they have two advantages over parametric tests: (i) they do not require the assumption of normality of distributions and (ii) they can deal with outliers. The trade-off is that nonparametric tests are usually less powerful than their corresponding parametric version when the normality assumption holds. Therefore, all else being equal, with a nonparametric test you are less likely to reject the null hypothesis when it is false if the data follow a normal distribution. It is thus preferred to use the parametric version when the assumptions are met.↩︎

Multiple linear regression made simple

Mon, 04 Oct 2021 00:00:00 +0000

Introduction

Remember that descriptive statistics is a branch of statistics that allows to describe your data at hand.

Inferential statistics (with the popular hypothesis tests and confidence intervals) is another branch of statistics that allows to make inferences, that is, to draw conclusions about a population based on a sample.

The last branch of statistics is about modeling the relationship between two or more variables.¹ The most common statistical tool to describe and evaluate the link between variables is linear regression.

There are two types of linear regression:

Simple linear regression is a statistical approach that allows to assess the linear relationship between two quantitative variables. More precisely, it enables the relationship to be quantified and its significance to be evaluated.
Multiple linear regression is a generalization of simple linear regression, in the sense that this approach makes it possible to evaluate the linear relationships between a response variable (quantitative) and several explanatory variables (quantitative or qualitative).

In the real world, multiple linear regression is used more frequently than simple linear regression. This is mostly the case because:

Multiple linear regression allows to evaluate the relationship between two variables, while controlling for the effect (i.e., removing the effect) of other variables.
With data collection becoming easier, more variables can be included and taken into account when analyzing data.

Multiple linear regression being such a powerful statistical tool, I would like to present it so that everyone understands it, and perhaps even use it when deemed necessary. However, I cannot afford to write about multiple linear regression without first presenting simple linear regression.

So after a reminder about the principle and the interpretations that can be drawn from a simple linear regression, I will illustrate how to perform multiple linear regression in R. I will also show, in the context of multiple linear regression, how to interpret the output and discuss about its conditions of application. I will then conclude the article by presenting more advanced topics directly linked to linear regression.

Simple linear regression: reminder

Simple linear regression is an asymmetric procedure in which:

one of the variable is considered the response or the variable to be explained. It is also called dependent variable, and is represented on the $y$-axis
the other variable is the explanatory or also called independent variable, and is represented on the $x$-axis

Simple linear regression allows to evaluate the existence of a linear relationship between two variables and to quantify this link. Note that linearity is a strong assumption in linear regression in the sense that it tests and quantifies whether the two variables are linearly dependent.

What makes linear regression a powerful statistical tool is that it allows to quantify by what quantity the response/dependent variable varies when the explanatory/independent variable increases by one unit.

This concept is key in linear regression and helps to answer the following questions:

Is there a link between the amount spent in advertising and the sales during a certain period?
Is the number of years of schooling valued, in financial terms, in the first job?
Will an increase in tobacco taxes reduce its consumption?
What is the most likely price of an apartment, depending on the area?
Does a person’s reaction time to a stimulus depend on gender?

Simple linear regression can be seen as an extension to the analysis of variance (ANOVA) and the Student’s t-test. ANOVA and t-test allow to compare groups in terms of a quantitative variable—2 groups for t-test and 3 or more groups for ANOVA.²

For these tests, the independent variable, that is, the grouping variable forming the different groups to compare must be a qualitative variable. Linear regression is an extension because in addition to be used to compare groups, it is also used with quantitative independent variables (which is not possible with t-test and ANOVA).

In this article, we are interested in assessing whether there is a linear relationship between the distance traveled with a gallon of fuel and the weight of cars. For this example, we use the mtcars dataset (preloaded in R).

The dataset includes fuel consumption and 10 aspects of automotive design and performance for 32 automobiles:³

mpg Miles/(US) gallon (with a gallon $\approx$ 3.79 liters)
cyl Number of cylinders
disp Displacement (cu.in.)
hp Gross horsepower
drat Rear axle ratio
wt Weight (1000 lbs, with 1000 lbs $\approx$ 453.59 kg)
qsec 1/4 mile time (with 1/4 mile $\approx$ 402.34 meters)
vs Engine (0 = V-shaped, 1 = straight)
am Transmission (0 = automatic, 1 = manual)
gear Number of forward gears
carb Number of carburetors

dat <- mtcars

library(ggplot2)
ggplot(dat, aes(x = wt, y = mpg)) +
  geom_point() +
  labs(
    y = "Miles per gallon",
    x = "Car's weight (1000 lbs)"
  ) +
  theme_minimal()

The scatterplot above shows that there seems to be a negative relationship between the distance traveled with a gallon of fuel and the weight of a car. This makes sense, as the heavier the car, the more fuel it consumes and thus the fewer miles it can drive with a gallon.

This is already a good overview of the relationship between the two variables, but a simple linear regression with the miles per gallon as dependent variable and the car’s weight as independent variable goes further. It will tell us by how many miles the distance varies, on average, when the weight varies by one unit (1000 lbs in this case). This is possible thanks to the regression line.

Principle

The principle of simple linear regression is to find the line (i.e., determine its equation) which passes as close as possible to the observations, that is, the set of points formed by the pairs $(x_i, y_i)$.

In the first step, there are many potential lines. Three of them are plotted:

To find the line which passes as close as possible to all the points, we take the square of the vertical distance between each point and each potential line. Note that we take the square of the distances to make sure that a negative gap (i.e., a point below the line) is not compensated by a positive gap (i.e., a point above the line). The line which passes closest to the set of points is the one which minimizes the sum of these squared distances.

The resulting regression line is presented in blue in the following plot, and the dashed gray lines represent the vertical distance between the points and the fitted line. These vertical distances between each observed point and the fitted line determined by the least squares method are called the residuals of the linear regression model and denoted $\epsilon$.

By definition, there is no other line with a smaller total distance between the points and the line. This method is called the least squares method, or OLS for ordinary least squares.

Equation

The regression model can be written in the form of the equation:

\[Y = \beta_0 + \beta_1 X + \epsilon\]

with:

$Y$ the dependent variable
$X$ the independent variable
$\beta_0$ the intercept (the mean value of $Y$ when $x = 0$), also sometimes denoted $\alpha$
$\beta_1$ the slope (the expected increase in $Y$ when $X$ increases by one unit)
$\epsilon$ the residuals (the error term of mean 0 which describes the variations of $Y$ not captured by the model, also referred as the noise)

When we determine the line which passes closest to all the points (we say that we fit a line to the observed data), we actually estimate the unknown parameters $\beta_0$ and $\beta_1$ based on the data at hand. Remember from your geometry classes, to draw a line you only need two parameters—the intercept and the slope.

These estimates (and thus the blue line shown in the previous scatterplot) can be computed by hand with the following formulas:

\[\begin{align} \widehat\beta_1 &= \frac{\sum^n_{i = 1} (x_i - \bar{x})(y_i - \bar{y})}{\sum^n_{i = 1}(x_i - \bar{x})^2} \\ &= \frac{\left(\sum^n_{i = 1}x_iy_i\right) - n\bar{x}\bar{y}}{\sum^n_{i = 1}(x_i - \bar{x})^2} \end{align}\]

and

\[\widehat\beta_0 = \bar{y} - \widehat\beta_1 \bar{x}\]

with $\bar{x}$ and $\bar{y}$ denoting the sample mean of $x$ and $y$, respectively.

(If you struggle to compute $\widehat\beta_0$ and $\widehat\beta_1$ by hand, see this Shiny app which helps you to easily find these estimates based on your data.)

Interpretations of coefficients $\widehat\beta$

The intercept $\widehat\beta_0$ is the mean value of the dependent variable $Y$ when the independent variable $X$ takes the value 0. Its estimation has no interest in evaluating whether there is a linear relationship between two variables. It has, however, an interest if you want to know what the mean value of $Y$ could be when $x = 0$.⁴

The slope $\widehat\beta_1$, on the other hand, corresponds to the expected variation of $Y$ when $X$ varies by one unit. It tells us two important informations:

The sign of the slope indicates the direction of the line—a positive slope ($\widehat\beta_1 > 0$) indicates that there is a positive relationship between the two variables of interest (they vary in the same direction), whereas a negative slope ($\widehat\beta_1 < 0$) means that there is a negative relationship between the two variables (they vary in opposite directions).
The value of the slope provides information on the speed of evolution of the variable $Y$ as a function of the variable $X$. The larger the slope in absolute value, the larger the expected variation of $Y$ for each unit of $X$. Note, however, that a large value does not necessarily mean that the relationship is statistically significant (more on that in the section about significance of the relationship).

This is similar to the correlation coefficient, which gives information about the direction and the strength of the relationship between two variables.

To perform a linear regression in R, we use the lm() function (which stands for linear model). The function requires to set the dependent variable first then the independent variable, separated by a tilde (~).

Applied to our example of weight and car’s consumption, we have:

model <- lm(mpg ~ wt, data = dat)

The summary() function gives the results of the model:

summary(model)

## 
## Call:
## lm(formula = mpg ~ wt, data = dat)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528,	Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

In practice, we usually check the conditions of application before interpreting the coefficients (because if they are not respected, results may be biased).

In this article, however, I present the interpretations before testing the conditions because the point is to show how to interpret the results, and less about finding a valid model.

The results can be summarized as follows (see the column Estimate in the table Coefficients):

The intercept $\widehat\beta_0 =$ 37.29 indicates that, for a hypothetical car weighting 0 lbs, we can expect, on average, a consumption of 37.29 miles/gallon. This interpretation is shown for illustrative purposes, but as a car weighting 0 lbs is impossible, the interpretation has no meaning. In practice, we would therefore refrain from interpreting the intercept in this case. See another interpretation of the intercept when the independent variable is centered around its mean in this section.
The slope $\widehat\beta_1 =$ -5.34 indicates that:
- There is a negative relationship between the weight and the distance a car can drive with a gallon (this was expected given the negative trend of the points in the scatterplot shown previously).
- But more importantly, a slope of -5.34 means that, for an increase of one unit in the weight (that is, an increase of 1000 lbs), the number of miles per gallon decreases, on average, by 5.34 units. In other words, for an increase of 1000 lbs, the number of miles/gallon decreases, on average, by 5.34.

Another interpretation of the intercept

Another useful interpretation of the intercept is when the independent variable is centered around its mean. In this case, the intercept is interpreted as the mean value of $Y$ for individuals who have a value of $X$ equal to the mean of $X$.

Let’s see it in practice.

We first center the wt variable around the mean then rerun a simple linear model with this new variable:

dat_centered <- dat

dat_centered$wt_centered <- dat$wt - mean(dat$wt)

mod_centered <- lm(mpg ~ wt_centered,
  data = dat_centered
)

summary(mod_centered)

## 
## Call:
## lm(formula = mpg ~ wt_centered, data = dat_centered)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  20.0906     0.5384  37.313  < 2e-16 ***
## wt_centered  -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528,	Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

Based on the results, we see that:

The slope has not changed, the interpretation is the same than without the centering (which makes sense since the regression line has simply been shifted to the right or left).
More importantly, the intercept is now $\widehat\beta_0 =$ 20.09, so we can expect, on average, a consumption of 20.09 miles/gallon for a car with an average weight (the mean of weight is 3.22 so 3220 lbs).

This centering is particularly interesting:

when the continuous independent variable has no meaningful value of 0 (which is the case here as a car with a weight of 0 lbs is not meaningful), or
when interpreting the intercept is important.

Note that centering does not have to be done around the mean only. The independent variable can also be centered at some value that is actually in the range of the data. The exact value you center on does not matter as long it’s meaningful and within the range of data (it is not recommended to center it on a value that is not in the range of the data because we are not sure about the type of relationship between the two variables outside that range).

For our example, we may find that choosing the lowest value or the highest value of weight is the best option. So it’s up to us to decide the weight at which it’s most meaningful to interpret the intercept.

Significance of the relationship

As mentioned earlier, the value of the slope does not, by itself, make it possible to assess the significance of the linear relationship.

In other words, a slope different from 0 does not necessarily mean it is significantly different from 0, so it does not mean that there is a significant relationship between the two variables in the population. There could be a slope of 10 that is not significant, and a slope of 2 that is significant.

Significance of the relationship also depends on the variability of the slope, which is measured by its standard error and generally noted $se(\widehat\beta_1)$.

Without going too much into details, to assess the significance of the linear relationship, we divide the slope by its standard error. This ratio is the test statistic and follows a Student distribution with $n - 2$ degrees of freedom:⁵

\[T_{n - 2} = \frac{\widehat\beta_1}{se(\widehat\beta_1)}\]

For a bilateral test, the null and alternative hypotheses are:⁶

$H_0 : \beta_1 = 0$ (there is no (linear) relationship between the two variables)
$H_1 : \beta_1 \ne 0$ (there is a (linear) relationship between the two variables)

Roughly speaking, if this ratio is greater than 2 in absolute value then the slope is significantly different from 0, and therefore the relationship between the two variables is significant (and in that case it is positive or negative depending on the sign of the estimate $\widehat\beta_1$).

The standard error and the test statistic are shown in the column Std. Error and t value in the table Coefficients.

Fortunately, R gives a more precise and easier way to assess to the significance of the relationship. The information is provided in the column Pr(>|t|) of the Coefficients table. This is the p-value of the test. As for any statistical test, if the p-value is greater than or equal to the significance level (usually $\alpha = 0.05$), we do not reject the null hypothesis, and if the p-value is lower than the significance level, we reject the null hypothesis.

If we do not reject the null hypothesis, we do not reject the hypothesis of no relationship between the two variables (because we do not reject the hypothesis of a slope of 0). On the contrary, if we reject the null hypothesis of no relationship, we can conclude that there is a significant linear relationship between the two variables.

In our example, the p-value = 1.29e-10 < 0.05 so we reject the null hypothesis at the significance level $\alpha = 5\%$. We therefore conclude that there is a significant relationship between a car’s weight and its fuel consumption.

Tip: In order to make sure I interpret only parameters that are significant, I tend to first check the significance of the parameters thanks to the p-values, and then interpret the estimates accordingly. For completeness, note that the test is also performed on the intercept. The p-value being smaller than 0.05, we also conclude that the intercept is significantly different from 0.

Correlation does not imply causation

Be careful that a significant relationship between two variables does not necessarily mean that there is an influence of one variable on the other or that there is a causal effect between these two variables!

A significant relationship between $X$ and $Y$ can appear in several cases:

$X$ causes $Y$
$Y$ causes $X$
a third variable cause $X$ and $Y$
a combination of these three reasons

A statistical model alone cannot establish a causal link between two variables. Demonstrating causality between two variables is more complex and requires, among others, a specific experimental design, the repeatability of the results over time, as well as various samples.

This is the reason you will often read “Correlation does not imply causation” and linear regression follows the same principle.

Conditions of application

Unfortunately, linear regression cannot be used in all situations.

In addition to the requirement that the dependent variable must be a continuous quantitative variables, simple linear regression requires that the data satisfy the following conditions:

Linearity: The relationship between the two variables should be linear (at least roughly). For this reason it is always necessary to represent graphically the data with a scatterplot before performing a simple linear regression.⁷

Independence: Observations must be independent. It is the sampling plan and the experimental design that usually provide information on this condition. If the data come from different individuals or experimental units, they are usually independent. On the other hand, if the same individuals are measured at different periods, the data are probably not independent.
Normality of the residuals: For large sample sizes, confidence intervals and tests on the coefficients are (approximately) valid whether the error follows a normal distribution or not (a consequence of the central limit theorem, see more in Ernst and Albers (2017) and Lumley et al. (2002))! For small sample sizes, residuals should follow a normal distribution. This condition can be tested visually (via a QQ-plot and/or a histogram), or more formally (via the Shapiro-Wilk test for instance).
Homoscedasticity of the residuals: The variance of the errors should be constant. There is a lack of homoscedasticity when the dispersion of the residuals increases with the predicted values (fitted values). This condition can be tested visually (by plotting the standardized residuals vs. the fitted values) or more formally (via the Breusch-Pagan test).
No influential points: If the data contain outliers, it is essential to identify them so that they do not, on their own, influence the results of the regression. Note that an outlier is not an issue per se if the point is in the alignment of the regression line for example because it does not influence the regression line. It becomes a problem in the context of linear regression if it influences in a substantial manner the estimates (and in particular the slope of the regression line). This can be tackled by identifying outliers (via the Cook’s distance⁸ or the leverage index⁹ for instance), and comparing the results with and without the potential outliers. Do the results remain the same with the two approaches? If yes, outliers are not really an issue in this case. If results are much different, you can use the Theil-Sen estimator, robust regression or quantile regression which are all more robust to outliers.

Tip: I remember the first 4 conditions thanks to the acronym “LINE”, for Linearity, Independence, Normality and Equality of variance.

If any of the condition is not met, the tests and the conclusions could be erroneous so it is best to avoid using and interpreting the model. If this is the case, sometimes the conditions can be met by transforming the data (e.g., logarithmic transformation, square or square root, Box-Cox transformation, etc.) or by adding a quadratic or cubic (or even a higher-order polynomial) term to the model.

If it does not help, it could be worth thinking about removing some variables or adding other variables, or even considering other types of models such as non-linear models.

Keep in mind that in practice, conditions of application should be verified before drawing any conclusion based on the model. I refrain here from testing the conditions on our data because it will be covered in details in the context of multiple linear regression (see this section).

Visualizations

If you are a frequent reader of the blog, you may know that I like to draw (simple but efficient) visualizations to illustrate my statistical analyses. Linear regression is not an exception.

There are numerous ways to visualize the relationship between the two variables of interest, but the easiest one I found so far is via the visreg() function from the package of the same name:

library(visreg)
visreg(model)

I like this approach for its simplicity—only a single line of code.

However, other elements could be displayed on the regression plot (for example the regression equation and the $R^2$). This can easily be done with the stat_regline_equation() and stat_cor() functions from the {ggpubr} package:

# load necessary libraries
library(ggpubr)

# create plot with regression line, regression equation and R^2
ggplot(dat, aes(x = wt, y = mpg)) +
  geom_smooth(method = "lm") +
  geom_point() +
  stat_regline_equation(label.x = 3, label.y = 32) + # for regression equation
  stat_cor(aes(label = after_stat(rr.label)), label.x = 3, label.y = 30) + # for R^2
  theme_minimal()

Multiple linear regression

Now that you understand the principle behind simple linear regression and you know how to interpret the results, it is time to discuss about multiple linear regression.

We also start with the underlying principle of multiple linear regression, then show how to interpret the results, how to test the conditions of application and finish with more advanced topics.

Principle

Multiple linear regression is a generalization of simple linear regression, in the sense that this approach makes it possible to relate one variable with several variables through a linear function in its parameters.

Multiple linear regression is used to assess the relationship between two variables while taking into account the effect of other variables. By taking into account the effect of other variables, we cancel out the effect of these other variables in order to isolate and measure the relationship between the two variables of interest. This point is the main difference with simple linear regression.

To illustrate how to perform a multiple linear regression in R, we use the same dataset than the one used for simple linear regression (mtcars). Below a short preview:

head(dat)

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

We have seen that there is a significant and negative linear relationship between the distance a car can drive with a gallon and its weight ($\widehat\beta_1 =$ -5.34, $p$-value < 0.001).

However, one may wonder whether there are not in reality other factors that could explain a car’s fuel consumption.

To explore this, we can visualize the relationship between a car’s fuel consumption (mpg) together with its weight (wt), horsepower (hp) and displacement (disp) (engine displacement is the combined swept (or displaced) volume of air resulting from the up-and-down movement of pistons in the cylinders, usually the higher the more powerful the car):

ggplot(dat) +
  aes(x = wt, y = mpg, colour = hp, size = disp) +
  geom_point() +
  scale_color_gradient() +
  labs(
    y = "Miles per gallon",
    x = "Weight (1000 lbs)",
    color = "Horsepower",
    size = "Displacement"
  ) +
  theme_minimal()

It seems that, in addition to the negative relationship between miles per gallon and weight, there is also:

a negative relationship between miles/gallon and horsepower (lighter points, indicating more horsepower, tend to be more present in low levels of miles per gallon)
a negative relationship between miles/gallon and displacement (bigger points, indicating larger values of displacement, tend to be more present in low levels of miles per gallon).

Therefore, we would like to evaluate the relation between the fuel consumption and the weight, but this time by adding information on the horsepower and displacement. By adding this additional information, we are able to capture only the direct relationship between miles/gallon and weight (the indirect effect due to horsepower and displacement is canceled out).

This is the whole point of multiple linear regression! In fact, in multiple linear regression, the estimated relationship between the dependent variable and an explanatory variable is an adjusted relationship, that is, free of the linear effects of the other explanatory variables.

Let’s illustrate this notion of adjustment by adding both horsepower and displacement in our linear regression model:

model2 <- lm(mpg ~ wt + hp + disp,
  data = dat
)

summary(model2)

## 
## Call:
## lm(formula = mpg ~ wt + hp + disp, data = dat)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -3.891 -1.640 -0.172  1.061  5.861 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 37.105505   2.110815  17.579  < 2e-16 ***
## wt          -3.800891   1.066191  -3.565  0.00133 ** 
## hp          -0.031157   0.011436  -2.724  0.01097 *  
## disp        -0.000937   0.010350  -0.091  0.92851    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.639 on 28 degrees of freedom
## Multiple R-squared:  0.8268,	Adjusted R-squared:  0.8083 
## F-statistic: 44.57 on 3 and 28 DF,  p-value: 8.65e-11

We can see that now, the relationship between miles/gallon and weight is weaker in terms of slope ($\widehat\beta_1 =$ -3.8 now, against $\widehat\beta_1 =$ -5.34 when only the weight was considered).

The effect of weight on fuel consumption was adjusted according to the effect of horsepower and displacement. This is the remaining effect between miles/gallon and weight after the effects of horsepower and displacement have been taken into account. More detailed interpretations in this section.

Equation

Multiple linear regression models are defined by the equation

\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon\]

It is similar than the equation of simple linear regression, except that there is more than one independent variables ($X_1, X_2, \dots, X_p$).

Estimation of the parameters $\beta_0, \dots, \beta_p$ by the method of least squares is based on the same principle as that of simple linear regression, but applied to $p$ dimensions. It is thus no longer a question of finding the best line (the one which passes closest to the pairs of points ($y_i, x_i$)), but finding the $p$-dimensional plane which passes closest to the coordinate points ($y_i, x_{i1}, \dots, x_{ip}$).

This is done by minimizing the sum of the squares of the deviations of the points on the plane:

Source: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013)

Interpretations of coefficients $\widehat\beta$

The least squares method results in an adjusted estimate of the coefficients. The term adjusted means after taking into account the linear effects of the other independent variables on the dependent variable, but also on the predictor variable.

In other words, the coefficient $\beta_1$ corresponds to the slope of the relationship between $Y$ and $X_1$ when the linear effects of the other explanatory variables ($X_2, \dots, X_p$) have been removed, both at the level of the dependent variable $Y$ but also at the level of $X_1$.

Applied to our model with weight, horsepower and displacement as independent variables, we have:

summary(model2)

## 
## Call:
## lm(formula = mpg ~ wt + hp + disp, data = dat)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -3.891 -1.640 -0.172  1.061  5.861 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 37.105505   2.110815  17.579  < 2e-16 ***
## wt          -3.800891   1.066191  -3.565  0.00133 ** 
## hp          -0.031157   0.011436  -2.724  0.01097 *  
## disp        -0.000937   0.010350  -0.091  0.92851    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.639 on 28 degrees of freedom
## Multiple R-squared:  0.8268,	Adjusted R-squared:  0.8083 
## F-statistic: 44.57 on 3 and 28 DF,  p-value: 8.65e-11

The table Coefficients gives the estimate for each parameter (column Estimate), together with the $p$-value of the nullity of the parameter (column Pr(>|t|)).

The hypotheses are the same as for simple linear regression, that is:

$H_0 : \beta_j = 0$
$H_1 : \beta_j \ne 0$

The test of $\beta_j = 0$ is equivalent to testing the hypothesis: is the dependent variable associated with the independent variable studied, all other things being equal, that is to say, at constant level of the other independent variables.

In other words:

the test of $\beta_1 = 0$ corresponds to testing the hypothesis: is fuel consumption associated with a car’s weight, at a constant level of horsepower and displacement
the test of $\beta_2 = 0$ corresponds to testing the hypothesis: is fuel consumption associated with horsepower, at a constant level of weight and displacement
the test of $\beta_3 = 0$ corresponds to testing the hypothesis: is fuel consumption associated with displacement, at a constant level of weight and horsepower
(for the sake of completeness: the test of $\beta_0 = 0$ corresponds to testing the hypothesis: is miles/gallon different from 0 when weight, horsepower and displacement are equal to 0)

In practice, we usually check the conditions of application before interpreting the coefficients (because if they are not respected, results may be biased). In this article, however, I present the interpretations before testing the conditions because the point is to show how to interpret the results, and less about finding a valid model.

Based on the output of our model, we conclude that:

There is a significant and negative relationship between miles/gallon and weight, all else being equal. So for an increase of one unit in the weight (that is, an increase of 1000 lbs), the number of miles/gallon decreases, on average, by 3.8, for a constant level of horsepower and displacement ($p$-value = 0.001).
There is a significant and negative relationship between miles/gallon and horsepower, all else being equal. So for an increase of one unit of horsepower, the distance traveled with a gallon decreases, on average, by 0.03 mile, for a constant level of weight and displacement ($p$-value = 0.011).
We do not reject the hypothesis of no relationship between miles/gallon and displacement when weight and horsepower stay constant (because $p$-value = 0.929 > 0.05).
(For completeness but it should be interpreted only when it makes sense: for a weight, horsepower and displacement = 0, we can expect that a car has, on average, a fuel consumption of 37.11 miles/gallon ($p$-value < 0.001). See a more useful interpretation of the intercept when the independent variables are centered in this section.)

This is how to interpret quantitative independent variables. Interpreting qualitative independent variables is slightly different in the sense that it quantifies the effect of a level in comparison with the reference level, sill all else being equal.

So it compares the different groups (formed by the different levels of the categorical variable) in terms of the dependent variable (this is why linear regression can be seen as an extension to the t-test and ANOVA).

For the illustration, we model the fuel consumption (mpg) on the weight (wt) and the shape of the engine (vs). The variable vs has two levels: V-shaped (the reference level) and straight engine.¹⁰

## Recoding dat$vs
library(forcats)
dat$vs <- as.character(dat$vs)
dat$vs <- fct_recode(dat$vs,
  "V-shaped" = "0",
  "Straight" = "1"
)

model3 <- lm(mpg ~ wt + vs,
  data = dat
)

summary(model3)

## 
## Call:
## lm(formula = mpg ~ wt + vs, data = dat)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.7071 -2.4415 -0.3129  1.4319  6.0156 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  33.0042     2.3554  14.012 1.92e-14 ***
## wt           -4.4428     0.6134  -7.243 5.63e-08 ***
## vsStraight    3.1544     1.1907   2.649   0.0129 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.78 on 29 degrees of freedom
## Multiple R-squared:  0.801,	Adjusted R-squared:  0.7873 
## F-statistic: 58.36 on 2 and 29 DF,  p-value: 6.818e-11

Based on the output of our model, we conclude that:

For a V-shaped engine and for an increase of one unit in the weight (that is, an increase of 1000 lbs), the number of miles/gallon decreases, on average, by 4.44 ($p$-value < 0.001).
The distance traveled with a gallon of fuel increases by, on average, 3.15 miles when the engine is straight compared to a V-shaped engine, for a constant weight ($p$-value = 0.013).
(For completeness but it should be interpreted only when it makes sense: for a weight = 0 and a V-shaped engine, we can expect that the car has, on average, a fuel consumption of 33 miles/gallon ($p$-value < 0.001). See a more useful interpretation of the intercept when the independent variables are centered in this section.)

Conditions of application

As for simple linear regression, multiple linear regression requires some conditions of application for the model to be usable and the results to be interpretable. Conditions for simple linear regression also apply to multiple linear regression, that is:

Linearity of the relationships between the dependent and independent variables¹¹
Independence of the observations
Normality of the residuals
Homoscedasticity of the residuals
No influential points (outliers)

But there is one more condition for multiple linear regression:

No multicollinearity: Multicollinearity arises when there is a strong linear correlation between the independent variables, conditional on the other variables in the model. It is important to check it because it may lead to an imprecision or an instability of the estimated parameters when a variable changes. It can be assessed by studying the correlation between each pair of independent variables, or even better, by computing the variance inflation factor (VIF). The VIF measures how much the variance of an estimated regression coefficient increases, relative to a situation in which the explanatory variables are strictly independent. A high value of VIF is a sign of multicollinearity (the threshold is generally admitted at 5 or 10 depending on the domain). The easiest way to reduce the VIF is to remove some correlated independent variables, or eventually to standardize the data.

You will often see that these conditions are verified by running plot(model, which = 1:6) and it is totally correct. However, I recently discovered the check_model() function from the {performance} package which tests these conditions all at the same time (and let’s be honest, in a more elegant way).¹²

Applied on our model2 with miles/gallon as dependent variable, and weight, horsepower and displacement as independent variables, we have:

# install.packages("performance")
# install.packages("see")
library(performance)

check_model(model2)

In addition to testing all conditions at the same time, it also gives insight on how to interpret the different diagnostic plots and what you should expect (see in the subtitles of each plot).

Based on these diagnostic plots, we see that:

Homogeneity of variance (middle left plot) is respected
Multicollinearity (bottom left plot) is not an issue (I tend to use the threshold of 10 for VIF, and all of them are below 10)¹³
There is no influential points (middle right plot)
Normality of the residuals (bottom right plot) is also not perfect due to 3 points deviating from the reference line but it still seems acceptable to me. In any case, the number of observations is large enough given the number of parameters¹⁴ and given the small deviation from normality so tests on the coefficients are (approximately) valid whether the error follows a normal distribution or not
Linearity (top right plot) is not perfect so let’s check each independent variable separately:

# weight
ggplot(dat, aes(x = wt, y = mpg)) +
  geom_point() +
  theme_minimal()

# horsepower
ggplot(dat, aes(x = hp, y = mpg)) +
  geom_point() +
  theme_minimal()

# displacement
ggplot(dat, aes(x = disp, y = mpg)) +
  geom_point() +
  theme_minimal()

It seems that the relationship between miles/gallon and horsepower is not linear, which could be the main component of the slight linearity defect of the model.

To improve linearity, the variable could be removed, a transformation could be applied (logarithmic and/or squared for instance) or a quadratic or cubic (or even a higher-order polynomial) term could be added to the model.¹⁵ If this does not fix the issue of linearity, other types of models could be considered.

If you want to read more about these conditions of applications and how to deal with them, here is a very complete chapter on diagnostics for linear models written by Prof. Dustin Fife.

For the sake of easiness and for illustrative purposes, we assume linearity for the rest of the article.

When the conditions of application are met, we usually say that the model is valid. But not all valid models are good models. The next section deals with model selection.

How to choose a good linear model?

A model which satisfies the conditions of application is the minimum requirement, but you will likely find several models that meet this criteria. So one may wonder how to choose between different models that are all valid?

The three most common tools to select a good linear model are according to:

the $p$-value associated to the model,
the coefficient of determination $R^2$ and
the Akaike Information Criterion

The approaches are detailed in the next sections. Note that the first two are applicable to simple and multiple linear regression, whereas the third is only applicable to multiple linear regression.

$P$-value associated to the model

Before interpreting the estimates of a model, it is a good practice to first check the $p$-value associated to the model. This $p$-value indicates if the model is better than a model with only the intercept.

The hypotheses of the test (called F-test) are:

$H_0: \beta_1 = \beta_2 = \dots = \beta_p = 0$
$H_1:$ at least one coefficient $\beta \ne 0$

This $p$-value can be found at the bottom of the summary() output:

summary(model2)

## 
## Call:
## lm(formula = mpg ~ wt + hp + disp, data = dat)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -3.891 -1.640 -0.172  1.061  5.861 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 37.105505   2.110815  17.579  < 2e-16 ***
## wt          -3.800891   1.066191  -3.565  0.00133 ** 
## hp          -0.031157   0.011436  -2.724  0.01097 *  
## disp        -0.000937   0.010350  -0.091  0.92851    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.639 on 28 degrees of freedom
## Multiple R-squared:  0.8268,	Adjusted R-squared:  0.8083 
## F-statistic: 44.57 on 3 and 28 DF,  p-value: 8.65e-11

The $p$-value = 8.65e-11. The null hypothesis is rejected, so we conclude that our model is better than a model with only the intercept because at least one coefficient $\beta$ is significantly different from 0.

If this $p$-value > 0.05 for one of your model, it means that none of the variables you selected help in explaining the dependent variable. In other words, you should completely forget about this model because it cannot do better than simply taking the mean of the dependent variable.

Coefficient of determination $R^2$

The coefficient of determination, $R^2$, is a measure of the goodness of fit of the model. It measures the proportion of the total variability that is explained by the model, or how well the model fits the data.

$R^2$ varies between 0 and 1:

$R^2 = 0$: the model explains nothing
$R^2 = 1$: the model explains everything
$0 < R^2 < 1$: the model explains part of the variability
the higher the $R^2$, the better the model explains the dependent variable. As a rule of thumb, a $R^2 > 0.7$ indicates a good fit of the model¹⁶

Note that in a simple linear regression model, the coefficient of determination is equal to the square of the Pearson correlation coefficient:

\[R^2 = corr(X, Y)^2\]

Applied on our model2 with miles/gallon as dependent variable, and weight, horsepower and displacement as independent variables, we have:

summary(model2)

## 
## Call:
## lm(formula = mpg ~ wt + hp + disp, data = dat)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -3.891 -1.640 -0.172  1.061  5.861 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 37.105505   2.110815  17.579  < 2e-16 ***
## wt          -3.800891   1.066191  -3.565  0.00133 ** 
## hp          -0.031157   0.011436  -2.724  0.01097 *  
## disp        -0.000937   0.010350  -0.091  0.92851    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.639 on 28 degrees of freedom
## Multiple R-squared:  0.8268,	Adjusted R-squared:  0.8083 
## F-statistic: 44.57 on 3 and 28 DF,  p-value: 8.65e-11

$R^2$ is displayed at the bottom of the summary() output or can be extracted with summary(model2)$r.squared.

$R^2$ for this model is 0.8268, which means that 82.68% of the variability of the distance traveled with a gallon is explained by the weight, horsepower and displacement of the car. The relatively high $R^2$ means that the weight, horsepower and displacement of a car are good characteristics to explain the distance it can drive with a gallon of fuel.

Note that if you want to compare models with different number of independent variables, it is best to refer to the adjusted $R^2$ (= 0.8083 here).

Indeed, adding variables to the model cannot make the $R^2$ to decrease, even if the variables are not related to the dependent variables (so the $R^2$ will artificially increase when adding variables to the model, or at least stay constant). Therefore, the adjusted $R^2$ takes into account the complexity of the model (the number of variables) by penalizing for additional variables, so it is a compromise between goodness of fit and parsimony.

Parsimony

A parsimonious model (few variables) is usually preferred over a complex model (many variables). There are two ways to obtain a parsimonious model from a model with many independent variables:

We can iteratively remove the independent variable least significantly related to the dependent variable (i.e., the one with the highest $p$-value in an analysis of variance table) until all of them are significantly associated to the response variable, or
We can select the model based on the Akaike Information Criterion (AIC). AIC expresses a desire to fit the model with the smallest number of coefficients possible and allows to compare models. According to this criterion, the best model is the one with the lowest AIC. This criterion is based on a compromise between the quality of the fit and its complexity. We usually start from a global model with many independent variables, and the procedure (referred as stepwise algorithm)¹⁷ automatically compares models then selects the best one according to the AIC.

We show how to do the second option in R. For the illustration, we start with a model with all variables in the dataset as independent variables (do not forget to transform the factor variables first):

## vs has already been transformed into factor
## so only am is transformed here

## Recoding dat$vs
library(forcats)
dat$am <- as.character(dat$am)
dat$am <- fct_recode(dat$am,
  "Automatic" = "0",
  "Manual" = "1"
)

model4 <- lm(mpg ~ .,
  data = dat
)

model4 <- step(model4, trace = FALSE)

(Tip: The formula mpg ~ . is a shortcut to consider all variables present in the dataset as independent variables, except the one that has been specified as the dependent variable (mpg here)).

The model that has been selected according to this criterion is the following:

summary(model4)

## 
## Call:
## lm(formula = mpg ~ wt + qsec + am, data = dat)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.4811 -1.5555 -0.7257  1.4110  4.6610 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   9.6178     6.9596   1.382 0.177915    
## wt           -3.9165     0.7112  -5.507 6.95e-06 ***
## qsec          1.2259     0.2887   4.247 0.000216 ***
## amManual      2.9358     1.4109   2.081 0.046716 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.459 on 28 degrees of freedom
## Multiple R-squared:  0.8497,	Adjusted R-squared:  0.8336 
## F-statistic: 52.75 on 3 and 28 DF,  p-value: 1.21e-11

Be careful when using an automatic procedure because, even though it is the best model that is selected, it is based:

on a single criterion (AIC in this case), but more importantly;
it is based on some set of mathematical rules, which means that industry knowledge or human expertise is not taken into consideration.

I believe that this kind of automatic procedure for model’s selection is a good starting point, but I also believe that the final model should always be checked and tested against other models to make sure it makes sense in practice (apply common sense).

Last but not least, do not forget to also verify the conditions of application because the stepwise procedure does not guarantee that they are respected.

Visualizations

There are many ways to visualize results of a linear regression. The easiest ones I am aware of are:

visreg() illustrates the relationships between the dependent and independent variables in different plots (one for each independent variable unless you specify which relationship you want to illustrate):

library(visreg)

visreg(model4)

ggcoefstats() illustrates the results in one single plot, with many statistical details:

library(ggstatsplot)

ggcoefstats(model4)

In this plot:

when the solid line does not cross the vertical dashed line, the estimates is significantly different from 0 at the 5% significance level (i.e., $p$-value < 0.05)
furthermore, a point to the right (left) of the vertical dashed line means that there is a positive (negative) relationship between the two variables
the more extreme the point, the stronger the relationship

plot_summs() which also illustrates the results but in a more concise way:

library(jtools)
library(ggstance)

plot_summs(model4,
  omit.coefs = NULL
)

The advantage of this approach is that it is possible to compare coefficients of multiple models simultaneously (particularly interesting when the models are nested):

model4bis <- lm(mpg ~ wt + qsec + am + hp,
  data = dat
)

plot_summs(model4,
  model4bis,
  omit.coefs = NULL
)

To go further

Below some more advanced topics related to linear regression. Feel free to comment at the end of the article if you believe I missed an important one.

Print model’s parameters

Thanks to the model_parameters() function from the {parameters} package, you can print a summary of the model in a nicely formatted way to make the output more readable:

library(parameters)

model_parameters(model4, summary = TRUE)

## Parameter   | Coefficient |   SE |         95% CI | t(28) |      p
## ------------------------------------------------------------------
## (Intercept) |        9.62 | 6.96 | [-4.64, 23.87] |  1.38 | 0.178 
## wt          |       -3.92 | 0.71 | [-5.37, -2.46] | -5.51 | < .001
## qsec        |        1.23 | 0.29 | [ 0.63,  1.82] |  4.25 | < .001
## am [Manual] |        2.94 | 1.41 | [ 0.05,  5.83] |  2.08 | 0.047

And if you are using R Markdown, you can use the print_html() function to get a compact and yet comprehensive summary table in your HTML file:

library(gt)

print_html(model_parameters(model4, summary = TRUE))

Parameter	Coefficient	SE	95% CI	t(28)	p
(Intercept)	9.62	6.96	(-4.64, 23.87)	1.38	0.178
wt	-3.92	0.71	(-5.37, -2.46)	-5.51	< .001
qsec	1.23	0.29	(0.63, 1.82)	4.25	< .001
am (Manual)	2.94	1.41	(0.05, 5.83)	2.08	0.047

Automatic reporting

The report() function from the package of the same name allows to automatically produces reports of models according to best practices guidelines:

library(report)

report(model4)[1]

## [1] "We fitted a linear model (estimated using OLS) to predict mpg with wt, qsec and am (formula: mpg ~ wt + qsec + am). The model explains a statistically significant and substantial proportion of variance (R2 = 0.85, F(3, 28) = 52.75, p < .001, adj. R2 = 0.83). The model's intercept, corresponding to wt = 0, qsec = 0 and am = Automatic, is at 9.62 (95% CI [-4.64, 23.87], t(28) = 1.38, p = 0.178). Within this model:\n\n  - The effect of wt is statistically significant and negative (beta = -3.92, 95% CI [-5.37, -2.46], t(28) = -5.51, p < .001; Std. beta = -0.64, 95% CI [-0.87, -0.40])\n  - The effect of qsec is statistically significant and positive (beta = 1.23, 95% CI [0.63, 1.82], t(28) = 4.25, p < .001; Std. beta = 0.36, 95% CI [0.19, 0.54])\n  - The effect of am [Manual] is statistically significant and positive (beta = 2.94, 95% CI [0.05, 5.83], t(28) = 2.08, p = 0.047; Std. beta = 0.49, 95% CI [7.59e-03, 0.97])\n\nStandardized parameters were obtained by fitting the model on a standardized version of the dataset. 95% Confidence Intervals (CIs) and p-values were computed using a Wald t-distribution approximation."

Note that the function also works for dataframes, statistical tests and other models.

Predictions

Linear regression is also very often used for predictive purposes. Confidence and prediction intervals for new data can be computed with the predict() function.

Suppose we want to predict the miles/gallon for a car with a manual transmission, weighting 3000 lbs and which drives a quarter of a mile (qsec) in 18 seconds:

# confidence interval for new data
predict(model4,
  new = data.frame(wt = 3, qsec = 18, am = "Manual"),
  interval = "confidence",
  level = .95
)

##        fit      lwr    upr
## 1 22.87005 21.09811 24.642

# prediction interval for new data
predict(model4,
  new = data.frame(wt = 3, qsec = 18, am = "Manual"),
  interval = "prediction",
  level = .95
)

##        fit      lwr      upr
## 1 22.87005 17.53074 28.20937

Based on our model, it is expected that this car will drive 22.87 miles with a gallon.

The difference between the confidence and prediction interval is that:

a confidence interval gives the predicted value for the mean of $Y$ for a new observation, whereas
a prediction interval gives the predicted value for an individual $Y$ for a new observation.

The prediction interval is wider than the confidence interval to account for the additional uncertainty due to predicting an individual response, and not the mean, for a given value of $X$.

Linear hypothesis tests

Linear hypothesis tests make it possible to generalize the F-test mentioned in this section, while offering the possibility to perform either tests of comparison of coefficients, or tests of equality of linear combinations of coefficients.

For example, to test the linear constraint:

$H_0: \beta_1 = \beta_2 = 0$
$H_1:$ not $H_0$

we use the linearHypothesis() function of the {car} package as follows:

library(car)
linearHypothesis(model4, c("wt = 0", "qsec = 0"))

## 
## Linear hypothesis test:
## wt = 0
## qsec = 0
## 
## Model 1: restricted model
## Model 2: mpg ~ wt + qsec + am
## 
##   Res.Df    RSS Df Sum of Sq      F   Pr(>F)    
## 1     30 720.90                                 
## 2     28 169.29  2    551.61 45.618 1.55e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We reject the null hypothesis and we conclude that at least one of $\beta_1$ and $\beta_2$ is different from 0 ($p$-value = 1.55e-09).

Overall effect of categorical variables

When the independent variables are categorical with $k$ categories, the regression table provides $k-1$ $p$-values:

model5 <- lm(mpg ~ vs + am + as.factor(cyl),
  data = dat
)

summary(model5)

## 
## Call:
## lm(formula = mpg ~ vs + am + as.factor(cyl), data = dat)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.2821 -1.4402  0.0391  1.8845  6.2179 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       22.809      2.928   7.789 2.24e-08 ***
## vsStraight         1.708      2.235   0.764  0.45135    
## amManual           3.165      1.528   2.071  0.04805 *  
## as.factor(cyl)6   -5.399      1.837  -2.938  0.00668 ** 
## as.factor(cyl)8   -8.161      2.892  -2.822  0.00884 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.097 on 27 degrees of freedom
## Multiple R-squared:  0.7701,	Adjusted R-squared:  0.736 
## F-statistic: 22.61 on 4 and 27 DF,  p-value: 2.741e-08

The variables vs and am have 2 levels so one is displayed in the regression output. The variable cyl has 3 levels (4, 6 and 8) so 2 of them are displayed. The overall effect of vs and am are reported in the Pr(>|t|) column, but not the overall effect of cyl because there are more than 2 levels for this variable.

To get the $p$-value of the overall effect of a categorical variable, we need to get an analysis of variance table via the Anova() function from the {car} package:¹⁸

library(car)
Anova(model5)

## Anova Table (Type II tests)
## 
## Response: mpg
##                 Sum Sq Df F value  Pr(>F)  
## vs               5.601  1  0.5841 0.45135  
## am              41.122  1  4.2886 0.04805 *
## as.factor(cyl)  94.591  2  4.9324 0.01493 *
## Residuals      258.895 27                  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

From this analysis of variance table, we conclude that:

vs is not significantly associated with mpg ($p$-value = 0.451)
am and cyl are significantly associated with mpg ($p$-values < 0.05)

Interaction

So far we have covered multiple linear regression without any interaction.

There is an interaction effect between factors A and B if the effect of factor A on the response depends on the level taken by factor B.

In R, interaction can be added as follows:

model6 <- lm(mpg ~ wt + am + wt:am,
  data = dat
)

# Or in a shorter way:
model6 <- lm(mpg ~ wt * am,
  data = dat
)

summary(model6)

## 
## Call:
## lm(formula = mpg ~ wt * am, data = dat)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.6004 -1.5446 -0.5325  0.9012  6.0909 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  31.4161     3.0201  10.402 4.00e-11 ***
## wt           -3.7859     0.7856  -4.819 4.55e-05 ***
## amManual     14.8784     4.2640   3.489  0.00162 ** 
## wt:amManual  -5.2984     1.4447  -3.667  0.00102 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.591 on 28 degrees of freedom
## Multiple R-squared:  0.833,	Adjusted R-squared:  0.8151 
## F-statistic: 46.57 on 3 and 28 DF,  p-value: 5.209e-11

From the output we conclude that there is an interaction between the weight and the transmission ($p$-value = 0.00102). This means that the effect of the weight on the distance traveled with a gallon depends on the transmission type.

The easiest way to handle interaction is to visualize the relationship for each level of the categorical variable:

visreg(model6, "wt", by = "am")

We see that the relationship between weight and miles/gallon is stronger (the slope is steeper) for cars with a manual transmission compared to cars with an automatic transmission.

This is a good example to illustrate the point that when studying a relationship between two variables, say $X$ and $Y$, if one also has data for other variables which are potentially associated with both $X$ and $Y$, it is important to include them in the regression and to analyze the relationship conditionally on these variables.

Omitting some variables that should be included in the model may lead to erroneous and misleading conclusions, up to the point that the relationship is completely reversed (a phenomenon referred as Simpson’s paradox).

Summary

In this article, we started with a reminder of simple linear regression and in particular its principle and how to interpret the results.

This laid the foundations for a better understanding of multiple linear regression. After explaining its principle, we showed how to interpret the output and how to choose a good linear model. We then mentioned a couple of visualizations and finished the article with some more advanced topics.

Thanks for reading.

I hope this article helped you to understand better linear regression and gave you the confidence to do your own linear regressions in R. If you need to model a binary variable instead of a quantitative continuous variable, see how to perform a binary logistic regression in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Austin, Peter C, and Ewout W Steyerberg. 2015. “The Number of Subjects Per Variable Required in Linear Regression Analyses.” Journal of Clinical Epidemiology 68 (6): 627–36.

Ernst, Anja F, and Casper J Albers. 2017. “Regression Assumptions in Clinical Psychology Research Practice?a Systematic Review of Common Misconceptions.” PeerJ 5: e3323.

James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning. Vol. 112. Springer.

Lumley, Thomas, Paula Diehr, Scott Emerson, and Lu Chen. 2002. “The Importance of the Normality Assumption in Large Public Health Data Sets.” Annual Review of Public Health 23 (1): 151–69.

Schmidt, Amand F, and Chris Finan. 2018. “Linear Regression and the Normality Assumption.” Journal of Clinical Epidemiology 98: 146–51.

Some people see regression analysis as a part of inferential statistics. It is true, as a sample is taken to evaluate the link between two or more variables in a population of interest. I tend to distinguish regression from inferential statistics for the simple reasons that (i) regressions are often used to a broader extent (for predictive analyses, among others), and because (ii) the main goal of linear regression (see this section) differs from the objectives of confidence intervals and hypothesis testing well known in the field of inferential statistics.↩︎
Formally, ANOVA can also be used to compare 2 groups, but in practice we tend to use it for 3 or more groups, leaving the t-test for 2 groups.↩︎
More information about the dataset can be found by executing ?mtcars.↩︎
Note that it best to avoid interpreting the intercept when $X$ cannot be equal to 0 or when it makes no sense in practice.↩︎
$n$ is the number of observations.↩︎
Other values than 0 are accepted as well. In that case, the test statistic becomes $T_{n - 2} = \frac{\widehat\beta - a}{se(\widehat\beta_1)}$ where $a$ is the hypothesized slope.↩︎
Note that linearity can be checked with a scatterplot of the two variables, or via a scatterplot of the residuals and the fitted values. See more about this in this section.↩︎
An observation is considered as an outlier based on the Cook’s distance if its value is > 1.↩︎
An observation has a high leverage value (and thus needs to be investigated) if it is greater than $2p/n$, where $p$ is the number of parameters in the model (intercept included) and $n$ is the number of observations.↩︎
You can always change the reference level with the relevel() function. See more data manipulation techniques.↩︎
Note that linearity can also be tested with a scatterplot of the residuals and the fitted values.↩︎
After installing the {performance} package, you will also need to install the {see} package manually. See how to install a R package if you need more help.↩︎
I use the threshold of 10 because, as shown by James et al. (2013), a value between 5 and 10 indicates a moderate correlation, while VIF values greater than 10 indicate a high and non-tolerable correlation.↩︎
Austin and Steyerberg (2015) showed that two subjects per variable tends to permit accurate estimation of regression coefficients in a linear regression model estimated using ordinary least squares. Moreover, the general rule of thumb says that there should be at least 10 observations per variable (Schmidt and Finan 2018). Our dataset contains 32 observations, above the minimum of 10 subjects per variable.↩︎
If you apply a logarithmic transformation, see two guides on how to interpret the results: in English and in French.↩︎
Note that a high $R^2$ does not guarantee that you selected the best variables or that your model is good. It simply tells that the model fits the data quite well. It is advised to apply common sense when comparing models and not only refer to $R^2$ (in particular when $R^2$ are close).↩︎
There are two main methods; backward and forward. The backward method consists in starting from the model containing all the explanatory variables likely to be relevant, then recursively removing the variable which reduces the information criterion of the model, until no reduction is possible. The forward method is the reverse of the backward method in the sense that we start from a one-variable model with the lowest information criterion and at each step, an explanatory variable is added. By default, the step() function in R combines the backward and forward methods.↩︎
To not be confused with the anova() function because it provides results that depend on the order in which the variables appear in the model.↩︎

Hypothesis test by hand

Wed, 27 Jan 2021 00:00:00 +0000

Descriptive versus inferential statistics

Remember that descriptive statistics is the branch of statistics aiming at describing and summarizing a set of data in the best possible manner, that is, by reducing it down to a few meaningful key measures and visualizations—with as little loss of information as possible. In other words, the branch of descriptive statistics helps to have a better understanding and a clear image about a set of observations thanks to summary statistics and graphics. With descriptive statistics, there is no uncertainty because we describe only the group of observations that we decided to work on and no attempt is made to generalize the observed characteristics to another or to a larger group of observations.

Inferential statistics, one the other hand, is the branch of statistics that uses a random sample of data taken from a population to make inferences, i.e., to draw conclusions about the population of interest (see the difference between population and sample if you need a refresh of the two concepts). In other words, information from the sample is used to make generalizations about the parameter of interest in the population.

The two most important tools used in the domain of inferential statistics are:

hypothesis test (which is the main subject of the present article), and
confidence interval (which is briefly discussed in this section)

Motivations and limitations

Via my teaching tasks, I realized that many students (especially in introductory statistic classes) struggle to perform hypothesis tests and interpret the results. It seems to me that these students often encounter difficulties mainly because hypothesis testing is rather unclear and abstract to them.

One of the reason it looks abstract to them is because they do not understand the final goal of hypothesis testing—the “why” behind this tool. They often do inferential statistics without understanding the reasoning behind it, as if they were following a cooking recipe which does not require any thinking. However, as soon as they understand the principle underlying hypothesis testing, it is much easier for them to apply the concepts and solve the exercises.

For this reason, I though it would be useful to write an article on the goal of hypothesis tests (the “why?”), in which context they should be used (the “when?”), how they work (the “how?”) and how to interpret the results (the “so what?”). Like anything else in statistics, it becomes much easier to apply a concept in practice when we understand what we are testing or what we are trying to demonstrate beforehand.

In this article, I present—as comprehensibly as possible—the different steps required to perform and conclude a hypothesis test by hand.

These steps are illustrated with a basic example. This will build the theoretical foundations of hypothesis testing, which will in turn be of great help for the understanding of most statistical tests.

Hypothesis tests come in many forms and can be used for many parameters or research questions. The steps I present in this article are not applicable to all hypothesis test, unfortunately.

They are however, appropriate for at least the most common hypothesis tests—the tests on:

One mean: $\mu$
Two means:
- independent samples: $\mu_1$ and $\mu_2$
- paired samples: $\mu_D$
One proportion: $p$
Two proportions: $p_1$ and $p_2$
One variance: $\sigma^2$
Two variances: $\sigma^2_1$ and $\sigma^2_2$

The good news is that the principles behind these 6 statistical tests (and many more) are exactly the same. So if you understand the intuition and the process for one of them, all others pretty much follow.

Hypothesis test

Why?

Unlike descriptive statistics where we only describe the data at hand, hypothesis tests use a subset of observations, referred as a sample, to draw conclusions about a population.

One may wonder why we would try to “guess” or make inference about a parameter of a population based on a sample, instead of simply collecting data for the entire population, compute statistics we are interested in and take decisions based upon that.

The main reason we actually use a sample instead of the entire population is because, most of the time, collecting data on the entire population is practically impossible, too complex, too expensive, it would take too long, or a combination of any of these.¹

So the overall objective of a hypothesis test is to draw conclusions in order to confirm or refute a belief about a population, based on a smaller group of observations.

In practice, we take some measurements of the variable of interest—representing the sample(s)—and we check whether our measurements are likely or not given our assumption (our belief). Based on the probability of observing the sample(s) we have, we decide whether we can trust our belief or not.

When?

Hypothesis tests have many practical applications.

Here are different situations illustrating when the 6 tests mentioned above would be appropriate:

One mean: suppose that a health professional would like to test whether the mean weight of Belgian adults is different than 80 kg (176.4 lbs).
Two means:
- Independent samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) for patients in a control group and patients in a treatment group, where patients in the two groups are different.
- Paired samples: suppose that a physiotherapist would like to test the effectiveness of a new treatment by measuring the mean response time (in seconds) before and after a treatment, where patients are measured twice—before and after treatment, so patients are the same in the 2 samples.
One proportion: suppose that a political pundit would like to test whether the proportion of citizens who are going to vote for a specific candidate is smaller than 30%.
Two proportions: suppose that a doctor would like to test whether the proportion of smokers is different between professional and amateur athletes.
One variance: suppose that an engineer would like to test whether a voltmeter has a lower variability than what is imposed by the safety standards.
Two variances: suppose that, in a factory, two production lines work independently from each other. The financial manager would like to test whether the costs of the weekly maintenance of these two machines have the same variance. Note that a test on two variances is also often performed to verify the assumption of equal variances, which is required for several other statistical tests, such as the Student’s t-test for instance.

Of course, this is a non-exhaustive list of potential applications and many research questions can be answered thanks to a hypothesis test.

One important point to remember is that in hypothesis testing we are always interested in the population and not in the sample. The sample is used for the aim of drawing conclusions about the population, so we always test in terms of the population.

Usually, hypothesis tests are used to answer research questions in confirmatory analyses. Confirmatory analyses refer to statistical analyses where hypotheses—deducted from theory—are defined beforehand (preferably before data collection). In this approach, the researcher has a specific idea about the variables under consideration and she is trying to see if her idea, specified as hypotheses, is supported by data.

On the other hand, hypothesis tests are rarely used in exploratory analyses.² Exploratory analyses aims to uncover possible relationships between the variables under investigation. In this approach, the researcher does not have any clear theory-driven assumptions or ideas in mind before data collection. This is the reason exploratory analyses are sometimes referred as hypothesis-generating analyses—they are used to create some hypotheses, which in turn may be tested via confirmatory analyses at a later stage.

How?

There are, to my knowledge, 3 different methods to perform a hypothesis tests:

Although the process for these 3 approaches may slightly differ, they all lead to the exact same conclusions. Using one method or another is, therefore, more often than not a matter of personal choice or a matter of context. See this section to know which method I use depending on the context.

I present the 3 methods in the following sections, starting with, in my opinion, the most comprehensive one when it comes to doing it by hand: comparing the test statistic with the critical value.

For the three methods, I will explain the required steps to perform a hypothesis test from a general point of view and illustrate them with the following situation:³

Suppose a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg.

Note that, as for most hypothesis tests, the test we are going to use as example below requires some assumptions. Since the aim of the present article is to explain a hypothesis test, we assume that all assumptions are met. For the interested reader, see the assumptions (and how to verify them) for this type of hypothesis test in the article presenting the one-sample t-test.

Method A: Comparing the test statistic with the critical value

Method A, which consists in comparing the test statistic with the critical value, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Finding the critical value
Concluding and interpreting the results

Each step is detailed below.

Step #1: Stating the null and alternative hypothesis

As discussed before, a hypothesis test first requires an idea, that is, an assumption about a phenomenon. This assumption, referred as hypothesis, is derived from the theory and/or the research question.

Since a hypothesis test is used to confirm or refute a prior belief, we need to formulate our belief so that there is a null and an alternative hypothesis. Those hypotheses must be mutually exclusive, which means that they cannot be true at the same time. This is step #1.

In the context of our scenario, the null and alternative hypothesis are thus:

Null hypothesis $H_0: \mu = 80$
Alternative hypothesis $H_1: \mu \ne 80$

When stating the null and alternative hypothesis, bear in mind the following three points:

We are always interested in the population and not in the sample. This is the reason $H_0$ and $H_1$ will always be written in terms of the population and not in terms of the sample (in this case, $\mu$ and not $\bar{x}$).
The assumption we would like to test is often the alternative hypothesis. If the researcher wanted to test whether the mean weight of Belgian adults was less than 80 kg, she would have stated $H_0: \mu = 80$ (or equivalently, $H_0: \mu \ge 80$) and $H_1: \mu < 80$.⁴ Do not mix the null with the alternative hypothesis, or the conclusions will be diametrically opposed!
The null hypothesis is often the status quo. For instance, suppose that a doctor wants to test whether the new treatment A is more efficient than the old treatment B. The status quo is that the new and old treatments are equally efficient. Assuming a larger value is better, she will then write $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A > \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B > 0$). On the opposite, if the lower the better, she would have written $H_0: \mu_A = \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B = 0$) and $H_1: \mu_A < \mu_B$ (or equivalently, $H_0: \mu_A - \mu_B < 0$).

Step #2: Computing the test statistic

The test statistic (often called t-stat) is, in some sense, a metric indicating how extreme the observations are compared to the null hypothesis. The higher the t-stat (in absolute value), the more extreme the observations are.

There are several formulas to compute the t-stat, with one formula for each type of hypothesis test—one or two means, one or two proportions, one or two variances. This means that there is a formula to compute the t-stat for a hypothesis test on one mean, another formula for a test on two means, another for a test on one proportion, etc.⁵

The only difficulty in this second step is to choose the appropriate formula. As soon as you know which formula to use based on the type of test, you simply have to apply it to the data. For the interested reader, see the different formulas to compute the t-stat for the most common tests in this Shiny app.

Luckily, formulas for hypothesis tests on one and two means, and one and two proportions follow the same structure.

Computing the test statistic for these tests is similar than scaling a random variable (a process also knows as “standardization” or “normalization”) which consists in subtracting the mean from that random variable, and dividing the result by the standard deviation:

\[Z = \frac{X - \mu}{\sigma}\]

For these 4 hypothesis tests (one/two means and one/two proportions), computing the test statistic is like scaling the estimator (computed from the sample) corresponding to the parameter of interest (in the population). So we basically subtract the target parameter from the point estimator and then divide the result by the standard error (which is equivalent to the standard deviation but for an estimator).

If this is unclear, here is how the test statistic (denoted $t_{obs}$) is computed in our scenario (assuming that the variance of the population is unknown):

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}}\]

where:

$\bar{x}$ is the sample mean (i.e., the estimator)
$\mu$ is the mean under the null hypothesis (i.e., the target parameter)
$s$ is the sample standard deviation
$n$ is the sample size
($\frac{s}{\sqrt{n}}$ is the standard error)

Notice the similarity between the formula of this test statistic and the formula used to standardize a random variable. This structure is the same for a test on two means, one proportion and two proportions, except that the estimator, the parameter and the standard error are, of course, slightly different for each type of test.

Suppose that in our case we have a sample mean of 71 kg ($\bar{x}$ = 71), a sample standard deviation of 13 kg ($s$ = 13) and a sample size of 10 adults ($n$ = 10). Remember that the population mean (the mean under the null hypothesis) is 80 kg ($\mu$ = 80).

The t-stat is thus:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Although formulas are different depending on which parameter you are testing, the value found for the test statistic gives us an indication on how extreme our observations are.

We keep this value of -2.189 in mind because it will be used again in step #4.

Step #3: Finding the critical value

Although the t-stat gives us an indication of how extreme our observations are, we cannot tell whether this “score of extremity” is too extreme or not based on its value only.

So, at this point, we cannot yet tell whether our data are too extreme or not. For this, we need to compare our t-stat with a threshold—referred as critical value—given by the probability distribution tables (and which can, of course, also be found with R).

In the same way that the formula to compute the t-stat is different for each parameter of interest, the underlying probability distribution—and thus the statistical table—on which the critical value is based is also different for each target parameter. This means that, in addition to choosing the appropriate formula to compute the t-stat, we also need to select the appropriate probability distribution depending on the parameter we are testing.

Luckily, there are only 4 different probability distributions for the 6 hypothesis tests covered in this article (one/two means, one/two proportions and one/two variances):

Standard Normal distribution:
- test on one and two means with known population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is known
- test on one and two proportions (given that some assumptions are met)
Student distribution:
- test on one and two means with unknown population variance(s)
- test on two paired samples where the variance of the difference between the 2 samples $\sigma^2_D$ is unknown
Chi-square distribution:
- test on one variance
Fisher distribution:
- test on two variances

Each probability distribution also has its own parameters (up to two parameters for the 4 distribution considered here), defining its shape and/or location. Parameter(s) of a probability distribution can be seen as its DNA; meaning that the distribution is entirely defined by its parameter(s).

Taking our initial scenario—a health professional who would like to test whether the mean weight of Belgian adults is different than 80 kg—as example.

The underlying probability distribution of a test on one mean is either the standard Normal or the Student distribution, depending on whether the variance of the population (not sample variance!) is known or unknown:⁶

If the population variance is known $\rightarrow$ the standard Normal distribution is used
If the population variance is unknown $\rightarrow$ the Student distribution is used

If no population variance is explicitly given, you can assume that it is unknown since you cannot compute it based on a sample. If you could compute it, that would mean you have access to the entire population and there is, in this case, no point in performing a hypothesis test (you could simply use some descriptive statistics to confirm or refute your belief).

In our example, no population variance is specified so it is assumed to be unknown. We therefore use the Student distribution.

The Student distribution has one parameter which defines it; the number of degrees of freedom. The number of degrees of freedom depends on the type of hypothesis test. For instance, the number of degrees of freedom for a test on one mean is equal to the number of observations minus one ($n$ - 1). Without going too far into the details, the - 1 comes from the fact that there is one quantity which is estimated (i.e., the mean).⁷ The sample size being equal to 10 in our example, the degrees of freedom is equal to $n$ - 1 = 10 - 1 = 9.

There is only one last element missing to find the critical value: the significance level. The significance level, denoted $\alpha$, is the probability of wrongly rejecting the null hypothesis, so the probability of rejecting the null hypothesis although it is in reality true. In this sense, it is an error (type I error, as opposed to the type II error⁸) that we accept to deal with, in order to be able to draw conclusions about a population based on a subset of it.

As you may have read in many statistical textbooks, the significance level is very often set to 5%.⁹ In some fields (such as medicine or engineering, among others), the significance level is also sometimes set to 1% to decrease the error rate.

It is best to specify the significance level before performing a hypothesis test to avoid the temptation to set the significance level in accordance to the results (the temptation is even bigger when the results are on the edge of being significant). As I always tell my students, you cannot “guess” nor compute the significance level. Therefore, if it is not explicitly specified, you can safely assume it is 5%. In our case, we did not indicate it, so we take $\alpha$ = 5% = 0.05.

Furthermore, in our example, we want to test whether the mean weight of Belgian adults is different than 80 kg. Since we do not specify the direction of the test, it is a two-sided test. If we wanted to test that the mean weight was less than 80 kg ($H_1: \mu <$ 80) or greater than 80 kg ($H_1: \mu >$ 80), we would have done a one-sided test.

Make sure that you perform the correct test (two-sided or one-sided) because it has an impact on how to find the critical value (see more in the following paragraphs).

So now that we know the appropriate distribution (Student distribution), its parameter (degrees of freedom (df) = 9), the significance level ($\alpha$ = 0.05) and the direction (two-sided), we have all we need to find the critical value in the statistical tables:

By looking at the row df = 9 and the column $t_.025$ in the Student’s distribution table, we find a critical value of:

\[t_{n-1; \alpha / 2} = t_{9; 0.025} = 2.262\]

One may wonder why we take $t_{\alpha/2} = t_.025$ and not $t_\alpha = t_.05$ since the significance level is 0.05. The reason is that we are doing a two-sided test ($H_1: \mu \ne$ 80), so the error rate of 0.05 must be divided in 2 to find the critical value to the right of the distribution. Since the Student’s distribution is symmetric, the critical value to the left of the distribution is simply: -2.262.

Visually, the error rate of 0.05 is partitioned into two parts:

0.025 to the left of -2.262 and
0.025 to the right of 2.262

We keep in mind these critical values of -2.262 and 2.262 for the fourth and last step.

Note that the red shaded areas in the previous plot are also known as the rejection regions. More on that in the following section.

These critical values can also be found in R, thanks to the qt() function:

qt(0.025, df = 9, lower.tail = TRUE)

## [1] -2.262157

qt(0.025, df = 9, lower.tail = FALSE)

## [1] 2.262157

The qt() function is used for the Student’s distribution (q stands for quantile and t for Student). There are other functions accompanying the different distributions:

qnorm() for the Normal distribution
qchisq() for the Chi-square distribution
qf() for the Fisher distribution

Step #4: Concluding and interpreting the results

In this fourth and last step, all we have to do is to compare the test statistic (computed in step #2) with the critical values (found in step #3) in order to conclude the hypothesis test.

The only two possibilities when concluding a hypothesis test are:

Rejection of the null hypothesis
Non-rejection of the null hypothesis

In our example of adult weight, remember that:

the t-stat is -2.189
the critical values are -2.262 and 2.262

Also remember that:

the t-stat gives an indication on how extreme our sample is compared to the null hypothesis
the critical values are the threshold from which the t-stat is considered as too extreme

To compare the t-stat with the critical values, I always recommend to plot them:

These two critical values form the rejection regions (the red shaded areas):

from $- \infty$ to -2.262, and
from 2.262 to $\infty$

If the t-stat lies within one of the rejection region, we reject the null hypothesis. On the contrary, if the t-stat does not lie within any of the rejection region, we do not reject the null hypothesis.

As we can see from the above plot, the t-stat is less extreme than the critical value and therefore does not lie within any of the rejection region. In conclusion, we do not reject the null hypothesis that $\mu = 80$.

This is the conclusion in statistical terms but they are meaningless without proper interpretation. So it is a good practice to also interpret the result in the context of the problem:

At the 5% significance level, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

Why don’t we accept $H_0$?

From a more philosophical (but still very important) perspective, note that we wrote “we do not reject the null hypothesis” and “we do not reject the hypothesis that the mean weight of Belgian adults is equal to 80 kg”. We did not write “we accept the null hypothesis” nor “the mean weight of Belgian adults is 80 kg”.

The reason is due to the fact that, in hypothesis testing, we conclude something about the population based on a sample. There is, therefore, always some uncertainty and we cannot be 100% sure that our conclusion is correct.

Perhaps it is the case that the mean weight of Belgian adults is in reality different than 80 kg, but we failed to prove it based on the data at hand. It may be the case that if we had more observations, we would have rejected the null hypothesis (since all else being equal, a larger sample size implies a more extreme t-stat). Or, it may be the case that even with more observations, we would not have rejected the null hypothesis because the mean weight of Belgian adults is in reality close to 80 kg. We cannot distinguish between the two.

So we can just say that we did not find enough evidence against the hypothesis that the mean weight of Belgian adults is 80 kg, but we do not conclude that the mean is equal to 80 kg.

If the difference is still not clear to you, the following example may help. Suppose a person is suspected of having committed a crime. This person is either innocent—the null hypothesis—or guilty—the alternative hypothesis. In the attempt to know if the suspect committed the crime, the police collects as much information and proof as possible. This is similar to the researcher collecting data to form a sample. And then the judge, based on the collected evidence, decides whether the suspect is considered as innocent or guilty. If there is enough evidence that the suspect committed the crime, the judge will conclude that the suspect is guilty. In other words, she will reject the null hypothesis of the suspect being innocent because there are enough evidence that the suspect committed the crime.

This is similar to the t-stat being more extreme than the critical value: we have enough information (based on the sample) to say that the null hypothesis is unlikely because our data would be too extreme if the null hypothesis were true. Since the sample cannot be “wrong” (it corresponds to the collected data), the only remaining possibility is that the null hypothesis is in fact wrong. This is the reason we write “we reject the null hypothesis”.

On the other hand, if there is not enough evidence that the suspect committed the crime (or no evidence at all), the judge will conclude that the suspect is considered as not guilty. In other words, she will not reject the null hypothesis of the suspect being innocent. But even if she concludes that the suspect is considered as not guilty, she will never be 100% sure that he is really innocent.

It may be the case that:

the suspect did not commit the crime, or
the suspect committed the crime but the police was not able to collect enough information against the suspect.

In the former case the suspect is really innocent, whereas in the latter case the suspect is guilty but the police and the judge failed to prove it because they failed to find enough evidence against him. Similar to hypothesis testing, the judge has to conclude the case by considering the suspect not guilty, without being able to distinguish between the two.

This is the main reason we write “we do not reject the null hypothesis” or “we fail to reject the null hypothesis” (you may even read in some textbooks conclusion such as “there is no sufficient evidence in the data to reject the null hypothesis”), and we do not write “we accept the null hypothesis”.

I hope this metaphor helped you to understand the reason why we reject the null hypothesis instead of accepting it.

In the following sections, we present two other methods used in hypothesis testing.

These methods will result in the exact same conclusion: non-rejection of the null hypothesis, that is, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg. It is thus presented only if you prefer to use these methods over the first one.

Method B: Comparing the p-value with the significance level $\alpha$

Method B, which consists in computing the p-value and comparing this p-value with the significance level $\alpha$, boils down to the following 4 steps:

Stating the null and alternative hypothesis
Computing the test statistic
Computing the p-value
Concluding and interpreting the results

In this second method which uses the p-value, the first and second steps are similar than in the first method.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the test statistic

Remember that the formula for the t-stat is different depending on the type of hypothesis test (one or two means, one or two proportions, one or two variances). In our case of one mean with unknown variance, we have:

\[t_{obs} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} = \frac{71 - 80}{\frac{13}{\sqrt{10}}} = -2.189\]

Step #3: Computing the p-value

The p-value is the probability (so it goes from 0 to 1) of observing a sample at least as extreme as the one we observed if the null hypothesis were true. In some sense, it gives you an indication on how likely your null hypothesis is. It is also defined as the smallest level of significance for which the data indicate rejection of the null hypothesis.

For more information about the p-value, I recommend reading this note about the p-value and the significance level $\alpha$.

Formally, the p-value is the area beyond the test statistic. Since we are doing a two-sided test, the p-value is thus the sum of the area above 2.189 and below -2.189.

Visually, the p-value is the sum of the two blue shaded areas in the following plot:

The p-value can computed with precision in R with the pt() function:

p_val <- pt(-2.189, df = 9, lower.tail = TRUE) + pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

# which is equivalent than:
p_val <- 2 * pt(2.189, df = 9, lower.tail = FALSE)
p_val

## [1] 0.05634202

The p-value is 0.0563, which indicates that there is a 5.63% chance to observe a sample at least as extreme as the one observed if the null hypothesis were true. This already gives us a hint on whether our t-stat is too extreme or not (and thus whether our null hypothesis is likely or not), but we formally conclude in step #4.

Like the qt() function to find the critical value, we use pt() to find the p-value because the underlying distribution is the Student’s distribution.

Use pnorm(), pchisq() and pf() for the Normal, Chi-square and Fisher distribution, respectively. See also this Shiny app to compute the p-value given a certain t-stat for most probability distributions.

If you do not have access to a computer (during exams for example) you will not be able to compute the p-value precisely, but you can bound it using the statistical table referring to your test.

In our case, we use the Student distribution and we look at the row df = 9 (since df = n - 1):

The test statistic is -2.189
We take the absolute value, which gives 2.189
The value 2.189 is between 1.833 and 2.262 (highlighted in blue in the above table)
From the column names $t_{.050}$ and $t_{.025}$ related to 1.833 and 2.262, we know that:
- the area to the right of 1.833 is 0.05
- the area to the right of 2.262 is 0.025
So we know that the area to the right of 2.189 must be between 0.025 and 0.05
Since the Student distribution is symmetric, we know that the area to the left of -2.189 must also be between 0.025 and 0.05
Therefore, the sum of the two areas must be between 0.05 and 0.10
In other words, the p-value is between 0.05 and 0.10 (i.e., 0.05 < p-value < 0.10)

Although we could not compute it precisely, it is enough to conclude our hypothesis test in the last step.

Step #4: Concluding and interpreting the results

The final step is now to simply compare the p-value (computed in step #3) with the significance level $\alpha$. As for all statistical tests:

If the p-value is smaller than $\alpha$ (p-value < 0.05) $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the p-value is greater than or equal to $\alpha$ (p-value $\ge$ 0.05) $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

No matter if we take into consideration the exact p-value (i.e., 0.0563) or the bounded one (0.05 < p-value < 0.10), it is larger than 0.05, so we do not reject the null hypothesis.¹⁰ In the context of the problem, we do not reject the null hypothesis that the mean weight of Belgian adults is 80 kg.

Remember that rejecting (or not rejecting) a null hypothesis at the significance level $\alpha$ using the critical value method (method A) is equivalent to rejecting (or not rejecting) the null hypothesis when the p-value is lower (equal or greater) than $\alpha$ (method B).

This is the reason we find the exact same conclusion than with method A, and why you should too if you use both methods on the same data and with the same significance level.

Method C: Comparing the target parameter with the confidence interval

Method C, which consists in computing the confidence interval and comparing this confidence interval with the target parameter (the parameter under the null hypothesis), boils down to the following 3 steps:

Stating the null and alternative hypothesis
Computing the confidence interval
Concluding and interpreting the results

In this last method which uses the confidence interval, the first step is similar than in the first two methods.

Step #1: Stating the null and alternative hypothesis

The null and alternative hypotheses remain the same:

$H_0: \mu = 80$
$H_1: \mu \ne 80$

Step #2: Computing the confidence interval

Like hypothesis testing, confidence intervals are a well-known tool in inferential statistics.

Confidence interval is an estimation procedure which produces an interval (i.e., a range of values) containing the true parameter with a certain—usually high—probability.

In the same way that there is a formula for each type of hypothesis test when computing the test statistics, there exists a formula for each type of confidence interval. Formulas for the different types of confidence intervals can be found in this Shiny app.

Here is the formula for a confidence interval on one mean $\mu$ (with unknown population variance):

\[ (1-\alpha)\text{% CI for } \mu = \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \]

where $t_{\alpha/2, n - 1}$ is found in the Student distribution table (and is similar to the critical value found in step #3 of method A).

Given our data and with $\alpha$ = 0.05, we have:

\[ \begin{aligned} 95\text{% CI for } \mu &= \bar{x} \pm t_{\alpha/2, n - 1} \frac{s}{\sqrt{n}} \\ &= 71 \pm 2.262 \frac{13}{\sqrt{10}} \\ &= [61.70; 80.30] \end{aligned} \]

The 95% confidence interval for $\mu$ is [61.70; 80.30] kg. But what does a 95% confidence interval mean?

We know that this estimation procedure has a 95% probability of producing an interval containing the true mean $\mu$. In other words, if we construct many confidence intervals (with different samples of the same size), 95% of them will, on average, include the mean of the population (the true parameter). So on average, 5% of these confidence intervals will not cover the true mean.

If you wish to decrease this last percentage, you can decrease the significance level (set $\alpha$ = 0.01 or 0.02 for instance). All else being equal, this will increase the range of the confidence interval and thus increase the probability that it includes the true parameter.

Step #3: Concluding and interpreting the results

The final step is simply to compare the confidence interval (constructed in step #2) with the value of the target parameter (the value under the null hypothesis, mentioned in step #1):

If the confidence interval does not include the hypothesized value $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the confidence interval includes the hypothesized value $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

In our example:

the hypothesized value is 80 (since $H_0: \mu$ = 80)
80 is included in the 95% confidence interval since it goes from 61.70 to 80.30 kg
So we do not reject the null hypothesis

In the terms of the problem, we do not reject the hypothesis that the mean weight of Belgian adults is 80 kg.

As you can see, the conclusion is equivalent than with the critical value method (method A) and the p-value method (method B). Again, this must be the case since we use the same data and the same significance level $\alpha$ for all three methods.

Which method to choose?

All three methods give the same conclusion. However, each method has its own advantage so I usually select the most convenient one depending on the situation:

Method A (comparing the test statistic with the critical value):
- It is, in my opinion, the easiest and most straightforward method of the three when I do not have access to R.
Method B (comparing the p-value with the significance level $\alpha$):
- In addition to being able to know whether the null hypothesis is rejected or not, computing the exact p-value can be very convenient so I tend to use this method if I have access to R.
Method C (comparing the target parameter with the confidence interval):
- If I need to test several hypothesized values, I tend to choose this method because I can construct one single confidence interval and compare it to as many values as I want. For example, with our 95% confidence interval [61.70; 80.30], I know that any value below 61.70 kg and above 80.30 kg will be rejected, without testing it for each value.

Summary

In this article, we reviewed the goals and when hypothesis testing is used. We then showed how to do a hypothesis test by hand through three different methods (A. critical value, B. p-value and C. confidence interval). We also showed how to interpret the results in the context of the initial problem.

Although all three methods give the exact same conclusion when using the same data and the same significance level (otherwise there is a mistake somewhere), I also presented my personal preferences when it comes to choosing one method over the other two.

Thanks for reading.

I hope this article helped you to understand the structure of a hypothesis by hand. I remind you that, at least for the 6 hypothesis tests covered in this article, the formulas are different, but the structure and the reasoning behind it remain the same. So you basically have to know which formulas to use, and simply follow the steps mentioned in this article.

For the interested reader, I created two accompanying Shiny apps:

Hypothesis testing and confidence intervals: after entering your data, the app illustrates all the steps in order to conclude the test and compute a confidence interval. See more information in this article.
How to read statistical tables: the app helps you to compute the p-value given a t-stat for most probability distributions. See more information in this article.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Suppose a researcher wants to test whether Belgian women are taller than French women. Suppose a health professional would like to know whether the proportion of smokers is different among athletes and non-athletes. It would take way too long to measure the height of all Belgian and French women and to ask all athletes and non-athletes their smoking habits. So most of the time, decisions are based on a representative sample of the population and not on the whole population. If we could measure the entire population in a reasonable time frame, we would not do any inferential statistics.↩︎
Don’t get me wrong, this does not mean that hypothesis tests are never used in exploratory analyses. It is just much less frequent in exploratory research than in confirmatory research.↩︎
You may see more or less steps in other articles or textbooks, depending on whether these steps are detailed or concise. Hypothesis testing should, however, follows the same process regardless of the number of steps.↩︎
For one-sided tests, writing $H_0: \mu = 80$ or $H_0: \mu \ge 80$ are both correct. The point is that the null and alternative hypothesis must be mutually exclusive since you are testing one hypothesis against the other, so both cannot be true at the same time.↩︎
To be complete, there are even different formulas within each type of test, depending on whether some assumptions are met or not. For the interested reader, see all the different scenarios and thus the different formulas for a test on one mean and on two means.↩︎
There are more uncertainty if the population variance is unknown than if it is known, and this greater uncertainty is taken into account by using the Student distribution instead of the standard Normal distribution. Also note that as the sample size increases, the degrees of freedom of the Student distribution increases and the two distributions become more and more similar. For large sample size (usually from $n >$ 30), the Student distribution becomes so close to the standard Normal distribution that, even if the population variance is unknown, the standard Normal distribution can be used.↩︎
For a test on two independent samples, the degrees of freedom is $n_1 + n_2 - 2$, where $n_1$ and $n_2$ are the size of the first and second sample, respectively. Note the - 2 due to the fact that in this case, two quantities are estimated.↩︎
The type II error is the probability of not rejecting the null hypothesis although it is in reality false.↩︎
Whether this is a good or a bad standard is a question that comes up often and is debatable. This is, however, beyond the scope of the article.↩︎
Again, p-values found via a statistical table or via R must be coherent.↩︎

Paper: 'Waiting period from diagnosis for mortgage insurance issued to cancer survivors'

Mon, 23 Nov 2020 00:00:00 +0000

I am happy to announce that our paper entitled “Waiting period from diagnosis for mortgage insurance issued to cancer survivors” has been published in the European Actuarial Journal (Soetewey et al. 2021).

Here is a brief summary of it:

Massart (2018) testimonial illustrates the difficulties faced by patients having survived cancer to access mortgage insurance securing home loan. Data collected by national registries nevertheless suggest that excess mortality due to some types of cancer becomes moderate or even negligible after some waiting period.

In relation to the insurance laws passed in France and more recently in Belgium creating a right to be forgotten for cancer survivors, the present study aims to determine the waiting period after which standard premium rates become applicable. Compared to the French and Belgian laws, a waiting period starting at diagnosis (as recorded in national databases) is favored over a waiting period starting at the end of the therapeutic treatment protocol. This aims to avoid disputes when a claim is filed. Since diagnosis is often recorded in the official registry database, as is the case for the Belgian Cancer Registry, its date is reliable and unquestionable in case of claim.

Based on 28,994 melanoma and thyroid cancer cases recorded by the Belgian Cancer Registry, the length of the waiting period is assessed with the help of widely-accepted tools from biostatistics, including relative survival models and time-to-cure indicators. It turns out for instance that a waiting period of 4 years after diagnosis is enough for 30-year-old thyroid cancer patients. This appears to be similar to the 3-year period starting at the end of treatment protocol according to the Belgian law in such a case.

Read the full article here.

Here is a video explaining the paper and our research in general:

The paper also led to a talk organized by the French National Cancer Institute (INCa).

This paper is written jointly with Prof. Catherine Legrand and Prof. Michel Denuit—my PhD supervisors—and Dr. Geert Silversmit from the Belgian Cancer Registry.

Thanks for reading. I hope this paper will, to some extent, be helpful for your research.

As always, if you have any question related to the topic covered in this paper, please add it as a comment so other readers can benefit from the discussion.

References

Soetewey, Antoine, Catherine Legrand, Michel Denuit, and Geert Silversmit. 2021. “Waiting Period from Diagnosis for Mortgage Insurance Issued to Cancer Survivors.” European Actuarial Journal 11: 135–60. https://doi.org/10.1007/s13385-020-00254-x.

ANOVA in R

Mon, 12 Oct 2020 00:00:00 +0000

Introduction

ANOVA (ANalysis Of VAriance) is a statistical test to determine whether two or more population means are different. In other words, it is used to compare two or more groups to see if they are significantly different.

In practice, however, the:

Student t-test is used to compare 2 groups;
ANOVA generalizes the t-test beyond 2 groups, so it is used to compare 3 or more groups.

Note that there are several versions of the ANOVA (e.g., one-way ANOVA, two-way ANOVA, mixed ANOVA, repeated measures ANOVA, etc.). In this article, we present the simplest form only—the one-way ANOVA¹—and we refer to it as ANOVA in the remaining of the article.

Although ANOVA is used to make inference about means of different groups, the method is called “analysis of variance”. It is called like this because it compares the “between” variance (the variance between the different groups) and the variance “within” (the variance within each group). If the between variance is significantly larger than the within variance, the group means are declared to be different. Otherwise, we cannot conclude one way or the other. The two variances are compared to each other by taking the ratio ($\frac{variance_{between}}{variance_{within}}$) and then by comparing this ratio to a threshold from the Fisher probability distribution (a threshold based on a specific significance level, usually 5%).

This is enough theory regarding the ANOVA method for now. In the remaining of this article, we discuss about it from a more practical point of view, and in particular we will cover the following points:

the aim of the ANOVA, when it should be used and the null/alternative hypothesis
the underlying assumptions of the ANOVA and how to check them
how to perform the ANOVA in R
how to interpret results of the ANOVA
understand the notion of post-hoc test and interpret the results
how to visualize results of ANOVA and post-hoc tests

Data

Data for the present article is the penguins dataset (an alternative to the well-known iris dataset), accessible via the {palmerpenguins} package:

# install.packages("palmerpenguins")
library(palmerpenguins)

The dataset contains data for 344 penguins of 3 different species (Adelie, Chinstrap and Gentoo). The dataset contains 8 variables, but we focus only on the flipper length and the species for this article, so we keep only those 2 variables:

library(tidyverse)

dat <- penguins %>%
  select(species, flipper_length_mm)

Below some basic descriptive statistics and a plot (made with the {ggplot2} package) of our dataset before we proceed to the goal of the ANOVA:

summary(dat)

##       species    flipper_length_mm
##  Adelie   :152   Min.   :172.0    
##  Chinstrap: 68   1st Qu.:190.0    
##  Gentoo   :124   Median :197.0    
##                  Mean   :200.9    
##                  3rd Qu.:213.0    
##                  Max.   :231.0    
##                  NA's   :2

Flipper length varies from 172 to 231 mm, with a mean of 200.9 mm. There are respectively 152, 68 and 124 penguins of the species Adelie, Chinstrap and Gentoo.

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm, color = species) +
  geom_jitter() +
  theme(legend.position = "none")

Here, the factor is the species variable which contains 3 modalities or groups (Adelie, Chinstrap and Gentoo).

Aim and hypotheses of ANOVA

As mentioned in the introduction, the ANOVA is used to compare groups (in practice, 3 or more groups). More generally, it is used to:

study whether measurements are similar across different modalities (also called levels or treatments in the context of ANOVA) of a categorical variable
compare the impact of the different levels of a categorical variable on a quantitative variable
explain a quantitative variable based on a qualitative variable

In this context and as an example, we are going to use an ANOVA to help us answer the question: “Is the length of the flippers different between the 3 species of penguins?”.

The null and alternative hypothesis of an ANOVA are:

$H_0$: $\mu_{Adelie} = \mu_{Chinstrap} = \mu_{Gentoo}$ ($\Rightarrow$ the 3 species are equal in terms of flipper length)
$H_1$: at least one mean is different ($\Rightarrow$ at least one species is different from the other 2 species in terms of flipper length)

Be careful that the alternative hypothesis is not that all means are different. The opposite of all means being equal ($H_0$) is that at least one mean is different from the others ($H_1$).

In this sense, if the null hypothesis is rejected, it means that at least one species is different from the other 2, but not necessarily that all 3 species are different from each other. It could be that flipper length for the species Gentoo is different than for the species Chinstrap and Adelie, but flipper length is similar between Chinstrap and Adelie. Other types of test (known as post-hoc tests and covered in this section) must be performed to test whether all 3 species differ.

Underlying assumptions of ANOVA

As for many statistical tests, there are some assumptions that need to be met in order to be able to interpret the results. When one or several assumptions are not met, although it is technically possible to perform these tests, it would be incorrect to interpret the results and trust the conclusions.

Below are the assumptions of the ANOVA, how to test them and which other tests exist if an assumption is not met:

Variable type: ANOVA requires a mix of one continuous quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with at least 2 levels which will determine the groups to compare).
Independence: the data, collected from a representative and randomly selected portion of the total population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if three measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before, (ii) during and (iii) after a treatment), the repeated measures ANOVA should be preferred in order to take into account the dependency between the samples.
Normality:
- In case of small samples, residuals² should follow approximately a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. If, even after a transformation of your data (e.g., logarithmic transformation, square root, Box-Cox, etc.), the residuals still do not follow approximately a normal distribution, the Kruskal-Wallis test can be applied (kruskal.test(variable ~ group, data = dat in R). This non-parametric test, robust to non normal distributions, has the same goal than the ANOVA—compare 3 or more groups—but it uses sample medians instead of sample means to compare groups.
- In case of large samples, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013).³ It is therefore not required to test the normality assumption when the number of observations in each group/sample is large (usually $n \ge 30$).
Equality of variances: the variances of the different groups should be equal in the populations (an assumption called homogeneity of the variances, or even sometimes referred as homoscedasticity, as opposed to heteroscedasticity if variances are different across groups). This assumption can be tested graphically (by comparing the dispersion in a boxplot or dotplot for instance), or more formally via the Levene’s test (leveneTest(variable ~ group) from the {car} package) or Bartlett’s test, among others. If the hypothesis of equal variances is rejected, another version of the ANOVA can be used: the Welch ANOVA (oneway.test(variable ~ group, var.equal = FALSE)). Note that the Welch ANOVA does not require homogeneity of the variances, but the distributions should still follow approximately a normal distribution. Note that the Kruskal-Wallis test does not require the assumptions of normality nor homoscedasticity of the variances.⁴
Outliers: An outlier is a value or an observation that is distant from the other observations. There should be no significant outliers in the different groups, or the conclusions of your ANOVA may be flawed. There are several methods to detect outliers in your data but in order to deal with them, it is your choice to either:
- use the non-parametric version (i.e., the Kruskal-Wallis test)
- transform your data (logarithmic or Box-Cox transformation, among others)
- or remove them (be careful)

Choosing the appropriate test depending on whether assumptions are met may be confusing so here is a brief summary:

Check that your observations are independent.
Sample sizes:
- In case of small samples, test the normality of residuals:
  - If normality is assumed, test the homogeneity of the variances:
    - If variances are equal, use ANOVA.
    - If variances are not equal, use the Welch ANOVA.
  - If normality is not assumed, use the Kruskal-Wallis test.
- In case of large samples normality is assumed, so test the homogeneity of the variances:
  - If variances are equal, use ANOVA.
  - If variances are not equal, use the Welch ANOVA.

Now that we have seen the underlying assumptions of the ANOVA, we review them specifically for our dataset before applying the appropriate version of the test.

Variable type

The dependent variable flipper_length_mm is a quantitative variable and the independent variable species is a qualitative one (with 3 levels corresponding to the 3 species). So we have a mix of the two types of variable and this assumption is met.

Independence

Independence of the observations is assumed as data have been collected from a randomly selected portion of the population and measurements within and between the 3 samples are not related.

The independence assumption is most often verified based on the design of the experiment and on the good control of experimental conditions, as it is the case here.

If you really want to test it more formally, you can, however, test it via a statistical test—the Durbin-Watson test (in R: durbinWatsonTest(res_lm) where res_lm is a linear model). The null hypothesis of this test specifies an autocorrelation coefficient = 0, while the alternative hypothesis specifies an autocorrelation coefficient $\ne$ 0.

Normality

Since the smallest sample size per group (i.e., per species) is 68, we have large samples. Therefore, we do not need to check normality.

Usually, we would directly test the homogeneity of the variances without testing normality. However, for the sake of illustration, we act as if the sample sizes were small in order to illustrate what would need to be done in that case.

Remember that normality of residuals can be tested visually via a histogram and a QQ-plot, and/or formally via a normality test (Shapiro-Wilk test for instance).

Before checking the normality assumption, we first need to compute the ANOVA (more on that in this section). We then save the results in res_aov :

res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

We can now check normality visually:

par(mfrow = c(1, 2)) # combine plots

# histogram
hist(res_aov$residuals)

# QQ-plot
library(car)
qqPlot(res_aov$residuals,
  id = FALSE # id = FALSE to remove point identification
)

From the histogram and QQ-plot above, we can already see that the normality assumption seems to be met. Indeed, the histogram roughly form a bell curve, indicating that the residuals follow a normal distribution. Furthermore, points in the QQ-plots roughly follow the straight line and most of them are within the confidence bands, also indicating that residuals follow approximately a normal distribution.

Some researchers stop here and assume that normality is met, while others also test the assumption via a formal normality test. It is your choice to test it (i) only visually, (ii) only via a normality test, or (iii) both visually AND via a normality test. Bear in mind, however, the two following points:

ANOVA is quite robust to small deviations from normality. This means that it is not an issue (from the perspective of the interpretation of the ANOVA results) if a small number of points deviates slightly from the normality,
normality tests are sometimes quite conservative, meaning that the null hypothesis of normality may be rejected due to a limited deviation from normality. This is especially the case with large samples as power of the test increases with the sample size.

In practice, I tend to prefer the (i) visual approach only, but again, this is a matter of personal choice and also depends on the context of the analysis.

Still for the sake of illustration, we also now test the normality assumption via a normality test. You can use the Shapiro-Wilk test or the Kolmogorov-Smirnov test, among others.

Remember that the null and alternative hypothesis of these tests are:

$H_0$: data come from a normal distribution
$H_1$: data do not come from a normal distribution

In R, we can test normality of the residuals with the Shapiro-Wilk test thanks to the shapiro.test() function:

shapiro.test(res_aov$residuals)

## 
## 	Shapiro-Wilk normality test
## 
## data:  res_aov$residuals
## W = 0.99452, p-value = 0.2609

P-value of the Shapiro-Wilk test on the residuals is larger than the usual significance level of $\alpha = 5\%$, so we do not reject the hypothesis that residuals follow a normal distribution (p-value = 0.261).

This result is in line with the visual approach. In our case, the normality assumption is thus met both visually and formally.

Side note: Remind that the p-value is the probability of having observations as extreme as the ones we have observed in the sample(s) given that the null hypothesis is true. If the p-value $< \alpha$ (indicating that it is not likely to observe the data we have in the sample given that the null hypothesis is true), the null hypothesis is rejected, otherwise the null hypothesis is not rejected. See more about p-value and significance level if you are unfamiliar with those important statistical concepts.

Remember that if the normality assumption was not reached, some transformation(s) would need to be applied on the raw data in the hope that residuals would better fit a normal distribution, or you would need to use the non-parametric version of the ANOVA—the Kruskal-Wallis test.

As pointed out by a reader (see comments at the very end of the article), the normality assumption can also be tested on the “raw” data (i.e., the observations) instead of the residuals. However, if you test the normality assumption on the raw data, it must be tested for each group separately as the ANOVA requires normality in each group.

Testing normality on all residuals or on the observations per group is equivalent, and will give similar results. Indeed, saying “The distribution of Y within each group is normally distributed” is the same as saying “The residuals are normally distributed”.

Remember that residuals are the distance between the actual value of Y and the mean value of Y for a specific value of X, so the grouping variable is induced in the computation of the residuals.

So in summary, in ANOVA you actually have two options for testing normality:

Checking normality separately for each group on the “raw” data (Y values)
Checking normality on all residuals (but not per group)

In practice, you will see that it is often easier to just use the residuals and check them all together, especially if you have many groups or few observations per group.

If you are still not convinced: remember that an ANOVA is a special case of a linear model. Suppose your independent variable is a continuous variable (instead of a categorical variable), the only option you have left is to check normality on the residuals, which is precisely what is done for testing normality in linear regression models.

Equality of variances - homogeneity

Assuming residuals follow a normal distribution, it is now time to check whether the variances are equal across species or not. The result will have an impact on whether we use the ANOVA or the Welch ANOVA.

This can again be verified visually—via a boxplot or dotplot—or more formally via a statistical test (Levene’s test, among others).

Visually, we have:

# Boxplot
boxplot(flipper_length_mm ~ species,
  data = dat
)

# Dotplot
library("lattice")

dotplot(flipper_length_mm ~ species,
  data = dat
)

Both the boxplot and the dotplot show a similar variance for the different species. In the boxplot, this can be seen by the fact that the boxes and the whiskers have a comparable size for all species.

There are a couple of outliers as shown by the points outside the whiskers, but this does not change the fact that the dispersion is more or less the same between the different species.

In the dotplot, this can be seen by the fact that points for all 3 species have more or less the same range, a sign of the dispersion and thus the variance being similar.

Like the normality assumption, if you feel that the visual approach is not sufficient, you can formally test for equality of the variances with a Levene’s or Bartlett’s test. Notice that the Levene’s test is less sensitive to departures from normal distribution than the Bartlett’s test.

The null and alternative hypothesis for both tests are:

$H_0$: variances are equal
$H_1$: at least one variance is different

In R, the Levene’s test can be performed thanks to the leveneTest() function from the {car} package:

# Levene's test
library(car)

leveneTest(flipper_length_mm ~ species,
  data = dat
)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   2  0.3306 0.7188
##       339

The p-value being larger than the significance level of 0.05, we do not reject the null hypothesis, so we cannot reject the hypothesis that variances are equal between species (p-value = 0.719).

This result is also in line with the visual approach, so the homogeneity of variances is met both visually and formally.

Another method to test normality and homogeneity

For your information, it is also possible to test the homogeneity of the variances and the normality of the residuals visually (and both at the same time) via the plot() function:

par(mfrow = c(1, 2)) # combine plots

# 1. Homogeneity of variances
plot(res_aov, which = 3)

# 2. Normality
plot(res_aov, which = 2)

Plot on the left hand side shows that there is no evident relationships between residuals and fitted values (the mean of each group), so homogeneity of variances is assumed. If homogeneity of variances was violated, the red line would not be flat (horizontal).

Plot on the right hand side shows that residuals follow approximately a normal distribution, so normality is assumed. If normality was violated, points would consistently deviate from the dashed line.

Outliers

There are several techniques to detect outliers. In this article, we focus on the most simple one (yet very efficient)—the visual approach via a boxplot:

boxplot(flipper_length_mm ~ species,
  data = dat
)

There is one outlier in the group Adelie, as defined by the interquartile range criterion. This point is, however, not seen as a significant outlier so we can assume that the assumption of no significant outliers is met.

ANOVA

We showed that all assumptions of the ANOVA are met.

We can thus proceed to the implementation of the ANOVA in R, but first, let’s do some preliminary analyses to better understand the research question.

Preliminary analyses

A good practice before actually performing the ANOVA in R is to visualize the data in relation to the research question. The best way to do so is to draw and compare boxplots of the quantitative variable flipper_length_mm for each species.

This can be done with the boxplot() function in base R (same code than the visual check of equal variances):

boxplot(flipper_length_mm ~ species,
  data = dat
)

Or with the {ggplot2} package:

library(ggplot2)

ggplot(dat) +
  aes(x = species, y = flipper_length_mm) +
  geom_boxplot()

The boxplots above show that, at least for our sample, penguins of the species Gentoo seem to have the biggest flipper, and Adelie species the smallest flipper.

Besides a boxplot for each species, it is also a good practice to compute some descriptive statistics such as the mean and standard deviation by species.

This can be done, for instance, with the aggregate() function:

aggregate(flipper_length_mm ~ species,
  data = dat,
  function(x) round(c(mean = mean(x), sd = sd(x)), 2)
)

##     species flipper_length_mm.mean flipper_length_mm.sd
## 1    Adelie                 189.95                 6.54
## 2 Chinstrap                 195.82                 7.13
## 3    Gentoo                 217.19                 6.48

or with the summarise() and group_by() functions from the {dplyr} package:

library(dplyr)

group_by(dat, species) %>%
  summarise(
    mean = mean(flipper_length_mm, na.rm = TRUE),
    sd = sd(flipper_length_mm, na.rm = TRUE)
  )

## # A tibble: 3 × 3
##   species    mean    sd
##   <fct>     <dbl> <dbl>
## 1 Adelie     190.  6.54
## 2 Chinstrap  196.  7.13
## 3 Gentoo     217.  6.48

Mean is also the lowest for Adelie and highest for Gentoo. Boxplots and descriptive statistics are, however, not enough to conclude that flippers are significantly different in the 3 populations of penguins.

ANOVA in R

As you guessed by now, only the ANOVA can help us to make inference about the population given the sample at hand, and help us to answer the initial research question “Is the length of the flippers different between the 3 species of penguins?”.

ANOVA in R can be done in several ways, of which two are presented below:

With the oneway.test() function:

# 1st method:
oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = TRUE # assuming equal variances
)

## 
## 	One-way analysis of means
## 
## data:  flipper_length_mm and species
## F = 594.8, num df = 2, denom df = 339, p-value < 2.2e-16

With the summary() and aov() functions:

# 2nd method:
res_aov <- aov(flipper_length_mm ~ species,
  data = dat
)

summary(res_aov)

##              Df Sum Sq Mean Sq F value Pr(>F)    
## species       2  52473   26237   594.8 <2e-16 ***
## Residuals   339  14953      44                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 2 observations deleted due to missingness

As you can see from the two outputs above, the test statistic (F = in the first method and F value in the second one) and the p-value (p-value in the first method and Pr(>F) in the second one) are exactly the same for both methods, which means that in case of equal variances, results and conclusions will be unchanged.

The advantage of the first method is that it is easy to switch from the ANOVA (used when variances are equal) to the Welch ANOVA (used when variances are unequal). This can be done by replacing var.equal = TRUE by var.equal = FALSE, as presented below:

oneway.test(flipper_length_mm ~ species,
  data = dat,
  var.equal = FALSE # assuming unequal variances
)

## 
## 	One-way analysis of means (not assuming equal variances)
## 
## data:  flipper_length_mm and species
## F = 614.01, num df = 2.00, denom df = 172.76, p-value < 2.2e-16

The advantage of the second method, however, is that:

the full ANOVA table (with degrees of freedom, mean squares, etc.) is printed, which may be of interest in some (theoritical) cases
results of the ANOVA (res_aov) can be saved for later use (especially useful for post-hoc tests)

Interpretations of ANOVA results

Given that the p-value is smaller than 0.05, we reject the null hypothesis, so we reject the hypothesis that all means are equal. Therefore, we can conclude that at least one species is different than the others in terms of flippers length (p-value < 2.2e-16).

(For the sake of illustration, if the p-value was larger than 0.05: we cannot reject the null hypothesis that all means are equal, so we cannot reject the hypothesis that the 3 considered species of penguins are equal in terms of flippers length.)

A nice and easy way to report results of an ANOVA in R is with the report() function from the {report} package:

# install.packages("remotes")
# remotes::install_github("easystats/report") # You only need to do that once
library("report") # Load the package every time you start R

report(res_aov)

## The ANOVA (formula: flipper_length_mm ~ species) suggests that:
## 
##   - The main effect of species is statistically significant and large (F(2, 339)
## = 594.80, p < .001; Eta2 = 0.78, 95% CI [0.75, 1.00])
## 
## Effect sizes were labelled following Field's (2013) recommendations.

As you can see, the function interprets the results for you and indicates a large and significant main effect of the species on the flipper length (p-value < .001).

Note that the report() function can be used for other analyses. See more tips and tricks in R if you find this one useful.

What’s next?

If the null hypothesis is not rejected (p-value $\ge$ 0.05), it means that we do not reject the hypothesis that all groups are equal. The ANOVA more or less stops here.

Other types of analyses can be performed of course, but—given the data at hand—we could not prove that at least one group was different so we usually do not go further with the ANOVA.

On the contrary, if the null hypothesis is rejected (as it is our case since the p-value < 0.05), we proved that at least one group is different. We can decide to stop here if we are only interested to test whether all species are equal in terms of flippers length.

But most of the time, when we showed thanks to an ANOVA that at least one group is different, we are also interested in knowing which one(s) is(are) different. Results of an ANOVA, however, do NOT tell us which group(s) is(are) different from the others.

To test this, we need to use other types of test, referred as post-hoc tests (in Latin, “after this”, so after obtaining statistically significant ANOVA results) or multiple pairwise-comparison tests.⁵

This family of statistical tests is the topic of the following sections.

Post-hoc test

Issue of multiple testing

In order to see which group(s) is(are) different from the others, we need to compare groups 2 by 2. In practice, since there are 3 species, we are going to compare species 2 by 2 as follows:

Chinstrap versus Adelie
Gentoo vs. Adelie
Gentoo vs. Chinstrap

In theory, we could compare species thanks to 3 Student’s t-tests since we need to compare 2 groups and a t-test is used precisely in that case.

However, if several t-tests are performed, the issue of multiple testing (also referred as multiplicity) arises. In short, when several statistical tests are performed, some will have p-values less than $\alpha$ purely by chance, even if all null hypotheses are in fact true.

To demonstrate the problem, consider our case where we have 3 hypotheses to test and a desired significance level of 0.05.

The probability of observing at least one significant result (at least one p-value < 0.05) just due to chance is:

\[\begin{equation} \begin{split} P(\text{at least 1 sig. result}) & = 1 - P(\text{no sig. results}) \\ & = 1 - (1 - 0.05)^3 \\ & = 0.142625 \end{split} \end{equation}\]

So, with as few as 3 tests being considered, we already have a 14.26% chance of observing at least one significant result, even if all of the tests are actually not significant.

And as the number of groups increases, the number of comparisons increases as well, so the probability of having a significant result simply due to chance keeps increasing.

For example, with 10 groups we need to make 45 comparisons and the probability of having at least one significant result by chance becomes $1 - (1 - 0.05)^{45} = 90\%$. So it is very likely to observe a significant result just by chance when comparing 10 groups, and when we have 14 groups or more we are almost certain (99%) to have a false positive!

Post-hoc tests take into account that multiple tests are done and deal with the problem by adjusting $\alpha$ in some way, so that the probability of observing at least one significant result due to chance remains below our desired significance level.⁶

Post-hoc tests in R and their interpretation

Post-hoc tests are a family of statistical tests so there are several of them. The most common ones are:

Tukey HSD, used to compare all groups to each other (so all possible comparisons of 2 groups).
Dunnett, used to make comparisons with a reference group. For example, consider 2 treatment groups and one control group. If you only want to compare the 2 treatment groups with respect to the control group, and you do not want to compare the 2 treatment groups to each other, the Dunnett’s test is preferred.
Bonferroni correction if one has a set of planned comparisons to do.

The Bonferroni correction is simple: you simply divide the desired global $\alpha$ level by the number of comparisons.

In our example, we have 3 comparisons so if we want to keep a global $\alpha = 0.05$, we have $\alpha' = \frac{0.05}{3} = 0.0167$. We can then simply perform a Student’s t-test for each comparison, and compare the obtained $p$-values with this new $\alpha'$.

The other two post-hoc tests are presented in the next sections.

Note that variances are assumed to be equal for all three methods (unless you use the Welch’s t-test instead of the Student’s t-test with the Bonferroni correction). If variances are not equal, you can use the Games-Howell test, among others.

Tukey HSD test

In our case, since there is no “reference” species and we are interested in comparing all species, we are going to use the Tukey HSD test.

In R, the Tukey HSD test is done as follows. This is where the second method to perform the ANOVA comes handy because the results (res_aov) are reused for the post-hoc test:

library(multcomp)

# Tukey HSD test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Tukey")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 1.03e-08 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-08 ***
## Gentoo - Chinstrap == 0  21.3635     1.0036  21.286  < 1e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

In the output of the Tukey HSD test, we are interested in the table displayed after Linear Hypotheses:, and more precisely, in the first and last column of the table. The first column shows the comparisons which have been made; the last column (Pr(>|t|)) shows the adjusted⁷ p-values for each comparison (with the null hypothesis being the two groups are equal and the alternative hypothesis being the two groups are different).

It is these adjusted p-values that are used to test whether two groups are significantly different or not, and we can be confident that the entire set of comparisons collectively has an error rate of 0.05.

In our example, we tested:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)
Gentoo vs. Chinstrap (line Gentoo - Chinstrap == 0)

All three ajusted p-values are smaller than 0.05, so we reject the null hypothesis for all comparisons, which means that all species are significantly different in terms of flippers length.

The results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that all groups are significantly different.

Note that the Tukey HSD test can also be done in R with the TukeyHSD() function:

TukeyHSD(res_aov)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## $species
##                       diff       lwr       upr p adj
## Chinstrap-Adelie  5.869887  3.586583  8.153191     0
## Gentoo-Adelie    27.233349 25.334376 29.132323     0
## Gentoo-Chinstrap 21.363462 19.000841 23.726084     0

With this code, it is the column p adj (also the last column) which is of interest. Notice that the conclusions are the same than above: all species are significantly different in terms of flippers length.

The results can also be visualized with the plot() function:

plot(TukeyHSD(res_aov))

Dunnett’s test

We have seen in this section that as the number of groups increases, the number of comparisons also increases. And as the number of comparisons increases, the post-hoc analysis must lower the individual significance level even further, which leads to lower statistical power (so a difference between group means in the population is less likely to be detected).

One method to mitigate this and increase the statistical power is by reducing the number of comparisons. This reduction allows the post-hoc procedure to use a larger individual error rate to achieve the desired global error rate.

While comparing all possible groups with a Tukey HSD test is a common approach, many studies have a control group and several treatment groups. For these studies, you may need to compare the treatment groups only to the control group, which reduces the number of comparisons.

Dunnett’s test does precisely this—it only compares a group taken as reference to all other groups, but it does not compare all groups to each others.

So to recap:

the Tukey HSD test allows to compares all groups but at the cost of less power
the Dunnett’s test allows to only make comparisons with a reference group, but with the benefit of more power

Now, again for the sake of illustration, consider that the species Adelie is the reference species and we are only interested in comparing the reference species against the other 2 species. In that scenario, we would use the Dunnett’s test.

In R, the Dunnett’s test is done as follows (the only difference with the code for the Tukey HSD test is in the line linfct = mcp(species = "Dunnett")):

library(multcomp)

# Dunnett's test:
post_test <- glht(res_aov,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Chinstrap - Adelie == 0   5.8699     0.9699   6.052 7.59e-09 ***
## Gentoo - Adelie == 0     27.2333     0.8067  33.760  < 1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

The interpretation is the same as for the Tukey HSD test’s except that in the Dunett’s test we only compare:

Chinstrap versus Adelie (line Chinstrap - Adelie == 0)
Gentoo vs. Adelie (line Gentoo - Adelie == 0)

Both adjusted p-values (displayed in the last column) are below 0.05, so we reject the null hypothesis for both comparisons.

This means that both the species Chinstrap and Gentoo are significantly different from the reference species Adelie in terms of flippers length. (Nothing can be said about the comparison between Chinstrap and Gentoo though.)

Again, the results of the post-hoc test can be visualized with the plot() function:

par(mar = c(3, 8, 3, 3))
plot(post_test)

We see that the confidence intervals do not cross the zero line, which indicate that both the species Gentoo and Chinstrap are significantly different from the reference species Adelie.

Note that in R, by default, the reference category for a factor variable is the first category in alphabetical order. This is the reason that, by default, the reference species is Adelie.

The reference category can be changed with the relevel() function (or with the {questionr} addin). Considering that we want Gentoo as the reference category instead of Adelie:

# Change reference category:
dat$species <- relevel(dat$species, ref = "Gentoo")

# Check that Gentoo is the reference category:
levels(dat$species)

## [1] "Gentoo"    "Adelie"    "Chinstrap"

Gentoo now being the first category of the three, it is indeed considered as the reference level.

In order to perform the Dunnett’s test with the new reference we first need to rerun the ANOVA to take into account the new reference:

res_aov2 <- aov(flipper_length_mm ~ species,
  data = dat
)

We can then run the Dunett’s test with the new results of the ANOVA:

# Dunnett's test:
post_test <- glht(res_aov2,
  linfct = mcp(species = "Dunnett")
)

summary(post_test)

## 
## 	 Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Dunnett Contrasts
## 
## 
## Fit: aov(formula = flipper_length_mm ~ species, data = dat)
## 
## Linear Hypotheses:
##                         Estimate Std. Error t value Pr(>|t|)    
## Adelie - Gentoo == 0    -27.2333     0.8067  -33.76   <1e-10 ***
## Chinstrap - Gentoo == 0 -21.3635     1.0036  -21.29   <1e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)

par(mar = c(3, 8, 3, 3))
plot(post_test)

From the results above we conclude that Adelie and Chinstrap species are significantly different from Gentoo species in terms of flippers length (adjusted p-values < 1e-10).

Note that even if your study does not have a reference group which you can compare to the other groups, it is still often better to do multiple comparisons determined by some research questions than to do all-pairwise tests. By reducing the number of post-hoc comparisons to what is necessary only, and no more, you maximize the statistical power.⁸

Other p-values adjustment methods

For the interested readers, note that you can use other p-values adjustment methods by using the pairwise.t.test() function:

pairwise.t.test(dat$flipper_length_mm, dat$species,
  p.adjust.method = "holm"
)

## 
## 	Pairwise comparisons using t tests with pooled SD 
## 
## data:  dat$flipper_length_mm and dat$species 
## 
##           Gentoo  Adelie 
## Adelie    < 2e-16 -      
## Chinstrap < 2e-16 3.8e-09
## 
## P value adjustment method: holm

By default, the Holm method is applied but other methods exist. See ?p.adjust for all available options.

Visualization of ANOVA and post-hoc tests on the same plot

If you are interested in including results of ANOVA and post-hoc tests on the same plot (directly on the boxplots), here are two pieces of code which may be of interest to you.

The first one is edited by me based on the code found in this article:

# Edit from here
x <- which(names(dat) == "species") # name of grouping variable
y <- which(
  names(dat) == "flipper_length_mm" # names of variables to test
)
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("Chinstrap", "Adelie"), c("Gentoo", "Adelie"), c("Gentoo", "Chinstrap")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
library(ggpubr)
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

And the second method is from the {ggstatsplot} package:

library(ggstatsplot)

ggbetweenstats(
  data = dat,
  x = species,
  y = flipper_length_mm,
  type = "parametric", # ANOVA or Kruskal-Wallis
  var.equal = TRUE, # ANOVA or Welch ANOVA
  plot.type = "box",
  pairwise.comparisons = TRUE,
  pairwise.display = "significant",
  centrality.plotting = FALSE,
  bf.message = FALSE
)

As you can see on the above plot, boxplots by species are presented together with p-values of the ANOVA (after p = in the subtitle of the plot) and p-values of the post-hoc tests (above each comparison).

Besides the fact that these methods can be used to combine a visual representation and statistical results on the same plot, they also have the advantage that you can perform multiple ANOVA tests at once. See more information in this article.

Summary

In this article, we reviewed the goals and hypotheses of an ANOVA, what are the assumptions which need to be verified before being able to trust the results (namely, independence, normality and homogeneity), we then showed how to do an ANOVA in R and how to interpret the results.

An article about ANOVA would not be complete without discussing about post-hoc tests, and in particular, the Tukey HSD—to compare all groups—and the Dunnett’s test—to compare a reference group to all other groups.

Last but not least, we showed how to visualize the data and the results of the ANOVA and post-hoc tests in the same plot.

Thanks for reading. See this tutorial if you would like to learn how to do an ANOVA by hand.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Hsu, Jason. 1996. Multiple Comparisons: Theory and Methods. CRC Press.

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Note that it is called one-way or one-factor ANOVA because the means relate to the different modalities of a single independent variable, or factor.↩︎
Residuals (denoted $\epsilon$) are the differences between the observed values of the dependent variable ($y$) and the predicted values ($\hat{y}$). In the context of ANOVA, residuals correspond to the differences between the observed values and the mean of all values for that group.↩︎
Stevens (2013) wrote, in p. 57, “Numerous studies have examined the effect of violations of assumptions in ANOVA, and an excellent summary of this literature has been provided by Glass, Peckham, and Sanders (1972). Their review indicates that non normality has only a slight effect on the type I error rate, even for very skewed or kurtotic distributions. For example, the actual $\alpha$s for some very non-normal populations were only .055 or .06: very minor deviations from the nominal level of .05. […] The basic reason is the Central Limit Theorem, which states that the sum of independent observations having any distribution whatsoever approaches a normal distribution as the number of observations increases. To be somewhat more specific, Bock (1975) notes,”even for distributions which depart markedly from normality, sums of 50 or more observations approximate to normality. For moderately non-normal distributions the approximation is good with as few as 10 to 20 observations” (p. 111). Now since the sums of independent observations approach normality rapidly, so do the means, and the sampling distribution of F is based on means. Thus the sampling distribution of F is only slightly affected, and therefore the critical values when sampling from normal and non-normal distributions will not differ by much. Lack of normality due to skewness also has only a slight effect on power (a few hundredths).”↩︎
As long as you use the Kruskal-Wallis test to, in fine, compare groups, homoscedasticity is not required. If you wish to compare medians, the Kruskal-Wallis test requires homoscedasticity. See more information about the difference in this article.↩︎
Note that, as discussed in the comments at the end of the article, post-hoc tests can under some circumstances be done directly (without an ANOVA). See the comments or Hsu (1996) for more details.↩︎
Note that you could in principle apply the Bonferroni correction to all tests. For example, in the example above, with 3 tests and a global desired significance level of $\alpha$ = 0.05, we would only reject a null hypothesis if the p-value is less than $\frac{0.05}{3}$ = 0.0167. This method is, however, known to be quite conservative, leading to a potentially high rate of false negatives.↩︎
The p-values are adjusted to keep the global significance level to the desired level.↩︎
Thanks Michael Friendly for this suggestion.↩︎

Outliers detection in R

Tue, 11 Aug 2020 00:00:00 +0000

Introduction

An outlier is a value or an observation that is distant from other observations, that is to say, a data point that differs significantly from other data points. Enderlein (1987) goes even further as the author considers outliers as values that deviate so much from other observations one might suppose a different underlying sampling mechanism.

An observation must always be compared to other observations made on the same phenomenon before actually calling it an outlier. Indeed, someone who is 200 cm tall (6’7” in US) will most likely be considered as an outlier compared to the general population, but that same person may not be considered as an outlier if we measured the height of basketball players.

An outlier may be due to the variability inherent in the observed phenomenon. For example, it is often the case that there are outliers when collecting data on salaries, as some people make much more money than the rest.

Outliers can also arise due to an experimental, measurement or encoding error. For instance, a human weighting 786 kg (1733 pounds) is clearly an error when encoding the weight of the subject. Her or his weight is most probably 78.6 kg (173 pounds) or 7.86 kg (17 pounds) depending on whether weights of adults or babies have been measured.

For this reason, it sometimes makes sense to formally distinguish two classes of outliers: (i) extreme values and (ii) mistakes. Extreme values are statistically and philosophically more interesting, because they are possible but unlikely responses.¹

In this article, I present several approaches to detect outliers in R, from simple techniques such as descriptive statistics (including minimum, maximum, histogram, boxplot and percentiles) to more formal techniques such as the Hampel filter, the Grubbs, the Dixon and the Rosner tests for outliers.

Although there is no strict or unique rule whether outliers should be removed or not from the dataset before doing statistical analyses, it is quite common to, at least, remove or impute outliers that are due to an experimental or measurement error (like the weight of 786 kg (1733 pounds) for a human). Some statistical tests require the absence of outliers in order to draw sound conclusions, but removing outliers is not recommended in all cases and must be done with caution.

This article will not tell you whether you should remove outliers or not (nor if you should impute them with the median, mean, mode or any other value), but it will help you to detect them in order to, as a first step, verify them. After their verification, it is then your choice to exclude or include them for your analyses (and this usually requires a thoughtful reflection on the researcher’s side).

Removing or keeping outliers mostly depend on three factors:

The domain/context of your analyses and the research question. In some domains, it is common to remove outliers as they often occur due to a malfunctioning process. In other fields, outliers are kept because they contain valuable information. It also happens that analyses are performed twice, once with and once without outliers to evaluate their impact on the conclusions. If results change drastically due to some influential values, this should caution the researcher to make overambitious claims.
Whether the tests you are going to apply are robust to the presence of outliers or not. For instance, the slope of a simple linear regression may significantly varies with just one outlier, whereas non-parametric tests such as the Wilcoxon test are usually robust to outliers.
How distant are the outliers from other observations? Some observations considered as outliers (according to the techniques presented below) are actually not really extreme compared to all other observations, while other potential outliers may be really distant from the rest of the observations.

The dataset mpg from the {ggplot2} package will be used to illustrate the different approaches of outliers detection in R, and in particular we will focus on the variable hwy (highway miles per gallon). We will also use some simulated data for the presentation of the outlier tests.

Descriptive statistics

Several methods using descriptive statistics exist. We present the most common ones below.

Minimum and maximum

The first step to detect outliers in R is to start with some descriptive statistics, and in particular with the minimum and maximum.

In R, this can easily be done with the summary() function:

dat <- ggplot2::mpg
summary(dat$hwy)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   12.00   18.00   24.00   23.44   27.00   44.00

where the minimum and maximum are respectively the first and last values in the output above.

Alternatively, they can also be computed with the min() and max(), or range() functions:²

min(dat$hwy)

## [1] 12

max(dat$hwy)

## [1] 44

range(dat$hwy)

## [1] 12 44

Some clear encoding mistake like a weight of 786 kg (1733 pounds) for a human will already be easily detected by this very simple technique.

Histogram

Another basic way to detect outliers is to draw a histogram of the data.

Using R base (with the number of bins corresponding to the square root of the number of observations in order to have more bins than the default option):

hist(dat$hwy,
  xlab = "hwy",
  main = "Histogram of hwy",
  breaks = sqrt(length(dat$hwy)) # set number of bins
)

or using ggplot2 (learn how to create plots with this package via the esquisse addin or via this tutorial):

library(ggplot2)

ggplot(dat) +
  aes(x = hwy) +
  geom_histogram(
    bins = round(sqrt(length(dat$hwy))), # set number of bins
    fill = "steelblue", color = "black"
  ) +
  theme_minimal()

From the histograms, we see that there seems to be a couple of observations larger than all other observations (see the bars on the right side of the plot).

Boxplot

In addition to histograms, boxplots are also useful to detect potential outliers.

Using R base:

boxplot(dat$hwy,
  ylab = "hwy"
)

or using ggplot2:

ggplot(dat) +
  aes(x = "", y = hwy) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal()

A boxplot helps to visualize a quantitative variable by displaying five common location summary (minimum, median, first and third quartiles and maximum) and any observation that was classified as a suspected outlier using the interquartile range (IQR) criterion.

The IQR criterion means that all observations above $q_{0.75} + 1.5 \cdot IQR$ or below $q_{0.25} - 1.5 \cdot IQR$ (where $q_{0.25}$ and $q_{0.75}$ correspond to first and third quartile respectively, and IQR is the difference between the third and first quartile) are considered as potential outliers by R.

In other words, all observations outside of the following interval will be considered as potential outliers:

\[I = [q_{0.25} - 1.5 \cdot IQR; q_{0.75} + 1.5 \cdot IQR]\]

Observations considered as potential outliers by the IQR criterion are displayed as points in the boxplot. Based on this criterion, there are 2 potential outliers (see the 2 points above the vertical line, at the top of the boxplot).

Remember that it is not because an observation is considered as a potential outlier by the IQR criterion that you should remove it. Removing or keeping an outlier depends on:

the context of your analysis,
whether the tests you are going to perform on the dataset are robust to outliers or not, and
how extreme is the outlier (so how far is the outlier from other observations).

It is also possible to extract the values of the potential outliers based on the IQR criterion thanks to the boxplot.stats()$out function:

boxplot.stats(dat$hwy)$out

## [1] 44 44 41

As you can see, there are actually 3 points considered as potential outliers: 2 observations with a value of 44 and 1 observation with a value of 41.

Thanks to the which() function it is possible to extract the row number corresponding to these outliers:

out <- boxplot.stats(dat$hwy)$out
out_ind <- which(dat$hwy %in% c(out))
out_ind

## [1] 213 222 223

With this information you can now easily go back to the specific rows in the dataset to verify them, or print all variables for these outliers:

dat[out_ind, ]

## # A tibble: 3 × 11
##   manufacturer model      displ  year   cyl trans  drv     cty   hwy fl    class
##   <chr>        <chr>      <dbl> <int> <int> <chr>  <chr> <int> <int> <chr> <chr>
## 1 volkswagen   jetta        1.9  1999     4 manua… f        33    44 d     comp…
## 2 volkswagen   new beetle   1.9  1999     4 manua… f        35    44 d     subc…
## 3 volkswagen   new beetle   1.9  1999     4 auto(… f        29    41 d     subc…

Another method to display these specific rows is with the identify_outliers() function from the {rstatix} package:³

library(rstatix)

identify_outliers(
  data = dat,
  variable = "hwy"
)

## # A tibble: 3 × 13
##   manufacturer model      displ  year   cyl trans  drv     cty   hwy fl    class
##   <chr>        <chr>      <dbl> <int> <int> <chr>  <chr> <int> <int> <chr> <chr>
## 1 volkswagen   jetta        1.9  1999     4 manua… f        33    44 d     comp…
## 2 volkswagen   new beetle   1.9  1999     4 manua… f        35    44 d     subc…
## 3 volkswagen   new beetle   1.9  1999     4 auto(… f        29    41 d     subc…
## # ℹ 2 more variables: is.outlier <lgl>, is.extreme <lgl>

It is also possible to print the values of the outliers directly on the boxplot with the mtext() function:

boxplot(dat$hwy,
  ylab = "hwy",
  main = "Boxplot of highway miles per gallon"
)
mtext(paste("Outliers: ", paste(out, collapse = ", ")))

Percentiles

This method of outliers detection is based on the percentiles.

With the percentiles method, all observations that lie outside the interval formed by the 2.5 and 97.5 percentiles will be considered as potential outliers. Other percentiles such as the 1 and 99, or the 5 and 95 percentiles can also be considered to construct the interval.

The values of the lower and upper percentiles (and thus the lower and upper limits of the interval) can be computed with the quantile() function:

lower_bound <- quantile(dat$hwy, 0.025)
lower_bound

## 2.5% 
##   14

upper_bound <- quantile(dat$hwy, 0.975)
upper_bound

##  97.5% 
## 35.175

According to this method, all observations below 14 and above 35.175 will be considered as potential outliers. The row numbers of the observations outside of the interval can then be extracted with the which() function:

outlier_ind <- which(dat$hwy < lower_bound | dat$hwy > upper_bound)
outlier_ind

##  [1]  55  60  66  70 106 107 127 197 213 222 223

Then their values of highway miles per gallon can be printed:

dat[outlier_ind, "hwy"]

## # A tibble: 11 × 1
##      hwy
##    <int>
##  1    12
##  2    12
##  3    12
##  4    12
##  5    36
##  6    36
##  7    12
##  8    37
##  9    44
## 10    44
## 11    41

Alternatively, all variables for these outliers can be printed:

dat[outlier_ind, ]

## # A tibble: 11 × 11
##    manufacturer model      displ  year   cyl trans drv     cty   hwy fl    class
##    <chr>        <chr>      <dbl> <int> <int> <chr> <chr> <int> <int> <chr> <chr>
##  1 dodge        dakota pi…   4.7  2008     8 auto… 4         9    12 e     pick…
##  2 dodge        durango 4…   4.7  2008     8 auto… 4         9    12 e     suv  
##  3 dodge        ram 1500 …   4.7  2008     8 auto… 4         9    12 e     pick…
##  4 dodge        ram 1500 …   4.7  2008     8 manu… 4         9    12 e     pick…
##  5 honda        civic        1.8  2008     4 auto… f        25    36 r     subc…
##  6 honda        civic        1.8  2008     4 auto… f        24    36 c     subc…
##  7 jeep         grand che…   4.7  2008     8 auto… 4         9    12 e     suv  
##  8 toyota       corolla      1.8  2008     4 manu… f        28    37 r     comp…
##  9 volkswagen   jetta        1.9  1999     4 manu… f        33    44 d     comp…
## 10 volkswagen   new beetle   1.9  1999     4 manu… f        35    44 d     subc…
## 11 volkswagen   new beetle   1.9  1999     4 auto… f        29    41 d     subc…

There are 11 potential outliers according to the percentiles method. To reduce this number, you can set the percentiles to 1 and 99:

lower_bound <- quantile(dat$hwy, 0.01)
upper_bound <- quantile(dat$hwy, 0.99)

outlier_ind <- which(dat$hwy < lower_bound | dat$hwy > upper_bound)

dat[outlier_ind, ]

## # A tibble: 3 × 11
##   manufacturer model      displ  year   cyl trans  drv     cty   hwy fl    class
##   <chr>        <chr>      <dbl> <int> <int> <chr>  <chr> <int> <int> <chr> <chr>
## 1 volkswagen   jetta        1.9  1999     4 manua… f        33    44 d     comp…
## 2 volkswagen   new beetle   1.9  1999     4 manua… f        35    44 d     subc…
## 3 volkswagen   new beetle   1.9  1999     4 auto(… f        29    41 d     subc…

Setting the percentiles to 1 and 99 gives the same potential outliers as with the IQR criterion.

Z-scores

If your data come from a normal distribution, you can use the z-scores, defined as

\[z_i = \frac{x_i - \overline{X}}{s_X}\]

where $\overline{X}$ is the mean and $s_X$ is the standard deviation of the random variable $X$.

This is referred as scaling, which can be done with the scale() function in R.

According to this method, any z-score:

< -2 or > 2 is considered as rare
< -3 or > 3 is considered as extremely rare

Other authors also use a z-score below -3.29 or above 3.29 to detect outliers. This value of 3.29 comes from the fact that 1 observation out of 1000 is out of this interval if the data follow a normal distribution.

In our case:

dat$z_hwy <- scale(dat$hwy)

hist(dat$z_hwy)

summary(dat$z_hwy)

##        V1          
##  Min.   :-1.92122  
##  1st Qu.:-0.91360  
##  Median : 0.09402  
##  Mean   : 0.00000  
##  3rd Qu.: 0.59782  
##  Max.   : 3.45274

We see that there are some observations above 3.29, but none below -3.29.

To identify the line in the dataset of these observations:

which(dat$z_hwy > 3.29)

## [1] 213 222

We see that observations 213 and 222 can be considered as outliers according to this method.

Hampel filter

Another method, known as Hampel filter, consists of considering as outliers the values outside the interval ($I$) formed by the median, plus or minus 3 median absolute deviations ($MAD$):⁴

\[I = [median - 3 \cdot MAD; median + 3 \cdot MAD]\]

where $MAD$ is the median absolute deviation and is defined as the median of the absolute deviations from the data’s median $\tilde{X} = median(X)$:

\[MAD = median(|X_i - \tilde{X}|)\]

For this method we first set the interval limits thanks to the median() and mad() functions:⁵

lower_bound <- median(dat$hwy) - 3 * mad(dat$hwy, constant = 1)
lower_bound

## [1] 9

upper_bound <- median(dat$hwy) + 3 * mad(dat$hwy, constant = 1)
upper_bound

## [1] 39

According to this method, all observations below 9 and above 39 will be considered as potential outliers. The row numbers of the observations outside of the interval can then be extracted with the which() function:

outlier_ind <- which(dat$hwy < lower_bound | dat$hwy > upper_bound)
outlier_ind

## [1] 213 222 223

According to the Hampel filter, there are 3 outliers for the hwy variable.

Statistical tests

In this section, we present 3 hypothesis tests to detect outliers:

Grubbs’s test
Dixon’s test
Rosner’s test

These 3 statistical tests are part of more formal techniques of outliers detection as they all involve the computation of a test statistic that is compared to tabulated critical values (that are based on the sample size and the desired confidence level).

Note that the 3 tests are appropriate only when the data, without any outliers, are approximately normally distributed. It is recommended to check normality visually, with a QQ-plot, a histogram and/or a boxplot for instance. Although it can also be checked with a formal test for normality (such as the Shapiro-Wilk test for example), the presence of one or more outliers may cause the normality test to reject normality when it is in fact a reasonable assumption for applying one of the 3 outlier tests mentioned above.

We check the normality of our data thanks to a QQ-plot:

library(car)
qqPlot(dat$hwy)

## [1] 213 222

Too many points deviate from the Henry’s line, so based on the QQ-plot we conclude that our data do not follow a normal distribution. Therefore, we should not use one of the outlier test on these data.

For the sake of completeness, here is an example of data that could be used with the three outlier tests, together with the QQ-plot of these data:

dat_tests <- c(rnorm(50), 5)

qqPlot(dat_tests)

## [1] 51 18

hist(dat_tests)

For the illustration of the three outlier tests in the next sections, we thus use the simulated data (dat_tests) instead of the data used so far (dat$hwy).

Grubbs’s test

The Grubbs test allows to detect whether the highest or lowest value in a dataset is an outlier.

The Grubbs test detects one outlier at a time (highest or lowest value), so the null and alternative hypotheses are as follows:

$H_0$: The highest value is not an outlier
$H_1$: The highest value is an outlier

if we want to test the highest value, or:

$H_0$: The lowest value is not an outlier
$H_1$: The lowest value is an outlier

if we want to test the lowest value.

As for any statistical test, if the p-value is less than the chosen significance threshold (generally $\alpha = 0.05$) then the null hypothesis is rejected and we will conclude that the lowest/highest value is an outlier.

On the contrary, if the p-value is greater or equal than the significance level, the null hypothesis is not rejected, and we will conclude that, based on the data, we do not reject the hypothesis that the lowest/highest value is not an outlier.

Note that the Grubbs test is not appropriate for sample size of 6 or less ($n \le 6$).

To perform the Grubbs test in R, we use the grubbs.test() function from the {outliers} package:

# install.packages("outliers")
library(outliers)

# Grubbs test
test <- grubbs.test(dat_tests)
test

## 
## 	Grubbs test for one outlier
## 
## data:  dat_tests
## G = 3.68326, U = 0.72325, p-value = 0.001873
## alternative hypothesis: highest value 5 is an outlier

The p-value is 0.002. At the 5% significance level, we reject the hypothesis that the highest value 5 is not an outlier. In other words, based on this test, we conclude that the highest value 5 is an outlier.

By default, the test is performed on the highest value (as shown in the R output: alternative hypothesis: highest value 5 is an outlier). If you want to do the test for the lowest value, simply add the argument opposite = TRUE in the grubbs.test() function:

test <- grubbs.test(dat_tests, opposite = TRUE)
test

## 
## 	Grubbs test for one outlier
## 
## data:  dat_tests
## G = 2.02893, U = 0.91602, p-value = 0.9981
## alternative hypothesis: lowest value -2.65645542090478 is an outlier

The R output indicates that the test is now performed on the lowest value (see alternative hypothesis: lowest value -2.6564554 is an outlier).

The p-value is 0.998. At the 5% significance level, we do not reject the hypothesis that the lowest value -2.66 is not an outlier. In other words, we cannot conlude that the lowest value -2.66 is an outlier.

Dixon’s test

Similar to the Grubbs test, Dixon test is used to test whether a single low or high value is an outlier. So if more than one outliers is suspected, the test has to be performed on these suspected outliers individually.

Note that Dixon test is most useful for small sample size (usually $n \le 25$).

To perform the Dixon’s test in R, we use the dixon.test() function from the {outliers} package.

For this illustration, as the Dixon test can only be done on small samples, we take a subset of our simulated data which consists of the 20 first observations and the outlier. Note that R will throw an error and accepts only a dataset consisting of 3 to 30 observations for this test.

# subset of simulated data
subdat <- c(dat_tests[1:20], max(dat_tests))

# Dixon test
test <- dixon.test(subdat)
test

## 
## 	Dixon test for outliers
## 
## data:  subdat
## Q = 0.46668, p-value = 0.06429
## alternative hypothesis: highest value 5 is an outlier

Results of the Dixon test show that we cannot conclude that the highest value 5 is an outlier (p-value = 0.064).

To test for the lowest value, simply add the opposite = TRUE argument to the dixon.test() function:

test <- dixon.test(subdat,
  opposite = TRUE
)
test

## 
## 	Dixon test for outliers
## 
## data:  subdat
## Q = 0.27115, p-value = 0.6872
## alternative hypothesis: lowest value -2.65645542090478 is an outlier

Results of the test show that we cannot conclude that the lowest value -2.66 is an outlier (p-value = 0.687).

It is a good practice to always check the results of the statistical test for outliers against the boxplot to make sure we tested all potential outliers:

out <- boxplot.stats(subdat)$out

boxplot(subdat)
mtext(paste("Outlier: ", paste(out, collapse = ", ")))

From the boxplot, we see that the Dixon test has been applied to all potential outliers.

If you need to perform the test again without the highest or lowest value, this can be done by finding the row number of the maximum or minimum value, excluding this row number from the dataset and then finally apply the Dixon test on this new dataset. We illustrate the process as if we needed to perform the test again, but this time on the data excluding the highest value:

# find and exclude highest value
remove_ind <- which.max(subdat)
subsubdat <- subdat[-remove_ind]

# Dixon test on dataset without the maximum
test <- dixon.test(subsubdat)
test

## 
## 	Dixon test for outliers
## 
## data:  subsubdat
## Q = 0.30413, p-value = 0.5547
## alternative hypothesis: lowest value -2.65645542090478 is an outlier

Results show that we cannot conclude that the lowest value -2.66 is an outlier (p-value = 0.555).

Rosner’s test

Rosner’s test for outliers has the advantages that:

it is used to detect several outliers at once (unlike Grubbs and Dixon test which must be performed iteratively to screen for multiple outliers), and
it is designed to avoid the problem of masking, where an outlier that is close in value to another outlier can go undetected.

Unlike Dixon test, note that Rosner test is most appropriate when the sample size is large ($n \ge 20$). We therefore use again the similated dataset (i.e., dat_tests), which includes 51 observations.

To perform the Rosner test, we use the rosnerTest() function from the {EnvStats} package. This function requires at least 2 arguments: the data and the number of suspected outliers k (with k = 3 as the default number of suspected outliers).

For this example, we set the number of suspected outliers to be equal to 1, as suggested by the number of potential outliers outlined in the boxplot.⁶

library(EnvStats)

# Rosner test
test <- rosnerTest(dat_tests,
  k = 1
)
test

## 
## Results of Outlier Test
## -------------------------
## 
## Test Method:                     Rosner's Test for Outliers
## 
## Hypothesized Distribution:       Normal
## 
## Data:                            dat_tests
## 
## Sample Size:                     51
## 
## Test Statistic:                  R.1 = 3.683258
## 
## Test Statistic Parameter:        k = 1
## 
## Alternative Hypothesis:          Up to 1 observations are not
##                                  from the same Distribution.
## 
## Type I Error:                    5%
## 
## Number of Outliers Detected:     1
## 
##   i     Mean.i     SD.i Value Obs.Num    R.i+1 lambda.i+1 Outlier
## 1 0 0.06306688 1.340371     5      51 3.683258   3.136165    TRUE

The interesting results are provided in the $all.stats table:

test$all.stats

##   i     Mean.i     SD.i Value Obs.Num    R.i+1 lambda.i+1 Outlier
## 1 0 0.06306688 1.340371     5      51 3.683258   3.136165    TRUE

Based on the results of the Rosner test, we see that there is only one outlier (as the table contains 1 line), and that it is the observation 51 (see Obs.Num) with a value of 5 (see Value). This finding aligns with the Grubbs test presented above, which also detected the value 5 as an outlier.

Additional remarks

You will find many other methods to detect outliers:

in the {outliers} packages,
via the lofactor() function from the {DMwR} package: Local Outlier Factor (LOF) is an algorithm used to identify outliers by comparing the local density of a point with that of its neighbors,
the outlierTest() from the {car} package gives the most extreme observation based on the given model and allows to test whether it is an outlier,
in the {OutlierDetection} package, and
with the aq.plot() function from the {mvoutlier} package (Thanks KTR for the suggestion.).

We present these methods with the initial dataset mpg, using the cyl and hwy variables:

library(mvoutlier)

Y <- as.matrix(ggplot2::mpg[, c("cyl", "hwy")])
res <- aq.plot(Y)

Note also that some transformations may “naturally” eliminate outliers. The natural log or square root of a value reduces the variation caused by extreme values, so in some cases applying these transformations will help in making potential outliers less extreme.

Conclusion

Thanks for reading.

I hope this article helped you to detect outliers in R via several descriptive statistics (including minimum, maximum, histogram, boxplot and percentiles) or thanks to more formal techniques of outliers detection (including Hampel filter, Grubbs, Dixon and Rosner tests).

It is now your turn to try to detect outliers in your data, and decide how to treat them (i.e., keeping, removing or imputing them) before conducting your analyses.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Enderlein, G. 1987. “Hawkins, DM: Identification of Outliers. Chapman and Hall, London–New York 1980, 188 s.,£ 14, 50.” Biometrical Journal 29 (2): 198–98.

Thanks Felix Kluxen for the valuable suggestion.↩︎
Thanks to Victor for pointing out the range() function.↩︎
Thanks to Marlenildo for pointing out the identify_outliers() function.↩︎
The default is 3 (according to Pearson’s rule), but another value is also possible.↩︎
The constant in the mad() function is 1.4826 by default, so it has to be set to 1 to find the median absolute deviation. See help(mad) for more details. Thanks to Elisei for pointing this out to me.↩︎
In order to avoid flawed conclusions, it is important to pre-screen the data (graphically with a boxplot for example) to make the selection of the number of potential outliers as accurate as possible prior to running Rosner’s test.↩︎

Wilcoxon test in R: how to compare 2 groups under the non-normality assumption?

Sun, 07 Jun 2020 00:00:00 +0000

Introduction

In a previous article, we showed how to compare two groups under different scenarios using the Student’s t-test. The Student’s t-test requires that the distributions follow a normal distribution when in presence of small samples.¹

In this article, we show how to compare two groups when the normality assumption is violated, using the Wilcoxon test.

The Wilcoxon test is a non-parametric test, meaning that it does not rely on data belonging to any particular parametric family of probability distributions. Non-parametric tests have the same objective as their parametric counterparts. However, they have two advantages over parametric tests: they do not require the assumption of normality of distributions and they can deal with outliers.

A Student’s t-test for instance is only applicable if the data are Gaussian or if the sample size is large enough (usually $n \ge 30$, thanks to the central limit theorem). A non-parametric test should be used in other cases.

One may wonder why we would not always use a non-parametric test so we do not have to bother about testing for normality. The reason is that non-parametric tests are usually less powerful than corresponding parametric tests when the normality assumption holds.

Therefore, all else being equal, with a non-parametric test you are less likely to reject the null hypothesis when it is false if the data follows a normal distribution. It is thus preferred to use the parametric version of a statistical test when the assumptions are met.

In the remaining of the article, we present the two scenarios of the Wilcoxon test and how to perform them in R through two examples.

Two different scenarios

As for the Student’s t-test, the Wilcoxon test is used to compare two groups and see whether they are significantly different from each other in terms of the variable of interest.

The two groups to be compared are either:

independent, or
paired (i.e., dependent)

There are actually two versions of the Wilcoxon test:

The Mann-Whitney-Wilcoxon test (also referred as Wilcoxon rank sum test or Mann-Whitney U test) is performed when the samples are independent (so this test is the non-parametric equivalent to the Student’s t-test for independent samples).
The Wilcoxon signed-rank test (also sometimes referred as Wilcoxon test for paired samples) is performed when the samples are paired/dependent (so this test is the non-parametric equivalent to the Student’s t-test for paired samples).

Luckily, those two tests can be done in R with the same function: wilcox.test(). They are presented in the following sections.

Independent samples

For the Wilcoxon test with independent samples, suppose that we want to test whether grades at the statistics exam differ between female and male students.

We have collected grades for 24 students (12 girls and 12 boys):

dat <- data.frame(
  Sex = as.factor(c(rep("Girl", 12), rep("Boy", 12))),
  Grade = c(
    19, 18, 9, 17, 8, 7, 16, 19, 20, 9, 11, 18,
    16, 5, 15, 2, 14, 15, 4, 7, 15, 6, 7, 14
  )
)

dat

##     Sex Grade
## 1  Girl    19
## 2  Girl    18
## 3  Girl     9
## 4  Girl    17
## 5  Girl     8
## 6  Girl     7
## 7  Girl    16
## 8  Girl    19
## 9  Girl    20
## 10 Girl     9
## 11 Girl    11
## 12 Girl    18
## 13  Boy    16
## 14  Boy     5
## 15  Boy    15
## 16  Boy     2
## 17  Boy    14
## 18  Boy    15
## 19  Boy     4
## 20  Boy     7
## 21  Boy    15
## 22  Boy     6
## 23  Boy     7
## 24  Boy    14

Here are the distributions of the grades by sex (using {ggplot2}):

library(ggplot2)

ggplot(dat) +
  aes(x = Sex, y = Grade) +
  geom_boxplot(fill = "#0c4c8a") +
  theme_minimal()

We first check whether the 2 samples follow a normal distribution via a histogram and the Shapiro-Wilk test:

hist(subset(dat, Sex == "Girl")$Grade,
  main = "Grades for girls",
  xlab = "Grades"
)

hist(subset(dat, Sex == "Boy")$Grade,
  main = "Grades for boys",
  xlab = "Grades"
)

shapiro.test(subset(dat, Sex == "Girl")$Grade)

## 
## 	Shapiro-Wilk normality test
## 
## data:  subset(dat, Sex == "Girl")$Grade
## W = 0.84548, p-value = 0.0323

shapiro.test(subset(dat, Sex == "Boy")$Grade)

## 
## 	Shapiro-Wilk normality test
## 
## data:  subset(dat, Sex == "Boy")$Grade
## W = 0.84313, p-value = 0.03023

The histograms show that both distributions do not seem to follow a normal distribution and the p-values of the Shapiro-Wilk tests confirm it (since we reject the null hypothesis of normality for both distributions at the 5% significance level).

We just showed that normality assumption is violated for both groups so it is now time to see how to perform the Wilcoxon test in R.

Note that in order to use the Student’s t-test (the parametric version of the Wilcoxon test), it is required that both samples follow a normal distribution if samples are small.² Therefore, even if one sample follows a normal distribution (and the other does not follow a normal distribution), it is recommended to use the non-parametric test.

Remember that the null and alternative hypothesis of the Wilcoxon test are as follows:

$H_0$: the 2 groups are equal in terms of the variable of interest
$H_1$: the 2 groups are different in terms of the variable of interest

Applied to our research question, we have:

$H_0$: grades of girls and boys are equal
$H_1$: grades of girls and boys are different

test <- wilcox.test(dat$Grade ~ dat$Sex)
test

## 
## 	Wilcoxon rank sum test with continuity correction
## 
## data:  dat$Grade by dat$Sex
## W = 31.5, p-value = 0.02056
## alternative hypothesis: true location shift is not equal to 0

We obtain the test statistic, the p-value and a reminder of the hypothesis tested.³

The p-value is 0.021. Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that grades are significantly different between girls and boys.

Given the boxplot presented above showing the grades by sex, one may see that girls seem to perform better than boys. This can be tested formally by adding the alternative = "less" argument to the wilcox.test() function:⁴

test <- wilcox.test(dat$Grade ~ dat$Sex,
  alternative = "less"
)
test

## 
## 	Wilcoxon rank sum test with continuity correction
## 
## data:  dat$Grade by dat$Sex
## W = 31.5, p-value = 0.01028
## alternative hypothesis: true location shift is less than 0

The p-value is 0.01. Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that boys performed significantly worse than girls (which is equivalent than concluding that girls performed significantly better than boys).

Paired samples

For this second scenario, consider that we administered a math test in a class of 12 students at the beginning of a semester, and that we administered a similar test at the end of the semester to the exact same students. We have the following data:

dat2 <- data.frame(
  Beginning = c(16, 5, 15, 2, 14, 15, 4, 7, 15, 6, 7, 14),
  End = c(19, 18, 9, 17, 8, 7, 16, 19, 20, 9, 11, 18)
)

dat2

##    Beginning End
## 1         16  19
## 2          5  18
## 3         15   9
## 4          2  17
## 5         14   8
## 6         15   7
## 7          4  16
## 8          7  19
## 9         15  20
## 10         6   9
## 11         7  11
## 12        14  18

We transform the dataset to have it in a tidy format:

dat2 <- data.frame(
  Time = c(rep("Before", 12), rep("After", 12)),
  Grade = c(dat2$Beginning, dat2$End)
)
dat2

##      Time Grade
## 1  Before    16
## 2  Before     5
## 3  Before    15
## 4  Before     2
## 5  Before    14
## 6  Before    15
## 7  Before     4
## 8  Before     7
## 9  Before    15
## 10 Before     6
## 11 Before     7
## 12 Before    14
## 13  After    19
## 14  After    18
## 15  After     9
## 16  After    17
## 17  After     8
## 18  After     7
## 19  After    16
## 20  After    19
## 21  After    20
## 22  After     9
## 23  After    11
## 24  After    18

The distribution of the grades at the beginning and after the semester:

# Reordering dat2$Time
dat2$Time <- factor(dat2$Time,
  levels = c("Before", "After")
)

ggplot(dat2) +
  aes(x = Time, y = Grade) +
  geom_boxplot(fill = "#0c4c8a") +
  theme_minimal()

(See the {esquisse} and {questionr} addins to help you reorder levels of a factor variable and to easily draw plots with the {ggplot2} package.)

In this example, it is clear that the two samples are not independent since the same 12 students took the exam before and after the semester. Supposing also that the normality assumption is violated (and given the small sample size), we thus use the Wilcoxon test for paired samples, with the following hypotheses:⁵

$H_0$: grades before and after the semester are equal
$H_1$: grades before and after the semester are different

We add the paired = TRUE argument to the wilcox.test() function to take into consideration the dependency between the 2 samples:

before <- dat2$Grade[dat2$Time == "Before"]
after <- dat2$Grade[dat2$Time == "After"]
test <- wilcox.test(before, after, paired = TRUE)

test

## 
## 	Wilcoxon signed rank test with continuity correction
## 
## data:  before and after
## V = 21, p-value = 0.1692
## alternative hypothesis: true location shift is not equal to 0

We obtain the test statistic, the p-value and a reminder of the hypothesis tested.

The p-value is 0.169. Therefore, at the 5% significance level, we do not reject the null hypothesis that the grades are similar before and after the semester.

Combination of plot and statistical test

After having written this article, I discovered the {ggstatsplot} package which I believe is worth mentioning here, in particular the ggbetweenstats() and ggwithinstats() functions for independent and paired samples, respectively.

These two functions combine a boxplot—representing the distribution for each group—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for independent and paired samples, using the same data than previously.

Independent samples

For independent samples, it is the ggbetweenstats() function which is used:

# load package
library(ggstatsplot)

# plot with statistical results
ggbetweenstats( # independent samples
  data = dat,
  x = Sex,
  y = Grade,
  plot.type = "box", # for boxplot
  type = "nonparametric", # for wilcoxon
  centrality.plotting = FALSE # remove median
)

The p-value (displayed after p = in the subtitle of the plot) indicates that we reject the null hypothesis, and we conclude that grades are significantly different between girls and boys (p-value = 0.02).

Paired samples

For paired samples, it is the ggwithinstats() function which is used:

# load package
library(ggstatsplot)

# plot with statistical results
ggwithinstats( # paired samples
  data = dat2,
  x = Time,
  y = Grade,
  type = "nonparametric", # for wilcoxon
  centrality.plotting = FALSE # remove median
)

The p-value (displayed after p = in the subtitle of the plot) indicates that we do not reject the null hypothesis, so we do not reject the hypothesis that grades are equal before and after the semester (p-value = 0.17).

The point of this section was to illustrate how to easily draw plots together with statistical results, which is exactly the aim of the {ggstatsplot} package. See more details and examples in this article.

Assumption of equal variances

As written at the beginning of the article, the Wilcoxon test does not require the assumption of normality in case of small samples.

Regarding the assumption of equal variances, this assumption may or may not be needed depending on your goal. If you only want to compare the two groups, you do not have to test the equality of variances because the two distributions do not have to have the same shape. However, if your goal is to compare medians of the two groups, then you will need to make sure that the two distributions have the same shape (and thus, the same variance).⁶

So results of your test of equality of variances will change your interpretation: differences in the “distributions” of two groups or differences in the “medians” of two groups.

In this article I do not wish to compare medians, I only want compare the groups by determining whether there are differences in the distributions of the two groups. This is the reason I do not test for equality of variances.

Note that this is equivalent when performing the Kruskal-Wallis test to compare three groups or more (i.e., the non-parametric version of the ANOVA): if you only want to test whether there are differences in the groups you do not need homoscedasticity, whereas if you want to compare the medians this assumption must be met.

Conclusion

Thanks for reading.

I hope this article helped you to compare two groups that do not follow a normal distribution in R using the Wilcoxon test. See also:

the one-sample Wilcoxon test if you have only one group and want to compare it to a default given value,
the Student’s t-test if you need to perform the parametric version of the two-sample Wilcoxon test,
and the ANOVA if you need to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Remember that the normality assumption can be tested via 3 complementary methods: (i) histogram, (ii) QQ-plot and (iii) normality tests (with the most common being the Shapiro-Wilk test). See how to determine if a distribution follows a normal distribution if you need a refresh.↩︎
In case of large samples, normality is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013).↩︎
Note that the presence of equal elements (ties) prevents an exact p-value calculation. This can be tackled by computing the exact or asymptotic Wilcoxon-Mann-Whitney test with adjustment for ties, using the wilcox_test() function from the {coin} package: wilcox_test(dat$Grade ~ dat$Sex, distribution = exact()) or wilcox_test(dat$Grade ~ dat$Sex). In our case, conclusions remain unchanged.↩︎
We add alternative = "less" (and not alternative = "greater") because we want to test that grades for boys are less than grade for girls. Using "less" or "greater" can be deducted from the reference level in the dataset.↩︎
Note that for paired samples (when in presence of a small sample), normality must be checked on the differences between the two paired samples, and not individually on the two samples like it is done for independent samples.↩︎
See these three articles for a more detailed discussion on the assumption of equal variances in Wilcoxon test: 1, 2 & 3.↩︎

Correlation coefficient and correlation test in R

Thu, 28 May 2020 00:00:00 +0000

Introduction

Correlations between variables play an important role in a descriptive analysis. A correlation measures the relationship between two variables, that is, how they are linked to each other. In this sense, a correlation allows to know which variables evolve in the same direction, which ones evolve in the opposite direction, and which ones are independent.

In this article, I show how to compute correlation coefficients, how to perform correlation tests and how to visualize relationships between variables in R.

Correlation is usually computed on two quantitative variables, but it can also be computed on two qualitative ordinal variables.¹ See the Chi-square test of independence if you need to study the relationship between two qualitative nominal variables.

If you need to quantify the relationship between two variables, I refer you to the article about linear regression.

Data

In this article, we use the mtcars dataset (loaded by default in R):

# display first 5 observations
head(mtcars, 5)

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2

The variables vs and am are categorical variables, so they are removed for this article:

# remove vs and am variables
library(tidyverse)
dat <- mtcars %>%
  select(-vs, -am)

# display 5 first obs. of new dataset
head(dat, 5)

##                    mpg cyl disp  hp drat    wt  qsec gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02    3    2

Correlation coefficient

Between two variables

The correlation between 2 variables is found with the cor() function.

Suppose we want to compute the correlation between horsepower (hp) and miles per gallon (mpg):

# Pearson correlation between 2 variables
cor(dat$hp, dat$mpg)

## [1] -0.7761684

Note that the correlation between variables X and Y is equal to the correlation between variables Y and X so the order of the variables in the cor() function does not matter.

The Pearson correlation is computed by default with the cor() function. If you want to compute the Spearman correlation, add the argument method = "spearman" to the cor() function:

# Spearman correlation between 2 variables
cor(dat$hp, dat$mpg,
  method = "spearman"
)

## [1] -0.8946646

The most common correlation methods (Run ?cor for more information about the different methods available in the cor() function) are:

Pearson correlation is often used for quantitative continuous variables that have a linear relationship
Spearman correlation (which is actually similar to Pearson but based on the ranked values for each variable rather than on the raw data) is often used to evaluate relationships involving at least one qualitative ordinal variable or two quantitative variables if the link is partially linear
Kendall’s tau-b which is computed from the number of concordant and discordant pairs is often used for qualitative ordinal variables

Note that there exists the point-biserial correlation (which can be used to measure the association between a continuous variable and a nominal variable of two levels), but this correlation is not covered here.

Correlation matrix: correlations for all variables

Suppose now that we want to compute correlations for several pairs of variables. We can easily do so for all possible pairs of variables in the dataset, again with the cor() function:

# correlation for all variables
round(cor(dat),
  digits = 2 # rounded to 2 decimals
)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg   1.00 -0.85 -0.85 -0.78  0.68 -0.87  0.42  0.48 -0.55
## cyl  -0.85  1.00  0.90  0.83 -0.70  0.78 -0.59 -0.49  0.53
## disp -0.85  0.90  1.00  0.79 -0.71  0.89 -0.43 -0.56  0.39
## hp   -0.78  0.83  0.79  1.00 -0.45  0.66 -0.71 -0.13  0.75
## drat  0.68 -0.70 -0.71 -0.45  1.00 -0.71  0.09  0.70 -0.09
## wt   -0.87  0.78  0.89  0.66 -0.71  1.00 -0.17 -0.58  0.43
## qsec  0.42 -0.59 -0.43 -0.71  0.09 -0.17  1.00 -0.21 -0.66
## gear  0.48 -0.49 -0.56 -0.13  0.70 -0.58 -0.21  1.00  0.27
## carb -0.55  0.53  0.39  0.75 -0.09  0.43 -0.66  0.27  1.00

This correlation matrix gives an overview of the correlations for all combinations of two variables.

Interpretation of a correlation coefficient

First of all, correlation ranges from -1 to 1. It gives us an indication on two things:

The direction of the relationship between the 2 variables
The strength of the relationship between the 2 variables

Regarding the direction of the relationship: On the one hand, a negative correlation implies that the two variables under consideration vary in opposite directions, that is, if a variable increases the other decreases and vice versa. On the other hand, a positive correlation implies that the two variables under consideration vary in the same direction, i.e., if a variable increases the other one increases and if one decreases the other one decreases as well.

Regarding the strength of the relationship: The more extreme the correlation coefficient (the closer to -1 or 1), the stronger the relationship. This also means that a correlation close to 0 indicates that the two variables are independent, that is, as one variable increases, there is no tendency in the other variable to either decrease or increase.

As an illustration, the Pearson correlation between horsepower (hp) and miles per gallon (mpg) found above is -0.78, meaning that the 2 variables vary in opposite direction. This makes sense, cars with more horsepower tend to consume more fuel (and thus have a lower mileage per gallon). On the contrary, from the correlation matrix we see that the correlation between miles per gallon (mpg) and the time to drive 1/4 of a mile (qsec) is 0.42, meaning that fast cars (low qsec) tend to have a worse mileage per gallon (low mpg). This again makes sense as fast cars tend to consume more fuel.

Note that it is a good practice to visualize the type of the relationship between the two variables before interpreting the correlation coefficients. The reason is that the correlation coefficient could be biased due to an outlier or due to the type of link between the two variables.

For instance, see the two Pearson correlation coefficients (denoted by R in the following plots) when the outlier is excluded and included:

The Pearson correlation coefficient changes drastically due to a single point, and thus the interpretation. It goes from a negative correlation coefficient, indicating a negative relationship between the 2 variables, to a positive coefficient, indicating a positive relationship. We would have missed this insight if we had not visualized the data in a scatterplot (see how to draw a scatterplot in this section).

A correlation coefficient may also miss a non-linear link between two variables:

The Pearson correlation coefficient is equal to 0, indicating no relationship between the two variables, because it measures the linear relationship and it is clear from the plot that the link is non-linear.

So to recap, it is a good practice to visualize the data via a scatterplot before interpreting a correlation coefficient (it does not tell the whole story) and see how the correlation coefficient changes when using the parametric (Pearson) or nonparametric version (Spearman or Kendall’s tau-b).

Visualizations

The correlation matrix presented above is not easily interpretable, especially when the dataset is composed of many variables. In the following sections, we present some alternatives to the correlation matrix for better readability.

A scatterplot for 2 variables

A good way to visualize a correlation between 2 variables is to draw a scatterplot of the two variables of interest. Suppose we want to examine the relationship between horsepower (hp) and miles per gallon (mpg):

# scatterplot
library(ggplot2)

ggplot(dat) +
  aes(x = hp, y = mpg) +
  geom_point(colour = "#0c4c8a") +
  theme_minimal()

If you are unfamiliar with the {ggplot2} package, you can draw the scatterplot using the plot() function from R base graphics:

plot(dat$hp, dat$mpg)

or use the esquisse addin to easily draw plots using the {ggplot2} package.

Scatterplots for several pairs of variables

Suppose that instead of visualizing the relationship between only 2 variables, we want to visualize the relationship for several pairs of variables. This is possible thanks to the pair() function.

For this illustration, we focus only on miles per gallon (mpg), horsepower (hp) and weight (wt):

# multiple scatterplots
pairs(dat[, c("mpg", "hp", "wt")])

The figure indicates that weight (wt) and horsepower (hp) are positively correlated, whereas miles per gallon (mpg) seems to be negatively correlated with horsepower (hp) and weight (wt).

Another simple correlation matrix

This version of the correlation matrix presents the correlation coefficients in a slightly more readable way, i.e., by coloring the coefficients based on their sign. Applied to our dataset, we have:

# improved correlation matrix
library(corrplot)

corrplot(cor(dat),
  method = "number",
  type = "upper" # show only upper side
)

Correlation test

For 2 variables

Unlike a correlation matrix which indicates the correlation coefficients between some pairs of variables in the sample, a correlation test is used to test whether the correlation (denoted $\rho$) between 2 variables is significantly different from 0 or not in the population.

Actually, a correlation coefficient different from 0 in the sample does not mean that the correlation is significantly different from 0 in the population. This needs to be tested with a hypothesis test—and known as the correlation test.

The null and alternative hypothesis for the correlation test are as follows:

$H_0$: $\rho = 0$ (meaning that there is no linear relationship between the two variables)
$H_1$: $\rho \ne 0$ (meaning that there is a linear relationship between the two variables)

Via this correlation test, what we are actually testing is whether:

the sample contains sufficient evidence to reject the null hypothesis and conclude that the correlation coefficient does not equal 0, so the relationship exists in the population.
or on the contrary, the sample does not contain enough evidence that the correlation coefficient does not equal 0, so in this case we do not reject the null hypothesis of no relationship between the variables in the population.

Note that there are 2 assumptions for this test to be valid:

Independence of the data
For small sample sizes (usually $n < 30$), the two variables should follow a normal distribution

Suppose that we want to test whether the rear axle ratio (drat) is correlated with the time to drive a quarter of a mile (qsec):

# Pearson correlation test
test <- cor.test(dat$drat, dat$qsec)
test

## 
## 	Pearson's product-moment correlation
## 
## data:  dat$drat and dat$qsec
## t = 0.50164, df = 30, p-value = 0.6196
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.265947  0.426340
## sample estimates:
##        cor 
## 0.09120476

The p-value of the correlation test between these 2 variables is 0.62. At the 5% significance level, we do not reject the null hypothesis of no correlation. We therefore conclude that we do not reject the hypothesis that there is no linear relationship between the 2 variables.²

This test proves that even if the correlation coefficient is different from 0 (the correlation is 0.09 in the sample), it is actually not significantly different from 0 in the population.

Note that the p-value of a correlation test is based on the correlation coefficient and the sample size. The larger the sample size and the more extreme the correlation (closer to -1 or 1), the more likely the null hypothesis of no correlation will be rejected.

With a small sample size, it is thus possible to obtain a relatively large correlation in the sample (based on the correlation coefficient), but still find a correlation not significantly different from 0 in the population (based on the correlation test). For this reason, it is recommended to always perform a correlation test before interpreting a correlation coefficient to avoid flawed conclusions.

For several pairs of variables

Similar to the correlation matrix used to compute correlation for several pairs of variables, the rcorr() function (from the {Hmisc} package) allows to compute p-values of the correlation test for several pairs of variables at once. Applied to our dataset, we have:

# correlation tests for whole dataset
library(Hmisc)
res <- rcorr(as.matrix(dat)) # rcorr() accepts matrices only

# display p-values (rounded to 3 decimals)
round(res$P, 3)

##        mpg   cyl  disp    hp  drat    wt  qsec  gear  carb
## mpg     NA 0.000 0.000 0.000 0.000 0.000 0.017 0.005 0.001
## cyl  0.000    NA 0.000 0.000 0.000 0.000 0.000 0.004 0.002
## disp 0.000 0.000    NA 0.000 0.000 0.000 0.013 0.001 0.025
## hp   0.000 0.000 0.000    NA 0.010 0.000 0.000 0.493 0.000
## drat 0.000 0.000 0.000 0.010    NA 0.000 0.620 0.000 0.621
## wt   0.000 0.000 0.000 0.000 0.000    NA 0.339 0.000 0.015
## qsec 0.017 0.000 0.013 0.000 0.620 0.339    NA 0.243 0.000
## gear 0.005 0.004 0.001 0.493 0.000 0.000 0.243    NA 0.129
## carb 0.001 0.002 0.025 0.000 0.621 0.015 0.000 0.129    NA

Only correlations with p-values smaller than the significance level (usually $\alpha = 0.05$) should be interpreted.

Combination of correlation coefficients and correlation tests

Now that we covered the concepts of correlation coefficients and correlation tests, let see if we can combine the two concepts.

If you need to do this for a few pairs of variables, I recommend using the ggscatterstats() function from the {ggstatsplot} package. Let’s see it in practice with one pair of variables—wt and mpg:

## plot with statistical results
library(ggstatsplot)

ggscatterstats(
  data = dat,
  x = wt,
  y = mpg,
  bf.message = FALSE,
  marginal = FALSE # remove histograms
)

Based on the result of the test, we conclude that there is a negative correlation between the weight and the number of miles per gallon ($r = - 0.87$, $p$-value < 0.001).

If you need to do it for many pairs of variables, I recommend using the the correlation function from the easystats {correlation} package.

This function allows to combine correlation coefficients and correlation tests for several pairs of variables, all in a single table (thanks to krzysiektr for pointing it out to me):

library(correlation)

correlation::correlation(dat,
  include_factors = TRUE, method = "auto"
)

## # Correlation Matrix (auto-method)
## 
## Parameter1 | Parameter2 |     r |         95% CI | t(30) |         p
## --------------------------------------------------------------------
## mpg        |        cyl | -0.85 | [-0.93, -0.72] | -8.92 | < .001***
## mpg        |       disp | -0.85 | [-0.92, -0.71] | -8.75 | < .001***
## mpg        |         hp | -0.78 | [-0.89, -0.59] | -6.74 | < .001***
## mpg        |       drat |  0.68 | [ 0.44,  0.83] |  5.10 | < .001***
## mpg        |         wt | -0.87 | [-0.93, -0.74] | -9.56 | < .001***
## mpg        |       qsec |  0.42 | [ 0.08,  0.67] |  2.53 | 0.137    
## mpg        |       gear |  0.48 | [ 0.16,  0.71] |  3.00 | 0.065    
## mpg        |       carb | -0.55 | [-0.75, -0.25] | -3.62 | 0.016*   
## cyl        |       disp |  0.90 | [ 0.81,  0.95] | 11.45 | < .001***
## cyl        |         hp |  0.83 | [ 0.68,  0.92] |  8.23 | < .001***
## cyl        |       drat | -0.70 | [-0.84, -0.46] | -5.37 | < .001***
## cyl        |         wt |  0.78 | [ 0.60,  0.89] |  6.88 | < .001***
## cyl        |       qsec | -0.59 | [-0.78, -0.31] | -4.02 | 0.007**  
## cyl        |       gear | -0.49 | [-0.72, -0.17] | -3.10 | 0.054    
## cyl        |       carb |  0.53 | [ 0.22,  0.74] |  3.40 | 0.027*   
## disp       |         hp |  0.79 | [ 0.61,  0.89] |  7.08 | < .001***
## disp       |       drat | -0.71 | [-0.85, -0.48] | -5.53 | < .001***
## disp       |         wt |  0.89 | [ 0.78,  0.94] | 10.58 | < .001***
## disp       |       qsec | -0.43 | [-0.68, -0.10] | -2.64 | 0.131    
## disp       |       gear | -0.56 | [-0.76, -0.26] | -3.66 | 0.015*   
## disp       |       carb |  0.39 | [ 0.05,  0.65] |  2.35 | 0.177    
## hp         |       drat | -0.45 | [-0.69, -0.12] | -2.75 | 0.110    
## hp         |         wt |  0.66 | [ 0.40,  0.82] |  4.80 | < .001***
## hp         |       qsec | -0.71 | [-0.85, -0.48] | -5.49 | < .001***
## hp         |       gear | -0.13 | [-0.45,  0.23] | -0.69 | > .999   
## hp         |       carb |  0.75 | [ 0.54,  0.87] |  6.21 | < .001***
## drat       |         wt | -0.71 | [-0.85, -0.48] | -5.56 | < .001***
## drat       |       qsec |  0.09 | [-0.27,  0.43] |  0.50 | > .999   
## drat       |       gear |  0.70 | [ 0.46,  0.84] |  5.36 | < .001***
## drat       |       carb | -0.09 | [-0.43,  0.27] | -0.50 | > .999   
## wt         |       qsec | -0.17 | [-0.49,  0.19] | -0.97 | > .999   
## wt         |       gear | -0.58 | [-0.77, -0.29] | -3.93 | 0.008**  
## wt         |       carb |  0.43 | [ 0.09,  0.68] |  2.59 | 0.132    
## qsec       |       gear | -0.21 | [-0.52,  0.15] | -1.19 | > .999   
## qsec       |       carb | -0.66 | [-0.82, -0.40] | -4.76 | < .001***
## gear       |       carb |  0.27 | [-0.08,  0.57] |  1.56 | 0.774    
## 
## p-value adjustment method: Holm (1979)
## Observations: 32

As you can see, it gives, among other useful information, the correlation coefficients (column r) and the result of the correlation test (column 95% CI for the confidence interval or p for the $p$-value) for all pairs of variables.

Correlograms

The table above is very useful and informative, but let see if it is possible to combine the concepts of correlation coefficients and correlations test in one single visualization. A visualization that would be easy to read and interpret.

Ideally, we would like to have a concise overview of correlations between all possible pairs of variables present in a dataset, with a clear distinction for correlations that are significantly different from 0.

The figure below, known as a correlogram and adapted from the corrplot() function, does precisely this:

# do not edit
corrplot2 <- function(data,
                      method = "pearson",
                      sig.level = 0.05,
                      order = "original",
                      diag = FALSE,
                      type = "upper",
                      tl.srt = 90,
                      number.font = 1,
                      number.cex = 1,
                      mar = c(0, 0, 0, 0)) {
  library(corrplot)
  data_incomplete <- data
  data <- data[complete.cases(data), ]
  mat <- cor(data, method = method)
  cor.mtest <- function(mat, method) {
    mat <- as.matrix(mat)
    n <- ncol(mat)
    p.mat <- matrix(NA, n, n)
    diag(p.mat) <- 0
    for (i in 1:(n - 1)) {
      for (j in (i + 1):n) {
        tmp <- cor.test(mat[, i], mat[, j], method = method)
        p.mat[i, j] <- p.mat[j, i] <- tmp$p.value
      }
    }
    colnames(p.mat) <- rownames(p.mat) <- colnames(mat)
    p.mat
  }
  p.mat <- cor.mtest(data, method = method)
  col <- colorRampPalette(c("#BB4444", "#EE9988", "#FFFFFF", "#77AADD", "#4477AA"))
  corrplot(mat,
    method = "color", col = col(200), number.font = number.font,
    mar = mar, number.cex = number.cex,
    type = type, order = order,
    addCoef.col = "black", # add correlation coefficient
    tl.col = "black", tl.srt = tl.srt, # rotation of text labels
    # combine with significance level
    p.mat = p.mat, sig.level = sig.level, insig = "blank",
    # hide correlation coefficients on the diagonal
    diag = diag
  )
}

# edit from here
corrplot2(
  data = dat,
  method = "pearson",
  sig.level = 0.05,
  order = "original",
  diag = FALSE,
  type = "upper",
  tl.srt = 75
)

The correlogram shows correlation coefficients for all pairs of variables (with more intense colors for more extreme correlations), and correlations not significantly different from 0 are represented by a white box.

To learn more about this plot and the code used, I invite you to read the article entitled “Correlogram in R: how to highlight the most correlated variables in a dataset”.

For those of you who are still not completely satisfied, I recently found two alternatives—one with the ggpairs() function from the {GGally} package and one with the ggcormat() function from the {ggstatsplot} package.

The two functions are illustrated with the variables mpg, hp and wt:

library(GGally)

ggpairs(dat[, c("mpg", "hp", "wt")])

The plot above combines correlation coefficients, correlation tests (via the asterisks next to the coefficients³) and scatterplots for all possible pairs of variables present in a dataset.

library(ggstatsplot)

ggcorrmat(
  data = dat[, c("mpg", "hp", "wt")],
  type = "parametric", # parametric for Pearson, nonparametric for Spearman's correlation
  colors = c("darkred", "white", "steelblue") # change default colors
)

The plot above also shows the correlation coefficients and if any, the non-significant correlations (by default at the 5% significance level with the Holm adjustment method) are shown by a big cross on the correlation coefficients.

The advantage of these two alternatives compared to the first one is that it is directly available within a package, so you do not need to run the code of the function first in order to draw the correlogram.

Correlation does not imply causation

I am pretty sure you have already heard the statement “Correlation does not imply causation” in statistics. An article about correlation would not be complete without discussing about causation.

A non-zero correlation between two variables does not necessarily mean that there is a cause and effect relationship between these two variables!

Indeed, a significant correlation between two variables means that changes in one variable are associated (positively or negatively) with changes in the other variable. Nonetheless, a significant correlation does not indicate that variations in one variable cause the variations in the other variable.

A non-zero correlation between X and Y can appear in several cases:

X causes Y
Y causes X
a third variable causes X and Y
a combination of these three reasons

Sometimes it is quite clear that there is a causal relationship between two variables. Take for example the correlation between the price of a consumer product such as milk and its consumption. It is quite obvious that there is a causal link between the two: if the price of milk increases, it is expected that its consumption will decrease.

However, this causal link is not always present even if the correlation is significant. Maurage et al. (2013) showed that, although there is a positive and significant correlation between chocolate consumption and the number of Nobel laureates, this correlation comes from the fact that a third variable, Gross Domestic Product (GDP), causes chocolate consumption and the number of Nobel laureates. They found that countries with higher GDP tend to have a higher level of chocolate consumption and scientific research (leading to more Nobel laureates).

This example shows that one must be very cautious when interpreting correlations and avoid over-interpreting a correlation as a causal relationship.

Conclusion

Thanks for reading.

I hope this article helped you to compute correlation coefficients and perform correlation tests in R. If you would like to learn how to compute the coefficients by hand, see this step-by-step tutorial.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

References

Maurage, Pierre, Alexandre Heeren, and Mauro Pesenti. 2013. “Does Chocolate Consumption Really Boost Nobel Award Chances? The Peril of over-Interpreting Correlations in Health Studies.” The Journal of Nutrition 143 (6): 931–33.

It is true that there is the point-biserial correlation which can be used with a nominal variable (consisting of two factors). Nonetheless, this type of correlation is much less known and usually not covered in introductory statistics classes; with one continuous and one nominal variable, it is much more frequent to learn about the Student’s t-test (for a nominal variable with 2 groups) or ANOVA (for a nominal variable with 3 or more groups). More information about choosing the most appropriate measure of association depending on the type of variable can be found in this article.↩︎
It is important to remember that we tested for a linear relationship between the two variables since we used the Pearson’s correlation. It may be the case that there is a relationship between the two variables in the population, but this relation may not be linear.↩︎
One asterisk means that the coefficient is significant at the 5% level, 2 is at the 1% significance level, and 3 is at the 0.1% significance level. This is usually the case in R; the more asterisks, the more it is significant.↩︎

One-proportion and chi-square goodness of fit test

Wed, 13 May 2020 00:00:00 +0000

Introduction

In a previous article, I presented the Chi-square test of independence in R which is used to test the independence between two categorical variables.

In this article, I show how to perform, first in R and then by hand, the:

one-proportion test (also referred as one-sample proportion test)
Chi-square goodness of fit test

The first test is used to compare an observed proportion to an expected proportion, when the qualitative variable has only two categories. The second test is used to compare multiple observed proportions to multiple expected proportions, in a situation where the qualitative variable has two or more categories.

Both tests allow to test the equality of proportions between the levels of the qualitative variable or to test the equality with given proportions. These given proportions could be determined arbitrarily or based on the theoretical probabilities of a known distribution.

In R

Data

For this section, we use the same dataset than in the article on descriptive statistics. It is the well-known iris dataset, to which we add the variable size. The variable size corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

# load iris dataset
dat <- iris

# create size variable
dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

# show first 5 observations
head(dat, n = 5)

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species  size
## 1          5.1         3.5          1.4         0.2  setosa small
## 2          4.9         3.0          1.4         0.2  setosa small
## 3          4.7         3.2          1.3         0.2  setosa small
## 4          4.6         3.1          1.5         0.2  setosa small
## 5          5.0         3.6          1.4         0.2  setosa small

One-proportion test

For this example, we have a sample of 150 flowers and we want to test whether the proportion of small flowers is different than the proportion of big flowers (measured by the variable size). Here are the number of flowers by size, and the corresponding proportions:

# barplot
library(ggplot2)
ggplot(dat) +
  aes(x = size) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by size
table(dat$size)

## 
##   big small 
##    77    73

# proportions by size, rounded to 2 decimals
round(prop.table(table(dat$size)), 2)

## 
##   big small 
##  0.51  0.49

Among the 150 flowers forming our sample, 51% and 49% are big and small, respectively. To test whether the proportions are different among both sizes, we use the prop.test() function which accepts the following arguments:

number of successes
number of observations/trials
expected probability (the one we want to test against)

The hypotheses in our example are:

$H_0$: proportions of big and small flowers are equal
$H_1$: proportions of big and small flowers are different

Considering (arbitrarily) that big is the success, we have:¹

# one-proportion test
test <- prop.test(
  x = 77, # number of successes
  n = 150, # total number of trials (77 + 73)
  p = 0.5 # we test for equal proportion so prob = 0.5 in each group
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  77 out of 150, null probability 0.5
## X-squared = 0.06, df = 1, p-value = 0.8065
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.4307558 0.5952176
## sample estimates:
##         p 
## 0.5133333

We obtain an output with:

the null probability (0.5),
the test statistic (X-squared = 0.06),
the degrees of freedom (df = 1),
the p-value (p-value = 0.8065),
the alternative hypothesis (true p is not equal to 0.5),
the 95% confidence interval (which can also be extracted with test$conf.int) and
the proportion in the sample (0.5133333).

The p-value is 0.806 so, at the 5% significance level, we do not reject the null hypothesis that the proportions of small and big flowers are the same.

An alternative is the ggpiestats() function from the {ggstatsplot} package:²

## plot with statistical results
library(ggstatsplot)
ggpiestats(
  data = dat,
  x = size,
  bf.message = FALSE
)

Note that the $p$-value (the value after p = in the subtitle of the plot) is slightly different because Yates’ continuity correction is not applied in ggpiestats() while it is applied by default in prop.test().

The conclusion remains however the same, that is, we do not reject the null hypothesis that proportions of big and small flowers are equal.

Assumption of `prop.test()` and `binom.test()`

Note that prop.test() uses a normal approximation to the binomial distribution. Therefore, one assumption of this test is that the sample size is large enough (usually, n > 30). If the sample size is small, it is recommended to use the exact binomial test.

The exact binomial test can be performed with the binom.test() function and accepts the same arguments as the prop.test() function.

For this example, suppose now that we have a sample of 12 big and 3 small flowers and we want to test whether the proportions are the same among both sizes:

# barplot
barplot(c(12, 3), # observed counts
  names.arg = c("big", "small"), # rename labels
  ylab = "Frequency", # y-axis label
  xlab = "Size" # x-axis label
)
abline(
  h = 15 / 2, # expected counts in each level
  lty = 2 # dashed line
)

# exact binomial test
test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5 # expected proportion
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.03516
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.5191089 0.9566880
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.035 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportions of small and big flowers are significantly different. This is equivalent than concluding that the proportion of big flowers is significantly different from 0.5 (since there are only two sizes).

If you want to test that the proportion of big flowers is greater than 50%, add the alternative = "greater" argument into the binom.test() function:³

test <- binom.test(
  x = 12, # counts of successes
  n = 15, # total counts (12 + 3)
  p = 0.5, # expected proportion
  alternative = "greater" # test that prop of big flowers is > 0.5
)

test

## 
##  Exact binomial test
## 
## data:  12 and 15
## number of successes = 12, number of trials = 15, p-value = 0.01758
## alternative hypothesis: true probability of success is greater than 0.5
## 95 percent confidence interval:
##  0.5602156 1.0000000
## sample estimates:
## probability of success 
##                    0.8

The p-value is 0.018 so, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of big flowers is significantly larger than 50%.

Chi-square goodness of fit test

Suppose now that the qualitative variable has more than two levels as it is the case for the variable Species:

# barplot
ggplot(dat) +
  aes(x = Species) +
  geom_bar(fill = "#0c4c8a") +
  theme_minimal()

# counts by Species
table(dat$Species)

## 
##     setosa versicolor  virginica 
##         50         50         50

The variable Species has 3 levels, with 50 observations in each level.

Suppose for this example that we want to test whether the 3 species are equally common. If they were equally common, they would be equally distributed and the expected proportions would be $\frac{1}{3}$ for each of the species.

The hypotheses are now:

$H_0$: proportions of each species are equal
$H_1$: there is at least one species with a different proportion⁴

This test can be done with the chisq.test() function, accepting the following arguments:

a numeric vector representing the observed proportions
a vector of probabilities (of the same length of the observed proportions) representing the expected proportions

Applied to our research question (i.e., are the 3 species equally common?), we have:

# chi-square goodness of fit test
test <- chisq.test(table(dat$Species), # observed proportions
  p = c(1 / 3, 1 / 3, 1 / 3) # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  table(dat$Species)
## X-squared = 0, df = 2, p-value = 1

The p-value is 1 so, at the 5% significance level, we do not reject the null hypothesis that the proportions are equal among all species.

This was quite obvious even before doing the statistical test given that there are exactly 50 flowers of each species, so it was easy to see that the species are equally common. We however still did the test to show how it works in practice.

Note that the alternative proposed by the {ggstatsplot} package can also be used for a Chi-square goodness of fit test:

## plot with statistical results
ggpiestats(
  data = dat,
  x = Species,
  bf.message = FALSE
)

Assumptions

One of the assumptions of the chi-square goodness of fit test is that the sample size is large enough in order for the chi-square approximation to be valid.

To be more precise, there must be at least 5 expected frequencies in each group of your categorical variable. This can be verified as follows:

chisq.test(table(dat$Species))$expected

##     setosa versicolor  virginica 
##         50         50         50

The assumption of sufficiently large sample size is met as all expected frequencies are above 5.

Does my distribution follow a given distribution?

In the previous section, we chose the proportions ourselves. The goodness of fit test is also particularly useful to compare observed proportions with expected proportions that are based on some known distribution.

Remember the hypotheses of the test:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

For this example, suppose that we measured the number of girls in 100 families of 5 children. We want to test whether the (observed) distribution of number girls follows a binomial distribution.

Observed frequencies

Here is the distribution of the number of girls per family in our sample of 100 families of 5 children:

And the corresponding frequencies and relative frequencies (remember that the relative frequency is the frequency divided by the total sample size):

# counts
dat

##   Girls Frequency Relative_freq
## 1     0         5          0.05
## 2     1        12          0.12
## 3     2        28          0.28
## 4     3        33          0.33
## 5     4        17          0.17
## 6     5         5          0.05

Expected frequencies

In order to compare the observed frequencies to a binomial distribution and see if both distributions match, we first need to determine the expected frequencies that would be obtained in case of a binomial distribution.

The expected frequencies assuming a probability of 0.5 of having a girl (for each of the 5 children) are as follows:

# create expected frequencies for a binomial distribution
x <- 0:5
df <- data.frame(
  Girls = factor(x),
  Expected_relative_freq = dbinom(x, size = 5, prob = 0.5)
)
df$Expected_freq <- df$Expected_relative_freq * 100 # *100 since there are 100 families

# create barplot
p <- ggplot(df, aes(x = Girls, y = Expected_freq)) +
  geom_bar(stat = "identity", fill = "#F8766D") +
  xlab("Number of girls per family") +
  ylab("Expected frequency") +
  labs(title = "Binomial distribution Bi(x, n = 5, p = 0.5)") +
  theme_minimal()
p

# expected relative frequencies and (absolute) frequencies
df

##   Girls Expected_relative_freq Expected_freq
## 1     0                0.03125         3.125
## 2     1                0.15625        15.625
## 3     2                0.31250        31.250
## 4     3                0.31250        31.250
## 5     4                0.15625        15.625
## 6     5                0.03125         3.125

Observed vs. expected frequencies

We now compare the observed frequencies to the expected frequencies to see whether the two differ significantly. If the two differ significantly, we reject the hypothesis that the number of girls per family of 5 children follows a binomial distribution. On the other hand, if the observed and expected frequencies are similar, we do not reject the hypothesis that the number of girls per family follows a binomial distribution.

Visually we have:

# create data
data <- data.frame(
  num_girls = factor(rep(c(0:5), times = 2)),
  Freq = c(dat$Freq, df$Expected_freq),
  obs_exp = c(rep("observed", 6), rep("expected", 6))
)

# create plot
ggplot() +
  geom_bar(
    data = data, aes(
      x = num_girls, y = Freq,
      fill = obs_exp
    ),
    position = "dodge", # bar next to each other
    stat = "identity"
  ) +
  ylab("Frequency") +
  xlab("Number of girls per family") +
  theme_minimal() +
  theme(legend.title = element_blank()) # remove legend title

We see that the observed and expected frequencies are quite similar, so we expect that the number of girls in families of 5 children follows a binomial distribution. However, only the goodness of fit test will confirm our belief:

# chi-square goodness of fit test
test <- chisq.test(dat$Freq, # observed frequencies
  p = df$Expected_relative_freq # expected proportions
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$Freq
## X-squared = 3.648, df = 5, p-value = 0.6011

The p-value is 0.601 so, at the 5% significance level, we do not reject the null hypothesis that the observed and expected frequencies are equal. This is equivalent than concluding that we cannot reject the hypothesis that the number of girls in families of 5 children follows a binomial distribution (since the expected frequencies were based on a binomial distribution).

Note that the chi-square goodness of fit test can of course be performed with other types of distribution than the binomial one. For instance, if you want to test whether an observed distribution follows a Poisson distribution, this test can be used to compare the observed frequencies with the expected proportions that would be obtained in case of a Poisson distribution.

By hand

Now that we showed how to perform the one-proportion and chi-square goodness of fit test in R, in this section we show how to do these tests by hand. We first illustrate the one-proportion test then the chi-square goodness of fit test.

One-proportion test

For this example, suppose that we tossed a coin 100 times and noted that it landed on heads 67 times. Following this, we want to test whether the coin is fair, that is, test whether the probability of landing on heads or tails is equal to 50%.

As for many hypothesis tests, we do it through 4 easy steps:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

In our example, the null and alternative hypotheses are:

$H_0$: $p_0 = 0.5$
$H_1$: $p_0 \ne 0.5$

where $p_0$ is the expected proportion of landing on heads.

Step 2.

The test statistic is:⁵

\[z_{obs} = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1 - p_0)}{n}}} = \frac{0.67 - 0.5}{\sqrt{\frac{0.5 \cdot (1 - 0.5)}{100}}} = 3.4\]

(See how to perform hypothesis tests in a Shiny app if you need more help in computing the test statistic.)

Step 3.

The rejection region is found via the normal distribution table. Assuming a significance level $\alpha = 0.05$, we have:

\[\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic lies within the rejection region (i.e., the grey shaded areas). Therefore, at the 5% significance level, we reject the null hypothesis and we conclude that the proportion of heads (and thus tails) is significantly different than 50%. In other words, still at the 5% significance level, we conclude that the coin is unfair.

If you prefer to compute the p-value instead of comparing the t-stat and the rejection region, you can use this Shiny app to easily compute p-values for different probability distributions. After having opened the app, set the t-stat, the corresponding alternative and you will find the p-value at the top of the page.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# one-proportion test
test <- prop.test(
  x = 67, # number of heads
  n = 100, # number of trials
  p = 0.5 # expected probability of heads
)

test

## 
##  1-sample proportions test with continuity correction
## 
## data:  67 out of 100, null probability 0.5
## X-squared = 10.89, df = 1, p-value = 0.0009668
## alternative hypothesis: true p is not equal to 0.5
## 95 percent confidence interval:
##  0.5679099 0.7588442
## sample estimates:
##    p 
## 0.67

The p-value is 0.001 so, at the 5% significance level, we reject the null hypothesis that the proportions of heads and tails are equal, and we conclude that the coin is biased. This is the same conclusion than the one found by hand.

Goodness of fit test

We now illustrate the chi-square goodness of fit test by hand with the following example.

Suppose that we toss a dice 100 times, we note how many times it lands on each face (1 to 6) and we test whether the dice is fair. Here are the observed counts by dice face:

## dice_face
##  1  2  3  4  5  6 
## 15 24 10 19 19 13

With a fair dice, we would expect it to land $\frac{100}{6} \approx 16.67$ times on each face (this expected value is represented by the dashed line in the above plot). Although the observed frequencies are different than the expected value of 16.67:

##   dice_face observed_freq expected_freq
## 1         1            15         16.67
## 2         2            24         16.67
## 3         3            10         16.67
## 4         4            19         16.67
## 5         5            19         16.67
## 6         6            13         16.67

we need to test whether they are significantly different. For this, we perform the appropriate hypothesis test following the 4 easy steps mentioned above:

State the null and alternative hypotheses
Compute the test-statistic (also known as t-stat)
Find the rejection region
Conclude by comparing the test-statistic with the rejection region

Step 1.

The null and alternative hypotheses of the chi-square goodness of fit test are:

$H_0$: there is no significant difference between the observed and the expected frequencies
$H_1$: there is a significant difference between the observed and the expected frequencies

Applied to our example, we have:

$H_0$: all faces occur in the same proportion
$H_1$: at least one proportion is not equal to 1/6

Step 2.

The test statistic is:

\[\chi^2 = \sum_{i = 1}^k \frac{(O_i - E_i)^2}{E_i}\]

where $O_i$ is the observed frequency, $E_i$ is the expected frequency and $k$ is the number of categories (in our case, there are 6 categories, representing the 6 dice faces).

This $\chi^2$ statistic is obtained by calculating the difference between the observed number of cases and the expected number of cases in each category. This difference is squared (to avoid negative and positive differences being compensated) and divided by the expected number of cases in that category. These values are then summed for all categories, and the total is referred to as the $\chi^2$ statistic. Large values of this test statistic lead to the rejection of the null hypothesis, small values mean that the null hypothesis cannot be rejected.⁶

Given our data, we have:

\[\chi^2 = \frac{(15 - 16.67)^2}{16.67} + \frac{(24 - 16.67)^2}{16.67} + \\ \frac{(10 - 16.67)^2}{16.67} +\frac{(19 - 16.67)^2}{16.67} + \\ \frac{(19 - 16.67)^2}{16.67} + \frac{(13 - 16.67)^2}{16.67} = 7.52\]

Step 3.

Whether the $\chi^2$ test statistic is small or large depends on the rejection region. The rejection region is found via the $\chi^2$ distribution table. With a degrees of freedom equals to $k - 1$ (where $k$ is the number of categories) and assuming a significance level $\alpha = 0.05$, we have:

\[\chi^2_{\alpha; k-1} = \chi^2_{0.05; 5} = 11.0705\]

Step 4.

We compare the test statistic (found in step 2) with the rejection region (found in step 3) and we conclude. Visually, we have:

The test statistic does not lie within the rejection region (i.e., the grey shaded area). Therefore, at the 5% significance level, we do not reject the null hypothesis that there is no significant difference between the observed and the expected frequencies. In other words, still at the 5% significance level, we cannot reject the hypothesis that the dice is fair.

Again, you can use the Shiny app to easily compute the p-value given the test statistic if you prefer this method over the comparison between the t-stat and the rejection region.

Verification in R

Just for the sake of illustration, here is the verification of the above example in R:

# chi-square goodness of fit test
test <- chisq.test(dat$observed_freq, # observed frequencies for each dice face
  p = rep(1 / 6, 6) # expected probabilities for each dice face
)

test

## 
##  Chi-squared test for given probabilities
## 
## data:  dat$observed_freq
## X-squared = 7.52, df = 5, p-value = 0.1847

The test statistic and degrees of freedom are exactly the same than the ones found by hand. The p-value is 0.185 which, still at the 5% significance level, leads to the same conclusion than by hand (i.e., failing to reject the null hypothesis).

Conclusion

Thanks for reading.

I hope this article helped you to understand and perform the one-proportion and chi-square goodness of fit test in R and by hand. Learn more about the Chi-square test of independence in R and by hand if you want to analyze two categorical variables instead of one.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Choosing big or small as the success event gives the exact same conclusion.↩︎
Note that if possible, it is best to avoid pie charts and use bar charts instead. Unfortunately, the ggbarstats() function works only for the independence Chi-square test.↩︎
Similarly, this argument can also be added to the prop.test() function to test whether the observed proportion is larger than the expected proportion. Use alternative = "less" if you want to test whether the observed proportion is smaller than the expected one.↩︎
Be careful that the alternative hypothesis is not that all proportions are different. One different from the others is sufficient to reject the null hypothesis. It is, in some sense, similar to the alternative hypothesis of the ANOVA which says that at least one mean is different than another.↩︎
One assumption of this test is that $n \cdot p \ge 5$ and $n \cdot (1 - p) \ge 5$. The assumption is met so we can use the normal approximation to the binomial distribution.↩︎
Source: http://uregina.ca/~gingrich/ch10.pdf.↩︎

How to do a t-test or ANOVA for more than one variable at once in R?

Thu, 19 Mar 2020 00:00:00 +0000

Introduction

As part of my teaching assistant position in a Belgian university, students often ask me for some help in their statistical analyses for their master’s thesis.

A frequent question is how to compare groups of patients in terms of several quantitative continuous variables. Most of us know that:

To compare two groups, a Student’s t-test should be used¹
To compare three groups or more, an ANOVA should be performed

These two tests are quite basic and have been extensively documented online and in statistical textbooks so the difficulty is not in how to perform these tests.

In the past, I used to do the analyses by following these 3 steps:

Draw boxplots illustrating the distributions by group (with the boxplot() function or thanks to the {esquisse} R Studio addin if I wanted to use the {ggplot2} package)
Perform a t-test or an ANOVA depending on the number of groups to compare (with the t.test() and oneway.test() functions for t-test and ANOVA, respectively)
Repeat steps 1 and 2 for each variable

This was feasible as long as there were only a couple of variables to test. Nonetheless, most students came to me asking to perform these kind of tests not on one or two variables, but on multiples variables. So when there were more than one variable to test, I quickly realized that I was wasting my time and that there must be a more efficient way to do the job.

Note: you must be very careful with the issue of multiple testing (also referred as multiplicity) which can arise when you perform multiple tests. In short, when a large number of statistical tests are performed, some will have $p$-values less than 0.05 purely by chance, even if all null hypotheses are in fact really true. This is known as multiplicity or multiple testing. You can tackle this problem by using the Bonferroni correction, among others. The Bonferroni correction is a simple method that allows many t-tests to be made while still assuring an overall confidence level is maintained. For this, instead of using the standard threshold of $\alpha = 5$% for the significance level, you can use $\alpha = \frac{0.05}{m}$ where $m$ is the number of t-tests. For example, if you perform 20 t-tests with a desired $\alpha = 0.05$, the Bonferroni correction implies that you would reject the null hypothesis for each individual test when the $p$-value is smaller than $\alpha = \frac{0.05}{20} = 0.0025$.

Note also that there is no universally accepted approach for dealing with the problem of multiple comparisons. Usually, you should choose a p-value adjustment measure familiar to your audience or in your field of study. The Bonferroni correction is easy to implement. It is however not appropriate if you have a very large number of tests to perform (imagine you want to do 10,000 t-tests, a p-value would have to be less than $\frac{0.05}{10000} = 0.000005$ to be significant). A more powerful method is also to adjust the false discovery rate using the Benjamini-Hochberg or Holm procedure (McDonald 2014).

Another option is to use a multivariate ANOVA (MANOVA), if your independent variable has more than two levels. This is particularly useful when your dependent variables are correlated. Correlation between the dependent variables provides MANOVA the following advantages:

Identify patterns between several dependent variables: The independent variables can influence the relationship between dependent variables instead of influencing a single dependent variable.
Address the issue of multiple testing: with MANOVA, the error rate equals the significance level (with no p-value adjustment method needed).
Greater statistical power: When the dependent variables are correlated, MANOVA can identify effects that are too small for the ANOVA to detect.

Note that MANOVA is used if your independent variable has more than two levels. If your independent variable has only two levels, the multivariate equivalent of the t-test is Hotelling’s $T^2$.

This article aims at presenting a way to perform multiple t-tests and ANOVA from a technical point of view (how to implement it in R). Discussion on which adjustment method to use or whether there is a more appropriate model to fit the data is beyond the scope of this article (so be sure to understand the implications of using the code below for your own analyses). Make sure also to test the assumptions of the ANOVA before interpreting results.

Perform multiple tests at once

I thus wrote a piece of code that automated the process, by drawing boxplots and performing the tests on several variables at once. Below is the code I used, illustrating the process with the iris dataset. The Species variable has 3 levels, so let’s remove one, and then draw a boxplot and apply a t-test on all 4 continuous variables at once. Note that the continuous variables that we would like to test are variables 1 to 4 in the iris dataset.

dat <- iris

# remove one level to have only two groups
dat <- subset(dat, Species != "setosa")
dat$Species <- factor(dat$Species)

# boxplots and t-tests for the 4 variables at once
for (i in 1:4) { # variables to compare are variables 1 to 4
  boxplot(dat[, i] ~ dat$Species, # draw boxplots by group
    ylab = names(dat[i]), # rename y-axis with variable's name
    xlab = "Species"
  )
  print(t.test(dat[, i] ~ dat$Species)) # print results of t-test
}

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -5.6292, df = 94.025, p-value = 1.866e-07
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.8819731 -0.4220269
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    5.936                    6.588

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -3.2058, df = 97.927, p-value = 0.001819
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.33028364 -0.07771636
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    2.770                    2.974

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -12.604, df = 95.57, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -1.49549 -1.08851
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    4.260                    5.552

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -14.625, df = 89.043, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.7951002 -0.6048998
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    1.326                    2.026

As you can see, the above piece of code draws a boxplot and then prints results of the test for each continuous variable, all at once.

At some point in the past, I even wrote code to:

draw a boxplot
test for the equality of variances (thanks to the Levene’s test)
depending on whether the variances were equal or unequal, the appropriate test was applied: the Welch test if the variances were unequal and the Student’s t-test in the case the variances were equal (see more details about the different versions of the t-test for two samples)
apply steps 1 to 3 for all continuous variables at once

I had a similar code for ANOVA in case I needed to compare more than two groups.

The code was doing the job relatively well. Indeed, thanks to this code I was able to test several variables in an automated way in the sense that it compared groups for all variables at once.

The only thing I had to change from one project to another is that I needed to modify the name of the grouping variable and the numbering of the continuous variables to test (Species and 1:4 in the above code).

Concise and easily interpretable results

T-test

Although it was working quite well and applicable to different projects with only minor changes, I was still unsatisfied with another point.

Someone who is proficient in statistics and R can read and interpret the output of a t-test without any difficulty. However, as you may have noticed with your own statistical projects, most people do not know what to look for in the results and are sometimes a bit confused when they see so many graphs, code, output, results and numeric values in a document. They are quite easily overwhelmed by this mass of information and unable to extract the key message.

With my old R routine, the time I was saving by automating the process of t-tests and ANOVA was (partially) lost when I had to explain R outputs to my students so that they could interpret the results correctly. Although most of the time it simply boiled down to pointing out what to look for in the outputs (i.e., p-values), I was still losing quite a lot of time because these outputs were, in my opinion, too detailed for most real-life applications and for students in introductory classes. In other words, too much information seemed to be confusing for many people so I was still not convinced that it was the most optimal way to share statistical results to nonscientists.

Of course, they came to me for statistical advices, so they expected to have these results and I needed to give them answers to their questions and hypotheses. Nonetheless, I wanted to find a better way to communicate these results to this type of audience, with the minimum of information required to arrive at a conclusion. No more and no less than that.

After a long time spent online trying to figure out a way to present results in a more concise and readable way, I discovered the {ggpubr} package. This package allows to indicate the test used and the p-value of the test directly on a ggplot2-based graph. It also facilitates the creation of publication-ready plots for non-advanced statistical audiences.

After many refinements and modifications of the initial code (available in this article), I finally came up with a rather stable and robust process to perform t-tests and ANOVA for more than one variable at once, and more importantly, make the results concise and easily readable by anyone (statisticians or not).

A graph is worth a thousand words, so here are the exact same tests than in the previous section, but this time with my new R routine:

library(ggpubr)

# Edit from here #
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method <- "t.test" # one of "wilcox.test" or "t.test"
paired <- FALSE # if paired make sure that in the dataframe you have first all individuals at T1, then all individuals again at T2
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    if (paired == TRUE) {
      p <- ggpaired(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]), line.color = "gray", line.size = 0.4,
        palette = "npg",
        legend = "none",
        xlab = colnames(dat[j]),
        ylab = colnames(dat[i]),
        add = "jitter"
      )
    } else {
      p <- ggboxplot(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]),
        palette = "npg",
        legend = "none",
        add = "jitter"
      )
    }
    #  Add p-value
    print(p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
      method = method,
      paired = paired,
      # group.by = NULL,
      ref.group = NULL
    ))
  }
}

As you can see from the graphs above, only the most important information is presented for each variable:

a visual comparison of the groups thanks to boxplots
the name of the statistical test
the p-value of the test

Of course, experts may be interested in more advanced results. However, this simple yet complete graph, which includes the name of the test and the p-value, gives all the necessary information to answer the question: “Are the groups different?”.

In my experience, I have noticed that students and professionals (especially those from a less scientific background) understand way better these results than the ones presented in the previous section.

The only lines of code that need to be modified for your own project is the name of the grouping variable (Species in the above code), the names of the variables you want to test (Sepal.Length, Sepal.Width, etc.),² whether you want to apply a t-test (t.test) or Wilcoxon test (wilcox.test) and whether the samples are paired or not (FALSE if samples are independent, TRUE if they are paired).

Based on these graphs, it is easy, even for non-experts, to interpret the results and conclude that the versicolor and virginica species are significantly different in terms of all 4 variables (since all p-values $< \frac{0.05}{4} = 0.0125$ (remind that the Bonferroni correction is applied to avoid the issue of multiple testing, so we divide the usual $\alpha$ level by 4 because there are 4 t-tests)).

Additional p-value adjustment methods

If you would like to use another p-value adjustment method, you can use the p.adjust() function. Below are the raw p-values found above, together with p-values derived from the main adjustment methods (presented in a dataframe):

raw_pvalue <- numeric(length = length(1:4))
for (i in (1:4)) {
  raw_pvalue[i] <- t.test(dat[, i] ~ dat$Species,
    alternative = "two.sided"
  )$p.value
}

df <- data.frame(
  Variable = names(dat[, 1:4]),
  raw_pvalue = round(raw_pvalue, 3)
)

df$Bonferroni <-
  p.adjust(df$raw_pvalue,
    method = "bonferroni"
  )
df$BH <-
  p.adjust(df$raw_pvalue,
    method = "BH"
  )
df$Holm <-
  p.adjust(df$raw_pvalue,
    method = "holm"
  )
df$Hochberg <-
  p.adjust(df$raw_pvalue,
    method = "hochberg"
  )
df$Hommel <-
  p.adjust(df$raw_pvalue,
    method = "hommel"
  )
df$BY <-
  round(p.adjust(df$raw_pvalue,
    method = "BY"
  ), 3)
df

##       Variable raw_pvalue Bonferroni    BH  Holm Hochberg Hommel    BY
## 1 Sepal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 2  Sepal.Width      0.002      0.008 0.002 0.002    0.002  0.002 0.004
## 3 Petal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 4  Petal.Width      0.000      0.000 0.000 0.000    0.000  0.000 0.000

Regardless of the p-value adjustment method, the two species are different for all 4 variables. Note that the adjustment method should be chosen before looking at the results to avoid choosing the method based on the results.

Below another function that allows to perform multiple Student’s t-tests or Wilcoxon tests at once and choose the p-value adjustment method. The function also allows to specify whether samples are paired or unpaired and whether the variances are assumed to be equal or not. (The code has been adapted from Mark White’s article.)

t_table <- function(data, dvs, iv,
                    var_equal = TRUE,
                    p_adj = "none",
                    alpha = 0.05,
                    paired = FALSE,
                    wilcoxon = FALSE) {
  if (!inherits(data, "data.frame")) {
    stop("data must be a data.frame")
  }

  if (!all(c(dvs, iv) %in% names(data))) {
    stop("at least one column given in dvs and iv are not in the data")
  }

  if (!all(sapply(data[, dvs], is.numeric))) {
    stop("all dvs must be numeric")
  }

  if (length(unique(na.omit(data[[iv]]))) != 2) {
    stop("independent variable must only have two unique values")
  }

  out <- lapply(dvs, function(x) {
    if (paired == FALSE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]], var.equal = var_equal)
    } else if (paired == FALSE & wilcoxon == TRUE) {
      tres <- wilcox.test(data[[x]] ~ data[[iv]])
    } else if (paired == TRUE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]],
        var.equal = var_equal,
        paired = TRUE
      )
    } else {
      tres <- wilcox.test(data[[x]] ~ data[[iv]],
        paired = TRUE
      )
    }

    c(
      p_value = tres$p.value
    )
  })

  out <- as.data.frame(do.call(rbind, out))
  out <- cbind(variable = dvs, out)
  names(out) <- gsub("[^0-9A-Za-z_]", "", names(out))

  out$p_value <- p.adjust(out$p_value, p_adj)
  out$conclusion <- ifelse(out$p_value < alpha,
    paste0("Reject H0 at ", alpha * 100, "%"),
    paste0("Do not reject H0 at ", alpha * 100, "%")
  )
  out$p_value <- ifelse(out$p_value < 0.001,
    "<0.001",
    round(out$p_value, 3)
  )

  return(out)
}

Applied to our dataset, with no adjustment method for the p-values:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

And with the Holm (1979) adjustment method:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species",
  p_adj = "holm"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

Again, with the Holm’s adjustment method, we conclude that, at the 5% significance level, the two species are significantly different from each other in terms of all 4 variables.

ANOVA

Below the same process with an ANOVA. Note that we reload the dataset iris to include all three Species this time:

dat <- iris

# Edit from here
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("setosa", "versicolor"), c("setosa", "virginica"), c("versicolor", "virginica")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

Like the improved routine for the t-test, I have noticed that students and non-expert professionals understand ANOVA results presented this way much more easily compared to the default R outputs.

With one graph for each variable, it is easy to see that all species are different from each other in terms of all 4 variables.³

If you want to apply the same automated process to your data, you will need to modify the name of the grouping variable (Species), the names of the variables you want to test (Sepal.Length, etc.), whether you want to perform an ANOVA (anova) or Kruskal-Wallis test (kruskal.test) and finally specify the comparisons for the post-hoc tests.⁴

To go even further

As we have seen, these two improved R routines allow to:

Perform t-tests and ANOVA on a small or large number of variables with only minor changes to the code. I basically only have to replace the variable names and the name of the test I want to use. It takes almost the same time to test one or several variables so it is quite an improvement compared to testing one variable at a time.
Share test results in a much proper and cleaner way. This is possible thanks to a graph showing the observations by group and the p-value of the appropriate test included directly on the graph. This is particularly important when communicating results to a wider audience or to people from diverse backgrounds.

However, like most of my R routines, these two pieces of code are still a work in progress. Below are some additional features I have been thinking of and which could be added in the future to make the process of comparing two or more groups even more optimal:

Add the possibility to select variables by their numbering in the dataframe. For the moment it is only possible to do it via their names. This will allow to automate the process even further because instead of typing all variable names one by one, we could simply type 4:25 (to test variables 4 to 25 for instance).
Add the possibility to choose a p-value adjustment method. Currently, raw p-values are displayed in the graphs and I manually adjust them afterwards or adjust the $\alpha$.
When comparing more than two groups, it is only possible to apply an ANOVA or Kruskal-Wallis test at the moment. A major improvement would be to add the possibility to perform a repeated measures ANOVA (i.e., an ANOVA when the samples are dependent). It is currently already possible to do a t-test with two paired samples, but it is not yet possible to do the same with more than two groups.
Another less important (yet still nice) feature when comparing more than 2 groups would be to automatically apply post-hoc tests only in the case where the null hypothesis of the ANOVA or Kruskal-Wallis test is rejected (so when there is at least one group different from the others, because if the null hypothesis of equal groups is not rejected we do not apply a post-hoc test). At the present time, I manually add or remove the code that displays the p-values of post-hoc tests depending on the global p-value of the ANOVA or Kruskal-Wallis test.

I will try to add these features in the future, or I would be glad to help if the author of the {ggpubr} package needs help in including these features (I hope he will see this article!).

Last but not least, the following packages may be of interest to some readers:

If you want to report statistical results on a graph, I advise you to check the {ggstatsplot} package and in particular the ggbetweenstats() and ggwithinstats() functions. These functions allow to compare a continuous variable across multiple groups or conditions (for both independent and paired samples). Two advantages of the functions is that:
- it is very easy to switch from parametric to nonparametric tests and
- it automatically runs an ANOVA or t-test depending on the number of groups to compare

Note that many different statistical results are displayed on the graph, not only the name of the test and the p-value so a bit of simplicity and clarity is lost for more precision. However, it is still very convenient to be able to include tests results on a graph in order to combine the advantages of a visualization and a sound statistical analysis. Something that I still need to figure out is how to run the code on several variables at once.

The {compareGroups} package also provides a nice way to compare groups. It comes with a really complete Shiny app, available with:

# install.packages("compareGroups")
library(compareGroups)
cGroupsWUI()

Update with the `{ggstatsplot}` package

Several months after having written this article, I finally found a way to plot and run analyses on several variables at once with the package {ggstatsplot} (Patil 2021). This was the main feature I was missing and which prevented me from using it more often.

Although I still find that too much statistical details are displayed (in particular for non experts), I still believe the ggbetweenstats() and ggwithinstats() functions are worth mentioning in this article. I actually now use those two functions almost as often as my previous routines because:

I do not have to care about the number of groups to compare, the functions automatically choose the appropriate test according to the number of groups (ANOVA for 3 groups or more, and t-test for 2 groups)
I can select variables based on their column numbering, and not based on their names anymore (which prevents me from writing those variable names manually)
When comparing 3 or more groups (so for ANOVA, Kruskal-Wallis, repeated measure ANOVA or Friedman), $p$-values of the post-hoc tests within each dependent variable are by default the adjusted $p$-values (Holm is the default but many adjustment methods are available)
It is possible to compare both independent and paired samples, no matter the number of groups (remember that with the ggpubr package I could only do paired samples for two samples, not for 3 samples)
They allow to easily switch between the parametric and nonparametric version
All this in a more concise manner using the {purrr} package

For those of you who are interested, below my updated R routine which include these functions and applied this time on the penguins dataset.

library(palmerpenguins)

dat <- penguins
str(dat)

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

We illustrate the routine for two groups with the variables sex (two factors) as independent variable, and the 4 quantitative continuous variables bill_length_mm, bill_depth_mm, flipper_length_mm and body_mass_g as dependent variables:

library(ggstatsplot)
library(tibble)

# Comparison between sexes

# edit from here
x <- "sex"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

We now illustrate the routine for 3 groups or more with the variable species (three factors) as independent variable, and the 4 same dependent variables:

# Comparison between species

# edit from here
x <- "species"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

As you can see, I only have to specify:

the name of the grouping variable (sex and species),
the number of the dependent variables (variables 3 to 6 in the dataset),
whether I want to use the parametric or nonparametric version and
whether samples are independent (paired = FALSE) or paired (paired = TRUE).

Everything else is automated—the outputs show a graphical representation of what we are comparing, together with the details of the statistical analyses in the subtitle of the plot (the $p$-value among others).

Note that the code shown above is actually the same if I want to compare 2 groups or more than 2 groups. I wrote twice the same code (once for 2 groups and once again for 3 groups) for illustrative purposes only, but they are the same and should be treated as one for your projects.

I must admit I am quite satisfied with this routine, now that:

I can automate it on many variables at once and I do not need to write the variable names manually anymore,
at the same time, I can choose the appropriate test among all the available ones (depending on the number of groups, whether they are paired or not, and whether I want to use the parametric or nonparametric version).

Nonetheless, I must also admit that I am still not satisfied with the level of details of the statistical results. As already mentioned, many students get confused and get lost in front of so much information (except the $p$-value and the number of observations, most of the details are rather obscure to them because they are not covered in introductory statistic classes).

I saved time thanks to all improvements in comparison to my previous routine, but I definitely lose time when I have to point out to them what they should look for. After discussing with other professors, I noticed that they have the same problem.

For the moment, you can only print all results or none. I have opened an issue kindly requesting to add the possibility to display only a summary (with the $p$-value and the name of the test for instance).⁵ I will update again this article if the maintainer of the package includes this feature in the future. So stay tuned!

Conclusion

Thanks for reading.

I hope this article will help you to perform t-tests and ANOVA for multiple variables at once and make the results more easily readable and interpretable by non-scientists. Learn more about the t-test to compare two groups, or the ANOVA to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 65–70.

McDonald, JH. 2014. “Multiple Tests.” Handbook of Biological Statistics. 3rd Ed Baltimore, Maryland: Sparky House Publishing, 233–36.

Patil, Indrajeet. 2021. “Visualizations with statistical details: The ’ggstatsplot’ approach.” Journal of Open Source Software 6 (61): 3167. https://doi.org/10.21105/joss.03167.

In theory, an ANOVA can also be used to compare two groups as it will give the same results compared to a Student’s t-test, but in practice we use the Student’s t-test to compare two groups and the ANOVA to compare three groups or more.↩︎
Do not forget to separate the variables you want to test with |.↩︎
Do not forget to adjust the $p$-values or the significance level $\alpha$. If you use the Bonferroni correction, the adjusted $\alpha$ is simply the desired $\alpha$ level divided by the number of comparisons.↩︎
Post-hoc test is only the name used to refer to a specific type of statistical tests. Post-hoc test includes, among others, the Tukey HSD test, the Bonferroni correction, Dunnett’s test. Even if an ANOVA or a Kruskal-Wallis test can determine whether there is at least one group that is different from the others, it does not allow us to conclude which are different from each other. For this purpose, there are post-hoc tests that compare all groups two by two to determine which ones are different, after adjusting for multiple comparisons. Concretely, post-hoc tests are performed to each possible pair of groups after an ANOVA or a Kruskal-Wallis test has shown that there is at least one group which is different (hence “post” in the name of this type of test). The null and alternative hypotheses and the interpretations of these tests are similar to a Student’s t-test for two samples.↩︎
I am open to contribute to the package if I can help!↩︎

How to perform a one-sample t-test by hand and in R: test on one mean

Mon, 09 Mar 2020 00:00:00 +0000

Introduction

After having written an article on the Student’s t-test for two samples (independent and paired samples), I believe it is time to explain in details how to perform one-sample t-tests by hand and in R.

One-sample t-test is an important part of inferential statistics (probably one of the first statistical test that students learn). Remind that, unlike descriptive statistics, inferential statistics is a branch of statistics aiming at drawing conclusions about one or two populations, based on a subset (or two) of that population (called samples). In other words, we first collect a random set of observations from a population, and then some measurements are calculated in order to generalize to the population the information found through the sample.

In this context, the one-sample t-test is used to determine whether the mean of a measurement variable is different from a specified value (a belief or a theoretical expectation for example). It works as follows: if the mean of the sample is too distant from the specified value (the value under the null hypothesis), it is considered that the mean of the population is different from what is expected. On the contrary, if the mean of the sample is close to the specified value, we cannot reject the hypothesis that the population mean is equal to what is expected.

Like the Student’s t-test for two samples and the ANOVA (for 3 or more samples), there are also different versions of the one-sample t-test. Luckily, there are only two different versions for this test (the Student’s t-test for two samples has 5 versions!). The difference between the two versions of the one-sample t-test lies in the fact that one version is used when the variance of the population (not the variance of the sample!) is known, the other version being used when the variance of the population is unknown.

In this article, I will first detail step by step how to perform both versions of the one-sample t-test by hand. The analyses will be done on a small set of observations for the sake of illustration and easiness. I will then show how to perform this test in R with the exact same data in order to verify the results found by hand. Reminders about the reasoning behind hypothesis tests, interpretations of the p-value and the results, and assumptions of this test will also be presented.

Note that the aim of this article is to show how to compute the one-sample t-test by hand and in R, so we refrain from testing the assumptions and we assume all assumptions are met for this exercise. For completeness, we still mention the assumptions and how to test them. Interested readers are invited to have a look at the end of the article for more information about these assumptions.

Null and alternative hypothesis

Before diving into the computations of the one-sample t-test by hand, let’s recap the null and alternative hypotheses of this test:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu \ne \mu_0$

where $\mu$ is the population mean and $\mu_0$ is the known or hypothesized value of the mean in the population.

This is in the general case where we simply want to determine whether the population mean is different (in terms of the dependent variable) compared to the hypothesized value. In this sense, we have no prior belief about the population mean being larger or smaller than the hypothesized value. This type of test is referred as a two-sided or bilateral test.

If we have some prior beliefs about the population mean being larger or smaller than the hypothesized value, the one-sample t-test also allows to test the following hypotheses:

$H_0$: $\mu = \mu_0$
$H_1$: $\mu > \mu_0$

$H_0$: $\mu = \mu_0$
$H_1$: $\mu < \mu_0$

In the first case, we want to test if the population mean is significantly larger than the hypothesized value, while in the latter case, we want to test if the population mean is significantly smaller than the hypothesized value. This type of test is referred as a one-sided or unilateral test.

Hypothesis testing

In statistics, many statistical tests is in the form of hypothesis tests. Hypothesis tests are used to determine whether a certain belief can be deemed as true (plausible) or not, based on the data at hand (i.e., the sample(s)). Most hypothesis tests boil down to the following 4 steps:¹

State the null and alternative hypothesis.
Compute the test statistic, denoted t-stat. Formulas to compute the test statistic differ among the different versions of the one-sample t-test but they have the same structure. See scenarios 1 and 2 below to see the different formulas.
Find the critical value given the theoretical statistical distribution of the test, the parameters of the distribution and the significance level $\alpha$. For the two versions of the one-sample t-test, it is either the normal or the Student’s t distribution (t denoting the Student distribution and z denoting the normal distribution).
Conclude by comparing the t-stat (found in step 2.) with the critical value (found in step. 3). If the t-stat lies in the rejection region (determined thanks to the critical value and the direction of the test), we reject the null hypothesis, otherwise we do not reject the null hypothesis. These two alternatives (reject or do not reject the null hypothesis) are the only two possible solutions, we never “accept” an hypothesis. It is also a good practice to always interpret the decision in the terms of the initial question.

For the interested reader, see these 4 steps of hypothesis testing in more details in this article.

Two versions of the one-sample t-test

There are two versions of the one-sample t-test, depending on whether the variance of the population (not the variance of the sample!) is known or unknown. This criteria is rather straightforward, we either know the variance of the population or we do not. The variance of the population cannot be computed because if you can compute the variance of a population, it means you have the data for the whole population, then there is no need to do a hypothesis test anymore…

So the variance of the population is either given in the statement (use them in that case), or there is no information about the variance and in that case, it is assumed that the variance is unknown. In practice, the variance of the population is most of the time unknown. However, we still illustrate how to do both versions of this test by hand and in R in the next sections following the 4 steps of a hypothesis test.

How to compute the one-sample t-test by hand?

Note that the data are artificial and do not represent any real variable. Furthermore, remind that the assumptions may or may not be met. The point of the article is to detail how to compute the different versions of the test by hand and in R, so all assumptions are assumed to be met. Moreover, we assume that for all tests the significance level, that is, the type I error is $\alpha = 5$%.

If you are interested in applying these tests by hand without having to do the computations yourself, here is a Shiny app which does it for you. You just need to enter the data and choose the appropriate version of the test thanks to the sidebar menu. There is also a graphical representation that helps you to visualize the test statistic and the rejection region. I hope you will find it useful!

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

value
0.9
-0.8
1.3
-0.3
1.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 0.56$
variance of the population: $\sigma^2 = 1$
$\mu_0 = 0$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 0$ and $H_1: \mu \ne 0$. ($\ne$ because we want to test whether the population mean is different from 0, we do not impose a direction in the test.)
Test statistic: \[z_{obs} = \frac{\bar{x} - \mu_0}{\frac{\sigma}{\sqrt{n}}} = \frac{0.56-0}{0.447} = 1.252\]
Critical value: $\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the population mean is equal to 0, or there is no sufficient evidence in the data to conclude that the population mean is different from 0.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

value
7.9
5.8
6.3
7.3
6.7

So we have:

5 observations: $n = 5$
mean of the sample: $\bar{x} = 6.8$
standard deviation of the sample: $s = 0.825$
$\mu_0 = 5$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu = 5$ and $H_1: \mu > 5$. (> because we want to test whether the population mean is larger than 5.)
Test statistic: \[t_{obs} = \frac{\bar{x} - \mu_0}{\frac{s}{\sqrt{n}}} = \frac{6.8-5}{0.369} = 4.881\]
Critical value: $t_{\alpha, n - 1} = t_{0.05, 4} = 2.132$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection region is thus from 2.132 to $+\infty$. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the population mean is larger than 5.

This concludes how to perform the two versions of the one-sample t-test by hand. In the next sections, we detail how to perform the exact same tests in R.

Different underlying distributions for the critical value

As you may have noticed, the underlying probability distributions used to find the critical value are different depending on whether the variance of the population is known or unknown.

The underlying probability distribution when the variance is known (scenario 1) is the normal distribution, while the probability distribution in the case where the variance is unknown (scenario 2) is the Student’s t distribution. This difference is partially explained by the fact that when the variance of the population is unknown, there is more “uncertainty” in the data, so we need to use the Student’s t distribution instead of the normal distribution.

Note that when the sample size is large (usually when n > 30), the Student’s t distribution tends to a normal distribution. Using a normal distribution when the variance is known and a Student’s t distribution when the variance is unknown also applies to a t-test for two samples.

How to compute the one-sample t-test in R?

A good practice before doing t-tests in R is to visualize the data thanks to a boxplot (or eventually a histogram or a density plot). A boxplot gives a first indication on the location of the sample, and thus, a first indication on whether the null hypothesis is likely to be rejected or not. However, even if a boxplot or a density plot is great in showing the distribution of a sample, only a sound statistical test will confirm our first impression.

After a visualization of the data, we replicate in R the results found by hand. Note that we use the same data, the same assumptions and the same question for both scenarios to facilitate the comparison between the tests performed by hand and in R.

Scenario 1: variance of the population is known

For the first scenario, suppose the data below. Moreover, suppose that the population variance $\sigma^2 = 1$ and that we would like to test whether the population mean is different from 0.

dat1 <- data.frame(
  value = c(0.9, -0.8, 1.3, -0.3, 1.7)
)

dat1

##   value
## 1   0.9
## 2  -0.8
## 3   1.3
## 4  -0.3
## 5   1.7

library(ggplot2)

ggplot(dat1) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Note that you can use the {esquisse} RStudio addin if you want to draw a boxplot with the package {ggplot2} without writing the code yourself. If you prefer the default graphics, use the boxplot() function:

boxplot(dat1$value)

The boxplot shows that the distribution of the sample is not distant from 0 (the hypothesized value), so we tend to believe that we will not be able to reject the null hypothesis that the population mean is equal to 0. However, only a formal statistical test will confirm this belief.

Below a function to perform a t-test with a known population variance, with arguments accepting the sample (x), the variance of the population (V), the mean under the null hypothesis (m0, default is 0), the significance level (alpha, default is 0.05) and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test2 <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test2(dat1$value,
  V = 1
)
test

## $mean
## [1] 0.56
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 1.252198
## 
## $p.value
## [1] 0.2104977
## 
## $LCL
## [1] -0.3165225
## 
## $UCL
## [1] 1.436523
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the sample mean, the hypothesized value and the population variance (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.2104977

The p-value is 0.21 so at the 5% significance level we do not reject the null hypothesis. There is no sufficient evidence in the data to reject the hypothesis that the population mean is equal to 0. This result confirms what we found by hand.

Note that a similar function exists in the {BSDA} package:²

library(BSDA)

z.test(dat1$value,
  alternative = "two.sided",
  mu = 0,
  sigma.x = 1,
  conf.level = 0.95
)

## 
## 	One-sample z-Test
## 
## data:  dat1$value
## z = 1.2522, p-value = 0.2105
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.3165225  1.4365225
## sample estimates:
## mean of x 
##      0.56

If you are unfamiliar with the concept of p-value, I invite you to read my note on p-value and significance level $\alpha$.

To sum up what have been said in that article about p-value and significance level $\alpha$:

If the p-value is smaller than the predetermined significance level $\alpha$ (usually 5%) so if p-value < 0.05, we reject the null hypothesis
If the p-value is greater than or equal to the predetermined significance level $\alpha$ (usually 5%) so if p-value $\ge$ 0.05, we do not reject the null hypothesis

This applies to all statistical tests without exception. Of course, the null and alternative hypotheses change depending on the test.

Scenario 2: variance of the population is unknown

For the second scenario, suppose the data below. Moreover, suppose that the variance in the population is unknown and that we would like to test whether the population mean is larger than 5.

dat2 <- data.frame(
  value = c(7.9, 5.8, 6.3, 7.3, 6.7)
)

dat2

##   value
## 1   7.9
## 2   5.8
## 3   6.3
## 4   7.3
## 5   6.7

ggplot(dat2) +
  aes(y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the box is quite distant from the hypothesized value of 5. From this boxplot, we can expect the test to reject the null hypothesis of the population mean being equal to 5. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” t-test for one-sample. Note that in our case the alternative hypothesis is $H_1: \mu > 5$ so we need to add the arguments mu = 5 and alternative = "greater" to the function because the default arguments are mu = 0 and the two-sided test:

test <- t.test(dat2$value,
  mu = 5,
  alternative = "greater"
)

test

## 
## 	One Sample t-test
## 
## data:  dat2$value
## t = 4.8809, df = 4, p-value = 0.004078
## alternative hypothesis: true mean is greater than 5
## 95 percent confidence interval:
##  6.013814      Inf
## sample estimates:
## mean of x 
##       6.8

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used, the hypothesized value and the sample mean (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.004077555

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the population mean is significantly larger than 5. This result confirms what we found by hand.

Confidence interval

Note that the confidence interval can be extracted with $conf.int:

test$conf.int

## [1] 6.013814      Inf
## attr(,"conf.level")
## [1] 0.95

You can see that the 95% confidence interval for the population mean is $[6.01; \infty]$, meaning that, at the significance level $\alpha = 5$%, we reject the null hypothesis as long as the hypothesized value $\mu_0$ is below 6.01, otherwise the null hypothesis cannot be rejected.

Combination of plot and statistical test

After having written this article, I discovered the {ggstatsplot} package which I believe is worth mentioning here, in particular the gghistostats() function for one-sample Student’s t-test.

This function combines a histogram—representing the distribution—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for scenario 2. Unfortunately, the package does not allow to run the test for scenario 1.

Scenario 2: variance of the population is unknown

# load packages
library(ggstatsplot)
library(ggplot2)

# plot with stat test
gghistostats(
  data = dat2, # dataframe from which variable is to be taken
  x = value, # numeric variable whose distribution is of interest
  type = "parametric", # for student's t-test
  test.value = 5 # default value is 0
) +
  labs(caption = NULL) # remove caption

The p-value is displayed after p = in the subtitle of the plot. Based on this plot and the p-value being lower than 5% (p-value = 0.008), we reject the null hypothesis that the population mean is equal to 5.

Note that, the p-value is two times as large as the one obtained with the t.test() function because when we ran t.test() we specified alternative = "greater" (i.e., a one-sided test). In our plot with the gghistostats() function, it is a two-sided test that is performed by default, that is, alternative = "two.sided".

Assumptions

Variable type: The dependent variable (i.e., the measured variable) must be measured on a continuous or ordinal scale.
Independence: The data, collected from a representative and randomly selected portion of the population, should be independent of one another. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another). If not, it is most likely that you have independent samples.
Normality:
- With a small sample size (usually $n < 30$), observations should follow a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. Some transformations, such as among others, the logarithm, the square root or the Box-Cox transformation can be applied on the observations to transform you data to better fit the normal distribution. If, even after a transformation, your data still do not follow a normal distribution, the one-sample Wilcoxon test (wilcox.test(variable_name, data = dat in R) can be applied. This non-parametric test is robust to non normal distributions so it does not require normality of the data.
- With a large sample size ($n \ge 30$), normality of the data is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). It is therefore not required to test the normality assumption when the number of observations is large.
Outliers: There should be no outliers in your data. An observation slightly different from the others does not pose a problem, but it starts to be an issue when you have at least one extreme outlier. In presence of at least one extreme outlier, it is best to transform your data (with the logarithm transformation for instance, as you would do with a non normal distribution) or use the non-parametric one-sample Wilcoxon test.

Conclusion

Thanks for reading.

I hope this article helped you to understand how the different versions of the one-sample t-test work and how to perform them by hand and in R.

If you are interested, here is a Shiny app to perform these tests by hand easily (you just need to enter your data and select the appropriate version of the test thanks to the sidebar menu). Moreover, read this article if you would like to know how to compute the Student’s t-test but this time, for two samples—in order to compare two dependent or independent groups—or this article if you want to use an ANOVA to compare 3 or more groups.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

It is a least the case regarding parametric hypothesis tests. A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. In the case of the one-sample t-test, it is based on the Student’s t distribution with a single parameter, the degrees of freedom ($df = n - 1$ where $n$ is the sample size), or the normal distribution.↩︎
Thanks gmacar for pointing it out to me.↩︎

The 9 concepts and formulas in probability that every data scientist should know

Tue, 03 Mar 2020 00:00:00 +0000

What is probability?

A probability is a number that reflects the chance that a particular event will occur. In other words, it quantifies (on a scale from 0 to 1, or from 0% to 100%) how likely an event is to occur.

Probability is a branch of mathematics that provides models to describe random processes. These mathematical tools allow to establish theoretical models for random phenomena and to use them to make predictions. Like every model, the probabilistic model is a simplification of the world. However, the model is useful as soon as it captures the essential features.

In this article, we present 9 fundamental formulas and concepts in probability that every data scientist should understand and master in order to appropriately handle any project involving probabilities.

1. A probability is always between 0 and 1

The probability of an event is always between 0 and 1 (or 0% and 100%). If we denote the probability that an event A (which could be any event) occurs by $P(A)$, we have:

\[0 \le P(A) \le 1\]

If an event is impossible: $P(A) = 0$
If an event is certain: $P(A) = 1$

For example, throwing a 7 with a standard six-sided dice (with faces ranging from 1 to 6) is impossible so its probability is equal to 0. Throwing head or tail with a coin is certain, so its probability is equal to 1.

2. Compute a probability

If the elements of a sample space (the set of all possible results of a randomized experiment) are equiprobable (= all elements have the same probability), then the probability of an event occurring is equal to the number of favourable cases (number of ways it can happen) divided by the number of possible cases (total number of outcomes):

\[P(A) = \frac{\text{number of favourable cases}}{\text{number of possible cases}}\]

For example, all numbers of a six-sided dice are equiprobable since they all have the same probability of occurring. The probability of rolling a 3 with a dice is thus

\[P(3) = \frac{\text{number of favourable cases}}{\text{number of possible cases}} = \frac{1}{6}\]

because there is only one favourable case (there is only one face with a 3 on it), and there are 6 possible cases (because there are 6 faces altogether).

3. Complement of an event

The probability of the complement (or opposite) of an event is:

\[P(\text{not A}) = P(\bar{A}) = 1 - P(A)\]

For instance, the probability of not throwing a 3 with a dice is:

\[P(\bar{A}) = 1 - P(A) = 1 - \frac{1}{6} = \frac{5}{6}\]

4. Union of two events

The probability of the union of two events is the probability of either occurring:

\[\begin{align} P(\text{A or B)} &= P(A \cup B) \\ &= P(A) + P(B) - P(A \cap B) \end{align}\]

Suppose that the probability of a fire breaking out in two houses in a given year is:

in house A: 60%, so $P(A) = 0.6$
in house B: 45%, so $P(B) = 0.45$
in at least one of the two houses: 80%, so $P(A \cup B) = 0.8$

Graphically we have

The probability of a fire breaking out in house A or house B is

\[P(A \cup B) = P(A) + P(B) - P(A \cap B)\] \[= 0.6 + 0.45 - 0.25 = 0.8\]

By summing $P(A)$ and $P(B)$, the intersection of A and B, i.e. $P(A \cap B)$, is counted twice. This is the reason we subtract it to count it only once.

If two events are mutually exclusive (i.e., two events that cannot occur simultaneously), the probability of both events occurring $P(A \cap B)$ is equal to 0, so the above formula becomes

\[P(A \cup B) = P(A) + P(B)\]

For example, the event “rolling a 3” and the event “rolling a 6” on a six-sided dice are two mutually exclusive events since they cannot both occur at the same time. Since their joint probability is equal to 0, the probability of rolling a 3 or 6 on a six-sided dice is

\[P(3 \cup 6) = P(3) + P(6) = \frac{1}{6} + \frac{1}{6} = \frac{1}{3}\]

5. Intersection of two events

If two events are independent, the probability of the intersection of the two events (i.e., the joint probability) is the probability of the two events occurring:

\[P(\text{A and B)} = P(A \cap B) = P(A) \cdot P(B)\]

For instance, if two coins are flipped, the probability of both coins being tails is

\[P(T_1 \cap T_2) = P(T_1) \cdot P(T_2) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\]

where $T_1$ ($T_2$) denotes the event that the first (second) coin is tail.

Note that $P(A \cap B) = P(B \cap A)$.

If two events are mutually exclusive, their joint probability is equal to 0:

\[P(A \cap B) = 0\]

6. Independence of two events

Another important concept in probability is the independence of two events. Formally, the events A and B are independent if and only if

\[P(A \cap B) = P(A) \cdot P(B)\]

If the equality holds, the two events are said to be independent, otherwise the two events are said to be dependent.

In the example of the two coins:

\[P(T_1 \cap T_2) = \frac{1}{4}\]

and

\[P(T_1) \cdot P(T_2) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\]

so the following equality holds

\[P(T_1 \cap T_2) = P(T_1) \cdot P(T_2) = \frac{1}{4}\]

The two events are thus independent, denoted $T_1{\perp\!\!\!\perp}T_2$.

In the example of the fire breaking out in two houses (see section 4):

\[P(A \cap B) = 0.25\]

and

\[P(A) \cdot P(B) = 0.6 \cdot 0.45 = 0.27\]

so the following equality does not hold

\[P(A \cap B) \ne P(A) \cdot P(B)\]

The two events are thus dependent (or not independent), denoted $A \not\!\perp\!\!\!\perp B$.

7. Conditional probability

Suppose two events A and B and $P(B) > 0$. The conditional probability of A given (knowing) B is the likelihood of event A occurring given that event B has occurred:

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\] \[= \frac{P(B \cap A)}{P(B)} \text{ (since } P(A \cap B) = P(B \cap A))\]

Note that, in general, the probability of A given B is not equal to the probability of B given A, that is, $P(A | B) \ne P(B | A)$.

From the formula of the conditional probability, we can derive the multiplicative law:

\[P(A | B) = \frac{P(A \cap B)}{P(B)} \text{ (Eq. 1)}\] \[P(A | B) \cdot P(B) = \frac{P(A \cap B)}{P(B)} \cdot P(B)\] \[P(A | B) \cdot P(B) = P(A \cap B) \text{ (multiplicative law)}\]

If two events are independent, $P(A \cap B) = P(A) \cdot P(B)$, and:

$P(B) > 0$, the conditional probability becomes

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\] \[P(A | B) = \frac{P(A) \cdot P(B)}{P(B)}\] \[P(A | B) = P(A) \text{ (Eq. 2)}\]

$P(A) > 0$, the conditional probability becomes

\[P(B | A) = \frac{P(B \cap A)}{P(A)}\] \[P(B | A) = \frac{P(B) \cdot P(A)}{P(A)}\] \[P(B | A) = P(B) \text{ (Eq. 3)}\] Equations 2 and 3 mean that knowing that one event occurred does not influence the probability of the outcome of the other event. This is in fact the definition of the independence: if knowing that one event occurred does not help to predict (does not influence) the outcome of the other event, the two events are by essence independent.

Bayes’ theorem

From the formulas of the conditional probability and the multiplicative law, we can derive the Bayes’ theorem:

\[\begin{align} P(B | A) &= \frac{P(B \cap A)}{P(A)} \\ & \text{(from conditional probability)} \\ &= \frac{P(A \cap B)}{P(A)} \\ & \text{(since } P(A \cap B) = P(B \cap A)) \\ &= \frac{P(A | B) \cdot P(B)}{P(A)} \\ & \text{ (from multiplicative law)} \end{align}\]

which is equivalent to

\[\begin{align} P(B | A) &= \frac{P(B | A) \cdot P(A)}{P(B)} \\ & \text{(Bayes' theorem)} \end{align}\]

Example

In order to illustrate the conditional probability and the Bayes’ theorem, suppose the following problem:

In order to determine the presence of a disease in a person, a blood test is performed. When a person has the disease, the test can reveal the disease in 80% of cases. When the disease is not present, the test is negative in 90% of cases. Experience has shown that the probability of the disease being present is 10%. A researcher would like to know the probability that an individual has the disease given that the result of the test is positive.

To answer this question, the following events are defined:

P: the test result is positive
D: the person has the disease

Moreover, we use a tree diagram to illustrate the statement:

(The sum of all 4 scenarios must be equal to 1 since these 4 scenarios cover all possible cases.)

We are looking for the probability that an individual has the disease given that the result of the test is positive, $P(D | P)$.

Following the formula of the conditional probability (Eq. 1) we have:

\[P(A | B) = \frac{P(A \cap B)}{P(B)}\]

In terms of our problem:

\[P(D | P) = \frac{P(D \cap P)}{P(P)}\] \[P(D | P) = \frac{0.08}{P(P)} \text{ (Eq. 4)}\]

From the tree diagram, we can see that a positive test result is possible under two scenarios:

when a person has the disease, or
when the person does not actually have the disease (because the test is not always correct).

In order to find the probability of a positive test result, $P(P)$, we need to sum up those two scenarios:

\[\begin{align} P(P) &= P(D \cap P) + P(\bar{D} \cap P) \\ &= 0.08 + 0.09 \\ &= 0.17 \end{align}\]

Eq. 4 then becomes

\[P(D | P) = \frac{0.08}{0.17} = 0.4706\]

The probability of having the disease given that the result of the test is positive is only 47.06%. This means that in this specific case (with the same percentages), an individual with a positive test has less than 1 chance out of 2 of having the disease!

This relatively small percentage is due to the facts that the disease is quite rare (only 10% of the population is affected) and that the test is not always correct (sometimes it detects the disease although it is not present, and sometimes it does not detect it although it is present).

As a consequence, a higher percentage of healthy people have a positive result (9%) compared to the percentage of people who have a positive result and who actually have the disease (8%). This explains why several diagnostic tests are often performed before announcing the diagnosis, especially for rare diseases.

8. Accuracy measures

Based on the example of the disease and the diagnostic test presented above, we explain the most common accuracy measures:

False negative (number and rate)
False positive (number and rate)
Sensitivity
Specificity
Positive predictive value
Negative predictive value

Before diving into the details of these accuracy measures, here is an overview of the measures and the tree diagram with the labels added for each of the 4 scenarios:

Adapted from Wikipedia

False negatives

The false negatives (FN) are the number of people incorrectly labeled as not having the disease or the condition, when in reality it is present. It is like telling a women who is 7 months pregnant that she is not pregnant.

From the tree diagram, we have:

\[FN = P(D \cap \bar{P}) = 0.02\]

Moreover, the false negative rate (FNR) is defined as

\[\begin{align} FNR &= \frac{FN}{FN + TP} \\ &= P(\bar{P} | D) \\ &= \frac{P(\bar{P} \cap D)}{P(D)} \\ &= \frac{0.02}{0.08 + 0.02} \\ &= 0.2 \end{align}\]

False positives

The false positives (FP) are the number of people incorrectly labeled as having the disease or the condition, when in reality it is not present. It is like telling a man he is pregnant.

From the tree diagram, we have:

\[FP = P(\bar{D} \cap P) = 0.09\]

Moreover, the false positive rate (FPR) is defined as

\[\begin{align} FPR &= \frac{FP}{FP + TN} \\ &= P(P | \bar{D}) \\ &= \frac{P(P \cap \bar{D})}{P(\bar{D})} \\ &= \frac{0.09}{0.09 + 0.81} \\ &= 0.1 \end{align}\]

Sensitivity

The sensitivity of a test, also referred as the recall, measures the ability of a test to detect the condition when the condition is present (the percentage of sick people who are correctly identified as having the disease):

\[ Sensitivity = \frac{TP}{TP + FN}\]

where TP is the true positives.

From the tree diagram, we have:

\[\begin{align} Sensitivity &= \frac{TP}{TP + FN} \\ &= P(P|D) \\ &= 0.8 \end{align}\]

Note also that $1 - sensitivity = FNR$ (false negative rate).

Specificity

The specificity of a test measures the ability of a test to correctly exclude the condition when the condition is absent (the percentage of healthy people who are correctly identified as not having the disease):

\[Specificity = \frac{TN}{TN + FP}\]

where TN is the true negatives.

From the tree diagram, we have:

\[\begin{align} Specificity &= \frac{TN}{TN + FP} \\ &= P(\bar{P} | \bar{D}) \\ &= 0.9 \end{align}\]

Note also that $1 - specificity = FPR$ (false positive rate).

Positive predictive value

The positive predictive value, also referred as the precision, is the proportion of positives that correspond to the presence of the condition, so the proportions of positive results that are true positive results:

\[PPV = \frac{TP}{TP+FP}\]

From the tree diagram, we have:

\[\begin{align} PPV &= \frac{TP}{TP+FP} \\ &= P(D | P) \\ &= \frac{P(D \cap P)}{P(P)} \\ &= \frac{0.08}{0.08+0.09} \\ &= 0.4706 \end{align}\]

Negative predictive value

The negative predictive value is the proportion of negatives that correspond to the absence of the condition, so the proportions of negative results that are true negative results:

\[NPV = \frac{TN}{TN + FN}\]

From the tree diagram, we have:

\[\begin{align} NPV &= \frac{TN}{TN + FN} \\ &= P(\bar{D} | \bar{P}) \\ &= \frac{P(\bar{D} \cap \bar{P})}{P(\bar{P})} \\ &= \frac{0.81}{0.81+0.02} \\ &= 0.9759 \end{align}\]

9. Counting techniques

In order to use the formula in section 2, one must know how to count the number of possible elements (both for favorable and possible cases).

There are 3 main counting techniques in probability:

Multiplication
Permutation
Combination

See below how to count the number of possible elements in case of equiprobable results.

Multiplication

The multiplication rule is as follows:

\[\#(A \times B) = (\#A) \times (\#B)\]

where $\#$ is the number of elements.

Example

In a restaurant, a customer has to choose a starter, a main course and a dessert. The restaurant offers 2 starters, 3 main courses and 2 desserts. How many different choices are possible?

There are $2 \cdot 3 \cdot 2 = 12$ different possible choices.

Permutation

The number of permutations is as follows:

\[\begin{align} P^r_n &= n \times (n - 1) \times \cdots \times (n - r + 1) \\ &= \frac{n !}{(n - r)!} \end{align}\]

with $r$ the length, $n$ the number of elements and $r \le n$. Note that $0! = 1$ and $k! = k \times (k - 1) \times (k - 2) \times \cdots \times 2 \times 1$ if $k = 1, 2, \dots$

The order is important in permutations!

Example

Count the permutations of length 2 of the set $A = \{a, b, c, d\}$, without a letter being repeated. How many permutations do you find?

By hand

\[P^4_2 = \frac{4!}{(4-2)!} = \frac{4\cdot3\cdot2\cdot1}{2\cdot1} = 12\]

In R

library(gtools)

x <- c("a", "b", "c", "d")

# See all different permutations
perms <- permutations(
  n = 4, r = 2, v = x,
  repeats.allowed = FALSE
)
perms

##       [,1] [,2]
##  [1,] "a"  "b" 
##  [2,] "a"  "c" 
##  [3,] "a"  "d" 
##  [4,] "b"  "a" 
##  [5,] "b"  "c" 
##  [6,] "b"  "d" 
##  [7,] "c"  "a" 
##  [8,] "c"  "b" 
##  [9,] "c"  "d" 
## [10,] "d"  "a" 
## [11,] "d"  "b" 
## [12,] "d"  "c"

# Count the number of permutations
nrow(perms)

## [1] 12

Combination

The number of combinations is as follows:

\[\begin{align} C^r_n &= \frac{P^r_n}{r!} \\ &= \frac{n !}{r!(n - r)!} \\ &= {n \choose r} \\ &= \frac{n}{r} \times \frac{n - 1}{r - 1} \times \dots \times \frac{n - r + 1}{1} \end{align}\]

with $r$ the length, $n$ the number of elements and $r \le n$.

The order is not important in combinations!

Example

In a family of 5 children, what is the probability that there are 3 girls and 2 boys? Assume that the probabilities of giving birth to a girl and a boy are equal.

By hand

Count of 3 girls and 2 boys (favourable cases): $C^3_5 = {5 \choose 3} = \frac{5!}{3!(5-3)!} = 10$
Count of possible cases: $2^5 = 32$

$\Rightarrow P(3 \text{ girls and 2 boys}) = \frac{\text{# of favourable cases}}{\text{# of possible cases}}$ \[= \frac{10}{32} = 0.3125\]

In R

Count of 3 girls and 2 boys:

choose(n = 5, k = 3)

## [1] 10

Count of possible cases:

2^5

## [1] 32

Probability of 3 girls and 2 boys:

choose(n = 5, k = 3) / 2^5

## [1] 0.3125

Conclusion

Thanks for reading.

I hope this article helped you to understand the most important formulas and concepts from probability theory.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Student's t-test in R and by hand: how to compare two groups under different scenarios?

Fri, 28 Feb 2020 00:00:00 +0000

Introduction

One of the most important test within the branch of inferential statistics is the Student’s t-test.¹ The Student’s t-test for two samples is used to test whether two groups (two populations) are different in terms of a quantitative variable, based on the comparison of two samples drawn from these two groups. In other words, a Student’s t-test for two samples allows to determine whether the two populations from which your two samples are drawn are different (with the two samples being measured on a quantitative continuous variable).²

The reasoning behind this statistical test is that if your two samples are markedly different from each other, it can be assumed that the two populations from which the samples are drawn are different. On the contrary, if the two samples are rather similar, we cannot reject the hypothesis that the two populations are similar, so there is no sufficient evidence in the data at hand to conclude that the two populations from which the samples are drawn are different. Note that this statistical tool belongs to the branch of inferential statistics because conclusions drawn from the study of the samples are generalized to the population, even though we do not have the data on the entire population.

To compare two samples, it is usual to compare a measure of central tendency computed for each sample. In the case of the Student’s t-test, the mean is used to compare the two samples. However, in some cases, the mean is not appropriate to compare two samples so the median is used to compare them via the Wilcoxon test. This article being already quite long and complete, the Wilcoxon test is covered in a separate article, together with some illustrations on when to use one test or the other.

These two tests (Student’s t-test and Wilcoxon test) have the same final goal, that is, compare two samples in order to determine whether the two populations from which they were drawn are different or not. Note that the Student’s t-test is more powerful than the Wilcoxon test (i.e., it more often detects a significant difference if there is a true difference, so a smaller difference can be detected with the Student’s t-test) but the Student’s t-test is sensitive to outliers and data asymmetry. Furthermore, within each of these two tests, several versions exist, with each version using different formulas to arrive at the final result. It is thus necessary to understand the difference between the two tests and which version to use in order to carry out the appropriate analyses depending on the question and the data at hand.

In this article, I will first detail step by step how to perform all versions of the Student’s t-test for independent and paired samples by hand. The analyses will be done on a small set of observations for the sake of illustration and easiness. I will then show how to perform this test in R with the exact same data in order to verify the results found by hand. Reminders about the reasoning behind hypothesis testing, interpretations of the p-value and the results, and assumptions of this test will also be presented.

Note that the aim of this article is to show how to compute the Student’s t-test by hand and in R, so we refrain from testing the assumptions and we assume all of them are met for this exercise. For completeness, we still mention the assumptions, how to test them and what other tests exist if one is not met. Interested readers are invited to have a look at the end of the article for more information about these assumptions.

Null and alternative hypothesis

Before diving into the computations of the Student’s t-test by hand, let’s recap the null and alternative hypotheses of this test:

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 \ne \mu_2$

where $\mu_1$ and $\mu_2$ are the means of the two populations from which the samples were drawn.

As mentioned in the introduction, although technically the Student’s t-test is based on the comparison of the means of the two samples, the final goal of this test is actually to test the following hypotheses:

$H_0$: the two populations are similar
$H_1$: the two populations are different

This is in the general case where we simply want to determine whether the two populations are different or not (in terms of the dependent variable). In this sense, we have no prior belief about a particular population mean being larger or smaller than the other. This type of test is referred as a two-sided or bilateral test.

If we have some prior beliefs about one population mean being larger or smaller than the other, the Student’s t-test also allows to test the following hypotheses:

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 > \mu_2$

$H_0$: $\mu_1 = \mu_2$
$H_1$: $\mu_1 < \mu_2$

In the first case, we want to test if the mean of the first population is significantly larger than the mean of the second, while in the latter case, we want to test if the mean of the first population is significantly smaller than the mean of the second. This type of test is referred as a one-sided or unilateral test.

Some authors argue that one-sided tests should not be used in practice for the simple reason that, if a researcher is so sure that the mean of one population is larger (smaller) than the mean of the other and would never be smaller (larger) than the other, why would she needs to test for significance at all? This a rather philosophical question and it is beyond the scope of this article. Interested readers are invited to see part of the discussion in Rowntree (2000).

Hypothesis testing

State the null and alternative hypothesis.
Compute the test statistic, denoted t-stat. Formulas to compute the test statistic differ among the different versions of the Student’s t-test but they have the same structure. See scenarios 1 to 5 below to see the different formulas.
Find the critical value given the theoretical statistical distribution of the test, the parameters of the distribution and the significance level $\alpha$. For a Student’s t-test and its extended version, it is either the normal or the Student’s t distribution (t denoting the Student distribution and z denoting the normal distribution).
Conclude by comparing the t-stat (found in step 2.) with the critical value (found in step. 3). If the t-stat lies in the rejection region (determined thanks to the critical value and the direction of the test), we reject the null hypothesis, otherwise we do not reject the null hypothesis. These two alternatives (reject or do not reject the null hypothesis) are the only two possible solutions, we never “accept” an hypothesis. It is also a good practice to always interpret the decision in the terms of the initial question.

For the interested reader, see these 4 steps of hypothesis testing in more details in this article.

Different versions of the Student’s t-test

There are several versions of the Student’s t-test for two samples, depending on whether the samples are independent or paired and depending on whether the variances of the populations are (un)equal and/or (un)known:

On the one hand, independent samples means that the two samples are collected on different experimental units or different individuals, for instance when we are working on women and men separately, or working on patients who have been randomly assigned to a control and a treatment group (and a patient belongs to only one group). On the other hand, we face paired samples when measurements are collected on the same experimental units, same individuals. This is often the case, for example in medical studies, when testing the efficiency of a treatment at two different times. The same patients are measured twice, before and after the treatment, and the dependency between the two samples must be taken into account in the computation of the test statistic by working on the differences of measurements for each subject. Paired samples are usually the result of measurements at two different times, but not exclusively. Suppose we want to test the difference in vision between the left and right eyes of 50 athletes. Although the measurements are not made at two different time (before-after), it is clear that both eyes are dependent within each subject. Therefore, the Student’s t-test for paired samples should be used to account for the dependency between the two samples instead of the standard Student’s t-test for independent samples.

Another criteria for choosing the appropriate version of the Student’s t-test is whether the variances of the populations (not the variances of the samples!) are known or unknown and equal or unequal. This criteria is rather straightforward, we either know the variances of the populations or we do not. The variances of the populations cannot be computed because if you can compute the variance of a population, it means you have the data for the whole population, then there is no need to do a hypothesis test anymore… So the variances of the populations are either given in the statement (use them in that case), or there is no information about these variances and in this case, it is assumed that the variances are unknown. In practice, the variances of the populations are most of the time unknown and the only thing to do in order to choose the appropriate version of the test is to check whether the variances are equal or not. However, we still illustrate how to do all versions of this test by hand and in R in the next sections following the 4 steps of hypothesis testing.

How to compute Student’s t-test by hand?

Note that the data are artificial and do not represent any real variable. Furthermore, remind that the assumptions may or may not be met. The point of the article is to detail how to compute the different versions of the test by hand and in R, so all assumptions are assumed to be met. Moreover, assume that the significance level $\alpha = 5$% for all tests.

Scenario 1: Independent samples with 2 known variances

For the first scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances $\sigma^2 = 1$ in both populations and that we would like to test whether the two population means are different.

value	sample
0.9	1
-0.8	1
0.1	1
-0.3	1
0.2	1
0.8	2
-0.9	2
-0.1	2
0.4	2
0.1	2

So we have:

5 observations in each sample: $n_1 = n_2 = 5$
mean of sample 1: $\bar{x}_1 = 0.02$
mean of sample 2: $\bar{x}_2 = 0.06$
variances of both populations: $\sigma^2_1 = \sigma^2_2 = 1$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 \ne 0$. ($\ne$ because we want to test whether the two means are different, we do not impose a direction in the test.)
Test statistic: \[z_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2}}}\] \[= \frac{0.02-0.06-0}{0.632} = -0.063\]
Critical value: $\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96$ (see a guide on how to read statistical tables if you struggle to find the critical value)
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the two population means are the same, or there is no sufficient evidence in the data to conclude that the two populations considered are different.

Scenario 2: Independent samples with 2 equal but unknown variances

For the second scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown but equal ($\sigma^2_1 = \sigma^2_1$) and that we would like to test whether the mean of population 1 is larger than the mean of population 2.

value	sample
1.78	1
1.5	1
0.9	1
0.6	1
0.8	1
1.9	1
0.8	2
-0.7	2
-0.1	2
0.4	2
0.1	2

So we have:

6 observations in sample 1: $n_1 = 6$
5 observations in sample 2: $n_2 = 5$
mean of sample 1: $\bar{x}_1 = 1.247$
mean of sample 2: $\bar{x}_2 = 0.1$
variance of sample 1: $s^2_1 = 0.303$
variance of sample 2: $s^2_1 = 0.315$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 > 0$. (> because we want to test if the mean of the first population is larger than the mean of the second population.)
Test statistic: \[t_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}\] where \[s_p = \sqrt{\frac{(n_1-1)s^2_1+ (n_2 - 1)s^2_2}{n_1 + n_2 - 2}} = 0.555\] so \[t_{obs} = \frac{1.247-0.1-0}{0.555 * 0.606} = 3.411\] (Note that as it is assumed the variances of the two populations are equal, a pooled (common) variance, denoted $s_p$, is computed.)
Critical value: $t_{\alpha, n_1 + n_2 - 2} = t_{0.05, 9} = 1.833$
Conclusion: The rejection region is thus from 1.833 to $+\infty$ (there is only one rejection region because it is a one-sided test). The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the mean of population 1 is larger than the mean of population 2.

Scenario 3: Independent samples with 2 unequal and unknown variances

For the third scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown and unequal ($\sigma^2_1 \ne \sigma^2_1$) and that we would like to test whether the mean of population 1 is smaller than the mean of population 2.

value	sample
0.8	1
0.7	1
0.1	1
0.4	1
0.1	1
1.78	2
1.5	2
0.9	2
0.6	2
0.8	2
1.9	2

So we have:

5 observations in sample 1: $n_1 = 5$
6 observations in sample 2: $n_2 = 6$
mean of sample 1: $\bar{x}_1 = 0.42$
mean of sample 2: $\bar{x}_2 = 1.247$
variance of sample 1: $s^2_1 = 0.107$
variance of sample 2: $s^2_1 = 0.303$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_1 = \mu_2$ and $H_1: \mu_1 - \mu_2 < 0$. (< because we want to test if the mean of the first population is smaller than the mean of the second population.)
Test statistic: \[t_{obs} = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}}\] \[= \frac{0.42-1.247-0}{0.268} = -3.084\]
Critical value: $-t_{\alpha, \upsilon}$ where \[\upsilon = \frac{\bigg(\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2} \bigg)^2}{\frac{\bigg(\frac{s^2_1}{n_1}\bigg)^2}{n_1 - 1} + \frac{\bigg(\frac{s^2_2}{n_2}\bigg)^2}{n_2 - 1}} = 8.28\] so \[-t_{0.05, 8.28} = -1.851\] Note: The degrees of freedom 8.28 does not exist in the standard Student distribution table, so simply take 8, or compute it in R with qt(p = 0.05, df = 8.28). For simplicity, this number of degrees of freedom is sometimes approximated as $df = min(n_1 - 1, n_2 - 1)$, so in this case it would be $df = 4$.
Conclusion: The rejection region is thus from $-\infty$ to -1.851. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the mean of population 1 is smaller than the mean of population 2.

Scenario 4: Paired samples where the variance of the differences is known

Student’s t-test with paired samples are a bit different than with independent samples, they are actually more similar to one sample Student’s t-test. Here is how it works. We actually compute the difference between the two samples for each pair of observations, and then we work on these differences as if we were doing a one sample Student’s t-test by computing the test statistic on these differences.

In case it is not clear, here is the fourth scenario as an illustration. Suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 ($\sigma^2_D = 1$) and that we would like to test whether the mean difference between the two populations is different than 0.

before	after
0.9	0.8
-0.8	-0.9
0.1	-0.1
-0.3	0.4
0.2	0.1

The first thing to do is to compute the differences for all pairs of observations:

before	after	difference
0.9	0.8	-0.1
-0.8	-0.9	-0.1
0.1	-0.1	-0.2
-0.3	0.4	0.7
0.2	0.1	-0.1

So we have:

number of pairs: $n = 5$
mean of the difference: $\bar{D} = 0.04$
variance of the difference in the population: $\sigma^2_D = 1$
standard deviation of the difference in the population: $\sigma_D = 1$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_D = 0$ and $H_1: \mu_D \ne 0$
Test statistic: \[z_{obs} = \frac{\bar{D} - \mu_0}{\frac{\sigma_D}{\sqrt{n}}} = \frac{0.04-0}{0.447} = 0.089\] (This formula is exactly the same than for one sample Student’s t-test with a known variance, except that we work on the mean of the differences.)
Critical value: $\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96$
Conclusion: The rejection regions are thus from $-\infty$ to -1.96 and from 1.96 to $+\infty$. The test statistic is outside the rejection regions so we do not reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the mean difference between the two populations is equal to 0.

Scenario 5: Paired samples where the variance of the differences is unknown

For the fifth and final scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is unknown and that we would like to test whether a treatment is effective in increasing running capabilities (the higher the value, the better in terms of running capabilities).

before	after
9	16
8	11
1	15
3	12
2	9

The first thing to do is to compute the differences for all pairs of observations:

before	after	difference
9	16	7
8	11	3
1	15	14
3	12	9
2	9	7

So we have:

number of pairs: $n = 5$
mean of the difference: $\bar{D} = 8$
variance of the difference in the sample: $s^2_D = 16$
standard deviation of the difference in the sample: $s_D = 4$

Following the 4 steps of hypothesis testing we have:

$H_0: \mu_D = 0$ and $H_1: \mu_D > 0$ (> because we would like to test whether the treatment is effective, so whether the treatment has a positive impact on the running capabilities.)
Test statistic: \[t_{obs} = \frac{\bar{D} - \mu_0}{\frac{s_D}{\sqrt{n}}} = \frac{8-0}{1.789} = 4.472\] (This formula is exactly the same than for one sample Student’s t-test with an unknown variance, except that we work on the mean of the differences.)
Critical value: $t_{\alpha, n-1} = t_{0.05, 4} = 2.132$ (n is the number of pairs, not the number of observations!)
Conclusion: The rejection regions are thus from 2.132 to $+\infty$. The test statistic lies within the rejection region so we reject the null hypothesis $H_0$. In terms of the initial question: At the 5% significance level, we conclude that the treatment has a positive impact on the running capabilities (because the mean of the differences is greater than 0)

This concludes how to perform the different versions of the Student’s t-test for two samples by hand. In the next sections, we detail how to perform the exact same tests in R.

How to compute Student’s t-test in R?

A good practice before doing t-tests in R is to visualize the data by group thanks to a boxplot (or a density plot, or eventually both). A boxplot with the two boxes overlapping each other gives a first indication that the two samples are similar, and thus, that the null hypothesis of equal means may not be rejected. On the contrary, if the two boxes are not overlapping, it indicates that the two samples are not similar, and thus, that the populations may be different in terms of the considered variable. However, even if boxplots or density plots are great in showing a comparison between the two groups, only a sound statistical test will confirm our first impression.

After a visualization of the data by group, we replicate in R the results found by hand. We will see that for some versions of the t-test, there is no default function built in R (at least to my knowledge, do not hesitate to let me know in the comments if I’m mistaken). In these cases, a function is written to replicate the results by hand.

Note that we use the same data, the same assumptions and the same question for all 5 scenarios to facilitate the comparison between the tests performed by hand and in R.

Scenario 1: Independent samples with 2 known variances

dat1 <- data.frame(
  sample1 = c(0.9, -0.8, 0.1, -0.3, 0.2),
  sample2 = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat1

##   sample1 sample2
## 1     0.9     0.8
## 2    -0.8    -0.9
## 3     0.1    -0.1
## 4    -0.3     0.4
## 5     0.2     0.1

dat_ggplot <- data.frame(
  value = c(0.9, -0.8, 0.1, -0.3, 0.2, 0.8, -0.9, -0.1, 0.4, 0.1),
  sample = c(rep("1", 5), rep("2", 5))
)

library(ggplot2)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

boxplot(value ~ sample,
  data = dat_ggplot
)

The two boxes seem to overlap which illustrate that the two samples are quite similar, so we tend to believe that we will not be able to reject the null hypothesis that the two population means are similar. However, only a formal statistical test will confirm this belief.

Below a function to perform a t-test with known variances, with arguments accepting:

the two samples (x and y),
the two variances of the populations (V1 and V2),
the difference in means under the null hypothesis (m0, default is 0),
the significance level (alpha, default is 0.05)
and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_knownvar <- function(x, y, V1, V2, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M1 <- mean(x)
  M2 <- mean(y)
  n1 <- length(x)
  n2 <- length(y)
  sigma1 <- sqrt(V1)
  sigma2 <- sqrt(V2)
  S <- sqrt((V1 / n1) + (V2 / n2))
  statistic <- (M1 - M2 - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M1 - M2 - S * qnorm(1 - alpha / 2))
  UCL <- (M1 - M2 + S * qnorm(1 - alpha / 2))
  value <- list(mean1 = M1, mean2 = M2, m0 = m0, sigma1 = sigma1, sigma2 = sigma2, S = S, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_knownvar(dat1$sample1, dat1$sample2,
  V1 = 1, V2 = 1
)
test

## $mean1
## [1] 0.02
## 
## $mean2
## [1] 0.06
## 
## $m0
## [1] 0
## 
## $sigma1
## [1] 1
## 
## $sigma2
## [1] 1
## 
## $S
## [1] 0.6324555
## 
## $statistic
## [1] -0.06324555
## 
## $p.value
## [1] 0.949571
## 
## $LCL
## [1] -1.27959
## 
## $UCL
## [1] 1.19959
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the two sample means and the two variances of the populations (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.949571

The p-value is 0.95 so at the 5% significance level we do not reject the null hypothesis of equal means. There is no sufficient evidence in the data to reject the hypothesis that the two means in the populations are similar. This result confirms what we found by hand.

Note that a similar function exists in the {BSDA} package:⁴

library(BSDA)

z.test(dat1$sample1,
  dat1$sample2,
  alternative = "two.sided",
  mu = 0,
  sigma.x = 1,
  sigma.y = 1,
  conf.level = 0.95
)

## 
##  Two-sample z-Test
## 
## data:  dat1$sample1 and dat1$sample2
## z = -0.063246, p-value = 0.9496
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.27959  1.19959
## sample estimates:
## mean of x mean of y 
##      0.02      0.06

A note on p-value and significance level $\alpha$

For those unfamiliar with the concept of p-value, the p-value is a probability and as any probability it goes from 0 to 1. The p-value is the probability of having observations at least as extreme as the one we measured (via the samples) if the null hypothesis were true. In other words, it is the probability of having a test statistic at least as extreme as the one we computed, given that the null hypothesis is true. In some sense, it gives you an indication on how likely your null hypothesis is. It is also defined as the smallest level of significance for which the data indicate rejection of the null hypothesis.

If the observations are not so extreme—not unlikely to occur if the null hypothesis were true—we do not reject this null hypothesis because it is deemed plausible to be true. And if the observations are considered too extreme—too unlikely to happen under the null hypothesis—we reject the null hypothesis because it is deemed too implausible to be true. Note that it does not mean that we are 100% sure that it is too unlikely, it happens sometimes that the null hypothesis is rejected although it is true (see the significance level $\alpha$ later on).

In our example above, the observations are not really extreme and the difference between the two means is not extreme, so the test statistic is not extreme (since the test statistic is partially based on the difference of the means of the two samples). Having a test statistic which is not extreme is not unlikely and that is the reason why the p-value is quite high. The p-value of 0.95 actually tells us that the probability of having two samples with a difference in means of -0.04 (= 0.02 - 0.06), given that the difference in means in the populations is 0 (the null hypothesis), equals 95%. A probability of 95% is definitely considered as plausible, so we do not reject the null hypothesis of equal means in the populations.

One may then wonder, “What is too extreme for a test statistic?” Most of the time, we consider that a test statistic is too extreme to happen just by chance when the probability of having such an extreme test statistic given that the null hypothesis is true is below 5%. The threshold of 5% ($\alpha = 0.05$) that you very often see in statistic courses or textbooks is the threshold used in many fields. With a p-value under that threshold of 5%, we consider that the observations (and thus the test statistic) is too unlikely to happen just by chance if the null hypothesis were true, so the null hypothesis is rejected. With a p-value above that threshold of 5%, we consider that it is not really implausible to face the observations we have if the null hypothesis were true, and we therefore do not reject the null hypothesis.

Note that I wrote “we do not reject the null hypothesis”, and not “we accept the null hypothesis”. This is because it may be the case that the null hypothesis is in fact false, but we failed to prove it with the samples. Suppose the analogy of a suspect accused of murder and we do not know the truth. On the one hand, if we have collected enough evidence that the suspect committed the murder, he is considered guilty: we reject the null hypothesis that he is innocent. On the other hand, if we have not collected enough evidence against the suspect, he is presumed to be innocent although he may in fact have committed the crime: we failed to reject the null hypothesis of him being innocent. We are never sure that he did not commit the crime even if he is released, we just did not find sufficient evidence against the null hypothesis of the suspect being innocent. This is the reason why we do not reject the null hypothesis instead of accepting it, and why you will often read things like “there is no sufficient evidence in the data to reject the null hypothesis” or “based on the samples we fail to reject the null hypothesis”.

The significance level $\alpha$, derived from the threshold of 5% mentioned earlier, is the probability of rejecting the null hypothesis when it is in fact true. In this sense, it is an error (of 5%) that we accept to deal with, in order to be able to draw conclusions. If we would accept no error (an error of 0%), we would not be able to draw any conclusion about the population(s) since we only have access to a limited portion of the population(s) via the sample(s). As a consequence, we will never be 100% sure when interpreting the result of a hypothesis test unless we have access to the data for the entire population, but then there is no reason to do a hypothesis test anymore since we can simply compare the two populations. We usually allow this error (called Type I error) to be 5%, but in order to be a bit more certain when concluding that we reject the null hypothesis, the alpha level can also be set to 1% (or even to 0.1% in some rare cases).

To sum up what you need to remember about p-value and significance level $\alpha$:

If the p-value is smaller than the predetermined significance level $\alpha$ (usually 5%) so if p-value < 0.05 $\rightarrow H_0$ is unlikely $\rightarrow$ we reject the null hypothesis
If the p-value is greater than or equal to the predetermined significance level $\alpha$ (usually 5%) so if p-value $\ge$ 0.05 $\rightarrow H_0$ is likely $\rightarrow$ we do not reject the null hypothesis

This applies to all statistical tests without exception. Of course, the null and alternative hypotheses change depending on the test.

A rule of thumb is that, for most hypothesis tests, the alternative hypothesis is what you want to test and the null hypothesis is the status quo. Take this with extreme caution (!) because, even if it works for all versions of the Student’s t-test it does not apply to ALL statistical tests. For example, when testing for normality, you usually want to test whether your distribution follows a normal distribution. Following this piece of advice, you would write the alternative hypothesis $H_1:$ the distribution follows a normal distribution. Nonetheless, for normality tests such as the Shapiro-Wilk or Kolmogorov-Smirnov test, it is the opposite; the alternative hypothesis is $H_1:$ the distribution does not follow a normal distribution. So for every test, make sure to use the correct hypotheses, otherwise the conclusion and interpretation of your test will be wrong.

Last but not least, note that statistical significance is not equal to scientific significance. To this end, a result may be statistically significant (a p-value < $\alpha$), but of little or no interest from a scientific point of view (because the effect is so small that it is negligible and/or useless for instance).

Scenario 2: Independent samples with 2 equal but unknown variances

dat2 <- data.frame(
  sample1 = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample2 = c(0.8, -0.7, -0.1, 0.4, 0.1, NA)
)
dat2

##   sample1 sample2
## 1    1.78     0.8
## 2    1.50    -0.7
## 3    0.90    -0.1
## 4    0.60     0.4
## 5    0.80     0.1
## 6    1.90      NA

dat_ggplot <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the two boxes do not overlap which illustrates that the two samples are different from each other. From this boxplot, we can expect the test to reject the null hypothesis of equal means in the populations. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” Student’s t-test for two samples. Note that it is assumed that the variances of the two populations are equal so we need to specify it in the function with the argument var.equal = TRUE (the default is FALSE) and the alternative hypothesis is $H_1: \mu_1 - \mu_2 > 0$ so we need to add the argument alternative = "greater" as well:

test <- t.test(dat2$sample1, dat2$sample2,
  var.equal = TRUE, alternative = "greater"
)
test

## 
##  Two Sample t-test
## 
## data:  dat2$sample1 and dat2$sample2
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean of x mean of y 
##  1.246667  0.100000

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used and the two sample means (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.003866756

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis of equal means. This result confirms what we found by hand.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the mean of population 1 is larger than the mean of population 2.

A nice and easy way to report results of a Student’s t-test in R is with the report() function from the {report} package:

# install.packages("remotes")
# remotes::install_github("easystats/report") # You only need to do that once
library("report") # Load the package every time you start R

report(test)

## Effect sizes were labelled following Cohen's (1988) recommendations.
## 
## The Two Sample t-test testing the difference between dat2$sample1 and
## dat2$sample2 (mean of x = 1.25, mean of y = 0.10) suggests that the effect is
## positive, statistically significant, and large (difference = 1.15, 95% CI
## [0.53, Inf], t(9) = 3.41, p = 0.004; Cohen's d = 2.07, 95% CI [0.75, Inf])

As you can see, the function interprets the test (together with the p-value) for you.

Note that the report() function can be used for other analyses. See more tips and tricks in R if you find this one useful.

If your data is formatted in the long format (which is even better), simply use the tilde (~). For instance, imagine the exact same data presented like this:

dat2bis <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)
dat2bis

##    value sample
## 1   1.78      1
## 2   1.50      1
## 3   0.90      1
## 4   0.60      1
## 5   0.80      1
## 6   1.90      1
## 7   0.80      2
## 8  -0.70      2
## 9  -0.10      2
## 10  0.40      2
## 11  0.10      2

Here is how to perform the Student’s t-test in R with data in the long format:

test <- t.test(value ~ sample,
  data = dat2bis,
  var.equal = TRUE,
  alternative = "greater"
)
test

## 
##  Two Sample t-test
## 
## data:  value by sample
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means between group 1 and group 2 is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean in group 1 mean in group 2 
##        1.246667        0.100000

test$p.value

## [1] 0.003866756

The results are exactly the same.

Scenario 3: Independent samples with 2 unequal and unknown variances

dat3 <- data.frame(
  value = c(0.8, 0.7, 0.1, 0.4, 0.1, 1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample = c(rep("1", 5), rep("2", 6))
)
dat3

##    value sample
## 1   0.80      1
## 2   0.70      1
## 3   0.10      1
## 4   0.40      1
## 5   0.10      1
## 6   1.78      2
## 7   1.50      2
## 8   0.90      2
## 9   0.60      2
## 10  0.80      2
## 11  1.90      2

ggplot(dat3) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test as well, and it is simply the t.test() function with the var.equal = FALSE argument. FALSE is the default option for the var.equal argument so you actually do not need to specify it. This version of the test is actually the Welch Student’s test, used when the variances of the populations are unknown and unequal. To test if two population variances are equal, you can use the Levene’s test (leveneTest(dat3$value, dat3$sample) from the {car} package, or simply by comparing the dispersion of the two samples via a dotplot or a boxplot). Note that the alternative hypothesis is $H_1: \mu_1 - \mu_2 < 0$ so we need to add the argument alternative = "less" as well:

test <- t.test(value ~ sample,
  data = dat3,
  var.equal = FALSE,
  alternative = "less"
)
test

## 
##  Welch Two Sample t-test
## 
## data:  value by sample
## t = -3.0841, df = 8.2796, p-value = 0.007206
## alternative hypothesis: true difference in means between group 1 and group 2 is less than 0
## 95 percent confidence interval:
##        -Inf -0.3304098
## sample estimates:
## mean in group 1 mean in group 2 
##        0.420000        1.246667

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.00720603

The p-value is 0.007 so at the 5% significance level we reject the null hypothesis of equal means, meaning that we can conclude that the mean of population 1 is smaller than the mean of population 2. This result confirms what we found by hand.

Scenario 4: Paired samples where the variance of the differences is known

For the fourth scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 ($\sigma^2_D = 1$) and that we would like to test whether the mean difference between the two populations is different than 0.

dat4 <- data.frame(
  before = c(0.9, -0.8, 0.1, -0.3, 0.2),
  after = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat4

##   before after
## 1    0.9   0.8
## 2   -0.8  -0.9
## 3    0.1  -0.1
## 4   -0.3   0.4
## 5    0.2   0.1

dat4$difference <- dat4$after - dat4$before

ggplot(dat4) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

Since there is no function in R to perform a t-test with paired samples where the variance of the differences is known, here is one with arguments accepting:

the differences between the two samples (x),
the variance of the differences in the population (V),
the mean of the differences under the null hypothesis (m0, default is 0),
the significance level (alpha, default is 0.05)
and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_pairedknownvar <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_pairedknownvar(dat4$after - dat4$before,
  V = 1
)
test

## $mean
## [1] 0.04
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 0.08944272
## 
## $p.value
## [1] 0.9287301
## 
## $LCL
## [1] -0.8365225
## 
## $UCL
## [1] 0.9165225
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.9287301

The p-value is 0.929 so at the 5% significance level we do not reject the null hypothesis of the mean of the differences being equal to 0. There is no sufficient evidence in the data to reject the hypothesis that the mean difference between the two populations is equal to 0. This result confirms what we found by hand.

Scenario 5: Paired samples where the variance of the differences is unknown

dat5 <- data.frame(
  before = c(9, 8, 1, 3, 2),
  after = c(16, 11, 15, 12, 9)
)
dat5

##   before after
## 1      9    16
## 2      8    11
## 3      1    15
## 4      3    12
## 5      2     9

dat5$difference <- dat5$after - dat5$before

ggplot(dat5) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test, and it is simply the t.test() function with the paired = TRUE argument. This version of the test is actually the standard version of the Student’s t-test with paired samples. Note that the alternative hypothesis is $H_1: \mu_D > 0$ so we need to add the argument alternative = "greater" as well:

test <- t.test(dat5$after, dat5$before,
  alternative = "greater",
  paired = TRUE
)
test

## 
##  Paired t-test
## 
## data:  dat5$after and dat5$before
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true mean difference is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean difference 
##               8

Note that we wrote after and then before in this order. If you write before and then after, make sure to change the alternative to alternative = "less".

If your data is in the long format, use the tilde ~:

dat5 <- data.frame(
  value = c(9, 8, 1, 3, 2, 16, 11, 15, 12, 9),
  time = c(rep("before", 5), rep("after", 5))
)
dat5

##    value   time
## 1      9 before
## 2      8 before
## 3      1 before
## 4      3 before
## 5      2 before
## 6     16  after
## 7     11  after
## 8     15  after
## 9     12  after
## 10     9  after

test <- t.test(value ~ time,
  data = dat5,
  alternative = "greater",
  paired = TRUE
)
test

## 
##  Paired t-test
## 
## data:  value by time
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true mean difference is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean difference 
##               8

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value

## [1] 0.005528247

The p-value is 0.006 so at the 5% significance level we reject the null hypothesis of the mean of the differences being equal to 0, meaning that we can conclude that the treatment is effective in increasing the running capabilities (because the mean of the differences is greater than 0). This result confirms what we found by hand.

Combination of plot and statistical test

These two functions combine a boxplot—representing the distribution for each group—and the results of the statistical test displayed in the subtitle of the plot.

See examples below for scenarios 2, 3 and 5. Unfortunately, the package does not allow to run tests for scenarios 1 and 4.

Scenario 2: Independent samples with 2 equal but unknown variances

The ggbetweenstats() function is used for independent samples:

# load package
library(ggstatsplot)
library(ggplot2)

# plot with statistical results
ggbetweenstats(
  data = dat2bis,
  x = sample,
  y = value,
  plot.type = "box", # for boxplot
  type = "parametric", # for student's t-test
  var.equal = TRUE, # equal variances
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

The p-value is displayed after p = in the subtitle of the plot. Based on this plot and the p-value being lower than 5% (p-value = 0.008), we reject the null hypothesis that the two population means are equal.

Note that, the p-value is two times as large as the one obtained with the t.test() function because when we ran t.test() we specified alternative = "greater" (i.e., a one-sided test). In our plot with the ggbetweenstats() function, it is a two-sided test that is performed by default, that is, alternative = "two.sided".

Scenario 3: Independent samples with 2 unequal and unknown variances

We also have independent samples so we use the ggbetweenstats() function again, but this time the two populations variances are not assumed to be equal so we specify the argument var.equal = FALSE:

# plot with statistical results
ggbetweenstats(
  data = dat3,
  x = sample,
  y = value,
  plot.type = "box", # for boxplot
  type = "parametric", # for student's t-test
  var.equal = FALSE, # unequal variances
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

Based on the output, we reject the null hypothesis that the two population means are equal (p-value = 0.01).

Note that the p-value displayed in the subtitle of the plot is also two times larger than with the t.test() function for the same reason than above.

Scenario 5: Paired samples where the variance of the differences is unknown

In this case, the samples are paired so we use the ggwithinstats() function:

ggwithinstats(
  data = dat5,
  x = time,
  y = value,
  type = "parametric", # for student's t-test
  centrality.plotting = FALSE # remove mean
) +
  labs(caption = NULL) # remove caption

Based on the output, we reject the null hypothesis that mean of the differences between the two populations is equal to 0 (p-value = 0.01).

Again the p-value in the subtitle of the plot is twice the one obtained with the t.test() function for the same reason than above.

Assumptions

As for many statistical tests, there are some assumptions that need to be met in order to be able to interpret the results. When one or several of them are not met, although it is technically possible to perform these tests, it would be incorrect to interpret the results or trust the conclusions.

Below are the assumptions of the Student’s t-test for two samples, how to test them and which other tests exist if an assumption is not met:

Variable type: A Student’s t-test requires a mix of one quantitative dependent variable (which corresponds to the measurements to which the question relates) and one qualitative independent variable (with exactly 2 levels which will determine the groups to compare).
Independence: The data, collected from a representative and randomly selected portion of the population, should be independent between groups and within each group. The assumption of independence is most often verified based on the design of the experiment and on the good control of experimental conditions rather than via a formal test. If you are still unsure about independence based on the experiment design, ask yourself if one observation is related to another (if one observation has an impact on another) within each group or between the groups themselves. If not, it is most likely that you have independent samples. If observations between samples (forming the different groups to be compared) are dependent (for example, if two measurements have been collected on the same individuals as it is often the case in medical studies when measuring a metric (i) before and (ii) after a treatment), the paired version of the Student’s t-test, called the Student’s t-test for paired samples, should be preferred in order to take into account the dependency between the two groups to be compared.
Normality:
- With small samples (usually $n < 30$), when the two samples are independent, observations in both samples should follow a normal distribution. When using the Student’s t-test for paired samples, it is the difference between the observations of the two samples that should follow a normal distribution. The normality assumption can be tested visually thanks to a histogram and a QQ-plot, and/or formally via a normality test such as the Shapiro-Wilk or Kolmogorov-Smirnov test. If, even after a transformation (e.g., logarithmic transformation, square root, etc.), your data still do not follow a normal distribution, the Wilcoxon test (wilcox.test(variable1 ~ variable2, data = dat in R) can be applied. This non-parametric test, robust to non normal distributions, compares the medians instead of the means in order to compare the two populations.
- With large samples (usually $n \ge 30$), normality of the data is not required (this is a common misconception!). By the central limit theorem, sample means of large samples are often well-approximated by a normal distribution even if the data are not normally distributed (Stevens 2013). It is therefore not required to test the normality assumption when the number of observations in each group/sample is large.
Equality of variances: When the two samples are independent, the variances of the two groups should be equal in the populations (an assumption called homogeneity of the variances, or even sometimes referred as homoscedasticity, as opposed to heteroscedasticity if variances are different across groups). This assumption can be tested graphically (by comparing the dispersion in a boxplot or dotplot for instance), or more formally via the Levene’s test (leveneTest(variable ~ group) from the {car} package) or via a F test (var.test(variable ~ group)). If the hypothesis of equal variances is rejected, another version of the Student’s t-test can be used: the Welch test (t.test(variable ~ group, var.equal = FALSE)). Note that the Welch test does not require homogeneity of the variances, but the distributions should still follow a normal distribution in case of small sample sizes. If your distributions are not normally distributed or the variances are unequal, the Wilcoxon test should be used. This test does not require the assumptions of normality nor homoscedasticity of the variances.
Outliers: An outlier is a value or an observation that is distant from the other observations. There should be no significant outliers in the two groups, or the conclusions of your t-test may be flawed. There are several methods to detect outliers in your data but in order to deal with them, it is your choice to either:
- use the non-parametric version (i.e., the Wilcoxon test)
- transform your data (logarithmic or Box-Cox transformation, among others)
- or remove them (be careful)

Conclusion

This concludes a relatively long article. Thanks for reading.

I hope this article helped you to understand how the different versions of the Student’s t-test for two samples work and how to perform them by hand and in R. If you are interested, here is a Shiny app to perform these tests by hand easily (you just need to enter your data and select the appropriate version of the test thanks to the sidebar menu).

Moreover, I invite you to read:

this article if you would like to know how to compute the Student’s t-test but this time, for one sample,
this article if you would like to compare 2 groups under the non-normality assumption, or
this article if you would like to use an ANOVA to compare 3 or more groups.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Rowntree, Derek. 2000. Statistics Without Tears.

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Remind that inferential statistics, as opposed to descriptive statistics, is a branch of statistics defined as the science of drawing conclusions about a population from observations made on a representative sample of that population. See the difference between population and sample.↩︎
For the rest of the present article, when we write Student’s t-test, we refer to the case of 2 samples. See one sample t-test if you want to compare only one sample.↩︎
It is a least the case for parametric hypothesis tests. A parametric test means that it is based on a theoretical statistical distribution, which depends on some defined parameters. In the case of the Student’s t-test for two samples, it is based on the Student’s t distribution with a single parameter, the degrees of freedom ($df = n_1 + n_2 - 2$ where $n_1$ and $n_2$ are the two sample sizes), or the normal distribution.↩︎
Thanks gmacar for pointing it out to me.↩︎

Correlogram in R: how to highlight the most correlated variables in a dataset

Sat, 22 Feb 2020 00:00:00 +0000

Introduction

Correlation, often computed as part of descriptive statistics, is a statistical tool used to study the relationship between two variables, that is, whether and how strongly couples of variables are associated.

Correlations are measured between 2 variables at a time. Therefore, for datasets with many variables, computing correlations can become quite cumbersome and time consuming.

Correlation matrix

A solution to this problem is to compute correlations and display them in a correlation matrix, which shows correlation coefficients for all possible combinations of two variables in the dataset.

For example, below is the correlation matrix for the dataset mtcars (which, as described by the help documentation of R, comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles).¹ For this article, we include only the continuous variables.

dat <- mtcars[, c(1, 3:7)]
round(cor(dat), 2)

##        mpg  disp    hp  drat    wt  qsec
## mpg   1.00 -0.85 -0.78  0.68 -0.87  0.42
## disp -0.85  1.00  0.79 -0.71  0.89 -0.43
## hp   -0.78  0.79  1.00 -0.45  0.66 -0.71
## drat  0.68 -0.71 -0.45  1.00 -0.71  0.09
## wt   -0.87  0.89  0.66 -0.71  1.00 -0.17
## qsec  0.42 -0.43 -0.71  0.09 -0.17  1.00

Even after rounding the correlation coefficients to 2 digits, you will conceive that this correlation matrix is not easily and quickly interpretable.

If you are using R Markdown, you can use the pander() function from the {pander} package to make it slightly more readable, but still, we must admit that this table is not optimal when it comes to visualizing correlations between several variables of a dataset, especially for large datasets.

Correlogram

To tackle this issue and make it much more insightful, let’s transform the correlation matrix into a correlation plot. A correlation plot (also referred as a correlogram or corrgram in Friendly (2002)) allows to highlight the variables that are most (positively and negatively) correlated. Below an example with the same dataset presented above:

The correlogram represents the correlations for all pairs of variables. Positive correlations are displayed in blue and negative correlations in red. The intensity of the color is proportional to the correlation coefficient so the stronger the correlation (i.e., the closer to -1 or 1), the darker the boxes. The color legend on the right hand side of the correlogram shows the correlation coefficients and the corresponding colors.

As a reminder, a negative correlation implies that the two variables under consideration vary in opposite directions, that is, if one variable increases the other decreases and vice versa. A positive correlation implies that the two variables under consideration vary in the same direction, that is, if one variable increases the other increases and if one variable decreases the other decreases as well. Furthermore, the stronger the correlation, the stronger the association between the two variables.

Correlation test

Finally, a white box in the correlogram indicates that the correlation is not significantly different from 0 at the specified significance level (in this example, at $\alpha = 5$%) for the couple of variables. A correlation not significantly different from 0 means that there is no linear relationship between the two variables considered in the population (there could be another kind of association, but not linear).

To determine whether a specific correlation coefficient is significantly different from 0, a correlation test has been performed. Remind that the null and alternative hypotheses of this test are:

$H_0$: $\rho = 0$
$H_1$: $\rho \ne 0$

where $\rho$ is denotes the correlation. The correlation test is based on two factors: the number of observations and the correlation coefficient. The more observations and the stronger the correlation between 2 variables, the more likely it is to reject the null hypothesis of no correlation between these 2 variables.

In the context of our example, the correlogram above shows that the variables wt (weight) and hp (horsepower) are positively correlated, while the variables mpg (miles per gallon) and wt (weight) are negatively correlated (both correlations make sense if we think about it). Furthermore, the variables wt and qsec are not correlated (indicated by a white box). Even if the correlation coefficient is -0.17 between the 2 variables, the correlation test has shown that we cannot reject the hypothesis of no correlation in the population. This is the reason the box for these two variable is white.

Although this correlogram presents exactly the same information than the correlation matrix, the correlogram presents a visual representation of the correlation matrix, allowing to quickly scan through it to see which variables are correlated and which are not.

Code

For those interested to draw this correlogram with their own data, here is the code of the function I adapted based on the corrplot() function from the {corrplot} package (thanks again to all contributors of this package):

The main arguments in the corrplot2() function are the following:

data: name of your dataset
method: the correlation method to be computed, one of “pearson” (default), “kendall”, or “spearman”. As a rule of thumb, if your dataset contains quantitative continuous variables that have a linear relationship, you can keep the Pearson method. If you have qualitative ordinal variables or quantitative variables with a partially linear link, the Spearman method is more appropriate
sig.level: the significance level for the correlation test, default is 0.05
order: order of the variables, one of “original” (default), “AOE” (angular order of the eigenvectors), “FPC” (first principal component order), “hclust” (hierarchical clustering order), “alphabet” (alphabetical order)
diag: display the correlation coefficients on the diagonal? The default is FALSE
type: display the entire correlation matrix or simply the upper/lower part, one of “upper” (default), “lower”, “full”
tl.srt: rotation of the variable labels
(note that missing values in the dataset are automatically removed)

`{ggstatsplot}` package

An alternative to the correlogram presented above is possible with the ggcorrmat() function from the {ggstatsplot} package:

# load package
library(ggstatsplot)

# correlogram
ggstatsplot::ggcorrmat(
  data = dat,
  type = "parametric", # parametric for Pearson, nonparametric for Spearman's correlation
  colors = c("darkred", "white", "steelblue") # change default colors
)

In this correlogram, the non-significant correlations (by default at the 5% significance level with the Holm adjustment method) are shown by a cross on the correlation coefficients.

The advantage of this alternative compared to the previous one is that it is directly available within a package, so you do not need to run the code of the function first in order to draw the correlogram.

`{lares}` package

Thanks to this article, I discovered the {lares} package which has really nice features regarding plotting correlations. Another advantage of this package is that it can be used to compute correlations with numerical, logical, categorical and date variables.

See more information about the package in this article.

All possible correlations

Use the corr_cross() function if you want to compute all correlations and return the highest and significant ones in a plot:

# devtools::install_github("laresbernardo/lares")
library(lares)

corr_cross(dat, # name of dataset
  max_pvalue = 0.05, # display only significant correlations (at 5% level)
  top = 10 # display top 10 couples of variables (by correlation coefficient)
)

Negative correlations are represented in red and positive correlations in blue.

Correlation of one variable against all others

Use the corr_var() function if you want to focus on the correlation of one variable against all others, and return the highest ones in a plot:

corr_var(dat, # name of dataset
  mpg, # name of variable to focus on
  top = 5 # display top 5 correlations
)

Conclusion

Thanks for reading.

I hope this article will help you to visualize correlations between variables in a dataset and to make correlation matrices more insightful and more appealing.

If you want to learn more about this topic, see:

how to compute correlation coefficients and perform correlation tests in R, or
how to compute correlation coefficients by hand.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Friendly, Michael. 2002. “Corrgrams: Exploratory Displays for Correlation Matrices.” The American Statistician 56 (4): 316–24.

The dataset mtcars is preloaded in R by default, so there is no need to import it into R. Check the article “How to import an Excel file in R” if you need help in importing your own dataset.↩︎

The complete guide to clustering analysis: k-means and hierarchical clustering by hand and in R

Thu, 13 Feb 2020 00:00:00 +0000

Note that this article is inspired from the lecture notes of Prof. Johan Segers and my personal notes as teaching assistant for his course entitled “Multivariate statistical analysis” given at UCLouvain.

What is clustering analysis?

Clustering analysis is a form of exploratory data analysis in which observations are divided into different groups that share common characteristics.

The purpose of cluster analysis (also known as classification) is to construct groups (or classes or clusters) while ensuring the following property: within a group the observations must be as similar as possible (intracluster similarity), while observations belonging to different groups must be as different as possible (intercluster similarity).

An illustration of intercluster and intracluster distance. Source: dinhanhthi.com

The two most common types of classification are:

k-means clustering
Hierarchical clustering

The first is generally used when the number of classes is fixed in advance, while the second is generally used for an unknown number of classes and helps to determine this optimal number. For this reason, k-means is considered as a supervised technique, while hierarchical clustering is considered as an unsupervised technique because the estimation of the number of clusters is part of the algorithm. See more clustering methods in this article.

Both methods are illustrated below through applications by hand and in R. Note that for hierarchical clustering, only the ascending classification is presented in this article.

Clustering algorithms use the distance in order to separate observations into different groups. Therefore, before diving into the presentation of the two classification methods, a reminder exercise on how to compute distances between points is presented.

Application 1: Computing distances

Let a data set containing the points $\boldsymbol{a} = (0, 0)'$, $\boldsymbol{b} = (1, 0)'$ and $\boldsymbol{c} = (5, 5)'$. Compute the matrix of Euclidean distances between the points by hand and in R.

Solution

The points are as follows:

# We create the points in R
a <- c(0, 0)
b <- c(1, 0)
c <- c(5, 5)

X <- rbind(a, b, c) # a, b and c are combined per row
colnames(X) <- c("x", "y") # rename columns

X # display the points

##   x y
## a 0 0
## b 1 0
## c 5 5

By the Pythagorean theorem, we will remember that the distance between 2 points $(x_a, y_a)$ and $(x_b, y_b)$ in $\mathbb{R}^2$ is given by $\sqrt{(x_a - x_b)^2 + (y_a - y_b)^2}$. So for instance, for the distance between the points $\boldsymbol{b} = (1, 0)'$ and $\boldsymbol{c} = (5, 5)'$ presented in the statement above, we have:

\[\begin{equation} \begin{split} d(b, c) &= \sqrt{(x_b - x_c)^2 + (y_b - y_c)^2} \\ &= \sqrt{(1-5)^2 + (0-5)^2}\\ &= 6.403124 \end{split} \end{equation}\]

We can proceed similarly for all pairs of points to find the distance matrix by hand. In R, the dist() function allows you to find the distance of points in a matrix or dataframe in a very simple way:

# The distance is found using the dist() function:
distance <- dist(X, method = "euclidean")
distance # display the distance matrix

##          a        b
## b 1.000000         
## c 7.071068 6.403124

Note that the argument method = "euclidean" is not mandatory because the Euclidean method is the default one.

The distance matrix resulting from the dist() function gives the distance between the different points. The Euclidean distance between the points $\boldsymbol{b}$ and $\boldsymbol{c}$ is 6.403124, which corresponds to what we found above via the Pythagorean formula.

Note: If two variables do not have the same units, one may have more weight in the calculation of the Euclidean distance than the other. In that case, it is preferable to scale the data. Scaling data allows to obtain variables independent of their unit, and this can be done with the scale() function.

Now that the distance has been presented, let’s see how to perform clustering analysis with the k-means algorithm.

k-means clustering

The first form of classification is the method called k-means clustering or the mobile center algorithm. As a reminder, this method aims at partitioning $n$ observations into $k$ clusters in which each observation belongs to the cluster with the closest average, serving as a prototype of the cluster.

We do not go too much into details about the mathematics. Instead, we focus on how to apply it in R and by hand.

Application 2: k-means clustering

Data

For this exercise, the Eurojobs.csv database available here is used.

This database contains the percentage of the population employed in different industries in 26 European countries in 1979. It contains 10 variables:

Country - the name of the country (identifier)
Agr - % of workforce employed in agriculture
Min - % in mining
Man - % in manufacturing
PS - % in power supplies industries
Con - % in construction
SI - % in service industries
Fin - % in finance
SPS - % in social and personal services
TC - % in transportation and communications

We first import the dataset. See how to import data into R if you need a reminder.

# Import data
Eurojobs <- read.csv(
  file = "https://statsandr.com/blog/data/Eurojobs.csv",
  sep = ",",
  dec = ".",
  header = TRUE
)

head(Eurojobs) # head() is used to display only the first 6 observations

##      Country  Agr Min  Man  PS  Con   SI Fin  SPS  TC
## 1    Belgium  3.3 0.9 27.6 0.9  8.2 19.1 6.2 26.6 7.2
## 2    Denmark  9.2 0.1 21.8 0.6  8.3 14.6 6.5 32.2 7.1
## 3     France 10.8 0.8 27.5 0.9  8.9 16.8 6.0 22.6 5.7
## 4 W. Germany  6.7 1.3 35.8 0.9  7.3 14.4 5.0 22.3 6.1
## 5    Ireland 23.2 1.0 20.7 1.3  7.5 16.8 2.8 20.8 6.1
## 6      Italy 15.9 0.6 27.6 0.5 10.0 18.1 1.6 20.1 5.7

Note that there is a numbering before the first variable Country. For more clarity, we will replace this numbering by the country. To do this, we add the argument row.names = 1 in the import function read.csv() to specify that the first column corresponds to the row names:

Eurojobs <- read.csv(
  file = "https://statsandr.com/blog/data/Eurojobs.csv",
  sep = ",",
  dec = ".",
  header = TRUE,
  row.names = 1
)

Eurojobs # displays dataset

##                 Agr Min  Man  PS  Con   SI  Fin  SPS  TC
## Belgium         3.3 0.9 27.6 0.9  8.2 19.1  6.2 26.6 7.2
## Denmark         9.2 0.1 21.8 0.6  8.3 14.6  6.5 32.2 7.1
## France         10.8 0.8 27.5 0.9  8.9 16.8  6.0 22.6 5.7
## W. Germany      6.7 1.3 35.8 0.9  7.3 14.4  5.0 22.3 6.1
## Ireland        23.2 1.0 20.7 1.3  7.5 16.8  2.8 20.8 6.1
## Italy          15.9 0.6 27.6 0.5 10.0 18.1  1.6 20.1 5.7
## Luxembourg      7.7 3.1 30.8 0.8  9.2 18.5  4.6 19.2 6.2
## Netherlands     6.3 0.1 22.5 1.0  9.9 18.0  6.8 28.5 6.8
## United Kingdom  2.7 1.4 30.2 1.4  6.9 16.9  5.7 28.3 6.4
## Austria        12.7 1.1 30.2 1.4  9.0 16.8  4.9 16.8 7.0
## Finland        13.0 0.4 25.9 1.3  7.4 14.7  5.5 24.3 7.6
## Greece         41.4 0.6 17.6 0.6  8.1 11.5  2.4 11.0 6.7
## Norway          9.0 0.5 22.4 0.8  8.6 16.9  4.7 27.6 9.4
## Portugal       27.8 0.3 24.5 0.6  8.4 13.3  2.7 16.7 5.7
## Spain          22.9 0.8 28.5 0.7 11.5  9.7  8.5 11.8 5.5
## Sweden          6.1 0.4 25.9 0.8  7.2 14.4  6.0 32.4 6.8
## Switzerland     7.7 0.2 37.8 0.8  9.5 17.5  5.3 15.4 5.7
## Turkey         66.8 0.7  7.9 0.1  2.8  5.2  1.1 11.9 3.2
## Bulgaria       23.6 1.9 32.3 0.6  7.9  8.0  0.7 18.2 6.7
## Czechoslovakia 16.5 2.9 35.5 1.2  8.7  9.2  0.9 17.9 7.0
## E. Germany      4.2 2.9 41.2 1.3  7.6 11.2  1.2 22.1 8.4
## Hungary        21.7 3.1 29.6 1.9  8.2  9.4  0.9 17.2 8.0
## Poland         31.1 2.5 25.7 0.9  8.4  7.5  0.9 16.1 6.9
## Rumania        34.7 2.1 30.1 0.6  8.7  5.9  1.3 11.7 5.0
## USSR           23.7 1.4 25.8 0.6  9.2  6.1  0.5 23.6 9.3
## Yugoslavia     48.7 1.5 16.8 1.1  4.9  6.4 11.3  5.3 4.0

dim(Eurojobs) # displays the number of rows and columns

## [1] 26  9

We now have a “clean” dataset of 26 observations and 9 quantitative continuous variables on which we can base the classification. Note that in this case it is not necessary to standardize the data because they are all expressed in the same unit (in percentage). If this was not the case, we would have had to standardize the data via the scale() function (do not forget it otherwise your results may be completely different!).

The so-called k-means clustering is done via the kmeans() function, with the argument centers that corresponds to the number of desired clusters. In the following we apply the classification with 2 classes and then 3 classes as examples.

`kmeans()` with 2 groups

model <- kmeans(Eurojobs, centers = 2)

# displays the class determined by
# the model for all observations:
model$cluster

##        Belgium        Denmark         France     W. Germany        Ireland 
##              1              1              1              1              2 
##          Italy     Luxembourg    Netherlands United Kingdom        Austria 
##              1              1              1              1              1 
##        Finland         Greece         Norway       Portugal          Spain 
##              1              2              1              2              2 
##         Sweden    Switzerland         Turkey       Bulgaria Czechoslovakia 
##              1              1              2              2              1 
##     E. Germany        Hungary         Poland        Rumania           USSR 
##              1              2              2              2              2 
##     Yugoslavia 
##              2

Note that the argument centers = 2 is used to set the number of clusters, determined in advance. In this exercise the number of clusters has been determined arbitrarily.

This number of clusters should be determined according to the context and goal of your analysis (so according to your expectations or hypotheses), or based on methods explained in this section. The output model$cluster specifies the group (i.e., 1 or 2) to which each country belongs to.

The cluster for each observation can be stored directly in the dataset as a column:

Eurojobs_cluster <- data.frame(Eurojobs,
  cluster = as.factor(model$cluster)
)

head(Eurojobs_cluster)

##             Agr Min  Man  PS  Con   SI Fin  SPS  TC cluster
## Belgium     3.3 0.9 27.6 0.9  8.2 19.1 6.2 26.6 7.2       1
## Denmark     9.2 0.1 21.8 0.6  8.3 14.6 6.5 32.2 7.1       1
## France     10.8 0.8 27.5 0.9  8.9 16.8 6.0 22.6 5.7       1
## W. Germany  6.7 1.3 35.8 0.9  7.3 14.4 5.0 22.3 6.1       1
## Ireland    23.2 1.0 20.7 1.3  7.5 16.8 2.8 20.8 6.1       2
## Italy      15.9 0.6 27.6 0.5 10.0 18.1 1.6 20.1 5.7       1

Quality of a k-means partition

The quality of a k-means partition is found by calculating the percentage of the TSS “explained” by the partition using the following formula:

\[\begin{equation} \dfrac{\operatorname{BSS}}{\operatorname{TSS}} \times 100\% \end{equation}\]

where BSS and TSS stand for Between Sum of Squares and Total Sum of Squares, respectively. The higher the percentage, the better the score (and thus the quality) because it means that BSS is large and/or WSS is small.

Here is how you can check the quality of the partition in R:

# BSS and TSS are extracted from the model and stored
(BSS <- model$betweenss)

## [1] 4823.535

(TSS <- model$totss)

## [1] 9299.59

# We calculate the quality of the partition
BSS / TSS * 100

## [1] 51.86826

The quality of the partition is 51.87%. This value has no real interpretation in absolute terms except that a higher quality means a higher explained percentage. However, it is more insightful when it is compared to the quality of other partitions (with the same number of clusters! see why at the end of this section) in order to determine the best partition among the ones considered.

`nstart` for several initial centers and better stability

Note that k-means is a non-deterministic algorithm so running it multiple times may result in different classification. This is the case because the k-means algorithm uses a random set of initial points to arrive at the final classification. Due to the fact that the initial centers are randomly chosen, the same command kmeans(Eurojobs, centers = 2) may give different results every time it is run, and thus slight differences in the quality of the partitions. The nstart argument in the kmeans() function allows to run the algorithm several times with different initial centers, in order to obtain a potentially better partition:

model2 <- kmeans(Eurojobs, centers = 2, nstart = 10)
100 * model2$betweenss / model2$totss

## [1] 54.2503

Depending on the initial random choices, this new partition will be better or not compared to the first one. In our example, the partition is better as the quality increased to 54.25%.

One of the main limitation often cited regarding k-means is the stability of the results. As the initial centers are randomly chosen, running the same command may yield different results. Adding the nstart argument in the kmeans() function limits this issue as it will generate several different initializations and take the most optimal one, leading to a better stability of the classification.

`kmeans()` with 3 groups

We now perform the k-means classification with 3 clusters and compute its quality:

model3 <- kmeans(Eurojobs, centers = 3)

BSS3 <- model3$betweenss
TSS3 <- model3$totss
BSS3 / TSS3 * 100

## [1] 74.59455

It can be seen that the classification into three groups allows for a higher explained percentage and a higher quality.

This will always be the case: with more classes, the partition will be finer, and the BSS contribution will be higher. On the other hand, the “model” will be more complex, requiring more classes. In the extreme case where k = n (each observation is a singleton class), we have BSS = TSS, but the partition has lost all interest.

This is the reason we compare partitions via their quality only for partitions that have the same number of clusters.

An alternative method to perform a k-means is to use the cluster_analysis() function from the {parameters} package:

library(parameters)

res_kmeans <- cluster_analysis(Eurojobs,
  n = 3,
  method = "kmeans"
)

predict(res_kmeans) # get clusters

##  [1] 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 1 3 3 3 3 3 3 3 1

An advantage of this method is that it is possible to visualize the centers (i.e., the average of each variable for each cluster):

plot(summary(res_kmeans))

Optimal number of clusters

In order to find the optimal number of clusters for a k-means, it is recommended to choose it based on:

the context of the problem at hand, for instance if you know that there is a specific number of groups in your data (you have strong expectations or hypotheses, this is option is however subjective), or
the following four approaches:
1. Elbow method (which uses the within cluster sums of squares)
2. Average silhouette method
3. Gap statistic method
4. Consensus-based algorithm

We show the R code for these 4 methods below, more theoretical information can be found here.

Elbow method

The Elbow method looks at the total within-cluster sum of square (WSS) as a function of the number of clusters.

# load required packages
library(factoextra)
library(NbClust)

# Elbow method
fviz_nbclust(Eurojobs, kmeans, method = "wss") +
  geom_vline(xintercept = 4, linetype = 2) + # add line for better visualisation
  labs(subtitle = "Elbow method") # add subtitle

The location of a knee in the plot is usually considered as an indicator of the appropriate number of clusters because it means that adding another cluster does not improve much better the partition. This method seems to suggest 4 clusters.

The Elbow method is sometimes ambiguous and an alternative is the average silhouette method.

Silhouette method

The Silhouette method measures the quality of a clustering and determines how well each point lies within its cluster.

# Silhouette method
fviz_nbclust(Eurojobs, kmeans, method = "silhouette") +
  labs(subtitle = "Silhouette method")

The Silhouette method suggests 2 clusters.

Gap statistic method

# Gap statistic
set.seed(42)
fviz_nbclust(Eurojobs, kmeans,
  nstart = 25,
  method = "gap_stat",
  nboot = 500 # reduce it for lower computation time (but less precise results)
) +
  labs(subtitle = "Gap statistic method")

The optimal number of clusters is the one that maximizes the gap statistic. This method suggests only 1 cluster (which is therefore a useless clustering).

As you can see these three methods do not necessarily lead to the same result. Here, the 3 approaches suggest a different number of clusters.

Consensus-based algorithm

Because no method is clearly better, a fourth alternative is to run many methods and take the number of clusters that is the most agreed upon (i.e., find the consensus).

This can easily be done with the n_clusters() function from the {parameters} package:

library(parameters)

n_clust <- n_clusters(Eurojobs,
  package = c("easystats", "NbClust", "mclust"),
  standardize = FALSE
)
n_clust

## # Method Agreement Procedure:
## 
## The choice of 3 clusters is supported by 15 (51.72%) methods out of 29 (kl, Ch, Hartigan, CCC, Scott, Marriot, trcovw, Tracew, Rubin, Beale, Ratkowsky, Ball, PtBiserial, Dunn, SDindex).

The optimal number of clusters to retain can also be visualized:

plot(n_clust)

Based on all indices, most methods suggest to retain 3 clusters.

Visualizations

To confirm that your number of classes is indeed optimal, there is a way to evaluate the quality of your clustering via the silhouette plot (which shows the silhouette coefficient on the y axis).

We draw the silhouette plot for 2 clusters, as suggested by the average silhouette method:

library(cluster)

set.seed(42)
km_res <- kmeans(Eurojobs, centers = 2, nstart = 20)

sil <- silhouette(km_res$cluster, dist(Eurojobs))
fviz_silhouette(sil)

##   cluster size ave.sil.width
## 1       1    5          0.33
## 2       2   21          0.54

As a reminder, the interpretation of the silhouette coefficient is as follows:

$> 0$ means that the observation is well grouped. The closer the coefficient is to 1, the better the observation is grouped.
$< 0$ means that the observation has been placed in the wrong cluster.
$= 0$ means that the observation is between two clusters.

The silhouette plot above and the average silhouette coefficient help to determine whether your clustering is good or not. If a large majority of the silhouette coefficients are positive, it indicates that the observations are placed in the correct group. This silhouette plot can therefore be used in the choice of the optimal number of classes.

It is also possible to plot clusters by using the fviz_cluster() function. Note that a principal component analysis is performed to represent the variables in a 2 dimensions plane.

library(factoextra)

fviz_cluster(km_res, Eurojobs, ellipse.type = "norm")

Now that the k-means clustering has been detailed in R, see how to do the algorithm by hand in the following sections.

Manual application and verification in R

Perform by hand the k-means algorithm for the points shown in the graph below, with k = 2 and with the points i = 5 and i = 6 as initial centers. Compute the quality of the partition you just found and then check your answers in R.

Assume that the variables have the same units so there is no need to scale the data.

Solution by hand

Step 1. Here are the coordinates of the 6 points:

point	x	y
1	7	3
2	4	5
3	2	4
4	0	1
5	9	7
6	6	8

And the initial centers:

Group 1: point 5 with center (9, 7)
Group 2: point 6 with center (6, 8)

Step 2. Compute the distance matrix point by point with the Pythagorean theorem. Remind that the distance between point a and point b is found with:

\[\sqrt{(x_a - x_b)^2 + (y_a - y_b)^2}\]

We apply this theorem to each pair of points, to finally have the following distance matrix (rounded to two decimals):

round(dist(X), 2)

##       1     2     3     4     5
## 2  3.61                        
## 3  5.10  2.24                  
## 4  7.28  5.66  3.61            
## 5  4.47  5.39  7.62 10.82      
## 6  5.10  3.61  5.66  9.22  3.16

Step 3. Based on the distance matrix computed in step 2, we can put each point to its closest group and compute the coordinates of the center.

We first put each point in its closest group:

point 1 is closer to point 5 than to point 6 because the distance between points 1 and 5 is 4.47 while the distance between points 1 and 6 is 5.10
point 2 is closer to point 6 than to point 5 because the distance between points 2 and 5 is 5.39 while the distance between points 2 and 6 is 3.61
point 3 is closer to point 6 than to point 5 because the distance between points 3 and 5 is 7.62 while the distance between points 3 and 6 is 5.66
point 4 is closer to point 6 than to point 5 because the distance between points 4 and 5 is 10.82 while the distance between points 4 and 6 is 9.22

Note that computing the distances between each point and the points 5 and 6 is sufficient. There is no need to compute the distance between the points 1 and 2 for example, as we compare each point to the initial centers (which are points 5 and 6).

We then compute the coordinates of the centers of the two groups by taking the mean of the coordinates x and y:

Group 1 includes the points 5 and 1 with (8, 5) as center ($8 = \frac{9+7}{2}$ and $5 = \frac{7+3}{2}$)
Group 2 includes the points 6, 2, 3 and 4 with (3, 4.5) as center ($3 = \frac{6+4+2+0}{4}$ and $4.5 = \frac{8+5+4+1}{4}$)

We thus have:

	points	center
cluster 1	5 & 1	(8, 5)
cluster 2	6, 2, 3 & 4	(3, 4.5)

Step 4. We make sure that the allocation is optimal by checking that each point is in the nearest cluster. The distance between a point and the center of a cluster is again computed thanks to the Pythagorean theorem. Thus, we have:

points	Distance to cluster 1	Distance to cluster 2
1	2.24	4.27
2	4	1.12
3	6.08	1.12
4	8.94	4.61
5	2.24	6.5
6	3.61	4.61

The minimum distance between the points and the two clusters is colored in green.

We check that each point is in the correct group (i.e., the closest cluster). According to the distance in the table above, point 6 seems to be closer to the cluster 1 than to the cluster 2. Therefore, the allocation is not optimal and point 6 should be reallocated to cluster 1.

Step 5. We compute again the centers of the clusters after this reallocation. The centers are found by taking the mean of the coordinates x and y of the points belonging to the cluster. We thus have:

	points	center
cluster 1	1, 5 & 6	(7.33, 6)
cluster 2	2, 3 & 4	(2, 3.33)

where, for instance, 3.33 is simply $\frac{5+4+1}{3}$.

Step 6. Repeat step 4 until the allocation is optimal. If the allocation is optimal, the algorithm stops. In our example we have:

points	Distance to cluster 1	Distance to cluster 2
1	3.02	5.01
2	3.48	2.61
3	5.69	0.67
4	8.87	3.07
5	1.95	7.9
6	2.4	6.15

All points are correctly allocated to its nearest cluster, so the allocation is optimal and the algorithm stops.

Step 7. State the final partition and the centers. In our example:

	points	center
cluster 1	1, 5 & 6	(7.33, 6)
cluster 2	2, 3 & 4	(2, 3.33)

Now that we have the clusters and the final centers, we compute the quality of the partition we just found. Remember that we need to compute the BSS and TSS to find the quality. Below the steps to compute the quality of this partition by k-means, based on this summary table:

cluster 1	cluster 2
point	x	y	point	x	y
1	7	3	2	4	5
5	9	7	3	2	4
6	6	8	4	0	1
mean	7.33	6		2	3.33

Step 1. Compute the overall mean of the x and y coordinates:

\[\overline{\overline{x}} = \frac{7+4+2+0+9+6}{12} \\ \frac{+3+5+4+1+7+8}{12} \\ = 4.67\]

Step 2. Compute TSS and WSS:

\[TSS = (7-4.67)^2 + (4-4.67)^2 + (2-4.67)^2 \\ + (0-4.67)^2 + (9-4.67)^2 + (6-4.67)^2 \\ + (3-4.67)^2 + (5-4.67)^2 + (4-4.67)^2 \\ + (1-4.67)^2 + (7-4.67)^2 + (8-4.67)^2 \\ = 88.67\]

Regarding WSS, it is splitted between cluster 1 and cluster 2. For cluster 1:

\[WSS[1] = (7-7.33)^2 + (9 - 7.33)^2 \\ + (6 - 7.33)^2 + (3-6)^2 \\ + (7-6)^2 + (8-6)^2 \\ = 18.67\]

For cluster 2:

\[WSS[2] = (4-2)^2 + (2-2)^2 + (0-2)^2 \\ + (5-3.33)^2 + (4-3.33)^2 + (1-3.33)^2 \\ = 16.67\]

And the total WSS is

\[\begin{align} WSS &= WSS[1] + WSS[2] \\ &= 18.67 + 16.67 \\ &= 35.34 \end{align}\]

To find the BSS:

\[\begin{align} BSS &= TSS - WSS \\ &= 88.67 - 35.34 \\ &= 53.33 \end{align}\]

Finally, the quality of the partition is:

\[Quality = \frac{BSS}{TSS} = \frac{53.33}{88.67} = 0.6014\]

So the quality of the partition is 60.14%.

We are now going to verify all these solutions (the partition, the final centers and the quality) in R.

Solution in R

As you can imagine, the solution in R us much shorter and requires much less computation on the user side. We first need to enter the data as a matrix or dataframe:

X <- matrix(c(7, 3, 4, 5, 2, 4, 0, 1, 9, 7, 6, 8),
  nrow = 6, byrow = TRUE
)
X # display the coordinates of the points

##      [,1] [,2]
## [1,]    7    3
## [2,]    4    5
## [3,]    2    4
## [4,]    0    1
## [5,]    9    7
## [6,]    6    8

We now perform the k-means via the kmeans() function with the point 5 and 6 as initial centers:

# take rows 5 and 6 of the X matrix as initial centers
res.k <- kmeans(X,
  centers = X[c(5, 6), ],
  algorithm = "Lloyd"
)

Unlike in the previous application with the dataset Eurojobs.csv where the initial centers are randomly chosen by R, in this second application we want to specify which points are going to be the two initial centers. For this, we need to set centers = X[c(5,6), ] to indicate that that there are 2 centers, and that they are going to be the points 5 and 6 (see a reminder on how to subset a dataframe if needed).

The reason for adding the argument algorithm = "Lloyd" can be found in the usage of the R function kmeans(). In fact, there are several variants of the k-means algorithm. The default choice is the Hartigan and Wong (1979) version, which is more sophisticated than the basic version detailed in the solution by hand. By using the original version of Lloyd (1982), we find the same solution in R and by hand. For more information, you can consult the documentation of the kmeans() function (via ?kmeans or help(kmeans)) and read the articles mentioned.

The solution in R is then found by extracting

the partition with $cluster:

res.k$cluster

## [1] 1 2 2 2 1 1

Points 1, 5 and 6 belong to cluster 1, points 2, 3 and 4 belong to cluster 2.

the coordinates of the final centers with $centers:

# We extract the coordinates of the 2 final centers, rounded to 2 decimals
round(res.k$centers, digits = 2)

##   [,1] [,2]
## 1 7.33 6.00
## 2 2.00 3.33

and then the quality of the partition by dividing the BSS to the TSS:

res.k$betweenss / res.k$totss

## [1] 0.6015038

The 3 results are equal to what we found by hand (except the quality which is slightly different due to rounding).

Hierarchical clustering

Remind that the difference with the partition by k-means is that for hierarchical clustering, the number of classes is not specified in advance. Hierarchical clustering will help to determine the optimal number of clusters.

Before applying hierarchical clustering by hand and in R, let’s see how the ascending hierarchical clustering works step by step:

It starts by putting every point in its own cluster, so each cluster is a singleton
It then merges the 2 points that are closest to each other based on the distances from the distance matrix. The consequence is that there is one less cluster
It then recalculates the distances between the new and old clusters and save them in a new distance matrix which will be used in the next step
Finally, steps 1 and 2 are repeated until all clusters are merged into one single cluster including all points.

There are 5 main methods to measure the distance between clusters, referred as linkage methods:

Single linkage: computes the minimum distance between clusters before merging them.
Complete linkage: computes the maximum distance between clusters before merging them.
Average linkage: computes the average distance between clusters before merging them.
Centroid linkage: calculates centroids for both clusters, then computes the distance between the two before merging them.
Ward’s (minimum variance) criterion: minimizes the total within-cluster variance and find the pair of clusters that leads to minimum increase in total within-cluster variance after merging.

In the following sections, only the three first linkage methods are presented (first by hand and then the results are verified in R).

Application 3: hierarchical clustering

Data

Using the data from the graph and the table below, we perform by hand the 3 algorithms (single, complete and average linkage) and we draw the dendrograms. Then we check our answers in R.

##      V1    V2
## 1  2.03  0.06
## 2 -0.64 -0.10
## 3 -0.42 -0.53
## 4 -0.36  0.07
## 5  1.14  0.37

We assume that the variables have the same units so there is no need to scale the data.

Solution by hand

Step 1. For all 3 algorithms, we first need to compute the distance matrix between the 5 points thanks to the Pythagorean theorem. Remind that the distance between point a and point b is found with:

\[\sqrt{(x_a - x_b)^2 + (y_a - y_b)^2}\]

We apply this theorem to each pair of points, to finally have the following distance matrix (rounded to three decimals):

##       1     2     3     4
## 2 2.675                  
## 3 2.520 0.483            
## 4 2.390 0.328 0.603      
## 5 0.942 1.841 1.801 1.530

Single linkage

Step 2. From the distance matrix computed in step 1, we see that the smallest distance = 0.328 between points 2 and 4. 0.328 corresponds to the first height (more on this later when drawing the dendrogram). Since points 2 and 4 are the closest to each other, these 2 points are put together to form a single group. The groups are thus: 1, 2 & 4, 3 and 5. The new distances between the group 2 & 4 and all other points are now:

	1	2 & 4	3	5
1	0
2 & 4	2.390	0
3	2.520	0.483	0
5	0.942	1.530	1.801	0

To construct this new distance matrix, proceed point by point:

the distance between points 1 and 3 has not changed, so the distance is unchanged compared to the initial distance matrix (found in step 1), which was 2.520
same goes for the distance between points 1 and 5 and points 3 and 5; the distances are the same than in the initial distance matrix since the points have not changed
the distance between points 1 and 2 & 4 has changed since points 2 & 4 are now together
since we are applying the single linkage criterion, the new distance between points 1 and 2 & 4 corresponds to the minimum distance between the distance between points 1 and 2 and the distance between points 1 and 4
the initial distance between points 1 and 2 is 2.675 and the initial distance between points 1 and 4 is 2.390
therefore, the minimum distance between these two distances is 2.390
2.390 is thus the new distance between points 1 and 2 & 4
we apply the same process for points 3 and 2 & 4: the initial distance between points 3 and 2 is 0.483 and the initial distance between points 3 and 4 is 0.603. The minimum distance between these 2 distances is 0.483 so the new distance between points 3 and 2 & 4 is 0.483
follow the same process for all other points

Step 3. Based on the distance matrix in step 2, the smallest distance is 0.483 between points 3 and 2 & 4 (the second height for the dendrogram). Since points 3 and 2 & 4 are the closest to each other, they are combined to form a new group, the group 2 & 3 & 4. The groups are thus: 1, 2 & 3 & 4 and 5. We construct the new distance matrix based on the same process detailed in step 2:

	1	2 & 3 & 4	5
1	0
2 & 3 & 4	2.390	0
5	0.942	1.530	0

points 1 and 5 have not change, so the distance between these two points are the same than in previous step
from step 2 we see that the distance between points 1 and 2 & 4 is 2.390 and the distance between points 1 and 3 is 2.520
since we apply the single linkage criterion, we take the minimum distance, which is 2.390
the distance between points 1 and 2 & 3 & 4 is thus 2.390
same process for points 5 and 2 & 3 & 4

Step 4. Based on the distance matrix in step 3, the smallest distance is 0.942 between points 1 and 5 (the third height in the dendrogram). Since points 1 and 5 are the closest to each other, they are combined to form a new group, the group 1 & 5. The groups are thus: 1 & 5 and 2 & 3 & 4. We construct the new distance matrix based on the same process detailed in steps 2 and 3:

	1 & 5	2 & 3 & 4
1 & 5	0
2 & 3 & 4	1.530	0

the only distance left to compute is the distance between points 1 & 5 and 2 & 3 & 4
from the previous step we see that the distance between points 1 and 2 & 3 & 4 is 2.390 and the distance between points 5 and 2 & 3 & 4 is 1.530
since we apply the single linkage criterion, we take the minimum distance, which is 1.530
the distance between points 1 & 5 and 2 & 3 & 4 is thus 1.530

Step 5. The final combination of points is the combination of points 1 & 5 and 2 & 3 & 4, with a final height of 1.530. Heights are used to draw the dendrogram in the sixth and final step.

Step 6. Draw the dendrogram thanks to the combination of points and heights found above. Remember that:

the first combination of points was between points 2 and 4, with a height of 0.328
the second combination was between points 3 and 2 & 4 with a height of 0.483
the third combination was between points 1 and 5 with a height of 0.942
the final combination was between points 1 & 5 and 2 & 3 & 4 with a height of 1.530
this is exactly what is illustrated in the following dendrogram:

In hierarchical clustering, dendrograms are used to show the sequence of combinations of the clusters. The distances of merge between clusters, called heights, are illustrated on the y-axis.

Complete linkage

Complete linkage is quite similar to single linkage, except that instead of taking the smallest distance when computing the new distance between points that have been grouped, the maximum distance is taken.

The steps to perform the hierarchical clustering with the complete linkage (maximum) are detailed below.

Step 1. Step 1 is exactly the same than for single linkage, that is, we compute the distance matrix of the 5 points thanks to the Pythagorean theorem. This gives us the following distance matrix:

##       1     2     3     4
## 2 2.675                  
## 3 2.520 0.483            
## 4 2.390 0.328 0.603      
## 5 0.942 1.841 1.801 1.530

Step 2. From the distance matrix computed in step 1, we see that the smallest distance = 0.328 between points 2 and 4. It is important to note that even if we apply the complete linkage, in the distance matrix the points are brought together based on the smallest distance. This is the case for all 3 algorithms. The difference between the 3 algorithms lies in how to compute the new distances between the new combination of points (the single linkage takes the minimum between the distances, the complete linkage takes the maximum distance and the average linkage takes the average distance). 0.328 corresponds to the first height (which will be used when drawing the dendrogram). Since points 2 and 4 are the closest to each other, these 2 points are put together to form a single group. The groups are thus: 1, 2 & 4, 3 and 5. The new distances between the group 2 & 4 and all other points are now:

	1	2 & 4	3	5
1	0
2 & 4	2.675	0
3	2.520	0.603	0
5	0.942	1.841	1.801	0

To construct this new distance matrix, proceed point by point as we did for single linkage:

the distance between points 1 and 3 has not changed, so the distance is unchanged compared to the initial distance matrix (found in step 1), which was 2.520
same goes for the distance between points 1 and 5 and points 3 and 5; the distances are the same than in the initial distance matrix since the points have not changed
the distance between points 1 and 2 & 4 has changed since points 2 & 4 are now together
since we are applying the complete linkage criterion, the new distance between points 1 and 2 & 4 corresponds to the maximum distance between the distance between points 1 and 2 and the distance between points 1 and 4
the initial distance between points 1 and 2 is 2.675 and the initial distance between points 1 and 4 is 2.390
therefore, the maximum distance between these two distances is 2.675
2.675 is thus the new distance between points 1 and 2 & 4
we apply the same process for points 3 and 2 & 4: the initial distance between points 3 and 2 is 0.483 and the initial distance between points 3 and 4 is 0.603. The maximum distance between these 2 distances is 0.603 so the new distance between points 3 and 2 & 4 is 0.603
follow the same process for all other points

Step 3. Based on the distance matrix in step 2, the smallest distance is 0.603 between points 3 and 2 & 4 (the second height for the dendrogram). Since points 3 and 2 & 4 are the closest to each other, they are combined to form a new group, the group 2 & 3 & 4. The groups are thus: 1, 2 & 3 & 4 and 5. We construct the new distance matrix based on the same process detailed in step 2:

	1	2 & 3 & 4	5
1	0
2 & 3 & 4	2.675	0
5	0.942	1.841	0

points 1 and 5 have not change, so the distance between these two points are the same than in previous step
from step 2 we see that the distance between points 1 and 2 & 4 is 2.675 and the distance between points 1 and 3 is 2.520
since we apply the complete linkage criterion, we take the maximum distance, which is 2.675
the distance between points 1 and 2 & 3 & 4 is thus 2.675
same process for points 5 and 2 & 3 & 4

	1 & 5	2 & 3 & 4
1 & 5	0
2 & 3 & 4	2.675	0

the only distance left to compute is the distance between points 1 & 5 and 2 & 3 & 4
from the previous step we see that the distance between points 1 and 2 & 3 & 4 is 2.675 and the distance between points 5 and 2 & 3 & 4 is 1.841
since we apply the complete linkage criterion, we take the maximum distance, which is 2.675
the distance between points 1 & 5 and 2 & 3 & 4 is thus 2.675

Step 5. The final combination of points is the combination of points 1 & 5 and 2 & 3 & 4, with a final height of 2.675. Heights are used to draw the dendrogram in the sixth and final step.

Step 6. Draw the dendrogram thanks to the combination of points and heights found above. Remember that:

the first combination of points was between points 2 and 4, with a height of 0.328
the second combination was between points 3 and 2 & 4 with a height of 0.603
the third combination was between points 1 and 5 with a height of 0.942
the final combination was between points 1 & 5 and 2 & 3 & 4 with a height of 2.675
this is exactly what is illustrated in the following dendrogram:

Average linkage

With the average linkage criterion, it is not the minimum nor the maximum distance that is taken when computing the new distance between points that have been grouped, but it is, as you guessed by now, the average distance between the points.

The steps to perform the hierarchical clustering with the average linkage are detailed below.

Step 1. Step 1 is exactly the same than for single and complete linkage, that is, we compute the distance matrix of the 5 points thanks to the Pythagorean theorem. This gives us the following distance matrix:

##       1     2     3     4
## 2 2.675                  
## 3 2.520 0.483            
## 4 2.390 0.328 0.603      
## 5 0.942 1.841 1.801 1.530

Step 2. From the distance matrix computed in step 1, we see that the smallest distance = 0.328 between points 2 and 4. It is important to note that even if we apply the average linkage, in the distance matrix the points are brought together based on the smallest distance. This is the case for all 3 algorithms. The difference between the 3 algorithms lies in how to compute the new distances between the new combination of points (the single linkage takes the minimum between the distances, the complete linkage takes the maximum distance and the average linkage takes the average distance). 0.328 corresponds to the first height (which will be used when drawing the dendrogram). Since points 2 and 4 are the closest to each other, these 2 points are put together to form a single group. The groups are thus: 1, 2 & 4, 3 and 5. The new distances between the group 2 & 4 and all other points are now:

	1	2 & 4	3	5
1	0
2 & 4	2.5325	0
3	2.520	0.543	0
5	0.942	1.6855	1.801	0

To construct this new distance matrix, proceed point by point as we did for the two previous criteria:

the distance between points 1 and 3 has not changed, so the distance is unchanged compared to the initial distance matrix (found in step 1), which was 2.520
same goes for the distance between points 1 and 5 and points 3 and 5; the distances are the same than in the initial distance matrix since the points have not changed
the distance between points 1 and 2 & 4 has changed since points 2 & 4 are now together
since we are applying the average linkage criterion, the new distance between points 1 and 2 & 4 corresponds to the average distance between the distance between points 1 and 2 and the distance between points 1 and 4
the initial distance between points 1 and 2 is 2.675 and the initial distance between points 1 and 4 is 2.390
therefore, the average distance between these two distances is $\frac{2.675 + 2.390}{2} = 2.5325$
2.5325 is thus the new distance between points 1 and 2 & 4
we apply the same process for points 3 and 2 & 4: the initial distance between points 3 and 2 is 0.483 and the initial distance between points 3 and 4 is 0.603. The average distance between these 2 distances is 0.543 so the new distance between points 3 and 2 & 4 is 0.543
follow the same process for all other points

Step 3. Based on the distance matrix in step 2, the smallest distance is 0.543 between points 3 and 2 & 4 (the second height for the dendrogram). Since points 3 and 2 & 4 are the closest to each other, they are combined to form a new group, the group 2 & 3 & 4. The groups are thus: 1, 2 & 3 & 4 and 5. We construct the new distance matrix based on the same process detailed in step 2:

	1	2 & 3 & 4	5
1	0
2 & 3 & 4	2.528333	0
5	0.942	1.724	0

points 1 and 5 have not change, so the distance between these two points are the same than in previous step
from step 2 we see that the distance between points 1 and 2 & 4 is 2.5325 and the distance between points 1 and 3 is 2.520
since we apply the average linkage criterion, we take the average distance
however, we have to take into the consideration that there are 2 points in the group 2 & 4, while there is only one point in the group 3
the average distance for the distance between 1 and 2 & 3 & 4 is thus: $\frac{(2 \cdot 2.5325) + (1 \cdot 2.520)}{3} = 2.528333$
same process for points 5 and 2 & 3 & 4: $\frac{(2 \cdot 1.6855) + (1 \cdot 1.801)}{3} = 1.724$

	1 & 5	2 & 3 & 4
1 & 5	0
2 & 3 & 4	2.126167	0

the only distance left to compute is the distance between points 1 & 5 and 2 & 3 & 4
from the previous step we see that the distance between points 1 and 2 & 3 & 4 is 2.528333 and the distance between points 5 and 2 & 3 & 4 is 1.724
since we apply the average linkage criterion, we take the average distance, which is $\frac{2.528333 + 1.724}{2} = 2.126167$
the distance between points 1 & 5 and 2 & 3 & 4 is thus 2.126167

Step 5. The final combination of points is the combination of points 1 & 5 and 2 & 3 & 4, with a final height of 2.126167. Heights are used to draw the dendrogram in the sixth and final step.

Step 6. Draw the dendrogram thanks to the combination of points and heights found above. Remember that:

the first combination of points was between points 2 and 4, with a height of 0.328
the second combination was between points 3 and 2 & 4 with a height of 0.543
the third combination was between points 1 and 5 with a height of 0.942
the final combination was between points 1 & 5 and 2 & 3 & 4 with a height of 2.126167
this is exactly what is illustrated in the following dendrogram:

Solution in R

To perform the hierarchical clustering with any of the 3 criterion in R, we first need to enter the data (in this case as a matrix format, but it can also be entered as a dataframe):

X <- matrix(c(2.03, 0.06, -0.64, -0.10, -0.42, -0.53, -0.36, 0.07, 1.14, 0.37),
  nrow = 5, byrow = TRUE
)

Single linkage

We can apply the hierarchical clustering with the single linkage criterion thanks to the hclust() function with the argument method = "single":

# Hierarchical clustering: single linkage
hclust <- hclust(dist(X), method = "single")

Note that the hclust() function requires a distance matrix. If your data is not already a distance matrix (like in our case, as the matrix X corresponds to the coordinates of the 5 points), you can transform it into a distance matrix with the dist() function.

We can now extract the heights and plot the dendrogram to check our results by hand found above:

round(hclust$height, 3)

## [1] 0.328 0.483 0.942 1.530

plot(hclust)

As we can see from the dendrogram, the combination of points and the heights are the same than the ones obtained by hand.

Optimal number of clusters

Remember that hierarchical clustering is used to determine the optimal number of clusters. This optimal number of clusters can be determined thanks to the dendrogram. For this, we usually look at the largest difference of heights:

How to determine the number of clusters from a dendrogram? Take the largest difference of heights and count how many vertical lines you see

The largest difference of heights in the dendrogram occurs before the final combination, that is, before the combination of the group 2 & 3 & 4 with the group 1 & 5. To determine the optimal number of clusters, simply count how many vertical lines you see within this largest difference. In our case, the optimal number of clusters is thus 2.

In R, we can even highlight these two clusters directly in the dendrogram with the rect.hclust() function:

plot(hclust)
rect.hclust(hclust,
  k = 2, # k is used to specify the number of clusters
  border = "blue"
)

Note that determining the optimal number of clusters via the dendrogram is not specific to the single linkage, it can be applied to other linkage methods too!

Below another figure explaining how to determine the optimal number of clusters:

How to determine the optimal numbers of cluster in hierarchical clustering? Source: Towards Data Science

(See this hierarchical clustering cheatsheet for more visualizations like this.)

Finally, we could also determine the optimal number of cluster thanks to a barplot of the heights (stored in $height of the clustering output):

barplot(hclust$height,
  names.arg = (nrow(X) - 1):1 # show the number of cluster below each bars
)

Again, look for the largest jump of heights. In our case, the largest jump is from 1 to 2 classes. Therefore, the optimal number of classes is 2.

Note that determining the number of clusters using the dendrogram or barplot is not a strict rule. You can also consider other methods such as the silhouette plot, elbow plot or some numerical measures like Dunn’s index, Hubert’s gamma, etc., which show the variation of the error with the number of clusters (k), and you choose the value of k where the error is smallest. Furthermore, measuring the goodness of clusters can be done thanks to the Dunn’s Index (the higher the index, the better).

However, these methods are beyond the scope of this article and the method presented with the dendrogram is generally sufficient. See more detailed information in this article.

Note that hierarchical clustering is a bit different than k-means in the sense that is does not return a vector containing the information about which cluster the observations belong to. Instead, it creates a hierarchical structure (a dendrogram), a tree from which we can cut branches to get a given number of clusters. This information can be found via the cutree() function:

clust <- cutree(hclust,
  k = 2
)
clust

## [1] 1 2 2 2 1

We see that:

points 1 and 5 belong to the cluster 1
points 2, 3 and 4 belong to the cluster 2

This is exactly what was drawn in the dendrogram above, and if needed, this information can be added to the initial data:

X_clust <- cbind(X, clust)
X_clust

##                  clust
## [1,]  2.03  0.06     1
## [2,] -0.64 -0.10     2
## [3,] -0.42 -0.53     2
## [4,] -0.36  0.07     2
## [5,]  1.14  0.37     1

Complete linkage

We can apply the hierarchical clustering with the complete linkage criterion thanks to the hclust() function with the argument method = "complete":

# Hierarchical clustering: complete linkage
hclust <- hclust(dist(X), method = "complete")

We can now extract the heights and plot the dendrogram to check our results by hand found above:

round(hclust$height, 3)

## [1] 0.328 0.603 0.942 2.675

plot(hclust)

As we can see from the dendrogram, the combination of points and the heights are the same than the ones obtained by hand.

Similar to the single linkage, the largest difference of heights in the dendrogram occurs before the final combination, that is, before the combination of the group 2 & 3 & 4 with the group 1 & 5. In this case, the optimal number of clusters is thus 2. In R, we can even highlight these two clusters directly in the dendrogram with the rect.hclust() function:

plot(hclust)
rect.hclust(hclust,
  k = 2, # k is used to specify the number of clusters
  border = "blue"
)

Average linkage

We can apply the hierarchical clustering with the average linkage criterion thanks to the hclust() function with the argument method = "average":

# Hierarchical clustering: average linkage
hclust <- hclust(dist(X), method = "average")

We can now extract the heights and plot the dendrogram to check our results by hand found above:

round(hclust$height, 3)

## [1] 0.328 0.543 0.942 2.126

plot(hclust)

As we can see from the dendrogram, the combination of points and the heights are the same than the ones obtained by hand.

Like the single and complete linkages, the largest difference of heights in the dendrogram occurs before the final combination, that is, before the combination of the group 2 & 3 & 4 with the group 1 & 5. In this case, the optimal number of clusters is thus 2. In R, we can even highlight these two clusters directly in the dendrogram with the rect.hclust() function:

plot(hclust)
rect.hclust(hclust,
  k = 2, # k is used to specify the number of clusters
  border = "blue"
)

k-means versus hierarchical clustering

Clustering is rather a subjective statistical analysis and there can be more than one appropriate algorithm, depending on the dataset at hand or the type of problem to be solved. So choosing between k-means and hierarchical clustering is not always easy. Moreover, no method is better than the other, each come with their sets of limitations and benefits.

If you have a good reason to think that there is a specific number of clusters in your dataset (for example if you would like to distinguish diseased and healthy patients depending on some characteristics but you do not know in which group patients belong to), you should probably opt for the k-means clustering as this technique is used when the number of groups is specified in advance. If you do not have any reason to believe there is a certain number of groups in your dataset (for instance in marketing when trying to distinguish clients without any prior belief on the number of different types of customers), then you should probably opt for the hierarchical clustering to determine in how many clusters your data should be divided.

In addition to this, if you are still undecided note that, on the one hand, with a large number of variables, k-means may be computationally faster than hierarchical clustering if the number of clusters is small. On the other hand, the result of a hierarchical clustering is a structure that is more informative and interpretable than the unstructured set of flat clusters returned by k-means. Therefore, it is easier to determine the optimal number of clusters by looking at the dendrogram of a hierarchical clustering than trying to predict this optimal number in advance in case of k-means.

What’s next?

We have seen that clustering is a nice tool to identify homogeneous observations with respect to the measured variables. Following your clustering, you may want to get a sense of the characteristics of each cluster. This can be done through some descriptive statistics for each cluster.

For example, for categorical variables, you could start to compute, for each cluster, the proportion of responses for each level of your categorical variables. You could also visualize this information via a barplot. For quantitative variables, the mean or median response for each variable in each cluster is usually a good starting point.

If your clustering performs well, you will see that within each cluster observations share a common pattern of responses across the variables. This is both a nice way to see what makes each cluster unique, and what makes clusters different.

Conclusion

Thanks for reading.

I hope this article helped you understand the different clustering methods and how to compute them by hand and in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Hartigan, J. A., and M. A. Wong. 1979. “A k-Means Clustering Algorithm.” Applied Statistics 28: 100–108.

Lloyd, Stuart. 1982. “Least Squares Quantization in PCM.” IEEE Transactions on Information Theory 28 (2): 129–37.

Do my data follow a normal distribution? A note on the most widely used distribution and how to test for normality in R

Wed, 29 Jan 2020 00:00:00 +0000

What is a normal distribution?

The normal distribution is a function that defines how a set of measurements is distributed around the center of these measurements (i.e., the mean). Many natural phenomena in real life can be approximated by a bell-shaped frequency distribution known as the normal distribution or the Gaussian distribution.

The normal distribution is a mount-shaped, unimodal and symmetric distribution where most measurements gather around the mean. Moreover, the further a measure deviates from the mean, the lower the probability of occurring. In this sense, for a given variable, it is common to find values close to the mean, but less and less likely to find values as we move away from the mean. Last but not least, since the normal distribution is symmetric around its mean, extreme values in both tails of the distribution are equivalently unlikely. For instance, given that adult height follows a normal distribution, most adults are close to the average height and extremely short adults occur as infrequently as extremely tall adults.

In this article, the focus is on understanding the normal distribution, the associated empirical rule, its parameters and how to compute $Z$-scores to find probabilities under the curve (illustrated with examples). As it is a requirement in some statistical tests and hypothesis tests, we also show 4 complementary methods to test the normality assumption in R.

Empirical rule

Data possessing an approximately normal distribution have a definite variation, as expressed by the following empirical rule:

$\mu \pm \sigma$ includes approximately 68% of the observations
$\mu \pm 2 \cdot \sigma$ includes approximately 95% of the observations
$\mu \pm 3 \cdot \sigma$ includes almost all of the observations (99.7% to be more precise)

Normal distribution & empirical rule (68-95-99.7% rule)

where $\mu$ and $\sigma$ correspond to the population mean and population standard deviation, respectively.

The empirical rule, also known as the 68-95-99.7% rule, is illustrated by the following 2 examples.

Suppose that the scores of an exam in statistics given to all students in a Belgian university are known to have, approximately, a normal distribution with mean $\mu = 67$ and standard deviation $\sigma = 9$. It can then be deduced that approximately 68% of the scores are between 58 and 76, that approximately 95% of the scores are between 49 and 85, and that almost all of the scores (99.7%) are between 40 and 94. Thus, knowing the mean and the standard deviation gives us a fairly good picture of the distribution of scores.

Now suppose that a single university student is randomly selected from those who took the exam. What is the probability that her score will be between 49 and 85? Based on the empirical rule, we find that 0.95 is a reasonable answer to this probability question.

The utility and value of the empirical rule are due to the common occurrence of approximately normal distributions of measurements in nature. For example, IQ, shoe size, height, birth weight, etc. are approximately normally-distributed. You will find that approximately 95% of these measurements will be within $2\sigma$ of their mean (Wackerly, Mendenhall, and Scheaffer 2014).

Parameters

Like many probability distributions, the shape and probabilities of the normal distribution is defined entirely by some parameters. The normal distribution has two parameters:

the mean $\mu$, and
the variance $\sigma^2$ (i.e., the square of the standard deviation $\sigma$).

The mean $\mu$ locates the center of the distribution, that is, the central tendency of the observations, and the variance $\sigma^2$ defines the width of the distribution, that is, the spread of the observations.

The mean $\mu$ can take on any finite value (i.e., $-\infty < \mu < \infty$), whereas the variance $\sigma^2$ can assume any positive finite value (i.e., $\sigma^2 > 0$). The shape of the normal distribution changes based on these two parameters. Since there is an infinite number of combinations of the mean and variance, there is an infinite number of normal distributions, and thus an infinite number of forms.

For instance, see how the shapes of the normal distributions vary when the two parameters change:

As you can see on the second graph, when the variance (or the standard deviation) decreases, the observations are closer to the mean. On the contrary, when the variance (or standard deviation) increases, it is more likely that observations will be further away from the mean.

A random variable $X$ which follows a normal distribution with a mean of 430 and a variance of 17 is denoted $X ~ \sim \mathcal{N}(\mu = 430, \sigma^2 = 17)$.

We have seen that, although different normal distributions have different shapes, all normal distributions have common characteristics:

They are symmetric, 50% of the population is above the mean and 50% of the population is below the mean
The mean, median and mode are equal
The empirical rule detailed earlier is applicable to all normal distributions

Probabilities and standard normal distribution

Probabilities and quantiles for random variables with normal distributions are easily found using R via the functions pnorm() and qnorm(). Probabilities associated with a normal distribution can also be found using this Shiny app. However, before computing probabilities, we need to learn more about the standard normal distribution and the $Z$-score.

Although there are infinitely many normal distributions (since there is a normal distribution for every combination of mean and variance), we need only one table to find the probabilities under the normal curve: the standard normal distribution.

The normal standard distribution is a special case of the normal distribution where the mean is equal to 0 and the variance is equal to 1. A normal random variable $X$ can always be transformed to a standard normal random variable $Z$, a process known as “scaling” or “standardization”, by subtracting the mean from the observation, and dividing the result by the standard deviation. Formally:

\[Z = \frac{x - \mu}{\sigma}\]

where $x$ is the observation, $\mu$ and $\sigma$ the mean and standard deviation of the population from which the observation was drawn. So the mean of the standard normal distribution is 0, and its variance is 1, denoted $Z ~ \sim \mathcal{N}(\mu = 0, \sigma^2 = 1)$.

From this formula, we see that $Z$, referred as standard score or $Z$-score, allows to see how far away one specific observation is from the mean of all observations, with the distance expressed in standard deviations. In other words, the $Z$-score corresponds to the number of standard deviations an observation is away from the mean. A positive $Z$-score means that the specific observation is above the mean, whereas a negative $Z$-score means that the specific observation is below the mean. $Z$-scores are often used to compare an individual to her peers, or more generally, a measurement compared to its distribution.

For instance, suppose a student scoring 60 at a statistics exam with the mean score of the class being 40, and scoring 65 at an economics exam with the mean score of the class being 80. Given the “raw” scores, one would say that the student performed better in economics than in statistics. However, taking into consideration her peers, it is clear that the student performed relatively better in statistics than in economics.

Computing $Z$-scores allows to take into consideration all other students (i.e., the entire distribution) and gives a better measure of comparison. Let’s compute the $Z$-scores for the two exams, assuming that the score for both exams follow a normal distribution with the following parameters:

	Statistics	Economics
Mean	40	80
Standard deviation	8	12.5
Student’s score	60	65

$Z$-scores for:

Statistics: $z_{stat} = \frac{60 - 40}{8} = 2.5$
Economics: $z_{econ} = \frac{65 - 80}{12.5} = -1.2$

On the one hand, the $Z$-score for the exam in statistics is positive ($z_{stat} = 2.5$) which means that she performed better than average. On the other hand, her score for the exam in economics is negative ($z_{econ} = -1.2$) which means that she performed worse than average. Below an illustration of her grades in a standard normal distribution for better comparison:

Although the score in economics is better in absolute terms, the score in statistics is actually relatively better when comparing each score within its own distribution.

Furthermore, $Z$-score also enables to compare observations that would otherwise be impossible because they have different units for example. Suppose you want to compare a salary in € with a weight in kg. Without standardization, there is no way to conclude whether someone is more extreme in terms of her wage or in terms of her weight. Thanks to $Z$-scores, we can compare two values that were in the first place not comparable to each other.

Final remark regarding the interpretation of a $Z$-score: a rule of thumb is that an observation with a $z$-score between -3 and -2 or between 2 and 3 is considered as a rare value. An observation with a $z$-score smaller than -3 or larger than 3 is considered as an extremely rare value. A value with any other $z$-score is considered as not rare nor extremely rare.

Areas under the normal distribution in R and by hand

Now that we have covered the $Z$-score, we are going to use it to determine the area under the curve of a normal distribution.

Note that there are several ways to arrive at the solution in the following exercises. You may therefore use other steps than the ones presented to obtain the same result.

Ex. 1

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(Z > 1)$.

We actually look for the shaded area in the following figure:

Standard normal distribution: $P(Z > 1)$

In R

pnorm(1,
  mean = 0,
  sd = 1, # sd stands for standard deviation
  lower.tail = FALSE
)

## [1] 0.1586553

We look for the probability of $Z$ being larger than 1 so we set the argument lower.tail = FALSE. The default lower.tail = TRUE would give the result for $P(Z < 1)$. Note that $P(Z = 1) = 0$ so writing $P(Z > 1)$ or $P(Z \ge 1)$ is equivalent.

By hand

See that the random variable $Z$ has already a mean of 0 and a standard deviation of 1, so no transformation is required. To find the probabilities by hand, we need to refer to the standard normal distribution table shown below:

Standard normal distribution table (Wackerly, Mendenhall, and Scheaffer 2014).

From the illustration at the top of the table, we see that the values inside the table correspond to the area under the normal curve above a certain $z$. Since we are looking precisely at the probability above $z = 1$ (since we look for $P(Z > 1)$), we can simply proceed down the first ($z$) column in the table until $z = 1.0$. The probability is 0.1587. Thus, $P(Z > 1) = 0.1587$. This is similar to what we found using R, except that values in the table are rounded to 4 digits.

Ex. 2

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(−1 \le Z \le 1)$.

We are looking for the shaded area in the following figure:

Standard normal distribution: $P(−1 \le Z \le 1)$

In R

pnorm(1, lower.tail = TRUE) - pnorm(-1, lower.tail = TRUE)

## [1] 0.6826895

Note that the arguments by default for the mean and the standard deviation are mean = 0 and sd = 1. Since this is what we need, we can omit them.¹

By hand

For this exercise we proceed by steps:

The shaded area corresponds to the entire area under the normal curve minus the two white areas in both tails of the curve.
We know that the normal distribution is symmetric.
Therefore, the shaded area is the entire area under the curve minus two times the white area in the right tail of the curve, the white area in the right tail of the curve being $P(Z > 1)$.
We also know that the entire area under the normal curve is 1.
Thus, the shaded area is 1 minus 2 times $P(Z > 1)$:

\[P(−1 \le Z \le 1) = 1 - 2 \cdot P(Z > 1)\] \[= 1 - 2 \cdot 0.1587 = 0.6826\]

where $P(Z > 1) = 0.1587$ has been found in the previous exercise.

Ex. 3

Let $Z$ denote a normal random variable with mean 0 and standard deviation 1, find $P(0 \le Z \le 1.37)$.

We are looking for the shaded area in the following figure:

Standard normal distribution: $P(0 \le Z \le 1.37)$

In R

pnorm(0, lower.tail = FALSE) - pnorm(1.37, lower.tail = FALSE)

## [1] 0.4146565

By hand

Again we proceed by steps for this exercise:

We know that $P(Z > 0) = 0.5$ since the entire area under the curve is 1, half of it is 0.5.
The shaded area is half of the entire area under the curve minus the area from 1.37 to infinity.
The area under the curve from 1.37 to infinity corresponds to $P(Z > 1.37)$.
Therefore, the shaded area is $0.5 - P(Z > 1.37)$.
To find $P(Z > 1.37)$, proceed down the $z$ column in the table to the entry 1.3 and then across the top of the table to the column labeled .07 to read $P(Z > 1.37) = .0853$
Thus,

\[P(0 \le Z \le 1.37) = P(Z > 0) - P(Z > 1.37)\] \[ = 0.5 - 0.0853 = 0.4147\]

Ex. 4

Recap the example presented in the empirical rule: Suppose that the scores of an exam in statistics given to all students in a Belgian university are known to have a normal distribution with mean $\mu = 67$ and standard deviation $\sigma = 9$. What fraction of the scores lies between 70 and 80?

We are looking for the shaded area in the following figure:

$P(70 \le X \le 80)$ where $X \sim \mathcal{N}(\mu = 67, \sigma^2 = 9^2)$

In R

pnorm(70, mean = 67, sd = 9, lower.tail = FALSE) - pnorm(80, mean = 67, sd = 9, lower.tail = FALSE)

## [1] 0.2951343

By hand

Remind that we are looking for $P(70 \le X \le 80)$ where $X \sim \mathcal{N}(\mu = 67, \sigma^2 = 9^2)$. The random variable $X$ is in its “raw” format, meaning that it has not been standardized yet since the mean is 67 and the variance is $9^2$. We thus need to first apply the transformation to standardize the endpoints 70 and 80 with the following formula:

\[Z = \frac{X - \mu}{\sigma}\]

After the standardization, $x = 70$ becomes (in terms of $z$, so in terms of deviation from the mean expressed in standard deviation):

\[z = \frac{70 - 67}{9} = 0.3333\]

and $x = 80$ becomes:

\[z = \frac{80 - 67}{9} = 1.4444\]

The figure above in terms of $X$ is now in terms of $Z$:

$P(0.3333 \le Z \le 1.4444)$ where $Z \sim \mathcal{N}(\mu = 0, \sigma^2 = 1)$

Finding the probability $P(0.3333 \le Z \le 1.4444)$ is similar to exercises 1 to 3:

The shaded area corresponds to the area under the curve from $z = 0.3333$ to $z = 1.4444$.
In other words, the shaded area is the area under the curve from $z = 0.3333$ to infinity minus the area under the curve from $z = 1.4444$ to infinity.
From the table, $P(Z > 0.3333) = 0.3707$ and $P(Z > 1.4444) = 0.0749$
Thus:

\[P(0.3333 \le Z \le 1.4444)\] \[= P(Z > 0.3333) - P(Z > 1.4444)\] \[= 0.3707 - 0.0749 = 0.2958\]

The difference with the probability found using in R comes from the rounding.

To conclude this exercise, we can say that, given that the mean scores is 67 and the standard deviation is 9, 29.58% of the students scored between 70 and 80.

Ex. 5

See another example in a context here.

Why is the normal distribution so crucial in statistics?

The normal distribution is important for three main reasons:

Some statistical hypothesis tests assume that the data follow a normal distribution
The central limit theorem states that, for a large number of observations (usually $n > 30$), no matter what is the underlying distribution of the original variable, the distribution of the sample means ($\overline{X}_n$) and of the sum ($S_n = \sum_{i = 1}^n X_i$) may be approached by a normal distribution (Stevens 2013)
Linear and nonlinear regression assume that the residuals are normally-distributed (for small sample sizes)

It is therefore useful to know how to test for normality in R, which is the topic of next sections.

How to test the normality assumption

As mentioned above, some statistical tests require that the data follow a normal distribution, or the result of the test may be flawed.

In this section, we show 4 complementary methods to determine whether your data follow a normal distribution in R.

Histogram

A histogram displays the spread and shape of a distribution, so it is a good starting point to evaluate normality.

Let’s have a look at the histogram of a distribution that we would expect to follow a normal distribution, the height of 1,000 adults in cm:

The normal curve with the corresponding mean and variance has been added to the histogram. The histogram follows the normal curve so the data seems to follow a normal distribution.

Below the minimal code for a histogram in R with the dataset iris:

data(iris)
hist(iris$Sepal.Length)

In {ggplot2}:

ggplot(iris) +
  aes(x = Sepal.Length) +
  geom_histogram()

Histograms are however not sufficient, particularly in the case of small samples because the number of bins greatly change its appearance. Histograms are not recommended when the number of observations is less than 20 because it does not always correctly illustrate the distribution. See two examples below with datasets of 10 and 12 observations:

Can you tell whether these datasets follow a normal distribution? Surprisingly, both series are generated from a normal distribution!

In the remaining of the article, we will use the dataset of the 12 adults. If you would like to follow my code in your own script, here is how I generated the data:

set.seed(42)
dat_hist <- data.frame(
  value = rnorm(12, mean = 165, sd = 5)
)

The rnorm() function generates random numbers from a normal distribution (12 random numbers with a mean of 165 and standard deviation of 5 in this case). These 12 observations are then saved in the dataset called dat_hist under the variable value. Note that set.seed(42) is important to obtain the exact same data as me.²

Density plot

Density plots also provide a visual judgment about whether the data follow a normal distribution.

They are similar to histograms as they also allow to analyze the spread and the shape of the distribution. However, they are a smoothed version of the histogram.

Here is the density plot drawn from the dataset on the height of the 12 adults discussed above:

plot(density(dat_hist$value))

In {ggpubr}:

library(ggpubr) # package must be installed first
ggdensity(dat_hist$value,
  main = "Density plot of adult height",
  xlab = "Height (cm)"
)

Since it is hard to test for normality from histograms and density plots only, it is recommended to corroborate these graphs with a QQ-plot. QQ-plot, also known as normality plot, is the third method presented to evaluate normality.

QQ-plot

Like histograms and density plots, QQ-plots allow to visually evaluate the normality assumption.

Here is the QQ-plot drawn from the dataset on the height of the 12 adults discussed above:

library(car)
qqPlot(dat_hist$value)

## [1] 12  2

In {ggpubr}:

library(ggpubr)
ggqqplot(dat_hist$value)

Instead of looking at the spread of the data (as it is the case with histograms and density plots), with QQ-plots we only need to ascertain whether the data points follow the line (sometimes referred as Henry’s line).

If points are close to the reference line and within the confidence bands, the normality assumption can be considered as met. The bigger the deviation between the points and the reference line and the more they lie outside the confidence bands, the less likely that the normality condition is met. The height of these 12 adults seem to follow a normal distribution because points follow the line and all of them lie within the confidence bands.

When facing a non-normal distribution as shown by the QQ-plot below (systematic departure from the reference line), the first step is usually to apply the logarithm transformation on the data and recheck to see whether the log-transformed data are normally distributed.

If this is the case, the data are said to follow a log-normal distribution. Applying the logarithm transformation can be done in R with the log() function.

Note that QQ-plots are also a convenient way to assess whether residuals from linear regression follow a normal distribution.

Normality test

The 3 tools presented above were a visual inspection of the normality. Nonetheless, visual inspection may sometimes be unreliable so it is also possible to formally test whether the data follow a normal distribution with statistical tests.

These normality tests compare the distribution of the data to a normal distribution in order to assess whether observations show an important deviation from normality.

The two most common normality tests are Shapiro-Wilk’s test and Kolmogorov-Smirnov test. Both tests have the same hypotheses, that is:

$H_0$: the data follow a normal distribution
$H_1$: the data do not follow a normal distribution

Shapiro-Wilk test is recommended for normality test as it provides better power than Kolmogorov-Smirnov test.³ In R, the Shapiro-Wilk test of normality can be done with the function shapiro.test():⁴

shapiro.test(dat_hist$value)

## 
##  Shapiro-Wilk normality test
## 
## data:  dat_hist$value
## W = 0.93968, p-value = 0.4939

From the output, we see that the $p$-value $> 0.05$ implying that we do not reject the null hypothesis that the data follow a normal distribution. This test goes in the same direction than the QQ-plot, which showed no significant deviation from the normality (as all points lied within the confidence bands).

It is important to note that, in practice, normality tests are often considered as too conservative in the sense that for large sample size ($n > 50$), a small deviation from the normality may cause the normality condition to be violated.

A normality test is a hypothesis test, so as the sample size increases, their capacity of detecting smaller differences increases. So as the number of observations increases, the Shapiro-Wilk test becomes very sensitive even to a small deviation from normality. As a consequence, it happens that according to the normality test the data do not follow a normal distribution although the departures from the normal distribution is negligible so the data could in fact be considered to follow approximately a normal distribution. For this reason, it is often the case that the normality condition is verified based on a combination of all methods presented in this article, that is, visual inspections (with histograms and QQ-plots) and a formal inspection (with the Shapiro-Wilk test for instance).

I personally tend to prefer QQ-plots over histograms and normality tests so I do not have to bother about the sample size. This article showed the different methods that are available, your choice will of course depends on the type of your data and the context of your analyses.

Conclusion

Thanks for reading.

I hope the article helped you to learn more about the normal distribution and how to test for normality in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Stevens, James P. 2013. Intermediate Statistics: A Modern Approach. Routledge.

Wackerly, Dennis, William Mendenhall, and Richard L Scheaffer. 2014. Mathematical Statistics with Applications. Cengage Learning.

The argument lower.tail = TRUE is also the default so we could omit it as well. However, for clarity and to make sure I compute the propabilities in the correct side of the curve, I used to keep this argument explicit by writing it.↩︎
The set.seed() function accepts any numeric as argument. Generating random numbers (via rnorm() for instance) implies that R will generates different random numbers every time you generate these random numbers (so every time you run the function rnorm()). To make sure R generates the exact same numbers every time you run the function, a seed can be set with the function set.seed(). Setting a seed implies that R will generate random numbers, but these numbers will always be the same as long as the seed is the same. This allows to replicate results that are based on a random generation. Change the seed if you want to generate other random values.↩︎
The Shapiro-Wilk test is based on the correlation between the sample and the corresponding normal scores.↩︎
In R, the Kolmogorov-Smirnov test is performed with the function ks.test().↩︎

Fisher's exact test in R: independence test for a small sample

Tue, 28 Jan 2020 00:00:00 +0000

Introduction

After presenting the Chi-square test of independence by hand and in R, this article focuses on the Fisher’s exact test.

Independence tests are used to determine if there is a significant relationship between two categorical variables. There exists two different types of independence test:

the Chi-square test (the most common)
the Fisher’s exact test

On the one hand, the Chi-square test is used when the sample is large enough (in this case the $p$-value is an approximation that becomes exact when the sample becomes infinite, which is the case for many statistical tests). On the other hand, the Fisher’s exact test is used when the sample is small (and in this case the $p$-value is exact and is not an approximation).

The literature indicates that the usual rule for deciding whether the $\chi^2$ approximation is good enough is that the Chi-square test is not appropriate when the expected values in one of the cells of the contingency table is less than 5, and in this case the Fisher’s exact test is preferred (McCrum-Gardner 2008; Bower 2003).

Hypotheses

The hypotheses of the Fisher’s exact test are the same than for the Chi-square test, that is:

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

Example

Data

For our example, we want to determine whether there is a statistically significant association between smoking and being a professional athlete. Smoking can only be “yes” or “no” and being a professional athlete can only be “yes” or “no”. The two variables of interest are qualitative variables and we collected data on 14 persons.¹

Observed frequencies

Our data are summarized in the contingency table below reporting the number of people in each subgroup:

dat <- data.frame(
  "smoke_no" = c(7, 0),
  "smoke_yes" = c(2, 5),
  row.names = c("Athlete", "Non-athlete"),
  stringsAsFactors = FALSE
)
colnames(dat) <- c("Non-smoker", "Smoker")

dat

##             Non-smoker Smoker
## Athlete              7      2
## Non-athlete          0      5

It is also a good practice to draw a mosaic plot to visually represent the data:

mosaicplot(dat,
  main = "Mosaic plot",
  color = TRUE
)

We can already see from the plot that the proportion of smokers in the sample is higher among non-athletes than athlete. The plot is however not sufficient to conclude that there is such a significant association in the population.

Like many statistical tests, this can be done via a hypothesis test. But before seeing how to do it in R, let’s see the concept of expected frequencies.

Expected frequencies

Remember that the Fisher’s exact test is used when there is at least one cell in the contingency table of the expected frequencies below 5. To retrieve the expected frequencies, use the chisq.test() function together with $expected:

chisq.test(dat)$expected

## Warning in chisq.test(dat): Chi-squared approximation may be incorrect

##             Non-smoker Smoker
## Athlete            4.5    4.5
## Non-athlete        2.5    2.5

The contingency table above confirms that we should use the Fisher’s exact test instead of the Chi-square test because there is at least one cell below 5.

Tip: although it is a good practice to check the expected frequencies before deciding between the Chi-square and the Fisher test, it is not a big issue if you forget. As you can see above, when doing the Chi-square test in R (with chisq.test()), a warning such as “Chi-squared approximation may be incorrect” will appear. This warning means that the smallest expected frequencies is lower than 5. Therefore, do not worry if you forgot to check the expected frequencies before applying the appropriate test to your data, R will warn you that you should use the Fisher’s exact test instead of the Chi-square test if that is the case.

(Remember that, as for the Chi-square test of independence, the observations must be independent in order for the Fisher’s exact test to be valid. See more details about the independence assumption in this section.)

Fisher’s exact test in R

To perform the Fisher’s exact test in R, use the fisher.test() function as you would do for the Chi-square test:

test <- fisher.test(dat)
test

## 
##  Fisher's Exact Test for Count Data
## 
## data:  dat
## p-value = 0.02098
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  1.449481      Inf
## sample estimates:
## odds ratio 
##        Inf

The most important in the output is the $p$-value. You can also retrieve the $p$-value with:

test$p.value

## [1] 0.02097902

Note that if your data is not already presented as a contingency table, you can simply use the following code:

fisher.test(table(dat$variable1, dat$variable2))

where dat is the name of your dataset, variable1 and variable2 correspond to the names of the two variables of interest.

Conclusion and interpretation

From the output and from test$p.value we see that the $p$-value is less than the significance level of 5%. Like any other statistical test, if the $p$-value is less than the significance level, we can reject the null hypothesis. If you are not familiar with $p$-values, I invite you to read this section.

$\Rightarrow$ In our context, rejecting the null hypothesis for the Fisher’s exact test of independence means that there is a significant relationship between the two categorical variables (smoking habits and being an athlete or not). Therefore, knowing the value of one variable helps to predict the value of the other variable.

Combination of plot and statistical test

It is possible print the results of the Fisher’s exact test directly on a barplot thanks to the ggbarstats() function from the {ggstatsplot} package (the function has been slightly edited to match our needs).

It is easier to work with the package when our data is not already in the form of a contingency table so we transform it to a data frame before plotting the results:

# create dataframe from contingency table
x <- c()
for (row in rownames(dat)) {
  for (col in colnames(dat)) {
    x <- rbind(x, matrix(rep(c(row, col), dat[row, col]), ncol = 2, byrow = TRUE))
  }
}
df <- as.data.frame(x)
colnames(df) <- c("Sport_habits", "Smoking_habits")
df

##    Sport_habits Smoking_habits
## 1       Athlete     Non-smoker
## 2       Athlete     Non-smoker
## 3       Athlete     Non-smoker
## 4       Athlete     Non-smoker
## 5       Athlete     Non-smoker
## 6       Athlete     Non-smoker
## 7       Athlete     Non-smoker
## 8       Athlete         Smoker
## 9       Athlete         Smoker
## 10  Non-athlete         Smoker
## 11  Non-athlete         Smoker
## 12  Non-athlete         Smoker
## 13  Non-athlete         Smoker
## 14  Non-athlete         Smoker

# Fisher's exact test with raw data
test <- fisher.test(table(df))

# combine plot and statistical test with ggbarstats
library(ggstatsplot)
ggbarstats(
  df, Smoking_habits, Sport_habits,
  results.subtitle = FALSE,
  subtitle = paste0(
    "Fisher's exact test", ", p-value = ",
    ifelse(test$p.value < 0.001, "< 0.001", round(test$p.value, 3))
  )
)

From the plot, it is clear that the proportion of smokers among non-athletes is higher than among athletes, suggesting that there is a relationship between the two variables.

This is confirmed thanks to the $p$-value displayed in the subtitle of the plot. As previously, we reject the null hypothesis and we conclude that the variables smoking habits and being an athlete or not are dependent ($p$-value = 0.021).

Conclusion

Thanks for reading.

I hope the article helped you to perform the Fisher’s exact test of independence in R and interpret its results. Learn more about the Chi-square test of independence by hand or in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Bower, Keith M. 2003. “When to Use Fisher’s Exact Test.” In American Society for Quality, Six Sigma Forum Magazine, 2:35–37. 4.

McCrum-Gardner, Evie. 2008. “Which Is the Correct Statistical Test to Use?” British Journal of Oral and Maxillofacial Surgery 46 (1): 38–41.

The data are the same than for the article covering the Chi-square test by hand, except that some observations have been removed to decrease the sample size.↩︎

Chi-square test of independence by hand

Mon, 27 Jan 2020 00:00:00 +0000

Introduction

Chi-square tests of independence test whether two qualitative variables are independent, that is, whether there exists a relationship between two categorical variables. In other words, this test is used to determine whether the values of one of the 2 qualitative variables depend on the values of the other qualitative variable.

If the test shows no association between the two variables (i.e., the variables are independent), it means that knowing the value of one variable gives no information about the value of the other variable. On the contrary, if the test shows a relationship between the variables (i.e., the variables are dependent), it means that knowing the value of one variable provides information about the value of the other variable.

This article focuses on how to perform a Chi-square test of independence by hand and how to interpret the results with a concrete example. To learn how to do this test in R, read the article “Chi-square test of independence in R”.

Hypotheses

The Chi-square test of independence is a hypothesis test so it has a null ($H_0$) and an alternative hypothesis ($H_1$):

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

How the test works?

The Chi-square test of independence works by comparing the observed frequencies (so the frequencies observed in your sample) to the expected frequencies if there was no relationship between the two categorical variables (so the expected frequencies if the null hypothesis was true).

If the difference between the observed frequencies and the expected frequencies is small, we cannot reject the null hypothesis of independence and thus we cannot reject the fact that the two variables are not related. On the other hand, if the difference between the observed frequencies and the expected frequencies is large, we can reject the null hypothesis of independence and thus we can conclude that the two variables are related.

The threshold between a small and large difference is a value that comes from the Chi-square distribution (hence the name of the test). This value, referred as the critical value, depends on the significance level $\alpha$ (usually set equal to 5%) and on the degrees of freedom. This critical value can be found in the statistical table of the Chi-square distribution. More on this critical value and the degrees of freedom later in the article.

Example

Note that we chose binary variables (binary variables = qualitative variables with two levels) for the sake of easiness, but the Chi-square test of independence can also be performed on qualitative variables with more than two levels. For instance, if the variable smoking had three levels: (i) non-smokers, (ii) moderate smokers and (iii) heavy smokers, the steps and the interpretation of the results of the test are similar than with two levels.

Observed frequencies

Our data are summarized in the contingency table below reporting the number of people in each subgroup, totals by row, by column and the grand total:

	Non-smoker	Smoker	Total
Athlete	14	4	18
Non-athlete	0	10	10
Total	14	14	28

Expected frequencies

Remember that for the Chi-square test of independence we need to determine whether the observed counts are significantly different from the counts that we would expect if there was no association between the two variables. We have the observed counts (see the table above), so we now need to compute the expected counts in the case the variables were independent. These expected frequencies are computed for each subgroup one by one with the following formula:

\[\text{exp. frequencies} = \frac{\text{total # of obs. for the row} \cdot \text{total # of obs. for the column}}{\text{total number of observations}}\]

where obs. correspond to observations. Given our table of observed frequencies above, below is the table of the expected frequencies computed for each subgroup:

	Non-smoker	Smoker	Total
Athlete	(18 * 14) / 28 = 9	(18 * 14) / 28 = 9	18
Non-athlete	(10 * 14) / 28 = 5	(10 * 14) / 28 = 5	10
Total	14	14	28

Note that the Chi-square test of independence should only be done when the expected frequencies in all groups are equal to or greater than 5. This assumption is met for our example as the minimum number of expected frequencies is 5. If the condition is not met, the Fisher’s exact test is preferred.

Talking about assumptions, the Chi-square test of independence requires that the observations are independent. This is usually not tested formally, but rather verified based on the design of the experiment and on the good control of experimental conditions. If you are not sure, ask yourself if one observation is related to another (if one observation has an impact on another). If not, it is most likely that you have independent observations.

If you have dependent observations (paired samples), the McNemar’s or Cochran’s Q tests should be used instead. The McNemar’s test is used when we want to know if there is a significant change in two paired samples (typically in a study with a measure before and after on the same subject) when the variables have only two categories. The Cochran’s Q tests is an extension of the McNemar’s test when we have more than two related measures.

Test statistic

We have the observed and expected frequencies. We now need to compare these frequencies to determine if they differ significantly. The difference between the observed and expected frequencies, referred as the test statistic (or t-stat) and denoted $\chi^2$, is computed as follows:

\[\chi^2 = \sum_{i, j} \frac{\big(O_{ij} - E_{ij}\big)^2}{E_{ij}}\]

where $O$ represents the observed frequencies and $E$ the expected frequencies. We use the square of the differences between the observed and expected frequencies to make sure that negative differences are not compensated by positive differences. The formula looks more complex than what it really is, so let’s illustrate it with our example. We first compute the difference in each subgroup one by one according to the formula:

in the subgroup of athlete and non-smoker: $\frac{(14 - 9)^2}{9} = 2.78$
in the subgroup of non-athlete and non-smoker: $\frac{(0 - 5)^2}{5} = 5$
in the subgroup of athlete and smoker: $\frac{(4 - 9)^2}{9} = 2.78$
in the subgroup of non-athlete and smoker: $\frac{(10 - 5)^2}{5} = 5$

and then we sum them all to obtain the test statistic:

\[\chi^2 = 2.78 + 5 + 2.78 + 5 = 15.56\]

Critical value

The test statistic alone is not enough to conclude for independence or dependence between the two variables. As previously mentioned, this test statistic (which in some sense is the difference between the observed and expected frequencies) must be compared to a critical value to determine whether the difference is large or small. One cannot tell that a test statistic is large or small without putting it in perspective with the critical value.

If the test statistic is above the critical value, it means that the probability of observing such a difference between the observed and expected frequencies is unlikely. On the other hand, if the test statistic is below the critical value, it means that the probability of observing such a difference is likely. If it is likely to observe this difference, we cannot reject the hypothesis that the two variables are independent, otherwise we can conclude that there exists a relationship between the variables.

The critical value can be found in the statistical table of the Chi-square distribution and depends on the significance level, denoted $\alpha$, and the degrees of freedom, denoted $df$. The significance level is usually set equal to 5%. The degrees of freedom for a Chi-square test of independence is found as follow:

\[df = (\text{number of rows} - 1) \cdot (\text{number of columns} - 1)\]

In our example, the degrees of freedom is thus $df = (2 - 1) \cdot (2 - 1) = 1$ since there are two rows and two columns in the contingency table (totals do not count as a row or column).

We now have all the necessary information to find the critical value in the Chi-square table ($\alpha = 0.05$ and $df = 1$). To find the critical value we need to look at the row $df = 1$ and the column $\chi^2_{0.050}$ (since $\alpha = 0.05$) in the picture below. The critical value is $3.84146$.¹

Chi-square table - Critical value for alpha = 5% and df = 1

Conclusion and interpretation

Now that we have the test statistic and the critical value, we can compare them to check whether the null hypothesis of independence of the variables is rejected or not. In our example,

\[\text{test statistic} = 15.56 > \text{critical value} = 3.84146\]

Like for many statistical tests, when the test statistic is larger than the critical value, we can reject the null hypothesis at the specified significance level.

In our case, we can therefore reject the null hypothesis of independence between the two categorical variables at the 5% significance level.

$\Rightarrow$ This means that there is a significant relationship between the smoking habit and being an athlete or not. Knowing the value of one variable helps to predict the value of the other variable.

Thanks for reading.

I hope the article helped you to perform the Chi-square test of independence by hand and interpret its results. If you would like to learn how to do this test in R, read the article “Chi-square test of independence in R”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

For readers that prefer to check the $p$-value in order to reject or not the null hypothesis, I also created a Shiny app to help you compute the $p$-value given a test statistic.↩︎

Chi-square test of independence in R

Mon, 27 Jan 2020 00:00:00 +0000

Introduction

This article explains how to perform the Chi-square test of independence in R and how to interpret its results. To learn more about how the test works and how to do it by hand, I invite you to read the article “Chi-square test of independence by hand”.

To briefly recap what have been said in that article, the Chi-square test of independence tests whether there is a relationship between two categorical variables. The null and alternative hypotheses are:

$H_0$ : the variables are independent, there is no relationship between the two categorical variables. Knowing the value of one variable does not help to predict the value of the other variable
$H_1$ : the variables are dependent, there is a relationship between the two categorical variables. Knowing the value of one variable helps to predict the value of the other variable

Data

For our example, let’s reuse the dataset introduced in the article “Descriptive statistics in R”. This dataset is the well-known iris dataset slightly enhanced. Since there is only one categorical variable and the Chi-square test of independence requires two categorical variables, we add the variable size which corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

dat <- iris

dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

We now create a contingency table of the two variables Species and size with the table() function:

table(dat$Species, dat$size)

##             
##              big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

The contingency table gives the observed number of cases in each subgroup. For instance, there is only one big setosa flower, while there are 49 small setosa flowers in the dataset.

It is also a good practice to draw a barplot to visually represent the data:

library(ggplot2)

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar()

If you prefer to visualize it in terms of proportions (so that bars all have a height of 1, or 100%):

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar(position = "fill")

This second barplot is particularly useful if there are a different number of observations in each level of the variable drawn on the $x$-axis because it allows to compare the two variables on the same ground.

If you prefer to have the bars next to each other:

ggplot(dat) +
  aes(x = Species, fill = size) +
  geom_bar(position = "dodge")

See the article “Graphics in R with ggplot2” to learn how to create this kind of barplot in {ggplot2}.

Chi-square test of independence in R

For this example, we are going to test in R if there is a relationship between the variables Species and size. For this, the chisq.test() function is used:

test <- chisq.test(table(dat$Species, dat$size))
test

## 
## 	Pearson's Chi-squared test
## 
## data:  table(dat$Species, dat$size)
## X-squared = 86.035, df = 2, p-value < 2.2e-16

Everything you need appears in this output:

the title of the test,
which variables have been used,
the test statistic,
the degrees of freedom and
the $p$-value of the test.

You can also retrieve the $\chi^2$ test statistic and the $p$-value with:

test$statistic # test statistic

## X-squared 
##  86.03451

test$p.value # p-value

## [1] 2.078944e-19

If you need to find the expected frequencies, use test$expected.

If a warning such as “Chi-squared approximation may be incorrect” appears, it means that the smallest expected frequencies is lower than 5. To avoid this issue, you can either:

gather some levels (especially those with a small number of observations) to increase the number of observations in the subgroups, or
use the Fisher’s exact test

The Fisher’s exact test does not require the assumption of a minimum of 5 expected counts in the contingency table. It can be applied in R thanks to the function fisher.test(). This test is similar to the Chi-square test in terms of hypothesis and interpretation of the results. Learn more about this test in this article dedicated to this type of test.

For your information, there are three other methods to perform the Chi-square test of independence in R:

with the summary() function
with the assocstats() function from the {vcd} package
with the ctable() function from the {summarytools} package

# second method:
summary(table(dat$Species, dat$size))

## Number of cases in table: 150 
## Number of factors: 2 
## Test for independence of all factors:
## 	Chisq = 86.03, df = 2, p-value = 2.079e-19

# third method:
library(vcd)

assocstats(table(dat$Species, dat$size))

##                      X^2 df P(> X^2)
## Likelihood Ratio 107.308  2        0
## Pearson           86.035  2        0
## 
## Phi-Coefficient   : NA 
## Contingency Coeff.: 0.604 
## Cramer's V        : 0.757

library(summarytools)
library(dplyr)

# fourth method:
dat %$%
  ctable(Species, size,
    prop = "r", chisq = TRUE, headings = FALSE
  ) %>%
  print(
    method = "render",
    style = "rmarkdown",
    footnote = NA
  )

As you can see all four methods give the same results.

If you do not have the same p-values with your data across the different methods, make sure to add the correct = FALSE argument in the chisq.test() function to prevent from applying the Yate’s continuity correction, which is applied by default in this method.¹

Conclusion and interpretation

$\Rightarrow$ In our context, rejecting the null hypothesis for the Chi-square test of independence means that there is a significant relationship between the species and the size. Therefore, knowing the value of one variable helps to predict the value of the other variable.

Combination of plot and statistical test

I recently discovered the mosaic() function from the {vcd} package. This function has the advantage that it combines a mosaic plot (to visualize a contingency table) and the result of the Chi-square test of independence:

library(vcd)

mosaic(~ Species + size,
  direction = c("v", "h"),
  data = dat,
  shade = TRUE
)

As you can see, the mosaic plot is similar to the barplot presented above, but the p-value of the Chi-square test is also displayed at the bottom right.

Moreover, this mosaic plot with colored cases shows where the observed frequencies deviates from the expected frequencies if the variables were independent. The red cases means that the observed frequencies are smaller than the expected frequencies, whereas the blue cases means that the observed frequencies are larger than the expected frequencies.

An alternative is the ggbarstats() function from the {ggstatsplot} package:

# load packages
library(ggstatsplot)
library(ggplot2)

# plot
ggbarstats(
  data = dat,
  x = size,
  y = Species
) +
  labs(caption = NULL) # remove caption

From the plot, it seems that big flowers are more likely to belong to the virginica species, while small flowers tend to belong to the setosa species. Species and size are thus expected to be dependent.

This is confirmed thanks to the statistical results displayed in the subtitle of the plot. There are several results, but we can in this case focus on the $p$-value which is displayed after p = at the top (in the subtitle of the plot).

As with the previous tests, we reject the null hypothesis and we conclude that species and size are dependent ($p$-value < 0.001).

Thanks for reading. I hope the article helped you to perform the Chi-square test of independence in R and interpret its results. If you would like to learn how to do this test by hand and how it works, read the article “Chi-square test of independence by hand”. If you want to go further and estimate the strength of the relationship between two categorical variables, see the binary logistic regression.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

Thanks Herivelto for pointing it out.↩︎

Descriptive statistics in R

Wed, 22 Jan 2020 00:00:00 +0000

Introduction

This article explains how to compute the main descriptive statistics in R and how to present them graphically. To learn more about the reasoning behind each descriptive statistics, how to compute them by hand and how to interpret them, read the article “Descriptive statistics by hand”.

To briefly recap what have been said in that article, descriptive statistics (in the broad sense of the term) is a branch of statistics aiming at summarizing, describing and presenting a series of values or a dataset. Descriptive statistics is often the first step and an important part in any statistical analysis. It allows to check the quality of the data and it helps to “understand” the data by having a clear overview of it. If well presented, descriptive statistics is already a good starting point for further analyses. There exists many measures to summarize a dataset. They are divided into two types:

location measures and
dispersion measures

Location measures give an understanding about the central tendency of the data, whereas dispersion measures give an understanding about the spread of the data. In this article, we focus only on the implementation in R of the most common descriptive statistics and their visualizations (when deemed appropriate). See online or in the above mentioned article for more information about the purpose and usage of each measure.

Data

We use the dataset iris throughout the article. This dataset is imported by default in R, you only need to load it by running iris:

dat <- iris # load the iris dataset and renamed it dat

Below a preview of this dataset and its structure:

head(dat) # first 6 observations

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa

str(dat) # structure of dataset

## 'data.frame':	150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

The dataset contains 150 observations and 5 variables, representing the length and width of the sepal and petal and the species of 150 flowers. Length and width of the sepal and petal are numeric variables and the species is a factor with 3 levels (indicated by num and Factor w/ 3 levels after the name of the variables). See the different variables types in R if you need a refresh.

Regarding plots, we present the default graphs and the graphs from the well-known {ggplot2} package. Graphs from the {ggplot2} package usually have a better look but it requires more advanced coding skills (see the article “Graphics in R with ggplot2” to learn more). If you need to publish or share your graphs, I suggest using {ggplot2} if you can, otherwise the default graphics will do the job.

Tip: I recently discovered the ggplot2 builder from the {esquisse} addins. See how you can easily draw graphs from the {ggplot2} package without having to code it yourself.

All plots displayed in this article can be customized. For instance, it is possible to edit the title, x and y-axis labels, color, etc. However, customizing plots is beyond the scope of this article so all plots are presented without any customization. Interested readers will find numerous resources online.

Minimum and maximum

Minimum and maximum can be found thanks to the min() and max() functions:

min(dat$Sepal.Length)

## [1] 4.3

max(dat$Sepal.Length)

## [1] 7.9

Alternatively the range() function:

rng <- range(dat$Sepal.Length)
rng

## [1] 4.3 7.9

gives you the minimum and maximum directly. Note that the output of the range() function is actually an object containing the minimum and maximum (in that order). This means you can actually access the minimum with:

rng[1] # rng = name of the object specified above

## [1] 4.3

and the maximum with:

rng[2]

## [1] 7.9

This reminds us that, in R, there are often several ways to arrive at the same result. The method that uses the shortest piece of code is usually preferred as a shorter piece of code is less prone to coding errors and more readable.

Range

The range can then be easily computed, as you have guessed, by subtracting the minimum from the maximum:

max(dat$Sepal.Length) - min(dat$Sepal.Length)

## [1] 3.6

To my knowledge, there is no default function to compute the range. However, if you are familiar with writing functions in R , you can create your own function to compute the range:

range2 <- function(x) {
  range <- max(x) - min(x)
  return(range)
}

range2(dat$Sepal.Length)

## [1] 3.6

which is equivalent than $max - min$ presented above.

Mean

The mean can be computed with the mean() function:

mean(dat$Sepal.Length)

## [1] 5.843333

Tips:

if there is at least one missing value in your dataset, use mean(dat$Sepal.Length, na.rm = TRUE) to compute the mean with the NA excluded. This argument can be used for most functions presented in this article, not only the mean
for a truncated mean, use mean(dat$Sepal.Length, trim = 0.10) and change the trim argument to your needs

Median

The median can be computed thanks to the median() function:

median(dat$Sepal.Length)

## [1] 5.8

or with the quantile() function:

quantile(dat$Sepal.Length, 0.5)

## 50% 
## 5.8

since the quantile of order 0.5 ($q_{0.5}$) corresponds to the median.

First and third quartile

As the median, the first and third quartiles can be computed thanks to the quantile() function and by setting the second argument to 0.25 or 0.75:

quantile(dat$Sepal.Length, 0.25) # first quartile

## 25% 
## 5.1

quantile(dat$Sepal.Length, 0.75) # third quartile

## 75% 
## 6.4

You may have seen that the results above are slightly different than the results you would have found if you compute the first and third quartiles by hand. It is normal, there are many methods to compute them (R actually has 7 methods to compute the quantiles!). However, the methods presented here and in the article “descriptive statistics by hand” are the easiest and most “standard” ones. Furthermore, results do not dramatically change between the two methods.

Other quantiles

As you have guessed, any quantile can also be computed with the quantile() function. For instance, the $4^{th}$ decile or the $98^{th}$ percentile:

quantile(dat$Sepal.Length, 0.4) # 4th decile

## 40% 
## 5.6

quantile(dat$Sepal.Length, 0.98) # 98th percentile

## 98% 
## 7.7

Interquartile range

The interquartile range (i.e., the difference between the first and third quartile) can be computed with the IQR() function:

IQR(dat$Sepal.Length)

## [1] 1.3

or alternatively with the quantile() function again:

quantile(dat$Sepal.Length, 0.75) - quantile(dat$Sepal.Length, 0.25)

## 75% 
## 1.3

As mentioned earlier, when possible it is usually recommended to use the shortest piece of code to arrive at the result. For this reason, the IQR() function is preferred to compute the interquartile range.

Standard deviation and variance

The standard deviation and the variance is computed with the sd() and var() functions:

sd(dat$Sepal.Length) # standard deviation

## [1] 0.8280661

var(dat$Sepal.Length) # variance

## [1] 0.6856935

Remember from the article descriptive statistics by hand that the standard deviation and the variance are different whether we compute it for a sample or a population (see the difference between sample and population). In R, the standard deviation and the variance are computed as if the data represent a sample (so the denominator is $n - 1$, where $n$ is the number of observations). To my knowledge, there is no function by default in R that computes the standard deviation or variance for a population.

Tip: to compute the standard deviation (or variance) of multiple variables at the same time, use lapply() with the appropriate statistics as second argument:

lapply(dat[, 1:4], sd)

## $Sepal.Length
## [1] 0.8280661
## 
## $Sepal.Width
## [1] 0.4358663
## 
## $Petal.Length
## [1] 1.765298
## 
## $Petal.Width
## [1] 0.7622377

The command dat[, 1:4] selects the variables 1 to 4 as the fifth variable is a qualitative variable and the standard deviation cannot be computed on such type of variable. See a recap of the different data types in R if needed.

Summary

You can compute the minimum, $1^{st}$ quartile, median, mean, $3^{rd}$ quartile and the maximum for all numeric variables of a dataset at once using summary():

summary(dat)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
##

Tip: if you need these descriptive statistics by group use the by() function:

by(dat, dat$Species, summary)

## dat$Species: setosa
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.300   Min.   :1.000   Min.   :0.100  
##  1st Qu.:4.800   1st Qu.:3.200   1st Qu.:1.400   1st Qu.:0.200  
##  Median :5.000   Median :3.400   Median :1.500   Median :0.200  
##  Mean   :5.006   Mean   :3.428   Mean   :1.462   Mean   :0.246  
##  3rd Qu.:5.200   3rd Qu.:3.675   3rd Qu.:1.575   3rd Qu.:0.300  
##  Max.   :5.800   Max.   :4.400   Max.   :1.900   Max.   :0.600  
##        Species  
##  setosa    :50  
##  versicolor: 0  
##  virginica : 0  
##                 
##                 
##                 
## ------------------------------------------------------------ 
## dat$Species: versicolor
##   Sepal.Length    Sepal.Width     Petal.Length   Petal.Width          Species  
##  Min.   :4.900   Min.   :2.000   Min.   :3.00   Min.   :1.000   setosa    : 0  
##  1st Qu.:5.600   1st Qu.:2.525   1st Qu.:4.00   1st Qu.:1.200   versicolor:50  
##  Median :5.900   Median :2.800   Median :4.35   Median :1.300   virginica : 0  
##  Mean   :5.936   Mean   :2.770   Mean   :4.26   Mean   :1.326                  
##  3rd Qu.:6.300   3rd Qu.:3.000   3rd Qu.:4.60   3rd Qu.:1.500                  
##  Max.   :7.000   Max.   :3.400   Max.   :5.10   Max.   :1.800                  
## ------------------------------------------------------------ 
## dat$Species: virginica
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.900   Min.   :2.200   Min.   :4.500   Min.   :1.400  
##  1st Qu.:6.225   1st Qu.:2.800   1st Qu.:5.100   1st Qu.:1.800  
##  Median :6.500   Median :3.000   Median :5.550   Median :2.000  
##  Mean   :6.588   Mean   :2.974   Mean   :5.552   Mean   :2.026  
##  3rd Qu.:6.900   3rd Qu.:3.175   3rd Qu.:5.875   3rd Qu.:2.300  
##  Max.   :7.900   Max.   :3.800   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    : 0  
##  versicolor: 0  
##  virginica :50  
##                 
##                 
##

where the arguments are the name of the dataset, the grouping variable and the summary function. Follow this order, or specify the name of the arguments if you do not follow this order.

If you need more descriptive statistics, use stat.desc() from the package {pastecs}:

library(pastecs)
stat.desc(dat)

##              Sepal.Length  Sepal.Width Petal.Length  Petal.Width Species
## nbr.val      150.00000000 150.00000000  150.0000000 150.00000000      NA
## nbr.null       0.00000000   0.00000000    0.0000000   0.00000000      NA
## nbr.na         0.00000000   0.00000000    0.0000000   0.00000000      NA
## min            4.30000000   2.00000000    1.0000000   0.10000000      NA
## max            7.90000000   4.40000000    6.9000000   2.50000000      NA
## range          3.60000000   2.40000000    5.9000000   2.40000000      NA
## sum          876.50000000 458.60000000  563.7000000 179.90000000      NA
## median         5.80000000   3.00000000    4.3500000   1.30000000      NA
## mean           5.84333333   3.05733333    3.7580000   1.19933333      NA
## SE.mean        0.06761132   0.03558833    0.1441360   0.06223645      NA
## CI.mean.0.95   0.13360085   0.07032302    0.2848146   0.12298004      NA
## var            0.68569351   0.18997942    3.1162779   0.58100626      NA
## std.dev        0.82806613   0.43586628    1.7652982   0.76223767      NA
## coef.var       0.14171126   0.14256420    0.4697441   0.63555114      NA

You can have even more statistics (i.e., skewness, kurtosis and normality test) by adding the argument norm = TRUE in the previous function. Note that the variable Species is not numeric, so descriptive statistics cannot be computed for this variable and NA are displayed.

Coefficient of variation

The coefficient of variation can be found with stat.desc() (see the line coef.var in the table above) or by computing manually (remember that the coefficient of variation is the standard deviation divided by the mean):

sd(dat$Sepal.Length) / mean(dat$Sepal.Length)

## [1] 0.1417113

Mode

To my knowledge there is no function to find the mode of a variable. However, we can easily find it thanks to the functions table() and sort():

tab <- table(dat$Sepal.Length) # number of occurrences for each unique value
sort(tab, decreasing = TRUE) # sort highest to lowest

## 
##   5 5.1 6.3 5.7 6.7 5.5 5.8 6.4 4.9 5.4 5.6   6 6.1 4.8 6.5 4.6 5.2 6.2 6.9 7.7 
##  10   9   9   8   8   7   7   7   6   6   6   6   6   5   5   4   4   4   4   4 
## 4.4 5.9 6.8 7.2 4.7 6.6 4.3 4.5 5.3   7 7.1 7.3 7.4 7.6 7.9 
##   3   3   3   3   2   2   1   1   1   1   1   1   1   1   1

table() gives the number of occurrences for each unique value, then sort() with the argument decreasing = TRUE displays the number of occurrences from highest to lowest. The mode of the variable Sepal.Length is thus 5. This code to find the mode can also be applied to qualitative variables such as Species:

sort(table(dat$Species), decreasing = TRUE)

## 
##     setosa versicolor  virginica 
##         50         50         50

or:

summary(dat$Species)

##     setosa versicolor  virginica 
##         50         50         50

Correlation

Another descriptive statistics is the correlation coefficient.

The correlation measures the linear relationship between two variables, and it can be computed with the cor() function:

cor(dat$Sepal.Length, dat$Sepal.Width)

## [1] -0.1175698

Computing correlation in R and interpreting the results deserve a detailed explanation, so I wrote an article covering correlation and correlation test.

Contingency table

table() introduced above can also be used on two qualitative variables to create a contingency table. The dataset iris has only one qualitative variable so we create a new qualitative variable just for this example. We create the variable size which corresponds to small if the length of the petal is smaller than the median of all flowers, big otherwise:

dat$size <- ifelse(dat$Sepal.Length < median(dat$Sepal.Length),
  "small", "big"
)

Here is a recap of the occurrences by size:

table(dat$size)

## 
##   big small 
##    77    73

We now create a contingency table of the two variables Species and size with the table() function:

table(dat$Species, dat$size)

##             
##              big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

or with the xtabs() function:

xtabs(~ dat$Species + dat$size)

##             dat$size
## dat$Species  big small
##   setosa       1    49
##   versicolor  29    21
##   virginica   47     3

The contingency table gives the number of cases in each subgroup. For instance, there is only one big setosa flower, while there are 49 small setosa flowers in the dataset.

To go further, we can see from the table that setosa flowers seem to be smaller in size than virginica flowers. In order to check whether size is significantly associated with species, we could perform a Chi-square test of independence since both variables are categorical variables. See how to do this test by hand and in R.

Note that Species are in rows and size in column because we specified Species and then size in table(). Change the order if you want to switch the two variables.

Instead of having the frequencies (i.e.. the number of cases) you can also have the relative frequencies (i.e., proportions) in each subgroup by adding the table() function inside the prop.table() function:

prop.table(table(dat$Species, dat$size))

##             
##                      big       small
##   setosa     0.006666667 0.326666667
##   versicolor 0.193333333 0.140000000
##   virginica  0.313333333 0.020000000

Note that you can also compute the percentages by row or by column by adding a second argument to the prop.table() function: 1 for row, or 2 for column:

# percentages by row:
round(prop.table(table(dat$Species, dat$size), 1), 2) # round to 2 digits with round()

##             
##               big small
##   setosa     0.02  0.98
##   versicolor 0.58  0.42
##   virginica  0.94  0.06

# percentages by column:
round(prop.table(table(dat$Species, dat$size), 2), 2) # round to 2 digits with round()

##             
##               big small
##   setosa     0.01  0.67
##   versicolor 0.38  0.29
##   virginica  0.61  0.04

See the section on advanced descriptive statistics for more advanced contingency tables.

Mosaic plot

A mosaic plot allows to visualize a contingency table of two qualitative variables:

mosaicplot(table(dat$Species, dat$size),
  color = TRUE,
  xlab = "Species", # label for x-axis
  ylab = "Size" # label for y-axis
)

The mosaic plot shows that, for our sample, the proportion of big and small flowers is clearly different between the three species. In particular, the virginica species is the biggest, and the setosa species is the smallest of the three species (in terms of sepal length since the variable size is based on the variable Sepal.Length).

For your information, a mosaic plot can also be done via the mosaic() function from the {vcd} package:

library(vcd)

mosaic(~ Species + size,
  data = dat,
  direction = c("v", "h")
)

Barplot

Barplots can only be done on qualitative variables (see the difference with a quantitative variable here). A barplot is a tool to visualize the distribution of a qualitative variable. We draw a barplot of the qualitative variable size:

barplot(table(dat$size)) # table() is mandatory

You can also draw a barplot of the relative frequencies instead of the frequencies by adding prop.table() as we did earlier:

barplot(prop.table(table(dat$size)))

In {ggplot2}:

library(ggplot2) # needed each time you open RStudio
# The package ggplot2 must be installed first

ggplot(dat) +
  aes(x = size) +
  geom_bar()

Histogram

A histogram gives an idea about the distribution of a quantitative variable. The idea is to break the range of values into intervals and count how many observations fall into each interval. Histograms are a bit similar to barplots, but histograms are used for quantitative variables whereas barplots are used for qualitative variables. To draw a histogram in R, use hist():

hist(dat$Sepal.Length)

Add the arguments breaks = inside the hist() function if you want to change the number of bins. A rule of thumb (known as the square-root rule) is that the number of bins should be the rounded value of the square root of the number of observations. The dataset includes 150 observations so in this case the number of bins can be set to 12.

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length) +
  geom_histogram()

By default, the number of bins is 30. You can change this value with geom_histogram(bins = 12) for instance.

Boxplot

Boxplots are really useful in descriptive statistics and are often underused (mostly because it is not well understood by the public). A boxplot graphically represents the distribution of a quantitative variable by visually displaying five common location summary (minimum, median, first/third quartiles and maximum) and any observation that was classified as a suspected outlier using the interquartile range (IQR) criterion.

The IQR criterion means that all observations above $q_{0.75} + 1.5 \cdot IQR$ or below $q_{0.25} - 1.5 \cdot IQR$ (where $q_{0.25}$ and $q_{0.75}$ correspond to first and third quartile respectively) are considered as potential outliers by R. The minimum and maximum in the boxplot are represented without these suspected outliers.

Seeing all these information on the same plot help to have a good first overview of the dispersion and the location of the data. Before drawing a boxplot of our data, see below a graph explaining the information present on a boxplot:

How to interpret a boxplot? Source: LFSAB1105

Now an example with our dataset:

boxplot(dat$Sepal.Length)

Boxplots are even more informative when presented side-by-side for comparing and contrasting distributions from two or more groups. For instance, we compare the length of the sepal across the different species:

boxplot(dat$Sepal.Length ~ dat$Species)

In {ggplot2}:

ggplot(dat) +
  aes(x = Species, y = Sepal.Length) +
  geom_boxplot()

Dotplot

A dotplot is more or less similar than a boxplot, except that:

observations are represented as points
it does not easily tell us about the median, first and third quartiles.

library(lattice)

dotplot(dat$Sepal.Length ~ dat$Species)

In {ggplot2}:

ggplot(dat) +
  aes(x = Species, y = Sepal.Length) +
  geom_dotplot(binaxis = "y", stackdir = "center")

The advantage of using {ggplot2} over {lattice} for this plot is that we can easily see the mode.

Note that a dotplot is particularly useful when there are a limited number of observations, whereas a boxplot is more appropriate with large datasets.

Scatterplot

Scatterplots allow to check whether there is a potential link between two quantitative variables. For this reason, scatterplots are often used to visualize a potential correlation between two variables. For instance, when drawing a scatterplot of the length of the sepal and the length of the petal:

plot(dat$Sepal.Length, dat$Petal.Length)

There seems to be a positive association between the two variables.

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length) +
  geom_point()

Like boxplots, scatterplots are even more informative when differentiating the points according to a factor, in this case the species:

ggplot(dat) +
  aes(x = Sepal.Length, y = Petal.Length, colour = Species) +
  geom_point() +
  scale_color_hue()

Line plot

Line plots, particularly useful in time series or finance, can be created by adding the type = "l" argument in the plot() function:

plot(dat$Sepal.Length,
  type = "l"
) # "l" for line

QQ-plot

For a single variable

In order to check the normality assumption of a variable (normality means that the data follow a normal distribution, also known as a Gaussian distribution), we usually use histograms and/or QQ-plots.¹ See an article discussing about the normal distribution and how to evaluate the normality assumption in R if you need a refresh on that subject.

Histograms have been presented earlier, so here is how to draw a QQ-plot:

# Draw points on the qq-plot:
qqnorm(dat$Sepal.Length)
# Draw the reference line:
qqline(dat$Sepal.Length)

Or a QQ-plot with confidence bands with the qqPlot() function from the {car} package:

library(car) # package must be installed first
qqPlot(dat$Sepal.Length)

## [1] 132 118

If points are close to the reference line (sometimes referred as Henry’s line) and within the confidence bands, the normality assumption can be considered as met. The bigger the deviation between the points and the reference line and the more they lie outside the confidence bands, the less likely that the normality condition is met. The variable Sepal.Length does not seem to follow a normal distribution because several points lie outside the confidence bands. When facing a non-normal distribution, the first step is usually to apply the logarithm transformation on the data and recheck to see whether the log-transformed data are normally distributed. Applying the logarithm transformation can be done with the log() function.

In {ggpubr}:

library(ggpubr)
ggqqplot(dat$Sepal.Length)

By groups

For some statistical tests, the normality assumption is required in all groups. One solution is to draw a QQ-plot for each group by manually splitting the dataset into different groups and then draw a QQ-plot for each subset of the data (with the methods shown above). Another (easier) solution is to draw a QQ-plot for each group automatically with the argument groups = in the function qqPlot() from the {car} package:

qqPlot(dat$Sepal.Length, groups = dat$size)

In {ggplot2}:

qplot(
  sample = Sepal.Length, data = dat,
  col = size, shape = size
)

It is also possible to differentiate groups by only shape or color. For this, remove one of the argument col or shape in the qplot() function above.

Density plot

Density plot is a smoothed version of the histogram and is used in the same concept, that is, to represent the distribution of a numeric variable. The functions plot() and density() are used together to draw a density plot:

plot(density(dat$Sepal.Length))

In {ggplot2}:

ggplot(dat) +
  aes(x = Sepal.Length) +
  geom_density()

Correlation plot

The last type of descriptive plot is a correlation plot, also called a correlogram. This type of graph is more complex than the ones presented above, so it is detailed in a separate article. See how to draw a correlogram to highlight the most correlated variables in a dataset.

Advanced descriptive statistics

We covered the main functions to compute the most common and basic descriptive statistics. There are, however, many more functions and packages to perform more advanced descriptive statistics in R. In this section, I present some of them with applications to our dataset.

`{summarytools}` package

One package for descriptive statistics I often use for my projects in R is the {summarytools} package. The package is centered around 4 functions:

freq() for frequencies tables
ctable() for cross-tabulations
descr() for descriptive statistics
dfSummary() for dataframe summaries

A combination of these 4 functions is usually more than enough for most descriptive analyses. Moreover, the package has been built with R Markdown in mind, meaning that outputs render well in HTML reports. And for non-English speakers, built-in translations exist for French, Portuguese, Spanish, Russian and Turkish.

I illustrate each of the 4 functions in the following sections. Outputs that follow display much better in R Markdown reports, but in this article I limit myself to the raw outputs as the goal is to show how the functions work, not how to make them render well. See the setup settings in the vignette of the package if you want to print the outputs in a nice way in R Markdown.²

Frequency tables with `freq()`

The freq() function produces frequency tables with frequencies, proportions, as well as missing data information.

library(summarytools)
freq(dat$Species)

## Frequencies  
## dat$Species  
## Type: Factor  
## 
##                    Freq   % Valid   % Valid Cum.   % Total   % Total Cum.
## ---------------- ------ --------- -------------- --------- --------------
##           setosa     50     33.33          33.33     33.33          33.33
##       versicolor     50     33.33          66.67     33.33          66.67
##        virginica     50     33.33         100.00     33.33         100.00
##             <NA>      0                               0.00         100.00
##            Total    150    100.00         100.00    100.00         100.00

If you do not need information about missing values, add the report.nas = FALSE argument:

freq(dat$Species,
  report.nas = FALSE # remove NA information
)

## Frequencies  
## dat$Species  
## Type: Factor  
## 
##                    Freq        %   % Cum.
## ---------------- ------ -------- --------
##           setosa     50    33.33    33.33
##       versicolor     50    33.33    66.67
##        virginica     50    33.33   100.00
##            Total    150   100.00   100.00

And for a minimalist output with only counts and proportions:

freq(dat$Species,
  report.nas = FALSE, # remove NA information
  totals = FALSE, # remove totals
  cumul = FALSE, # remove cumuls
  headings = FALSE # remove headings
)

## 
##                    Freq       %
## ---------------- ------ -------
##           setosa     50   33.33
##       versicolor     50   33.33
##        virginica     50   33.33

Cross-tabulations with `ctable()`

The ctable() function produces cross-tabulations (also known as contingency tables) for pairs of categorical variables. Using the two categorical variables in our dataset:

ctable(
  x = dat$Species,
  y = dat$size
)

## Cross-Tabulation, Row Proportions  
## Species * size  
## Data Frame: dat  
## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 2.0%)   49 (98.0%)    50 (100.0%)
##   versicolor          29 (58.0%)   21 (42.0%)    50 (100.0%)
##    virginica          47 (94.0%)    3 ( 6.0%)    50 (100.0%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------

Row proportions are shown by default. To display column or total proportions, add the prop = "c" or prop = "t" arguments, respectively:

ctable(
  x = dat$Species,
  y = dat$size,
  prop = "t" # total proportions
)

## Cross-Tabulation, Total Proportions  
## Species * size  
## Data Frame: dat  
## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 0.7%)   49 (32.7%)    50 ( 33.3%)
##   versicolor          29 (19.3%)   21 (14.0%)    50 ( 33.3%)
##    virginica          47 (31.3%)    3 ( 2.0%)    50 ( 33.3%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------

To remove proportions altogether, add the argument prop = "n". Furthermore, to display only the bare minimum, add the totals = FALSE and headings = FALSE arguments:

ctable(
  x = dat$Species,
  y = dat$size,
  prop = "n", # remove proportions
  totals = FALSE, # remove totals
  headings = FALSE # remove headings
)

## 
## ------------ ------ ----- -------
##                size   big   small
##      Species                     
##       setosa            1      49
##   versicolor           29      21
##    virginica           47       3
## ------------ ------ ----- -------

This is equivalent than table(dat$Species, dat$size) and xtabs(~ dat$Species + dat$size) performed in the section on contingency tables.

To display results of the Chi-square test of independence, add the chisq = TRUE argument:³

ctable(
  x = dat$Species,
  y = dat$size,
  chisq = TRUE, # display results of Chi-square test of independence
  headings = FALSE # remove headings
)

## 
## ------------ ------ ------------ ------------ --------------
##                size          big        small          Total
##      Species                                                
##       setosa           1 ( 2.0%)   49 (98.0%)    50 (100.0%)
##   versicolor          29 (58.0%)   21 (42.0%)    50 (100.0%)
##    virginica          47 (94.0%)    3 ( 6.0%)    50 (100.0%)
##        Total          77 (51.3%)   73 (48.7%)   150 (100.0%)
## ------------ ------ ------------ ------------ --------------
## 
## ----------------------------
##  Chi.squared   df   p.value 
## ------------- ---- ---------
##    86.0345     2       0    
## ----------------------------

The p-value is close to 0 so we reject the null hypothesis of independence between the two variables. In our context, this indicates that species and size are dependent and that there is a significant relationship between the two variables.

It is also possible to create a contingency table for each level of a third categorical variable thanks to the combination of the stby() and ctable() functions. There are only 2 categorical variables in our dataset, so let’s use the tabacco dataset which has 4 categorical variables (i.e., gender, age group, smoker, diseased). For this example, we would like to create a contingency table of the variables smoker and diseased, and this for each gender:

stby(
  list(
    x = tobacco$smoker, # smoker and diseased
    y = tobacco$diseased
  ),
  INDICES = tobacco$gender, # for each gender
  FUN = ctable # ctable for cross-tabulation
)

## Cross-Tabulation, Row Proportions  
## smoker * diseased  
## Data Frame: tobacco  
## Group: gender = F  
## 
## -------- ---------- ------------- ------------- --------------
##            diseased           Yes            No          Total
##   smoker                                                      
##      Yes               62 (42.2%)    85 (57.8%)   147 (100.0%)
##       No               49 (14.3%)   293 (85.7%)   342 (100.0%)
##    Total              111 (22.7%)   378 (77.3%)   489 (100.0%)
## -------- ---------- ------------- ------------- --------------
## 
## Group: gender = M  
## 
## -------- ---------- ------------- ------------- --------------
##            diseased           Yes            No          Total
##   smoker                                                      
##      Yes               63 (44.1%)    80 (55.9%)   143 (100.0%)
##       No               47 (13.6%)   299 (86.4%)   346 (100.0%)
##    Total              110 (22.5%)   379 (77.5%)   489 (100.0%)
## -------- ---------- ------------- ------------- --------------

Descriptive statistics with `descr()`

The descr() function produces descriptive (univariate) statistics with common central tendency statistics and measures of dispersion. (See the difference between a measure of central tendency and dispersion if you need a reminder.)

A major advantage of this function is that it accepts single vectors as well as data frames. If a data frame is provided, all non-numerical columns are ignored so you do not have to remove them yourself before running the function.

The descr() function allows to display:

only a selection of descriptive statistics of your choice, with the stats = c("mean", "sd") argument for mean and standard deviation for example
the minimum, first quartile, median, third quartile and maximum with stats = "fivenum"
the most common descriptive statistics (mean, standard deviation, minimum, median, maximum, number and percentage of valid observations), with stats = "common":

descr(dat,
  headings = FALSE, # remove headings
  stats = "common" # most common descriptive statistics
)

## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           3.76          1.20           5.84          3.06
##         Std.Dev           1.77          0.76           0.83          0.44
##             Min           1.00          0.10           4.30          2.00
##          Median           4.35          1.30           5.80          3.00
##             Max           6.90          2.50           7.90          4.40
##         N.Valid         150.00        150.00         150.00        150.00
##               N         150.00        150.00         150.00        150.00
##       Pct.Valid         100.00        100.00         100.00        100.00

Tip: if you have a large number of variables, add the transpose = TRUE argument for a better display.

In order to compute these descriptive statistics by group (e.g., Species in our dataset), use the descr() function in combination with the stby() function:

stby(
  data = dat,
  INDICES = dat$Species, # by Species
  FUN = descr, # descriptive statistics
  stats = "common" # most common descr. stats
)

## Descriptive Statistics  
## dat  
## Group: Species = setosa  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           1.46          0.25           5.01          3.43
##         Std.Dev           0.17          0.11           0.35          0.38
##             Min           1.00          0.10           4.30          2.30
##          Median           1.50          0.20           5.00          3.40
##             Max           1.90          0.60           5.80          4.40
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00
## 
## Group: Species = versicolor  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           4.26          1.33           5.94          2.77
##         Std.Dev           0.47          0.20           0.52          0.31
##             Min           3.00          1.00           4.90          2.00
##          Median           4.35          1.30           5.90          2.80
##             Max           5.10          1.80           7.00          3.40
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00
## 
## Group: Species = virginica  
## N: 50  
## 
##                   Petal.Length   Petal.Width   Sepal.Length   Sepal.Width
## --------------- -------------- ------------- -------------- -------------
##            Mean           5.55          2.03           6.59          2.97
##         Std.Dev           0.55          0.27           0.64          0.32
##             Min           4.50          1.40           4.90          2.20
##          Median           5.55          2.00           6.50          3.00
##             Max           6.90          2.50           7.90          3.80
##         N.Valid          50.00         50.00          50.00         50.00
##               N          50.00         50.00          50.00         50.00
##       Pct.Valid         100.00        100.00         100.00        100.00

Data frame summaries with `dfSummary()`

The dfSummary() function generates a summary table with statistics, frequencies and graphs for all variables in a dataset. The information shown depends on the type of the variables (character, factor, numeric, date) and also varies according to the number of distinct values.

dfSummary(dat)

## Data Frame Summary  
## dat  
## Dimensions: 150 x 6  
## Duplicates: 1  
## 
## -----------------------------------------------------------------------------------------------------------
## No   Variable       Stats / Values          Freqs (% of Valid)   Graph                 Valid      Missing  
## ---- -------------- ----------------------- -------------------- --------------------- ---------- ---------
## 1    Sepal.Length   Mean (sd) : 5.8 (0.8)   35 distinct values     . . : :             150        0        
##      [numeric]      min < med < max:                               : : : :             (100.0%)   (0.0%)   
##                     4.3 < 5.8 < 7.9                                : : : : :                               
##                     IQR (CV) : 1.3 (0.1)                           : : : : :                               
##                                                                  : : : : : : : :                           
## 
## 2    Sepal.Width    Mean (sd) : 3.1 (0.4)   23 distinct values           :             150        0        
##      [numeric]      min < med < max:                                     :             (100.0%)   (0.0%)   
##                     2 < 3 < 4.4                                        . :                                 
##                     IQR (CV) : 0.5 (0.1)                             : : : :                               
##                                                                  . . : : : : : :                           
## 
## 3    Petal.Length   Mean (sd) : 3.8 (1.8)   43 distinct values   :                     150        0        
##      [numeric]      min < med < max:                             :         . :         (100.0%)   (0.0%)   
##                     1 < 4.3 < 6.9                                :         : : .                           
##                     IQR (CV) : 3.5 (0.5)                         : :       : : : .                         
##                                                                  : :   . : : : : : .                       
## 
## 4    Petal.Width    Mean (sd) : 1.2 (0.8)   22 distinct values   :                     150        0        
##      [numeric]      min < med < max:                             :                     (100.0%)   (0.0%)   
##                     0.1 < 1.3 < 2.5                              :       . .   :                           
##                     IQR (CV) : 1.5 (0.6)                         :       : :   :   .                       
##                                                                  : :   : : : . : : :                       
## 
## 5    Species        1. setosa               50 (33.3%)           IIIIII                150        0        
##      [factor]       2. versicolor           50 (33.3%)           IIIIII                (100.0%)   (0.0%)   
##                     3. virginica            50 (33.3%)           IIIIII                                    
## 
## 6    size           1. big                  77 (51.3%)           IIIIIIIIII            150        0        
##      [character]    2. small                73 (48.7%)           IIIIIIIII             (100.0%)   (0.0%)   
## -----------------------------------------------------------------------------------------------------------

`describeBy()` from the `{psych}` package

The describeBy() function from the {psych} package allows to report several summary statistics (i.e., number of valid cases, mean, standard deviation, median, trimmed mean, mad: median absolute deviation (from the median), minimum, maximum, range, skewness and kurtosis) by a grouping variable.

library(psych)
describeBy(
  dat,
  dat$Species # grouping variable
)

## 
##  Descriptive statistics by group 
## group: setosa
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 5.01 0.35    5.0    5.00 0.30 4.3 5.8   1.5  0.11    -0.45
## Sepal.Width     2 50 3.43 0.38    3.4    3.42 0.37 2.3 4.4   2.1  0.04     0.60
## Petal.Length    3 50 1.46 0.17    1.5    1.46 0.15 1.0 1.9   0.9  0.10     0.65
## Petal.Width     4 50 0.25 0.11    0.2    0.24 0.00 0.1 0.6   0.5  1.18     1.26
## Species         5 50 1.00 0.00    1.0    1.00 0.00 1.0 1.0   0.0   NaN      NaN
## size            6 50 1.98 0.14    2.0    2.00 0.00 1.0 2.0   1.0 -6.65    43.12
##                se
## Sepal.Length 0.05
## Sepal.Width  0.05
## Petal.Length 0.02
## Petal.Width  0.01
## Species      0.00
## size         0.02
## ------------------------------------------------------------ 
## group: versicolor
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 5.94 0.52   5.90    5.94 0.52 4.9 7.0   2.1  0.10    -0.69
## Sepal.Width     2 50 2.77 0.31   2.80    2.78 0.30 2.0 3.4   1.4 -0.34    -0.55
## Petal.Length    3 50 4.26 0.47   4.35    4.29 0.52 3.0 5.1   2.1 -0.57    -0.19
## Petal.Width     4 50 1.33 0.20   1.30    1.32 0.22 1.0 1.8   0.8 -0.03    -0.59
## Species         5 50 2.00 0.00   2.00    2.00 0.00 2.0 2.0   0.0   NaN      NaN
## size            6 50 1.42 0.50   1.00    1.40 0.00 1.0 2.0   1.0  0.31    -1.94
##                se
## Sepal.Length 0.07
## Sepal.Width  0.04
## Petal.Length 0.07
## Petal.Width  0.03
## Species      0.00
## size         0.07
## ------------------------------------------------------------ 
## group: virginica
##              vars  n mean   sd median trimmed  mad min max range  skew kurtosis
## Sepal.Length    1 50 6.59 0.64   6.50    6.57 0.59 4.9 7.9   3.0  0.11    -0.20
## Sepal.Width     2 50 2.97 0.32   3.00    2.96 0.30 2.2 3.8   1.6  0.34     0.38
## Petal.Length    3 50 5.55 0.55   5.55    5.51 0.67 4.5 6.9   2.4  0.52    -0.37
## Petal.Width     4 50 2.03 0.27   2.00    2.03 0.30 1.4 2.5   1.1 -0.12    -0.75
## Species         5 50 3.00 0.00   3.00    3.00 0.00 3.0 3.0   0.0   NaN      NaN
## size            6 50 1.06 0.24   1.00    1.00 0.00 1.0 2.0   1.0  3.59    11.15
##                se
## Sepal.Length 0.09
## Sepal.Width  0.05
## Petal.Length 0.08
## Petal.Width  0.04
## Species      0.00
## size         0.03

`aggregate()` function

The aggregate() function allows to split the data into subsets and then to compute summary statistics for each. For instance, if we want to compute the mean for the variables Sepal.Length and Sepal.Width by Species and Size:

aggregate(cbind(Sepal.Length, Sepal.Width) ~ Species + size,
  data = dat,
  mean
)

##      Species  size Sepal.Length Sepal.Width
## 1     setosa   big     5.800000    4.000000
## 2 versicolor   big     6.282759    2.868966
## 3  virginica   big     6.663830    2.997872
## 4     setosa small     4.989796    3.416327
## 5 versicolor small     5.457143    2.633333
## 6  virginica small     5.400000    2.600000

`summaryBy()` from `{doBy}`

An alternative is the summaryBy() function from the {doBy} package:

# summary statistics by group
library(doBy)
summaryBy(Sepal.Length + Sepal.Width ~ Species,
  data = dat,
  FUN = summary
)

##      Species Sepal.Length.Min. Sepal.Length.1st Qu. Sepal.Length.Median
## 1     setosa               4.3                4.800                 5.0
## 2 versicolor               4.9                5.600                 5.9
## 3  virginica               4.9                6.225                 6.5
##   Sepal.Length.Mean Sepal.Length.3rd Qu. Sepal.Length.Max. Sepal.Width.Min.
## 1             5.006                  5.2               5.8              2.3
## 2             5.936                  6.3               7.0              2.0
## 3             6.588                  6.9               7.9              2.2
##   Sepal.Width.1st Qu. Sepal.Width.Median Sepal.Width.Mean Sepal.Width.3rd Qu.
## 1               3.200                3.4            3.428               3.675
## 2               2.525                2.8            2.770               3.000
## 3               2.800                3.0            2.974               3.175
##   Sepal.Width.Max.
## 1              4.4
## 2              3.4
## 3              3.8

If you are interested in some specific descriptive statistics, you can easily specify them via the FUN argument:

summaryBy(Sepal.Length + Sepal.Width ~ Species,
  data = dat,
  FUN = c(mean, var)
)

##      Species Sepal.Length.mean Sepal.Width.mean Sepal.Length.var
## 1     setosa             5.006            3.428        0.1242490
## 2 versicolor             5.936            2.770        0.2664327
## 3  virginica             6.588            2.974        0.4043429
##   Sepal.Width.var
## 1      0.14368980
## 2      0.09846939
## 3      0.10400408

`group_by()` and `summarise()` from `{dplyr}`

Another alternative is with the summarise() and group_by() functions from the {dplyr} package:

library(dplyr)

group_by(dat, Species) %>%
  summarise(
    mean = mean(Sepal.Length, na.rm = TRUE),
    sd = sd(Sepal.Length, na.rm = TRUE)
  )

## # A tibble: 3 × 3
##   Species     mean    sd
##   <fct>      <dbl> <dbl>
## 1 setosa      5.01 0.352
## 2 versicolor  5.94 0.516
## 3 virginica   6.59 0.636

Conclusion

Thanks for reading.

I hope this article helped you to do descriptive statistics in R. If you would like to do the same by hand or understand what these statistics represent, I invite you to read the article “Descriptive statistics by hand”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

(Note that this article is available for download on my Gumroad page.)

Normality tests such as Shapiro-Wilk or Kolmogorov-Smirnov tests can also be used to test whether the data follow a normal distribution or not. However, in practice, normality tests are often considered as too conservative in the sense that for large sample size, a small deviation from the normality may cause the normality condition to be violated. For this reason, it is often the case that the normality condition is verified based on a combination of visual inspections (with histograms and QQ-plots) and formal test (Shapiro-Wilk test for instance).↩︎
Note that the plain.ascii and style arguments are needed for this package. In our examples, these arguments are added in the settings of each chunk so they are not visible.↩︎
Note that it is also possible to compute odds ratio and risk ratio. See the vignette of the package for more information on this matter as these ratios are beyond the scope of this article.↩︎

Descriptive statistics by hand

Sun, 19 Jan 2020 00:00:00 +0000

Introduction

This article explains how to compute the main descriptive statistics by hand and how to interpret them. To learn how to compute these measures in R, read the article “Descriptive statistics in R”.

Descriptive statistics (in the broad sense of the term) is a branch of statistics aiming at summarizing, describing and presenting a series of values or a dataset. Long series of values without any preparation or without any summary measures are often not informative due to the difficulty of recognizing any pattern in the data. Below an example with the height (in cm) of a population of 100 adults:

188.7, 169.4, 178.6, 181.3, 179, 173.9, 190.1, 174.1, 195.2, 174.4, 188, 197.9, 161.1, 172.2, 173.7, 181.4, 172.2, 148.4, 150.6, 188.2, 171.9, 157.2, 173.3, 187.1, 194, 170.7, 172.4, 157.4, 179.6, 168.6, 179.6, 182, 185.4, 168.9, 180, 157.8, 167.2, 166.5, 150.9, 175.4, 177.1, 171.4, 182.6, 167.7, 161.3, 179.3, 166.9, 189.4, 170.7, 181.6, 178.2, 167.2, 190.8, 181.4, 175.9, 177.8, 181.8, 175.9, 145.1, 177.8, 171.3, 176.9, 180.8, 189, 167.7, 188, 178.4, 185.4, 184.2, 182.2, 164.6, 174.1, 181.2, 165.5, 169.6, 180.8, 182.7, 179.6, 166.1, 164, 190.1, 177.6, 175.9, 173.8, 163.1, 181.1, 172.8, 173.2, 184.3, 183.2, 188.9, 170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

Facing this series, it is hard (not to say impossible) for anyone to understand the data and have a clear view of the size of these adults in a reasonable amount of time. Descriptive statistics allow to summarize, and thus have a better overview of the data. Of course, by summarizing data through one or several measures, some information will inevitably be lost. However, in many cases it is generally better to lose some information but in return gain an overview.

Descriptive statistics is often the first step and an important part in any statistical analysis. It allows to check the quality of the data by detecting potential outliers (i.e., data points that appear to be separated from the rest of the data), collection or encoding errors. It also helps to “understand” the data and if well presented, descriptive statistics is already a good starting point for further analyses.

Location versus dispersion measures

Several different measures (called statistics if we are analyzing a sample) are used to summarize the data. Some of them give an understanding about the location of the data, others give and understanding about the dispersion of the data. In practice, both types of measures are often used together in order to summarize the data in the most concise but complete way.

We illustrate this point with the graph below, representing the height (in cm) of 100 persons divided into two groups (50 persons in each group):

The black line corresponds to the mean. The mean height (in cm) is similar in both groups. However, it is clear that the dispersion of heights are very different in the two groups. For this reason, location or dispersion measures are often not enough if presented individually and it is a good practice to present several statistics from both types of measures.

In the following sections, we detail the most common location and dispersion measures and illustrate them with examples. Note that for the sake of simplicity, we consider only series of values (i.e., univariate data) and not bivariate or multivariate data, and we do not consider the case of series grouped in classes.

Location

Location measures allow to see “where” the data are located, around which values. In other words, location measures give an understanding on what is the central tendency, the “position” of the data as a whole. It includes the following statistics (others exist but we focus on the most common ones):

minimum
maximum
mean
median
first quartile
third quartile
mode

We detail and compute by hand each of them in the following sections.

Minimum and maximum

Minimum ($min$) and maximum ($max$) are simply the lowest and largest values, respectively. Given the height (in cm) of a sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

The minimum is 169.4 cm and the maximum is 188.7 cm. These two basic statistics give a clear idea about the size of the smallest and tallest of these 6 adults.

Mean

The mean, also known as average, is probably the most common statistics. It gives an idea on what is the average value, that is, the central value of the data or in other words the center of gravity:¹

The mean is found by summing all values and dividing the total by the number of observations (denoted $n$):

\[mean = \bar{x} = \frac{\text{sum of all values}}{\text{number of values}} = \frac{1}{n}\sum^{n}_{i = 1} x_i\]

Given our sample of 6 adults presented above, the mean is:

\[\bar{x} = \frac{188.7 + 169.4 + 178.6}{6}\] \[\frac{+ 181.3 + 179 + 173.9}{6}\\ = 178.4833\]

In conclusion, the mean size, that is, the average size of our sample of 6 adults is 178.48 cm (rounded to 2 decimals).

Median

The median is another measure of location so it also gives an idea about the central tendency of the data. The interpretation of the median is that there are as many observations below as above the median. In other words, 50% of the observations lie below the median, and 50% of the observations lie above the median. Below a visual representation of the median:

The easiest way to compute the median is by first sorting the data from lowest to highest (i.e., in ascending order) then take the middle point as the median. From the sorted values, for an odd number of observations, the middle point is easy to find: it is the value with as many observations below as above. Still from the sorted values, for an even number of observations, the middle point is exactly between the two middle values. Formally, after sorting, the median is:

if $n$ (number of observations) is odd: \[med(x) = x_{\frac{n+1}{2}}\]
if $n$ is even: \[med(x) = \frac{1}{2}\big(x_{\frac{n}{2}} + x_{\frac{n}{2} + 1}\big)\]

where the subscript of $x$ denotes the numbering of the sorted data. The formulas look harder than they really are, so let’s see with two concrete examples.

Odd number of observations

Given the height of a sample of 7 adults taken from the 100 adults presented in the introduction:

188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

We first sort the order from lowest to highest:

160.4, 163.7, 163.9, 166.4, 175.8, 181.5 and 188.9

Given that the number of observations $n$ is odd (since $n = 7$), the median is

\[med(x) = x_\frac{7 + 1}{2} = x_4\]

So we take the fourth value from the sorted values, which corresponds to 166.4. In conclusion, the median size of these 7 adults is 166.4 cm. As you can see, there are 3 observations below 166.4 and 3 observations above 166.4 cm.

Even number of observations

Now let’s see when the number of observations is even, which is slightly more complicated than when the number of observations is odd. Given the height of a sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

We sort the values in ascending order:

169.4, 173.9, 178.6, 179, 181.3 and 188.7

Given that the number of observations $n$ is even (since $n = 6$), the median is

\[med(x) = \frac{1}{2}\big(x_{\frac{6}{2}} + x_{\frac{6}{2} + 1}\big) = \frac{1}{2}\big(x_{3} + x_{4}\big)\]

So we sum the third and fourth values from the sorted values and divide the total by 2 (which is equivalent than taking the mean of these two middle values):

\[\frac{1}{2}(178.6 + 179) = 178.8\]

In conclusion, the median size of these 6 adults is 178.8 cm. Again, remark that there are as many observations below as above 178.8 cm.

Mean vs. median

Although the mean and median are often relatively close to each other (in particular when the distribution is symmetric) they should not be confused since they both have advantages and disadvantages in different contexts. Besides the fact that almost everyone knows (or at least have heard about) the mean, it has the advantage that it gives a unique picture for each different series of data. However, it has the disadvantage that the mean is sensible to outliers (i.e., extreme values). On the other hand, the advantage of the median is that it is resistant to outliers and the inconvenient is that it may be the exact same value for very different series of data (so not unique to the data).

To illustrate the “sensible to outlier” argument, consider 3 friends in a bar comparing their salaries. Their salaries are 1800, 2000 and 2100€, for an average (mean) salary of 1967€. A friend of them (who happens to be friend with Bill Gates as well) joins them in the bar. Their salaries are now 1800, 2000, 2100 and 1000000€. The average salary of the 4 friends is now 251475€, compared to 1967€ without the rich friend. Although it is statistically correct to say that the mean salary of the 4 friends is 251475€, you will concede that this measure does not represent a fair image of the salaries of the 4 friends, as 3 of them earn much less than the mean salary. As we have just seen, the mean is sensible to outliers. (Note: this example also shows how a large majority of citizens earn less than the mean salary reported in the news. For the french-speaking readers, see this video for more information.)

On the other hand, if we report the medians, we see that the median salary of the 3 first friends is 2000€, and the median salary of the 4 friends is 2050€. As you can see with this example, the median is not sensible to outliers and for series with such extreme value(s), the median is more appropriate compared to the mean as it often gives a better representation of the data.

Given the previous example, one may then choose to always use the median instead of the mean. However, the median has it own inconvenient which the mean does not have: the median is less unique and less specific to its underlying data than the mean. Consider the following data, representing the grades of 5 students taking a statistics and economics exam:

studentID	economics	statistics
1	10	5
2	10	7
3	10	10
4	18	10
5	20	11

The median of the grades is the same in economics and statistics (median = 10). Therefore, had we computed only the medians, we could have concluded that the students performed as well in economics as in statistics. However, although the medians are exactly the same for both classes, it is clear that students performed better in economics than in statistics (compare both grades for each student to see for yourself). In fact, the mean of the grades in economics is 13.6 and the mean of the grades in statistics is 8.6.

What we have just shown here is that the median is based only on one single value, the middle value, or on the two middle values if there are an even number of observations, while the mean is based on all values (and thus includes more information). The median is therefore not sensible to outliers, but it is also not unique (i.e., not specific) to different series of data, whereas the mean is much more likely to be different and unique for different series of data. This difference in terms of specificity and uniqueness between the two measures may make the mean more useful for data with no outlier.

In conclusion, depending on the context and the data, it is often more interesting to report the mean or the median, or both. As a last remark regarding the comparison between the two most important location measures, note that when the mean and median are equal, the distribution of your data can often be considered to follow a normal distribution (also referred as a Gaussian distribution).

$1^{st}$ and $3^{rd}$ quartiles

The first and third quartiles are similar to the median in the sense that they also divide the observations into two parts, except that these parts are not equal. Remind that the median divides the data into two equal parts (with 50% of the observations below and 50% above the median).

The first quartile cuts the observations such that there are 25% of the observations below and thus 75% above the first quartile. The third quartile, as you have guessed by now, represents the value with 75% of the observations below it and thus 25% of the observations above it. There exists several methods to compute the first and third quartile (which sometimes give slight differences, R for instance uses a different method), but here is I believe the easiest one when computing these statistics by hand:

sort the data in ascending order
compute $0.25 \cdot n$ and $0.75 \cdot n$ (i.e., 0.25 and 0.75 times the number of observations)
round up these two numbers to the next whole number
these two numbers represent the rank of the first and third quartile (in the sorted series), respectively

The steps are the same for both an odd and even number of observations. Here is an example with the following series, representing the height in cm of 9 adults:

170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

We first order from lowest to highest:

160.4, 163.7, 163.9, 166.4, 170.2, 175.8, 181.5, 181.5 and 188.9

There are 9 observations so \[0.25 \cdot 9 = 2.25\] and \[0.75 \cdot 9 = 6.75\]

Rounding up to the whole number gives 3 and 7, which represent the rank of the first and third quartiles, respectively. Therefore, the first quartile is 163.9 cm and the third quartile is 181.5 cm.

In conclusion, 25% of adults are less than 163.9 cm tall (and thus 75% of them are more than 163.9 cm tall), while 75% of adults are less than 181.5 cm tall (and thus 25% of them are more than 181.5 cm tall).

$q_{0.25}$, $q_{0.75}$ and $q_{0.5}$

Note that the first quartile is denoted $q_{0.25}$ and the third quartile is denoted $q_{0.75}$ (where $q$ stands for quartile). As you can see, the median is actually the second quartile and for this reason it is also sometimes denoted $q_{0.5}$.

A note on deciles and percentiles

Deciles and percentiles are similar to quartiles except that they cuts the data in 10 and 100 equal parts. For instance, the $4^{th}$ decile ($q_{0.4}$) is the value such that there are 40% of the observations below it and thus 60% of the observations above it.

Percentiles follow the same logic. For example, the $98^{th}$ percentile ($q_{0.98}$, also sometimes denoted $P98$) is the value such that there are 98% of the observations below it and thus 2% of the observations above it. Percentiles are often used for the weight and height of babies, giving precise information to the parents about where their child stands compared to other children of the same age.

Mode

The mode of a series is the value that appears most often. In other words, it is the value that has the highest number of occurrences.

Unlike some descriptive statistics that can only be computed for quantitative variables (the mean for instance), the mode can be computed for quantitative and qualitative variables (see a recap of the different types of variables if you do not remember the difference).

Quantitative variables

Given the height of 9 adults:

170, 168, 171, 170, 182, 165, 170, 189 and 167

The mode is 170 because it is the most common value with 3 occurrences. All other values appear only once.

Note that it is possible that a series has no mode or more than one mode:

the series 4, 7, 2 and 10 has no mode,
the series 4, 2, 2, 8, 11 and 11 has two modes.

Data with two modes are often called bimodal and data with more than two modes are often called multimodal, as opposed to series with one mode which are referred as unimodal.

Qualitative variables

Given the eye color of the 9 adults presented above:

brown, brown, brown, brown, blue, blue, blue, brown and green

Brown is the most frequent color, so the mode is brown.

Dispersion

All previous descriptive statistics helps to get a sense of the location and position of the data. We now present the most common dispersion measures, which help to get a sense of the dispersion and the variability of the data (to which extent a distribution is squeezed or stretched):

range
standard deviation
variance
interquartile range
coefficient of variation

As for location measures, we detail and compute by hand each of these statistics one by one.

Range

The range is the difference between the maximum and the minimum value:

\[range = max - min\]

Given the height (in cm) of our sample of 6 adults:

188.7, 169.4, 178.6, 181.3, 179 and 173.9

The range is 188.7 $-$ 169.4 $=$ 19.3 cm. The advantage of the range is that it is extremely easy to compute it and it gives a precise idea about the “length” of the data. The disadvantage is that it relies on the two most extreme values only, so it is highly sensible to outliers.

Standard deviation

The standard deviation is the most common dispersion measure in statistics. Like the mean for the location measures, if we have to present one statistics which summarizes the spread of the data, it is usually the standard deviation.

As its name suggests, the standard deviation tells what is the “normal” deviation of the data. It actually computes the mean deviation from the global mean. The larger the standard deviation, the more scattered the data are. On the contrary, the smaller the standard deviation, the more the data are centered around the mean. Below a visual representation of the standard deviation:

The standard deviation is a bit more complex than the previous statistics in the sense that there are two formulas depending on whether we face a sample or a population. A population includes all members from a specified group, all possible outcomes or measurements that are of interest. A sample consists of some observations drawn from the population, so a part or a subset of the population. For instance, the population may be “all people living in Belgium” and the sample may be “some people living in Belgium”. Read this article on the difference between population and sample if you want to learn more.

Standard deviation for a population

The standard deviation for a population, denoted $\sigma$, is:

\[\sigma = \sqrt{\frac{1}{n}\sum^n_{i = 1}(x_i - \mu)^2}\]

As you can see from the formula, the standard deviation is actually the mean deviation of the data from the global mean $\mu$. Note the square for the difference between the observations ($x_i$) and the mean ($\mu$) to avoid that negative differences are compensated by positive differences.

For the sake of easiness, imagine a population of only 3 adults (the steps are the same with a large population, the computation is just longer). Below their heights (in cm):

160.4, 175.8 and 181.5

The mean is 172.6 (rounded to 1 decimal). The standard deviation is thus:

\[\sigma = \sqrt{\frac{1}{3} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big]}\] \[\sigma = 8.91\]

In conclusion, the standard deviation for the heights of these 3 adults is 8.91 cm. This means that, on average, the height of the adults in this population deviates from the mean by 8.91 cm.

Standard deviation for a sample

The standard deviation for a sample is similar to the standard deviation for a population except that we divide by $n -1$ instead of $n$ and it is denoted $s$:

\[s = \sqrt{\frac{1}{n-1}\sum^n_{i = 1}(x_i - \bar{x})^2}\]

Imagine now that the 3 adults presented in the previous section is a sample instead of a population:

160.4, 175.8 and 181.5

The mean is still 172.6 (rounded to 1 decimal) since the mean is the same whether it is a population or a sample. The standard deviation is now:

\[s = \sqrt{\frac{1}{3 - 1} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big]}\] \[s = 10.92\]

In conclusion, the standard deviation for the heights of these 3 adults is 10.92 cm. The interpretation is the same than for a population.

Variance

The variance is simply the square of the standard deviation. Put it another way, the standard deviation is the square root of the variance. We also distinguish between the variance for a population and for a sample in the next sections.

Variance for a population

The variance for a population, denoted $\sigma^2$, is:

\[\sigma^2 = \frac{1}{n}\sum^n_{i = 1}(x_i - \mu)^2\]

As you can see, the formula for variance is the same than for standard deviation, except that the square root is removed for the variance. Remember the heights of our population of 3 adults:

160.4, 175.8 and 181.5

The standard deviation was 8.91 cm, so the variance of the height of these adults is $8.91^2 = 79.39$ $cm^2$ (see below why the unit of a variance is $unit^2$).

If you did not know the standard deviation of the population and needed to compute the variance of the population by hand, here is how:

\[\sigma^2 = \frac{1}{3} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big] \\ = 79.43\]

(The difference with the above result is due to rounding.)

Variance for a sample

Again, the variance for a sample is similar to the variance for a population except that we divide by $n - 1$ instead of $n$ and it is denoted $s^2$:

\[s^2 = \frac{1}{n-1}\sum^n_{i = 1}(x_i - \bar{x})^2\]

Imagine again that the 3 adults in the previous section is a sample instead of a population:

160.4, 175.8 and 181.5

The standard deviation for this sample was 10.92 cm, so the variance of the height of these adults is $10.92^2 = 119.25$ $cm^2$.

If you did not know the standard deviation of the sample and needed to compute the variance of the sample by hand, here is how:

\[s^2 = \frac{1}{3 - 1} \big[(160.4 - 172.6)^2 \\ + (175.8 - 172.6)^2 \\ + (181.5 - 172.6)^2 \big] \\ = 119.15\]

(The difference with the above result is due to rounding.)

Standard deviation vs. variance

Standard deviation and variance are often used interchangeably and both quantify the spread of a given dataset by measuring how far the observations are from their mean. However, the standard deviation can be more easily interpreted because the unit for the standard deviation is the same than the unit of measurement of the data (while it is the $unit^2$ for the variance).

Following our example of adult heights in cm, the standard deviation is measured in cm while the variance is measured in $cm^2$. The fact that the standard deviation keeps the same unit than the initial unit of measurement makes it more interpretable and thus more often used in practice.

Notations

For completeness, below a table showing the different notations for variance and standard deviation in case of population and sample:

	Population	Sample
Standard deviation	$\sigma$	$s$
Variance	$\sigma^2$	$s^2$

Interquartile range

Remember the first $q_{0.25}$ and third quartile $q_{0.75}$ presented earlier (see this section). The interquartile range is another measure of dispersion of the data, using the quartiles. It is the difference between the third and first quartile:

\[IQR = q_{0.75} - q_{0.25}\]

Considering the height of the 9 adults presented in the section about the first and third quartile:

170.2, 181.5, 188.9, 163.9, 166.4, 163.7, 160.4, 175.8 and 181.5

The first quartile was 163.9 cm and the third quartile was 181.5 cm. The IQR is thus:

\[IQR = 181.5 - 163.9 = 17.6\]

In conclusion, the interquartile range is 17.6 cm. The interquartile range is actually the range (since it is the difference between a higher and a lower value) of the middle data. The graph below may help to understand better the IQR and the quartiles:

Coefficient of variation

The last dispersion measure is the coefficient of variation. The coefficient of variation, denoted $CV$, is the standard deviation divided by the mean. Formally:

\[CV = \frac{s}{\bar{x}}\]

Consider the height of a sample of 4 adults:

163.7, 160.4, 175.8 and 181.5

The mean is $\bar{x} =$ 170.35 cm and the standard deviation is $s =$ 9.95 cm. (Find the same values as an exercise!) The coefficient of variation is

\[CV = \frac{9.95 \text{ cm}}{170.35 \text{ cm}} = 0.058 = 5.8\%\]

In conclusion, the coefficient of variation is 5.8%. Note that, as a rule of thumb, a coefficient of variation greater than 15% usually means that the data are heterogeneous while a coefficient of variation equal to or less than 15% means that the data are homogeneous. Given that the coefficient of variation equals 5.8% in this case, we can conclude that these 4 adults are homogeneous in terms of height.

Note that the coefficient of variation for a population follows the same formula, except that notations for the mean and the standard deviation differ:

\[CV = \frac{\sigma}{\mu}\]

The interpretation is also the same as for a sample.

Coefficient of variation vs. standard deviation

Although the coefficient of variation is rather unknown to the public, it is, in fact, worth presenting when making descriptive statistics.

The standard deviation should always be understood in the context of the mean of the data and is dependent on its unit. The standard deviation has the advantage that it tells by how far on average the data is from the mean in terms of unit in which the data has been measured. Standard deviation is useful when considering variables with same units and approximately same means. However, standard deviation becomes less useful when comparing variables with different units or widely different means. For instance, a variable with a standard deviation of 10 cm cannot be compared to a variable with a standard deviation of 12€ to conclude which one of the two is the most dispersed.

The coefficient of variation is a ratio of two statistics with the same units. It has thus no unit and is independent of the unit in which the data has been measured. Being unit-free, coefficients of variation computed on datasets or variables with different units or widely different means can be compared to conclude, in fine, which data or variables is more (or less) dispersed. For instance, consider a sample of 10 women with their heights in cm and their salaries in €. We cannot compare the dispersion of their weights with the dispersion of their salaries because it is not measured on the same unit/scale. Now suppose that the coefficients of variation are 0.032 and 0.061 respectively for the height and the salary. Based on that, we can conclude that, relative to their respective average, their salaries vary more than their heights for these women. This is the case because the coefficient of variation is larger for the salary compared to the coefficient variation for the height, and a coefficient of variation has no unit (it is a ratio).

Conclusion

Remember that descriptive statistics are useful to describe and present a series of observations in a concise and informative way. There are two families of descriptive statistics: location and dispersion measures. Location measures give information about the position of the data, whereas dispersion measures give information about the variability of the data.

We showed how to compute the most common descriptive statistics by hand with concrete examples. We also discussed differences between some measures and when it is more appropriate to use one or the other depending on the context and the data at hand.

This concludes a relatively long article, thanks for reading! If you would like to learn how to compute these measures in R, read the article “Descriptive statistics in R”. A book I recommend for further reading is “The Art of Statistics” by David Spiegelhalter. It is a great book for beginners in statistics and covers a wide range of topics.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

This image and the following ones are taken from the LFSAB1105 course syllabus at UCLouvain.↩︎

What is the difference between population and sample?

Sat, 18 Jan 2020 00:00:00 +0000

Introduction

People often fail to properly distinguish between population and sample. It is however essential in any statistical analysis, starting from descriptive statistics with different formulas for variance and standard deviation depending on whether we face a sample or a population.

Moreover, the branch of statistics called inferential statistics is often defined as the science of drawing conclusions about a population from observations made on a representative sample of that population. It is therefore crucial to properly distinguish between the two concepts. So, what exactly is the difference between population and sample?

Sample vs. population

Population versus sample. Source: towardsdatascience.com

A population includes all members from a specified group, all possible outcomes or measurements that are of interest. The exact population will depend on the scope of the study. For example, say you would like to know whether there is an association between job performance and the amount of home working hours per week in the specific case of Belgian data scientists. In this case, the population might be Belgian data scientists. However, if the scope of the study is more narrow (e.g., the study focuses on french-speaking Belgian data scientists who live at least 30km away from their workplace), then the population will be more specific and include only workers who meet the criteria. The point is that the population should only include people to whom the results will apply.

A sample consists of some observations drawn from the population, so a part or a subset of the population. The sample is the group of elements who actually participated in the study.

Members and elements are defined in the broad sense of the term. It may be human. For instance, the population may be “all people living in Belgium” and the sample may be “some people living in Belgium”. It can be anything else too. Say you are testing the effect of a new fertilizer on crop yield. All the crop fields represent your population, whereas the 10 crop fields you tested correspond to your sample. Since a sample is a subset of a population, a sample is always smaller than the population.¹ Note that, a population must not necessarily be large. It might be the case that you study such a narrow population (e.g., first-year male bachelor students from your university who passed the statistics exam in June and for whom their parents have been divorced for more than 5 years), that the size of the population is actually rather small.

Why a sample?

As mentioned at the beginning of this article, one of the main concern in statistics is being able to draw conclusions about a population from a representative sample. Why using a sample of the population and not directly the population? In general it is almost always impossible to carry out measurements for the entire study population because:

the population is too large. Example: the population of pregnant women. If we want to take measurements on all pregnant women in the world, it will most likely either take too long or cost too much
the population is virtual. In this case “virtual” population is understood as a “hypothetical” population: it is unlimited in size. Example: for an experimental study, we focus on men with prostate cancer treated with a new treatment. We do not know how many people will be treated, so the population varies, is infinite and uncountable at the present time, and therefore virtual
the population is not easily reachable. Example: the population of homeless persons in Belgium

For these reasons, measurements are made on a subgroup of observations from the population, i.e., on a sample of our population. These measures are then used to draw conclusions about the population of interest. With an appropriate methodology and a sufficiently large sample size, the results obtained on a sample are often almost as accurate as those that would be obtained on the entire population.

Representative sample

Of course, the sample must be selected to be representative of the population under study. If participants are included in a study on a voluntary basis, there is a serious concern that the resulting sample may not be representative of the population. It may be the case that volunteers are different in terms of the parameter² of interest, leading to a selection bias. Another selection bias can occur when, for instance, a researcher collects citizens’ wage, by the means of internet. It might be the case that people having access to internet have different wages than people who do not have access.

The gold standard to select a sample representative of the population under study is by selecting a random sample. A random sample is a sample selected at random from the population so that each member of the population has an equal chance of being selected. A random sample is usually an unbiased sample, that is, a sample whose randomness is not in doubt.

In some situations or domains (e.g., in medicine, psychology, etc.) it is complicated or even impossible to obtain a random sample of the population. In such cases, it will be important to consider how representative the resulting sample will be.

Paired samples

Last but not least, paired samples are samples in which groups (often pairs) of experimental units are linked together by the same experimental conditions. For example, one may measure the hours of sleep for 20 patients before taking a sleeping pill (forming sample A), and then repeat the measurements on the same individuals after they have taken a sleeping pill (forming sample B). Another example is the grades of 25 students at their statistics and economics exam. Among each student, grades for these two exams are undoubtedly associated (to some extent) to each other.

Paired samples are usually formed when a variable of interest is measured at different times on the same experimental unit. But, paired samples can also occur even when measurements are made at one specific point in time. For example, if we measure the strength in the right and left arm of 30 athletes. Strength in the right and left arm for the same individual are linked to each other, so sample A (for the right arm) and sample B (for the left arm) are paired samples.

The two measurements for each individual (hours of sleep before and after the sleeping pill, grades at two different exams, or the strength in the right and left arm) and the two samples are of course related. Statistical tools accounting for a relation between the samples exist and should be preferred in that case.

Conclusion

To summarize, the sample is the group of individuals who participated in the study and the population is the broader group to whom the results will apply. Measurements on the entire population is often too complex or impossible, so representative samples are used to draw conclusions about the population. Samples based on a random selection are often the most representative samples.

Thanks for reading.

I hope the article helped you to understand the difference between population and sample. For the interested reader, see the most common descriptive statistics that can be made on a sample or a population.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There can be, however, many different samples from the same population. This is beyond the scope of this article, and at the moment we assume there is only one sample from a specified population.↩︎
The tools used to describe a population are called parameters, whereas the tools used to describe a sample are referred as statistics. See the most common descriptive statistics for a sample and a population.↩︎

A Shiny app for inferential statistics by hand

Wed, 15 Jan 2020 00:00:00 +0000

Statistics is divided into four main branches:

Descriptive statistics
Inferential statistics
Predictive analysis
Exploratory analysis

Descriptive statistics provide a summary of the data; it helps explaining the data in a concise way without losing too much information. Data can be summarized numerically or graphically. See descriptive statistics by hand or in R to learn more about this branch of statistics.

The branch of predictive analysis aims at predicting a dependent variable based on one or several independent variables. Depending on the type of data to be predicted, it often encompasses methods such as linear regression or classification.

Exploratory analyses focus on using graphical approaches to delve into the data and identify the relationships that exist between the different variables in the dataset. They are therefore more akin to data visualization.

Inferential statistics uses a random sample of data taken from a population to make inferences, i.e., to draw conclusions about the population (see the difference between population and sample). In other words, information from the sample is used to make generalizations about the parameter of interest in the population.

The two major tools in inferential statistics are:

confidence intervals, and
hypothesis tests.

Here is a Shiny app which helps you to use these two tools:

Statistics-201

This Shiny app focuses on confidence intervals and hypothesis tests for:

1 and 2 means (with unpaired and paired samples)
1 and 2 proportions
1 and 2 variances

How to use this app?

Follow these steps:

Open the app via this link
Choose the parameter(s) you want to do inference for (i.e., mean(s), proportion(s) or variance(s))
Write your data in Sample. Observations are separated by a comma and the decimal is a point
Set the null and alternative hypothesis
Select the significance level (most of the time $\alpha = 0.05$)

In the results panel (on the right side or below depending on the size of your screen), you will see:

a recap of your sample together with some appropriate descriptive statistics
the confidence interval
the hypothesis test
the interpretation
and an illustration of the hypothesis test

All formulas, steps and computations to arrive at the final results are also provided.

Code

See the last version on GitHub.

Note that the link may not work if the app has hit the monthly usage limit. Try again later if that is the case.

Conclusion

Thanks for reading.

I hope you will find this app useful to do inferential statistics and in particular confidence interval and hypothesis testing by hand.

If you need to learn more about the structure of a hypothesis test by hand, see this article. See also this flowchart to have an overview of the most common statistical tests.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A Shiny app for simple linear regression by hand and in R

Wed, 15 Jan 2020 00:00:00 +0000

Simple linear regression is a statistical method to summarize and study relationships between two variables. When more than two variables are of interest, it is referred as multiple linear regression. See this article on linear regression for more details.

In this article, we focus only on a Shiny app which allows to perform simple linear regression by hand and in R:

Statistics-202

How to use this app?

Follow these steps:

Open the app via this link
Enter your data in the x and y fields. The x field corresponds to the independent variable, while the y field corresponds to the dependent variable
If you do not want to display the confidence interval around the regression line, uncheck the checkbox under Plot
Change the x and y-axis labels for the regression plot if needed

In the results panel (on the right side or below depending on the size of your screen), you will see:

a recap of your dataset together with some appropriate descriptive statistics
the estimates $\beta_0$, $\beta_1$ and the regression model computed by hand
the results of the model computed in R
the regression plot with some key measures
the interpretations
and the assumptions to check the validity of the model

All formulas, steps and computations to arrive at the final results are also provided.

Note that although the assumptions are displayed, it is your responsibility to check them to assess the validity of the linear model.

Last but not least, you can download a report of the results (in HTML) by clicking on the Download button, and you can choose whether you want to include the R code or not.

Code

See the last version on GitHub.

Note that the link may not work if the app has hit the monthly usage limit. Try again later if that is the case.

Conclusion

For further details about what is linear regression and when it is used, please see:

this post, and
the numerous resources on the topic available in textbooks and online.

Thanks for reading.

I hope you will find this app useful to do simple linear regression by hand and in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A guide on how to read statistical tables

Mon, 06 Jan 2020 00:00:00 +0000

Below a Shiny app to help you read the main statistical tables:

Statistics-101

This Shiny app helps you to compute probabilities for the main probability distributions.

How to use this app?

Open the app via this link
Choose the distribution
Set the parameter(s) of the distribution (the parameters depend of course on the chosen distribution)
Select whether you want to find the lower tail, upper tail or an interval
Choose the value of x

On the right panel (or below depending on the size of your screen) you will see:

a recap of the data you just entered
the numerical solution (i.e., the probability)
a visualization of the solution
the probability density function together with the mean, the standard deviation and the variance

Example

Here is an example with the most common distribution: the normal distribution.

Suppose the following problem: The cost of weekly maintenance and repair of a business has been observed over a long period of time and turns out to be distributed according to a normal distribution with an average of 402€ and a standard deviation of 22€. Having set a budget of 439€ for next week, what is the probability that the cost exceeds this budget?

To solve this problem, follow these steps in the app:

Choose the normal distribution, as it is said that the costs follow a normal distribution
Set the mean $\mu$ equal to 402, as it is said that the average cost is 402€
In the statement, the standard deviation is given (and not the variance) so select “Standard deviation $\sigma$” and set it equal to 22
We are asked what is the probability the the cost exceeds the budget. Therefore, we look for the probability above a certain x, so select upper tail $P(X > x)$
We are now asked to find the probability that the cost exceeds 439€, so set x equal to 439

The solution panel gives a recap of the data:

\[X ∼ \mathcal{N}(\mu = 402, \sigma^2 = 484)\] where $484 = 22^2$, and the solution: \[P(X > 439) = P(Z > 1.68) = 0.0463\] where $Z = \frac{X - \mu}{\sigma} = \frac{439 - 402}{22} = 1.68$ and $Z ∼ \mathcal{N}(\mu = 0, \sigma^2 = 1)$ (known as the standard normal distribution). Thus, the probability that the cost next week exceeds the budget of 439€ is 0.0463, or 4.63%.

It also shows the normal distribution (with $\mu = 402$ and $\sigma^2 = 484$) with the shaded area corresponding to the probability we are looking for. It then gives some details about the density function, the mean, the standard deviation and the variance.

Code

See the last version on GitHub.

Note that the link may not work if the app has hit the monthly usage limit. Try again later if that is the case.

Conclusion

Thanks for reading.

I hope you will find this app useful to compute probabilities for the main distributions.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Variable types and examples

Mon, 30 Dec 2019 00:00:00 +0000

Introduction

If you happen to work with datasets frequently, you probably know that each row of your dataset represents a different experimental unit (also called observation) and each column represents a different characteristic (called variable):

Structure of a dataset. Source: R for Data Science by Hadley Wickham & Garrett Grolemund

If you do some research on the weight and height of 100 students of your university, for example, you will most likely have a dataset containing 100 rows and 3 columns:

one for the student’s ID (could be anonymized or not),
one for the weight,
and one for the height.

These three columns represent three characteristics of the 100 students. They are called variables.

In this article, we are going to focus on variables, and in particular on the different types of variable that exist in statistics. (To learn about the different data types in R, read “Data types in R”.)

Different types of variables for different types of statistical analysis

First, one may wonder why we are interested in defining the types of our variables of interest.

The reason why we often class variables into different types is because not all statistical analyses can be performed on all variable types. For instance, it is impossible to compute the mean of the variable “hair color” as you cannot sum brown and blond hair.

On the other hand, finding the mode of a continuous variable does not really make any sense because most of the time there will not be two exact same values, so there will be no mode. And even in the case there is a mode, there will be very few observations with this value. As an example, try finding the mode of the height of the students in your class. If you are lucky, a couple of students will have the same size. However, most of the time, every student will have a different size (especially if heights have been measured in millimeters) and thus there will be no mode. To see what kind of analysis is possible on each type of variable, see more details in the articles “Descriptive statistics by hand” and “Descriptive statistics in R”.

Similarly, some statistical tests can only be performed on certain type of variables. For example, the Pearson correlation is usually computed on two quantitative variables, while a Chi-square test of independence is done with two qualitative variables, and a Student t-test or ANOVA requires a mix of one quantitative and one qualitative variable.

Big picture

In statistics, variables are classified into 4 different types:

We present each type together with examples in the following sections.

Quantitative

A quantitative variable is a variable that reflects a notion of magnitude, that is, if the values it can take are numbers. A quantitative variable represents thus a measure and is numerical.

Quantitative variables are divided into two types: discrete and continuous. The difference is explained in the following two sections.

Discrete

Quantitative discrete variables are variables for which the values it can take are countable and have a finite number of possibilities. The values are often (but not always) integers. Here are some examples of discrete variables:

Number of children per family
Number of students in a class
Number of citizens of a country

Even if it would take a long time to count the citizens of a large country, it is still technically doable. Moreover, for all examples, the number of possibilities is finite. Whatever the number of children in a family, it will never be 3.58 or 7.912 so the number of possibilities is a finite number and thus countable.

Continuous

On the other hand, quantitative continuous variables are variables for which the values are not countable and have an infinite number of possibilities. For example:

Age
Weight
Height

For simplicity, we usually referred to years, kilograms (or pounds) and centimeters (or feet and inches) for age, weight and height respectively. However, a 28-year-old man could actually be 28 years, 7 months, 16 days, 3 hours, 4 minutes, 5 seconds, 31 milliseconds, 9 nanoseconds old.

For all measurements, we usually stop at a standard level of granularity, but nothing (except our measurement tools) prevents us from going deeper, leading to an infinite number of potential values. The fact that the values can take an infinite number of possibilities makes it uncountable.

Qualitative

In opposition to quantitative variables, qualitative variables (also referred as categorical variables or factors in R) are variables that are not numerical and which values fit into categories.

In other words, a qualitative variable is a variable which takes as its values modalities, categories or even levels, in contrast to quantitative variables which measure a quantity on each individual.

Qualitative variables are divided into two types: nominal and ordinal.

Nominal

A qualitative nominal variable is a qualitative variable where no ordering is possible or implied in the levels.

For example, the variable gender is nominal because there is no order in the levels (no matter how many levels you consider for the gender—only two with female/male, or more than two with female/male/ungendered/others, levels are unordered). Eye color is another example of a nominal variable because there is no order among blue, brown or green eyes.

A nominal variable can have:

two levels (e.g., do you smoke? Yes/No, or are you pregnant? Yes/No), or
a large number of levels (what is your college major? Each major is a level in that case).

Note that a qualitative variable with exactly 2 levels is also referred as a binary or dichotomous variable.

Ordinal

On the other hand, a qualitative ordinal variable is a qualitative variable with an order implied in the levels. For instance, if the severity of road accidents has been measured on a scale such as light, moderate and fatal accidents, this variable is a qualitative ordinal variable because there is a clear order in the levels.

Another good example is health, which can take values such as poor, reasonable, good, or excellent. Again, there is a clear order in these levels so health is in this case a qualitative ordinal variable.

Variable transformations

There are two main variable transformations:

From a continuous to a discrete variable
From a quantitative to a qualitative variable

From continuous to discrete

Let’s say we are interested in babies’ ages. The data collected is the age of the babies, so a quantitative continuous variable. However, we may work with only the number of weeks since birth and thus transforming the age into a discrete variable. The variable age remains a quantitative continuous variable but the variable we are working on (i.e., the number of weeks since birth) can be seen as a quantitative discrete variable.

From quantitative to qualitative

Let’s say we are interested in the Body Mass Index (BMI). For this, a researcher collects data on height and weight of individuals and computes the BMI. The BMI is a quantitative continuous variable but the researcher may want to turn it into a qualitative variable by categorizing individuals below a certain threshold as underweight, above a certain threshold as overweight and the rest as normal weight. The raw BMI is a quantitative continuous variable but the categorization of the BMI makes the transformed variable a qualitative (ordinal) variable, where the levels are in this case underweight < normal < overweight.

Same goes for age when age is transformed to a qualitative ordinal variable with levels such as minors, adults and seniors. It is also often the case (especially in surveys) that the variable salary (quantitative continuous) is transformed into a qualitative ordinal variable with different range of salaries (e.g., < 1000€, 1000 - 2000€, > 2000€).

Additional notes

Misleading data encoding

Last but not least, in datasets it is very often the case that numbers are used for qualitative variables. For instance, a researcher may assign the number “1” to women and the number “2” to men (or “0” to the answer “No” and “1” to the answer “Yes”). Despite the numerical classification, the variable gender is still a qualitative variable and not a discrete variable as it may look. The numerical classification is only used to facilitate data collection and data management. It is indeed easier to write the number “1” or “2” instead of “women” or “men”, and thus less prone to encoding errors.

The same goes for the identification of each observation. Suppose you collected information on 100 students. You may use their student’s ID to identify them in the dataset (so that you can trace them back). Most of the time, students’ ID (or ID in general) are encoded as numeric values. At first sight, it may thus look like a quantitative variable (because it goes from 1 to 100 for example). However, ID is clearly not a quantitative variable because it actually corresponds to an anonymized version of the student’s first and last name. If you think about it, it would make no sense to compute the mean or median on the IDs, as it does not represent a numerical measurement (but rather just an easier way to identify students than with their names).

If you face this kind of setup, do not forget to transform your variable into the right type before performing any statistical analyses. Usually, a basic descriptive analysis (and knowledge about the variables which have been measured) prior to the main statistical analyses is enough to check that all variable types are correct.

Conclusion

Thanks for reading.

I hope this article helped you to understand the different types of variable. If you would like to learn more about the different data types in R, read the article “Data types in R”.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

	Population	Sample
Standard deviation	\(\sigma\)	\(s\)
Variance	\(\sigma^2\)	\(s^2\)