Professional on Stats and R

EM-DAT, the world's disaster memory, is at risk

Tue, 07 Apr 2026 00:00:00 +0000

I do not usually write posts that are calls to action. But sometimes, something important enough comes along that it would feel wrong to stay silent. This is one of those times.

What is EM-DAT?

EM-DAT, the Emergency Events Database, is the world’s most widely used and trusted global database for tracking natural and technological disasters. It has been maintained since 1988 by the Centre for Research on the Epidemiology of Disasters (CRED), which is part of UCLouvain.

The database currently contains data on the occurrence and impacts of over 27,000 mass disasters worldwide, from 1900 to the present day. It covers floods, storms, earthquakes, droughts, wildfires, extreme temperatures, landslides, volcanic activity, and technological accidents, across virtually every country on earth.

Crucially, it is:

Open access (for non-commercial use)
Globally comparable, using transparent and consistent inclusion criteria
Cross-verified across multiple sources (UN agencies, NGOs, reinsurance companies, research institutes, press agencies)
The reference dataset for thousands of peer-reviewed studies, national risk assessments, and international policy processes

If you have ever read a paper or report about global disaster trends, the probability is high that EM-DAT was the data source behind it.

Why is it at risk?

For more than 25 years, EM-DAT was primarily funded by the United States Agency for International Development (USAID). Following the recent dismantling of USAID, that funding is gone, and no sustainable alternative has been secured.

This is not a minor budget shortfall. Without a replacement funding mechanism, EM-DAT risks shutting down entirely.

Why does it matter?

The open letter drafted in support of EM-DAT puts it well: in an era of intensifying climate extremes, cascading risks, and compounding crises, reliable data are not a luxury. They are the infrastructure for informed decision-making.

Concretely, EM-DAT underpins:

Disaster risk reduction and prevention policies, used by governments to assess national risks and prioritise investments
Humanitarian operations, relied upon by multilateral agencies and NGOs to plan and forecast needs
Climate research, providing historical baselines for understanding trends in extreme weather events
Monitoring of global commitments, such as the Sendai Framework for Disaster Risk Reduction, the SDGs, and the Paris Agreement
Insurance and risk modelling, used by the private sector alongside other data to benchmark losses and refine exposure models

EM-DAT’s value is not just in the quantity of records. It lies in the rigour and consistency of its methodology over time and across countries. That is exactly what makes it irreplaceable. In a world awash with data, curated and quality-controlled datasets of this kind are rare. If EM-DAT were to close, the result would not be a smooth substitution. It would be fragmentation, proprietary data silos, and reduced access, particularly for lower-income countries that are already under-represented in global evidence.

A personal note

I signed the open letter after being informed of the issue by my colleague Prof. Niko Speybroeck, a leading epidemiologist at UCLouvain and program director of the CRED.

I do not have direct expertise in disaster epidemiology. But I do care about open data, open science, and the integrity of global research infrastructure. And EM-DAT is exactly the kind of resource that the whole scientific community relies on, often without fully realising it.

How you can help

If you share these values, I encourage you to sign the open letter: “The World’s collective disaster memory must be preserved”.

The letter calls on governments, multilateral development banks, philanthropic foundations, and international organisations to step forward with a coordinated and sustainable funding arrangement for EM-DAT. The cost of maintaining the world’s primary disaster database is modest set against the billions spent on disaster response and recovery each year. The cost of losing it would be profound.

Please also consider sharing this post or the open letter with your own network (researchers, policymakers, students, practitioners, or anyone who cares about data-driven approaches to global challenges).

More information

EM-DAT website: https://www.emdat.be/
Open letter: https://openletter.earth/the-worlds-collective-disaster-memory-must-be-preserved-66c88c44
CRED at UCLouvain: https://www.uclouvain.be/en/research-institutes/irss/cred-epidemiology-of-disasters

As always, if you have any thoughts or questions related to this post, feel free to leave a comment below.

What is the probability that two persons have the same initials?

Wed, 06 Dec 2023 00:00:00 +0000

Introduction

Last week, I joined a team to work on a collaborative project. The team was already established for a few months, with several scientists working together on the project. For simplicity, they used to sign documents, mention colleagues in emails, etc. with their initials (the first letter of their first name followed by the first letter of their last name).

A couple of days after joining the project, when I needed to sign my first document with my initials, we realized that another person in the team had the exact same initials than me.

This was not really an issue, as we decided that I would write my initials backward, that is, “SA” instead of “AS”, and the other person would keep signing with “AS” as usual.

It could have stopped here. However, the idea to write a post about this rather trivial anecdote came to me when the team leader claimed, in the middle of a meeting: “That’s very unfortunate that you two have the same initials! What are the chances of this happening to us?!”.

We spent a couple of minutes trying to estimate this probability, which in the end were mostly based on our intuitions rather than on a formal calculation. This piqued my curiosity.

Given that the project we are working on requires the use of simulations, I decided to focus on answering this question via simulations in R. That being said, as for most simulations, it is a good practice to verify these results. This is done using probability theory. This comparison will allow to assess the truthfulness of results obtained through simulations.

Furthermore, I thought that it would be a nice way to illustrate methods not often presented in my posts: for loops, replications and writing functions in R.

How likely is it?

Before answering the question raised by the team leader, there are three things to note:

Although the team leader was curious to know the probability that exactly two persons have the same initials, we are actually more interested in the probability that at least two persons have the same initials (as the problem also occurs if more than two persons within a team have the same initials).
The team consists of 8 people.
We restrict ourselves to two-letters initials (the first letter being the first letter of the first name, the second letter being the first letter of the last name). This means that middle names are not taken into account, and only the first letter is considered for compound names.

In this post, we will show how to compute this probability:

in our context, that is, for a team of 8 persons, and
for completeness, for teams of all sizes from 2 to 100 persons.

As stated in the introduction, we will compute these probabilities first through simulations and then through probability theory.

For our team

We start by creating a vector of size 8, corresponding to the initials of a team of 8 persons randomly sampled among all 26 letters of the Latin alphabet:

# number of persons
n_persons <- 8

# create vector of initials
initials <- replicate(
  n = n_persons, # number of replications
  paste0(sample(LETTERS, size = 1), sample(LETTERS, size = 1)) # sample letters
)

# display initials
initials

## [1] "UJ" "MN" "XD" "CY" "BB" "ZB" "CU" "HQ"

# are there duplicates?
any(duplicated(initials))

## [1] FALSE

As we can see, everyone has different initials in this simulated team of 8 persons, but this will not always be the case.

To estimate, via simulations, how likely is that at least two persons have the same initials among the team, we need to replicate this vector of 8 sampled initials a large number of times (say 1,000 replications):¹

# number of replications
reps <- 1000

# create and save replications
dat <- replicate(
  n = reps, # number of replications
  replicate(n_persons, paste0(sample(LETTERS, size = 1), sample(LETTERS, size = 1)))
)

# dimensions
dim(dat)

## [1]    8 1000

# display first 4 simulated teams
dat[, 1:4]

##      [,1] [,2] [,3] [,4]
## [1,] "VA" "BU" "LU" "PT"
## [2,] "JG" "SM" "HM" "OL"
## [3,] "BY" "NA" "VJ" "OT"
## [4,] "RT" "CM" "WT" "YT"
## [5,] "PS" "CT" "NB" "QJ"
## [6,] "MG" "KR" "SV" "US"
## [7,] "PL" "SN" "PN" "XW"
## [8,] "NJ" "BR" "DD" "ZC"

The result is a matrix of 8 rows and 1000 columns, where:

each rows corresponds to the sampled initials of a person, and
each column corresponds to one simulated team of 8 people.

For better readability, we rename:

the row names as M1 to M8, corresponding to persons 1 to 8, and
the column names as T1 to T1000, corresponding to teams 1 to 1000.

# rename rows
rownames(dat) <- paste0("M", 1:n_persons)

# rename columns
colnames(dat) <- paste0("T", 1:reps)

# display first 4 simulated teams
dat[, 1:4]

##    T1   T2   T3   T4  
## M1 "VA" "BU" "LU" "PT"
## M2 "JG" "SM" "HM" "OL"
## M3 "BY" "NA" "VJ" "OT"
## M4 "RT" "CM" "WT" "YT"
## M5 "PS" "CT" "NB" "QJ"
## M6 "MG" "KR" "SV" "US"
## M7 "PL" "SN" "PN" "XW"
## M8 "NJ" "BR" "DD" "ZC"

We now need to compute, among the 1000 teams simulated, how many of them have at least two persons with the same initials:

# transform to data frame
dat <- as.data.frame(dat)

# save which teams have duplicates
duplicates <- rep(NA, reps) # create empty vector
for (i in 1:reps) { # for loop over i from 1 to 1,000
  duplicates[i] <- any(duplicated(dat[, i])) # save results TRUE/FALSE in duplicates vector
}

# count how many teams have duplicates
sum(duplicates)

## [1] 41

Here, for each column of our data frame dat (from the first to the 1000th column), we ask whether there are duplicates or not. This is done repeatedly over all columns thanks to a for loop. For each column, the result is TRUE if there are duplicates, otherwise it is FALSE. The result of each iteration is saved in the duplicates vector. As TRUE = 1 and FALSE = 0 in R, we can then count how many columns (and thus teams) have duplicates by summing the number of TRUE in the duplicates vector.

As we can see from the output above, among the 1000 simulated teams, 41 of them have duplicates, that is, 41 of them have at least two persons with the same initials.

Therefore, based on the simulations, we can expect the probability that at least two persons with the same initials in a team of 8 persons to be close to 4.1%.

This is a good starting point. Notice, however, that I wrote close to 4.1% because this probability will vary each time it is computed via simulations.

For instance, if we repeat the exact same process a second time:

# create and save replications
dat <- replicate(
  n = reps, # number of replications
  replicate(n_persons, paste0(sample(LETTERS, size = 1), sample(LETTERS, size = 1)))
)

# transform to data frame
dat <- as.data.frame(dat)

# save which teams have duplicates
duplicates <- rep(NA, reps) # create empty vector
for (i in 1:reps) { # for loop over i from 1 to 1,000
  duplicates[i] <- any(duplicated(dat[, i])) # save results in the duplicates vector (as TRUE/FALSE)
}

# count how many teams have duplicates
sum(duplicates)

## [1] 44

We now find a probability of 4.4%. This is not an error, but it is due to randomness when sampling initials.

Luckily, we can make the computation of this probability more robust thanks to replications. Intuitively, it works as follows. We repeat the same computation multiple times, giving us a range of possible probabilities. This allows us to assess the uncertainty of our result, and understand how the probability might vary due to taking different random samples of initials.

So the goal is to compute our probability multiple times (say 100 times), and see its distribution.

To repeat the same computation multiple times, it is best to write a function in order to avoid copy pasting the same code over and over. So we first write a function (called initials) which computes the probability that at least two persons share the same initials among a team of \(n\) people:

initials <- function(n_persons, reps = 1000) {
  # simulate data
  dat <- as.data.frame(replicate(
    reps,
    replicate(n_persons, paste0(sample(LETTERS, size = 1), sample(LETTERS, size = 1)))
  ))

  # save which teams have duplicates
  duplicates <- rep(NA, reps)
  for (i in 1:reps) {
    duplicates[i] <- any(duplicated(dat[, i]))
  }

  # proportion of teams with duplicates
  return(mean(duplicates))
}

A function in R requires to include:

the parameters inside (), and
the computation inside {}.

We can then use our function to compute the probability that at least two persons share the same initials among a team of 8 people. And we combine it with the replicate() function to compute this probability 100 times.

# compute and save probabilities
probs <- replicate(100, initials(n_persons = 8))

# display probabilities
probs

##   [1] 0.032 0.037 0.040 0.043 0.033 0.042 0.039 0.047 0.045 0.038 0.052 0.042
##  [13] 0.042 0.040 0.023 0.044 0.041 0.039 0.036 0.048 0.041 0.037 0.027 0.030
##  [25] 0.052 0.038 0.043 0.035 0.038 0.045 0.047 0.044 0.030 0.036 0.036 0.048
##  [37] 0.038 0.045 0.044 0.034 0.031 0.043 0.045 0.034 0.049 0.047 0.051 0.036
##  [49] 0.051 0.040 0.043 0.044 0.038 0.049 0.043 0.050 0.035 0.043 0.048 0.038
##  [61] 0.041 0.044 0.039 0.045 0.033 0.057 0.036 0.043 0.041 0.041 0.041 0.041
##  [73] 0.038 0.044 0.031 0.034 0.049 0.041 0.040 0.034 0.032 0.036 0.049 0.047
##  [85] 0.048 0.038 0.038 0.037 0.036 0.037 0.043 0.040 0.026 0.049 0.046 0.044
##  [97] 0.048 0.038 0.026 0.029

Finally, we visualize the distribution of these 100 probabilities thanks to a histogram and a boxplot (with the {ggplot2} package):

# visualize distribution of the computed probabilities
# build and save plots
library(ggplot2)

p1 <- ggplot(mapping = aes(x = probs)) +
  geom_histogram(color = "black", fill = "steelblue", bins = 8) +
  labs(
    x = "Probabilities",
    y = "Frequencies"
  ) +
  scale_x_continuous(labels = scales::percent) # format x-axis in %

p2 <- ggplot(mapping = aes(x = probs)) +
  geom_boxplot(color = "black", fill = "steelblue") +
  labs(x = "Probabilities") +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank()
  ) +
  scale_x_continuous(labels = scales::percent) # format x-axis in %

# combine plots
library(patchwork)

p1 + p2

These two plots show that the probability that at least two persons share the same initials among a team of 8 people is most likely between 3.5% and 4.5%.

For the record, during the meeting at the root of all this thinking, most of us thought that it was much less likely. Indeed, I believe we were tempted to compute the probability that someone who joins the team has “AS” as initials. This is indeed much less likely, as the probability is only \(\frac{1}{26} \times \frac{1}{26} \simeq 0.15\%\).

However, this does not take into account the fact:

that the newcomer can have the same initials as any other person, and
that it is not only the newcomer who can have the same initials as another person (2 people already working in the team when the newcomer arrives could have the same initials as well).

If you are puzzled by this finding, I recommend reading about the birthday’s paradox. The birthday’s paradox states that the probability of two people sharing the same birthday becomes surprisingly high with a relatively small group of individuals. In practice, in a group of just 23 people, there is a greater than 50% chance that at least two individuals share the same birthday, illustrating our counterintuitive intuitions about the likelihood of such coincidences. This phenomenon arises due to the multitude of possible birthday pairs within the group, similar to the multitude of possible pairs if initials within a team.

For teams of different sizes

We are now interested in computing this probability not just for a team of 8 persons, but for teams of different sizes. We can do this with the help of our function defined earlier.

For the illustration, let’s compute the probability that at least two persons have the same initials, for teams of 2 and up to 100 persons:

# set lower and upper bounds of number of persons
min_persons <- 2
max_persons <- 100

# create empty vector of probabilities
probs <- rep(NA, length(min_persons:max_persons))

# compute and save probabilities for teams of size 2 to 100
for (i in min_persons:max_persons) {
  probs[i] <- initials(n_persons = i)
}

# display probabilities
probs

##   [1]    NA 0.001 0.005 0.013 0.012 0.019 0.036 0.040 0.047 0.057 0.074 0.083
##  [13] 0.103 0.128 0.158 0.166 0.178 0.215 0.232 0.260 0.275 0.296 0.300 0.329
##  [25] 0.357 0.392 0.405 0.405 0.439 0.478 0.495 0.536 0.535 0.563 0.578 0.599
##  [37] 0.653 0.656 0.686 0.693 0.715 0.711 0.767 0.760 0.786 0.784 0.814 0.817
##  [49] 0.825 0.826 0.842 0.845 0.867 0.893 0.901 0.920 0.919 0.911 0.917 0.942
##  [61] 0.950 0.951 0.946 0.947 0.969 0.959 0.965 0.964 0.977 0.984 0.977 0.977
##  [73] 0.985 0.978 0.986 0.981 0.989 0.991 0.989 0.988 0.992 0.993 0.994 0.996
##  [85] 0.997 0.995 0.994 0.999 0.999 0.999 1.000 0.999 0.997 1.000 0.999 0.999
##  [97] 0.999 1.000 1.000 0.999

We are left with storing these probabilities together with the number of persons in the team in a data frame:

# create data frame with saved probabilities and number of persons
dat_plot_sim <- data.frame(
  n_persons = (min_persons - 1):max_persons,
  prob = probs
)

# display first 6 rows
head(dat_plot_sim)

##   n_persons  prob
## 1         1    NA
## 2         2 0.001
## 3         3 0.005
## 4         4 0.013
## 5         5 0.012
## 6         6 0.019

Of course, two people having the same initials in a team of 1 (if we can call this a team…) is impossible.

An event which is impossible has a probability equal to 0. We thus impute this probability in our data frame, in the first row:

# set proba = 0 when n_person = 1
dat_plot_sim[1, 2] <- 0

# display first 6 rows
head(dat_plot_sim)

##   n_persons  prob
## 1         1 0.000
## 2         2 0.001
## 3         3 0.005
## 4         4 0.013
## 5         5 0.012
## 6         6 0.019

Finally, we visualize these probabilities in function of the number of persons in the team:

# visualize probabilities
ggplot(dat_plot_sim) +
  aes(x = n_persons, y = probs) +
  geom_line(linewidth = 1) +
  labs(
    x = "# of persons in the team",
    y = "Probability",
    title = "What is the probability that at least 2 persons have the same initials?"
  ) +
  scale_y_continuous(labels = scales::percent) # format y-axis in %

From the plot above, we see that the probability that at least two persons have the same initials reaches 50% when the team exceeds around 30 people.

Moreover, notice that this probability becomes close to 100% when the team reaches around 75 people.

Verification

For the sake of completeness, we now compare results obtained through simulations with results obtained from probability theory.²

We first define the function that will be used to compare results found above:

# define function
have_same <- function(s, n) {
  sample_space <- s
  probability <- 1
  for (i in 0:(n - 1)) {
    probability <- probability * (sample_space - i) / sample_space
  }
  1 - probability
}

For our team

# number of possible two-letter initials
n_initials <- 26^2

# apply function
have_same(n_initials, n_persons)

## [1] 0.0407218

The probability that at least two persons have the same initials in a team of 8 is 4.07%. This is close to the probability found with simulations, and within the range of 3.5%–4.5%.

For teams of different sizes

We now compute the probability for teams between 1 and 100 persons:

# compute and save probabilities for teams between 1 and 100 persons
probs <- vector(length = max_persons)
for (i in 1:max_persons) {
  probs[i] <- have_same(n_initials, i)
}

# create data frame with saved probabilities and number of persons
dat_plot_theory <- data.frame(
  n_persons = (min_persons - 1):max_persons,
  prob = probs
)

# display first 6 rows
head(dat_plot_theory)

##   n_persons        prob
## 1         1 0.000000000
## 2         2 0.001479290
## 3         3 0.004433493
## 4         4 0.008851688
## 5         5 0.014716471
## 6         6 0.022004071

Finally, we visualize these probabilities in function of the number of persons in the team:

# visualize probabilities
ggplot(dat_plot_theory) +
  aes(x = n_persons, y = probs) +
  geom_line(linewidth = 1) +
  labs(
    x = "# of persons in the team",
    y = "Probability",
    title = "What is the probability that at least 2 persons have the same initials?"
  ) +
  scale_y_continuous(labels = scales::percent) # format y-axis in %

For an easier comparison, we plot probabilities found thanks to simulations and thanks to probability theory on the same plot:

# combine the two data frames into one and add the method as variable
dat_plot_sim$Method <- "Simulations"
dat_plot_theory$Method <- "Theory"
dat_plot_all <- rbind(dat_plot_sim, dat_plot_theory)

# visualize probabilities on same plot
ggplot(dat_plot_all) +
  aes(x = n_persons, y = prob, color = Method) +
  geom_line(linewidth = 1) +
  labs(
    x = "# of persons in the team",
    y = "Probability",
    title = "What is the probability that at least 2 persons have the same initials?"
  ) +
  scale_y_continuous(labels = scales::percent) # format y-axis in %

The plot above shows that results using probability theory are relatively similar to results obtained through simulations, indicating that the simulations are trustworthy.

Conclusion

The initial question, raised during a meeting, was “What is the probability that, among our team consisting of 8 persons, two have the same initials?”.

In this post, we first showed how to compute this probability through simulations in R. Secondly, we verified the veracity of the simulations thanks to probability theory. Furthermore, we illustrated how for loops, replications and writing a function can be used in R to answer a probability problem.

As a side note, it is important to keep in mind that in this post, we assumed the following:

All letters of the alphabet had the same probability of occurring, meaning that all pairs of initials were equally probable. This is probably not the case in reality, as a first and last name starting both with X is not as probable as a first and last name starting respectively with M and K. This bias could be limited by specifying different weights when sampling initials.
We restricted ourselves to two-letters initials. Therefore, for compound first or last names, only the first letter is considered. Middle names are also not taken into account.

Last but not least, note that you will find slightly different results than mine, even if you use the exact same code. This is due to randomness. To replicate results as shown in this post, use set.seed(6).

Thanks for reading.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

You can always use a larger number of replications, but in our case the final result is similar with more replications, and the aim of the post is more to show the development than the final answer.↩︎
I thank Richard for writing the first version of the code used for the verifications.↩︎

10 potential career options with a degree in statistics

Fri, 24 Mar 2023 00:00:00 +0000

This post has been written in collaboration with Daniel Williams.

Introduction

A reader recently contacted me because he was hesitating between starting his studies in statistics or economics. I studied economics and I am now doing a PhD in statistics, so he reached out to me to know what I thought about these two fields, and in particular, what were my feelings about the career opportunities available with these two degrees.

Before considering pursuing a graduate degree, it is important to have a clear understanding of the skills you will develop and the career opportunities that will become available to you.

I leave it to those who have studied economics and work in this field to discuss career opportunities with a degree in economics. In this post, I will focus on the career options that are available for people with a degree in statistics.

Bear in mind that there are many professional opportunities with a degree in statistics. Indeed, statistical expertise is highly sought after in a wide range of computing and data analysis job roles; as soon as there are data, statistical expertise is required. Moreover, there are almost as many different jobs as people. Therefore, the list of jobs below is non-exhaustive and you may get a job that is not mentioned in this list. However, I hope it will still give you an overview of what you can expect after your studies.

If you believe I missed one—if you studied statistics and your job is not mentioned here for example, feel free to leave a comment at the end of the post.

What types of jobs are available?

If you are skilled in statistics and R programming, there are a plethora of job opportunities available to you.

Some of the roles you can pursue include:

Statistician: Collect, analyze and interpret data in a wide range of fields, including economics, finance, marketing, social and medical sciences.
Data scientist, data/business analyst, data engineer or machine learning engineer: Collect, analyze and interpret large and complex data sets using statistical and machine learning techniques to gain insights and inform decision-making to business and organizations. Mastering these techniques is key to the role, and in industry, turning statistical models into production-ready solutions often requires collaboration with a professional machine learning development company.
Actuary or actuarial analyst: Use statistical modeling to assess risk and uncertainty, and to create financial projections for insurance and investment companies.
Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst: Use statistical and mathematical modeling to analyze financial and market data, identify and evaluate risks for investment and trading decisions.
Business intelligence analyst: Use data analysis and visualization tools to create reports and dashboards that help businesses make informed decisions.
Operational researcher or quality control analyst: Use mathematical and statistical techniques to monitor and optimize the quality of products and processes in manufacturing.
Market or survey researcher: Conduct research and surveys, gather and analyze data to help businesses develop and improve their marketing strategies.
Economist or econometrician: Use statistical methods to analyze (socio)economic data and develop economic models to inform policy and decision-making.
Teacher: Facilitate learning and provide guidance to students in order to help them acquire knowledge, skills, and values that will prepare them for their future.
(Freelance) consultant: Provide statistical consulting services to clients, helping them solve problems using a combination of statistical methods and business expertise.

In the following sections, we delve deeper into the aforementioned job roles and outline the qualifications expected of prospective candidates.

Statistician

As a graduate in statistics, you have the potential to land a coveted statistician’s job in a reputable private company or public agency. These types of roles typically involve analyzing and assessing data, and utilizing various tools and software to manage data effectively.

With data proliferation and more and more companies/organizations that rely heavily on data analysis, your skills will be valuable in a broad range of fields and industries. Being a statistician means that you could work for instance as an:

environmental statistician: analyze data related to environmental issues, such as climate change and pollution, to inform policy and decision-making,
sports statistician: analyze data related to sports performance, such as player statistics and game outcomes, to inform coaching and strategy decisions,
government statistician: collect, analyze and interpret data for government agencies and public departments to guide policymakers in their decisions,
biostatistician: use statistical methods and data analysis techniques to study and interpret medical and (public) health-related data,
etc.

Statistical expertise is required wherever there is data, so feel free to specialize in the field or industry you care about most!

Data scientist, data/business analyst, data engineer or machine learning engineer

As a relatively new job role in various organizations, data scientists and data analysts are experts in manipulating, analyzing and interpreting data. These roles typically require the analysis of large datasets using statistical methods to assist organizations in making more informed business decisions. These professionals dive into unstructured information to uncover valuable insights for businesses, ultimately increasing their revenue.

To pursue this career, one would usually require a graduate degree in science, statistics, or mathematics, as well as training in data mining. It is also expected that you are familiar with one or several programming languages such as R, Python or SAS.¹

Statistics graduates can also pursue their career as machine learning engineers. Job roles in this field typically involve conducting experiments and implementing machine learning algorithms. Consequently, many organizations hire these professionals to create a wide range of AI products. To do so, they must possess strong programming and statistical skills. Data science and software engineering knowledge are also beneficial.

Actuary or actuarial analyst

Actuaries usually apply statistical modeling tools on data related to retirement, financial/insurance products, accidents or mortality. As an actuary, one would be responsible for analyzing financial costs and risks. It means examining uncertainties associated with investments and other financial products and making predictions about the risks involved in a particular venture.

Actuaries are also in charge of writing insurance proposals, determining policy terms, and calculating premiums for different products, among other responsibilities. You may also play a vital role in deciding whether to accept or reject insurance applications by evaluating their risks. You will draw upon your statistical, actuarial, and background information to accomplish these tasks.

Typically, actuaries have a range of financial companies as clients, and they must identify potential risks and recommend compensation strategies accordingly. It means that their work is closely related to insurance products, as they must assess the likelihood of certain events occurring and the resulting financial impact.

Typically, those who pursue this role hold a degree in actuarial science, mathematics or statistics, and are trained in statistical analysis tools and software. Note that each country has its own specific accreditation related to this profession.

Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst

As a financial analyst, one is required to gather data from various sources, organize and analyze historical results, and make projections and forecasts. Furthermore, in addition to analyzing data, their role often involves reporting financial results to the board of directors/management to help them in setting the overall strategy and direction of the company.

Investment analysts are experts in evaluating different financial assets such as stocks, securities, and bonds. They do not stop there, though. They also conduct research and make crucial decisions about purchasing business financials.

As a financial trader, you will have a keen understanding of financial markets and the ability to execute trades (i.e., buy or sell shares, bonds, and other assets) on behalf of clients. Note that there are also various sub-roles to explore within this job segment.

Financial managers combine their passion for finance with their love of numbers and statistics. Financial managers are experts at creating and interpreting complex financial reports, providing invaluable insights into a company’s financial health. They can also advise management on investment strategies and assist with vital financial decisions. With their keen analytical skills, financial managers keep a watchful eye on daily financial activities and help ensure a company’s long-term success.

These skills are in high demand in financial organizations, such as banks, actuarial firms, insurance companies (commercial insurance, reinsurance, general insurance, life insurance, car insurance, etc.), and other similar establishments.

The educational background of these professionals is diverse. Graduates in finance, mathematics, and statistics all bring unique perspectives to the table.

Business intelligence analyst

A business intelligence (BI) analyst is responsible for collecting, organizing, analyzing, and presenting large amounts of data from various sources such as databases, spreadsheets, and software applications. They use various tools and techniques to identify trends, patterns, and relationships in the data to create reports, dashboards, and visualizations to communicate their findings to stakeholders. They may also be responsible for monitoring business performance metrics and KPIs, identifying areas for improvement, and making recommendations to optimize business operations.

Additionally, BI analysts may also be involved in the development and implementation of data-driven software applications, such as business intelligence platforms and data warehouses, to improve data accessibility, accuracy, and efficiency. Overall, the role of a BI analyst is crucial in filling the gap between business analysts and the IT team.

The most common software tools and platforms used by BI analysts are, at the time of writing this post, R Shiny, Microsoft Power BI, Tableau, QlikView and SAS BI.

Operational researcher or quality control analyst

Operational researchers are responsible for using mathematical and analytical methods to solve complex problems and optimize business operations. They collect and analyze data, develop models and algorithms, test and validate them, and make recommendations for improvements.

On the other hand, quality control analysts are responsible for ensuring that products or services meet the required standards of quality. They monitor and analyze product or service performance, identify areas for improvement, and develop strategies for maintaining or improving quality. They may also work with production teams to develop and implement quality control processes and procedures.

While operational researchers focus on optimizing processes and systems, quality control analysts focus on ensuring that the end product or service meets the required quality standards. Both roles require analytical skills and the ability to work with data, but operational researchers focus more on mathematical modeling and optimization, while quality control analysts focus more on monitoring and improving quality.

Individuals who pursue job roles of this nature typically possess strong mathematical abilities, often having graduated with degrees in statistics or mathematics. They are trained to analyze vast amounts of data and are well-versed in various analytical tools, such as simulation, mathematical modeling, and data science.

Market or survey researcher

Market or survey researchers are professionals who specialize in conducting research to gather information about consumer behavior and preferences, market trends, and competitive landscapes. They use various research methods and techniques to collect and analyze data, and provide insights and recommendations to businesses and organizations.

Market research professionals typically work with marketing agencies on various projects for clients across different sectors. In addition to a degree in statistics, knowledge in marketing and familiarity with the industry is beneficial.

Economist or econometrician

Economists and econometricians study and analyze economic systems, markets, and policies using quantitative methods and models. They collect and analyze data, develop economic models, conduct economic analysis, provide recommendations to policymakers and organizations, and communicate their findings to a variety of audiences. Their role is to inform economic policies and decisions in both public and private sectors.

If you have a background in statistics in addition to knowledge in economics, there is a whole world of vocational opportunities in economics just waiting for you. Indeed, armed with the ability to appropriately analyze data (thanks to your background in statistics) and the ability to understand (socio)economic issues and financial data, you will be a hot commodity as an advisor to governments and businesses on all economic decisions.

(Freelance) consultant

From a general point of view, a consultant (or freelance consultant) is a professional who provides expert advice and guidance to organizations or individuals on a contractual basis. They analyze problems, develop solutions, and provide recommendations to clients in a specific field or industry.

A consultant specializing in statistics provides expert advice and guidance to organizations or individuals on statistical analysis, data management, and modeling. They use statistical tools and techniques to analyze and interpret complex data, and provide insights and recommendations based on their analyses. The role of a statistical consultant may include identifying research questions, designing studies, collecting and managing data, analyzing data using appropriate statistical methods, interpreting results, and presenting findings to clients. They may also assist clients with the implementation of statistical methods and tools, provide training on statistical software and techniques, and develop custom statistical models for specific applications.

As a side note, note that many consultants in statistics provide their service as data visualization consultants. These consultants are responsible for helping clients to effectively communicate complex data through the creation of clear and visually appealing graphics, charts, and other forms of visual representation that can aid in decision-making and enhance understanding.

You can either be a consultant in a consulting firm, or a freelance consultant. If you choose to be a freelance consultant, you can more easily select the projects you want to work on and the industry you want to focus on, but you will also need to take care of all the administrative tasks such as finding and building relationships with clients, do your bookkeeping, etc.

If you are not ready to be a full-time freelance consultant but would like to experience a taste of it, you can always keep your primary job and accept a couple of side projects. This is known as a side hustle. This way, you will gradually build your portfolio and create relationships with clients, all that with the advantage of keeping a safe source of income. This is what I am doing with datanalyze.be, and I recommend it to anyone who is considering the option of being a freelance in the future.

Teacher

If you care about education, you like to transmit your knowledge and explain complex things in a simple manner, you may be interested in being a teacher in high school or a university professor. With a degree in statistics, you may be eligible to teach different subjects such as mathematics, statistics, physics, or science in general. Your ultimate goal is to educate and inspire students to develop an understanding and appreciation of these subjects (which are often not well appreciated by students), as well as to develop critical thinking skills, problem-solving abilities, and scientific literacy.

Like actuaries, note that each country has its own specific accreditation related to this profession.

To know whether this job suits you, you can start by being a private tutor. This way, you will experience what the job is like and you will be more able to tell whether it is the direction you want to take.

To become a university professor, note that a PhD (and even sometimes a postdoc) is usually required. This brings me to the last career option I would like to mention: a PhD.

I do not include a PhD in the list of the 10 potential career options because it is more seen as an additional degree rather than a job, and even for those who see it as a job, it is a temporary one. However, I would still like to mention it because it is worth considering with a degree in statistics (I am of course biased, but my biased opinion may be of interest to some readers).

PhD student

For those who enjoy academic research, pursuing a career as a researcher after taking a statistics degree may be a worthwhile consideration. Typically, pursuing a PhD in statistics involves working in close collaboration with one or two university professors (your supervisors) on a specific topic.

I cannot speak for all PhD students, but I can speak from my experience. It involves conducting original research in statistical theory and methodology, developing new statistical models and methods, and applying statistical techniques to solve real-world problems or to advance knowledge in a specific field. My research focuses on applying biostatistical procedures to cancer patients, so it is a rather applied PhD, but many of my colleagues work on a more theoretical subject.

We also sometimes participate in academic conferences, present our research findings, and collaborate with other researchers in our field of research.

After completing our PhD, we can continue to evolve in academia (and become a university professor for instance), or decide to work in a private or public research organization. Because professionals with a PhD in statistics become experts in statistical theory and methodology, commercial and public organizations often seek such profiles.

The length of a PhD depends on the country and the type of contract. If you are interested to know more, feel free to contact me, I might be able to help you (at least with how it goes in Belgium). Otherwise, the program director at your university will definitely be able to answer the questions you may have.

Conclusion

As you have seen, with a degree in statistics and knowledge in programming, you will be able to pursue positions at various companies or organizations. You can also always enhance your qualifications with additional courses if you feel that you miss some skills that are required for the job you want to apply to.

Statisticians are in demand in government departments across many regions and sectors. Actuarial firms, banks, investment firms, and market research organizations are also great potential employers. The demand for statistical and analytical skills is on the rise across various fields. Job seekers with these skills are particularly needed. However, it is becoming increasingly common for these positions to require specialized education and training. For example, if you have a degree in statistics, pursuing a finance specialization could open doors to opportunities in banking, investment, accountancy, or insurance firms. Last but not least, those passionate about solving complex problems may find roles as researchers (within or outside academia) appealing.

Thanks for reading.

I hope this article helped you to get an overview of the career options you have with a degree in statistics.

If you are looking for a job, I wish you the best of luck in finding your dream job! Remember that you will find a variety of job openings in academic, research-oriented, and analytical fields on online and offline job boards. Moreover, do not underestimate the power of friends, colleagues and family: make sure to network with people in your field to increase the chance of getting a job.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There are many programming languages for data analysis, and it will certainly continue to evolve in the future. The most common ones at the time of writing this post are: R, Python, SAS, Jamovi, SPSS, JMP, Stata and Matlab. Depending on the role and the organization, more or less programming skills will be required.↩︎

Google Analytics in R: Review of 2022

Fri, 16 Dec 2022 00:00:00 +0000

Introduction

It is almost the end of the year, which means it is time to do a review of Stats and R and look back on the past year.

This year’s review will be much shorter compared to previous years because there are already a lot of examples in those previous reviews.

Note that beginning of September 2022, I moved from Universal Analytics (known as UA) to Google Analytics 4 (known as GA4). As you may know if you often use Google Analytics; on July 1, 2023, standard Universal Analytics properties will stop processing new hits. This means that you will need to move to GA4 if you want to continue using it after this date.

This year, I will be brief about the different metrics and how to analyze your Google Analytics (GA) data. However, this post will perhaps be helpful for people who also moved (or will move) to GA4 and still want to analyze their GA data from both UA and GA4.

I have to admit that it is quite time consuming to combine data from both, in particular since metrics and dimensions are not the same between UA and GA4. So if you have not moved yet, I recommend doing it at the end or beginning of a year. This will prevent you from having to fetch data from two GA IDs, and then having to combine both data in R.

As usual, I analyze my GA data in R thanks to the amazing {googleAnalyticsR} package from Mark Edmondson.

Note that the actual number presented below will not be very useful for you. Every blog and website is different, so every audience is different. This post is more about illustrating the process of analyzing GA data in R (and this year, also to illustrate the process of combining GA data from two GA IDs), rather than about showing my numbers.

Prerequisites

Before anything else, you have to specify the GA ID for which you want to analyze data. Mine are as follows:

ga_id <- c("208126346", "250795226")

You will find your(s) in the admin section of your GA account.

Analytics

In this section, I present some visualizations that might be useful when analyzing your GA data. Feel free to comment at the end of the post if you use other interesting visualizations.

Note that I focus on the page views metrics in this post, but you can edit my code to show other metrics as well. See the metrics available for UA and the ones for GA4.

Page views over time

Let’s start with the evolution of the number of page views over the past year, so from December 16, 2021 to December 15, 2022.

library(googleAnalyticsR)

# set date range
start_date <- as.Date("2021-12-16")
end_date <- as.Date("2022-12-15")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
library(dplyr)
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# scatter plot with a trend line
library(ggplot2)
dat %>%
  ggplot(aes(x = date, y = page_views)) +
  geom_point(size = 1L, color = "gray") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see in the code above, we need to:

extract data from two different IDs: this is done with google_analytics() for UA and ga_data() for GA4 (with each function having its own arguments)
join data from the two IDs (with a full_join() because even after moving to GA4, there were still some hits on UA, so I need to keep data from the 2 IDs for the entire year)
sum up the metric for both IDs (with rowSums())

This process will be done repeatedly for each visualization.

From the plot above, we see that the number of page views follows a cyclical evolution (with a decrease during summer). This trend follows the same pattern than last year, but the increase in the last quarter of the year is larger in 2022.

Note that there seems to be consistently some days with a lower number of page views than the rest. This is actually the weekends. See more on that in this section. Note also the presence of an outlier with a number of page views above 5,000. No post has been published on that day, so it probably comes from an old post that was shared to a large audience.

Finally, summing all days give a total page views of 938,630. Last year, it was 876,280, so 2022 saw an increase of 7.12%.

Page views per month and year

Comparison of the number of page views per month for all previous years can also be useful. Below two different types of visualizations.

# set new date range to include previous years
start_date_launch <- as.Date("2019-12-16")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date_launch, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date_launch, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# add year and month columns to dataframe
dat$month <- format(dat$date, "%m")
dat$year <- format(dat$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
dat %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(page_views = sum(page_views)) %>%
  ggplot(aes(x = month, y = page_views, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) # better y labels

Another possibility is as follows:

dat2 <- dat

library(lubridate)
dat2$day <- day(dat2$date)
dat2$day_month <- as.Date(paste0(dat2$month, "-", dat2$day), format = "%m-%d")

dat2 %>%
  filter(year != 2019 & page_views < 7500) %>% # remove 2019 and outliers
  ggplot(aes(x = day_month, y = page_views, color = year)) +
  geom_point(size = 1L, alpha = 0.25) + # change size and alpha of points
  geom_smooth(se = FALSE) + # remove confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    color = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) + # better y labels
  scale_x_date(date_labels = "%b") # show only months

From the 2 plots above, we see that the blog globally performed better in terms of page views compared to 2020-2021 (with an exception in the first quarter for which it performed better in 2021).

Note that a comparison between 3 years is fine, but with more years to compare the plots would quickly become unreadable!

Top performing pages

In case you are interested to know the top performing pages (still in terms of page views):

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "pageTitle",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "pageTitle",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "pageTitle")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)


most_viewed_posts <- dat %>%
  mutate(Title = str_sub(pageTitle, start = 1, end = -nchar(" - Stats and R"))) %>% # remove blog site in pageTitle
  count(Title, wt = page_views, sort = TRUE) %>%
  mutate(Title = str_trunc(Title, width = 30)) # keep maximum 30 characters

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Top performing posts are quite similar than last year, that is:

Since they are the most read posts for several years, I have made them available to download via Gumroad.

Page views by country

Knowing from which country your readers come from can also be useful, in particular for e-commerce or blogs that sell physical products in addition to writing posts.

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "country",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "country")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

# table
countries <- dat %>%
  mutate(Country = str_trunc(country, width = 30)) %>% # keep maximum 30 characters
  count(Country, wt = pageviews, sort = TRUE)

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

For Stats and R, most readers are from the US, and to a large extent since the second top country (UK) is quite far in terms of page views.

This trend is the same than last years, except that:

UK has surpassed India, and
Belgium (which used to rank 3rd and 6th in 2020 and 2021, respectively) is now only 8th. Since I am from Belgium, this indicates that the blog is attracting more and more international readers through the years (and proportionally less and less from my country).

Page views per day of week

As discussed in the section about the evolution of daily page views (view this section), there seem to be some days which consistently perform worse than other days.

The following plot will show what are these days:

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("date"),
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = c("date"),
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)
# find day of week
dat$weekday <- wday(dat$date, label = TRUE, abbr = TRUE)

## Reordering dat$weekday so Monday is first
dat$weekday <- factor(dat$weekday,
  levels = c("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
)

# boxplot
library(scales)
dat %>%
  ggplot(aes(x = weekday, y = page_views)) +
  geom_boxplot(fill = "steelblue", outlier.colour = alpha(0.25)) +
  geom_jitter(alpha = 0.25) + # adds transparency
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As expected for a technical blog, there are much more readers during the week than during the weekend.

Thank you note

Thank you to all readers who came to Stats and R this year. You made this journey incredibly more enriching. For next year and the many more to come, I will keep writing about topics for which I am familiar and interested in. So stay tuned!

Thanks for reading. I hope this article helped you to analyze your Google Analytics data in R, or helped you to combine your Universal Analytics and Google Analytics 4 data. For more examples of visualizations or summaries of your GA data in R, see also previous years’ reviews.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

How to keep yourself updated with the latest R news?

Thu, 13 Oct 2022 00:00:00 +0000

Introduction

At the end of one of the training sessions I gave on R, a student asked me the following question:

How do you keep yourself updated with the latest R news?

It is true that R, being open source (meaning that everyone can contribute), is evolving rapidly. This means that even if I am using R for several years and on a daily basis, I like to stay informed in order to stay up to date with the program and the latest coding practices.

In fact, I learn about new packages, new functions and new features almost everyday. Most of them are not particularly useful for my research or my teaching tasks, but sometimes I discover such a nice package or function that I replace my code with new one.¹

The training was an advanced one, so the student had a good knowledge of R and was not looking for more tutorials or courses. She was interested in knowing where to look for updates about current and new R packages and functions.

After sharing my sources with all students following the training, I thought it would be useful to others. In this article, I share my sources—from where I get the latest R updates and news.

The sources are divided into two main categories: Twitter and newsletters.

How do I keep track?

Twitter

To be honest, I mostly use Twitter to keep up to date with R news.

Twitter allows me to follow discussions about statistical methods or approaches, and to keep me informed about publications of new blog posts.

What I particularly like with Twitter is that there is a mix between:

short messages about new functions or packages (most of the time with an illustration or an example), and
announcements of new blog posts that cover specific subjects in details.

For this, I follow people (researchers, professors, bloggers, statisticians, data scientists, etc.) that are working in my domains of interest. For instance, I am mostly interested in the application of statistics in R, data science, biostatistics and data visualization. I thus follow accounts which regularly post about these topics. I also avoid following accounts that cover topics I am not interested in, so that my Twitter feed really shows information I am most likely to be interested in.

Below, you will see a list of some of the accounts I follow, classified by themes. Of course, this is a non-exhaustive list! There are plenty of very inspiring and intelligent people that are not in the list, simply because I cannot afford to put them all.

If you follow people that post regularly about the themes covered below, feel free to add them in the comments. I am always looking for new inspiring accounts to follow.

Note that the accounts are displayed in alphabetical order and the table is searchable.

You will also find many news when exploring #rstats on Twitter.

Newsletters

Besides Twitter, I also read new blog posts and stay informed through these newsletters:

The 2 first newsletters are actually blog aggregators, so collections of many many blogs. If you have your own blog, don’t hesitate to submit it to make it accessible to more people.

You can always subscribe to a blog you like, but if it is related to R, it will most likely be shared via R-Bloggers or R Weekly.

Conclusion

Thanks for reading.

I hope this article will help you to keep track of great blogs, tutorials and other resources about R. Feel free to follow me on Twitter (@statsandr), where I tweet my new articles and retweet everything I find interesting or worth mentioning.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

See for example this article. The package I discovered was so useful and interesting (to me), that I added a new section to the post and now use it instead of the older package.↩︎

Stats and R is 2 years old!

Thu, 16 Dec 2021 00:00:00 +0000

Introduction

Stats and R has been launched exactly two years ago. Like last year, I think it is a good time to do a review of the past 12 months by sharing some figures about the audience of the blog.

This article is not about showing off my numbers, but rather a way to illustrate how to analyze your blog or your website’s traffic using Google Analytics data. Figures regarding the audience of my blog is probably useless to you (and I believe, should not be compared with). However, the code used in this post can be reused for your own blog or website (provided you also use Google Analytics to track your audience).

Note that I use the {googleAnalyticsR} R package to analyze my blog’s Google Analytics data. If you are unfamiliar with this package, see the prerequisites first.

If you have already used that package, you can select your account as followed (make sure to edit the code with your own property name):

library(googleAnalyticsR)

accounts <- ga_account_list()

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

On top of that, I also assume that you have basic knowledge of {ggplot2}—a popular R package to draw nice plots and visualizations.

Analytics

Last year, I mainly focused on the number of sessions. This year, I mainly concentrate on the number of page views to illustrate a different metrics.

For your information, a session is a group of user interactions with your website that take place within a given time frame, whereas a page view, as the name suggests, is defined as a view of a page on your site.

You can always change the metrics by editing metrics = c("pageviews") in the code below. See all available metrics provided by Google Analytics in this article.

Users and page views

As for last year’s review, let’s start with some general numbers, such as the number of users and page views for the entire site.

Note that we analyze traffic over the last year only so we extract data from December 16, 2020 to yesterday (December 15, 2021):

# set date range
start_date <- as.Date("2020-12-16")
end_date <- as.Date("2021-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users pageviews
## 1 549360    876280

Over this past year, Stats and R has attracted 549,360 users (number of new and returning people who visited the site), who generated a total of 876,280 page views.

That is an average of 2401 page views per day in 2021, compared to 1,531 page views per day in 2020 (an increase of 56.81%).

Page views over time

One of the first interesting metrics to analyze your blog’s audience is the evolution of traffic over time.

The daily number of page views over time can be presented in a scatterplot—together with a smoothed line—to analyze the evolution of the audience of your blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = pageviews)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

Although the number of page views varies quite a bit (with an outlier at more than 5,000 page views in a day and as low as less than 1,000 page views for some days), it seems to be cyclical with a dip during summer. It is worth noting that the same dip appeared last year, probably due to the fact that people are less likely to read posts about statistics and R during summer holidays.

So if you write about technical stuff in your blog, low numbers during summer may be expected and does not necessarily mean something is broken on your website.

Page views per channel

Knowing how people come to your blog is also a pretty important factor.

Here is how to visualize the evolution of daily page views per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "pageviews",
  pivots = pivot_ga4("medium", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Page_views, -Date)

# Build up the line plot
trend_long %>%
  filter(Channel != "Total") %>%
  ggplot() +
  aes(x = Date, y = Page_views, group = Channel) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As we can see from the plot above, the large majority of page views come from the organic channel (so from search engines such as Google, Bing, etc.), with some peaks from referral (mainly from R-bloggers and RWeekly) when an article is published. (By the way, you can always subscribe to the newsletter if you want to be informed by email when a new post goes out.)

If you happen to write tutorials, you can also expect that most visitors come from the organic channel. If you are very present on social media, you will most likely attract more visitors from the social channel.

Page views per day of week and month of year

As seen in the previous plot, there are many ups and downs and traffic seems to be cyclical.

To investigate this further, we draw a boxplot of the number of page views for each day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We can also compute the sum and the mean number of page views per day to have a numerical summary instead of a plot:

# compute sum
dat_sum <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Day of week", "Sum", "Mean")

# display table
dat_summary

##   Day of week    Sum     Mean
## 1      Monday 141115 2713.750
## 2     Tuesday 143145 2752.788
## 3   Wednesday 146472 2763.623
## 4    Thursday 140712 2706.000
## 5      Friday 128223 2465.827
## 6    Saturday  85766 1649.346
## 7      Sunday  90847 1747.058

As expected, there are more readers during the week compared to the weekends.

The same analysis can be done for each month of the year instead of days of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("month", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Boxplot
gadata %>%
  ggplot(aes(x = month, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

# compute sum
dat_sum <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Month", "Sum", "Mean")

# display table
dat_summary

##    Month   Sum     Mean
## 1     01 68534 2210.774
## 2     02 80953 2891.179
## 3     03 92629 2988.032
## 4     04 88679 2955.967
## 5     05 81739 2636.742
## 6     06 64460 2148.667
## 7     07 49772 1605.548
## 8     08 46389 1496.419
## 9     09 68046 2268.200
## 10    10 79237 2556.032
## 11    11 77745 2591.500
## 12    12 78097 2519.258

From the plots and the numerical summaries, it is clear that the number of page views is not the same between the days of the week and the months of the year.

Page views per month and year

If you have data over more than a year, it could be useful to compare your monthly blog’s traffic over the years.

With the following code, we create a barplot of the number of daily page views per month and year:

# set new date range
start_date_launch <- as.Date("2019-12-16")

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date_launch, end_date),
  metrics = c("pageviews"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(pageviews = sum(pageviews)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = pageviews, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of page views over the months, but more importantly, compare this evolution across different years.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views per pages.

It is true that the top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

However, for the interested reader, here is how to get the data in R (note that you can change n = 7 in the code below to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)

head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title      n
## 1    Outliers detection in R - Stats and R 119747
## 2 Descriptive statistics in R - Stats a... 109473
## 3 Variable types and examples - Stats a...  83025
## 4 Correlation coefficient and correlati...  65703
## 5 Chi-square test of independence in R ...  62100
## 6 The complete guide to clustering anal...  40440
## 7                 ANOVA in R - Stats and R  32914

If like me you prefer a visualization over a table, here is how to draw this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. The top 3 articles in the past year were:

Be careful that this ranking is based on the total number of page views over the last 12 months. A recent article may thus be found at the bottom of the list simply because it collected page views over a shorter period of time compared to an old article. So it is best to avoid comparing recent articles with older ones, or you can compare articles after having “time-normalized” the number of page views. See previous year’s review for more details and illustrations of this metrics.

You could also be interested in knowing the worst performing ones (to eventually improve them or include them in higher quality posts):

tail(subset(most_viewed_posts, n > 1000),
  n = 7
)

##                                       Title    n
## 49 A guide on how to read statistical ta... 1477
## 50                      About - Stats and R 1470
## 51 How to embed a Shiny app in blogdown?... 1455
## 52                   About me - Stats and R 1277
## 53 One-proportion and chi-square goodnes... 1226
## 54 Running pace calculator in R Shiny - ... 1049
## 55                      Shiny - Stats and R 1018

Note that I intentionally excluded pages with less than 1000 views to remove deleted or hidden pages from the ranking.

Another issue with this ranking is that it may be biased due to some pages which have been duplicated (if you edited the title for example), but at least you have a broad idea of the worst performing pages.

Page views by country

Knowing the country where your readers come from may also be handy for some content creators or marketers.

Location of my readers is not really important for me because I intend to write for everyone, but this may be completely the opposite if you are selling things or running a business/ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)

head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 244752
## 2           India  71797
## 3  United Kingdom  53272
## 4         Germany  38322
## 5          Canada  33812
## 6         Belgium  26421
## 7       Australia  26225
## 8     Philippines  23193
## 9     Netherlands  21944
## 10         Brazil  17956

Again, if you prefer a plot over a table, you can visualize the top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that a large share of readers are from the US—and Belgium (my country) comes only in \(6^{th}\) place in terms of number of page views.

Be careful that, as the number of people located in different countries differs widely, this ranking may hide some insights if you are comparing page views by countries in absolute terms. See why in this section of last year’s review.

User engagement by devices

One may also be interested in checking how users are engaged depending on device’s type. To investigate this, we plot 3 charts describing:

How page views are distributed by type of device?
The average time on page (in seconds) by type of device
The number of page views per session by device type

So first, how page views are distributed by device type?

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot pageviews by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, pageviews)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the large majority of readers visited the blog from a desktop and only a very small proportion comes from a tablet.

This makes sense since I guess many visitors are reading my articles or tutorials while using R (which is only available on desktop).

However, this information of total number of page views per device type does not tell me anything about the time spent on each page and thus the engagement by device type. The following plot answers this question:

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 125,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 3,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

Most readers coming from a tablet actually leave the page very quickly (see the peak around 0 second in the tablet facet).
Distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 125 seconds (= 2 minutes and 5 seconds) and 375 seconds (= 6 minutes and 15 seconds).
Quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (median = 316 seconds on mobile and 266 seconds on desktop, see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop (as shown by the total number of page views by device), it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article more carefully and try to reproduce the code on their own computer. At least that is what I do when I read blogs on mobile versus reading them on desktop. What is even more intriguing, is that it was already the case last year. If someone finds similar results and have a possible explanation, I would be glad to hear from her (if possible, in the comments at the end of the article so everyone can benefit from the discussion).
As a side note, we see that these medians are higher this year compared to last year (266, 316 and 114 seconds in 2021 compared to 190, 228 and 106 seconds in 2020 on desktop, mobile and tablet, respectively). This is somewhat encouraging because it indicates that people spend more time on each page (which is an indication, at least partially, of the quality of the blog for Google).¹

Given this result, I also believe it is helpful to illustrate the number of page views during a session by device type.

In fact, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session (remember that a session is a set of interactions with your website that take place within a given time frame).

We verify this belief via a density plot, and for better readability we exclude visits from tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views per session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nDashed lines represent the medians",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  )

This last plot shows that readers on desktop and mobile visit approximately the same number of pages per session, as indicated by the fact that the two distributions overlap each other and are not distant from each other. It is true that the median is higher for people on desktop than on mobile, but to a very small margin only (and the difference between the two is smaller than in last year’s review).

So to summarize what we learned based on the 3 last plots, we now know that

most readers visited the blog from desktop;
readers on mobile spent more time on each page than readers on desktop (and even more compared to readers on tablet);
users on desktop and mobile seem to have visited approximately the same number of pages per session.

One may wonder why I chose to compare medians instead of means. The main reason is that the median is a more robust way to represent non-normal data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.

Browser information

From a more technical perspective, you could also be interested in the number of page views by browser. I am personally not really interested in knowing which browser my visitors are using the most (mostly because this blog is available on all common browsers), but the most geeky among you may be so.

This information can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most readers visited the site using Chrome, Safari and Firefox (this was expected since they are the most common browsers).

This was the last metrics presented in this review. Of course, many more are possible depending on your R skills (mainly, data manipulation and {ggplot2}) and your expertise in SEO or analyzing Google Analytics data. Hopefully, thanks to this review and possibly from last year too, you will be able to analyze your own blog or website using R and the {googleAnalyticsR} package. At least, this was the aim of the present article.

For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.²

End note

I would also like to add that these figures are not done to be compared with. Every website or blog is unique, every author is unique (with different priorities and different agendas) and more is not always better. I learn a lot thanks to this blog, I use it for personal purposes and for my students as part of my teaching tasks. I will keep writing on it as long as I enjoy it and as long as I have the time to do so, not matter how low or high the number of clicks.

Thanks to all readers of the past year, and see you in a year for another review! In the meantime, if you maintain a blog I would be really happy to hear how you track its performance.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

More time spend on each page is a favorable factor for Google because it means that people are reading it more carefully. If the blog or the post is of mediocre quality, users tend to bounce back quickly (known as bounce rate) and look for an answer to their question somewhere else (leading ultimately to less time spend on the page or site).↩︎
Thanks to the RStudio blog for the inspiration.↩︎

How to track the performance of your blog in R?

Wed, 16 Dec 2020 00:00:00 +0000

Introduction

Stats and R has been launched on December 16, 2019. Since the blog is officially one year old today and after having discussed the main benefits of maintaining a technical blog, I thought it would be a good time to share some numbers and thoughts about it.

In this article, I show how to analyze a blog and its blog posts with the {googleAnalyticsR} R package (see package’s full documentation). After sharing some analytics about the blog, I will also discuss about content creation/distribution and, to a smaller extent, the future plans. This is a way to share my journey as a data science blogger and a way to give you an insight about how Stats and R is doing.

I decided to share with you some numbers through the {googleAnalyticsR} package instead of the regular Google Analytics dashboards for several reasons:

There are plenty of data analysts who are much more experienced than me when it comes to analyzing Google Analytics data via their dedicated platform
I recently discovered the {googleAnalyticsR} package in R and I would like to present its possibilities, and perhaps convince marketing specialists familiar with R to complement their Google Analytics dashboards with some data visualizations made in R (via some ggplot2 visualizations for instance)
I would like to automate the process such in a way that I can easily replicate the same types of analysis across the years. This will allow to see how the blog evolves throughout the years. We know that using R is a pretty good starting point when it comes to automation and replication—especially thanks to R Markdown reports

I am not an expert in the field of digital marketing, but who knows, it may still give some ideas to data analysts, SEO specialists or other bloggers on how to track the performance of their own blog or website using R. For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.¹

Before going further, I would like to remind that I am not making a living from my blog (far from it!) and it is definitely not my goal as I do not believe that I would be the same kind of writer if it was my main occupation.

Prerequisites

As for any package in R, we first need to install it—with install.packages()—and load it—with library():²

# install.packages('googleAnalyticsR', dependencies = TRUE)
library(googleAnalyticsR)

Next, we need to authorize the access of the Google Analytics account using the ga_auth() function:

ga_auth()

Running this code will open a browser window on which you will be able to authorize the access. This step will save an authorization token so you only have to do it once.

Make sure to run the ga_auth() function in a R script and not in a R Markdown document. Follow this procedure if you want to use the package and its functions in a R Markdown report or in a blog post like I did for this article.

Once we have completed the Google Analytics authorization, we will need the ID of the Google Analytics account we want to access. All the available accounts linked to your email address (after authentication) are stored in ga_account_list():

accounts <- ga_account_list()

accounts

## # A tibble: 2 × 10
##   accountId account…¹ inter…² level websi…³ type  webPr…⁴ webPr…⁵ viewId viewN…⁶
##   <chr>     <chr>     <chr>   <chr> <chr>   <chr> <chr>   <chr>   <chr>  <chr>  
## 1 86997981  Antoine … 129397… STAN… https:… WEB   UA-869… Antoin… 13318… All We…
## 2 86997981  Antoine … 218214… STAN… https:… WEB   UA-869… statsa… 20812… All We…
## # … with abbreviated variable names ¹accountName, ²internalWebPropertyId,
## #   ³websiteUrl, ⁴webPropertyId, ⁵webPropertyName, ⁶viewName

accounts$webPropertyName

## [1] "Antoine Soetewey" "statsandr.com"

As you can see I have two accounts linked to my Google Analytics profile: one for my personal website (antoinesoetewey.com) and one for this blog.

Of course, I select the account linked to this blog:

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

Make sure to edit the code with your own property name.

We are now finally ready to use our Google Analytics data in R for a better analysis!

Analytics

Users, page views and sessions

Let’s start with some general numbers, such as the number of users, sessions and page views for the entire site. Note that for the present article, we use data over the past year, so from December 16, 2019 to December 15, 2020:

# set date range
start_date <- as.Date("2019-12-16")
end_date <- as.Date("2020-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "sessions", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users sessions pageviews
## 1 321940   428217    560491

In its first year, Stats and R has attracted 321,940 users, who generated a total of 428,217 sessions and 560,491 page views (that is an average of 1531 page views per day).

For those unfamiliar with Google Analytics data and the difference between these metrics, remember that:

a user is the number of new and returning people who visit your site during a set period of time
a session is a group of user interactions with your website that take place within a given time frame
a page view, as the name suggests, is defined as a view of a page on your site

So if person A reads three blog posts then leave the site and person B reads one blog post, your about page then leave the site, Google Analytics data will show 2 users, 2 sessions and 5 page views.

Sessions over time

In addition to the rather general metrics presented above, it is also interesting to illustrate the daily number of sessions over time in a scatterplot—together with a smoothed line—to analyze the evolution of the blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = sessions)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "darkgrey", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 15.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

(See how to draw plots with the {ggplot2} package, or with the {esquisse} addin if you are not familiar with the package.)

As you can see, there was a huge peak of traffic around end of April, with almost 12,000 users in a single day. Yes, you read it well and there is no bug. The blog post “A package to download free Springer books during Covid-19 quarantine” went viral and generated a massive traffic for a few days. The daily number of sessions returned to a more normal level after a couple of days. We also observe an upward trend in the last months (since end of August/beginning of September), which indicates that the blog is growing in terms of number of daily sessions.

Note that I decided to focus on the number of sessions and the number of page views in this section and the following ones, but you can always change to your preferred metrics by editing metrics = c("sessions") in the code. See all available metrics provided by Google Analytics in this article.

Sessions per channel

Knowing how people come to your blog is a pretty important factor. Here is how to visualize the evolution of daily sessions per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "sessions",
  pivots = pivot_ga4("medium", "sessions"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Sessions, -Date)

# Build up the line plot
ggplot(trend_long, aes(x = Date, y = Sessions, group = Channel)) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We see that a large share of the traffic is from the organic channel, which indicates that most readers visit the blog after a query on search engines (mostly Google). In my case, where most of my posts are tutorials and which help people with specific problems, it is thus not a surprise that most of my traffic comes from organic search.

We also notice some small peaks of sessions generated from the referral and direct channels, which are probably happening on the date of publication of each article.

We also see that there seems to be a recurrent pattern of ups and downs in the number of daily sessions. Those are weekly cycles, with less readers during the weekend and which indicates that people are working on improving their statistical or R knowledge mostly during the week (which actually makes sense!).

Sessions per day of week

As shown above, traffic seems to be different depending on the day of week. To investigate this further, we draw a boxplot of the number of sessions for every day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "sessions",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = sessions)) +
  geom_boxplot() +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see there are some outliers, probably due (in part at least) to the article that went viral. For the sake of illustration, below the same plot after removing points considered as potential outliers according to the interquartile range (IQR) criterion (i.e., points above or below the whiskers), and after a couple of visual improvements:

# boxplot
gadata %>%
  filter(sessions <= 3000) %>% # filter out sessions > 3,000
  ggplot(aes(x = dayOfWeek, y = sessions, fill = dayOfWeek)) + # fill boxplot by dayOfWeek
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 3,000 excluded"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(legend.position = "none") # remove legend

After excluding data points above 3,000, it is now easier to see that the median number of sessions (represented by the horizontal bold line in the boxes) is the highest on Wednesdays, and lowest on Saturdays and Sundays.

The difference in sessions between weekdays is, however, not as large as expected.

Sessions per day and time

We have seen the traffic per day of week. The example below shows a visualization of traffic, broken down this time by day of week and hour of day in a density plot. In this plot, the device type has also been added for more insights.

## Get data by deviceCategory, day of week and hour
weekly_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("deviceCategory", "dayOfWeekName", "hour"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Manipulation using dplyr
weekly_data_sessions <- weekly_data %>%
  group_by(deviceCategory, dayOfWeekName, hour)

## Reordering weekly_data_sessions$dayOfWeekName to have Monday as first day of the week
weekly_data_sessions$dayOfWeekName <- factor(weekly_data_sessions$dayOfWeekName,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

## Plotting using ggplot2
weekly_data_sessions %>%
  ggplot(aes(hour, sessions, fill = deviceCategory, group = deviceCategory)) +
  geom_area(position = "stack") +
  labs(
    title = "Sessions per day and time",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    x = "Time",
    y = "Sessions",
    fill = "Device" # edit legend title
  ) +
  theme_minimal() +
  facet_wrap(~dayOfWeekName, ncol = 2, scales = "fixed") +
  theme(
    legend.position = "bottom", # move legend
    axis.text = element_text(size = 7) # change font of axis text
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

The above plot shows that:

traffic increases in the afternoon then declines in the late evening
traffic is the highest from Monday to Thursday, and lowest on Saturday and Sunday

In addition to that, thanks to the additional information on the device category, we also see that:

the number of sessions on tablet is low (it is so low compared to desktop and mobile that it is not visible on the plot), and
the number of sessions on mobile seems to be quite stable during the entire day,
as opposed to sessions on desktop which are highest at the end of the day.

Sessions per month and year

In the following code, we create a barplot of the number of daily sessions per month and year:

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# sessions by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  group_by(year, month) %>%
  summarize(sessions = sum(sessions)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = sessions, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "Month",
    title = "Sessions per month and year",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of sessions over the months, and compare this evolution across different years.

At the moment, since the blog is online only since December 2019, the year factor is not relevant. However, I still present the visualization for other users who work on older websites, and also to remind my future self to create this interesting barplot when there will be data for more than a year.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views for the different pages. The top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

For the interested reader, here is how to get the data in R (note that you can change n = 7 in the code to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)
head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title     n
## 1 A package to download free Springer b... 85684
## 2 Variable types and examples - Stats a... 44951
## 3 Descriptive statistics in R - Stats a... 43621
## 4    Outliers detection in R - Stats and R 32560
## 5 The complete guide to clustering anal... 27184
## 6 Correlation coefficient and correlati... 21581
## 7                              Stats and R 16786

Here is how to visualize this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. For instance, I never thought that the post illustrating the different types of variables that exist in statistics (ranked #2) would be so appreciated when I wrote it.

This is something I learned with this blog: at the time of writing, there are some posts which I think no one will care about (and which I mostly write as a personal note for myself), and others which I think people will find very useful. However, after a couple of weeks after publication, sometimes I realize that it is actually exactly the opposite.

Time-normalized page views

When it comes to comparing blog posts, however, this is not as simple.

Based on the above barplot and without any further analysis, I would conclude that my article about variable types is performing much better than the one about outliers detection in R.

However, if I tell you that the article on outliers detection has been published on August 11 and the one about variable types on December 30, you will agree that the comparison does not make much sense anymore since page views for these articles were counted over a different length of time. One could also argue that a recent article had less time to generate backlinks, so it is unfair to compare it with an old post which had plenty of time to be ranked high by Google.

In order to make the comparison more “fair”, we would need to compare the number of page views for each post since their date of publication. The following code does precisely this:³

it pulls daily data for a bunch of pages,
then tries to detect their publication date,
time-normalizes the traffic for each page based on that presumed publication date,
and finally, plots the daily traffic from the publication date on out, as well as overall cumulative traffic for the top n pages

# figure out when a page actually launched by finding the
# first day where the page had at least 2 unique pageviews
first_day_pageviews_min <- 2

# exclude pages that have total traffic (daily unique pageviews) that are relatively low
total_unique_pageviews_cutoff <- 500

# set how many "days since publication" we want to include in our plot
days_live_range <- 180

# set number of top pages to display
n <- 7

# Create a dimension filter object
# You need to update the "expressions" value to be a regular expression that filters to
# the appropriate set of content on your site
page_filter_object <- dim_filter("pagePath",
  operator = "REGEXP",
  expressions = "/blog/.+"
)

# Now, put that filter object into a filter clause. The "operator" argument can be AND # or OR...but you have to have it be something, even though it doesn't do anything
# when there is only a single filter object.
page_filter <- filter_clause_ga4(list(page_filter_object),
  operator = "AND"
)

# Pull the GA data
ga_data <- google_analytics(
  viewId = view_id,
  date_range = c(start_date, end_date),
  metrics = "uniquePageviews",
  dimensions = c("date", "pagePath"),
  dim_filters = page_filter,
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Find the first date for each post. This is actually a little tricky, so we're going to write a
# function that takes each page as an input, filters the data to just include those
# pages, finds the first page, and then puts a "from day 1" count on that data and
# returns it.
normalize_date_start <- function(page) {
  # Filter all the data to just be the page being processed
  ga_data_single_page <- ga_data %>% filter(pagePath == page)

  # Find the first value in the result that is greater than first_day_pageviews_min. In many
  # cases, this will be the first row, but, if there has been testing/previews before it
  # actually goes live, some noise may sneak in where the page may have been live, technically,
  # but wasn't actually being considered live.
  first_live_row <- min(which(ga_data_single_page$uniquePageviews > first_day_pageviews_min))

  # Filter the data to start with that page
  ga_data_single_page <- ga_data_single_page[first_live_row:nrow(ga_data_single_page), ]

  # As the content ages, there may be days that have ZERO traffic. Those days won't show up as
  # rows at all in our data. So, we actually need to create a data frame that includes
  # all dates in the range from the "publication" until the last day traffic was recorded. There's
  # a little trick here where we're going to make a column with a sequence of *dates* (date) and,
  # with a slightly different "seq," a "days_live" that corresponds with each date.
  normalized_results <- data.frame(
    date = seq.Date(
      from = min(ga_data_single_page$date),
      to = max(ga_data_single_page$date),
      by = "day"
    ),
    days_live = seq(min(ga_data_single_page$date):
    max(ga_data_single_page$date)),
    page = page
  ) %>%
    # Join back to the original data to get the uniquePageviews
    left_join(ga_data_single_page) %>%
    # Replace the "NAs" (days in the range with no uniquePageviews) with 0s (because
    # that's exactly what happened on those days!)
    mutate(uniquePageviews = ifelse(is.na(uniquePageviews), 0, uniquePageviews)) %>%
    # We're going to plot both the daily pageviews AND the cumulative total pageviews,
    # so let's add the cumulative total
    mutate(cumulative_uniquePageviews = cumsum(uniquePageviews)) %>%
    # Grab just the columns we need for our visualization!
    select(page, days_live, uniquePageviews, cumulative_uniquePageviews)
}

# We want to run the function above on each page in our dataset. So, we need to get a list
# of those pages. We don't want to include pages with low traffic overall, which we set
# earlier as the 'total_unique_pageviews_cutoff' value, so let's also filter our
# list to only include the ones that exceed that cutoff. We also select the top n pages
# in terms of page views to display in the visualization.
library(dplyr)
pages_list <- ga_data %>%
  group_by(pagePath) %>%
  summarise(total_traffic = sum(uniquePageviews)) %>%
  filter(total_traffic > total_unique_pageviews_cutoff) %>%
  top_n(n = n, total_traffic)

# The first little bit of magic can now occur. We'll run our normalize_date_start function on
# each value in our list of pages and get a data frame back that has our time-normalized
# traffic by page!
library(purrr)
ga_data_normalized <- map_dfr(pages_list$pagePath, normalize_date_start)

# We specified earlier -- in the `days_live_range` object -- how many "days since publication" we
# actually want to include, so let's do one final round of filtering to only include those
# rows.
ga_data_normalized <- ga_data_normalized %>% filter(days_live <= days_live_range)

Now that our data is ready, we create two visualizations:

Number of page views by day since publication: this plot shows how quickly interest in a particular piece of content drops off. If it is not declining as rapidly as the other posts, it means you are getting sustained value from it
Cumulative number of page views by day since publication: this plot can be used to compare blog posts on the same ground since number of page views is shown based on the publication date. To see which pages have generated the most traffic over time, simply look from top to bottom (at the right edge of the plot)

Note that both plots use the {plotly} package to make them interactive so that you can mouse over a line and find out exactly what page it is (together with its values). The interactivity of the plot makes it also possible to zoom in to see, for instance, the number of page views in the first days after publication (instead of the default length of 180 days),⁴ or zoom in to see only the evolution of the number of page views below/above a certain threshold.

# Create first plot
library(ggplot2)

gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Page views by day since publication",
    x = "Days since publication",
    y = "Page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
library(plotly)
ggplotly(gg, dynamicTicks = TRUE)

The plot above shows again a huge spike for the post that went viral (orange line). If we zoom in to include only page views below 1500, comparison between posts is easier and we see that:

The post on the top 100 R resources on Coronavirus (purple line) generated comparatively more traffic than the other posts in the first 50 days (\(\approx\) 1 month and 3 weeks) after publication. However, traffic gradually decreased up to the point that after 180 days (\(\approx\) 6 months), it attracted less traffic than other more performing posts.
The post on outliers detection in R (blue line) did not get a lot of attention in the first weeks after its publication. However, it gradually generated more and more traffic up to the point that, after 60 days (\(\approx\) 2 months) after publication, it actually generated more traffic than any other post (and by a relatively large margin).
The post on correlation coefficient and correlation test in R (green line) took approximately 100 days (\(\approx\) 3 months and 1 week) to take off, but after that it generated quite a lot of traffic. This is interesting to keep in mind when analyzing recent posts, because they may actually follow the same trend in the long run.

# Create second plot: cumulative
gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = cumulative_uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Cumulative page views by day since publication",
    x = "Days since publication",
    y = "Cumulative page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
ggplotly(gg, dynamicTicks = TRUE)

Compared to the previous plot, this one shows the cumulative number of page views since the date of publication of the post.

Without taking into consideration the post that went viral and which, by the way, does not really generate much traffic anymore (which makes sense since the incredible campaign from Springer to offer their books for free during the COVID-19 quarantine has ended), we see that:

The post on outliers detection has surpassed the post on top R resources on Coronavirus in terms of cumulative number of page views after around 120 days (\(\approx\) 4 months) after publication, which indicates that people are looking at this specific problem. I remember that I wrote this post because, at that time, I had to deal with the problems of outliers in R and I did not found a neat solution online. So I guess, the fact that resources on a specific topic are missing helps to attract visitors looking for an answer to their question.
Among the remaining posts, the ranking of the most performing ones in terms of cumulative number of page views within 180 days (\(\approx\) 6 months) after publication is the following:

The analyses so far give you already a good understanding of the performance of your blog. However, for the interested readers, we show other important metrics in the following sections.

Page views by country

In the following we are interested in seeing where the traffic comes from. This is particularly interesting to get to know your audience, and even more important if you are running a business or an ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)
head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 131306
## 2           India  38800
## 3         Belgium  32983
## 4  United Kingdom  25833
## 5          Brazil  18961
## 6         Germany  17851
## 7           Spain  17355
## 8          Canada  13880
## 9          Mexico  12882
## 10    Philippines  12540

To visualize this table of top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that readers from the US take the largest share of the number of page views (by quite a lot actually!), and Belgium (my country) comes in third place in terms of number of page views.

Given that the US population is much larger than the Belgian population (\(\approx\) 331 million people compared to \(\approx\) 11.5 million people, respectively), the above result is not really surprising. Again, for a better comparison, it would be interesting to take into consideration the size of the population when comparing countries.

Indeed, it could be that a large share of the traffic comes from a country with a large population, but that the number of page views per person (or per 100,000 inhabitants) is higher for another country. This information about top performing countries in terms of page views per person could give you insights on which country do the most avid readers come from. This is beyond the scope of this article, but you can see examples of plots which include the information on the population size in these COVID-19 visualizations. I recommend to apply the same methodology to the above plot for a better comparison.

If you are thinking about doing the extra step of including population size when comparing countries, I believe that it would be even better to take into account the information on computer access in each country as well. If we take India as example: at the time of writing this article, its population amounts to almost 1.4 billion. However, the percentage of Indian people having access to a computer is undoubtedly lower than in US or Belgium (again, at least at the moment). It would therefore make more sense to compare countries by comparing the number of page views per people having access to a computer.⁵

Browser information

For more technical aspects, you could also be interested in the number of page views by browser. This can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most visits were, as expected, from Chrome, Safari and Firefox browsers.

User engagement by devices

One may also be interested in checking how users are engaged on different types of devices. To do so, we plot 3 charts describing:

How many sessions were made from the different types of devices
The average time on page (in seconds) by type of device
The number of page views per session by device type

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot sessions by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, sessions)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the majority of readers visited the blog from a desktop, and a small number of readers from a tablet.

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 25,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 2,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

some readers on tablet have a very low average time spent on each page (see the peak around 0 second in the tablet facet)
distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 100 seconds (= 1 minute 40 seconds) and 400 seconds (= 6 minutes 40 seconds)
quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop, it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article carefully and try to reproduce the code on their computer.⁶

Given this result, it would be interesting to also illustrate the number of page views during a session, represented by device type.

Indeed, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session. We verify this via a density plot, and for better readability we exclude data points above 2.5 page views/session and we exclude visits from a tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(pageviewsPerSession <= 2.5 & deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views/session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 2.5 excluded",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  ) +
  geom_text(
    aes(
      x = med, y = 2.75,
      label = paste0("Median = ", round(med, 2), " page views/session"),
      color = deviceCategory
    ),
    angle = 90,
    vjust = 2,
    size = 3,
    show.legend = FALSE # remove legend
  )

This last plot confirms our thoughts, that is, although people on mobile seem to spend more time on each page, people on desktop tend to visit more pages per session.

(One may wonder why I chose to compare medians instead of means. The main reason is that not all distributions considered here are bell-shaped (especially for data on tablet) and there are many outliers. In these cases, the mean is usually not the most appropriate descriptive statistics and the median is a more robust way to represent such data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.)

This is the end of the analytics section. Of course, many more visualizations and data analyses are possible, depending on the site that is tracked and the marketing expertise of the analyst. This was an overview of what is possible, and I hope it will give you some ideas to explore your Google Analytics data further. Next year, I may also include forecasts and make annual comparisons. See also some examples of other analyses in this GitHub repository and this website.

As a side note, I would like to add the following: even if tracking the performance of your blog is important to understand how you attract visitors and how they engage with your site, I also believe that looking at your Google Analytics stats too often is not optimal, nor sane.

Talking about personal experience: at the beginning of the blog I used to look very often at the number of visitors in real-time and the audience. I was kind of obsessed to know how many people were right now on my blog and I was constantly checking if it performed better than the day before in terms of number of visitors. I remember that I was spending so much time looking at these metrics in the first weeks that I felt I was wasting my time. And the time I was wasting looking at my Google Analytics stats was lost not creating good quality content for the blog, working on my thesis/classes, or other projects.

So when I realized that, I deleted the Google Analytics app from my smartphone and forced myself not to look at my stats more than once a month (just to make sure there is no critical issues that need to be fixed). From that moment onward, I stopped wasting my time on things I cannot control, and I got more satisfaction from writing articles because I was writing them for myself, not for the sake of seeing people reading them. This change was like a relief, and I am now more satisfied with my work on this blog compared to the first weeks or months.

Everyone is different and unique so I am not saying that you should do the same. However, if you feel that you look too much at your stats and sometimes lose motivation in writing, perhaps this is one potential solution.

Content

Now that we have seen how we can analyze Google Analytics data and track the performance of a website or blog in details, I would like to share some thoughts about content creation, content distribution and the future plans.

Finding topics

One of the biggest challenges I face with my blog is creating proper content. In the best case scenario, I would like to:

create useful content
about subjects I am really familiar with,
which I enjoy and
which fit into the blog.

By sharing only articles about topics I am familiar with, the writing process is easier and quicker. Indeed, most of the articles I wrote cover topics I teach at university, so a large part of the preliminary research is already done when I decide to write about it. Also (and this is not negligible), questions and incomprehension from students allow me to see the points I should focus on when writing about it, and how to present it to make it accessible and comprehensible to most people.

Moreover, by making my thoughts public, I often have the chance to confront them with other points of view, which allows me to study the topic even further. This in turn accelerates the writing process even more when writing about a related topic.

I also tend to write only about things I enjoy or I am interested in. I prefer quality over quantity, so writing an article from A to Z takes quite a long time depending on the depth of the subject. Since it takes time (even without taking into account the time spent after publication) and I have a full-time job, I really focus on topics I enjoy in order to keep seeing this blog as a source of pleasure, and not as a work or an obligation.

The fact that I write only about things I am familiar with, which I enjoy and when I have some free time (which mostly depend on the ongoing projects related to my PhD thesis) makes it hard for me to follow a defined pace for posts. This is why my writing schedule has been a bit inconsistent during this first year and is likely to be similar in the future as I do not want to be in the position to force myself to write.⁷

Content distribution

Even if we all agree that bloggers should write for themselves first and not for the sake of having lots of readers, it is still appreciated when your content is being read by others.

So although I do not like abusive self-promotion and I do not feel comfortable sharing my blog posts to all existing Facebook, LinkedIn and Reddit groups, somehow people need to get informed that you have written something.

For this reason, I created a Twitter account where I share new posts immediately after publishing them on the blog. In addition to posting them on Twitter, I also share the link by email to people who subscribed to the newsletter of the blog.

I also managed to get my content published on Medium through the Towards Data Science publication, R-bloggers and R Weekly. A non-negligible part of my audience comes from these referral, especially during the couple of days after publication.

By distributing the content this way, I do not feel pushy (something I want to avoid at all costs!) because people decided by themselves to receive the content I publish (e.g., they subscribed to the newsletter, they followed the blog on Twitter, they chose to read blog aggregators, etc.). So in some sense, I do not distribute my content “without their prior consent” and they can always choose not to see my posts anymore.

However, as you can see from the plot of the number of session per channel, you see that most readers come from the organic channel, so from search engines. And for this channel, apart from creating quality content (and some basic knowledge in SEO), I do not have any control on how well it is presented nor distributed to people. For this channel, it is basically search engine algorithms which decide to promote my content or not, and if they do, how well it is promoted. The only control I have regarding ranking factors is simply to create quality content.

So even if I believe that having a content distribution strategy definitely helps in reaching more people and growing your blog, it does not make everything. Creating quality content is, in my view of a beginner in SEO and marketing, the best strategy for my posts to be read.

For this very specific reason, now that I feel I have settled a decent distribution strategy, I no longer spend my energy and time on this matter.

So I just simply try to enjoy writing on my blog, and the rest will eventually follow. If not, I am still learning a lot from this blog anyway so I do not see it as wasted time.

A small note about ads

I know that an easy way to get money out of your blog is to run ads and I understand that people do it if it is profitable. However, I often find ads too intrusive or annoying and I personally do not like to read blog posts which include many ads.

To keep the reading process as enjoyable as possible, as you can see, I do not display any advertising on my blog. As long as the costs of running this website and the related open source projects (e.g., my Shiny apps, etc.) are not too high, I do not expect to include ads.

If in the future costs were to increase, I will still try to avoid putting ads as much as possible and I will try to rely on sponsorship programs & donations and on paid side projects for people who need help for their statistical analyses.

Future plans

After exactly one year of blogging, I am asking myself the following:

What do I want statsandr.com to be?

As already said, as long as I enjoy writing stuff I am passionate about on this blog, I will continue. This is very important to me.

In addition to that, I would like it to be a place to share knowledge. The field of statistics and R (and data science in general) is evolving at an extremely fast pace—and more and more people have many interesting things to say.

Moreover, I learned a lot since the launch this blog. But I learned even more when working in collaboration with someone else (see for instance these collaborations). I see so many learning opportunities when collaborating that I would love to see this blog as a place to share knowledge, but not only my knowledge.

To be more precise, I would love to hear about other researchers, statisticians, R lovers, data scientists, authors, etc. who want to collaborate with me. This could lead, in the end, to:

a guest post if you have a good idea on what you want to write about but need a way to share it (and since you have access to my Google Analytics data in the past year through this article you have a good idea of the number of visitors who will see your content),
a piece of content written together if you believe our skills and knowledge are complementary (see all articles for an overview of what I like to write about),
a piece of code, a R package, a visualization or a Shiny app you want to create together (or just share),
a book or a course on statistics and/or R,
or anything else you have in mind (I am open to new ideas and challenges).

With this article, you see the figures and the audience of the blog. With all my articles, you can see my strengths and weaknesses.

So I will finish this section by saying that, if anyone is willing to work on something together (which does not need to be huge), you can always contact me or leave a comment at the end of this post. I am looking forward to hearing from you.

Thank you note

Last but not least, I also wanted to leave a short thank you note to the Towards Data Science, R-bloggers and R Weekly teams that have cross-posted most of my articles in their respective publications and therefore allowed me to share my thoughts to a larger—and very knowledgeable—audience.

Also, I would like to thank all active readers for their comments, constructive feedback and support so far. I am looking forward to sharing more quality content through this blog and I hope it will keep being useful to many of you, in parallel to being useful to me.

A special thanks to Mark Edmondson, the author of the {googleAnalyticsR} package and all people who wrote tutorials using the package, which were used as inspiration for this blog post. More broadly, thanks also to the open source community which is of great help in learning R.

Thanks for reading. I hope that you learned how to track the performance of your website or blog in R using the {googleAnalyticsR} package. See you next year for a second review, and in the meantime, if you maintain a blog I would be really happy to hear how you track its performance. Feel free to let me know via the comments below!

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Thanks to the RStudio blog for the inspiration.↩︎
See more ways to install and load R packages.↩︎
Many thanks to dartistics.com for the code. Note that for better readability of the plots, I slightly edited the code: (i) to display only the top n pages in terms of traffic instead of all pages, (ii) to change the theme to theme_minimal() and (iii) to make the axis ticks dynamic when zooming in or out (in the ggplotly() function).↩︎
Note that the default period of 180 days can also be changed in the code.↩︎
This may require some assumptions if the percentage of people having access to a computer is not readily available. I believe, however, that it would still be more appropriate than just taking into account the population size—especially when comparing developed with developing countries.↩︎
At least that is what I do when I read blogs on mobile versus reading them on desktop.↩︎
Note, however, that research has shown that scheduling time for writing is a good way to productive writing (Silvia, 2019). This is why, unlike for my blog, I have set regular writing periods in my calendar (and I try to stick to it no matter how busy I am at that time) dedicated to my PhD thesis.↩︎

Why do I have a data science blog? 7 benefits of sharing your code

Wed, 02 Sep 2020 00:00:00 +0000

My blog statsandr.com was launched in December 2019. Although 9 months of writing is a very short period compared to others, I can already say that it’s been an incredible and very enriching adventure!

With 45 articles published (at the time of writing this article) and topics ranging from descriptive statistics, probability, inferential statistics to R Markdown and data visualization, I have seen many benefits of sharing my code through a technical blog.

In this article, I highlight 7 of them (in no particular order) with the hope that it will give ideas and incentives to some of you. In the end of this article, I also mention a couple of modern solutions needed for starting your own blog.

Note that in these 9 months, I did not make a living from my blog and this is not my goal as I would need to make it a priority.¹ However, I have received enough positive feedback from readers, and more importantly, I have learned enough to continue writing.

(For the interested reader, see a review of the blog after one year—and some thoughts about the future plans. In this review, I track its performance in R by analyzing page views, sessions, users and engagement with the {googleAnalyticsR} package.)

#1 Learn by writing

I really enjoy learning new stuff in many different domains. I learn (and I am still learning) a lot by teaching statistics to students from diverse backgrounds as part of my teaching assistant position at university.

Before launching this blog, I believed that I understood a statistical concept as soon as I was able to teach it to my students. If I was not able to explain it in a clear and understandable way, it meant that I needed to study it more thoroughly because I actually did not fully understand it.

This is often referred as the Feynman technique. This method of learning is based on the fact that in order to fully master a topic you need to be able to explain it back to someone in simple terms.

Throughout this blog, I actually realized that in order to learn and fully understand something new, one must:

be able to clearly communicate it and teach it in simple terms,
but also be able to write it down in a precise and concise manner

So although this blog was first launched to share statistical concepts I am most familiar with (hoping that it would be useful to some people), I now also use it to learn by writing. I think that this additional way of learning is actually as powerful as teaching because writing allows me to consolidate my understanding.

Obviously, I am learning mainly about statistics and its applications in R as they are the main topics of the blog. However, I never thought that I would also learn so much about:

web development and SEO/analytics (which are increasingly important skills nowadays)
project management (as you build something from scratch and wish to develop it)
writing (a skill I still need to improve as a non-native English speaker)²
communicating results (as any data scientist will tell you, results without proper communication are useless and writing a blog is a great practice)
marketing/public relation/brand management (think about social networking and how to deal with all sorts of questions from readers)
etc.

Maintaining a blog teaches me essential skills that are usually taught in full-time jobs. With a blog, you are responsible for everything from the content to readers’ inquiries, similar to an employee in charge of a project and who has to deal and communicate with end users.

Don’t get me wrong, apart from exceptions, I do not think that a blog can completely replace a job in terms of personal development, but it definitely helps to learn a broad range of important skills. Reading books and completing data science online courses are other examples to gain skills in addition to a job, but a blog tends to be more diverse and applied, making it more beneficial (in my opinion).

Another way I am learning is by doing research about the topic I am writing about.

An example of it is my post on outliers detection in R. As part of any descriptive analysis, I am used to check for potential outliers. Since I was familiar with this topic, I decided to write about it. However, I found the post not complete enough so I did some further research.

It turned out that there were in fact several statistical tests that I did not know about. I wrote about these tests and I now include these new techniques whenever I check for potential outliers.

Last but not least, I often receive emails from readers asking to write about a topic of their choice. It sometimes happens that I never heard about their suggested topic, so I do some research out of curiosity. Even if I still do not write about it because I am not familiar enough with it, at least I am aware that it exists and I know more or less what this is about.

If the sole benefit of learning did not convince you to start your blog, see other benefits I have experienced in the following sections.

#2 Get feedback

Get feedback, suggestions and constructive critics from more experienced users. Remarks greatly help in correcting typos and bugs present in my code, and they also help in improving my R skills.

You will be amazed to see that, although you spent countless hours to check your code, there will always be someone who will spot a typo that you missed. Valuable feedback from people all over the world has, for instance, definitely improved the quality and completeness of my R Shiny applications that I use on a daily basis.

A blog can therefore be seen as a powerful peer-review method of your understanding of a concept, code or R practices. It is also better to make mistakes when working on a toy example and correct them, than to make mistakes on a real project at your workplace.

Furthermore, some people aggregate their blog posts into articles or books, so blogging can be seen as a way to:

make incremental progress toward a longer-term publishing goal, and
receive feedback at each step of this long-term goal.

#3 Personal note to remind my future self

How many times you searched for a piece of code in the folders of your computer, to finally look for the solution on Google because you could not remember for which project you wrote that piece of code? It happened to me every day.

With blog posts organized by topics, it now takes me much less time (and less frustration!) to find code snippets that I wrote several months ago. This also allows me to keep my code and R practices up-to-date, as I only have to edit them in one place.

An example of this is my article about graphics in R with {ggplot2}. I prefer plots with the {ggplot2} package over plots available by default in R base, but I cannot remember all layers and their arguments.

Now every time I struggle with a plot using this package, I simply revisit the corresponding article to find the solution. Same goes for many of my articles, every time I have forgotten the code or a nuance around how it works.

You could store your code in different files (R Markdown documents or R scripts for example, as I used to do in the past), but blog posts have the advantages that:

code snippets are highlighted, and
by making it available to the world, you are forced to keep it tidy, complete and up-to-date.

#4 Contribute to the open source community

I have learned so much and I keep learning everyday about R thanks to great resources made available for free by developers and scientists who believe in open source and free materials.

Making all my code and articles freely available through a blog is in some sense, my way of:

“paying back” people who helped me to learn and thanks to whom I stand where I am now, and
pay in advance the many more from who I will learn in the future.

If you also believe in sharing your knowledge or expertise, having a blog is definitely a great way to contribute to the community. Your contribution does not necessarily have to be huge, as long as it adds something, someone will use it.

Remember that everyone started as a beginner, and even world experts are beginners in other domains. And you will see that as you keep sharing your knowledge with others, some people will appreciate it because they can learn from it.

As far as I am concerned, if my small contribution is useful for some people in having a better understanding of statistics or in learning R, my goal will be reached. Others who believe passionately in a topic may want to inform people about it, in the hope that more people in turn contribute to the topic.

You are totally free to choose the contribution you want to make to the community.

As a side note, Tom Preston-Werner (Github’s founder) wrote a great post about why we should open source (almost) everything! I can only agree with his arguments and although the post was written in 2011, I believe it still holds today.

#5 Stay humble, stay curious

Since launching my blog, I discovered plenty of other data science blogs of high quality. The more I see new things about different topics and the more I see people doing incredible stuff, the more I realize that I actually do not know much. This reminds me to stay humble.

In addition to that, by sharing my code with scientists from diverse backgrounds and coming from all over the world, it allows me to consider other perspectives and practice open-mindedness, which in turn helps me to stay curious.

Being humble and curious is, in my opinion, a good starting point to keep learning. I would not want to lose my curiosity, otherwise I may lose my appetite for learning.

For your information, R-bloggers and R Weekly are two great blog aggregators focusing on R, and the Towards Data Science publication on Medium is full of high quality blog posts covering many topics related to data science. Subscribe to these if you are interested in discovering more technical blogs.

#6 Learn to be less perfectionist and to prioritize

No matter how much time you spend on your blog, remember that everything cannot always be perfect. As a perfectionist, I cannot deny that I would love everything to run perfectly in all aspects of my life.

At the time of writing my first articles, my tendency for perfectionism was sometimes so prevalent that it was a real weakness: I could spend several minutes thinking whether a comma between two words was needed or not (which would of course make no difference at all).

However, at some point, improving something takes so much time and energy that the added value is not as large as the added value you could create by devoting your time in producing something new.

This gets even worse as you keep learning every day. As a consequence, an article which seemed to be perfect in the past may no longer be viewed as perfect today, so you want to change or add a small detail in all your previous articles.

I am not saying you should not edit a blog post, it is even recommended! Nonetheless, I advise you to edit it only if it really adds a significant value to it. Otherwise, I believe it is best to spend that time and effort in creating something else.

With a blog you will thus gradually learn to put your effort and time (which are limited resources and seem to be scarcer with the years) where it is most productive. In other words, you will learn to prioritize.

#7 Build connections and professional relationships

By sharing your practices in a specific field, you sometimes come across people who actually work on the same topic than you or have similar research interests. This was the case, among others, with:

My collection of top R resources on Coronavirus. It started with only 5 resources, then as I was gradually adding more and more resources, Rees Morrison who was collecting blog posts on the same topic contacted me and I added his blog posts to the original collection.
The presentation of a R package developed by Renan Xavier Cortes which could be used to download free Springer books during Covid-19 quarantine.
My article on the analysis of the spread of COVID-19 in Belgium, which led, in collaboration with three other researchers, to a seminar and the creation of graphs representing COVID-19 hospital admissions in Belgium (which were later used by a Belgian news television channel).

These collaborations extended my professional network and allowed me to build connections with new researchers from different universities and from different countries. I would never have known these people without first sharing my analyses on a specific topic.

I am sure that a blog can lead to numerous collaborations, making the whole journey even more interesting and enriching. And who knows, the next chapter of your professional life may be shaped by one person with whom you have worked in the past.

Even if your blog does not lead to any collaboration, it is still a great tool for self-promotion. Doing some research, practicing, and writing about a topic help to become (and to be seen as) an expert in the field.

Moreover, your blog will become your portfolio when applying for jobs or projects. Recruiters will definitely appreciate to see what you are capable of.

Whether your blog is at the origin of some fruitful collaborations or a showcase of your work, it is a valuable asset when applying for your dream job.

How to start your own blog?

I have seen an acceleration in my learning curve, a deeper understanding of statistics and R, and a large improvement in my communication skills since starting my blog. I never realized how useful maintaining a blog could be as a learning technique.

This comes with the added benefit of being able to store information and code for myself and for others to also use, which is essential for getting feedback and connecting with other researchers.

If these 7 benefits have convinced you, the good news is that nowadays it is cheaper and easier than ever to start your own blog.

For non-technical blogs I recommend Medium or WordPress because it is easy to set up and it does not require code. For technical blogs, there are plenty of static site generators to choose from but I highly recommend using Hugo and the {blogdown} package (I created my blog after reading this book). You can then host it on GitHub and publish it using Netlify.³

Conclusion

Thanks for reading.

I hope this article answered questions such as “Why do you have a blog?” or “What is the purpose of it?”, and who knows, gave you the motivation to start your own blog. If you are still undecided, I truly recommend to do it and remember that the hardest part is to start. If you are patient and passionate about a topic, you simply have to start and the rest will follow.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

I intentionally avoided to write about financial incentives in this article because I believe that it is not a good motivation to start a blog. First, many people do not make money with their blog and never will. Second, it is true that there are also a lot of people who make money with their blog, but for most of them the money is not worth the time. I see from my personal experience that a blog can sometimes be so time consuming that I would definitely make money more easily somewhere else. Moreover, as you can see, I chose not to put advertising or banner as I think that it deteriorates the reading experience (I find ads and banners extremely annoying when I see them on other blogs). This is of course my opinion it remains a matter of personal choice, and a matter of trade off: worse reading experience and more money versus better reading experience and less money. I also tend to prefer to leave the choice to readers to support my projects than to impose them ads or banners.↩︎
Even native English speakers can improve their writing skills with a blog.↩︎
Note that I am not a programmer nor a computer scientist so I do not have extensive knowledge in creating websites. There must be many more options available but I find these options the most optimal choices given my computer skills and my goals.↩︎

Mortgage calculator in R Shiny

Fri, 14 Aug 2020 00:00:00 +0000

Introduction

I recently moved out and bought my first apartment. Of course, I could not pay it entirely with my own savings, so I had to borrow money from the bank. I visited a couple of banks operating in my country and asked for a mortgage.

If you already bought your house or apartment in the past, you know how it goes: the bank analyzes your financial and personal situation and make an offer based on your propensity to repay the bank. You then either accept the offer if you are satisfied with the rate and conditions, or visit another bank if you believe you could receive a better offer. Mortgages and loans are more complicated than that of course, but let’s keep it simple here.

As I kind of like to control and keep a close eye on my personal finances (sometimes a bit too close I must admit), I knew precisely how much I could spend for my monthly mortgage repayment while still being able to cover my living expenses. However, I had no clue how much I could borrow in total for my new apartment given these housing repayments.

I knew I was not the first person in this case, so I looked online if I could find a R script which would answer my question (and potentially also give me the total cost of the housing loan, including the loan amount and the accumulated interests). I finally found a R script created a while ago by Prof. Thomas Girke.

Mortgage calculator

The function in the script was functional and solved my main issue, but I wanted to be able to play more easily with the different settings such as the amount, the duration and the interest rate of the loan.

For this reason, I created a R Shiny app which is available here:

Mortgage calculator

Mortgage calculator in R Shiny

In the meantime, I received an Excel file from a friend working in a Belgian bank which does precisely the same task. I am not an actuary nor an expert in mortgage loan, so with his file I was able to cross check the results and edit the code accordingly.

The app greatly helped me to know the maximum amount I could borrow from the bank by playing with the three main settings of a mortgage, so it gave me a precise price limit when looking for apartments online.

Note that the app can of course be used for any loan, not only for mortgage.

How to use the mortgage calculator?

First, you can find the mortgage calculator here.

I try to keep all my Shiny apps easy to use for everyone. However, here is how to use it in case it is not intuitive enough:

Enter the amount of the loan (i.e., the amount you would like to borrow, do not include downpayment)
Enter the annual interest rate in %
Enter the duration of the loan in years

On the right panel (or bottom if you use the app on mobile) you will see:

a summary repeating the settings you entered,
the total cost of the loan (principal and interests included), and more importantly
the amount of the monthly payments

A plot representing the percentage attributed to the repayment of the interests and the capital is also displayed. You see that (especially in the first years of the loan), the higher the interest rate and the duration of the loan, the higher the percentage of the monthly repayments is attributed to the repayments of the interests.

Finally, the amortization table showing the remaining balance month by month is displayed after the summary and the plot. You can copy, export (in PDF, CSV or Excel) or print this amortization table for further use.

Code of the app

Here is the entire code (or see the last version on GitHub) in case you would like to enhance it (feel free to send me your app if you happen to improve it!).

Conclusion

Thanks for reading.

I hope this mortgage calculator helped you to play with the different settings of a mortgage, and who knows, helped you to decide which house or apartment to buy.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Disclosure: Note that this application does not include investment advice or recommendations, nor a financial analysis. This application is intended for information only and you invest at your own risks. I cannot be held liable for any decision made based on the information contained in this application, nor for its use by third parties.

How to do a t-test or ANOVA for more than one variable at once in R?

Thu, 19 Mar 2020 00:00:00 +0000

Introduction

As part of my teaching assistant position in a Belgian university, students often ask me for some help in their statistical analyses for their master’s thesis.

A frequent question is how to compare groups of patients in terms of several quantitative continuous variables. Most of us know that:

To compare two groups, a Student’s t-test should be used¹
To compare three groups or more, an ANOVA should be performed

These two tests are quite basic and have been extensively documented online and in statistical textbooks so the difficulty is not in how to perform these tests.

In the past, I used to do the analyses by following these 3 steps:

Draw boxplots illustrating the distributions by group (with the boxplot() function or thanks to the {esquisse} R Studio addin if I wanted to use the {ggplot2} package)
Perform a t-test or an ANOVA depending on the number of groups to compare (with the t.test() and oneway.test() functions for t-test and ANOVA, respectively)
Repeat steps 1 and 2 for each variable

This was feasible as long as there were only a couple of variables to test. Nonetheless, most students came to me asking to perform these kind of tests not on one or two variables, but on multiples variables. So when there were more than one variable to test, I quickly realized that I was wasting my time and that there must be a more efficient way to do the job.

Note: you must be very careful with the issue of multiple testing (also referred as multiplicity) which can arise when you perform multiple tests. In short, when a large number of statistical tests are performed, some will have \(p\)-values less than 0.05 purely by chance, even if all null hypotheses are in fact really true. This is known as multiplicity or multiple testing. You can tackle this problem by using the Bonferroni correction, among others. The Bonferroni correction is a simple method that allows many t-tests to be made while still assuring an overall confidence level is maintained. For this, instead of using the standard threshold of \(\alpha = 5\)% for the significance level, you can use \(\alpha = \frac{0.05}{m}\) where \(m\) is the number of t-tests. For example, if you perform 20 t-tests with a desired \(\alpha = 0.05\), the Bonferroni correction implies that you would reject the null hypothesis for each individual test when the \(p\)-value is smaller than \(\alpha = \frac{0.05}{20} = 0.0025\).

Note also that there is no universally accepted approach for dealing with the problem of multiple comparisons. Usually, you should choose a p-value adjustment measure familiar to your audience or in your field of study. The Bonferroni correction is easy to implement. It is however not appropriate if you have a very large number of tests to perform (imagine you want to do 10,000 t-tests, a p-value would have to be less than \(\frac{0.05}{10000} = 0.000005\) to be significant). A more powerful method is also to adjust the false discovery rate using the Benjamini-Hochberg or Holm procedure (McDonald 2014).

Another option is to use a multivariate ANOVA (MANOVA), if your independent variable has more than two levels. This is particularly useful when your dependent variables are correlated. Correlation between the dependent variables provides MANOVA the following advantages:

Identify patterns between several dependent variables: The independent variables can influence the relationship between dependent variables instead of influencing a single dependent variable.
Address the issue of multiple testing: with MANOVA, the error rate equals the significance level (with no p-value adjustment method needed).
Greater statistical power: When the dependent variables are correlated, MANOVA can identify effects that are too small for the ANOVA to detect.

Note that MANOVA is used if your independent variable has more than two levels. If your independent variable has only two levels, the multivariate equivalent of the t-test is Hotelling’s \(T^2\).

This article aims at presenting a way to perform multiple t-tests and ANOVA from a technical point of view (how to implement it in R). Discussion on which adjustment method to use or whether there is a more appropriate model to fit the data is beyond the scope of this article (so be sure to understand the implications of using the code below for your own analyses). Make sure also to test the assumptions of the ANOVA before interpreting results.

Perform multiple tests at once

I thus wrote a piece of code that automated the process, by drawing boxplots and performing the tests on several variables at once. Below is the code I used, illustrating the process with the iris dataset. The Species variable has 3 levels, so let’s remove one, and then draw a boxplot and apply a t-test on all 4 continuous variables at once. Note that the continuous variables that we would like to test are variables 1 to 4 in the iris dataset.

dat <- iris

# remove one level to have only two groups
dat <- subset(dat, Species != "setosa")
dat$Species <- factor(dat$Species)

# boxplots and t-tests for the 4 variables at once
for (i in 1:4) { # variables to compare are variables 1 to 4
  boxplot(dat[, i] ~ dat$Species, # draw boxplots by group
    ylab = names(dat[i]), # rename y-axis with variable's name
    xlab = "Species"
  )
  print(t.test(dat[, i] ~ dat$Species)) # print results of t-test
}

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -5.6292, df = 94.025, p-value = 1.866e-07
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.8819731 -0.4220269
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    5.936                    6.588

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -3.2058, df = 97.927, p-value = 0.001819
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.33028364 -0.07771636
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    2.770                    2.974

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -12.604, df = 95.57, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -1.49549 -1.08851
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    4.260                    5.552

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -14.625, df = 89.043, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.7951002 -0.6048998
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    1.326                    2.026

As you can see, the above piece of code draws a boxplot and then prints results of the test for each continuous variable, all at once.

At some point in the past, I even wrote code to:

draw a boxplot
test for the equality of variances (thanks to the Levene’s test)
depending on whether the variances were equal or unequal, the appropriate test was applied: the Welch test if the variances were unequal and the Student’s t-test in the case the variances were equal (see more details about the different versions of the t-test for two samples)
apply steps 1 to 3 for all continuous variables at once

I had a similar code for ANOVA in case I needed to compare more than two groups.

The code was doing the job relatively well. Indeed, thanks to this code I was able to test several variables in an automated way in the sense that it compared groups for all variables at once.

The only thing I had to change from one project to another is that I needed to modify the name of the grouping variable and the numbering of the continuous variables to test (Species and 1:4 in the above code).

Concise and easily interpretable results

T-test

Although it was working quite well and applicable to different projects with only minor changes, I was still unsatisfied with another point.

Someone who is proficient in statistics and R can read and interpret the output of a t-test without any difficulty. However, as you may have noticed with your own statistical projects, most people do not know what to look for in the results and are sometimes a bit confused when they see so many graphs, code, output, results and numeric values in a document. They are quite easily overwhelmed by this mass of information and unable to extract the key message.

With my old R routine, the time I was saving by automating the process of t-tests and ANOVA was (partially) lost when I had to explain R outputs to my students so that they could interpret the results correctly. Although most of the time it simply boiled down to pointing out what to look for in the outputs (i.e., p-values), I was still losing quite a lot of time because these outputs were, in my opinion, too detailed for most real-life applications and for students in introductory classes. In other words, too much information seemed to be confusing for many people so I was still not convinced that it was the most optimal way to share statistical results to nonscientists.

Of course, they came to me for statistical advices, so they expected to have these results and I needed to give them answers to their questions and hypotheses. Nonetheless, I wanted to find a better way to communicate these results to this type of audience, with the minimum of information required to arrive at a conclusion. No more and no less than that.

After a long time spent online trying to figure out a way to present results in a more concise and readable way, I discovered the {ggpubr} package. This package allows to indicate the test used and the p-value of the test directly on a ggplot2-based graph. It also facilitates the creation of publication-ready plots for non-advanced statistical audiences.

After many refinements and modifications of the initial code (available in this article), I finally came up with a rather stable and robust process to perform t-tests and ANOVA for more than one variable at once, and more importantly, make the results concise and easily readable by anyone (statisticians or not).

A graph is worth a thousand words, so here are the exact same tests than in the previous section, but this time with my new R routine:

library(ggpubr)

# Edit from here #
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method <- "t.test" # one of "wilcox.test" or "t.test"
paired <- FALSE # if paired make sure that in the dataframe you have first all individuals at T1, then all individuals again at T2
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    if (paired == TRUE) {
      p <- ggpaired(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]), line.color = "gray", line.size = 0.4,
        palette = "npg",
        legend = "none",
        xlab = colnames(dat[j]),
        ylab = colnames(dat[i]),
        add = "jitter"
      )
    } else {
      p <- ggboxplot(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]),
        palette = "npg",
        legend = "none",
        add = "jitter"
      )
    }
    #  Add p-value
    print(p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
      method = method,
      paired = paired,
      # group.by = NULL,
      ref.group = NULL
    ))
  }
}

As you can see from the graphs above, only the most important information is presented for each variable:

a visual comparison of the groups thanks to boxplots
the name of the statistical test
the p-value of the test

Of course, experts may be interested in more advanced results. However, this simple yet complete graph, which includes the name of the test and the p-value, gives all the necessary information to answer the question: “Are the groups different?”.

In my experience, I have noticed that students and professionals (especially those from a less scientific background) understand way better these results than the ones presented in the previous section.

The only lines of code that need to be modified for your own project is the name of the grouping variable (Species in the above code), the names of the variables you want to test (Sepal.Length, Sepal.Width, etc.),² whether you want to apply a t-test (t.test) or Wilcoxon test (wilcox.test) and whether the samples are paired or not (FALSE if samples are independent, TRUE if they are paired).

Based on these graphs, it is easy, even for non-experts, to interpret the results and conclude that the versicolor and virginica species are significantly different in terms of all 4 variables (since all p-values \(< \frac{0.05}{4} = 0.0125\) (remind that the Bonferroni correction is applied to avoid the issue of multiple testing, so we divide the usual \(\alpha\) level by 4 because there are 4 t-tests)).

Additional p-value adjustment methods

If you would like to use another p-value adjustment method, you can use the p.adjust() function. Below are the raw p-values found above, together with p-values derived from the main adjustment methods (presented in a dataframe):

raw_pvalue <- numeric(length = length(1:4))
for (i in (1:4)) {
  raw_pvalue[i] <- t.test(dat[, i] ~ dat$Species,
    alternative = "two.sided"
  )$p.value
}

df <- data.frame(
  Variable = names(dat[, 1:4]),
  raw_pvalue = round(raw_pvalue, 3)
)

df$Bonferroni <-
  p.adjust(df$raw_pvalue,
    method = "bonferroni"
  )
df$BH <-
  p.adjust(df$raw_pvalue,
    method = "BH"
  )
df$Holm <-
  p.adjust(df$raw_pvalue,
    method = "holm"
  )
df$Hochberg <-
  p.adjust(df$raw_pvalue,
    method = "hochberg"
  )
df$Hommel <-
  p.adjust(df$raw_pvalue,
    method = "hommel"
  )
df$BY <-
  round(p.adjust(df$raw_pvalue,
    method = "BY"
  ), 3)
df

##       Variable raw_pvalue Bonferroni    BH  Holm Hochberg Hommel    BY
## 1 Sepal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 2  Sepal.Width      0.002      0.008 0.002 0.002    0.002  0.002 0.004
## 3 Petal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 4  Petal.Width      0.000      0.000 0.000 0.000    0.000  0.000 0.000

Regardless of the p-value adjustment method, the two species are different for all 4 variables. Note that the adjustment method should be chosen before looking at the results to avoid choosing the method based on the results.

Below another function that allows to perform multiple Student’s t-tests or Wilcoxon tests at once and choose the p-value adjustment method. The function also allows to specify whether samples are paired or unpaired and whether the variances are assumed to be equal or not. (The code has been adapted from Mark White’s article.)

t_table <- function(data, dvs, iv,
                    var_equal = TRUE,
                    p_adj = "none",
                    alpha = 0.05,
                    paired = FALSE,
                    wilcoxon = FALSE) {
  if (!inherits(data, "data.frame")) {
    stop("data must be a data.frame")
  }

  if (!all(c(dvs, iv) %in% names(data))) {
    stop("at least one column given in dvs and iv are not in the data")
  }

  if (!all(sapply(data[, dvs], is.numeric))) {
    stop("all dvs must be numeric")
  }

  if (length(unique(na.omit(data[[iv]]))) != 2) {
    stop("independent variable must only have two unique values")
  }

  out <- lapply(dvs, function(x) {
    if (paired == FALSE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]], var.equal = var_equal)
    } else if (paired == FALSE & wilcoxon == TRUE) {
      tres <- wilcox.test(data[[x]] ~ data[[iv]])
    } else if (paired == TRUE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]],
        var.equal = var_equal,
        paired = TRUE
      )
    } else {
      tres <- wilcox.test(data[[x]] ~ data[[iv]],
        paired = TRUE
      )
    }

    c(
      p_value = tres$p.value
    )
  })

  out <- as.data.frame(do.call(rbind, out))
  out <- cbind(variable = dvs, out)
  names(out) <- gsub("[^0-9A-Za-z_]", "", names(out))

  out$p_value <- p.adjust(out$p_value, p_adj)
  out$conclusion <- ifelse(out$p_value < alpha,
    paste0("Reject H0 at ", alpha * 100, "%"),
    paste0("Do not reject H0 at ", alpha * 100, "%")
  )
  out$p_value <- ifelse(out$p_value < 0.001,
    "<0.001",
    round(out$p_value, 3)
  )

  return(out)
}

Applied to our dataset, with no adjustment method for the p-values:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

And with the Holm (1979) adjustment method:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species",
  p_adj = "holm"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

Again, with the Holm’s adjustment method, we conclude that, at the 5% significance level, the two species are significantly different from each other in terms of all 4 variables.

ANOVA

Below the same process with an ANOVA. Note that we reload the dataset iris to include all three Species this time:

dat <- iris

# Edit from here
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("setosa", "versicolor"), c("setosa", "virginica"), c("versicolor", "virginica")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

Like the improved routine for the t-test, I have noticed that students and non-expert professionals understand ANOVA results presented this way much more easily compared to the default R outputs.

With one graph for each variable, it is easy to see that all species are different from each other in terms of all 4 variables.³

If you want to apply the same automated process to your data, you will need to modify the name of the grouping variable (Species), the names of the variables you want to test (Sepal.Length, etc.), whether you want to perform an ANOVA (anova) or Kruskal-Wallis test (kruskal.test) and finally specify the comparisons for the post-hoc tests.⁴

To go even further

As we have seen, these two improved R routines allow to:

Perform t-tests and ANOVA on a small or large number of variables with only minor changes to the code. I basically only have to replace the variable names and the name of the test I want to use. It takes almost the same time to test one or several variables so it is quite an improvement compared to testing one variable at a time.
Share test results in a much proper and cleaner way. This is possible thanks to a graph showing the observations by group and the p-value of the appropriate test included directly on the graph. This is particularly important when communicating results to a wider audience or to people from diverse backgrounds.

However, like most of my R routines, these two pieces of code are still a work in progress. Below are some additional features I have been thinking of and which could be added in the future to make the process of comparing two or more groups even more optimal:

Add the possibility to select variables by their numbering in the dataframe. For the moment it is only possible to do it via their names. This will allow to automate the process even further because instead of typing all variable names one by one, we could simply type 4:25 (to test variables 4 to 25 for instance).
Add the possibility to choose a p-value adjustment method. Currently, raw p-values are displayed in the graphs and I manually adjust them afterwards or adjust the \(\alpha\).
When comparing more than two groups, it is only possible to apply an ANOVA or Kruskal-Wallis test at the moment. A major improvement would be to add the possibility to perform a repeated measures ANOVA (i.e., an ANOVA when the samples are dependent). It is currently already possible to do a t-test with two paired samples, but it is not yet possible to do the same with more than two groups.
Another less important (yet still nice) feature when comparing more than 2 groups would be to automatically apply post-hoc tests only in the case where the null hypothesis of the ANOVA or Kruskal-Wallis test is rejected (so when there is at least one group different from the others, because if the null hypothesis of equal groups is not rejected we do not apply a post-hoc test). At the present time, I manually add or remove the code that displays the p-values of post-hoc tests depending on the global p-value of the ANOVA or Kruskal-Wallis test.

I will try to add these features in the future, or I would be glad to help if the author of the {ggpubr} package needs help in including these features (I hope he will see this article!).

Last but not least, the following packages may be of interest to some readers:

If you want to report statistical results on a graph, I advise you to check the {ggstatsplot} package and in particular the ggbetweenstats() and ggwithinstats() functions. These functions allow to compare a continuous variable across multiple groups or conditions (for both independent and paired samples). Two advantages of the functions is that:
- it is very easy to switch from parametric to nonparametric tests and
- it automatically runs an ANOVA or t-test depending on the number of groups to compare

Note that many different statistical results are displayed on the graph, not only the name of the test and the p-value so a bit of simplicity and clarity is lost for more precision. However, it is still very convenient to be able to include tests results on a graph in order to combine the advantages of a visualization and a sound statistical analysis. Something that I still need to figure out is how to run the code on several variables at once.

The {compareGroups} package also provides a nice way to compare groups. It comes with a really complete Shiny app, available with:

# install.packages("compareGroups")
library(compareGroups)
cGroupsWUI()

Update with the `{ggstatsplot}` package

Several months after having written this article, I finally found a way to plot and run analyses on several variables at once with the package {ggstatsplot} (Patil 2021). This was the main feature I was missing and which prevented me from using it more often.

Although I still find that too much statistical details are displayed (in particular for non experts), I still believe the ggbetweenstats() and ggwithinstats() functions are worth mentioning in this article. I actually now use those two functions almost as often as my previous routines because:

I do not have to care about the number of groups to compare, the functions automatically choose the appropriate test according to the number of groups (ANOVA for 3 groups or more, and t-test for 2 groups)
I can select variables based on their column numbering, and not based on their names anymore (which prevents me from writing those variable names manually)
When comparing 3 or more groups (so for ANOVA, Kruskal-Wallis, repeated measure ANOVA or Friedman), \(p\)-values of the post-hoc tests within each dependent variable are by default the adjusted \(p\)-values (Holm is the default but many adjustment methods are available)
It is possible to compare both independent and paired samples, no matter the number of groups (remember that with the ggpubr package I could only do paired samples for two samples, not for 3 samples)
They allow to easily switch between the parametric and nonparametric version
All this in a more concise manner using the {purrr} package

For those of you who are interested, below my updated R routine which include these functions and applied this time on the penguins dataset.

library(palmerpenguins)

dat <- penguins
str(dat)

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

We illustrate the routine for two groups with the variables sex (two factors) as independent variable, and the 4 quantitative continuous variables bill_length_mm, bill_depth_mm, flipper_length_mm and body_mass_g as dependent variables:

library(ggstatsplot)
library(tibble)

# Comparison between sexes

# edit from here
x <- "sex"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

We now illustrate the routine for 3 groups or more with the variable species (three factors) as independent variable, and the 4 same dependent variables:

# Comparison between species

# edit from here
x <- "species"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

As you can see, I only have to specify:

the name of the grouping variable (sex and species),
the number of the dependent variables (variables 3 to 6 in the dataset),
whether I want to use the parametric or nonparametric version and
whether samples are independent (paired = FALSE) or paired (paired = TRUE).

Everything else is automated—the outputs show a graphical representation of what we are comparing, together with the details of the statistical analyses in the subtitle of the plot (the \(p\)-value among others).

Note that the code shown above is actually the same if I want to compare 2 groups or more than 2 groups. I wrote twice the same code (once for 2 groups and once again for 3 groups) for illustrative purposes only, but they are the same and should be treated as one for your projects.

I must admit I am quite satisfied with this routine, now that:

I can automate it on many variables at once and I do not need to write the variable names manually anymore,
at the same time, I can choose the appropriate test among all the available ones (depending on the number of groups, whether they are paired or not, and whether I want to use the parametric or nonparametric version).

Nonetheless, I must also admit that I am still not satisfied with the level of details of the statistical results. As already mentioned, many students get confused and get lost in front of so much information (except the \(p\)-value and the number of observations, most of the details are rather obscure to them because they are not covered in introductory statistic classes).

I saved time thanks to all improvements in comparison to my previous routine, but I definitely lose time when I have to point out to them what they should look for. After discussing with other professors, I noticed that they have the same problem.

For the moment, you can only print all results or none. I have opened an issue kindly requesting to add the possibility to display only a summary (with the \(p\)-value and the name of the test for instance).⁵ I will update again this article if the maintainer of the package includes this feature in the future. So stay tuned!

Conclusion

Thanks for reading.

I hope this article will help you to perform t-tests and ANOVA for multiple variables at once and make the results more easily readable and interpretable by non-scientists. Learn more about the t-test to compare two groups, or the ANOVA to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 65–70.

McDonald, JH. 2014. “Multiple Tests.” Handbook of Biological Statistics. 3rd Ed Baltimore, Maryland: Sparky House Publishing, 233–36.

Patil, Indrajeet. 2021. “Visualizations with statistical details: The ’ggstatsplot’ approach.” Journal of Open Source Software 6 (61): 3167. https://doi.org/10.21105/joss.03167.

In theory, an ANOVA can also be used to compare two groups as it will give the same results compared to a Student’s t-test, but in practice we use the Student’s t-test to compare two groups and the ANOVA to compare three groups or more.↩︎
Do not forget to separate the variables you want to test with |.↩︎
Do not forget to adjust the \(p\)-values or the significance level \(\alpha\). If you use the Bonferroni correction, the adjusted \(\alpha\) is simply the desired \(\alpha\) level divided by the number of comparisons.↩︎
Post-hoc test is only the name used to refer to a specific type of statistical tests. Post-hoc test includes, among others, the Tukey HSD test, the Bonferroni correction, Dunnett’s test. Even if an ANOVA or a Kruskal-Wallis test can determine whether there is at least one group that is different from the others, it does not allow us to conclude which are different from each other. For this purpose, there are post-hoc tests that compare all groups two by two to determine which ones are different, after adjusting for multiple comparisons. Concretely, post-hoc tests are performed to each possible pair of groups after an ANOVA or a Kruskal-Wallis test has shown that there is at least one group which is different (hence “post” in the name of this type of test). The null and alternative hypotheses and the interpretations of these tests are similar to a Student’s t-test for two samples.↩︎
I am open to contribute to the package if I can help!↩︎

How to create a timeline of your CV in R?

Sun, 26 Jan 2020 00:00:00 +0000

Introduction

In this article, I show how to create a timeline of your CV in R. A CV timeline illustrates key information about your education, work experiences and extra activities. The main advantage of CV timelines compared to regular CV is that they make you stand out immediately by being visually appealing and easier to scan. It also allows you to better present your “story” by showing the chronology of your jobs and activities and thus explain how you got to where you are today. (It can also be part of your portfolio to show your R skills.)

We show below how to create such CV in R with a minimal reproducible example. Feel free to use the code and adapt it to you. For a more complete example (together with the code) you can check my own CV timeline.

Note that I wrote this article after reading this original post by Bernardo Lares, and in particular his package, i.e., {lares} package. A special thanks for his amazing work which was used to create a slightly modified version of the plot_timeline() function!

Minimal reproducible example

Here is the code and the result of a minimal reproducible example:

# All packages used below must be installed first
library(devtools)
# devtools::install_github("laresbernardo/lares")
library(lares)
library(ggplot2)


today <- as.character(Sys.Date())


### Edit from here ###
cv <- data.frame(rbind(
  c("PhD in Statistics", "University3", "Academic", "2017-09-01", today),
  c("MSc in Econometrics", "University2", "Academic", "2015-09-01", "2017-08-31"),
  c("BSc in Economics", "University1", "Academic", "2010-09-01", "2013-08-31"),
  c("Job title2", "Company2", "Work Experience", "2016-09-01", today),
  c("Job title1", "Company1", "Work Experience", "2013-08-31", "2015-08-31"),
  c("Extra1", "Place1", "Extra", "2015-05-01", today),
  c("Extra2", "Place2", "Extra", "2019-01-01", today),
  c("Extra3", NA, "Extra", "2019-12-01", today)
))
### Edit until here ###


order <- c("Role", "Place", "Type", "Start", "End")
colnames(cv) <- order


plot_timeline2 <- function(event, start, end = start + 1, label = NA, group = NA,
                           title = "Curriculum Vitae Timeline", subtitle = "Antoine Soetewey",
                           size = 7, colour = "orange", save = FALSE, subdir = NA) {
  df <- data.frame(
    Role = as.character(event), Place = as.character(label),
    Start = lubridate::date(start), End = lubridate::date(end),
    Type = group
  )
  cvlong <- data.frame(pos = rep(
    as.numeric(rownames(df)),
    2
  ), name = rep(as.character(df$Role), 2), type = rep(factor(df$Type,
    ordered = TRUE
  ), 2), where = rep(
    as.character(df$Place),
    2
  ), value = c(df$Start, df$End), label_pos = rep(df$Start +
    floor((df$End - df$Start) / 2), 2))
  maxdate <- max(df$End)
  p <- ggplot(cvlong, aes(
    x = value, y = reorder(name, -pos),
    label = where, group = pos
  )) +
    geom_vline(
      xintercept = maxdate,
      alpha = 0.8, linetype = "dotted"
    ) +
    labs(
      title = title,
      subtitle = subtitle, x = NULL, y = NULL, colour = NULL
    ) +
    theme_minimal() +
    theme(panel.background = element_rect(
      fill = "white",
      colour = NA
    ), axis.ticks = element_blank(), panel.grid.major.x = element_line(
      linewidth = 0.25,
      colour = "grey80"
    ))
  if (!is.na(cvlong$type)[1] | length(unique(cvlong$type)) >
    1) {
    p <- p + geom_line(aes(color = type), linewidth = size) +
      facet_grid(type ~ ., scales = "free", space = "free") +
      guides(colour = "none") +
      scale_colour_manual(values = c("#F8766D", "#00BA38", "#619CFF")) +
      theme(strip.text.y = element_text(size = 10))
  } else {
    p <- p + geom_line(linewidth = size)
  }
  p <- p + geom_label(aes(x = label_pos),
    colour = "black",
    size = 2, alpha = 0.7
  )
  if (save) {
    file_name <- "cv_timeline.png"
    if (!is.na(subdir)) {
      dir.create(file.path(getwd(), subdir), recursive = T)
      file_name <- paste(subdir, file_name, sep = "/")
    }
    p <- p + ggsave(file_name, width = 8, height = 6)
    message(paste("Saved plot as", file_name))
  }
  return(p)
}




plot_timeline2(
  event = cv$Role,
  start = cv$Start,
  end = cv$End,
  label = cv$Place,
  group = cv$Type,
  save = FALSE,
  subtitle = "Antoine Soetewey" # replace with your name
)

How to personalize it

If you want to edit the example with your own academic, extra and work experiences you basically just have to edit the dataframe called cv in the code above. Each row of the dataset cv is a different academic program, job or activity. Rows should include:

the name of the academic program, job title or activity
the name of the university, school, company or workplace
the category: academic, work experience or extra
the starting date (dates must be in format yyyy-mm-dd)
the ending date. If the role has not yet ended, type today instead of the date. By using today your CV timeline will automatically adapt to today’s date

Add or remove a row in the dataframe if you want to add or remove a role. Indicate NA if you do not want to specify any workplace (as it has been done for Extra3). Last, do not forget to replace my name with yours for the subtitle of the timeline at the end of the code.

Experienced R users may wish to edit the plot_timeline2 function to their needs. However, if you are happy with the template and design of the example, you only have to change things mentioned above.

Additional note

Instead of editing the code according to your roles directly in the script, you can also create an Excel file with the required data (job title, workplace, type, start date, end date) and then import it into R. Editing the Excel file is easier and less prone to coding errors. Moreover, if you have a long career, the code may become long while if you import the Excel file, it will always stay short and concise.

Conclusion

Thanks for reading.

I hope this article helped you to create a timeline of your CV in R. If you would like to see a more complete and live example, see my timeline CV.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A special thanks to Prof. Job N Nmadu for the suggestion about creating the file that holds the data.