Visualization on Stats and R

AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns

Tue, 16 Dec 2025 00:00:00 +0000

I am pleased to announce the publication of our paper “AssociationExplorer: A user-friendly Shiny application for exploring associations and visual patterns” in the journal SoftwareX, together with the official release of the AssociationExplorer2 R package on CRAN.

Both the paper and the software are part of an open-science effort aimed at making exploratory data analysis more accessible to non-technical users.

Why AssociationExplorer?

Exploring multivariate datasets is now central in social sciences, data journalism, and education. However, identifying and interpreting associations between variables often requires programming skills and a solid background in statistics, which can represent a substantial barrier for many users.

AssociationExplorer was designed to lower this barrier by providing an interactive, visual, and statistically grounded tool for exploring associations between quantitative and qualitative variables, without requiring users to write any code.

The application is primarily intended for:

journalists and data journalism practitioners,
teachers and students,
researchers in the exploratory phase of an analysis,
engaged citizens interested in understanding public or survey data.

What does the app do?

AssociationExplorer follows a simple and guided workflow:

Import data (CSV or Excel files)
Interactively select variables of interest
Automatically compute association measures adapted to variable types:
- Pearson’s \(r\) correlation for numeric–numeric pairs,
- Cramer’s V for categorical–categorical pairs,
- the correlation ratio \(\eta\) for mixed numeric–categorical pairs
Filter associations using user-defined thresholds
Visualize results through:
- an interactive correlation network,
- contextual bivariate visualizations (scatter plots, mean plots, and colored contingency tables)

This workflow is designed to support transparent, reactive, and interpretable exploratory data analysis.

A paper published in SoftwareX

The SoftwareX paper provides a detailed description of:

the motivation and intended audience of the tool,
the software architecture,
the methodological choices underlying the association measures,
an illustrative case study based on the European Social Survey,
and perspectives for future development.

Link to the paper: https://doi.org/10.1016/j.softx.2025.102483

In line with the journal’s standards, the code, documentation, and example data are fully open and reproducible.

The R package is also on CRAN

Alongside the paper, an R package is also available on CRAN, making installation and use straightforward:

install.packages("AssociationExplorer2")
library(AssociationExplorer2)
run_associationexplorer()

The CRAN release ensures:

standardized installation,
better integration with existing R workflows,
clearer versioning and dependency management.

Who is it for, and how can it be used?

AssociationExplorer is particularly useful for:

exploratory analysis prior to formal modeling,
teaching concepts related to association and dependence,
data storytelling and journalistic exploration,
the analysis of survey data and public datasets.

The goal is not to replace confirmatory statistical analyses, but to provide a robust tool for understanding the structure of the data before modeling.

Acknowledgements

This work was carried out in collaboration with Cédric Heuchenne, Arnaud Claes, and Antonin Descampe, whom I warmly thank.

The project is supported by the Walloon Region and SPW Recherche within the ODALON research project.

Useful links

📄 SoftwareX paper: https://doi.org/10.1016/j.softx.2025.102483
📦 CRAN package: https://CRAN.R-project.org/package=AssociationExplorer2
💻 GitHub repository:
- of the paper
- of the R package

As always, feedback, bug reports, and suggestions are very welcome.

Thanks for reading!

References

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2026). AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns. SoftwareX, 33(102483). https://doi.org/10.1016/j.softx.2025.102483

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2025). AssociationExplorer2: A User-Friendly ‘shiny’ Application for Exploring Associations and Visual Patterns. R package version 0.1.4, https://github.com/AntoineSoetewey/AssociationExplorer2

Google Analytics in R: Review of 2022

Fri, 16 Dec 2022 00:00:00 +0000

Introduction

It is almost the end of the year, which means it is time to do a review of Stats and R and look back on the past year.

This year’s review will be much shorter compared to previous years because there are already a lot of examples in those previous reviews.

Note that beginning of September 2022, I moved from Universal Analytics (known as UA) to Google Analytics 4 (known as GA4). As you may know if you often use Google Analytics; on July 1, 2023, standard Universal Analytics properties will stop processing new hits. This means that you will need to move to GA4 if you want to continue using it after this date.

This year, I will be brief about the different metrics and how to analyze your Google Analytics (GA) data. However, this post will perhaps be helpful for people who also moved (or will move) to GA4 and still want to analyze their GA data from both UA and GA4.

I have to admit that it is quite time consuming to combine data from both, in particular since metrics and dimensions are not the same between UA and GA4. So if you have not moved yet, I recommend doing it at the end or beginning of a year. This will prevent you from having to fetch data from two GA IDs, and then having to combine both data in R.

As usual, I analyze my GA data in R thanks to the amazing {googleAnalyticsR} package from Mark Edmondson.

Note that the actual number presented below will not be very useful for you. Every blog and website is different, so every audience is different. This post is more about illustrating the process of analyzing GA data in R (and this year, also to illustrate the process of combining GA data from two GA IDs), rather than about showing my numbers.

Prerequisites

Before anything else, you have to specify the GA ID for which you want to analyze data. Mine are as follows:

ga_id <- c("208126346", "250795226")

You will find your(s) in the admin section of your GA account.

Analytics

In this section, I present some visualizations that might be useful when analyzing your GA data. Feel free to comment at the end of the post if you use other interesting visualizations.

Note that I focus on the page views metrics in this post, but you can edit my code to show other metrics as well. See the metrics available for UA and the ones for GA4.

Page views over time

Let’s start with the evolution of the number of page views over the past year, so from December 16, 2021 to December 15, 2022.

library(googleAnalyticsR)

# set date range
start_date <- as.Date("2021-12-16")
end_date <- as.Date("2022-12-15")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
library(dplyr)
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# scatter plot with a trend line
library(ggplot2)
dat %>%
  ggplot(aes(x = date, y = page_views)) +
  geom_point(size = 1L, color = "gray") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see in the code above, we need to:

extract data from two different IDs: this is done with google_analytics() for UA and ga_data() for GA4 (with each function having its own arguments)
join data from the two IDs (with a full_join() because even after moving to GA4, there were still some hits on UA, so I need to keep data from the 2 IDs for the entire year)
sum up the metric for both IDs (with rowSums())

This process will be done repeatedly for each visualization.

From the plot above, we see that the number of page views follows a cyclical evolution (with a decrease during summer). This trend follows the same pattern than last year, but the increase in the last quarter of the year is larger in 2022.

Note that there seems to be consistently some days with a lower number of page views than the rest. This is actually the weekends. See more on that in this section. Note also the presence of an outlier with a number of page views above 5,000. No post has been published on that day, so it probably comes from an old post that was shared to a large audience.

Finally, summing all days give a total page views of 938,630. Last year, it was 876,280, so 2022 saw an increase of 7.12%.

Page views per month and year

Comparison of the number of page views per month for all previous years can also be useful. Below two different types of visualizations.

# set new date range to include previous years
start_date_launch <- as.Date("2019-12-16")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date_launch, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date_launch, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# add year and month columns to dataframe
dat$month <- format(dat$date, "%m")
dat$year <- format(dat$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
dat %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(page_views = sum(page_views)) %>%
  ggplot(aes(x = month, y = page_views, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) # better y labels

Another possibility is as follows:

dat2 <- dat

library(lubridate)
dat2$day <- day(dat2$date)
dat2$day_month <- as.Date(paste0(dat2$month, "-", dat2$day), format = "%m-%d")

dat2 %>%
  filter(year != 2019 & page_views < 7500) %>% # remove 2019 and outliers
  ggplot(aes(x = day_month, y = page_views, color = year)) +
  geom_point(size = 1L, alpha = 0.25) + # change size and alpha of points
  geom_smooth(se = FALSE) + # remove confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    color = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) + # better y labels
  scale_x_date(date_labels = "%b") # show only months

From the 2 plots above, we see that the blog globally performed better in terms of page views compared to 2020-2021 (with an exception in the first quarter for which it performed better in 2021).

Note that a comparison between 3 years is fine, but with more years to compare the plots would quickly become unreadable!

Top performing pages

In case you are interested to know the top performing pages (still in terms of page views):

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "pageTitle",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "pageTitle",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "pageTitle")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)


most_viewed_posts <- dat %>%
  mutate(Title = str_sub(pageTitle, start = 1, end = -nchar(" - Stats and R"))) %>% # remove blog site in pageTitle
  count(Title, wt = page_views, sort = TRUE) %>%
  mutate(Title = str_trunc(Title, width = 30)) # keep maximum 30 characters

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Top performing posts are quite similar than last year, that is:

Since they are the most read posts for several years, I have made them available to download via Gumroad.

Page views by country

Knowing from which country your readers come from can also be useful, in particular for e-commerce or blogs that sell physical products in addition to writing posts.

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "country",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "country")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

# table
countries <- dat %>%
  mutate(Country = str_trunc(country, width = 30)) %>% # keep maximum 30 characters
  count(Country, wt = pageviews, sort = TRUE)

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

For Stats and R, most readers are from the US, and to a large extent since the second top country (UK) is quite far in terms of page views.

This trend is the same than last years, except that:

UK has surpassed India, and
Belgium (which used to rank 3rd and 6th in 2020 and 2021, respectively) is now only 8th. Since I am from Belgium, this indicates that the blog is attracting more and more international readers through the years (and proportionally less and less from my country).

Page views per day of week

As discussed in the section about the evolution of daily page views (view this section), there seem to be some days which consistently perform worse than other days.

The following plot will show what are these days:

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("date"),
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = c("date"),
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)
# find day of week
dat$weekday <- wday(dat$date, label = TRUE, abbr = TRUE)

## Reordering dat$weekday so Monday is first
dat$weekday <- factor(dat$weekday,
  levels = c("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
)

# boxplot
library(scales)
dat %>%
  ggplot(aes(x = weekday, y = page_views)) +
  geom_boxplot(fill = "steelblue", outlier.colour = alpha(0.25)) +
  geom_jitter(alpha = 0.25) + # adds transparency
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As expected for a technical blog, there are much more readers during the week than during the weekend.

Thank you note

Thank you to all readers who came to Stats and R this year. You made this journey incredibly more enriching. For next year and the many more to come, I will keep writing about topics for which I am familiar and interested in. So stay tuned!

Thanks for reading. I hope this article helped you to analyze your Google Analytics data in R, or helped you to combine your Universal Analytics and Google Analytics 4 data. For more examples of visualizations or summaries of your GA data in R, see also previous years’ reviews.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Stats and R is 2 years old!

Thu, 16 Dec 2021 00:00:00 +0000

Introduction

Stats and R has been launched exactly two years ago. Like last year, I think it is a good time to do a review of the past 12 months by sharing some figures about the audience of the blog.

This article is not about showing off my numbers, but rather a way to illustrate how to analyze your blog or your website’s traffic using Google Analytics data. Figures regarding the audience of my blog is probably useless to you (and I believe, should not be compared with). However, the code used in this post can be reused for your own blog or website (provided you also use Google Analytics to track your audience).

Note that I use the {googleAnalyticsR} R package to analyze my blog’s Google Analytics data. If you are unfamiliar with this package, see the prerequisites first.

If you have already used that package, you can select your account as followed (make sure to edit the code with your own property name):

library(googleAnalyticsR)

accounts <- ga_account_list()

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

On top of that, I also assume that you have basic knowledge of {ggplot2}—a popular R package to draw nice plots and visualizations.

Analytics

Last year, I mainly focused on the number of sessions. This year, I mainly concentrate on the number of page views to illustrate a different metrics.

For your information, a session is a group of user interactions with your website that take place within a given time frame, whereas a page view, as the name suggests, is defined as a view of a page on your site.

You can always change the metrics by editing metrics = c("pageviews") in the code below. See all available metrics provided by Google Analytics in this article.

Users and page views

As for last year’s review, let’s start with some general numbers, such as the number of users and page views for the entire site.

Note that we analyze traffic over the last year only so we extract data from December 16, 2020 to yesterday (December 15, 2021):

# set date range
start_date <- as.Date("2020-12-16")
end_date <- as.Date("2021-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users pageviews
## 1 549360    876280

Over this past year, Stats and R has attracted 549,360 users (number of new and returning people who visited the site), who generated a total of 876,280 page views.

That is an average of 2401 page views per day in 2021, compared to 1,531 page views per day in 2020 (an increase of 56.81%).

Page views over time

One of the first interesting metrics to analyze your blog’s audience is the evolution of traffic over time.

The daily number of page views over time can be presented in a scatterplot—together with a smoothed line—to analyze the evolution of the audience of your blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = pageviews)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

Although the number of page views varies quite a bit (with an outlier at more than 5,000 page views in a day and as low as less than 1,000 page views for some days), it seems to be cyclical with a dip during summer. It is worth noting that the same dip appeared last year, probably due to the fact that people are less likely to read posts about statistics and R during summer holidays.

So if you write about technical stuff in your blog, low numbers during summer may be expected and does not necessarily mean something is broken on your website.

Page views per channel

Knowing how people come to your blog is also a pretty important factor.

Here is how to visualize the evolution of daily page views per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "pageviews",
  pivots = pivot_ga4("medium", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Page_views, -Date)

# Build up the line plot
trend_long %>%
  filter(Channel != "Total") %>%
  ggplot() +
  aes(x = Date, y = Page_views, group = Channel) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As we can see from the plot above, the large majority of page views come from the organic channel (so from search engines such as Google, Bing, etc.), with some peaks from referral (mainly from R-bloggers and RWeekly) when an article is published. (By the way, you can always subscribe to the newsletter if you want to be informed by email when a new post goes out.)

If you happen to write tutorials, you can also expect that most visitors come from the organic channel. If you are very present on social media, you will most likely attract more visitors from the social channel.

Page views per day of week and month of year

As seen in the previous plot, there are many ups and downs and traffic seems to be cyclical.

To investigate this further, we draw a boxplot of the number of page views for each day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We can also compute the sum and the mean number of page views per day to have a numerical summary instead of a plot:

# compute sum
dat_sum <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Day of week", "Sum", "Mean")

# display table
dat_summary

##   Day of week    Sum     Mean
## 1      Monday 141115 2713.750
## 2     Tuesday 143145 2752.788
## 3   Wednesday 146472 2763.623
## 4    Thursday 140712 2706.000
## 5      Friday 128223 2465.827
## 6    Saturday  85766 1649.346
## 7      Sunday  90847 1747.058

As expected, there are more readers during the week compared to the weekends.

The same analysis can be done for each month of the year instead of days of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("month", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Boxplot
gadata %>%
  ggplot(aes(x = month, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

# compute sum
dat_sum <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Month", "Sum", "Mean")

# display table
dat_summary

##    Month   Sum     Mean
## 1     01 68534 2210.774
## 2     02 80953 2891.179
## 3     03 92629 2988.032
## 4     04 88679 2955.967
## 5     05 81739 2636.742
## 6     06 64460 2148.667
## 7     07 49772 1605.548
## 8     08 46389 1496.419
## 9     09 68046 2268.200
## 10    10 79237 2556.032
## 11    11 77745 2591.500
## 12    12 78097 2519.258

From the plots and the numerical summaries, it is clear that the number of page views is not the same between the days of the week and the months of the year.

Page views per month and year

If you have data over more than a year, it could be useful to compare your monthly blog’s traffic over the years.

With the following code, we create a barplot of the number of daily page views per month and year:

# set new date range
start_date_launch <- as.Date("2019-12-16")

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date_launch, end_date),
  metrics = c("pageviews"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(pageviews = sum(pageviews)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = pageviews, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of page views over the months, but more importantly, compare this evolution across different years.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views per pages.

It is true that the top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

However, for the interested reader, here is how to get the data in R (note that you can change n = 7 in the code below to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)

head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title      n
## 1    Outliers detection in R - Stats and R 119747
## 2 Descriptive statistics in R - Stats a... 109473
## 3 Variable types and examples - Stats a...  83025
## 4 Correlation coefficient and correlati...  65703
## 5 Chi-square test of independence in R ...  62100
## 6 The complete guide to clustering anal...  40440
## 7                 ANOVA in R - Stats and R  32914

If like me you prefer a visualization over a table, here is how to draw this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. The top 3 articles in the past year were:

Be careful that this ranking is based on the total number of page views over the last 12 months. A recent article may thus be found at the bottom of the list simply because it collected page views over a shorter period of time compared to an old article. So it is best to avoid comparing recent articles with older ones, or you can compare articles after having “time-normalized” the number of page views. See previous year’s review for more details and illustrations of this metrics.

You could also be interested in knowing the worst performing ones (to eventually improve them or include them in higher quality posts):

tail(subset(most_viewed_posts, n > 1000),
  n = 7
)

##                                       Title    n
## 49 A guide on how to read statistical ta... 1477
## 50                      About - Stats and R 1470
## 51 How to embed a Shiny app in blogdown?... 1455
## 52                   About me - Stats and R 1277
## 53 One-proportion and chi-square goodnes... 1226
## 54 Running pace calculator in R Shiny - ... 1049
## 55                      Shiny - Stats and R 1018

Note that I intentionally excluded pages with less than 1000 views to remove deleted or hidden pages from the ranking.

Another issue with this ranking is that it may be biased due to some pages which have been duplicated (if you edited the title for example), but at least you have a broad idea of the worst performing pages.

Page views by country

Knowing the country where your readers come from may also be handy for some content creators or marketers.

Location of my readers is not really important for me because I intend to write for everyone, but this may be completely the opposite if you are selling things or running a business/ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)

head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 244752
## 2           India  71797
## 3  United Kingdom  53272
## 4         Germany  38322
## 5          Canada  33812
## 6         Belgium  26421
## 7       Australia  26225
## 8     Philippines  23193
## 9     Netherlands  21944
## 10         Brazil  17956

Again, if you prefer a plot over a table, you can visualize the top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that a large share of readers are from the US—and Belgium (my country) comes only in \(6^{th}\) place in terms of number of page views.

Be careful that, as the number of people located in different countries differs widely, this ranking may hide some insights if you are comparing page views by countries in absolute terms. See why in this section of last year’s review.

User engagement by devices

One may also be interested in checking how users are engaged depending on device’s type. To investigate this, we plot 3 charts describing:

How page views are distributed by type of device?
The average time on page (in seconds) by type of device
The number of page views per session by device type

So first, how page views are distributed by device type?

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot pageviews by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, pageviews)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the large majority of readers visited the blog from a desktop and only a very small proportion comes from a tablet.

This makes sense since I guess many visitors are reading my articles or tutorials while using R (which is only available on desktop).

However, this information of total number of page views per device type does not tell me anything about the time spent on each page and thus the engagement by device type. The following plot answers this question:

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 125,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 3,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

Most readers coming from a tablet actually leave the page very quickly (see the peak around 0 second in the tablet facet).
Distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 125 seconds (= 2 minutes and 5 seconds) and 375 seconds (= 6 minutes and 15 seconds).
Quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (median = 316 seconds on mobile and 266 seconds on desktop, see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop (as shown by the total number of page views by device), it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article more carefully and try to reproduce the code on their own computer. At least that is what I do when I read blogs on mobile versus reading them on desktop. What is even more intriguing, is that it was already the case last year. If someone finds similar results and have a possible explanation, I would be glad to hear from her (if possible, in the comments at the end of the article so everyone can benefit from the discussion).
As a side note, we see that these medians are higher this year compared to last year (266, 316 and 114 seconds in 2021 compared to 190, 228 and 106 seconds in 2020 on desktop, mobile and tablet, respectively). This is somewhat encouraging because it indicates that people spend more time on each page (which is an indication, at least partially, of the quality of the blog for Google).¹

Given this result, I also believe it is helpful to illustrate the number of page views during a session by device type.

In fact, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session (remember that a session is a set of interactions with your website that take place within a given time frame).

We verify this belief via a density plot, and for better readability we exclude visits from tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views per session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nDashed lines represent the medians",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  )

This last plot shows that readers on desktop and mobile visit approximately the same number of pages per session, as indicated by the fact that the two distributions overlap each other and are not distant from each other. It is true that the median is higher for people on desktop than on mobile, but to a very small margin only (and the difference between the two is smaller than in last year’s review).

So to summarize what we learned based on the 3 last plots, we now know that

most readers visited the blog from desktop;
readers on mobile spent more time on each page than readers on desktop (and even more compared to readers on tablet);
users on desktop and mobile seem to have visited approximately the same number of pages per session.

One may wonder why I chose to compare medians instead of means. The main reason is that the median is a more robust way to represent non-normal data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.

Browser information

From a more technical perspective, you could also be interested in the number of page views by browser. I am personally not really interested in knowing which browser my visitors are using the most (mostly because this blog is available on all common browsers), but the most geeky among you may be so.

This information can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most readers visited the site using Chrome, Safari and Firefox (this was expected since they are the most common browsers).

This was the last metrics presented in this review. Of course, many more are possible depending on your R skills (mainly, data manipulation and {ggplot2}) and your expertise in SEO or analyzing Google Analytics data. Hopefully, thanks to this review and possibly from last year too, you will be able to analyze your own blog or website using R and the {googleAnalyticsR} package. At least, this was the aim of the present article.

For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.²

End note

I would also like to add that these figures are not done to be compared with. Every website or blog is unique, every author is unique (with different priorities and different agendas) and more is not always better. I learn a lot thanks to this blog, I use it for personal purposes and for my students as part of my teaching tasks. I will keep writing on it as long as I enjoy it and as long as I have the time to do so, not matter how low or high the number of clicks.

Thanks to all readers of the past year, and see you in a year for another review! In the meantime, if you maintain a blog I would be really happy to hear how you track its performance.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

More time spend on each page is a favorable factor for Google because it means that people are reading it more carefully. If the blog or the post is of mediocre quality, users tend to bounce back quickly (known as bounce rate) and look for an answer to their question somewhere else (leading ultimately to less time spend on the page or site).↩︎
Thanks to the RStudio blog for the inspiration.↩︎

How to track the performance of your blog in R?

Wed, 16 Dec 2020 00:00:00 +0000

Introduction

Stats and R has been launched on December 16, 2019. Since the blog is officially one year old today and after having discussed the main benefits of maintaining a technical blog, I thought it would be a good time to share some numbers and thoughts about it.

In this article, I show how to analyze a blog and its blog posts with the {googleAnalyticsR} R package (see package’s full documentation). After sharing some analytics about the blog, I will also discuss about content creation/distribution and, to a smaller extent, the future plans. This is a way to share my journey as a data science blogger and a way to give you an insight about how Stats and R is doing.

I decided to share with you some numbers through the {googleAnalyticsR} package instead of the regular Google Analytics dashboards for several reasons:

There are plenty of data analysts who are much more experienced than me when it comes to analyzing Google Analytics data via their dedicated platform
I recently discovered the {googleAnalyticsR} package in R and I would like to present its possibilities, and perhaps convince marketing specialists familiar with R to complement their Google Analytics dashboards with some data visualizations made in R (via some ggplot2 visualizations for instance)
I would like to automate the process such in a way that I can easily replicate the same types of analysis across the years. This will allow to see how the blog evolves throughout the years. We know that using R is a pretty good starting point when it comes to automation and replication—especially thanks to R Markdown reports

I am not an expert in the field of digital marketing, but who knows, it may still give some ideas to data analysts, SEO specialists or other bloggers on how to track the performance of their own blog or website using R. For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.¹

Before going further, I would like to remind that I am not making a living from my blog (far from it!) and it is definitely not my goal as I do not believe that I would be the same kind of writer if it was my main occupation.

Prerequisites

As for any package in R, we first need to install it—with install.packages()—and load it—with library():²

# install.packages('googleAnalyticsR', dependencies = TRUE)
library(googleAnalyticsR)

Next, we need to authorize the access of the Google Analytics account using the ga_auth() function:

ga_auth()

Running this code will open a browser window on which you will be able to authorize the access. This step will save an authorization token so you only have to do it once.

Make sure to run the ga_auth() function in a R script and not in a R Markdown document. Follow this procedure if you want to use the package and its functions in a R Markdown report or in a blog post like I did for this article.

Once we have completed the Google Analytics authorization, we will need the ID of the Google Analytics account we want to access. All the available accounts linked to your email address (after authentication) are stored in ga_account_list():

accounts <- ga_account_list()

accounts

## # A tibble: 2 × 10
##   accountId account…¹ inter…² level websi…³ type  webPr…⁴ webPr…⁵ viewId viewN…⁶
##   <chr>     <chr>     <chr>   <chr> <chr>   <chr> <chr>   <chr>   <chr>  <chr>  
## 1 86997981  Antoine … 129397… STAN… https:… WEB   UA-869… Antoin… 13318… All We…
## 2 86997981  Antoine … 218214… STAN… https:… WEB   UA-869… statsa… 20812… All We…
## # … with abbreviated variable names ¹accountName, ²internalWebPropertyId,
## #   ³websiteUrl, ⁴webPropertyId, ⁵webPropertyName, ⁶viewName

accounts$webPropertyName

## [1] "Antoine Soetewey" "statsandr.com"

As you can see I have two accounts linked to my Google Analytics profile: one for my personal website (antoinesoetewey.com) and one for this blog.

Of course, I select the account linked to this blog:

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

Make sure to edit the code with your own property name.

We are now finally ready to use our Google Analytics data in R for a better analysis!

Analytics

Users, page views and sessions

Let’s start with some general numbers, such as the number of users, sessions and page views for the entire site. Note that for the present article, we use data over the past year, so from December 16, 2019 to December 15, 2020:

# set date range
start_date <- as.Date("2019-12-16")
end_date <- as.Date("2020-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "sessions", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users sessions pageviews
## 1 321940   428217    560491

In its first year, Stats and R has attracted 321,940 users, who generated a total of 428,217 sessions and 560,491 page views (that is an average of 1531 page views per day).

For those unfamiliar with Google Analytics data and the difference between these metrics, remember that:

a user is the number of new and returning people who visit your site during a set period of time
a session is a group of user interactions with your website that take place within a given time frame
a page view, as the name suggests, is defined as a view of a page on your site

So if person A reads three blog posts then leave the site and person B reads one blog post, your about page then leave the site, Google Analytics data will show 2 users, 2 sessions and 5 page views.

Sessions over time

In addition to the rather general metrics presented above, it is also interesting to illustrate the daily number of sessions over time in a scatterplot—together with a smoothed line—to analyze the evolution of the blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = sessions)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "darkgrey", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 15.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

(See how to draw plots with the {ggplot2} package, or with the {esquisse} addin if you are not familiar with the package.)

As you can see, there was a huge peak of traffic around end of April, with almost 12,000 users in a single day. Yes, you read it well and there is no bug. The blog post “A package to download free Springer books during Covid-19 quarantine” went viral and generated a massive traffic for a few days. The daily number of sessions returned to a more normal level after a couple of days. We also observe an upward trend in the last months (since end of August/beginning of September), which indicates that the blog is growing in terms of number of daily sessions.

Note that I decided to focus on the number of sessions and the number of page views in this section and the following ones, but you can always change to your preferred metrics by editing metrics = c("sessions") in the code. See all available metrics provided by Google Analytics in this article.

Sessions per channel

Knowing how people come to your blog is a pretty important factor. Here is how to visualize the evolution of daily sessions per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "sessions",
  pivots = pivot_ga4("medium", "sessions"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Sessions, -Date)

# Build up the line plot
ggplot(trend_long, aes(x = Date, y = Sessions, group = Channel)) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We see that a large share of the traffic is from the organic channel, which indicates that most readers visit the blog after a query on search engines (mostly Google). In my case, where most of my posts are tutorials and which help people with specific problems, it is thus not a surprise that most of my traffic comes from organic search.

We also notice some small peaks of sessions generated from the referral and direct channels, which are probably happening on the date of publication of each article.

We also see that there seems to be a recurrent pattern of ups and downs in the number of daily sessions. Those are weekly cycles, with less readers during the weekend and which indicates that people are working on improving their statistical or R knowledge mostly during the week (which actually makes sense!).

Sessions per day of week

As shown above, traffic seems to be different depending on the day of week. To investigate this further, we draw a boxplot of the number of sessions for every day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "sessions",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = sessions)) +
  geom_boxplot() +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see there are some outliers, probably due (in part at least) to the article that went viral. For the sake of illustration, below the same plot after removing points considered as potential outliers according to the interquartile range (IQR) criterion (i.e., points above or below the whiskers), and after a couple of visual improvements:

# boxplot
gadata %>%
  filter(sessions <= 3000) %>% # filter out sessions > 3,000
  ggplot(aes(x = dayOfWeek, y = sessions, fill = dayOfWeek)) + # fill boxplot by dayOfWeek
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 3,000 excluded"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(legend.position = "none") # remove legend

After excluding data points above 3,000, it is now easier to see that the median number of sessions (represented by the horizontal bold line in the boxes) is the highest on Wednesdays, and lowest on Saturdays and Sundays.

The difference in sessions between weekdays is, however, not as large as expected.

Sessions per day and time

We have seen the traffic per day of week. The example below shows a visualization of traffic, broken down this time by day of week and hour of day in a density plot. In this plot, the device type has also been added for more insights.

## Get data by deviceCategory, day of week and hour
weekly_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("deviceCategory", "dayOfWeekName", "hour"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Manipulation using dplyr
weekly_data_sessions <- weekly_data %>%
  group_by(deviceCategory, dayOfWeekName, hour)

## Reordering weekly_data_sessions$dayOfWeekName to have Monday as first day of the week
weekly_data_sessions$dayOfWeekName <- factor(weekly_data_sessions$dayOfWeekName,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

## Plotting using ggplot2
weekly_data_sessions %>%
  ggplot(aes(hour, sessions, fill = deviceCategory, group = deviceCategory)) +
  geom_area(position = "stack") +
  labs(
    title = "Sessions per day and time",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    x = "Time",
    y = "Sessions",
    fill = "Device" # edit legend title
  ) +
  theme_minimal() +
  facet_wrap(~dayOfWeekName, ncol = 2, scales = "fixed") +
  theme(
    legend.position = "bottom", # move legend
    axis.text = element_text(size = 7) # change font of axis text
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

The above plot shows that:

traffic increases in the afternoon then declines in the late evening
traffic is the highest from Monday to Thursday, and lowest on Saturday and Sunday

In addition to that, thanks to the additional information on the device category, we also see that:

the number of sessions on tablet is low (it is so low compared to desktop and mobile that it is not visible on the plot), and
the number of sessions on mobile seems to be quite stable during the entire day,
as opposed to sessions on desktop which are highest at the end of the day.

Sessions per month and year

In the following code, we create a barplot of the number of daily sessions per month and year:

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# sessions by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  group_by(year, month) %>%
  summarize(sessions = sum(sessions)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = sessions, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "Month",
    title = "Sessions per month and year",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of sessions over the months, and compare this evolution across different years.

At the moment, since the blog is online only since December 2019, the year factor is not relevant. However, I still present the visualization for other users who work on older websites, and also to remind my future self to create this interesting barplot when there will be data for more than a year.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views for the different pages. The top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

For the interested reader, here is how to get the data in R (note that you can change n = 7 in the code to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)
head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title     n
## 1 A package to download free Springer b... 85684
## 2 Variable types and examples - Stats a... 44951
## 3 Descriptive statistics in R - Stats a... 43621
## 4    Outliers detection in R - Stats and R 32560
## 5 The complete guide to clustering anal... 27184
## 6 Correlation coefficient and correlati... 21581
## 7                              Stats and R 16786

Here is how to visualize this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. For instance, I never thought that the post illustrating the different types of variables that exist in statistics (ranked #2) would be so appreciated when I wrote it.

This is something I learned with this blog: at the time of writing, there are some posts which I think no one will care about (and which I mostly write as a personal note for myself), and others which I think people will find very useful. However, after a couple of weeks after publication, sometimes I realize that it is actually exactly the opposite.

Time-normalized page views

When it comes to comparing blog posts, however, this is not as simple.

Based on the above barplot and without any further analysis, I would conclude that my article about variable types is performing much better than the one about outliers detection in R.

However, if I tell you that the article on outliers detection has been published on August 11 and the one about variable types on December 30, you will agree that the comparison does not make much sense anymore since page views for these articles were counted over a different length of time. One could also argue that a recent article had less time to generate backlinks, so it is unfair to compare it with an old post which had plenty of time to be ranked high by Google.

In order to make the comparison more “fair”, we would need to compare the number of page views for each post since their date of publication. The following code does precisely this:³

it pulls daily data for a bunch of pages,
then tries to detect their publication date,
time-normalizes the traffic for each page based on that presumed publication date,
and finally, plots the daily traffic from the publication date on out, as well as overall cumulative traffic for the top n pages

# figure out when a page actually launched by finding the
# first day where the page had at least 2 unique pageviews
first_day_pageviews_min <- 2

# exclude pages that have total traffic (daily unique pageviews) that are relatively low
total_unique_pageviews_cutoff <- 500

# set how many "days since publication" we want to include in our plot
days_live_range <- 180

# set number of top pages to display
n <- 7

# Create a dimension filter object
# You need to update the "expressions" value to be a regular expression that filters to
# the appropriate set of content on your site
page_filter_object <- dim_filter("pagePath",
  operator = "REGEXP",
  expressions = "/blog/.+"
)

# Now, put that filter object into a filter clause. The "operator" argument can be AND # or OR...but you have to have it be something, even though it doesn't do anything
# when there is only a single filter object.
page_filter <- filter_clause_ga4(list(page_filter_object),
  operator = "AND"
)

# Pull the GA data
ga_data <- google_analytics(
  viewId = view_id,
  date_range = c(start_date, end_date),
  metrics = "uniquePageviews",
  dimensions = c("date", "pagePath"),
  dim_filters = page_filter,
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Find the first date for each post. This is actually a little tricky, so we're going to write a
# function that takes each page as an input, filters the data to just include those
# pages, finds the first page, and then puts a "from day 1" count on that data and
# returns it.
normalize_date_start <- function(page) {
  # Filter all the data to just be the page being processed
  ga_data_single_page <- ga_data %>% filter(pagePath == page)

  # Find the first value in the result that is greater than first_day_pageviews_min. In many
  # cases, this will be the first row, but, if there has been testing/previews before it
  # actually goes live, some noise may sneak in where the page may have been live, technically,
  # but wasn't actually being considered live.
  first_live_row <- min(which(ga_data_single_page$uniquePageviews > first_day_pageviews_min))

  # Filter the data to start with that page
  ga_data_single_page <- ga_data_single_page[first_live_row:nrow(ga_data_single_page), ]

  # As the content ages, there may be days that have ZERO traffic. Those days won't show up as
  # rows at all in our data. So, we actually need to create a data frame that includes
  # all dates in the range from the "publication" until the last day traffic was recorded. There's
  # a little trick here where we're going to make a column with a sequence of *dates* (date) and,
  # with a slightly different "seq," a "days_live" that corresponds with each date.
  normalized_results <- data.frame(
    date = seq.Date(
      from = min(ga_data_single_page$date),
      to = max(ga_data_single_page$date),
      by = "day"
    ),
    days_live = seq(min(ga_data_single_page$date):
    max(ga_data_single_page$date)),
    page = page
  ) %>%
    # Join back to the original data to get the uniquePageviews
    left_join(ga_data_single_page) %>%
    # Replace the "NAs" (days in the range with no uniquePageviews) with 0s (because
    # that's exactly what happened on those days!)
    mutate(uniquePageviews = ifelse(is.na(uniquePageviews), 0, uniquePageviews)) %>%
    # We're going to plot both the daily pageviews AND the cumulative total pageviews,
    # so let's add the cumulative total
    mutate(cumulative_uniquePageviews = cumsum(uniquePageviews)) %>%
    # Grab just the columns we need for our visualization!
    select(page, days_live, uniquePageviews, cumulative_uniquePageviews)
}

# We want to run the function above on each page in our dataset. So, we need to get a list
# of those pages. We don't want to include pages with low traffic overall, which we set
# earlier as the 'total_unique_pageviews_cutoff' value, so let's also filter our
# list to only include the ones that exceed that cutoff. We also select the top n pages
# in terms of page views to display in the visualization.
library(dplyr)
pages_list <- ga_data %>%
  group_by(pagePath) %>%
  summarise(total_traffic = sum(uniquePageviews)) %>%
  filter(total_traffic > total_unique_pageviews_cutoff) %>%
  top_n(n = n, total_traffic)

# The first little bit of magic can now occur. We'll run our normalize_date_start function on
# each value in our list of pages and get a data frame back that has our time-normalized
# traffic by page!
library(purrr)
ga_data_normalized <- map_dfr(pages_list$pagePath, normalize_date_start)

# We specified earlier -- in the `days_live_range` object -- how many "days since publication" we
# actually want to include, so let's do one final round of filtering to only include those
# rows.
ga_data_normalized <- ga_data_normalized %>% filter(days_live <= days_live_range)

Now that our data is ready, we create two visualizations:

Number of page views by day since publication: this plot shows how quickly interest in a particular piece of content drops off. If it is not declining as rapidly as the other posts, it means you are getting sustained value from it
Cumulative number of page views by day since publication: this plot can be used to compare blog posts on the same ground since number of page views is shown based on the publication date. To see which pages have generated the most traffic over time, simply look from top to bottom (at the right edge of the plot)

Note that both plots use the {plotly} package to make them interactive so that you can mouse over a line and find out exactly what page it is (together with its values). The interactivity of the plot makes it also possible to zoom in to see, for instance, the number of page views in the first days after publication (instead of the default length of 180 days),⁴ or zoom in to see only the evolution of the number of page views below/above a certain threshold.

# Create first plot
library(ggplot2)

gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Page views by day since publication",
    x = "Days since publication",
    y = "Page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
library(plotly)
ggplotly(gg, dynamicTicks = TRUE)

The plot above shows again a huge spike for the post that went viral (orange line). If we zoom in to include only page views below 1500, comparison between posts is easier and we see that:

The post on the top 100 R resources on Coronavirus (purple line) generated comparatively more traffic than the other posts in the first 50 days (\(\approx\) 1 month and 3 weeks) after publication. However, traffic gradually decreased up to the point that after 180 days (\(\approx\) 6 months), it attracted less traffic than other more performing posts.
The post on outliers detection in R (blue line) did not get a lot of attention in the first weeks after its publication. However, it gradually generated more and more traffic up to the point that, after 60 days (\(\approx\) 2 months) after publication, it actually generated more traffic than any other post (and by a relatively large margin).
The post on correlation coefficient and correlation test in R (green line) took approximately 100 days (\(\approx\) 3 months and 1 week) to take off, but after that it generated quite a lot of traffic. This is interesting to keep in mind when analyzing recent posts, because they may actually follow the same trend in the long run.

# Create second plot: cumulative
gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = cumulative_uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Cumulative page views by day since publication",
    x = "Days since publication",
    y = "Cumulative page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
ggplotly(gg, dynamicTicks = TRUE)

Compared to the previous plot, this one shows the cumulative number of page views since the date of publication of the post.

Without taking into consideration the post that went viral and which, by the way, does not really generate much traffic anymore (which makes sense since the incredible campaign from Springer to offer their books for free during the COVID-19 quarantine has ended), we see that:

The post on outliers detection has surpassed the post on top R resources on Coronavirus in terms of cumulative number of page views after around 120 days (\(\approx\) 4 months) after publication, which indicates that people are looking at this specific problem. I remember that I wrote this post because, at that time, I had to deal with the problems of outliers in R and I did not found a neat solution online. So I guess, the fact that resources on a specific topic are missing helps to attract visitors looking for an answer to their question.
Among the remaining posts, the ranking of the most performing ones in terms of cumulative number of page views within 180 days (\(\approx\) 6 months) after publication is the following:

The analyses so far give you already a good understanding of the performance of your blog. However, for the interested readers, we show other important metrics in the following sections.

Page views by country

In the following we are interested in seeing where the traffic comes from. This is particularly interesting to get to know your audience, and even more important if you are running a business or an ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)
head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 131306
## 2           India  38800
## 3         Belgium  32983
## 4  United Kingdom  25833
## 5          Brazil  18961
## 6         Germany  17851
## 7           Spain  17355
## 8          Canada  13880
## 9          Mexico  12882
## 10    Philippines  12540

To visualize this table of top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that readers from the US take the largest share of the number of page views (by quite a lot actually!), and Belgium (my country) comes in third place in terms of number of page views.

Given that the US population is much larger than the Belgian population (\(\approx\) 331 million people compared to \(\approx\) 11.5 million people, respectively), the above result is not really surprising. Again, for a better comparison, it would be interesting to take into consideration the size of the population when comparing countries.

Indeed, it could be that a large share of the traffic comes from a country with a large population, but that the number of page views per person (or per 100,000 inhabitants) is higher for another country. This information about top performing countries in terms of page views per person could give you insights on which country do the most avid readers come from. This is beyond the scope of this article, but you can see examples of plots which include the information on the population size in these COVID-19 visualizations. I recommend to apply the same methodology to the above plot for a better comparison.

If you are thinking about doing the extra step of including population size when comparing countries, I believe that it would be even better to take into account the information on computer access in each country as well. If we take India as example: at the time of writing this article, its population amounts to almost 1.4 billion. However, the percentage of Indian people having access to a computer is undoubtedly lower than in US or Belgium (again, at least at the moment). It would therefore make more sense to compare countries by comparing the number of page views per people having access to a computer.⁵

Browser information

For more technical aspects, you could also be interested in the number of page views by browser. This can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most visits were, as expected, from Chrome, Safari and Firefox browsers.

User engagement by devices

One may also be interested in checking how users are engaged on different types of devices. To do so, we plot 3 charts describing:

How many sessions were made from the different types of devices
The average time on page (in seconds) by type of device
The number of page views per session by device type

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot sessions by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, sessions)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the majority of readers visited the blog from a desktop, and a small number of readers from a tablet.

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 25,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 2,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

some readers on tablet have a very low average time spent on each page (see the peak around 0 second in the tablet facet)
distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 100 seconds (= 1 minute 40 seconds) and 400 seconds (= 6 minutes 40 seconds)
quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop, it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article carefully and try to reproduce the code on their computer.⁶

Given this result, it would be interesting to also illustrate the number of page views during a session, represented by device type.

Indeed, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session. We verify this via a density plot, and for better readability we exclude data points above 2.5 page views/session and we exclude visits from a tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(pageviewsPerSession <= 2.5 & deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views/session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 2.5 excluded",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  ) +
  geom_text(
    aes(
      x = med, y = 2.75,
      label = paste0("Median = ", round(med, 2), " page views/session"),
      color = deviceCategory
    ),
    angle = 90,
    vjust = 2,
    size = 3,
    show.legend = FALSE # remove legend
  )

This last plot confirms our thoughts, that is, although people on mobile seem to spend more time on each page, people on desktop tend to visit more pages per session.

(One may wonder why I chose to compare medians instead of means. The main reason is that not all distributions considered here are bell-shaped (especially for data on tablet) and there are many outliers. In these cases, the mean is usually not the most appropriate descriptive statistics and the median is a more robust way to represent such data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.)

This is the end of the analytics section. Of course, many more visualizations and data analyses are possible, depending on the site that is tracked and the marketing expertise of the analyst. This was an overview of what is possible, and I hope it will give you some ideas to explore your Google Analytics data further. Next year, I may also include forecasts and make annual comparisons. See also some examples of other analyses in this GitHub repository and this website.

As a side note, I would like to add the following: even if tracking the performance of your blog is important to understand how you attract visitors and how they engage with your site, I also believe that looking at your Google Analytics stats too often is not optimal, nor sane.

Talking about personal experience: at the beginning of the blog I used to look very often at the number of visitors in real-time and the audience. I was kind of obsessed to know how many people were right now on my blog and I was constantly checking if it performed better than the day before in terms of number of visitors. I remember that I was spending so much time looking at these metrics in the first weeks that I felt I was wasting my time. And the time I was wasting looking at my Google Analytics stats was lost not creating good quality content for the blog, working on my thesis/classes, or other projects.

So when I realized that, I deleted the Google Analytics app from my smartphone and forced myself not to look at my stats more than once a month (just to make sure there is no critical issues that need to be fixed). From that moment onward, I stopped wasting my time on things I cannot control, and I got more satisfaction from writing articles because I was writing them for myself, not for the sake of seeing people reading them. This change was like a relief, and I am now more satisfied with my work on this blog compared to the first weeks or months.

Everyone is different and unique so I am not saying that you should do the same. However, if you feel that you look too much at your stats and sometimes lose motivation in writing, perhaps this is one potential solution.

Content

Now that we have seen how we can analyze Google Analytics data and track the performance of a website or blog in details, I would like to share some thoughts about content creation, content distribution and the future plans.

Finding topics

One of the biggest challenges I face with my blog is creating proper content. In the best case scenario, I would like to:

create useful content
about subjects I am really familiar with,
which I enjoy and
which fit into the blog.

By sharing only articles about topics I am familiar with, the writing process is easier and quicker. Indeed, most of the articles I wrote cover topics I teach at university, so a large part of the preliminary research is already done when I decide to write about it. Also (and this is not negligible), questions and incomprehension from students allow me to see the points I should focus on when writing about it, and how to present it to make it accessible and comprehensible to most people.

Moreover, by making my thoughts public, I often have the chance to confront them with other points of view, which allows me to study the topic even further. This in turn accelerates the writing process even more when writing about a related topic.

I also tend to write only about things I enjoy or I am interested in. I prefer quality over quantity, so writing an article from A to Z takes quite a long time depending on the depth of the subject. Since it takes time (even without taking into account the time spent after publication) and I have a full-time job, I really focus on topics I enjoy in order to keep seeing this blog as a source of pleasure, and not as a work or an obligation.

The fact that I write only about things I am familiar with, which I enjoy and when I have some free time (which mostly depend on the ongoing projects related to my PhD thesis) makes it hard for me to follow a defined pace for posts. This is why my writing schedule has been a bit inconsistent during this first year and is likely to be similar in the future as I do not want to be in the position to force myself to write.⁷

Content distribution

Even if we all agree that bloggers should write for themselves first and not for the sake of having lots of readers, it is still appreciated when your content is being read by others.

So although I do not like abusive self-promotion and I do not feel comfortable sharing my blog posts to all existing Facebook, LinkedIn and Reddit groups, somehow people need to get informed that you have written something.

For this reason, I created a Twitter account where I share new posts immediately after publishing them on the blog. In addition to posting them on Twitter, I also share the link by email to people who subscribed to the newsletter of the blog.

I also managed to get my content published on Medium through the Towards Data Science publication, R-bloggers and R Weekly. A non-negligible part of my audience comes from these referral, especially during the couple of days after publication.

By distributing the content this way, I do not feel pushy (something I want to avoid at all costs!) because people decided by themselves to receive the content I publish (e.g., they subscribed to the newsletter, they followed the blog on Twitter, they chose to read blog aggregators, etc.). So in some sense, I do not distribute my content “without their prior consent” and they can always choose not to see my posts anymore.

However, as you can see from the plot of the number of session per channel, you see that most readers come from the organic channel, so from search engines. And for this channel, apart from creating quality content (and some basic knowledge in SEO), I do not have any control on how well it is presented nor distributed to people. For this channel, it is basically search engine algorithms which decide to promote my content or not, and if they do, how well it is promoted. The only control I have regarding ranking factors is simply to create quality content.

So even if I believe that having a content distribution strategy definitely helps in reaching more people and growing your blog, it does not make everything. Creating quality content is, in my view of a beginner in SEO and marketing, the best strategy for my posts to be read.

For this very specific reason, now that I feel I have settled a decent distribution strategy, I no longer spend my energy and time on this matter.

So I just simply try to enjoy writing on my blog, and the rest will eventually follow. If not, I am still learning a lot from this blog anyway so I do not see it as wasted time.

A small note about ads

I know that an easy way to get money out of your blog is to run ads and I understand that people do it if it is profitable. However, I often find ads too intrusive or annoying and I personally do not like to read blog posts which include many ads.

To keep the reading process as enjoyable as possible, as you can see, I do not display any advertising on my blog. As long as the costs of running this website and the related open source projects (e.g., my Shiny apps, etc.) are not too high, I do not expect to include ads.

If in the future costs were to increase, I will still try to avoid putting ads as much as possible and I will try to rely on sponsorship programs & donations and on paid side projects for people who need help for their statistical analyses.

Future plans

After exactly one year of blogging, I am asking myself the following:

What do I want statsandr.com to be?

As already said, as long as I enjoy writing stuff I am passionate about on this blog, I will continue. This is very important to me.

In addition to that, I would like it to be a place to share knowledge. The field of statistics and R (and data science in general) is evolving at an extremely fast pace—and more and more people have many interesting things to say.

Moreover, I learned a lot since the launch this blog. But I learned even more when working in collaboration with someone else (see for instance these collaborations). I see so many learning opportunities when collaborating that I would love to see this blog as a place to share knowledge, but not only my knowledge.

To be more precise, I would love to hear about other researchers, statisticians, R lovers, data scientists, authors, etc. who want to collaborate with me. This could lead, in the end, to:

a guest post if you have a good idea on what you want to write about but need a way to share it (and since you have access to my Google Analytics data in the past year through this article you have a good idea of the number of visitors who will see your content),
a piece of content written together if you believe our skills and knowledge are complementary (see all articles for an overview of what I like to write about),
a piece of code, a R package, a visualization or a Shiny app you want to create together (or just share),
a book or a course on statistics and/or R,
or anything else you have in mind (I am open to new ideas and challenges).

With this article, you see the figures and the audience of the blog. With all my articles, you can see my strengths and weaknesses.

So I will finish this section by saying that, if anyone is willing to work on something together (which does not need to be huge), you can always contact me or leave a comment at the end of this post. I am looking forward to hearing from you.

Thank you note

Last but not least, I also wanted to leave a short thank you note to the Towards Data Science, R-bloggers and R Weekly teams that have cross-posted most of my articles in their respective publications and therefore allowed me to share my thoughts to a larger—and very knowledgeable—audience.

Also, I would like to thank all active readers for their comments, constructive feedback and support so far. I am looking forward to sharing more quality content through this blog and I hope it will keep being useful to many of you, in parallel to being useful to me.

A special thanks to Mark Edmondson, the author of the {googleAnalyticsR} package and all people who wrote tutorials using the package, which were used as inspiration for this blog post. More broadly, thanks also to the open source community which is of great help in learning R.

Thanks for reading. I hope that you learned how to track the performance of your website or blog in R using the {googleAnalyticsR} package. See you next year for a second review, and in the meantime, if you maintain a blog I would be really happy to hear how you track its performance. Feel free to let me know via the comments below!

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Thanks to the RStudio blog for the inspiration.↩︎
See more ways to install and load R packages.↩︎
Many thanks to dartistics.com for the code. Note that for better readability of the plots, I slightly edited the code: (i) to display only the top n pages in terms of traffic instead of all pages, (ii) to change the theme to theme_minimal() and (iii) to make the axis ticks dynamic when zooming in or out (in the ggplotly() function).↩︎
Note that the default period of 180 days can also be changed in the code.↩︎
This may require some assumptions if the percentage of people having access to a computer is not readily available. I believe, however, that it would still be more appropriate than just taking into account the population size—especially when comparing developed with developing countries.↩︎
At least that is what I do when I read blogs on mobile versus reading them on desktop.↩︎
Note, however, that research has shown that scheduling time for writing is a good way to productive writing (Silvia, 2019). This is why, unlike for my blog, I have set regular writing periods in my calendar (and I try to stick to it no matter how busy I am at that time) dedicated to my PhD thesis.↩︎

Graphics in R with ggplot2

Fri, 21 Aug 2020 00:00:00 +0000

Note that this article is inspired from the lecture notes of Prof. Anouar El Ghouch and my personal notes as teaching assistant for his course entitled “Statistics and data sciences with R: Advanced programming” given at UCLouvain.

Introduction

R is known to be a really powerful programming language when it comes to graphics and visualizations (in addition to statistics and data science of course!).

To keep it short, graphics in R can be done in three ways, via the:

{graphics} package (the base graphics in R, loaded by default)
{lattice} package which adds more functionalities to the base package
{ggplot2} package (which needs to be installed and loaded beforehand)

The {graphics} package comes with a large choice of plots (such as plot, hist, barplot, boxplot, pie, mosaicplot, etc.) and additional related features (e.g., abline, lines, legend, mtext, rect, etc.). It is often the preferred way to draw plots for most R users, and in particular for beginners to intermediate users.

Since its creation in 2005 by Hadley Wickham, {ggplot2} has grown in use to become one of the most popular R packages and the most popular package for graphics and data visualizations. The {ggplot2} package is a much more modern approach to creating professional-quality graphics. More information about the package can be found at ggplot2.tidyverse.org.

In this article, we will see how to create common plots such as scatter plots, line plots, histograms, boxplots, barplots, density plots in R with this package. If you are unfamiliar with any of these types of graph, you will find more information about each one (when to use it, its purpose, what does it show, etc.) in my article about descriptive statistics in R.

Data

To illustrate plots with the {ggplot2} package we will use the mpg dataset available in the package.

The dataset contains observations collected by the US Environmental Protection Agency on fuel economy from 1999 to 2008 for 38 popular models of cars (run ?mpg for more information about the data):

library(ggplot2)
dat <- ggplot2::mpg

Before going further, let’s transform the cyl, drv, fl, year and class variables in factor with the transform() function:

dat <- transform(dat,
  cyl = factor(cyl),
  drv = factor(drv),
  fl = factor(fl),
  year = factor(year),
  class = factor(class)
)

For the interested reader, see more data manipulation techniques in R.

Basic principles of `{ggplot2}`

The {ggplot2} package is based on the principles of “The Grammar of Graphics” (hence “gg” in the name of {ggplot2}), that is, a coherent system for describing and building graphs. The main idea is to design a graphic as a succession of layers.

The main layers are:

The dataset that contains the variables that we want to represent. This is done with the ggplot() function and comes first.
The variable(s) to represent on the x and/or y-axis, and the aesthetic elements (such as color, size, fill, shape and transparency) of the objects to be represented. This is done with the aes() function (abbreviation of aesthetic).
The type of graphical representation (scatter plot, line plot, barplot, histogram, boxplot, etc.). This is done with the functions geom_point(), geom_line(), geom_bar(), geom_histogram(), geom_boxplot(), etc.
If needed, additional layers (such as labels, annotations, scales, axis ticks, legends, themes, facets, etc.) can be added to personalize the plot.

To create a plot, we thus first need to specify the data in the ggplot() function and then add the required layers such as the variables, the aesthetic elements and the type of plot:

ggplot(data) +
  aes(x = var_x, y = var_y) +
  geom_x()

data in ggplot() is the name of the data frame which contains the variables var_x and var_y.
The + symbol is used to indicate the different layers that will be added to the plot. Make sure to write the + symbol at the end of the line of code and not at the beginning of the line, otherwise R throws an error.
The layer aes() indicates what variables will be used in the plot and more generally, the aesthetic elements of the plot.
Finally, x in geom_x() represents the type of plot.
Other layers are usually not required unless we want to personalize the plot further.

Note that it is a good practice to write one line of code per layer to improve code readability.

Create plots with `{ggplot2}`

In the following sections we will show how to draw the following plots:

scatter plot
line plot
histogram
density plot
dotplot
boxplot
barplot
raincloud plot

In order to focus on the construction of the different plots and the use of {ggplot2}, we will restrict ourselves to drawing basic (yet beautiful) plots without unnecessary layers. For the sake of completeness, we will briefly discuss and illustrate different layers to further personalize a plot at the end of the article (see this section).

Note that if you still struggle to create plots with {ggplot2} after reading this tutorial, you may find the {esquisse} addin useful. This addin allows you to interactively (that is, by dragging and dropping variables) create plots with the {ggplot2} package. Give it a try!

Scatter plot

We start by creating a scatter plot using geom_point. Remember that a scatter plot is used to visualize the relation between two quantitative variables.

We start by specifying the data:

ggplot(dat) # data

Then we add the variables to be represented with the aes() function:

ggplot(dat) + # data
  aes(x = displ, y = hwy) # variables

Finally, we indicate the type of plot:

ggplot(dat) + # data
  aes(x = displ, y = hwy) + # variables
  geom_point() # type of plot

You will also sometimes see the aesthetic elements (aes() with the variables) inside the ggplot() function in addition to the dataset:

ggplot(mpg, aes(x = displ, y = hwy)) +
  geom_point()

This second method gives the exact same plot than the first method. I tend to prefer the first method over the second for better readability, but this is more a matter of taste so the choice is up to you.

Line plot

Line plots, particularly useful in time series or finance, can be created similarly but by using geom_line():

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_line()

(Note that this might not be the most appropriate plot since there are multiple points for each value of displ, but this is just an example to show how to create a line plot.)

Combination of line and points

An advantage of {ggplot2} is the ability to combine several types of plots and its flexibility in designing it. For instance, we can add a line to a scatter plot by simply adding a layer to the initial scatter plot:

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point() +
  geom_line() # add line

Histogram

A histogram (useful to visualize distributions and detect potential outliers) can be plotted using geom_histogram():

ggplot(dat) +
  aes(x = hwy) +
  geom_histogram()

By default, the number of bins is equal to 30. You can change this value using the bins argument inside the geom_histogram() function:

ggplot(dat) +
  aes(x = hwy) +
  geom_histogram(bins = round(sqrt(nrow(dat))))

Here I specify the number of bins to be equal to the square root of the number of observations (following the square-root rule) but you can specify any integer number.

Density plot

Density plots can be created using geom_density():

ggplot(dat) +
  aes(x = hwy) +
  geom_density()

Combination of histogram and densities

We can also superimpose a histogram and a density curve on the same plot:

ggplot(dat) +
  aes(x = hwy, y = after_stat(density)) +
  geom_histogram() +
  geom_density()

Or superimpose several densities:

ggplot(dat) +
  aes(x = hwy, color = drv, fill = drv) +
  geom_density(alpha = 0.25) # add transparency

The argument alpha = 0.25 has been added for some transparency. More information about this argument can be found in this section.

Dotplot

A dotplot in {ggplot2} can be built with geom_dotplot():

# Dotplot for one variable
ggplot(dat) +
  aes(x = "", y = hwy) +
  geom_dotplot(binaxis = "y", stackdir = "center")

# Dotplot by factor
ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_dotplot(
    binaxis = "y", stackdir = "center",
    dotsize = 0.75 # decrease dot size
  )

Dotplots are more appropriate with small samples (because the plot may be hard to read with too many points). For large samples, a boxplot may be used.

For the interested reader, see many personalization that is possible with a dotplot in this tutorial.

Boxplot

A boxplot (also very useful to visualize distributions and detect potential outliers) can be plotted using geom_boxplot():

# Boxplot for one variable
ggplot(dat) +
  aes(x = "", y = hwy) +
  geom_boxplot()

# Boxplot by factor
ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot()

It is also possible to plot the points on the boxplot with geom_jitter(), and to vary the width of the boxes according to the size (i.e., the number of observations) of each level with varwidth = TRUE:

ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) # adds random noise and limit its width

The geom_jitter() layer adds some random variation to each point in order to prevent them from overlapping (an issue known as overplotting).¹ Moreover, the alpha argument adds some transparency to the points (see more in this section) to keep the focus on the boxes and not on the points.

Finally, it is also possible to divide boxplots into several panels according to the levels of a qualitative variable:

ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) # divide into 2 panels

For a visually more appealing plot, it is also possible to use some colors for the boxes depending on the x variable:

ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) + # add color to boxes with fill
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) + # divide into 2 panels
  theme(legend.position = "none") # remove legend

In that case, it best to remove the legend as it becomes redundant. See more information about the legend in this section.

If you are unhappy with the default colors provided in {ggplot2}, you can change them manually with the scale_fill_manual() layer:

ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) + # add color to boxes with fill
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) + # divide into 2 panels
  theme(legend.position = "none") + # remove legend
  scale_fill_manual(values = c("darkred", "darkgreen", "steelblue")) # change fill color manually

Barplot

A barplot (useful to visualize qualitative variables) can be plotted using geom_bar():

ggplot(dat) +
  aes(x = drv) +
  geom_bar()

Bars’ heights correspond to the observed frequencies (i.e., the number of observations) for each level of the variable of interest (drv in our case).

Tip: by default, the order of the bars follows the initial order (by alphabetical order or numerical order if you did not change it). If you want to order the levels by frequency (largest first), use the fct_infreq() function from the {forcats} package.

library(forcats)

ggplot(dat) +
  aes(x = fct_infreq(drv)) + # order by frequency
  geom_bar()

If you want to order levels in an increasing order (i.e., category with the smallest frequency first), use the fct_rev() in addition to the fct_infreq() function:

ggplot(dat) +
  aes(x = fct_rev(fct_infreq(drv))) + # order by frequency
  geom_bar()

(Label for the x-axis can then easily be edited with the labs() function. See below for more information.)

Again, for a more appealing plot, we can add some colors to the bars with the fill argument:

ggplot(dat) +
  aes(x = drv, fill = drv) + # add colors to bars
  geom_bar() +
  theme(legend.position = "none") # remove legend

We can also create a barplot with two qualitative variables:

ggplot(dat) +
  aes(x = drv, fill = year) + # fill by years
  geom_bar()

In order to compare proportions across groups, it is best to make each bar the same height using position = "fill":

ggplot(dat) +
  geom_bar(aes(x = drv, fill = year), position = "fill")

To draw the bars next to each other for each group, use position = "dodge":

ggplot(dat) +
  geom_bar(aes(x = drv, fill = year), position = "dodge")

Raincloud plot

A raincloud plot is a graph that combines 3 visualizations:

a density plot,
a boxplot,
and the raw data in the form of a dotplot or jittered points.

The advantage of this plot is that it illustrates, all at once, the distribution (with the density curve), the summary measures (first, second and third quartiles, and maximum/mininum without outliers thanks to the boxplot) and the number of observations (either via a dotplot or via jittered points).

Let’s illustrate the raincloud plot, first with jittered points (more appropriate with large samples):²

library(tidyverse)
library(ggdist)

# density plot:
dat %>%
  ggplot(aes(x = drv, y = hwy, fill = drv)) +
  stat_halfeye(
    adjust = 0.5, # set the smoothing parameter
    width = 0.5, # set the height of the curves
    justification = -0.2, # move curves to the right
    .width = 0, point_colour = NA # remove interval present by default
  ) +
  # boxplot:
  geom_boxplot(
    width = 0.12, # width of boxes
    outlier.color = NA, # remove color of outliers
    alpha = 0.5 # add transparency
  ) +
  # jittered points:
  geom_point(aes(colour = drv), # add color on points
    size = 1.3, # size of points
    alpha = .3, # add transparency
    position = position_jitter( # obtain shifted points
      seed = 1, # set seed for same random representation
      width = .09 # manage the width of the offset
    )
  ) +
  # further personalization:
  coord_flip() + # rotate plot
  theme(legend.position = "none") # remove legend

Now the same chart but with dotplots this time (more appropriate with small samples):

# density plot:
dat %>%
  sample_n(100) %>% # random sample of size 100
  ggplot(aes(x = drv, y = hwy, fill = drv)) +
  stat_halfeye(
    adjust = 0.5, # set the smoothing parameter
    width = 0.5, # set the height of the curves
    justification = -0.2, # move curves to the right
    .width = 0, point_colour = NA # remove interval present by default
  ) +
  # boxplot:
  geom_boxplot(
    width = 0.12, # width of boxes
    outlier.color = NA, # remove color of outliers
    alpha = 0.5 # add transparency
  ) +
  # dotplot:
  stat_dots(
    dotsize = 0.5, # size of points
    side = "left", # place points on opposite side of density curve
    justification = 1.1, # move points away from boxplot
    binwidth = 1 # group points together
  ) +
  # further personalization:
  coord_flip() + # rotate plot
  theme(legend.position = "none") # remove legend

The code is much longer compared to the other plots, but the only line(s) to edit to adapt to your dataset is the aesthetics (aes()). The rest is mainly adjustments that should not be changed.

You may have notice that at the end of the code, there are some personalization which allow to improve the plot even further. The most common personalization are presented in the next section.

Further personalization

Title and axis labels

The first things to personalize in a plot is the labels to make the plot more informative to the audience. We can easily add a title, subtitle, caption and edit axis labels with the labs() function:

p <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

p + labs(
  title = "Fuel efficiency for 38 popular models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
)

As you can see in the above code, you can save one or more layers of the plot in an object for later use.

This way, you can save your “main” plot, and add more layers of personalization until you get the desired output. Here we saved the main scatter plot in an object called p and we will refer to it for the subsequent personalization.

You can also edit the alignment, the size and the shape of the title and subtitle via the theme() layer and the element_text() function:

p + labs(
  title = "Fuel efficiency for 38 popular models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
) +
  theme(
    plot.title = element_text(
      hjust = 0.5, # center
      size = 12,
      color = "steelblue",
      face = "bold"
    ),
    plot.subtitle = element_text(
      hjust = 0.5, # center
      size = 10,
      color = "gray",
      face = "italic"
    )
  )

If the title or subtitle is long and you want to divide it into multiple lines, use \n:

p + labs(
  title = "Fuel efficiency for 38 popular \n models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
) +
  theme(
    plot.title = element_text(
      hjust = 0.5, # center
      size = 12,
      color = "steelblue",
      face = "bold"
    ),
    plot.subtitle = element_text(
      hjust = 0.5, # center
      size = 10,
      color = "gray",
      face = "italic"
    )
  )

Axis ticks

Axis ticks can be adjusted using scale_x_continuous() and scale_y_continuous() for the x and y-axis, respectively:

# Adjust ticks
p + scale_x_continuous(breaks = seq(from = 1, to = 7, by = 0.5)) + # x-axis
  scale_y_continuous(breaks = seq(from = 10, to = 45, by = 5)) # y-axis

Log transformations

In some cases, it is useful to plot the log transformation of the variables. This can be done with the scale_x_log10() and scale_y_log10() functions:

p + scale_x_log10() +
  scale_y_log10()

Limits

The most convenient way to control the limits of the plot is to use again the scale_x_continuous() and scale_y_continuous() functions in addition to the limits argument:

p + scale_x_continuous(limits = c(3, 6)) +
  scale_y_continuous(limits = c(20, 30))

It is also possible to simply take a subset of the dataset with the subset() or filter() function. See how to subset a dataset if you need a reminder.

Scales for better axis formats

Depending on your data, it is possible to format axes in a certain way with the {scales} package. The format I use the most is comma which formats large numbers in a more-readable way.

For this example, we multiply both variables by 10000 to have larger numbers and then we apply the format to the y-axis (only to the y-axis so we can see the difference with the x-axis which is not formatted):

ggplot(dat) +
  aes(x = displ * 10000, y = hwy * 10000) +
  geom_point() +
  scale_y_continuous(labels = scales::comma) # format y-axis

As you can see, numbers on the y-axis are displayed as 200,000, 300,000, etc. instead of 200000, 300000, etc., which makes it more readable.

Another common format is percent to display numbers as percentages:

ggplot(dat) +
  aes(x = displ, y = hwy / 100) +
  geom_point() +
  scale_y_continuous(labels = scales::percent) # format y-axis

These two formats make large numbers and percentages easier to read. Other formats are possible such as using dollar signs, dates etc. See more information in the package’s documentation.

Legend

By default, the legend is located to the right side of the plot (when there is a legend to be displayed of course).

To control the position of the legend, we need to use the theme() function in addition to the legend.position argument:

p + aes(color = class) +
  theme(legend.position = "top")

Replace "top" by "left" or "bottom" to change its position and by "none" to remove it.

The title of the legend can be edited with the labs() layer:

p + aes(color = class) +
  labs(color = "Car's class")

Note that the argument inside labs() must match the one inside the aes() layer (in this case: color).

The title of the legend can also be removed with legend.title = element_blank() inside the theme() layer:

p + aes(color = class) +
  theme(
    legend.title = element_blank(),
    legend.position = "bottom"
  )

The legend now appears at the bottom of the plot, without the legend title.

Shape, color, size and transparency

There are a very large number of options to improve the quality of the plot or to add additional information. These include:

shape,
size,
color, and
alpha (transparency).

We can for instance change the shape of all points in a scatter plot by adding shape to geom_point(), or vary the shape according to the values taken by another variable (in that case, the shape argument must be inside aes()):³

# Change shape of all points
ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point(shape = 4)

# Change shape of points based on a categorical variable
ggplot(dat) +
  aes(x = displ, y = hwy, shape = drv) +
  geom_point()

Following the same principle, we can modify the color, size and transparency of the points based on a qualitative or quantitative variable. Here are some examples:

p <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

# Change color for all points
p + geom_point(color = "steelblue")

# Change color based on a qualitative variable
p + aes(color = drv)

# Change color based on a quantitative variable
p + aes(color = cty)

# Change color based on a criterion (median of cty variable)
p + aes(color = cty > median(cty))

# Change size of all points
p + geom_point(size = 4)

# Change size of points based on a quantitative variable
p + aes(size = cty)

# Change transparency based on a quantitative variable
p + aes(alpha = cty)

We can of course mix several options (shape, color, size, alpha) to build more complex graphics:

p + geom_point(size = 0.5) +
  aes(color = drv, shape = year, alpha = cty)

If you are unhappy with the default colors, you can change them manually with the scale_colour_manual() layer (for qualitative variables) and the scale_colour_gradient2() layer (for quantitative variables):

# Change color based on a qualitative variable
p + aes(color = drv) +
  scale_colour_manual(values = c("red", "blue", "green"))

# Change color based on a quantitative variable
p + aes(color = cty) +
  scale_colour_gradient2(
    low = "green",
    mid = "gray",
    high = "red",
    midpoint = median(dat$cty)
  )

For your information, you can emulate {ggplot2} default color palette for a desired number of colors and produce a character vector of HEX colors. For example, with 4 colors:

library(scales)
show_col(hue_pal()(4))

Text and labels

To add a label on a point (for example the row number), we can use the geom_text() and aes() functions:

p + geom_text(aes(label = rownames(dat)),
  check_overlap = TRUE,
  size = 2,
  vjust = -1
)

To add text on the plot, we use the annotate() function:

p + annotate("text",
  x = 6,
  y = 40,
  label = "hwy and displ are \n negatively correlated \n (rho = -0.77, p-value < 0.001)",
  size = 3
)

Read the article on correlation coefficient and correlation test in R to see how I computed the correlation coefficient (rho) and the p-value of the correlation test.

Smooth and regression lines

In a scatter plot, it is possible to add a smooth line fitted to the data:

p + geom_smooth()

In the context of simple linear regression, it is often the case that the regression line is displayed on the plot.

This can be done by adding method = lm (lm stands for linear model) in the geom_smooth() layer:

p + geom_smooth(method = lm)

It is also possible to draw a regression line for each level of a categorical variable:

p + aes(color = drv, shape = drv) +
  geom_smooth(method = lm, se = FALSE)

The se = FALSE argument removes the confidence interval around the regression lines.

Facets

facet_grid allows you to divide the same graphic into several panels according to the values of one or two qualitative variables:

# According to one variable
p + facet_grid(. ~ drv)

# According to 2 variables
p + facet_grid(drv ~ year)

It is then possible to add a regression line to each facet:

p + facet_grid(. ~ drv) +
  geom_smooth(method = lm)

facet_wrap() can also be used, as illustrated in this section.

Themes

Several functions are available in the {ggplot2} package to change the theme of the plot.

The most common themes after the default theme (i.e., theme_gray()) are the black and white (theme_bw()), minimal (theme_minimal()) and classic (theme_classic()) themes:

# Black and white theme
p + theme_bw()

# Minimal theme
p + theme_minimal()

# Classic theme
p + theme_classic()

I tend to use the minimal theme for most of my R Markdown reports as it brings out the patterns and points and not the layout of the plot, but again this is a matter of personal taste. See more themes at ggplot2.tidyverse.org/reference/ggtheme.html and in the {ggthemes} package.

In order to avoid having to change the theme for each plot you create, you can change the theme for the current R session using the theme_set() function as follows:

theme_set(theme_minimal())

Interactive plot with `{plotly}`

You can easily make your plots created with {ggplot2} interactive with the {plotly} package:

library(plotly)
ggplotly(p + aes(color = year))

You can now hover over a point to display more information about that point. There is also the possibility to zoom in and out, to download the plot, to select some observations, etc. More information about {plotly} for R can be found here.

Combine plots with `{patchwork}`

There are several ways to combine plots made in {ggplot2}. In my opinion, the most convenient way is with the {patchwork} package using symbols such as +, / and parentheses.

We first need to create some plots and save them:

p_a <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

p_b <- ggplot(dat) +
  aes(x = hwy) +
  geom_histogram()

p_c <- ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot()

Now that we have 3 plots saved in our environment, we can combine them. To have plots next to each other simply use the + symbol:

library(patchwork)
p_a + p_b + p_c

To display them above each other simply use the / symbol:

p_a / p_b / p_c

And finally, to combine them above and next to each other, mix +, / and parentheses:

p_a + p_b / p_c

(p_a + p_b) / p_c

See more ways to combine plots with:

grid.arrange() from the {gridExtra} package
plot_grid() from the {cowplot} package

Flip coordinates

Flipping coordinates of your plot is useful to create horizontal boxplots, or when labels of a variable are so long that they overlap each other on the x-axis. See with and without flipping coordinates below:

# without flipping coordinates
p1 <- ggplot(dat) +
  aes(x = class, y = hwy) +
  geom_boxplot()

# with flipping coordinates
p2 <- ggplot(dat) +
  aes(x = class, y = hwy) +
  geom_boxplot() +
  coord_flip()

library(patchwork)
p1 + p2 # left: without flipping, right: with flipping

This can be done with many types of plot, not only with boxplots. For instance, if a categorical variable has many levels or the labels are long, it is usually best to flip the coordinates for a better visual:

ggplot(dat) +
  aes(x = class) +
  geom_bar() +
  coord_flip()

Save plot

The ggsave() function will save the most recent plot in your current working directory unless you specify a path to another folder:

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

ggsave("plot1.pdf")

You can also specify the width, height and resolution (dpi) as follows:

ggsave("plot1.pdf",
  width = 12,
  height = 12,
  units = "cm",
  dpi = 300
)

Managing dates

If the time variable in your dataset is in date format, the {ggplot2} package recognizes the date format and automatically uses a specific type for the axis ticks.

There is no time variable with a date format in our dataset, so let’s create a new variable of this type thanks to the as.Date() function:

dat$date <- as.Date("2020-08-21") - 0:(nrow(dat) - 1)

See the first 6 observations of this date variable and its class:

head(dat$date)

## [1] "2020-08-21" "2020-08-20" "2020-08-19" "2020-08-18" "2020-08-17"
## [6] "2020-08-16"

str(dat$date)

##  Date[1:234], format: "2020-08-21" "2020-08-20" "2020-08-19" "2020-08-18" "2020-08-17" ...

The new variable date is correctly specified in a date format.

Most of the time, with a time variable, we want to create a line plot with the date on the X-axis and another continuous variable on the Y-axis, like the following plot for example:

p <- ggplot(dat) +
  aes(x = date, y = hwy) +
  geom_line()
p

As soon as the time variable is recognized as a date, we can use the scale_x_date() layer to change the format displayed on the X-axis. The following table shows the most frequent date formats:

Source: www.statmethods.net

Run ?strptime() to see many more date formats available in R.

For this example, let’s add the year in addition to the unabbreviated month:

p + scale_x_date(date_labels = "%B %Y")

It also possible to control the breaks to display on the X-axis with the date_breaks argument. For this example, let’s say we want to display the day as number and the abbreviated month for each interval of 10 days:

p + scale_x_date(date_breaks = "10 days", date_labels = "%d %b")

If labels displayed on the X-axis are unreadable because they overlap each other, you can rotate them with the theme() layer and the angle argument:

p + scale_x_date(date_breaks = "10 days", date_labels = "%d %b") +
  theme(axis.text.x = element_text(angle = 60, hjust = 1))

Highlight data with `{gghighlight}`

The {gghighlight} package allows, as its name suggests, to highlight some data directly on your ggplot. The highlighted data (that you define) are shown in bright color and the rest in gray.

Below example of how it works for a scatter plot, boxplot, barplot and histogram.

library(gghighlight)

# scatter plot
ggplot(mpg, aes(x = displ, y = hwy, color = cyl)) +
  geom_point() +
  gghighlight(cyl == "8") +
  theme(legend.position = "none") # remove legend

# boxplot
ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) +
  geom_boxplot() +
  gghighlight(drv %in% c("r", "4")) +
  theme(legend.position = "none") # remove legend

# barplot
ggplot(dat) +
  aes(x = drv, fill = drv) +
  geom_bar() +
  gghighlight(drv == "f")

As you can see, the gghighlight() layer accepts different types of conditions, but also several of them at the same time:

# histogram
ggplot(dat) +
  aes(x = hwy, fill = year) +
  geom_histogram() +
  gghighlight(displ > 2 & year == "1999")

Tip

I recently learned a tip very useful when drawing plots with {ggplot2}. If like me, you often comment and uncomment some lines of code in your plot, you know that you cannot transform the last line into a comment without removing the + sign in the line just above.

Adding a line NULL at the end of your plots will avoid an error if you forget to remove the + sign in the last line of your code. See with this basic example:

ggplot(dat) +
  aes(x = date, y = hwy) +
  geom_line() + # I do not have to remove the + sign
  # theme_minimal() + # this line is a comment
  NULL # adding this line doesn't change anything to the plot

This trick saves me a lot of time as I do not need to worry about making sure to remove the last + sign after commenting some lines of code in my plots.

If you find this trick useful, you may like these other tips and tricks in RStudio and R Markdown.

To go further

By now you have seen that {ggplot2} is a very powerful and complete package to create plots in R. This article illustrated only the tip of the iceberg, and you will find many tutorials on how to create more advanced plots and visualizations with {ggplot2} online. If you want to learn more than what is described in the present article, I highly recommend starting with:

the chapters Data visualisation and Graphics for communication from the book R for Data Science from Garrett Grolemund and Hadley Wickham
the book ggplot2: Elegant Graphics for Data Analysis from Hadley Wickham
the book R Graphics Cookbook from Winston Chang
the ggplot2 extensions guide which lists many of the packages that extend {ggplot2}
the {ggplot2} cheat sheet
this presentation by Garrick Aden-Buie
a detailed tutorial by Cédric Scherer

Conclusion

Thanks for reading.

I hope this article helped you to create your first plots with the {ggplot2} package. As a reminder, for simple graphs, it is sometimes easier to draw them via the {esquisse} addin. After some time, you will quickly learn how to create them by yourselves and in no time you will be able to build complex and sophisticated data visualizations.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Use the geom_jitter() layer with caution because, although it makes a plot more revealing at large scales, it also makes it slightly less accurate at small scales since some randomness is added to the points.↩︎
Code inspired from Claire Della Vedova (DellaData) and Cédric Scherer. Thanks to both of them for this nice plot!↩︎
There are (at the time of writing) 26 shapes accepted in the shape argument. See this documentation for all available shapes.↩︎

COVID-19 in Belgium: is it over yet?

Fri, 22 May 2020 00:00:00 +0000

Introduction

Note 1: The present article has been written on May 22, 2020 and has been updated infrequently. The current situation regarding COVID-19 in Belgium may therefore be different to what is presented below. See my Twitter profile for more frequent updates of the plots.

Note 2: This is a joint work with Prof. Niko Speybroeck, Prof. Catherine Linard, Prof. Simon Dellicour and Angel Rosas-Aguirre.

Belgium recently started to lift its lockdown measures initially imposed to contain the spread of the Covid-19. Following this decision taken by Belgian authorities, we analyze how the situation evolved so far.

Contrarily to a previous article in which I analyzed the outbreak of the Coronavirus in Belgium using the SIR model, in this article we focus on the evolution of the number of:

hospital admissions
patients in hospitals
patients in intensive care
new confirmed cases

at the province and national level.

Data is from Sciensano and all plots were created with the {ggplot2} package.

New hospital admissions

Overall

Belgian hospitalizations COVID-19

From the above figure, we see that the rate of hospitalizations continue with a decreasing trend in all provinces (and in Belgium as well).

Download the figure, or see the code.

Update of October 27, 2020:

COVID19 hospitalizations in Belgium

Download the figure, or see the code.

The detailed situation in Brabant:

Download the figure, or see the code.

By period

Daily COVID19 hospitalizations in Belgium from March to October 2020

Download the figure, or see the code.

Update of November 16, 2020:

Daily COVID19 hospitalizations in Belgium by period

Download the figure.

In the first wave, the province of Limburg recorded on average the highest number of COVID19 hospital admissions per million inhabitants. During the second wave, Liège and Hainaut struggled with the highest rates. With two exceptions (Antwerp and Limburg), last month was worse than in March-April. In three provinces (Hainaut, Namur and Liège), the number has more than doubled.

During the period from June 14 to July 15, 2020, the number of COVID19 hospital admissions in Belgium fell to very low relative levels, but we have failed to maintain them. Now that hospital admissions are no longer increasing, we hope that the colors will lighten up again a bit as the end of the year approaches.

Zooming in

Hospital admissions COVID-19 - Belgium

Download the figure or see the code.

Update of February 26, 2021:

There is some ongoing debate in Belgium on whether or not to ease restrictions. On February 26, 2021, Belgian authorities will meet, discuss, debate and decide. Current levels and trends of COVID-19 hospitalizations may guide them:

There is still no strong evidence that COVID-19 hospitalization curves decrease in Belgium. The comparison between the first (in gray - dates & curve) and second wave (in blue - dates & curve) needs to be done with care, but indicates that current hospitalization levels are not as low as some may like:

Zooming in provides some additional insights on the COVID-19 levels during the first and second waves at province level:

This shows that the second wave resulted in more hospitalizations than the first one in most Belgian provinces, despite the warning of a first deadly wave. It also illustrates the fact that daily hospitalizations in Belgium are currently still higher than what was observed at the end of the first wave.

Put simply, the bad news is that the combination of the number of contacts and the risk of transmission by contact seems (at the moment) not sufficiently low to result in a considerable decrease of hospitalizations. Yet (put simply), the good news today is that there is already some immunity in the population and that vaccinations may increase this immunity considerably. This can help in pushing curves down. Let’s not lose hope.

Download figures (1, 2 and 3) or see the code.

Update of May 10, 2021:

COVID19 hospitalizations - Wave 1 and 2

When looking at the above plot, bad news are that:

current levels correspond to levels of October 2020 and
current levels are still about double the target of 75 hospitalizations per day.

There are, however, three good news (compared to October 2020):

decreasing curve,
vaccination and
good weather.

Update of June 4, 2021

COVID-19 hospitalizations in Belgium below 75/day

The good news is that the number of COVID-19 hospitalizations in Belgium is now below the well-known threshold of 75 hospitalizations per day (which is a target defined by the Belgian government). This is the way to go, and we hope this trend will continue in the coming days/weeks.

Patients in hospitals

Below the evolution of the number of patients in hospitals in Belgium:

COVID19 patients in hospitals in Belgium

Download the figure or see the code.

We see that, as of October 28, 2020, the number of COVID19 patients in Belgian hospitals reached the peak of the first wave. So although patients stay shorter at the hospital during the second wave compared to the first wave, hospitals are still getting crowded.

Therefore, if the number of patients in hospitals follows the same path in the coming weeks, hospitals will quickly become too crowded and will not be able to accept new patients as their maximum capacity will soon be reached (if this is not already the case…).

Patients in intensive care

Below the evolution of COVID19 patients in intensive care in Belgium, with short-term projections and 99% confidence interval:

Evolution of COVID19 patients in intensive care in Belgium

Download the figure.

Short-term projections indicate what may have happened without the slow-down in transmission. This slow-down is positive news.

The maps show total intensive care patients by province if these would have had the Belgian population. Map at the top shows maximum levels in March-April and map at the bottom shows current levels. The maps indicate high intensive care use due to COVID19. In most Belgian provinces, numbers are still higher today than March-April peak numbers.

Observations are in line with other preliminary indications, such as trends of COVID19 hospitalizations (currently relatively volatile), indicating that transmission is slowing down:

Download the figure.

Confirmed cases

Note that the reported number of new confirmed cases is probably underestimated. This number does not take into account undiagnosed (without or with few symptoms) or untested cases. Therefore, figures with number of cases should be interpreted with extreme caution.

By province

New confirmed COVID-19 cases in Belgium

Download the figure or see the code.

By age group and sex

Static

Below another visualization of the number of cases by age group and sex in Belgium, for three different periods:

COVID-19 cases by age group and sex in Belgium

Download the figure or see the code.

This visualization shows the importance to report ages of cases and not just total number.

Moreover, we see that the distribution of cases per week by age group at the beginning of September is similar than during the summer holidays, but the number of cases per week is higher. The distribution of cases per week by age group at the beginning of September is however different from the “first wave” (period from March 1, 2020 to May 31, 2020). During the fist period, majority of cases were elderly, while at the beginning of September majority of cases are young people. It would be interesting to see how the distribution of cases by age group evolves during winter.

The figure above may be put in relation with the structure of the Belgian population:

Structure of Belgian population (2019)

Download the figure or see the code.

Dynamic

Additionally, these can be seen dynamically:

COVID-19 cases by age group and sex in Belgium - dynamic version

Download the figure or see the code.

With an update of the second wave:

Age and sex specific incidence per 100 000 of COVID19 cases in Belgium - dynamic version

Download the figure or see the code.

By age group, sex and province

Download the figure or see the code.

Conclusion

Thanks for reading.

We hope that these figures will evolve in the right direction. In the meantime, take care and stay safe!

If you would like to be further updated on the evolution of the COVID-19 epidemic, two options:

visit the blog from time to time, and
join Twitter and follow us: statsandr, NikoSpeybroeck & arosas_aguirre

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

How to create a simple Coronavirus dashboard specific to your country in R?

Mon, 23 Mar 2020 00:00:00 +0000

Coronavirus dashboard: the case of Belgium

Introduction

The Novel COVID-19 Coronavirus is the hottest topic right now. Every day, the media and newspapers share the number of new cases and deaths in several countries, try to measure the impacts of the virus on citizens and remind us to stay home in order to stay safe. The Coronavirus is on everyone’s lips.

In addition to governments, media and companies discussing about it, data scientists and data professionals in general are putting their knowledge and time at the service of the virus. This leads to a proliferation of applications, dashboards, blog posts, videos, datasets and code analyzing, in one way or another, the expansion of the COVID-19 and how it spreads in the population.

Top R resources on Coronavirus

As a data lover myself, I discovered a multitude of great resources about the Coronavirus. However, these resources were spread all over the internet and were often hidden by the mass of information of another type (e.g., alarming headlines, names of infected celebrities, companies showing off how they helped health care agencies, etc.). To tackle this issue, I collected and then shared in a previous article the best R resources on the Coronavirus I came across.

Note that only resources on R are shared in this article as R is my favorite statistical program and the one I am most familiar with. The fact that I use this program almost daily makes it easier for me to realize the complexity and time put behind a resource, and appreciate its quality and its potential.

I am sure that there are other very interesting resources online (see for example the probably most popular dashboard by the Johns Hopkins Coronavirus Resource Center).¹ Nonetheless, a lot of people are in a better position than I am to judge the quality of resources made with programming languages for which I am not competent.

This article made me discover so many great resources about the Coronavirus and I keep receiving data visualizations and data analyses from scientists all over the world so that I include them in the collection. Thanks for that, it continuously improves the quality and completeness of the collection.

In addition to receiving R resources, a question often asked by readers was “How can I create a dashboard myself?” or “How can I build a dashboard specific to my country?”. I therefore thought it would serve some people if I created a dashboard specific to my country (Belgium) and detailed the steps on how to build it.

Questions on how to develop such dashboards came mostly from R beginners as advanced R users most probably know how to do one, or at least can easily use the resources I collected here as sources of inspiration for their own work. Furthermore, in response to the craze about the Coronavirus, interested users were quite in a hurry and wanted to have their own dashboard running as quickly as possible.

These questions led me to the idea of creating a simple (yet powerful and visually appealing) dashboard, as opposed to a Shiny app. Shiny apps have the advantage that they are interactive in the sense that users can edit the outputs and the visualizations by simply changing some inputs in a user-friendly way, while dashboards are static and cannot be modified by the final user. On the other hand, the advantage of a dashboard over a Shiny app is that it is much easier to code, especially if you are already proficient in R Markdown.

Coronavirus dashboard: the case of Belgium

Amongst all the visualizations I have seen so far, one is standing out by its simplicity and at the same time, by its completeness and by the quality of its visualizations. I thus decided to create a Coronavirus dashboard based on this already existing dashboard by Rami Krispin (which comes with a license that allows to be freely adapted and shared) and adapt it so that it is specific to Belgium. Note that I also removed some visualizations and tables from the initial dashboard to keep it really simple and straight to the point.

Before reading further, here is my Coronavirus dashboard adapted to Belgium and previews of the main sections below:

The dashboard is segmented into several sections that can be selected at the top:

The summary section provides key measures about the Coronavirus (total cases, active cases and deaths) and a plot displaying the cumulative number of active cases and deaths from January 22, 2020 to the latest available date.
The comparison section presents a comparison of the number of daily new cases (left panel) and the distribution of cases by type (right panel) with other European countries (you can also change these countries by replacing them in the code).
The map section shows a world map of the confirmed cases and deaths. You can uncheck one or several types of cases (top right corner) and zoom in or out (top left corner) to adapt the map to your needs.
The about section gives more information about the data, the dashboard in general and how often it is updated.

I believe this simple dashboard is easy enough to be adapted to any country (and by anyone from beginner to expert), and still communicate key measures about the virus throughout some visualizations.

A little extra which is worth mentioning is the fact that all plots are generated with the {plotly} package. This package allows to enhance plots by displaying additional relevant information when hovering over them (try by yourself!).

How to create your own Coronavirus dashboard

If you want to build your own dashboard specific to a country, follow these steps:

Open the dashboard here
See the entire code via the button “Source code” located in the top right corner of the dashboard, or see the code on GitHub. Copy that code.
Open a new R Markdown file (.Rmd), type any title and author (they will be replaced in the next step anyway), select HTML as the output format and click on OK:

Remove all the template code already present and paste the code you copied in step 1.
Make sure that the required packages are installed:

install.packages(c("devtools", "flexdashboard", "leaflet", "leafpop"))
devtools::install_github("RamiKrispin/coronavirus", force = TRUE)

If R asks you which package you would like to update, there should be no need to update them: type 3 for “None”.

In the code, replace Belgium with your country. Here is the list of all available countries in the dataset:

Canada, United Kingdom, China, Netherlands, Australia, New Zealand, Denmark, France, Afghanistan, Albania, Algeria, Andorra, Angola, Antarctica, Antigua and Barbuda, Argentina, Armenia, Austria, Azerbaijan, Bahamas, Bahrain, Bangladesh, Barbados, Belarus, Belgium, Belize, Benin, Bhutan, Bolivia, Bosnia and Herzegovina, Botswana, Brazil, Brunei, Bulgaria, Burkina Faso, Burma, Burundi, Cabo Verde, Cambodia, Cameroon, Central African Republic, Chad, Chile, Colombia, Comoros, Congo (Brazzaville), Congo (Kinshasa), Costa Rica, Cote d’Ivoire, Croatia, Cuba, Cyprus, Czechia, Diamond Princess, Djibouti, Dominica, Dominican Republic, Ecuador, Egypt, El Salvador, Equatorial Guinea, Eritrea, Estonia, Eswatini, Ethiopia, Fiji, Finland, Gabon, Gambia, Georgia, Germany, Ghana, Greece, Grenada, Guatemala, Guinea, Guinea-Bissau, Guyana, Haiti, Holy See, Honduras, Hungary, Iceland, India, Indonesia, Iran, Iraq, Ireland, Israel, Italy, Jamaica, Japan, Jordan, Kazakhstan, Kenya, Kiribati, Korea, North, Korea, South, Kosovo, Kuwait, Kyrgyzstan, Laos, Latvia, Lebanon, Lesotho, Liberia, Libya, Liechtenstein, Lithuania, Luxembourg, Madagascar, Malawi, Malaysia, Maldives, Mali, Malta, Marshall Islands, Mauritania, Mauritius, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Morocco, Mozambique, MS Zaandam, Namibia, Nepal, Nicaragua, Niger, Nigeria, North Macedonia, Norway, Oman, Pakistan, Palau, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Poland, Portugal, Qatar, Romania, Russia, Rwanda, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, Samoa, San Marino, Sao Tome and Principe, Saudi Arabia, Senegal, Serbia, Seychelles, Sierra Leone, Singapore, Slovakia, Slovenia, Solomon Islands, Somalia, South Africa, South Sudan, Spain, Sri Lanka, Sudan, Summer Olympics 2020, Suriname, Sweden, Switzerland, Syria, _Taiwan*_, Tajikistan, Tanzania, Thailand, Timor-Leste, Togo, Tonga, Trinidad and Tobago, Tunisia, Turkey, Uganda, Ukraine, United Arab Emirates, Uruguay, US, Uzbekistan, Vanuatu, Venezuela, Vietnam, West Bank and Gaza, Winter Olympics 2022, Yemen, Zambia and Zimbabwe

Note that if your country is spelled in two words or more, you will need to surround it by a backtick (but only at one specific line in the code, see an example with United Kingdom):

#----------------------------------------
# Plotting the data

daily_confirmed %>%
  plotly::plot_ly() %>%
  plotly::add_trace(
    x = ~date,
    y = ~`United Kingdom`,
    type = "scatter",
    mode = "lines+markers",
    name = "United Kingdom"
  ) %>%

Do not add backticks in the rest of the code as everywhere else the country’s name is surrounded by double quotes "".

Do not forget to also change the title and the author at the top of the document, and edit the about section at the bottom of the document. Last but not least, as you can see on the plot in the summary section, the arrows point to different (sad) “milestones” in Belgium (i.e., first case, first death and new containment measures).

You will need to adapt these milestones for your country (or remove them if you do not want to have any milestone displayed on the plot). Change this in the code following the plotly::add_annotations() functions.

Knit the document (see this article if you are unfamiliar with R Markdown). Your dashboard should appear in HTML format.

Following these 7 steps, you should already have a simple dashboard specific to your country. I have intentionally kept it simple so that everyone could copy it and have their own dashboard in a limited amount of time.

If you are familiar with the {flexdashboard}, {plotly} and {leaflet} packages for the dashboard interface and the visualizations, and the {dplyr} and {tidyr} packages for the data manipulation, feel free to edit the code according to your needs and improve your dashboard.

Additional notes

Data

The input data for this dashboard is the dataset available from the {coronavirus} R package. Make sure to download the development version of the package to have the latest data:

install.packages("devtools")
devtools::install_github("RamiKrispin/coronavirus")

To update your dashboard with the latest data, you have to manually update the data by reinstalling the {coronavirus} package with devtools::install_github("RamiKrispin/coronavirus", force = TRUE). Again, if R asks you whether you would like to update other packages, type 3 for “None”.

This question was often raised so I repeat, your dashboard will not update by itself every day, you need to manually update it. After updating the data, you may also need to restart your R session in order to have the last available data.

The raw data is pulled from the Johns Hopkins University Center for Systems Science and Engineering (JHU CCSE) Coronavirus repository.

Open source

This dashboard and the code available on GitHub are open source so feel free to copy it, adapt it and share it as much as you want.

Accuracy

Please note that this dashboard has been built mainly for educational purposes. I update the dashboard as often as possible to keep it accurate. However, there is some uncertainty concerning the COVID-19 case numbers and the testing methods vary between countries so the figures on this dashboard may be slightly different compared to other sources. Currently, the maintainer of the dataset updates it on a daily basis, but updates may become less frequent in the future.

Publish your dashboard

If you want to share your dashboard, you can either:

Upload it on your website if you have one (and I strongly suggest you to create one if you do not already have one)
Publish it through RPubs (it is free and easy to publish your work directly from RStudio)

Conclusion

Thanks for reading.

I hope this article helped you to build your first Coronavirus dashboard in R. See these top R resources on Coronavirus if you need inspiration to enhance further your dashboard.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

I would be glad to mention a collection of Python resources if someone is willing to create a collection of resources about the Coronavirus made with this programming language. Feel free to contact me if this is the case.↩︎

How to do a t-test or ANOVA for more than one variable at once in R?

Thu, 19 Mar 2020 00:00:00 +0000

Introduction

As part of my teaching assistant position in a Belgian university, students often ask me for some help in their statistical analyses for their master’s thesis.

A frequent question is how to compare groups of patients in terms of several quantitative continuous variables. Most of us know that:

To compare two groups, a Student’s t-test should be used¹
To compare three groups or more, an ANOVA should be performed

These two tests are quite basic and have been extensively documented online and in statistical textbooks so the difficulty is not in how to perform these tests.

In the past, I used to do the analyses by following these 3 steps:

Draw boxplots illustrating the distributions by group (with the boxplot() function or thanks to the {esquisse} R Studio addin if I wanted to use the {ggplot2} package)
Perform a t-test or an ANOVA depending on the number of groups to compare (with the t.test() and oneway.test() functions for t-test and ANOVA, respectively)
Repeat steps 1 and 2 for each variable

This was feasible as long as there were only a couple of variables to test. Nonetheless, most students came to me asking to perform these kind of tests not on one or two variables, but on multiples variables. So when there were more than one variable to test, I quickly realized that I was wasting my time and that there must be a more efficient way to do the job.

Note: you must be very careful with the issue of multiple testing (also referred as multiplicity) which can arise when you perform multiple tests. In short, when a large number of statistical tests are performed, some will have \(p\)-values less than 0.05 purely by chance, even if all null hypotheses are in fact really true. This is known as multiplicity or multiple testing. You can tackle this problem by using the Bonferroni correction, among others. The Bonferroni correction is a simple method that allows many t-tests to be made while still assuring an overall confidence level is maintained. For this, instead of using the standard threshold of \(\alpha = 5\)% for the significance level, you can use \(\alpha = \frac{0.05}{m}\) where \(m\) is the number of t-tests. For example, if you perform 20 t-tests with a desired \(\alpha = 0.05\), the Bonferroni correction implies that you would reject the null hypothesis for each individual test when the \(p\)-value is smaller than \(\alpha = \frac{0.05}{20} = 0.0025\).

Note also that there is no universally accepted approach for dealing with the problem of multiple comparisons. Usually, you should choose a p-value adjustment measure familiar to your audience or in your field of study. The Bonferroni correction is easy to implement. It is however not appropriate if you have a very large number of tests to perform (imagine you want to do 10,000 t-tests, a p-value would have to be less than \(\frac{0.05}{10000} = 0.000005\) to be significant). A more powerful method is also to adjust the false discovery rate using the Benjamini-Hochberg or Holm procedure (McDonald 2014).

Another option is to use a multivariate ANOVA (MANOVA), if your independent variable has more than two levels. This is particularly useful when your dependent variables are correlated. Correlation between the dependent variables provides MANOVA the following advantages:

Identify patterns between several dependent variables: The independent variables can influence the relationship between dependent variables instead of influencing a single dependent variable.
Address the issue of multiple testing: with MANOVA, the error rate equals the significance level (with no p-value adjustment method needed).
Greater statistical power: When the dependent variables are correlated, MANOVA can identify effects that are too small for the ANOVA to detect.

Note that MANOVA is used if your independent variable has more than two levels. If your independent variable has only two levels, the multivariate equivalent of the t-test is Hotelling’s \(T^2\).

This article aims at presenting a way to perform multiple t-tests and ANOVA from a technical point of view (how to implement it in R). Discussion on which adjustment method to use or whether there is a more appropriate model to fit the data is beyond the scope of this article (so be sure to understand the implications of using the code below for your own analyses). Make sure also to test the assumptions of the ANOVA before interpreting results.

Perform multiple tests at once

I thus wrote a piece of code that automated the process, by drawing boxplots and performing the tests on several variables at once. Below is the code I used, illustrating the process with the iris dataset. The Species variable has 3 levels, so let’s remove one, and then draw a boxplot and apply a t-test on all 4 continuous variables at once. Note that the continuous variables that we would like to test are variables 1 to 4 in the iris dataset.

dat <- iris

# remove one level to have only two groups
dat <- subset(dat, Species != "setosa")
dat$Species <- factor(dat$Species)

# boxplots and t-tests for the 4 variables at once
for (i in 1:4) { # variables to compare are variables 1 to 4
  boxplot(dat[, i] ~ dat$Species, # draw boxplots by group
    ylab = names(dat[i]), # rename y-axis with variable's name
    xlab = "Species"
  )
  print(t.test(dat[, i] ~ dat$Species)) # print results of t-test
}

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -5.6292, df = 94.025, p-value = 1.866e-07
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.8819731 -0.4220269
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    5.936                    6.588

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -3.2058, df = 97.927, p-value = 0.001819
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.33028364 -0.07771636
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    2.770                    2.974

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -12.604, df = 95.57, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -1.49549 -1.08851
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    4.260                    5.552

## 
## 	Welch Two Sample t-test
## 
## data:  dat[, i] by dat$Species
## t = -14.625, df = 89.043, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group versicolor and group virginica is not equal to 0
## 95 percent confidence interval:
##  -0.7951002 -0.6048998
## sample estimates:
## mean in group versicolor  mean in group virginica 
##                    1.326                    2.026

As you can see, the above piece of code draws a boxplot and then prints results of the test for each continuous variable, all at once.

At some point in the past, I even wrote code to:

draw a boxplot
test for the equality of variances (thanks to the Levene’s test)
depending on whether the variances were equal or unequal, the appropriate test was applied: the Welch test if the variances were unequal and the Student’s t-test in the case the variances were equal (see more details about the different versions of the t-test for two samples)
apply steps 1 to 3 for all continuous variables at once

I had a similar code for ANOVA in case I needed to compare more than two groups.

The code was doing the job relatively well. Indeed, thanks to this code I was able to test several variables in an automated way in the sense that it compared groups for all variables at once.

The only thing I had to change from one project to another is that I needed to modify the name of the grouping variable and the numbering of the continuous variables to test (Species and 1:4 in the above code).

Concise and easily interpretable results

T-test

Although it was working quite well and applicable to different projects with only minor changes, I was still unsatisfied with another point.

Someone who is proficient in statistics and R can read and interpret the output of a t-test without any difficulty. However, as you may have noticed with your own statistical projects, most people do not know what to look for in the results and are sometimes a bit confused when they see so many graphs, code, output, results and numeric values in a document. They are quite easily overwhelmed by this mass of information and unable to extract the key message.

With my old R routine, the time I was saving by automating the process of t-tests and ANOVA was (partially) lost when I had to explain R outputs to my students so that they could interpret the results correctly. Although most of the time it simply boiled down to pointing out what to look for in the outputs (i.e., p-values), I was still losing quite a lot of time because these outputs were, in my opinion, too detailed for most real-life applications and for students in introductory classes. In other words, too much information seemed to be confusing for many people so I was still not convinced that it was the most optimal way to share statistical results to nonscientists.

Of course, they came to me for statistical advices, so they expected to have these results and I needed to give them answers to their questions and hypotheses. Nonetheless, I wanted to find a better way to communicate these results to this type of audience, with the minimum of information required to arrive at a conclusion. No more and no less than that.

After a long time spent online trying to figure out a way to present results in a more concise and readable way, I discovered the {ggpubr} package. This package allows to indicate the test used and the p-value of the test directly on a ggplot2-based graph. It also facilitates the creation of publication-ready plots for non-advanced statistical audiences.

After many refinements and modifications of the initial code (available in this article), I finally came up with a rather stable and robust process to perform t-tests and ANOVA for more than one variable at once, and more importantly, make the results concise and easily readable by anyone (statisticians or not).

A graph is worth a thousand words, so here are the exact same tests than in the previous section, but this time with my new R routine:

library(ggpubr)

# Edit from here #
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method <- "t.test" # one of "wilcox.test" or "t.test"
paired <- FALSE # if paired make sure that in the dataframe you have first all individuals at T1, then all individuals again at T2
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    if (paired == TRUE) {
      p <- ggpaired(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]), line.color = "gray", line.size = 0.4,
        palette = "npg",
        legend = "none",
        xlab = colnames(dat[j]),
        ylab = colnames(dat[i]),
        add = "jitter"
      )
    } else {
      p <- ggboxplot(dat,
        x = colnames(dat[j]), y = colnames(dat[i]),
        color = colnames(dat[j]),
        palette = "npg",
        legend = "none",
        add = "jitter"
      )
    }
    #  Add p-value
    print(p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
      method = method,
      paired = paired,
      # group.by = NULL,
      ref.group = NULL
    ))
  }
}

As you can see from the graphs above, only the most important information is presented for each variable:

a visual comparison of the groups thanks to boxplots
the name of the statistical test
the p-value of the test

Of course, experts may be interested in more advanced results. However, this simple yet complete graph, which includes the name of the test and the p-value, gives all the necessary information to answer the question: “Are the groups different?”.

In my experience, I have noticed that students and professionals (especially those from a less scientific background) understand way better these results than the ones presented in the previous section.

The only lines of code that need to be modified for your own project is the name of the grouping variable (Species in the above code), the names of the variables you want to test (Sepal.Length, Sepal.Width, etc.),² whether you want to apply a t-test (t.test) or Wilcoxon test (wilcox.test) and whether the samples are paired or not (FALSE if samples are independent, TRUE if they are paired).

Based on these graphs, it is easy, even for non-experts, to interpret the results and conclude that the versicolor and virginica species are significantly different in terms of all 4 variables (since all p-values \(< \frac{0.05}{4} = 0.0125\) (remind that the Bonferroni correction is applied to avoid the issue of multiple testing, so we divide the usual \(\alpha\) level by 4 because there are 4 t-tests)).

Additional p-value adjustment methods

If you would like to use another p-value adjustment method, you can use the p.adjust() function. Below are the raw p-values found above, together with p-values derived from the main adjustment methods (presented in a dataframe):

raw_pvalue <- numeric(length = length(1:4))
for (i in (1:4)) {
  raw_pvalue[i] <- t.test(dat[, i] ~ dat$Species,
    alternative = "two.sided"
  )$p.value
}

df <- data.frame(
  Variable = names(dat[, 1:4]),
  raw_pvalue = round(raw_pvalue, 3)
)

df$Bonferroni <-
  p.adjust(df$raw_pvalue,
    method = "bonferroni"
  )
df$BH <-
  p.adjust(df$raw_pvalue,
    method = "BH"
  )
df$Holm <-
  p.adjust(df$raw_pvalue,
    method = "holm"
  )
df$Hochberg <-
  p.adjust(df$raw_pvalue,
    method = "hochberg"
  )
df$Hommel <-
  p.adjust(df$raw_pvalue,
    method = "hommel"
  )
df$BY <-
  round(p.adjust(df$raw_pvalue,
    method = "BY"
  ), 3)
df

##       Variable raw_pvalue Bonferroni    BH  Holm Hochberg Hommel    BY
## 1 Sepal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 2  Sepal.Width      0.002      0.008 0.002 0.002    0.002  0.002 0.004
## 3 Petal.Length      0.000      0.000 0.000 0.000    0.000  0.000 0.000
## 4  Petal.Width      0.000      0.000 0.000 0.000    0.000  0.000 0.000

Regardless of the p-value adjustment method, the two species are different for all 4 variables. Note that the adjustment method should be chosen before looking at the results to avoid choosing the method based on the results.

Below another function that allows to perform multiple Student’s t-tests or Wilcoxon tests at once and choose the p-value adjustment method. The function also allows to specify whether samples are paired or unpaired and whether the variances are assumed to be equal or not. (The code has been adapted from Mark White’s article.)

t_table <- function(data, dvs, iv,
                    var_equal = TRUE,
                    p_adj = "none",
                    alpha = 0.05,
                    paired = FALSE,
                    wilcoxon = FALSE) {
  if (!inherits(data, "data.frame")) {
    stop("data must be a data.frame")
  }

  if (!all(c(dvs, iv) %in% names(data))) {
    stop("at least one column given in dvs and iv are not in the data")
  }

  if (!all(sapply(data[, dvs], is.numeric))) {
    stop("all dvs must be numeric")
  }

  if (length(unique(na.omit(data[[iv]]))) != 2) {
    stop("independent variable must only have two unique values")
  }

  out <- lapply(dvs, function(x) {
    if (paired == FALSE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]], var.equal = var_equal)
    } else if (paired == FALSE & wilcoxon == TRUE) {
      tres <- wilcox.test(data[[x]] ~ data[[iv]])
    } else if (paired == TRUE & wilcoxon == FALSE) {
      tres <- t.test(data[[x]] ~ data[[iv]],
        var.equal = var_equal,
        paired = TRUE
      )
    } else {
      tres <- wilcox.test(data[[x]] ~ data[[iv]],
        paired = TRUE
      )
    }

    c(
      p_value = tres$p.value
    )
  })

  out <- as.data.frame(do.call(rbind, out))
  out <- cbind(variable = dvs, out)
  names(out) <- gsub("[^0-9A-Za-z_]", "", names(out))

  out$p_value <- p.adjust(out$p_value, p_adj)
  out$conclusion <- ifelse(out$p_value < alpha,
    paste0("Reject H0 at ", alpha * 100, "%"),
    paste0("Do not reject H0 at ", alpha * 100, "%")
  )
  out$p_value <- ifelse(out$p_value < 0.001,
    "<0.001",
    round(out$p_value, 3)
  )

  return(out)
}

Applied to our dataset, with no adjustment method for the p-values:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

And with the Holm (1979) adjustment method:

result <- t_table(
  data = dat,
  c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"),
  "Species",
  p_adj = "holm"
)

result

##       variable p_value      conclusion
## 1 Sepal.Length  <0.001 Reject H0 at 5%
## 2  Sepal.Width   0.002 Reject H0 at 5%
## 3 Petal.Length  <0.001 Reject H0 at 5%
## 4  Petal.Width  <0.001 Reject H0 at 5%

Again, with the Holm’s adjustment method, we conclude that, at the 5% significance level, the two species are significantly different from each other in terms of all 4 variables.

ANOVA

Below the same process with an ANOVA. Note that we reload the dataset iris to include all three Species this time:

dat <- iris

# Edit from here
x <- which(names(dat) == "Species") # name of grouping variable
y <- which(names(dat) == "Sepal.Length" # names of variables to test
| names(dat) == "Sepal.Width" |
  names(dat) == "Petal.Length" |
  names(dat) == "Petal.Width")
method1 <- "anova" # one of "anova" or "kruskal.test"
method2 <- "t.test" # one of "wilcox.test" or "t.test"
my_comparisons <- list(c("setosa", "versicolor"), c("setosa", "virginica"), c("versicolor", "virginica")) # comparisons for post-hoc tests
# Edit until here


# Edit at your own risk
for (i in y) {
  for (j in x) {
    p <- ggboxplot(dat,
      x = colnames(dat[j]), y = colnames(dat[i]),
      color = colnames(dat[j]),
      legend = "none",
      palette = "npg",
      add = "jitter"
    )
    print(
      p + stat_compare_means(aes(label = paste0(after_stat(method), ", p-value = ", after_stat(p.format))),
        method = method1, label.y = max(dat[, i], na.rm = TRUE)
      )
      + stat_compare_means(comparisons = my_comparisons, method = method2, label = "p.format") # remove if p-value of ANOVA or Kruskal-Wallis test >= alpha
    )
  }
}

Like the improved routine for the t-test, I have noticed that students and non-expert professionals understand ANOVA results presented this way much more easily compared to the default R outputs.

With one graph for each variable, it is easy to see that all species are different from each other in terms of all 4 variables.³

If you want to apply the same automated process to your data, you will need to modify the name of the grouping variable (Species), the names of the variables you want to test (Sepal.Length, etc.), whether you want to perform an ANOVA (anova) or Kruskal-Wallis test (kruskal.test) and finally specify the comparisons for the post-hoc tests.⁴

To go even further

As we have seen, these two improved R routines allow to:

Perform t-tests and ANOVA on a small or large number of variables with only minor changes to the code. I basically only have to replace the variable names and the name of the test I want to use. It takes almost the same time to test one or several variables so it is quite an improvement compared to testing one variable at a time.
Share test results in a much proper and cleaner way. This is possible thanks to a graph showing the observations by group and the p-value of the appropriate test included directly on the graph. This is particularly important when communicating results to a wider audience or to people from diverse backgrounds.

However, like most of my R routines, these two pieces of code are still a work in progress. Below are some additional features I have been thinking of and which could be added in the future to make the process of comparing two or more groups even more optimal:

Add the possibility to select variables by their numbering in the dataframe. For the moment it is only possible to do it via their names. This will allow to automate the process even further because instead of typing all variable names one by one, we could simply type 4:25 (to test variables 4 to 25 for instance).
Add the possibility to choose a p-value adjustment method. Currently, raw p-values are displayed in the graphs and I manually adjust them afterwards or adjust the \(\alpha\).
When comparing more than two groups, it is only possible to apply an ANOVA or Kruskal-Wallis test at the moment. A major improvement would be to add the possibility to perform a repeated measures ANOVA (i.e., an ANOVA when the samples are dependent). It is currently already possible to do a t-test with two paired samples, but it is not yet possible to do the same with more than two groups.
Another less important (yet still nice) feature when comparing more than 2 groups would be to automatically apply post-hoc tests only in the case where the null hypothesis of the ANOVA or Kruskal-Wallis test is rejected (so when there is at least one group different from the others, because if the null hypothesis of equal groups is not rejected we do not apply a post-hoc test). At the present time, I manually add or remove the code that displays the p-values of post-hoc tests depending on the global p-value of the ANOVA or Kruskal-Wallis test.

I will try to add these features in the future, or I would be glad to help if the author of the {ggpubr} package needs help in including these features (I hope he will see this article!).

Last but not least, the following packages may be of interest to some readers:

If you want to report statistical results on a graph, I advise you to check the {ggstatsplot} package and in particular the ggbetweenstats() and ggwithinstats() functions. These functions allow to compare a continuous variable across multiple groups or conditions (for both independent and paired samples). Two advantages of the functions is that:
- it is very easy to switch from parametric to nonparametric tests and
- it automatically runs an ANOVA or t-test depending on the number of groups to compare

Note that many different statistical results are displayed on the graph, not only the name of the test and the p-value so a bit of simplicity and clarity is lost for more precision. However, it is still very convenient to be able to include tests results on a graph in order to combine the advantages of a visualization and a sound statistical analysis. Something that I still need to figure out is how to run the code on several variables at once.

The {compareGroups} package also provides a nice way to compare groups. It comes with a really complete Shiny app, available with:

# install.packages("compareGroups")
library(compareGroups)
cGroupsWUI()

Update with the `{ggstatsplot}` package

Several months after having written this article, I finally found a way to plot and run analyses on several variables at once with the package {ggstatsplot} (Patil 2021). This was the main feature I was missing and which prevented me from using it more often.

Although I still find that too much statistical details are displayed (in particular for non experts), I still believe the ggbetweenstats() and ggwithinstats() functions are worth mentioning in this article. I actually now use those two functions almost as often as my previous routines because:

I do not have to care about the number of groups to compare, the functions automatically choose the appropriate test according to the number of groups (ANOVA for 3 groups or more, and t-test for 2 groups)
I can select variables based on their column numbering, and not based on their names anymore (which prevents me from writing those variable names manually)
When comparing 3 or more groups (so for ANOVA, Kruskal-Wallis, repeated measure ANOVA or Friedman), \(p\)-values of the post-hoc tests within each dependent variable are by default the adjusted \(p\)-values (Holm is the default but many adjustment methods are available)
It is possible to compare both independent and paired samples, no matter the number of groups (remember that with the ggpubr package I could only do paired samples for two samples, not for 3 samples)
They allow to easily switch between the parametric and nonparametric version
All this in a more concise manner using the {purrr} package

For those of you who are interested, below my updated R routine which include these functions and applied this time on the penguins dataset.

library(palmerpenguins)

dat <- penguins
str(dat)

## tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
##  $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
##  $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
##  $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
##  $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
##  $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
##  $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

We illustrate the routine for two groups with the variables sex (two factors) as independent variable, and the 4 quantitative continuous variables bill_length_mm, bill_depth_mm, flipper_length_mm and body_mass_g as dependent variables:

library(ggstatsplot)
library(tibble)

# Comparison between sexes

# edit from here
x <- "sex"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

We now illustrate the routine for 3 groups or more with the variable species (three factors) as independent variable, and the 4 same dependent variables:

# Comparison between species

# edit from here
x <- "species"
cols <- 3:6 # the 4 continuous dependent variables
type <- "parametric" # given the large number of observations, we use the parametric version
paired <- FALSE # FALSE for independent samples, TRUE for paired samples
# edit until here

# edit at your own risk
plotlist <-
  purrr::pmap(
    .l = list(
      data = list(as_tibble(dat)),
      x = x,
      y = as.list(colnames(dat)[cols]),
      plot.type = "box", # for boxplot
      type = type, # parametric or nonparametric
      pairwise.comparisons = TRUE, # to run post-hoc tests if more than 2 groups
      pairwise.display = "significant", # show only significant differences
      bf.message = FALSE, # remove message about Bayes Factor
      centrality.plotting = FALSE # remove central measure
    ),
    .f = ifelse(paired, # automatically use ggwithinstats if paired samples, ggbetweenstats otherwise
      ggstatsplot::ggwithinstats,
      ggstatsplot::ggbetweenstats
    ),
    violin.args = list(width = 0, linewidth = 0) # remove violin plots and keep only boxplots
  )

# print all plots together with statistical results
for (i in 1:length(plotlist)) {
  print(plotlist[[i]])
}

As you can see, I only have to specify:

the name of the grouping variable (sex and species),
the number of the dependent variables (variables 3 to 6 in the dataset),
whether I want to use the parametric or nonparametric version and
whether samples are independent (paired = FALSE) or paired (paired = TRUE).

Everything else is automated—the outputs show a graphical representation of what we are comparing, together with the details of the statistical analyses in the subtitle of the plot (the \(p\)-value among others).

Note that the code shown above is actually the same if I want to compare 2 groups or more than 2 groups. I wrote twice the same code (once for 2 groups and once again for 3 groups) for illustrative purposes only, but they are the same and should be treated as one for your projects.

I must admit I am quite satisfied with this routine, now that:

I can automate it on many variables at once and I do not need to write the variable names manually anymore,
at the same time, I can choose the appropriate test among all the available ones (depending on the number of groups, whether they are paired or not, and whether I want to use the parametric or nonparametric version).

Nonetheless, I must also admit that I am still not satisfied with the level of details of the statistical results. As already mentioned, many students get confused and get lost in front of so much information (except the \(p\)-value and the number of observations, most of the details are rather obscure to them because they are not covered in introductory statistic classes).

I saved time thanks to all improvements in comparison to my previous routine, but I definitely lose time when I have to point out to them what they should look for. After discussing with other professors, I noticed that they have the same problem.

For the moment, you can only print all results or none. I have opened an issue kindly requesting to add the possibility to display only a summary (with the \(p\)-value and the name of the test for instance).⁵ I will update again this article if the maintainer of the package includes this feature in the future. So stay tuned!

Conclusion

Thanks for reading.

I hope this article will help you to perform t-tests and ANOVA for multiple variables at once and make the results more easily readable and interpretable by non-scientists. Learn more about the t-test to compare two groups, or the ANOVA to compare 3 groups or more.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 65–70.

McDonald, JH. 2014. “Multiple Tests.” Handbook of Biological Statistics. 3rd Ed Baltimore, Maryland: Sparky House Publishing, 233–36.

Patil, Indrajeet. 2021. “Visualizations with statistical details: The ’ggstatsplot’ approach.” Journal of Open Source Software 6 (61): 3167. https://doi.org/10.21105/joss.03167.

In theory, an ANOVA can also be used to compare two groups as it will give the same results compared to a Student’s t-test, but in practice we use the Student’s t-test to compare two groups and the ANOVA to compare three groups or more.↩︎
Do not forget to separate the variables you want to test with |.↩︎
Do not forget to adjust the \(p\)-values or the significance level \(\alpha\). If you use the Bonferroni correction, the adjusted \(\alpha\) is simply the desired \(\alpha\) level divided by the number of comparisons.↩︎
Post-hoc test is only the name used to refer to a specific type of statistical tests. Post-hoc test includes, among others, the Tukey HSD test, the Bonferroni correction, Dunnett’s test. Even if an ANOVA or a Kruskal-Wallis test can determine whether there is at least one group that is different from the others, it does not allow us to conclude which are different from each other. For this purpose, there are post-hoc tests that compare all groups two by two to determine which ones are different, after adjusting for multiple comparisons. Concretely, post-hoc tests are performed to each possible pair of groups after an ANOVA or a Kruskal-Wallis test has shown that there is at least one group which is different (hence “post” in the name of this type of test). The null and alternative hypotheses and the interpretations of these tests are similar to a Student’s t-test for two samples.↩︎
I am open to contribute to the package if I can help!↩︎

Top 100 R resources on COVID-19 Coronavirus

Thu, 12 Mar 2020 00:00:00 +0000

Warning: Some links or resources may have been moved or deleted, and are thus not accessible anymore. If you are the author and would like to update the URL, feel free to contact me so I can update the link.

The Coronavirus is a serious concern around the globe. With its expansion, there are also more and more online resources about it. This article presents a selection of the best R resources on the COVID-19 virus.

This list is by no means exhaustive. I am not aware of all R resources available online about the Coronavirus, so please feel free to let me know in the comments or by contacting me if you believe that another resource (R package, Shiny app, R code, blog posts, datasets, etc.) deserves to be on this list.

R Shiny apps and dashboards

Coronavirus tracker

Developed by John Coene, this Shiny app tracks the spread of the Coronavirus, based on three data sources (John Hopkins, Weixin and DXY Data). The Shiny app, built with shinyMobile (which makes it responsive on different screen sizes), presents in a really nice way the number of deaths, confirmed, suspected and recovered cases by time and region.

The code is available on GitHub.

Coronavirus dashboard from the `{coronavirus} package`

Developed by the author of the {coronavirus} package, this dashboard provides an overview of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) epidemic. The data and dashboard are refreshed on a daily basis.

The code is available on GitHub.

From this dashboard, I created another dashboard specific to Belgium. Feel free to use the code available on GitHub to build one specific to your country. See more details in this article.

Visualization of Covid-19 Cases

Developed by Nico Hahn, this Shiny app uses leaflet, plotly and the data from Johns Hopkins University to visualize the outbreak of the novel Coronavirus and shows data for the entire world or singular countries.

The code is available on GitHub.

Modeling COVID-19 Spread vs Healthcare Capacity

Developed by Dr. Alison Hill, this Shiny app uses an epidemiological model based on the classic SEIR model to describe the spread and clinical progression of COVID-19. It includes different clinical trajectories of infection, interventions to reduce transmission, and comparisons to healthcare capacity.

The code is available on GitHub.

COVID-19 Data Visualization Platform

Developed by Shubhram Pandey, this Shiny app provides a clear visualization of Covid19 impact all over the world and it also provides a sentiment analysis using natural language processing from Twitter.

The code is available on GitHub.

Coronavirus 10-day forecast

Developed by the Spatial Ecology and Evolution Lab, this Shiny app gives a ten-day forecast, by country, on likely numbers of Coronavirus cases and gives citizens a sense of how fast this epidemic is progressing.

See a detailed explanation of the app and how to read it in this blog post. The code is available on GitHub.

Coronavirus (COVID-19) across the world

Developed by Anisa Dhana in collaboration with datascience+, this Shiny app monitors the spread of COVID-19 across the world via a map visualization of the confirmed cases and some graphs on the growth of the virus.

The dataset used is from Johns Hopkins CSSE and part of the code is available in this blog post.

COVID-19 outbreak

Developed by Dr. Thibaut Fabacher in collaboration with the department of Public Health of the Strasbourg University Hospital and the Laboratory of Biostatistics and Medical Informatics of the Strasbourg Medicine Faculty, this Shiny app shows an interactive map for global monitoring of the infection. It focuses on the evolution of the number of cases per country and for a given period in terms of incidence and prevalence.

The code is available on GitHub and this blog post discusses it in more detail.

Flatten the Curve

Developed by Tinu Schneider, this Shiny app illustrates, in an interactive way, the different scenarios behind the #FlattenTheCurve message.

The app has been built upon Michael Höhle’s article and the code is available on GitHub.

Explore the spread of Covid-19

Developed by Joachim Gassen, this Shiny app allows you to visualize confirmed, recovered cases and reported deaths for several countries via one summary graph.

The Shiny app is based on data from:

Johns Hopkins University CSSE team on the spread of the SARS-CoV-2 virus
ACAPS governmental measures database
World Bank

This blog post explains the Shiny app in further details and in particular the {tidycovid19} R package behind it.

Governments and COVID-19

Developed by Sebastian Engel-Wolf, this Shiny app presents in a elegant way the following measurements:

Maximum time of exponential growth in a row
Days to double infections
Exponential growth today
Confirmed cases
Deaths
Population
Confirmed cases on 100,000 inhabitants
Mortality rate

The code is available on GitHub and this article explains it in further details.

Simulating COVID-19 Epidemic in Togo - West Africa

Developed by Dr. Kankoé Sallah, this Shiny app uses SEIR metapopulation model with mobility between catchment areas to describe country level spread of COVID-19 and the impact of interventions in Togo, West Africa.

Covid-19 Prediction

Developed by Manuel Oviedo and Manuel Febrero (Modestya research group of the University of Santiago de Compostela), this Shiny app predicts the growth rate at 5-day horizon using the evolution during the last 15 days of growth rate. Three functional regression models are fitted and re-estimated when new data is available. The app also shows an interactive plot and table for the expected number of accumulated cases and new daily cases to each horizon (for confirmed and deaths responses) by country (from Johns Hopkins CSSE) and Spanish region (from ISCII).

See an explanation of the methodology in the About tab.

Covid-19 Dashboard

Developed by Philippe De Brouwer, this dashboard displays several key measures regarding the outbreak of the virus (by country or for all countries combined), together with some forecasts, a world map and other interactive plots.

Healthcare worker deaths from novel Coronavirus (COVID-19) in the US

Developed by Jonathan Gross, this Shiny app visualizes healthcare worker deaths from Coronavirus (COVID-19) in the US reported in the news. It is updated daily and the code is available on GitHub.

Covid-19 Hospitalizations in Belgium

Developed by Jean-Michel Bodart, this dashboard provides an overview of the evolution of Covid-19-related hospitalizations in Belgium, by region and province.

The code is available on GitHub.

COVIDMINDER: Where you live matters!

Developed by The Rensselaer Institute for Data Exploration and Applications, this Shiny app reveals the regional disparities in outcomes, determinants and medications (e.g., mortality rates, test cases, diabetes, and hospital beds) across United States, with a special focus on New York.

This blog post explains the Shiny app in more detail and the code is available on GitHub.

COVID-19 Canada Data Explorer Tool

Developed by Petr Baranovskiy from Data Enthusiast’s Blog, this Shiny app processes the official dataset available from the Government of Canada and shows several indicators related to the SARS-CoV-2 epidemic in Canada.

This blog post details the application in further detail.

Philippine COVID-19 Case Forecasting

Developed by Jamal Kay Rogers and Yvonne Grace Arandela, this Shiny app provides a 5-day forecast of confirmed positive, deaths, and recoveries of COVID-19 cases in The Philippines.

The app also delivers graphical plots of a 10-day forecast and the daily and cumulated cases of COVID-19 in The Philippines. The data source is Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE).

COVID-19 Case & Death Report Number Corrector

Developed by Matt Maciejewski, this Shiny app focuses on the correction of underreported Covid-19 case and death counts using a reference country based on Lachmann et al. (2020), and via a multiplicative estimator for total deaths and cases. The estimator will be turned into a posterior prediction once data becomes available.

The app is explained in more detail in this article and the code of the Shiny app can be found on GitHub.

Covid-19: the SPQEIR model

Developed by several researchers from the Luxembourg Centre for Systems Biomedicine (LCSB) of the University of Luxembourg, the KU Leuven and the UGent, this Shiny app uses the new SPQEIR model to simulate the impact of various suppression strategies (social distancing, lockdown, protection, etc.) on the development of COVID-19.

The Belgian Covid Cases Tracker

Developed by Patrick Sciortino, this Shiny app aims at estimating the curve of true Covid-19 cases based on the idea that a hospitalized case in time t informs us about an infection that took place a few days earlier.

COVID-19 monitor

Developed by Trafford Data Lab, this Shiny app visualizes daily confirmed Coronavirus cases and deaths in the UK.

It uses the following data sources:

The code can be found on GitHub.

COVID-19 Bulletin Board

Developed by Wei Su, this dashboard shows real-time visualization of the COVID-19 epidemic in Japan. It mainly shows various indicators including, but not limited to, PCR test, positive confirmed, hospital discharge and death, as well as trends in each prefecture in Japan. There are also a variety of charts such as cluster network, new confirmed cases in log scale for users’ reference.

The dashboard is based on this Japanese version (developed by the same author). The code can be found on GitHub.

Covid-19 Statistics Displayer

Created by Carl Sansfaçon, this Wordpress plugin associates R {ggplot2} graphics with ARIMA forecast and PHP coding to show evolution of the confirmed, death and recovered cases in different countries, states/provinces and US cities.

It uses data from the COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. The plugin can be installed as a Wordpress plugin.

CoronaMapper

Supported by OxyLabs, created by Paolo Montemurro and Peter, this visualization displays the four days average growth indicator, which clearly shows how a certain statistic of the virus is evolving over time, filtering out the noise.

The website receives data each hour from several official data-sources, and visualizes the historical evolution of COVID19 in an intuitive & interactive way.

CoronaDash

Developed by Peter Laurinec, this Shiny app provides various data mining and visualization techniques for comparing countries’ COVID-19 data statistics as:

extrapolating total confirmed cases by exponential smoothing model,
trajectories of cases/deaths spread,
multidimensional clustering of countries’ data/ statistics - with dendrogram and table of clusters averages,
aggregated views for the whole world,
hierarchical clustering of countries’ trajectories based on DTW distance and preprocessing by SMA (+ normalization), for fast comparison of a large number of countries’ COVID-19 magnitudes and trends.

This blog post explained in further detail the last point of the above list. The code of the app is available on GitHub.

Covidfrance

Developed by Guillaume Pressiat, this Shiny app illustrates the evolution of number of hospitalizations, intensive care units, recoveries and deaths in France (by department).

This blog post presents the application and the code of the app is available on GitHub.

The Impact of COVID-19 on Mobility

Developed by Dimiter Toshkov, this Shiny app shows relative changes in mobility (visits and length of stay) for a specific category of places within a country compared to a baseline. The baseline is computed as the median for the day of the week during the 5-week period between January 3 and February 6, 2020. Hence, the plot shows how mobility has changed relative to the situation in the same country in the beginning of the year.

The author also developed a version for the US states.

Data is from Google Community Mobility Reports.

Coronavirus Analysis Platform

Developed by Khaled M Alqahtani, this Shiny app offers a powerful analysis tools, including:

Descriptive analysis
Growth rate and curve flatting
Cumulative forecast
Daily cases forecasting containing 16 different models
Newspaper analysis
TV analysis

Data is from Johns Hopkins CSSE and some GDELT APIs.

COVID-19 Tracker

Developed by Dr. Magda Bucholc from Ulster University, this dashboard reports cases at the local government district in Northern Ireland and county level across the island of Ireland, providing gender and age breakdowns of reported cases, growth rates, and statistics per 100,000 of the population; it also has daily mobility data from Google and Apple.

More information about this dashboard can be found here.

COVID-19 Overview

Developed by Fabian Dablander, Alexandra Rusu, Marcel Schreiner, and Aleksandar Tomasevic as part of the Science versus Corona project, this dashboard provides an overview of confirmed cases, deaths, and measures that countries have taken to curb the spread of the virus.

For a more information about this dashboard, see this blog post.

COVID-19 Exit Strategies

Developed as part of the Science versus Corona project, this Shiny app compares several alternative exit strategies that either aim to keep the number of infections as low as possible (e.g., contact tracing), or that aim to develop herd immunity without exceeding health care capacity.

The Shiny app is based on the stochastic individual-based SEIR model developed by de Vlas and Coffeng (2020). This post explains the Shiny app in more detail.

Corona Virus Statistics : COVID19

Developed by Dr. Mohammed N. Alenezi, this dashboard displays the latest information about Coronavirus for Kuwait, GCC, and the world. The dashboard contains a collection of different plots and models presented under 8 tabs.

Covid19 Data

Developed by Dhafer Malouche, this dashboard presents statistics for more than 200 countries and regions, with the estimation of the reproduction number \(R(t)\) in the previous 60 days and a Covid19 country classification.

WHO COVID-19 Explorer

Developed by the World Health Organization (WHO), this Shiny app aims to provide frequently updated data visualizations about confirmed cases and deaths at the global, regional and country level.

COVID-19 Scenario Analysis Tool

Developed by the MRC Centre for Global Infectious Disease Analysis (Imperial College London), this dashboard illustrates the epidemic trajectory, the healthcare demand and the \(R_t\) & \(R_{eff}\) measures for many countries over time in interactive plots.

The dashboard uses the squire R package, among others.

Covid-19 track

Developed by Redzuan, this Shiny dashboard analyses, combines and harmonizes the latest Covid-19 data into progress timeline, comparative analysis on various areas, mapping, latest news, forecasting using various models, prediction on Herd Immunity target, summary table, downloadable fact sheet and other features.

R packages

`{nCov2019}`

The {nCov2019} package gives you access to epidemiological data on the Coronavirus outbreak.¹ The package gives real-time statistics, includes historical data and a Shiny app. The vignette explains the main functions and possibilities of the package.

Furthermore, the authors of the package also developed a website with interactive plots and time-series forecasts, which could be useful in informing the public and studying how the virus spread in populous countries.

`{coronavirus}`

Developed by Rami Krispin, the {coronavirus} package provides a tidy format dataset of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) epidemic. Pulled from the dataset of John Hopkins, the R package gives a daily summary of the Coronavirus cases by state/province. The data set contains various variables such as confirmed cases, death, and recovered across different countries and states.

More details are available here, a csv format of the package dataset is available here and a summary dashboard is available here.

`{tidycovid19}`

Developed by Joachim Gassen, the {tidycovid19} package allows you to download, tidy and visualize Covid-19 related data (including data on governmental measures) directly from authoritative sources. It also provides a flexible function and an accompanying Shiny app to visualize the spreading of the virus.

The package is available on GitHub and these blog posts here and here explain it in more detail.

R packages from R Epidemics Consortium

These R packages from R Epidemics Consortium allows you to find the most advanced tools used by professional epidemiologists and experts in the domain of analyzing disease outbreaks.

`{covdata}`

Published by Prof. Kieran Healy, the {covdata} package is a R package providing COVID-19 case data from multiple sources:

National level data from the European Centers for Disease Control
State-level data for the United States from the COVID Tracking Project
State-level and county-level data for the United States from the New York Times
Data from the US Centers for Disease Control’s Coronavirus Disease 2019 (COVID-19)-Associated Hospitalization Surveillance Network (COVID-NET)
Data from Apple on relative trends in mobility in cities and countries since mid-January of 2020, based on usage of their Maps application
Data from Google on relative trends in mobility in regions and countries since mid-January of 2020, based on location and activity information

The code is available on GitHub.

`{covid19italy}`

The covid19italy R package provides a tidy format dataset of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) pandemic outbreak in Italy. The package includes the following three datasets:

italy_total: daily summary of the outbreak on the national level
italy_region: daily summary of the outbreak on the region level
italy_province: daily summary of the outbreak on the province level

More information about the package datasets available in this vignette, this blog post, and this supporting dashboard.

Data source: Italy Department of Civil Protection

`{COVID19}`

The goal of COVID-19 Data Hub is to provide the research community with a unified data hub by collecting worldwide fine-grained case data, merged with exogenous variables helpful for a better understanding of COVID-19. Featured by the University of Milano and funded by the Institute for Data Valorization IVADO, Canada.

The package collects COVID-19 data across governmental sources, includes policy measures from Oxford COVID-19 Government Response Tracker, and extends the dataset via an interface to World Bank Open Data, Google Mobility Reports and Apple Mobility Reports.

The package is available on CRAN, it is 100% open source and external contributors are welcomed to join.

`{COVOID}`

COVOID (for COVID-19 Open-source Infection Dynamics project) is a R package for modelling COVID-19 and other infectious diseases using deterministic compartmental models (DCMs).

It contains a built-in Shiny app enabling easy use and demonstration of key concepts to those without R programming backgrounds, along with an expanding API for simulating and estimating homogeneous and age-structured SIR, SEIR and extended models. In particular COVOID allows the simultaneous simulation of age specific (e.g. school closures) and general interventions over varying time intervals.

The code is available on GitHub.

`{cdccovidview}`

Published by Bob Rudis, the {cdccovidview} package can be used to work with the U.S. CDC’s New COVID-19 Trackers: COVIDView and COVID-NET.

`{babsim.hospital}`

Published by several researchers from TH Köln, the {babsim.hospital} package implements a discrete-event simulation model for a hospital resource planning problem. It can be used by health departments to forecast demand for intensive care beds, ventilators, and staff resources.

The team also developed a Shiny app which predicts COVID-19 ICU bed resources in hospitals. The app is available in English and German.

`{EpiLPS}`

Published by several researchers working in the field of epidemiology, the package shows how to smooth epidemic curves and estimate the time-varying reproduction number in a flexible way.

More information can be found in:

the research paper,
the accompanying website, and
this summary written by one of the authors.

R code and blog posts

Analyzing COVID-19 outbreak data with R

Written by Tim Churches, these two articles (part 1 and part 2) explore the R tools and packages that might be used to analyze the COVID-19 data. In particular, the author considers when the pandemic will subside in China, and then turns the analysis on Japan, South Korea, Italy and Iran. He also shows improvements on the cumulative incidence plots that are so common. Moreover, he presents R code to analyze how contagious is the Coronavirus thanks to the classic SIR (Susceptible-Infectious-Recovered) compartmental model of communicable disease outbreaks.²

The code is available on GitHub (part 1 and part 2).

Part 1 is actually based on another shorter blog post by Prof. Dr. Holger K. von Jouanne-Diedrich from Learning Machines. Read his article for a more concise analysis on how to model the outbreak of the Coronavirus and discover how contagious it is. Note that I have personally written an article analyzing COVID-19 in Belgium based on articles from these two authors.

More recently, Tim Churches published a series of other interesting articles:

Modelling the effect of various public health interventions on the local epidemic spread of COVID-19 infection using stochastic individual compartmental models (ICMs) implemented by the {EpiModel} package for R:
- Part 1 (code here)
- Part 2 (code here)
The use of simulations to explore the effects of various interventions on COVID-19 spread
COVID-19 epidemiology with R: in this blog post, the author, using relatively early and partial US data, separates out inbound from community cases, and predicts the next few weeks of incident numbers. He also highlights several R functions to analyze a disease outbreak.
We can “shrink” the COVID-19 curve, rather than just flatten it (in collaboration with Louisa Jorm)

COVID-19 Data Analysis with `{tidyverse}` and `{ggplot2}`

Dr. Yanchang Zhao from RDataMining published a data analysis around the Coronavirus with the {tidyverse} and {ggplot2} packages, for China and world wide.

Both documents are a mix of data cleaning, data processing and visualizations of the confirmed/cured cases and death rates across countries or regions.

COVID-19 cumulative observed case fatality rate over time

Written by Peter Ellis, this article focuses on how the observed case fatality rate of COVID-19 has evolved over time across 7 countries and comments on why the rates vary (low testing rates, age of the population, overwhelmed hospitals, etc.).

The code is available at the end of the article. The data is from John Hopkins and it uses the {coronavirus} package.

More recently, the author published a series of other articles:

Impact of a country’s age breakdown on COVID-19 case fatality rate: it looks at estimated fatalities in different countries according to the age distributions in those countries (based on Italy’s data). The data is from The Istituto Superiore di Sanità (Roma) and all the code is shown in the post.
How to make that crazy Fox News y axis chart with ggplot2 and scales: less about COVID19 than about how a bizarre Fox News graph can be re-created with the correct transformations needed to make its scale appropriate.
Test positivity rates and actual incidence and growth of diseases: this blog post looks at several different ways of accounting for the information given to us by high positive testing rates for COVID-19 and looks at the impact on estimates of effective reproduction number at a point in time.
Incidence of COVID-19 in Texas after adjusting for test positivity: the author examines the trends in COVID-19 cases in Texas, with and without being adjusted by a multiplier of the square root of the test positivity rate.

Covid 19 Tracking

Written by Prof. Kieran Healy, this article discusses how to get an overview of best-available counts of deaths, using the COVID-19 Data from the European Centers for Disease Control.

Code can be found in the article and on GitHub.

More recently, the author published three other articles:

A COVID Small Multiple: this article discusses how to create a small-multiple plot of cases by country, showing the trajectory of the outbreak for a large number of countries, with the background of each small-multiple panel also showing (in grey) the trajectory of every other country for comparison.

Apple’s COVID Mobility Data: this article uses Apple’s time series mobility data for several cities and countries (via the {covdata} package) to plot three modes of getting around: driving, public transit, and walking. The series begins on January 13th indexed to 100 at the beginning of the series, so trends are relative to that baseline.

New Orleans and Normalization: this article responds to a thoughtful post by Dr. Drang regarding an improvement in normalization of the data.

Infectious diseases and nonlinear differential equations

Published by Fabian Dablander, this math intensive blog post explains what SIR and SIRS models take into account and how they calculate their results.

From a pandemic perspective, the author writes “The SIRS model extends the SIR model, allowing the recovered population to become susceptible again (hence the extra ‘S’). It assumes that the susceptible population increases proportional to the recovered population”.

More recently, the author, in collaboration with other researchers, published another blog post outlining a number of excellent visualizations of the COVID19 pandemic, as well as presenting their own dashboard.

Epidemic modelling of COVID-19 in the UK using an SIR model

Published by Thomas Wilding, this blog post applies the SIR model to UK data.

As further extensions to the model, the author suggests:

Using an SEIR model (adding an Exposed compartment for people who are infected but not yet infectious)
Adding a “Q” layer since a lot of people are being Quarantined or isolated
Considering the “hidden”” population that is infected but is denied being tested due to shortage of tests
Feasibility of a second wave / outbreak of the epidemic later in the year (as seen in previous outbreaks, such as Swine Flu)

Data sources:

Modeling Pandemics

Published by Arthur Charpentier, this series of 3 blog post (part 1, part 2, part 3) walks through the SIR model and its parameters, how ODEquations solves it, and generating the reproductive rate. It also gives a mathematical explanation of a model for how quickly a pandemic will return, albeit with diminishing intensity. Last, it explains a model that is more sophisticated than SIR, the SEIR model, and illustrates it with Ebola data.

More recently, the author published another article which examines what proportion of the population in various U.S. states have been tested for the novel Coronavirus and tries to answer the following two questions:

How many people are tested on a daily basis?
What are we actually testing for?

Last but not least, this post reproduces one of his scientific paper entitled “COVID-19 pandemic control: balancing detection policy and lockdown intervention under ICU sustainability”.

COVID-19: The Case of Germany

Published by Prof. Dr. Holger K. von Jouanne-Diedrich from Learning Machines, this blog post uses the SIR model and German data to estimate the duration and severity of the pandemic.

Download the data from Morgenpost.

More recently, the author published other articles:

COVID-19 in the US: Back-of-the-Envelope Calculation of Actual Infections and Future Deaths: Working back from reported deaths from Covid19, the post shows how to estimate infections at a prior date, based on several assumptions about fatality rates and infected periods (and acknowledging many unknowns and data problems).
How to analyze mobility trends with R using anonymized and aggregated Apple’s mobility data available to the public. The article presents a R function to return the data in a well-structured format for countries and major cities, and to visualize the drop in vehicular and pedestrian movement caused by the pandemic.

COVID-19: False Positive Alarm, demonstrating the importance of infection rates on the likelihood that someone testing positive for the Coronavirus is actually positive.

Flatten the COVID-19 Curve

Published by Michael Höhle from Theory meets practice, this blog post discusses why the message of flattening the COVID-19 curve is right, but why some of the visualizations used to show the effect are wrong: Reducing the basic reproduction number does not just stretch the outbreak, it also reduces the final size of the outbreak.

From a pandemic point of view, the author writes “Because of limited health capacities, stretching out the outbreak over a longer time period will ensure, that a larger proportion of those in need of hospital treatment will actually get it. Other advantages of this approach are to win time in order to find better treatment forms and, possibly, to eventually develop a vaccine”.

A Shiny app has also been built upon this article to investigate different scenarios.

In a second article entitled “Effective reproduction number estimation”, Michael Höhle estimates with the {R0} package the time-varying effective reproduction number during an infectious disease outbreak such as COVID-19. Using a single simulated outbreak he compares the performance of three different estimation methods.

More recently, in this article the author looks at “superspreading” in infectious disease transmission from a statistical point of view. He characterises heterogeneity in the offspring distribution [who becomes infected by the superspreader person] by the Gini coefficient instead of the usual dispersion parameter of the negative binomial distribution. This allows us to consider more flexible offspring distributions.

Flattening vs shrinking: the math of #FlattenTheCurve

Published by Ben Bolker and Jonathan Dushoff, this blog post gives a clear explanation of physical distancing and explains how physical distancing makes several beneficial outcomes possible.

The code is available on GitHub.

explainCovid19 challenge

Published by Przemyslaw Biecek, this blog post gives an overview of a model that uses gradient boosting to predict survival based on age, country, and gender. It also shows how older people are more at risk and it lets you play with the model yourself with a modelStudio interactive dashboard.

Data sources:

Google sheet (with most recent data at the end of February)
Kaggle dataset

An R Package to explore the Novel Coronavirus

Published by Patrick Tung via Towards Data Science, this blog post translates into English an R package originally written in Chinese.

Data is collected from Tencent, at https://news.qq.com/zt2020/page/feiyan.htm, which contains one of the most up-to-date public information of the Coronavirus.

Coronavirus model using R – Colombia

Published by Daniel Pena Chavez, this blog post uses the code from Prof. Dr. Holger K. von Jouanne-Diedrich to model height of pandemic in Colombia and projected deaths. The author also points out that a huge number of other variables need to be considered, such as density, climate and government response.

Data is from Rami Krispin’s GitHub.

More recently, the author published another article comparing China and Italy’s rates on log scales.

COVID-19: The Case of Spain

Written by Jose from Diarium - Statistics and R software, this blog post, using data for Spain, applies the SIR model, and then a cubic polynomial regression model to predict infections, hospitalizations, deaths and peak date.

Tidying the new Johns Hopkins Covid-19 time-series datasets

Written by Joachim Gassen, this blog post provides functions and code to deal with different country names and changes on the Johns Hopkins site.

More recently, the author published a series of other interesting articles:

Merge Covid-19 Data with Governmental Interventions Data: this article analyzes five kinds of intervention on the spread of COVID-19.

Scraping Google Covid-19 community movement data from PDF figures: this article explains how to scrape data from a Google site that tracks movements of people. The author uses the {tidycovid19} R package and prepares an analysis of Germany and then across countries.

Covid-19: Explore Your Visualizer Degrees of Freedom: in this article, the author uses COVID-19 data to demonstrate how graphs can communicate very differently, and be manipulated. He shows that getting a ‘neutral’ message to the reader is far from trivial and that visualizations without guidance can be particularly misleading.
{tidycovid19} New data and documentation: A recent update to the {tidycovid19} package brings data on testing, alternative case data, some regional data and proper data documentation. Using all this, you can use the package to explore the associations of (the lifting of) governmental measures, citizen behavior and the Covid-19 spread.
Exploring and Benchmarking Oxford Government Response Data: A post assessing the impact of non-pharmaceutical interventions on the spread of Covid-19 based on the Assessment Capacities Project (ACAPS) and the Oxford Covid-19 Government Response Tracker.

COVID-19 in Belgium

Based on Tim Churches’ article, I published an analysis of the COVID-19 specifically for Belgium. In this article, I also use the most common epidemiological model, the SIR model (to its simplest form), to analyze the outbreak of the disease in the case where there would be no public health intervention. I also show how to compute the reproduction number and I present some additional improvements that can be made to further analyze the epidemic.

The code is available on GitHub, so feel free to use it as starting point for an analysis of the virus outbreak in your own country.

Facts About Coronavirus Disease 2019 (COVID-19) in 5 Charts created with R and ggplot2

Written by Gregory Kanevsky, this blog post compiles some useful facts about COVID-19 into 5 charts, including gauge charts, and discusses R and {ggplot2} techniques used to create them.

Contagiousness of COVID-19 Part I: Improvements of Mathematical Fitting

Written by Martijn Weterings on Learning Machines, this guest post describes the fitting of Covid-19 data with the SIR model and explains tricky parts of the fitting methodology and how we can mitigate some of the problems (e.g., early stopping of the algorithm or an ill-conditioned problem). It provides a very clear explanation of some tweaks to the standard model.

The code is available here.

Coronavirus : spatially smoothed decease in France and decease animation map

Published by Michael Ires, this blog post shows R code on how to use kernel weighted smoothing with arbitrary bounding areas to display a map of deaths from Covid-19 in France.

The author also published two other articles on how to build an animated map of deaths from Covid-19 in France and in Europe.

More recently, the author published an article where he uses the {geogrid} package to show the incidence of Covid19 by French departments as well as spread.

Another “flatten the COVID-19 curve” simulation… in R

Written by Javier Fernandez-Lopez, this blog post shows R code to create static plots and then simulations to demonstrate how social distancing could help to “flat the curve” of COVID-19 infections.

Tracking Covid19 Cases Throughout NJ with R

Written by Kevin Zolea, this blog post shows how to use the {gganimate} package to create an animated time series map showing how Covid19 spread throughout the U.S. state of New Jersey’s counties.

It’s fun to look at the YACM (Yet Another COVID Model)

Written by Adrian Barnett from Median Watch, all the models in this blog post are based on the excellent ordinary differential equation models by Alison Hill. They are microsimulations of those models that make heavy use of the {MicSim} package for running microsimulations in R.

Is COVID-19 as bad as all that? Yes it probably is

Written by Dr. Francis Smart from Econometrics By Simulation, this article explains clearly some of the factors that determine the infections and deaths from COVID19, with different scenarios of seriousness.

Potential Long-Term Intervention Strategies for COVID-19

On this website, several professors and members of Stanford University (Marissa Childs, Morgan Kain, Devin Kirk, Mallory Harris, Jacob Ritchie, Lisa Couper, Isabel Delwel, Nicole Nova, Erin Mordecai) developed a compartmental model of COVID-19 to evaluate possible outcomes of non-pharmaceutical interventions such as social distancing.

The website presents an introduction to the problem, the possibility to play around with the model to predict the effects of COVID intervention strategies (thanks to a Shiny app), the model details, and predictions for Santa Clara County, California.

The code is available on GitHub.

Animations in the time of Coronavirus

Written by Martin Henze from the Heads or Tails blog, this article describes how to extract and prepare the necessary data to animate the spread of the virus over time in Germany, using {gganimate} and {sf} R packages to create animated map visuals.

The author posted the dataset associated with the Germany maps to Kaggle, where he is maintaining it on a daily basis. In addition, he posted a version of the JHU US county level dataset where he added some key demographic info from the US census. This dataset is also updated daily.

COVID-19 Data and Prediction for Michigan

Written by Nagdev Amruthnath, this blog post builds and tests an exponential regression model based on (not much) State of Michigan data.

Data Visualization of COVID-19 in the US

Written by Daniel Reiff, this article examines COVID-19 growth dynamics using exponential and logistic curves.

The spread of COVID-19 across countries visualization with R

Written by Sergey Bryl, this article examines the speed and spreading of the virus across countries. One animated visualization and two stationary charts show:

how long and intensive were previous phases and
compare the effectiveness against COVID-19 for different countries

The code can be found at the end of the article.

Covid-19 and Rural Areas in the U.S

Written by Elliot Meador from Deltanomics, this blog post focuses on cases of Covid-19 in rural areas of the U.S, including whether in the South any particular state appears to be an outlier.

Covid Death Rates: Is the data correct?

Written by Sam Weiss, this post raises questions about the accuracy of reports of case numbers, in that they may fail to backfill for corrected data.

More recently, the author published two articles (part 1 and part 2) in which he finds and visualizes an association between number of people that tested positive for COVID-19 in a country and imports from China. In addition he finds that there are particular industries that are particularly correlated with COVID-19 rates.

COVID-19 Risk Heat Maps with Location Data, Apache Arrow, Markov Chain Modeling, and R Shiny

Written by Filip Stachura, this post describes Appsilon’s solution (CoronaRank) submitted to the recent Pandemic Response Hackathon. Inspired by Google’s PageRank, it uses geolocation data in the Apache Parquet format from Veraset for effective exposure risk assessment using Markov Chain modeling.

COVID-19 Tracker Indonesia

Written by Dio Ariadi from DataWizArt, this post shows some very nice plots on regional variation in COVID-19 cases and deaths in Indonesia, with very neatly integrated R code for each plot.

COVID-19 Projections Using Machine Learning

Developed by Youyang Gu, this website presents an intuitive model that builds machine learning techniques on top of a classic infectious disease model to make COVID-19 infections and deaths projections for the US, all 50 US states, and more than 60 countries.

The code can be found on GitHub.

COVID-19 in Belgium: is it over yet?

Written by myself in collaboration with Prof. Niko Speybroeck and Angel Rosas-Aguirre, this article shows the evolution of the number of hospital admissions and the number of confirmed cases in Belgium (by province and at the national level).

Code of the plots is available in the article.

COVID-19 Cases by Ethnicity

Written by Tommi Suvitaival, this article investigates Covid19 deaths as a function of the percentage of a county’s population that comes from various ethnic backgrounds.

Tennessee COVID-19 Update

Written by Prof. James M. Luther, this document presents a summary of the daily data for the state of Tennessee. The author uses an interactive map, a seven-day rolling average of various metrics, and facet charts.

Simulating Coronavirus Outbreak in Cities with Origin-Destination Matrix and SEIR Model

Written by Yihui Fan, this blog post lays out a step-by-step guide on simulating and visualising the spread of Coronavirus in the Greater Tokyo Area based on Origin-Destination Matrix and SEIR Model.

Another post from the same author focuses on the effectiveness of reducing population movement in managing Coronavirus outbreak.

COVID-19 Population Mobility - How has human mobility changed under the COVID-19 Pandemic?

Written by Kelsey E. Gonzalez, this visualization aims to understand population behavior during the COVID-19 pandemic.

Data is from Cuebiq and the code is available on GitHub.

How to Build COVID-19 Data-Driven Shiny Apps in 5 minutes

Written by Emanuele Guidotti, this tutorial shows how to build a simple yet complete Shiny application using the R Package COVID19: R Interface to COVID-19 Data Hub.

In another post, the author explores the R package {COVID19} in further detail.

Analyzing data from COVID19 R package

Written by Pablo Cánovas, this blog post explores whether deaths from COVID19 are being reported accurately.

It uses data from The Human Mortality Database and the COVID-19 Data Hub.

Body Mass and Risk from COVID-19 and Influenza

Written by Prof. Radford Neal, this blog post looks at data from flu-like illnesses and some preliminary Covid19 data. The author concludes that being underweight and being seriously obese are both risk factors for serious respiratory illness.

More recently, the author published a series of other articles:

The Puzzling Linearity of COVID-19 discussing the fact that for many countries, the linear plots of total cases or total deaths go up exponentially at first, and then approach a straight line that is not horizontal
Seasonality of COVID-19, Other Coronaviruses, and Influenza: this post looks at the evidence for seasonality in influenza and the common cold Coronaviruses, and to what extent one might expect COVID-19 to also be seasonal
Critique of “Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period”: this post analyzes and criticizes an earlier paper by Kissler et al. (2020). See also part 2 and part 3

HMD – Weekly Data

Written by Ronald Richman, this blog post explores the highly improbable level of deaths currently being reported using the Human Mortality Database and its recently begun special time series of weekly death data across 13 countries.

Guest posts on Chris Muir’s blog

Written by Skylar Hara, this blog post examines whether the incidence of Covid19 changed after the Governor of Hawaii issued a stay-at-home order.

Written by Ronja Steinbach, this blog post gathers data on Trump or Clinton states, percentages of minorities, and median household income to answer the following questions:

Does party affiliation of a state in the 2016 election have a significant impact on the incidence rate of the virus in that state?
Does the proportion of the population that is minority and median household income affect the incident rate of the virus across states?

Written by Akemi Santiago, this post shows a statistically different mortality rate between African-Americans in the United States and white Americans.

Written by Masha Rutenberg, this post focuses on the correlation between the population of a country and the number of confirmed infections in the country.

All authors are students of Prof. Chris Muir.

An R View into Epidemiology

Written by Joseph Rickert in R Views, this blog post tracks down R packages that help epidemiology research and shows the number of downloads in recent months for the five most popular packages.

Articles by Rob J Hyndman

Rob J. Hyndman, Professor of Statistics and Head of the Department of Econometrics and Business Statistics at Monash University (Australia), published a series of Covid19 related articles:

Forecasting COVID-19: this blog post does not use R, although Prof. Hyndman is an expert with it, but it does explain some of the problems with time series forecasting or other methods of forecasting
Why log ratios are useful for tracking COVID-19: this post presents the benefits of reporting log-scale graphics
Excess deaths for 2020: the reported COVID19 deaths in each country are often underestimated. One way to explore the true mortality effect of the pandemic is to look at “excess deaths” — the difference between death rates this year and the same time in previous years
Seasonal mortality rates: this post shows how the weekly mortality data published by the Human Mortality Database can be used to explore seasonality in mortality rates. Mortality rates are known to be seasonal due to temperatures and other weather-related effects

Turkey vs. Germany: COVID-19

Written by Selcuk Disci from DataGeeek, this article compares efforts by Turkey and Germany to control the pandemic, and tests several regression models.

Hands-on: How to build an interactive map in R-Shiny: An example for the COVID-19 Dashboard

Written by Sangmeng, this post explains how to build an interactive dashboard with Shiny with an example for the COVID-19 Dashboard.

Modelling COVID-19 in Morocco

Written by Zakariah Gassasse, this blog post presents data on cases of Covid19 in Morocco and applies the SIR model to the data.

More recently, the author published two other articles. The first article provides a short-term forecast of COVID-19 cases and deaths in Morocco by using simple but effective time-series analyses. The second article maps the outbreak to see which regions are suffering the most.

SIR models with Kermack and McKendrick

Written by Pierre Jacob, this blog post is mostly a retrospective look at the origins of the much-used SIR model.

Johns Hopkins Covid-19 Data and R

Written by Steve Miller, these blog posts (part 1 & part 2) showcase the handling of daily data of cases/deaths from Covid-19 in the U.S. published by Johns Hopkins University, and visualize moving averages of cases and deaths.

The author also published a blog post putting unemployment claims in the U.S. in perspective.

Estimating COVID-19’s \(R_t\) in Real-Time

Written by Ramnath Vaidyanathan, this tutorial shows how to estimate \(R_t\), the measure known as effective reproduction number, which is the number of people who become infected per infectious person at time \(t\).

From static to animated time series: the tidyverse way

Written by Giulia Ruggeri, this post goes through the steps necessary to create an animated COVID-19 time series plot.

Sneak peek: new Summit data tool helps clients visualize US areas that are most heavily impacted by the COVID-19 virus

Written by Colby Ziegler, this blog post describes a tool (built primarily in R using the {leaflet}, {tidyverse}, and {tigris} packages) tracking and displaying the total number of confirmed COVID-19 cases and deaths by U.S. county, overlaid with the locations of Certified Community Development Financial Institutions (CDFIs).

How to Reproduce Financial Times Style COVID19 Daily Reporting?

Written by Peyman Kor, this blog post shows how to reproduce in R the Financial Times facet plot by country.

Visualizing COVID cases in Belgium

In this blog post, Koen Hufkens plots cases in Belgium and addresses a challenge in geo-spatial plotting.

A spatio-temporal analysis of the environmental correlates of COVID-19 incidence in Spain

Written by Antonio Paez and several co-authors, this blog post looks at weather, humidity and other factors in Spain to create a SUR model.

Another post by Antonio Paez investigates the incidence of COVID-19 in the United States using Google Community Mobility Reports.

Covid-19 Analysis

Written by Rizami Annuar, this post compares data on cases and deaths in Malaysia to other countries, including correlations.

Australian governments can choose to slow the spread of coronavirus, but they would need to act immediately

Written by Matt Cowgill, this blog post shows that Australia early on had relatively few cases, but the post argues that the country needed to act urgently. Data is from the {gtrendsR} package.

In a more recent post, using Google trends data for key words associated with unemployment, the authors trace the effects of COVID19 in Australia in the early months.

In addition to these posts, a series of other posts have been published by Stephen Duckett and Brendan Coates:

Does Covid raise everyone’s relative risk of dying by a similar amount? More evidence

Written by Prof. David Spiegelhalter (author of the great book “The Art of Statistics”), this post looks at relative mortality rates by age and gender, using data from the U.K.’s Office for National Statistics.

Tracking Coronavirus: Building Parameterized Reports to Analyze Changing Data Sources

In this post, using Johns Hopkins data, Tyler Sanders builds a virus dashboard that can be updated each day with just the click of a button as an example of how to build parameterized reports with R Markdown.

Mapping NZ cases of COVID-19

In this post, using his own choropleth package ({DHBins}) and its hexagonal bins, Thomas Lumley maps cases of COVID19 by Health Boards in New Zealand.

Visualize the Pandemic with R #COVID-19

In this post, Xinhan Qian carries out a variety of explorations with Covid19 data, including the precipitous declines in U.S. movie box office revenue and restaurant reservations.

Exploring the Temporal Evolution of COVID-19 Cases in the United States

Written by Robert Winkelman and Colin Waltz, this post shows clearly how to create animated plots of the spread of COVID19 infections in the United States.

R Data Analysis: COVID-19

Published on the blog of Sharp Sight, this series of blog posts (part 1, 2, 3, 4, 5 and 6) explain how to rename and reorder columns, standardize dates (with the {lubridate} package), merge datasets, take other preparatory steps, plot with the {ggplot2} package and finally, reproduce in R a plot that shows the relative progress of 16 countries in coping with the pandemic.

All posts use the Johns Hopkins data and data from the company’s own collection.

GA COVID-19 Reports

Based on data from the GA Department of Public Health’s report, Andrew Benesh posts a daily analysis of the U.S. state Georgia’s cases, deaths, ICU usage etc.

All his reports are posted on Medium and the code is available here.

Corona in Belgium

Published by bnosac, this post covers the early exploration of the exponential spread of Covid19, with a focus on Belgium and the Netherlands.

The Coronavirus in Italy from the Twitter’s Point of View

Written by the Kode team, these two posts (here and here) use text-mining techniques to analyze tweets in Italy early in the pandemic and on speeches by the Prime Minister of Italy regarding Covid19.

They also published an interactive dashboard (in Italian) allowing to explore the data released daily by the Civil Protection.

Use R and Tidycensus to Look at COVID-19 Risk Factors

Written by René F. Najera, this post uses US census data to identify “overcrowded” areas and considers them in terms of Covid19 risk.

Animating U.S. COVID-19 hotspots over time

Written by Nathan Chaney, this post presents an animated map of the 7-day rolling average of new COVID-19 cases in US. Code for the animated map is available directly at the end of the post.

This post is an extension of his previous post on visualizing COVID-19 in Arkansas.

Understanding COVID19 in Connecticut. It takes a town

Based on these two posts by Nathan Chaney, Chuck Powell shows in his post how to create an animated map of 7-day rolling average of new COVID19 cases per 100,000 in Connecticut (by town).

Data

2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE: this dataset is used by many resources mentioned in this article and has become the gold standard for COVID-19 modeling
World Health Organization (WHO). See also their accompanying Shiny app
COVID-19 Open Research Dataset Challenge (CORD-19) (via Kaggle)
Novel Corona Virus 2019 Dataset: Day level information on Covid-19 affected cases (via Kaggle)
COVID-19 Data from the European Centers for Disease Control
The COVID Tracking Project
Tidying the John Hopkins Covid-19 data to long format and merging some World Bank data
A Short Review of COVID-19 Data Sources
COVID-19 datasets by esri
beoutbreakprepared/nCoV2019: one of the very few non-aggregated dataset available online. Such a dataset of individual-level information on patients with confirmed COVID-19, (including their travel history, location, symptoms, reported onset and confirmation dates and basic demographics) is important to understand, among others, transmissibility, risk of geographic spread, routes of transmission and risk factors for infection
Fighting the Covid-19: All the datasets and data efforts in one place: this post gathers many relevant datasets and data efforts
A Google Sheet which helps with tracking of the local lockdowns in the UK

Other lists or collections of resources

With so many great resources about the Coronavirus, other people also collected and organized similar lists.³ Below some collections I have been fortunate enough to discover:

COVID-19 Blog Post Directory: developed by Connor Rothschild and Rees Morrison, this Shiny app lets users interactively search a collection of over 400 posts by primary topic, post title, date, and whether the post uses a particular mathematical technique or data source. See also the accompanying blog post.
COVID-19 Modelling Resources, Data and Challenges by IDEA (not only R)
GitHub repo covid19-r by Mine Cetinkaya-Rundel (only R)
Amplifying people I trust on COVID-19: written by Jeff Leek from Simply Statistics, this article is a collection of trustworthy people and experts who share good information about the COVID-19 pandemic
Some Select COVID-19 Modeling Resources and More Select COVID-19 Resources by Joseph Rickert, assembling dashboards, Shiny apps, blog posts, packages, datasets, videos and conference proceedings that pertain to the Covid19 pandemic
CovidR Contest: launched by the European R users meeting (eRum), this contest is an open-source contest and pre-conference event, featuring any work done with R around the topic of the COVID-19 pandemic
Turning COVID-19 into a data visualization exercise for your students: written by Daniela Duca, this blog post presents a variety of methods to visualize data, drawing on several blog posts, Shiny apps and dashboards
Outbreak.info is an open source tool built by Scripps Research that standardizes and aggregates COVID-19 data. The interface aggregates journal articles, preprints, datasets, clinical trials, protocols, and other resources in one place, standardizing metadata and applying NLP to make these sources searchable and more accessible. The interactive data dashboards allow for quick comparison of countries, states, counties, and metro areas, and includes an API and R Package for researchers who want to access all of the raw data

I hope that, in addition to my collection, these rich lists done by others will give you enough background materials to analyze the outbreak of COVID-19 on your own (or at least some ideas)!

Non-english resources

This section may be of interested to only a limited number or people, but still, there are great resources in languages other than English. See a collection of them below listed by language:

Japanese:
- Coronavirus infection bulletin: original version of this dashboard
German:
- Developed by Prof. Dr. Helmut Küchenhoff, this CoronaMaps presents the situation of the Coronavirus in the world, in Europe and in Germany via a map and a table
Spanish:
- GitHub repository with official government data and R code used to extract it, see here and here
- COVID-19 en Chile: this Shiny app shows the accumulated confirmed contagion cases and provides an estimate of the growth rate for each municipality
- Developed by Que Oferton, this Shiny app provides an overview of the 2019 Novel Coronavirus COVID-19 epidemic in Central America, including statistics, forecast, SIR and SEIR models. The data and dashboard are refreshed on a daily basis
French:
- Written by Arthur Charpentier, this blog post shows how to quantify excess mortality using french mortality data

Conclusion

Thanks for reading.

I hope you will find these R resources on the COVID-19 Coronavirus useful. Feel free to let me know in the comments if I missed one.

A special thanks to Rees Morrison for his tremendous work on collecting and organizing several articles, which greatly helped in improving the section about blog posts. Read his articles (part 1 and 2) presenting a descriptive analysis of all the posts collected.

Although I have carefully read all resources, inclusion on the list does not mean that I endorse the findings. Moreover, some of the analyses, code, dashboards, packages or datasets might be out of date, so these should not be viewed, by default, as current findings. If you are the author of one of these resources, do not hesitate to contact me if you see any inconsistency or if you would like to remove it from this article.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

de Vlas, Sake J, and Luc E Coffeng. 2020. “A Phased Lift of Control: A Practical Strategy to Achieve Herd Immunity Against Covid-19 at the Country Level.” medRxiv.

Kissler, Stephen M, Christine Tedijanto, Edward Goldstein, Yonatan H Grad, and Marc Lipsitch. 2020. “Projecting the Transmission Dynamics of SARS-CoV-2 Through the Postpandemic Period.” Science 368 (6493): 860–68.

The package has also been the subject of a preprint.↩︎
See more information about this epidemiological model in this post by Marc Choisy.↩︎
Note that unlike my list, collections by others may include resources on COVID-19 using other tools than R.↩︎

Correlogram in R: how to highlight the most correlated variables in a dataset

Sat, 22 Feb 2020 00:00:00 +0000

Introduction

Correlation, often computed as part of descriptive statistics, is a statistical tool used to study the relationship between two variables, that is, whether and how strongly couples of variables are associated.

Correlations are measured between 2 variables at a time. Therefore, for datasets with many variables, computing correlations can become quite cumbersome and time consuming.

Correlation matrix

A solution to this problem is to compute correlations and display them in a correlation matrix, which shows correlation coefficients for all possible combinations of two variables in the dataset.

For example, below is the correlation matrix for the dataset mtcars (which, as described by the help documentation of R, comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles).¹ For this article, we include only the continuous variables.

dat <- mtcars[, c(1, 3:7)]
round(cor(dat), 2)

##        mpg  disp    hp  drat    wt  qsec
## mpg   1.00 -0.85 -0.78  0.68 -0.87  0.42
## disp -0.85  1.00  0.79 -0.71  0.89 -0.43
## hp   -0.78  0.79  1.00 -0.45  0.66 -0.71
## drat  0.68 -0.71 -0.45  1.00 -0.71  0.09
## wt   -0.87  0.89  0.66 -0.71  1.00 -0.17
## qsec  0.42 -0.43 -0.71  0.09 -0.17  1.00

Even after rounding the correlation coefficients to 2 digits, you will conceive that this correlation matrix is not easily and quickly interpretable.

If you are using R Markdown, you can use the pander() function from the {pander} package to make it slightly more readable, but still, we must admit that this table is not optimal when it comes to visualizing correlations between several variables of a dataset, especially for large datasets.

Correlogram

To tackle this issue and make it much more insightful, let’s transform the correlation matrix into a correlation plot. A correlation plot (also referred as a correlogram or corrgram in Friendly (2002)) allows to highlight the variables that are most (positively and negatively) correlated. Below an example with the same dataset presented above:

The correlogram represents the correlations for all pairs of variables. Positive correlations are displayed in blue and negative correlations in red. The intensity of the color is proportional to the correlation coefficient so the stronger the correlation (i.e., the closer to -1 or 1), the darker the boxes. The color legend on the right hand side of the correlogram shows the correlation coefficients and the corresponding colors.

As a reminder, a negative correlation implies that the two variables under consideration vary in opposite directions, that is, if one variable increases the other decreases and vice versa. A positive correlation implies that the two variables under consideration vary in the same direction, that is, if one variable increases the other increases and if one variable decreases the other decreases as well. Furthermore, the stronger the correlation, the stronger the association between the two variables.

Correlation test

Finally, a white box in the correlogram indicates that the correlation is not significantly different from 0 at the specified significance level (in this example, at \(\alpha = 5\)%) for the couple of variables. A correlation not significantly different from 0 means that there is no linear relationship between the two variables considered in the population (there could be another kind of association, but not linear).

To determine whether a specific correlation coefficient is significantly different from 0, a correlation test has been performed. Remind that the null and alternative hypotheses of this test are:

\(H_0\): \(\rho = 0\)
\(H_1\): \(\rho \ne 0\)

where \(\rho\) is denotes the correlation. The correlation test is based on two factors: the number of observations and the correlation coefficient. The more observations and the stronger the correlation between 2 variables, the more likely it is to reject the null hypothesis of no correlation between these 2 variables.

In the context of our example, the correlogram above shows that the variables wt (weight) and hp (horsepower) are positively correlated, while the variables mpg (miles per gallon) and wt (weight) are negatively correlated (both correlations make sense if we think about it). Furthermore, the variables wt and qsec are not correlated (indicated by a white box). Even if the correlation coefficient is -0.17 between the 2 variables, the correlation test has shown that we cannot reject the hypothesis of no correlation in the population. This is the reason the box for these two variable is white.

Although this correlogram presents exactly the same information than the correlation matrix, the correlogram presents a visual representation of the correlation matrix, allowing to quickly scan through it to see which variables are correlated and which are not.

Code

For those interested to draw this correlogram with their own data, here is the code of the function I adapted based on the corrplot() function from the {corrplot} package (thanks again to all contributors of this package):

The main arguments in the corrplot2() function are the following:

data: name of your dataset
method: the correlation method to be computed, one of “pearson” (default), “kendall”, or “spearman”. As a rule of thumb, if your dataset contains quantitative continuous variables that have a linear relationship, you can keep the Pearson method. If you have qualitative ordinal variables or quantitative variables with a partially linear link, the Spearman method is more appropriate
sig.level: the significance level for the correlation test, default is 0.05
order: order of the variables, one of “original” (default), “AOE” (angular order of the eigenvectors), “FPC” (first principal component order), “hclust” (hierarchical clustering order), “alphabet” (alphabetical order)
diag: display the correlation coefficients on the diagonal? The default is FALSE
type: display the entire correlation matrix or simply the upper/lower part, one of “upper” (default), “lower”, “full”
tl.srt: rotation of the variable labels
(note that missing values in the dataset are automatically removed)

`{ggstatsplot}` package

An alternative to the correlogram presented above is possible with the ggcorrmat() function from the {ggstatsplot} package:

# load package
library(ggstatsplot)

# correlogram
ggstatsplot::ggcorrmat(
  data = dat,
  type = "parametric", # parametric for Pearson, nonparametric for Spearman's correlation
  colors = c("darkred", "white", "steelblue") # change default colors
)

In this correlogram, the non-significant correlations (by default at the 5% significance level with the Holm adjustment method) are shown by a cross on the correlation coefficients.

The advantage of this alternative compared to the previous one is that it is directly available within a package, so you do not need to run the code of the function first in order to draw the correlogram.

`{lares}` package

Thanks to this article, I discovered the {lares} package which has really nice features regarding plotting correlations. Another advantage of this package is that it can be used to compute correlations with numerical, logical, categorical and date variables.

See more information about the package in this article.

All possible correlations

Use the corr_cross() function if you want to compute all correlations and return the highest and significant ones in a plot:

# devtools::install_github("laresbernardo/lares")
library(lares)

corr_cross(dat, # name of dataset
  max_pvalue = 0.05, # display only significant correlations (at 5% level)
  top = 10 # display top 10 couples of variables (by correlation coefficient)
)

Negative correlations are represented in red and positive correlations in blue.

Correlation of one variable against all others

Use the corr_var() function if you want to focus on the correlation of one variable against all others, and return the highest ones in a plot:

corr_var(dat, # name of dataset
  mpg, # name of variable to focus on
  top = 5 # display top 5 correlations
)

Conclusion

Thanks for reading.

I hope this article will help you to visualize correlations between variables in a dataset and to make correlation matrices more insightful and more appealing.

If you want to learn more about this topic, see:

how to compute correlation coefficients and perform correlation tests in R, or
how to compute correlation coefficients by hand.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Friendly, Michael. 2002. “Corrgrams: Exploratory Displays for Correlation Matrices.” The American Statistician 56 (4): 316–24.

The dataset mtcars is preloaded in R by default, so there is no need to import it into R. Check the article “How to import an Excel file in R” if you need help in importing your own dataset.↩︎

How to create a timeline of your CV in R?

Sun, 26 Jan 2020 00:00:00 +0000

Introduction

In this article, I show how to create a timeline of your CV in R. A CV timeline illustrates key information about your education, work experiences and extra activities. The main advantage of CV timelines compared to regular CV is that they make you stand out immediately by being visually appealing and easier to scan. It also allows you to better present your “story” by showing the chronology of your jobs and activities and thus explain how you got to where you are today. (It can also be part of your portfolio to show your R skills.)

We show below how to create such CV in R with a minimal reproducible example. Feel free to use the code and adapt it to you. For a more complete example (together with the code) you can check my own CV timeline.

Note that I wrote this article after reading this original post by Bernardo Lares, and in particular his package, i.e., {lares} package. A special thanks for his amazing work which was used to create a slightly modified version of the plot_timeline() function!

Minimal reproducible example

Here is the code and the result of a minimal reproducible example:

# All packages used below must be installed first
library(devtools)
# devtools::install_github("laresbernardo/lares")
library(lares)
library(ggplot2)


today <- as.character(Sys.Date())


### Edit from here ###
cv <- data.frame(rbind(
  c("PhD in Statistics", "University3", "Academic", "2017-09-01", today),
  c("MSc in Econometrics", "University2", "Academic", "2015-09-01", "2017-08-31"),
  c("BSc in Economics", "University1", "Academic", "2010-09-01", "2013-08-31"),
  c("Job title2", "Company2", "Work Experience", "2016-09-01", today),
  c("Job title1", "Company1", "Work Experience", "2013-08-31", "2015-08-31"),
  c("Extra1", "Place1", "Extra", "2015-05-01", today),
  c("Extra2", "Place2", "Extra", "2019-01-01", today),
  c("Extra3", NA, "Extra", "2019-12-01", today)
))
### Edit until here ###


order <- c("Role", "Place", "Type", "Start", "End")
colnames(cv) <- order


plot_timeline2 <- function(event, start, end = start + 1, label = NA, group = NA,
                           title = "Curriculum Vitae Timeline", subtitle = "Antoine Soetewey",
                           size = 7, colour = "orange", save = FALSE, subdir = NA) {
  df <- data.frame(
    Role = as.character(event), Place = as.character(label),
    Start = lubridate::date(start), End = lubridate::date(end),
    Type = group
  )
  cvlong <- data.frame(pos = rep(
    as.numeric(rownames(df)),
    2
  ), name = rep(as.character(df$Role), 2), type = rep(factor(df$Type,
    ordered = TRUE
  ), 2), where = rep(
    as.character(df$Place),
    2
  ), value = c(df$Start, df$End), label_pos = rep(df$Start +
    floor((df$End - df$Start) / 2), 2))
  maxdate <- max(df$End)
  p <- ggplot(cvlong, aes(
    x = value, y = reorder(name, -pos),
    label = where, group = pos
  )) +
    geom_vline(
      xintercept = maxdate,
      alpha = 0.8, linetype = "dotted"
    ) +
    labs(
      title = title,
      subtitle = subtitle, x = NULL, y = NULL, colour = NULL
    ) +
    theme_minimal() +
    theme(panel.background = element_rect(
      fill = "white",
      colour = NA
    ), axis.ticks = element_blank(), panel.grid.major.x = element_line(
      linewidth = 0.25,
      colour = "grey80"
    ))
  if (!is.na(cvlong$type)[1] | length(unique(cvlong$type)) >
    1) {
    p <- p + geom_line(aes(color = type), linewidth = size) +
      facet_grid(type ~ ., scales = "free", space = "free") +
      guides(colour = "none") +
      scale_colour_manual(values = c("#F8766D", "#00BA38", "#619CFF")) +
      theme(strip.text.y = element_text(size = 10))
  } else {
    p <- p + geom_line(linewidth = size)
  }
  p <- p + geom_label(aes(x = label_pos),
    colour = "black",
    size = 2, alpha = 0.7
  )
  if (save) {
    file_name <- "cv_timeline.png"
    if (!is.na(subdir)) {
      dir.create(file.path(getwd(), subdir), recursive = T)
      file_name <- paste(subdir, file_name, sep = "/")
    }
    p <- p + ggsave(file_name, width = 8, height = 6)
    message(paste("Saved plot as", file_name))
  }
  return(p)
}




plot_timeline2(
  event = cv$Role,
  start = cv$Start,
  end = cv$End,
  label = cv$Place,
  group = cv$Type,
  save = FALSE,
  subtitle = "Antoine Soetewey" # replace with your name
)

How to personalize it

If you want to edit the example with your own academic, extra and work experiences you basically just have to edit the dataframe called cv in the code above. Each row of the dataset cv is a different academic program, job or activity. Rows should include:

the name of the academic program, job title or activity
the name of the university, school, company or workplace
the category: academic, work experience or extra
the starting date (dates must be in format yyyy-mm-dd)
the ending date. If the role has not yet ended, type today instead of the date. By using today your CV timeline will automatically adapt to today’s date

Add or remove a row in the dataframe if you want to add or remove a role. Indicate NA if you do not want to specify any workplace (as it has been done for Extra3). Last, do not forget to replace my name with yours for the subtitle of the timeline at the end of the code.

Experienced R users may wish to edit the plot_timeline2 function to their needs. However, if you are happy with the template and design of the example, you only have to change things mentioned above.

Additional note

Instead of editing the code according to your roles directly in the script, you can also create an Excel file with the required data (job title, workplace, type, start date, end date) and then import it into R. Editing the Excel file is easier and less prone to coding errors. Moreover, if you have a long career, the code may become long while if you import the Excel file, it will always stay short and concise.

Conclusion

Thanks for reading.

I hope this article helped you to create a timeline of your CV in R. If you would like to see a more complete and live example, see my timeline CV.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A special thanks to Prof. Job N Nmadu for the suggestion about creating the file that holds the data.

World map of visited countries in R

Thu, 09 Jan 2020 00:00:00 +0000

Like me, if you like traveling as much as R you might want to draw a world map of the countries you have visited in R. Below an example with the countries I have visited as of January 2020:

To draw this map in R, you will need the following packages:

library(highcharter)
library(dplyr)
library(maps)

As usual, you need the packages to be installed on your machine before loading them with library(). You can install a package with the command install.packages("name_of_package").

After having loaded the packages, we are going to use the dataset called iso3166 from the {maps} package and rename it dat. Here are the first 6 rows of the dataset:

dat <- iso3166
head(dat)

##   a2  a3       ISOname               mapname sovereignty
## 1 AW ABW         Aruba                 Aruba Netherlands
## 2 AF AFG   Afghanistan           Afghanistan Afghanistan
## 3 AO AGO        Angola                Angola      Angola
## 4 AI AIA      Anguilla              Anguilla    Anguilla
## 5 AX ALA Aland Islands Finland:Aland Islands     Finland
## 6 AL ALB       Albania               Albania     Albania

We rename the variable a3 by iso-a3:

dat <- rename(dat, "iso-a3" = a3)
head(dat)

##   a2 iso-a3       ISOname               mapname sovereignty
## 1 AW    ABW         Aruba                 Aruba Netherlands
## 2 AF    AFG   Afghanistan           Afghanistan Afghanistan
## 3 AO    AGO        Angola                Angola      Angola
## 4 AI    AIA      Anguilla              Anguilla    Anguilla
## 5 AX    ALA Aland Islands Finland:Aland Islands     Finland
## 6 AL    ALB       Albania               Albania     Albania

We save the visited countries in a vector called countries_visited. To know the ISO codes of the countries you have visited, check the column ISOname in the dataset and extract the ISO codes from the column iso-a3:

countries_visited <- c("AUS", "AUT", "BEL", "CAN", "CZE", "DNK", "FIN", "FRA", "DEU", "GRC", "HUN", "IDN", "IRL", "ITA", "LVA", "LUX", "MCO", "MMR", "NLD", "NZL", "NOR", "PRT", "ROU", "SGP", "ESP", "SWE", "CHE", "TWN", "THA", "GBR", "USA")

We now create a new variable called visited which equals to 1 if you have visited the country and 0 otherwise:

dat$visited <- ifelse(dat$`iso-a3` %in% countries_visited, 1, 0)
head(dat)

##   a2 iso-a3       ISOname               mapname sovereignty visited
## 1 AW    ABW         Aruba                 Aruba Netherlands       0
## 2 AF    AFG   Afghanistan           Afghanistan Afghanistan       0
## 3 AO    AGO        Angola                Angola      Angola       0
## 4 AI    AIA      Anguilla              Anguilla    Anguilla       0
## 5 AX    ALA Aland Islands Finland:Aland Islands     Finland       0
## 6 AL    ALB       Albania               Albania     Albania       0

Finally, we are ready to draw the world map thanks to the hcmap() command from the {highcharter} package:

hcmap(
  map = "custom/world-highres3", # high resolution world map
  data = dat, # name of dataset
  joinBy = "iso-a3",
  value = "visited",
  showInLegend = FALSE, # hide legend
  nullColor = "#DADADA",
  download_map_data = TRUE
) %>%
  hc_mapNavigation(enabled = FALSE) %>%
  hc_legend("none") %>%
  hc_title(text = "World map") # title

Change the arguments to your needs and you are good to go.

To go even further, you can also add a list including all visited countries thanks to this code:

dat <- subset(dat, dat$visited == 1)
sort(unique(dat$ISOname)) # sort to have the visited countries in alphabetical order

##  [1] "Australia"                                           
##  [2] "Austria"                                             
##  [3] "Belgium"                                             
##  [4] "Canada"                                              
##  [5] "Clipperton Island"                                   
##  [6] "Czech Republic"                                      
##  [7] "Denmark"                                             
##  [8] "Finland"                                             
##  [9] "France"                                              
## [10] "Germany"                                             
## [11] "Greece"                                              
## [12] "Hungary"                                             
## [13] "Indonesia"                                           
## [14] "Ireland"                                             
## [15] "Italy"                                               
## [16] "Latvia"                                              
## [17] "Luxembourg"                                          
## [18] "Monaco"                                              
## [19] "Myanmar"                                             
## [20] "Netherlands"                                         
## [21] "New Zealand"                                         
## [22] "Norway"                                              
## [23] "Portugal"                                            
## [24] "Romania"                                             
## [25] "Singapore"                                           
## [26] "Spain"                                               
## [27] "Sweden"                                              
## [28] "Switzerland"                                         
## [29] "Taiwan"                                              
## [30] "Thailand"                                            
## [31] "United Kingdom of Great Britain and Northern Ireland"
## [32] "United States"

I never went to Clipperton Island, so I’ll remove it from the list:

dat <- subset(dat, dat$ISOname != "Clipperton Island")
sort(unique(dat$ISOname))

##  [1] "Australia"                                           
##  [2] "Austria"                                             
##  [3] "Belgium"                                             
##  [4] "Canada"                                              
##  [5] "Czech Republic"                                      
##  [6] "Denmark"                                             
##  [7] "Finland"                                             
##  [8] "France"                                              
##  [9] "Germany"                                             
## [10] "Greece"                                              
## [11] "Hungary"                                             
## [12] "Indonesia"                                           
## [13] "Ireland"                                             
## [14] "Italy"                                               
## [15] "Latvia"                                              
## [16] "Luxembourg"                                          
## [17] "Monaco"                                              
## [18] "Myanmar"                                             
## [19] "Netherlands"                                         
## [20] "New Zealand"                                         
## [21] "Norway"                                              
## [22] "Portugal"                                            
## [23] "Romania"                                             
## [24] "Singapore"                                           
## [25] "Spain"                                               
## [26] "Sweden"                                              
## [27] "Switzerland"                                         
## [28] "Taiwan"                                              
## [29] "Thailand"                                            
## [30] "United Kingdom of Great Britain and Northern Ireland"
## [31] "United States"

And finally, count the number of countries visited:

paste(
  "Total: ",
  length(unique(dat$ISOname)),
  " countries."
)

## [1] "Total:  31  countries."

In conclusion, here is the entire code to draw a world map with visited countries highlighted, a list of all countries in alphabetical order and the number of countries visited:

library(highcharter)
library(dplyr)
library(maps)

dat <- iso3166
dat <- rename(dat, "iso-a3" = a3)
countries_visited <- c("AUS", "AUT", "BEL", "CAN", "CZE", "DNK", "FIN", "FRA", "DEU", "GRC", "HUN", "IDN", "IRL", "ITA", "LVA", "LUX", "MCO", "MMR", "NLD", "NZL", "NOR", "PRT", "ROU", "SGP", "ESP", "SWE", "CHE", "TWN", "THA", "GBR", "USA")
dat$visited <- ifelse(dat$`iso-a3` %in% countries_visited, 1, 0)

hcmap(
  map = "custom/world-highres3", # high resolution world map
  data = dat, # name of dataset
  joinBy = "iso-a3",
  value = "visited",
  showInLegend = FALSE, # hide legend
  nullColor = "#DADADA",
  download_map_data = TRUE
) %>%
  hc_mapNavigation(enabled = FALSE) %>%
  hc_legend("none") %>%
  hc_title(text = "World map") # title

dat <- subset(dat, dat$visited == 1 & dat$ISOname != "Clipperton Island")
sort(unique(dat$ISOname)) # sort to have the visited countries in alphabetical order

paste(
  "Total: ",
  length(unique(dat$ISOname)),
  " countries."
)

Thanks for reading. I hope this article helped you to draw a world map with visited countries highlighted in R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Visualization on Stats and R

AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns

Why AssociationExplorer?

What does the app do?

A paper published in SoftwareX

The R package is also on CRAN

Who is it for, and how can it be used?

Acknowledgements

Useful links

References

Google Analytics in R: Review of 2022

Introduction

Prerequisites

Analytics

Page views over time

Page views per month and year

Top performing pages

Page views by country

Page views per day of week

Thank you note

Stats and R is 2 years old!

Introduction

Analytics

Users and page views

Page views over time

Page views per channel

Page views per day of week and month of year

Page views per month and year

Top performing pages

Page views by country

User engagement by devices

Browser information

End note

How to track the performance of your blog in R?

Introduction

Prerequisites

Analytics

Users, page views and sessions

Sessions over time

Sessions per channel

Sessions per day of week

Sessions per day and time

Sessions per month and year

Top performing pages

Time-normalized page views

Page views by country

Browser information

User engagement by devices

Content

Finding topics

Content distribution

A small note about ads

Future plans

Thank you note

Graphics in R with ggplot2

Introduction

Data

Basic principles of {ggplot2}

Create plots with {ggplot2}

Scatter plot

Line plot

Combination of line and points

Histogram

Density plot

Combination of histogram and densities

Dotplot

Boxplot

Barplot

Raincloud plot

Further personalization

Title and axis labels

Axis ticks

Log transformations

Limits

Scales for better axis formats

Legend

Shape, color, size and transparency

Text and labels

Smooth and regression lines

Facets

Basic principles of `{ggplot2}`

Create plots with `{ggplot2}`

Interactive plot with `{plotly}`

Combine plots with `{patchwork}`

Highlight data with `{gghighlight}`

Update with the `{ggstatsplot}` package

Coronavirus dashboard from the `{coronavirus} package`