Package on Stats and R

Bayesian Neural Networks in {tidymodels} with {kindling}

Fri, 20 Mar 2026 00:00:00 +0000

What Are Bayesian Neural Networks?

Standard neural networks learn fixed weights during training and produce a single point estimate for each input — with no sense of how confident the model is. Bayesian Neural Networks (BNNs) replace those fixed weights with probability distributions (Neal 2012). Instead of learning a single value per weight, the network learns a mean and a variance, and samples from that distribution at every forward pass.

Formally, a BNN places a prior $p(\mathbf{w})$ over the weights and updates it with observed data $\mathcal{D}$ via Bayes’ theorem:

\[p(\mathbf{w} \mid \mathcal{D}) = \frac{p(\mathcal{D} \mid \mathbf{w})\, p(\mathbf{w})}{p(\mathcal{D})}\]

For large and deep networks, the posterior $p(\mathbf{w} \mid \mathcal{D})$ is intractable. BNNs typically use variational inference to approximate it with a tractable distribution $q_\phi(\mathbf{w})$ by minimising the KL divergence between the two (Blundell et al. 2015).

From point estimates to distributions

A standard linear layer computes $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$, where $\mathbf{W}$ and $\mathbf{b}$ are fixed. In a Bayesian linear layer, they are random variables. We place a Gaussian prior over the weights:

\[\mathbf{W} \sim \mathcal{N}(\mu_{\text{prior}},\ \sigma_{\text{prior}}^2)\]

and during training we learn a variational posterior:

\[q(\mathbf{W}) = \mathcal{N}(\mu_W,\ \sigma_W^2)\]

Each weight therefore has two learnable scalars: a mean $\mu_W$ and a log-standard-deviation $\log \sigma_W$.

The reparameterisation trick

Sampling directly from $q(\mathbf{W})$ is not differentiable with respect to $\mu_W$ and $\sigma_W$. We resolve this with the reparameterisation trick (Kingma and Welling 2013):

\[\mathbf{W} = \mu_W + \sigma_W \odot \varepsilon, \quad \varepsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\]

We store $\log \sigma_W$ (not $\sigma_W$ directly) to ensure positivity during unconstrained optimisation:

\[\sigma_W = \exp(\texttt{weight}_{\log\sigma})\]

This yields the sampled weight used in the forward pass:

\[\mathbf{W}_{\text{sample}} = \mu_W + \exp(\texttt{weight}_{\log\sigma}) \odot \varepsilon_W\]

and similarly for the bias:

\[\mathbf{b}_{\text{sample}} = \mu_b + \exp(\texttt{bias}_{\log\sigma}) \odot \varepsilon_b\]

Is it possible?

At the moment, {tidymodels} offers limited APIs for neural network architectures, mostly around standard MLPs. A few packages make {torch} easier to use at a higher level:

{brulee} bridges {torch} with {tidymodels}. It is ergonomic, but currently focused on MLPs for tabular data.
{cito} does not integrate with {tidymodels} and leans more toward statistical applications.
The {torch} team maintains {luz}.
{tabnet} integrates with {tidymodels}, but is dedicated to the TabNet architecture.

{kindling} does not replace these packages. However, it helps close some of these gaps by supporting custom architectures with flexible depth. As a result, custom neural network architectures, including BNNs, can be used within {tidymodels} workflows. For a broader overview of what {kindling} enables in R, see this earlier post.

Setup

On March 3, 2026, {kindling} v0.3.0 was released on CRAN. Remember that utils::install.packages() in R always installs the latest version.

install.packages("kindling")

If you want the development version, you can retrieve the package by installing it from GitHub:

pak::pak("joshuamarie/kindling")

I recommend using {pak}: it is fast and very good at resolving package dependencies across environments, including system libraries and {renv}-managed projects.

Before using {kindling}, install LibTorch — the C++ backend shared by PyTorch and the {torch} R package (Falbel and Luraschi 2023):

torch::install_torch()

Loading namespace

When teaching R, I recommend using box::use() and qualifying the names you import. Create a bnn folder in your project root, then clone this repository to use BNNs in R (this module is adapted from torchbnn (Kim 2020)): https://github.com/joshuamarie/RTorchBNN.

You can run a specific command:

git clone --branch main https://github.com/joshuamarie/RTorchBNN bnn

Then, load the following:

box::use(
  bnn = . / bnn,
  kindling[train_nnsnip, nn_arch, act_funs],
  parsnip[fit, augment],
  yardstick[metrics]
)

Then register the models in {parsnip} with:

loadNamespace("kindling")

The `train_nnsnip()` Interface

In {kindling} v0.3.0, train_nnsnip() was introduced as a {parsnip}-compatible model specification that bridges the generalized neural network trainer with the {tidymodels} ecosystem. Unlike mlp_kindling(), which is specific to feedforward networks, train_nnsnip() is architecture-agnostic: you describe the layer topology via nn_arch(), allowing BNN-style or other custom layer types to slot in without changing the training logic.

Classification: Iris

We train a Bayesian Neural Network model to predict species in the iris dataset via train_nnsnip(), by configuring arch with nn_arch(). Inside nn_arch(), set nn_layer to bnn$BayesLinear instead of the default linear layer, then define the arguments for each layer with layer_arg_fn.

nn_model <-
  train_nnsnip(
    mode = "classification",
    arch = nn_arch(
      nn_layer = bnn$BayesLinear,
      out_nn_layer = torch::nn_linear,
      layer_arg_fn = ~ if (.is_output) {
        list(.in, .out)
      } else {
        list(
          in_features = .in,
          out_features = .out,
          prior_mu = 0,
          prior_sigma = 0.1
        )
      }
    ),
    hidden_neurons = c(64, 32),
    activations = act_funs(relu, elu),
    loss = "cross_entropy",
    epochs = 50,
    verbose = TRUE,
    learn_rate = 1e-3,
    optimizer_args = list(weight_decay = 0.01)
  ) |>
  fit(Species ~ ., data = iris)

nn_model |>
  augment(new_data = iris) |>
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass     0.667
## 2 kap      multiclass     0.5

Averaging predictions over multiple forward passes gives both a point estimate and a measure of predictive spread — all within a standard {tidymodels} pipeline.

The drawbacks

For now, there is no built-in option to update the loss dynamically during training. To use a custom objective such as the Evidence Lower Bound (ELBO), you currently need to define it manually and retrain with that loss.

Updating the model

That said, you can still apply a custom loss by setting the loss parameter to a torch function, for example torch::nnf_mse_loss(). If you want to train the model with an ELBO-style loss, do the following:

make_elbo_loss <- function(bnn_model, n_obs, kl_weight = 1.0) {
  box::use(
    torch[nnf_cross_entropy, torch_tensor]
  )

  function(input, target) {
    ce <- nnf_cross_entropy(input, target)
    device <- input$device
    kl_val <- torch_tensor(0.0, device = device, requires_grad = FALSE)
    for (m in bnn_model$modules) {
      if (inherits(m, "BayesLinear") && !is.null(m$kl)) {
        kl_val <- kl_val + m$kl
      }
    }

    ce + kl_weight * kl_val / n_obs
  }
}

elbo_loss <- make_elbo_loss(nn_model$fit$model, n_obs = nrow(iris), kl_weight = 1.0)

nn_model2 <-
  train_nnsnip(
    mode = "classification",
    arch = nn_arch(
      nn_layer = bnn$BayesLinear,
      out_nn_layer = torch::nn_linear,
      layer_arg_fn = ~ if (.is_output) {
        list(.in, .out)
      } else {
        list(
          in_features = .in,
          out_features = .out,
          prior_mu = 0,
          prior_sigma = 0.1
        )
      }
    ),
    hidden_neurons = c(64, 32),
    activations = act_funs(relu, elu),
    loss = elbo_loss,
    epochs = 50,
    verbose = TRUE,
    learn_rate = 1e-3,
    optimizer_args = list(weight_decay = 0.01)
  ) |>
  fit(Species ~ ., data = iris)

nn_model2 |>
  augment(new_data = iris) |>
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass      0.72
## 2 kap      multiclass      0.58

As always, if you have any question related to the topic covered in this post, please add it as a comment so other readers can benefit from the discussion.

References

Blundell, Charles, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. “Weight Uncertainty in Neural Network.” International Conference on Machine Learning, 1613–22.

Falbel, Daniel, and Javier Luraschi. 2023. Torch: Tensors and Neural Networks with GPU Acceleration. https://github.com/mlverse/torch.

Kim, Harry. 2020. Bayesian Neural Network for PyTorch [Torchbnn]. https://github.com/Harry24k/bayesian-neural-network-pytorch.

Kingma, Diederik P, and Max Welling. 2013. “Auto-Encoding Variational Bayes.” arXiv Preprint arXiv:1312.6114.

Neal, Radford M. 2012. Bayesian Learning for Neural Networks. Vol. 118. Springer Science & Business Media.

You can do more for neural networks in R with {kindling}

Thu, 19 Feb 2026 00:00:00 +0000

Why this post matters

Neural networks in R are no longer niche. Today, we can choose among:

{nnet} for classic, small-scale neural nets,
{neuralnet} another classic neural nets package besides {nnet},
{keras} / {keras3} for the Keras API (typically with Python backends such as TensorFlow/JAX/Torch),
{torch} for native-R deep learning with explicit model and training control.

So why discuss another package?

In our experience, many day-to-day projects sit in the middle: we want more flexibility than a classical model, but less boilerplate than writing a full {torch} training loop. {kindling} is designed for that middle ground.

In this article, we focus on what {kindling} does well, where it helps, and where it is still limited.

What problem `{kindling}` solves (and what it does not)

{kindling} is a higher-level interface built on top of {torch}. In practice, it reduces repetitive code for:

defining common network architectures,
training feedforward and recurrent models,
integrating with {tidymodels} ({parsnip}, {recipes}, {workflows}, {tune}).

It does not replace low-level {torch} for highly customized research code, and it does not make hardware setup disappear. You still need a working LibTorch installation and an environment that can use CPU/GPU properly.

At the time of writing (CRAN version 0.2.0), core built-in architectures include FFNN/MLP and recurrent variants (RNN/LSTM/GRU).

Setup

Install this package in two different ways:

Solely install {kindling} (as it installs the package dependencies internally)

install.packages("kindling")

You can install the package dependencies separately:

install.packages(c(
  "kindling",
  # dependencies
  "torch", "dplyr", "rsample", "recipes", "yardstick",
  "workflows", "parsnip", "tibble", "ggplot2", "mlbench", "vip"
))

Then load the packages in two ways:

Using library() traditionally:

library(kindling)
library(torch)
library(dplyr)
library(rsample)
library(recipes)
library(yardstick)
library(workflows)
library(parsnip)
library(tibble)
library(ggplot2)
library(mlbench)
library(vip)

In this article, we use library() for clarity and copy-paste reproducibility.

If you prefer a module-based and more explicit import style, use box::use() (read Box: Placing module system into R for more details).

Here’s the equivalent code as above:

box::use(
kindling[...],
dplyr[...],
rsample[...],
recipes[...],
yardstick[...],
workflows[...],
parsnip[...],
tibble[...],
ggplot2[...],
mlbench[...],
vip[...]
)

{kindling} uses {torch} as backend, so LibTorch must be installed once:

if (!torch::torch_is_installed()) {
  torch::install_torch()
}

Three levels of interaction

{kindling} is useful because you can work at different abstraction levels. Here are three ways to interact with the package, ordered from lowest to highest level of abstraction:

1) Generate model code (`*_generator()`)

If you want to inspect architecture code before training:

ffnn_generator(
  nn_name = "MyNet",
  hd_neurons = c(64, 32, 16),
  no_x = 10,
  no_y = 1,
  activations = act_funs(
    relu,
    "softplus(beta = 0.5)",
    selu
  )
)

## torch::nn_module("MyNet", initialize = function () 
## {
##     self$fc1 = torch::nn_linear(10, 64, bias = TRUE)
##     self$fc2 = torch::nn_linear(64, 32, bias = TRUE)
##     self$fc3 = torch::nn_linear(32, 16, bias = TRUE)
##     self$out = torch::nn_linear(16, 1, bias = TRUE)
## }, forward = function (x) 
## {
##     x = self$fc1(x)
##     x = torch::nnf_relu(x)
##     x = self$fc2(x)
##     x = torch::nnf_softplus(x, beta = 0.5)
##     x = self$fc3(x)
##     x = torch::nnf_selu(x)
##     x = self$out(x)
##     x
## })

Technical note: To specify a parametric activation functions like Softplus under act_funs() function, set softplus = args(beta = 0.5) or softplus[beta = 0.5] (available in v0.3.x and later), not just in a stringly typed expression, e.g. "softplus(beta = 0.5)".

This returns an unevaluated torch::nn_module expression you can inspect or modify.

2) Direct Training

The functions available (for now) are ffnn() and rnn(). For many applied tasks, this is the fastest way to fit a network from a formula.

mini_direct <- ffnn(
  mpg ~ .,
  data = mtcars,
  hidden_neurons = 8,
  activations = act_funs(relu),
  epochs = 20,
  verbose = FALSE
)

predict(mini_direct, newdata = head(mtcars, 3))

## [1] 7.887171 7.877418 6.773977

3) ML Framework Integration: `{tidymodels}` with `mlp_kindling()` / `rnn_kindling()`

{kindling} functions to directly train the said models also (currently) integrates with {tidymodels}. This level is ideal when we want recipes, workflows, resampling, and tuning.

mini_spec <- mlp_kindling(
  mode = "regression",
  hidden_neurons = 8,
  activations = act_funs(relu), # or just "relu"
  epochs = 20,
  verbose = FALSE
)

mini_wf <- workflow() |>
  add_formula(mpg ~ .) |>
  add_model(mini_spec)

mini_fit <- fit(mini_wf, data = mtcars)

predict(mini_fit, new_data = head(mtcars, 3))

## # A tibble: 3 × 1
##   .pred
##   <dbl>
## 1  9.68
## 2  9.70
## 3  7.70

Now that we’ve covered the three levels of interaction, let’s see {kindling} in action with two “realistic” examples. We’ll demonstrate how to structure reproducible workflows, handle feature preprocessing, and evaluate results properly. Along the way, you’ll see which abstraction level works best for different scenarios.

Example 1: Iris classification with reproducible good practice

This first example uses direct training with ffnn(), but still follows practical steps:

train/test split,
feature scaling,
validation split during training,
regularization,
out-of-sample evaluation.

Data Preprocessing

Train/test split is performed to prevent evaluating only on training data.

data(iris)

iris_split <- initial_split(iris, prop = 0.8, strata = Species)
iris_train <- training(iris_split)
iris_test <- testing(iris_split)

iris_recipe <- recipe(Species ~ ., data = iris_train) |>
  step_normalize(all_predictors())
iris_prep <- prep(iris_recipe, training = iris_train)
iris_train_processed <- bake(iris_prep, new_data = NULL)
iris_test_processed <- bake(iris_prep, new_data = iris_test)

MLP with 2 Hidden Layers

set.seed(20260218)

iris_mlp <- ffnn(
  Species ~ .,
  data = iris_train_processed,
  hidden_neurons = c(32, 16),
  activations = act_funs(relu, "softplus(beta = 0.5)"),
  loss = "cross_entropy",
  optimizer = "adam",
  learn_rate = 0.01,
  penalty = 1e-4,
  mixture = 0,
  batch_size = 16,
  epochs = 200,
  validation_split = 0.2,
  verbose = FALSE
)

iris_mlp

## 
## ======================= Feedforward Neural Networks (MLP) ======================
## 
## 
## -- FFNN Model Summary ----------------------------------------------------------
## 
## 
## -------------------------------------------------------------------------------------
##   NN Model Type           :             FFNN    n_predictors :                    4
##   Number of Epochs        :              200    n_response   :                    3
##   Hidden Layer Units      :           32, 16    reg.         :   [λ = 1e-04, α = 0]
##   Number of Hidden Layers :                2    Device       :                  mps
##   Pred. Type              :   classification                 :                     
## -------------------------------------------------------------------------------------
## 
## 
## 
## -- Activation function ---------------------------------------------------------
## 
## 
##                -------------------------------------------------
##                  1st Layer {32}    :                      relu
##                  2nd Layer {16}    :      softplus(beta = 0.5)
##                  Output Activation :   No act function applied
##                -------------------------------------------------

iris_pred <- predict(iris_mlp, newdata = iris_test_processed, type = "response")

iris_eval <- tibble(
  truth = iris_test_processed$Species,
  .pred_class = iris_pred
)

metrics(iris_eval, truth = truth, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass     0.933
## 2 kap      multiclass     0.9

conf_mat(iris_eval, truth = truth, estimate = .pred_class)

##             Truth
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         1
##   virginica       0          1         9

Why these choices?

Here’s the brief explanation on how we set up the models from {kindling}:

Architecture (c(32, 16)): The architecture has 2 layers with 32 units for the 1st layer and 16 for the 2nd layer. It is enough capacity for Iris, but still simple.
Activations: The architecture has 2 hidden layers, each layer is specified with different activation functions.
1. relu: Also called Rectified Linear Unit (ReLU); the most stable default for hidden layers.
2. softplus: Similar to ReLU but much smoother in approximation; beta is adjusted to 0.5.
Scaling: crucial because neural networks are sensitive to predictor scale.
Regularization (penalty and mixture): helps reduce overfitting even on small data. mixture is set to 0, which means the optimization is performed with L2 regularization.
Validation split: monitors generalization during training.

Example 2: A more realistic tabular benchmark (Sonar)

Iris is useful for teaching, but sometimes considered (too) easy. The Sonar dataset (60 numeric predictors, binary outcome) is a better stress test for tabular classification.

data("Sonar", package = "mlbench")

sonar <- Sonar |>
  mutate(Class = factor(Class))

sonar_split <- initial_split(sonar, prop = 0.8, strata = Class)
sonar_train <- training(sonar_split)
sonar_test <- testing(sonar_split)

Now fit through {tidymodels} so preprocessing and modeling stay in one pipeline:

sonar_rec <- recipe(Class ~ ., data = sonar_train) |>
  step_normalize(all_predictors())

sonar_spec <- mlp_kindling(
  mode = "classification",
  hidden_neurons = c(64, 32),
  activations = act_funs(relu, relu),
  optimizer = "adam",
  learn_rate = 0.001,
  penalty = 1e-4,
  mixture = 0,
  batch_size = 16,
  epochs = 250,
  validation_split = 0.2,
  verbose = FALSE
)

sonar_wf <- workflow() |>
  add_recipe(sonar_rec) |>
  add_model(sonar_spec)

set.seed(20260218)
sonar_fit <- fit(sonar_wf, data = sonar_train)

sonar_pred <- augment(sonar_fit, new_data = sonar_test)

metrics(sonar_pred, truth = Class, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy binary         0.860
## 2 kap      binary         0.720

In practice, this setup is often a strong baseline for tabular binary classification before trying larger architectures.

About early stopping and callbacks

If you come from Keras/TensorFlow, you may be used to callback objects (early stopping, learning-rate schedules, etc.).

With {kindling} 0.2.0, practical overfitting control is usually handled through:

validation_split,
regularization (penalty, mixture),
tuning model size and training length (epochs),
proper resampling with {tidymodels}.

In other words, we can approximate early-stopping behavior by selecting epochs via validation/resampling, even without a callback-heavy workflow.

Supported architectures (current scope)

Architecture	Main function(s)	Typical use
Feedforward (MLP/FFNN)	ffnn(), mlp_kindling()	Tabular regression/classification
RNN	rnn_kindling(rnn_type = "rnn")	Sequential patterns
LSTM	rnn_kindling(rnn_type = "lstm")	Longer-range sequence dependencies
GRU	rnn_kindling(rnn_type = "gru")	Sequence modeling with fewer parameters

Variable importance (FFNN)

Interpretability for neural networks is imperfect, but {kindling} integrates established approaches for FFNN models.

# For FFNN fits:
garson(iris_mlp, bar_plot = FALSE)

##        x_names y_names  rel_imp
## 1 Petal.Length       y 36.29246
## 2  Sepal.Width       y 23.15108
## 3  Petal.Width       y 21.63324
## 4 Sepal.Length       y 18.92322

olden(iris_mlp, bar_plot = FALSE)

##        x_names y_names    rel_imp
## 1 Petal.Length       y -2.8066132
## 2  Petal.Width       y -2.1824838
## 3  Sepal.Width       y  1.2130474
## 4 Sepal.Length       y -0.7235911

# Via vip (Olden/Garson methods supported by kindling S3 methods)
vi(iris_mlp, type = "olden") |>
  vip()

We recommend using these as directional diagnostics, not as causal evidence.

When `{kindling}` is a good fit

{kindling} is a practical choice when our projects:

work mostly in R and want to stay inside {tidymodels},
need neural nets for tabular or moderate sequence tasks,
want less boilerplate than raw {torch} but still meaningful control.

It may be less ideal when our projects need:

custom research architectures and training loops,
mature callback ecosystems similar to high-level Keras workflows,
highly optimized distributed production pipelines.

Limitations to keep in mind

Hardware setup still matters: GPU usage depends on a correct {torch}/LibTorch installation and supported hardware.
Ecosystem maturity: this is a young package; interfaces can evolve.
Debugging depth: for deeply custom debugging, low-level {torch} remains the reference path.
Model class assumptions: recurrent models are for sequence-structured data; using them on plain tabular data is usually not appropriate.

Takeaways

{kindling} is not about replacing {torch} or {keras3}. It is about reducing friction for common deep-learning workflows in R.

For many applied projects, a robust pattern is:

preprocess with {recipes},
start with a modest MLP architecture,
use validation split + regularization,
evaluate on held-out test data,
tune only after you have a strong baseline.

If this matches your workflow, {kindling} is worth trying.

As always, if you have any question related to the topic covered in this post, please add it as a comment so other readers can benefit from the discussion.

nycOpenData: A unified R interface to NYC Open Data APIs

Tue, 27 Jan 2026 00:00:00 +0000

Guest post by Christian Martinez, developer of the nycOpenData package in R.

nycOpenData: A unified R interface to NYC Open Data APIs

I am pleased to announce the release of nycOpenData, an R package providing convenient, tidy access to dozens of datasets from the New York City Open Data platform.

The package is designed as part of an open-science and reproducible-research effort, with the goal of lowering the friction between public data and statistical analysis—especially for teaching, exploratory research, and applied civic work.

Why nycOpenData?

NYC Open Data hosts hundreds of datasets covering topics such as public safety, housing, transportation, education, health, and city services. While these datasets are publicly accessible through the Socrata API, working with them directly often requires:

knowing dataset identifiers,
manually constructing API queries,
handling pagination, timeouts, and rate limits,
and performing repetitive data-cleaning steps.

These barriers can slow down exploratory analysis and make public data less accessible to students, researchers, and practitioners who primarily work in R.

nycOpenData was built to remove these obstacles by providing a consistent, user-friendly interface that returns clean tibbles ready for analysis—without requiring users to interact directly with the API.

What does the package do?

The package provides a growing collection of wrapper functions, each corresponding to a specific NYC Open Data dataset or dataset family. All functions follow a shared design pattern and support:

row limits,
optional filtering via named lists,
sorting,
and graceful handling of API errors and timeouts.

Examples of currently supported domains include:

311 service requests
Transportation and for-hire vehicles
Motor vehicle collisions
Department of Buildings permits and complaints
Education and school reporting
Juvenile justice and public safety
Street trees and environmental data
Permitted events (historical)

A typical call looks like this:

library(nycOpenData)

nyc_311(
  limit = 1000,
  filters = list(borough = "BROOKLYN")
)

## # A tibble: 1,000 × 40
##    unique_key created_date          agency agency_name complaint_type descriptor
##    <chr>      <chr>                 <chr>  <chr>       <chr>          <chr>     
##  1 67613985   2026-01-26T02:06:05.… NYPD   New York C… Noise - Resid… Banging/P…
##  2 67609553   2026-01-26T02:02:09.… NYPD   New York C… Noise - Resid… Banging/P…
##  3 67610990   2026-01-26T01:58:58.… NYPD   New York C… Illegal Parki… Blocked H…
##  4 67615428   2026-01-26T01:56:49.… NYPD   New York C… Noise - Resid… Banging/P…
##  5 67609568   2026-01-26T01:48:16.… NYPD   New York C… Noise - Resid… Loud Musi…
##  6 67612476   2026-01-26T01:47:10.… NYPD   New York C… Noise - Resid… Loud Musi…
##  7 67614152   2026-01-26T01:46:26.… DSNY   Department… Snow or Ice    Snow Trac…
##  8 67614054   2026-01-26T01:44:50.… DSNY   Department… Dirty Conditi… Trash     
##  9 67606570   2026-01-26T01:41:32.… NYPD   New York C… Noise - Resid… Banging/P…
## 10 67610091   2026-01-26T01:35:51.… NYPD   New York C… Noise - Vehic… Car/Truck…
## # ℹ 990 more rows
## # ℹ 34 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>, …

The result is returned as a tidy tibble of the 1,000 most recent NYC 311 requests, making it immediately compatible with the tidyverse ecosystem for visualization, modeling, and reporting.

Mini analysis

One of the strongest qualities this function has is its ability to filter based on multiple columns. Let’s put everything together and get a dataset of the last 1,000 311 requests from the New York Police Department in Brooklyn.

# Creating the dataset
brooklyn_nypd <- nyc_311(limit = 1000, filters = list(agency = "NYPD", borough = "BROOKLYN"))

# Calling head of our new dataset
head(brooklyn_nypd)

## # A tibble: 6 × 39
##   unique_key created_date           agency agency_name complaint_type descriptor
##   <chr>      <chr>                  <chr>  <chr>       <chr>          <chr>     
## 1 67613985   2026-01-26T02:06:05.0… NYPD   New York C… Noise - Resid… Banging/P…
## 2 67609553   2026-01-26T02:02:09.0… NYPD   New York C… Noise - Resid… Banging/P…
## 3 67610990   2026-01-26T01:58:58.0… NYPD   New York C… Illegal Parki… Blocked H…
## 4 67615428   2026-01-26T01:56:49.0… NYPD   New York C… Noise - Resid… Banging/P…
## 5 67609568   2026-01-26T01:48:16.0… NYPD   New York C… Noise - Resid… Loud Musi…
## 6 67612476   2026-01-26T01:47:10.0… NYPD   New York C… Noise - Resid… Loud Musi…
## # ℹ 33 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>,
## #   x_coordinate_state_plane <chr>, y_coordinate_state_plane <chr>, …

# Quick check to make sure our filtering worked
nrow(brooklyn_nypd)

## [1] 1000

unique(brooklyn_nypd$agency)

## [1] "NYPD"

unique(brooklyn_nypd$borough)

## [1] "BROOKLYN"

We successfully created our dataset that contains the 1,000 most recent requests regarding the NYPD in the borough Brooklyn.

Now that we have successfully pulled the data and have it in R, let’s figure out what NYC residents in Brooklyn are complaining about to the NYPD.

To do this, we will create a bar graph of the complaint types.

# Visualizing the distribution, ordered by frequency
library(ggplot2)

ggplot(brooklyn_nypd, aes(y = reorder(complaint_type, complaint_type, length))) +
  geom_bar(fill = "steelblue") +
  theme_minimal() +
  labs(
    title = "Most Recent NYPD 311 Complaints (Brooklyn)",
    subtitle = "Top 1,000 service requests",
    x = "Number of Complaints",
    y = "Type of Complaint"
  )

Figure 1: Bar chart showing the frequency of NYPD-related 311 complaint types in Brooklyn from the 1,000 most recent service requests.

This graph shows us not only which complaints were made, but how many of each complaint were made.

Designed for reproducible workflows

A core design principle of nycOpenData is reproducibility. Rather than downloading static CSV files that can change over time or be accidentally modified, analyses can explicitly document:

which dataset was used,
how many rows were requested,
which filters were applied,
and when the data were accessed.

This makes the package particularly useful for:

reproducible research projects,
classroom assignments,
data journalism,
and exploratory civic analysis.

The package is also designed to be API-polite, with configurable timeouts and safeguards that help prevent common failure modes when querying large public datasets.

Who is it for?

nycOpenData is intended for a broad audience, including:

students learning statistics or data science using real-world data,
instructors teaching reproducible research or applied data analysis,
researchers conducting exploratory or descriptive analyses,
data journalists and civic technologists,
and anyone interested in working with NYC public data in R.

The goal is not to abstract away the data itself, but to make access predictable, transparent, and easy to integrate into standard R workflows.

Availability

The package is available on CRAN and can be installed using:

install.packages("nycOpenData")

Development continues on GitHub, where new datasets and improvements are added regularly.

Acknowledgements

This package was developed alongside teaching and applied research projects in reproducible data science, with inspiration from open-source contributors across the R community and the NYC Open Data program.

Useful links

CRAN package: https://CRAN.R-project.org/package=nycOpenData
pkgdown site: https://martinezc1.github.io/nycOpenData/
GitHub repository: https://github.com/martinezc1/nycOpenData
NYC Open Data portal: https://opendata.cityofnewyork.us/

As always, feedback, bug reports, and dataset requests are very welcome.

Thanks for reading!

AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns

Tue, 16 Dec 2025 00:00:00 +0000

I am pleased to announce the publication of our paper “AssociationExplorer: A user-friendly Shiny application for exploring associations and visual patterns” in the journal SoftwareX, together with the official release of the AssociationExplorer2 R package on CRAN.

Both the paper and the software are part of an open-science effort aimed at making exploratory data analysis more accessible to non-technical users.

Why AssociationExplorer?

Exploring multivariate datasets is now central in social sciences, data journalism, and education. However, identifying and interpreting associations between variables often requires programming skills and a solid background in statistics, which can represent a substantial barrier for many users.

AssociationExplorer was designed to lower this barrier by providing an interactive, visual, and statistically grounded tool for exploring associations between quantitative and qualitative variables, without requiring users to write any code.

The application is primarily intended for:

journalists and data journalism practitioners,
teachers and students,
researchers in the exploratory phase of an analysis,
engaged citizens interested in understanding public or survey data.

What does the app do?

AssociationExplorer follows a simple and guided workflow:

Import data (CSV or Excel files)
Interactively select variables of interest
Automatically compute association measures adapted to variable types:
- Pearson’s $r$ correlation for numeric–numeric pairs,
- Cramer’s V for categorical–categorical pairs,
- the correlation ratio $\eta$ for mixed numeric–categorical pairs
Filter associations using user-defined thresholds
Visualize results through:
- an interactive correlation network,
- contextual bivariate visualizations (scatter plots, mean plots, and colored contingency tables)

This workflow is designed to support transparent, reactive, and interpretable exploratory data analysis.

A paper published in SoftwareX

The SoftwareX paper provides a detailed description of:

the motivation and intended audience of the tool,
the software architecture,
the methodological choices underlying the association measures,
an illustrative case study based on the European Social Survey,
and perspectives for future development.

Link to the paper: https://doi.org/10.1016/j.softx.2025.102483

In line with the journal’s standards, the code, documentation, and example data are fully open and reproducible.

The R package is also on CRAN

Alongside the paper, an R package is also available on CRAN, making installation and use straightforward:

install.packages("AssociationExplorer2")
library(AssociationExplorer2)
run_associationexplorer()

The CRAN release ensures:

standardized installation,
better integration with existing R workflows,
clearer versioning and dependency management.

Who is it for, and how can it be used?

AssociationExplorer is particularly useful for:

exploratory analysis prior to formal modeling,
teaching concepts related to association and dependence,
data storytelling and journalistic exploration,
the analysis of survey data and public datasets.

The goal is not to replace confirmatory statistical analyses, but to provide a robust tool for understanding the structure of the data before modeling.

Acknowledgements

This work was carried out in collaboration with Cédric Heuchenne, Arnaud Claes, and Antonin Descampe, whom I warmly thank.

The project is supported by the Walloon Region and SPW Recherche within the ODALON research project.

Useful links

📄 SoftwareX paper: https://doi.org/10.1016/j.softx.2025.102483
📦 CRAN package: https://CRAN.R-project.org/package=AssociationExplorer2
💻 GitHub repository:
- of the paper
- of the R package

As always, feedback, bug reports, and suggestions are very welcome.

Thanks for reading!

References

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2026). AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns. SoftwareX, 33(102483). https://doi.org/10.1016/j.softx.2025.102483

Soetewey, A., Heuchenne, C., Claes, A. and Descampe, A. (2025). AssociationExplorer2: A User-Friendly ‘shiny’ Application for Exploring Associations and Visual Patterns. R package version 0.1.4, https://github.com/AntoineSoetewey/AssociationExplorer2

Nonparametric serial interval estimation

Mon, 18 Aug 2025 00:00:00 +0000

Motivation

Epidemiological delays inform about the time between two well-defined events related to a disease. The serial interval (SI) of an infectious disease is defined as the time between symptom onset in a primary case (infector) and symptom onset in a secondary case (infectee). It is a widely used epidemiological delay quantity and plays a central role in mathematical/statistical models of disease transmission. There exists a tight link between the reproduction number (average number of secondary infections generated by an infected individual) and the serial interval. Therefore, getting accurate knowledge about the SI distribution is key to gain a clear understanding of transmission dynamics during outbreaks. Timings of symptom onset for infector-infectee pairs can be obtained from line list data and observations usually consist of calendar dates. From a mathematical perspective, it is more convenient to work with numbers than with calendar dates and the latter are typically transformed to integers for the sake of statistical analysis.

The main challenge when working with SI data is censoring in the sense that exact symptom onset times are usually unobserved and only known to have occurred between two time points. If the time resolution of a reported timing of illness onset is a calendar day, for instance July 15, there is not enough information to determine the exact time of illness onset within that day. As such, symptom onset is assumed to have occurred between July 15 and July 16 and we say that serial interval data are interval-censored. The figure below illustrates the coarse structure of SI data that adds a layer of complexity to the estimation problem.

Source: Gressani O, Hens N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Comput Biol 21(8): e1013338.

A recent article by Gressani and Hens (2025) published in PLOS Computational Biology proposes a new estimator of the cumulative distribution function of the serial interval without making parametric assumptions regarding the underlying SI distribution. The estimator is based on mixtures of uniform distributions and only requires left and right bounds of serial interval windows of infector-infectee pairs as a main input ($s_{iL}$ and $s_{iR}$ in the above figure). Point estimates of different serial interval features are available in closed-form and the bootstrap is used to compute confidence intervals. The nonparametric methodology is relatively simple and computationally fast and stable. Moreover, a user-friendly routine is available in the EpiDelays package written in R. This post aims at giving users a simple first experience with this new nonparametric methodology for serial interval estimation. The package can be installed from GitHub (using devtools) as follows:

install.packages("devtools")
devtools::install_github("oswaldogressani/EpiDelays")

Simulated data

The estimSI() routine of the EpiDelays package can be used to compute nonparametric estimates (point estimates with standard errors and confidence intervals) of different serial interval features (e.g. the mean, median, standard deviation). The routine is simple to use and requires only two inputs:

x: A data frame with $n$ rows (corresponding to the number of transmission pairs for which illness onset data is available) and two columns containing the lower bound of the SI window $s_{iL}$ (first column) and the upper bound of the SI window $s_{iR}$ (second column).
nboot: An integer for the bootstrap sample size (default is 2000) used to construct ($90\%$ and $95\%$) confidence intervals (CIs).

We start by illustrating the use of estimSI() on simulated data. The simSI() routine can be used to simulate artificial serial interval data with SI windows having a width (coarseness) of at least two days. The underlying target SI distribution is assumed to have a Gaussian distribution with mean muS and standard deviation sdS that have to be specified by the user. The code below can be used to generate $n=15$ SI windows from a Gaussian distribution with a mean of $3$ days and standard deviation of $2$ days. More details regarding the data generating mechanism can be found in the article.

set.seed(2025)
simdata <- simSI(muS = 3, sdS = 2, n = 15)
gt::gt(round(simdata, 2))

s	sl	sr	sw
4.24	3	5	2
3.07	2	4	2
4.55	4	6	2
5.54	5	7	2
3.74	3	5	2
2.67	2	4	2
3.79	3	5	2
2.84	2	4	2
2.31	1	5	4
4.40	3	6	3
2.21	1	3	2
-0.51	-2	1	3
2.16	1	3	2
4.53	2	5	3
5.13	3	7	4

The first column of the simulated dataset contains the true (unobserved) serial interval value generated from the chosen Gaussian distribution. The second and third columns contain the left and right bound of the SI window (sl and sr). Finally, the last column contains the width of the observed SI window, i.e. sw=sr-sl. The underlying target SI distribution is specified to be Gaussian with a mean of $3$ days and standard deviation of $2$ days. The 5th, 25th, 75th and 95th quantiles of the latter distribution are:

round(qnorm(p = c(0.05, 0.25, 0.75, 0.95), mean = 3, sd = 2), 1)

## [1] -0.3  1.7  4.3  6.3

We now create a data frame containing sl and sr and use the latter as an input in the estimSI() routine. Nonparametric estimates of different SI features can be accessed with $npestim$ .

xdf <- data.frame(sl = simdata$sl, sr = simdata$sr)
SIfit <- estimSI(x = xdf, nboot = 2000)
gt::gt(round(SIfit$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	3.4	1.7	0.3	2.5	3.5	4.6	6.0
se	0.4	0.3	1.1	0.4	0.3	0.4	0.5
ci90l	2.8	1.1	-1.3	1.8	2.9	3.9	5.0
ci90r	4.0	2.1	2.1	3.2	4.1	5.2	6.6
ci95l	2.7	1.1	-1.3	1.5	2.8	3.8	4.9
ci95r	4.2	2.2	2.1	3.3	4.2	5.4	6.6

The output shows point estimates (point), standard errors (se) and confidence intervals bounds (ci) for the serial interval mean, standard deviation (sd) and 5th, 25th, 50th, 75th and 95th quantiles denoted by q0.05, q0.25, etc. We can also plot the estimated cumulative distribution function (cdf) obtained with the nonparametric approach and compare it with the target Gaussian cdf. The quality of the fit will typically depend on the sample size $n$ and on the degree of coarseness present in the data. Note that the nonparametric methodology naturally deals with negative SI values.

sl <- simdata$sl
sr <- simdata$sr
Fhat <- function(s) (1 / SIfit$n) * sum((s - sl) / (sr - sl) * (s >= sl & s <= sr) + (s > sr))
sf <- seq(-3, 8, length = 100)
plot(sf, sapply(sf, Fhat), type = "l", lwd = 2, xlab = "Serial interval", ylab = "cdf")
grid()
lines(sf, pnorm(sf, mean = 3, sd = 2), col = "blue", lwd = 2)
legend("topleft", c("Estimated cdf of SI", "Target cdf of SI"), col = c("black", "blue"), lwd = c(2, 2), bty = "n")

Real data

Lessler et al. (2009) share a dataset containing serial interval windows obtained from $n=16$ infector-infectee pairs for Influenza A (2009 H1N1 influenza) at a New York City school. The SI windows are directly available from the supplementary appendix of the latter reference and are encoded in a data frame xNY. Nonparametric estimates of serial interval features are then obtained with estimSI().

xNY <- data.frame(sl = c(1, 1, 1, 0, 0, 4, 2, 3, 0, 3, 0, 3, 4, 1, 3, 3), sr = c(3, 3, 3, 2, 2, 6, 4, 5, 2, 5, 2, 5, 6, 3, 5, 5))
set.seed(123)
SIfitNY <- estimSI(xNY, nboot = 2000)
gt::gt(round(SIfitNY$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	2.8	1.5	0.4	1.5	2.8	4.1	5.2
se	0.3	0.1	0.2	0.4	0.6	0.4	0.3
ci90l	2.2	1.3	0.2	1.1	1.8	3.2	4.7
ci90r	3.4	1.7	1.1	2.2	3.7	4.6	5.6
ci95l	2.1	1.2	0.2	1.0	1.8	3.0	4.6
ci95r	3.5	1.7	1.1	2.5	3.8	4.7	5.7

Interested readers can find more real data examples in Gressani and Hens (2025) and learn about the strengths and limitations of this new nonparametric methodology for serial interval estimation.

References

Gressani, O. and Hens, N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Computational Biology 21(8):e101338. https://doi.org/10.1371/journal.pcbi.1013338

Gressani; O. (2025). EpiDelays: A Software for Estimation of Epidemiological Delays (version 0.0.1). https://github.com/oswaldogressani/EpiDelays

Lessler, J., Reich, N. G., Cummings, D. A., and the New York City Department of Health and Mental Hygiene Swine Influenza Investigation Team. (2009). Outbreak of 2009 pandemic influenza A (H1N1) at a New York City school. New England Journal of Medicine 361(27), 2628-2636. https://www.nejm.org/doi/full/10.1056/NEJMoa0906089

EpiLPS for estimation of incubation times

Thu, 01 Aug 2024 00:00:00 +0000

Motivation

A group of researchers from the Data Science Institute (DSI) at Hasselt University developed a new statistical model to estimate the incubation period of a pathogenic organism based on coarse data. The incubation period of an infectious disease (defined as the time elapsed between infection and the manifestation of first symptoms) is of great importance as it permits to shed light on the epidemic potential of a disease and to optimize the length of quarantine periods to freeze transmission. The article (Gressani et al. 2024) was recently published in the American Journal of Epidemiology with practical implementation of the methodology accessible through the EpiLPS package (Gressani et al. 2022).

Coarse data

What makes estimation of incubation times so challenging in the first place? The devil lies in the data. True infection times are stealthy and rarely observed. In information-theoretic jargon this phenomenon is called “imperfect information’’ but statisticians prefer to call it censoring. To be more precise, infection times are interval censored, i.e. one part of the collected data contains exposure intervals $\mathcal{E}=[t^{E_L},t^{E_R}]$ reported by individuals that are part of the study, where $t^{E_L}$ and $t^{E_R}$ stand for the left and right bound, respectively, of the exposure window. The other part of the data contains symptom onset times $t^{\mathcal{S}}$. This is a more easily accessible piece of information -people tend to remember the day when first symptoms appeared- and so the timing of symptom onset is assumed to be exactly observed. Subtracting the exposure bounds from the symptom onset time, one obtains the incubation interval $\mathcal{I}=[t^{\mathcal{I}_L}, t^{\mathcal{I}_R}]$ with lower bound $t^{\mathcal{I}_L}=t^{\mathcal{S}}-t^{E_R}$ and upper bound $t^{\mathcal{I}_R}=t^{\mathcal{S}}-t^{E_L}$, characterizing the coarse data structure which will be the main model input.

Simulated example

In EpiLPS, the estimIncub() routine is designed to compute an estimate of the incubation density based on the methodology of Gressani et al. (2024). Giving a detailed account of the methodology would be out of scope for this blog and the reader is redirected to the article for technicalities. In a nutshell, it is a Bayesian approach making use of (penalized) B-splines, Laplace approximations and Markov chain Monte Carlo (MCMC) methods to derive a semi-parametric estimate of the incubation density. An attractive feature of the estimIncub() routine for the end-user is the minimalistic input it requires to work, namely:

x: A data frame containing the lower and upper bound of the incubation interval.
K: An integer specifying the number of B-splines to smooth the incubation density.
niter: The number of MCMC samples required.

In practice, only the data frame x is required as the remaining inputs are assigned default values. Computationally, the routine requires a small amount of resources as costly subroutines are coded in C++ and integrated in R via the Rcpp package. The structure of x is quite simple. It should be a data frame with two columns containing the left bound $t^{\mathcal{I}_L}$ of the incubation interval (in the first column) and the right bound $t^{\mathcal{I}_R}$.

Let’s start with a simple example where x is simulated based on a data generating mechanism assuming a known incubation distribution. This can be achieved with the incubsim() routine in EpiLPS. We choose x to be generated according to a Lognormal incubation distribution with a mean of 5.5 days and a standard deviation of 2.1 days following Ferretti et al. (2020). Simulation of $n=40$ observations with an average exposure window of 2 days is implemented as follows:

set.seed(2024)
simdat <- incubsim(incubdist = "LogNormal", n = 40, coarseness = 2)
gt(head(simdat$Dobsincub, 5))

tL	tR
3.724500	5.510304
4.381377	6.654224
4.588020	5.483614
3.847835	6.003948
3.522282	5.033487

By typing simdat$Dobsincub, the user has access to the generated incubation intervals (expressed in days), corresponding here to a data frame with two columns and $n=40$ rows. This is the data frame that is injected in the estimIncub() routine:

fit <- estimIncub(x = simdat$Dobsincub, verbose = TRUE)

## ----------------------------------------------------------------
## Time elapsed: 1.339 seconds.
## Fitted density is Log-Normal with meanlog=1.617 and sdlog=0.317.
## Mean incubation period (days): 5.298 with 95% CI: 5.086-5.615.
## 95th percentile (days): 8.484 with 95% CI: 8.121-9.089.
## ----------------------------------------------------------------

The output in the R console can easily be interpreted. It tells us that the model chooses a Lognormal density fit for the incubation period with a mean of 5.3 days (95% CI: 5.0-5.6 days). More detailed summary statistics are accessible by typing fit$stats, such as the posterior standard deviation and additional percentiles. What happens under the hood? Basically, the model computes a semi-parametric fit to the data and compares it with classic parametric fits (Lognormal, Weibull and Gamma) used for incubation modeling. The candidate with the lowest Bayesian information criterion (BIC) wins the game and is finally selected (here the Lognormal distribution). The incubation windows and the fitted incubation density can be obtained by typing:

grid.arrange(plot(fit, typ = "incubwin"), plot(fit, type = "pdf"), nrow = 1)

Real data example

The flexible Bayesian methodology is illustrated on SARS-CoV-2 symptom onset and exposure window data extracted from cases in Vietnam. The dataset was analyzed in Bui et al. (2020) and is publicly available on the GitHub repository provided in the article (https://github.com/longbui/Covid19IncubVN; last accessed July 17, 2024). The dataset contains information about $n=19$ cases identified from January 23, 2020 to April 13, 2020. After continuity corrections (required to change calendar dates into continuous time points), the left and right incubation bounds are given by:

# Left incubation bound
tL <- c(0.504, 2.983, 5.343, 6.969, 6.990, 2.570, 6.870, 1.263, 0.693, 1.869, 1.151, 2.748, 1.209, 1.161, 4.982, 4.017, 2.170, 9.805, 1.659)

# Right incubation bound
tR <- c(1.491, 7.762, 11.777, 12.821, 11.359, 8.551, 24.954, 4.144, 6.408, 7.083, 4.191, 10.326, 9.942, 7.254, 12.690, 10.825, 14.478, 16.592, 18.649)

# Data set
dataVietnam <- data.frame(tL = tL, tR = tR)

The estimIncub() routine provides a Weibull fit with a mean incubation period of 6.7 days (95% CI: 5.8-7.4 days) and the standard deviation (extracted by typing incubfit$stats) is 3.3 days (95% CI: 3.0-3.8 days). A figure of the incubation windows and the fitted Weibull density (with 95% credible interval) is also provided.

set.seed(2024)
incubfit <- EpiLPS::estimIncub(x = dataVietnam, verbose = TRUE, tmax = 25)

## ----------------------------------------------------------------
## Time elapsed: 1.13 seconds.
## Fitted density is Weibull with shape=2.142 and scale=7.521.
## Mean incubation period (days): 6.661 with 95% CI: 5.753-7.375.
## 95th percentile (days): 12.552 with 95% CI: 11.931-14.087.
## ----------------------------------------------------------------

grid.arrange(plot(incubfit, typ = "incubwin"), plot(incubfit, type = "pdf"), nrow = 1)

References

Gressani, O., Torneri, A., Hens, N. and Faes, C. (2024). Flexible Bayesian estimation of incubation times. American Journal of Epidemiology (Accepted manuscript). https://doi.org/10.1093/aje/kwae192

Gressani, O., Wallinga, J., Althaus, C. L., Hens, N. and Faes, C. (2022). EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number. PLoS Comput Biol 18(10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618

Eddelbuettel, D. and Francois, R. (2011). Rcpp: Seamless R and C++ Integration. Journal of Statistical Software, 40(8), 1–18. https://doi.org/10.18637/jss.v040.i08

Ferretti, L. et al. (2020). Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science 368, eabb6936. https://doi.org/10.1126/science.abb6936

Bui LV, Nguyen HT, Levine H, Nguyen HN, Nguyen T-A, Nguyen TP, et al. (2020) Estimation of the incubation period of COVID-19 in Vietnam. PLoS ONE 15(12): e0243889. https://doi.org/10.1371/journal.pone.0243889

Google Analytics in R: Review of 2022

Fri, 16 Dec 2022 00:00:00 +0000

Introduction

It is almost the end of the year, which means it is time to do a review of Stats and R and look back on the past year.

This year’s review will be much shorter compared to previous years because there are already a lot of examples in those previous reviews.

Note that beginning of September 2022, I moved from Universal Analytics (known as UA) to Google Analytics 4 (known as GA4). As you may know if you often use Google Analytics; on July 1, 2023, standard Universal Analytics properties will stop processing new hits. This means that you will need to move to GA4 if you want to continue using it after this date.

This year, I will be brief about the different metrics and how to analyze your Google Analytics (GA) data. However, this post will perhaps be helpful for people who also moved (or will move) to GA4 and still want to analyze their GA data from both UA and GA4.

I have to admit that it is quite time consuming to combine data from both, in particular since metrics and dimensions are not the same between UA and GA4. So if you have not moved yet, I recommend doing it at the end or beginning of a year. This will prevent you from having to fetch data from two GA IDs, and then having to combine both data in R.

As usual, I analyze my GA data in R thanks to the amazing {googleAnalyticsR} package from Mark Edmondson.

Note that the actual number presented below will not be very useful for you. Every blog and website is different, so every audience is different. This post is more about illustrating the process of analyzing GA data in R (and this year, also to illustrate the process of combining GA data from two GA IDs), rather than about showing my numbers.

Prerequisites

Before anything else, you have to specify the GA ID for which you want to analyze data. Mine are as follows:

ga_id <- c("208126346", "250795226")

You will find your(s) in the admin section of your GA account.

Analytics

In this section, I present some visualizations that might be useful when analyzing your GA data. Feel free to comment at the end of the post if you use other interesting visualizations.

Note that I focus on the page views metrics in this post, but you can edit my code to show other metrics as well. See the metrics available for UA and the ones for GA4.

Page views over time

Let’s start with the evolution of the number of page views over the past year, so from December 16, 2021 to December 15, 2022.

library(googleAnalyticsR)

# set date range
start_date <- as.Date("2021-12-16")
end_date <- as.Date("2022-12-15")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
library(dplyr)
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# scatter plot with a trend line
library(ggplot2)
dat %>%
  ggplot(aes(x = date, y = page_views)) +
  geom_point(size = 1L, color = "gray") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see in the code above, we need to:

extract data from two different IDs: this is done with google_analytics() for UA and ga_data() for GA4 (with each function having its own arguments)
join data from the two IDs (with a full_join() because even after moving to GA4, there were still some hits on UA, so I need to keep data from the 2 IDs for the entire year)
sum up the metric for both IDs (with rowSums())

This process will be done repeatedly for each visualization.

From the plot above, we see that the number of page views follows a cyclical evolution (with a decrease during summer). This trend follows the same pattern than last year, but the increase in the last quarter of the year is larger in 2022.

Note that there seems to be consistently some days with a lower number of page views than the rest. This is actually the weekends. See more on that in this section. Note also the presence of an outlier with a number of page views above 5,000. No post has been published on that day, so it probably comes from an old post that was shared to a large audience.

Finally, summing all days give a total page views of 938,630. Last year, it was 876,280, so 2022 saw an increase of 7.12%.

Page views per month and year

Comparison of the number of page views per month for all previous years can also be useful. Below two different types of visualizations.

# set new date range to include previous years
start_date_launch <- as.Date("2019-12-16")

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date_launch, end_date),
  metrics = "pageviews",
  dimensions = "date",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date_launch, end_date),
  metrics = "screenPageViews",
  dimensions = "date",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)


# add year and month columns to dataframe
dat$month <- format(dat$date, "%m")
dat$year <- format(dat$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
dat %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(page_views = sum(page_views)) %>%
  ggplot(aes(x = month, y = page_views, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) # better y labels

Another possibility is as follows:

dat2 <- dat

library(lubridate)
dat2$day <- day(dat2$date)
dat2$day_month <- as.Date(paste0(dat2$month, "-", dat2$day), format = "%m-%d")

dat2 %>%
  filter(year != 2019 & page_views < 7500) %>% # remove 2019 and outliers
  ggplot(aes(x = day_month, y = page_views, color = year)) +
  geom_point(size = 1L, alpha = 0.25) + # change size and alpha of points
  geom_smooth(se = FALSE) + # remove confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    color = "" # remove legend title
  ) +
  theme(legend.position = "top") + # change legend position
  scale_y_continuous(labels = scales::comma) + # better y labels
  scale_x_date(date_labels = "%b") # show only months

From the 2 plots above, we see that the blog globally performed better in terms of page views compared to 2020-2021 (with an exception in the first quarter for which it performed better in 2021).

Note that a comparison between 3 years is fine, but with more years to compare the plots would quickly become unreadable!

Top performing pages

In case you are interested to know the top performing pages (still in terms of page views):

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "pageTitle",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "pageTitle",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "pageTitle")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)


most_viewed_posts <- dat %>%
  mutate(Title = str_sub(pageTitle, start = 1, end = -nchar(" - Stats and R"))) %>% # remove blog site in pageTitle
  count(Title, wt = page_views, sort = TRUE) %>%
  mutate(Title = str_trunc(Title, width = 30)) # keep maximum 30 characters

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Top performing posts are quite similar than last year, that is:

Since they are the most read posts for several years, I have made them available to download via Gumroad.

Page views by country

Knowing from which country your readers come from can also be useful, in particular for e-commerce or blogs that sell physical products in addition to writing posts.

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = "country",
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "country")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)

# table
countries <- dat %>%
  mutate(Country = str_trunc(country, width = 30)) %>% # keep maximum 30 characters
  count(Country, wt = pageviews, sort = TRUE)

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

For Stats and R, most readers are from the US, and to a large extent since the second top country (UK) is quite far in terms of page views.

This trend is the same than last years, except that:

UK has surpassed India, and
Belgium (which used to rank 3rd and 6th in 2020 and 2021, respectively) is now only 8th. Since I am from Belgium, this indicates that the blog is attracting more and more international readers through the years (and proportionally less and less from my country).

Page views per day of week

As discussed in the section about the evolution of daily page views (view this section), there seem to be some days which consistently perform worse than other days.

The following plot will show what are these days:

# extract data from both IDs
dat1 <- google_analytics(ga_id[1],
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("date"),
  anti_sample = TRUE # avoid sampling
)

dat2 <- ga_data(ga_id[2],
  date_range = c(start_date, end_date),
  metrics = "screenPageViews",
  dimensions = c("date"),
  limit = -1 # return all data (no limit)
)

# combine data from both IDs
dat <- full_join(dat1, dat2, by = "date")
dat$page_views <- rowSums(select(dat, pageviews, screenPageViews),
  na.rm = TRUE
)
# find day of week
dat$weekday <- wday(dat$date, label = TRUE, abbr = TRUE)

## Reordering dat$weekday so Monday is first
dat$weekday <- factor(dat$weekday,
  levels = c("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
)

# boxplot
library(scales)
dat %>%
  ggplot(aes(x = weekday, y = page_views)) +
  geom_boxplot(fill = "steelblue", outlier.colour = alpha(0.25)) +
  geom_jitter(alpha = 0.25) + # adds transparency
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As expected for a technical blog, there are much more readers during the week than during the weekend.

Thank you note

Thank you to all readers who came to Stats and R this year. You made this journey incredibly more enriching. For next year and the many more to come, I will keep writing about topics for which I am familiar and interested in. So stay tuned!

Thanks for reading. I hope this article helped you to analyze your Google Analytics data in R, or helped you to combine your Universal Analytics and Google Analytics 4 data. For more examples of visualizations or summaries of your GA data in R, see also previous years’ reviews.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Paper: 'EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number'

Wed, 19 Oct 2022 00:00:00 +0000

Introduction

A colleague (and friend) of mine recently published a research paper entitled “EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number” in PLoS Computational Biology.

I am not in the habit of sharing research paper to which I did not contribute. Nevertheless, I would like to make an exception with this one because I strongly believe that the method developed in the paper deserves to be known, especially for anyone working in epidemiology.

Below is the motivation behind the article, as well as an illustration on simulated and real data (US hospitalization data). More information can be found in the paper and on the accompanying website.

Motivation

EpiLPS (Gressani et al. 2022) is a methodology for flexible Bayesian inference of the time-varying reproduction number $\mathcal{R}_t$; the average number of secondary cases generated by an infected agent at time $t$. This is a key epidemiological parameter that informs about the transmission potential of an infectious disease and can be used by public health authorities to gauge the effectiveness of interventions and propose an orientation for future control strategies.

This metric has gained in popularity during the SARS-CoV-2 pandemic with wide media coverage as its meaning is easily and intuitively grasped. Put simply, when $\mathcal{R} < 1$, the signal is encouraging as the epidemic is under control and will eventually vanish. On the contrary, a value of $\mathcal{R} > 1$ means that the disease keeps spreading and infections are witnessing an expansionary impact. Having a robust and reliable tool to compute the reproduction number from infectious disease data is therefore crucial.

A group of researchers in the EpiPose team from Hasselt University (Belgium), Leiden University (The Netherlands), and the University of Bern (Switzerland) have recently developed a new methodology for estimating the instantaneous reproduction number from incidence time series data for a given serial interval distribution (the time elapsed between the onset of symptoms in an infector and the onset of symptoms of secondary cases).

They termed their approach EpiLPS for “Epidemiological modeling with Laplacian-P-Splines” as Laplace approximations and P-splines smoothers are key ingredients that form the backbone of the proposed methodology.

The EpiLPS model assumes that the observed reported cases (by reporting date or date of symptom onset) are governed by a negative binomial distribution. As such, it allows to take the feature of overdispersion into account, contrary to a Poisson model. The epidemic curve is smoothed with P-splines (where posterior estimates of latent variables are computed via Laplace approximations) in a first step and a renewal equation model is used in a second step as a bridge between the reproduction number and the estimated spline coefficients through a “plug-in” method.

The authors also explain the main difference between EpiLPS and EpiEstim, a well established approached for estimating $\mathcal{R}_t$ in real-time developed by Cori et al. (2013) and make extensive comparisons between the two approaches under different epidemic scenarios.

An interesting feature of EpiLPS is that the user can choose between a fully “sampling-free” path, where model hyperparameters are fixed at their maximum a posteriori (LPSMAP) or a fully stochastic path (LPSMALA) based on a Metropolis-adjusted Langevin algorithm (LPSMALA). Talking about efficiency, routines for Laplace approximations and B-splines evaluations have been coded in C++ and integrated in R via the Rcpp package, so that the underlying algorithm can be executed in negligible time.

Below, we provide a short example of how to use the EpiLPS routines to estimate $\mathcal{R}_t$.

Getting started

The EpiLPS package is available from CRAN (see https://cran.r-project.org/web/packages/EpiLPS/index.html) and can be installed from the R console by typing:

install.packages("EpiLPS")

The package can then be loaded as follows:

library("EpiLPS")

The EpiLPS package structure is fairly simple as it consists in a few routines:

The function epilps() is the core routine for fitting the reproduction number.
With plot.epilps(), the user can plot the estimated epidemic curve and $\mathcal{R}_t$.
Finally, two ancillary routines, episim() and perfcheck() have been developed to essentially reproduce the simulation results of the associated paper.

A simulated example

A set of epidemic data can be simulated with the episim() routine by specifying a serial interval distribution and by choosing among a set of available patterns for the true reproduction number curve (here we choose pattern number 5 corresponding to a rather wiggly curve).

The simulated outbreak is for a duration of 40 days as specified in the endepi option. By setting the option plotsim = TRUE, the routine returns a figure summarizing the incidence time series, a bar plot for the specified serial interval distribution and the true underlying reproduction number curve.

set.seed(1234)

SI <- c(0.344, 0.316, 0.168, 0.104, 0.068)
simepidemic <- episim(
  serial_interval = SI,
  Rpattern = 5,
  plotsim = TRUE,
  verbose = TRUE,
  endepi = 40
)

## Chosen scenario: 5 'Wiggly then stable Rt'.
## Incidence of cases generated from a Poisson distribution. 
## Total number of days of epidemic: 40.

If you want to have an overview of the generated incidence time series, just type:

simepidemic$y

##  [1]  10   6  15  24  37  43  54  46  47  28  20   8  10  10   3   5   3   4   6
## [20]   6  15  21  44  75 135 217 329 409 453 487 457 443 297 290 255 246 246 339
## [39] 395 573

Smoothing the epidemic curve and estimating $\mathcal{R}_t$

Let us now use the epilps() routine to smooth the epidemic curve and estimate the reproduction number.

We will do this through LPSMAP (a fully sampling-free approach) and via LPSMALA (a fully stochastic approach relying on a MCMC algorithm with Langevin dynamics), where we specify a chain of length 10000 and a burn-in of size 4000.

LPSMAP_fit <- epilps(
  incidence = simepidemic$y,
  serial_interval = SI,
  tictoc = TRUE
)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.327.
## Mean 95% CI of Rt discarding first 7 days: (1.164,1.527) 
## Elapsed real time (wall clock time): 0.261 seconds.

LPSMALA_fit <- epilps(
  incidence = simepidemic$y, serial_interval = SI,
  method = "LPSMALA", chain_length = 10000, burn = 4000
)

## Inference method chosen: LPSMALA with chain length 10000 and warmup 4000.
## MCMC acceptance rate: 56.41%. 
## Geweke z-score < 2.33 for:  32 / 33  variables. 
## Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.326.
## Mean 95% CI of Rt discarding first 7 days: (1.117,1.555). 
## Timing of routine not requested.

After execution, each routine prints in the console a brief summary of the method that has been requested by the user.

For LPSMALA, it summarizes the chain length, the acceptance rate (should be around 56%) and other basic information. As can be seen from the printed output, the mean reproduction number for the simulated epidemic is around 1.33.

We can now use, say, the LPSMALA_fit object together with the plot() routine to obtain the smoothed epidemic curve and the estimated reproduction number (by default the credible interval is at a 5% level of significance but this can be changed by the user).

days <- seq(8, 40)

#--- Smoothed epidemic curve
gridExtra::grid.arrange(
  plot(LPSMALA_fit,
    plotout = "epicurve", incibars = TRUE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "Smoothed epidemic curve", titlesize = 13, barwidth = 0.25
  ),

  #--- Estimated reproduction number
  plot(LPSMALA_fit,
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "Estimated R (LPSMALA)"
  ),
  nrow = 1, ncol = 2
)

The figure can be customized in various ways:

Users can specify the theme under themetype. Available options are gray (the default), classic, light and dark.
Other choices, such as whether or not to show the incidence bars, the color of the credible interval envelope, the color of the smoothed epidemic curve and the estimated reproduction number are also available.

The figure above was generated within the ggplot2 package, but there is also another way of extracting information directly from the LPSMAP_fit and LPSMALA_fit objects. In fact, the estimated reproduction number values and their associated credible interval for each day can be extracted and plotted.

Below, we make the exercise and plot the estimated $\mathcal{R}_t$ obtained with LPSMAP and LPSMALA, respectively and compare it with the true underlying reproduction number curve. The fit is quite good.

par(mfrow = c(1, 2))

#--- LPSMAP vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMAP_fit$epifit$R95CI_low[8:40],
    rev(LPSMAP_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(0.23, 0.54, 1, 0.3), border = NA
)
lines(days, LPSMAP_fit$epifit$R_estim[8:40], type = "l", col = "cornflowerblue", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "blue", rgb(0.23, 0.54, 1, 0.3)),
  c("Target R", "LPSMAP", "LPSMAP 95% CI"), bty = "n", cex = 0.9
)

#--- LPSMALA vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMALA_fit$epifit$R95CI_low[8:40],
    rev(LPSMALA_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(1, 0.23, 0.31, 0.3), border = NA
)
lines(days, LPSMALA_fit$epifit$R_estim[8:40], type = "l", col = "red", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "red", rgb(1, 0.23, 0.31, 0.3)),
  c("Target R", "LPSMALA", "LPSMALA 95% CI"), bty = "n", cex = 0.9
)

You can access, say, the results of the last week of the epidemic by typing:

# Estimated R of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 1:4], 7), 3)

##    Date R_estim R95CI_low R95CI_up
## 34   34   0.708     0.663    0.756
## 35   35   0.724     0.676    0.775
## 36   36   0.809     0.755    0.868
## 37   37   0.971     0.908    1.039
## 38   38   1.205     1.135    1.279
## 39   39   1.461     1.384    1.541
## 40   40   1.671     1.557    1.794

# Estimated mean number of cases of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 5:7], 7))

##    mu_estim mu95CI_low mu95CI_up
## 34      284        225       357
## 35      254        202       319
## 36      248        196       312
## 37      267        211       338
## 38      319        253       404
## 39      411        325       520
## 40      552        394       774

USA hospitalization data

To illustrate EpiLPS on real data, we download hospitalization data from the COVID19 package for the USA in the period ranging from 2021-09-01 to 2022-09-01 and apply the epilps() routine to estimate the reproduction number.

install.packages("COVID19")
library("COVID19")

# Get data and specify serial interval distribution
USADat <- COVID19::covid19(
  country = "US", level = 1, start = "2021-09-01",
  end = "2022-09-01", verbose = FALSE
)

si <- c(0.344, 0.316, 0.168, 0.104, 0.068)

inciUSA <- USADat$hosp
dateUSA <- USADat$date

We use the epilps() routine with method LPSMAP (default) and plot the smoothed epidemic curve and the estimated reproduction number with a 95% credible interval.

epifit <- epilps(incidence = inciUSA, serial_interval = si, K = 20)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 366. 
## Mean Rt discarding first 7 days: 0.994.
## Mean 95% CI of Rt discarding first 7 days: (0.983,1.005) 
## Timing of routine not requested.

gridExtra::grid.arrange(
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "epicurve", incibars = FALSE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "USA smoothed epidemic curve", titlesize = 13
  ),
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "USA Estimated R"
  ),
  nrow = 1, ncol = 2
)

Conclusion

Thanks for reading.

I hope you will find the method developed in the paper as useful as I do. Feel free to reach out to me and to the authors if you happen to use it for your own research.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Cori, Anne, Neil M Ferguson, Christophe Fraser, and Simon Cauchemez. 2013. “A New Framework and Software to Estimate Time-Varying Reproduction Numbers During Epidemics.” American Journal of Epidemiology 178 (9): 1505–12.

Gressani, Oswaldo, Jacco Wallinga, Christian L Althaus, Niel Hens, and Christel Faes. 2022. “EpiLPS: A Fast and Flexible Bayesian Tool for Estimation of the Time-Varying Reproduction Number.” PLoS Computational Biology 18 (10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618.

Stats and R is 2 years old!

Thu, 16 Dec 2021 00:00:00 +0000

Introduction

Stats and R has been launched exactly two years ago. Like last year, I think it is a good time to do a review of the past 12 months by sharing some figures about the audience of the blog.

This article is not about showing off my numbers, but rather a way to illustrate how to analyze your blog or your website’s traffic using Google Analytics data. Figures regarding the audience of my blog is probably useless to you (and I believe, should not be compared with). However, the code used in this post can be reused for your own blog or website (provided you also use Google Analytics to track your audience).

Note that I use the {googleAnalyticsR} R package to analyze my blog’s Google Analytics data. If you are unfamiliar with this package, see the prerequisites first.

If you have already used that package, you can select your account as followed (make sure to edit the code with your own property name):

library(googleAnalyticsR)

accounts <- ga_account_list()

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

On top of that, I also assume that you have basic knowledge of {ggplot2}—a popular R package to draw nice plots and visualizations.

Analytics

Last year, I mainly focused on the number of sessions. This year, I mainly concentrate on the number of page views to illustrate a different metrics.

For your information, a session is a group of user interactions with your website that take place within a given time frame, whereas a page view, as the name suggests, is defined as a view of a page on your site.

You can always change the metrics by editing metrics = c("pageviews") in the code below. See all available metrics provided by Google Analytics in this article.

Users and page views

As for last year’s review, let’s start with some general numbers, such as the number of users and page views for the entire site.

Note that we analyze traffic over the last year only so we extract data from December 16, 2020 to yesterday (December 15, 2021):

# set date range
start_date <- as.Date("2020-12-16")
end_date <- as.Date("2021-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users pageviews
## 1 549360    876280

Over this past year, Stats and R has attracted 549,360 users (number of new and returning people who visited the site), who generated a total of 876,280 page views.

That is an average of 2401 page views per day in 2021, compared to 1,531 page views per day in 2020 (an increase of 56.81%).

Page views over time

One of the first interesting metrics to analyze your blog’s audience is the evolution of traffic over time.

The daily number of page views over time can be presented in a scatterplot—together with a smoothed line—to analyze the evolution of the audience of your blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = pageviews)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "steelblue", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

Although the number of page views varies quite a bit (with an outlier at more than 5,000 page views in a day and as low as less than 1,000 page views for some days), it seems to be cyclical with a dip during summer. It is worth noting that the same dip appeared last year, probably due to the fact that people are less likely to read posts about statistics and R during summer holidays.

So if you write about technical stuff in your blog, low numbers during summer may be expected and does not necessarily mean something is broken on your website.

Page views per channel

Knowing how people come to your blog is also a pretty important factor.

Here is how to visualize the evolution of daily page views per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "pageviews",
  pivots = pivot_ga4("medium", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Page_views, -Date)

# Build up the line plot
trend_long %>%
  filter(Channel != "Total") %>%
  ggplot() +
  aes(x = Date, y = Page_views, group = Channel) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Page views",
    x = "",
    title = "Evolution of daily page views per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As we can see from the plot above, the large majority of page views come from the organic channel (so from search engines such as Google, Bing, etc.), with some peaks from referral (mainly from R-bloggers and RWeekly) when an article is published. (By the way, you can always subscribe to the newsletter if you want to be informed by email when a new post goes out.)

If you happen to write tutorials, you can also expect that most visitors come from the organic channel. If you are very present on social media, you will most likely attract more visitors from the social channel.

Page views per day of week and month of year

As seen in the previous plot, there are many ups and downs and traffic seems to be cyclical.

To investigate this further, we draw a boxplot of the number of page views for each day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We can also compute the sum and the mean number of page views per day to have a numerical summary instead of a plot:

# compute sum
dat_sum <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ dayOfWeek,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Day of week", "Sum", "Mean")

# display table
dat_summary

##   Day of week    Sum     Mean
## 1      Monday 141115 2713.750
## 2     Tuesday 143145 2752.788
## 3   Wednesday 146472 2763.623
## 4    Thursday 140712 2706.000
## 5      Friday 128223 2465.827
## 6    Saturday  85766 1649.346
## 7      Sunday  90847 1747.058

As expected, there are more readers during the week compared to the weekends.

The same analysis can be done for each month of the year instead of days of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = c("month", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Boxplot
gadata %>%
  ggplot(aes(x = month, y = pageviews)) +
  geom_boxplot(fill = "steelblue") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

# compute sum
dat_sum <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = sum
)

# compute mean
dat_mean <- aggregate(pageviews ~ month,
  data = gadata,
  FUN = mean
)

# combine both in one table
dat_summary <- cbind(dat_sum, dat_mean[, 2])

# rename columns
names(dat_summary) <- c("Month", "Sum", "Mean")

# display table
dat_summary

##    Month   Sum     Mean
## 1     01 68534 2210.774
## 2     02 80953 2891.179
## 3     03 92629 2988.032
## 4     04 88679 2955.967
## 5     05 81739 2636.742
## 6     06 64460 2148.667
## 7     07 49772 1605.548
## 8     08 46389 1496.419
## 9     09 68046 2268.200
## 10    10 79237 2556.032
## 11    11 77745 2591.500
## 12    12 78097 2519.258

From the plots and the numerical summaries, it is clear that the number of page views is not the same between the days of the week and the months of the year.

Page views per month and year

If you have data over more than a year, it could be useful to compare your monthly blog’s traffic over the years.

With the following code, we create a barplot of the number of daily page views per month and year:

# set new date range
start_date_launch <- as.Date("2019-12-16")

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date_launch, end_date),
  metrics = c("pageviews"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# page views by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  filter(year != 2019) %>% # remove 2019 because there are data for December only
  group_by(year, month) %>%
  summarize(pageviews = sum(pageviews)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = pageviews, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "Month",
    title = "Page views per month and year",
    subtitle = paste0(format(start_date_launch, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of page views over the months, but more importantly, compare this evolution across different years.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views per pages.

It is true that the top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

However, for the interested reader, here is how to get the data in R (note that you can change n = 7 in the code below to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)

head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title      n
## 1    Outliers detection in R - Stats and R 119747
## 2 Descriptive statistics in R - Stats a... 109473
## 3 Variable types and examples - Stats a...  83025
## 4 Correlation coefficient and correlati...  65703
## 5 Chi-square test of independence in R ...  62100
## 6 The complete guide to clustering anal...  40440
## 7                 ANOVA in R - Stats and R  32914

If like me you prefer a visualization over a table, here is how to draw this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Page title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 17.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. The top 3 articles in the past year were:

Be careful that this ranking is based on the total number of page views over the last 12 months. A recent article may thus be found at the bottom of the list simply because it collected page views over a shorter period of time compared to an old article. So it is best to avoid comparing recent articles with older ones, or you can compare articles after having “time-normalized” the number of page views. See previous year’s review for more details and illustrations of this metrics.

You could also be interested in knowing the worst performing ones (to eventually improve them or include them in higher quality posts):

tail(subset(most_viewed_posts, n > 1000),
  n = 7
)

##                                       Title    n
## 49 A guide on how to read statistical ta... 1477
## 50                      About - Stats and R 1470
## 51 How to embed a Shiny app in blogdown?... 1455
## 52                   About me - Stats and R 1277
## 53 One-proportion and chi-square goodnes... 1226
## 54 Running pace calculator in R Shiny - ... 1049
## 55                      Shiny - Stats and R 1018

Note that I intentionally excluded pages with less than 1000 views to remove deleted or hidden pages from the ranking.

Another issue with this ranking is that it may be biased due to some pages which have been duplicated (if you edited the title for example), but at least you have a broad idea of the worst performing pages.

Page views by country

Knowing the country where your readers come from may also be handy for some content creators or marketers.

Location of my readers is not really important for me because I intend to write for everyone, but this may be completely the opposite if you are selling things or running a business/ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)

head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 244752
## 2           India  71797
## 3  United Kingdom  53272
## 4         Germany  38322
## 5          Canada  33812
## 6         Belgium  26421
## 7       Australia  26225
## 8     Philippines  23193
## 9     Netherlands  21944
## 10         Brazil  17956

Again, if you prefer a plot over a table, you can visualize the top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that a large share of readers are from the US—and Belgium (my country) comes only in $6^{th}$ place in terms of number of page views.

Be careful that, as the number of people located in different countries differs widely, this ranking may hide some insights if you are comparing page views by countries in absolute terms. See why in this section of last year’s review.

User engagement by devices

One may also be interested in checking how users are engaged depending on device’s type. To investigate this, we plot 3 charts describing:

How page views are distributed by type of device?
The average time on page (in seconds) by type of device
The number of page views per session by device type

So first, how page views are distributed by device type?

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot pageviews by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, pageviews)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Page views",
    x = "",
    title = "Page views per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the large majority of readers visited the blog from a desktop and only a very small proportion comes from a tablet.

This makes sense since I guess many visitors are reading my articles or tutorials while using R (which is only available on desktop).

However, this information of total number of page views per device type does not tell me anything about the time spent on each page and thus the engagement by device type. The following plot answers this question:

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 125,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 3,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

Most readers coming from a tablet actually leave the page very quickly (see the peak around 0 second in the tablet facet).
Distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 125 seconds (= 2 minutes and 5 seconds) and 375 seconds (= 6 minutes and 15 seconds).
Quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (median = 316 seconds on mobile and 266 seconds on desktop, see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop (as shown by the total number of page views by device), it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article more carefully and try to reproduce the code on their own computer. At least that is what I do when I read blogs on mobile versus reading them on desktop. What is even more intriguing, is that it was already the case last year. If someone finds similar results and have a possible explanation, I would be glad to hear from her (if possible, in the comments at the end of the article so everyone can benefit from the discussion).
As a side note, we see that these medians are higher this year compared to last year (266, 316 and 114 seconds in 2021 compared to 190, 228 and 106 seconds in 2020 on desktop, mobile and tablet, respectively). This is somewhat encouraging because it indicates that people spend more time on each page (which is an indication, at least partially, of the quality of the blog for Google).¹

Given this result, I also believe it is helpful to illustrate the number of page views during a session by device type.

In fact, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session (remember that a session is a set of interactions with your website that take place within a given time frame).

We verify this belief via a density plot, and for better readability we exclude visits from tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views per session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nDashed lines represent the medians",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  )

This last plot shows that readers on desktop and mobile visit approximately the same number of pages per session, as indicated by the fact that the two distributions overlap each other and are not distant from each other. It is true that the median is higher for people on desktop than on mobile, but to a very small margin only (and the difference between the two is smaller than in last year’s review).

So to summarize what we learned based on the 3 last plots, we now know that

most readers visited the blog from desktop;
readers on mobile spent more time on each page than readers on desktop (and even more compared to readers on tablet);
users on desktop and mobile seem to have visited approximately the same number of pages per session.

One may wonder why I chose to compare medians instead of means. The main reason is that the median is a more robust way to represent non-normal data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.

Browser information

From a more technical perspective, you could also be interested in the number of page views by browser. I am personally not really interested in knowing which browser my visitors are using the most (mostly because this blog is available on all common browsers), but the most geeky among you may be so.

This information can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most readers visited the site using Chrome, Safari and Firefox (this was expected since they are the most common browsers).

This was the last metrics presented in this review. Of course, many more are possible depending on your R skills (mainly, data manipulation and {ggplot2}) and your expertise in SEO or analyzing Google Analytics data. Hopefully, thanks to this review and possibly from last year too, you will be able to analyze your own blog or website using R and the {googleAnalyticsR} package. At least, this was the aim of the present article.

For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.²

End note

I would also like to add that these figures are not done to be compared with. Every website or blog is unique, every author is unique (with different priorities and different agendas) and more is not always better. I learn a lot thanks to this blog, I use it for personal purposes and for my students as part of my teaching tasks. I will keep writing on it as long as I enjoy it and as long as I have the time to do so, not matter how low or high the number of clicks.

Thanks to all readers of the past year, and see you in a year for another review! In the meantime, if you maintain a blog I would be really happy to hear how you track its performance.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

More time spend on each page is a favorable factor for Google because it means that people are reading it more carefully. If the blog or the post is of mediocre quality, users tend to bounce back quickly (known as bounce rate) and look for an answer to their question somewhere else (leading ultimately to less time spend on the page or site).↩︎
Thanks to the RStudio blog for the inspiration.↩︎

How to track the performance of your blog in R?

Wed, 16 Dec 2020 00:00:00 +0000

Introduction

Stats and R has been launched on December 16, 2019. Since the blog is officially one year old today and after having discussed the main benefits of maintaining a technical blog, I thought it would be a good time to share some numbers and thoughts about it.

In this article, I show how to analyze a blog and its blog posts with the {googleAnalyticsR} R package (see package’s full documentation). After sharing some analytics about the blog, I will also discuss about content creation/distribution and, to a smaller extent, the future plans. This is a way to share my journey as a data science blogger and a way to give you an insight about how Stats and R is doing.

I decided to share with you some numbers through the {googleAnalyticsR} package instead of the regular Google Analytics dashboards for several reasons:

There are plenty of data analysts who are much more experienced than me when it comes to analyzing Google Analytics data via their dedicated platform
I recently discovered the {googleAnalyticsR} package in R and I would like to present its possibilities, and perhaps convince marketing specialists familiar with R to complement their Google Analytics dashboards with some data visualizations made in R (via some ggplot2 visualizations for instance)
I would like to automate the process such in a way that I can easily replicate the same types of analysis across the years. This will allow to see how the blog evolves throughout the years. We know that using R is a pretty good starting point when it comes to automation and replication—especially thanks to R Markdown reports

I am not an expert in the field of digital marketing, but who knows, it may still give some ideas to data analysts, SEO specialists or other bloggers on how to track the performance of their own blog or website using R. For those of you who are interested in a more condensed analysis, see my custom Google Analytics dashboard.¹

Before going further, I would like to remind that I am not making a living from my blog (far from it!) and it is definitely not my goal as I do not believe that I would be the same kind of writer if it was my main occupation.

Prerequisites

As for any package in R, we first need to install it—with install.packages()—and load it—with library():²

# install.packages('googleAnalyticsR', dependencies = TRUE)
library(googleAnalyticsR)

Next, we need to authorize the access of the Google Analytics account using the ga_auth() function:

ga_auth()

Running this code will open a browser window on which you will be able to authorize the access. This step will save an authorization token so you only have to do it once.

Make sure to run the ga_auth() function in a R script and not in a R Markdown document. Follow this procedure if you want to use the package and its functions in a R Markdown report or in a blog post like I did for this article.

Once we have completed the Google Analytics authorization, we will need the ID of the Google Analytics account we want to access. All the available accounts linked to your email address (after authentication) are stored in ga_account_list():

accounts <- ga_account_list()

accounts

## # A tibble: 2 × 10
##   accountId account…¹ inter…² level websi…³ type  webPr…⁴ webPr…⁵ viewId viewN…⁶
##   <chr>     <chr>     <chr>   <chr> <chr>   <chr> <chr>   <chr>   <chr>  <chr>  
## 1 86997981  Antoine … 129397… STAN… https:… WEB   UA-869… Antoin… 13318… All We…
## 2 86997981  Antoine … 218214… STAN… https:… WEB   UA-869… statsa… 20812… All We…
## # … with abbreviated variable names ¹accountName, ²internalWebPropertyId,
## #   ³websiteUrl, ⁴webPropertyId, ⁵webPropertyName, ⁶viewName

accounts$webPropertyName

## [1] "Antoine Soetewey" "statsandr.com"

As you can see I have two accounts linked to my Google Analytics profile: one for my personal website (antoinesoetewey.com) and one for this blog.

Of course, I select the account linked to this blog:

# select the view ID by property name
view_id <- accounts$viewId[which(accounts$webPropertyName == "statsandr.com")]

Make sure to edit the code with your own property name.

We are now finally ready to use our Google Analytics data in R for a better analysis!

Analytics

Users, page views and sessions

Let’s start with some general numbers, such as the number of users, sessions and page views for the entire site. Note that for the present article, we use data over the past year, so from December 16, 2019 to December 15, 2020:

# set date range
start_date <- as.Date("2019-12-16")
end_date <- as.Date("2020-12-15")

# get Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("users", "sessions", "pageviews"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

gadata

##    users sessions pageviews
## 1 321940   428217    560491

In its first year, Stats and R has attracted 321,940 users, who generated a total of 428,217 sessions and 560,491 page views (that is an average of 1531 page views per day).

For those unfamiliar with Google Analytics data and the difference between these metrics, remember that:

a user is the number of new and returning people who visit your site during a set period of time
a session is a group of user interactions with your website that take place within a given time frame
a page view, as the name suggests, is defined as a view of a page on your site

So if person A reads three blog posts then leave the site and person B reads one blog post, your about page then leave the site, Google Analytics data will show 2 users, 2 sessions and 5 page views.

Sessions over time

In addition to the rather general metrics presented above, it is also interesting to illustrate the daily number of sessions over time in a scatterplot—together with a smoothed line—to analyze the evolution of the blog:

# get the Google Analytics (GA) data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"), # edit for other metrics
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# load required libraries
library(dplyr)
library(ggplot2)

# scatter plot with a trend line
gadata %>%
  ggplot(aes(x = date, y = sessions)) +
  geom_point(size = 1L, color = "steelblue") + # change size and color of points
  geom_smooth(color = "darkgrey", alpha = 0.25) + # change color of smoothed line and transparency of confidence interval
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme(plot.margin = unit(c(5.5, 15.5, 5.5, 5.5), "pt")) + # to avoid the plot being cut on the right edge
  scale_y_continuous(labels = scales::comma) # better y labels

(See how to draw plots with the {ggplot2} package, or with the {esquisse} addin if you are not familiar with the package.)

As you can see, there was a huge peak of traffic around end of April, with almost 12,000 users in a single day. Yes, you read it well and there is no bug. The blog post “A package to download free Springer books during Covid-19 quarantine” went viral and generated a massive traffic for a few days. The daily number of sessions returned to a more normal level after a couple of days. We also observe an upward trend in the last months (since end of August/beginning of September), which indicates that the blog is growing in terms of number of daily sessions.

Note that I decided to focus on the number of sessions and the number of page views in this section and the following ones, but you can always change to your preferred metrics by editing metrics = c("sessions") in the code. See all available metrics provided by Google Analytics in this article.

Sessions per channel

Knowing how people come to your blog is a pretty important factor. Here is how to visualize the evolution of daily sessions per channel in a line plot:

# Get the data
trend_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  dimensions = c("date"),
  metrics = "sessions",
  pivots = pivot_ga4("medium", "sessions"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# edit variable names
names(trend_data) <- c("Date", "Total", "Organic", "Referral", "Direct", "Email", "Social")

# Change the data into a long format
library(tidyr)
trend_long <- gather(trend_data, Channel, Sessions, -Date)

# Build up the line plot
ggplot(trend_long, aes(x = Date, y = Sessions, group = Channel)) +
  theme_minimal() +
  geom_line(aes(colour = Channel)) +
  labs(
    y = "Sessions",
    x = "",
    title = "Evolution of daily sessions per channel",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

We see that a large share of the traffic is from the organic channel, which indicates that most readers visit the blog after a query on search engines (mostly Google). In my case, where most of my posts are tutorials and which help people with specific problems, it is thus not a surprise that most of my traffic comes from organic search.

We also notice some small peaks of sessions generated from the referral and direct channels, which are probably happening on the date of publication of each article.

We also see that there seems to be a recurrent pattern of ups and downs in the number of daily sessions. Those are weekly cycles, with less readers during the weekend and which indicates that people are working on improving their statistical or R knowledge mostly during the week (which actually makes sense!).

Sessions per day of week

As shown above, traffic seems to be different depending on the day of week. To investigate this further, we draw a boxplot of the number of sessions for every day of the week:

# get data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "sessions",
  dimensions = c("dayOfWeek", "date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Recoding gadata$dayOfWeek following GA naming conventions
gadata$dayOfWeek <- recode_factor(gadata$dayOfWeek,
  "0" = "Sunday",
  "1" = "Monday",
  "2" = "Tuesday",
  "3" = "Wednesday",
  "4" = "Thursday",
  "5" = "Friday",
  "6" = "Saturday"
)

## Reordering gadata$dayOfWeek to have Monday as first day of the week
gadata$dayOfWeek <- factor(gadata$dayOfWeek,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

# Boxplot
gadata %>%
  ggplot(aes(x = dayOfWeek, y = sessions)) +
  geom_boxplot() +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

As you can see there are some outliers, probably due (in part at least) to the article that went viral. For the sake of illustration, below the same plot after removing points considered as potential outliers according to the interquartile range (IQR) criterion (i.e., points above or below the whiskers), and after a couple of visual improvements:

# boxplot
gadata %>%
  filter(sessions <= 3000) %>% # filter out sessions > 3,000
  ggplot(aes(x = dayOfWeek, y = sessions, fill = dayOfWeek)) + # fill boxplot by dayOfWeek
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per day of week",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 3,000 excluded"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(legend.position = "none") # remove legend

After excluding data points above 3,000, it is now easier to see that the median number of sessions (represented by the horizontal bold line in the boxes) is the highest on Wednesdays, and lowest on Saturdays and Sundays.

The difference in sessions between weekdays is, however, not as large as expected.

Sessions per day and time

We have seen the traffic per day of week. The example below shows a visualization of traffic, broken down this time by day of week and hour of day in a density plot. In this plot, the device type has also been added for more insights.

## Get data by deviceCategory, day of week and hour
weekly_data <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("deviceCategory", "dayOfWeekName", "hour"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Manipulation using dplyr
weekly_data_sessions <- weekly_data %>%
  group_by(deviceCategory, dayOfWeekName, hour)

## Reordering weekly_data_sessions$dayOfWeekName to have Monday as first day of the week
weekly_data_sessions$dayOfWeekName <- factor(weekly_data_sessions$dayOfWeekName,
  levels = c(
    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday",
    "Sunday"
  )
)

## Plotting using ggplot2
weekly_data_sessions %>%
  ggplot(aes(hour, sessions, fill = deviceCategory, group = deviceCategory)) +
  geom_area(position = "stack") +
  labs(
    title = "Sessions per day and time",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    x = "Time",
    y = "Sessions",
    fill = "Device" # edit legend title
  ) +
  theme_minimal() +
  facet_wrap(~dayOfWeekName, ncol = 2, scales = "fixed") +
  theme(
    legend.position = "bottom", # move legend
    axis.text = element_text(size = 7) # change font of axis text
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

The above plot shows that:

traffic increases in the afternoon then declines in the late evening
traffic is the highest from Monday to Thursday, and lowest on Saturday and Sunday

In addition to that, thanks to the additional information on the device category, we also see that:

the number of sessions on tablet is low (it is so low compared to desktop and mobile that it is not visible on the plot), and
the number of sessions on mobile seems to be quite stable during the entire day,
as opposed to sessions on desktop which are highest at the end of the day.

Sessions per month and year

In the following code, we create a barplot of the number of daily sessions per month and year:

# get data
df2 <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions"),
  dimensions = c("date"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add in year month columns to dataframe
df2$month <- format(df2$date, "%m")
df2$year <- format(df2$date, "%Y")

# sessions by month by year using dplyr then graph using ggplot2 barplot
df2 %>%
  group_by(year, month) %>%
  summarize(sessions = sum(sessions)) %>%
  # print table steps by month by year
  # print(n = 100) %>%
  # graph data by month by year
  ggplot(aes(x = month, y = sessions, fill = year)) +
  geom_bar(position = "dodge", stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "Month",
    title = "Sessions per month and year",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This barplot allows to easily see the evolution of the number of sessions over the months, and compare this evolution across different years.

At the moment, since the blog is online only since December 2019, the year factor is not relevant. However, I still present the visualization for other users who work on older websites, and also to remind my future self to create this interesting barplot when there will be data for more than a year.

Top performing pages

Another important factor when measuring the performance of your blog or website is the number of page views for the different pages. The top performing pages in terms of page views over the year can easily be found in Google Analytics (you can access it via Behavior > Site Content > All pages).

For the interested reader, here is how to get the data in R (note that you can change n = 7 in the code to change the number of top performing pages to display):

## Make the request to GA
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("pageTitle"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

## Create a table of the most viewed posts
library(lubridate)
library(reactable)
library(stringr)

most_viewed_posts <- data_fetch %>%
  mutate(Title = str_trunc(pageTitle, width = 40)) %>% # keep maximum 40 characters
  count(Title, wt = pageviews, sort = TRUE)
head(most_viewed_posts, n = 7) # edit n for more or less pages to display

##                                      Title     n
## 1 A package to download free Springer b... 85684
## 2 Variable types and examples - Stats a... 44951
## 3 Descriptive statistics in R - Stats a... 43621
## 4    Outliers detection in R - Stats and R 32560
## 5 The complete guide to clustering anal... 27184
## 6 Correlation coefficient and correlati... 21581
## 7                              Stats and R 16786

Here is how to visualize this table of top performing pages in a barplot:

# plot
top_n(most_viewed_posts, n = 7, n) %>% # edit n for more or less pages to display
  ggplot(., aes(x = reorder(Title, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Title",
    title = "Top performing pages in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

This gives me a good first overview on how posts performed in terms of page views, so in some sense, what people find useful. For instance, I never thought that the post illustrating the different types of variables that exist in statistics (ranked #2) would be so appreciated when I wrote it.

This is something I learned with this blog: at the time of writing, there are some posts which I think no one will care about (and which I mostly write as a personal note for myself), and others which I think people will find very useful. However, after a couple of weeks after publication, sometimes I realize that it is actually exactly the opposite.

Time-normalized page views

When it comes to comparing blog posts, however, this is not as simple.

Based on the above barplot and without any further analysis, I would conclude that my article about variable types is performing much better than the one about outliers detection in R.

However, if I tell you that the article on outliers detection has been published on August 11 and the one about variable types on December 30, you will agree that the comparison does not make much sense anymore since page views for these articles were counted over a different length of time. One could also argue that a recent article had less time to generate backlinks, so it is unfair to compare it with an old post which had plenty of time to be ranked high by Google.

In order to make the comparison more “fair”, we would need to compare the number of page views for each post since their date of publication. The following code does precisely this:³

it pulls daily data for a bunch of pages,
then tries to detect their publication date,
time-normalizes the traffic for each page based on that presumed publication date,
and finally, plots the daily traffic from the publication date on out, as well as overall cumulative traffic for the top n pages

# figure out when a page actually launched by finding the
# first day where the page had at least 2 unique pageviews
first_day_pageviews_min <- 2

# exclude pages that have total traffic (daily unique pageviews) that are relatively low
total_unique_pageviews_cutoff <- 500

# set how many "days since publication" we want to include in our plot
days_live_range <- 180

# set number of top pages to display
n <- 7

# Create a dimension filter object
# You need to update the "expressions" value to be a regular expression that filters to
# the appropriate set of content on your site
page_filter_object <- dim_filter("pagePath",
  operator = "REGEXP",
  expressions = "/blog/.+"
)

# Now, put that filter object into a filter clause. The "operator" argument can be AND # or OR...but you have to have it be something, even though it doesn't do anything
# when there is only a single filter object.
page_filter <- filter_clause_ga4(list(page_filter_object),
  operator = "AND"
)

# Pull the GA data
ga_data <- google_analytics(
  viewId = view_id,
  date_range = c(start_date, end_date),
  metrics = "uniquePageviews",
  dimensions = c("date", "pagePath"),
  dim_filters = page_filter,
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# Find the first date for each post. This is actually a little tricky, so we're going to write a
# function that takes each page as an input, filters the data to just include those
# pages, finds the first page, and then puts a "from day 1" count on that data and
# returns it.
normalize_date_start <- function(page) {
  # Filter all the data to just be the page being processed
  ga_data_single_page <- ga_data %>% filter(pagePath == page)

  # Find the first value in the result that is greater than first_day_pageviews_min. In many
  # cases, this will be the first row, but, if there has been testing/previews before it
  # actually goes live, some noise may sneak in where the page may have been live, technically,
  # but wasn't actually being considered live.
  first_live_row <- min(which(ga_data_single_page$uniquePageviews > first_day_pageviews_min))

  # Filter the data to start with that page
  ga_data_single_page <- ga_data_single_page[first_live_row:nrow(ga_data_single_page), ]

  # As the content ages, there may be days that have ZERO traffic. Those days won't show up as
  # rows at all in our data. So, we actually need to create a data frame that includes
  # all dates in the range from the "publication" until the last day traffic was recorded. There's
  # a little trick here where we're going to make a column with a sequence of *dates* (date) and,
  # with a slightly different "seq," a "days_live" that corresponds with each date.
  normalized_results <- data.frame(
    date = seq.Date(
      from = min(ga_data_single_page$date),
      to = max(ga_data_single_page$date),
      by = "day"
    ),
    days_live = seq(min(ga_data_single_page$date):
    max(ga_data_single_page$date)),
    page = page
  ) %>%
    # Join back to the original data to get the uniquePageviews
    left_join(ga_data_single_page) %>%
    # Replace the "NAs" (days in the range with no uniquePageviews) with 0s (because
    # that's exactly what happened on those days!)
    mutate(uniquePageviews = ifelse(is.na(uniquePageviews), 0, uniquePageviews)) %>%
    # We're going to plot both the daily pageviews AND the cumulative total pageviews,
    # so let's add the cumulative total
    mutate(cumulative_uniquePageviews = cumsum(uniquePageviews)) %>%
    # Grab just the columns we need for our visualization!
    select(page, days_live, uniquePageviews, cumulative_uniquePageviews)
}

# We want to run the function above on each page in our dataset. So, we need to get a list
# of those pages. We don't want to include pages with low traffic overall, which we set
# earlier as the 'total_unique_pageviews_cutoff' value, so let's also filter our
# list to only include the ones that exceed that cutoff. We also select the top n pages
# in terms of page views to display in the visualization.
library(dplyr)
pages_list <- ga_data %>%
  group_by(pagePath) %>%
  summarise(total_traffic = sum(uniquePageviews)) %>%
  filter(total_traffic > total_unique_pageviews_cutoff) %>%
  top_n(n = n, total_traffic)

# The first little bit of magic can now occur. We'll run our normalize_date_start function on
# each value in our list of pages and get a data frame back that has our time-normalized
# traffic by page!
library(purrr)
ga_data_normalized <- map_dfr(pages_list$pagePath, normalize_date_start)

# We specified earlier -- in the `days_live_range` object -- how many "days since publication" we
# actually want to include, so let's do one final round of filtering to only include those
# rows.
ga_data_normalized <- ga_data_normalized %>% filter(days_live <= days_live_range)

Now that our data is ready, we create two visualizations:

Number of page views by day since publication: this plot shows how quickly interest in a particular piece of content drops off. If it is not declining as rapidly as the other posts, it means you are getting sustained value from it
Cumulative number of page views by day since publication: this plot can be used to compare blog posts on the same ground since number of page views is shown based on the publication date. To see which pages have generated the most traffic over time, simply look from top to bottom (at the right edge of the plot)

Note that both plots use the {plotly} package to make them interactive so that you can mouse over a line and find out exactly what page it is (together with its values). The interactivity of the plot makes it also possible to zoom in to see, for instance, the number of page views in the first days after publication (instead of the default length of 180 days),⁴ or zoom in to see only the evolution of the number of page views below/above a certain threshold.

# Create first plot
library(ggplot2)

gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Page views by day since publication",
    x = "Days since publication",
    y = "Page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
library(plotly)
ggplotly(gg, dynamicTicks = TRUE)

The plot above shows again a huge spike for the post that went viral (orange line). If we zoom in to include only page views below 1500, comparison between posts is easier and we see that:

The post on the top 100 R resources on Coronavirus (purple line) generated comparatively more traffic than the other posts in the first 50 days ($\approx$ 1 month and 3 weeks) after publication. However, traffic gradually decreased up to the point that after 180 days ($\approx$ 6 months), it attracted less traffic than other more performing posts.
The post on outliers detection in R (blue line) did not get a lot of attention in the first weeks after its publication. However, it gradually generated more and more traffic up to the point that, after 60 days ($\approx$ 2 months) after publication, it actually generated more traffic than any other post (and by a relatively large margin).
The post on correlation coefficient and correlation test in R (green line) took approximately 100 days ($\approx$ 3 months and 1 week) to take off, but after that it generated quite a lot of traffic. This is interesting to keep in mind when analyzing recent posts, because they may actually follow the same trend in the long run.

# Create second plot: cumulative
gg <- ggplot(ga_data_normalized, mapping = aes(x = days_live, y = cumulative_uniquePageviews, color = page)) +
  geom_line() + # The main "plot" operation
  scale_y_continuous(labels = scales::comma) + # Include commas in the y-axis numbers
  labs(
    title = "Cumulative page views by day since publication",
    x = "Days since publication",
    y = "Cumulative page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  theme_minimal() + # minimal theme
  theme(
    legend.position = "none", # remove legend
  )

# Output the plot, wrapped in ggplotly so we will get some interactivity in the plot
ggplotly(gg, dynamicTicks = TRUE)

Compared to the previous plot, this one shows the cumulative number of page views since the date of publication of the post.

Without taking into consideration the post that went viral and which, by the way, does not really generate much traffic anymore (which makes sense since the incredible campaign from Springer to offer their books for free during the COVID-19 quarantine has ended), we see that:

The post on outliers detection has surpassed the post on top R resources on Coronavirus in terms of cumulative number of page views after around 120 days ($\approx$ 4 months) after publication, which indicates that people are looking at this specific problem. I remember that I wrote this post because, at that time, I had to deal with the problems of outliers in R and I did not found a neat solution online. So I guess, the fact that resources on a specific topic are missing helps to attract visitors looking for an answer to their question.
Among the remaining posts, the ranking of the most performing ones in terms of cumulative number of page views within 180 days ($\approx$ 6 months) after publication is the following:

The analyses so far give you already a good understanding of the performance of your blog. However, for the interested readers, we show other important metrics in the following sections.

Page views by country

In the following we are interested in seeing where the traffic comes from. This is particularly interesting to get to know your audience, and even more important if you are running a business or an ecommerce.

# get GA data
data_fetch <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = "pageviews",
  dimensions = "country",
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
countries <- data_fetch %>%
  mutate(Country = str_trunc(country, width = 40)) %>% # keep maximum 40 characters
  count(Country, wt = pageviews, sort = TRUE)
head(countries, n = 10) # edit n for more or less countries to display

##           Country      n
## 1   United States 131306
## 2           India  38800
## 3         Belgium  32983
## 4  United Kingdom  25833
## 5          Brazil  18961
## 6         Germany  17851
## 7           Spain  17355
## 8          Canada  13880
## 9          Mexico  12882
## 10    Philippines  12540

To visualize this table of top countries in terms of page views in a barplot:

# plot
top_n(countries, n = 10, n) %>% # edit n for more or less countries to display
  ggplot(., aes(x = reorder(Country, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Country",
    title = "Top performing countries in terms of page views",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  theme(plot.margin = unit(c(5.5, 7.5, 5.5, 5.5), "pt")) # to avoid the plot being cut on the right edge

We see that readers from the US take the largest share of the number of page views (by quite a lot actually!), and Belgium (my country) comes in third place in terms of number of page views.

Given that the US population is much larger than the Belgian population ($\approx$ 331 million people compared to $\approx$ 11.5 million people, respectively), the above result is not really surprising. Again, for a better comparison, it would be interesting to take into consideration the size of the population when comparing countries.

Indeed, it could be that a large share of the traffic comes from a country with a large population, but that the number of page views per person (or per 100,000 inhabitants) is higher for another country. This information about top performing countries in terms of page views per person could give you insights on which country do the most avid readers come from. This is beyond the scope of this article, but you can see examples of plots which include the information on the population size in these COVID-19 visualizations. I recommend to apply the same methodology to the above plot for a better comparison.

If you are thinking about doing the extra step of including population size when comparing countries, I believe that it would be even better to take into account the information on computer access in each country as well. If we take India as example: at the time of writing this article, its population amounts to almost 1.4 billion. However, the percentage of Indian people having access to a computer is undoubtedly lower than in US or Belgium (again, at least at the moment). It would therefore make more sense to compare countries by comparing the number of page views per people having access to a computer.⁵

Browser information

For more technical aspects, you could also be interested in the number of page views by browser. This can be visualized with the following barplot:

# get data
browser_info <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviews"),
  dimensions = c("browser"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# table
browser <- browser_info %>%
  mutate(Browser = str_trunc(browser, width = 40)) %>% # keep maximum 40 characters
  count(Browser, wt = pageviews, sort = TRUE)

# plot
top_n(browser, n = 10, n) %>% # edit n for more or less browser to display
  ggplot(., aes(x = reorder(Browser, n), y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  theme_minimal() +
  coord_flip() +
  labs(
    y = "Page views",
    x = "Browser",
    title = "Which browsers are our visitors using?",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

Most visits were, as expected, from Chrome, Safari and Firefox browsers.

User engagement by devices

One may also be interested in checking how users are engaged on different types of devices. To do so, we plot 3 charts describing:

How many sessions were made from the different types of devices
The average time on page (in seconds) by type of device
The number of page views per session by device type

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("sessions", "avgTimeOnPage"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# plot sessions by deviceCategory
gadata %>%
  ggplot(aes(deviceCategory, sessions)) +
  geom_bar(aes(fill = deviceCategory), stat = "identity") +
  theme_minimal() +
  labs(
    y = "Sessions",
    x = "",
    title = "Sessions per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com",
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) # better y labels

From the above plot, we see that the majority of readers visited the blog from a desktop, and a small number of readers from a tablet.

# add median of average time on page per device
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(avgTimeOnPage))

# plot avgTimeOnPage by deviceCategory
ggplot(gadata) +
  aes(x = avgTimeOnPage, fill = deviceCategory) +
  geom_histogram(bins = 30L) +
  scale_fill_hue() +
  theme_minimal() +
  theme(legend.position = "none") +
  facet_wrap(vars(deviceCategory)) +
  labs(
    y = "Frequency",
    x = "Average time on page (in seconds)",
    title = "Average time on page per device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com"
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory),
    color = "darkgrey",
    linetype = "dashed"
  ) +
  geom_text(
    aes(
      x = med, y = 25,
      label = paste0("Median = ", round(med), " seconds")
    ),
    angle = 90,
    vjust = 2,
    color = "darkgrey",
    size = 3
  )

From the above plot, we see that:

some readers on tablet have a very low average time spent on each page (see the peak around 0 second in the tablet facet)
distributions of the average time on page for readers on desktop and mobile were quite similar, with an average time on page mostly between 100 seconds (= 1 minute 40 seconds) and 400 seconds (= 6 minutes 40 seconds)
quite surprisingly, the median of the average time spent on page is slightly higher for visitors on mobile than on desktop (see the dashed vertical lines representing the medians in the desktop and mobile facets). This indicates that, although more people visit the blog from desktop, it seems that people on mobile spend more time per page. I find this result quite surprising given that most of my articles include R code and require a computer to run the code. Therefore, I expected that people would spend more time on desktop than on mobile because on mobile they would quickly scan the article, while on desktop they would read the article carefully and try to reproduce the code on their computer.⁶

Given this result, it would be interesting to also illustrate the number of page views during a session, represented by device type.

Indeed, it may be the case that visitors on mobile spend, on average, more time on each page but people on desktop visit more pages per session. We verify this via a density plot, and for better readability we exclude data points above 2.5 page views/session and we exclude visits from a tablet:

# GA data
gadata <- google_analytics(view_id,
  date_range = c(start_date, end_date),
  metrics = c("pageviewsPerSession"),
  dimensions = c("date", "deviceCategory"),
  anti_sample = TRUE # slows down the request but ensures data isn't sampled
)

# add median of number of page views/session
gadata <- gadata %>%
  group_by(deviceCategory) %>%
  mutate(med = median(pageviewsPerSession))

## Reordering gadata$deviceCategory
gadata$deviceCategory <- factor(gadata$deviceCategory,
  levels = c("mobile", "desktop", "tablet")
)

# plot pageviewsPerSession by deviceCategory
gadata %>%
  filter(pageviewsPerSession <= 2.5 & deviceCategory != "tablet") %>% # filter out pageviewsPerSession > 2.5 and visits from tablet
  ggplot(aes(x = pageviewsPerSession, fill = deviceCategory, color = deviceCategory)) +
  geom_density(alpha = 0.5) +
  scale_fill_hue() +
  theme_minimal() +
  labs(
    y = "Frequency",
    x = "Page views per session",
    title = "Page views/session by device",
    subtitle = paste0(format(start_date, "%b %d, %Y"), " to ", format(end_date, "%b %d, %Y")),
    caption = "Data: Google Analytics data of statsandr.com\nPoints > 2.5 excluded",
    color = "Device", # edit legend title
    fill = "Device" # edit legend title
  ) +
  scale_y_continuous(labels = scales::comma) + # better y labels
  geom_vline(aes(xintercept = med, group = deviceCategory, color = deviceCategory),
    linetype = "dashed",
    show.legend = FALSE # remove legend
  ) +
  geom_text(
    aes(
      x = med, y = 2.75,
      label = paste0("Median = ", round(med, 2), " page views/session"),
      color = deviceCategory
    ),
    angle = 90,
    vjust = 2,
    size = 3,
    show.legend = FALSE # remove legend
  )

This last plot confirms our thoughts, that is, although people on mobile seem to spend more time on each page, people on desktop tend to visit more pages per session.

(One may wonder why I chose to compare medians instead of means. The main reason is that not all distributions considered here are bell-shaped (especially for data on tablet) and there are many outliers. In these cases, the mean is usually not the most appropriate descriptive statistics and the median is a more robust way to represent such data. For the interested reader, see a note on the difference between mean and median, and the context in which each measure is more appropriate.)

This is the end of the analytics section. Of course, many more visualizations and data analyses are possible, depending on the site that is tracked and the marketing expertise of the analyst. This was an overview of what is possible, and I hope it will give you some ideas to explore your Google Analytics data further. Next year, I may also include forecasts and make annual comparisons. See also some examples of other analyses in this GitHub repository and this website.

As a side note, I would like to add the following: even if tracking the performance of your blog is important to understand how you attract visitors and how they engage with your site, I also believe that looking at your Google Analytics stats too often is not optimal, nor sane.

Talking about personal experience: at the beginning of the blog I used to look very often at the number of visitors in real-time and the audience. I was kind of obsessed to know how many people were right now on my blog and I was constantly checking if it performed better than the day before in terms of number of visitors. I remember that I was spending so much time looking at these metrics in the first weeks that I felt I was wasting my time. And the time I was wasting looking at my Google Analytics stats was lost not creating good quality content for the blog, working on my thesis/classes, or other projects.

So when I realized that, I deleted the Google Analytics app from my smartphone and forced myself not to look at my stats more than once a month (just to make sure there is no critical issues that need to be fixed). From that moment onward, I stopped wasting my time on things I cannot control, and I got more satisfaction from writing articles because I was writing them for myself, not for the sake of seeing people reading them. This change was like a relief, and I am now more satisfied with my work on this blog compared to the first weeks or months.

Everyone is different and unique so I am not saying that you should do the same. However, if you feel that you look too much at your stats and sometimes lose motivation in writing, perhaps this is one potential solution.

Content

Now that we have seen how we can analyze Google Analytics data and track the performance of a website or blog in details, I would like to share some thoughts about content creation, content distribution and the future plans.

Finding topics

One of the biggest challenges I face with my blog is creating proper content. In the best case scenario, I would like to:

create useful content
about subjects I am really familiar with,
which I enjoy and
which fit into the blog.

By sharing only articles about topics I am familiar with, the writing process is easier and quicker. Indeed, most of the articles I wrote cover topics I teach at university, so a large part of the preliminary research is already done when I decide to write about it. Also (and this is not negligible), questions and incomprehension from students allow me to see the points I should focus on when writing about it, and how to present it to make it accessible and comprehensible to most people.

Moreover, by making my thoughts public, I often have the chance to confront them with other points of view, which allows me to study the topic even further. This in turn accelerates the writing process even more when writing about a related topic.

I also tend to write only about things I enjoy or I am interested in. I prefer quality over quantity, so writing an article from A to Z takes quite a long time depending on the depth of the subject. Since it takes time (even without taking into account the time spent after publication) and I have a full-time job, I really focus on topics I enjoy in order to keep seeing this blog as a source of pleasure, and not as a work or an obligation.

The fact that I write only about things I am familiar with, which I enjoy and when I have some free time (which mostly depend on the ongoing projects related to my PhD thesis) makes it hard for me to follow a defined pace for posts. This is why my writing schedule has been a bit inconsistent during this first year and is likely to be similar in the future as I do not want to be in the position to force myself to write.⁷

Content distribution

Even if we all agree that bloggers should write for themselves first and not for the sake of having lots of readers, it is still appreciated when your content is being read by others.

So although I do not like abusive self-promotion and I do not feel comfortable sharing my blog posts to all existing Facebook, LinkedIn and Reddit groups, somehow people need to get informed that you have written something.

For this reason, I created a Twitter account where I share new posts immediately after publishing them on the blog. In addition to posting them on Twitter, I also share the link by email to people who subscribed to the newsletter of the blog.

I also managed to get my content published on Medium through the Towards Data Science publication, R-bloggers and R Weekly. A non-negligible part of my audience comes from these referral, especially during the couple of days after publication.

By distributing the content this way, I do not feel pushy (something I want to avoid at all costs!) because people decided by themselves to receive the content I publish (e.g., they subscribed to the newsletter, they followed the blog on Twitter, they chose to read blog aggregators, etc.). So in some sense, I do not distribute my content “without their prior consent” and they can always choose not to see my posts anymore.

However, as you can see from the plot of the number of session per channel, you see that most readers come from the organic channel, so from search engines. And for this channel, apart from creating quality content (and some basic knowledge in SEO), I do not have any control on how well it is presented nor distributed to people. For this channel, it is basically search engine algorithms which decide to promote my content or not, and if they do, how well it is promoted. The only control I have regarding ranking factors is simply to create quality content.

So even if I believe that having a content distribution strategy definitely helps in reaching more people and growing your blog, it does not make everything. Creating quality content is, in my view of a beginner in SEO and marketing, the best strategy for my posts to be read.

For this very specific reason, now that I feel I have settled a decent distribution strategy, I no longer spend my energy and time on this matter.

So I just simply try to enjoy writing on my blog, and the rest will eventually follow. If not, I am still learning a lot from this blog anyway so I do not see it as wasted time.

A small note about ads

I know that an easy way to get money out of your blog is to run ads and I understand that people do it if it is profitable. However, I often find ads too intrusive or annoying and I personally do not like to read blog posts which include many ads.

To keep the reading process as enjoyable as possible, as you can see, I do not display any advertising on my blog. As long as the costs of running this website and the related open source projects (e.g., my Shiny apps, etc.) are not too high, I do not expect to include ads.

If in the future costs were to increase, I will still try to avoid putting ads as much as possible and I will try to rely on sponsorship programs & donations and on paid side projects for people who need help for their statistical analyses.

Future plans

After exactly one year of blogging, I am asking myself the following:

What do I want statsandr.com to be?

As already said, as long as I enjoy writing stuff I am passionate about on this blog, I will continue. This is very important to me.

In addition to that, I would like it to be a place to share knowledge. The field of statistics and R (and data science in general) is evolving at an extremely fast pace—and more and more people have many interesting things to say.

Moreover, I learned a lot since the launch this blog. But I learned even more when working in collaboration with someone else (see for instance these collaborations). I see so many learning opportunities when collaborating that I would love to see this blog as a place to share knowledge, but not only my knowledge.

To be more precise, I would love to hear about other researchers, statisticians, R lovers, data scientists, authors, etc. who want to collaborate with me. This could lead, in the end, to:

a guest post if you have a good idea on what you want to write about but need a way to share it (and since you have access to my Google Analytics data in the past year through this article you have a good idea of the number of visitors who will see your content),
a piece of content written together if you believe our skills and knowledge are complementary (see all articles for an overview of what I like to write about),
a piece of code, a R package, a visualization or a Shiny app you want to create together (or just share),
a book or a course on statistics and/or R,
or anything else you have in mind (I am open to new ideas and challenges).

With this article, you see the figures and the audience of the blog. With all my articles, you can see my strengths and weaknesses.

So I will finish this section by saying that, if anyone is willing to work on something together (which does not need to be huge), you can always contact me or leave a comment at the end of this post. I am looking forward to hearing from you.

Thank you note

Last but not least, I also wanted to leave a short thank you note to the Towards Data Science, R-bloggers and R Weekly teams that have cross-posted most of my articles in their respective publications and therefore allowed me to share my thoughts to a larger—and very knowledgeable—audience.

Also, I would like to thank all active readers for their comments, constructive feedback and support so far. I am looking forward to sharing more quality content through this blog and I hope it will keep being useful to many of you, in parallel to being useful to me.

A special thanks to Mark Edmondson, the author of the {googleAnalyticsR} package and all people who wrote tutorials using the package, which were used as inspiration for this blog post. More broadly, thanks also to the open source community which is of great help in learning R.

Thanks for reading. I hope that you learned how to track the performance of your website or blog in R using the {googleAnalyticsR} package. See you next year for a second review, and in the meantime, if you maintain a blog I would be really happy to hear how you track its performance. Feel free to let me know via the comments below!

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Thanks to the RStudio blog for the inspiration.↩︎
See more ways to install and load R packages.↩︎
Many thanks to dartistics.com for the code. Note that for better readability of the plots, I slightly edited the code: (i) to display only the top n pages in terms of traffic instead of all pages, (ii) to change the theme to theme_minimal() and (iii) to make the axis ticks dynamic when zooming in or out (in the ggplotly() function).↩︎
Note that the default period of 180 days can also be changed in the code.↩︎
This may require some assumptions if the percentage of people having access to a computer is not readily available. I believe, however, that it would still be more appropriate than just taking into account the population size—especially when comparing developed with developing countries.↩︎
At least that is what I do when I read blogs on mobile versus reading them on desktop.↩︎
Note, however, that research has shown that scheduling time for writing is a good way to productive writing (Silvia, 2019). This is why, unlike for my blog, I have set regular writing periods in my calendar (and I try to stick to it no matter how busy I am at that time) dedicated to my PhD thesis.↩︎

Graphics in R with ggplot2

Fri, 21 Aug 2020 00:00:00 +0000

Note that this article is inspired from the lecture notes of Prof. Anouar El Ghouch and my personal notes as teaching assistant for his course entitled “Statistics and data sciences with R: Advanced programming” given at UCLouvain.

Introduction

R is known to be a really powerful programming language when it comes to graphics and visualizations (in addition to statistics and data science of course!).

To keep it short, graphics in R can be done in three ways, via the:

{graphics} package (the base graphics in R, loaded by default)
{lattice} package which adds more functionalities to the base package
{ggplot2} package (which needs to be installed and loaded beforehand)

The {graphics} package comes with a large choice of plots (such as plot, hist, barplot, boxplot, pie, mosaicplot, etc.) and additional related features (e.g., abline, lines, legend, mtext, rect, etc.). It is often the preferred way to draw plots for most R users, and in particular for beginners to intermediate users.

Since its creation in 2005 by Hadley Wickham, {ggplot2} has grown in use to become one of the most popular R packages and the most popular package for graphics and data visualizations. The {ggplot2} package is a much more modern approach to creating professional-quality graphics. More information about the package can be found at ggplot2.tidyverse.org.

In this article, we will see how to create common plots such as scatter plots, line plots, histograms, boxplots, barplots, density plots in R with this package. If you are unfamiliar with any of these types of graph, you will find more information about each one (when to use it, its purpose, what does it show, etc.) in my article about descriptive statistics in R.

Data

To illustrate plots with the {ggplot2} package we will use the mpg dataset available in the package.

The dataset contains observations collected by the US Environmental Protection Agency on fuel economy from 1999 to 2008 for 38 popular models of cars (run ?mpg for more information about the data):

library(ggplot2)
dat <- ggplot2::mpg

Before going further, let’s transform the cyl, drv, fl, year and class variables in factor with the transform() function:

dat <- transform(dat,
  cyl = factor(cyl),
  drv = factor(drv),
  fl = factor(fl),
  year = factor(year),
  class = factor(class)
)

For the interested reader, see more data manipulation techniques in R.

Basic principles of `{ggplot2}`

The {ggplot2} package is based on the principles of “The Grammar of Graphics” (hence “gg” in the name of {ggplot2}), that is, a coherent system for describing and building graphs. The main idea is to design a graphic as a succession of layers.

The main layers are:

The dataset that contains the variables that we want to represent. This is done with the ggplot() function and comes first.
The variable(s) to represent on the x and/or y-axis, and the aesthetic elements (such as color, size, fill, shape and transparency) of the objects to be represented. This is done with the aes() function (abbreviation of aesthetic).
The type of graphical representation (scatter plot, line plot, barplot, histogram, boxplot, etc.). This is done with the functions geom_point(), geom_line(), geom_bar(), geom_histogram(), geom_boxplot(), etc.
If needed, additional layers (such as labels, annotations, scales, axis ticks, legends, themes, facets, etc.) can be added to personalize the plot.

To create a plot, we thus first need to specify the data in the ggplot() function and then add the required layers such as the variables, the aesthetic elements and the type of plot:

ggplot(data) +
  aes(x = var_x, y = var_y) +
  geom_x()

data in ggplot() is the name of the data frame which contains the variables var_x and var_y.
The + symbol is used to indicate the different layers that will be added to the plot. Make sure to write the + symbol at the end of the line of code and not at the beginning of the line, otherwise R throws an error.
The layer aes() indicates what variables will be used in the plot and more generally, the aesthetic elements of the plot.
Finally, x in geom_x() represents the type of plot.
Other layers are usually not required unless we want to personalize the plot further.

Note that it is a good practice to write one line of code per layer to improve code readability.

Create plots with `{ggplot2}`

In the following sections we will show how to draw the following plots:

scatter plot
line plot
histogram
density plot
dotplot
boxplot
barplot
raincloud plot

In order to focus on the construction of the different plots and the use of {ggplot2}, we will restrict ourselves to drawing basic (yet beautiful) plots without unnecessary layers. For the sake of completeness, we will briefly discuss and illustrate different layers to further personalize a plot at the end of the article (see this section).

Note that if you still struggle to create plots with {ggplot2} after reading this tutorial, you may find the {esquisse} addin useful. This addin allows you to interactively (that is, by dragging and dropping variables) create plots with the {ggplot2} package. Give it a try!

Scatter plot

We start by creating a scatter plot using geom_point. Remember that a scatter plot is used to visualize the relation between two quantitative variables.

We start by specifying the data:

ggplot(dat) # data

Then we add the variables to be represented with the aes() function:

ggplot(dat) + # data
  aes(x = displ, y = hwy) # variables

Finally, we indicate the type of plot:

ggplot(dat) + # data
  aes(x = displ, y = hwy) + # variables
  geom_point() # type of plot

You will also sometimes see the aesthetic elements (aes() with the variables) inside the ggplot() function in addition to the dataset:

ggplot(mpg, aes(x = displ, y = hwy)) +
  geom_point()

This second method gives the exact same plot than the first method. I tend to prefer the first method over the second for better readability, but this is more a matter of taste so the choice is up to you.

Line plot

Line plots, particularly useful in time series or finance, can be created similarly but by using geom_line():

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_line()

(Note that this might not be the most appropriate plot since there are multiple points for each value of displ, but this is just an example to show how to create a line plot.)

Combination of line and points

An advantage of {ggplot2} is the ability to combine several types of plots and its flexibility in designing it. For instance, we can add a line to a scatter plot by simply adding a layer to the initial scatter plot:

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point() +
  geom_line() # add line

Histogram

A histogram (useful to visualize distributions and detect potential outliers) can be plotted using geom_histogram():

ggplot(dat) +
  aes(x = hwy) +
  geom_histogram()

By default, the number of bins is equal to 30. You can change this value using the bins argument inside the geom_histogram() function:

ggplot(dat) +
  aes(x = hwy) +
  geom_histogram(bins = round(sqrt(nrow(dat))))

Here I specify the number of bins to be equal to the square root of the number of observations (following the square-root rule) but you can specify any integer number.

Density plot

Density plots can be created using geom_density():

ggplot(dat) +
  aes(x = hwy) +
  geom_density()

Combination of histogram and densities

We can also superimpose a histogram and a density curve on the same plot:

ggplot(dat) +
  aes(x = hwy, y = after_stat(density)) +
  geom_histogram() +
  geom_density()

Or superimpose several densities:

ggplot(dat) +
  aes(x = hwy, color = drv, fill = drv) +
  geom_density(alpha = 0.25) # add transparency

The argument alpha = 0.25 has been added for some transparency. More information about this argument can be found in this section.

Dotplot

A dotplot in {ggplot2} can be built with geom_dotplot():

# Dotplot for one variable
ggplot(dat) +
  aes(x = "", y = hwy) +
  geom_dotplot(binaxis = "y", stackdir = "center")

# Dotplot by factor
ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_dotplot(
    binaxis = "y", stackdir = "center",
    dotsize = 0.75 # decrease dot size
  )

Dotplots are more appropriate with small samples (because the plot may be hard to read with too many points). For large samples, a boxplot may be used.

For the interested reader, see many personalization that is possible with a dotplot in this tutorial.

Boxplot

A boxplot (also very useful to visualize distributions and detect potential outliers) can be plotted using geom_boxplot():

# Boxplot for one variable
ggplot(dat) +
  aes(x = "", y = hwy) +
  geom_boxplot()

# Boxplot by factor
ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot()

It is also possible to plot the points on the boxplot with geom_jitter(), and to vary the width of the boxes according to the size (i.e., the number of observations) of each level with varwidth = TRUE:

ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) # adds random noise and limit its width

The geom_jitter() layer adds some random variation to each point in order to prevent them from overlapping (an issue known as overplotting).¹ Moreover, the alpha argument adds some transparency to the points (see more in this section) to keep the focus on the boxes and not on the points.

Finally, it is also possible to divide boxplots into several panels according to the levels of a qualitative variable:

ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) # divide into 2 panels

For a visually more appealing plot, it is also possible to use some colors for the boxes depending on the x variable:

ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) + # add color to boxes with fill
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) + # divide into 2 panels
  theme(legend.position = "none") # remove legend

In that case, it best to remove the legend as it becomes redundant. See more information about the legend in this section.

If you are unhappy with the default colors provided in {ggplot2}, you can change them manually with the scale_fill_manual() layer:

ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) + # add color to boxes with fill
  geom_boxplot(varwidth = TRUE) + # vary boxes width according to n obs.
  geom_jitter(alpha = 0.25, width = 0.2) + # adds random noise and limit its width
  facet_wrap(~year) + # divide into 2 panels
  theme(legend.position = "none") + # remove legend
  scale_fill_manual(values = c("darkred", "darkgreen", "steelblue")) # change fill color manually

Barplot

A barplot (useful to visualize qualitative variables) can be plotted using geom_bar():

ggplot(dat) +
  aes(x = drv) +
  geom_bar()

Bars’ heights correspond to the observed frequencies (i.e., the number of observations) for each level of the variable of interest (drv in our case).

Tip: by default, the order of the bars follows the initial order (by alphabetical order or numerical order if you did not change it). If you want to order the levels by frequency (largest first), use the fct_infreq() function from the {forcats} package.

library(forcats)

ggplot(dat) +
  aes(x = fct_infreq(drv)) + # order by frequency
  geom_bar()

If you want to order levels in an increasing order (i.e., category with the smallest frequency first), use the fct_rev() in addition to the fct_infreq() function:

ggplot(dat) +
  aes(x = fct_rev(fct_infreq(drv))) + # order by frequency
  geom_bar()

(Label for the x-axis can then easily be edited with the labs() function. See below for more information.)

Again, for a more appealing plot, we can add some colors to the bars with the fill argument:

ggplot(dat) +
  aes(x = drv, fill = drv) + # add colors to bars
  geom_bar() +
  theme(legend.position = "none") # remove legend

We can also create a barplot with two qualitative variables:

ggplot(dat) +
  aes(x = drv, fill = year) + # fill by years
  geom_bar()

In order to compare proportions across groups, it is best to make each bar the same height using position = "fill":

ggplot(dat) +
  geom_bar(aes(x = drv, fill = year), position = "fill")

To draw the bars next to each other for each group, use position = "dodge":

ggplot(dat) +
  geom_bar(aes(x = drv, fill = year), position = "dodge")

Raincloud plot

A raincloud plot is a graph that combines 3 visualizations:

a density plot,
a boxplot,
and the raw data in the form of a dotplot or jittered points.

The advantage of this plot is that it illustrates, all at once, the distribution (with the density curve), the summary measures (first, second and third quartiles, and maximum/mininum without outliers thanks to the boxplot) and the number of observations (either via a dotplot or via jittered points).

Let’s illustrate the raincloud plot, first with jittered points (more appropriate with large samples):²

library(tidyverse)
library(ggdist)

# density plot:
dat %>%
  ggplot(aes(x = drv, y = hwy, fill = drv)) +
  stat_halfeye(
    adjust = 0.5, # set the smoothing parameter
    width = 0.5, # set the height of the curves
    justification = -0.2, # move curves to the right
    .width = 0, point_colour = NA # remove interval present by default
  ) +
  # boxplot:
  geom_boxplot(
    width = 0.12, # width of boxes
    outlier.color = NA, # remove color of outliers
    alpha = 0.5 # add transparency
  ) +
  # jittered points:
  geom_point(aes(colour = drv), # add color on points
    size = 1.3, # size of points
    alpha = .3, # add transparency
    position = position_jitter( # obtain shifted points
      seed = 1, # set seed for same random representation
      width = .09 # manage the width of the offset
    )
  ) +
  # further personalization:
  coord_flip() + # rotate plot
  theme(legend.position = "none") # remove legend

Now the same chart but with dotplots this time (more appropriate with small samples):

# density plot:
dat %>%
  sample_n(100) %>% # random sample of size 100
  ggplot(aes(x = drv, y = hwy, fill = drv)) +
  stat_halfeye(
    adjust = 0.5, # set the smoothing parameter
    width = 0.5, # set the height of the curves
    justification = -0.2, # move curves to the right
    .width = 0, point_colour = NA # remove interval present by default
  ) +
  # boxplot:
  geom_boxplot(
    width = 0.12, # width of boxes
    outlier.color = NA, # remove color of outliers
    alpha = 0.5 # add transparency
  ) +
  # dotplot:
  stat_dots(
    dotsize = 0.5, # size of points
    side = "left", # place points on opposite side of density curve
    justification = 1.1, # move points away from boxplot
    binwidth = 1 # group points together
  ) +
  # further personalization:
  coord_flip() + # rotate plot
  theme(legend.position = "none") # remove legend

The code is much longer compared to the other plots, but the only line(s) to edit to adapt to your dataset is the aesthetics (aes()). The rest is mainly adjustments that should not be changed.

You may have notice that at the end of the code, there are some personalization which allow to improve the plot even further. The most common personalization are presented in the next section.

Further personalization

Title and axis labels

The first things to personalize in a plot is the labels to make the plot more informative to the audience. We can easily add a title, subtitle, caption and edit axis labels with the labs() function:

p <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

p + labs(
  title = "Fuel efficiency for 38 popular models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
)

As you can see in the above code, you can save one or more layers of the plot in an object for later use.

This way, you can save your “main” plot, and add more layers of personalization until you get the desired output. Here we saved the main scatter plot in an object called p and we will refer to it for the subsequent personalization.

You can also edit the alignment, the size and the shape of the title and subtitle via the theme() layer and the element_text() function:

p + labs(
  title = "Fuel efficiency for 38 popular models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
) +
  theme(
    plot.title = element_text(
      hjust = 0.5, # center
      size = 12,
      color = "steelblue",
      face = "bold"
    ),
    plot.subtitle = element_text(
      hjust = 0.5, # center
      size = 10,
      color = "gray",
      face = "italic"
    )
  )

If the title or subtitle is long and you want to divide it into multiple lines, use \n:

p + labs(
  title = "Fuel efficiency for 38 popular \n models of car",
  subtitle = "Period 1999-2008",
  caption = "Data: ggplot2::mpg. See more at statsandr.com",
  x = "Engine displacement (litres)",
  y = "Highway miles per gallon (mpg)"
) +
  theme(
    plot.title = element_text(
      hjust = 0.5, # center
      size = 12,
      color = "steelblue",
      face = "bold"
    ),
    plot.subtitle = element_text(
      hjust = 0.5, # center
      size = 10,
      color = "gray",
      face = "italic"
    )
  )

Axis ticks

Axis ticks can be adjusted using scale_x_continuous() and scale_y_continuous() for the x and y-axis, respectively:

# Adjust ticks
p + scale_x_continuous(breaks = seq(from = 1, to = 7, by = 0.5)) + # x-axis
  scale_y_continuous(breaks = seq(from = 10, to = 45, by = 5)) # y-axis

Log transformations

In some cases, it is useful to plot the log transformation of the variables. This can be done with the scale_x_log10() and scale_y_log10() functions:

p + scale_x_log10() +
  scale_y_log10()

Limits

The most convenient way to control the limits of the plot is to use again the scale_x_continuous() and scale_y_continuous() functions in addition to the limits argument:

p + scale_x_continuous(limits = c(3, 6)) +
  scale_y_continuous(limits = c(20, 30))

It is also possible to simply take a subset of the dataset with the subset() or filter() function. See how to subset a dataset if you need a reminder.

Scales for better axis formats

Depending on your data, it is possible to format axes in a certain way with the {scales} package. The format I use the most is comma which formats large numbers in a more-readable way.

For this example, we multiply both variables by 10000 to have larger numbers and then we apply the format to the y-axis (only to the y-axis so we can see the difference with the x-axis which is not formatted):

ggplot(dat) +
  aes(x = displ * 10000, y = hwy * 10000) +
  geom_point() +
  scale_y_continuous(labels = scales::comma) # format y-axis

As you can see, numbers on the y-axis are displayed as 200,000, 300,000, etc. instead of 200000, 300000, etc., which makes it more readable.

Another common format is percent to display numbers as percentages:

ggplot(dat) +
  aes(x = displ, y = hwy / 100) +
  geom_point() +
  scale_y_continuous(labels = scales::percent) # format y-axis

These two formats make large numbers and percentages easier to read. Other formats are possible such as using dollar signs, dates etc. See more information in the package’s documentation.

Legend

By default, the legend is located to the right side of the plot (when there is a legend to be displayed of course).

To control the position of the legend, we need to use the theme() function in addition to the legend.position argument:

p + aes(color = class) +
  theme(legend.position = "top")

Replace "top" by "left" or "bottom" to change its position and by "none" to remove it.

The title of the legend can be edited with the labs() layer:

p + aes(color = class) +
  labs(color = "Car's class")

Note that the argument inside labs() must match the one inside the aes() layer (in this case: color).

The title of the legend can also be removed with legend.title = element_blank() inside the theme() layer:

p + aes(color = class) +
  theme(
    legend.title = element_blank(),
    legend.position = "bottom"
  )

The legend now appears at the bottom of the plot, without the legend title.

Shape, color, size and transparency

There are a very large number of options to improve the quality of the plot or to add additional information. These include:

shape,
size,
color, and
alpha (transparency).

We can for instance change the shape of all points in a scatter plot by adding shape to geom_point(), or vary the shape according to the values taken by another variable (in that case, the shape argument must be inside aes()):³

# Change shape of all points
ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point(shape = 4)

# Change shape of points based on a categorical variable
ggplot(dat) +
  aes(x = displ, y = hwy, shape = drv) +
  geom_point()

Following the same principle, we can modify the color, size and transparency of the points based on a qualitative or quantitative variable. Here are some examples:

p <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

# Change color for all points
p + geom_point(color = "steelblue")

# Change color based on a qualitative variable
p + aes(color = drv)

# Change color based on a quantitative variable
p + aes(color = cty)

# Change color based on a criterion (median of cty variable)
p + aes(color = cty > median(cty))

# Change size of all points
p + geom_point(size = 4)

# Change size of points based on a quantitative variable
p + aes(size = cty)

# Change transparency based on a quantitative variable
p + aes(alpha = cty)

We can of course mix several options (shape, color, size, alpha) to build more complex graphics:

p + geom_point(size = 0.5) +
  aes(color = drv, shape = year, alpha = cty)

If you are unhappy with the default colors, you can change them manually with the scale_colour_manual() layer (for qualitative variables) and the scale_colour_gradient2() layer (for quantitative variables):

# Change color based on a qualitative variable
p + aes(color = drv) +
  scale_colour_manual(values = c("red", "blue", "green"))

# Change color based on a quantitative variable
p + aes(color = cty) +
  scale_colour_gradient2(
    low = "green",
    mid = "gray",
    high = "red",
    midpoint = median(dat$cty)
  )

For your information, you can emulate {ggplot2} default color palette for a desired number of colors and produce a character vector of HEX colors. For example, with 4 colors:

library(scales)
show_col(hue_pal()(4))

Text and labels

To add a label on a point (for example the row number), we can use the geom_text() and aes() functions:

p + geom_text(aes(label = rownames(dat)),
  check_overlap = TRUE,
  size = 2,
  vjust = -1
)

To add text on the plot, we use the annotate() function:

p + annotate("text",
  x = 6,
  y = 40,
  label = "hwy and displ are \n negatively correlated \n (rho = -0.77, p-value < 0.001)",
  size = 3
)

Read the article on correlation coefficient and correlation test in R to see how I computed the correlation coefficient (rho) and the p-value of the correlation test.

Smooth and regression lines

In a scatter plot, it is possible to add a smooth line fitted to the data:

p + geom_smooth()

In the context of simple linear regression, it is often the case that the regression line is displayed on the plot.

This can be done by adding method = lm (lm stands for linear model) in the geom_smooth() layer:

p + geom_smooth(method = lm)

It is also possible to draw a regression line for each level of a categorical variable:

p + aes(color = drv, shape = drv) +
  geom_smooth(method = lm, se = FALSE)

The se = FALSE argument removes the confidence interval around the regression lines.

Facets

facet_grid allows you to divide the same graphic into several panels according to the values of one or two qualitative variables:

# According to one variable
p + facet_grid(. ~ drv)

# According to 2 variables
p + facet_grid(drv ~ year)

It is then possible to add a regression line to each facet:

p + facet_grid(. ~ drv) +
  geom_smooth(method = lm)

facet_wrap() can also be used, as illustrated in this section.

Themes

Several functions are available in the {ggplot2} package to change the theme of the plot.

The most common themes after the default theme (i.e., theme_gray()) are the black and white (theme_bw()), minimal (theme_minimal()) and classic (theme_classic()) themes:

# Black and white theme
p + theme_bw()

# Minimal theme
p + theme_minimal()

# Classic theme
p + theme_classic()

I tend to use the minimal theme for most of my R Markdown reports as it brings out the patterns and points and not the layout of the plot, but again this is a matter of personal taste. See more themes at ggplot2.tidyverse.org/reference/ggtheme.html and in the {ggthemes} package.

In order to avoid having to change the theme for each plot you create, you can change the theme for the current R session using the theme_set() function as follows:

theme_set(theme_minimal())

Interactive plot with `{plotly}`

You can easily make your plots created with {ggplot2} interactive with the {plotly} package:

library(plotly)
ggplotly(p + aes(color = year))

You can now hover over a point to display more information about that point. There is also the possibility to zoom in and out, to download the plot, to select some observations, etc. More information about {plotly} for R can be found here.

Combine plots with `{patchwork}`

There are several ways to combine plots made in {ggplot2}. In my opinion, the most convenient way is with the {patchwork} package using symbols such as +, / and parentheses.

We first need to create some plots and save them:

p_a <- ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

p_b <- ggplot(dat) +
  aes(x = hwy) +
  geom_histogram()

p_c <- ggplot(dat) +
  aes(x = drv, y = hwy) +
  geom_boxplot()

Now that we have 3 plots saved in our environment, we can combine them. To have plots next to each other simply use the + symbol:

library(patchwork)
p_a + p_b + p_c

To display them above each other simply use the / symbol:

p_a / p_b / p_c

And finally, to combine them above and next to each other, mix +, / and parentheses:

p_a + p_b / p_c

(p_a + p_b) / p_c

See more ways to combine plots with:

grid.arrange() from the {gridExtra} package
plot_grid() from the {cowplot} package

Flip coordinates

Flipping coordinates of your plot is useful to create horizontal boxplots, or when labels of a variable are so long that they overlap each other on the x-axis. See with and without flipping coordinates below:

# without flipping coordinates
p1 <- ggplot(dat) +
  aes(x = class, y = hwy) +
  geom_boxplot()

# with flipping coordinates
p2 <- ggplot(dat) +
  aes(x = class, y = hwy) +
  geom_boxplot() +
  coord_flip()

library(patchwork)
p1 + p2 # left: without flipping, right: with flipping

This can be done with many types of plot, not only with boxplots. For instance, if a categorical variable has many levels or the labels are long, it is usually best to flip the coordinates for a better visual:

ggplot(dat) +
  aes(x = class) +
  geom_bar() +
  coord_flip()

Save plot

The ggsave() function will save the most recent plot in your current working directory unless you specify a path to another folder:

ggplot(dat) +
  aes(x = displ, y = hwy) +
  geom_point()

ggsave("plot1.pdf")

You can also specify the width, height and resolution (dpi) as follows:

ggsave("plot1.pdf",
  width = 12,
  height = 12,
  units = "cm",
  dpi = 300
)

Managing dates

If the time variable in your dataset is in date format, the {ggplot2} package recognizes the date format and automatically uses a specific type for the axis ticks.

There is no time variable with a date format in our dataset, so let’s create a new variable of this type thanks to the as.Date() function:

dat$date <- as.Date("2020-08-21") - 0:(nrow(dat) - 1)

See the first 6 observations of this date variable and its class:

head(dat$date)

## [1] "2020-08-21" "2020-08-20" "2020-08-19" "2020-08-18" "2020-08-17"
## [6] "2020-08-16"

str(dat$date)

##  Date[1:234], format: "2020-08-21" "2020-08-20" "2020-08-19" "2020-08-18" "2020-08-17" ...

The new variable date is correctly specified in a date format.

Most of the time, with a time variable, we want to create a line plot with the date on the X-axis and another continuous variable on the Y-axis, like the following plot for example:

p <- ggplot(dat) +
  aes(x = date, y = hwy) +
  geom_line()
p

As soon as the time variable is recognized as a date, we can use the scale_x_date() layer to change the format displayed on the X-axis. The following table shows the most frequent date formats:

Source: www.statmethods.net

Run ?strptime() to see many more date formats available in R.

For this example, let’s add the year in addition to the unabbreviated month:

p + scale_x_date(date_labels = "%B %Y")

It also possible to control the breaks to display on the X-axis with the date_breaks argument. For this example, let’s say we want to display the day as number and the abbreviated month for each interval of 10 days:

p + scale_x_date(date_breaks = "10 days", date_labels = "%d %b")

If labels displayed on the X-axis are unreadable because they overlap each other, you can rotate them with the theme() layer and the angle argument:

p + scale_x_date(date_breaks = "10 days", date_labels = "%d %b") +
  theme(axis.text.x = element_text(angle = 60, hjust = 1))

Highlight data with `{gghighlight}`

The {gghighlight} package allows, as its name suggests, to highlight some data directly on your ggplot. The highlighted data (that you define) are shown in bright color and the rest in gray.

Below example of how it works for a scatter plot, boxplot, barplot and histogram.

library(gghighlight)

# scatter plot
ggplot(mpg, aes(x = displ, y = hwy, color = cyl)) +
  geom_point() +
  gghighlight(cyl == "8") +
  theme(legend.position = "none") # remove legend

# boxplot
ggplot(dat) +
  aes(x = drv, y = hwy, fill = drv) +
  geom_boxplot() +
  gghighlight(drv %in% c("r", "4")) +
  theme(legend.position = "none") # remove legend

# barplot
ggplot(dat) +
  aes(x = drv, fill = drv) +
  geom_bar() +
  gghighlight(drv == "f")

As you can see, the gghighlight() layer accepts different types of conditions, but also several of them at the same time:

# histogram
ggplot(dat) +
  aes(x = hwy, fill = year) +
  geom_histogram() +
  gghighlight(displ > 2 & year == "1999")

Tip

I recently learned a tip very useful when drawing plots with {ggplot2}. If like me, you often comment and uncomment some lines of code in your plot, you know that you cannot transform the last line into a comment without removing the + sign in the line just above.

Adding a line NULL at the end of your plots will avoid an error if you forget to remove the + sign in the last line of your code. See with this basic example:

ggplot(dat) +
  aes(x = date, y = hwy) +
  geom_line() + # I do not have to remove the + sign
  # theme_minimal() + # this line is a comment
  NULL # adding this line doesn't change anything to the plot

This trick saves me a lot of time as I do not need to worry about making sure to remove the last + sign after commenting some lines of code in my plots.

If you find this trick useful, you may like these other tips and tricks in RStudio and R Markdown.

To go further

By now you have seen that {ggplot2} is a very powerful and complete package to create plots in R. This article illustrated only the tip of the iceberg, and you will find many tutorials on how to create more advanced plots and visualizations with {ggplot2} online. If you want to learn more than what is described in the present article, I highly recommend starting with:

the chapters Data visualisation and Graphics for communication from the book R for Data Science from Garrett Grolemund and Hadley Wickham
the book ggplot2: Elegant Graphics for Data Analysis from Hadley Wickham
the book R Graphics Cookbook from Winston Chang
the ggplot2 extensions guide which lists many of the packages that extend {ggplot2}
the {ggplot2} cheat sheet
this presentation by Garrick Aden-Buie
a detailed tutorial by Cédric Scherer

Conclusion

Thanks for reading.

I hope this article helped you to create your first plots with the {ggplot2} package. As a reminder, for simple graphs, it is sometimes easier to draw them via the {esquisse} addin. After some time, you will quickly learn how to create them by yourselves and in no time you will be able to build complex and sophisticated data visualizations.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Use the geom_jitter() layer with caution because, although it makes a plot more revealing at large scales, it also makes it slightly less accurate at small scales since some randomness is added to the points.↩︎
Code inspired from Claire Della Vedova (DellaData) and Cédric Scherer. Thanks to both of them for this nice plot!↩︎
There are (at the time of writing) 26 shapes accepted in the shape argument. See this documentation for all available shapes.↩︎

A package to download free Springer books during Covid-19 quarantine

Sun, 26 Apr 2020 00:00:00 +0000

Update

The promotion has ended so it is not possible to download the books through R. If you did not download the books in time, you can still have access to them via this link.¹

Introduction

You probably already have seen that Springer released about 500 books for free following the COVID-19 pandemic. According to Springer, these textbooks will be available free of charge until at least the end of July.

Following this announcement, I already downloaded a couple of statistics and R programming textbooks from their website and I will probably download a few more in the coming weeks.

In this article, I present a package that saved me a lot of time and which may be of interest to many of us: the {springerQuarantineBooksR} package, developed by Renan Xavier Cortes.

This package allows you to easily download all (or a selection of) Springer books made available free of charge during the COVID-19 quarantine.

With this large collection of high quality resources and my collection of top R resources about the Coronavirus, we do not have any excuse to not read and learn during this quarantine.

In this article, I show:

how to download all available textbooks at once and
how to download a subset of books, given a specific title, author or subject

Without further ado, here is how the package works in practice.

Installation

After having installed the {devtools} package, you can install the {springerQuarantineBooksR} package from GitHub and load it with:

# install.packages("devtools")
devtools::install_github("renanxcortes/springerQuarantineBooksR", force = TRUE)
library(springerQuarantineBooksR)

Download all books at once

First, set the path where you would like to save all books with the setwd() function then download all of them at once with the download_springer_book_files() function. Note that it takes several minutes (depending on the speed of your internet connection) since all books combined amount for almost 8GB.

setwd("path_of_your_choice") # where you want to save the books
download_springer_book_files() # download all of them at once

You will find all downloaded books (in PDF format) in a folder named “springer_quarantine_books”, organized by category.²

If you want to download the EPUB version (or both the PDF and EPUB versions), add the filetype argument to the function:

# for EPUB version:
download_springer_book_files(filetype = "epub")

# for both PDF and EPUB versions:
download_springer_book_files(filetype = "both")

By default, it downloads only the English books. However, it is also possible to download all German books by adding the argument lan = 'ger':

download_springer_book_files(lan = "ger")

Note that in total, there are 407 unique titles in English and 52 in German.

Create a table of Springer books

Like me, if you do not know which books are offered by Springer and you do not want to download all of them, you probably may want to have an overview or a list of the released books before downloading any.

For this, you can load a table containing all the titles made available by Springer into an R session with the download_springer_table() function:

springer_table <- springerQuarantineBooksR::download_springer_table()

This table can then be improved with the {DT} package to:

keep only a minimum of information,
allow searching a book by its title, author, classification or year,
allow downloading the list of available books, and
make the Springer links clickable for instance

# install.packages("DT")
library(DT)

springer_table$open_url <- paste0(
  '<a target="_blank" href="', # opening HTML tag
  springer_table$open_url, # href link
  '">SpringerLink</a>' # closing HTML tag
)

springer_table <- springer_table[, c(1:3, 19, 20)] # keep only relevant information

datatable(springer_table,
  rownames = FALSE, # remove row numbers
  filter = "top", # add filter on top of columns
  extensions = "Buttons", # add download buttons
  options = list(
    autoWidth = TRUE,
    dom = "Blfrtip", # location of the download buttons
    buttons = c("copy", "csv", "excel", "pdf", "print"), # download buttons
    pageLength = 5, # show first 5 entries, default is 10
    order = list(0, "asc") # order the title column by ascending order
  ),
  escape = FALSE # make URLs clickable
)

This table allows you to see which textbooks Springer offers (together with some information) and allows you to find the ones that you are most likely to be interested in.

Note that you can create a similar table for German books with the download_springer_table(lan = "ger") function.

Download only specific books

Now that you have a better idea about the books you are interested in, you can download them by their title, author or subject.

By title

Say that you are interested in downloading only one specific book and you know its title. For instance, suppose you want to download the book entitled “All of Statistics”:

download_springer_book_files(springer_books_titles = "All of Statistics")

If you are interested to download more than one book, run the following command:

download_springer_book_files(
  springer_books_titles = c(
    "All of Statistics",
    "A Modern Introduction to Probability and Statistics"
  )
)

Alternatively, if you do not have a specific title in mind but you are interested in downloading all books with the word “Statistics” in the title, you can run:

springer_table <- download_springer_table()

library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    book_title, # look for a pattern in the book_title column
    "Statistics" # specify the word
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

Tip: If you would like to download all books with the word “Statistics” or “Data Science” in the title, replace "Statistics" in the above code by "Statistics|Data Science".

By author

If you want to download all books from a specific author, you can run:

springer_table <- download_springer_table()

# library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    author, # look for a pattern in the author column
    "John Hunt" # specify the author
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

By subject

You can also download all textbooks covering a specific subject (see all subjects in the subject_classification column in the summary table). For instance, here is how to download all books categorized in the Statistics subject:

springer_table <- download_springer_table()

# library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    subject_classification, # look for a pattern in the subject_classification column
    "Statistics" # specify the subject
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

Improvements

Below a list of features that can potentially be implemented in order to improve the package:

Add the possibility to download all editions of a book. Currently, only the latest edition can be downloaded.
Add the possibility to resume downloading if it stopped. Currently, if the code is executed again, the downloads start from scratch.
Add the possibility of downloading books by topic. Currently, it is only possible by title, author or subject.

Feel free to open a pull request on GitHub if you have another improvement in mind.

Acknowledgments

I would like to thank:

Renan Xavier Cortes (and all contributors) for providing this package
The springer_free_books Python project which was used as inspiration to the {springerQuarantineBooksR} package
And last but not least, Springer who offers many of their excellent books for free!

Conclusion

Thanks for reading.

I hope this article will help you to download and read more high quality materials made available by Springer during this Covid-19 quarantine.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Note that I am not the author nor the maintainer of this shared folder. Therefore, I do not know how long the books will be available through this link, and I am not responsible if some (or all) books are removed.↩︎
Note that you can change the folder name by specifying the argument destination_folder = "name_of_your_choice".↩︎

An efficient way to install and load R packages

Fri, 31 Jan 2020 00:00:00 +0000

What is an R package and how to use it?

Unlike other programs, only fundamental functionalities come by default with R. You will thus often need to install some “extensions” to perform the analyses you want. These extensions which are collections of functions and datasets developed and published by R users are called packages.

Packages extend existing base R functionalities by adding new ones. R is open source so everyone can write code and publish it as a package, and everyone can install a package and start using the functions or datasets built inside the package, all this for free.

In order to use a package, it needs to be installed on your computer by running install.packages("name_of_package") (do not forget "" around the name of the package, otherwise R will look for an object saved under that name!). Once the package is installed, you must load the package and only after it has been loaded you can use all the functions and datasets it contains. To load a package, run library(name_of_package) (this time "" around the name of the package are optional, but can still be used if you wish).

Note that packages must be installed only once (until you update your R, then you have to install them again), whereas packages must be loaded every time you open R.¹

Inefficient way to install and load R packages

Depending on how long you have been using R, you may use a limited amount of packages or, on the contrary, a large amount of them. As you use more and more packages you will soon start to have (too) many lines of code just for installing and loading them.

Here is a preview of the code from my PhD thesis showing how the installation and loading of R packages looked like when I started working on R (only a fraction of them are displayed to shorten the code):

# Installation of required packages
install.packages("tidyverse")
install.packages("ggplot2")
install.packages("readxl")
install.packages("dplyr")
install.packages("tidyr")
install.packages("ggfortify")
install.packages("DT")
install.packages("reshape2")
install.packages("knitr")
install.packages("lubridate")

# Load packages
library("tidyverse")
library("ggplot2")
library("readxl")
library("dplyr")
library("tidyr")
library("ggfortify")
library("DT")
library("reshape2")
library("knitr")
library("lubridate")

As you can guess the code became longer and longer as I needed more and more packages for my analyses. Moreover, I tended to reinstall all packages as I was working on 4 different computers and I could not remember which packages were already installed on which machine. Reinstalling all packages every time I opened my script or R Markdown document was a waste of time.

More efficient way

Then one day, a colleague of mine shared some of his code with me. I am glad he did as he introduced me to a much more efficient way to install and load R packages. He gave me the permission to share the tip, so here is the code I now use to perform the task of installing and loading R packages:

# Package names
packages <- c("ggplot2", "readxl", "dplyr", "tidyr", "ggfortify", "DT", "reshape2", "knitr", "lubridate", "pwr", "psy", "car", "doBy", "imputeMissings", "RcmdrMisc", "questionr", "vcd", "multcomp", "KappaGUI", "rcompanion", "FactoMineR", "factoextra", "corrplot", "ltm", "goeveg", "corrplot", "FSA", "MASS", "scales", "nlme", "psych", "ordinal", "lmtest", "ggpubr", "dslabs", "stringr", "assist", "ggstatsplot", "forcats", "styler", "remedy", "snakecaser", "addinslist", "esquisse", "here", "summarytools", "magrittr", "tidyverse", "funModeling", "pander", "cluster", "abind")

# Install packages not yet installed
installed_packages <- packages %in% rownames(installed.packages())
if (any(installed_packages == FALSE)) {
  install.packages(packages[!installed_packages])
}

# Packages loading
invisible(lapply(packages, library, character.only = TRUE))

This code for installing and loading R packages is more efficient in several ways:

The function install.packages() accepts a vector as argument, so one line of code for each package in the past is now one line including all packages
In the second part of the code, it checks whether a package is already installed or not, and then install only the missing ones
Regarding the packages loading (the last part of the code), the lapply() function is used to call the library() function on all packages at once, which makes the code more condense.
The output when loading a package is rarely useful. The invisible() function removes this output.

From that day on, every time I need to use a new package, I simply add it to the vector packages at the top of the code, which is located at the top of my scripts and R Markdown documents. No matter on which computer I am working on, running the entire code will install only the missing packages and will load all of them. This greatly reduced the running time for the installation and loading of my R packages.

Most efficient way

`{pacman}` package

After this article was published, a reader informed me about the {pacman} package. After having read the documentation and try it out myself, I learned that the function p_load() from {pacman} checks to see if a package is installed, if not it attempts to install the package and then loads it. It can also be applied to several packages at once, all this in a very condensed way:

install.packages("pacman")

pacman::p_load(ggplot2, tidyr, dplyr)

Find more about this package on CRAN.

`{librarian}` package

Like {pacman}, the shelf() function from the {librarian} package automatically installs, updates, and loads R packages that are not yet installed in a single function. The function accepts packages from CRAN, GitHub, and Bioconductor (only if Bioconductor’s Biobase package is installed). The function also accepts multiple package entries, provided as a comma-separated list of unquoted names (so no "" around package names).

Last but not least, the {librarian} package allows to load packages automatically at the start of every R session (thanks to the lib_startup() function) and search for new packages on CRAN by keywords or regular expressions (thanks to the browse_cran() function).

Here is an example of how to install missing packages and load them with the shelf() function:

# From CRAN:
install.packages("librarian")

librarian::shelf(ggplot2, DesiQuintans / desiderata, pander)

For CRAN packages, provide the package name as normal without "" and for GitHub packages, provide the username and package name separated by / (i.e., UserName/RepoName as shown for the desiderata package).

Find more about this package on CRAN.

Conclusion

Thanks for reading.

I hope the article helped you to install and load R packages in a more efficient way.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A special thanks Danilo and James for informing me about the {pacman} and {librarian} packages.

As suggested by Patrick, it is a good practice to comment (with a # in front of the line of code) the installation of your packages after they have been installed on your computer. This avoids installing packages on someone else’s computer when you share your code. If they want to install them before running your code, they will need to do it by themselves.↩︎

Package on Stats and R

Bayesian Neural Networks in {tidymodels} with {kindling}

What Are Bayesian Neural Networks?

From point estimates to distributions

The reparameterisation trick

Is it possible?

Setup

Loading namespace

The train_nnsnip() Interface

Classification: Iris

The drawbacks

Updating the model

References

You can do more for neural networks in R with {kindling}

Why this post matters

What problem {kindling} solves (and what it does not)

Setup

Three levels of interaction

1) Generate model code (*_generator())

2) Direct Training

3) ML Framework Integration: {tidymodels} with mlp_kindling() / rnn_kindling()

Example 1: Iris classification with reproducible good practice

Data Preprocessing

MLP with 2 Hidden Layers

Why these choices?

Example 2: A more realistic tabular benchmark (Sonar)

About early stopping and callbacks

Supported architectures (current scope)

Variable importance (FFNN)

When {kindling} is a good fit

Limitations to keep in mind

Takeaways

nycOpenData: A unified R interface to NYC Open Data APIs

nycOpenData: A unified R interface to NYC Open Data APIs

Why nycOpenData?

What does the package do?

Mini analysis

Designed for reproducible workflows

Who is it for?

Availability

Acknowledgements

AssociationExplorer: A user-friendly shiny application for exploring associations and visual patterns

Why AssociationExplorer?

What does the app do?

A paper published in SoftwareX

The R package is also on CRAN

Who is it for, and how can it be used?

Acknowledgements

Useful links

References

Nonparametric serial interval estimation

Motivation

Simulated data

Real data

References

EpiLPS for estimation of incubation times

Motivation

Coarse data

Simulated example

Real data example

References

Google Analytics in R: Review of 2022

Introduction

Prerequisites

Analytics

Page views over time

Page views per month and year

Top performing pages

Page views by country

Page views per day of week

Thank you note

Paper: 'EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number'

Introduction

Motivation

Getting started

A simulated example

Smoothing the epidemic curve and estimating \(\mathcal{R}_t\)

USA hospitalization data

Conclusion

References

The `train_nnsnip()` Interface

What problem `{kindling}` solves (and what it does not)

1) Generate model code (`*_generator()`)

3) ML Framework Integration: `{tidymodels}` with `mlp_kindling()` / `rnn_kindling()`

When `{kindling}` is a good fit

Basic principles of `{ggplot2}`

Create plots with `{ggplot2}`

Interactive plot with `{plotly}`

Combine plots with `{patchwork}`

Highlight data with `{gghighlight}`