Collaboration on Stats and R

EM-DAT, the world's disaster memory, is at risk

Tue, 07 Apr 2026 00:00:00 +0000

I do not usually write posts that are calls to action. But sometimes, something important enough comes along that it would feel wrong to stay silent. This is one of those times.

What is EM-DAT?

EM-DAT, the Emergency Events Database, is the world’s most widely used and trusted global database for tracking natural and technological disasters. It has been maintained since 1988 by the Centre for Research on the Epidemiology of Disasters (CRED), which is part of UCLouvain.

The database currently contains data on the occurrence and impacts of over 27,000 mass disasters worldwide, from 1900 to the present day. It covers floods, storms, earthquakes, droughts, wildfires, extreme temperatures, landslides, volcanic activity, and technological accidents, across virtually every country on earth.

Crucially, it is:

Open access (for non-commercial use)
Globally comparable, using transparent and consistent inclusion criteria
Cross-verified across multiple sources (UN agencies, NGOs, reinsurance companies, research institutes, press agencies)
The reference dataset for thousands of peer-reviewed studies, national risk assessments, and international policy processes

If you have ever read a paper or report about global disaster trends, the probability is high that EM-DAT was the data source behind it.

Why is it at risk?

For more than 25 years, EM-DAT was primarily funded by the United States Agency for International Development (USAID). Following the recent dismantling of USAID, that funding is gone, and no sustainable alternative has been secured.

This is not a minor budget shortfall. Without a replacement funding mechanism, EM-DAT risks shutting down entirely.

Why does it matter?

The open letter drafted in support of EM-DAT puts it well: in an era of intensifying climate extremes, cascading risks, and compounding crises, reliable data are not a luxury. They are the infrastructure for informed decision-making.

Concretely, EM-DAT underpins:

Disaster risk reduction and prevention policies, used by governments to assess national risks and prioritise investments
Humanitarian operations, relied upon by multilateral agencies and NGOs to plan and forecast needs
Climate research, providing historical baselines for understanding trends in extreme weather events
Monitoring of global commitments, such as the Sendai Framework for Disaster Risk Reduction, the SDGs, and the Paris Agreement
Insurance and risk modelling, used by the private sector alongside other data to benchmark losses and refine exposure models

EM-DAT’s value is not just in the quantity of records. It lies in the rigour and consistency of its methodology over time and across countries. That is exactly what makes it irreplaceable. In a world awash with data, curated and quality-controlled datasets of this kind are rare. If EM-DAT were to close, the result would not be a smooth substitution. It would be fragmentation, proprietary data silos, and reduced access, particularly for lower-income countries that are already under-represented in global evidence.

A personal note

I signed the open letter after being informed of the issue by my colleague Prof. Niko Speybroeck, a leading epidemiologist at UCLouvain and program director of the CRED.

I do not have direct expertise in disaster epidemiology. But I do care about open data, open science, and the integrity of global research infrastructure. And EM-DAT is exactly the kind of resource that the whole scientific community relies on, often without fully realising it.

How you can help

If you share these values, I encourage you to sign the open letter: “The World’s collective disaster memory must be preserved”.

The letter calls on governments, multilateral development banks, philanthropic foundations, and international organisations to step forward with a coordinated and sustainable funding arrangement for EM-DAT. The cost of maintaining the world’s primary disaster database is modest set against the billions spent on disaster response and recovery each year. The cost of losing it would be profound.

Please also consider sharing this post or the open letter with your own network (researchers, policymakers, students, practitioners, or anyone who cares about data-driven approaches to global challenges).

More information

EM-DAT website: https://www.emdat.be/
Open letter: https://openletter.earth/the-worlds-collective-disaster-memory-must-be-preserved-66c88c44
CRED at UCLouvain: https://www.uclouvain.be/en/research-institutes/irss/cred-epidemiology-of-disasters

As always, if you have any thoughts or questions related to this post, feel free to leave a comment below.

Bayesian Neural Networks in {tidymodels} with {kindling}

Fri, 20 Mar 2026 00:00:00 +0000

What Are Bayesian Neural Networks?

Standard neural networks learn fixed weights during training and produce a single point estimate for each input — with no sense of how confident the model is. Bayesian Neural Networks (BNNs) replace those fixed weights with probability distributions (Neal 2012). Instead of learning a single value per weight, the network learns a mean and a variance, and samples from that distribution at every forward pass.

Formally, a BNN places a prior $p(\mathbf{w})$ over the weights and updates it with observed data $\mathcal{D}$ via Bayes’ theorem:

\[p(\mathbf{w} \mid \mathcal{D}) = \frac{p(\mathcal{D} \mid \mathbf{w})\, p(\mathbf{w})}{p(\mathcal{D})}\]

For large and deep networks, the posterior $p(\mathbf{w} \mid \mathcal{D})$ is intractable. BNNs typically use variational inference to approximate it with a tractable distribution $q_\phi(\mathbf{w})$ by minimising the KL divergence between the two (Blundell et al. 2015).

From point estimates to distributions

A standard linear layer computes $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$, where $\mathbf{W}$ and $\mathbf{b}$ are fixed. In a Bayesian linear layer, they are random variables. We place a Gaussian prior over the weights:

\[\mathbf{W} \sim \mathcal{N}(\mu_{\text{prior}},\ \sigma_{\text{prior}}^2)\]

and during training we learn a variational posterior:

\[q(\mathbf{W}) = \mathcal{N}(\mu_W,\ \sigma_W^2)\]

Each weight therefore has two learnable scalars: a mean $\mu_W$ and a log-standard-deviation $\log \sigma_W$.

The reparameterisation trick

Sampling directly from $q(\mathbf{W})$ is not differentiable with respect to $\mu_W$ and $\sigma_W$. We resolve this with the reparameterisation trick (Kingma and Welling 2013):

\[\mathbf{W} = \mu_W + \sigma_W \odot \varepsilon, \quad \varepsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\]

We store $\log \sigma_W$ (not $\sigma_W$ directly) to ensure positivity during unconstrained optimisation:

\[\sigma_W = \exp(\texttt{weight}_{\log\sigma})\]

This yields the sampled weight used in the forward pass:

\[\mathbf{W}_{\text{sample}} = \mu_W + \exp(\texttt{weight}_{\log\sigma}) \odot \varepsilon_W\]

and similarly for the bias:

\[\mathbf{b}_{\text{sample}} = \mu_b + \exp(\texttt{bias}_{\log\sigma}) \odot \varepsilon_b\]

Is it possible?

At the moment, {tidymodels} offers limited APIs for neural network architectures, mostly around standard MLPs. A few packages make {torch} easier to use at a higher level:

{brulee} bridges {torch} with {tidymodels}. It is ergonomic, but currently focused on MLPs for tabular data.
{cito} does not integrate with {tidymodels} and leans more toward statistical applications.
The {torch} team maintains {luz}.
{tabnet} integrates with {tidymodels}, but is dedicated to the TabNet architecture.

{kindling} does not replace these packages. However, it helps close some of these gaps by supporting custom architectures with flexible depth. As a result, custom neural network architectures, including BNNs, can be used within {tidymodels} workflows. For a broader overview of what {kindling} enables in R, see this earlier post.

Setup

On March 3, 2026, {kindling} v0.3.0 was released on CRAN. Remember that utils::install.packages() in R always installs the latest version.

install.packages("kindling")

If you want the development version, you can retrieve the package by installing it from GitHub:

pak::pak("joshuamarie/kindling")

I recommend using {pak}: it is fast and very good at resolving package dependencies across environments, including system libraries and {renv}-managed projects.

Before using {kindling}, install LibTorch — the C++ backend shared by PyTorch and the {torch} R package (Falbel and Luraschi 2023):

torch::install_torch()

Loading namespace

When teaching R, I recommend using box::use() and qualifying the names you import. Create a bnn folder in your project root, then clone this repository to use BNNs in R (this module is adapted from torchbnn (Kim 2020)): https://github.com/joshuamarie/RTorchBNN.

You can run a specific command:

git clone --branch main https://github.com/joshuamarie/RTorchBNN bnn

Then, load the following:

box::use(
  bnn = . / bnn,
  kindling[train_nnsnip, nn_arch, act_funs],
  parsnip[fit, augment],
  yardstick[metrics]
)

Then register the models in {parsnip} with:

loadNamespace("kindling")

The `train_nnsnip()` Interface

In {kindling} v0.3.0, train_nnsnip() was introduced as a {parsnip}-compatible model specification that bridges the generalized neural network trainer with the {tidymodels} ecosystem. Unlike mlp_kindling(), which is specific to feedforward networks, train_nnsnip() is architecture-agnostic: you describe the layer topology via nn_arch(), allowing BNN-style or other custom layer types to slot in without changing the training logic.

Classification: Iris

We train a Bayesian Neural Network model to predict species in the iris dataset via train_nnsnip(), by configuring arch with nn_arch(). Inside nn_arch(), set nn_layer to bnn$BayesLinear instead of the default linear layer, then define the arguments for each layer with layer_arg_fn.

nn_model <-
  train_nnsnip(
    mode = "classification",
    arch = nn_arch(
      nn_layer = bnn$BayesLinear,
      out_nn_layer = torch::nn_linear,
      layer_arg_fn = ~ if (.is_output) {
        list(.in, .out)
      } else {
        list(
          in_features = .in,
          out_features = .out,
          prior_mu = 0,
          prior_sigma = 0.1
        )
      }
    ),
    hidden_neurons = c(64, 32),
    activations = act_funs(relu, elu),
    loss = "cross_entropy",
    epochs = 50,
    verbose = TRUE,
    learn_rate = 1e-3,
    optimizer_args = list(weight_decay = 0.01)
  ) |>
  fit(Species ~ ., data = iris)

nn_model |>
  augment(new_data = iris) |>
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass      0.68
## 2 kap      multiclass      0.52

Averaging predictions over multiple forward passes gives both a point estimate and a measure of predictive spread — all within a standard {tidymodels} pipeline.

The drawbacks

For now, there is no built-in option to update the loss dynamically during training. To use a custom objective such as the Evidence Lower Bound (ELBO), you currently need to define it manually and retrain with that loss.

Updating the model

That said, you can still apply a custom loss by setting the loss parameter to a torch function, for example torch::nnf_mse_loss(). If you want to train the model with an ELBO-style loss, do the following:

make_elbo_loss <- function(bnn_model, n_obs, kl_weight = 1.0) {
  box::use(
    torch[nnf_cross_entropy, torch_tensor]
  )

  function(input, target) {
    ce <- nnf_cross_entropy(input, target)
    device <- input$device
    kl_val <- torch_tensor(0.0, device = device, requires_grad = FALSE)
    for (m in bnn_model$modules) {
      if (inherits(m, "BayesLinear") && !is.null(m$kl)) {
        kl_val <- kl_val + m$kl
      }
    }

    ce + kl_weight * kl_val / n_obs
  }
}

elbo_loss <- make_elbo_loss(nn_model$fit$model, n_obs = nrow(iris), kl_weight = 1.0)

nn_model2 <-
  train_nnsnip(
    mode = "classification",
    arch = nn_arch(
      nn_layer = bnn$BayesLinear,
      out_nn_layer = torch::nn_linear,
      layer_arg_fn = ~ if (.is_output) {
        list(.in, .out)
      } else {
        list(
          in_features = .in,
          out_features = .out,
          prior_mu = 0,
          prior_sigma = 0.1
        )
      }
    ),
    hidden_neurons = c(64, 32),
    activations = act_funs(relu, elu),
    loss = elbo_loss,
    epochs = 50,
    verbose = TRUE,
    learn_rate = 1e-3,
    optimizer_args = list(weight_decay = 0.01)
  ) |>
  fit(Species ~ ., data = iris)

nn_model2 |>
  augment(new_data = iris) |>
  metrics(truth = Species, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass      0.72
## 2 kap      multiclass      0.58

As always, if you have any question related to the topic covered in this post, please add it as a comment so other readers can benefit from the discussion.

References

Blundell, Charles, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. “Weight Uncertainty in Neural Network.” International Conference on Machine Learning, 1613–22.

Falbel, Daniel, and Javier Luraschi. 2023. Torch: Tensors and Neural Networks with GPU Acceleration. https://github.com/mlverse/torch.

Kim, Harry. 2020. Bayesian Neural Network for PyTorch [Torchbnn]. https://github.com/Harry24k/bayesian-neural-network-pytorch.

Kingma, Diederik P, and Max Welling. 2013. “Auto-Encoding Variational Bayes.” arXiv Preprint arXiv:1312.6114.

Neal, Radford M. 2012. Bayesian Learning for Neural Networks. Vol. 118. Springer Science & Business Media.

You can do more for neural networks in R with {kindling}

Thu, 19 Feb 2026 00:00:00 +0000

Why this post matters

Neural networks in R are no longer niche. Today, we can choose among:

{nnet} for classic, small-scale neural nets,
{neuralnet} another classic neural nets package besides {nnet},
{keras} / {keras3} for the Keras API (typically with Python backends such as TensorFlow/JAX/Torch),
{torch} for native-R deep learning with explicit model and training control.

So why discuss another package?

In our experience, many day-to-day projects sit in the middle: we want more flexibility than a classical model, but less boilerplate than writing a full {torch} training loop. {kindling} is designed for that middle ground.

In this article, we focus on what {kindling} does well, where it helps, and where it is still limited.

What problem `{kindling}` solves (and what it does not)

{kindling} is a higher-level interface built on top of {torch}. In practice, it reduces repetitive code for:

defining common network architectures,
training feedforward and recurrent models,
integrating with {tidymodels} ({parsnip}, {recipes}, {workflows}, {tune}).

It does not replace low-level {torch} for highly customized research code, and it does not make hardware setup disappear. You still need a working LibTorch installation and an environment that can use CPU/GPU properly.

At the time of writing (CRAN version 0.2.0), core built-in architectures include FFNN/MLP and recurrent variants (RNN/LSTM/GRU).

Setup

Install this package in two different ways:

Solely install {kindling} (as it installs the package dependencies internally)

install.packages("kindling")

You can install the package dependencies separately:

install.packages(c(
  "kindling",
  # dependencies
  "torch", "dplyr", "rsample", "recipes", "yardstick",
  "workflows", "parsnip", "tibble", "ggplot2", "mlbench", "vip"
))

Then load the packages in two ways:

Using library() traditionally:

library(kindling)
library(torch)
library(dplyr)
library(rsample)
library(recipes)
library(yardstick)
library(workflows)
library(parsnip)
library(tibble)
library(ggplot2)
library(mlbench)
library(vip)

In this article, we use library() for clarity and copy-paste reproducibility.

If you prefer a module-based and more explicit import style, use box::use() (read Box: Placing module system into R for more details).

Here’s the equivalent code as above:

box::use(
kindling[...],
dplyr[...],
rsample[...],
recipes[...],
yardstick[...],
workflows[...],
parsnip[...],
tibble[...],
ggplot2[...],
mlbench[...],
vip[...]
)

{kindling} uses {torch} as backend, so LibTorch must be installed once:

if (!torch::torch_is_installed()) {
  torch::install_torch()
}

Three levels of interaction

{kindling} is useful because you can work at different abstraction levels. Here are three ways to interact with the package, ordered from lowest to highest level of abstraction:

1) Generate model code (`*_generator()`)

If you want to inspect architecture code before training:

ffnn_generator(
  nn_name = "MyNet",
  hd_neurons = c(64, 32, 16),
  no_x = 10,
  no_y = 1,
  activations = act_funs(
    relu,
    "softplus(beta = 0.5)",
    selu
  )
)

## torch::nn_module("MyNet", initialize = function () 
## {
##     self$fc1 = torch::nn_linear(10, 64, bias = TRUE)
##     self$fc2 = torch::nn_linear(64, 32, bias = TRUE)
##     self$fc3 = torch::nn_linear(32, 16, bias = TRUE)
##     self$out = torch::nn_linear(16, 1, bias = TRUE)
## }, forward = function (x) 
## {
##     x = self$fc1(x)
##     x = torch::nnf_relu(x)
##     x = self$fc2(x)
##     x = torch::nnf_softplus(x, beta = 0.5)
##     x = self$fc3(x)
##     x = torch::nnf_selu(x)
##     x = self$out(x)
##     x
## })

Technical note: To specify a parametric activation functions like Softplus under act_funs() function, set softplus = args(beta = 0.5) or softplus[beta = 0.5] (available in v0.3.x and later), not just in a stringly typed expression, e.g. "softplus(beta = 0.5)".

This returns an unevaluated torch::nn_module expression you can inspect or modify.

2) Direct Training

The functions available (for now) are ffnn() and rnn(). For many applied tasks, this is the fastest way to fit a network from a formula.

mini_direct <- ffnn(
  mpg ~ .,
  data = mtcars,
  hidden_neurons = 8,
  activations = act_funs(relu),
  epochs = 20,
  verbose = FALSE
)

predict(mini_direct, newdata = head(mtcars, 3))

## [1] -0.01706841 -0.04320815 -0.71570069

3) ML Framework Integration: `{tidymodels}` with `mlp_kindling()` / `rnn_kindling()`

{kindling} functions to directly train the said models also (currently) integrates with {tidymodels}. This level is ideal when we want recipes, workflows, resampling, and tuning.

mini_spec <- mlp_kindling(
  mode = "regression",
  hidden_neurons = 8,
  activations = act_funs(relu), # or just "relu"
  epochs = 20,
  verbose = FALSE
)

mini_wf <- workflow() |>
  add_formula(mpg ~ .) |>
  add_model(mini_spec)

mini_fit <- fit(mini_wf, data = mtcars)

predict(mini_fit, new_data = head(mtcars, 3))

## # A tibble: 3 × 1
##   .pred
##   <dbl>
## 1  9.28
## 2  9.31
## 3  5.63

Now that we’ve covered the three levels of interaction, let’s see {kindling} in action with two “realistic” examples. We’ll demonstrate how to structure reproducible workflows, handle feature preprocessing, and evaluate results properly. Along the way, you’ll see which abstraction level works best for different scenarios.

Example 1: Iris classification with reproducible good practice

This first example uses direct training with ffnn(), but still follows practical steps:

train/test split,
feature scaling,
validation split during training,
regularization,
out-of-sample evaluation.

Data Preprocessing

Train/test split is performed to prevent evaluating only on training data.

data(iris)

iris_split <- initial_split(iris, prop = 0.8, strata = Species)
iris_train <- training(iris_split)
iris_test <- testing(iris_split)

iris_recipe <- recipe(Species ~ ., data = iris_train) |>
  step_normalize(all_predictors())
iris_prep <- prep(iris_recipe, training = iris_train)
iris_train_processed <- bake(iris_prep, new_data = NULL)
iris_test_processed <- bake(iris_prep, new_data = iris_test)

MLP with 2 Hidden Layers

set.seed(20260218)

iris_mlp <- ffnn(
  Species ~ .,
  data = iris_train_processed,
  hidden_neurons = c(32, 16),
  activations = act_funs(relu, "softplus(beta = 0.5)"),
  loss = "cross_entropy",
  optimizer = "adam",
  learn_rate = 0.01,
  penalty = 1e-4,
  mixture = 0,
  batch_size = 16,
  epochs = 200,
  validation_split = 0.2,
  verbose = FALSE
)

iris_mlp

## 
## ======================= Feedforward Neural Networks (MLP) ======================
## 
## 
## -- FFNN Model Summary ----------------------------------------------------------
## 
## 
## -------------------------------------------------------------------------------------
##   NN Model Type           :             FFNN    n_predictors :                    4
##   Number of Epochs        :              200    n_response   :                    3
##   Hidden Layer Units      :           32, 16    reg.         :   [λ = 1e-04, α = 0]
##   Number of Hidden Layers :                2    Device       :                  mps
##   Pred. Type              :   classification                 :                     
## -------------------------------------------------------------------------------------
## 
## 
## 
## -- Activation function ---------------------------------------------------------
## 
## 
##                -------------------------------------------------
##                  1st Layer {32}    :                      relu
##                  2nd Layer {16}    :      softplus(beta = 0.5)
##                  Output Activation :   No act function applied
##                -------------------------------------------------

iris_pred <- predict(iris_mlp, newdata = iris_test_processed, type = "response")

iris_eval <- tibble(
  truth = iris_test_processed$Species,
  .pred_class = iris_pred
)

metrics(iris_eval, truth = truth, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy multiclass     0.933
## 2 kap      multiclass     0.9

conf_mat(iris_eval, truth = truth, estimate = .pred_class)

##             Truth
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         1
##   virginica       0          1         9

Why these choices?

Here’s the brief explanation on how we set up the models from {kindling}:

Architecture (c(32, 16)): The architecture has 2 layers with 32 units for the 1st layer and 16 for the 2nd layer. It is enough capacity for Iris, but still simple.
Activations: The architecture has 2 hidden layers, each layer is specified with different activation functions.
1. relu: Also called Rectified Linear Unit (ReLU); the most stable default for hidden layers.
2. softplus: Similar to ReLU but much smoother in approximation; beta is adjusted to 0.5.
Scaling: crucial because neural networks are sensitive to predictor scale.
Regularization (penalty and mixture): helps reduce overfitting even on small data. mixture is set to 0, which means the optimization is performed with L2 regularization.
Validation split: monitors generalization during training.

Example 2: A more realistic tabular benchmark (Sonar)

Iris is useful for teaching, but sometimes considered (too) easy. The Sonar dataset (60 numeric predictors, binary outcome) is a better stress test for tabular classification.

data("Sonar", package = "mlbench")

sonar <- Sonar |>
  mutate(Class = factor(Class))

sonar_split <- initial_split(sonar, prop = 0.8, strata = Class)
sonar_train <- training(sonar_split)
sonar_test <- testing(sonar_split)

Now fit through {tidymodels} so preprocessing and modeling stay in one pipeline:

sonar_rec <- recipe(Class ~ ., data = sonar_train) |>
  step_normalize(all_predictors())

sonar_spec <- mlp_kindling(
  mode = "classification",
  hidden_neurons = c(64, 32),
  activations = act_funs(relu, relu),
  optimizer = "adam",
  learn_rate = 0.001,
  penalty = 1e-4,
  mixture = 0,
  batch_size = 16,
  epochs = 250,
  validation_split = 0.2,
  verbose = FALSE
)

sonar_wf <- workflow() |>
  add_recipe(sonar_rec) |>
  add_model(sonar_spec)

set.seed(20260218)
sonar_fit <- fit(sonar_wf, data = sonar_train)

sonar_pred <- augment(sonar_fit, new_data = sonar_test)

metrics(sonar_pred, truth = Class, estimate = .pred_class)

## # A tibble: 2 × 3
##   .metric  .estimator .estimate
##   <chr>    <chr>          <dbl>
## 1 accuracy binary         0.860
## 2 kap      binary         0.720

In practice, this setup is often a strong baseline for tabular binary classification before trying larger architectures.

About early stopping and callbacks

If you come from Keras/TensorFlow, you may be used to callback objects (early stopping, learning-rate schedules, etc.).

With {kindling} 0.2.0, practical overfitting control is usually handled through:

validation_split,
regularization (penalty, mixture),
tuning model size and training length (epochs),
proper resampling with {tidymodels}.

In other words, we can approximate early-stopping behavior by selecting epochs via validation/resampling, even without a callback-heavy workflow.

Supported architectures (current scope)

Architecture	Main function(s)	Typical use
Feedforward (MLP/FFNN)	ffnn(), mlp_kindling()	Tabular regression/classification
RNN	rnn_kindling(rnn_type = "rnn")	Sequential patterns
LSTM	rnn_kindling(rnn_type = "lstm")	Longer-range sequence dependencies
GRU	rnn_kindling(rnn_type = "gru")	Sequence modeling with fewer parameters

Variable importance (FFNN)

Interpretability for neural networks is imperfect, but {kindling} integrates established approaches for FFNN models.

# For FFNN fits:
garson(iris_mlp, bar_plot = FALSE)

##        x_names y_names  rel_imp
## 1 Petal.Length       y 34.31493
## 2  Sepal.Width       y 24.68379
## 3  Petal.Width       y 22.43818
## 4 Sepal.Length       y 18.56311

olden(iris_mlp, bar_plot = FALSE)

##        x_names y_names    rel_imp
## 1 Petal.Length       y -5.1100721
## 2  Petal.Width       y -2.9793542
## 3  Sepal.Width       y  0.8302629
## 4 Sepal.Length       y -0.1882139

# Via vip (Olden/Garson methods supported by kindling S3 methods)
vi(iris_mlp, type = "olden") |>
  vip()

We recommend using these as directional diagnostics, not as causal evidence.

When `{kindling}` is a good fit

{kindling} is a practical choice when our projects:

work mostly in R and want to stay inside {tidymodels},
need neural nets for tabular or moderate sequence tasks,
want less boilerplate than raw {torch} but still meaningful control.

It may be less ideal when our projects need:

custom research architectures and training loops,
mature callback ecosystems similar to high-level Keras workflows,
highly optimized distributed production pipelines.

Limitations to keep in mind

Hardware setup still matters: GPU usage depends on a correct {torch}/LibTorch installation and supported hardware.
Ecosystem maturity: this is a young package; interfaces can evolve.
Debugging depth: for deeply custom debugging, low-level {torch} remains the reference path.
Model class assumptions: recurrent models are for sequence-structured data; using them on plain tabular data is usually not appropriate.

Takeaways

{kindling} is not about replacing {torch} or {keras3}. It is about reducing friction for common deep-learning workflows in R.

For many applied projects, a robust pattern is:

preprocess with {recipes},
start with a modest MLP architecture,
use validation split + regularization,
evaluate on held-out test data,
tune only after you have a strong baseline.

If this matches your workflow, {kindling} is worth trying.

As always, if you have any question related to the topic covered in this post, please add it as a comment so other readers can benefit from the discussion.

nycOpenData: A unified R interface to NYC Open Data APIs

Tue, 27 Jan 2026 00:00:00 +0000

Guest post by Christian Martinez, developer of the nycOpenData package in R.

nycOpenData: A unified R interface to NYC Open Data APIs

I am pleased to announce the release of nycOpenData, an R package providing convenient, tidy access to dozens of datasets from the New York City Open Data platform.

The package is designed as part of an open-science and reproducible-research effort, with the goal of lowering the friction between public data and statistical analysis—especially for teaching, exploratory research, and applied civic work.

Why nycOpenData?

NYC Open Data hosts hundreds of datasets covering topics such as public safety, housing, transportation, education, health, and city services. While these datasets are publicly accessible through the Socrata API, working with them directly often requires:

knowing dataset identifiers,
manually constructing API queries,
handling pagination, timeouts, and rate limits,
and performing repetitive data-cleaning steps.

These barriers can slow down exploratory analysis and make public data less accessible to students, researchers, and practitioners who primarily work in R.

nycOpenData was built to remove these obstacles by providing a consistent, user-friendly interface that returns clean tibbles ready for analysis—without requiring users to interact directly with the API.

What does the package do?

The package provides a growing collection of wrapper functions, each corresponding to a specific NYC Open Data dataset or dataset family. All functions follow a shared design pattern and support:

row limits,
optional filtering via named lists,
sorting,
and graceful handling of API errors and timeouts.

Examples of currently supported domains include:

311 service requests
Transportation and for-hire vehicles
Motor vehicle collisions
Department of Buildings permits and complaints
Education and school reporting
Juvenile justice and public safety
Street trees and environmental data
Permitted events (historical)

A typical call looks like this:

library(nycOpenData)

nyc_311(
  limit = 1000,
  filters = list(borough = "BROOKLYN")
)

## # A tibble: 1,000 × 40
##    unique_key created_date          agency agency_name complaint_type descriptor
##    <chr>      <chr>                 <chr>  <chr>       <chr>          <chr>     
##  1 67613985   2026-01-26T02:06:05.… NYPD   New York C… Noise - Resid… Banging/P…
##  2 67609553   2026-01-26T02:02:09.… NYPD   New York C… Noise - Resid… Banging/P…
##  3 67610990   2026-01-26T01:58:58.… NYPD   New York C… Illegal Parki… Blocked H…
##  4 67615428   2026-01-26T01:56:49.… NYPD   New York C… Noise - Resid… Banging/P…
##  5 67609568   2026-01-26T01:48:16.… NYPD   New York C… Noise - Resid… Loud Musi…
##  6 67612476   2026-01-26T01:47:10.… NYPD   New York C… Noise - Resid… Loud Musi…
##  7 67614152   2026-01-26T01:46:26.… DSNY   Department… Snow or Ice    Snow Trac…
##  8 67614054   2026-01-26T01:44:50.… DSNY   Department… Dirty Conditi… Trash     
##  9 67606570   2026-01-26T01:41:32.… NYPD   New York C… Noise - Resid… Banging/P…
## 10 67610091   2026-01-26T01:35:51.… NYPD   New York C… Noise - Vehic… Car/Truck…
## # ℹ 990 more rows
## # ℹ 34 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>, …

The result is returned as a tidy tibble of the 1,000 most recent NYC 311 requests, making it immediately compatible with the tidyverse ecosystem for visualization, modeling, and reporting.

Mini analysis

One of the strongest qualities this function has is its ability to filter based on multiple columns. Let’s put everything together and get a dataset of the last 1,000 311 requests from the New York Police Department in Brooklyn.

# Creating the dataset
brooklyn_nypd <- nyc_311(limit = 1000, filters = list(agency = "NYPD", borough = "BROOKLYN"))

# Calling head of our new dataset
head(brooklyn_nypd)

## # A tibble: 6 × 39
##   unique_key created_date           agency agency_name complaint_type descriptor
##   <chr>      <chr>                  <chr>  <chr>       <chr>          <chr>     
## 1 67613985   2026-01-26T02:06:05.0… NYPD   New York C… Noise - Resid… Banging/P…
## 2 67609553   2026-01-26T02:02:09.0… NYPD   New York C… Noise - Resid… Banging/P…
## 3 67610990   2026-01-26T01:58:58.0… NYPD   New York C… Illegal Parki… Blocked H…
## 4 67615428   2026-01-26T01:56:49.0… NYPD   New York C… Noise - Resid… Banging/P…
## 5 67609568   2026-01-26T01:48:16.0… NYPD   New York C… Noise - Resid… Loud Musi…
## 6 67612476   2026-01-26T01:47:10.0… NYPD   New York C… Noise - Resid… Loud Musi…
## # ℹ 33 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>,
## #   x_coordinate_state_plane <chr>, y_coordinate_state_plane <chr>, …

# Quick check to make sure our filtering worked
nrow(brooklyn_nypd)

## [1] 1000

unique(brooklyn_nypd$agency)

## [1] "NYPD"

unique(brooklyn_nypd$borough)

## [1] "BROOKLYN"

We successfully created our dataset that contains the 1,000 most recent requests regarding the NYPD in the borough Brooklyn.

Now that we have successfully pulled the data and have it in R, let’s figure out what NYC residents in Brooklyn are complaining about to the NYPD.

To do this, we will create a bar graph of the complaint types.

# Visualizing the distribution, ordered by frequency
library(ggplot2)

ggplot(brooklyn_nypd, aes(y = reorder(complaint_type, complaint_type, length))) +
  geom_bar(fill = "steelblue") +
  theme_minimal() +
  labs(
    title = "Most Recent NYPD 311 Complaints (Brooklyn)",
    subtitle = "Top 1,000 service requests",
    x = "Number of Complaints",
    y = "Type of Complaint"
  )

Figure 1: Bar chart showing the frequency of NYPD-related 311 complaint types in Brooklyn from the 1,000 most recent service requests.

This graph shows us not only which complaints were made, but how many of each complaint were made.

Designed for reproducible workflows

A core design principle of nycOpenData is reproducibility. Rather than downloading static CSV files that can change over time or be accidentally modified, analyses can explicitly document:

which dataset was used,
how many rows were requested,
which filters were applied,
and when the data were accessed.

This makes the package particularly useful for:

reproducible research projects,
classroom assignments,
data journalism,
and exploratory civic analysis.

The package is also designed to be API-polite, with configurable timeouts and safeguards that help prevent common failure modes when querying large public datasets.

Who is it for?

nycOpenData is intended for a broad audience, including:

students learning statistics or data science using real-world data,
instructors teaching reproducible research or applied data analysis,
researchers conducting exploratory or descriptive analyses,
data journalists and civic technologists,
and anyone interested in working with NYC public data in R.

The goal is not to abstract away the data itself, but to make access predictable, transparent, and easy to integrate into standard R workflows.

Availability

The package is available on CRAN and can be installed using:

install.packages("nycOpenData")

Development continues on GitHub, where new datasets and improvements are added regularly.

Acknowledgements

This package was developed alongside teaching and applied research projects in reproducible data science, with inspiration from open-source contributors across the R community and the NYC Open Data program.

Useful links

CRAN package: https://CRAN.R-project.org/package=nycOpenData
pkgdown site: https://martinezc1.github.io/nycOpenData/
GitHub repository: https://github.com/martinezc1/nycOpenData
NYC Open Data portal: https://opendata.cityofnewyork.us/

As always, feedback, bug reports, and dataset requests are very welcome.

Thanks for reading!

Scrape Yahoo search engine results with R

Thu, 24 Aug 2023 00:00:00 +0000

Note: This is a guest post by Manthan Koolwal, founder of Scrapingdog.

Introduction

Web scraping is the process of extracting data from websites. It is usually done in an automated manner to obtain large amounts of data through various websites, without the need to gather data by hand.

In a previous post, we introduced this method and illustrated it with a Wikipedia page. Although there are a lot of use cases of web scraping, in this blog post, we are restricting ourselves to scraping search results from Yahoo using R. Scraping search engine results can help you with SEO analysis, competitor analysis, keyword research, trend analysis, etc.

Scraping Yahoo search engine results with R

After installing R and RStudio, we first need to load the necessary packages by running the following commands:¹

# install.packages("rvest")
# install.packages("jsonlite")
# install.packages("purrr")

library(rvest)
library(jsonlite)
library(purrr)

The {rvest} package is for web scraping, the {jsonlite} package is for working with JSON data and the {purrr} package is for working with functions and vectors.

It is always better to decide in advance what exactly we are going to scrape. For this tutorial, we are going to scrape search results from this URL:

We are going to scrape the following data points from this page:

Link
Title
Description

For this, we define the URL of the Yahoo search results page that we want to scrape. In this case, we are searching for the word “pizza”.

# URL of the Yahoo search results page
url <- "https://search.yahoo.com/search?p=pizza"

We then use the read_html() function from the {rvest} package to read the HTML content of the provided URL:

# Read the HTML content of the page
page <- read_html(url)

This creates an HTML document object that we can work with:

str(page)

## List of 2
##  $ node:<externalptr> 
##  $ doc :<externalptr> 
##  - attr(*, "class")= chr [1:2] "xml_document" "xml_node"

Here’s where we start the process of extracting the search results:

# Extract search results
results <- page %>%
  html_nodes(".algo-sr") %>% # Selector for search result elements
  html_nodes("a") %>% # Select the <a> elements within the search results
  # Extract link, title, and description attributes
  map_df(~ data.frame(
    link = .x %>% html_attr("href"),
    title = .x %>% html_text(),
    description = .x %>% html_attr("title"),
    stringsAsFactors = FALSE
  ))

In the code above:

We use %>% (pipe operator) to chain multiple operations together for clarity.
First, we use html_nodes(".algo-sr") to select the search result elements with the class .algo-sr. These elements contain the links to the search results.
Within each search result element, we further select the <a> elements using html_nodes("a"). These <a> elements contain the link, title, and description information.
Using map_df(), we iterate through each <a> element and extract the link, title, and description attributes.
We create a data frame with these attributes for each search result.

Finally, we convert the results data frame into JSON format using the toJSON() function from the {jsonlite} package. The pretty = TRUE argument adds indentation for better readability. We use cat() to print the JSON-formatted results to the console.

# Print the results in JSON format
cat(toJSON(results, pretty = TRUE))

## [
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.yelp.com%2fsearch%3ffind_desc%3dpizza%26find_loc%3dBrussels%252C%2bR%25C3%25A9gion%2bde%2bBruxelles-Capitale/RK=2/RS=xW.v1O1REkKKLsO7DRIWoIGq4bA-",
##     "title": "yelp.comhttps://www.yelp.com › searchTop 10 Best pizza Near Brussels, Région de Bruxelles-Capitale"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzIEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.yelp.com%2fsearch%3fcflt%3dpizza%26find_loc%3dBrussels%252C%2bR%25C3%25A9gion%2bde%2bBruxelles-Capitale/RK=2/RS=zynV6ypY6PoaPj4cg5yD9a1x.LU-",
##     "title": "yelp.comhttps://www.yelp.com › searchTHE BEST 10 PIZZA PLACES IN BRUSSELS, RÉGION DE ... - Yelp"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.dominos.com%2fen/RK=2/RS=darBKTs8REd78kRRFhZb8jRT4jY-",
##     "title": "Domino'shttps://www.dominos.com › enPizza Delivery & Carryout, Pasta, Chicken & More | Domino's"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAS_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3ds2TCGJY8%26id%3d7236CCFBE7C0B5849BA9DE100BE4F1F32FCD231D%26thid%3dOIP.s2TCGJY8bjV7G5kps1FEwQHaHa%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.98.1%2fimages%2ftiles%2fmixAndMatchDeal%2fhero.webp%26exph%3d530%26expw%3d530%26q%3dpizza%26ck%3d8B813BA6312D3F558421CB691D0C6657%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=xsbqmguwIFBhAOzGrm.qrjLykcA-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3d%2f6Qs28Cq%26id%3d7236CCFBE7C0B5849BA9E9CE4472C5692CE3F420%26thid%3dOIP._6Qs28CqWWCt4S-yapYAXwHaHa%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.116.0%2fimages%2ftiles%2fmixAndMatchDealPSC%2fhero.webp%26exph%3d530%26expw%3d530%26q%3dpizza%26ck%3d758519383F5F3AF0EA83707449E54C87%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=UHqcKBQXuq8AtTptzc.7rkut3rY-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3dZ33s1urJ%26id%3d7236CCFBE7C0B5849BA9D44071E0F214A85C5EAF%26thid%3dOIP.Z33s1urJ3ZHwJtgKZtWHJAHaDS%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fcms%2fassets%2f58a8e864-d1aa-457a-9d37-fcc1912b10ad%3fim%3dCrop%2crect%3d%28774%2c1015%2c4331%2c1920%29%3bResize%3d%281700%29%2callowExpansion%26exph%3d754%26expw%3d1700%26q%3dpizza%26ck%3d1274A2028C287B1E2E3A8B0EB2F9DE7D%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=d9hH9.gzp8cqI8s0XYrAtGSVGXg-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3deK8nC663%26id%3d7236CCFBE7C0B5849BA99FA63EED83A3990E2BF8%26thid%3dOIP.eK8nC6636jB9TYTyjaHNGgHaFj%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.83.3%2fimages%2ftiles%2fperfectComboDeal%2fside.webp%26exph%3d458%26expw%3d610%26q%3dpizza%26ck%3dCD805B6ADACFFC2697D3BE3D36098B7E%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=KY2oqWHmG3Y6yqMzcVPRlshaVsg-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAT_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3d%2bf29F0Yv%26id%3d7236CCFBE7C0B5849BA985F6EAF3041631C46CA1%26thid%3dOIP.-f29F0YvYvZaKklmamQDSwHaDS%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fcms%2fassets%2f43778453-e8b1-4db6-83fa-00f2dc43d6e3%3fim%3dCrop%2crect%3d%28374%2c325%2c3250%2c1441%29%3bResize%3d%281700%29%2callowExpansion%26exph%3d754%26expw%3d1700%26q%3dpizza%26ck%3d66B92DE33E5F605B85BE7C83BE11836C%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=15I_QuI3QQbRtWZj5auaBfXOKWQ-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAYfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzQEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.albacioixelles.com%2f/RK=2/RS=GTpw.m6z.2eGOKxyo3K8xVrqSPE-",
##     "title": "Albacioixelleshttps://www.albacioixelles.comAuthentic | Albacioixelles | Elsene"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAYvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzUEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.nonalife.com%2f/RK=2/RS=d8sI_7Fy4oweSN3IQQ32zJSfgRY-",
##     "title": "Nonahttps://www.nonalife.comNona"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAY_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzYEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.tripadvisor.com%2fRestaurant_Review-g188644-d2480305-Reviews-Pizza_Saco-Brussels.html/RK=2/RS=52RUI7KZf9vt_R13PVY7sv04RMQ-",
##     "title": "Tripadvisorhttps://www.tripadvisor.com › Restaurant_Review-g188644-dPIZZA SACO, Brussels - Avenue Milcamps 154 - Restaurant ..."
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAZPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzcEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=http%3a%2f%2fwww.pizzasaco.com%2f/RK=2/RS=Zi2tEoNCQKejdFUdm0zsZwuqoig-",
##     "title": "pizzasaco.comhttp://www.pizzasaco.comSaco Pizza Bar"
##   }
## ]

Conclusion

Thanks for reading.

This was a simple tutorial in which we scraped Yahoo search results with R. Following the same process, you can create your own web crawler which can scrape search results from Yahoo for any web query.

Of course, you can scrape other search engines with almost the same technique. Also, you can check out this tutorial on web scraping Google search results using Python.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Note that, as for any R package, it must first be installed (with the install.packages() function) before being loaded (with the library() function).↩︎

10 potential career options with a degree in statistics

Fri, 24 Mar 2023 00:00:00 +0000

This post has been written in collaboration with Daniel Williams.

Introduction

A reader recently contacted me because he was hesitating between starting his studies in statistics or economics. I studied economics and I am now doing a PhD in statistics, so he reached out to me to know what I thought about these two fields, and in particular, what were my feelings about the career opportunities available with these two degrees.

Before considering pursuing a graduate degree, it is important to have a clear understanding of the skills you will develop and the career opportunities that will become available to you.

I leave it to those who have studied economics and work in this field to discuss career opportunities with a degree in economics. In this post, I will focus on the career options that are available for people with a degree in statistics.

Bear in mind that there are many professional opportunities with a degree in statistics. Indeed, statistical expertise is highly sought after in a wide range of computing and data analysis job roles; as soon as there are data, statistical expertise is required. Moreover, there are almost as many different jobs as people. Therefore, the list of jobs below is non-exhaustive and you may get a job that is not mentioned in this list. However, I hope it will still give you an overview of what you can expect after your studies.

If you believe I missed one—if you studied statistics and your job is not mentioned here for example, feel free to leave a comment at the end of the post.

What types of jobs are available?

If you are skilled in statistics and R programming, there are a plethora of job opportunities available to you.

Some of the roles you can pursue include:

Statistician: Collect, analyze and interpret data in a wide range of fields, including economics, finance, marketing, social and medical sciences.
Data scientist, data/business analyst, data engineer or machine learning engineer: Collect, analyze and interpret large and complex data sets using statistical and machine learning techniques to gain insights and inform decision-making to business and organizations. Mastering these techniques is key to the role, and in industry, turning statistical models into production-ready solutions often requires collaboration with a professional machine learning development company.
Actuary or actuarial analyst: Use statistical modeling to assess risk and uncertainty, and to create financial projections for insurance and investment companies.
Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst: Use statistical and mathematical modeling to analyze financial and market data, identify and evaluate risks for investment and trading decisions.
Business intelligence analyst: Use data analysis and visualization tools to create reports and dashboards that help businesses make informed decisions.
Operational researcher or quality control analyst: Use mathematical and statistical techniques to monitor and optimize the quality of products and processes in manufacturing.
Market or survey researcher: Conduct research and surveys, gather and analyze data to help businesses develop and improve their marketing strategies.
Economist or econometrician: Use statistical methods to analyze (socio)economic data and develop economic models to inform policy and decision-making.
Teacher: Facilitate learning and provide guidance to students in order to help them acquire knowledge, skills, and values that will prepare them for their future.
(Freelance) consultant: Provide statistical consulting services to clients, helping them solve problems using a combination of statistical methods and business expertise.

In the following sections, we delve deeper into the aforementioned job roles and outline the qualifications expected of prospective candidates.

Statistician

As a graduate in statistics, you have the potential to land a coveted statistician’s job in a reputable private company or public agency. These types of roles typically involve analyzing and assessing data, and utilizing various tools and software to manage data effectively.

With data proliferation and more and more companies/organizations that rely heavily on data analysis, your skills will be valuable in a broad range of fields and industries. Being a statistician means that you could work for instance as an:

environmental statistician: analyze data related to environmental issues, such as climate change and pollution, to inform policy and decision-making,
sports statistician: analyze data related to sports performance, such as player statistics and game outcomes, to inform coaching and strategy decisions,
government statistician: collect, analyze and interpret data for government agencies and public departments to guide policymakers in their decisions,
biostatistician: use statistical methods and data analysis techniques to study and interpret medical and (public) health-related data,
etc.

Statistical expertise is required wherever there is data, so feel free to specialize in the field or industry you care about most!

Data scientist, data/business analyst, data engineer or machine learning engineer

As a relatively new job role in various organizations, data scientists and data analysts are experts in manipulating, analyzing and interpreting data. These roles typically require the analysis of large datasets using statistical methods to assist organizations in making more informed business decisions. These professionals dive into unstructured information to uncover valuable insights for businesses, ultimately increasing their revenue.

To pursue this career, one would usually require a graduate degree in science, statistics, or mathematics, as well as training in data mining. It is also expected that you are familiar with one or several programming languages such as R, Python or SAS.¹

Statistics graduates can also pursue their career as machine learning engineers. Job roles in this field typically involve conducting experiments and implementing machine learning algorithms. Consequently, many organizations hire these professionals to create a wide range of AI products. To do so, they must possess strong programming and statistical skills. Data science and software engineering knowledge are also beneficial.

Actuary or actuarial analyst

Actuaries usually apply statistical modeling tools on data related to retirement, financial/insurance products, accidents or mortality. As an actuary, one would be responsible for analyzing financial costs and risks. It means examining uncertainties associated with investments and other financial products and making predictions about the risks involved in a particular venture.

Actuaries are also in charge of writing insurance proposals, determining policy terms, and calculating premiums for different products, among other responsibilities. You may also play a vital role in deciding whether to accept or reject insurance applications by evaluating their risks. You will draw upon your statistical, actuarial, and background information to accomplish these tasks.

Typically, actuaries have a range of financial companies as clients, and they must identify potential risks and recommend compensation strategies accordingly. It means that their work is closely related to insurance products, as they must assess the likelihood of certain events occurring and the resulting financial impact.

Typically, those who pursue this role hold a degree in actuarial science, mathematics or statistics, and are trained in statistical analysis tools and software. Note that each country has its own specific accreditation related to this profession.

Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst

As a financial analyst, one is required to gather data from various sources, organize and analyze historical results, and make projections and forecasts. Furthermore, in addition to analyzing data, their role often involves reporting financial results to the board of directors/management to help them in setting the overall strategy and direction of the company.

Investment analysts are experts in evaluating different financial assets such as stocks, securities, and bonds. They do not stop there, though. They also conduct research and make crucial decisions about purchasing business financials.

As a financial trader, you will have a keen understanding of financial markets and the ability to execute trades (i.e., buy or sell shares, bonds, and other assets) on behalf of clients. Note that there are also various sub-roles to explore within this job segment.

Financial managers combine their passion for finance with their love of numbers and statistics. Financial managers are experts at creating and interpreting complex financial reports, providing invaluable insights into a company’s financial health. They can also advise management on investment strategies and assist with vital financial decisions. With their keen analytical skills, financial managers keep a watchful eye on daily financial activities and help ensure a company’s long-term success.

These skills are in high demand in financial organizations, such as banks, actuarial firms, insurance companies (commercial insurance, reinsurance, general insurance, life insurance, car insurance, etc.), and other similar establishments.

The educational background of these professionals is diverse. Graduates in finance, mathematics, and statistics all bring unique perspectives to the table.

Business intelligence analyst

A business intelligence (BI) analyst is responsible for collecting, organizing, analyzing, and presenting large amounts of data from various sources such as databases, spreadsheets, and software applications. They use various tools and techniques to identify trends, patterns, and relationships in the data to create reports, dashboards, and visualizations to communicate their findings to stakeholders. They may also be responsible for monitoring business performance metrics and KPIs, identifying areas for improvement, and making recommendations to optimize business operations.

Additionally, BI analysts may also be involved in the development and implementation of data-driven software applications, such as business intelligence platforms and data warehouses, to improve data accessibility, accuracy, and efficiency. Overall, the role of a BI analyst is crucial in filling the gap between business analysts and the IT team.

The most common software tools and platforms used by BI analysts are, at the time of writing this post, R Shiny, Microsoft Power BI, Tableau, QlikView and SAS BI.

Operational researcher or quality control analyst

Operational researchers are responsible for using mathematical and analytical methods to solve complex problems and optimize business operations. They collect and analyze data, develop models and algorithms, test and validate them, and make recommendations for improvements.

On the other hand, quality control analysts are responsible for ensuring that products or services meet the required standards of quality. They monitor and analyze product or service performance, identify areas for improvement, and develop strategies for maintaining or improving quality. They may also work with production teams to develop and implement quality control processes and procedures.

While operational researchers focus on optimizing processes and systems, quality control analysts focus on ensuring that the end product or service meets the required quality standards. Both roles require analytical skills and the ability to work with data, but operational researchers focus more on mathematical modeling and optimization, while quality control analysts focus more on monitoring and improving quality.

Individuals who pursue job roles of this nature typically possess strong mathematical abilities, often having graduated with degrees in statistics or mathematics. They are trained to analyze vast amounts of data and are well-versed in various analytical tools, such as simulation, mathematical modeling, and data science.

Market or survey researcher

Market or survey researchers are professionals who specialize in conducting research to gather information about consumer behavior and preferences, market trends, and competitive landscapes. They use various research methods and techniques to collect and analyze data, and provide insights and recommendations to businesses and organizations.

Market research professionals typically work with marketing agencies on various projects for clients across different sectors. In addition to a degree in statistics, knowledge in marketing and familiarity with the industry is beneficial.

Economist or econometrician

Economists and econometricians study and analyze economic systems, markets, and policies using quantitative methods and models. They collect and analyze data, develop economic models, conduct economic analysis, provide recommendations to policymakers and organizations, and communicate their findings to a variety of audiences. Their role is to inform economic policies and decisions in both public and private sectors.

If you have a background in statistics in addition to knowledge in economics, there is a whole world of vocational opportunities in economics just waiting for you. Indeed, armed with the ability to appropriately analyze data (thanks to your background in statistics) and the ability to understand (socio)economic issues and financial data, you will be a hot commodity as an advisor to governments and businesses on all economic decisions.

(Freelance) consultant

From a general point of view, a consultant (or freelance consultant) is a professional who provides expert advice and guidance to organizations or individuals on a contractual basis. They analyze problems, develop solutions, and provide recommendations to clients in a specific field or industry.

A consultant specializing in statistics provides expert advice and guidance to organizations or individuals on statistical analysis, data management, and modeling. They use statistical tools and techniques to analyze and interpret complex data, and provide insights and recommendations based on their analyses. The role of a statistical consultant may include identifying research questions, designing studies, collecting and managing data, analyzing data using appropriate statistical methods, interpreting results, and presenting findings to clients. They may also assist clients with the implementation of statistical methods and tools, provide training on statistical software and techniques, and develop custom statistical models for specific applications.

As a side note, note that many consultants in statistics provide their service as data visualization consultants. These consultants are responsible for helping clients to effectively communicate complex data through the creation of clear and visually appealing graphics, charts, and other forms of visual representation that can aid in decision-making and enhance understanding.

You can either be a consultant in a consulting firm, or a freelance consultant. If you choose to be a freelance consultant, you can more easily select the projects you want to work on and the industry you want to focus on, but you will also need to take care of all the administrative tasks such as finding and building relationships with clients, do your bookkeeping, etc.

If you are not ready to be a full-time freelance consultant but would like to experience a taste of it, you can always keep your primary job and accept a couple of side projects. This is known as a side hustle. This way, you will gradually build your portfolio and create relationships with clients, all that with the advantage of keeping a safe source of income. This is what I am doing with datanalyze.be, and I recommend it to anyone who is considering the option of being a freelance in the future.

Teacher

If you care about education, you like to transmit your knowledge and explain complex things in a simple manner, you may be interested in being a teacher in high school or a university professor. With a degree in statistics, you may be eligible to teach different subjects such as mathematics, statistics, physics, or science in general. Your ultimate goal is to educate and inspire students to develop an understanding and appreciation of these subjects (which are often not well appreciated by students), as well as to develop critical thinking skills, problem-solving abilities, and scientific literacy.

Like actuaries, note that each country has its own specific accreditation related to this profession.

To know whether this job suits you, you can start by being a private tutor. This way, you will experience what the job is like and you will be more able to tell whether it is the direction you want to take.

To become a university professor, note that a PhD (and even sometimes a postdoc) is usually required. This brings me to the last career option I would like to mention: a PhD.

I do not include a PhD in the list of the 10 potential career options because it is more seen as an additional degree rather than a job, and even for those who see it as a job, it is a temporary one. However, I would still like to mention it because it is worth considering with a degree in statistics (I am of course biased, but my biased opinion may be of interest to some readers).

PhD student

For those who enjoy academic research, pursuing a career as a researcher after taking a statistics degree may be a worthwhile consideration. Typically, pursuing a PhD in statistics involves working in close collaboration with one or two university professors (your supervisors) on a specific topic.

I cannot speak for all PhD students, but I can speak from my experience. It involves conducting original research in statistical theory and methodology, developing new statistical models and methods, and applying statistical techniques to solve real-world problems or to advance knowledge in a specific field. My research focuses on applying biostatistical procedures to cancer patients, so it is a rather applied PhD, but many of my colleagues work on a more theoretical subject.

We also sometimes participate in academic conferences, present our research findings, and collaborate with other researchers in our field of research.

After completing our PhD, we can continue to evolve in academia (and become a university professor for instance), or decide to work in a private or public research organization. Because professionals with a PhD in statistics become experts in statistical theory and methodology, commercial and public organizations often seek such profiles.

The length of a PhD depends on the country and the type of contract. If you are interested to know more, feel free to contact me, I might be able to help you (at least with how it goes in Belgium). Otherwise, the program director at your university will definitely be able to answer the questions you may have.

Conclusion

As you have seen, with a degree in statistics and knowledge in programming, you will be able to pursue positions at various companies or organizations. You can also always enhance your qualifications with additional courses if you feel that you miss some skills that are required for the job you want to apply to.

Statisticians are in demand in government departments across many regions and sectors. Actuarial firms, banks, investment firms, and market research organizations are also great potential employers. The demand for statistical and analytical skills is on the rise across various fields. Job seekers with these skills are particularly needed. However, it is becoming increasingly common for these positions to require specialized education and training. For example, if you have a degree in statistics, pursuing a finance specialization could open doors to opportunities in banking, investment, accountancy, or insurance firms. Last but not least, those passionate about solving complex problems may find roles as researchers (within or outside academia) appealing.

Thanks for reading.

I hope this article helped you to get an overview of the career options you have with a degree in statistics.

If you are looking for a job, I wish you the best of luck in finding your dream job! Remember that you will find a variety of job openings in academic, research-oriented, and analytical fields on online and offline job boards. Moreover, do not underestimate the power of friends, colleagues and family: make sure to network with people in your field to increase the chance of getting a job.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There are many programming languages for data analysis, and it will certainly continue to evolve in the future. The most common ones at the time of writing this post are: R, Python, SAS, Jamovi, SPSS, JMP, Stata and Matlab. Depending on the role and the organization, more or less programming skills will be required.↩︎

Web scraping in R

Mon, 16 Jan 2023 00:00:00 +0000

Note: This post has been written in collaboration with Pietro Zanotta.

Introduction

Almost anyone is familiar with web pages (otherwise you would not be here), but what if we tell you that how you see a site is different from how Google or your browser does?

In fact, when you type any site address in your browser, your browser will download and render the page for you, but for rendering the page it needs some instructions.

There are 3 types of instructions:

HTML: describes a web page’s infrastructure;
CSS: defines the appearance of a site;
JavaScript: decides the behavior of the page.

Web scraping is the art of extracting information from the HTML, CSS and Javascript lines of code. The term usually refers to an automated process, which is less error-prone and faster than gathering data by hand.

It is important to note that web scraping can raise ethical concerns, as it involves accessing and using data from websites without the explicit permission of the website owner. It is a good practice to respect the terms of use for a website, and to seek written permission before scraping large amounts of data.

This article aims to cover the basics of how to do web scraping in R. We will conclude by creating a database on Formula 1 drivers from Wikipedia.

Note that this article doesn’t want to be exhaustive on topic. To learn more, see this section.

HTML and CSS

Before starting it is important to have a basic knowledge of HTML and CSS. This section aims to briefly explain how HTML and CSS work, to learn more we leave you some resources at the bottom of this article.

Feel free to skip this section if you already are knowledgeable in this topic.

Starting from HTML, an HTML file looks like the following piece of code.

<!DOCTYPE html>
<html lang="en">
<body>

<h1 href="https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss"> Carl Friedrich Gauss</h1>
<h2> Biography </h2>
<p> Johann Carl Friedrich Gauss was born on 30 April 1777 in Brunswick. </p>
<h2> Profession </h2>
<p> Gauss is considered as one of the greatest mathematician, statistician and physicist of all time. </p>

</body>
</html>

Those instructions produce the following:

As you read above, HTML is used to describe the infrastructure of a web page, for example we may want to define the headings, the paragraphs, etc.

This infrastructure is represented by what are called tags (for example <h1>...</h1> or <p>...</p> are tags). Tags are the core of an HTML document as they represent the nature of what is inside the tag (for example h1 stands for heading 1). It is important to observe that there are two types of tags:

starting tags (e.g. <h1>)
ending tags (e.g. </h1>)

This is what allows to nest different tags.

Tags can also have attributes, for example in <h1 href="https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss">Carl Friedrich Gauss</h1>, href is an attribute of the tag h1 that specifies an URL.

As the output of the above HTML code is not super elegant, CSS is used to style the final website. For example CSS is used to define the font, the color, the size, the spacing and many more features of a website.

What is important for this article are CSS selectors, which are patterns used to select elements. The most important is the .class selector, which selects all elements with the same class. For example the .xyz selector selects all elements with class="xyz".

Web scraping vs. APIs

Going back to web scraping, you may know that APIs are another way to access data from websites and online services.

In fact an API is a set of rules and protocols that allows two different software systems to communicate with each other. When a website or online service provides an API, it means that they have made it possible for developers to access their data in a structured and controlled way.

Why does web scraping exist if APIs are so powerful and do exactly the same work?

The main difference between web scraping and using APIs is that APIs are typically provided by the website or service to allow access to their data, while web scraping involves accessing data without the explicit permission of the website owner.

This means that using APIs is generally considered more ethical than web scraping, as it is done with the explicit permission of the website or service.

However, there are also some limitations to using APIs:

many APIs have rate limits, which means that they will only allow a certain number of requests to be made within a certain time period, i.e. you may not access large amounts of data;
not all websites or online services provide APIs, which means the only way to access their data is via web scraping.

Web scraping in R

There are several packages for web scraping in R, every package has its strengths and limitations. We will cover only the rvest package since it is the most used.

To get started with web scraping in R you will first need R and RStudio installed (if needed, see here). Once you have R and RStudio installed, you need to install the rvest package:

install.packages("rvest")

rvest

Inspired by beautiful soup and RoboBrowser (two Python libraries for web scraping), rvest has a similar syntax, which makes it the most eligible package for those who come from Python.

rvest provides functions to access a web page and specific elements using CSS selectors and XPath. The library is a part of the Tidyverse collection of packages, i.e. it shares some coding conventions (e.g. the pipes) with other libraries as tibble and ggplot2.

Before the real scraping it is necessary to load the rvest package:

library(rvest)

Now that everything is settled down, we can start the web scraping operation, which is usually made in 3 steps:

HTTP GET request
Parsing HTML content
Getting HTML element attributes

These steps are detailed in the following sections.

HTTP GET request

The HTTP GET method is a method used to send a server a question to get certain data and information. It is important to notice that this method does not change the state of the server.

To send a GET request we need the link (as a character) to the page we want to scrape:

link <- "https://www.nytimes.com/"

Sending the request to the page is simple, rvest provides the read_html function, which returns an object of html_document type:

NYT_page <- read_html(link)

NYT_page

## {html_document}
## <html lang="en" class="  nytapp-vi-homepage  tpl-always-light" xmlns:og="http://opengraphprotocol.org/schema/">
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
## [2] <body>\n    \n    <div id="app">\n<link rel="preload" as="image" href="/v ...

Parsing HTML content

As we saw in the last chunk of code, NYT_page contains the raw HTML code, which is not so easily readable.

In order to make it readable from R it has to be parsed, which means generating a Document Object Model (DOM) from the raw HTML. DOM is what connects scripts and web pages by representing the structure of a document in memory. If you retrieve the HTTP request using Node.js, you can give the raw HTML response to R for parsing and further analysis.

rvest provides 2 ways to select HTML elements:

XPath
CSS selectors

Selecting elements with rvest is simple, for XPath we use the following syntax:

NYT_page %>%
  html_elements(xpath = "")

while for CSS elector we need:

NYT_page %>%
  html_elements(css = "")

CSS selector

Suppose that for a project you need the summaries of the articles of the NYT (note that what is in the following picture is not what you see in the New York Times web page).

Searching in the HTML code, it is not that complex to find <p class="summary-class">, which is the markup of what we are looking for. To parse the HTML using this selector we use the html_element function:

summaries_css <- NYT_page %>%
  html_elements(css = ".summary-class")

head(summaries_css)

## {xml_nodeset (6)}
## [1] <p class="summary-class css-crclbt">At least 18 were killed in Kyiv, offi ...
## [2] <p class="summary-class css-1vqq9lj">Long-delayed funeral ceremonies for  ...
## [3] <p class="summary-class css-1vqq9lj">Senior U.S. officials have said Iran ...
## [4] <p class="summary-class css-1vqq9lj">One of England’s greatest World Cup  ...
## [5] <p class="summary-class css-1vqq9lj">America’s 2-0 win over Bosnia and He ...
## [6] <p class="summary-class css-1vqq9lj">Some data suggest artificial intelli ...

The easiest way to obtain a CSS selector is opening the inspect mode, find the element you desire and right click on it. Then click on copy and copy selector.

XPath

Parsing with XPath is similar to parsing using selectors. In fact, we just need to repeat what we did above using XPath of the element of interest. Moreover, obtaining an element’s XPath is not different form selector: inspector mode -> right click on element of interest -> copy -> copy XPath.

Repeating what we did above with XPath:

summaries_xpath <- NYT_page %>%
  html_elements(xpath = "//*[contains(@class, 'summary-class')]")

head(summaries_xpath)

## {xml_nodeset (6)}
## [1] <p class="summary-class css-crclbt">At least 18 were killed in Kyiv, offi ...
## [2] <p class="summary-class css-1vqq9lj">Long-delayed funeral ceremonies for  ...
## [3] <p class="summary-class css-1vqq9lj">Senior U.S. officials have said Iran ...
## [4] <p class="summary-class css-1vqq9lj">One of England’s greatest World Cup  ...
## [5] <p class="summary-class css-1vqq9lj">America’s 2-0 win over Bosnia and He ...
## [6] <p class="summary-class css-1vqq9lj">Some data suggest artificial intelli ...

Obviously the data we collected with CSS selector and XPath are exactly the same.

Getting attributes

Since the chunk of code above collect all the elements p with the class summary, we render all the elements of NYT_summary_css as a text using the html_text function:

NYT_summaries_css <- html_text(summaries_css)
NYT_summaries_xpath <- html_text(summaries_xpath)

We only print some of them:

head(NYT_summaries_css)

## [1] "At least 18 were killed in Kyiv, officials said. Ukraine’s president had warned of a “massive strike” as his country’s forces hit deeper into Russian territory."
## [2] "Long-delayed funeral ceremonies for Ayatollah Ali Khamenei, killed during U.S.-Israeli strikes at the war’s outset, are set to begin Friday."                    
## [3] "Senior U.S. officials have said Iran would be richly rewarded for changing its stance on America. But Tehran has rejected such a bargain in the past."           
## [4] "One of England’s greatest World Cup moments unfolded on Wednesday, a columnist for The Athletic writes."                                                         
## [5] "America’s 2-0 win over Bosnia and Herzegovina sets up a round-of-16 showdown with Belgium."                                                                      
## [6] "Some data suggest artificial intelligence is already causing job losses. Other sources show the opposite. Why is it so hard to figure out what’s going on?"

A real application of web scraping in R

To conclude this brief introduction to web scraping we want to use the rvest package in a real word application of web scraping. The goal is to scrape data from Formula 1 Wikipedia’s voice and create a CSV file containing the name, the nationality, the number of podiums and some other statistics for every pilot.

The table we are going to scrape is the following:

If you haven’t done so, you need to install the rvest package:

install.packages("rvest")

and then load it:

library(rvest)

HTTP GET request

The GET request is the easiest part of scraping, we just need the following line of code:

link <- "https://en.wikipedia.org/wiki/List_of_Formula_One_drivers"

Parsing HTML content and getting attributes

Again we repeat what we did before with the NYT example:

page <- read_html(link)

Searching in the HTML code we find that the table is a table element with the sortable attribute:

Therefore we run the following lines of code:

drivers_F1 <- html_element(page, "table.sortable") %>%
  html_table()

In the chunk of code above, the html_table function is used to render the HTML code into tables.

To inspect it, we display the first and last observations, and the structure of the dataset:

head(drivers_F1) # first 6 rows

## # A tibble: 6 × 11
##   `Driver name`     Nationality    `Seasons competed` `Drivers' Championships`
##   <chr>             <chr>          <chr>              <chr>                   
## 1 Carlo Abate       Italy          1962–1963          0                       
## 2 George Abecassis  United Kingdom 1951–1952          0                       
## 3 Kenny Acheson     United Kingdom 1983, 1985         0                       
## 4 Andrea de Adamich Italy          1968, 1970–1973    0                       
## 5 Philippe Adams    Belgium        1994               0                       
## 6 Walt Ader         United States  1950               0                       
## # ℹ 7 more variables: `Race entries` <chr>, `Race starts` <chr>,
## #   `Pole positions` <chr>, `Race wins` <chr>, Podiums <chr>,
## #   `Fastest laps` <chr>, `Points[a]` <chr>

tail(drivers_F1) # last 6 rows

## # A tibble: 6 × 11
##   `Driver name`  Nationality `Seasons competed`   `Drivers' Championships`
##   <chr>          <chr>       <chr>                <chr>                   
## 1 Emilio Zapico  Spain       1976                 0                       
## 2 Zhou Guanyu    China       2022–2024            0                       
## 3 Ricardo Zonta  Brazil      1999–2001, 2004–2005 0                       
## 4 Renzo Zorzi    Italy       1975–1977            0                       
## 5 Ricardo Zunino Argentina   1979–1981            0                       
## 6 Driver name    Nationality Seasons competed     Drivers' Championships  
## # ℹ 7 more variables: `Race entries` <chr>, `Race starts` <chr>,
## #   `Pole positions` <chr>, `Race wins` <chr>, Podiums <chr>,
## #   `Fastest laps` <chr>, `Points[a]` <chr>

str(drivers_F1) # structure of the dataset

## tibble [878 × 11] (S3: tbl_df/tbl/data.frame)
##  $ Driver name           : chr [1:878] "Carlo Abate" "George Abecassis" "Kenny Acheson" "Andrea de Adamich" ...
##  $ Nationality           : chr [1:878] "Italy" "United Kingdom" "United Kingdom" "Italy" ...
##  $ Seasons competed      : chr [1:878] "1962–1963" "1951–1952" "1983, 1985" "1968, 1970–1973" ...
##  $ Drivers' Championships: chr [1:878] "0" "0" "0" "0" ...
##  $ Race entries          : chr [1:878] "3" "2" "10" "36" ...
##  $ Race starts           : chr [1:878] "0" "2" "3" "30" ...
##  $ Pole positions        : chr [1:878] "0" "0" "0" "0" ...
##  $ Race wins             : chr [1:878] "0" "0" "0" "0" ...
##  $ Podiums               : chr [1:878] "0" "0" "0" "0" ...
##  $ Fastest laps          : chr [1:878] "0" "0" "0" "0" ...
##  $ Points[a]             : chr [1:878] "0" "0" "0" "6" ...

Now that we have a tibble (a sort of dataframe used in the tidyverse universe), we just need to select the variables of interest and eliminate the last row that contains the name of the variables:

drivers_F1 <- drivers_F1[c(1:4, 7:9)] # select variables

drivers_F1 <- drivers_F1[-nrow(drivers_F1), ] # remove last row

At this point we may want to clean our data. For example, we notice that Drivers' Championships has a small formatting issue: it returns not only the number of championships the driver won, but also the years of the victories. To extract only the number of victories (without the years) we use the substr() function:

drivers_F1$`Drivers' Championships` <- substr(drivers_F1$`Drivers' Championships`,
  start = 1, stop = 1
)

With this code, we actually extract only the first character since we start at 1 and stop at 1. At the moment, the maximum number of championships won by a driver is 7 (Lewis Hamilton & Michael Schumacher), so it is fine to extract only the first digit.

Et voila! With only a few lines of code, we scraped a table and we are now ready to perform our analysis.

If you want to save the dataset, you can always do so:

write.csv(drivers_F1, "F1_drivers.csv", row.names = FALSE)

Analysis on the database

To convince you that this is a real database, we will now answer some simple questions.

First of all, we load the tidyverse package:

library(tidyverse)

Which country has the largest number of wins?

drivers_F1 %>%
  group_by(Nationality) %>%
  summarise(championship_country = sum(as.double(`Drivers' Championships`))) %>%
  arrange(desc(championship_country))

## # A tibble: 48 × 2
##    Nationality    championship_country
##    <chr>                         <dbl>
##  1 United Kingdom                   21
##  2 Germany                          12
##  3 Brazil                            8
##  4 Argentina                         5
##  5 Australia                         4
##  6 Austria                           4
##  7 Finland                           4
##  8 France                            4
##  9 Netherlands                       4
## 10 Italy                             3
## # ℹ 38 more rows

Who has the most Championships?

drivers_F1 %>%
  group_by(`Driver name`) %>%
  summarise(championship_pilot = sum(as.double(`Drivers' Championships`))) %>%
  arrange(desc(championship_pilot))

## # A tibble: 877 × 2
##    `Driver name`       championship_pilot
##    <chr>                            <dbl>
##  1 Lewis Hamilton~                      7
##  2 Michael Schumacher^                  7
##  3 Juan Manuel Fangio^                  5
##  4 Alain Prost^                         4
##  5 Max Verstappen~                      4
##  6 Sebastian Vettel^                    4
##  7 Ayrton Senna^                        3
##  8 Jack Brabham^                        3
##  9 Jackie Stewart^                      3
## 10 Nelson Piquet^                       3
## # ℹ 867 more rows

Sorry Michael, it looks like Lewis dethroned you.

Is there a relation between the number of Championships won and the number of race pole positions?

drivers_F1 %>%
  filter(`Pole positions` > 1) %>%
  ggplot(aes(x = as.double(`Pole positions`), y = as.double(`Drivers' Championships`))) +
  geom_point(position = "jitter") +
  labs(y = "Championships won", x = "Pole positions") +
  theme_minimal()

As expected, there seems to be a positive relationship between the number of pole positions and the number of Championships won. To quantify this relationship, we could build a linear model but this is beyond the scope of the article.

To go further

As you have seen, rvest is a powerful tool. The goal of the article is to show just the tip of the iceberg regarding web scraping in R.

There are many resources online that you can read if you want to know more:

Web scraping: Basics by Paul Bauer
rvest CRAN documentation
xml2 CRAN documentation
httr CRAN documentation
rvest CRAN vignette
httr CRAN vignette
Beautiful Soup documentation
RoboBrowser documentation
HTML documentation
CSS documentation

Conclusion

Thanks for reading.

I hope this article helped you to learn about web scraping in R, and gave you the incentive to use it for your projects. If you are interested in seeing another example, see how to scrape Yahoo search engine results with R.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Paper: 'EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number'

Wed, 19 Oct 2022 00:00:00 +0000

Introduction

A colleague (and friend) of mine recently published a research paper entitled “EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number” in PLoS Computational Biology.

I am not in the habit of sharing research paper to which I did not contribute. Nevertheless, I would like to make an exception with this one because I strongly believe that the method developed in the paper deserves to be known, especially for anyone working in epidemiology.

Below is the motivation behind the article, as well as an illustration on simulated and real data (US hospitalization data). More information can be found in the paper and on the accompanying website.

Motivation

EpiLPS (Gressani et al. 2022) is a methodology for flexible Bayesian inference of the time-varying reproduction number $\mathcal{R}_t$; the average number of secondary cases generated by an infected agent at time $t$. This is a key epidemiological parameter that informs about the transmission potential of an infectious disease and can be used by public health authorities to gauge the effectiveness of interventions and propose an orientation for future control strategies.

This metric has gained in popularity during the SARS-CoV-2 pandemic with wide media coverage as its meaning is easily and intuitively grasped. Put simply, when $\mathcal{R} < 1$, the signal is encouraging as the epidemic is under control and will eventually vanish. On the contrary, a value of $\mathcal{R} > 1$ means that the disease keeps spreading and infections are witnessing an expansionary impact. Having a robust and reliable tool to compute the reproduction number from infectious disease data is therefore crucial.

A group of researchers in the EpiPose team from Hasselt University (Belgium), Leiden University (The Netherlands), and the University of Bern (Switzerland) have recently developed a new methodology for estimating the instantaneous reproduction number from incidence time series data for a given serial interval distribution (the time elapsed between the onset of symptoms in an infector and the onset of symptoms of secondary cases).

They termed their approach EpiLPS for “Epidemiological modeling with Laplacian-P-Splines” as Laplace approximations and P-splines smoothers are key ingredients that form the backbone of the proposed methodology.

The EpiLPS model assumes that the observed reported cases (by reporting date or date of symptom onset) are governed by a negative binomial distribution. As such, it allows to take the feature of overdispersion into account, contrary to a Poisson model. The epidemic curve is smoothed with P-splines (where posterior estimates of latent variables are computed via Laplace approximations) in a first step and a renewal equation model is used in a second step as a bridge between the reproduction number and the estimated spline coefficients through a “plug-in” method.

The authors also explain the main difference between EpiLPS and EpiEstim, a well established approached for estimating $\mathcal{R}_t$ in real-time developed by Cori et al. (2013) and make extensive comparisons between the two approaches under different epidemic scenarios.

An interesting feature of EpiLPS is that the user can choose between a fully “sampling-free” path, where model hyperparameters are fixed at their maximum a posteriori (LPSMAP) or a fully stochastic path (LPSMALA) based on a Metropolis-adjusted Langevin algorithm (LPSMALA). Talking about efficiency, routines for Laplace approximations and B-splines evaluations have been coded in C++ and integrated in R via the Rcpp package, so that the underlying algorithm can be executed in negligible time.

Below, we provide a short example of how to use the EpiLPS routines to estimate $\mathcal{R}_t$.

Getting started

The EpiLPS package is available from CRAN (see https://cran.r-project.org/web/packages/EpiLPS/index.html) and can be installed from the R console by typing:

install.packages("EpiLPS")

The package can then be loaded as follows:

library("EpiLPS")

The EpiLPS package structure is fairly simple as it consists in a few routines:

The function epilps() is the core routine for fitting the reproduction number.
With plot.epilps(), the user can plot the estimated epidemic curve and $\mathcal{R}_t$.
Finally, two ancillary routines, episim() and perfcheck() have been developed to essentially reproduce the simulation results of the associated paper.

A simulated example

A set of epidemic data can be simulated with the episim() routine by specifying a serial interval distribution and by choosing among a set of available patterns for the true reproduction number curve (here we choose pattern number 5 corresponding to a rather wiggly curve).

The simulated outbreak is for a duration of 40 days as specified in the endepi option. By setting the option plotsim = TRUE, the routine returns a figure summarizing the incidence time series, a bar plot for the specified serial interval distribution and the true underlying reproduction number curve.

set.seed(1234)

SI <- c(0.344, 0.316, 0.168, 0.104, 0.068)
simepidemic <- episim(
  serial_interval = SI,
  Rpattern = 5,
  plotsim = TRUE,
  verbose = TRUE,
  endepi = 40
)

## Chosen scenario: 5 'Wiggly then stable Rt'.
## Incidence of cases generated from a Poisson distribution. 
## Total number of days of epidemic: 40.

If you want to have an overview of the generated incidence time series, just type:

simepidemic$y

##  [1]  10   6  15  24  37  43  54  46  47  28  20   8  10  10   3   5   3   4   6
## [20]   6  15  21  44  75 135 217 329 409 453 487 457 443 297 290 255 246 246 339
## [39] 395 573

Smoothing the epidemic curve and estimating $\mathcal{R}_t$

Let us now use the epilps() routine to smooth the epidemic curve and estimate the reproduction number.

We will do this through LPSMAP (a fully sampling-free approach) and via LPSMALA (a fully stochastic approach relying on a MCMC algorithm with Langevin dynamics), where we specify a chain of length 10000 and a burn-in of size 4000.

LPSMAP_fit <- epilps(
  incidence = simepidemic$y,
  serial_interval = SI,
  tictoc = TRUE
)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.327.
## Mean 95% CI of Rt discarding first 7 days: (1.164,1.527) 
## Elapsed real time (wall clock time): 0.261 seconds.

LPSMALA_fit <- epilps(
  incidence = simepidemic$y, serial_interval = SI,
  method = "LPSMALA", chain_length = 10000, burn = 4000
)

## Inference method chosen: LPSMALA with chain length 10000 and warmup 4000.
## MCMC acceptance rate: 56.41%. 
## Geweke z-score < 2.33 for:  32 / 33  variables. 
## Total number of days: 40. 
## Mean Rt discarding first 7 days: 1.326.
## Mean 95% CI of Rt discarding first 7 days: (1.117,1.555). 
## Timing of routine not requested.

After execution, each routine prints in the console a brief summary of the method that has been requested by the user.

For LPSMALA, it summarizes the chain length, the acceptance rate (should be around 56%) and other basic information. As can be seen from the printed output, the mean reproduction number for the simulated epidemic is around 1.33.

We can now use, say, the LPSMALA_fit object together with the plot() routine to obtain the smoothed epidemic curve and the estimated reproduction number (by default the credible interval is at a 5% level of significance but this can be changed by the user).

days <- seq(8, 40)

#--- Smoothed epidemic curve
gridExtra::grid.arrange(
  plot(LPSMALA_fit,
    plotout = "epicurve", incibars = TRUE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "Smoothed epidemic curve", titlesize = 13, barwidth = 0.25
  ),

  #--- Estimated reproduction number
  plot(LPSMALA_fit,
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "Estimated R (LPSMALA)"
  ),
  nrow = 1, ncol = 2
)

The figure can be customized in various ways:

Users can specify the theme under themetype. Available options are gray (the default), classic, light and dark.
Other choices, such as whether or not to show the incidence bars, the color of the credible interval envelope, the color of the smoothed epidemic curve and the estimated reproduction number are also available.

The figure above was generated within the ggplot2 package, but there is also another way of extracting information directly from the LPSMAP_fit and LPSMALA_fit objects. In fact, the estimated reproduction number values and their associated credible interval for each day can be extracted and plotted.

Below, we make the exercise and plot the estimated $\mathcal{R}_t$ obtained with LPSMAP and LPSMALA, respectively and compare it with the true underlying reproduction number curve. The fit is quite good.

par(mfrow = c(1, 2))

#--- LPSMAP vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMAP_fit$epifit$R95CI_low[8:40],
    rev(LPSMAP_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(0.23, 0.54, 1, 0.3), border = NA
)
lines(days, LPSMAP_fit$epifit$R_estim[8:40], type = "l", col = "cornflowerblue", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "blue", rgb(0.23, 0.54, 1, 0.3)),
  c("Target R", "LPSMAP", "LPSMAP 95% CI"), bty = "n", cex = 0.9
)

#--- LPSMALA vs target R
plot(days, sapply(days, simepidemic$Rtrue),
  type = "l", lwd = 2, ylim = c(0, 4),
  ylab = "Estimated R", xlab = "Time"
)
polygon(
  x = c(days, rev(days)), y = c(
    LPSMALA_fit$epifit$R95CI_low[8:40],
    rev(LPSMALA_fit$epifit$R95CI_up[8:40])
  ),
  col = rgb(1, 0.23, 0.31, 0.3), border = NA
)
lines(days, LPSMALA_fit$epifit$R_estim[8:40], type = "l", col = "red", lwd = 2)
lines(days, sapply(days, simepidemic$Rtrue), type = "l", lwd = 2)

grid(nx = 10, ny = 10)
legend("topright",
  lty = c(1, 1), lwd = c(2, 2),
  col = c("black", "red", rgb(1, 0.23, 0.31, 0.3)),
  c("Target R", "LPSMALA", "LPSMALA 95% CI"), bty = "n", cex = 0.9
)

You can access, say, the results of the last week of the epidemic by typing:

# Estimated R of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 1:4], 7), 3)

##    Date R_estim R95CI_low R95CI_up
## 34   34   0.708     0.663    0.756
## 35   35   0.724     0.676    0.775
## 36   36   0.809     0.755    0.868
## 37   37   0.971     0.908    1.039
## 38   38   1.205     1.135    1.279
## 39   39   1.461     1.384    1.541
## 40   40   1.671     1.557    1.794

# Estimated mean number of cases of the last week (with LPSMAP)
round(tail(LPSMAP_fit$epifit[, 5:7], 7))

##    mu_estim mu95CI_low mu95CI_up
## 34      284        225       357
## 35      254        202       319
## 36      248        196       312
## 37      267        211       338
## 38      319        253       404
## 39      411        325       520
## 40      552        394       774

USA hospitalization data

To illustrate EpiLPS on real data, we download hospitalization data from the COVID19 package for the USA in the period ranging from 2021-09-01 to 2022-09-01 and apply the epilps() routine to estimate the reproduction number.

install.packages("COVID19")
library("COVID19")

# Get data and specify serial interval distribution
USADat <- COVID19::covid19(
  country = "US", level = 1, start = "2021-09-01",
  end = "2022-09-01", verbose = FALSE
)

si <- c(0.344, 0.316, 0.168, 0.104, 0.068)

inciUSA <- USADat$hosp
dateUSA <- USADat$date

We use the epilps() routine with method LPSMAP (default) and plot the smoothed epidemic curve and the estimated reproduction number with a 95% credible interval.

epifit <- epilps(incidence = inciUSA, serial_interval = si, K = 20)

## Inference method chosen: LPSMAP. 
## CI for LPSMAP computed via lognormal posterior approx. of Rt.Total number of days: 366. 
## Mean Rt discarding first 7 days: 0.994.
## Mean 95% CI of Rt discarding first 7 days: (0.983,1.005) 
## Timing of routine not requested.

gridExtra::grid.arrange(
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "epicurve", incibars = FALSE, themetype = "light",
    epicol = "darkgreen", cicol = rgb(0.3, 0.73, 0.3, 0.2),
    epititle = "USA smoothed epidemic curve", titlesize = 13
  ),
  plot(epifit,
    dates = dateUSA, datelab = "3m",
    plotout = "rt", theme = "light", rtcol = "black",
    titlesize = 13, Rtitle = "USA Estimated R"
  ),
  nrow = 1, ncol = 2
)

Conclusion

Thanks for reading.

I hope you will find the method developed in the paper as useful as I do. Feel free to reach out to me and to the authors if you happen to use it for your own research.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Cori, Anne, Neil M Ferguson, Christophe Fraser, and Simon Cauchemez. 2013. “A New Framework and Software to Estimate Time-Varying Reproduction Numbers During Epidemics.” American Journal of Epidemiology 178 (9): 1505–12.

Gressani, Oswaldo, Jacco Wallinga, Christian L Althaus, Niel Hens, and Christel Faes. 2022. “EpiLPS: A Fast and Flexible Bayesian Tool for Estimation of the Time-Varying Reproduction Number.” PLoS Computational Biology 18 (10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618.

COVID-19 in Belgium: is it over yet?

Fri, 22 May 2020 00:00:00 +0000

Introduction

Note 1: The present article has been written on May 22, 2020 and has been updated infrequently. The current situation regarding COVID-19 in Belgium may therefore be different to what is presented below. See my Twitter profile for more frequent updates of the plots.

Note 2: This is a joint work with Prof. Niko Speybroeck, Prof. Catherine Linard, Prof. Simon Dellicour and Angel Rosas-Aguirre.

Belgium recently started to lift its lockdown measures initially imposed to contain the spread of the Covid-19. Following this decision taken by Belgian authorities, we analyze how the situation evolved so far.

Contrarily to a previous article in which I analyzed the outbreak of the Coronavirus in Belgium using the SIR model, in this article we focus on the evolution of the number of:

hospital admissions
patients in hospitals
patients in intensive care
new confirmed cases

at the province and national level.

Data is from Sciensano and all plots were created with the {ggplot2} package.

New hospital admissions

Overall

Belgian hospitalizations COVID-19

From the above figure, we see that the rate of hospitalizations continue with a decreasing trend in all provinces (and in Belgium as well).

Download the figure, or see the code.

Update of October 27, 2020:

COVID19 hospitalizations in Belgium

Download the figure, or see the code.

The detailed situation in Brabant:

Download the figure, or see the code.

By period

Daily COVID19 hospitalizations in Belgium from March to October 2020

Download the figure, or see the code.

Update of November 16, 2020:

Daily COVID19 hospitalizations in Belgium by period

Download the figure.

In the first wave, the province of Limburg recorded on average the highest number of COVID19 hospital admissions per million inhabitants. During the second wave, Liège and Hainaut struggled with the highest rates. With two exceptions (Antwerp and Limburg), last month was worse than in March-April. In three provinces (Hainaut, Namur and Liège), the number has more than doubled.

During the period from June 14 to July 15, 2020, the number of COVID19 hospital admissions in Belgium fell to very low relative levels, but we have failed to maintain them. Now that hospital admissions are no longer increasing, we hope that the colors will lighten up again a bit as the end of the year approaches.

Zooming in

Hospital admissions COVID-19 - Belgium

Download the figure or see the code.

Update of February 26, 2021:

There is some ongoing debate in Belgium on whether or not to ease restrictions. On February 26, 2021, Belgian authorities will meet, discuss, debate and decide. Current levels and trends of COVID-19 hospitalizations may guide them:

There is still no strong evidence that COVID-19 hospitalization curves decrease in Belgium. The comparison between the first (in gray - dates & curve) and second wave (in blue - dates & curve) needs to be done with care, but indicates that current hospitalization levels are not as low as some may like:

Zooming in provides some additional insights on the COVID-19 levels during the first and second waves at province level:

This shows that the second wave resulted in more hospitalizations than the first one in most Belgian provinces, despite the warning of a first deadly wave. It also illustrates the fact that daily hospitalizations in Belgium are currently still higher than what was observed at the end of the first wave.

Put simply, the bad news is that the combination of the number of contacts and the risk of transmission by contact seems (at the moment) not sufficiently low to result in a considerable decrease of hospitalizations. Yet (put simply), the good news today is that there is already some immunity in the population and that vaccinations may increase this immunity considerably. This can help in pushing curves down. Let’s not lose hope.

Download figures (1, 2 and 3) or see the code.

Update of May 10, 2021:

COVID19 hospitalizations - Wave 1 and 2

When looking at the above plot, bad news are that:

current levels correspond to levels of October 2020 and
current levels are still about double the target of 75 hospitalizations per day.

There are, however, three good news (compared to October 2020):

decreasing curve,
vaccination and
good weather.

Update of June 4, 2021

COVID-19 hospitalizations in Belgium below 75/day

The good news is that the number of COVID-19 hospitalizations in Belgium is now below the well-known threshold of 75 hospitalizations per day (which is a target defined by the Belgian government). This is the way to go, and we hope this trend will continue in the coming days/weeks.

Patients in hospitals

Below the evolution of the number of patients in hospitals in Belgium:

COVID19 patients in hospitals in Belgium

Download the figure or see the code.

We see that, as of October 28, 2020, the number of COVID19 patients in Belgian hospitals reached the peak of the first wave. So although patients stay shorter at the hospital during the second wave compared to the first wave, hospitals are still getting crowded.

Therefore, if the number of patients in hospitals follows the same path in the coming weeks, hospitals will quickly become too crowded and will not be able to accept new patients as their maximum capacity will soon be reached (if this is not already the case…).

Patients in intensive care

Below the evolution of COVID19 patients in intensive care in Belgium, with short-term projections and 99% confidence interval:

Evolution of COVID19 patients in intensive care in Belgium

Download the figure.

Short-term projections indicate what may have happened without the slow-down in transmission. This slow-down is positive news.

The maps show total intensive care patients by province if these would have had the Belgian population. Map at the top shows maximum levels in March-April and map at the bottom shows current levels. The maps indicate high intensive care use due to COVID19. In most Belgian provinces, numbers are still higher today than March-April peak numbers.

Observations are in line with other preliminary indications, such as trends of COVID19 hospitalizations (currently relatively volatile), indicating that transmission is slowing down:

Download the figure.

Confirmed cases

Note that the reported number of new confirmed cases is probably underestimated. This number does not take into account undiagnosed (without or with few symptoms) or untested cases. Therefore, figures with number of cases should be interpreted with extreme caution.

By province

New confirmed COVID-19 cases in Belgium

Download the figure or see the code.

By age group and sex

Static

Below another visualization of the number of cases by age group and sex in Belgium, for three different periods:

COVID-19 cases by age group and sex in Belgium

Download the figure or see the code.

This visualization shows the importance to report ages of cases and not just total number.

Moreover, we see that the distribution of cases per week by age group at the beginning of September is similar than during the summer holidays, but the number of cases per week is higher. The distribution of cases per week by age group at the beginning of September is however different from the “first wave” (period from March 1, 2020 to May 31, 2020). During the fist period, majority of cases were elderly, while at the beginning of September majority of cases are young people. It would be interesting to see how the distribution of cases by age group evolves during winter.

The figure above may be put in relation with the structure of the Belgian population:

Structure of Belgian population (2019)

Download the figure or see the code.

Dynamic

Additionally, these can be seen dynamically:

COVID-19 cases by age group and sex in Belgium - dynamic version

Download the figure or see the code.

With an update of the second wave:

Age and sex specific incidence per 100 000 of COVID19 cases in Belgium - dynamic version

Download the figure or see the code.

By age group, sex and province

Download the figure or see the code.

Conclusion

Thanks for reading.

We hope that these figures will evolve in the right direction. In the meantime, take care and stay safe!

If you would like to be further updated on the evolution of the COVID-19 epidemic, two options:

visit the blog from time to time, and
join Twitter and follow us: statsandr, NikoSpeybroeck & arosas_aguirre

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

A package to download free Springer books during Covid-19 quarantine

Sun, 26 Apr 2020 00:00:00 +0000

Update

The promotion has ended so it is not possible to download the books through R. If you did not download the books in time, you can still have access to them via this link.¹

Introduction

You probably already have seen that Springer released about 500 books for free following the COVID-19 pandemic. According to Springer, these textbooks will be available free of charge until at least the end of July.

Following this announcement, I already downloaded a couple of statistics and R programming textbooks from their website and I will probably download a few more in the coming weeks.

In this article, I present a package that saved me a lot of time and which may be of interest to many of us: the {springerQuarantineBooksR} package, developed by Renan Xavier Cortes.

This package allows you to easily download all (or a selection of) Springer books made available free of charge during the COVID-19 quarantine.

With this large collection of high quality resources and my collection of top R resources about the Coronavirus, we do not have any excuse to not read and learn during this quarantine.

In this article, I show:

how to download all available textbooks at once and
how to download a subset of books, given a specific title, author or subject

Without further ado, here is how the package works in practice.

Installation

After having installed the {devtools} package, you can install the {springerQuarantineBooksR} package from GitHub and load it with:

# install.packages("devtools")
devtools::install_github("renanxcortes/springerQuarantineBooksR", force = TRUE)
library(springerQuarantineBooksR)

Download all books at once

First, set the path where you would like to save all books with the setwd() function then download all of them at once with the download_springer_book_files() function. Note that it takes several minutes (depending on the speed of your internet connection) since all books combined amount for almost 8GB.

setwd("path_of_your_choice") # where you want to save the books
download_springer_book_files() # download all of them at once

You will find all downloaded books (in PDF format) in a folder named “springer_quarantine_books”, organized by category.²

If you want to download the EPUB version (or both the PDF and EPUB versions), add the filetype argument to the function:

# for EPUB version:
download_springer_book_files(filetype = "epub")

# for both PDF and EPUB versions:
download_springer_book_files(filetype = "both")

By default, it downloads only the English books. However, it is also possible to download all German books by adding the argument lan = 'ger':

download_springer_book_files(lan = "ger")

Note that in total, there are 407 unique titles in English and 52 in German.

Create a table of Springer books

Like me, if you do not know which books are offered by Springer and you do not want to download all of them, you probably may want to have an overview or a list of the released books before downloading any.

For this, you can load a table containing all the titles made available by Springer into an R session with the download_springer_table() function:

springer_table <- springerQuarantineBooksR::download_springer_table()

This table can then be improved with the {DT} package to:

keep only a minimum of information,
allow searching a book by its title, author, classification or year,
allow downloading the list of available books, and
make the Springer links clickable for instance

# install.packages("DT")
library(DT)

springer_table$open_url <- paste0(
  '<a target="_blank" href="', # opening HTML tag
  springer_table$open_url, # href link
  '">SpringerLink</a>' # closing HTML tag
)

springer_table <- springer_table[, c(1:3, 19, 20)] # keep only relevant information

datatable(springer_table,
  rownames = FALSE, # remove row numbers
  filter = "top", # add filter on top of columns
  extensions = "Buttons", # add download buttons
  options = list(
    autoWidth = TRUE,
    dom = "Blfrtip", # location of the download buttons
    buttons = c("copy", "csv", "excel", "pdf", "print"), # download buttons
    pageLength = 5, # show first 5 entries, default is 10
    order = list(0, "asc") # order the title column by ascending order
  ),
  escape = FALSE # make URLs clickable
)

This table allows you to see which textbooks Springer offers (together with some information) and allows you to find the ones that you are most likely to be interested in.

Note that you can create a similar table for German books with the download_springer_table(lan = "ger") function.

Download only specific books

Now that you have a better idea about the books you are interested in, you can download them by their title, author or subject.

By title

Say that you are interested in downloading only one specific book and you know its title. For instance, suppose you want to download the book entitled “All of Statistics”:

download_springer_book_files(springer_books_titles = "All of Statistics")

If you are interested to download more than one book, run the following command:

download_springer_book_files(
  springer_books_titles = c(
    "All of Statistics",
    "A Modern Introduction to Probability and Statistics"
  )
)

Alternatively, if you do not have a specific title in mind but you are interested in downloading all books with the word “Statistics” in the title, you can run:

springer_table <- download_springer_table()

library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    book_title, # look for a pattern in the book_title column
    "Statistics" # specify the word
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

Tip: If you would like to download all books with the word “Statistics” or “Data Science” in the title, replace "Statistics" in the above code by "Statistics|Data Science".

By author

If you want to download all books from a specific author, you can run:

springer_table <- download_springer_table()

# library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    author, # look for a pattern in the author column
    "John Hunt" # specify the author
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

By subject

You can also download all textbooks covering a specific subject (see all subjects in the subject_classification column in the summary table). For instance, here is how to download all books categorized in the Statistics subject:

springer_table <- download_springer_table()

# library(dplyr)
specific_titles_list <- springer_table %>%
  filter(str_detect(
    subject_classification, # look for a pattern in the subject_classification column
    "Statistics" # specify the subject
  )) %>%
  pull(book_title)

download_springer_book_files(springer_books_titles = specific_titles_list)

Improvements

Below a list of features that can potentially be implemented in order to improve the package:

Add the possibility to download all editions of a book. Currently, only the latest edition can be downloaded.
Add the possibility to resume downloading if it stopped. Currently, if the code is executed again, the downloads start from scratch.
Add the possibility of downloading books by topic. Currently, it is only possible by title, author or subject.

Feel free to open a pull request on GitHub if you have another improvement in mind.

Acknowledgments

I would like to thank:

Renan Xavier Cortes (and all contributors) for providing this package
The springer_free_books Python project which was used as inspiration to the {springerQuarantineBooksR} package
And last but not least, Springer who offers many of their excellent books for free!

Conclusion

Thanks for reading.

I hope this article will help you to download and read more high quality materials made available by Springer during this Covid-19 quarantine.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Note that I am not the author nor the maintainer of this shared folder. Therefore, I do not know how long the books will be available through this link, and I am not responsible if some (or all) books are removed.↩︎
Note that you can change the folder name by specifying the argument destination_folder = "name_of_your_choice".↩︎

COVID-19 in Belgium

Tue, 31 Mar 2020 00:00:00 +0000

Introduction

The Novel COVID-19 Coronavirus is still spreading quickly in several countries and it does not seem like it is going to stop anytime soon as the peak has not yet been reached in many countries.

Since the beginning of its expansion, a large number of scientists across the world have been analyzing this Coronavirus from different perspectives and with different technologies with the hope of coming up with a cure in order to stop its expansion and limit its impact on citizens.

Top R resources on Coronavirus

In the meantime, epidemiologists, statisticians and data scientists are working towards a better understanding of the spread of the virus in order to help governments and health agencies in taking the most optimal decisions. This led to the publication of a great deal of online resources about the virus, which I collected and organized in an article covering the top R resources on Coronavirus. This article is a collection of the best resources I’ve had the chance to discover, with a brief summary for each of them. It includes Shiny apps, dashboards, R packages, blog posts and datasets.

Publishing this collection led many readers to submit their piece of work, which made the article even more complete and more insightful for anyone interested in analyzing the virus from a quantitative perspective. Thanks to everyone who contributed and who helped me in collecting and summarizing these R resources about COVID-19!

Given my field of expertise, I am not able to help in this fight against the virus from a medical point of view. However, I still wanted to contribute as much as I could. From understanding better the disease to bringing scientists and doctors together to build something bigger and more impactful, I truly hope that this collection will, to a small extent, help to fight the pandemic.

Coronavirus dashboard for your own country

Besides receiving analyses, blog posts, R code and Shiny apps from people across the world, I realized that many people were trying to create a dashboard tracking the spread of the Coronavirus for their own country. So in addition to the collection of top R resources, I also published an article detailing the steps to follow to create a dashboard specific to a country. See how to create such dashboard in this article and an example with Belgium.

The code has been made available on GitHub and is open source so everyone can copy it and adapt it to their own country. The dashboard was intentionally kept simple so anyone with a minimum knowledge in R could easily replicate it, and advanced users could enhance it according to their needs.

Motivations, limitations and structure of the article

By seeing and organizing many R resources about COVID-19, I am fortunate enough to have read a lot of excellent analyses on the disease outbreak, the impact of different health measures, forecasts of the number of cases, projections about the length of the pandemic, hospitals capacity, etc.

Furthermore, I must admit that some countries such as China, South Korea, Italy, Spain, UK and Germany received a lot of attention as shown by the number of analyses done on these countries. However, to my knowledge and at the date of publication of this article, I am not aware of any analysis of the spread of the Coronavirus specifically for Belgium.¹ The present article aims at filling that gap.

Throughout my PhD thesis in statistics, my main research interest is about survival analysis applied to cancer patients (more information in the research section of my personal website). I am not an epidemiologist and I have no extensive knowledge in modelling disease outbreaks via epidemiological models.

I usually write articles only about things I consider myself familiar with, mainly statistics and its applications in R. At the time of writing this article, I was however curious where Belgium stands regarding the spread of this virus, I wanted to play with this kind of data in R (which is new to me) and see what comes out.

In order to satisfy my curiosity while not being an expert, in this article I am going to replicate analyses done by more knowledgeable people and apply them to my country, that is, Belgium. From all the analyses I have read so far, I decided to replicate the analyses done by Tim Churches and Prof. Dr. Holger K. von Jouanne-Diedrich. This article is based on a mix of their articles which can be found here and here. They both present a very informative analysis on how to model the outbreak of the Coronavirus and show how contagious it is. Their articles also allowed me to gain an understanding of the topic and in particular an understanding of the most common epidemiological model. I strongly advise interested readers to also read their more recent articles for more advanced analyses and for an even deeper understanding of the spread of the COVID-19 pandemic.

Other more complex analyses are possible and even preferable, but I leave this to experts in this field. Note also that the following analyses take into account only the data until the date of publication of this article, so the results should not be viewed, by default, as current findings.

In the remaining of the article, we first introduce the model which will be used to analyze the Coronavirus outbreak in Belgium. We also briefly discuss and show how to compute an important epidemiological measure, the reproduction number. We then use our model to analyze the outbreak of the disease in the case where there would be no public health intervention. We conclude the article by summarizing more advanced tools and techniques that could be used to further model COVID-19 in Belgium.

Analysis of Coronavirus in Belgium

A classic epidemiological model: the SIR model

Before diving into the real-life application, we first introduce the model that will be used.

There are many epidemiological models but we will use one of the most common one, the SIR model. The SIR model can be complexified to incorporate more specificities of the virus outbreak, but in this article we keep its simplest version. Tim Churches’ explanation of this model and how to fit it using R is so nice, I will reproduce it here with a few minor changes.

The basic idea behind the SIR model (Susceptible - Infectious - Recovered) of communicable disease outbreaks is that there are three groups (also called compartments) of individuals:

S: those who are healthy but susceptible to the disease (i.e., at risk of being contaminated). At the start of the pandemic, S is the entire population since no one is immune to the virus.
I: the infectious (and thus, infected) people
R: individuals who were contaminated but who have either recovered or died. They are not infectious anymore.

These groups evolve over time as the virus progresses in the population:

S decreases when individuals are contaminated and move to the infectious group I
As people recover or die, they go from the infected group I to the recovered group R

To model the dynamics of the outbreak we need three differential equations to describe the rates of change in each group, parameterised by:

$\beta$, the infection rate, which controls the transition between S and I
$\gamma$, the removal or recovery rate, which controls the transition between I and R

Formally, this gives:

\[\frac{dS}{dt} = - \frac{\beta IS}{N} \text{ (Eq. 1)}\]

\[\frac{dI}{dt} = \frac{\beta IS}{N} - \gamma I \text{ (Eq. 2)}\]

\[\frac{dR}{dt} = \gamma I \text{ (Eq. 3)}\]

The first equation (Eq. 1) states that the number of susceptible individuals (S) decreases with the number of newly infected individuals, where new infected cases are the result of the infection rate ($\beta$) multiplied by the number of susceptible individuals (S) who had a contact with infectious individuals (I).

The second equation (Eq. 2) states that the number of infectious individuals (I) increases with the newly infected individuals ($\beta I S$), minus the previously infected people who recovered (i.e., $\gamma I$ which is the removal rate $\gamma$ multiplied by the infectious individuals I).

Finally, the last equation (Eq. 3) states that the recovered group (R) increases with the number of individuals who were infectious and who either recovered or died ($\gamma I$).

An epidemic develops as follows:

Before the start of the disease outbreak, S equals the entire population as no one has anti-bodies.
At the beginning of the outbreak, as soon as the first individual is infected, S decreases by 1 and I increases by 1 as well.
This first infectious individual contaminates (before recovering or dying) other individuals who were susceptible.
The dynamic continues, with recently contaminated individuals who in turn infect other susceptible people before they recover.

Visually, we have:

SIR model. Source: Kai Sasaki.

Before fitting the SIR model to the data, the first step is to express these differential equations as an R function, with respect to time t.

SIR <- function(time, state, parameters) {
  par <- as.list(c(state, parameters))
  with(par, {
    dS <- -beta * I * S / N
    dI <- beta * I * S / N - gamma * I
    dR <- gamma * I
    list(c(dS, dI, dR))
  })
}

Fitting a SIR model to the Belgium data

To fit the model to the data we need two things:

a solver for these differential equations
an optimiser to find the optimal values for our two unknown parameters, $\beta$ and $\gamma$

The function ode() (for ordinary differential equations) from the {deSolve} R package makes solving the system of equations easy, and to find the optimal values for the parameters we wish to estimate, we can just use the optim() function built into base R.

Specifically, what we need to do is minimise the sum of the squared differences between $I(t)$, which is the number of people in the infectious compartment $I$ at time $t$, and the corresponding number of cases as predicted by our model $\hat{I}(t)$. This quantity is known as the residual sum of squares (RSS):

\[RSS(\beta, \gamma) = \sum_t \big(I(t) - \hat{I}(t) \big)^2\]

In order to fit a model to the incidence data for Belgium, we need a value N for the initial uninfected population. The population of Belgium in November 2019 was 11,515,793 people, according to Wikipedia. We will thus use N = 11515793 as the initial uninfected population.

Next, we need to create a vector with the daily cumulative incidence for Belgium, from February 4 (when our daily incidence data starts), through to March 30 (last available date at the time of publication of this article). We will then compare the predicted incidence from the SIR model fitted to these data with the actual incidence since February 4. We also need to initialise the values for N, S, I and R. Note that the daily cumulative incidence for Belgium is extracted from the {coronavirus} R package developed by Rami Krispin.

# devtools::install_github("RamiKrispin/coronavirus")
library(coronavirus)
data(coronavirus)

`%>%` <- magrittr::`%>%`

# extract the cumulative incidence
df <- coronavirus %>%
  dplyr::filter(country == "Belgium") %>%
  dplyr::group_by(date, type) %>%
  dplyr::summarise(total = sum(cases, na.rm = TRUE)) %>%
  tidyr::pivot_wider(
    names_from = type,
    values_from = total
  ) %>%
  dplyr::arrange(date) %>%
  dplyr::ungroup() %>%
  dplyr::mutate(active = confirmed - death - recovered) %>%
  dplyr::mutate(
    confirmed_cum = cumsum(confirmed),
    death_cum = cumsum(death),
    recovered_cum = cumsum(recovered),
    active_cum = cumsum(active)
  )

# put the daily cumulative incidence numbers for Belgium from
# Feb 4 to March 30 into a vector called Infected
library(lubridate)

sir_start_date <- "2020-02-04"
sir_end_date <- "2020-03-30"

Infected <- subset(df, date >= ymd(sir_start_date) & date <= ymd(sir_end_date))$active_cum

# Create an incrementing Day vector the same length as our
# cases vector
Day <- 1:(length(Infected))

# now specify initial values for N, S, I and R
N <- 11515793
init <- c(
  S = N - Infected[1],
  I = Infected[1],
  R = 0
)

Note that what is needed are currently infected persons (cumulative infected minus the removed, i.e. recovered or dead). However, numbers of recovered persons are hard to obtain and probably underestimated due to underreporting bias. I thus consider the cumulative number of infected people, which is probably not an issue here since the number of recovered cases is negligible at the time of the analysis.

Then we need to define a function to calculate the RSS, given a set of values for $\beta$ and $\gamma$.

# define a function to calculate the residual sum of squares
# (RSS), passing in parameters beta and gamma that are to be
# optimised for the best fit to the incidence data
RSS <- function(parameters) {
  names(parameters) <- c("beta", "gamma")
  out <- ode(y = init, times = Day, func = SIR, parms = parameters)
  fit <- out[, 3]
  sum((Infected - fit)^2)
}

Finally, we can fit the SIR model to our data by finding the values for $\beta$ and $\gamma$ that minimise the residual sum of squares between the observed cumulative incidence (observed in Belgium) and the predicted cumulative incidence (predicted by our model). We also need to check that our model has converged, as indicated by the message shown below:

# now find the values of beta and gamma that give the
# smallest RSS, which represents the best fit to the data.
# Start with values of 0.5 for each, and constrain them to
# the interval 0 to 1.0

# install.packages("deSolve")
library(deSolve)

Opt <- optim(c(0.5, 0.5),
  RSS,
  method = "L-BFGS-B",
  lower = c(0, 0),
  upper = c(1, 1)
)

# check for convergence
Opt$message

## [1] "CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH"

Convergence is confirmed. Note that you may find different estimates for different choices of initial values or constraints. This proves that the fitting process is not stable. Here is a potential solution for a better fitting process.

Now we can examine the fitted values for $\beta$ and $\gamma$:

Opt_par <- setNames(Opt$par, c("beta", "gamma"))
Opt_par

##      beta     gamma 
## 0.5841185 0.4158816

Remember that $\beta$ controls the transition between S and I (i.e., susceptible and infectious) and $\gamma$ controls the transition between I and R (i.e., infectious and recovered). However, those values do not mean a lot but we use them to get the fitted numbers of people in each compartment of our SIR model for the dates up to March 30 that were used to fit the model, and compare those fitted values with the observed (real) data.

# time in days for predictions
t <- 1:as.integer(ymd(sir_end_date) + 1 - ymd(sir_start_date))

# get the fitted values from our SIR model
fitted_cumulative_incidence <- data.frame(ode(
  y = init, times = t,
  func = SIR, parms = Opt_par
))

# add a Date column and the observed incidence data
library(dplyr)
fitted_cumulative_incidence <- fitted_cumulative_incidence %>%
  mutate(
    Date = ymd(sir_start_date) + days(t - 1),
    Country = "Belgium",
    cumulative_incident_cases = Infected
  )

# plot the data
library(ggplot2)
fitted_cumulative_incidence %>%
  ggplot(aes(x = Date)) +
  geom_line(aes(y = I), colour = "red") +
  geom_point(aes(y = cumulative_incident_cases), colour = "blue") +
  labs(
    y = "Cumulative incidence",
    title = "COVID-19 fitted vs observed cumulative incidence, Belgium",
    subtitle = "(Red = fitted from SIR model, blue = observed)"
  ) +
  theme_minimal()

From the above graph we see that the number of observed confirmed cases follows (unfortunately) the number of confirmed cases expected by our model. The fact that both trends are overlapping indicates that the pandemic is clearly in an exponential phase in Belgium. More data would be needed to see whether this trend is confirmed in the long term.

The following graph is similar than the previous one, except that the y-axis is measured on a log scale. This kind of plot is called a semi-log plot or more precisely a log-linear plot because only the y-axis is transformed with a logarithm scale. Transforming the scale in log has the advantage that it is more easily readable in terms of difference between the observed and expected number of confirmed cases and it also shows how the number of observed confirmed cases differs from an exponential trend.

fitted_cumulative_incidence %>%
  ggplot(aes(x = Date)) +
  geom_line(aes(y = I), colour = "red") +
  geom_point(aes(y = cumulative_incident_cases), colour = "blue") +
  labs(
    y = "Cumulative incidence",
    title = "COVID-19 fitted vs observed cumulative incidence, Belgium",
    subtitle = "(Red = fitted from SIR model, blue = observed)"
  ) +
  theme_minimal() +
  scale_y_log10(labels = scales::comma)

The plot indicates that, at the beginning of the pandemic and until March 12, the number of confirmed cases stayed below what would be expected in an exponential phase. In particular, the number of confirmed cases stayed constant at 1 case from February 4 to February 29. From March 13 and until March 30, the number of confirmed cases kept increasing at a rate close to an exponential rate.

We also notice a small jump between March 12 and March 13, which may potentially indicate an error in the data collection, or a change in the testing/screening methods.

Reproduction number $R_0$

Our SIR model looks like a good fit to the observed cumulative incidence data in Belgium, so we can now use our fitted model to calculate the basic reproduction number $R_0$, also referred as basic reproduction ratio, and which is closely linked to $\beta$ and $\gamma$.²

The basic reproduction number $R_0$ gives the average number of susceptible people who are infected by each infectious person where all individuals are susceptible to infection. In other words, the reproduction number refers to the number of healthy people that get infected per number of sick people. When $R_0 > 1$ the disease starts spreading in a population, but not if $R_0 < 1$. Usually, the larger the value of $R_0$, the harder it is to control the epidemic and the higher the probability of a pandemic.

Formally, we have:

\[R_0 = \frac{\beta}{\gamma}\]

We can compute it in R:

Opt_par

##      beta     gamma 
## 0.5841185 0.4158816

R0 <- as.numeric(Opt_par[1] / Opt_par[2])
R0

## [1] 1.404531

An $R_0$ of 1.4 is below values found by others for COVID-19 and the $R_0$ for SARS and MERS, which are similar diseases also caused by coronavirus. Furthermore, in the literature, it has been estimated that the reproduction number for COVID-19 is approximately 2.7 (with $\beta$ close to 0.54 and $\gamma$ close to 0.2). Our reproduction number being lower is mainly due to the fact that the number of confirmed cases stayed constant and equal to 1 at the beginning of the pandemic.

A $R_0$ of 1.4 means that, on average in Belgium, 1.4 persons are infected for each infected person.

For simple models, the proportion of the population that needs to be effectively immunized to prevent sustained spread of the disease, known as the “herd immunity threshold”, has to be larger than $1 - \frac{1}{R_0}$ (Fine, Eames, and Heymann 2011).

The reproduction number of 1.4 we just calculated suggests that, given the formula 1 - (1 / 1.4), 28.8% of the population should be immunized to stop the spread of the infection. With a population in Belgium of approximately 11.5 million, this translates into roughly 3.3 million people.

Using our model to analyze the outbreak if there was no intervention

It is instructive to use our model fitted to the first 56 days of available data on confirmed cases in Belgium, to see what would happen if the outbreak were left to run its course, without public health intervention.

# time in days for predictions
t <- 1:120

# get the fitted values from our SIR model
fitted_cumulative_incidence <- data.frame(ode(
  y = init, times = t,
  func = SIR, parms = Opt_par
))

# add a Date column and join the observed incidence data
fitted_cumulative_incidence <- fitted_cumulative_incidence %>%
  mutate(
    Date = ymd(sir_start_date) + days(t - 1),
    Country = "Belgium",
    cumulative_incident_cases = c(Infected, rep(NA, length(t) - length(Infected)))
  )

# plot the data
fitted_cumulative_incidence %>%
  ggplot(aes(x = Date)) +
  geom_line(aes(y = I), colour = "red") +
  geom_line(aes(y = S), colour = "black") +
  geom_line(aes(y = R), colour = "green") +
  geom_point(aes(y = cumulative_incident_cases),
    colour = "blue"
  ) +
  scale_y_continuous(labels = scales::comma) +
  labs(y = "Persons", title = "COVID-19 fitted vs observed cumulative incidence, Belgium") +
  scale_colour_manual(name = "", values = c(
    red = "red", black = "black",
    green = "green", blue = "blue"
  ), labels = c(
    "Susceptible",
    "Recovered", "Observed", "Infectious"
  )) +
  theme_minimal()

The same graph in log scale for the y-axis and with a legend for better readability:

# plot the data
fitted_cumulative_incidence %>%
  ggplot(aes(x = Date)) +
  geom_line(aes(y = I, colour = "red")) +
  geom_line(aes(y = S, colour = "black")) +
  geom_line(aes(y = R, colour = "green")) +
  geom_point(aes(y = cumulative_incident_cases, colour = "blue")) +
  scale_y_log10(labels = scales::comma) +
  labs(
    y = "Persons",
    title = "COVID-19 fitted vs observed cumulative incidence, Belgium"
  ) +
  scale_colour_manual(
    name = "",
    values = c(red = "red", black = "black", green = "green", blue = "blue"),
    labels = c("Susceptible", "Observed", "Recovered", "Infectious")
  ) +
  theme_minimal()

More summary statistics

Other interesting statistics can be computed from the fit of our model. For example:

the peak of the pandemic
the number of severe cases
the number of people in need of intensive care
the number of deaths

fit <- fitted_cumulative_incidence

# peak of pandemic
fit[fit$I == max(fit$I), c("Date", "I")]

##          Date        I
## 89 2020-05-02 531000.4

# severe cases
max_infected <- max(fit$I)
max_infected * 0.2

## [1] 106200.1

# cases with need for intensive care
max_infected * 0.06

## [1] 31860.03

# deaths with supposed 4.5% fatality rate
max_infected * 0.045

## [1] 23895.02

Given these predictions, with the exact same settings and no intervention at all to limit the spread of the pandemic, the peak in Belgium is expected to be reached by the beginning of May. About 530,000 people would be infected by then, which translates to about 106,000 severe cases, about 32,000 persons in need of intensive care (given that there are about 2000 intensive care units in Belgium, the health sector would be completely overwhelmed) and up to 24,000 deaths (assuming a 4.5% fatality rate, as suggested by this source).

At this point, we understand why such strict containment measures and regulations are taken in Belgium!

Note that those predictions should be taken with a lot of caution. On the one hand, as mentioned above, they are based on rather unrealistic assumptions (for example, no public health interventions, fixed reproduction number $R_0$, etc.). More advanced projections are possible with the {projections} package, among others (see this section for more information on this matter). On the other hand, we still have to be careful and strictly follow public health interventions because previous pandemics such as the Spanish and swine flu have shown that incredibly high numbers are not impossible!

The purpose of this article was to give an illustration of how such analyses are done in R with a simple epidemiological model. Those are the numbers our simple model produces and we hope they are wrong because the cost in terms of lives would be enormous.

Additional considerations

As previously mentioned, the SIR model and the analyses done above are rather simplistic and may not give a true representation of the reality. In the following sections, we highlight five improvements that could be done to enhance theses analyses and lead to a better overview of the spread of the Coronavirus in Belgium.

Ascertainment rates

In the previous analyses and graphs, it is assumed that the number of confirmed cases represent all the cases that are infectious. This is far from reality as only a proportion of all cases are screened, detected and counted in the official figures. This proportion is known as the ascertainment rate.

The ascertainment rate is likely to vary during the course of an outbreak, in particular if testing and screening efforts are increased, or if detections methods are changed. Such changing ascertainment rates can be easily incorporated into the model by using a weighting function for the incidence cases.

In his first article, Tim Churches demonstrates that a fixed ascertainment rates of 20% makes little difference to the modelled outbreak with no intervention, except that it all happens a bit more quickly.

More sophisticated models

More sophisticated models could also be used to better reflect real-life transmission processes. For instance, another classical model in disease outbreak is the SEIR model. This extended model is similar to the SIR model, where S stands for Susceptible and R stands for Recovered, but the infected people are divided into two compartments:

E for the Exposed/infected but asymptomatic
I for the Infected and symptomatic

These models belong to the continuous-time dynamic models that assume fixed transition rates. There are other stochastic models that allow for varying transition rates depending on attributes of individuals, social networking, etc.

Modelling the epidemic trajectory using log-linear models

As noted above, the initial exponential phase of an outbreak, when shown in a log-linear plot (the y-axis on a log scale and the x-axis without transformation), appears (somewhat) linear. This suggests that we can model epidemic growth, and decay, using a simple log-linear model of the form:

\[log(y)=rt+b\]

where y is the incidence, r is the growth rate, t is the number of days since a specific point in time (typically the start of the outbreak), and b is the intercept. In this context, two log-linear models:

one to the growth phase (before the peak), and
one to the decay phase (after the peak)

are fitted to the epidemic (incidence cases) curve.

The doubling and halving time estimates which you very often hear in the news can be estimated from these log-linear models. Furthermore, these log-linear models can also be used on the epidemic trajectory to estimate the reproduction number $R_0$ in the growth and decay phases of the epidemic.

The {incidence} package in R, part of the R Epidemics Consortium (RECON) suite of packages for epidemic modelling and control, makes the fitting of this kind of models very convenient.

Estimating changes in the effective reproduction number $R_e$

In our model, we set a reproduction number $R_0$ and kept it constant. It would nonetheless be useful to estimate the current effective reproduction number $R_e$ on a day-by-day basis so as to track the effectiveness of public health interventions, and possibly predict when an incidence curve will start to decrease.

The {EpiEstim} package in R can be used to estimate $R_e$ and allow to take into consideration human travel from other geographical regions in addition to local transmission (Cori et al. 2013; Thompson et al. 2019).

More sophisticated projections

In addition to naïve predictions based on a simple SIR model, more advanced and complex projections are also possible, notably, with the {projections} package. This packages uses data on daily incidence, the serial interval and the reproduction number to simulate plausible epidemic trajectories and project future incidence.

Conclusion

This article started with (i) a description of a couple of R resources on the Coronavirus pandemic (i.e., a collection and a dashboard) that can be used as background materials and (ii) the motivations behind this article. We then detailed the most common epidemiological model, i.e. the SIR model, before actually applying it on Belgium incidence data.

This resulted in a visual comparison of the fitted and observed cumulative incidence in Belgium. It showed that the COVID-19 pandemic is clearly in an exponential phase in Belgium in terms of number of confirmed cases.

We then explained what is the reproduction number and how to compute it in R. Finally, our model was used to analyze the outbreak of the Coronavirus if there was no public health intervention at all.

Under this (probably too) simplistic scenario, the peak of the COVID-19 in Belgium is expected to be reached by the beginning of May, 2020, with around 530,000 infected people and about 24,000 deaths. These very alarmist naïve predictions highlight the importance of restrictive public health actions taken by governments, and the urgency for citizens to follow these health actions in order to mitigate the spread of the virus in Belgium (or at least slow it enough to allow health care systems to cope with it).

We concluded this article by describing five improvements that could be implemented to further analyze the disease outbreak.

Note that this article has been subject to a talk at UCLouvain.

Thanks for reading. I hope this article gave you a good understanding of the spread of the COVID-19 Coronavirus in Belgium. Feel free to use this article as a starting point for analyzing the outbreak of this disease in your own country.

For the interested readers, see also:

the evolution of hospital admissions and number of confirmed cases in Belgium
a collection of top R resources on Coronavirus to gain even further knowledge

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

Fine, Paul, Ken Eames, and David L Heymann. 2011. “"Herd Immunity": A Rough Guide.” Clinical Infectious Diseases 52 (7): 911–16.

Thompson, RN, JE Stockwin, RD van Gaalen, JA Polonsky, ZN Kamvar, PA Demarsh, E Dahlqwist, et al. 2019. “Improved Inference of Time-Varying Reproduction Numbers During Infectious Disease Outbreaks.” Epidemics 29: 100356.

Feel free to let me know in the comments or by contacting me if you performed some analyses specifically for Belgium and which I could include in my article covering the top R resources on the Coronavirus.↩︎
See a more detailed note on the reproduction number by James Holland Jones if you need a deeper understanding.↩︎

Top 100 R resources on COVID-19 Coronavirus

Thu, 12 Mar 2020 00:00:00 +0000

Warning: Some links or resources may have been moved or deleted, and are thus not accessible anymore. If you are the author and would like to update the URL, feel free to contact me so I can update the link.

The Coronavirus is a serious concern around the globe. With its expansion, there are also more and more online resources about it. This article presents a selection of the best R resources on the COVID-19 virus.

This list is by no means exhaustive. I am not aware of all R resources available online about the Coronavirus, so please feel free to let me know in the comments or by contacting me if you believe that another resource (R package, Shiny app, R code, blog posts, datasets, etc.) deserves to be on this list.

R Shiny apps and dashboards

Coronavirus tracker

Developed by John Coene, this Shiny app tracks the spread of the Coronavirus, based on three data sources (John Hopkins, Weixin and DXY Data). The Shiny app, built with shinyMobile (which makes it responsive on different screen sizes), presents in a really nice way the number of deaths, confirmed, suspected and recovered cases by time and region.

The code is available on GitHub.

Coronavirus dashboard from the `{coronavirus} package`

Developed by the author of the {coronavirus} package, this dashboard provides an overview of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) epidemic. The data and dashboard are refreshed on a daily basis.

The code is available on GitHub.

From this dashboard, I created another dashboard specific to Belgium. Feel free to use the code available on GitHub to build one specific to your country. See more details in this article.

Visualization of Covid-19 Cases

Developed by Nico Hahn, this Shiny app uses leaflet, plotly and the data from Johns Hopkins University to visualize the outbreak of the novel Coronavirus and shows data for the entire world or singular countries.

The code is available on GitHub.

Modeling COVID-19 Spread vs Healthcare Capacity

Developed by Dr. Alison Hill, this Shiny app uses an epidemiological model based on the classic SEIR model to describe the spread and clinical progression of COVID-19. It includes different clinical trajectories of infection, interventions to reduce transmission, and comparisons to healthcare capacity.

The code is available on GitHub.

COVID-19 Data Visualization Platform

Developed by Shubhram Pandey, this Shiny app provides a clear visualization of Covid19 impact all over the world and it also provides a sentiment analysis using natural language processing from Twitter.

The code is available on GitHub.

Coronavirus 10-day forecast

Developed by the Spatial Ecology and Evolution Lab, this Shiny app gives a ten-day forecast, by country, on likely numbers of Coronavirus cases and gives citizens a sense of how fast this epidemic is progressing.

See a detailed explanation of the app and how to read it in this blog post. The code is available on GitHub.

Coronavirus (COVID-19) across the world

Developed by Anisa Dhana in collaboration with datascience+, this Shiny app monitors the spread of COVID-19 across the world via a map visualization of the confirmed cases and some graphs on the growth of the virus.

The dataset used is from Johns Hopkins CSSE and part of the code is available in this blog post.

COVID-19 outbreak

Developed by Dr. Thibaut Fabacher in collaboration with the department of Public Health of the Strasbourg University Hospital and the Laboratory of Biostatistics and Medical Informatics of the Strasbourg Medicine Faculty, this Shiny app shows an interactive map for global monitoring of the infection. It focuses on the evolution of the number of cases per country and for a given period in terms of incidence and prevalence.

The code is available on GitHub and this blog post discusses it in more detail.

Flatten the Curve

Developed by Tinu Schneider, this Shiny app illustrates, in an interactive way, the different scenarios behind the #FlattenTheCurve message.

The app has been built upon Michael Höhle’s article and the code is available on GitHub.

Explore the spread of Covid-19

Developed by Joachim Gassen, this Shiny app allows you to visualize confirmed, recovered cases and reported deaths for several countries via one summary graph.

The Shiny app is based on data from:

Johns Hopkins University CSSE team on the spread of the SARS-CoV-2 virus
ACAPS governmental measures database
World Bank

This blog post explains the Shiny app in further details and in particular the {tidycovid19} R package behind it.

Governments and COVID-19

Developed by Sebastian Engel-Wolf, this Shiny app presents in a elegant way the following measurements:

Maximum time of exponential growth in a row
Days to double infections
Exponential growth today
Confirmed cases
Deaths
Population
Confirmed cases on 100,000 inhabitants
Mortality rate

The code is available on GitHub and this article explains it in further details.

Simulating COVID-19 Epidemic in Togo - West Africa

Developed by Dr. Kankoé Sallah, this Shiny app uses SEIR metapopulation model with mobility between catchment areas to describe country level spread of COVID-19 and the impact of interventions in Togo, West Africa.

Covid-19 Prediction

Developed by Manuel Oviedo and Manuel Febrero (Modestya research group of the University of Santiago de Compostela), this Shiny app predicts the growth rate at 5-day horizon using the evolution during the last 15 days of growth rate. Three functional regression models are fitted and re-estimated when new data is available. The app also shows an interactive plot and table for the expected number of accumulated cases and new daily cases to each horizon (for confirmed and deaths responses) by country (from Johns Hopkins CSSE) and Spanish region (from ISCII).

See an explanation of the methodology in the About tab.

Covid-19 Dashboard

Developed by Philippe De Brouwer, this dashboard displays several key measures regarding the outbreak of the virus (by country or for all countries combined), together with some forecasts, a world map and other interactive plots.

Healthcare worker deaths from novel Coronavirus (COVID-19) in the US

Developed by Jonathan Gross, this Shiny app visualizes healthcare worker deaths from Coronavirus (COVID-19) in the US reported in the news. It is updated daily and the code is available on GitHub.

Covid-19 Hospitalizations in Belgium

Developed by Jean-Michel Bodart, this dashboard provides an overview of the evolution of Covid-19-related hospitalizations in Belgium, by region and province.

The code is available on GitHub.

COVIDMINDER: Where you live matters!

Developed by The Rensselaer Institute for Data Exploration and Applications, this Shiny app reveals the regional disparities in outcomes, determinants and medications (e.g., mortality rates, test cases, diabetes, and hospital beds) across United States, with a special focus on New York.

This blog post explains the Shiny app in more detail and the code is available on GitHub.

COVID-19 Canada Data Explorer Tool

Developed by Petr Baranovskiy from Data Enthusiast’s Blog, this Shiny app processes the official dataset available from the Government of Canada and shows several indicators related to the SARS-CoV-2 epidemic in Canada.

This blog post details the application in further detail.

Philippine COVID-19 Case Forecasting

Developed by Jamal Kay Rogers and Yvonne Grace Arandela, this Shiny app provides a 5-day forecast of confirmed positive, deaths, and recoveries of COVID-19 cases in The Philippines.

The app also delivers graphical plots of a 10-day forecast and the daily and cumulated cases of COVID-19 in The Philippines. The data source is Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE).

COVID-19 Case & Death Report Number Corrector

Developed by Matt Maciejewski, this Shiny app focuses on the correction of underreported Covid-19 case and death counts using a reference country based on Lachmann et al. (2020), and via a multiplicative estimator for total deaths and cases. The estimator will be turned into a posterior prediction once data becomes available.

The app is explained in more detail in this article and the code of the Shiny app can be found on GitHub.

Covid-19: the SPQEIR model

Developed by several researchers from the Luxembourg Centre for Systems Biomedicine (LCSB) of the University of Luxembourg, the KU Leuven and the UGent, this Shiny app uses the new SPQEIR model to simulate the impact of various suppression strategies (social distancing, lockdown, protection, etc.) on the development of COVID-19.

The Belgian Covid Cases Tracker

Developed by Patrick Sciortino, this Shiny app aims at estimating the curve of true Covid-19 cases based on the idea that a hospitalized case in time t informs us about an infection that took place a few days earlier.

COVID-19 monitor

Developed by Trafford Data Lab, this Shiny app visualizes daily confirmed Coronavirus cases and deaths in the UK.

It uses the following data sources:

The code can be found on GitHub.

COVID-19 Bulletin Board

Developed by Wei Su, this dashboard shows real-time visualization of the COVID-19 epidemic in Japan. It mainly shows various indicators including, but not limited to, PCR test, positive confirmed, hospital discharge and death, as well as trends in each prefecture in Japan. There are also a variety of charts such as cluster network, new confirmed cases in log scale for users’ reference.

The dashboard is based on this Japanese version (developed by the same author). The code can be found on GitHub.

Covid-19 Statistics Displayer

Created by Carl Sansfaçon, this Wordpress plugin associates R {ggplot2} graphics with ARIMA forecast and PHP coding to show evolution of the confirmed, death and recovered cases in different countries, states/provinces and US cities.

It uses data from the COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. The plugin can be installed as a Wordpress plugin.

CoronaMapper

Supported by OxyLabs, created by Paolo Montemurro and Peter, this visualization displays the four days average growth indicator, which clearly shows how a certain statistic of the virus is evolving over time, filtering out the noise.

The website receives data each hour from several official data-sources, and visualizes the historical evolution of COVID19 in an intuitive & interactive way.

CoronaDash

Developed by Peter Laurinec, this Shiny app provides various data mining and visualization techniques for comparing countries’ COVID-19 data statistics as:

extrapolating total confirmed cases by exponential smoothing model,
trajectories of cases/deaths spread,
multidimensional clustering of countries’ data/ statistics - with dendrogram and table of clusters averages,
aggregated views for the whole world,
hierarchical clustering of countries’ trajectories based on DTW distance and preprocessing by SMA (+ normalization), for fast comparison of a large number of countries’ COVID-19 magnitudes and trends.

This blog post explained in further detail the last point of the above list. The code of the app is available on GitHub.

Covidfrance

Developed by Guillaume Pressiat, this Shiny app illustrates the evolution of number of hospitalizations, intensive care units, recoveries and deaths in France (by department).

This blog post presents the application and the code of the app is available on GitHub.

The Impact of COVID-19 on Mobility

Developed by Dimiter Toshkov, this Shiny app shows relative changes in mobility (visits and length of stay) for a specific category of places within a country compared to a baseline. The baseline is computed as the median for the day of the week during the 5-week period between January 3 and February 6, 2020. Hence, the plot shows how mobility has changed relative to the situation in the same country in the beginning of the year.

The author also developed a version for the US states.

Data is from Google Community Mobility Reports.

Coronavirus Analysis Platform

Developed by Khaled M Alqahtani, this Shiny app offers a powerful analysis tools, including:

Descriptive analysis
Growth rate and curve flatting
Cumulative forecast
Daily cases forecasting containing 16 different models
Newspaper analysis
TV analysis

Data is from Johns Hopkins CSSE and some GDELT APIs.

COVID-19 Tracker

Developed by Dr. Magda Bucholc from Ulster University, this dashboard reports cases at the local government district in Northern Ireland and county level across the island of Ireland, providing gender and age breakdowns of reported cases, growth rates, and statistics per 100,000 of the population; it also has daily mobility data from Google and Apple.

More information about this dashboard can be found here.

COVID-19 Overview

Developed by Fabian Dablander, Alexandra Rusu, Marcel Schreiner, and Aleksandar Tomasevic as part of the Science versus Corona project, this dashboard provides an overview of confirmed cases, deaths, and measures that countries have taken to curb the spread of the virus.

For a more information about this dashboard, see this blog post.

COVID-19 Exit Strategies

Developed as part of the Science versus Corona project, this Shiny app compares several alternative exit strategies that either aim to keep the number of infections as low as possible (e.g., contact tracing), or that aim to develop herd immunity without exceeding health care capacity.

The Shiny app is based on the stochastic individual-based SEIR model developed by de Vlas and Coffeng (2020). This post explains the Shiny app in more detail.

Corona Virus Statistics : COVID19

Developed by Dr. Mohammed N. Alenezi, this dashboard displays the latest information about Coronavirus for Kuwait, GCC, and the world. The dashboard contains a collection of different plots and models presented under 8 tabs.

Covid19 Data

Developed by Dhafer Malouche, this dashboard presents statistics for more than 200 countries and regions, with the estimation of the reproduction number $R(t)$ in the previous 60 days and a Covid19 country classification.

WHO COVID-19 Explorer

Developed by the World Health Organization (WHO), this Shiny app aims to provide frequently updated data visualizations about confirmed cases and deaths at the global, regional and country level.

COVID-19 Scenario Analysis Tool

Developed by the MRC Centre for Global Infectious Disease Analysis (Imperial College London), this dashboard illustrates the epidemic trajectory, the healthcare demand and the $R_t$ & $R_{eff}$ measures for many countries over time in interactive plots.

The dashboard uses the squire R package, among others.

Covid-19 track

Developed by Redzuan, this Shiny dashboard analyses, combines and harmonizes the latest Covid-19 data into progress timeline, comparative analysis on various areas, mapping, latest news, forecasting using various models, prediction on Herd Immunity target, summary table, downloadable fact sheet and other features.

R packages

`{nCov2019}`

The {nCov2019} package gives you access to epidemiological data on the Coronavirus outbreak.¹ The package gives real-time statistics, includes historical data and a Shiny app. The vignette explains the main functions and possibilities of the package.

Furthermore, the authors of the package also developed a website with interactive plots and time-series forecasts, which could be useful in informing the public and studying how the virus spread in populous countries.

`{coronavirus}`

Developed by Rami Krispin, the {coronavirus} package provides a tidy format dataset of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) epidemic. Pulled from the dataset of John Hopkins, the R package gives a daily summary of the Coronavirus cases by state/province. The data set contains various variables such as confirmed cases, death, and recovered across different countries and states.

More details are available here, a csv format of the package dataset is available here and a summary dashboard is available here.

`{tidycovid19}`

Developed by Joachim Gassen, the {tidycovid19} package allows you to download, tidy and visualize Covid-19 related data (including data on governmental measures) directly from authoritative sources. It also provides a flexible function and an accompanying Shiny app to visualize the spreading of the virus.

The package is available on GitHub and these blog posts here and here explain it in more detail.

R packages from R Epidemics Consortium

These R packages from R Epidemics Consortium allows you to find the most advanced tools used by professional epidemiologists and experts in the domain of analyzing disease outbreaks.

`{covdata}`

Published by Prof. Kieran Healy, the {covdata} package is a R package providing COVID-19 case data from multiple sources:

National level data from the European Centers for Disease Control
State-level data for the United States from the COVID Tracking Project
State-level and county-level data for the United States from the New York Times
Data from the US Centers for Disease Control’s Coronavirus Disease 2019 (COVID-19)-Associated Hospitalization Surveillance Network (COVID-NET)
Data from Apple on relative trends in mobility in cities and countries since mid-January of 2020, based on usage of their Maps application
Data from Google on relative trends in mobility in regions and countries since mid-January of 2020, based on location and activity information

The code is available on GitHub.

`{covid19italy}`

The covid19italy R package provides a tidy format dataset of the 2019 Novel Coronavirus COVID-19 (2019-nCoV) pandemic outbreak in Italy. The package includes the following three datasets:

italy_total: daily summary of the outbreak on the national level
italy_region: daily summary of the outbreak on the region level
italy_province: daily summary of the outbreak on the province level

More information about the package datasets available in this vignette, this blog post, and this supporting dashboard.

Data source: Italy Department of Civil Protection

`{COVID19}`

The goal of COVID-19 Data Hub is to provide the research community with a unified data hub by collecting worldwide fine-grained case data, merged with exogenous variables helpful for a better understanding of COVID-19. Featured by the University of Milano and funded by the Institute for Data Valorization IVADO, Canada.

The package collects COVID-19 data across governmental sources, includes policy measures from Oxford COVID-19 Government Response Tracker, and extends the dataset via an interface to World Bank Open Data, Google Mobility Reports and Apple Mobility Reports.

The package is available on CRAN, it is 100% open source and external contributors are welcomed to join.

`{COVOID}`

COVOID (for COVID-19 Open-source Infection Dynamics project) is a R package for modelling COVID-19 and other infectious diseases using deterministic compartmental models (DCMs).

It contains a built-in Shiny app enabling easy use and demonstration of key concepts to those without R programming backgrounds, along with an expanding API for simulating and estimating homogeneous and age-structured SIR, SEIR and extended models. In particular COVOID allows the simultaneous simulation of age specific (e.g. school closures) and general interventions over varying time intervals.

The code is available on GitHub.

`{cdccovidview}`

Published by Bob Rudis, the {cdccovidview} package can be used to work with the U.S. CDC’s New COVID-19 Trackers: COVIDView and COVID-NET.

`{babsim.hospital}`

Published by several researchers from TH Köln, the {babsim.hospital} package implements a discrete-event simulation model for a hospital resource planning problem. It can be used by health departments to forecast demand for intensive care beds, ventilators, and staff resources.

The team also developed a Shiny app which predicts COVID-19 ICU bed resources in hospitals. The app is available in English and German.

`{EpiLPS}`

Published by several researchers working in the field of epidemiology, the package shows how to smooth epidemic curves and estimate the time-varying reproduction number in a flexible way.

More information can be found in:

the research paper,
the accompanying website, and
this summary written by one of the authors.

R code and blog posts

Analyzing COVID-19 outbreak data with R

Written by Tim Churches, these two articles (part 1 and part 2) explore the R tools and packages that might be used to analyze the COVID-19 data. In particular, the author considers when the pandemic will subside in China, and then turns the analysis on Japan, South Korea, Italy and Iran. He also shows improvements on the cumulative incidence plots that are so common. Moreover, he presents R code to analyze how contagious is the Coronavirus thanks to the classic SIR (Susceptible-Infectious-Recovered) compartmental model of communicable disease outbreaks.²

The code is available on GitHub (part 1 and part 2).

Part 1 is actually based on another shorter blog post by Prof. Dr. Holger K. von Jouanne-Diedrich from Learning Machines. Read his article for a more concise analysis on how to model the outbreak of the Coronavirus and discover how contagious it is. Note that I have personally written an article analyzing COVID-19 in Belgium based on articles from these two authors.

More recently, Tim Churches published a series of other interesting articles:

Modelling the effect of various public health interventions on the local epidemic spread of COVID-19 infection using stochastic individual compartmental models (ICMs) implemented by the {EpiModel} package for R:
- Part 1 (code here)
- Part 2 (code here)
The use of simulations to explore the effects of various interventions on COVID-19 spread
COVID-19 epidemiology with R: in this blog post, the author, using relatively early and partial US data, separates out inbound from community cases, and predicts the next few weeks of incident numbers. He also highlights several R functions to analyze a disease outbreak.
We can “shrink” the COVID-19 curve, rather than just flatten it (in collaboration with Louisa Jorm)

COVID-19 Data Analysis with `{tidyverse}` and `{ggplot2}`

Dr. Yanchang Zhao from RDataMining published a data analysis around the Coronavirus with the {tidyverse} and {ggplot2} packages, for China and world wide.

Both documents are a mix of data cleaning, data processing and visualizations of the confirmed/cured cases and death rates across countries or regions.

COVID-19 cumulative observed case fatality rate over time

Written by Peter Ellis, this article focuses on how the observed case fatality rate of COVID-19 has evolved over time across 7 countries and comments on why the rates vary (low testing rates, age of the population, overwhelmed hospitals, etc.).

The code is available at the end of the article. The data is from John Hopkins and it uses the {coronavirus} package.

More recently, the author published a series of other articles:

Impact of a country’s age breakdown on COVID-19 case fatality rate: it looks at estimated fatalities in different countries according to the age distributions in those countries (based on Italy’s data). The data is from The Istituto Superiore di Sanità (Roma) and all the code is shown in the post.
How to make that crazy Fox News y axis chart with ggplot2 and scales: less about COVID19 than about how a bizarre Fox News graph can be re-created with the correct transformations needed to make its scale appropriate.
Test positivity rates and actual incidence and growth of diseases: this blog post looks at several different ways of accounting for the information given to us by high positive testing rates for COVID-19 and looks at the impact on estimates of effective reproduction number at a point in time.
Incidence of COVID-19 in Texas after adjusting for test positivity: the author examines the trends in COVID-19 cases in Texas, with and without being adjusted by a multiplier of the square root of the test positivity rate.

Covid 19 Tracking

Written by Prof. Kieran Healy, this article discusses how to get an overview of best-available counts of deaths, using the COVID-19 Data from the European Centers for Disease Control.

Code can be found in the article and on GitHub.

More recently, the author published three other articles:

A COVID Small Multiple: this article discusses how to create a small-multiple plot of cases by country, showing the trajectory of the outbreak for a large number of countries, with the background of each small-multiple panel also showing (in grey) the trajectory of every other country for comparison.

Apple’s COVID Mobility Data: this article uses Apple’s time series mobility data for several cities and countries (via the {covdata} package) to plot three modes of getting around: driving, public transit, and walking. The series begins on January 13th indexed to 100 at the beginning of the series, so trends are relative to that baseline.

New Orleans and Normalization: this article responds to a thoughtful post by Dr. Drang regarding an improvement in normalization of the data.

Infectious diseases and nonlinear differential equations

Published by Fabian Dablander, this math intensive blog post explains what SIR and SIRS models take into account and how they calculate their results.

From a pandemic perspective, the author writes “The SIRS model extends the SIR model, allowing the recovered population to become susceptible again (hence the extra ‘S’). It assumes that the susceptible population increases proportional to the recovered population”.

More recently, the author, in collaboration with other researchers, published another blog post outlining a number of excellent visualizations of the COVID19 pandemic, as well as presenting their own dashboard.

Epidemic modelling of COVID-19 in the UK using an SIR model

Published by Thomas Wilding, this blog post applies the SIR model to UK data.

As further extensions to the model, the author suggests:

Using an SEIR model (adding an Exposed compartment for people who are infected but not yet infectious)
Adding a “Q” layer since a lot of people are being Quarantined or isolated
Considering the “hidden”” population that is infected but is denied being tested due to shortage of tests
Feasibility of a second wave / outbreak of the epidemic later in the year (as seen in previous outbreaks, such as Swine Flu)

Data sources:

Modeling Pandemics

Published by Arthur Charpentier, this series of 3 blog post (part 1, part 2, part 3) walks through the SIR model and its parameters, how ODEquations solves it, and generating the reproductive rate. It also gives a mathematical explanation of a model for how quickly a pandemic will return, albeit with diminishing intensity. Last, it explains a model that is more sophisticated than SIR, the SEIR model, and illustrates it with Ebola data.

More recently, the author published another article which examines what proportion of the population in various U.S. states have been tested for the novel Coronavirus and tries to answer the following two questions:

How many people are tested on a daily basis?
What are we actually testing for?

Last but not least, this post reproduces one of his scientific paper entitled “COVID-19 pandemic control: balancing detection policy and lockdown intervention under ICU sustainability”.

COVID-19: The Case of Germany

Published by Prof. Dr. Holger K. von Jouanne-Diedrich from Learning Machines, this blog post uses the SIR model and German data to estimate the duration and severity of the pandemic.

Download the data from Morgenpost.

More recently, the author published other articles:

COVID-19 in the US: Back-of-the-Envelope Calculation of Actual Infections and Future Deaths: Working back from reported deaths from Covid19, the post shows how to estimate infections at a prior date, based on several assumptions about fatality rates and infected periods (and acknowledging many unknowns and data problems).
How to analyze mobility trends with R using anonymized and aggregated Apple’s mobility data available to the public. The article presents a R function to return the data in a well-structured format for countries and major cities, and to visualize the drop in vehicular and pedestrian movement caused by the pandemic.

COVID-19: False Positive Alarm, demonstrating the importance of infection rates on the likelihood that someone testing positive for the Coronavirus is actually positive.

Flatten the COVID-19 Curve

Published by Michael Höhle from Theory meets practice, this blog post discusses why the message of flattening the COVID-19 curve is right, but why some of the visualizations used to show the effect are wrong: Reducing the basic reproduction number does not just stretch the outbreak, it also reduces the final size of the outbreak.

From a pandemic point of view, the author writes “Because of limited health capacities, stretching out the outbreak over a longer time period will ensure, that a larger proportion of those in need of hospital treatment will actually get it. Other advantages of this approach are to win time in order to find better treatment forms and, possibly, to eventually develop a vaccine”.

A Shiny app has also been built upon this article to investigate different scenarios.

In a second article entitled “Effective reproduction number estimation”, Michael Höhle estimates with the {R0} package the time-varying effective reproduction number during an infectious disease outbreak such as COVID-19. Using a single simulated outbreak he compares the performance of three different estimation methods.

More recently, in this article the author looks at “superspreading” in infectious disease transmission from a statistical point of view. He characterises heterogeneity in the offspring distribution [who becomes infected by the superspreader person] by the Gini coefficient instead of the usual dispersion parameter of the negative binomial distribution. This allows us to consider more flexible offspring distributions.

Flattening vs shrinking: the math of #FlattenTheCurve

Published by Ben Bolker and Jonathan Dushoff, this blog post gives a clear explanation of physical distancing and explains how physical distancing makes several beneficial outcomes possible.

The code is available on GitHub.

explainCovid19 challenge

Published by Przemyslaw Biecek, this blog post gives an overview of a model that uses gradient boosting to predict survival based on age, country, and gender. It also shows how older people are more at risk and it lets you play with the model yourself with a modelStudio interactive dashboard.

Data sources:

Google sheet (with most recent data at the end of February)
Kaggle dataset

An R Package to explore the Novel Coronavirus

Published by Patrick Tung via Towards Data Science, this blog post translates into English an R package originally written in Chinese.

Data is collected from Tencent, at https://news.qq.com/zt2020/page/feiyan.htm, which contains one of the most up-to-date public information of the Coronavirus.

Coronavirus model using R – Colombia

Published by Daniel Pena Chavez, this blog post uses the code from Prof. Dr. Holger K. von Jouanne-Diedrich to model height of pandemic in Colombia and projected deaths. The author also points out that a huge number of other variables need to be considered, such as density, climate and government response.

Data is from Rami Krispin’s GitHub.

More recently, the author published another article comparing China and Italy’s rates on log scales.

COVID-19: The Case of Spain

Written by Jose from Diarium - Statistics and R software, this blog post, using data for Spain, applies the SIR model, and then a cubic polynomial regression model to predict infections, hospitalizations, deaths and peak date.

Tidying the new Johns Hopkins Covid-19 time-series datasets

Written by Joachim Gassen, this blog post provides functions and code to deal with different country names and changes on the Johns Hopkins site.

More recently, the author published a series of other interesting articles:

Merge Covid-19 Data with Governmental Interventions Data: this article analyzes five kinds of intervention on the spread of COVID-19.

Scraping Google Covid-19 community movement data from PDF figures: this article explains how to scrape data from a Google site that tracks movements of people. The author uses the {tidycovid19} R package and prepares an analysis of Germany and then across countries.

Covid-19: Explore Your Visualizer Degrees of Freedom: in this article, the author uses COVID-19 data to demonstrate how graphs can communicate very differently, and be manipulated. He shows that getting a ‘neutral’ message to the reader is far from trivial and that visualizations without guidance can be particularly misleading.
{tidycovid19} New data and documentation: A recent update to the {tidycovid19} package brings data on testing, alternative case data, some regional data and proper data documentation. Using all this, you can use the package to explore the associations of (the lifting of) governmental measures, citizen behavior and the Covid-19 spread.
Exploring and Benchmarking Oxford Government Response Data: A post assessing the impact of non-pharmaceutical interventions on the spread of Covid-19 based on the Assessment Capacities Project (ACAPS) and the Oxford Covid-19 Government Response Tracker.

COVID-19 in Belgium

Based on Tim Churches’ article, I published an analysis of the COVID-19 specifically for Belgium. In this article, I also use the most common epidemiological model, the SIR model (to its simplest form), to analyze the outbreak of the disease in the case where there would be no public health intervention. I also show how to compute the reproduction number and I present some additional improvements that can be made to further analyze the epidemic.

The code is available on GitHub, so feel free to use it as starting point for an analysis of the virus outbreak in your own country.

Facts About Coronavirus Disease 2019 (COVID-19) in 5 Charts created with R and ggplot2

Written by Gregory Kanevsky, this blog post compiles some useful facts about COVID-19 into 5 charts, including gauge charts, and discusses R and {ggplot2} techniques used to create them.

Contagiousness of COVID-19 Part I: Improvements of Mathematical Fitting

Written by Martijn Weterings on Learning Machines, this guest post describes the fitting of Covid-19 data with the SIR model and explains tricky parts of the fitting methodology and how we can mitigate some of the problems (e.g., early stopping of the algorithm or an ill-conditioned problem). It provides a very clear explanation of some tweaks to the standard model.

The code is available here.

Coronavirus : spatially smoothed decease in France and decease animation map

Published by Michael Ires, this blog post shows R code on how to use kernel weighted smoothing with arbitrary bounding areas to display a map of deaths from Covid-19 in France.

The author also published two other articles on how to build an animated map of deaths from Covid-19 in France and in Europe.

More recently, the author published an article where he uses the {geogrid} package to show the incidence of Covid19 by French departments as well as spread.

Another “flatten the COVID-19 curve” simulation… in R

Written by Javier Fernandez-Lopez, this blog post shows R code to create static plots and then simulations to demonstrate how social distancing could help to “flat the curve” of COVID-19 infections.

Tracking Covid19 Cases Throughout NJ with R

Written by Kevin Zolea, this blog post shows how to use the {gganimate} package to create an animated time series map showing how Covid19 spread throughout the U.S. state of New Jersey’s counties.

It’s fun to look at the YACM (Yet Another COVID Model)

Written by Adrian Barnett from Median Watch, all the models in this blog post are based on the excellent ordinary differential equation models by Alison Hill. They are microsimulations of those models that make heavy use of the {MicSim} package for running microsimulations in R.

Is COVID-19 as bad as all that? Yes it probably is

Written by Dr. Francis Smart from Econometrics By Simulation, this article explains clearly some of the factors that determine the infections and deaths from COVID19, with different scenarios of seriousness.

Potential Long-Term Intervention Strategies for COVID-19

On this website, several professors and members of Stanford University (Marissa Childs, Morgan Kain, Devin Kirk, Mallory Harris, Jacob Ritchie, Lisa Couper, Isabel Delwel, Nicole Nova, Erin Mordecai) developed a compartmental model of COVID-19 to evaluate possible outcomes of non-pharmaceutical interventions such as social distancing.

The website presents an introduction to the problem, the possibility to play around with the model to predict the effects of COVID intervention strategies (thanks to a Shiny app), the model details, and predictions for Santa Clara County, California.

The code is available on GitHub.

Animations in the time of Coronavirus

Written by Martin Henze from the Heads or Tails blog, this article describes how to extract and prepare the necessary data to animate the spread of the virus over time in Germany, using {gganimate} and {sf} R packages to create animated map visuals.

The author posted the dataset associated with the Germany maps to Kaggle, where he is maintaining it on a daily basis. In addition, he posted a version of the JHU US county level dataset where he added some key demographic info from the US census. This dataset is also updated daily.

COVID-19 Data and Prediction for Michigan

Written by Nagdev Amruthnath, this blog post builds and tests an exponential regression model based on (not much) State of Michigan data.

Data Visualization of COVID-19 in the US

Written by Daniel Reiff, this article examines COVID-19 growth dynamics using exponential and logistic curves.

The spread of COVID-19 across countries visualization with R

Written by Sergey Bryl, this article examines the speed and spreading of the virus across countries. One animated visualization and two stationary charts show:

how long and intensive were previous phases and
compare the effectiveness against COVID-19 for different countries

The code can be found at the end of the article.

Covid-19 and Rural Areas in the U.S

Written by Elliot Meador from Deltanomics, this blog post focuses on cases of Covid-19 in rural areas of the U.S, including whether in the South any particular state appears to be an outlier.

Covid Death Rates: Is the data correct?

Written by Sam Weiss, this post raises questions about the accuracy of reports of case numbers, in that they may fail to backfill for corrected data.

More recently, the author published two articles (part 1 and part 2) in which he finds and visualizes an association between number of people that tested positive for COVID-19 in a country and imports from China. In addition he finds that there are particular industries that are particularly correlated with COVID-19 rates.

COVID-19 Risk Heat Maps with Location Data, Apache Arrow, Markov Chain Modeling, and R Shiny

Written by Filip Stachura, this post describes Appsilon’s solution (CoronaRank) submitted to the recent Pandemic Response Hackathon. Inspired by Google’s PageRank, it uses geolocation data in the Apache Parquet format from Veraset for effective exposure risk assessment using Markov Chain modeling.

COVID-19 Tracker Indonesia

Written by Dio Ariadi from DataWizArt, this post shows some very nice plots on regional variation in COVID-19 cases and deaths in Indonesia, with very neatly integrated R code for each plot.

COVID-19 Projections Using Machine Learning

Developed by Youyang Gu, this website presents an intuitive model that builds machine learning techniques on top of a classic infectious disease model to make COVID-19 infections and deaths projections for the US, all 50 US states, and more than 60 countries.

The code can be found on GitHub.

COVID-19 in Belgium: is it over yet?

Written by myself in collaboration with Prof. Niko Speybroeck and Angel Rosas-Aguirre, this article shows the evolution of the number of hospital admissions and the number of confirmed cases in Belgium (by province and at the national level).

Code of the plots is available in the article.

COVID-19 Cases by Ethnicity

Written by Tommi Suvitaival, this article investigates Covid19 deaths as a function of the percentage of a county’s population that comes from various ethnic backgrounds.

Tennessee COVID-19 Update

Written by Prof. James M. Luther, this document presents a summary of the daily data for the state of Tennessee. The author uses an interactive map, a seven-day rolling average of various metrics, and facet charts.

Simulating Coronavirus Outbreak in Cities with Origin-Destination Matrix and SEIR Model

Written by Yihui Fan, this blog post lays out a step-by-step guide on simulating and visualising the spread of Coronavirus in the Greater Tokyo Area based on Origin-Destination Matrix and SEIR Model.

Another post from the same author focuses on the effectiveness of reducing population movement in managing Coronavirus outbreak.

COVID-19 Population Mobility - How has human mobility changed under the COVID-19 Pandemic?

Written by Kelsey E. Gonzalez, this visualization aims to understand population behavior during the COVID-19 pandemic.

Data is from Cuebiq and the code is available on GitHub.

How to Build COVID-19 Data-Driven Shiny Apps in 5 minutes

Written by Emanuele Guidotti, this tutorial shows how to build a simple yet complete Shiny application using the R Package COVID19: R Interface to COVID-19 Data Hub.

In another post, the author explores the R package {COVID19} in further detail.

Analyzing data from COVID19 R package

Written by Pablo Cánovas, this blog post explores whether deaths from COVID19 are being reported accurately.

It uses data from The Human Mortality Database and the COVID-19 Data Hub.

Body Mass and Risk from COVID-19 and Influenza

Written by Prof. Radford Neal, this blog post looks at data from flu-like illnesses and some preliminary Covid19 data. The author concludes that being underweight and being seriously obese are both risk factors for serious respiratory illness.

More recently, the author published a series of other articles:

The Puzzling Linearity of COVID-19 discussing the fact that for many countries, the linear plots of total cases or total deaths go up exponentially at first, and then approach a straight line that is not horizontal
Seasonality of COVID-19, Other Coronaviruses, and Influenza: this post looks at the evidence for seasonality in influenza and the common cold Coronaviruses, and to what extent one might expect COVID-19 to also be seasonal
Critique of “Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period”: this post analyzes and criticizes an earlier paper by Kissler et al. (2020). See also part 2 and part 3

HMD – Weekly Data

Written by Ronald Richman, this blog post explores the highly improbable level of deaths currently being reported using the Human Mortality Database and its recently begun special time series of weekly death data across 13 countries.

Guest posts on Chris Muir’s blog

Written by Skylar Hara, this blog post examines whether the incidence of Covid19 changed after the Governor of Hawaii issued a stay-at-home order.

Written by Ronja Steinbach, this blog post gathers data on Trump or Clinton states, percentages of minorities, and median household income to answer the following questions:

Does party affiliation of a state in the 2016 election have a significant impact on the incidence rate of the virus in that state?
Does the proportion of the population that is minority and median household income affect the incident rate of the virus across states?

Written by Akemi Santiago, this post shows a statistically different mortality rate between African-Americans in the United States and white Americans.

Written by Masha Rutenberg, this post focuses on the correlation between the population of a country and the number of confirmed infections in the country.

All authors are students of Prof. Chris Muir.

An R View into Epidemiology

Written by Joseph Rickert in R Views, this blog post tracks down R packages that help epidemiology research and shows the number of downloads in recent months for the five most popular packages.

Articles by Rob J Hyndman

Rob J. Hyndman, Professor of Statistics and Head of the Department of Econometrics and Business Statistics at Monash University (Australia), published a series of Covid19 related articles:

Forecasting COVID-19: this blog post does not use R, although Prof. Hyndman is an expert with it, but it does explain some of the problems with time series forecasting or other methods of forecasting
Why log ratios are useful for tracking COVID-19: this post presents the benefits of reporting log-scale graphics
Excess deaths for 2020: the reported COVID19 deaths in each country are often underestimated. One way to explore the true mortality effect of the pandemic is to look at “excess deaths” — the difference between death rates this year and the same time in previous years
Seasonal mortality rates: this post shows how the weekly mortality data published by the Human Mortality Database can be used to explore seasonality in mortality rates. Mortality rates are known to be seasonal due to temperatures and other weather-related effects

Turkey vs. Germany: COVID-19

Written by Selcuk Disci from DataGeeek, this article compares efforts by Turkey and Germany to control the pandemic, and tests several regression models.

Hands-on: How to build an interactive map in R-Shiny: An example for the COVID-19 Dashboard

Written by Sangmeng, this post explains how to build an interactive dashboard with Shiny with an example for the COVID-19 Dashboard.

Modelling COVID-19 in Morocco

Written by Zakariah Gassasse, this blog post presents data on cases of Covid19 in Morocco and applies the SIR model to the data.

More recently, the author published two other articles. The first article provides a short-term forecast of COVID-19 cases and deaths in Morocco by using simple but effective time-series analyses. The second article maps the outbreak to see which regions are suffering the most.

SIR models with Kermack and McKendrick

Written by Pierre Jacob, this blog post is mostly a retrospective look at the origins of the much-used SIR model.

Johns Hopkins Covid-19 Data and R

Written by Steve Miller, these blog posts (part 1 & part 2) showcase the handling of daily data of cases/deaths from Covid-19 in the U.S. published by Johns Hopkins University, and visualize moving averages of cases and deaths.

The author also published a blog post putting unemployment claims in the U.S. in perspective.

Estimating COVID-19’s $R_t$ in Real-Time

Written by Ramnath Vaidyanathan, this tutorial shows how to estimate $R_t$, the measure known as effective reproduction number, which is the number of people who become infected per infectious person at time $t$.

From static to animated time series: the tidyverse way

Written by Giulia Ruggeri, this post goes through the steps necessary to create an animated COVID-19 time series plot.

Sneak peek: new Summit data tool helps clients visualize US areas that are most heavily impacted by the COVID-19 virus

Written by Colby Ziegler, this blog post describes a tool (built primarily in R using the {leaflet}, {tidyverse}, and {tigris} packages) tracking and displaying the total number of confirmed COVID-19 cases and deaths by U.S. county, overlaid with the locations of Certified Community Development Financial Institutions (CDFIs).

How to Reproduce Financial Times Style COVID19 Daily Reporting?

Written by Peyman Kor, this blog post shows how to reproduce in R the Financial Times facet plot by country.

Visualizing COVID cases in Belgium

In this blog post, Koen Hufkens plots cases in Belgium and addresses a challenge in geo-spatial plotting.

A spatio-temporal analysis of the environmental correlates of COVID-19 incidence in Spain

Written by Antonio Paez and several co-authors, this blog post looks at weather, humidity and other factors in Spain to create a SUR model.

Another post by Antonio Paez investigates the incidence of COVID-19 in the United States using Google Community Mobility Reports.

Covid-19 Analysis

Written by Rizami Annuar, this post compares data on cases and deaths in Malaysia to other countries, including correlations.

Australian governments can choose to slow the spread of coronavirus, but they would need to act immediately

Written by Matt Cowgill, this blog post shows that Australia early on had relatively few cases, but the post argues that the country needed to act urgently. Data is from the {gtrendsR} package.

In a more recent post, using Google trends data for key words associated with unemployment, the authors trace the effects of COVID19 in Australia in the early months.

In addition to these posts, a series of other posts have been published by Stephen Duckett and Brendan Coates:

Does Covid raise everyone’s relative risk of dying by a similar amount? More evidence

Written by Prof. David Spiegelhalter (author of the great book “The Art of Statistics”), this post looks at relative mortality rates by age and gender, using data from the U.K.’s Office for National Statistics.

Tracking Coronavirus: Building Parameterized Reports to Analyze Changing Data Sources

In this post, using Johns Hopkins data, Tyler Sanders builds a virus dashboard that can be updated each day with just the click of a button as an example of how to build parameterized reports with R Markdown.

Mapping NZ cases of COVID-19

In this post, using his own choropleth package ({DHBins}) and its hexagonal bins, Thomas Lumley maps cases of COVID19 by Health Boards in New Zealand.

Visualize the Pandemic with R #COVID-19

In this post, Xinhan Qian carries out a variety of explorations with Covid19 data, including the precipitous declines in U.S. movie box office revenue and restaurant reservations.

Exploring the Temporal Evolution of COVID-19 Cases in the United States

Written by Robert Winkelman and Colin Waltz, this post shows clearly how to create animated plots of the spread of COVID19 infections in the United States.

R Data Analysis: COVID-19

Published on the blog of Sharp Sight, this series of blog posts (part 1, 2, 3, 4, 5 and 6) explain how to rename and reorder columns, standardize dates (with the {lubridate} package), merge datasets, take other preparatory steps, plot with the {ggplot2} package and finally, reproduce in R a plot that shows the relative progress of 16 countries in coping with the pandemic.

All posts use the Johns Hopkins data and data from the company’s own collection.

GA COVID-19 Reports

Based on data from the GA Department of Public Health’s report, Andrew Benesh posts a daily analysis of the U.S. state Georgia’s cases, deaths, ICU usage etc.

All his reports are posted on Medium and the code is available here.

Corona in Belgium

Published by bnosac, this post covers the early exploration of the exponential spread of Covid19, with a focus on Belgium and the Netherlands.

The Coronavirus in Italy from the Twitter’s Point of View

Written by the Kode team, these two posts (here and here) use text-mining techniques to analyze tweets in Italy early in the pandemic and on speeches by the Prime Minister of Italy regarding Covid19.

They also published an interactive dashboard (in Italian) allowing to explore the data released daily by the Civil Protection.

Use R and Tidycensus to Look at COVID-19 Risk Factors

Written by René F. Najera, this post uses US census data to identify “overcrowded” areas and considers them in terms of Covid19 risk.

Animating U.S. COVID-19 hotspots over time

Written by Nathan Chaney, this post presents an animated map of the 7-day rolling average of new COVID-19 cases in US. Code for the animated map is available directly at the end of the post.

This post is an extension of his previous post on visualizing COVID-19 in Arkansas.

Understanding COVID19 in Connecticut. It takes a town

Based on these two posts by Nathan Chaney, Chuck Powell shows in his post how to create an animated map of 7-day rolling average of new COVID19 cases per 100,000 in Connecticut (by town).

Data

2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE: this dataset is used by many resources mentioned in this article and has become the gold standard for COVID-19 modeling
World Health Organization (WHO). See also their accompanying Shiny app
COVID-19 Open Research Dataset Challenge (CORD-19) (via Kaggle)
Novel Corona Virus 2019 Dataset: Day level information on Covid-19 affected cases (via Kaggle)
COVID-19 Data from the European Centers for Disease Control
The COVID Tracking Project
Tidying the John Hopkins Covid-19 data to long format and merging some World Bank data
A Short Review of COVID-19 Data Sources
COVID-19 datasets by esri
beoutbreakprepared/nCoV2019: one of the very few non-aggregated dataset available online. Such a dataset of individual-level information on patients with confirmed COVID-19, (including their travel history, location, symptoms, reported onset and confirmation dates and basic demographics) is important to understand, among others, transmissibility, risk of geographic spread, routes of transmission and risk factors for infection
Fighting the Covid-19: All the datasets and data efforts in one place: this post gathers many relevant datasets and data efforts
A Google Sheet which helps with tracking of the local lockdowns in the UK

Other lists or collections of resources

With so many great resources about the Coronavirus, other people also collected and organized similar lists.³ Below some collections I have been fortunate enough to discover:

COVID-19 Blog Post Directory: developed by Connor Rothschild and Rees Morrison, this Shiny app lets users interactively search a collection of over 400 posts by primary topic, post title, date, and whether the post uses a particular mathematical technique or data source. See also the accompanying blog post.
COVID-19 Modelling Resources, Data and Challenges by IDEA (not only R)
GitHub repo covid19-r by Mine Cetinkaya-Rundel (only R)
Amplifying people I trust on COVID-19: written by Jeff Leek from Simply Statistics, this article is a collection of trustworthy people and experts who share good information about the COVID-19 pandemic
Some Select COVID-19 Modeling Resources and More Select COVID-19 Resources by Joseph Rickert, assembling dashboards, Shiny apps, blog posts, packages, datasets, videos and conference proceedings that pertain to the Covid19 pandemic
CovidR Contest: launched by the European R users meeting (eRum), this contest is an open-source contest and pre-conference event, featuring any work done with R around the topic of the COVID-19 pandemic
Turning COVID-19 into a data visualization exercise for your students: written by Daniela Duca, this blog post presents a variety of methods to visualize data, drawing on several blog posts, Shiny apps and dashboards
Outbreak.info is an open source tool built by Scripps Research that standardizes and aggregates COVID-19 data. The interface aggregates journal articles, preprints, datasets, clinical trials, protocols, and other resources in one place, standardizing metadata and applying NLP to make these sources searchable and more accessible. The interactive data dashboards allow for quick comparison of countries, states, counties, and metro areas, and includes an API and R Package for researchers who want to access all of the raw data

I hope that, in addition to my collection, these rich lists done by others will give you enough background materials to analyze the outbreak of COVID-19 on your own (or at least some ideas)!

Non-english resources

This section may be of interested to only a limited number or people, but still, there are great resources in languages other than English. See a collection of them below listed by language:

Japanese:
- Coronavirus infection bulletin: original version of this dashboard
German:
- Developed by Prof. Dr. Helmut Küchenhoff, this CoronaMaps presents the situation of the Coronavirus in the world, in Europe and in Germany via a map and a table
Spanish:
- GitHub repository with official government data and R code used to extract it, see here and here
- COVID-19 en Chile: this Shiny app shows the accumulated confirmed contagion cases and provides an estimate of the growth rate for each municipality
- Developed by Que Oferton, this Shiny app provides an overview of the 2019 Novel Coronavirus COVID-19 epidemic in Central America, including statistics, forecast, SIR and SEIR models. The data and dashboard are refreshed on a daily basis
French:
- Written by Arthur Charpentier, this blog post shows how to quantify excess mortality using french mortality data

Conclusion

Thanks for reading.

I hope you will find these R resources on the COVID-19 Coronavirus useful. Feel free to let me know in the comments if I missed one.

A special thanks to Rees Morrison for his tremendous work on collecting and organizing several articles, which greatly helped in improving the section about blog posts. Read his articles (part 1 and 2) presenting a descriptive analysis of all the posts collected.

Although I have carefully read all resources, inclusion on the list does not mean that I endorse the findings. Moreover, some of the analyses, code, dashboards, packages or datasets might be out of date, so these should not be viewed, by default, as current findings. If you are the author of one of these resources, do not hesitate to contact me if you see any inconsistency or if you would like to remove it from this article.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

References

de Vlas, Sake J, and Luc E Coffeng. 2020. “A Phased Lift of Control: A Practical Strategy to Achieve Herd Immunity Against Covid-19 at the Country Level.” medRxiv.

Kissler, Stephen M, Christine Tedijanto, Edward Goldstein, Yonatan H Grad, and Marc Lipsitch. 2020. “Projecting the Transmission Dynamics of SARS-CoV-2 Through the Postpandemic Period.” Science 368 (6493): 860–68.

The package has also been the subject of a preprint.↩︎
See more information about this epidemiological model in this post by Marc Choisy.↩︎
Note that unlike my list, collections by others may include resources on COVID-19 using other tools than R.↩︎

Collaboration on Stats and R

EM-DAT, the world's disaster memory, is at risk

What is EM-DAT?

Why is it at risk?

Why does it matter?

A personal note

How you can help

More information

Bayesian Neural Networks in {tidymodels} with {kindling}

What Are Bayesian Neural Networks?

From point estimates to distributions

The reparameterisation trick

Is it possible?

Setup

Loading namespace

The train_nnsnip() Interface

Classification: Iris

The drawbacks

Updating the model

References

You can do more for neural networks in R with {kindling}

Why this post matters

What problem {kindling} solves (and what it does not)

Setup

Three levels of interaction

1) Generate model code (*_generator())

2) Direct Training

3) ML Framework Integration: {tidymodels} with mlp_kindling() / rnn_kindling()

Example 1: Iris classification with reproducible good practice

Data Preprocessing

MLP with 2 Hidden Layers

Why these choices?

Example 2: A more realistic tabular benchmark (Sonar)

About early stopping and callbacks

Supported architectures (current scope)

Variable importance (FFNN)

When {kindling} is a good fit

Limitations to keep in mind

Takeaways

nycOpenData: A unified R interface to NYC Open Data APIs

nycOpenData: A unified R interface to NYC Open Data APIs

Why nycOpenData?

What does the package do?

Mini analysis

Designed for reproducible workflows

Who is it for?

Availability

Acknowledgements

Scrape Yahoo search engine results with R

Introduction

Scraping Yahoo search engine results with R

Conclusion

10 potential career options with a degree in statistics

Introduction

What types of jobs are available?

Statistician

Data scientist, data/business analyst, data engineer or machine learning engineer

Actuary or actuarial analyst

Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst

Business intelligence analyst

Operational researcher or quality control analyst

Market or survey researcher

Economist or econometrician

(Freelance) consultant

Teacher

PhD student

Conclusion

Web scraping in R

Introduction

HTML and CSS

Web scraping vs. APIs

Why does web scraping exist if APIs are so powerful and do exactly the same work?

Web scraping in R

rvest

HTTP GET request

Parsing HTML content

CSS selector

XPath

Getting attributes

A real application of web scraping in R

The `train_nnsnip()` Interface

What problem `{kindling}` solves (and what it does not)

1) Generate model code (`*_generator()`)

3) ML Framework Integration: `{tidymodels}` with `mlp_kindling()` / `rnn_kindling()`

When `{kindling}` is a good fit

Coronavirus dashboard from the `{coronavirus} package`