Guest post on Stats and R

nycOpenData: A unified R interface to NYC Open Data APIs

Tue, 27 Jan 2026 00:00:00 +0000

Guest post by Christian Martinez, developer of the nycOpenData package in R.

nycOpenData: A unified R interface to NYC Open Data APIs

I am pleased to announce the release of nycOpenData, an R package providing convenient, tidy access to dozens of datasets from the New York City Open Data platform.

The package is designed as part of an open-science and reproducible-research effort, with the goal of lowering the friction between public data and statistical analysis—especially for teaching, exploratory research, and applied civic work.

Why nycOpenData?

NYC Open Data hosts hundreds of datasets covering topics such as public safety, housing, transportation, education, health, and city services. While these datasets are publicly accessible through the Socrata API, working with them directly often requires:

knowing dataset identifiers,
manually constructing API queries,
handling pagination, timeouts, and rate limits,
and performing repetitive data-cleaning steps.

These barriers can slow down exploratory analysis and make public data less accessible to students, researchers, and practitioners who primarily work in R.

nycOpenData was built to remove these obstacles by providing a consistent, user-friendly interface that returns clean tibbles ready for analysis—without requiring users to interact directly with the API.

What does the package do?

The package provides a growing collection of wrapper functions, each corresponding to a specific NYC Open Data dataset or dataset family. All functions follow a shared design pattern and support:

row limits,
optional filtering via named lists,
sorting,
and graceful handling of API errors and timeouts.

Examples of currently supported domains include:

311 service requests
Transportation and for-hire vehicles
Motor vehicle collisions
Department of Buildings permits and complaints
Education and school reporting
Juvenile justice and public safety
Street trees and environmental data
Permitted events (historical)

A typical call looks like this:

library(nycOpenData)

nyc_311(
  limit = 1000,
  filters = list(borough = "BROOKLYN")
)

## # A tibble: 1,000 × 40
##    unique_key created_date          agency agency_name complaint_type descriptor
##    <chr>      <chr>                 <chr>  <chr>       <chr>          <chr>     
##  1 67613985   2026-01-26T02:06:05.… NYPD   New York C… Noise - Resid… Banging/P…
##  2 67609553   2026-01-26T02:02:09.… NYPD   New York C… Noise - Resid… Banging/P…
##  3 67610990   2026-01-26T01:58:58.… NYPD   New York C… Illegal Parki… Blocked H…
##  4 67615428   2026-01-26T01:56:49.… NYPD   New York C… Noise - Resid… Banging/P…
##  5 67609568   2026-01-26T01:48:16.… NYPD   New York C… Noise - Resid… Loud Musi…
##  6 67612476   2026-01-26T01:47:10.… NYPD   New York C… Noise - Resid… Loud Musi…
##  7 67614152   2026-01-26T01:46:26.… DSNY   Department… Snow or Ice    Snow Trac…
##  8 67614054   2026-01-26T01:44:50.… DSNY   Department… Dirty Conditi… Trash     
##  9 67606570   2026-01-26T01:41:32.… NYPD   New York C… Noise - Resid… Banging/P…
## 10 67610091   2026-01-26T01:35:51.… NYPD   New York C… Noise - Vehic… Car/Truck…
## # ℹ 990 more rows
## # ℹ 34 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>, …

The result is returned as a tidy tibble of the 1,000 most recent NYC 311 requests, making it immediately compatible with the tidyverse ecosystem for visualization, modeling, and reporting.

Mini analysis

One of the strongest qualities this function has is its ability to filter based on multiple columns. Let’s put everything together and get a dataset of the last 1,000 311 requests from the New York Police Department in Brooklyn.

# Creating the dataset
brooklyn_nypd <- nyc_311(limit = 1000, filters = list(agency = "NYPD", borough = "BROOKLYN"))

# Calling head of our new dataset
head(brooklyn_nypd)

## # A tibble: 6 × 39
##   unique_key created_date           agency agency_name complaint_type descriptor
##   <chr>      <chr>                  <chr>  <chr>       <chr>          <chr>     
## 1 67613985   2026-01-26T02:06:05.0… NYPD   New York C… Noise - Resid… Banging/P…
## 2 67609553   2026-01-26T02:02:09.0… NYPD   New York C… Noise - Resid… Banging/P…
## 3 67610990   2026-01-26T01:58:58.0… NYPD   New York C… Illegal Parki… Blocked H…
## 4 67615428   2026-01-26T01:56:49.0… NYPD   New York C… Noise - Resid… Banging/P…
## 5 67609568   2026-01-26T01:48:16.0… NYPD   New York C… Noise - Resid… Loud Musi…
## 6 67612476   2026-01-26T01:47:10.0… NYPD   New York C… Noise - Resid… Loud Musi…
## # ℹ 33 more variables: location_type <chr>, incident_zip <chr>,
## #   incident_address <chr>, street_name <chr>, cross_street_1 <chr>,
## #   cross_street_2 <chr>, intersection_street_1 <chr>,
## #   intersection_street_2 <chr>, address_type <chr>, city <chr>,
## #   landmark <chr>, status <chr>, community_board <chr>,
## #   council_district <chr>, police_precinct <chr>, bbl <chr>, borough <chr>,
## #   x_coordinate_state_plane <chr>, y_coordinate_state_plane <chr>, …

# Quick check to make sure our filtering worked
nrow(brooklyn_nypd)

## [1] 1000

unique(brooklyn_nypd$agency)

## [1] "NYPD"

unique(brooklyn_nypd$borough)

## [1] "BROOKLYN"

We successfully created our dataset that contains the 1,000 most recent requests regarding the NYPD in the borough Brooklyn.

Now that we have successfully pulled the data and have it in R, let’s figure out what NYC residents in Brooklyn are complaining about to the NYPD.

To do this, we will create a bar graph of the complaint types.

# Visualizing the distribution, ordered by frequency
library(ggplot2)

ggplot(brooklyn_nypd, aes(y = reorder(complaint_type, complaint_type, length))) +
  geom_bar(fill = "steelblue") +
  theme_minimal() +
  labs(
    title = "Most Recent NYPD 311 Complaints (Brooklyn)",
    subtitle = "Top 1,000 service requests",
    x = "Number of Complaints",
    y = "Type of Complaint"
  )

Figure 1: Bar chart showing the frequency of NYPD-related 311 complaint types in Brooklyn from the 1,000 most recent service requests.

This graph shows us not only which complaints were made, but how many of each complaint were made.

Designed for reproducible workflows

A core design principle of nycOpenData is reproducibility. Rather than downloading static CSV files that can change over time or be accidentally modified, analyses can explicitly document:

which dataset was used,
how many rows were requested,
which filters were applied,
and when the data were accessed.

This makes the package particularly useful for:

reproducible research projects,
classroom assignments,
data journalism,
and exploratory civic analysis.

The package is also designed to be API-polite, with configurable timeouts and safeguards that help prevent common failure modes when querying large public datasets.

Who is it for?

nycOpenData is intended for a broad audience, including:

students learning statistics or data science using real-world data,
instructors teaching reproducible research or applied data analysis,
researchers conducting exploratory or descriptive analyses,
data journalists and civic technologists,
and anyone interested in working with NYC public data in R.

The goal is not to abstract away the data itself, but to make access predictable, transparent, and easy to integrate into standard R workflows.

Availability

The package is available on CRAN and can be installed using:

install.packages("nycOpenData")

Development continues on GitHub, where new datasets and improvements are added regularly.

Acknowledgements

This package was developed alongside teaching and applied research projects in reproducible data science, with inspiration from open-source contributors across the R community and the NYC Open Data program.

Useful links

CRAN package: https://CRAN.R-project.org/package=nycOpenData
pkgdown site: https://martinezc1.github.io/nycOpenData/
GitHub repository: https://github.com/martinezc1/nycOpenData
NYC Open Data portal: https://opendata.cityofnewyork.us/

As always, feedback, bug reports, and dataset requests are very welcome.

Thanks for reading!

Nonparametric serial interval estimation

Mon, 18 Aug 2025 00:00:00 +0000

Motivation

Epidemiological delays inform about the time between two well-defined events related to a disease. The serial interval (SI) of an infectious disease is defined as the time between symptom onset in a primary case (infector) and symptom onset in a secondary case (infectee). It is a widely used epidemiological delay quantity and plays a central role in mathematical/statistical models of disease transmission. There exists a tight link between the reproduction number (average number of secondary infections generated by an infected individual) and the serial interval. Therefore, getting accurate knowledge about the SI distribution is key to gain a clear understanding of transmission dynamics during outbreaks. Timings of symptom onset for infector-infectee pairs can be obtained from line list data and observations usually consist of calendar dates. From a mathematical perspective, it is more convenient to work with numbers than with calendar dates and the latter are typically transformed to integers for the sake of statistical analysis.

The main challenge when working with SI data is censoring in the sense that exact symptom onset times are usually unobserved and only known to have occurred between two time points. If the time resolution of a reported timing of illness onset is a calendar day, for instance July 15, there is not enough information to determine the exact time of illness onset within that day. As such, symptom onset is assumed to have occurred between July 15 and July 16 and we say that serial interval data are interval-censored. The figure below illustrates the coarse structure of SI data that adds a layer of complexity to the estimation problem.

Source: Gressani O, Hens N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Comput Biol 21(8): e1013338.

A recent article by Gressani and Hens (2025) published in PLOS Computational Biology proposes a new estimator of the cumulative distribution function of the serial interval without making parametric assumptions regarding the underlying SI distribution. The estimator is based on mixtures of uniform distributions and only requires left and right bounds of serial interval windows of infector-infectee pairs as a main input ($s_{iL}$ and $s_{iR}$ in the above figure). Point estimates of different serial interval features are available in closed-form and the bootstrap is used to compute confidence intervals. The nonparametric methodology is relatively simple and computationally fast and stable. Moreover, a user-friendly routine is available in the EpiDelays package written in R. This post aims at giving users a simple first experience with this new nonparametric methodology for serial interval estimation. The package can be installed from GitHub (using devtools) as follows:

install.packages("devtools")
devtools::install_github("oswaldogressani/EpiDelays")

Simulated data

The estimSI() routine of the EpiDelays package can be used to compute nonparametric estimates (point estimates with standard errors and confidence intervals) of different serial interval features (e.g. the mean, median, standard deviation). The routine is simple to use and requires only two inputs:

x: A data frame with $n$ rows (corresponding to the number of transmission pairs for which illness onset data is available) and two columns containing the lower bound of the SI window $s_{iL}$ (first column) and the upper bound of the SI window $s_{iR}$ (second column).
nboot: An integer for the bootstrap sample size (default is 2000) used to construct ($90\%$ and $95\%$) confidence intervals (CIs).

We start by illustrating the use of estimSI() on simulated data. The simSI() routine can be used to simulate artificial serial interval data with SI windows having a width (coarseness) of at least two days. The underlying target SI distribution is assumed to have a Gaussian distribution with mean muS and standard deviation sdS that have to be specified by the user. The code below can be used to generate $n=15$ SI windows from a Gaussian distribution with a mean of $3$ days and standard deviation of $2$ days. More details regarding the data generating mechanism can be found in the article.

set.seed(2025)
simdata <- simSI(muS = 3, sdS = 2, n = 15)
gt::gt(round(simdata, 2))

s	sl	sr	sw
4.24	3	5	2
3.07	2	4	2
4.55	4	6	2
5.54	5	7	2
3.74	3	5	2
2.67	2	4	2
3.79	3	5	2
2.84	2	4	2
2.31	1	5	4
4.40	3	6	3
2.21	1	3	2
-0.51	-2	1	3
2.16	1	3	2
4.53	2	5	3
5.13	3	7	4

The first column of the simulated dataset contains the true (unobserved) serial interval value generated from the chosen Gaussian distribution. The second and third columns contain the left and right bound of the SI window (sl and sr). Finally, the last column contains the width of the observed SI window, i.e. sw=sr-sl. The underlying target SI distribution is specified to be Gaussian with a mean of $3$ days and standard deviation of $2$ days. The 5th, 25th, 75th and 95th quantiles of the latter distribution are:

round(qnorm(p = c(0.05, 0.25, 0.75, 0.95), mean = 3, sd = 2), 1)

## [1] -0.3  1.7  4.3  6.3

We now create a data frame containing sl and sr and use the latter as an input in the estimSI() routine. Nonparametric estimates of different SI features can be accessed with $npestim$ .

xdf <- data.frame(sl = simdata$sl, sr = simdata$sr)
SIfit <- estimSI(x = xdf, nboot = 2000)
gt::gt(round(SIfit$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	3.4	1.7	0.3	2.5	3.5	4.6	6.0
se	0.4	0.3	1.1	0.4	0.3	0.4	0.5
ci90l	2.8	1.1	-1.3	1.8	2.9	3.9	5.0
ci90r	4.0	2.1	2.1	3.2	4.1	5.2	6.6
ci95l	2.7	1.1	-1.3	1.5	2.8	3.8	4.9
ci95r	4.2	2.2	2.1	3.3	4.2	5.4	6.6

The output shows point estimates (point), standard errors (se) and confidence intervals bounds (ci) for the serial interval mean, standard deviation (sd) and 5th, 25th, 50th, 75th and 95th quantiles denoted by q0.05, q0.25, etc. We can also plot the estimated cumulative distribution function (cdf) obtained with the nonparametric approach and compare it with the target Gaussian cdf. The quality of the fit will typically depend on the sample size $n$ and on the degree of coarseness present in the data. Note that the nonparametric methodology naturally deals with negative SI values.

sl <- simdata$sl
sr <- simdata$sr
Fhat <- function(s) (1 / SIfit$n) * sum((s - sl) / (sr - sl) * (s >= sl & s <= sr) + (s > sr))
sf <- seq(-3, 8, length = 100)
plot(sf, sapply(sf, Fhat), type = "l", lwd = 2, xlab = "Serial interval", ylab = "cdf")
grid()
lines(sf, pnorm(sf, mean = 3, sd = 2), col = "blue", lwd = 2)
legend("topleft", c("Estimated cdf of SI", "Target cdf of SI"), col = c("black", "blue"), lwd = c(2, 2), bty = "n")

Real data

Lessler et al. (2009) share a dataset containing serial interval windows obtained from $n=16$ infector-infectee pairs for Influenza A (2009 H1N1 influenza) at a New York City school. The SI windows are directly available from the supplementary appendix of the latter reference and are encoded in a data frame xNY. Nonparametric estimates of serial interval features are then obtained with estimSI().

xNY <- data.frame(sl = c(1, 1, 1, 0, 0, 4, 2, 3, 0, 3, 0, 3, 4, 1, 3, 3), sr = c(3, 3, 3, 2, 2, 6, 4, 5, 2, 5, 2, 5, 6, 3, 5, 5))
set.seed(123)
SIfitNY <- estimSI(xNY, nboot = 2000)
gt::gt(round(SIfitNY$npestim, 1),
  rownames_to_stub = TRUE
)

	mean	sd	q0.05	q0.25	q0.5	q0.75	q0.95
point	2.8	1.5	0.4	1.5	2.8	4.1	5.2
se	0.3	0.1	0.2	0.4	0.6	0.4	0.3
ci90l	2.2	1.3	0.2	1.1	1.8	3.2	4.7
ci90r	3.4	1.7	1.1	2.2	3.7	4.6	5.6
ci95l	2.1	1.2	0.2	1.0	1.8	3.0	4.6
ci95r	3.5	1.7	1.1	2.5	3.8	4.7	5.7

Interested readers can find more real data examples in Gressani and Hens (2025) and learn about the strengths and limitations of this new nonparametric methodology for serial interval estimation.

References

Gressani, O. and Hens, N. (2025). Nonparametric serial interval estimation with uniform mixtures. PLoS Computational Biology 21(8):e101338. https://doi.org/10.1371/journal.pcbi.1013338

Gressani; O. (2025). EpiDelays: A Software for Estimation of Epidemiological Delays (version 0.0.1). https://github.com/oswaldogressani/EpiDelays

Lessler, J., Reich, N. G., Cummings, D. A., and the New York City Department of Health and Mental Hygiene Swine Influenza Investigation Team. (2009). Outbreak of 2009 pandemic influenza A (H1N1) at a New York City school. New England Journal of Medicine 361(27), 2628-2636. https://www.nejm.org/doi/full/10.1056/NEJMoa0906089

EpiLPS for estimation of incubation times

Thu, 01 Aug 2024 00:00:00 +0000

Motivation

A group of researchers from the Data Science Institute (DSI) at Hasselt University developed a new statistical model to estimate the incubation period of a pathogenic organism based on coarse data. The incubation period of an infectious disease (defined as the time elapsed between infection and the manifestation of first symptoms) is of great importance as it permits to shed light on the epidemic potential of a disease and to optimize the length of quarantine periods to freeze transmission. The article (Gressani et al. 2024) was recently published in the American Journal of Epidemiology with practical implementation of the methodology accessible through the EpiLPS package (Gressani et al. 2022).

Coarse data

What makes estimation of incubation times so challenging in the first place? The devil lies in the data. True infection times are stealthy and rarely observed. In information-theoretic jargon this phenomenon is called “imperfect information’’ but statisticians prefer to call it censoring. To be more precise, infection times are interval censored, i.e. one part of the collected data contains exposure intervals $\mathcal{E}=[t^{E_L},t^{E_R}]$ reported by individuals that are part of the study, where $t^{E_L}$ and $t^{E_R}$ stand for the left and right bound, respectively, of the exposure window. The other part of the data contains symptom onset times $t^{\mathcal{S}}$. This is a more easily accessible piece of information -people tend to remember the day when first symptoms appeared- and so the timing of symptom onset is assumed to be exactly observed. Subtracting the exposure bounds from the symptom onset time, one obtains the incubation interval $\mathcal{I}=[t^{\mathcal{I}_L}, t^{\mathcal{I}_R}]$ with lower bound $t^{\mathcal{I}_L}=t^{\mathcal{S}}-t^{E_R}$ and upper bound $t^{\mathcal{I}_R}=t^{\mathcal{S}}-t^{E_L}$, characterizing the coarse data structure which will be the main model input.

Simulated example

In EpiLPS, the estimIncub() routine is designed to compute an estimate of the incubation density based on the methodology of Gressani et al. (2024). Giving a detailed account of the methodology would be out of scope for this blog and the reader is redirected to the article for technicalities. In a nutshell, it is a Bayesian approach making use of (penalized) B-splines, Laplace approximations and Markov chain Monte Carlo (MCMC) methods to derive a semi-parametric estimate of the incubation density. An attractive feature of the estimIncub() routine for the end-user is the minimalistic input it requires to work, namely:

x: A data frame containing the lower and upper bound of the incubation interval.
K: An integer specifying the number of B-splines to smooth the incubation density.
niter: The number of MCMC samples required.

In practice, only the data frame x is required as the remaining inputs are assigned default values. Computationally, the routine requires a small amount of resources as costly subroutines are coded in C++ and integrated in R via the Rcpp package. The structure of x is quite simple. It should be a data frame with two columns containing the left bound $t^{\mathcal{I}_L}$ of the incubation interval (in the first column) and the right bound $t^{\mathcal{I}_R}$.

Let’s start with a simple example where x is simulated based on a data generating mechanism assuming a known incubation distribution. This can be achieved with the incubsim() routine in EpiLPS. We choose x to be generated according to a Lognormal incubation distribution with a mean of 5.5 days and a standard deviation of 2.1 days following Ferretti et al. (2020). Simulation of $n=40$ observations with an average exposure window of 2 days is implemented as follows:

set.seed(2024)
simdat <- incubsim(incubdist = "LogNormal", n = 40, coarseness = 2)
gt(head(simdat$Dobsincub, 5))

tL	tR
3.724500	5.510304
4.381377	6.654224
4.588020	5.483614
3.847835	6.003948
3.522282	5.033487

By typing simdat$Dobsincub, the user has access to the generated incubation intervals (expressed in days), corresponding here to a data frame with two columns and $n=40$ rows. This is the data frame that is injected in the estimIncub() routine:

fit <- estimIncub(x = simdat$Dobsincub, verbose = TRUE)

## ----------------------------------------------------------------
## Time elapsed: 1.339 seconds.
## Fitted density is Log-Normal with meanlog=1.617 and sdlog=0.317.
## Mean incubation period (days): 5.298 with 95% CI: 5.086-5.615.
## 95th percentile (days): 8.484 with 95% CI: 8.121-9.089.
## ----------------------------------------------------------------

The output in the R console can easily be interpreted. It tells us that the model chooses a Lognormal density fit for the incubation period with a mean of 5.3 days (95% CI: 5.0-5.6 days). More detailed summary statistics are accessible by typing fit$stats, such as the posterior standard deviation and additional percentiles. What happens under the hood? Basically, the model computes a semi-parametric fit to the data and compares it with classic parametric fits (Lognormal, Weibull and Gamma) used for incubation modeling. The candidate with the lowest Bayesian information criterion (BIC) wins the game and is finally selected (here the Lognormal distribution). The incubation windows and the fitted incubation density can be obtained by typing:

grid.arrange(plot(fit, typ = "incubwin"), plot(fit, type = "pdf"), nrow = 1)

Real data example

The flexible Bayesian methodology is illustrated on SARS-CoV-2 symptom onset and exposure window data extracted from cases in Vietnam. The dataset was analyzed in Bui et al. (2020) and is publicly available on the GitHub repository provided in the article (https://github.com/longbui/Covid19IncubVN; last accessed July 17, 2024). The dataset contains information about $n=19$ cases identified from January 23, 2020 to April 13, 2020. After continuity corrections (required to change calendar dates into continuous time points), the left and right incubation bounds are given by:

# Left incubation bound
tL <- c(0.504, 2.983, 5.343, 6.969, 6.990, 2.570, 6.870, 1.263, 0.693, 1.869, 1.151, 2.748, 1.209, 1.161, 4.982, 4.017, 2.170, 9.805, 1.659)

# Right incubation bound
tR <- c(1.491, 7.762, 11.777, 12.821, 11.359, 8.551, 24.954, 4.144, 6.408, 7.083, 4.191, 10.326, 9.942, 7.254, 12.690, 10.825, 14.478, 16.592, 18.649)

# Data set
dataVietnam <- data.frame(tL = tL, tR = tR)

The estimIncub() routine provides a Weibull fit with a mean incubation period of 6.7 days (95% CI: 5.8-7.4 days) and the standard deviation (extracted by typing incubfit$stats) is 3.3 days (95% CI: 3.0-3.8 days). A figure of the incubation windows and the fitted Weibull density (with 95% credible interval) is also provided.

set.seed(2024)
incubfit <- EpiLPS::estimIncub(x = dataVietnam, verbose = TRUE, tmax = 25)

## ----------------------------------------------------------------
## Time elapsed: 1.13 seconds.
## Fitted density is Weibull with shape=2.142 and scale=7.521.
## Mean incubation period (days): 6.661 with 95% CI: 5.753-7.375.
## 95th percentile (days): 12.552 with 95% CI: 11.931-14.087.
## ----------------------------------------------------------------

grid.arrange(plot(incubfit, typ = "incubwin"), plot(incubfit, type = "pdf"), nrow = 1)

References

Gressani, O., Torneri, A., Hens, N. and Faes, C. (2024). Flexible Bayesian estimation of incubation times. American Journal of Epidemiology (Accepted manuscript). https://doi.org/10.1093/aje/kwae192

Gressani, O., Wallinga, J., Althaus, C. L., Hens, N. and Faes, C. (2022). EpiLPS: A fast and flexible Bayesian tool for estimation of the time-varying reproduction number. PLoS Comput Biol 18(10): e1010618. https://doi.org/10.1371/journal.pcbi.1010618

Eddelbuettel, D. and Francois, R. (2011). Rcpp: Seamless R and C++ Integration. Journal of Statistical Software, 40(8), 1–18. https://doi.org/10.18637/jss.v040.i08

Ferretti, L. et al. (2020). Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science 368, eabb6936. https://doi.org/10.1126/science.abb6936

Bui LV, Nguyen HT, Levine H, Nguyen HN, Nguyen T-A, Nguyen TP, et al. (2020) Estimation of the incubation period of COVID-19 in Vietnam. PLoS ONE 15(12): e0243889. https://doi.org/10.1371/journal.pone.0243889

Scrape Yahoo search engine results with R

Thu, 24 Aug 2023 00:00:00 +0000

Note: This is a guest post by Manthan Koolwal, founder of Scrapingdog.

Introduction

Web scraping is the process of extracting data from websites. It is usually done in an automated manner to obtain large amounts of data through various websites, without the need to gather data by hand.

In a previous post, we introduced this method and illustrated it with a Wikipedia page. Although there are a lot of use cases of web scraping, in this blog post, we are restricting ourselves to scraping search results from Yahoo using R. Scraping search engine results can help you with SEO analysis, competitor analysis, keyword research, trend analysis, etc.

Scraping Yahoo search engine results with R

After installing R and RStudio, we first need to load the necessary packages by running the following commands:¹

# install.packages("rvest")
# install.packages("jsonlite")
# install.packages("purrr")

library(rvest)
library(jsonlite)
library(purrr)

The {rvest} package is for web scraping, the {jsonlite} package is for working with JSON data and the {purrr} package is for working with functions and vectors.

It is always better to decide in advance what exactly we are going to scrape. For this tutorial, we are going to scrape search results from this URL:

We are going to scrape the following data points from this page:

Link
Title
Description

For this, we define the URL of the Yahoo search results page that we want to scrape. In this case, we are searching for the word “pizza”.

# URL of the Yahoo search results page
url <- "https://search.yahoo.com/search?p=pizza"

We then use the read_html() function from the {rvest} package to read the HTML content of the provided URL:

# Read the HTML content of the page
page <- read_html(url)

This creates an HTML document object that we can work with:

str(page)

## List of 2
##  $ node:<externalptr> 
##  $ doc :<externalptr> 
##  - attr(*, "class")= chr [1:2] "xml_document" "xml_node"

Here’s where we start the process of extracting the search results:

# Extract search results
results <- page %>%
  html_nodes(".algo-sr") %>% # Selector for search result elements
  html_nodes("a") %>% # Select the <a> elements within the search results
  # Extract link, title, and description attributes
  map_df(~ data.frame(
    link = .x %>% html_attr("href"),
    title = .x %>% html_text(),
    description = .x %>% html_attr("title"),
    stringsAsFactors = FALSE
  ))

In the code above:

We use %>% (pipe operator) to chain multiple operations together for clarity.
First, we use html_nodes(".algo-sr") to select the search result elements with the class .algo-sr. These elements contain the links to the search results.
Within each search result element, we further select the <a> elements using html_nodes("a"). These <a> elements contain the link, title, and description information.
Using map_df(), we iterate through each <a> element and extract the link, title, and description attributes.
We create a data frame with these attributes for each search result.

Finally, we convert the results data frame into JSON format using the toJSON() function from the {jsonlite} package. The pretty = TRUE argument adds indentation for better readability. We use cat() to print the JSON-formatted results to the console.

# Print the results in JSON format
cat(toJSON(results, pretty = TRUE))

## [
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.yelp.com%2fsearch%3ffind_desc%3dpizza%26find_loc%3dBrussels%252C%2bR%25C3%25A9gion%2bde%2bBruxelles-Capitale/RK=2/RS=xW.v1O1REkKKLsO7DRIWoIGq4bA-",
##     "title": "yelp.comhttps://www.yelp.com › searchTop 10 Best pizza Near Brussels, Région de Bruxelles-Capitale"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzIEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.yelp.com%2fsearch%3fcflt%3dpizza%26find_loc%3dBrussels%252C%2bR%25C3%25A9gion%2bde%2bBruxelles-Capitale/RK=2/RS=zynV6ypY6PoaPj4cg5yD9a1x.LU-",
##     "title": "yelp.comhttps://www.yelp.com › searchTHE BEST 10 PIZZA PLACES IN BRUSSELS, RÉGION DE ... - Yelp"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIASvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.dominos.com%2fen/RK=2/RS=darBKTs8REd78kRRFhZb8jRT4jY-",
##     "title": "Domino'shttps://www.dominos.com › enPizza Delivery & Carryout, Pasta, Chicken & More | Domino's"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAS_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3ds2TCGJY8%26id%3d7236CCFBE7C0B5849BA9DE100BE4F1F32FCD231D%26thid%3dOIP.s2TCGJY8bjV7G5kps1FEwQHaHa%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.98.1%2fimages%2ftiles%2fmixAndMatchDeal%2fhero.webp%26exph%3d530%26expw%3d530%26q%3dpizza%26ck%3d8B813BA6312D3F558421CB691D0C6657%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=xsbqmguwIFBhAOzGrm.qrjLykcA-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3d%2f6Qs28Cq%26id%3d7236CCFBE7C0B5849BA9E9CE4472C5692CE3F420%26thid%3dOIP._6Qs28CqWWCt4S-yapYAXwHaHa%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.116.0%2fimages%2ftiles%2fmixAndMatchDealPSC%2fhero.webp%26exph%3d530%26expw%3d530%26q%3dpizza%26ck%3d758519383F5F3AF0EA83707449E54C87%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=UHqcKBQXuq8AtTptzc.7rkut3rY-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3dZ33s1urJ%26id%3d7236CCFBE7C0B5849BA9D44071E0F214A85C5EAF%26thid%3dOIP.Z33s1urJ3ZHwJtgKZtWHJAHaDS%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fcms%2fassets%2f58a8e864-d1aa-457a-9d37-fcc1912b10ad%3fim%3dCrop%2crect%3d%28774%2c1015%2c4331%2c1920%29%3bResize%3d%281700%29%2callowExpansion%26exph%3d754%26expw%3d1700%26q%3dpizza%26ck%3d1274A2028C287B1E2E3A8B0EB2F9DE7D%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=d9hH9.gzp8cqI8s0XYrAtGSVGXg-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIATvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3deK8nC663%26id%3d7236CCFBE7C0B5849BA99FA63EED83A3990E2BF8%26thid%3dOIP.eK8nC6636jB9TYTyjaHNGgHaFj%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fstatic%2f1.83.3%2fimages%2ftiles%2fperfectComboDeal%2fside.webp%26exph%3d458%26expw%3d610%26q%3dpizza%26ck%3dCD805B6ADACFFC2697D3BE3D36098B7E%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=KY2oqWHmG3Y6yqMzcVPRlshaVsg-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAT_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzMEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fview%3ddetailV2%26ccid%3d%2bf29F0Yv%26id%3d7236CCFBE7C0B5849BA985F6EAF3041631C46CA1%26thid%3dOIP.-f29F0YvYvZaKklmamQDSwHaDS%26mediaurl%3dhttps%3a%2f%2fwww.dominos.com%2fcms%2fassets%2f43778453-e8b1-4db6-83fa-00f2dc43d6e3%3fim%3dCrop%2crect%3d%28374%2c325%2c3250%2c1441%29%3bResize%3d%281700%29%2callowExpansion%26exph%3d754%26expw%3d1700%26q%3dpizza%26ck%3d66B92DE33E5F605B85BE7C83BE11836C%26idpp%3drc%26idpview%3dsingleimage%26form%3drc2idp/RK=2/RS=15I_QuI3QQbRtWZj5auaBfXOKWQ-",
##     "title": ""
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAYfJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzQEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.albacioixelles.com%2f/RK=2/RS=GTpw.m6z.2eGOKxyo3K8xVrqSPE-",
##     "title": "Albacioixelleshttps://www.albacioixelles.comAuthentic | Albacioixelles | Elsene"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAYvJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzUEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.nonalife.com%2f/RK=2/RS=d8sI_7Fy4oweSN3IQQ32zJSfgRY-",
##     "title": "Nonahttps://www.nonalife.comNona"
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAY_JXNyoA;_ylu=Y29sbwNiZjEEcG9zAzYEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=https%3a%2f%2fwww.tripadvisor.com%2fRestaurant_Review-g188644-d2480305-Reviews-Pizza_Saco-Brussels.html/RK=2/RS=52RUI7KZf9vt_R13PVY7sv04RMQ-",
##     "title": "Tripadvisorhttps://www.tripadvisor.com › Restaurant_Review-g188644-dPIZZA SACO, Brussels - Avenue Milcamps 154 - Restaurant ..."
##   },
##   {
##     "link": "https://r.search.yahoo.com/_ylt=AwrEteLKS0ZqawIAZPJXNyoA;_ylu=Y29sbwNiZjEEcG9zAzcEdnRpZAMEc2VjA3Ny/RV=2/RE=1784201418/RO=10/RU=http%3a%2f%2fwww.pizzasaco.com%2f/RK=2/RS=Zi2tEoNCQKejdFUdm0zsZwuqoig-",
##     "title": "pizzasaco.comhttp://www.pizzasaco.comSaco Pizza Bar"
##   }
## ]

Conclusion

Thanks for reading.

This was a simple tutorial in which we scraped Yahoo search results with R. Following the same process, you can create your own web crawler which can scrape search results from Yahoo for any web query.

Of course, you can scrape other search engines with almost the same technique. Also, you can check out this tutorial on web scraping Google search results using Python.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

Note that, as for any R package, it must first be installed (with the install.packages() function) before being loaded (with the library() function).↩︎