Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions content/blog/2023-09-27-epidatr.Rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
title: New package released! Epidatr, an R client for Delphi Epidata API
author: Dmitry Shemetov, David Weber
date: 2023-09-27
tags:
- r
- epidata
- covidcast
authors:
- dmitry
- davidweb
heroImage: blog-lg-epidatr.jpg
heroImageThumb: blog-thumb-epidatr.jpg
summary: |
`epidatr` is designed to streamline the downloading and usage of data from the [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/). It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format.

output:
blogdown::html_page:
toc: true
---

The [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/) provides
real-time access to epidemiological surveillance data for influenza, COVID-19,
and other diseases from both official government sources such as the [Centers
for Disease Control and Prevention
(CDC)](https://www.cdc.gov/datastatistics/index.html), private partners such as
[Facebook (now
Meta)](https://delphi.cmu.edu/blog/2020/08/26/covid-19-symptom-surveys-through-facebook/)
and [Change Healthcare](https://www.changehealthcare.com/), and other public
datasets like [Google
Trends](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-search-trends).
It is built and maintained by the Carnegie Mellon University [Delphi Research
Group](https://delphi.cmu.edu/).

Today we introduce the R package `epidatr`, available [on
CRAN](https://cloud.r-project.org/web/packages/epidatr/index.html), with the
source and development [on github](https://github.com/cmu-delphi/epidatr).

This package is designed to streamline the downloading and usage of data from
the Delphi Epidata API. It provides a simple R interface to the API, including
functions for downloading data, parsing the results, and converting the data
into a tidy format. The API stores a historical record of all data, including
corrections and updates, which is particularly useful for accurately backtesting
forecasting models. We also provide packages for downstream data processing
([epiprocess](https://github.com/cmu-delphi/epiprocess)) and modeling
([epipredict](https://github.com/cmu-delphi/epipredict)).

## Usage

```
library(epidatr)
# Obtain the smoothed covid-like illness (CLI) signal from Delphi's US COVID-19
# Trends and Impact Survey (CTIS), in partnership with Facebook, as it was on
# April 10, 2021 for the US at the national level
epidata <- pub_covidcast(
source = "fb-survey",
signals = "smoothed_cli",
geo_type = "nation",
time_type = "day",
geo_values = "us",
time_values = epirange(20210101, 20210601),
as_of = 20210601
)
epidata
```

```
# A tibble: 151 × 15
geo_value signal source geo_type time_type time_value
<chr> <chr> <chr> <fct> <fct> <date>
1 us smoothed_cli fb-su… nation day 2021-01-01
2 us smoothed_cli fb-su… nation day 2021-01-02
3 us smoothed_cli fb-su… nation day 2021-01-03
4 us smoothed_cli fb-su… nation day 2021-01-04
5 us smoothed_cli fb-su… nation day 2021-01-05
6 us smoothed_cli fb-su… nation day 2021-01-06
7 us smoothed_cli fb-su… nation day 2021-01-07
8 us smoothed_cli fb-su… nation day 2021-01-08
9 us smoothed_cli fb-su… nation day 2021-01-09
10 us smoothed_cli fb-su… nation day 2021-01-10
# ℹ 141 more rows
# ℹ 9 more variables: direction <dbl>, issue <date>,
# lag <dbl>, missing_value <dbl>, missing_stderr <dbl>,
# missing_sample_size <dbl>, value <dbl>, stderr <dbl>,
# sample_size <dbl>
# ℹ Use `print(n = ...)` to see more rows
```

## Installation

Installing the package is straightforward.

```
# Install the CRAN version
pak::pkg_install("epidatr")

# Install the development version from the GitHub dev branch
pak::pkg_install("cmu-delphi/epidatr@dev")
```

### API Keys

The Delphi API requires a (free) API key for full functionality. To generate
your key, register for a pseudo-anonymous account
[here](https://api.delphi.cmu.edu/epidata/admin/registration_form) and see more
discussion on the [general API
website](https://cmu-delphi.github.io/delphi-epidata/api/api_keys.html). The
`epidatr` client will automatically look for this key in the environment
variable `DELPHI_EPIDATA_KEY`. We recommend storing your key in your `.Renviron`
file, which R will read by default.

Note that for the time being, the private endpoints (i.e. those prefixed with
`pvt`) will require a separate key that needs to be passed as an argument.

## For users of the covidcast R package

The `covidcast` package is deprecated and will no longer be updated. The
`epidatr` package is a complete rewrite of the [`covidcast`
package](https://cmu-delphi.github.io/covidcast/covidcastR/), with a focus on
speed, reliability, and ease of use. It also supports more endpoints and data
sources than `covidcast`. When migrating from that package, you will need to use
the
[`pub_covidcast`](https://cmu-delphi.github.io/epidatr/reference/pub_covidcast.html)
function in `epidatr`.
123 changes: 123 additions & 0 deletions content/blog/2023-09-27-epidatr.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
---
title: New package released! Epidatr, an R client for Delphi Epidata API
author: Dmitry Shemetov, David Weber
date: 2023-09-27
tags:
- r
- epidata
- covidcast
authors:
- dmitry
- davidweb
heroImage: blog-lg-epidatr.jpg
heroImageThumb: blog-thumb-epidatr.jpg
summary: |
`epidatr` is designed to streamline the downloading and usage of data from the [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/). It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format.

output:
blogdown::html_page:
toc: true
---

<div id="TOC">
<ul>
<li><a href="#usage">Usage</a></li>
<li><a href="#installation">Installation</a><ul>
<li><a href="#api-keys">API Keys</a></li>
</ul></li>
<li><a href="#for-users-of-the-covidcast-r-package">For users of the covidcast R package</a></li>
</ul>
</div>

<p>The <a href="https://cmu-delphi.github.io/delphi-epidata/">Delphi Epidata API</a> provides
real-time access to epidemiological surveillance data for influenza, COVID-19,
and other diseases from both official government sources such as the <a href="https://www.cdc.gov/datastatistics/index.html">Centers
for Disease Control and Prevention
(CDC)</a>, private partners such as
<a href="https://delphi.cmu.edu/blog/2020/08/26/covid-19-symptom-surveys-through-facebook/">Facebook (now
Meta)</a>
and <a href="https://www.changehealthcare.com/">Change Healthcare</a>, and other public
datasets like <a href="https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-search-trends">Google
Trends</a>.
It is built and maintained by the Carnegie Mellon University <a href="https://delphi.cmu.edu/">Delphi Research
Group</a>.</p>
<p>Today we introduce the R package <code>epidatr</code>, available <a href="https://cloud.r-project.org/web/packages/epidatr/index.html">on
CRAN</a>, with the
source and development <a href="https://github.com/cmu-delphi/epidatr">on github</a>.</p>
<p>This package is designed to streamline the downloading and usage of data from
the Delphi Epidata API. It provides a simple R interface to the API, including
functions for downloading data, parsing the results, and converting the data
into a tidy format. The API stores a historical record of all data, including
corrections and updates, which is particularly useful for accurately backtesting
forecasting models. We also provide packages for downstream data processing
(<a href="https://github.com/cmu-delphi/epiprocess">epiprocess</a>) and modeling
(<a href="https://github.com/cmu-delphi/epipredict">epipredict</a>).</p>
<div id="usage" class="section level2">
<h2>Usage</h2>
<pre><code>library(epidatr)
# Obtain the smoothed covid-like illness (CLI) signal from Delphi&#39;s US COVID-19
# Trends and Impact Survey (CTIS), in partnership with Facebook, as it was on
# April 10, 2021 for the US at the national level
epidata &lt;- pub_covidcast(
source = &quot;fb-survey&quot;,
signals = &quot;smoothed_cli&quot;,
geo_type = &quot;nation&quot;,
time_type = &quot;day&quot;,
geo_values = &quot;us&quot;,
time_values = epirange(20210101, 20210601),
as_of = 20210601
)
epidata</code></pre>
<pre><code># A tibble: 151 × 15
geo_value signal source geo_type time_type time_value
&lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;fct&gt; &lt;date&gt;
1 us smoothed_cli fb-su… nation day 2021-01-01
2 us smoothed_cli fb-su… nation day 2021-01-02
3 us smoothed_cli fb-su… nation day 2021-01-03
4 us smoothed_cli fb-su… nation day 2021-01-04
5 us smoothed_cli fb-su… nation day 2021-01-05
6 us smoothed_cli fb-su… nation day 2021-01-06
7 us smoothed_cli fb-su… nation day 2021-01-07
8 us smoothed_cli fb-su… nation day 2021-01-08
9 us smoothed_cli fb-su… nation day 2021-01-09
10 us smoothed_cli fb-su… nation day 2021-01-10
# ℹ 141 more rows
# ℹ 9 more variables: direction &lt;dbl&gt;, issue &lt;date&gt;,
# lag &lt;dbl&gt;, missing_value &lt;dbl&gt;, missing_stderr &lt;dbl&gt;,
# missing_sample_size &lt;dbl&gt;, value &lt;dbl&gt;, stderr &lt;dbl&gt;,
# sample_size &lt;dbl&gt;
# ℹ Use `print(n = ...)` to see more rows</code></pre>
</div>
<div id="installation" class="section level2">
<h2>Installation</h2>
<p>Installing the package is straightforward.</p>
<pre><code># Install the CRAN version
pak::pkg_install(&quot;epidatr&quot;)

# Install the development version from the GitHub dev branch
pak::pkg_install(&quot;cmu-delphi/epidatr@dev&quot;)</code></pre>
<div id="api-keys" class="section level3">
<h3>API Keys</h3>
<p>The Delphi API requires a (free) API key for full functionality. To generate
your key, register for a pseudo-anonymous account
<a href="https://api.delphi.cmu.edu/epidata/admin/registration_form">here</a> and see more
discussion on the <a href="https://cmu-delphi.github.io/delphi-epidata/api/api_keys.html">general API
website</a>. The
<code>epidatr</code> client will automatically look for this key in the environment
variable <code>DELPHI_EPIDATA_KEY</code>. We recommend storing your key in your <code>.Renviron</code>
file, which R will read by default.</p>
<p>Note that for the time being, the private endpoints (i.e. those prefixed with
<code>pvt</code>) will require a separate key that needs to be passed as an argument.</p>
</div>
</div>
<div id="for-users-of-the-covidcast-r-package" class="section level2">
<h2>For users of the covidcast R package</h2>
<p>The <code>covidcast</code> package is deprecated and will no longer be updated. The
<code>epidatr</code> package is a complete rewrite of the <a href="https://cmu-delphi.github.io/covidcast/covidcastR/"><code>covidcast</code>
package</a>, with a focus on
speed, reliability, and ease of use. It also supports more endpoints and data
sources than <code>covidcast</code>. When migrating from that package, you will need to use
the
<a href="https://cmu-delphi.github.io/epidatr/reference/pub_covidcast.html"><code>pub_covidcast</code></a>
function in <code>epidatr</code>.</p>
</div>
Binary file added content/blog/images/blog-lg-epidatr.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added content/blog/images/blog-thumb-epidatr.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.