Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions content/blog/2023-09-27-epidatr.Rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
title: New package released! Epidatr, an R client for Delphi Epidata API
author: Dmitry Shemetov, David Weber
date: 2023-09-27
tags:
- r
- epidata
- covidcast
authors:
- dmitry
- davidweb
heroImage: blog-lg-epidatr.jpg
heroImageThumb: blog-thumb-epidatr.jpg
summary: |
`epidatr` is designed to streamline the downloading and usage of data from the [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/). It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format.

output:
blogdown::html_page:
toc: true
---

The [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/) provides real-time access to epidemiological surveillance data for influenza, COVID-19, and other diseases from both official government sources such as the [Center for Disease Control (CDC)](https://www.cdc.gov/datastatistics/index.html), private partners such as [Facebook](https://delphi.cmu.edu/blog/2020/08/26/covid-19-symptom-surveys-through-facebook/) and [Change Healthcare](https://www.changehealthcare.com/), and other public datasets like [Google Trends](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-search-trends). It is built and maintained by the Carnegie Mellon University [Delphi research group](https://delphi.cmu.edu/).

Today we introduce the R package "`epidatr`", available [on CRAN](https://cloud.r-project.org/web/packages/epidatr/index.html), with the source and development [on github](https://github.com/cmu-delphi/epidatr).

This package is designed to streamline the downloading and usage of data from the Delphi Epidata API. It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format. The API stores a historical record of all data, including corrections and updates, which is particularly useful for accurately backtesting forecasting models. We also provide packages for downstream data processing ([epiprocess](https://github.com/cmu-delphi/epiprocess)) and modeling ([epipredict](https://github.com/cmu-delphi/epipredict)).

## Usage

```R
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this code block seems to be automatically collapsed/folded/hidden, which is kinda silly because the article is for a package that people will use in their own code (and the resulting output block below it is not collapsed).

I think we can fix this by removing the R at the top of this block, or by replacing the R with {r class.source = 'fold-show'} as described here. The former is much simpler and i think i would go with that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried this with no luck. I have a feeling that we're not really using blogdown, but instead just rendering html that is then passed through our Hugo blog's styling.

library(epidatr)
# Obtain the smoothed covid-like illness (CLI) signal from the Facebook survey as it was on April 10, 2021 for the US

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Obtain the smoothed covid-like illness (CLI) signal from the Facebook survey as it was on April 10, 2021 for the US
# Obtain the smoothed covid-like illness (CLI) signal from the Delphi's US
# COVID-19 Trends and Impact Survey (CTIS), in partnership with Facebook, as it
# was reported on April 10, 2021, for the US at a national level

trying to get closer to the language here. Not sure about suggesting "Meta" above & "Facebook" here though

epidata <- covidcast(
source = "fb-survey",
signals = "smoothed_cli",
geo_type = "nation",
time_type = "day",
geo_values = "us",
time_values = epirange(20210101, 20210601),
as_of = "2021-06-01")
epidata
```

```
# A tibble: 6 × 15
geo_value signal source geo_type time_type time_value
<chr> <chr> <chr> <fct> <fct> <date>
1 us smoothed_cli fb-surv… nation day 2021-04-05
2 us smoothed_cli fb-surv… nation day 2021-04-06
3 us smoothed_cli fb-surv… nation day 2021-04-07
4 us smoothed_cli fb-surv… nation day 2021-04-08
5 us smoothed_cli fb-surv… nation day 2021-04-09
6 us smoothed_cli fb-surv… nation day 2021-04-10
# ℹ 9 more variables: direction <dbl>, issue <date>,
# lag <int>, missing_value <int>, missing_stderr <int>,
# missing_sample_size <int>, value <dbl>, stderr <dbl>,
# sample_size <dbl>
```

## Installation

You can install the stable version of this package from CRAN:

``` r
install.packages("epidatr")
pak::pkg_install("epidatr")
renv::install("epidatr")
```

Or if you want the development version, install from GitHub:

``` r
# Install the dev version using `pak` or `remotes`
pak::pkg_install("cmu-delphi/epidatr")
remotes::install_github("cmu-delphi/epidatr")
renv::install("cmu-delphi/epidatr")
```

### API Keys

The Delphi API requires a (free) API key for full functionality. To generate
your key, register for a pseudo-anonymous account
[here](https://api.delphi.cmu.edu/epidata/admin/registration_form) and see more
discussion on the [general API
website](https://cmu-delphi.github.io/delphi-epidata/api/api_keys.html). The
`epidatr` client will automatically look for this key in the R option
`delphi.epidata.key` or in the environment variable
`DELPHI_EPIDATA_KEY`. We recommend storing your key in `.Renviron` file, which R
will read by default.

Note that for the time being, the private endpoints (i.e. those prefixed with
`pvt`) will require a separate key that needs to be passed as an argument.

## For users of the `covidcast` R package

The `epidatr` package is a complete rewrite of the [`covidcast` package](https://cmu-delphi.github.io/covidcast/covidcastR/), with a focus on speed, reliability, and ease of use. The `covidcast` package is deprecated and will no longer be updated.
93 changes: 93 additions & 0 deletions content/blog/2023-09-27-epidatr.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: New package released! Epidatr, an R client for Delphi Epidata API
author: Dmitry Shemetov, David Weber
date: 2023-09-27
tags:
- r
- epidata
- covidcast
authors:
- dmitry
- davidweb
heroImage: blog-lg-epidatr.jpg
heroImageThumb: blog-thumb-epidatr.jpg
summary: |
`epidatr` is designed to streamline the downloading and usage of data from the [Delphi Epidata API](https://cmu-delphi.github.io/delphi-epidata/). It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format.

output:
blogdown::html_page:
toc: true
---


<div id="TOC">
<ul>
<li><a href="#usage" id="toc-usage">Usage</a></li>
<li><a href="#installation" id="toc-installation">Installation</a>
<ul>
<li><a href="#api-keys" id="toc-api-keys">API Keys</a></li>
</ul></li>
<li><a href="#for-users-of-the-covidcast-r-package" id="toc-for-users-of-the-covidcast-r-package">For users of the <code>covidcast</code> R package</a></li>
</ul>
</div>

<p>The <a href="https://cmu-delphi.github.io/delphi-epidata/">Delphi Epidata API</a> provides real-time access to epidemiological surveillance data for influenza, COVID-19, and other diseases from both official government sources such as the <a href="https://www.cdc.gov/datastatistics/index.html">Center for Disease Control (CDC)</a>, private partners such as <a href="https://delphi.cmu.edu/blog/2020/08/26/covid-19-symptom-surveys-through-facebook/">Facebook</a> and <a href="https://www.changehealthcare.com/">Change Healthcare</a>, and other public datasets like <a href="https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-search-trends">Google Trends</a>. It is built and maintained by the Carnegie Mellon University <a href="https://delphi.cmu.edu/">Delphi research group</a>.</p>
<p>Today we introduce the R package “<code>epidatr</code>”, available <a href="https://cloud.r-project.org/web/packages/epidatr/index.html">on CRAN</a>, with the source and development <a href="https://github.com/cmu-delphi/epidatr">on github</a>.</p>
<p>This package is designed to streamline the downloading and usage of data from the Delphi Epidata API. It provides a simple R interface to the API, including functions for downloading data, parsing the results, and converting the data into a tidy format. The API stores a historical record of all data, including corrections and updates, which is particularly useful for accurately backtesting forecasting models. We also provide packages for downstream data processing (<a href="https://github.com/cmu-delphi/epiprocess">epiprocess</a>) and modeling (<a href="https://github.com/cmu-delphi/epipredict">epipredict</a>).</p>
<div id="usage" class="section level2">
<h2>Usage</h2>
<pre class="r"><code>library(epidatr)
# Obtain the smoothed covid-like illness (CLI) signal from the Facebook survey as it was on April 10, 2021 for the US
epidata &lt;- covidcast(
source = &quot;fb-survey&quot;,
signals = &quot;smoothed_cli&quot;,
geo_type = &quot;nation&quot;,
time_type = &quot;day&quot;,
geo_values = &quot;us&quot;,
time_values = epirange(20210101, 20210601),
as_of = &quot;2021-06-01&quot;)
epidata</code></pre>
<pre><code># A tibble: 6 × 15
geo_value signal source geo_type time_type time_value
&lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;fct&gt; &lt;date&gt;
1 us smoothed_cli fb-surv… nation day 2021-04-05
2 us smoothed_cli fb-surv… nation day 2021-04-06
3 us smoothed_cli fb-surv… nation day 2021-04-07
4 us smoothed_cli fb-surv… nation day 2021-04-08
5 us smoothed_cli fb-surv… nation day 2021-04-09
6 us smoothed_cli fb-surv… nation day 2021-04-10
# ℹ 9 more variables: direction &lt;dbl&gt;, issue &lt;date&gt;,
# lag &lt;int&gt;, missing_value &lt;int&gt;, missing_stderr &lt;int&gt;,
# missing_sample_size &lt;int&gt;, value &lt;dbl&gt;, stderr &lt;dbl&gt;,
# sample_size &lt;dbl&gt;</code></pre>
</div>
<div id="installation" class="section level2">
<h2>Installation</h2>
<p>You can install the stable version of this package from CRAN:</p>
<pre class="r"><code>install.packages(&quot;epidatr&quot;)
pak::pkg_install(&quot;epidatr&quot;)
renv::install(&quot;epidatr&quot;)</code></pre>
<p>Or if you want the development version, install from GitHub:</p>
<pre class="r"><code># Install the dev version using `pak` or `remotes`
pak::pkg_install(&quot;cmu-delphi/epidatr&quot;)
remotes::install_github(&quot;cmu-delphi/epidatr&quot;)
renv::install(&quot;cmu-delphi/epidatr&quot;)</code></pre>
<div id="api-keys" class="section level3">
<h3>API Keys</h3>
<p>The Delphi API requires a (free) API key for full functionality. To generate
your key, register for a pseudo-anonymous account
<a href="https://api.delphi.cmu.edu/epidata/admin/registration_form">here</a> and see more
discussion on the <a href="https://cmu-delphi.github.io/delphi-epidata/api/api_keys.html">general API
website</a>. The
<code>epidatr</code> client will automatically look for this key in the R option
<code>delphi.epidata.key</code> or in the environment variable
<code>DELPHI_EPIDATA_KEY</code>. We recommend storing your key in <code>.Renviron</code> file, which R
will read by default.</p>
<p>Note that for the time being, the private endpoints (i.e. those prefixed with
<code>pvt</code>) will require a separate key that needs to be passed as an argument.</p>
</div>
</div>
<div id="for-users-of-the-covidcast-r-package" class="section level2">
<h2>For users of the <code>covidcast</code> R package</h2>
<p>The <code>epidatr</code> package is a complete rewrite of the <a href="https://cmu-delphi.github.io/covidcast/covidcastR/"><code>covidcast</code> package</a>, with a focus on speed, reliability, and ease of use. The <code>covidcast</code> package is deprecated and will no longer be updated.</p>
</div>
Binary file added content/blog/images/blog-lg-epidatr.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added content/blog/images/blog-thumb-epidatr.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.