Merge pull request #154 from cmu-delphi/quidel_covidtest

pipeline for Quidel covidtest

Author: krivard

quidel_covidtest/.pylintrc

@@ -0,0 +1,8 @@
+[DESIGN]
+
+min-public-methods=1
+
+
+[MESSAGES CONTROL]
+
+disable=R0801, C0330, E1101, E0611, C0114, C0116, C0103, R0913, R0914, W0702

quidel_covidtest/DETAILS.md

@@ -0,0 +1,44 @@
+# Quidel COVID Test
+
+### Background
+Starting May 9, 2020, we began getting Quidel COVID Test data and started reporting it from May 26, 2020 due to limitation in the data volume. The data contains a number of features for every test, including localization at 5-digit Zip Code level, a TestDate and StorageDate, patient age, and several identifiers that uniquely identify the device on which the test was performed (SofiaSerNum, the individual test (FluTestNum), and the result (ResultID). Multiple tests are stored on each device. The present Quidel COVID Test sensor concerns the positive rate in the test result.
+
+### Signal names
+- raw_pct_positive: estimates of the percentage of positive tests in total tests 
+- smoothed_pct_positive: same as in the first one, but where the estimates are formed by pooling together the last 7 days of data
+
+### Estimating percent positive test proportion
+Let n be the number of total COVID tests taken over a given time period and a given location (the test result can be negative/positive/invalid). Let x be the number of tests taken with positive results in this location over the given time period. We are interested in estimating the percentage of positive tests which is defined as:
+```
+p = 100 * x / n 
+```
+We estimate p across 3 temporal-spatial aggregation schemes:
+- daily, at the MSA (metropolitan statistical area) level;
+- daily, at the HRR (hospital referral region) level;
+- daily, at the state level.
+We are able to make these aggregations accurately because each test is reported with its 5-digit ZIP code. We do not report estimates for individual counties, as typically each county has too few tests to make the estimated value statistically meaningful.
+
+**MSA and HRR levels**: In a given MSA or HRR, suppose N flu tests are taken in a certain time period, X is the number of tests taken with positive results. If N >= 50, we simply use:
+```
+p = 100 * X / N 
+```
+If N < 50, we lend 50 - N  fake samples from its home state to shrink the estimate to the state's mean, which means:
+```
+p = 100 * [ N /50 * X/N + (50 - N)/50  * Xs /Ns ] 
+```
+where Ns, Xs are the number of flu tests and the number of flu tests taken with positive results taken in its home state in the same time period.
+
+**State level**:  the states with sample sizes smaller than a certain threshold are discarded. (The threshold is set to be 50 temporarily). For the rest of the states with big enough sample sizes,
+```
+p = 100 * X / N
+```
+
+The estimated standard error is simply:
+```
+se = 1/100 * sqrt{ p*(1-p)/N } 
+```
+where we assume for each time point, the estimates follow a binomial distribution.
+
+
+### Temporal Pooling
+Additionally, as with the Quidel COVID Test signal, we consider smoothed estimates formed by pooling data over time. That is, daily, for each location, we first pool all data available in that location over the last 7 days, and we then recompute everything described in the last two subsections. Pooling in this data makes estimates available in more geographic areas, as many areas report very few tests per day, but have enough data to report when 7 days are considered. 

quidel_covidtest/README.md

@@ -0,0 +1,61 @@
+# Quidel COVID Test Indicators
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. A template is
+included as `params.json.template`. At a minimum, you will need to include a
+password for the datadrop email account and the email address of the data sender. 
+Note that setting `export_end_date` to an empty string will export data through 
+today (GMT) minus 5 days. Setting `pull_end_date` to an empty string will pull data 
+through today (GMT).
+
+To execute the module and produce the output datasets (by default, in
+`receiving`), run the following:
+
+```
+env/bin/python -m delphi_quidel_covidtest
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_quidel_covidtest
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_quidel_covidtest --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.

quidel_covidtest/REVIEW.md

@@ -0,0 +1,39 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

quidel_covidtest/cache/.gitignore

@@ -0,0 +1 @@
+*.csv

quidel_covidtest/delphi_quidel_covidtest/__init__.py

@@ -0,0 +1,16 @@
+# -*- coding: utf-8 -*-
+"""Module to pull and clean indicators from the Quidel COVID Test.
+
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import geo_maps
+from . import data_tools
+from . import generate_sensor
+from . import export
+from . import pull
+from . import run

quidel_covidtest/delphi_quidel_covidtest/__main__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m MODULE_NAME`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover