New Google Symptoms indicator

[krivard]

Google has released a new search trends product. We should create an indicator for it, and swap out the existing GHT indicator. The new product is substantially more complex, and will require some modeling & analysis work before we are ready to actually write the indicator.

Notes from @huisaddison on his presentation to Modeling on 10 September:

• Discussion outline: https://github.com/cmu-delphi/covidcast-modeling/blob/master/google_symptoms_exploration/plans.md
• Notebooks with initial EDA and the a correlations analysis: https://github.com/cmu-delphi/covidcast-modeling/tree/master/google_symptoms_exploration
• Use these notebooks to jumpstart future correlations analysis (my helper functions allow data to be downloaded / cached and then turned into a COVIDCAST-style dataframe)
Open projects (also in the DAP Google Doc):
• Google Symptoms: Comprehensive correlations analysis for new Google Symptoms data: (“Open problems”: https://github.com/cmu-delphi/covidcast-modeling/blob/master/google_symptoms_exploration/plans.md)
• Google Symptoms: Clinically guided approach to creating a new COVID signal from Google Symptoms data: (“Two approaches”, bullet 1: https://github.com/cmu-delphi/covidcast-modeling/blob/master/google_symptoms_exploration/plans.md) combined with clinical data, e.g., Nigam’s Stanford data: https://docs.google.com/spreadsheets/d/1iZZvbv94fpZdC6XaiPosiniMOh18etSPliAXVlLLr1w/edit#gid=344371264
• Google Symptoms: Supervised learning approach to creating a new COVID signal from Google Symptoms data: (“Two approaches”, bullet 2: https://github.com/cmu-delphi/covidcast-modeling/blob/master/google_symptoms_exploration/plans.md)

[krivard]

Construct an analog of the current GHT signal and compare the results.

Evaluate the forecasting usefulness re:cases.

[nmdefries]

I created a Google Symptoms (GS)-based version of the existing GHT indicator by summing the search popularity of the topics anosmia and ageusia (loss of smell and lost of taste, respectively). I compared this new indicator (GS A+A) to the GHT indicator at the state level (the only geo region reported by both data sources).

Initial takeaways:

• GS A+A is consistently smoother than both raw and smoothed GHT, especially notable in areas with small populations.
• GS A+A and GHT are largely the same shape, although GS A+A shows a stronger signal (elevation compared to baseline in Jan + Feb 2020) in some states/time periods. More info here.
• GS A+A and GHT are positively correlated, as we'd expect, but the amount of correlation varies widely by state and over time.
• GS data redacts data where user privacy couldn't be guaranteed, indicating rare search terms and small sample sizes. However, some of these missing obs correspond to very high values in GHT, which is concerning. I'll look into this more.

Plots & more detail here.

[krivard]

Possible missingness problem discrepancy between GHT and GS is resolved: high GHT areas that were missing in GS turn out to be places with very low sample size, indicating that the high GHT value is both spurious and should be filtered out anyway.

More details in #covid-19-ght.

[nmdefries]

Detail on missingness investigation results here.

[krivard]

GS improves predictions more than both GHT and GHT+GS, which is pretty excellent.

Still todo:

• Should we include more search topics? Currently a straight duplicate

Will discuss with @huisaddison and pull in @jingjtang when ready.

[nmdefries]

Detail on predictive power of Google Symptoms (anosmia + ageusia)

I used the procedures in Ryan's symptom survey blog post to compare predictive power of a series of simple time series models (cases only, cases + GS, cases + GHT, and cases + GS + GHT) at the state level.

Findings:

• Cases + GS outperforms all other models
• Cases + GS provides an additional 5 days of lead time in forecasting compared to the cases-only model.
• Cases + GHT adds a small amount of predictive power vs cases-only model
• But cases + GS + GHT is not better than cases + GS
• Model evaluation plots

Next steps:

• Put into production dupe of GHT indicator using GS anosmia + ageusia terms only
• Investigate which additional search terms to include in GS indicator. Repeat model analysis.

[huisaddison]

Blocked by #344 which appears to be ready. What is the timeline for release? I would like to include Google Symptoms in cmu-delphi/covidcast-modeling#18 . Thanks!