Nevada

• [X] make GH issue
• [X] double check the metrics aren't being reported somewhere by the state
• [X] add public note
• [x] remove carried over values
• [x] add chart annotation
• [x] null out source notes
• [x] null out WS2 values
• [x] revisit public note

Values removed: Changes.txt

NV remove recoveries.txt

We backfilled our Total Test Encounters (PCR) for Nevada using the data behind the dashboard above. Attached are the analysis and raw data sheets used during this backfill.

NV - Analysis PCR Encounters 20201029.csv.txt NV - Sheet12.csv.txt

This issue is complicated by the fact that we do not have a screenshot or reliably historical source for Total PCR Tests (People), normally found on p.3 of the dashboard

Since we do not have a reliable source from which to backfill this value, we are carrying over the testing value from 8/14 (538202).

BEFORE:

AFTER:

There are a number of discrepancies but they all seem to be human error, e.g. transposing digits and such. Based on screenshots, the error seems to have been with the state and not with our data entry team. Since these were mistake corrections by the state instead of continuously revised data, the backfill team agreed we can pull them in.

In addition to the chart Daniel screenshotted, on the previous page the dashboard has data for total tests that similarly matches our data for total tests (PCR) except with a few corrections and going back into March.

The result is three updated columns: positive cases (PCR), positives, and total tests (PCR). (Our positives for NV are identical to positive cases.) Our positive cases (PCR) now extends back to 3/5 as a result of this backfill. Total tests (PCR) now extends back to 3/22 (formerly it ended at 4/29).

Spreadsheet with analysis and before/after data: https://docs.google.com/spreadsheets/d/1QLtUsLbO0HZ4xQQJlVGzf9nbmc31QVy9lqp1sNaZs7s/edit#gid=0

The 23:59 timestamp was found to be relevant through 7/15. States Daily was changed accordingly to account for current timestamp practices.

Before:

After:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

This issue has been closed because it was stale for 15 days, and there was no further activity on it for 10 days. You can feel free to re-open it if the issue is important, and label it as "not stale."

Confirmed they were identical to "Deaths", then removed "Deaths (confirmed)" from today back through 5/12.

Updated NV's source note for Deaths (confirmed) to Not Provided and explained in NV private notes

DC'ed by JJA 6/26 9:44:00 a.m. ET

This looks like a screenshot from the dashboard (page5). At least there's no need to copy numbers from PDFs. Here's the CSV version, that I took from the dashboard nv_deaths_cumsum.zip

Nevada looks like they revise historicals to account for reporting lags and such; the curves for cases and deaths on their dashboard (you can right click and view as table) don't match ours until the current day. (edit: the total tests curve on the other hand does match ours, just offset by one day)

Small adjustment to what I said: they only revise historical deaths.

Cases and total tests are mostly offset by one day because of when we update. There are some small differences in cases, especially back in March, but I don't think correcting minor errors is worth the "bubble" (one day with double cases) it will introduce, because it corrects the off-by-one effect for the data that exists only to have future data entry shifts reintroduce it.

So: I've updated deaths all the way back and the confirmed deaths column. These were fairly off from what we had because of the revisions. For example, in between 19 June and today, they added a new death back in early May! Data and analysis here: https://docs.google.com/spreadsheets/d/10jJ5Fq9M6CamVAELZK2dfendTMYfJy5bEZMFdEsuBf0/edit#gid=0

Hey Michal, Thank you for writing. We've updated the data for cumulative positive cases for March 6, 2020.

BEFORE:

AFTER:

We have also updated the data for cumulative deaths for June 1 and June 2, 2020 as reported by the Nevada Health Response team on June 4. BEFORE:

AFTER:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

This issue has been closed because it was stale for 15 days, and there was no further activity on it for 10 days. You can feel free to re-open it if the issue is important, and label it as "not stale."

Uploading a spreadsheet of all decreases (more than documented previously), current as of today: covidtracking_problemdates.xlsx

States affected: AK, AL, AZ, CO, DC, DE, FL, HI, IA, KS, MA, MD, MI, NJ, NM, NV, NY, OH, OK, PR, RI, SC, WI

This is my R code to calculate new cases and pull records that decreased from the prior day (any variable) AND the prior day's row for comparison:

library(tidyverse)
covidtracking %>% 
    arrange(state, date) %>% 
    group_by(state) %>% 
    mutate_at(vars(c("positive", "death", "total")), 
              list(new = ~ coalesce(. - lag(.), .))) %>%
    filter_at(vars(ends_with("new")), any_vars(. < 0 | lead(.) < 0)) %>% 
    ungroup()

I wonder if the problem is an error in the data source in which corrections for prior day results are included as adjustments on the day the error was discovered. This is common in the banking world because of the value of keeping past transactions immutable. It's poor practice for scientific data, however, because the test counts on a given day matter.

If the problem is deferred adjustment, the idea solution is to inform the data sources and ask for better quality data. Short of that, an corrective approach is to reverse the error to the best extent that the data allows: Where a daily result is negative, set that day to zero, and decrease the count of the previous day by the corresponding amount.

Such a correction would not be perfect since (a) you don't know for sure whether the error was from the previous day versus earlier and (b) it doesn't correct any of the cases where adjustments didn't cause negative result. Still, it leads to better data quality than making no correction and avoids the confusion of negative daily counts.

Hello, and thank you for helping us clean our data. Please see the following:

New York:

• [ ] 3/7 to 3/8 the total changed due to pending tests no longer being reported.
• [ ] 3/10 to 3/11 should be correct

Oklahoma

• [ ] 3/20 to 3/21 the variance in totals is tied to the unreliable Pending category. It may be that the data point was phased out and our data was affected by this transition;

Ohio

• [ ] 3/16 to 3/17 variation in data is due to Pending data no longer being published.

New Jersey

• [ ] 3/16 is infact incorrect:
• [ ] corrected

Hawaii

• [ ] Pending for 3/19 decrease per the state

Michigan stopped reporting pending data as of 3/17

Kansas stopped reporting pending data as of 3/11

Iowa stopped reporting pending data as of 3/14 Deleware stopped reporting pending data as of 3/17 DC has fluctuating pending data

Thank you for these clarifications, @careeningspace, and for working on providing these data so accessibly! In terms of cleaning up these records for use:

1. Has the NJ correction been applied to the live data?
2. Given the inconsistencies with pending data, do you foresee any problems with subtracting pending cases from the totals?

The NJ correction should be in the live feed. Going forward, our API will no longer be focusing on including Pending in our "Total". You can find more detail on our API Page

• [ ] totalTestResults - Calculated value (positive + negative) of total test results.
• [ ] total - DEPRECATED Will be removed in the future. (positive + negative + pending). Pending has been an unstable value and should not count in any totals.

As for subtracting historical "Pending" data - if you want a clean Total, you can sum Positive and Negative.

Great, thanks. That fixes problems in many states and leaves only AK, DC, HI, ID, KY, MI, NV, and SC with negative "increases": covidtracking_problemdates.xlsx

covidtracking %>% arrange(state, date) %>% group_by(state) %>% filter_at(vars(ends_with("Increase")), any_vars(. < 0 | lead(.) < 0)) %>% ungroup()

-[ ] Alaska has a period of data flux that needs more research 3/17 - 3/19: Data Log:

Daily Report:

State Data from 3/17 14:09 ET:

State Data from 3/17 18:00 ET:

State Data from 3/18:

Updated Daily after correction:

District of Columbia 3/10 - 3/11

• [ ] We do not have screen grabs from this time period
• [ ] It looks like DC changed how they were reporting data. I am going to make both days match

DC Before update:

DC after update:

• [ ] ID has an issue with a change in data from 3/18 to 3/19

• [ ] Screen cap of State Data from 3/18 14:04:

• [ ] Screencap of State Data from 3/19 14:04:

• [ ] The positives increase, while the total tests reported did not. Our methodology is to leave the negatives unchanged in this case. Fixed data below:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

This issue has been closed because it was stale for 15 days, and there was no further activity on it for 10 days. You can feel free to re-open it if the issue is important, and label it as "not stale."