Maine

Source: https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml

The testing time series, available in the "Daily Testing Data" tab is a complete time series for PCR and antigen testing. The difference between this time series and the values we were previously using is that the previous values (the daily snapshot data table at the bottom of the page) contains both residents and non residents tested in Maine. This time series includes only residents.

Source: https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml

The testing time series, available in the "Daily Testing Data" tab is a complete time series for PCR and antigen testing. The difference between this time series and the values we were previously using is that the previous values (the daily snapshot data table at the bottom of the page) contains both residents and non residents tested in Maine. This time series includes only residents.

State: Maine

Issue: Maine started Reporting Total, Positive, and Negative Antigen tests. Backfilling today's values.

Source: State Page: https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml Screenshot: screencapture-maine-gov-dhhs-mecdc-infectious-disease-epi-airborne-coronavirus-data-shtml-2020-11-02-22_44_58.pdf

State or US: Maine

Describe the problem On 9/22, due to technical difficulties, ME did not update their testing data, so CTP carried over values for total, positive, and negative antibody and PCR tests. On 9/23, CTP recorded data as of 9/21 where we should have recorded data as of 9/22.

We need to:

• Take the testing values from 9/23 (as of 9/21) and record them for 9/22
• Record testing values in 9/23 with data as of 9/22

Link to data source https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml

State or US: ME

Describe the problem CTP started reporting probables on 7/28. ME started reporting values for probables on 5/4, and we can reasonably backfill by calculating positive (confirmed + probables) - positive PCR cases back until 5/4.

Link to data source

• CTP screenshots [https://covidtracking.com/data/state/maine/screenshots] 5/4-7/28

PROPOSED VALUES (from 5/4-9/15)

503 502 487 485 475 473 476 469 471 472 470 468 472 467 467 466 465 457 455 453 447 449 446 445 445 438 441 435 432 430 421 418 418 410 408 406 408 417 417 415 415 424 427 429 423 421 413 411 409 405 410 406 403 400 403 402 423 424 421 394 397 391 392 398 399 396 389 389 394 395 390 389 387 385 388 377 372 360 356 353 345 344 339 337 339 331 328 328 327 323 327 320 315 307 305 301 287 287 284 283 275 271 272 265 266 259 256 258 257 255 238 223 215 216 210 209 199 199 187 180 180 176 171 166 160 143 139 134 124 121 110 99 80 76 69

State: ME

Describe the problem We had the value 2903 for positive cases (PCR) on 6/3, which contradicts screenshots [https://covidtracking.com/screenshots/ME/ME-20200603-183749.png] for that day. Similarly, we had the value of 2606 for positive cases (PCR) on 6/9, which is not correct according to screenshots [https://covidtracking.com/screenshots/ME/ME-20200609-124232.png].

Link to data source

• Screenshots linked above

State or US: Maine

Describe the problem

1. ME data capture good after 4/29, with separated confirmed and confirmed+positive reporting.
2. CTP Capture of Cases (Confirmed and Probable) can be used to fill in Positive Cases (PCR) values between 3/15-4/28, as Maine has always reported confirmed cases.

Link to data source http://covid-tracking-project-data.s3-website.us-east-1.amazonaws.com/state_screenshots/ME/

ME did not update its data on PCR or tests 7/11 and provided the figures on 7/12. Historicals need to be updated https://covidtracking.com/screenshots/ME/ME-20200712-181559.png

ME began separating confirmed and probable deaths on 6/23. We were previously recording state reported deaths as Deaths (confirmed) but it is now unclear whether they were lumped (confirmed + probable) or if those numbers were only confirmed. This should be looked into further to determine whether our historical Deaths (confirmed) values should be updated.

Some notes:

• Deaths (confirmed) decreased from 6/22 (102) to 6/23 (101)
• Since there was only 1 probable death on 6/23 it is possible that it was counted as confirmed from 6/17-6/22 until it was separated out on 6/23
• I watched the beginning of ME press releases from 6/22 and 6/24 and there was no mention of the change to confirmed and probable deaths, though this may be addressed later in the video

As of 6/23, ME now provides separate confirmed and probable deaths. Deaths provided in the "Cumulative COVID-19 Cases by County" table are lumped but Confirmed and Probable can be separate by changing the "Case Status" in the "COVID-19 Case Trends" chart.

Death values are historically recorded in both the "Deaths" and "Deaths (Confirmed)" columns for ME. However, ME’s total death values represent lumped probable and confirmed figures, so they should only be recorded in the main "Deaths" field and confirmed and probably deaths should be separated beginning on 6/23.

The Maine CDC page that hosts daily data updates changed a few days ago.

The new link is: https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml

The historical data seems to be ok, but the screenshots are broken since the URL change.

History

Alexis Madrigal brought two pain points to my attention: 1. He gets a lot of questions about why our data is out-of-sync with the state's dashboards. 2. He feels publishing to history at 5PM (in retrospect) feels like a mistake. 3. He likes the idea of a midnight update but worries that it would be too hard on data entry.

A group of us (Elliot/Zach/Julia/me) looked at how we could fix this.

Elliot is close to the data entry process. He said that 5PM ET is a problem because the west coast often does not report on/at the deadline. He thinks 8PM ET (5PM PT) is sufficient to insure that we get an update for each state every day in the history table (assuming they make one) .

The next step in fixing this is for Alexis to find the right people to hash out the best publish date.

RECOMMENDATION

Our current recommendation is that we move the publish step to 8PM ET but do not change the process otherwise (keep it in the 2nd push) in the near future. This would reduce operational pressure and make the numbers a lot more consistent. There are other variations that I will add as comments for separate consideration.

We also recommend adding a field, called Source Date perhaps, to the published data that tells the consumers the 'as-of' date. It would be on a per-state per-day basis. It would clear up both the questions Alexis is getting and the questions about inconsistencies between Current and History.

My only concern about this change is if we have consumers who are news organizations with evening edition cutoffs. I will list some alternatives in the comments.

IMPLEMENTATION

1. [Now] Add a column called "Source Date (ET)" to the worksheet that tracks the timestamp of the data from the state and display that in the history table. This is very close to the "Last Updated (ET)" conceptually but we may want the flexibility to put a different date into the history from the actual date that we last looked at it.

That would enable us to be clear on the website about when each record was from. It would also enable us to leave the old date if the state fails to make an update in the day.

2. [Now] Back fill all the history records so that "Source Date (ET)" is the time of the 5PM push. Initially, we should make it be the same time (5PM) but eventually we may update it based on the data in the CHECKS tab that records the actual time of update.

3. [Now] Tell everyone that consumes is a high profile consumer of our data to that we are planning to switch to an 8 PM ET publish time in the near future and listen for objections.

4. [Next Week] Move the history out of the spreadsheet into a relational database. This is discussed as a separate issue but mentioned here because it is what will enable us to take the next steps without breaking the current process.

5. [Following week] Rebuild the history using an 8PM ET publish time using the Checks tab, the screen shots, and backups of the worksheet. Apply the other data fixes in the same process. Publish it on a 'version 2' of the API.

6. [Go Live] Data Entry switches to an 8PM ET publish time. The web site changes to use the new version of the API. If you use the old version, the only change is that from this date forward you have data from 8PM. If you use the new version, you get all the changes and fixes.