05 October 2020

COVID-19: WHY DID IT TAKE SO LONG TO SPOT THE EXCEL PROBLEM?

Wow.  England's testing statistics have been understated by over 15 thousand, mainly for tests in the North West of England, because of an IT problem.  

Positive tests have been notified to people being tested by the people carrying out the testing, but the test details notified to the central system were not loaded properly.  That caused two problems:

  1. The test results were not counted.
  2. Contact tracers were not aware of omitted tests, so contacts were not followed up.  Which is a key point of testing, in getting contacts to self-isolate promptly to reduce transmission of the virus.

I'm presuming the actual technical reason in the linked article is the correct reason for these omissions, I'm 99% certain I know why the problem was not spotted immediately. 

 

WHY IT TOOK SO LONG TO SPOT THE TESTING STATISTICS HAD BEEN UNDERSTATED

It just happens that one of the last projects I was involved in was using Excel as the reporting engine for a £100 million food service company, covering their many depots around England.  A very high volume of low value transactions.  

The issue was getting data OUT of a central database to analyse and report using Excel, as is common for financial reporting.  But this was plagued with the same basic problem.  We were using an old version of the database software that output data manually as the modern .xlsx format, but scheduled outputs were using the older .xls.  The .xls version not being suitable for the task, due to the low number of rows that format could handle.  So we had to find a different way to automate scheduled data extracts each morning.

The Test and Trace problem is about loading data INTO the central system.  But exactly the same basic problem, using .xls rather than .xlsx, and finding the .xls format would not handle enough rows.  Data just gets dropped.  I've seen it with my own eyes. 

The difference was that I had implemented control totals for the food service company, which were not there before, which immediately identified there was a problem.    For example, if total revenues for the day was £X, so the extract had to total £X too.

I've previously mentioned that I reviewed the development of Sony's main trading system "STN", running on an IBM mainframe.  Data had to be passed from one part of the system to another, under the control of the Operations team.  There were few if any control total checks for completeness.  The IT Director was grateful for me identifying this issue.  My recommendations for implementing control totals were immediately implemented.  Avoiding a similar understatement problem at Sony.

Control totals for completeness is a basic technique.  Yet this new Test and Trace problem is the third time I've seen basic completeness checks seemingly omitted.  Or were they there, but the issue not resolved?  Face. Palms.


No comments:

Post a Comment

Popular Posts