Governor Bob McDonnell’s office says last summer’s massive computer failure that created havoc at the Department of Motor Vehicles was caused by both hardware failure and human error. The findings come in an independent audit released Tuesday. As News/Talk 960’s Evan Jones reports, Virginia plans to use the audit results to seek computer system changes — and compensation.
RICHMOND- The McDonnell Administration and the Joint Legislative Audit and Review Committee (JLARC) released today the results of the external audit of the events surrounding the August 2010 state computer disruption.
On August 25, 2010, the failure of a key hardware component owned and operated by Northrop Grumman at the state data center –and subsequent delays in restoring operations — caused a significant impact to the operation of several state agencies. These delays had an adverse effect on the citizens of the Commonwealth and resulted in unexpected expense to state agencies. All told, outages related to the disruption impacted 13% of the Commonwealth’s executive branch file servers and 26 of 89 applicable executive branch agencies, including, most visibly, the Virginia Department of Motor Vehicles.
In the wake of the disruption, Governor McDonnell ordered that immediate steps be taken. First he directed Department of Motor Vehicle offices in every region of the state to hold unprecedented weekend open hours in order to accommodate Virginians inconvenienced by the outages. Second, he called for a comprehensive external audit of the situation, to be overseen jointly by the Executive and Legislative branches, and paid for by Northrop Grumman. Working in conjunction with JLARC, the McDonnell Administration, represented by Secretary of Technology Jim Duffey, chose Agilysys, a provider of information technology services to Fortune 50, 500, and mid-tier customers, to conduct the audit.
The audits major findings are:
- The outage resulted from the failure of a key data storage system (EMC DMX-3) and subsequent human error during its repair.
- Loss of data, and the delay in restoring operations and data, resulted from the failure to follow two key industry best practices. Also, that there was an insufficient degree of self-governance towards continuous process improvement and the management of risk.
- Many components selected for the data center and its IT infrastructure meet or exceed industry best practices, but that implementation falls short in certain areas.
Governor McDonnell issued the following statement regarding the results of the audit:
“The disruption to our state computer system last August caused the Commonwealth to incur significant expenses. It impacted a broad array of state agencies, including some of the agencies our citizens most commonly interact with on a daily basis. Many Virginians were inconvenienced by this disruption and lost hours of their time in dealing with the outages. It was an unacceptable failure and one that cannot be allowed to reoccur.
In conjunction with Northrop Grumman, the Virginia Information Technologies Agency, led by Sam Nixon, worked around the clock to bring all agencies back on line and fix the situation as quickly as possible. When that was completed, we moved immediately to take every step necessary to assist impacted Virginians. We also called for a thorough external audit that would tell us exactly how this situation occurred and what after-action steps should take place to both address the recent outage and to prevent future disruptions of service.
The findings from the external audit conducted by Agilysys, and formally released today, confirm our basic understanding of the circumstances that led to the computer outage and subsequent delays in recovery and full restoration of agency operations. Further, the report that has been presented provides a path forward for the Commonwealth to recover losses from Northrop Grumman for the outage pursuant to the contract between the parties.
To this end, I have directed Secretary of Technology Jim Duffey to accelerate, and bring to a prompt conclusion, discussions with senior Northrop Grumman executives on compensation and corrective actions. We are committed to holding all state contractors accountable for the performance of their duties on behalf of the Commonwealth and its citizens.”
Here is the full statement from Northrop Grumman:
Northrop Grumman is committed to this partnership and dedicated to meeting our obligations with regard to the quality of the Commonwealth’s infrastructure and the services we deliver. We have been working hard since the August 2010 outage to make the appropriate improvements to help avoid or mitigate similar disruptions. We are ready to engage in discussions with the Commonwealth about how the report recommendations could be best implemented.
