Texas-Sized Data Center Problem

Posted by Mark Brousseau

IBM Corp. has been given a month to fix service problems with a mammoth Texas state data center project or possibly have its $863 million contract terminated, according to an article in Friday's Austin American-Statesman.

The state notified IBM this week that the company has "breached its contractual duties and obligations to the State of Texas" and outlines specific issues with IBM's work.

"The current, unremediated situation is untenable for the State of Texas, as critical state data continues to be at risk," according to the notice written by Brian Rawson, executive director of the Department of Information Resources.

The Statesman says the most persistent problem involves the company's failure to back up data stored on agencies' servers as it consolidates 27 state agencies' data centers into two facilities in Austin and San Angelo. The objective is to streamline and modernize the agencies' technology operations while saving money.

But the company has been repeatedly warned by the state that it had fallen short on requirements in the contract and has been penalized $5.2 million, including almost $902,000 for the data backup problems. The company so far has been paid $175 million under the contract.

The notice, released Thursday, is the first required step to terminate the contract.

It is a serious move by the state, following Gov. Rick Perry's directive two weeks ago to stop work on the contract while a plan to fix the problems was developed. That plan is expected to be completed by Nov. 17.

"The governor expects the issue will be resolved to the full satisfaction of the state," said Allison Castle, the governor's spokeswoman, so that termination of the contract will not be necessary.

IBM spokesman Jeff Tieszen said the company has been working to address the data backup and recovery issues, and would respond accordingly to the state's letter.

"The state's data center infrastructure is more stable and secure now and we plan to work with (the Department of Information Resources) to continue to improve the system to better serve the state's citizens," Tieszen said in a statement.

Rawson wrote in a letter to Perry that he expects "IBM to have restored the confidence of the state leadership and the agencies upon implementation" of the governor's plan.

But Rawson said that the state must be prepared if the results are not satisfactory and said a contingency plan is being developed to ensure data center services without IBM.

IBM assumed responsibility for servers and mainframes at 1,300 locations across the state in April 2007 as it moves the work to the centralized facilities.

It must provide security, backup and disaster recovery at all of those locations. That is where many of the problems have developed.

This summer, the attorney general's office lost a significant amount of data in a server crash at its Medicaid fraud division in Tyler. IBM had failed to back up the data, as required by the contract.

Eight months of data files from the office's fraud investigations were initially lost because the office had stopped using its previous file backup software in November. Ninety percent of the data has since been recovered, but it is unclear whether all of that data is usable.