Following the crash of a critical system at the sheriff’s office in late October, and the temporary loss of 18 months of data, the bill is now in for necessary upgrades to the county’s and sheriff’s office’s computer infrastructure: $1.1 million. Of that, $600,000 was budgeted ahead of the crash. But $500,000 was not. The county commission will have to find that money. And that’s just for this year.
“We’re going to have IT investments over the next four, five years,” County Administrator Craig Coffey told the county commission in a workshop Monday as he summed up the long-checkered history of the system and briefed them on what’s needed to bring it out of technology’s equivalent of a bygone age.
There was good news. The system known as computer-assisted dispatching, or CAD—which enables the information deputies and firefighters share and depend on while on patrol–had crashed on Oct. 29, taking down with it much of the sheriff’s office’s daily functionality and access to such things as jail records, ongoing investigation records and criminal background databases. But after sending the damaged hard drives to a forensic recovery company (and paying about $20,000), all files were recovered and, by Dec. 2, restored.
“We did lose them for that period of time but we were able to fully recover all the files,” Coffey said. “We now have a double-backup redundancy in place,” with back-ups taking place hourly, and a triple back-up system planned. But the process revealed the needs for much broader improvements.
The CAD system was installed in 2007 at a cost of $2 million to the county budget. Even then, it was not configured correctly—at least not according to New World systems, the vendor, as Coffey relayed the history to commissioners. “From the fire side all the way to the main system we never trained the maintenance folks, never trained the end users properly in order to manage the system,” he said, “so we bought this expensive system, no one really knew how to use it functionally or the way it should work, so we’ve always had some issues.”
The sheriff’s office managed it until 2014, when the county took it over and bought new hardware for over $100,000. “We’re now trying to correct all the problems that we’ve had since the beginning,” Coffey said. One of the problems that exemplify its discordance with current needs: the system was running on two servers when it should have been running on 12.
The necessary fix’s cost is to be split between the county’s budget and the sheriff’s, though in the end it all comes down to the same pot: the general revenue of county government, funded in large part by county residents’ property taxes. “Where does the sheriff get his money? Back to you guys,” Coffey told commissioners.
The part that has not been budgeted covers hardware. For example, computer network switches are a decade old. They have to be replaced. The system’s backbone servers that carry mail, store files and include backups, all need replacement. Much of the operating system is not supported anymore. Backup equipment and off-site back-up storage will have to be put in place. That batch of improvements will cost $275,000 over the next six to 12 months. The county’s IT backbone infrastructure has its own similar needs, adding another $175,000.
And more hardware is needed, including an emergency phone system at the sheriff’s office, an internal alarm system with panic buttons, a camera system for the new interview rooms at the sheriff’s office, a cable television system and other related infrastructure improvements will add more costs. That figure is undetermined. “The sheriff is paying for a lot of these things,” Coffey said, not putting a price tag on them.
Coffey was preparing commissioners to be presented with heavy IT budget requests every year for the near future. “Our telephone system is now eight years old. We can’t do paperless management, stuff like that, more online stuff,” he said, though some departments—such as the building division—may be able to tap into grants for those upgrades. Other constitutional officers such as the tax collector will likely have needs of their own for their IT infrastructure.
The costs are a reflection of years of neglect when, during and after the Great Recession, the county delayed serious capital improvements, including improvements to its computer systems. That delay is now costing the county. Spending money now will or should conversely prevent the sort of crises that took the sheriff by surprise in October.
“You’re putting out less fires,” Jarrod Shupe, the county’s new innovation technology director, said. “We’re spending so much labor hours, man hours right now, putting out fires, trying to keep stuff together, we’re paying for extra support on equipment that really isn’t supported any longer, so we have to pay a third-party vendor to provide the back-up support on pretty substantial equipment.”
The projected fix is expected to be in place by the end of next year. Next, however, comes the commission’s numbers challenge: where to get the money for the fixes, and what other projects to delay or forego. Commissioners, none of whom is more conversant with IT than the overwhelming majority of sentient beings, did not take on those issues at the end of a long workshop Monday, declining so much as another workshop on the subject.