“Catastrophic Failure” in Sheriff’s Computer System Jeopardizing Records and Stalling Data

Last Updated: 1:47 p.m.

The Flagler County Sheriff’s Office’s computer servers, including parts of its 911 system, experienced a “catastrophic failure” on Friday, a government official familiar with the system said, with numerous records such as warrants currently difficult to access—or inaccessible—and key records dating back to February 2014 in jeopardy of being lost, including the mugshots of thousands of individuals booked through the county jail.

The failure has nothing to do with the sheriff’s offices move to its new operations center in downtown Bunnell—scene of a grand opening featuring Gov. Rick Scott and numerous local officials at 11 today.

The failure has also not affected calls for service to residents’ homes or businesses. In other words, deputies, ambulances and fire trucks are still being dispatched, but the 911 center is operating with pen and paper in many regards, and it has been unable to issue so-called run numbers to document calls for service. At the other end, cops and firefighters have contended with an extremely slow communications system. Still, one deputy described the problem as “an officer safety issue,” with deputies unable to keep track of each other’s whereabouts as they normally would through the CAD, or computer-assisted dispatching, system.

Friday afternoon the server that runs the sheriff’s office’s computer assisted dispatching and stores massive amounts of data ran out of space and crashed. It was not unforeseen. It’s the same server system that was in stalled when the Emergency Operations Center was built almost a decade ago. County officials were well aware of the needed upgrade and, in fact, budgeted and paid for it in a contract with New World Systems. But the upgrade had not yet been fully implemented.

The potentially lost data has to do with absent back-ups dating back to February 2014—roughly the time when Bob Urie, until then the director for Information Technology at the sheriff’s office, left the agency for another job in Brevard County. (Urie has consulted with the sheriff’s office since, and has been helping with the problem since Friday.) The same month, Noreen Kiesla, who’d been Urie’s deputy in the IT department, was laid off as the sheriff decided to merge the department’s IT operations with that of county government in a cost-saving measure. Kevin Archambault is now IT’s point person at the sheriff’s office.

The county’s IT went through similar convulsions earlier this year, with Donnie Wines, who’d run the county’s IT operations for years, pushed out in January. He was replaced a few months ago by Jarrod Shupe (the son of Marshall Shupe, chairman of the Flagler Beach City Commission).

And the problem revealed another issue: the sheriff’s office’s data was not being backed up off-site (as the county’s data is, for example), only internally. Shupe this afternoon said “a complete restructuring of the entire network” was under way. “It’s coming.”

(Even as the system was approaching its critical failure on Friday, the county administration had been on what some employees described as “a witch hunt” to locate the origin of an anonymous email sent last month to county commissioners, criticizing County Administrator Craig Coffey and his deputy, Sally Sherman.)

Sheriff’s office and county officials are scrambling on a temporary solution with New World—including the use of an old server that is now functioning, but at a snail’s pace, requiring deputies to wait for minutes on data that usually takes seconds to access. By this morning, Shupe said, the back-up of the old server’s data to a hard drive was completed, thus freeing resources and speeding up operations, though it’s still not clear what data was recovered: it has not been accessed.

Asked if the data is safe and recoverable, Shupe said: “That is yet to be determined.”

“The New World system is still experiencing difficulties and it is not expected to be back in full operation any time soon,” Jim Troiano, the Sheriff’s Office’s Operational Support Section director, wrote sheriff’s employees in an email this morning. “Unfortunately, estimates are between a few days to several weeks.  CAD may be up or down depending on the status of the work being done behind the scenes.

“We had a conference call today with county IT, EOC, New World and our public safety partners and are now waiting on New World to give us guidance on how to handle the issues created by the server running out of disc space.  This to include, but not limited to merging and assigning cases as well as looking up/researching cases.  Just to name a few.”

In an interview this afternoon, Troiano disputed the characterization of the problem. “We haven’t had a catastrophic failure at all,” he said. “It’s too premature to say we had a catastrophic failure, because we didn’t, and it’s premature to say we’ve lost the data.”

He added: “There’s a potential that we lost those data, a great potential, but we’re not there yet.”

When Urie left and the sheriff’s office changed his password, the office was unaware that the password was linked to back-up systems that Urie had been responsible for, which explains the lack of back-ups since, an official familiar with the system said.

Shupe confirmed that Urie’s departure revealed that some of the systems he had in place were not followed up after he left.

“This is not a New World System problem, this is an IT infrastructure problem,” the official said.

Since Friday, deputies have been filing reports, but those reports have not been able to go through the normal follow-up steps as they’ve “sat” in a temporary hamper, which itself, officials fear, has no guarantee against failure. If that server fails, the reports are lot.

New World, meanwhile, has pledged to expedite the upgrade. “This doesn’t happen overnight. It was to happen over the course of a year,” the official said.

“I ask you to be patient with the process and understand everything that can be done is being done to get the system operational,” Troiano’s email told sheriff’s staff. “If you begin to experience issues related with the New World system that you had not experienced before (especially since Thursday of last week), please contact the help desk as soon as possible.”