Last Updated: 1:47 p.m.
The Flagler County Sheriff’s Office’s computer servers, including parts of its 911 system, experienced a “catastrophic failure” on Friday, a government official familiar with the system said, with numerous records such as warrants currently difficult to access—or inaccessible—and key records dating back to February 2014 in jeopardy of being lost, including the mugshots of thousands of individuals booked through the county jail.
The failure has nothing to do with the sheriff’s offices move to its new operations center in downtown Bunnell—scene of a grand opening featuring Gov. Rick Scott and numerous local officials at 11 today.
The failure has also not affected calls for service to residents’ homes or businesses. In other words, deputies, ambulances and fire trucks are still being dispatched, but the 911 center is operating with pen and paper in many regards, and it has been unable to issue so-called run numbers to document calls for service. At the other end, cops and firefighters have contended with an extremely slow communications system. Still, one deputy described the problem as “an officer safety issue,” with deputies unable to keep track of each other’s whereabouts as they normally would through the CAD, or computer-assisted dispatching, system.
Friday afternoon the server that runs the sheriff’s office’s computer assisted dispatching and stores massive amounts of data ran out of space and crashed. It was not unforeseen. It’s the same server system that was in stalled when the Emergency Operations Center was built almost a decade ago. County officials were well aware of the needed upgrade and, in fact, budgeted and paid for it in a contract with New World Systems. But the upgrade had not yet been fully implemented.
The potentially lost data has to do with absent back-ups dating back to February 2014—roughly the time when Bob Urie, until then the director for Information Technology at the sheriff’s office, left the agency for another job in Brevard County. (Urie has consulted with the sheriff’s office since, and has been helping with the problem since Friday.) The same month, Noreen Kiesla, who’d been Urie’s deputy in the IT department, was laid off as the sheriff decided to merge the department’s IT operations with that of county government in a cost-saving measure. Kevin Archambault is now IT’s point person at the sheriff’s office.
The county’s IT went through similar convulsions earlier this year, with Donnie Wines, who’d run the county’s IT operations for years, pushed out in January. He was replaced a few months ago by Jarrod Shupe (the son of Marshall Shupe, chairman of the Flagler Beach City Commission).
And the problem revealed another issue: the sheriff’s office’s data was not being backed up off-site (as the county’s data is, for example), only internally. Shupe this afternoon said “a complete restructuring of the entire network” was under way. “It’s coming.”
(Even as the system was approaching its critical failure on Friday, the county administration had been on what some employees described as “a witch hunt” to locate the origin of an anonymous email sent last month to county commissioners, criticizing County Administrator Craig Coffey and his deputy, Sally Sherman.)
Sheriff’s office and county officials are scrambling on a temporary solution with New World—including the use of an old server that is now functioning, but at a snail’s pace, requiring deputies to wait for minutes on data that usually takes seconds to access. By this morning, Shupe said, the back-up of the old server’s data to a hard drive was completed, thus freeing resources and speeding up operations, though it’s still not clear what data was recovered: it has not been accessed.
Asked if the data is safe and recoverable, Shupe said: “That is yet to be determined.”
“The New World system is still experiencing difficulties and it is not expected to be back in full operation any time soon,” Jim Troiano, the Sheriff’s Office’s Operational Support Section director, wrote sheriff’s employees in an email this morning. “Unfortunately, estimates are between a few days to several weeks. CAD may be up or down depending on the status of the work being done behind the scenes.
“We had a conference call today with county IT, EOC, New World and our public safety partners and are now waiting on New World to give us guidance on how to handle the issues created by the server running out of disc space. This to include, but not limited to merging and assigning cases as well as looking up/researching cases. Just to name a few.”
In an interview this afternoon, Troiano disputed the characterization of the problem. “We haven’t had a catastrophic failure at all,” he said. “It’s too premature to say we had a catastrophic failure, because we didn’t, and it’s premature to say we’ve lost the data.”
He added: “There’s a potential that we lost those data, a great potential, but we’re not there yet.”
When Urie left and the sheriff’s office changed his password, the office was unaware that the password was linked to back-up systems that Urie had been responsible for, which explains the lack of back-ups since, an official familiar with the system said.
Shupe confirmed that Urie’s departure revealed that some of the systems he had in place were not followed up after he left.
“This is not a New World System problem, this is an IT infrastructure problem,” the official said.
Since Friday, deputies have been filing reports, but those reports have not been able to go through the normal follow-up steps as they’ve “sat” in a temporary hamper, which itself, officials fear, has no guarantee against failure. If that server fails, the reports are lot.
New World, meanwhile, has pledged to expedite the upgrade. “This doesn’t happen overnight. It was to happen over the course of a year,” the official said.
“I ask you to be patient with the process and understand everything that can be done is being done to get the system operational,” Troiano’s email told sheriff’s staff. “If you begin to experience issues related with the New World system that you had not experienced before (especially since Thursday of last week), please contact the help desk as soon as possible.”
Sheesh! How did this world ever function before computers? Back to 1975!
An this is an accredited agency?
Seriously? They knew the server was going down and did nothing? The IT people didn’t check to be sure the back ups were actually being done – whether it was in house or New World? This is absolutely shameful and such a joke on the people of Flagler County. They were too busy worrying about getting a shiny new building. There is just no excuse for this.
Epic management failure, this one is not just on Manfre, but probably more so on Sheriff candidate Rick Staly who as undersheriff was in charge of daily operations. Not only should this system had daily backups, kept off site for fire and security reasons, but also redundancy. There are not to many Law Enforcement Agencies in the last 10 years that do not build in their Record Management Systems (RMS) a redundant server for security and backup purposes. There should be a second site with radio and RMS servers in case a hurricane or other natural or manmade disaster occurs at the operational 911 center. These two systems can run simultaneously with fiber optic technology and a second command center can be established with the “flip of a switch” theoretically. Sounds like the finger pointing has begun, blame it on Urie, after all he’s not a Sheriff’s candidate, no the buck should stop with Staly/Manfre.
Somebody dropped the ball!
this is what cost cutting gets you.
This is an area where cost cutting should not be an option. How could something like this happen in 2015….. How much is it costing now Sheriff
Another manfre messup…
Oh yes… nothing to do with the move.Wonder who is witch hunting who, in reality. One IT leaves and then the problems start…very peculiar to say the least.
Sounds like yet another example of the people who knew what they were doing got fired because they liked the old sheriff.
Truth be told says
Staly ran Bob Urie off because Urie would not jump to Slaly’s every whim. Urie is and always was a true professional. The computer system failure is another proof positive of a lack of leadership. This doesn’t or hasn’t happened in any of the surrounding agencies….ever. Let’s see who Manfre finds to blame after he completely revamped the IT department himself. It was working great under Fleming.
maybe they wanted certain “records” to disappear…
a complete and thorough audit of the entire department needs to be done-from the top on down
Not Good says
And this my dear friends is what awaits the whole damn country when the Russians cut the underwater internet cables and the Chinese release an EMP over the United States……..So you had to have a computer, ah mate ?
The Truth says
This is the sad reality of IT in many organizations. IT budgets are cut and their employees are forced to work with what they have until something like this happens. Afterwards, the upper management realizes how important IT is to their daily operations. This is disappointing to see and I hope they can recover the lost data, but if they can’t it’s going to be very difficult to rebound.
Ray Thorne says
This is what happens when you fire the very people who knew what they were doing with that system. Manfre does not have any foresight. Doesn’t matter what’s good for the agency and/or citizens….he doesn’t care about future repercussions before he makes decisions.
If the FBI can resurrect emails from Hillary that has already been deleted from servers someone can find these lost records.
Ray Thorne says
much bigger issue than they’re letting on….deputies are hindered and frustrated…they simply cannot do their jobs. But keep smiling over that new building Sheriff…it looks good on the outside but its a mere shell to all that is crumbling inside.
All they needed was a copy of Norton Utilities for Windows 3.1.
Maybe they can hire a kid to come in after school to fix the servers.
(within the budget of course)
I have a Gateway 2000, 386 PC that I’d be willing to donate.
And it includes Norton Utilities.