Sheriff Rethinks Joint IT Operations With County Amid Discord Over Computer Failure

The computer crash that crippled major parts of the Flagler County Sheriff’s communications and records infrastructure since Oct. 29 has been only one part of a crisis that so far is without a resolution. The other has been a failure of communication between the sheriff’s office and the county administration, even though the two sides are in a co-dependent relationship for their IT systems, with the sheriff depending on the county more than the other way around.

That communications failure has led Sheriff Jim Manfre to strongly suggest that he is no longer interested in continuing joint operations with the county that began less than two years ago as a cost-saving and efficiency measure.

The original failure, it appears, may have been preventable and the loss of data was the work of an employee who was trying to make space on the server after it crashed, according to a former director of the sheriff’s system who surveyed the damage soon afterward. But relations between the she sheriff’s staff and the county’s have not mended any faster than the computer infrastructure both sides are desperately trying to fix, records show.

Summing up the situation to county commissioners at a meeting Monday evening, County Administrator Craig Coffey took some responsibility for the crash, conceded that “there are some data lost” and that “there are a lot more issues outside of CAD even that need attention,” a reference to the Computer Assisted Dispatching system that the 911 operators and deputies and firefighters in the field rely on to coordinate their operations, and which failed for more than a week after the crash. He said the administration is preparing a budget request for the commission, but hinted at no figures and gave no timetable for when the issues might be resolved.

“Did we make mistakes in back-ups? I think we did,” he said. “And that’s something that we’ve of course already corrected. It’s something we were moving towards but things didn’t get done by everyone that should have got done. We have a great IT director in Jerrod [Shupe]. He’s got a plan to lead us forward and correct a lot of problems, even a lot of stuff on the sheriff’s side, outside of CAD, it really needs some help.”

Aside from one or two clarifications Commission Chairman Frank Meeker asked for, none of the commissioners asked any questions about the issue. Meeker praised Coffey for taking some responsibility about the crash, and moved on.

But moving on is the last thing taking place behind the scenes.

Two days after critical parts of the computer system crashed, Jim Troiano, the operations director at the sheriff’s office, called Bob Urie in a panic. Urie had been the office’s IT director until February 2014, when he took a job elsewhere and the sheriff had agreed to merge his IT operations with the county’s.

“I was traveling back from vacation and told Jim that I would stop by to get ‘eyes on the server’ to see what was up,” Urie wrote in an email he sent County Administrator Craig Coffey on Saturday. “I met Brian McCoppin at [9:30 p.m.] and it didn’t take long to determine that the server had run out of hard drive space, the backups were not being done and someone had deleted a file from the server to free up space that I think was needed to bring the system back up. Brian was telling me what happened and threw a steno pad at Kevin’s desk, cussing about Kevin not doing the backups.”

Urie drafted an analytical assessment of the situation for Troiano, and on Nov. 12 took part in online discussions within the New World System’s user group, where the problem was being hashed out. “I was asked a series of questions on how to get the system back up, and every one of my suggestions were scoffed at,” Urie wrote. “I finally told Jerrod Shupe that he obviously didn’t need my help, and since then I have blocked their numbers in my phone. I was not going to put this email together until I was barraged with people calling and coming up to me saying that for some reason I was the “scapegoat” for the server crash.”

Urie objected to any intimation that he somehow had been responsible for the absence of back-ups, which he, in fact, had set up. “Let me make sure that you know that the backups of the New World server were completed on a four hour interval and the files that you are missing (file storage) were being backed up to a redundant server immediately, as they were being written to the production server during my employment.

Urie concluded: “In reality, since my departure, a thorough assessment of the system and backups should have been completed immediately, the suggestion for a walk-through of the system by me should have been taken advantage of, the free training should have been attended, and at the least, a five minute call to New World should have happened asking what to back up. My assessment of this is that their ego (County I.T.) and pride got in the way of their success, thinking that taking help from me was somehow going to make people think that they were not competent, but in all actuality, the server crash did that for them.”

In reply, and after taking a shot at FlaglerLive’s reporting of the issue (which Coffey claims “has a way of organizing things that often makes them sound more sinister than they might otherwise be”), agreed with Urie. “The backup to recover files from the server crash lies with my folks here and primarily with the individual you mentioned,” Coffey wrote, referring to Kevin Archambault, who Urie said never responded to his offer of a walk-through the system so Urie could pass on his institutional knowledge. “While some have shifted onto who is to blame and I have had to address that, my focus is still getting everything straight now to avoid any further problems.”

Coffey, of course, has not shied away from placing blame, either, as he did Monday evening, even as he claimed not to, when blaming the sheriff for shirking on licenses necessary to run part of the system: “They refused to do some maintenance on us because we were out of license for about $187,000 worth of licensing for example at the sheriff’s office for the end users, that you’re required to have access, you’re required to have a license for all that stuff,” Coffey said, explaining why new World had not done what was considered required maintenance on the system, an issue the sheriff raised with Coffey, too. “This is not about blame, and I know that’s been a focus, to me it’s about trying to get it fixed and getting it fixed right.”

Sheriff Jim Manfre has not hesitated to find and place blame. He faces a difficult re-election campaign next year and would rather not have a campaign issue served to his opponents. He wrote a critical, four-page letter to Coffey last week. Coffey responded much more briefly, summarizing the needed steps in days and weeks ahead, including a re-evaluation of the sheriff’s IT infrastructure and attempted recovery of lost data. Part of the work, he said, will require taking down the system for reboots, and contingency plans on the sheriff’s part.

Manfre, who’s never had a friendly relationship with Coffey, was not mollified. He immediately wrote back, saying his original questions were not answered, putting the responsibilities for the system squarely on the county’s end, and more significantly, showing a lack of confidence in the county’s ability to resolve the issue.

“Too many preventable failures have occurred and we need to know what is operational, what is borderline and what is critical,” Manfre wrote before citing parts of Urie’s email back to the administrator. He disputed the administrator’s claim that Shupe was keeping the sheriff’s office in the loop “on what they are going to do to determine the viability of the recovery” of lost data. (Monday evening, however, Coffey said the affected hardware is being sent to two companies in an attempt to recover the data.)

Meanwhile, the sheriff said, various external and internal investigations are being delayed because of the problem.

He ended his letter by noting his “intent to discuss the potential transition of IT services from Flagler County at some point and time in the near future,” a different way of saying that he is not as interested in continuing his joint relationship with the county.