The computer crash that crippled major parts of the Flagler County Sheriff’s communications and records infrastructure since Oct. 29 has been only one part of a crisis that so far is without a resolution. The other has been a failure of communication between the sheriff’s office and the county administration, even though the two sides are in a co-dependent relationship for their IT systems, with the sheriff depending on the county more than the other way around.
That communications failure has led Sheriff Jim Manfre to strongly suggest that he is no longer interested in continuing joint operations with the county that began less than two years ago as a cost-saving and efficiency measure.
The original failure, it appears, may have been preventable and the loss of data was the work of an employee who was trying to make space on the server after it crashed, according to a former director of the sheriff’s system who surveyed the damage soon afterward. But relations between the she sheriff’s staff and the county’s have not mended any faster than the computer infrastructure both sides are desperately trying to fix, records show.
Summing up the situation to county commissioners at a meeting Monday evening, County Administrator Craig Coffey took some responsibility for the crash, conceded that “there are some data lost” and that “there are a lot more issues outside of CAD even that need attention,” a reference to the Computer Assisted Dispatching system that the 911 operators and deputies and firefighters in the field rely on to coordinate their operations, and which failed for more than a week after the crash. He said the administration is preparing a budget request for the commission, but hinted at no figures and gave no timetable for when the issues might be resolved.
“Did we make mistakes in back-ups? I think we did,” he said. “And that’s something that we’ve of course already corrected. It’s something we were moving towards but things didn’t get done by everyone that should have got done. We have a great IT director in Jerrod [Shupe]. He’s got a plan to lead us forward and correct a lot of problems, even a lot of stuff on the sheriff’s side, outside of CAD, it really needs some help.”
Aside from one or two clarifications Commission Chairman Frank Meeker asked for, none of the commissioners asked any questions about the issue. Meeker praised Coffey for taking some responsibility about the crash, and moved on.
But moving on is the last thing taking place behind the scenes.
Two days after critical parts of the computer system crashed, Jim Troiano, the operations director at the sheriff’s office, called Bob Urie in a panic. Urie had been the office’s IT director until February 2014, when he took a job elsewhere and the sheriff had agreed to merge his IT operations with the county’s.
“I was traveling back from vacation and told Jim that I would stop by to get ‘eyes on the server’ to see what was up,” Urie wrote in an email he sent County Administrator Craig Coffey on Saturday. “I met Brian McCoppin at [9:30 p.m.] and it didn’t take long to determine that the server had run out of hard drive space, the backups were not being done and someone had deleted a file from the server to free up space that I think was needed to bring the system back up. Brian was telling me what happened and threw a steno pad at Kevin’s desk, cussing about Kevin not doing the backups.”
Urie drafted an analytical assessment of the situation for Troiano, and on Nov. 12 took part in online discussions within the New World System’s user group, where the problem was being hashed out. “I was asked a series of questions on how to get the system back up, and every one of my suggestions were scoffed at,” Urie wrote. “I finally told Jerrod Shupe that he obviously didn’t need my help, and since then I have blocked their numbers in my phone. I was not going to put this email together until I was barraged with people calling and coming up to me saying that for some reason I was the “scapegoat” for the server crash.”
Urie objected to any intimation that he somehow had been responsible for the absence of back-ups, which he, in fact, had set up. “Let me make sure that you know that the backups of the New World server were completed on a four hour interval and the files that you are missing (file storage) were being backed up to a redundant server immediately, as they were being written to the production server during my employment.
Urie concluded: “In reality, since my departure, a thorough assessment of the system and backups should have been completed immediately, the suggestion for a walk-through of the system by me should have been taken advantage of, the free training should have been attended, and at the least, a five minute call to New World should have happened asking what to back up. My assessment of this is that their ego (County I.T.) and pride got in the way of their success, thinking that taking help from me was somehow going to make people think that they were not competent, but in all actuality, the server crash did that for them.”
In reply, and after taking a shot at FlaglerLive’s reporting of the issue (which Coffey claims “has a way of organizing things that often makes them sound more sinister than they might otherwise be”), agreed with Urie. “The backup to recover files from the server crash lies with my folks here and primarily with the individual you mentioned,” Coffey wrote, referring to Kevin Archambault, who Urie said never responded to his offer of a walk-through the system so Urie could pass on his institutional knowledge. “While some have shifted onto who is to blame and I have had to address that, my focus is still getting everything straight now to avoid any further problems.”
Coffey, of course, has not shied away from placing blame, either, as he did Monday evening, even as he claimed not to, when blaming the sheriff for shirking on licenses necessary to run part of the system: “They refused to do some maintenance on us because we were out of license for about $187,000 worth of licensing for example at the sheriff’s office for the end users, that you’re required to have access, you’re required to have a license for all that stuff,” Coffey said, explaining why new World had not done what was considered required maintenance on the system, an issue the sheriff raised with Coffey, too. “This is not about blame, and I know that’s been a focus, to me it’s about trying to get it fixed and getting it fixed right.”
Sheriff Jim Manfre has not hesitated to find and place blame. He faces a difficult re-election campaign next year and would rather not have a campaign issue served to his opponents. He wrote a critical, four-page letter to Coffey last week. Coffey responded much more briefly, summarizing the needed steps in days and weeks ahead, including a re-evaluation of the sheriff’s IT infrastructure and attempted recovery of lost data. Part of the work, he said, will require taking down the system for reboots, and contingency plans on the sheriff’s part.
Manfre, who’s never had a friendly relationship with Coffey, was not mollified. He immediately wrote back, saying his original questions were not answered, putting the responsibilities for the system squarely on the county’s end, and more significantly, showing a lack of confidence in the county’s ability to resolve the issue.
“Too many preventable failures have occurred and we need to know what is operational, what is borderline and what is critical,” Manfre wrote before citing parts of Urie’s email back to the administrator. He disputed the administrator’s claim that Shupe was keeping the sheriff’s office in the loop “on what they are going to do to determine the viability of the recovery” of lost data. (Monday evening, however, Coffey said the affected hardware is being sent to two companies in an attempt to recover the data.)
Meanwhile, the sheriff said, various external and internal investigations are being delayed because of the problem.
He ended his letter by noting his “intent to discuss the potential transition of IT services from Flagler County at some point and time in the near future,” a different way of saying that he is not as interested in continuing his joint relationship with the county.
YankeeExPat says
Could you imagine being stuck on a Life raft with these two? I would opt to go overboard and take my chances with the Sharks.
Anonymous says
“Too many preventable failures have occurred…….”
The first preventable failure occurred in November of 2012. It was all down hill from there. Manfre Out in 2016.
Wake up call says
The Sheriff is independently elected and is ultimately responsible. For Manfre to enter into an agreement with Coffey that was not first determined to be stable is a big no no. Craig Coffey has proven again and again to mislead the BOCC and others, and his true colors are again shining through. To think that the BOCC allowed him to remain on the payroll for several more years is a disgrace….there is no over site over him. This is a serious issue and the Sheriff has every reason to be concerned. Craig Coffey needs to be held accountable, but as we can see again the BOCC is allowing him to run wild. Remember all of this in the upcoming 2016 elections when the Sheriff and County Commissioners will be up for reelection. All of this should have been avoidable and those responsible need to be held accountable!
Ray Thorne says
Manfre can blame whomever he wishes but the bottom line is this….when it comes to law enforcement sensitive material,documents etc.., HE is the gate keeper. He is ultimately responsible for ensuring things like this don’t happen. HE is responsible for the keeping of records. This is pretty big folks…It’s a loss of mass proportions that will have long lasting affects.
Ray Thorne says
Failed to provide records in 2003 during first term in office.
Totally lost records in 2015 during second term in office.
Declared to run for a third term in office.
Ethics violations, Whistleblower lawsuits, Wrongful Termination lawsuits……failing to adhere to the Sheriffs Office /City of Palm Coast contract for providing law enforcement services…..enough is enough. Manfre Out in 2016
Typical says
Why do I envision 2 employees pointing the finger at each other and shrugging shoulders saying “I didn’t do it?” Everyone in an organization has a role and I’m sure at the County it’s no different. For such an intricate system I can’t believe they don’t have a person or persons solely responsible for it. They better hire someone ASAP. Someone (not a County Administrator) should’ve taken responsibility for this mess way before this was played out in the media. I’m a nurse and if a patient falls ill as a direct result of my care, bet your bottom I would be suspended or fired while the hospital gets their legal team together. Even if the result was in termination, accountability is a characteristic every employee should possess.
Jim Neuenfeldt says
Have any of you ever priced data recovery from a professional source? My guess is that the cost of the required harddrives will end up being less than 10% of what the companies attempting to recover data end up charging for their service and any cost savings by switching to the county IT will be long ago wiped off.
Yes it is the person in charge’s responsibility, and we have an elected official who the citizens elected who is in charge. While I’m certainly no fan of Mr. Manfre, I also think it is entirely possible he was sold a bill of goods by the county administration, and was likely pressured by the B.O.C.C. to utilize those services.
Bottom line now is fix it, and make sure it doesn’t happen again.
There ultimately is way to much at stake to let it just fester and continue.
Anonymous says
@ Jim Nuenfeldt,
You can’t fix what you can never get back. It’s a public records nightmare and the Sheriff is ultimately responsible (no matter who he blames)
communication is key says
Agree Agree Agree!! Only a select few in Flagler actually dedicated and direct with the communities needs!! Local Government is Not a Game!! DO SOMETHING TO SHOW YOU’RE MAKING MOVES TO HELP!!
Shamed on the BOCC says
Jim-We wouldn’t have to worry about the price of recovery software or the cost of recovery service if we had a competent County Administrator who was responsible and had an interest in protecting and preserving public records because if we did this would never have happened. Remember he fired Donnie Wines with nearly 30 years with the county just 2 years ago. This is the same administrator who gave Supervisor of Elections Weeks a hard time when she expressed concerns about her server and wanted to replace it but has freely given Supervisor Lenhart more than a $100,000 budget increase with no server upgrade request being made. Bottom line is this administrator fails us time and time again and the Board of County Commissioner’s act brain dead not seeming to care. He should be immediately fired!!!!