The City of Dallas says the recent loss of police department data was the result of poor management, supervision and an attempt to save money, among other factors.
A report by the Department of Information and Technology Services (ITS) detailed the causes that immediately led to the deletion of millions of files. It also addressed underlying systemic factors at play.
In March, a now-fired staffer at ITS deleted 22 terabytes (TB) of data. The city, with help from Microsoft, recovered 14.49 TBs, but deemed 7.51 TBs “unrecoverable.” The data included photos, videos, audio, notes and other evidence collected for police department cases.
Then, in a subsequent audit, the Information and Technology Services department found an additional 13.167 TBs of data had been lost in separate incidents.
An attempt to recover that data is still ongoing.
Best practices
The original data had been stored in a cloud service run by a vendor. According to the report, the data losses happened when the technician migrated files from the cloud to an on-site archive with the city. One of the city’s vendors, Commvault, told investigators the technician did not follow the company’s best practices in performing the migration.
Furthermore, the report added that three ITS managers had reviewed the employee’s request to migrate files, and those supervisors “either did not understand the actions to be performed; the potential risk of failure; or negligently reviewed the Change Request” before allowing the technician to proceed.
The report cites the National Institute of Standards and Technology in stating that moving large amounts of data is particularly risky.
Disaster recovery measures in place were “inadequate … for a large volume migration,” the report says.
Cost control
The report stated that ITS had been exceeding its monthly budget for cloud computing, and the department migrated the data off the cloud in order to reduce costs.
“It was also communicated that as a result of the spend analysis ITS leadership’s decision was to return certain workloads back to a more cost controlled on-premises storage model even though adequate risk assessments had not been performed,” the report said.
A Dallas city spokesperson said no one at the city was available to respond to the report.
“The report speaks for itself,” she said.
Recommendations
More broadly, the report criticized the city for not having enough policies in place for managing its data.
It recommended many steps toward developing stricter protocols and becoming more concrete with vendors about the scope of work.
The report also said in order for leadership at ITS to have better oversight, it “must set the appropriate environmental tone that actions have repercussions.”
The lost files could affect thousands of ongoing cases, including 1,000 cases that the Dallas County District Attorney’s office has prioritized. The “majority” of the unrecoverable 7.51 TB of data affected the Family Violence Unit, said the report.
“The District Attorney’s office and DPD are continually providing ITS list of priority cases to search against for the recovery effort,” the report stated.
Dallas Mayor Eric Johnson said he wants the city council's ad hoc committee on General Investigating and Ethics to get a detailed briefing on the report later this month.
"All of it raises further questions," Johnson said in a statement. "We need action and accountability."
This story was updated since first published to include comments from Dallas Mayor Eric Johnson.
KERA News is made possible through the generosity of our members. If you find this reporting valuable, consider making a tax-deductible gifttoday. Thank you.
Got a tip? Email Bret Jaspers at bjaspers@kera.org. You can follow Bret on Twitter @bretjaspers.
