Police Are Telling ShotSpotter to Alter Evidence From Gunshot-Detecting AI

On May 31 last year, 25-year-old Safarain Herring was shot in the head and dropped off at St. Bernard Hospital in Chicago by a man named Michael Williams. He died two days later.

Chicago police eventually arrested the 64-year-old Williams and charged him with murder (Williams maintains that Herring was hit in a drive-by shooting). A key piece of evidence in the case is video surveillance footage showing Williams’ car stopped on the 6300 block of South Stony Island Avenue at 11:46 p.m.—the time and location where police say they know Herring was shot.

How did they know that’s where the shooting happened? Police said ShotSpotter, a surveillance system that uses hidden microphone sensors to detect the sound and location of gunshots, generated an alert for that time and place.

Except that’s not entirely true, according to recent court filings.

That night, 19 ShotSpotter sensors detected a percussive sound at 11:46 p.m. and determined the location to be 5700 South Lake Shore Drive—a mile away from the site where prosecutors say Williams committed the murder, according to a motion filed by Williams’ public defender. The company’s algorithms initially classified the sound as a firework. That weekend had seen widespread protests in Chicago in response to George Floyd’s murder, and some of those protesting lit fireworks.

But after the 11:46 p.m. alert came in, a ShotSpotter analyst manually overrode the algorithms and “reclassified” the sound as a gunshot. Then, months later and after “post-processing,” another ShotSpotter analyst changed the alert’s coordinates to a location on South Stony Island Drive near where Williams’ car was seen on camera.

A screenshot of the ShotSpotter alert from 11:46 PM, May 31, 2020 showing that the sound was manually reclassified from a firecracker to a gunshot.

“Through this human-involved method, the ShotSpotter output in this case was dramatically transformed from data that did not support criminal charges of any kind to data that now forms the centerpiece of the prosecution’s murder case against Mr. Williams,” the public defender wrote in the motion.

[…]

The case isn’t an anomaly, and the pattern it represents could have huge ramifications for ShotSpotter in Chicago, where the technology generates an average of 21,000 alerts each year. The technology is also currently in use in more than 100 cities.

Motherboard’s review of court documents from the Williams case and other trials in Chicago and New York State, including testimony from ShotSpotter’s favored expert witness, suggests that the company’s analysts frequently modify alerts at the request of police departments—some of which appear to be grasping for evidence that supports their narrative of events.

[…]

Untested evidence

Had the Cook County State’s Attorney’s office not withdrawn the evidence in the Williams case, it would likely have become the first time an Illinois court formally examined the science and source code behind ShotSpotter, Jonathan Manes, an attorney at the MacArthur Justice Center, told Motherboard.

“Rather than defend the evidence, [prosecutors] just ran away from it,” he said. “Right now, nobody outside of ShotSpotter has ever been able to look under the hood and audit this technology. We wouldn’t let forensic crime labs use a DNA test that hadn’t been vetted and audited.”

[…]

A pattern of alterations

In 2016, Rochester, New York, police looking for a suspicious vehicle stopped the wrong car and shot the passenger, Silvon Simmons, in the back three times. They charged him with firing first at officers.

The only evidence against Simmons came from ShotSpotter. Initially, the company’s sensors didn’t detect any gunshots, and the algorithms ruled that the sounds came from helicopter rotors. After Rochester police contacted ShotSpotter, an analyst ruled that there had been four gunshots—the number of times police fired at Simmons, missing once.

Paul Greene, ShotSpotter’s expert witness and an employee of the company, testified at Simmons’ trial that “subsequently he was asked by the Rochester Police Department to essentially search and see if there were more shots fired than ShotSpotter picked up,” according to a civil lawsuit Simmons has filed against the city and the company. Greene found a fifth shot, despite there being no physical evidence at the scene that Simmons had fired. Rochester police had also refused his multiple requests for them to test his hands and clothing for gunshot residue.

Curiously, the ShotSpotter audio files that were the only evidence of the phantom fifth shot have disappeared.

Both the company and the Rochester Police Department “lost, deleted and/or destroyed the spool and/or other information containing sounds pertaining to the officer-involved shooting,”

[…]

Greene—who has testified as a government witness in dozens of criminal trials—was involved in another altered report in Chicago, in 2018, when Ernesto Godinez, then 27, was charged with shooting a federal agent in the city.

The evidence against him included a report from ShotSpotter stating that seven shots had been fired at the scene, including five from the vicinity of a doorway where video surveillance showed Godinez to be standing and near where shell casings were later found. The video surveillance did not show any muzzle flashes from the doorway, and the shell casings could not be matched to the bullets that hit the agent, according to court records.

During the trial, Greene testified under cross-examination that the initial ShotSpotter alert only indicated two gunshots (those fired by an officer in response to the original shooting). But after Chicago police contacted ShotSpotter, Greene re-analyzed the audio files.

[…]

Prior to the trial, the judge ruled that Godinez could not contest ShotSpotter’s accuracy or Greene’s qualifications as an expert witness. Godinez has appealed the conviction, in large part due to that ruling.

“The reliability of their technology has never been challenged in court and nobody is doing anything about it,” Gal Pissetzky, Godinez’s attorney, told Motherboard. “Chicago is paying millions of dollars for their technology and then, in a way, preventing anybody from challenging it.”

The evidence

At the core of the opposition to ShotSpotter is the lack of empirical evidence that it works—in terms of both its sensor accuracy and the system’s overall effect on gun crime.

The company has not allowed any independent testing of its algorithms, and there’s evidence that the claims it makes in marketing materials about accuracy may not be entirely scientific.

Over the years, ShotSpotter’s claims about its accuracy have increased, from 80 percent accurate to 90 percent accurate to 97 percent accurate. According to Greene, those numbers aren’t actually calculated by engineers, though.

“Our guarantee was put together by our sales and marketing department, not our engineers,” Greene told a San Francisco court in 2017. “We need to give them [customers] a number … We have to tell them something. … It’s not perfect. The dot on the map is simply a starting point.”

In May, the MacArthur Justice Center analyzed ShotSpotter data and found that over a 21-month period 89 percent of the alerts the technology generated in Chicago led to no evidence of a gun crime and 86 percent of the alerts led to no evidence a crime had been committed at all.

[..]

Meanwhile, a growing body of research suggests that ShotSpotter has not led to any decrease in gun crime in cities where it’s deployed, and several customers have dropped the company, citing too many false alarms and the lack of return on investment.

[…]

a 2021 study by New York University School of Law’s Policing Project that determined that assaults (which include some gun crime) decreased by 30 percent in some districts in St. Louis County after ShotSpotter was installed. The study authors disclosed that ShotSpotter has been providing the Policing Project unrestricted funding since 2018, that ShotSpotter’s CEO sits on the Policing Project’s advisory board, and that ShotSpotter has previously compensated Policing Project researchers.

[…]

Motherboard recently obtained data demonstrating the stark racial disparity in how Chicago has deployed ShotSpotter. The sensors have been placed almost exclusively in predominantly Black and brown communities, while the white enclaves in the north and northwest of the city have no sensors at all, despite Chicago police data that shows gun crime is spread throughout the city.

Community members say they’ve seen little benefit from the technology in the form of less gun violence—the number of shootings in 2021 is on pace to be the highest in four years—or better interactions with police officers.

[…]