It was a Friday night in February 2021. His wife called an advice nurse at their health care provider to schedule an emergency consultation for the next morning, by video because it was a Saturday and there was a pandemic going on. The nurse said to send photos so the doctor could review them in advance.
Mark’s wife grabbed her husband’s phone and texted a few high-quality close-ups of their son’s groin area to her iPhone so she could upload them to the health care provider’s messaging system. In one, Mark’s hand was visible, helping to better display the swelling. Mark and his wife gave no thought to the tech giants that made this quick capture and exchange of digital data possible, or what those giants might think of the images.
[…]
the episode left Mark with a much larger problem, one that would cost him more than a decade of contacts, emails and photos, and make him the target of a police investigation. Mark, who asked to be identified only by his first name for fear of potential reputational harm, had been caught in an algorithmic net designed to snare people exchanging child sexual abuse material.
[…]
“There could be tens, hundreds, thousands more of these,” he said.
Given the toxic nature of the accusations, Callas speculated that most people wrongfully flagged would not publicize what had happened.
“I knew that these companies were watching and that privacy is not what we would hope it to be,” Mark said. “But I haven’t done anything wrong.”
Police agreed. Google did not.
[…]
Two days after taking the photos of his son, Mark’s phone made a blooping notification noise: His account had been disabled because of “harmful content” that was “a severe violation of Google’s policies and might be illegal.” A “learn more” link led to a list of possible reasons, including “child sexual abuse and exploitation.”
Mark was confused at first but then remembered his son’s infection. “Oh, God, Google probably thinks that was child porn,” he thought.
[…]
He filled out a form requesting a review of Google’s decision, explaining his son’s infection. At the same time, he discovered the domino effect of Google’s rejection. Not only did he lose emails, contact information for friends and former colleagues, and documentation of his son’s first years of life, his Google Fi account shut down, meaning he had to get a new phone number with another carrier. Without access to his old phone number and email address, he couldn’t get the security codes he needed to sign in to other internet accounts, locking him out of much of his digital life.
[…]
A few days after Mark filed the appeal, Google responded that it would not reinstate the account, with no further explanation.
Mark didn’t know it, but Google’s review team had also flagged a video he made and the San Francisco Police Department had already started to investigate him.
[…]
Cassio was in the middle of buying a house, and signing countless digital documents, when his Gmail account was disabled. He asked his mortgage broker to switch his email address, which made the broker suspicious until Cassio’s real estate agent vouched for him.
[…]
In December, Mark received a manila envelope in the mail from the San Francisco Police Department. It contained a letter informing him that he had been investigated as well as copies of the search warrants served on Google and his internet service provider. An investigator, whose contact information was provided, had asked for everything in Mark’s Google account: his internet searches, his location history, his messages and any document, photo and video he’d stored with the company.
The search, related to “child exploitation videos,” had taken place in February, within a week of his taking the photos of his son.
Mark called the investigator, Nicholas Hillard, who said the case was closed. Hillard had tried to get in touch with Mark, but his phone number and email address hadn’t worked.
“I determined that the incident did not meet the elements of a crime and that no crime occurred,” Hillard wrote in his report. Police had access to all the information Google had on Mark and decided it did not constitute child abuse or exploitation.
Mark asked if Hillard could tell Google that he was innocent so he could get his account back.
“You have to talk to Google,” Hillard said, according to Mark. “There’s nothing I can do.”
Mark appealed his case to Google again, providing the police report, but to no avail. After getting a notice two months ago that his account was being permanently deleted, Mark spoke with a lawyer about suing Google and how much it might cost.
“I decided it was probably not worth $7,000,” he said.
[…]
False positives, when people are erroneously flagged, are inevitable given the billions of images being scanned. While most people would probably consider that trade-off worthwhile, given the benefit of identifying abused children, Klonick said companies need a “robust process” for clearing and reinstating innocent people who are mistakenly flagged.
“This would be problematic if it were just a case of content moderation and censorship,” Klonick said. “But this is doubly dangerous in that it also results in someone being reported to law enforcement.”
It could have been worse, she said, with a parent potentially losing custody of a child. “You could imagine how this might escalate,” Klonick said.
Cassio was also investigated by police. A detective from the Houston Police department called this past fall, asking him to come into the station.
After Cassio showed the detective his communications with the pediatrician, he was quickly cleared. But he, too, was unable to get his decade-old Google account back, despite being a paying user of Google’s web services.
[…]
