In 2010, the U.S. Department of Defense found thousands of its computer servers sending military network data to China—the result of code hidden in chips that handled the machines’ startup process.
In 2014, Intel Corp. discovered that an elite Chinese hacking group breached its network through a single server that downloaded malware from a supplier’s update site.
And in 2015, the Federal Bureau of Investigation warned multiple companies that Chinese operatives had concealed an extra chip loaded with backdoor code in one manufacturer’s servers.
Each of these distinct attacks had two things in common: China and Super Micro Computer Inc., a computer hardware maker in San Jose, California. They shared one other trait; U.S. spymasters discovered the manipulations but kept them largely secret as they tried to counter each one and learn more about China’s capabilities.
[…]
Around early 2010, a Pentagon security team noticed unusual behavior in Supermicro servers in its unclassified networks.
Implant in the Startup Process
The machines turned out to be loaded with unauthorized instructions directing each one to secretly copy data about itself and its network and send that information to China, according to six former senior officials who described a confidential probe of the incident. The Pentagon found the implant in thousands of servers, one official said; another described it as “ubiquitous.”
Investigators attributed the rogue code to China’s intelligence agencies, the officials said. A former senior Pentagon official said there was “no ambiguity” in that attribution.
[…]
As military experts investigated the Pentagon breach, they determined that the malicious instructions guiding the Pentagon’s servers were hidden in the machines’ basic input-output system, or BIOS, part of any computer that tells it what to do at startup.
Two people with direct knowledge said the manipulation combined two pieces of code: The first was embedded in instructions that manage the order of the startup and can’t be easily erased or updated. That code fetched additional instructions that were tucked into the BIOS chip’s unused memory, where they were unlikely to be found even by security-conscious customers. When the server was turned on, the implant would load into the machine’s main memory, where it kept sending out data periodically.
Manufacturers like Supermicro typically license most of their BIOS code from third parties. But government experts determined that part of the implant resided in code customized by workers associated with Supermicro, according to six former U.S. officials briefed on the findings.
[…]
By 2014, investigators across the U.S. government were looking for any additional forms of manipulation—anything they might have missed, as one former Pentagon official put it. Within months, working with information provided by American intelligence agencies, the FBI found another type of altered equipment: malicious chips added to Supermicro motherboards.
Warnings Delivered
Government experts regarded the use of these devices as a significant advance in China’s hardware-hacking capabilities, according to seven former American officials who were briefed about them between 2014 and 2017. The chips injected only small amounts of code into the machines, opening a door for attackers, the officials said.
Small batches of motherboards with the added chips were detected over time, and many Supermicro products didn’t include them, two of the officials said.
[…]
“The agents said it was not a one-off case; they said this was impacting thousands of servers,” Kumar said of his own discussion with FBI agents.
It remains unclear how many companies were affected by the added-chip attack. Bloomberg’s 2018 story cited one official who put the number at almost 30, but no customer has acknowledged finding malicious chips on Supermicro motherboards.
Several executives who received warnings said the information contained too few details about how to find any rogue chips. Two former senior officials said technical details were kept classified.
[…]
“This wasn’t a case of a guy stealing a board and soldering a chip on in his hotel room; it was architected onto the final device,” Quinn said, recalling details provided by Air Force officials. The chip “was blended into the trace on a multilayered board,” he said.
“The attackers knew how that board was designed so it would pass” quality assurance tests, Quinn said.
[…]
Corporate investigators uncovered yet another way that Chinese hackers were exploiting Supermicro products. In 2014, executives at Intel traced a security breach in their network to a seemingly routine firmware update downloaded from Supermicro’s website.
[…]
A contact in the U.S. intelligence community alerted the company to the breach, according to a person familiar with the matter. The tip helped Intel investigators determine that the attackers were from a state-sponsored group known as APT 17.
APT 17 specializes in complex supply-chain attacks, and it often hits multiple targets to reach its intended victims, according to cybersecurity firms including Symantec and FireEye. In 2012, the group hacked the cybersecurity firm Bit9 in order to get to defense contractors protected by Bit9’s products.
Intel’s investigators found that a Supermicro server began communicating with APT 17 shortly after receiving a firmware patch from an update website that Supermicro had set up for customers. The firmware itself hadn’t been tampered with; the malware arrived as part of a ZIP file downloaded directly from the site, according to accounts of Intel’s presentation.
[…]
Breaches involving Supermicro’s update site continued after the Intel episode, according to two consultants who participated in corporate investigations and asked not to be named.
In incidents at two non-U.S. companies, one in 2015 and the other in 2018, attackers infected a single Supermicro server through the update site, according to a person who consulted on both cases. The companies were involved in the steel industry, according to the person, who declined to identify them, citing non-disclosure agreements. The chief suspect in the intrusions was China, the person said.
In 2018, a major U.S. contract manufacturer found malicious code in a BIOS update from the Supermicro site, according to a consultant who participated in that probe. The consultant declined to share the manufacturer’s name. Bloomberg reviewed portions of a report on the investigation.
It’s unclear whether the three companies informed Supermicro about their issues with the update site, and Supermicro didn’t respond to questions about them.
