Bloomberg the Company & Its Products Bloomberg Terminal Demo Request Bloomberg Anywhere Remote Login Bloomberg Customer SupportHelp

In 2010, the U.S. Department of Defense found thousandsof its computer servers sending military network data toChina—the result of code hidden in chips that handled themachines’ startup process.

In 2014, Intel Corp. discovered that an elite Chinese hackinggroup breached its network through a single server thatdownloaded malware from a supplier’s update site.

And in 2015, the Federal Bureau of Investigation warnedmultiple companies that Chinese operatives had concealed anextra chip loaded with backdoor code in one manufacturer'sservers.

Each of these distinct attacks had two things in common:China and Super Micro Computer Inc., a computer hardwaremaker in San Jose, California. They shared one other trait; U.S.spymasters discovered the manipulations but kept themlargely secret as they tried to counter each one and learnmore about China’s capabilities.

China’s exploitation of products made by Supermicro, as theU.S. company is known, has been under federal scrutiny formuch of the past decade, according to 14 former lawenforcement and intelligence officials familiar with the matter.That included an FBI counterintelligence investigation thatbegan around 2012, when agents started monitoring thecommunications of a small group of Supermicro workers,using warrants obtained under the Foreign IntelligenceSurveillance Act, or FISA, according to five of the officials.

Whether that probe continues is unknown, as is a full accountof its findings. But as recently as 2018, the FBI enlisted private-sector help in analyzing Supermicro equipment thatcontained added chips, according to an adviser to twosecurity firms that did the work.

The Supermicro saga demonstrates a widespread risk inglobal supply chains, said Jay Tabb, a former senior FBIofficial who agreed to speak generally about China’sinterference with the company’s products.

“Supermicro is the perfect illustration of how susceptibleAmerican companies are to potential nefarious tampering ofany products they choose to have manufactured in China,”said Tabb, who was the executive assistant director of theFBI’s national security branch from 2018 until he retired inJanuary 2020. “It’s an example of the worst-case scenario ifyou don’t have complete supervision over where your devicesare manufactured.”

Tabb declined to address specifics of the FBI’s probe. “TheChinese government has been doing this for a long time, andcompanies need to be aware that China is doing this,” he said.“And Silicon Valley in particular needs to quit pretending thatthis isn’t happening.”

Neither Supermicro nor any of its employees has beenaccused of wrongdoing, and former U.S. officials whoprovided information for this story emphasized that thecompany itself has not been the target of anycounterintelligence investigation.

“Supermicro is an American success story and the securityand integrity of our products is a top priority,” the companysaid.

A spokesperson for the Chinese Foreign Ministry calledaccounts of these attacks “attempts to discredit China andChinese enterprises” and accused U.S. officials of “makingthings up to hype up the ‘China threat.’”

“China has never and will never require enterprises orindividuals to collect or provide data, information andintelligence from other countries for the Chinese governmentby installing ‘back doors,’” the spokesperson said in a writtenstatement.

This story is drawn from interviews with more than 50 peoplefrom law enforcement, the military, Congress, intelligenceagencies and the private sector. Most asked not to be namedin order to share sensitive information. Some details wereconfirmed in corporate documents Bloomberg Newsreviewed.

Bloomberg Businessweek first reported on China’s meddlingwith Supermicro products in October 2018, in an article thatfocused on accounts of added malicious chips found on servermotherboards in 2015. That story said Apple Inc. andAmazon.com Inc. had discovered the chips on equipmentthey’d purchased. Supermicro, Apple and Amazon publiclycalled for a retraction. U.S. government officials also disputedthe article.

With additional reporting, it’s now clear that the Businessweekreport captured only part of a larger chain of events in whichU.S. officials first suspected, then investigated, monitored andtried to manage China’s repeated manipulation ofSupermicro’s products.

Throughout, government officials kept their findings from thegeneral public. Supermicro itself wasn’t told about the FBI’scounterintelligence investigation, according to three formerU.S. officials.

The secrecy lifted occasionally, as the bureau and othergovernment agencies warned a select group of companies andsought help from outside experts.

“In early 2018, two security companies that I advise werebriefed by the FBI’s counterintelligence division investigatingthis discovery of added malicious chips on Supermicro’smotherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These twocompanies were subsequently involved in the governmentinvestigation, where they used advanced hardware forensicson the actual tampered Supermicro boards to validate theexistence of the added malicious chips.”

Janke, whose firm has incubated startups with formermembers of the U.S. intelligence community, said the twocompanies are not allowed to speak publicly about that workbut they did share details from their analysis with him. Heagreed to discuss their findings generally to raise awarenessabout the threat of Chinese espionage within technologysupply chains.

“This is real,” Janke said, “and the government knows it.”

Supermicro, founded in 1993 by Taiwanese immigrant CharlesLiang, was built to take advantage of global supply chains.Many of its motherboards—the clusters of chips and circuitrythat run modern electronics—were manufactured in China bycontractors, then assembled into servers in the U.S. andelsewhere.

The company, which earned $3.3 billion in revenue last year,has seen its computer gear become pervasive in the cloudcomputing era. Its motherboards sit in products ranging frommedical imaging scanners to cybersecurity devices.Supermicro declined to address questions about whether itrelies on contract manufacturers in China today.

In an unusual disclosure for any public company, Supermicrotold investors in May 2019 that its own computer networkshad been breached over multiple years. “We experiencedunauthorized intrusions into our network between 2011 and2018,” the company wrote. “None of these intrusions,individually or in the aggregate, has had a material adverseeffect on our business, operations, or products.” Thecompany didn’t respond to requests for additional detailsabout those intrusions.

Federal officials had concerns about China’s dominant role inglobal electronics manufacturing before Supermicro’sproducts drew sustained U.S. government scrutiny.

Another Pentagon supplier that received attention was China’sLenovo Group Ltd. In 2008, U.S. investigators found thatmilitary units in Iraq were using Lenovo laptops in which thehardware had been altered. The discovery surfaced later inlittle-noticed testimony during a U.S. criminal case—a rarepublic description of a Chinese hardware hack.

“A large amount of Lenovo laptops were sold to the U.S.military that had a chip encrypted on the motherboard thatwould record all the data that was being inputted into thatlaptop and send it back to China,” Lee Chieffalo, whomanaged a Marine network operations center near Fallujah,Iraq, testified during that 2010 case. “That was a huge securitybreach. We don’t have any idea how much data they got, butwe had to take all those systems off the network.”

Three former U.S. officials confirmed Chieffalo’s description ofan added chip on Lenovo motherboards. The episode was awarning to the U.S. government about altered hardware, theysaid.

Lenovo was unaware of the testimony and the U.S. militaryhasn’t told the company of any security concerns about itsproducts, spokeswoman Charlotte West said in an email. U.S.officials conducted “an extensive probe into Lenovo'sbackground and trustworthiness” while reviewing its 2014acquisitions of businesses from IBM and Google, West said.Both purchases were approved.

“As there have been no reports of any problems, we have noway to assess the allegations you cite or whether securityconcerns may have been triggered by third-partyinterference,” West said.

After the discovery in 2008, the Defense Department quietlyblocked Lenovo hardware from some sensitive projects, thethree U.S. officials said, but the company was not removedfrom a list of approved vendors to the Pentagon.

In 2018, the Army and Air Force bought $2.2 million worth ofLenovo products—purchases the Pentagon’s inspector generalcriticized in a 2019 report that cited “known cybersecurityrisks.”

The Defense Department needs a better process for evaluatingtechnology purchases and imposing bans when necessary,according to the report.

Around early 2010, a Pentagon security team noticed unusualbehavior in Supermicro servers in its unclassified networks.

The machines turned out to be loaded withunauthorized instructions directing each one tosecretly copy data about itself and its network andsend that information to China, according to sixformer senior officials who described aconfidential probe of the incident. The Pentagonfound the implant in thousands of servers, oneofficial said; another described it as “ubiquitous.”

Investigators attributed the rogue code to China’sintelligence agencies, the officials said. A formersenior Pentagon official said there was “noambiguity” in that attribution.

There was no evidence that the implant siphonedany details on military operations. But theattackers did get something of value: data thatamounted to a partial map of the DefenseDepartment’s unclassified networks. Analystswere also concerned that the implant—which theattackers had taken pains to hide—might be adigital weapon that could shut down thosesystems during a conflict.

Without a fix on China’s ultimate purpose, U.S.leaders decided in 2013 to keep the discoverysecret and let the attack run, according to threeofficials who were informed of the plan. KeithAlexander, then-director of the National SecurityAgency, played a central role in the decision, theofficials said. The Pentagon devised undetectablecountermeasures to protect its networks, two of them said.

The moves allowed America’s own spies to begingathering intelligence on China’s plans withoutalerting Beijing, the two officials said.

A spokesman for Alexander referred questions tothe NSA. The agency declined to commentbeyond a one-sentence statement: “NSA cannotconfirm that this incident—or the subsequentresponse actions described—ever occurred.”

A senior White House official declined tocomment on a detailed description of the

information in this story. “We will not have a comment on thisspecific issue,” the official said in an emailed statement. “As ageneral matter, the President has made a commitment that hisadministration will conduct a wide-ranging supply chainreview on a variety of goods and sectors to identify criticalnational security risks. We’ll have more details on that reviewwhen we are ready to share.”

Other federal agencies, including the Office of the Director ofNational Intelligence, the Department of Homeland Securityand the FBI, declined to comment for this story.

A Defense Department spokeswoman said officials generallydon’t comment on investigations, intelligence matters orparticular suppliers. In response to questions about thePentagon’s 2010 investigation, one official said thegovernment has sought to safeguard its supply chain.

“When confronting adversarial effort, the Department takesmany steps to continually work to exclude products orcompanies that pose a threat to our national security,” saidEllen Lord, who served as the under secretary of defense foracquisition and sustainment before she stepped down on Jan.20. She didn't name Supermicro or any other company.

As they investigated the Pentagon’s data centers,government officials took discreet steps to try toprevent the use of Supermicro products insensitive national-security networks—even thoughthe company remained on public lists of approvedsuppliers.

Adrian Gardner, who was chief information officerfor NASA’s Goddard Space Flight Center inGreenbelt, Maryland, said he learned of theintelligence community’s concerns aboutSupermicro products before he left NASA in 2013,during a review of Goddard’s computer systems.

Gardner declined to discuss exactly what he was told orwhether NASA removed any hardware. But he said themessage was clear: “The U.S. government must use everycontrol at its disposal to ensure that it does not deployequipment from Supermicro within the system boundary ofhigh-valued assets and sensitive networks,” he said.

▲ Super Micro Computer Inc. headquarters in San Jose. Photographer: David PaulMorris/Bloomberg

▲ Jay Tabb Photographer: Chona Kasinger/Bloomberg

In response to detailed questions, Supermicro said it has“never been contacted by the U.S. government, or by any ofour customers, about these alleged investigations.” Thecompany said Bloomberg had assembled “a mishmash ofdisparate and inaccurate allegations” that “draws farfetchedconclusions.” Federal agencies, including those described inthis article as conducting investigations, still buy Supermicroproducts, the company said. And it noted that this account ofa counterintelligence investigation lacks full details, includingthe probe’s outcome or whether it’s ongoing. The fullresponse is published here.

Discover the

IMPORTANT DISCLOSURES >

‘Unauthorized Intrusions’

▲ Charles Liang in 1998. Photographer: Jim Gensheimer/The Mercury News/Getty Images

▲ Lenovo assembly line in Beijing in July 2008. Photographer: Tony Law/Redux

Pentagon Attack

IMPORTANT DISCLOSURES >

The Long Hack: How ChinaExploited a U.S. Tech

SupplierFor years, U.S. investigators found tampering in productsmade by Super Micro Computer Inc. The company says it

was never told. Neither was the public.

By Jordan Robertson and Michael Riley

February 11, 2021, 11:00 PM

Implant in the Startup Process

Former U.S. officialssay that thousands ofservers in unclassifiednetworks at theDepartment ofDefense...

...were collectingunique data about themachines and thenetworks and sendingit to...

China

...through beaconsthat occurred everyfew weeks.

U.S. investigatorsfound unauthorizedcode in part of theservers’ bootsequence...

...which theydetermined wascustomized byworkers associatedwith Super MicroComputer

▲ Keith Alexander in 2013. Photographer: AndrewHarrer/Bloomberg

▲ Ellen Lord, under secretary of defense for acquisition andsustainment, testifies during a Senate hearing on supply-chainintegrity on Oct. 1. Photographer: Tom Williams/CQ-RollCall/Getty Images

BloombergMenu Search Sign In Subscribe

Terms of Service Do Not Sell My Info (California) Trademarks Privacy Policy ©2021 Bloomberg L.P. All Rights Reserved

Careers Made in NYC Advertise Ad Choices Contact Us Help

U.S. agencies continued to purchase Supermicro products.News releases from the company show that NASA’s GoddardCenter bought some for an unclassified network devoted toclimate research in 2017. And last year, Lawrence LivermoreNational Laboratory, which does classified work on nuclearweapons, bought Supermicro equipment for unclassifiedresearch into Covid-19.

As military experts investigated the Pentagon breach, theydetermined that the malicious instructions guiding thePentagon’s servers were hidden in the machines’ basic input-output system, or BIOS, part of any computer that tells it whatto do at startup.

Two people with direct knowledge said the manipulationcombined two pieces of code: The first was embedded ininstructions that manage the order of the startup and can’t beeasily erased or updated. That code fetched additionalinstructions that were tucked into the BIOS chip’s unusedmemory, where they were unlikely to be found even bysecurity-conscious customers. When the server was turnedon, the implant would load into the machine’s main memory,where it kept sending out data periodically.

Manufacturers like Supermicro typically license most of theirBIOS code from third parties. But government expertsdetermined that part of the implant resided in codecustomized by workers associated with Supermicro, accordingto six former U.S. officials briefed on the findings.

Investigators examined the BIOS code in Defense Departmentservers made by other vendors and found no similar issues.And they discovered the same unusual code in Supermicroservers made by different factories at different times,suggesting the implant was introduced in the design phase.

Overall, the findings pointed to infiltration of Supermicro’sBIOS engineering by China’s intelligence agencies, the sixofficials said.

By 2012, the FBI had opened a counterintelligence probe, andagents in the San Francisco field office used FISA warrants tomonitor the communications of several people connected toSupermicro, according to five former U.S. officials.

Three of the officials said the FBI had evidence suggesting thatthe company had been infiltrated by people working—wittingly or unwittingly—for China. They declined to detailthat evidence.

The FISA surveillance included individuals in a position toalter the company’s technology, and didn’t focus on seniorexecutives, the officials said.

It’s not clear how long that monitoring continued. The JusticeDepartment hasn’t acknowledged the probe or announcedany charges linked to it. Counterintelligence investigationsaim to monitor and disrupt foreign intelligence operations onU.S. soil and rarely result in criminal cases.

By 2014, investigators across the U.S. government werelooking for any additional forms of manipulation—anythingthey might have missed, as one former Pentagon official putit. Within months, working with information provided byAmerican intelligence agencies, the FBI found another type ofaltered equipment: malicious chips added to Supermicromotherboards.

Government experts regarded the use of these devices as asignificant advance in China’s hardware-hacking capabilities,according to seven former American officials who werebriefed about them between 2014 and 2017. The chips injectedonly small amounts of code into the machines, opening a doorfor attackers, the officials said.

Small batches of motherboards with the added chips weredetected over time, and many Supermicro products didn’tinclude them, two of the officials said.

Alarmed by the devices’ sophistication, officialsopted to warn a small number of potential targetsin briefings that identified Supermicro by name.Executives from 10 companies and one largemunicipal utility told Bloomberg News that they’dreceived such warnings. While most executivesasked not to be named to discuss sensitivecybersecurity matters, some agreed to go on therecord.

“This was espionage on the board itself,” saidMukul Kumar, who said he received one suchwarning during an unclassified briefing in 2015when he was the chief security officer for AlteraCorp., a chip designer in San Jose. “There was achip on the board that was not supposed to bethere that was calling home—not to Supermicrobut to China.”

Altera, which was purchased by Intel in December2015, didn’t use Supermicro products, Kumarsaid, so the company determined it wasn’t at risk.

After his in-person briefing, Kumar said, helearned that peers at two other Silicon Valleysemiconductor companies had already receivedthe same FBI warning.

“The agents said it was not a one-off case; theysaid this was impacting thousands of servers,”Kumar said of his own discussion with FBI agents.

It remains unclear how many companies wereaffected by the added-chip attack. Bloomberg’s 2018 storycited one official who put the number at almost 30, but nocustomer has acknowledged finding malicious chips onSupermicro motherboards.

Several executives who received warnings said theinformation contained too few details about how to find anyrogue chips. Two former senior officials said technical detailswere kept classified.

Mike Quinn, a cybersecurity executive who served in seniorroles at Cisco Systems Inc. and Microsoft Corp., said he wasbriefed about added chips on Supermicro motherboards byofficials from the U.S. Air Force. Quinn was working for acompany that was a potential bidder for Air Force contracts,and the officials wanted to ensure that any work would notinclude Supermicro equipment, he said. Bloomberg agreednot to specify when Quinn received the briefing or identify thecompany he was working for at the time.

“This wasn’t a case of a guy stealing a board and soldering achip on in his hotel room; it was architected onto the finaldevice,” Quinn said, recalling details provided by Air Forceofficials. The chip “was blended into the trace on amultilayered board,” he said.

“The attackers knew how that board was designed so it wouldpass” quality assurance tests, Quinn said.

An Air Force spokesman said in an email that Supermicroequipment hasn’t been excluded from USAF contracts underany public legal authority. In general, he said, the DefenseDepartment has non-public options for managing supply-chain risks in contracts for national security systems.

In its written response to questions, Supermicro said that nocustomer or government agency has ever informed thecompany about the discovery of malicious chips in itsequipment. It also said it has “never found any maliciouschips, even after engaging a third-party security firm toconduct an independent investigation on our products.” Thecompany didn’t respond to a question about who chose thesamples that were investigated.

After Bloomberg reported on the added-chip threat in October2018, officials for the U.S. Department of Homeland Security,the FBI, the Office of the Director of National Intelligence andthe NSA made public statements either discounting thereport’s validity or saying they had no knowledge of the attackas described. The NSA said at the time it was “befuddled” byBloomberg’s report and was unable to corroborate it; theagency said last month that it stands by those comments.

Alerts about added chips weren’t limited to the private sector.Former chief information officers at four U.S. agencies toldBloomberg they took part in briefings delivered by theDefense Department between 2015 and 2017 about addedchips on Supermicro motherboards.

And the FBI was examining samples of manipulatedSupermicro motherboards as recently as 2018, according toJanke, the adviser to two companies that assisted with theanalysis.

Darren Mott, who oversaw counterintelligence investigationsin the bureau’s Huntsville, Alabama, satellite office, said awell-placed FBI colleague described key details about theadded chips for him in October 2018.

“What I was told was there was an additional little componenton the Supermicro motherboards that was not supposed to bethere,” said Mott, who has since retired. He emphasized thatthe information was shared in an unclassified setting. “TheFBI knew the activity was being conducted by China, knew itwas concerning, and alerted certain entities about it.”

Mott said that at the time, he advised companies that hadasked him about the chips to take the issue seriously.

Corporate investigators uncovered yet another way thatChinese hackers were exploiting Supermicro products. In2014, executives at Intel traced a security breach in theirnetwork to a seemingly routine firmware update downloadedfrom Supermicro’s website.

Intel security executives concluded that an elite Chinesehacking group perpetrated the attack, according to aslideshow they presented to a gathering of tech industry peersin 2015. Two participants agreed to share details of thepresentation.

In response to questions about the incident, anIntel spokeswoman said it was caught early andcaused no data loss.

“In 2014, Intel IT identified and quickly addressedan issue found in non-Intel software on twosystems in a contained part of our network,”spokeswoman Tara Smith said. “There was noimpact to our network or data.” She declined toelaborate.

Intel’s presentation focused on the identity of theattackers and their use of a trusted supplier’supdate site, according to people who saw theslideshow. A contact in the U.S. intelligencecommunity alerted the company to the breach,according to a person familiar with the matter.The tip helped Intel investigators determine thatthe attackers were from a state-sponsored groupknown as APT 17.

APT 17 specializes in complex supply-chainattacks, and it often hits multiple targets to reachits intended victims, according to cybersecurityfirms including Symantec and FireEye. In 2012,the group hacked the cybersecurity firm Bit9 inorder to get to defense contractors protected byBit9’s products.

Intel’s investigators found that a Supermicro server begancommunicating with APT 17 shortly after receiving a firmwarepatch from an update website that Supermicro had set up forcustomers. The firmware itself hadn’t been tampered with;the malware arrived as part of a ZIP file downloaded directlyfrom the site, according to accounts of Intel’s presentation.

This delivery mechanism is similar to the one used in therecent SolarWinds hack, in which Russians allegedly targetedgovernment agencies and private companies through softwareupdates. But there was a key difference: In Intel’s case, themalware initially turned up in just one of the firm’s thousandsof servers—and then in just one other a few months later.Intel’s investigators concluded that the attackers could targetspecific machines, making detection much less likely. Bycontrast, malicious code went to as many as 18,000SolarWinds users.

Intel executives told Supermicro about the attack shortly afterit occurred, according to descriptions of the company’spresentation.

Supermicro didn't respond to detailed questions about theincident, but said: “Intel raised a question we were not able toverify, but out of an abundance of caution, we promptly tooksteps to address.” The two companies continue to doextensive amounts of business with each other.

Breaches involving Supermicro’s update site continued afterthe Intel episode, according to two consultants whoparticipated in corporate investigations and asked not to benamed.

In incidents at two non-U.S. companies, one in 2015 and theother in 2018, attackers infected a single Supermicro serverthrough the update site, according to a person who consultedon both cases. The companies were involved in the steelindustry, according to the person, who declined to identifythem, citing non-disclosure agreements. The chief suspect inthe intrusions was China, the person said.

In 2018, a major U.S. contract manufacturer found maliciouscode in a BIOS update from the Supermicro site, according toa consultant who participated in that probe. The consultantdeclined to share the manufacturer’s name. Bloombergreviewed portions of a report on the investigation.

It’s unclear whether the three companies informedSupermicro about their issues with the update site, andSupermicro didn’t respond to questions about them.

Today, with the SolarWinds hack still under investigation,national-security concerns about the technology supply chainhave erupted into U.S. politics. American officials are callingfor stricter supply-chain policing and cajoling manufacturersto ensure their code and hardware are safe.

“Supermicro’s tale of woe is a chilling wake-up call for theindustry,” said Frank Figliuzzi, who was the FBI’s assistantdirector for counterintelligence until 2012. Figliuzzi declinedto address specifics, but agreed to speak publicly about theimplications of Supermicro’s history with Chinese tampering.

“If you think this story has been about only one company,you’re missing the point,” he said. “This is a ‘don’t let thishappen to you’ moment for anyone in the tech sector supplychain.”

—With assistance from Jennifer Jacobs

More On Bloomberg

Oonee Wants to Fix New YorkBike Parking with Free, SecureStorage Pods

MacKenzie Scott’s RemarkableGiveaway Is Transforming theBezos Fortune

Bankruptcy Claimed Their Jobs,and Now They’re Out forPayback

China Needs to Hit Peak Oil LongBefore It Reaches Net-ZeroEmissions

Customized Code

Warnings Delivered

Altered Updates

▲ Frank Figliuzzi Photographer: Cheney Orr/Bloomberg

Editors: John Voskuhl, Robert Blau and Otis BilodeauGraphics: Christopher Cannon

"

FEB. 11, 2021 FEB. 11, 2021 FEB. 10, 2021 FEB. 9, 2021

Added Chips With Malicious Code

Former U.S. officialsand corporate securityexecutives say...

Chinese operativestampered with SuperMicro Computer’sproducts...

...by placing addedchips loaded withmalicious code on anunknown number ofmotherboards.

Once installed inservers and shippedto Supermicrocustomers, theadded chips...

...sent beacons backto China and...

...created an openingfor malware to beinstalled undetected.

Malware Sent With an Update

A corporateinvestigation foundthat a Chinesehacking group...

...tampered with thesoftware and firmwareupdate portal ofSuper MicroComputer.

Malicious code wasbundled withlegitimate firmwareand downloaded to...

...precisely targetedservers.

The malware openeda back door...

...and begancommunicating withthe attackers.

Your monthly limit of free content is about to expire. Stay on top of historic market volatility. Try 3 months for$8.75 $0.50 per week. Cancel anytime.

Already a subscriber or Bloomberg Anywhere

client? Sign In3