SolarWinds Supply Chain Hit: Victims Include Cisco, Intel'Kill Switch' Effort Disrupts Some Malware, But Espionage Teams' Access May Persist
American technology giants Cisco and Intel are among the thousands of organizations that have been affected by the supply chain attack targeting software vendor SolarWinds and, by extension, its customers.
The attack campaign, which was first revealed Sunday by FireEye, one of its victims, centers on the Orion network monitoring software from SolarWinds, a technology firm based in Austin, Texas, that until recently had a valuation of about $1 billion.
While SolarWinds may be relatively unknown, the company has 300,000 customers, of which nearly 18,000 may have been caught up in the supply chain attack, which involved attackers adding a backdoor to the company's Orion software, apparently by having infiltrated its software development pipeline (see: SolarWinds: The Hunt to Figure Out Who Was Breached).
As a result, any customer that installed a Trojanized software update - present in versions introduced from March through May - would have been vulnerable to their endpoint phoning home to an attacker-controlled command-and-control server, after which attackers could have pushed further malware to the endpoint, exfiltrated data from the network and left additional tools to allow remote access. Potentially, attackers could have used such intrusions as a beachhead for attacking an organization's business partners too.
The roster of victims remains incomplete, but it already features a number of notable organizations, including FireEye and, reportedly, at least five U.S. government agencies: the Commerce, Homeland Security, State and Treasury departments, as well as the National Institutes of Health.
As security researchers unravel the malicious infrastructure used in the attack, including communications between infected endpoints and the attackers' command-and-control - or C2 - servers, they're also identifying more victims, now including technology firms, universities and government entities.
FireEye reports that a "kill switch" has now been deployed to block attackers from accessing the backdoor they added to at least some instances of the Orion network monitoring software. But the vulnerable software appears to remain widely installed, and the supply chain attack may remain active.
Individuals with knowledge of the investigation, speaking on background with news outlets, have suggested that Russia's foreign intelligence service, the SVR, may have been responsible for this apparent cyberespionage operation. The Russian government has denied those assertions.
'Kill Switch' Efforts
FireEye on Wednesday said that it had helped spearhead an effort that successfully blocked attackers' access to at least some endpoints by seizing one of the C2 domains, avsvmcloud[.]com, that attackers were using to communicate with systems running the backdoored software, which it calls SUNBURST.
Working with registrar GoDaddy and Microsoft, "we identified a kill switch that would prevent SUNBURST from continuing to operate," a FireEye spokesperson tells Information Security Media Group. "This kill switch will affect new and previous SUNBURST infections by disabling SUNBURST deployments that are still beaconing to avsvmcloud[.]com."
The malware includes a list of IP addresses with which it will not communicate. So, to use the kill switch, FireEye says any time the malware beacons to the seized avsvmcloud[.]com, one of the IP addresses on its blocklist gets returned to it. But while this disrupts attackers' access to infected systems, "this kill switch will not remove the actor from victim networks where they have established other backdoors," FireEye says. "However, it will make it more difficult for the actor to leverage the previously distributed versions of SUNBURST."
Cybersecurity blogger Brian Krebs first reported on the kill switch effort, as well as efforts by researchers to crack the obfuscated communications being used by the malware.
By decoding the #DGA domain names, we discovered nearly a hundred domains suspected to be attacked by #UNC2452 #SolarWinds, including universities, governments and high tech companies such as @Intel and @Cisco. Visit our github project to get the script.https://t.co/jsnOldynCV pic.twitter.com/40VfXuR6JI— RedDrip Team (@RedDrip7) December 16, 2020
On the heels of those efforts, Chinese cybersecurity firm RedDrip Team on Wednesday released on GitHub a decoder tool that can be used to crack a list of obscured hostnames - in other words, a partial list of victims - with which the C2 communicated.
The more than 1,700 hostnames listed include endpoints - sometimes more than one - inside such U.S. technology firms as Belkin, Cisco, Intel and Nvidia; government agencies and universities; as well as organizations such as the Hewlett Foundation.
Potential Insider Trading Questions
The value of SolarWinds' stock has dropped by 23% since FireEye issued its Sunday alert. In the days before that happened, however, and just before a leadership change inside SolarWinds, trading of the company's stock by two firms that own 70% of the company surged. The Washington Post reports that, on Dec. 7, investor Silver Lake sold $158 million of its shares in SolarWinds, while privacy equity firm Thoma Bravo sold $128 million in shares in SolarWinds on the same day. In a statement, both companies said those trades had been preplanned.
But former regulators say the timing of the trades is suspicious. “Of course, the SEC is going to look at that,” Jacob S. Frenkel, a former senior counsel in the SEC’s Division of Enforcement, tells The Washington Post. “Large trades in advance of a major announcement, then an announcement: That is a formula for an insider trading investigation.”
FBI Investigating Campaign
The U.S. government has said it is taking a coordinated approach to investigating and mitigating the campaign.
The effort is being led by the FBI, the Cybersecurity and Infrastructure Security Agency, and the Office of the Director of National Intelligence, which have formed a Cyber Unified Coordination Group with the following responsibilities:
- FBI: The bureau "is investigating and gathering intelligence in order to attribute, pursue and disrupt the responsible threat actors," as well as working with "known and suspected victims" and gathering relevant information.
- CISA: On the heels of issuing an emergency directive on Sunday that required federal civilian agencies "to immediately disconnect or power down affected SolarWinds Orion products from their network," CISA is providing on-demand technical help as well as "engaging with our public and private stakeholders across the critical infrastructure community to ensure they understand their exposure and are taking steps to identify and mitigate any compromises."
- ODNI: The office is helping to coordinate the country's intelligence agencies to support the response as well as to share information across the government.
Officials say any U.S. organizations that detect "suspicious or criminal activity" tied to the campaign should contact their local FBI field office, while any organization seeking incident response technical assistance should contact CISA or send an email to email@example.com.
But some top lawmakers have been asking why government cybersecurity officials weren't already doing more to seek out this type of supply chain hack, especially following the devastating 2017 NotPetya attack - also blamed on Russia - which had a similar MO.
“As we learned in the NotPetya attacks, software supply chain attacks of this nature can have devastating and wide-ranging effects - whether it’s via niche Ukrainian tax software or, as here, network management tools relied upon by some of the world’s largest companies," Sen. Mark Warner, D-Va., said in a statement. "As we gather more information on the impact and goals of these malign efforts, we should make clear that there will be consequences for any broader impact on private networks, critical infrastructure or other sensitive sectors.”
UK Issues Guidance
Investigations into the campaign are ongoing in many countries.
“We are continuing to investigate this incident and have produced guidance for SolarWinds’ Orion suite customers," a spokesperson for Britain's National Cyber Security Center, the public-facing arm of intelligence agency GCHQ, tells ISMG. “While it is important to note this issue has only been reported for the Orion product suite and will therefore not impact all SolarWinds customers, we strongly urge those who are affected to follow our guidance.”
For starters, NCSC recommends that all Orion software be immediately placed behind a firewall that disables all inbound and outbound internet access to the product. It also offers guidance on how customers should investigate and respond (see: SolarWinds Incident Response: 4 Essential Security Alerts).
For organizations that "are not able to deal with a suspected server compromise of this nature," the NCSC notes that it maintains a list of cyber incident response companies that can help.
Attackers Emulated SolarWinds 'Coding Style'
Evidence that whoever subverted SolarWinds' code was a sophisticated actor continues to mount. Reverse-engineering expert Itay Cohen, who works as a security researcher for Check Point Software Technologies' threat intelligence group, says the coders appear to have carefully studied SolarWinds' network topology and coding style and successfully emulated both.
"Their efforts to stay undetected are impressive," Cohen says via Twitter. "They carefully tested the feasibility of the attack by first deploying a backdoor without malicious capabilities, wrote their code with SolarWinds coding style and avoided the infection of the company's internal networks."
These checks might indicate that the attackers not only deeply learned the source code of #SolarWinds, but also learned the topology of their networks and internal development domain names to minimize the risk that a vigilant employee will notice the anomaly >> pic.twitter.com/qjGPeT33ZD— Itay Cohen (@megabeets_) December 16, 2020
"This is both a wonderful demonstration of proper cyber techniques and also the bare minimum you’d expect for a cyber operation," tweets the operational security expert known as the Grugq. "In some ways the most astounding thing is that this is considered astounding."
Cleanup May Take Years
Users and government officials are still struggling to identify all victims, and security experts say the scale of such efforts is massive. "None of this work is impossible, but the scale issues are daunting," tweets Alex Stamos, the former CSO of Facebook, who's now a professor at Stanford University's Center for International Security and Cooperation.
One challenge is that, while there may be up to 18,000 victim organizations, the number of available incident response teams is far smaller, he notes (see: Surviving a Breach: 8 Incident Response Essentials).
In addition, the backdoor added to Orion would have given attackers initial access to a network. Since then, they may have dropped additional malware, created exclusive-use administrator accounts or left other mechanisms allowing them to return.
I hope that the responders find some more high-leverage indicators of compromise, although I expect only a few hundred of those orgs have enough artifacts collected. We are going to be reaping an "iron harvest" of second-stage malware for years from this one.— Alex Stamos (@alexstamos) December 16, 2020
Stamos likens the cleanup to the so-called "iron harvest" that happens every year whenever French and Belgian farmers plough fields across which World War I battles occurred from 1914 to 1918 and turn up unexploded ordinance.
"I hope that the responders find some more high-leverage indicators of compromise, although I expect only a few hundred of those organizations have enough artifacts collected," Stamos says. "We are going to be reaping an 'iron harvest' of second-stage malware for years from this one."